Question: A sliding technique based on conditional gradients for solving convex optimization problems.

This document explores the approximation capabilities of a specific category of artificial neural networks, demonstrating that feedforward networks equipped with a single layer of sigmoidal nonlinearities can attain an integrated squared error of O(1/n), where n signifies the quantity of nodes. This finding is contingent on the assumption that the function being approximated exhibits a bounded first moment for the magnitude distribution of its Fourier transform. Both the nonlinear parameters linked to the sigmoidal nodes and the linear combination coefficients are optimized during the approximation process. Conversely, for series expansions featuring n terms, where solely the linear combination parameters are fine-tuned, the integrated squared approximation error cannot be reduced to less than O(1/n^(2/d)) for functions adhering to the identical smoothness criteria, with d denoting the dimension of the input to the function. For the examined class of functions, the approximation accuracy and the efficiency of the parameterization within the networks exhibit a remarkable advantage, particularly in high-dimensional contexts. This suggests that for certain high-dimensional problems, neural networks with sigmoidal functions can offer superior performance in terms of approximation quality and parameter efficiency compared to traditional series expansions, even when the functions are relatively smooth.

In this paper, we consider the problem of federated (or decentralized) learning using ADMM with multiple agents. We consider a scenario where a certain fraction of agents (referred to as Byzantines) provide falsified data to the system. In this context, we study the convergence behavior of the decentralized ADMM algorithm. We show that ADMM converges linearly to a neighborhood of the solution to the problem under certain conditions. We next provide guidelines for network structure design to achieve faster convergence. Next, we provide necessary conditions on the falsified updates for exact convergence to the true solution. To tackle the data falsification problem, we propose a robust variant of ADMM. We also provide simulation results to validate the analysis and show the resilience of the proposed algorithm to Byzantines.

Kernel method has been developed as one of the standard approaches for nonlinear learning, which however, does not scale to large data set due to its quadratic complexity in the number of samples. A number of kernel approximation methods have thus been proposed in the recent years, among which the random features method gains much popularity due to its simplicity and direct reduction of nonlinear problem to a linear one. Different random feature functions have since been proposed to approximate a variety of kernel functions. Among them the Random Binning (RB) feature, proposed in the first random-feature paper [21], has drawn much less attention than the Random Fourier (RF) feature proposed also in [21]. In this work, we observe that the RB features, with right choice of optimization solver, could be orders-of-magnitude more efficient than other random features and kernel approximation methods under the same requirement of accuracy. We thus propose the first analysis of RB from the perspective of optimization, which by interpreting RB as a Randomized Block Coordinate Descent in the infinite-dimensional space, gives a faster convergence rate compared to that of other random features. In particular, we show that by drawing R random grids with at least κ number of non-empty bins per grid in expectation, RB method achieves a convergence rate of O(1/κ R)), which not only sharpens its O(1/√R) rate from Monte Carlo analysis, but also shows a κ times speedup over other random features under the same analysis framework. In addition, we demonstrate another advantage of RB in the L1-regularized setting, where unlike other random features, a RB-based Coordinate Descent solver can be parallelized with guaranteed speedup proportional to κ. Our extensive experiments demonstrate the superior performance of the RB features over other random features and kernel approximation methods.

Chapter 1 strongly advocates the stochastic back-propagation method to train neural networks. This is in fact an instance of a more general technique called stochastic gradient descent (SGD). This chapter provides background material, explains why SGD is a good learning algorithm when the training set is large, and provides useful recommendations.

A novel algorithm is presented to compute the convex hull of a point set in ℝ3 using the graphics processing unit (GPU). By exploiting the relationship between the Voronoi diagram and the convex hull, the algorithm derives the approximation of the convex hull from the former. The other extreme vertices of the convex hull are then found by using a two-round checking in the digital and the continuous space successively. The algorithm does not need explicit locking or any other concurrency control mechanism, thus it can maximize the parallelism available on the modern GPU.
 The implementation using the CUDA programming model on NVIDIA GPUs is exact and efficient. The experiments show that it is up to an order of magnitude faster than other sequential convex hull implementations running on the CPU for inputs of millions of points. The works demonstrate that the GPU can be used to solve nontrivial computational geometry problems with significant performance benefit.

Several problems arising in control system analysis and design, such as reduced order controller synthesis, involve minimizing the rank of a matrix variable subject to linear matrix inequality (LMI) constraints. Except in some special cases, solving this rank minimization problem (globally) is very difficult. One simple and surprisingly effective heuristic, applicable when the matrix variable is symmetric and positive semidefinite, is to minimize its trace in place of its rank. This results in a semidefinite program (SDP) which can be efficiently solved. In this paper we describe a generalization of the trace heuristic that applies to general nonsymmetric, even non-square, matrices, and reduces to the trace heuristic when the matrix is positive semidefinite. The heuristic is to replace the (nonconvex) rank objective with the sum of the singular values of the matrix, which is the dual of the spectral norm. We show that this problem can be reduced to an SDP, hence efficiently solved. To motivate the heuristic, we show that the dual spectral norm is the convex envelope of the rank on the set of matrices with norm less than one. We demonstrate the method on the problem of minimum order system approximation.

Matrix rank minimizing subject to affine constraints arises in many application areas, ranging from signal processing to machine learning. Nuclear norm is a convex relaxation for this problem which can recover the rank exactly under some restricted and theoretically interesting conditions. However, for many real-world applications, nuclear norm approximation to the rank function can only produce a result far from the optimum. To seek a solution of higher accuracy than the nuclear norm, in this letter, we propose a rank approximation based on Logarithm-Determinant. We consider using this rank approximation for subspace clustering application. Our framework can model different kinds of errors and noise. Effective optimization strategy is developed with theoretical guarantee to converge to a stationary point. The proposed method gives promising results on face clustering and motion segmentation tasks compared to the state-of-the-art subspace clustering algorithms.

In this article, we discuss the numerical solution of the Stokes and Navier-Stokes equations completed by nonlinear slip boundary conditions of friction type in two and three dimensions. To solve the Stokes system, we first reduce the related variational inequality into a saddle point-point problem for a well chosen augmented Lagrangian. To solve this saddle point problem we suggest an alternating direction method of multiplier together with finite element approximations. The solution of the Navier Stokes system combines finite element approximations, time discretization by operator splitting and augmented Lagrangian method. Numerical experiment results for two and three dimensional flow confirm the interest of these approaches.

The skyline of a d-dimensional dataset contains the points that are not dominated by any other point on all dimensions. Skyline computation has recently received considerable attention in the database community, especially for progressive methods that can quickly return the initial results without reading the entire database. All the existing algorithms, however, have some serious shortcomings which limit their applicability in practice. In this article we develop branch-and-bound skyline (BBS), an algorithm based on nearest-neighbor search, which is I/O optimal, that is, it performs a single access only to those nodes that may contain skyline points. BBS is simple to implement and supports all types of progressive processing (e.g., user preferences, arbitrary dimensionality, etc). Furthermore, we propose several interesting variations of skyline computation, and show how BBS can be applied for their efficient processing.

An efficient and reliable algorithm for computing the Euclidean distance between a pair of convex sets in Rm is described. Extensive numerical experience with a broad family of polytopes in R 3 shows that the computational cost is approximately linear in the total number of vertices specifying the two polytopes. The algorithm has special features which makes its application in a variety of robotics problems attractive. These are discussed and an example of collision

In this paper we develop new Newton and conjugate gradient algorithms on the Grassmann and Stiefel manifolds. These manifolds represent the constraints that arise in such areas as the symmetric eigenvalue problem, nonlinear eigenvalue problems, electronic structures computations, and signal processing. In addition to the new algorithms, we show how the geometrical framework gives penetrating new insights allowing us to create, understand, and compare algorithms. The theory proposed here provides a taxonomy for numerical linear algebra algorithms that provide a top level mathematical view of previously unrelated algorithms. It is our hope that developers of new algorithms and perturbation theories will benefit from the theory, methods, and examples in this paper.


Question: A sliding conditional gradient method in convex optimization problems.

This paper unveils fresh insights into the efficacy of a greedy algorithm, specifically the Orthogonal Matching Pursuit (OMP), in addressing the sparse approximation challenge over redundant dictionaries. It introduces a criterion that guarantees both OMP and Donoho's Basis Pursuit (BP) method can accurately reconstruct the optimal representation of signals that are exactly sparse. Building on this foundation, the paper demonstrates that both OMP and BP are capable of successfully processing every sparse signal within a broad category of dictionaries, which are characterized as quasi-incoherent. This classification offers a refined perspective on incoherent dictionaries, with the cumulative coherence function serving as a metric to gauge the degree of incoherence. This analysis not only consolidates the existing findings on BP but also extends them to encompass OMP. Moreover, the paper outlines a criterion that ensures OMP can effectively select atoms for an optimal approximation of nonsparse signals. It further posits that OMP functions as an approximation algorithm for the sparse problem over quasi-incoherent dictionaries. This means that, for any given signal, OMP is able to generate a sparse approximation whose error margin is only marginally higher than the least possible error achievable with a matching number of terms.

We describe linear-time algorithms for solving a class of problems that involve transforming a cost function on a grid using spatial information. These problems can be viewed as a generalization of classical distance transforms of binary images, where the binary image is replaced by an arbitrary function on a grid. Alternatively they can be viewed in terms of the minimum convolution of two functions, which is an important operation in grayscale morphology. A consequence of our techniques is a simple and fast method for computing the Euclidean distance transform of a binary image. Our algorithms are also applicable to Viterbi decoding, belief propagation, and optimal control. ACM Classification: F.2.1, I.4 AMS Classification: 68T45, 68W40

In many applications, optimization of a collection of problems is required where each problem is structurally the same, but in which some or all of the data defining the instance is updated. Such models are easily specified within modern modeling systems, but have often been slow to solve due to the time needed to regenerate the instance, and the inability to use advance solution information (such as basis factorizations) from previous solves as the collection is processed. We describe a new language extension, GUSS, that gathers data from different sources/symbols to define the collection of models (called scenarios), updates a base model instance with this scenario data and solves the updated model instance and scatters the scenario results to symbols in the GAMS database. We demonstrate the utility of this approach in three applications, namely data envelopment analysis, cross validation and stochastic dual dynamic programming. The language extensions are available for general use in all versions of GAMS starting with release 23.7.

We study in this paper a class of constrained linear-quadratic (LQ) optimal control problem formulations for the scalar-state stochastic system with multiplicative noise, which has various applications, especially in the financial risk management. The linear constraint on both the control and state variables considered in our model destroys the elegant structure of the conventional LQ formulation and has blocked the derivation of an explicit control policy so far in the literature. We successfully derive in this paper the analytical control policy for such a class of problems by utilizing the state separation property induced from its structure. We reveal that the optimal control policy is a piece-wise affine function of the state and can be computed off-line efficiently by solving two coupled Riccati equations. Under some mild conditions, we also obtain the stationary control policy for infinite time horizon. We demonstrate the implementation of our method via some illustrative examples and show how to calibrate our model to solve dynamic constrained portfolio optimization problems.

We introduce the Stochastic Asynchronous Proximal Alternating Linearized Minimization (SAPALM) method, a block coordinate stochastic proximal-gradient method for solving nonconvex, nonsmooth optimization problems. SAPALM is the first asynchronous parallel optimization method that provably converges on a large class of nonconvex, nonsmooth problems. We prove that SAPALM matches the best known rates of convergence — among synchronous or asynchronous methods — on this problem class. We provide upper bounds on the number of workers for which we can expect to see a linear speedup, which match the best bounds known for less complex problems, and show that in practice SAPALM achieves this linear speedup. We demonstrate state-of-the-art performance on several matrix factorization problems.

We study the numerical performance of a limited memory quasi Newton method for large scale optimization which we call the L BFGS method We compare its performance with that of the method developed by Buckley and LeNir which combines cyles of BFGS steps and conjugate direction steps Our numerical tests indicate that the L BFGS method is faster than the method of Buckley and LeNir and is better able to use additional storage to accelerate convergence We show that the L BFGS method can be greatly accelerated by means of a simple scaling We then compare the L BFGSmethod with the partitioned quasi Newton method of Griewank and Toint a The results show that for some problems the partitioned quasi Newton method is clearly superior to the L BFGS method However we nd that for other problems the L BFGS method is very competitive due to its low iteration cost We also study the convergence properties of the L BFGS method and prove global convergence on uniformly convex problems

Random sampling is used for several new geometric algorithms. The algorithms are “Las Vegas,” and their expected bounds are with respect to the random behavior of the algorithms. One algorithm reports all the intersecting pairs of a set of line segments in the plane, and requires <italic>&Ogr;</italic>(<italic>A</italic> + <italic>n</italic> log <italic>n</italic>) expected time, where <italic>A</italic> is the size of the answer, the number of intersecting pairs reported. The algorithm requires <italic>&Ogr;</italic>(<italic>n</italic>) space in the worst case. Another algorithm computes the convex hull of a point set in <italic>E</italic><supscrpt>3</supscrpt> in <italic>&Ogr;</italic>(<italic>n</italic> log <italic>A</italic>) expected time, where <italic>n</italic> is the number of points and <italic>A</italic> is the number of points on the surface of the hull. A simple Las Vegas algorithm triangulates simple polygons in <italic>&Ogr;</italic>(<italic>n</italic> log log <italic>n</italic>) expected time. Algorithms for half-space range reporting are also given. In addition, this paper gives asymptotically tight bounds for a combinatorial quantity of interest in discrete and computational geometry, related to halfspace partitions of point sets.

We propose a new stochastic algorithm (generalized simulated annealing) for computationally finding the global minimum of a given (not necessarily convex) energy/cost function defined in a continuous D-dimensional space. This algorithm recovers, as particular cases, the so called classical (“Boltzmann machine”) and fast (“Cauchy machine”) simulated annealings, and can be quicker than both. Key-words: Simulated annealing; Nonconvex optimization; Gradient descent; Generalized Statistical Mechanics.

Chapter 1 strongly advocates the stochastic back-propagation method to train neural networks. This is in fact an instance of a more general technique called stochastic gradient descent (SGD). This chapter provides background material, explains why SGD is a good learning algorithm when the training set is large, and provides useful recommendations.

We propose a new fast algorithm for solving one of the standard formulations of image restoration and reconstruction which consists of an unconstrained optimization problem where the objective includes an l2 data-fidelity term and a nonsmooth regularizer. This formulation allows both wavelet-based (with orthogonal or frame-based representations) regularization or total-variation regularization. Our approach is based on a variable splitting to obtain an equivalent constrained optimization formulation, which is then addressed with an augmented Lagrangian method. The proposed algorithm is an instance of the so-called alternating direction method of multipliers, for which convergence has been proved. Experiments on a set of image restoration and reconstruction benchmark problems show that the proposed algorithm is faster than the current state of the art methods.

Scalable distributed dataflow systems have recently experienced widespread adoption, with commodity dataflow engines such as Hadoop and Spark, and even commodity SQL engines routinely supporting increasingly sophisticated analytics tasks (e.g., support vector machines, logistic regression, collaborative filtering). However, these systems’ synchronous (often Bulk Synchronous Parallel) dataflow execution model is at odds with an increasingly important trend in the machine learning community: the use of asynchrony via shared, mutable state (i.e., data races) in convex programming tasks, which has—in a single-node context—delivered noteworthy empirical performance gains and inspired new research into asynchronous algorithms. In this work, we attempt to bridge this gap by evaluating the use of lightweight, asynchronous state transfer within a commodity dataflow engine. Specifically, we investigate the use of asynchronous sideways information passing (ASIP) that presents single-stage parallel iterators with a Volcano-like intra-operator iterator that can be used for asynchronous information passing. We port two synchronous convex programming algorithms, stochastic gradient descent and the alternating direction method of multipliers (ADMM), to use ASIPs. We evaluate an implementation of ASIPs within on Apache Spark that exhibits considerable speedups as well as a rich set of performance trade-offs in the use of these asynchronous algorithms.


Question: A sliding conditional gradient technique for solving convex optimization problems.

This document introduces a novel methodology for devising subgradient techniques tailored to various nonsmooth problems possessing a convex structure. Distinguished by their primal-dual nature, these methods inherently generate feasible approximations to the optimum of an associated dual problem formulation, offering a significant advantage—a robust stopping criterion. Unlike conventional approaches, such as divergent series methods and mirror descent methods, the proposed schemes incorporate two control sequences. One sequence facilitates the aggregation of support functions within the dual space, while the other dynamically adjusts the scale between the primal and dual spaces. This additional layer of adaptability ensures the boundedness of the primal test point sequence, even in scenarios with unbounded feasible sets. The paper outlines subgradient scheme variants for a range of problem types, including nonsmooth convex minimization, minimax problems, saddle point problems, variational inequalities, and stochastic optimization. Across these applications, the methods are proven to achieve optimality from the perspective of worst-case black-box lower complexity bounds.

**Question**: A Multi-Objective Evolutionary Algorithm for Portfolio Optimization

**Documents**: A new multiobjective evolutionary algorithm for the multiobjective traveling salesman problem
The multiobjective traveling salesman problem (MTSP) is a generalization of the well-known traveling salesman problem (TSP), in which the traveling salesman has to visit a set of cities exactly once and return to the starting city in order to minimize a set of objectives. In this paper, we present a new multiobjective evolutionary algorithm (MOEA) to solve the MTSP. The MOEA is based on the nondominated sorting genetic algorithm II (NSGA-II) and uses a new local search operator. The proposed algorithm is called NSGA-II with local search (NSGA-II/LS). The local search operator is used to improve the solutions generated by the evolutionary algorithm. We compare the proposed algorithm with two other MOEAs, namely the nondominated sorting genetic algorithm (NSGA) and the strength Pareto evolutionary algorithm (SPEA). The experimental results show that the proposed algorithm is able to find solutions that are close to the Pareto front and that it outperforms the other two algorithms in terms of the hypervolume and the inverted generational distance.

**Similar Documents**: This paper introduces a novel multiobjective evolutionary algorithm (MOEA) designed to address the multiobjective traveling salesman problem (MTSP), a generalization of the classic traveling salesman problem (TSP). In the MTSP, the objective is to optimize multiple criteria while ensuring that each city is visited exactly once and the journey returns

A constrained optimization type of numerical algorithm for removing noise from images is presented. The total variation of the image is minimized subject to constraints involving the statistics of the noise. The constraints are imposed using Lagrange multipliers. The solution is obtained using the gradient-projection method. This amounts to solving a time dependent partial differential equation on a manifold determined by the constraints. As t → ∞ the solution converges to a steady state which is the denoised image. The numerical algorithm is simple and relatively fast. The results appear to be state-of-the-art for very noisy images. The method is noninvasive, yielding sharp edges in the image. The technique could be interpreted as a first step of moving each level set of the image normal to itself with velocity equal to the curvature of the level set divided by the magnitude of the gradient of the image, and a second step which projects the image back onto the constraint set.

This paper discusses efficient techniques for computing PageRank, a ranking metric for hypertext documents. We show that PageRank can be computed for very large subgraphs of the web (up to hundreds of millions of nodes) on machines with limited main memory. Running-time measurements on various memory configurations are presented for PageRank computation over the 24-million-page Stanford WebBase archive. We discuss several methods for analyzing the convergence of PageRank based on the induced ordering of the pages. We present convergence results helpful for determining the number of iterations necessary to achieve a useful PageRank assignment, both in the absence and presence of search queries.

Function approximation is viewed from the perspective of numerical optimization in function space, rather than parameter space. A connection is made between stagewise additive expansions and steepest{descent minimization. A general gradient{descent \boosting" paradigm is developed for additive expansions based on any tting criterion. Speci c algorithms are presented for least{squares, least{absolute{deviation, and Huber{M loss functions for regression, and multi{class logistic likelihood for classi cation. Special enhancements are derived for the particular case where the individual additive components are decision trees, and tools for interpreting such \TreeBoost" models are presented. Gradient boosting of decision trees produces competitive, highly robust, interpretable procedures for regression and classi cation, especially appropriate for mining less than clean data. Connections between this approach and the boosting methods of Freund and Shapire 1996, and Friedman, Hastie, and Tibshirani 1998 are discussed. 1 Function estimation In the function estimation problem one has a system consisting of a random \output" or \response" variable y and a set of random \input" or \explanatory" variables x = fx1; ; xng. Given a \training" sample fyi;xig N 1 of known (y;x){values, the goal is to nd a function F (x) that maps x to y, such that over the joint distribution of all (y;x){values, the expected value of some speci ed loss function (y; F (x)) is minimized F (x) = argmin F (x) Ey;x (y; F (x)) = argmin F (x) Ex [Ey( (y; F (x)) jx] : (1) Frequently employed loss functions (y; F ) include squared{error (y F ) and absolute error jy F j for y 2 R (regression), and negative binomial log{likelihood, log(1 + e 2yF ), when y 2 f 1; 1g (classi cation). A common procedure is to take F (x) to be a member of a parameterized class of functions F (x;P), where P = fP1; P2; g is a set of parameters. In this paper we focus on \additive" expansions of the form

We discuss the use in machine learning of a general type of convex optimisation problem known as semi-definite programming (SDP) [1]. We intend to argue that SDP’s arise quite naturally in a variety of situations, accounting for their omnipresence in modern machine learning approaches, and we provide examples in support.

Full terms and conditions of use: http://pubsonline.informs.org/page/terms-and-conditions This article may be used only for the purposes of research, teaching, and/or private study. Commercial use or systematic downloading (by robots or other automatic processes) is prohibited without explicit Publisher approval, unless otherwise noted. For more information, contact permissions@informs.org. The Publisher does not warrant or guarantee the article’s accuracy, completeness, merchantability, fitness for a particular purpose, or non-infringement. Descriptions of, or references to, products or publications, or inclusion of an advertisement in this article, neither constitutes nor implies a guarantee, endorsement, or support of claims made of that product, publication, or service. © 1990 INFORMS

Hadoop, employing the MapReduce programming paradigm, has been widely accepted as the standard framework for analyzing big data in distributed environments. Unfortunately, this rich framework was not truly exploited towards processing large-scale computational geometry operations. This paper introduces CG_Hadoop; a suite of scalable and efficient MapReduce algorithms for various fundamental computational geometry problems, namely, polygon union, skyline, convex hull, farthest pair, and closest pair, which present a set of key components for other geometric algorithms. For each computational geometry operation, CG_Hadoop has two versions, one for the Apache Hadoop system and one for the SpatialHadoop system; a Hadoop-based system that is more suited for spatial operations. These proposed algorithms form a nucleus of a comprehensive MapReduce library of computational geometry operations. Extensive experimental results on a cluster of 25 machines of datasets up to 128GB show that CG_Hadoop achieves up to 29x and 260x better performance than traditional algorithms when using Hadoop and SpatialHadoop systems, respectively.

In many applications of black-box optimization, one can evaluate multiple points simultaneously, e.g. when evaluating the performances of several different neural networks in a parallel computing environment. In this paper, we develop a novel batch Bayesian optimization algorithm — the parallel knowledge gradient method. By construction, this method provides the one-step Bayes optimal batch of points to sample. We provide an efficient strategy for computing this Bayes-optimal batch of points, and we demonstrate that the parallel knowledge gradient method finds global optima significantly faster than previous batch Bayesian optimization algorithms on both synthetic test functions and when tuning hyperparameters of practical machine learning algorithms, especially when function evaluations are noisy.

We present methods for online linear optimization that take advantage of benign (as opposed to worst-case) sequences. Specifically if the sequence encountered by the learner is described well by a known “predictable process”, the algorithms presented enjoy tighter bounds as compared to the typical worst case bounds. Additionally, the methods achieve the usual worst-case regret bounds if the sequence is not benign. Our approach can be seen as a way of adding prior knowledge about the sequence within the paradigm of online learning. The setting is shown to encompass partial and side information. Variance and path-length bounds [11, 9] can be seen as particular examples of online learning with simple predictable sequences. We further extend our methods and results to include competing with a set of possible predictable processes (models), that is “learning” the predictable process itself concurrently with using it to obtain better regret guarantees. We show that such model selection is possible under various assumptions on the available feedback. Our results suggest a promising direction of further research with potential applications to stock market and time series prediction.

The `1-regularized maximum likelihood estimation problem has recently become a topic of great interest within the machine learning, statistics, and optimization communities as a method for producing sparse inverse covariance estimators. In this paper, a proximal gradient method (G-ISTA) for performing `1-regularized covariance matrix estimation is presented. Although numerous algorithms have been proposed for solving this problem, this simple proximal gradient method is found to have attractive theoretical and numerical properties. G-ISTA has a linear rate of convergence, resulting in an O(log ε) iteration complexity to reach a tolerance of ε. This paper gives eigenvalue bounds for the G-ISTA iterates, providing a closed-form linear convergence rate. The rate is shown to be closely related to the condition number of the optimal point. Numerical convergence results and timing comparisons for the proposed method are presented. G-ISTA is shown to perform very well, especially when the optimal point is well-conditioned.

This paper studies the problem of simultaneously aligning a batch of linearly correlated images despite gross corruption (such as occlusion). Our method seeks an optimal set of image domain transformations such that the matrix of transformed images can be decomposed as the sum of a sparse matrix of errors and a low-rank matrix of recovered aligned images. We reduce this extremely challenging optimization problem to a sequence of convex programs that minimize the sum of ℓ1-norm and nuclear norm of the two component matrices, which can be efficiently solved by scalable convex optimization techniques. We verify the efficacy of the proposed robust alignment algorithm with extensive experiments on both controlled and uncontrolled real data, demonstrating higher accuracy and efficiency than existing methods over a wide range of realistic misalignments and corruptions.


Question: Basic constraints regarding adversarial robustness.

This paper delves into the enigma of adversarial examples, a critical issue where machine learning models, notably neural networks, are deceived by inputs that are subtly altered. These adversarial examples are created by introducing minimal yet meticulously crafted perturbations to dataset samples, causing the model to confidently produce incorrect outputs. Initially, explanations for this phenomenon pointed to nonlinearity and overfitting. However, we propose a different perspective, attributing the susceptibility of neural networks to adversarial perturbations to their inherent linearity. This hypothesis is substantiated by novel quantitative findings, offering the first coherent explanation for a perplexing characteristic of adversarial examples: their consistent appearance across various architectures and training datasets. Furthermore, this viewpoint leads to a straightforward and efficient technique for generating adversarial examples. By incorporating these examples into adversarial training, we achieve a reduction in the test set error of a maxout network applied to the MNIST dataset, showcasing the practical implications of our theoretical insights.

**Question**: The Impact of Social Media on Mental Health

**Documents**: The relationship between social media use and depression among U.S. young adults
Social media use is increasingly prevalent in the lives of U.S. young adults. However, its impact on mental health, particularly depression, remains unclear. This study examines the association between social media use and depression among a nationally representative sample of U.S. young adults. The data come from the National Longitudinal Study of Adolescent to Adult Health (Add Health), a longitudinal study of a nationally representative sample of U.S. adolescents. The sample for this study includes 1,787 young adults who participated in Wave IV of Add Health, which was conducted in 2008. The main outcome variable is depression, which is measured using the Center for Epidemiologic Studies Depression Scale (CES-D). The key independent variable is social media use, which is measured using a series of questions about the frequency and duration of use of various social media platforms. The results show a significant positive association between social media use and depression. Young adults who spend more time on social media are more likely to experience symptoms of depression. This association remains significant even after controlling for demographic characteristics, social support, and other potential confounding variables. These findings suggest that social media use may be a risk factor for depression among young adults. Given the widespread use of social media among this population, these results have important implications for mental health interventions and policies.

**Similar Documents**: This research investigates the connection between social media utilization and depressive symptoms among young adults in

Information bottleneck (IB) is a method for extracting information from one random variable X that is relevant for predicting another random variable Y . To do so, IB identifies an intermediate “bottleneck” variable T that has low mutual information I(X;T ) and high mutual information I(Y ;T ). The IB curve characterizes the set of bottleneck variables that achieve maximal I(Y ;T ) for a given I(X;T ), and is typically explored by optimizing the IB Lagrangian, I(Y ;T )− βI(X;T ). Recently, there has been interest in applying IB to supervised learning, particularly for classification problems that use neural networks. In most classification problems, the output class Y is a deterministic function of the input X , which we refer to as “deterministic supervised learning”. We demonstrate three pathologies that arise when IB is used in any scenario where Y is a deterministic function of X: (1) the IB curve cannot be recovered by optimizing the IB Lagrangian for different values of β; (2) there are “uninteresting” solutions at all points of the IB curve; and (3) for classifiers that achieve low error rates, the activity of different hidden layers will not exhibit a strict trade-off between compression and prediction, contrary to a recent proposal. To address problem (1), we propose a functional that, unlike the IB Lagrangian, can recover the IB curve in all cases. We finish by demonstrating these issues on the MNIST dataset.

We present a polynomial-time algorithm that, given as a input the description of a game with incomplete information and any number of players, produces a protocol for playing the game that leaks no partial information, provided the majority of the players is honest.
Our algorithm automatically solves all the multi-party protocol problems addressed in complexity-based cryptography during the last 10 years. It actually is a completeness theorem for the class of distributed protocols with honest majority. Such completeness theorem is optimal in the sense that, if the majority of the players is not honest, some protocol problems have no efficient solution [C].

Recent work by Krawczyk [12] and Menezes [16] has highlighted the importance of understanding well the guarantees and limitations of formal security models when using them to prove the security of protocols. In this paper we focus on security models for authenticated key exchange (AKE) protocols. We observe that there are several classes of attacks on AKE protocols that lie outside the scope of the Canetti-Krawczyk model. Some of these additional attacks have already been considered by Krawczyk [12]. In an attempt to bring these attacks within the scope of the security model we extend the Canetti-Krawczyk model for AKE security by providing significantly greater powers to the adversary. Our contribution is a more compact, integrated, and comprehensive formulation of the security model. We then introduce a new AKE protocol called NAXOS and prove that it is secure against these stronger

With the development of neural networks based machine learning and their usage in mission critical applications, voices are rising against the black box aspect of neural networks as it becomes crucial to understand their limits and capabilities. With the rise of neuromorphic hardware, it is even more critical to understand how a neural network, as a distributed system, tolerates the failures of its computing nodes, neurons, and its communication channels, synapses. Experimentally assessing the robustness of neural networks involves the quixotic venture of testing all the possible failures, on all the possible inputs, which ultimately hits a combinatorial explosion for the first, and the impossibility to gather all the possible inputs for the second.In this paper, we prove an upper bound on the expected error of the output when a subset of neurons crashes. This bound involves dependencies on the network parameters that can be seen as being too pessimistic in the average case. It involves a polynomial dependency on the Lipschitz coefficient of the neurons' activation function, and an exponential dependency on the depth of the layer where a failure occurs. We back up our theoretical results with experiments illustrating the extent to which our prediction matches the dependencies between the network parameters and robustness. Our results show that the robustness of neural networks to the average crash can be estimated without the need to neither test the network on all failure configurations, nor access the training set used to train the network, both of which are practically impossible requirements.

This paper presents a game-theoretic method for analyzing the security of computer networks. We view the interactions between an attacker and the administrator as a two-player stochastic game and construct a model for the game. Using a nonlinear program, we compute Nash equilibria or best-response strategies for the players (attacker and administrator). We then explain why the strategies are realistic and how administrators can use these results to enhance the security of their network.

We survey promising directions for research in the area of formal methods and its applications, including fundamental questions about the combination of formal methods and machine learning, and applications to cyberphysical systems and education.

Many resource allocation problems can be formulated as an optimization problem whose constraints contain sensitive information about participating users. This paper concerns a class of resource allocation problems whose objective function depends on the aggregate allocation (i.e., the sum of individual allocations); in particular, we investigate distributed algorithmic solutions that preserve the privacy of participating users. Without privacy considerations, existing distributed algorithms normally consist of a central entity computing and broadcasting certain public coordination signals to participating users. However, the coordination signals often depend on user information, so that an adversary who has access to the coordination signals can potentially decode information on individual users and put user privacy at risk. We present a distributed optimization algorithm that preserves differential privacy, which is a strong notion that guarantees user privacy regardless of any auxiliary information an adversary may have. The algorithm achieves privacy by perturbing the public signals with additive noise, whose magnitude is determined by the sensitivity of the projection operation onto user-specified constraints. By viewing the differentially private algorithm as an implementation of stochastic gradient descent, we are able to derive a bound for the suboptimality of the algorithm. We illustrate the implementation of our algorithm via a case study of electric vehicle charging. Specifically, we derive the sensitivity and present numerical simulations for the algorithm. Through numerical simulations, we are able to investigate various aspects of the algorithm when being used in practice, including the choice of step size, number of iterations, and the trade-off between privacy level and suboptimality.

This paper takes an initial step forward in bringing to life the certification mechanisms according to Art. 42 of the General Data Protection Regulation (GDPR). These newly established methods of legal specification act not only as a central vehicle for overcoming widely articulated and discussed legal challenges, but also as a sandbox for the much needed close collaboration between computer sciences and legal studies. In order to illustrate, for example, what data protection seals could look like in the future, the authors propose a methodology for "translating" legal requirements into technical guidelines: architectural blueprints designed using legal requirements. The purpose of these blueprints is to show developers how their solutions might comply with the principle of Privacy by Design (Art. 25 GDPR). To demonstrate this methodology, the authors propose an architectural blueprint that embodies the legal concept of the data subject’s consent (Art. 6 sec. 1 lit. a GDPR) and elevates best practice to a high standard of Privacy by Design. Finally, the authors highlight further legal problems concerning blockchain technology under the GDPR that will have to be addressed in order to achieve a comprehensive certification mechanism for Privacy by Blockchain Design in the future. ACM Classification

Encrypted databases, a popular approach to protecting data from compromised database management systems (DBMS's), use abstract threat models that capture neither realistic databases, nor realistic attack scenarios. In particular, the "snapshot attacker" model used to support the security claims for many encrypted databases does not reflect the information about past queries available in any snapshot attack on an actual DBMS.
 We demonstrate how this gap between theory and reality causes encrypted databases to fail to achieve their "provable security" guarantees.

We prove a lower bound of ( ln + VCdim(C) ) on the number of random examples required for distribution-free learning of a concept class C, where VCdim(C) is the Vapnik-Chervonenkis dimension and and are the accuracy and con dence parameters. This improves the previous best lower bound of ( ln + VCdim(C)), and comes close to the known general upper bound of O( ln + VCdim(C) ln ) for consistent algorithms. We show that for many interesting concept classes, including kCNF and kDNF, our bound is actually tight to within a constant factor. A. Ehrenfeucht was supported by NSF grant MCS-8305245, D. Haussler by ONR grant N00014-86-K-0454, M. Kearns by ONR grant N00014-86-K-0454 and an A.T. & T. Bell Laboratories Scholarship, and L. Valiant by grants ONR-N00014-85-K-0445, NSF-DCR-8600379, DAAL03-86-K-0171 and by the SERC of the U.K. Part of this research was done while M. Kearns was visiting U.C. Santa Cruz.


Question: Inherent bounds on robustness against adversarial attacks

Exploring Robust Deep Neural Network Architectures Against Adversarial Attacks

The vulnerability of deep neural networks (DNNs) to adversarial examples, which are carefully crafted perturbations at the input layer, has been a recent focus of study. These perturbations, often undetectable by human perception, can lead to significant misclassifications by even the most advanced DNNs. This paper delves into the characteristics of adversarial examples and examines various strategies to enhance DNN robustness, including network architecture modifications, preprocessing techniques, and specialized training methods. Through a series of experiments, we evaluate the effectiveness of removing adversarial effects by introducing additional noise and preprocessing with denoising autoencoders (DAEs). Our findings indicate that DAEs are capable of significantly mitigating adversarial noise. However, when these DAEs are integrated with the original DNN, the resulting architecture remains susceptible to new adversarial examples with potentially smaller distortions. To address this issue, we introduce the Deep Contractive Network, a novel model that incorporates a smoothness penalty during end-to-end training, inspired by the contractive autoencoder (CAE) principle. This approach not only boosts the network's resilience against adversarial examples but also maintains high performance with minimal impact on classification accuracy.

Recent trojan attacks on deep neural network (DNN) models are one insidious variant of data poisoning attacks. Trojan attacks exploit an effective backdoor created in a DNN model by leveraging the difficulty in interpretability of the learned model to misclassify any inputs signed with the attacker’s chosen trojan trigger input. Trojan attacks are easy to craft; survive even in adverse conditions such as different viewpoints, and lighting conditions on images; and threaten real world applications such as autonomous vehicles and robotics. The trojan trigger is a secret guarded and exploited by the attacker. Therefore, detecting such trojaned inputs is a challenge, especially at run-time when models are in active operation. We focus on vision systems and build the STRong Intentional Perturbation (STRIP) based run-time trojan attack detection system. We intentionally perturb the incoming input, for instance by superimposing various image patterns, and observe the randomness of the predicted classes for the perturbed inputs for a given model—malicious or benign—under deployment. A low entropy in the predicted classes violates the input-dependence property of a benign model and implies the presence of a malicious input—a characteristic of a trojaned input. The high efficacy of our method is validated through case studies on two popular and contrasting datasets: MNIST and CIFAR10. We achieve an overall false acceptance rate (FAR) of less than 1%, given a preset false rejection rate (FRR) of 1%, for four different tested trojan trigger types—three triggers are identified in previous attack works and one dedicated trigger is crafted by us to demonstrate the trigger-size insensitivity of the STRIP detection approach. In particular, on the dataset of natural images in CIFAR10, we have empirically achieved the desired result of 0% for both FRR and FAR.

Past works have shown that, somewhat surprisingly, over-parametrization can help generalization in neural networks. Towards explaining this phenomenon, we adopt a margin-based perspective. We establish: 1) for multilayer feedforward relu networks, the global minimizer of a weakly-regularized cross-entropy loss has the maximum normalized margin among all networks, 2) as a result, increasing the over-parametrization improves the normalized margin and generalization error bounds for two-layer networks. In particular, an infinite-size neural network enjoys the best generalization guarantees. The typical infinite feature methods are kernel methods; we compare the neural net margin with that of kernel methods and construct natural instances where kernel methods have much weaker generalization guarantees. We validate this gap between the two approaches empirically. Finally, this infinite-neuron viewpoint is also fruitful for analyzing optimization. We show that a perturbed gradient flow on infinite-size networks finds a global optimizer in polynomial time.

We define notions of stability for learning algorithms and show how to use these notions to derive generalization error bounds based on the empirical error and the leave-one-out error. The methods we use can be applied in the regression framework as well as in the classification one when the classifier is obtained by thresholding a real-valued function. We study the stability properties of large classes of learning algorithms such as regularization based algorithms. In particular we focus on Hilbert space regularization and Kullback-Leibler regularization. We demonstrate how to apply the results to SVM for regression and classification.

A power grid is a complex system connecting electric power generators to consumers through power transmission and distribution networks across a large geographical area. System monitoring is necessary to ensure the reliable operation of power grids, and state estimation is used in system monitoring to best estimate the power grid state through analysis of meter measurements and power system models. Various techniques have been developed to detect and identify bad measurements, including interacting bad measurements introduced by arbitrary, nonrandom causes. At first glance, it seems that these techniques can also defeat malicious measurements injected by attackers.
 In this article, we expose an unknown vulnerability of existing bad measurement detection algorithms by presenting and analyzing a new class of attacks, called false data injection attacks, against state estimation in electric power grids. Under the assumption that the attacker can access the current power system configuration information and manipulate the measurements of meters at physically protected locations such as substations, such attacks can introduce arbitrary errors into certain state variables without being detected by existing algorithms. Moreover, we look at two scenarios, where the attacker is either constrained to specific meters or limited in the resources required to compromise meters. We show that the attacker can systematically and efficiently construct attack vectors in both scenarios to change the results of state estimation in arbitrary ways. We also extend these attacks to generalized false data injection attacks, which can further increase the impact by exploiting measurement errors typically tolerated in state estimation. We demonstrate the success of these attacks through simulation using IEEE test systems, and also discuss the practicality of these attacks and the real-world constraints that limit their effectiveness.

Split manufacturing can mitigate security vulnerabilities at untrusted foundries by exposing only partial designs. Even so, attackers can make educated guess according to design conventions and thereby recover entire chip designs. In this work, a routing perturbation-based defense method is proposed such that such attacks become very difficult while wirelength/timing overhead is restricted to be very small. Experimental results on benchmark circuits confirm the effectiveness of the proposed techniques. The new techniques also significantly outperform the latest previous work.

Nowadays, convolutional neural network (CNN) based steganalysis methods achieved great performance. While those methods are also facing security problems. In this paper, we proposed an attack scheme aiming at CNN based steganalyzer including two different attack methods 1) the LSB-Jstego Gradient Based Attack; 2) LSB-Jstego Evolutionary Algorithms Based Attack. The experiment results show that the attack strategies could achieve 96.02% and 90.25% success ratio separately on the target CNN. The proposed attack scheme is an effective way to fool the CNN based steganalyzer and in addition demonstrates the vulnerability of the neural networks in steganalysis.

Deep neural networks are often trained in the over-parametrized regime (i.e. with far more parameters than training examples), and understanding why the training converges to solutions that generalize remains an open problem Zhang et al. [2017]. Several studies have highlighted the fact that the training procedure, i.e. mini-batch Stochastic Gradient Descent (SGD) leads to solutions that have specific properties in the loss landscape. However, even with plain Gradient Descent (GD) the solutions found in the over-parametrized regime are pretty good and this phenomenon is poorly understood. We propose an analysis of this behavior for feedforward networks with a ReLU activation function under the assumption of small initialization and learning rate and uncover a quantization effect: The weight vectors tend to concentrate at a small number of directions determined by the input data. As a consequence, we show that for given input data there are only finitely many, “simple” functions that can be obtained, independent of the network size. This puts these functions in analogy to linear interpolations (for given input data there are finitely many triangulations, which each determine a function by linear interpolation). We ask whether this analogy extends to the generalization properties while the usual distribution-independent generalization property does not hold, it could be that for e.g. smooth functions with bounded second derivative an approximation property holds which could “explain” generalization of networks (of unbounded size) to unseen inputs.

An obfuscator is a compiler that transforms any program (which we will view in this work as a boolean circuit) into an obfuscated program (also a circuit) that has the same input-output functionality as the original program, but is “unintelligible”. Obfuscation has applications for cryptography and for software protection. Barak et al. (CRYPTO 2001, pp. 1–18, 2001) initiated a theoretical study of obfuscation, which focused on black-box obfuscation, where the obfuscated circuit should leak no information except for its (black-box) input-output functionality. A family of functionalities that cannot be obfuscated was demonstrated. Subsequent research has showed further negative results as well as positive results for obfuscating very specific families of circuits, all with respect to black box obfuscation. This work is a study of a new notion of obfuscation, which we call best-possible obfuscation. Best possible obfuscation makes the relaxed requirement that the obfuscated program leaks as little information as any other program with the same functionality (and of similar size). In particular, this definition allows the program to leak information that cannot be obtained from a black box. Best-possible obfuscation guarantees that any information that is not hidden by the obfuscated program is also not hidden by any other similar-size program computing the same functionality, and thus the obfuscation is (literally) the best possible. In this work we study best-possible obfuscation and its relationship to previously studied definitions. Our main results are: (1) A separation between black-box and best-possible obfuscation. We show a natural obfuscation task that can be achieved under the best-possible definition, but cannot be achieved under the black-box definition. (2) A hardness result for best-possible obfuscation, showing that strong (information-theoretic) best-possible obfuscation implies a collapse in the Polynomial-Time Hierarchy. (3) An impossibility result for efficient best-possible (and black-box) obfuscation in the presence of random oracles. This impossibility result uses a random oracle to construct hard-to-obfuscate circuits, and thus it does not imply impossibility in the standard model.

In practice, there are often explicit constraints on what representations or decisions are acceptable in an application of machine learning. For example it may be a legal requirement that a decision must not favour a particular group. Alternatively it can be that that representation of data must not have identifying information. We address these two related issues by learning flexible representations that minimize the capability of an adversarial critic. This adversary is trying to predict the relevant sensitive variable from the representation, and so minimizing the performance of the adversary ensures there is little or no information in the representation about the sensitive variable. We demonstrate this adversarial approach on two problems: making decisions free from discrimination and removing private information from images. We formulate the adversarial model as a minimax problem, and optimize that minimax objective using a stochastic gradient alternate min-max optimizer. We demonstrate the ability to provide discriminant free representations for standard test problems, and compare with previous state of the art methods for fairness, showing statistically significant improvement across most cases. The flexibility of this method is shown via a novel problem: removing annotations from images, from separate training examples of annotated and unannotated images, and with no a priori knowledge of the form of annotation provided to the model.

In this paper, we propose a novel robust estimator, called ASEE (Adaptive Scale based Entropy-like Estimator) which minimizes the entropy of inliers. This estimator is based on IKOSE (Iterative Kth Ordered Scale Estimator) and LEL (Least Entropy-Like Estimator). Unlike LEL, ASEE only considers inliers' entropy while excluding outliers, which makes it very robust in parametric model estimation. Compared with other robust estimators, ASEE is simple and computationally efficient. From the experiments on both synthetic and real-image data, ASEE is more robust than several state-of-the-art robust estimators, especially in handling extreme outliers.


Question: Intrinsic bounds on robustness against adversarial attacks

Deep Neural Networks (DNNs) have emerged as powerful tools in pattern recognition, particularly in visual classification tasks, where they often rival human performance. This raises questions about the disparities between human and computer vision systems. A recent investigation [30] uncovered that DNNs can be misled by images that appear unchanged to human observers, resulting in misclassifications (e.g., a lion being incorrectly identified as a library). Building on this, we present a finding that further elucidates the limitations of DNNs: it is remarkably simple to generate images that are utterly unrecognizable to humans but are confidently classified by state-of-the-art DNNs as specific objects (e.g., identifying white noise as a lion with absolute certainty). Specifically, we employ evolutionary algorithms or gradient ascent techniques to create images that convolutional neural networks, trained on either the ImageNet or MNIST datasets, label with near-perfect confidence as belonging to each category. These images, which are indiscernible to the human eye, are referred to as "fooling images" (or more broadly, fooling examples). Our study highlights significant distinctions between human vision and current DNN capabilities, and raises important questions about the comprehensiveness and robustness of DNN-based computer vision.

Deep neural networks (DNNs) have recently been achieving state-of-the-art performance on a variety of pattern-recognition tasks, most notably visual classification problems. Given that DNNs are now able to classify objects in images with near-human-level performance, questions naturally arise as to what differences remain between computer and human vision. A recent study [30] revealed that changing an image (e.g. of a lion) in a way imperceptible to humans can cause a DNN to label the image as something else entirely (e.g. mislabeling a lion a library). Here we show a related result: it is easy to produce images that are completely unrecognizable to humans, but that state-of-the-art DNNs believe to be recognizable objects with 99.99% confidence (e.g. labeling with certainty that white noise static is a lion). Specifically, we take convolutional neural networks trained to perform well on either the ImageNet or MNIST datasets and then find images with evolutionary algorithms or gradient ascent that DNNs label with high confidence as belonging to each dataset class. It is possible to produce images totally unrecognizable to human eyes that DNNs believe with near certainty are familiar objects, which we call “fooling images” (more generally, fooling examples). Our results shed light on interesting differences between human vision and current DNNs, and raise questions about the generality of DNN computer vision.

We study the behavior of weight-tied multilayer vanilla autoencoders under the assumption of random weights. Via an exact characterization in the limit of large dimensions, our analysis reveals interesting phase transition phenomena when the depth becomes large. This, in particular, provides quantitative answers and insights to three questions that were yet fully understood in the literature. Firstly, we provide a precise answer on how the random deep weight-tied autoencoder model performs “approximate inference” as posed by Scellier et al. (2018), and its connection to reversibility considered by several theoretical studies. Secondly, we show that deep autoencoders display a higher degree of sensitivity to perturbations in the parameters, distinct from the shallow counterparts. Thirdly, we obtain insights on pitfalls in training initialization practice, and demonstrate experimentally that it is possible to train a deep autoencoder, even with the tanh activation and a depth as large as 200 layers, without resorting to techniques such as layer-wise pre-training or batch normalization. Our analysis is not specific to any depths or any Lipschitz activations, and our analytical techniques may have broader applicability.

Large-scale learning problems require a plethora of labels that can be efficiently collected from crowdsourcing services at low cost. However, labels annotated by crowdsourced workers are often noisy, which inevitably degrades the performance of large-scale optimizations including the prevalent stochastic gradient descent (SGD). Specifically, these noisy labels adversely affect updates of the primal variable in conventional SGD. To solve this challenge, we propose a robust SGD mechanism called progressive stochastic learning (POSTAL), which naturally integrates the learning regime of curriculum learning (CL) with the update process of vanilla SGD. Our inspiration comes from the progressive learning process of CL, namely learning from “easy” tasks to “complex” tasks. Through the robust learning process of CL, POSTAL aims to yield robust updates of the primal variable on an ordered label sequence, namely, from “reliable” labels to “noisy” labels. To realize POSTAL mechanism, we design a cluster of “screening losses,” which sorts all labels from the reliable region to the noisy region. To sum up, POSTAL using screening losses ensures robust updates of the primal variable on reliable labels first, then on noisy labels incrementally until convergence. In theory, we derive the convergence rate of POSTAL realized by screening losses. Meanwhile, we provide the robustness analysis of representative screening losses. Experimental results on UCI1 simulated and Amazon Mechanical Turk crowdsourcing data sets show that the POSTAL using screening losses is more effective and robust than several existing baselines.1UCI is the abbreviation of University of California Irvine.

Human visual object recognition is typically rapid and seemingly effortless, as well as largely independent of viewpoint and object orientation. Until very recently, animate visual systems were the only ones capable of this remarkable computational feat. This has changed with the rise of a class of computer vision algorithms called deep neural networks (DNNs) that achieve human-level classification performance on object recognition tasks. Furthermore, a growing number of studies report similarities in the way DNNs and the human visual system process objects, suggesting that current DNNs may be good models of human visual object recognition. Yet there clearly exist important architectural and processing differences between stateof-the-art DNNs and the primate visual system. The potential behavioural consequences of these differences are not well understood. We aim to address this issue by comparing human and DNN generalisation abilities towards image degradations. We find the human visual system to be more robust to image manipulations like contrast reduction, additive noise or novel eidolon-distortions. In addition, we find progressively diverging classification error-patterns between man and DNNs when the signal gets weaker, indicating that there may still be marked differences in the way humans and current DNNs perform visual object recognition. We envision that our findings as well as our carefully measured and freely available behavioural datasets1 provide a new useful benchmark for the computer vision community to improve the robustness of DNNs and a motivation for neuroscientists to search for mechanisms in the brain that could facilitate this robustness. Data and materials available at https://github.com/rgeirhos/object-recognition 1 ar X iv :1 70 6. 06 96 9v 1 [ cs .C V ] 2 1 Ju n 20 17

Training deep neural networks generally requires massive amounts of data and is very computation intensive. We show here that it may be possible to circumvent the expensive gradient descent procedure and derive the parameters of a neural network directly from properties of the training data. We show that, near convergence, the gradient descent equations for layers close to the input can be linearized and become stochastic equations with noise related to the covariance of data for each class. We derive the distribution of solutions to these equations and discover that it is related to a “supervised principal component analysis.” We implement these results on image datasets MNIST, CIFAR10 and CIFAR100 and find that, indeed, pretrained layers using our findings performs comparable or superior to neural networks of the same size and architecture trained with gradient descent. Moreover, our pretrained layers can often be calculated using a fraction of the training data, owing to the quick convergence of the covariance matrix. Thus, our findings indicate that we can cut the training time both by requiring only a fraction of the data used for gradient descent, and by eliminating layers in the costly backpropagation step of the training. Additionally, these findings partially elucidate the inner workings of deep neural networks and allow us to mathematically calculate optimal solutions for some stages of classification problems, thus significantly boosting our ability to solve such problems efficiently.

Detection and rejection of adversarial examples in security sensitive and safety-critical systems using deep CNNs is essential. In this paper, we propose an approach to augment CNNs with out-distribution learning in order to reduce misclassification rate by rejecting adversarial examples. We empirically show that our augmented CNNs can either reject or classify correctly most adversarial examples generated using well-known methods (> 95% for MNIST and > 75% for CIFAR-10 on average). Furthermore, we achieve this without requiring to train using any specific type of adversarial examples and without sacrificing the accuracy of models on clean samples significantly (< 4%).

We analyze the problem of designing pattern recognition systems in adversarial settings, under an engineering viewpoint, motivated by their increasing exploitation in security-sensitive applications like spam and malware detection, despite their vulnerability to potential attacks has not yet been deeply understood. We first review previous work and report examples of how a complex system may be evaded either by leveraging on trivial vulnerabilities of its untrained components, e.g., parsing errors in the pre-processing steps, or by exploiting more subtle vulnerabilities of learning algorithms. We then discuss the need of exploiting both reactive and proactive security paradigms complementarily to improve the security by design. Our ultimate goal is to provide some useful guidelines for improving the security of pattern recognition in adversarial settings, and to suggest related open issues to foster research in this area.

Estimating the pose of a camera is a core problem in many geometric vision applications. While there has been much progress in the last two decades, the main difficulty is still dealing with data contaminated by outliers. For many scenes, e.g. with poor lightning conditions or repetitive textures, it is common that most of the correspondences are outliers. For real applications it is therefore essential to have robust estimation methods. In this paper we present an outlier rejection method for absolute pose estimation. We focus on the special case when the orientation of the camera is known. The problem is solved by projecting to a lower dimensional subspace where we are able to efficiently compute upper bounds on the maximum number of inliers. The method guarantees that only correspondences which cannot belong to an optimal pose are removed. In a number of challenging experiments we evaluate our method on both real and synthetic data and show improved performance compared to competing methods.

In this paper, we analyze the numerics of common algorithms for training Generative Adversarial Networks (GANs). Using the formalism of smooth two-player games we analyze the associated gradient vector field of GAN training objectives. Our findings suggest that the convergence of current algorithms suffers due to two factors: i) presence of eigenvalues of the Jacobian of the gradient vector field with zero real-part, and ii) eigenvalues with big imaginary part. Using these findings, we design a new algorithm that overcomes some of these limitations and has better convergence properties. Experimentally, we demonstrate its superiority on training common GAN architectures and show convergence on GAN architectures that are known to be notoriously hard to train.

In this paper, we present a theoretical framework for understanding vector embedding, a fundamental building block of many deep learning models, especially in NLP. We discover a natural unitary-invariance in vector embeddings, which is required by the distributional hypothesis. This unitary-invariance states the fact that two embeddings are essentially equivalent if one can be obtained from the other by performing a relative-geometry preserving transformation, for example a rotation. This idea leads to the Pairwise Inner Product (PIP) loss, a natural unitary-invariant metric for the distance between two embeddings. We demonstrate that the PIP loss captures the difference in functionality between embeddings, and that it is tightly connect to two fundamental properties, namely similarity and compositionality. By formulating the embedding training process as matrix factorization under noise, we reveal a fundamental bias-variance tradeoff in dimensionality selection. With tools from perturbation and stability theory, we provide an upper bound on the PIP loss using the signal spectrum and noise variance, both of which can be readily inferred from data. Our framework sheds light on many empirical phenomena, including the existence of an optimal dimension, and the robustness of embeddings against over-parametrization. The bias-variance tradeoff of PIP loss explicitly answers the fundamental open problem of dimensionality selection for vector embeddings. 1 ar X iv :1 80 3. 00 50 2v 3 [ st at .M L ] 1 2 A pr 2 01 8


Question: Basic constraints regarding opposition resilience.

This paper delves into the puzzling fragility of classifiers when subjected to adversarial perturbations, a phenomenon initially highlighted by Szegedy et al. (International conference on learning representations (ICLR), 2014). We present a theoretical framework designed to scrutinize the resilience of classifiers to such perturbations, complemented by fundamental upper bounds on classifier robustness. Specifically, we derive a universal upper bound on the robustness of classifiers against adversarial perturbations, and subsequently, we apply this bound to two practical classifier categories: linear and quadratic. In both instances, our upper bound is contingent upon a distinguishability metric, which encapsulates the classification task's complexity. Our findings for both classifier types suggest that in scenarios characterized by low distinguishability, no classifier within the considered set will exhibit robustness to adversarial perturbations, notwithstanding the achievement of high accuracy. Our theoretical analysis further implies that the vulnerability to adversarial perturbations is attributable to the limited flexibility of classifiers, in juxtaposition to the classification task's difficulty, quantified by the distinguishability measure. We also highlight the dichotomy between a classifier's robustness to random noise and its resilience to adversarial perturbations. Specifically, we demonstrate that the robustness to random noise exceeds that to adversarial perturbations by a factor proportional to the square root of the signal dimension ($$\sqrt{d}$$ d) for linear classifiers. This result offers a theoretical rationale for the observed disparity between the two robustness facets in high-dimensional contexts, as noted empirically by Szegedy et al. in relation to neural networks. Finally, we corroborate our theoretical insights with experimental outcomes on both controlled and real-world datasets, which align with our theoretical analysis and extend its implications to more intricate classification methodologies.


Question: Basic constraints regarding opposition resilience.

Intriguing characteristics of deep learning architectures

Deep learning architectures, notably deep neural networks, have emerged as highly expressive models that have recently achieved remarkable performance in speech and visual recognition tasks. Their expressiveness is both the source of their success and the reason for their propensity to learn solutions that are not easily interpretable and may possess unexpected properties. This paper highlights two such peculiarities. Firstly, we observe that there is no discernible distinction between individual high-level units and random linear combinations of these units when evaluated through various unit analysis techniques. This suggests that the semantic information in the high layers of neural networks is embedded in the space itself, rather than in the individual units. Secondly, we discover that deep neural networks often learn input-output mappings that exhibit a significant degree of discontinuity. Specifically, we can induce the network to misclassify an image by introducing a subtle, yet precise, perturbation that is designed to maximize the network's prediction error. Interestingly, the nature of these perturbations is not a mere coincidence of the learning process; the same perturbation can lead a different network, trained on a distinct subset of the data, to misclassify the same input. These findings underscore the need for a deeper understanding of the fundamental limits on adversarial robustness in deep learning models.


Question: From depth information to head orientation prediction: a Siamese network strategy.

A Real-Time Deep Neural Network Approach for Head Orientation Estimation from Monocular Camera Data

In this paper, we present a novel real-time method for estimating head orientation using a monocular camera, leveraging the power of deep neural networks. Our approach is inspired by recent advances in computer vision and machine learning, particularly in the context of facial feature detection and analysis. Building upon foundational work in the field, such as the Real-Time Combined 2D + 3D Active Appearance Models by Xiao et al. [1], which combines 2D and 3D models for real-time facial analysis, and the work by Gourier et al. [2] on estimating face orientation from robust detection of salient facial features, we aim to enhance the accuracy and efficiency of head orientation estimation. We also draw from pioneering studies in 3D face analysis using random forests by Fanelli et al. [3] and the seminal work on gradient-based learning for document recognition by LeCun et al. [4]. The success of deep convolutional neural networks in image classification, as demonstrated by Krizhevsky et al. [5], further motivates our choice of deep learning techniques.

Our proposed method differs from conventional head pose estimation approaches by employing a deep neural network specifically trained to process monocular camera data for real-time head orientation estimation. The network is designed to be computationally efficient, making it suitable for real-world applications with stringent processing requirements. We focus on optimizing the network architecture and training methodologies to ensure robust performance across various lighting conditions and head orientations.

The research presented here is supported by the National Research Foundation of Korea (NRF) grant, funded by the Korean government (MSIP) under grant number 2010-0028680. Our goal is to contribute to the field of head pose estimation by offering a practical and accurate solution that can be seamlessly integrated into existing systems, enhancing user experience in applications such as augmented reality, virtual reality, and human-computer interaction.

References:
[1] J. Xiao, S. Baker, I. Matthews, T. Kanade, "Real-Time Combined 2D + 3D Active Appearance Models," CVPR, pp.535-542, 2004.
[2] N. Gourier, D. Hall, J. L. Crowley, "Estimating Face Orientation from Robust Detection of Salient Facial Features", Proceedings of Pointing 2004, ICPR, International Workshop on

Human activity recognition using depth information is an emerging and challenging technology in computer vision due to its considerable attention by many practical applications such as smart home/office system, personal health care and 3D video games. This paper presents a novel framework of 3D human body detection, tracking and recognition from depth video sequences using spatiotemporal features and modified HMM. To detect human silhouette, raw depth data is examined to extract human silhouette by considering spatial continuity and constraints of human motion information. While, frame differentiation is used to track human movements. Features extraction mechanism consists of spatial depth shape features and temporal joints features are used to improve classification performance. Both of these features are fused together to recognize different activities using the modified hidden Markov model (M-HMM). The proposed approach is evaluated on two challenging depth video datasets. Moreover, our system has significant abilities to handle subject's body parts rotation and body parts missing which provide major contributions in human activity recognition.

Face alignment has witnessed substantial progress in the last decade. One of the recent focuses has been aligning a dense 3D face shape to face images with large head poses. The dominant technology used is based on the cascade of regressors, e.g., CNNs, which has shown promising results. Nonetheless, the cascade of CNNs suffers from several drawbacks, e.g., lack of end-to-end training, handcrafted features and slow training speed. To address these issues, we propose a new layer, named visualization layer, which can be integrated into the CNN architecture and enables joint optimization with different loss functions. Extensive evaluation of the proposed method on multiple datasets demonstrates state-of-the-art accuracy, while reducing the training time by more than half compared to the typical cascade of CNNs. In addition, we compare across multiple CNN architectures, all with the visualization layer, to further demonstrate the advantage of its utilization.

Our goal is to classify 3D models directly using convolutional neural network. Most of existing approaches rely on a set of human-engineered features. We use 3D convolutional neural network to let the network learn the features over 3D space to minimize classification error. We trained and tested over ShapeNet dataset with data augmentation by applying random transformations. We made various visual analysis to find out what the network has learned. We extended our work to extract additional information such as pose of the 3D model.

A real-time solution for estimating and tracking the 3D pose of a rigid object is presented for image-based visual servo with natural landmarks. The many state-of-the-art technologies that are available for recognizing the 3D pose of an object in a natural setting are not suitable for real-time servo due to their time lags. This paper demonstrates that a real-time solution of 3D pose estimation become feasible by combining a fast tracker such as KLT [7] [8] with a method of determining the 3D coordinates of tracking points on an object at the time of SIFT based tracking point initiation, assuming that a 3D geometric model with SIFT description of an object is known a-priori. Keeping track of tracking points with KLT, removing the tracking point outliers automatically, and reinitiating the tracking points using SIFT once deteriorated, the 3D pose of an object can be estimated and tracked in real-time. This method can be applied to both mono and stereo camera based 3D pose estimation and tracking. The former guarantees higher frame rates with about 1 ms of local pose estimation, while the latter assures of more precise pose results but with about 16 ms of local pose estimation. The experimental investigations have shown the effectiveness of the proposed approach with real-time performance.

State-of-the-art methods for 3D hand pose estimation from depth images require large amounts of annotated training data. We propose to model the statistical relationships of 3D hand poses and corresponding depth images using two deep generative models with a shared latent space. By design, our architecture allows for learning from unlabeled image data in a semi-supervised manner. Assuming a one-to-one mapping between a pose and a depth map, any given point in the shared latent space can be projected into both a hand pose and a corresponding depth map. Regressing the hand pose can then be done by learning a discriminator to estimate the posterior of the latent pose given some depth map. To improve generalization and to better exploit unlabeled depth maps, we jointly train a generator and a discriminator. At each iteration, the generator is updated with the back-propagated gradient from the discriminator to synthesize realistic depth maps of the articulated hand, while the discriminator benefits from an augmented training set of synthesized and unlabeled samples. The proposed discriminator network architecture is highly efficient and runs at 90ḞPS on the CPU with accuracies comparable or better than state-of-art on 3 publicly available benchmarks.

In this paper, we first review the approaches to recover 3D shape and related movements of a human and then we present an easy and reliable approach to recover a 3D model using just one image or monocular video sequence. A simplification of the perspective camera model is required, due to the absence of stereo view. The human figure is reconstructed in a skeleton form and to improve the visual quality, a pre-defined human model is also fitted to the recovered 3D data.

With the presence of ubiquitous devices in our daily lives, effectively capturing and managing user attention becomes a critical device requirement. While gaze-tracking is typically employed to determine the user’s focus of attention, gaze-lock detection to sense eyecontact with a device is proposed in [16]. This work proposes eye contact detection using deep neural networks, and makes the following contributions: (1) With a convolutional neural network (CNN) architecture, we achieve superior eye-contact detection performance as compared to [16] with minimal data pre-processing ; our algorithm is furthermore validated on multiple datasets, (2) Gaze-lock detection is improved by combining head pose and eye-gaze information consistent with social attention literature, and (3) We demonstrate gaze-locking on an Android mobile platform via CNN model compression.

The present study investigated the ability to inter actively locate points in a three dimensional computer environment using a six degree of freedom input device. Four different visu al feedback modes were tested: fixed viewpoint monoscopic persp ective, fixed viewpoint stereoscopic perspective, head-tracked mo noscopic perspective and head-tracked stereoscopic perspecti ve. Targets were located at plus and minus 10cm along the X, Y or Z axes, from a fixed starting location. Data about the time to complete the task and positioning accuracy (error) are gathered for each trial. In addition, subjective feedback regarding the apparat us, visual mode and task difficulty was solicited from subject s. The results indicate that stereoscopic performance is superior to monoscopic performance and that asymmetries exist both across and within axes. Head tracking had no appreciable effect upon performance. Subjective feedback regarding performance is usuall y consistent with objective measures, but some inconsistencies a re present.

There have been increasing demands for developing robotic system combining camera and inertial measurement unit in navigation task, due to their low-cost, lightweight and complementary properties. In this paper, we present a Visual Inertial Odometry (VIO) system which can utilize sparse depth to estimate 6D pose in GPS-denied and unstructured environments. The system is based on Multi-State Constraint Kalman Filter (MSCKF), which benefits from low computation load when compared to optimization-based method, especially on resource-constrained platform. Features are enhanced with depth information forming 3D landmark position measurements in space, which reduces uncertainty of position estimate. And we derivate measurement model to access compatibility with both 2D and 3D measurements. In experiments, we evaluate the performance of the system in different in-flight scenarios, both cluttered room and industry environment. The results suggest that the estimator is consistent, substantially improves the accuracy compared with original monocular-based MSKCF and achieves competitive accuracy with other research.

Person-independent and pose-invariant estimation of eye-gaze is important for situation analysis and for automated video annotation. We propose a fast cascade regression based method that first estimates the location of a dense set of markers and their visibility, then reconstructs face shape by fitting a part-based 3D model. Next, the reconstructed 3D shape is used to estimate a canonical view of the eyes for 3D gaze estimation. The model operates in a feature space that naturally encodes local ordinal properties of pixel intensities leading to photometric invariant estimation of gaze. To evaluate the algorithm in comparison with alternative approaches, three publicly-available databases were used, Boston University Head Tracking, Multi-View Gaze and CAVE Gaze datasets. Precision for head pose and gaze averaged 4 degrees or less for pitch, yaw, and roll. The algorithm outperformed alternative methods in both datasets.