Question: Variational sequential labelers in semi-supervised learning scenarios.

This paper introduces a unified neural network framework and learning methodology capable of addressing a wide range of natural language processing tasks, such as part-of-speech tagging, chunking, named entity recognition, and semantic role labeling. Our approach eschews task-specific engineering and the reliance on extensive prior knowledge, aiming for a more generalized and adaptable system. Rather than utilizing manually crafted features tailored to each individual task, our model learns to generate its own internal representations through exposure to large volumes of primarily unlabeled data. This strategy allows the system to automatically discover and utilize the most salient linguistic features for various tasks. As a practical outcome, we have developed a freely accessible tagging system that delivers strong performance while maintaining low computational demands, making it suitable for deployment in resource-constrained environments. This system's effectiveness and efficiency demonstrate the potential of leveraging deep learning techniques for natural language processing tasks without the need for extensive task-specific feature engineering.

Auto-encoder is a popular representation learning technique which can capture the generative model of data via a encoding and decoding procedure typically driven by reconstruction errors in an unsupervised way. In this paper, we propose a semi-supervised manifold learning based auto-encoder (named semAE). semAE is based on a regularized auto-encoder framework which leverages semi-supervised manifold learning to impose regularization based on the encoded representation. Our proposed approach suits more practical scenarios in which a small number of labeled data are available in addition to a large number of unlabeled data. Experiments are conducted on several well-known benchmarking datasets to validate the efficacy of semAE from the aspects of both representation and classification. The comparisons to state-of-the-art representation learning methods on classification performance in semi-supervised settings demonstrate the superiority of our approach.

This paper discusses the use of unlabeled examples for the problem of named entity classification. A large number of rules is needed for coverage of the domain, suggesting that a fairly large number of labeled examples should be required to train a classifier. However, we show that the use of unlabeled data can reduce the requirements for supervision to just 7 simple “seed” rules. The approach gains leverage from natural redundancy in the data: for many named-entity instances both the spelling of the name and the context in which it appears are sufficient to determine its type. We present two algorithms. The first method uses a similar algorithm to that of (Yarowsky 95), with modifications motivated by (Blum and Mitchell 98). The second algorithm extends ideas from boosting algorithms, designed for supervised learning tasks, to the framework suggested by (Blum and Mitchell 98).

We develop a fully discriminative learning approach for supervised Latent Dirichlet Allocation (LDA) model, which maximizes the posterior probability of the prediction variable given the input document. Different from traditional variational learning or Gibbs sampling approaches, the proposed learning method applies (i) the mirror descent algorithm for exact maximum a posterior inference and (ii) back propagation with stochastic gradient descent for model parameter estimation, leading to scalable learning of the model in an end-to-end discriminative manner. As a byproduct, we also apply this technique to develop a new learning method for the traditional unsupervised LDA model. Experimental results on two real-world regression and classification tasks show that the proposed methods significantly outperform the previous supervised/unsupervised LDA learning methods.

In many machine learning settings, labeled examples are difficult to collect while unlabeled data are abundant. Also, for some binary classification problems, positive examples which are elements of the target concept are available. Can these additional data be used to improve accuracy of supervised learning algorithms? We investigate in this paper the design of learning algorithms from positive and unlabeled data only. Many machine learning and data mining algorithms, such as decision tree induction algorithms and naive Bayes algorithms, use examples only to evaluate statistical queries (SQ-like algorithms). Kearns designed the statistical query learning model in order to describe these algorithms. Here, we design an algorithm scheme which transforms any SQ-like algorithm into an algorithm based on positive statistical queries (estimate for probabilities over the set of positive instances) and instance statistical queries (estimate for probabilities over the instance space). We prove that any class learnable in the statistical query learning model is learnable from positive statistical queries and instance statistical queries only if a lower bound on the weight of any target concept f can be estimated in polynomial time. Then, we design a decision tree induction algorithm POSC4.5, based on C4.5, that uses only positive and unlabeled examples and we give experimental results for this algorithm. In the case of imbalanced classes in the sense that one of the two classes (say the positive class) is heavily underrepresented compared to the other class, the learning problem remains open. This problem is challenging because it is encountered in many real-world applications. © 2005 Elsevier B.V. All rights reserved.

Many problems in real-world applications involve predicting several random variables that are statistically related. Markov random fields (MRFs) are a great mathematical tool to encode such dependencies. The goal of this paper is to combine MRFs with deep learning to estimate complex representations while taking into account the dependencies between the output random variables. Towards this goal, we propose a training algorithm that is able to learn structured models jointly with deep features that form the MRF potentials. Our approach is efficient as it blends learning and inference and makes use of GPU acceleration. We demonstrate the effectiveness of our algorithm in the tasks of predicting words from noisy images, as well as tagging of Flickr photographs. We show that joint learning of the deep features and the MRF parameters results in significant performance gains.

Many approaches to Bayesian image segmentation have used maximum a posteriori (MAP) estimation in conjunction with Markov random fields (MRF). Although this approach performs well, it has a number of disadvantages. In particular, exact MAP estimates cannot be computed, approximate MAP estimates are computationally expensive to compute, and unsupervised parameter estimation of the MRF is difficult. The authors propose a new approach to Bayesian image segmentation that directly addresses these problems. The new method replaces the MRF model with a novel multiscale random field (MSRF) and replaces the MAP estimator with a sequential MAP (SMAP) estimator derived from a novel estimation criteria. Together, the proposed estimator and model result in a segmentation algorithm that is not iterative and can be computed in time proportional to MN where M is the number of classes and N is the number of pixels. The also develop a computationally efficient method for unsupervised estimation of model parameters. Simulations on synthetic images indicate that the new algorithm performs better and requires much less computation than MAP estimation using simulated annealing. The algorithm is also found to improve classification accuracy when applied to the segmentation of multispectral remotely sensed images with ground truth data.

Recent approaches in classifying evolving data streams are based on supervised learning algorithms, which can be trained with labeled data only. Manual labeling of data is both costly and time consuming. Therefore, in a real streaming environment, where huge volumes of data appear at a high speed, labeled data may be very scarce. Thus, only a limited amount of training data may be available for building the classification models, leading to poorly trained classifiers. We apply a novel technique to overcome this problem by building a classification model from a training set having both unlabeled and a small amount of labeled instances. This model is built as micro-clusters using semi-supervised clustering technique and classification is performed with kappa-nearest neighbor algorithm. An ensemble of these models is used to classify the unlabeled data. Empirical evaluation on both synthetic data and real botnet traffic reveals that our approach, using only a small amount of labeled data for training, outperforms state-of-the-art stream classification algorithms that use twenty times more labeled data than our approach.

Recently, several works in the domain of natural language processing presented successful methods for word embedding. Among them, the Skip-Gram with negative sampling, known also as word2vec, advanced the state-of-the-art of various linguistics tasks. In this paper, we propose a scalable Bayesian neural word embedding algorithm. The algorithm relies on a Variational Bayes solution for the SkipGram objective and a detailed step by step description is provided. We present experimental results that demonstrate the performance of the proposed algorithm for word analogy and similarity tasks on six different datasets and show it is competitive with the original Skip-Gram method.

Many supervised learning problems involve learning from samples whose labels are corrupted in some way. For example, each label may be flipped with some constant probability (learning with label noise), or one may have a pool of unlabelled samples in lieu of negative samples (learning from positive and unlabelled data). This paper uses class-probability estimation to study these and other corruption processes belonging to the mutually contaminated distributions framework (Scott et al., 2013), with three conclusions. First, one can optimise balanced error and AUC without knowledge of the corruption parameters. Second, given estimates of the corruption parameters, one can minimise a range of classification risks. Third, one can estimate corruption parameters via a class-probability estimator (e.g. kernel logistic regression) trained solely on corrupted data. Experiments on label noise tasks corroborate our analysis. 1. Learning from corrupted binary labels In many practical scenarios involving learning from binary labels, one observes samples whose labels are corrupted versions of the actual ground truth. For example, in learning from class-conditional label noise (CCN learning), the labels are flipped with some constant probability (Angluin & Laird, 1988). In positive and unlabelled learning (PU learning), we have access to some positive samples, but in lieu of negative samples only have a pool of samples whose label is unknown (Denis, 1998). More generally, suppose there is a notional clean distribution D over instances and labels. We say a problem involves learning from corrupted Proceedings of the 32 International Conference on Machine Learning, Lille, France, 2015. JMLR: W&CP volume 37. Copyright 2015 by the author(s). binary labels if we observe training samples drawn from some corrupted distribution Dcorr such that the observed labels do not represent those we would observe under D. A fundamental question is whether one can minimise a given performance measure with respect to D, given access only to samples from Dcorr. Intuitively, in general this requires knowledge of the parameters of the corruption process that determines Dcorr. This yields two further questions: are there measures for which knowledge of these corruption parameters is unnecessary, and for other measures, can we estimate these parameters? In this paper, we consider corruption problems belonging to the mutually contaminated distributions framework (Scott et al., 2013). We then study the above questions through the lens of class-probability estimation, with three conclusions. First, optimising balanced error (BER) as-is on corrupted data equivalently optimises BER on clean data, and similarly for the area under the ROC curve (AUC). That is, these measures can be optimised without knowledge of the corruption process parameters; further, we present evidence that these are essentially the only measures with this property. Second, given estimates of the corruption parameters, a range of classification measures can be minimised by thresholding corrupted class-probabilities. Third, under some assumptions, these corruption parameters may be estimated from the range of the corrupted class-probabilities. For all points above, observe that learning requires only corrupted data. Further, corrupted class-probability estimation can be seen as treating the observed samples as if they were uncorrupted. Thus, our analysis gives justification (under some assumptions) for this apparent heuristic in problems such as CCN and PU learning. While some of our results are known for the special cases of CCN and PU learning, our interest is in determining to what extent they generalise to other label corruption problems. This is a step towards a unified treatment of these problems. We now fix notation and formalise the problem. Learning from Corrupted Binary Labels via Class-Probability Estimation 2. Background and problem setup Fix an instance space X. We denote byD some distribution over X × {±1}, with (X,Y) ∼ D a pair of random variables. Any D may be expressed via the class-conditional distributions (P,Q) = (P(X | Y = 1),P(X | Y = −1)) and base rate π = P(Y = 1), or equivalently via marginal distribution M = P(X) and class-probability function η : x 7→ P(Y = 1 | X = x). When referring to these constituent distributions, we write D as DP,Q,π or DM,η. 2.1. Classifiers, scorers, and risks A classifier is any function f : X→ {±1}. A scorer is any function s : X → R. Many learning methods (e.g. SVMs) output a scorer, from which a classifier is formed by thresholding about some t ∈ R. We denote the resulting classifier by thresh(s, t) : x 7→ sign(s(x)− t). The false positive and false negative rates of a classifier f are denoted FPR(f),FNR(f), and are defined by P X∼Q (f(X) = 1) and P X∼P (f(X) = −1) respectively. Given a function Ψ: [0, 1] → [0, 1], a classification performance measure ClassΨ : {±1} → [0, 1] assesses the performance of a classifier f via (Narasimhan et al., 2014) ClassΨ(f) = Ψ(FPR (f),FNR(f), π). A canonical example is the misclassification error, where Ψ: (u, v, p) 7→ p · v+ (1− p) · u. Given a scorer s, we use ClassΨ(s; t) to refer to Class D Ψ(thresh(s, t)). The Ψ-classification regret of a classifier f : X→ {±1} is regretΨ(f) = Class D Ψ(f)− inf g : X→{±1} ClassΨ(g). A loss is any function ` : {±1} ×R→ R+. Given a distribution D, the `-risk of a scorer s is defined as L` (s) = E (X,Y)∼D [`(Y, s(X))] . (1) The `-regret of a scorer, regret` , is as per the Ψ-regret. We say ` is strictly proper composite (Reid & Williamson, 2010) if argmins L` (s) is some strictly monotone transformation ψ of η, i.e. we can recover class-probabilities from the optimal prediction via the link function ψ. We call class-probability estimation (CPE) the task of minimising Equation 1 for some strictly proper composite `. The conditional Bayes-risk of a strictly proper composite ` is L` : η 7→ η`1(ψ(η)) + (1 − η)`−1(ψ(η)). We call ` strongly proper composite with modulus λ if L` is λstrongly concave (Agarwal, 2014). Canonical examples of such losses are the logistic and exponential loss, as used in logistic regression and AdaBoost respectively. Quantity Clean Corrupted Joint distribution D Corr(D,α, β, πcorr) or Dcorr Class-conditionals P,Q Pcorr, Qcorr Base rate π πcorr Class-probability η ηcorr Ψ-optimal threshold tΨ t D corr,Ψ Table 1. Common quantities on clean and corrupted distributions. 2.2. Learning from contaminated distributions Suppose DP,Q,π is some “clean” distribution where performance will be assessed. (We do not assume that D is separable.) In MC learning (Scott et al., 2013), we observe samples from some corrupted distribution Corr(D,α, β, πcorr) over X × {±1}, for some unknown noise parameters α, β ∈ [0, 1] with α + β < 1; where the parameters are clear from context, we occasionally refer to the corrupted distribution as Dcorr. The corrupted classconditional distributions Pcorr, Qcorr are Pcorr = (1− α) · P + α ·Q Qcorr = β · P + (1− β) ·Q, (2) and the corrupted base rate πcorr in general has no relation to the clean base rate π. (If α+β = 1, then Pcorr = Qcorr, making learning impossible, whereas if α+ β > 1, we can swap Pcorr, Qcorr.) Table 1 summarises common quantities on the clean and corrupted distributions. From (2), we see that none of Pcorr, Qcorr or πcorr contain any information about π in general. Thus, estimating π from Corr(D,α, β, πcorr) is impossible in general. The parameters α, β are also non-identifiable, but can be estimated under some assumptions on D (Scott et al., 2013). 2.3. Special cases of MC learning Two special cases of MC learning are notable. In learning from class-conditional label noise (CCN learning) (Angluin & Laird, 1988), positive samples have labels flipped with probability ρ+, and negative samples with probability ρ−. This can be shown to reduce to MC learning with α = π−1 corr · (1− π) · ρ− , β = (1− πcorr) · π · ρ+, (3) and the corrupted base rate πcorr = (1−ρ+)·π+ρ−·(1−π). (See Appendix C for details.) In learning from positive and unlabelled data (PU learning) (Denis, 1998), one has access to unlabelled samples in lieu of negative samples. There are two subtly different settings: in the case-controlled setting (Ward et al., 2009), the unlabelled samples are drawn from the marginal distribution M , corresponding to MC learning with α = 0, β = π, Learning from Corrupted Binary Labels via Class-Probability Estimation and πcorr arbitrary. In the censoring setting (Elkan & Noto, 2008), observations are drawn from D followed by a label censoring procedure. This is in fact a special of CCN (and hence MC) learning with ρ− = 0. 3. BER and AUC are immune to corruption We first show that optimising balanced error and AUC on corrupted data is equivalent to doing so on clean data. Thus, with a suitably rich function class, one can optimise balanced error and AUC from corrupted data without knowledge of the corruption process parameters. 3.1. BER minimisation is immune to label corruption The balanced error (BER) (Brodersen et al., 2010) of a classifier is simply the mean of the per-class error rates, BER(f) = FPR(f) + FNR(f) 2 . This is a popular measure in imbalanced learning problems (Cheng et al., 2002; Guyon et al., 2004) as it penalises sacrificing accuracy on the rare class in favour of accuracy on the dominant class. The negation of the BER is also known as the AM (arithmetic mean) metric (Menon et al., 2013). The BER-optimal classifier thresholds the class-probability function at the base rate (Menon et al., 2013), so that: argmin f : X→{±1} BER(f) = thresh(η, π) (4) argmin f : X→{±1} BERcorr(f) = thresh(ηcorr, πcorr), (5) where ηcorr denotes the corrupted class-probability function. As Equation 4 depends on π, it may appear that one must know π to minimise the clean BER from corrupted data. Surprisingly, the BER-optimal classifiers in Equations 4 and 5 coincide. This is because of the following relationship between

Our inspiration comes from Nell (Never Ending Language Learning), a computer program running at Carnegie Mellon University to extract structured information from unstructured web pages. We consider the problem of semi-supervised learning approach to extract category instances (e.g. country(USA), city(New York)) from web pages, starting with a handful of labeled training examples of each category or relation, plus hundreds of millions of unlabeled web documents. Semisupervised approaches using a small number of labeled examples together with many unlabeled examples are often unreliable as they frequently produce an internally consistent, but nevertheless, incorrect set of extractions. We believe that this problem can be overcome by simultaneously learning independent classifiers in a new approach named Coupled Bayesian Sets algorithm, based on Bayesian Sets, for many different categories and relations (in the presence of an ontology defining constraints that couple the training of these classifiers). Experimental results show that simultaneously learning a coupled collection of classifiers for random 11 categories resulted in much more accurate extractions than training classifiers through original Bayesian Sets algorithm, Naive Bayes, BaS-all and Coupled Pattern Learner (the category extractor used in NELL).


Question: Variational sequential labelers in semi-supervised learning scenarios.

This paper focuses on the task of part-of-speech tagging for English content sourced from Twitter, the widely-used microblogging platform. We have devised a specialized tagset, manually annotated a dataset, crafted a set of features, and achieved tagging accuracy approaching 90%. In an effort to facilitate more sophisticated textual analysis of Twitter and similar social media datasets, we have released both the annotated data and the tagging tools to the academic community. Our work lays the groundwork for enhanced linguistic processing of informal and rapidly evolving online language, contributing to the broader field of natural language processing.

Many modern unsupervised or semi-supervised machine learning algorithms rely on Bayesian probabilistic models. These models are usually intractable and thus require approximate inference. Variational inference (VI) lets us approximate a high-dimensional Bayesian posterior with a simpler variational distribution by solving an optimization problem. This approach has been successfully used in various models and large-scale applications. In this review, we give an overview of recent trends in variational inference. We first introduce standard mean field variational inference, then review recent advances focusing on the following aspects: (a) scalable VI, which includes stochastic approximations, (b) generic VI, which extends the applicability of VI to a large class of otherwise intractable models, such as non-conjugate models, (c) accurate VI, which includes variational models beyond the mean field approximation or with atypical divergences, and (d) amortized VI, which implements the inference over local latent variables with inference networks. Finally, we provide a summary of promising future research directions.

Our inspiration comes from Nell (Never Ending Language Learning), a computer program running at Carnegie Mellon University to extract structured information from unstructured web pages. We consider the problem of semi-supervised learning approach to extract category instances (e.g. country(USA), city(New York)) from web pages, starting with a handful of labeled training examples of each category or relation, plus hundreds of millions of unlabeled web documents. Semisupervised approaches using a small number of labeled examples together with many unlabeled examples are often unreliable as they frequently produce an internally consistent, but nevertheless, incorrect set of extractions. We believe that this problem can be overcome by simultaneously learning independent classifiers in a new approach named Coupled Bayesian Sets algorithm, based on Bayesian Sets, for many different categories and relations (in the presence of an ontology defining constraints that couple the training of these classifiers). Experimental results show that simultaneously learning a coupled collection of classifiers for random 11 categories resulted in much more accurate extractions than training classifiers through original Bayesian Sets algorithm, Naive Bayes, BaS-all and Coupled Pattern Learner (the category extractor used in NELL).

Graph-based semi-supervised learning improves classification by combining labeled and unlabeled data through label propagation. It was shown that the sparse representation of graph by weighted local neighbors provides a better similarity measure between data points for label propagation. However, selecting local neighbors can lead to disjoint components and incorrect neighbors in graph, and thus, fail to capture the underlying global structure. In this paper, we propose to learn a nonnegative low-rank graph to capture global linear neighborhoods, under the assumption that each data point can be linearly reconstructed from weighted combinations of its direct neighbors and reachable indirect neighbors. The global linear neighborhoods utilize information from both direct and indirect neighbors to preserve the global cluster structures, while the low-rank property retains a compressed representation of the graph. An efficient algorithm based on a multiplicative update rule is designed to learn a nonnegative low-rank factorization matrix minimizing the neighborhood reconstruction error. Large scale simulations and experiments on UCI datasets and high-dimensional gene expression datasets showed that label propagation based on global linear neighborhoods captures the global cluster structures better and achieved more accurate classification results.

We present a general theoretical analysis of structured prediction. By introducing a new complexity measure that explicitly factors in the structure of the output space and the loss function, we are able to derive new data-dependent learning guarantees for a broad family of losses and for hypothesis sets with an arbitrary factor graph decomposition. To the best of our knowledge, these are both the most favorable and the most general guarantees for structured prediction (and multiclass classification) currently known. We also extend this theory by leveraging the principle of Voted Risk Minimization (VRM) and showing that learning is possible with complex factor graphs. We both present new learning bounds in this advanced setting as well as derive two new families of algorithms, Voted Conditional Random Field and Voted Structured Boosting, which can make use of very complex features and factor graphs without overfitting. Finally, we validate our theory through experiments on several datasets.

Ensemble methods have been shown to be an effectiv e tool for solving multi-label classification tasks. In the RAndom k-labELsets (RAKEL) algorithm, each member of the en semble is associated with a small randomly-selected subset of k labels. Then, a single label classifier is trained according to each combination of elements in the subset. In this paper we adopt a similar approach, however, instead of rando mly choosing subsets, we select the minimum required su bsets of k labels that cover all labels and meet additional constraints such as coverage of inter-label correla tions. Construction of the cover is achieved by form ulating the subset selection as a minimum set covering prob lem (SCP) and solving it by using approximation algo rithms. Every cover needs only to be prepared once by offline algorithms. Once prepared, a cover may b e applied to the classification of any given multi-la bel dataset whose properties conform with those of the cover. The contribution of this paper is two-fold. Fir st, we introduce SCP as a general framework for con structing label covers while allowing the user to incorpo rate cover construction constraints. We demonstrate he effectiveness of this framework by proposing two co nstruction constraints whose enforcement produces c overs that improve the prediction performance of ran dom selection. Second, we provide theoretical bound s that quantify the probabilities of random selection t produce covers that meet the proposed construct ion riteria. The experimental results indicate that the p roposed methods improve multi-label classification accuracy and stability compared with the RAKEL algorithm a nd to other state-of-the-art algorithms.

Deep networks are successfully used as classification models yielding state-ofthe-art results when trained on a large number of labeled samples. These models, however, are usually much less suited for semi-supervised problems because of their tendency to overfit easily when trained on small amounts of data. In this work we will explore a new training objective that is targeting a semi-supervised regime with only a small subset of labeled data. This criterion is based on a deep metric embedding over distance relations within the set of labeled samples, together with constraints over the embeddings of the unlabeled set. The final learned representations are discriminative in euclidean space, and hence can be used with subsequent nearest-neighbor classification using the labeled samples.

Variational methods are widely used for approximate posterior inference. However, their use is typically limited to families of distributions that enjoy particular conjugacy properties. To circumvent this limitation, we propose a family of variational approximations inspired by nonparametric kernel density estimation. The locations of these kernels and their bandwidth are treated as variational parameters and optimized to improve an approximate lower bound on the marginal likelihood of the data. Unlike most other variational approximations, using multiple kernels allows the approximation to capture multiple modes of the posterior. We demonstrate the efficacy of the nonparametric approximation with a hierarchical logistic regression model and a nonlinear matrix factorization model. We obtain predictive performance as good as or better than more specialized variational methods and MCMC approximations. The method is easy to apply to graphical models for which standard variational methods are difficult to derive.

We develop a fully discriminative learning approach for supervised Latent Dirichlet Allocation (LDA) model, which maximizes the posterior probability of the prediction variable given the input document. Different from traditional variational learning or Gibbs sampling approaches, the proposed learning method applies (i) the mirror descent algorithm for exact maximum a posterior inference and (ii) back propagation with stochastic gradient descent for model parameter estimation, leading to scalable learning of the model in an end-to-end discriminative manner. As a byproduct, we also apply this technique to develop a new learning method for the traditional unsupervised LDA model. Experimental results on two real-world regression and classification tasks show that the proposed methods significantly outperform the previous supervised/unsupervised LDA learning methods.

Many problems in real-world applications involve predicting several random variables that are statistically related. Markov random fields (MRFs) are a great mathematical tool to encode such dependencies. The goal of this paper is to combine MRFs with deep learning to estimate complex representations while taking into account the dependencies between the output random variables. Towards this goal, we propose a training algorithm that is able to learn structured models jointly with deep features that form the MRF potentials. Our approach is efficient as it blends learning and inference and makes use of GPU acceleration. We demonstrate the effectiveness of our algorithm in the tasks of predicting words from noisy images, as well as tagging of Flickr photographs. We show that joint learning of the deep features and the MRF parameters results in significant performance gains.

In multilabel learning, each instance in the training set is associated with a set of labels and the task is to output a label set whose size is unknown a priori for each unseen instance. In this paper, this problem is addressed in the way that a neural network algorithm named BP-MLL, i.e., backpropagation for multilabel learning, is proposed. It is derived from the popular backpropagation algorithm through employing a novel error function capturing the characteristics of multilabel learning, i.e., the labels belonging to an instance should be ranked higher than those not belonging to that instance. Applications to two real-world multilabel learning problems, i.e., functional genomics and text categorization, show that the performance of BP-MLL is superior to that of some well-established multilabel learning algorithms


Question: Variational sequential labelers in semi-supervised learning scenarios.

This paper explores a fundamental goal in statistical inference and learning: uncovering the underlying structure within a set of patterns. To achieve this, we propose constructing a stochastic generative model with parameters that can independently generate these patterns. However, for all but the simplest models, the number of ways a pattern can be generated grows exponentially, making it computationally infeasible to adjust the parameters to directly maximize the probability of the observed patterns. To circumvent this combinatorial explosion, we introduce a novel approach that maximizes a lower bound on the probability of the observations that can be easily computed. This method can be interpreted as a form of hierarchical self-supervised learning, potentially mirroring the function of bottom-up and top-down pathways in cortical processing. Specifically, our technique allows for the refinement of model parameters in a way that optimally explains the observed data, even in scenarios with complex generative processes. This self-supervised learning framework not only addresses the computational challenges associated with maximizing likelihood in intricate models but also provides a theoretical foundation for understanding the interplay between bottom-up and top-down information processing in biological systems.

Multi-label classification methods are increasingly required by modern applications, such as protein function classification, music categorization, and semantic scene classification. This article introduces the task of multi-label classification, organizes the sparse related literature into a structured presentation and performs comparative experimental results of certain multi-label classification methods. It also contributes the definition of concepts for the quantification of the multi-label nature of a data set.

We marry ideas from deep neural networks and approximate Bayesian inference to derive a generalised class of deep, directed generative models, endowed with a new algorithm for scalable inference and learning. Our algorithm introduces a recognition model to represent an approximate posterior distribution and uses this for optimisation of a variational lower bound. We develop stochastic backpropagation – rules for gradient backpropagation through stochastic variables – and derive an algorithm that allows for joint optimisation of the parameters of both the generative and recognition models. We demonstrate on several real-world data sets that by using stochastic backpropagation and variational inference, we obtain models that are able to generate realistic samples of data, allow for accurate imputations of missing data, and provide a useful tool for high-dimensional data visualisation.

In this paper, we theoretically study the problem of binary classification in the presence of random classification noise — the learner, instead of seeing the true labels, sees labels that have independently been flipped with some small probability. Moreover, random label noise is class-conditional — the flip probability depends on the class. We provide two approaches to suitably modify any given surrogate loss function. First, we provide a simple unbiased estimator of any loss, and obtain performance bounds for empirical risk minimization in the presence of iid data with noisy labels. If the loss function satisfies a simple symmetry condition, we show that the method leads to an efficient algorithm for empirical minimization. Second, by leveraging a reduction of risk minimization under noisy labels to classification with weighted 0-1 loss, we suggest the use of a simple weighted surrogate loss, for which we are able to obtain strong empirical risk bounds. This approach has a very remarkable consequence — methods used in practice such as biased SVM and weighted logistic regression are provably noise-tolerant. On a synthetic non-separable dataset, our methods achieve over 88% accuracy even when 40% of the labels are corrupted, and are competitive with respect to recently proposed methods for dealing with label noise in several benchmark datasets.

We propose a hypergraph-based active learning scheme which we term HS; HS generalizes the previously reported algorithm S originally proposed for graph-based active learning with pointwise queries [1]. Our HS method can accommodate hypergraph structures and allows one to ask both pointwise queries and pairwise queries. Based on a novel parametric system particularly designed for hypergraphs, we derive theoretical results on the query complexity of HS for the above described generalized settings. Both the theoretical and empirical results show that HS requires a significantly fewer number of queries than S when one uses S over a graph obtained from the corresponding hypergraph via clique expansion.

We propose a generative model for text and other collections of discrete data that generalizes or improves on several previous models including naive Bayes/unigram, mixture of unigrams [6], and Hofmann's aspect model , also known as probabilistic latent semantic indexing (pLSI) [3]. In the context of text modeling, our model posits that each document is generated as a mixture of topics, where the continuous-valued mixture proportions are distributed as a latent Dirichlet random variable. Inference and learning are carried out efficiently via variational algorithms. We present empirical results on applications of this model to problems in text modeling, collaborative filtering, and text classification.

The recently proposed Temporal Ensembling has achieved state-of-the-art results in several semi-supervised learning benchmarks. It maintains an exponential moving average of label predictions on each training example, and penalizes predictions that are inconsistent with this target. However, because the targets change only once per epoch, Temporal Ensembling becomes unwieldy when learning large datasets. To overcome this problem, we propose Mean Teacher, a method that averages model weights instead of label predictions. As an additional benefit, Mean Teacher improves test accuracy and enables training with fewer labels than Temporal Ensembling. Without changing the network architecture, Mean Teacher achieves an error rate of 4.35% on SVHN with 250 labels, outperforming Temporal Ensembling trained with 1000 labels. We also show that a good network architecture is crucial to performance. Combining Mean Teacher and Residual Networks, we improve the state of the art on CIFAR-10 with 4000 labels from 10.55% to 6.28%, and on ImageNet 2012 with 10% of the labels from 35.24% to 9.11%.

Sequence-discriminative training of deep neural networks (DNNs) is investigated on a standard 300 hour American English conversational telephone speech task. Different sequencediscriminative criteria — maximum mutual information (MMI), minimum phone error (MPE), state-level minimum Bayes risk (sMBR), and boosted MMI — are compared. Two different heuristics are investigated to improve the performance of the DNNs trained using sequence-based criteria — lattices are regenerated after the first iteration of training; and, for MMI and BMMI, the frames where the numerator and denominator hypotheses are disjoint are removed from the gradient computation. Starting from a competitive DNN baseline trained using cross-entropy, different sequence-discriminative criteria are shown to lower word error rates by 7-9% relative, on average. Little difference is noticed between the different sequencebased criteria that are investigated. The experiments are done using the open-source Kaldi toolkit, which makes it possible for the wider community to reproduce these results.

We address the problem of learning classifiers when observat ions have multiple views, some of which may not be observed for all examples. We a ssume the existence of view generating functions which may complete t he missing views in an approximate way. This situation corresponds for examp le to learning text classifiers from multilingual collections where documents are not available in all languages. In that case, Machine Translation (MT) systems m ay be used to translate each document in the missing languages. We derive a gene ralization error bound for classifiers learned on examples with multiple arti ficially created views. Our result uncovers a trade-off between the size of the train ing set, the number of views, and the quality of the view generating functions. A s a consequence, we identify situations where it is more interesting to use mu ltiple views for learning instead of classical single view learning. An extension of this framework is a natural way to leverage unlabeled multi-view data in semisupervised learning. Experimental results on a subset of the Reuters RCV1/RCV2 co llecti ns support our findings by showing that additional views obtained from M T may significantly improve the classification performance in the cases identifi d by our trade-off.

We study the problem of hierarchical classification when labels corresponding to partial and/or multiple paths in the underlying taxonomy are allowed. We introduce a new hierarchical loss function, the H-loss, implementing the simple intuition that additional mistakes in the subtree of a mistaken class should not be charged for. Based on a probabilistic data model introduced in earlier work, we derive the Bayes-optimal classifier for the H-loss. We then empirically compare two incremental approximations of the Bayes-optimal classifier with a flat SVM classifier and with classifiers obtained by using hierarchical versions of the Perceptron and SVM algorithms. The experiments show that our simplest incremental approximation of the Bayes-optimal classifier performs, after just one training epoch, nearly as well as the hierarchical SVM classifier (which performs best). For the same incremental algorithm we also derive an H-loss bound showing, when data are generated by our probabilistic data model, exponentially fast convergence to the H-loss of the hierarchical classifier based on the true model parameters.

Learning from positive and unlabeled data or PU learning is the setting where a learner only has access to positive examples and unlabeled data. The assumption is that the unlabeled data can contain both positive and negative examples. This setting has attracted increasing interest within the machine learning literature as this type of data naturally arises in applications such as medical diagnosis and knowledge base completion. This article provides a survey of the current state of the art in PU learning. It proposes seven key research questions that commonly arise in this field and provides a broad overview of how the field has tried to address them.


Question: A method for performing one-sided unsupervised domain mapping.

This paper explores the challenge of recognizing and extracting text, specifically digits, from natural images, a task critical for various emerging applications in computer vision. While document character recognition, particularly for handwritten digits, has been extensively studied and effectively solved for practical uses, the recognition of characters in more intricate scenes, such as those captured in photographs, remains significantly more complex. Despite advancements, current methods fall short of human-level performance in such scenarios. To address this gap, we focus on an application that requires the reading of house numbers from street-level images, employing unsupervised feature learning techniques. In support of this research, we introduce a comprehensive benchmark dataset, comprising over 600,000 labeled digits extracted from Street View images. We highlight the limitations of traditional approaches using hand-designed features in tackling this problem. Subsequently, we apply and evaluate two recent unsupervised feature learning methods, which prove to be markedly superior in our comparative benchmarks. This work underscores the potential of unsupervised feature learning in advancing the field of character recognition in natural images.

We study the problem of learning domain invariant representations for time series data while transferring the complex temporal latent dependencies between domains. Our model termed as Variational Recurrent Adversarial Deep Domain Adaptation (VRADA) is built atop a variational recurrent neural network (VRNN) and trains adversarially to capture complex temporal relationships that are domain-invariant. This is (as far as we know) the first to capture and transfer temporal latent dependencies of multivariate time-series data. Through experiments on real-world multivariate healthcare time-series datasets, we empirically demonstrate that learning temporal dependencies helps our model’s ability to create domain-invariant representations, allowing our model to outperform current state-of-the-art deep domain adaptation approaches.

In this paper, we present an overview of generalized expectation criteria (GE), a simple, robust, scalable method for semi-supervised training using weakly -labeled data. GE fits model parameters by favoring models that match certain expectation constrai nts, such as marginal label distributions, on the unlabeled data. This paper shows how to apply generali z d expectation criteria to two classes of parametric models: maximum entropy models and condition al random fields. Experimental results demonstrate accuracy improvements over supervise d training and a number of other stateof-the-art semi-supervised learning methods for these mod els.

Given a pre-trained CNN without any testing samples, this paper proposes a simple yet effective method to diagnose feature representations of the CNN. We aim to discover representation flaws caused by potential dataset bias. More specifically, when the CNN is trained to estimate image attributes, we mine latent relationships between representations of different attributes inside the CNN. Then, we compare the mined attribute relationships with ground-truth attribute relationships to discover the CNN’s blind spots and failure modes due to dataset bias. In fact, representation flaws caused by dataset bias cannot be examined by conventional evaluation strategies based on testing images, because testing images may also have a similar bias. Experiments have demonstrated the effectiveness of our method.

Recently there has been signi cant interest in supervised learning algorithms that combine labeled and unlabeled data for text learning tasks. The co-training setting [1] applies to datasets that have a natural separation of their features into two disjoint sets. We demonstrate that when learning from labeled and unlabeled data, algorithms explicitly leveraging a natural independent split of the features outperform algorithms that do not. When a natural split does not exist, co-training algorithms that manufacture a feature split may out-perform algorithms not using a split. These results help explain why co-training algorithms are both discriminative in nature and robust to the assumptions of their embedded classi ers.

Thanks to the success of deep learning, cross-modal retrieval has made significant progress recently. However, there still remains a crucial bottleneck: how to bridge the modality gap to further enhance the retrieval accuracy. In this paper, we propose a self-supervised adversarial hashing (SSAH) approach, which lies among the early attempts to incorporate adversarial learning into cross-modal hashing in a self-supervised fashion. The primary contribution of this work is that two adversarial networks are leveraged to maximize the semantic correlation and consistency of the representations between different modalities. In addition, we harness a self-supervised semantic network to discover high-level semantic information in the form of multi-label annotations. Such information guides the feature learning process and preserves the modality relationships in both the common semantic space and the Hamming space. Extensive experiments carried out on three benchmark datasets validate that the proposed SSAH surpasses the state-of-the-art methods.

The goal of transfer learning is to exploit previous experiences and knowledge in order to improve learning in a novel domain. This is especially beneficial for the challenging task of learning classifiers that generalize well when only few training examples are available. In such a case, knowledge transfer methods can help to compensate for the lack of data. The performance and robustness against negative transfer of these approaches is influenced by the interdependence between knowledge representation and transfer type. However, this important point is usually neglected in the literature; instead the focus lies on either of the two aspects. In contrast, we study in this work the effect of various high-level semantic knowledge representations on different transfer types in a novel generic transfer metric learning framework. Furthermore, we introduce a hierarchical knowledge representation model based on the embedded structure in the semantic attribute space. The evaluation of the framework on challenging transfer settings in the context of action similarity demonstrates the effectiveness of our approach compared to state-of-the-art. c © 2015 Elsevier Ltd. All rights reserved.

The recently proposed Temporal Ensembling has achieved state-of-the-art results in several semi-supervised learning benchmarks. It maintains an exponential moving average of label predictions on each training example, and penalizes predictions that are inconsistent with this target. However, because the targets change only once per epoch, Temporal Ensembling becomes unwieldy when learning large datasets. To overcome this problem, we propose Mean Teacher, a method that averages model weights instead of label predictions. As an additional benefit, Mean Teacher improves test accuracy and enables training with fewer labels than Temporal Ensembling. Without changing the network architecture, Mean Teacher achieves an error rate of 4.35% on SVHN with 250 labels, outperforming Temporal Ensembling trained with 1000 labels. We also show that a good network architecture is crucial to performance. Combining Mean Teacher and Residual Networks, we improve the state of the art on CIFAR-10 with 4000 labels from 10.55% to 6.28%, and on ImageNet 2012 with 10% of the labels from 35.24% to 9.11%.

In this paper we introduce the deep kernelized autoencoder, a neural network model that allows an explicit approximation of (i) the mapping from an input space to an arbitrary, user-specified kernel space and (ii) the back-projection from such a kernel space to input space. The proposed method is based on traditional autoencoders and is trained through a new unsupervised loss function. During training, we optimize both the reconstruction accuracy of input samples and the alignment between a kernel matrix given as prior and the inner products of the hidden representations computed by the autoencoder. Kernel alignment provides control over the hidden representation learned by the autoencoder. Experiments have been performed to evaluate both reconstruction and kernel alignment performance. Additionally, we applied our method to emulate kPCA on a denoising task obtaining promising results.

We present the first domain adaptation model for authorship attribution to leverage unlabeled data. The model includes extensions to structural correspondence learning needed to make it appropriate for the task. For example, we propose a median-based classification instead of the standard binary classification used in previous work. Our results show that punctuation-based character n-grams form excellent pivot features. We also show how singular value decomposition plays a critical role in achieving domain adaptation, and that replacing (instead of concatenating) non-pivot features with correspondence features yields better performance.

In Proc. of IEEE Conf. on CVPR’2000, Vol.II, pp.88-94, Hilton Head Island, SC, 2000 Since human hand is highly articulated and deformable, hand posture recognition is a challenging example in the research of view-independent object recognition. Due to the difficulties of the modelbased approach, the appearance-based learning approach is promising to handle large variation in visual inputs. However, the generalization of many proposed supervised learning methods to this problem often suffers from the insufficiency of labeled training data. This paper describes an approach to alleviate this difficulty by adding a large unlabeled training set. Combining supervised and unsupervised learning paradigms, a novel and powerful learning approach, the Discriminant-EM (D-EM) algorithm, is proposed in this paper to handle the case of small labeled training set. Experiments show that D-EM outperforms many other learning methods. Based on this approach, we implement a gesture interface to recognize a set of predefined gesture commands, and it is also extended to hand detection. This algorithm can also apply to other object recognition tasks.


Question: A method for performing one-sided unsupervised domain mapping.

This paper presents an innovative approach using generative adversarial networks (GANs) to identify and learn cross-domain relationships between unpaired datasets. Inspired by the human capacity to effortlessly discern connections across different domains without supervision, our method, termed DiscoGAN, aims to automate this process with minimal reliance on ground-truth paired examples, which can be both costly and time-consuming to acquire. DiscoGAN's core functionality revolves around its ability to learn and extract the underlying relationships that exist between various domains. By harnessing these learned relationships, DiscoGAN facilitates the transfer of stylistic attributes from one domain to another, all while preserving critical characteristics such as orientation and identity in the case of faces. This capability not only enhances the versatility of GANs but also broadens their application in a multitude of cross-domain scenarios, showcasing DiscoGAN's potential in advancing unsupervised domain mapping techniques.

Domain dependence of NLP systems is one of the major obstacles to their application in large-scale text analysis, also restricting the applicability of FrameNet semantic role labeling (SRL) systems. Yet, current FrameNet SRL systems are still only evaluated on a single in-domain test set. For the first time, we study the domain dependence of FrameNet SRL on a wide range of benchmark sets. We create a novel test set for FrameNet SRL based on user-generated web text and find that the major bottleneck for out-of-domain FrameNet SRL is the frame identification step. To address this problem, we develop a simple, yet efficient system based on distributed word representations. Our system closely approaches the state-of-the-art in-domain while outperforming the best available frame identification system out-of-domain. We publish our system and test data for research purposes.1

Domain adaptation aims to train a model on labeled data from a source domain while minimizing test error on a target domain. Most of existing domain adaptation methods only focus on reducing domain shift of single-modal data. In this paper, we consider a new problem of multimodal domain adaptation and propose a unified framework to solve it. The proposed multimodal domain adaptation neural networks(MDANN) consist of three important modules. (1) A covariant multimodal attention is designed to learn a common feature representation for multiple modalities. (2) A fusion module adaptively fuses attended features of different modalities. (3) Hybrid domain constraints are proposed to comprehensively learn domain-invariant features by constraining single modal features, fused features, and attention scores. Through jointly attending and fusing under an adversarial objective, the most discriminative and domain-adaptive parts of the features are adaptively fused together. Extensive experimental results on two real-world cross-domain applications (emotion recognition and cross-media retrieval) demonstrate the effectiveness of the proposed method.

In this paper, we propose a new framework called domain adaptation machine (DAM) for the multiple source domain adaption problem. Under this framework, we learn a robust decision function (referred to as target classifier) for label prediction of instances from the target domain by leveraging a set of base classifiers which are prelearned by using labeled instances either from the source domains or from the source domains and the target domain. With the base classifiers, we propose a new domain-dependent regularizer based on smoothness assumption, which enforces that the target classifier shares similar decision values with the relevant base classifiers on the unlabeled instances from the target domain. This newly proposed regularizer can be readily incorporated into many kernel methods (e.g., support vector machines (SVM), support vector regression, and least-squares SVM (LS-SVM)). For domain adaptation, we also develop two new domain adaptation methods referred to as FastDAM and UniverDAM. In FastDAM, we introduce our proposed domain-dependent regularizer into LS-SVM as well as employ a sparsity regularizer to learn a sparse target classifier with the support vectors only from the target domain, which thus makes the label prediction on any test instance very fast. In UniverDAM, we additionally make use of the instances from the source domains as Universum to further enhance the generalization ability of the target classifier. We evaluate our two methods on the challenging TRECIVD 2005 dataset for the large-scale video concept detection task as well as on the 20 newsgroups and email spam datasets for document retrieval. Comprehensive experiments demonstrate that FastDAM and UniverDAM outperform the existing multiple source domain adaptation methods for the two applications.

We describe a unified active transfer learning framework called Hierarchical Active Transfer Learning (HATL). HATL exploits cluster structure shared between different data domains to perform transfer learning by imputing labels for unlabeled target data and to generate effective label queries during active learning. The resulting framework is flexible enough to perform not only adaptive transfer learning and accelerated active learning but also unsupervised and semi-supervised transfer learning. We derive an intuitive and useful upper bound on HATL’s error when used to infer labels for unlabeled target points. We also present results on synthetic data that confirm both intuition and our analysis. Finally, we demonstrate HATL’s empirical effectiveness on a benchmark data set for sentiment classification.

The applicability of machine learning methods is often limi ted by the amount of available labeled data, and by the ability (or inability) of the de signer to produce good internal representations and good similarity measures for the input data vectors. The aim of this thesis is to alleviate these two limitations by proposing al gorithms tolearngood internal representations, and invariant feature hierarchies from u nlabeled data. These methods go beyond traditional supervised learning algorithms, and rely on unsupervised, and semi-supervised learning. In particular, this work focuses on “deep learning” methods , a et of techniques and principles to train hierarchical models. Hierarchical mod els produce feature hierarchies that can capture complex non-linear dependencies among the observed data variables in a concise and efficient manner. After training, these mode ls can be employed in real-time systems because they compute the representation by a very fast forward propagation of the input through a sequence of non-linear transf ormations. When the paucity of labeled data does not allow the use of traditional supervi s d algorithms, each layer of the hierarchy can be trained in sequence starting at the bott om by using unsupervised or semi-supervised algorithms. Once each layer has been train ed, the whole system can be fine-tuned in an end-to-end fashion. We propose several unsu pervised algorithms that can be used as building block to train such feature hierarchi es. We investigate algorithms that produce sparse overcomplete representations a nd fe tures that are invariant to known and learned transformations. These algorithms are designed using the Energy-

Domain adaptation (DA) algorithms utilize a label-rich old dataset (domain) to build a machine learning model (classification, detection etc.) in a label-scarce new dataset with different data distribution. Recent approaches transform cross-domain data into a shared subspace by minimizing the shift between their marginal distributions. In this paper, we propose a novel iterative method to learn a common subspace based on non-parametric quadratic mutual information (QMI) between data and corresponding class labels. We extend a prior work of discriminative subspace learning based on maximization of QMI and integrate instance weighting into the QMI formulation. We propose an adaptive weighting model to identify relevant samples that share underlying similarity across domains and ignore irrelevant ones. Due to difficulty of applying cross-validation, an alternative strategy is integrated with the proposed algorithm to setup model parameters. A set of comprehensive experiments on benchmark datasets is conducted to prove the efficacy of our proposed framework over state-of-the-art approaches.

The applicability of computer vision to real paintings and artworks has been rarely investigated, even though a vast heritage would greatly benefit from techniques which can understand and process data from the artistic domain. This is partially due to the small amount of annotated artistic data, which is not even comparable to that of natural images captured by cameras. In this paper, we propose a semantic-aware architecture which can translate artworks to photo-realistic visualizations, thus reducing the gap between visual features of artistic and realistic data. Our architecture can generate natural images by retrieving and learning details from real photos through a similarity matching strategy which leverages a weakly-supervised semantic understanding of the scene. Experimental results show that the proposed technique leads to increased realism and to a reduction in domain shift, which improves the performance of pre-trained architectures for classification, detection, and segmentation. Code will be made publicly available.

Generative Adversarial Net is a frontier method of generative models for images, audios and videos. In this paper, we focus on conditional image generation and introduce conditional Feature-Matching Generative Adversarial Net to generate images from category labels. By visualizing state-of-art discriminative conditional generative models, we find these networks do not gain clear semantic concepts. Thus we design the loss function in the light of metric learning to measure semantic distance. The proposed model is evaluated on several well-known datasets. It is shown to be of higher perceptual quality and better diversity then existing generative models.

A number of vision problems such as zero-shot learning and person re-identification can be considered as cross-class transfer learning problems. As mid-level semantic properties shared cross different object classes, attributes have been studied extensively for knowledge transfer across classes. Most previous attribute learning methods focus only on human-defined/nameable semantic attributes, whilst ignoring the fact there also exist undefined/latent shareable visual properties, or latent attributes. These latent attributes can be either discriminative or non-discriminative parts depending on whether they can contribute to an object recognition task. In this work, we argue that learning the latent attributes jointly with user-defined semantic attributes not only leads to better representation but also helps semantic attribute prediction. A novel dictionary learning model is proposed which decomposes the dictionary space into three parts corresponding to semantic, latent discriminative and latent background attributes respectively. Such a joint attribute learning model is then extended by following a multi-task transfer learning framework to address a more challenging unsupervised domain adaptation problem, where annotations are only available on an auxiliary dataset and the target dataset is completely unlabelled. Extensive experiments show that the proposed models, though being linear and thus extremely efficient to compute, produce state-of-the-art results on both zero-shot learning and person re-identification.

Deep generative models such as Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs) have recently been applied to style and domain transfer for images, and in the case of VAEs, music. GAN-based models employing several generators and some form of cycle consistency loss have been among the most successful for image domain transfer. In this paper we apply such a model to symbolic music and show the feasibility of our approach for music genre transfer. Evaluations using separate genre classifiers show that the style transfer works well. In order to improve the fidelity of the transformed music, we add additional discriminators that cause the generators to keep the structure of the original music mostly intact, while still achieving strong genre transfer. Visual and audible results further show the potential of our approach. To the best of our knowledge, this paper represents the first application of GANs to symbolic music domain transfer.


Question: A single-sided unsupervised technique for domain mapping.

This paper introduces the Basel Face Model (BFM), a comprehensive 3D generative model for facial recognition, designed to overcome the challenges posed by varying poses and lighting conditions. Traditional generative 3D face models, while effective, are often hindered by the complexities and costs associated with their construction, leading researchers to opt for simpler, albeit less potent, alternatives. The BFM, however, represents a significant advancement, offering enhanced shape and texture accuracy achieved through the use of a superior scanning device and a refined registration algorithm that minimizes correspondence artifacts. This model's versatility is further demonstrated by its ability to be fitted to both 2D and 3D images captured under diverse conditions and with different sensors, employing an analysis by synthesis method. The fitting process yields model parameters that distinctly separate pose, lighting, imaging, and identity variables, making it particularly adept at invariant face recognition across multiple sensors and datasets by comparing solely the identity parameters. The release of this meticulously registered face model is anticipated to invigorate research in the field of generative models. Accompanying the model are comprehensive recognition and reconstruction outcomes on established databases, facilitating direct algorithm comparisons and promoting advancements in facial recognition technology.

Question: A Direct Search Method to solve Economic Dispatch Problem with Valve-Point Effect

Documents: An evolutionary recurrent network which automates the design of recurrent neural/fuzzy networks using a new evolutionary learning algorithm is proposed in this paper. This new evolutionary learning algorithm is based on a hybrid of genetic algorithm (GA) and particle swarm optimization (PSO), and is thus called HGAPSO. In HGAPSO, individuals in a new generation are created, not only by crossover and mutation operation as in GA, but also by PSO. The concept of elite strategy is adopted in HGAPSO, where the upper-half of the best-performing individuals in a population are regarded as elites. However, instead of being reproduced directly to the next generation, these elites are first enhanced. The group constituted by the elites is regarded as a swarm, and each elite corresponds to a particle within it. In this regard, the elites are enhanced by PSO, an operation which mimics the maturing phenomenon in nature. These enhanced elites constitute half of the population in the new generation, whereas the other half is generated by performing crossover and mutation operation on these enhanced elites. HGAPSO is applied to recurrent neural/fuzzy network design as follows. For recurrent neural network, a fully connected recurrent neural network is designed and applied to a temporal sequence production problem. For recurrent fuzzy

Choosing optimal (or at least better) policies is an important problem in domains from medicine to education to finance and many others. One approach to this problem is through controlled experiments/trials but controlled experiments are expensive. Hence it is important to choose the best policies on the basis of observational data. This presents two difficult challenges: (i) missing counterfactuals, and (ii) selection bias. This paper presents theoretical bounds on estimation errors of counterfactuals from observational data by making connections to domain adaptation theory. It also presents a principled way of choosing optimal policies using domain adversarial neural networks. This illustrates the effectiveness of domain adversarial training together with various features of our algorithm on a semi-synthetic breast cancer dataset.

of “Any Domain Parsing: Automatic Domain Adaptation for Natural Language Parsing” by David McClosky , Ph.D., Brown University, May, 2010. Current efforts in syntactic parsing are largely data-driv en. These methods require labeled examples of syntactic structures to learn statistical patterns governing these structures. Labeled data typically requires expert annotators which makes it both time consuming and costly to p roduce. Furthermore, once training data has been created for one textual domain, portability to similar domains is limited. This domain-dependence has inspired a large body of work since syntactic parsing aims to capture syntactic patterns across an entire language rather than just a specific domain. The simplest approach to this task is to assume that the targe t domain is essentially the same as the source domain. No additional knowledge about the target domain is i cluded. A more realistic approach assumes that only raw text from the target domain is available. This a s umption lends itself well to semi-supervised learning methods since these utilize both labeled and unlab eled examples. This dissertation focuses on a family of semi-supervised me thods called self-training. Self-training creates semi-supervised learners from existing supervised learne rs with minimal effort. We first show results on self-training for constituency parsing within a single dom ain. While self-training has failed here in the past, we present a simple modification which allows it to succeed, p roducing state-of-the-art results for English constituency parsing. Next, we show how self-training is be neficial when parsing across domains and helps further when raw text is available from the target domain. On e of the remaining issues is that one must choose a training corpus appropriate for the target domain or perfo rmance may be severely impaired. Humans can do this in some situations, but this strategy becomes less prac tical as we approach larger data sets. We present a technique, Any Domain Parsing, which automatically detect s useful source domains and mixes them together to produce a customized parsing model. The resulting models perform almost as well as the best seen parsing models (oracle) for each target domain. As a result, we have a fully automatic syntactic constituency parser which can produce high-quality parses for all types of text, regardless of domain. Abstract of “Any Domain Parsing: Automatic Domain Adaptation for Natural Language Parsing” by David McClosky , Ph.D., Brown University, May, 2010.of “Any Domain Parsing: Automatic Domain Adaptation for Natural Language Parsing” by David McClosky , Ph.D., Brown University, May, 2010. Current efforts in syntactic parsing are largely data-driv en. These methods require labeled examples of syntactic structures to learn statistical patterns governing these structures. Labeled data typically requires expert annotators which makes it both time consuming and costly to p roduce. Furthermore, once training data has been created for one textual domain, portability to similar domains is limited. This domain-dependence has inspired a large body of work since syntactic parsing aims to capture syntactic patterns across an entire language rather than just a specific domain. The simplest approach to this task is to assume that the targe t domain is essentially the same as the source domain. No additional knowledge about the target domain is i cluded. A more realistic approach assumes that only raw text from the target domain is available. This a s umption lends itself well to semi-supervised learning methods since these utilize both labeled and unlab eled examples. This dissertation focuses on a family of semi-supervised me thods called self-training. Self-training creates semi-supervised learners from existing supervised learne rs with minimal effort. We first show results on self-training for constituency parsing within a single dom ain. While self-training has failed here in the past, we present a simple modification which allows it to succeed, p roducing state-of-the-art results for English constituency parsing. Next, we show how self-training is be neficial when parsing across domains and helps further when raw text is available from the target domain. On e of the remaining issues is that one must choose a training corpus appropriate for the target domain or perfo rmance may be severely impaired. Humans can do this in some situations, but this strategy becomes less prac tical as we approach larger data sets. We present a technique, Any Domain Parsing, which automatically detect s useful source domains and mixes them together to produce a customized parsing model. The resulting models perform almost as well as the best seen parsing models (oracle) for each target domain. As a result, we have a fully automatic syntactic constituency parser which can produce high-quality parses for all types of text, regardless of domain. Any Domain Parsing: Automatic Domain Adaptation for Natural Language Parsing by David McClosky B. S., University of Rochester, 2004 Sc. M., Brown University, 2006 A dissertation submitted in partial fulfillment of the requirements for the Degree of Doctor of Philosophy in the Department of Computer Science at Brown University. Providence, Rhode Island May, 2010 c © Copyright 2010 by David McClosky This dissertation by David McClosky is accepted in its prese nt form by the Department of Computer Science as satisfying the disser tation requirement for the degree of Doctor of Philosophy. Date Eugene Charniak, Director Recommended to the Graduate Council Date Mark Johnson, Reader Cognitive and Linguistic Sciences Date Dan Klein, Reader University of California at Berkeley Approved by the Graduate Council Date Sheila Bond Dean of the Graduate School

What machine learning (ML) technique is used for system intelligence implementation in ADAS (advanced driving assistance system)? This paper tries to answer this question. This paper analyzes ADAS and ML independently and then relate which ML technique is applicable to what ADAS component and why. The paper gives a good grasp of the current state-of-the-art. Sample works in supervised, unsupervised, deep and reinforcement learnings are presented; their strengths and rooms for improvements are also discussed. This forms part of the basics in understanding autonomous vehicle. This work is a contribution to the ongoing research in ML aimed at reducing road traffic accidents and fatalities, and the invocation of safe driving.

Cross-domain visual data matching is one of the fundamental problems in many real-world vision tasks, e.g., matching persons across ID photos and surveillance videos. Conventional approaches to this problem usually involves two steps: i) projecting samples from different domains into a common space, and ii) computing (dis-)similarity in this space based on a certain distance. In this paper, we present a novel pairwise similarity measure that advances existing models by i) expanding traditional linear projections into affine transformations and ii) fusing affine Mahalanobis distance and Cosine similarity by a data-driven combination. Moreover, we unify our similarity measure with feature representation learning via deep convolutional neural networks. Specifically, we incorporate the similarity measure matrix into the deep architecture, enabling an end-to-end way of model optimization. We extensively evaluate our generalized similarity model in several challenging cross-domain matching tasks: person re-identification under different views and face verification over different modalities (i.e., faces from still images and videos, older and younger faces, and sketch and photo portraits). The experimental results demonstrate superior performance of our model over other state-of-the-art methods.

We evaluate whether features extracted from the activation of a deep convolutional network trained in a fully supervised fashion on a large, fixed set of object recognition tasks can be repurposed to novel generic tasks. Our generic tasks may differ significantly from the originally trained tasks and there may be insufficient labeled or unlabeled data to conventionally train or adapt a deep architecture to the new tasks. We investigate and visualize the semantic clustering of deep convolutional features with respect to a variety of such tasks, including scene recognition, domain adaptation, and fine-grained recognition challenges. We compare the efficacy of relying on various network levels to define a fixed feature, and report novel results that significantly outperform the state-of-the-art on several important vision challenges. We are releasing DeCAF, an open-source implementation of these deep convolutional activation features, along with all associated network parameters to enable vision researchers to be able to conduct experimentation with deep representations across a range of visual concept learning paradigms.

1College of Computer Science, Sichuan Normal University, Chengdu, Sichuan, China 2Developmental Neurocognition Lab, Department of Psychological Sciences, Birkbeck, University of London, London, UK 3College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing, China Correspondence Juan Yang, College of Computer Science, Sichuan Normal University, Chengdu, Sichuan, China. Email: jkxy_yjuan@sicnu.edu.cn Funding information National Natural Science Foundation of China, Grant/Award Number: (61402309); UK Economic and Social Research Council, Grant/Award Number: RES-062-23-2721 Summary Mapping problems are typical research topics related to natural language learning, and they include not only classification mappings but also nonclassification mappings, such as verbs and their past tenses. Connectionist computational models are one of the most popular approaches for simulating those mapping problems; however, their lack of explanatory ability has prevented them from being further used to understand the language learning process. Therefore, the work of extracting rational rules from a connectionist model is as important as simulating the mapping behaviors. Unfortunately, there is no available technique that can be directly applied in those computational models to simulate nonclassification problems. In this paper, an autoencoder-based connectionist computational model is proposed to derive a rule extraction method that can construct “If-Then” rational relations with high fidelity for nonclassification mapping problems. To demonstrate its generalizability, this computational model is extended to a modified version to address a multi-label classification mapping problem related to cognitive style prediction. Experiments prove this computational model's simulation ability and its explanatory ability on nonclassification problems by comparing its fidelity performances with those of the classical connectionist computational model (multilayer perceptron artificial neural network), and its similar ability on a multi-label classification problem (Felder-Silverman learning style classification) by comparing its prediction accuracy with those of other rule induction techniques.

In Proc. of IEEE Conf. on CVPR’2000, Vol.II, pp.88-94, Hilton Head Island, SC, 2000 Since human hand is highly articulated and deformable, hand posture recognition is a challenging example in the research of view-independent object recognition. Due to the difficulties of the modelbased approach, the appearance-based learning approach is promising to handle large variation in visual inputs. However, the generalization of many proposed supervised learning methods to this problem often suffers from the insufficiency of labeled training data. This paper describes an approach to alleviate this difficulty by adding a large unlabeled training set. Combining supervised and unsupervised learning paradigms, a novel and powerful learning approach, the Discriminant-EM (D-EM) algorithm, is proposed in this paper to handle the case of small labeled training set. Experiments show that D-EM outperforms many other learning methods. Based on this approach, we implement a gesture interface to recognize a set of predefined gesture commands, and it is also extended to hand detection. This algorithm can also apply to other object recognition tasks.

A lack of generalizability is one key limitation of deep learning based segmentation. Typically, one manually labels new training images when segmenting organs in different imaging modalities or segmenting abnormal organs from distinct disease cohorts. The manual efforts can be alleviated if one is able to reuse manual labels from one modality (e.g., MRI) to train a segmentation network for a new modality (e.g., CT). Previously, two stage methods have been proposed to use cycle generative adversarial networks (CycleGAN) to synthesize training images for a target modality. Then, these efforts trained a segmentation network independently using synthetic images. However, these two independent stages did not use the complementary information between synthesis and segmentation. Herein, we proposed a novel end-to-end synthesis and segmentation network (EssNet) to achieve the unpaired MRI to CT image synthesis and CT splenomegaly segmentation simultaneously without using manual labels on CT. The end-to-end EssNet achieved significantly higher median Dice similarity coefficient (0.9188) than the two stages strategy (0.8801), and even higher than canonical multi-atlas segmentation (0.9125) and ResNet method (0.9107), which used the CT manual labels.

We propose a recurrent neural network architecture with a Forward Kinematics layer and cycle consistency based adversarial training objective for unsupervised motion retargetting. Our network captures the high-level properties of an input motion by the forward kinematics layer, and adapts them to a target character with different skeleton bone lengths (e.g., shorter, longer arms etc.). Collecting paired motion training sequences from different characters is expensive. Instead, our network utilizes cycle consistency to learn to solve the Inverse Kinematics problem in an unsupervised manner. Our method works online, i.e., it adapts the motion sequence on-the-fly as new frames are received. In our experiments, we use the Mixamo animation data1 to test our method for a variety of motions and characters and achieve state-of-the-art results. We also demonstrate motion retargetting from monocular human videos to 3D characters using an off-the-shelf 3D pose estimator.

The process of learning good features for machine learning applications can be very computationally expensive and may prove difficult in cases where little data is available. A prototypical example of this is the one-shot learning setting, in which we must correctly make predictions given only a single example of each new class. In this paper, we explore a method for learning siamese neural networks which employ a unique structure to naturally rank similarity between inputs. Once a network has been tuned, we can then capitalize on powerful discriminative features to generalize the predictive power of the network not just to new data, but to entirely new classes from unknown distributions. Using a convolutional architecture, we are able to achieve strong results which exceed those of other deep learning models with near state-of-the-art performance on one-shot classification tasks. Humans exhibit a strong ability to acquire and recognize new patterns. In particular, we observe that when presented with stimuli, people seem to be able to understand new concepts quickly and then recognize variations on these concepts in future percepts (Lake et al., 2011). Machine learning has been successfully used to achieve state-ofthe-art performance in a variety of applications such as web search, spam detection, caption generation, and speech and image recognition. However, these algorithms often break down when forced to make predictions about data for which little supervised information is available. We desire to generalize to these unfamiliar categories without necessitating extensive retraining which may be either expensive or impossible due to limited data or in an online prediction setting, such as web retrieval. Proceedings of the 32 International Conference on Machine Learning, Lille, France, 2015. JMLR: W&CP volume 37. Copyright 2015 by the author(s). Figure 1. Example of a 20-way one-shot classification task using the Omniglot dataset. The lone test image is shown above the grid of 20 images representing the possible unseen classes that we can choose for the test image. These 20 images are our only known examples of each of those classes. One particularly interesting task is classification under the restriction that we may only observe a single example of each possible class before making a prediction about a test instance. This is called one-shot learning and it is the primary focus of our model presented in this work (Fei-Fei et al., 2006; Lake et al., 2011). This should be distinguished from zero-shot learning, in which the model cannot look at any examples from the target classes (Palatucci et al., 2009). One-shot learning can be directly addressed by developing domain-specific features or inference procedures which possess highly discriminative properties for the target task. As a result, systems which incorporate these methods tend to excel at similar instances but fail to offer robust solutions that may be applied to other types of problems. In this paper, we present a novel approach which limits assumptions on the structure of the inputs while automatically acquiring features which enable the model to generalize successfully from few examples. We build upon the deep learnSiamese Neural Networks for One-shot Image Recognition Figure 2. Our general strategy. 1) Train a model to discriminate between a collection of same/different pairs. 2) Generalize to evaluate new categories based on learned feature mappings for