Question: Enhancing the sparse spectrum approximation of Gaussian processes by incorporating uncertainty in frequency inputs.

This paper explores the integration of Bayesian principles into supervised learning for classification networks, highlighting three key concepts. Firstly, it advocates for the use of the posterior distribution of parameters to derive the output of a classifier, proposing a straightforward approximation method for this integration. This approach involves a form of "moderation" for the outputs of the most probable classifier, resulting in enhanced classification accuracy. Secondly, the paper demonstrates the applicability of the Bayesian framework for model comparison, initially developed for regression models by MacKay (1992a,b), to classification tasks. This framework effectively determines the optimal weight decay terms and ranks solutions obtained with varying numbers of hidden units, showcasing its versatility in model selection. Thirdly, an information-theoretic data selection criterion is derived and validated within this Bayesian context, providing a principled method for selecting informative data points for training. These contributions underscore the potential of Bayesian methods to improve the robustness and efficiency of classification networks.

**Question**: A Novel Method for Identifying and Mitigating Adversarial Attacks in Deep Learning Models

**Documents**: Deep Learning with Differential Privacy: A Survey
Deep learning (DL) has achieved remarkable success in various application domains. However, the data used to train these models often contain sensitive information, raising concerns about privacy. Differential privacy (DP) is a rigorous privacy definition that has been extensively studied in the context of data publishing and statistical analysis. Recently, DP has been applied to deep learning to protect the privacy of the training data. This survey provides a comprehensive overview of the existing work on deep learning with differential privacy. We first introduce the background of differential privacy and its application to deep learning. Then, we review the state-of-the-art techniques for training deep learning models with differential privacy, including DP-SGD, DP-GAN, and DP-VAE. We also discuss the challenges and limitations of applying differential privacy to deep learning, such as the trade-off between privacy and utility, the high computational cost, and the difficulty of tuning the privacy parameters. Finally, we highlight the open problems and future research directions in this field.

**Similar Documents**: This paper offers an extensive survey on the intersection of deep learning (DL) and differential privacy (DP), focusing on the critical issue of safeguarding sensitive information in training data. Despite DL's impressive achievements across numerous fields, the privacy implications of utilizing data that may contain confidential details have become a pressing concern. DP, as a robust privacy definition, has gained significant attention for its potential to mitigate these privacy risks. The paper begins by laying the groundwork

Kernel approximation via nonlinear random feature maps is w idely used in speeding up kernel machines. There are two main challenges for the conventional kernel ap proximation methods. First, before performing kernel approximation, a good kernel has to be chosen. Pickin g a good kernel is a very challenging problem in itself. Second, high-dimensional maps are often required i n order to achieve good performance. This leads to high computational cost in both generating the nonlinear ma ps, nd in the subsequent learning and prediction process. In this work, we propose to optimize the nonlinear m ps directly with respect to the classification objective in a data-dependent fashion. The proposed approa ch achieves kernel approximation and kernel learning in a joint framework. This leads to much more compact maps wit hout hurting the performance. As a by-product, the same framework can also be used to achieve more compact ke rnel maps to approximate a known kernel. We also introduce Circulant Nonlinear Maps, which uses a circu lant-structured projection matrix to speed up the nonlinear maps for high-dimensional data.

While the accuracy of feature measurements heavily depends on changing environmental conditions, studying the consequences of this fact in pattern recognition tasks has received relatively little attention to date. In this paper, we explicitly take feature measurement uncertainty into account and show how multimodal classification and learning rules should be adjusted to compensate for its effects. Our approach is particularly fruitful in multimodal fusion scenarios, such as audiovisual speech recognition, where multiple streams of complementary time-evolving features are integrated. For such applications, provided that the measurement noise uncertainty for each feature stream can be estimated, the proposed framework leads to highly adaptive multimodal fusion rules which are easy and efficient to implement. Our technique is widely applicable and can be transparently integrated with either synchronous or asynchronous multimodal sequence integration architectures. We further show that multimodal fusion methods relying on stream weights can naturally emerge from our scheme under certain assumptions; this connection provides valuable insights into the adaptivity properties of our multimodal uncertainty compensation approach. We show how these ideas can be practically applied for audiovisual speech recognition. In this context, we propose improved techniques for person-independent visual feature extraction and uncertainty estimation with active appearance models, and also discuss how enhanced audio features along with their uncertainty estimates can be effectively computed. We demonstrate the efficacy of our approach in audiovisual speech recognition experiments on the CUAVE database using either synchronous or asynchronous multimodal integration models.

This paper makes available a concise review of data windows and their affect on the detection of harmonic signals in the presence of broad-band noise, and in the presence of nearby strong harmonic interference. We also call attention to a number of common errors in the application of windows when used with the fast Fourier transform. This paper includes a comprehensive catalog of data windows along with their significant performance parameters from which the different windows can be compared. Finally, an example demonstrates the use and value of windows to resolve closely spaced harmonic signals characterized by large differences in amplitude.

An alternative derivation for the well-known approximate message passing (AMP) algorithm proposed by Donoho is presented in this letter. Compared with the original derivation, which exploits central limit theorem and Taylor expansion to simplify belief propagation (BP), our derivation resorts to expectation propagation (EP) and the neglect of high-order terms in large system limit. This alternative derivation leads to a different yet provably equivalent form of message passing, which explicitly establishes the intrinsic connection between AMP and EP, thereby offering some new insights in the understanding and improvement of AMP.

State of the art Bayesian classification approaches typically maintain a posterior distribution over possible classes given available sensor observations (images). Yet, while these approaches fuse all classifier outputs thus far, they do not provide any indication regarding how reliable the posterior classification is, thus limiting its functionality in terms of autonomous systems and robotics. On the other hand, current deep learning based classifiers provide an uncertainty measure, thereby quantifying model uncertainty. However, they do so on a single frame basis and do not consider a sequential framework. In this letter, we develop a novel approach that infers a distribution over posterior class probabilities, while accounting for model uncertainty. This distribution enables reasoning about uncertainty in the posterior classification and, therefore, is of prime importance for robust classification, object-level perception in uncertain and ambiguous scenarios, and for safe autonomy in general. The distribution of the posterior class probability has no known analytical solution; thus, we propose to approximate this distribution via sampling. We evaluate our approach in simulation and using real images fed into a convolutional neural network classifier.

When applying sparse representation techniques to images, the standard approach is to independently compute the representations for a set of overlapping image patches. This method performs very well in a variety of applications, but results in a representation that is multi-valued and not optimized with respect to the entire image. An alternative representation structure is provided by a convolutional sparse representation, in which a sparse representation of an entire image is computed by replacing the linear combination of a set of dictionary vectors by the sum of a set of convolutions with dictionary filters. The resulting representation is both single-valued and jointly optimized over the entire image. While this form of a sparse representation has been applied to a variety of problems in signal and image processing and computer vision, the computational expense of the corresponding optimization problems has restricted application to relatively small signals and images. This paper presents new, efficient algorithms that substantially improve on the performance of other recent methods, contributing to the development of this type of representation as a practical tool for a wider range of problems.

The management of uncertainty is crucial when harvesting structured content from unstructured and noisy sources. Knowledge Graphs (KGs) are a prominent example. KGs maintain both numerical and non-numerical facts, with the support of an underlying schema. These facts are usually accompanied by a confidence score that witnesses how likely is for them to hold. Despite their popularity, most of existing KGs focus on static data thus impeding the availability of timewise knowledge. What is missing is a comprehensive solution for the management of uncertain and temporal data in KGs. The goal of this paper is to fill this gap. We rely on two main ingredients. The first is a numerical extension of Markov Logic Networks (MLNs) that provide the necessary underpinning to formalize the syntax and semantics of uncertain temporal KGs. The second is a set of Datalog constraints with inequalities that extend the underlying schema of the KGs and help to detect inconsistencies. From a theoretical point of view, we discuss the complexity of two important classes of queries for uncertain temporal KGs: maximum a-posteriori and conditional probability inference. Due to the hardness of these problems and the fact that MLN solvers do not scale well, we also explore the usage of Probabilistic Soft Logics (PSL) as a practical tool to support our reasoning tasks. We report on an experimental evaluation comparing the MLN and PSL approaches.

Choosing appropriate architectures and regularization strategies of deep networks is crucial to good predictive performance. To shed light on this problem, we analyze the analogous problem of constructing useful priors on compositions of functions. Specifically, we study the deep Gaussian process, a type of infinitely-wide, deep neural network. We show that in standard architectures, the representational capacity of the network tends to capture fewer degrees of freedom as the number of layers increases, retaining only a single degree of freedom in the limit. We propose an alternate network architecture which does not suffer from this pathology. We also examine deep covariance functions, obtained by composing infinitely many feature transforms. Lastly, we characterize the class of models obtained by performing dropout on Gaussian processes.

This article presents new results on using a greedy algorithm, orthogonal matching pursuit (OMP), to solve the sparse approximation problem over redundant dictionaries. It provides a sufficient condition under which both OMP and Donoho's basis pursuit (BP) paradigm can recover the optimal representation of an exactly sparse signal. It leverages this theory to show that both OMP and BP succeed for every sparse input signal from a wide class of dictionaries. These quasi-incoherent dictionaries offer a natural generalization of incoherent dictionaries, and the cumulative coherence function is introduced to quantify the level of incoherence. This analysis unifies all the recent results on BP and extends them to OMP. Furthermore, the paper develops a sufficient condition under which OMP can identify atoms from an optimal approximation of a nonsparse signal. From there, it argues that OMP is an approximation algorithm for the sparse problem over a quasi-incoherent dictionary. That is, for every input signal, OMP calculates a sparse approximant whose error is only a small factor worse than the minimal error that can be attained with the same number of terms.

Absstroct-This paper focuses on the class of speech enhancement systems which capitalize on the major importance of the short-time spectral amplitude (STSA) of the speech signal in its perception. A system which utilizes a minimum mean-square error (MMSE) STSA estimator is proposed and then compared with other widely used systems which are based on Wiener filtering and the " spectral subtraction " algorithm. In this paper we derive the MMSE STSA estimator, based on modeling speech and noise spectral components as statistically independent Gaussian random variables. We analyze the performance of the proposed STSA estimator and compare it with a STSA estimator derived from the Wiener estimator. We also examine the MMSE STSA estimator under uncertainty of signal presence in the noisy observations. In constructing the enhanced signal, the MMSE STSA estimator is combined with the complex exponential of the noisy phase. It is shown here that the latter is the MMSE estimator of the complex exponential of the original phase, which does not affect the STSA estimation. The proposed approach results in a significant reduction of the noise, and provides enhanced speech with colorless residual noise. The complexity of the proposed algorithm is approximately that of other systems in the discussed class.


Question: Enhancing the sparse spectrum approximation of Gaussian processes by incorporating uncertainty in frequency inputs.

This paper offers a comprehensive perspective that encompasses all existing probabilistic sparse approximations for Gaussian process regression, providing a fresh insight into the field. Our methodology is centered around elucidating the effective prior utilized by these techniques, which not only unveils new understandings but also underscores the interconnections among them. This approach facilitates a clear, theoretically grounded assessment of the proximity of these approximations to their corresponding full Gaussian processes. Moreover, it paves the way for the design of novel, superior sparse approximations that synthesize the best features of current strategies, all while adhering to favorable computational boundaries. Specifically, by representing uncertainty in frequency inputs, we can enhance the Gaussian Process Sparse Spectrum Approximation, leading to more accurate and reliable predictions. This novel approach not only improves upon existing methods but also opens up new avenues for research and application in the realm of Gaussian process regression.

The description of a novel type of rn-gram language model is given. The model offers, via a nonlinear recursive procedure, a computation and space efficient solution to the problem of estimating probabilities from sparse data. This solution compares favorably to other proposed methods. While the method has been developed for and successfully implemented in the IBM Real Time Speech Recognizers, its generality makes it applicable in other areas where the problem of estimating probabilities from sparse data arises. Sparseness of data is an inherent property of any real text, and it is a problem that one always encounters while collecting frequency statistics on words and word sequences (m-grams) from a text of finite size. This means that even for a very large data collection, the maximum likelihood estimation method does not allow us to adequately estimate probabilities of rare but nevertheless possible word sequences-many sequences occur only once (“singletons”); many more do not occur at all. Inadequacy of the maximum likelihood estimator and the necessity to estimate the probabilities of m-grams which did not occur in the text constitute the essence of the problem. The main idea of the proposed solution to the problem is to reduce unreliable probability estimates given by the observed frequencies and redistribute the “freed” probability “mass” among m-grams which never occurred in the text. The reduction is achieved by replacing maximum likelihood estimates for m-grams having low counts with renormalized Turing’s estimates [l], and the redistribution is done via the recursive utilization of lower level conditional distributions. We found Turing’s method attractive because of its simplicity and its characterization as the optimal empirical Bayes’ estimator of a multinomial probability. Robbins in [2] introduces the empirical Bayes’ methodology and Nadas in [3] gives various derivations of the Turing’s formula. Let N be a sample text size and let n, be the number of words (m-grams) which occurred in the text exactly r times, so that N = C rn,. (1) Turing’s estimate PT for a probability of a word (m-gram) which occurred in the sample r times is r

Kernel approximation via nonlinear random feature maps is w idely used in speeding up kernel machines. There are two main challenges for the conventional kernel ap proximation methods. First, before performing kernel approximation, a good kernel has to be chosen. Pickin g a good kernel is a very challenging problem in itself. Second, high-dimensional maps are often required i n order to achieve good performance. This leads to high computational cost in both generating the nonlinear ma ps, nd in the subsequent learning and prediction process. In this work, we propose to optimize the nonlinear m ps directly with respect to the classification objective in a data-dependent fashion. The proposed approa ch achieves kernel approximation and kernel learning in a joint framework. This leads to much more compact maps wit hout hurting the performance. As a by-product, the same framework can also be used to achieve more compact ke rnel maps to approximate a known kernel. We also introduce Circulant Nonlinear Maps, which uses a circu lant-structured projection matrix to speed up the nonlinear maps for high-dimensional data.

Fitting data by a bounded complexity linear model is equivalent to low-rank approximation of a matrix constructed from the data. The data matrix being Hankel structured is equivalent to the existence of a linear time-invariant system that fits the data and the rank constraint is related to a bound on the model complexity. In the special case of fitting by a static model, the data matrix and its low-rank approximation are unstructured. We outline applications in system theory (approximate realization, model reduction, output error, and errors-in-variables identification), signal processing (harmonic retrieval, sum-of-damped exponentials, and finite impulse response modeling), and computer algebra (approximate common divisor). Algorithms based on heuristics and local optimization methods are presented. Generalizations of the low-rank approximation problem result from different approximation criteria (e.g., weighted norm) and constraints on the data matrix (e.g., nonnegativity). Related problems are rank minimization and structured pseudospectra. 2007 Elsevier Ltd. All rights reserved.

We consider the problem of sparse canonical correlation analysis (CCA), i.e., the search for two linear combi nations, one for each multivariate, that yield maximum correlation using a specified number of variables. We propose an efficient numeri cal approximation based on a direct greedy approach which bound s the correlation at each stage. The method is specifically des igned to cope with large data sets and its computational complexit y depends only on the sparsity levels. We analyze the algorith m’s performance through the tradeoff between correlation and parsimony. The results of numerical simulation suggest that a significant portion of the correlation may be captured using a relatively small number of variables. In addition, we exami ne the use of sparse CCA as a regularization method when the number of available samples is small compared to the dimensions of t he multivariates. I. I NTRODUCTION Canonical correlation analysis (CCA), introduced by Harol d Hotelling [1], is a standard technique in multivariate data n lysis for extracting common features from a pair of data sourc es [2], [3]. Each of these data sources generates a random vecto r that we call a multivariate. Unlike classical dimensionali ty reduction methods which address one multivariate, CCA take s into account the statistical relations between samples fro m two spaces of possibly different dimensions and structure. In particular, it searches for two linear combinations, one fo r each multivariate, in order to maximize their correlation. It is used in different disciplines as a stand-alone tool or as a preprocessing step for other statistical methods. Further more, CCA is a generalized framework which includes numerous classical methods in statistics, e.g., Principal Componen t Analysis (PCA), Partial Least Squares (PLS) and Multiple Linear Regression (MLR) [4]. CCA has recently regained attention with the advent of kernel CCA and its application to independent component analysis [5], [6]. The last decade has witnessed a growing interest in the search for sparse representations of signals and sparse numerical methods. Thus, we consider the problem of sparse CCA, i.e., the search for linear combinations with maximal correlation using a small number of variables. The quest for sparsity can be motivated through various reasonings. First is the ability to interpret and visualize the results. A small number of variables allows us to get the “big picture”, while sacrificing some of the small details. Moreover, spars e representations enable the use of computationally efficien t The first two authors contributed equally to this manuscript . This work was supported in part by an AFOSR MURI under Grant FA9550-06-1-0 324. numerical methods, compression techniques, as well as nois e reduction algorithms. The second motivation for sparsity i s regularization and stability. One of the main vulnerabilit ies of CCA is its sensitivity to a small number of observations. Thu s, regularized methods such as ridge CCA [7] must be used. In this context, sparse CCA is a subset selection scheme which allows us to reduce the dimensions of the vectors and obtain a stable solution. To the best of our knowledge the first reference to sparse CCA appeared in [2] where backward and stepwise subset selection were proposed. This discussion was of qualitativ e nature and no specific numerical algorithm was proposed. Recently, increasing demands for multidimensional data pr ocessing and decreasing computational cost has caused the topic to rise to prominence once again [8]–[13]. The main disadvantages with these current solutions is that there is no direct control over the sparsity and it is difficult (and nonintuitive) to select their optimal hyperparameters. In add ition, the computational complexity of most of these methods is too high for practical applications with high dimensional data sets. Sparse CCA has also been implicitly addressed in [9], [14] an d is intimately related to the recent results on sparse PCA [9] , [15]–[17]. Indeed, our proposed solution is an extension of the results in [17] to CCA. The main contribution of this work is twofold. First, we derive CCA algorithms with direct control over the sparsity in each of the multivariates and examine their performance. Our computationally efficient methods are specifically aime d at understanding the relations between two data sets of larg e dimensions. We adopt a forward (or backward) greedy approach which is based on sequentially picking (or dropping) variables. At each stage, we bound the optimal CCA solution and bypass the need to resolve the full problem. Moreover, the computational complexity of the forward greedy method does not depend on the dimensions of the data but only on the sparsity parameters. Numerical simulation results show th at a significant portion of the correlation can be efficiently cap tured using a relatively low number of non-zero coefficients. Our second contribution is investigation of sparse CCA as a regularization method. Using empirical simulations we examin e the use of the different algorithms when the dimensions of the multivariates are larger than (or of the same order of) the number of samples and demonstrate the advantage of sparse CCA. In this context, one of the advantages of the greedy approach is that it generates the full sparsity path i n a single run and allows for efficient parameter tuning using

Extreme learning machine (ELM) has become a popular topic in machine learning in recent years. ELM is a new kind of single-hidden layer feedforward neural network with an extremely low computational cost. ELM, however, has two evident drawbacks: 1) the output weights solved by Moore-Penrose generalized inverse is a least squares minimization issue, which easily suffers from overfitting and 2) the accuracy of ELM is drastically sensitive to the number of hidden neurons so that a large model is usually generated. This brief presents a sparse Bayesian approach for learning the output weights of ELM in classification. The new model, called Sparse Bayesian ELM (SBELM), can resolve these two drawbacks by estimating the marginal likelihood of network outputs and automatically pruning most of the redundant hidden neurons during learning phase, which results in an accurate and compact model. The proposed SBELM is evaluated on wide types of benchmark classification problems, which verifies that the accuracy of SBELM model is relatively insensitive to the number of hidden neurons; and hence a much more compact model is always produced as compared with other state-of-the-art neural network classifiers.

Can one parallelize complex exploration– exploitation tradeoffs? As an example, consider the problem of optimal highthroughput experimental design, where we wish to sequentially design batches of experiments in order to simultaneously learn a surrogate function mapping stimulus to response and identify the maximum of the function. We formalize the task as a multiarmed bandit problem, where the unknown payoff function is sampled from a Gaussian process (GP), and instead of a single arm, in each round we pull a batch of several arms in parallel. We develop GP-BUCB, a principled algorithm for choosing batches, based on the GP-UCB algorithm for sequential GP optimization. We prove a surprising result; as compared to the sequential approach, the cumulative regret of the parallel algorithm only increases by a constant factor independent of the batch size B. Our results provide rigorous theoretical support for exploiting parallelism in Bayesian global optimization. We demonstrate the effectiveness of our approach on two real-world applications.

A novel measurement technique of the transfer function of weakly not-linear, approximately time-invariant systems is presented. The method is implemented with low-cost instrumentation; it is based on an exponentially-swept sine signal. It is applicable to loudspeakers and other audio components, but also to room acoustics measurements. The paper presents theoretical description of the method and experimental verification in comparison with MLS.

In this paper, we propose a novel speech dereverberation framework that utilizes deep neural network (DNN)-based spectrum estimation to construct linear inverse filters. The proposed dereverberation framework is based on the state-of-the-art inverse filter estimation algorithm called weighted prediction error (WPE) algorithm, which is known to effectively reduce reverberation and greatly boost the ASR performance in various conditions. In WPE, the accuracy of the inverse filter estimation, and thus the deverberation performance, is largely dependent on the estimation of the power spectral density (PSD) of the target signal. Therefore, the conventional WPE iteratively performs the inverse filter estimation, actual dereverberation and the PSD estimation to gradually improve the PSD estimate. However, while such iterative procedure works well when sufficiently long acoustically-stationary observed signals are available, WPE’s performance degrades when the duration of observed/accessible data is short, which typically is the case for real-time applications using online block-batch processing with small batches. To solve this problem, we incorporate the DNN-based spectrum estimator into the framework ofWPE, because a DNN can estimate the PSD robustly even from very short observed data. We experimentally show that the proposed framework outperforms the conventional WPE, and improves the ASR performance in real noisy reverberant environments in both single-channel and multichannel cases.

The management of uncertainty is crucial when harvesting structured content from unstructured and noisy sources. Knowledge Graphs (KGs) are a prominent example. KGs maintain both numerical and non-numerical facts, with the support of an underlying schema. These facts are usually accompanied by a confidence score that witnesses how likely is for them to hold. Despite their popularity, most of existing KGs focus on static data thus impeding the availability of timewise knowledge. What is missing is a comprehensive solution for the management of uncertain and temporal data in KGs. The goal of this paper is to fill this gap. We rely on two main ingredients. The first is a numerical extension of Markov Logic Networks (MLNs) that provide the necessary underpinning to formalize the syntax and semantics of uncertain temporal KGs. The second is a set of Datalog constraints with inequalities that extend the underlying schema of the KGs and help to detect inconsistencies. From a theoretical point of view, we discuss the complexity of two important classes of queries for uncertain temporal KGs: maximum a-posteriori and conditional probability inference. Due to the hardness of these problems and the fact that MLN solvers do not scale well, we also explore the usage of Probabilistic Soft Logics (PSL) as a practical tool to support our reasoning tasks. We report on an experimental evaluation comparing the MLN and PSL approaches.

Uncertainty is intrinsic to a wide spectrum of real-life applications, which inevitably applies to graph data. Representative uncertain graphs are seen in bio-informatics, social networks, etc. This paper motivates the problem of frequent subgraph mining on single uncertain graphs. We present an enumeration-evaluation algorithm to solve the problem. By showing support computation on an uncertain graph is #P-hard, we develop an approximation algorithm with accuracy guarantee for this purpose. To enhance the solution, we devise optimization techniques to achieve better mining performance. Experiment results on real-life data confirm the usability of the algorithm.


Question: Enhancing the sparse spectrum approximation of Gaussian processes by incorporating uncertainty in frequency inputs.

This paper introduces an efficient technique for scaling up kernel machines by utilizing random features to approximate the kernel function, thereby enabling the application of fast linear methods. Our approach involves projecting the input data onto a randomized low-dimensional feature space, with the specific design of these features ensuring that the inner products of the transformed data closely resemble those in the high-dimensional feature space of a chosen shift-invariant kernel. We investigate two distinct sets of random features and present theoretical bounds on their approximation accuracy for various radial basis kernels. Through extensive experiments on large-scale classification and regression tasks, we demonstrate that linear machine learning algorithms, when augmented with our randomized features, significantly outperform current state-of-the-art large-scale kernel machines in terms of both computational efficiency and predictive performance. This innovative method not only accelerates the training process but also maintains the accuracy of kernel machines, making it a promising solution for handling big data scenarios where traditional kernel methods become computationally prohibitive.

Declarative queries are proving to be an attractive paradigm for interacting with networks of wireless sensors. The metaphor that “the sensornet is a database” is problematic, however, because sensors do not exhaustively represent the data in the real world. In order to map the raw sensor readings onto physical reality, a modelof that reality is required to complement the readings. In this paper, we enrich interactive sensor querying with statistical modeling techniques. We demonstrate that such models can help provide answers that are both more meaningful, and, by introducing approximations with probabilistic confidences, significantly more efficient to compute in both time and energy. Utilizing the combination of a model and live data acquisition raises the challenging optimization problem of selecting the best sensor readings to acquire, balancing the increase in the confidence of our answer against the communication and data acquisition costs in the network. We describe an exponential time algorithm for finding the optimal solution to this optimization problem, and a polynomial-time heuristic for identifying solutions that perform well in practice. We evaluate our approach on several real-world sensor-network data sets, taking into account the real measured data and communication quality, demonstrating that our model-based approach provides a high-fidelity representation of the real phenomena and leads to significant performance gains versus traditional data acquisition techniques.

This paper introduces an approach for flexible, robust Bayesian modeling of structure in spherical data sets. Our method is based upon a recent construction called the needlet, which is a particular form of spherical wavelet with many favorable statistical and computational properties. We perform shrinkage and selection of needlet coefficients, focusing on two main alternatives: empirical-Bayes thresholding for selection, and the horseshoe prior for shrinkage. We study the performance of the proposed methodology both on simulated data and on a real data set involving the cosmic microwave background radiation. Horseshoe shrinkage of needlet coefficients is shown to yield the best overall performance against some common benchmarks.

We present a new method for solving stochastic differential equations based on Galerkin projections and extensions of Wiener’s polynomial chaos. Specifically, we represent the stochastic processes with an optimum trial basis from the Askey family of orthogonal polynomials that reduces the dimensionality of the system and leads to exponential convergence of the error. Several continuous and discrete processes are treated, and numerical examples show substantial speed-up compared to Monte Carlo simulations for low dimensional stochastic inputs.

Active and semi-supervised learning are important techniques when labeled data are scarce. We combine the two under a Gaussian random field model. Labeled and unlabeled data are represented as vertices in a weighted graph, with edge weights encoding the similarity between instances. The semi-supervised learning problem is then formulated in terms of a Gaussian random field on this graph, the mean of which is characterized in terms of harmonic functions. Active learning is performed on top of the semisupervised learning scheme by greedily selecting queries from the unlabeled data to minimize the estimated expected classification error (risk); in the case of Gaussian fields the risk is efficiently computed using matrix methods. We present experimental results on synthetic data, handwritten digit recognition, and text classification tasks. The active learning scheme requires a much smaller number of queries to achieve high accuracy compared with random query selection. 1. Introduction Semi-supervised learning targets the common situation where labeled data are scarce but unlabeled data are abundant. Under suitable assumptions, it uses unlabeled data to help supervised learning tasks. Various semi-supervised learning methods have been proposed and show promising results; Seeger (2001) gives a survey. These methods typically assume that the labeled data set is given and fixed. In practice, it may make sense to utilize active learning in conjunction with semi-supervised learning. That is, we might allow the learning algorithm to pick a set of unlabeled instances to be labeled by a domain expert, which will then be used as (or to augment) the labeled data set. In other words, if we have to label a few instances for semisupervised learning, it may be attractive to let the learning algorithm tell us which instances to label, rather than selecting them randomly. We will limit the range of query selection to the unlabeled data set, a practice known as poolbased active learning or selective sampling. There has been a great deal of research in active learning. For example, Tong and Koller (2000) select queries to minimize the version space size for support vector machines; Cohn et al. (1996) minimize the variance component of the estimated generalization error; Freund et al. (1997) employ a committee of classifiers, and query a point whenever the committee members disagree. Most of the active learning methods do not take further advantage of the large amount of unlabeled data once the queries are selected. Exceptions include McCallum and Nigam (1998) who use EM with unlabeled data integrated into the active learning, and Muslea et al. (2002) who use a semi-supervised learning method during training. In addition to this body of work from the machine learning community, there is a large literature on the closely related topic of experimental design in statistics; Chaloner and Verdinelli (1995) give a survey of experimental design from a Bayesian perspective. Recently Zhu et al. (2003) introduced a semi-supervised learning framework which is based on Gaussian random fields and harmonic functions. In this paper we demonstrate how this framework allows a combination of active learning and semi-supervised learning. In brief, the framework allows one to efficiently estimate the expected generalization error after querying a point, which leads to a better query selection criterion than naively selecting the point with maximum label ambiguity. Then, once the queries are selected and added to the labeled data set, the classifier can be trained using both the labeled and remaining unlabeled data. We present results on synthetic data, text classificaProceedings of the ICML-2003 Workshop on The Continuum from Labeled to Unlabeled Data, Washington DC, 2003. tion and image classification tasks that indicate the combination of these techniques can result in highly effective learning schemes. 2. Gaussian random fields and harmonic energy minimizing functions We begin by briefly describing the semi-supervised learning framework of Zhu et al. (2003). There are labeled points , and unlabeled points ; usually . We will use , ! to denote the labeled and unlabeled set, and "$#% '&( the total number of points. We assume the labels are binary: *),+.-0/1 243 . We assume a connected graph 56#6 87' 9: is given with nodes 7 corresponding to the " data points, of which the nodes in are labeled with the corresponding ’s. The edges 9 are represented by an "<;=" weight matrix > which is given. For example if ?+A@CB , > can be the radial basis function (RBF): DFEHG #JI KML NPO 2 Q R B STVU E T O G T R W (1) so that nearby points in Euclidean space are assigned large edge weights. Other weightings are possible, of course, and may be more appropriate when is discrete or symbolic. For our purposes the matrix > fully specifies the data manifold structure. We note that a method for learning the scale parameter Q is proposed in (Zhu et al., 2003). The semi-supervised algorithm in this paper is based on a relaxation of the requirement that labels are binary, resulting in a simple and tractable family of algorithms. We allow continuous labels on unlabeled nodes MXZY[7]\^@ . The labels on labeled data are still constrained to be 0 or 1: _` C#a 4)b c_ d+A-e/ 243 for _P#(24 f . We also denote the constraint by g ) U h i . Since we want unlabeled points that are nearby in the graph to have similar labels, we define the energy to be 9j c 1 C# 2 k S E l G D EmG _` O onp R (2) so that low energy corresponds to a slowly varying function over the graph. Define the diagonal matrix q6#sr*_`t*u[ 8r E whose entries r E #wv G DFEmG are the weighted degrees of the nodes in 5 . The combinatorial Laplacian is the "x;y" matrix z{#aq O > . We can rewrite the energy function in matrix form as 9| c 1 #( } z: . We consider the Gaussian random field ~ c 1 C# 2 P I KML< OF 9j c 1 f where  is an “inverse temperature” parameter, and   is the partition function  #s h  i U h i I KpL< OF 9j M r4 . The Gaussian random field differs from the “standard” discrete Markov random field in that the field configurations are over a continuous state space. Moreover, for a Gaussian field the joint probability distribution over unlabeled nodes is Gaussian with covariance matrix  z| 0 . z 0 is the submatrix of z corresponding to unlabeled data. The minimum energy function # arg min h  i U h i 9| c 1 of the Gaussian field is harmonic; that is, it satisfies zA#/ on unlabeled data points ! , and is equal to ) on the labeled data points . The harmonic property means that the value at each unlabeled node is the average of neighboring nodes: b onp C# 2 r G S EG DEmG b _` for n#Z &a24 f &? which is consistent with our prior notion of smoothness with respect to the graph. Because of the maximum principle of harmonic functions (Doyle & Snell, 1984),  is unique. Furthermore,  satisfies /(b onp 2 for n+! when ! is connected and labeled nodes from both classes are present at the boundary (the usual case; otherwise  takes on the extremum 0 or 1). By definition  is the mode of the Gaussian random field, but since the joint distribution is Gaussian,  is also the mean of the field. The harmonic energy minimizing function  can be computed with matrix methods. We partition the Laplacian matrix z into blocks for labeled and unlabeled nodes, z# z m z z e z 0  and let ,#  0   where  # ) , and  denotes the mean values on the unlabeled data points. The solution is given by  # O z  0 z e  (3) It is not hard to show that the Gaussian field, conditioned on labeled data, is a multivariate normal: * ya   fz  0 . To carry out classification with a Gaussian field, we note that the harmonic energy minimizing function  is the mean of the field. Therefore the Bayes classification rule is to label node _ as class 1 in case b c_ dJ/1 H , and to label node _ class 0 otherwise. The harmonic function  has many nice interpretations, of which the random walk view is particularly relevant here. Define the transition matrix # q  > . Consider the random walk on graph 5 by a particle. Starting from an unlabeled node _ , it moves to a node n with probability  EHG after one step. The walk continues until the particle reaches a labeled node. Then  E is the probability that the particle, starting from node _ , reaches a labeled node with label 1. The labeled data are the absorbing boundary for the random walk. More on this semi-supervised learning framework can be found in (Zhu et al., 2003). 3. Active learning We propose to perform active learning with the Gaussian random field model by greedily selecting queries from the unlabeled data to minimize the risk of the harmonic energy minimization function. The risk is the estimated generalization error of the Bayes classifier, and can be efficiently computed with matrix methods. We define the true risk  of the Bayes classifier based on the harmonic function  to be  # S E U S h U l sgn  E # E ~ E g where sgn 8 E is the Bayes decision rule, where (with a slight abuse of notation) sgn  E # 2 if  E ^/  and sgn  E #,/ otherwise. Here ~ E g is the unknown true label distribution at node _ , given the labeled data. Because of this 8 is not computable. In order to proceed, it is necessary to make assumptions. We begin by assuming that we can estimate the unknown distribution ~ c E g d with the mean of the Gaussian field model: ~ c E # 2pg d Z E Intuitively, recalling  E is the probability of reaching 1 in a random walk on the graph, our assumption is that we can approximate the distribution using a biased coin at each node, whose probability of heads is  E . With this assumption we can compute the estimated risk 8 as  8 # S E U sgn  E # / 2 O  E & sgn 8 E #(2  E # S E U  E 2 O  E (4) If we perform active learning and query an unlabeled node , we will receive an answer (0 or 1). Adding this point to the tr

Weighted-norms can arise in several situations. A zero/one weighted-norm, for example, arises when some of the entries in the matrix are not observed. External estimates of the noise variance associated with each measurement may be available (e.g. gene expression analysis) and using weights inversely proportional to the noise variance can lead to better reconstruction of the underlying structure. In other applications, entries in the target matrix represent aggregates of many samples. When using unweighted low-rank approximations (e.g. for separating style and content [4]), we assume a uniform number of samples for each entry. By incorporating weights, we can account for varying numbers of samples in such situations. Low-rank approximations are also used in the design of two-dimensional digital filters, in which case weights might arise from constraints of varying importance [3].

This paper reports our first set of results on managing uncertainty in data integration. We posit that data-integration systems need to handle uncertainty at three levels and do so in a principled fashion. First, the semantic mappings between the data sources and the mediated schema may be approximate because there may be too many of them to be created and maintained or because in some domains (e.g., bioinformatics) it is not clear what the mappings should be. Second, the data from the sources may be extracted using information extraction techniques and so may yield erroneous data. Third, queries to the system may be posed with keywords rather than in a structured form. As a first step to building such a system, we introduce the concept of probabilistic schema mappings and analyze their formal foundations. We show that there are two possible semantics for such mappings: by-table semantics assumes that there exists a correct mapping but we do not know what it is; by-tuple semantics assumes that the correct mapping may depend on the particular tuple in the source data. We present the query complexity and algorithms for answering queries in the presence of probabilistic schema mappings, and we describe an algorithm for efficiently computing the top-k answers to queries in such a setting. Finally, we consider using probabilistic mappings in the scenario of data exchange.

We study the notion of Compressed Sensing (CS) as put forward in [14] and related work [20, 3, 4]. The basic idea behind CS is that a signal or image, unknown but supposed to be compressible by a known transform, (eg. wavelet or Fourier), can be subjected to fewer measurements than the nominal number of pixels, and yet be accurately reconstructed. The samples are nonadaptive and measure ‘random’ linear combinations of the transform coefficients. Approximate reconstruction is obtained by solving for the transform coefficients consistent with measured data and having the smallest possible ` norm. We perform a series of numerical experiments which validate in general terms the basic idea proposed in [14, 3, 5], in the favorable case where the transform coefficients are sparse in the strong sense that the vast majority are zero. We then consider a range of less-favorable cases, in which the object has all coefficients nonzero, but the coefficients obey an ` bound, for some p ∈ (0, 1]. These experiments show that the basic inequalities behind the CS method seem to involve reasonable constants. We next consider synthetic examples modelling problems in spectroscopy and image processing, and note that reconstructions from CS are often visually “noisy” . We post-process using translation-invariant de-noising, and find the visual appearance considerably improved. We also consider a multiscale deployment of compressed sensing, in which various scales are segregated and CS applied separately to each; this gives much better quality reconstructions than a literal deployment of the CS methodology. We also report that several workable families of ‘random’ linear combinations all behave equivalently, including random spherical, random signs, partial Fourier and partial Hadamard. These results show that, when appropriately deployed in a favorable setting, the CS framework is able to save significantly over traditional sampling, and there are many useful extensions of the basic idea.

Three Bayesian ideas are presented for supervised adaptive classifiers. First, it is argued that the output of a classifier should be obtained by marginalizing over the posterior distribution of the parameters; a simple approximation to this integral is proposed and demonstrated. This involves a "moderation" of the most probable classifier's outputs, and yields improved performance. Second, it is demonstrated that the Bayesian framework for model comparison described for regression models in MacKay (1992a,b) can also be applied to classification problems. This framework successfully chooses the magnitude of weight decay terms, and ranks solutions found using different numbers of hidden units. Third, an information-based data selection criterion is derived and demonstrated within this framework.

Neural networks are known to be a class of highly expressive functions able to fit even random input-output mappings with 100% accuracy. In this work we present properties of neural networks that complement this aspect of expressivity. By using tools from Fourier analysis, we show that deep ReLU networks are biased towards low frequency functions, meaning that they cannot have local fluctuations without affecting their global behavior. Intuitively, this property is in line with the observation that over-parameterized networks find simple patterns that generalize across data samples. We also investigate how the shape of the data manifold affects expressivity by showing evidence that learning high frequencies gets easier with increasing manifold complexity, and present a theoretical understanding of this behavior. Finally, we study the robustness of the frequency components with respect to parameter perturbation, to develop the intuition that the parameters must be finely tuned to express high frequency functions.

A novel measurement technique of the transfer function of weakly not-linear, approximately time-invariant systems is presented. The method is implemented with low-cost instrumentation; it is based on an exponentially-swept sine signal. It is applicable to loudspeakers and other audio components, but also to room acoustics measurements. The paper presents theoretical description of the method and experimental verification in comparison with MLS.


Question: Enhancing the sparse spectrum approximation of Gaussian processes by incorporating uncertainty in frequency inputs.

This paper outlines a comprehensive and practical Bayesian framework for the training of feedforward networks to learn mappings. The framework enables (1) the objective comparison of solutions derived from different network architectures, (2) the development of objective stopping criteria for network pruning or expansion, (3) the objective determination of the scale and nature of weight decay terms or additional regularizers to prevent overlarge weights, (4) a quantifiable assessment of the number of parameters in a model that are well-determined, (5) precise estimates of the uncertainty associated with network parameters and outputs, and (6) the objective evaluation of alternative learning and interpolation methods, such as splines and radial basis functions. The Bayesian "evidence" inherently incorporates "Occam's razor," discouraging overly complex or flexible models. The Bayesian methodology facilitates the identification of weaknesses in underlying learning models. When the learning models are well-suited to the problem at hand, a strong correlation is observed between the Bayesian evidence and the model's generalization capability.

Given i.i.d. observations of a random vector X ∈ R, we study the problem of estimating both its covariance matrix Σ, and its inverse covariance or concentration matrix Θ = (Σ). When X is multivariate Gaussian, the non-zero structure of Θ is specified by the graph of an associated Gaussian Markov random field; and a popular estimator for such sparse Θ is the l1-regularized Gaussian MLE. This estimator is sensible even for for non-Gaussian X, since it corresponds to minimizing an l1-penalized log-determinant Bregman divergence. We analyze its performance under high-dimensional scaling, in which the number of nodes in the graph p, the number of edges s, and the maximum node degree d, are allowed to grow as a function of the sample size n. In addition to the parameters (p, s, d), our analysis identifies other key quantities that control rates: (a) the l∞-operator norm of the true covariance matrix Σ; and (b) the l∞ operator norm of the sub-matrix Γ∗SS , where S indexes the graph edges, and Γ∗ = (Θ) ⊗ (Θ); and (c) a mutual incoherence or irrepresentability measure on the matrix Γ∗ and (d) the rate of decay 1/f(n, δ) on the probabilities {|Σ̂ij − Σ∗ij | > δ}, where Σ̂ is the sample covariance based on n samples. Our first result establishes consistency of our estimate Θ̂ in the elementwise maximum-norm. This in turn allows us to derive convergence rates in Frobenius and spectral norms, with improvements upon existing results for graphs with maximum node degrees d = o( √ s). In our second result, we show that with probability converging to one, the estimate Θ̂ correctly specifies the zero pattern of the concentration matrix Θ∗. We illustrate our theoretical results via simulations for various graphs and problem parameters, showing good correspondences between the theoretical predictions and behavior in simulations. AMS 2000 subject classifications: Primary 62F12; secondary 62F30.

A continuous spatial model can be constructed by convolving a very simple, perhaps independent, process with a kernel or point spread function. This approach for constructing a spatial process offers a number of advantages over specification through a spatial covariogram. In particular, this process convolution specification leads to compuational simplifications and easily extends beyond simple stationary models. This paper uses process convolution models to build space and space-time models that are flexible and able to accomodate large amounts of data. Data from environmental monitoring is considered.

In this paper, we propose a method for robust kernel density estimation. We interpret a KDE with Gaussian kernel as the inner product between a mapped test point and the centroid of mapped training points in kernel feature space. Our robust KDE replaces the centroid with a robust estimate based on M-estimation (P. Huber, 1981), The iteratively re-weighted least squares (IRWLS) algorithm for M-estimation depends only on inner products, and can therefore be implemented using the kernel trick. We prove the IRWLS method monotonically decreases its objective value at every iteration for a broad class of robust loss functions. Our proposed method is applied to synthetic data and network traffic volumes, and the results compare favorably to the standard KDE.

In this paper we model the components of the compressive sensing (CS) problem using the Bayesian framework by utilizing a hierarchical form of the Laplace prior to model sparsity of the unknown signal. This signal prior includes some of the existing models as special cases and achieves a high degree of sparsity. We develop a constructive (greedy) algorithm resulting from this formulation where necessary parameters are estimated solely from the observation and therefore no user-intervention is needed. We provide experimental results with synthetic 1D signals and images, and compare with the state-of-the-art CS reconstruction algorithms demonstrating the superior performance of the proposed approach.

In this paper, we will consider the approximation properties of a recently introduced neural network model called graph neural network (GNN), which can be used to process-structured data inputs, e.g., acyclic graphs, cyclic graphs, and directed or undirected graphs. This class of neural networks implements a function tau(G, n) isin R m that maps a graph G and one of its nodes n onto an m-dimensional Euclidean space. We characterize the functions that can be approximated by GNNs, in probability, up to any prescribed degree of precision. This set contains the maps that satisfy a property called preservation of the unfolding equivalence, and includes most of the practically useful functions on graphs; the only known exception is when the input graph contains particular patterns of symmetries when unfolding equivalence may not be preserved. The result can be considered an extension of the universal approximation property established for the classic feedforward neural networks (FNNs). Some experimental examples are used to show the computational capabilities of the proposed model.

Spectral feature selection identifies relevant features by measuring their capability of preserving sample similarity. It provides a powerful framework for both supervised and unsupervised feature selection, and has been proven to be effective in many real-world applications. One common drawback associated with most existing spectral feature selection algorithms is that they evaluate features individually and cannot identify redundant features. Since redundant features can have significant adverse effect on learning performance, it is necessary to address this limitation for spectral feature selection. To this end, we propose a novel spectral feature selection algorithm to handle feature redundancy, adopting an embedded model. The algorithm is derived from a formulation based on a sparse multi-output regression with a L2;1-norm constraint. We conduct theoretical analysis on the properties of its optimal solutions, paving the way for designing an efficient path-following solver. Extensive experiments show that the proposed algorithm can do well in both selecting relevant features and removing redundancy.

In this paper, we develop a framework for information theore tic l arning based on infinitely divisible matrices. We formulate an entropy-lik e functional on positive definite matrices based on Renyi’s axiomatic definition of en tropy and examine some key properties of this functional that lead to the conce pt of infinite divisibility. The proposed formulation avoids the plug in estimation of density and brings along the representation power of reproducing kernel Hilbe rt spaces. As an application example, we derive a supervised metric learning algo rithm using a matrix based analogue to conditional entropy achieving results co mparable with the state of the art.

We consider the problem of estimating sparse graphs by a lasso penalty applied to the inverse covariance matrix. Using a coordinate descent procedure for the lasso, we develop a simple algorithm--the graphical lasso--that is remarkably fast: It solves a 1000-node problem ( approximately 500,000 parameters) in at most a minute and is 30-4000 times faster than competing methods. It also provides a conceptual link between the exact problem and the approximation suggested by Meinshausen and Bühlmann (2006). We illustrate the method on some cell-signaling data from proteomics.

We focus in this work on the estimation of the first k eigenvectors of any graph Laplacian using filtering of Gaussian random signals. We prove that we only need k such signals to be able to exactly recover as many of the smallest eigenvectors, regardless of the number of nodes in the graph. In addition, we address key issues in implementing the theoretical concepts in practice using accurate approximated methods. We also propose fast algorithms both for eigenspace approximation and for the determination of the kth smallest eigenvalue λk. The latter proves to be extremely efficient under the assumption of locally uniform distribution of the eigenvalue over the spectrum. Finally, we present experiments which show the validity of our method in practice and compare it to state-of-the-art methods for clustering and visualization both on synthetic smallscale datasets and larger real-world problems of millions of nodes. We show that our method allows a better scaling with the number of nodes than all previous methods while achieving an almost perfect reconstruction of the eigenspace formed by the first k eigenvectors. Keywords—Graph signal processing, low-rank reconstruction, partitionning, spectral graph theory, spectrum analysis, subspace approximation, visualization

Introduction In this chapter we will examine how we can generalize the idea of transforming a time series into an alternative representation, such as the Fourier (frequency) domain, to facilitate systematic methods of either removing (filtering) or adding (interpolating) data. In particular, we will examine the techniques of Principal Component Analysis (PCA) using Singular Value Decomposition (SVD), and Independent Component Analysis (ICA). Both of these techniques utilize a representation of the data in a statistical domain rather than a time or frequency domain. That is, the data are projected onto a new set of axes that fulfill some statistical criterion, which implies independence, rather than a set of axes that represent discrete frequencies such as with the Fourier transform, where the independence is assumed. Another important difference between these statistical techniques and Fourier-based techniques is that the Fourier components onto which a data segment is projected are fixed, whereas PCA-or ICA-based transformations depend on the structure of the data being analyzed. The axes onto which the data are projected are therefore discovered. If the structure of the data (or rather the statistics of the underlying sources) changes over time, then the axes onto which the data are projected will change too 1. Any projection onto another set of axes (or into another space) is essentially a method for separating the data out into separate components or sources which will hopefully allow us to see important structure more clearly in a particular projection. That is, the direction of projection increases the signal-to-noise ratio (SNR) for a particular signal source. For example, by calculating the power spectrum of a segment of data, we hope to see peaks at certain frequencies. The power (amplitude squared) along certain frequency vectors is therefore high, meaning we have a strong component in the signal at that frequency. By discarding the projections that correspond to the unwanted sources (such as the noise or artifact sources) and inverting the transformation, we effectively perform a filtering of the recorded observation. This is true for both ICA and PCA as well as Fourier-based techniques. However, one important difference between these techniques is that Fourier techniques assume that the projections onto each frequency component are independent of the other frequency components. In PCA and ICA we attempt to find a set of axes which are independent of one another in some sense. We assume there are a set of independent 1 …


Question: Specific appliance power consumption classification and disaggregation.

This paper presents an investigation into the application of deep neural networks for energy disaggregation, a process aimed at estimating the individual electricity consumption of appliances from aggregate household data. Inspired by the success of deep learning in domains like image classification and speech recognition, we explore three distinct neural network architectures for this task: 1) Long Short-Term Memory (LSTM) networks, a type of recurrent neural network; 2) denoising autoencoders; and 3) a regression network that predicts the start time, end time, and average power demand of appliance usage. To evaluate the performance of these algorithms, we employ a comprehensive set of seven metrics on real-world aggregate power data from five different appliances. Our tests are conducted on both a house that was not included in the training dataset and on houses that were part of the training set. The results demonstrate that all three neural network models outperform traditional combinatorial optimization and factorial hidden Markov models in terms of F1 scores, averaged across the five appliances. Moreover, our neural network algorithms exhibit strong generalization capabilities, performing well even on a house not encountered during the training phase.

Energy-proportional designs would enable large energy savings in servers, potentially doubling their efficiency in real-life use. Achieving energy proportionality will require significant improvements in the energy usage profile of every system component, particularly the memory and disk subsystems.

The present paper considers some technical issues related to the “energy blockchain” paradigm applied to microgrids. In particular, what appears from the study is that the superposition of energy transactions in a microgrid creates a variation of the power losses in all the branches of the microgrid. Traditional power losses allocation in distribution systems takes into account only generators while, in this paper, a real-time attribution of power losses to each transaction involving one generator and one load node is done by defining some suitable indices. Besides, the presence of P–V nodes increases the level of reactive flows and provides a more complex technical perspective. For this reason, reactive power generation for voltage support at P–V nodes poses a further problem of reactive power flow exchange, which is worth of investigation in future works in order to define a possible way of remuneration. The experimental section of the paper considers a medium voltage microgrid and two different operational scenarios.

The double-layer capacitor (DLC) for power applications is a new device. A simple resistive capacitive equivalent circuit is insufficient to characterize its terminal behaviour. Based on physical reasoning, an equivalent circuit is proposed to provide the power electronics engineer with a model for the terminal behaviour of the DLC. The equivalent model consists of three Re branches, one of them with a voltage dependent capacitor. A method to identify the circuit parameters is presented. Measurements of carbon-based DLCs for power applications are presented, analysed, and the equivalent circuit response is compared with experimental results.

As electric vehicles (EVs) take a greater share in the personal automobile market, their penetration will cause overload conditions at the distribution transformer. This paper focuses on the impacts of charging EVs on residential distribution networks including the transformer. The cost to accommodate a large-scale EV penetration by upgrading distribution transformers can be prohibitive. To alleviate the potential new load peaks with minimal infrastructure investments, a demand response strategy is proposed as a load shaping tool that allows improvement in distribution transformer usage. With the proposed strategy, consumers' preferences, load priorities, and privacy can be taken into account.

We present results of a systematic investigation of leakage of compromising information via electromagnetic (EM) emanations from CMOS devices. These emanations are shown to consist of a multiplicity of signals, each leaking somewhat different information about the underlying computation. We show that not only can EM emanations be used to attack cryptographic devices where the power side–channel is unavailable, they can even be used to break power analysis countermeasures.

Demand-side management, together with the integration of distributed energy generation and storage, are considered increasingly essential elements for implementing the smart grid concept and balancing massive energy production from renewable sources. We focus on a smart grid in which the demand-side comprises traditional users as well as users owning some kind of distributed energy sources and/or energy storage devices. By means of a day-ahead optimization process regulated by an independent central unit, the latter users intend to reduce their monetary energy expense by producing or storing energy rather than just purchasing their energy needs from the grid. In this paper, we formulate the resulting grid optimization problem as a noncooperative game and analyze the existence of optimal strategies. Furthermore, we present a distributed algorithm to be run on the users' smart meters, which provides the optimal production and/or storage strategies, while preserving the privacy of the users and minimizing the required signaling with the central unit. Finally, the proposed day-ahead optimization is tested in a realistic situation.

Gesture recognition is a well-establish topic of research that is widely adopted for a broad range of applications. For instance, it can be exploited for the command of a smart environment without any remote control unit or even for the recognition of human activities from a set of video cameras deployed in strategic position. Many researchers working on assistive smart home, such as our team, believe that the intrusiveness of that technology will prevent the future adoption and commercialization of smart homes. In this paper, we propose a novel gesture recognition algorithm that is solely based on passive RFID technology. This technology enables the localization of small tags that can be embedded in everyday life objects (a cup or a book, for instance) while remaining non intrusive. However, until now, this technology has been largely ignored by researchers on gesture recognition, mostly because it is easily disturbed by noise (metal, human, etc.) and offer limited precision. Despite these issues, the localization algorithms have improved over the years, and our recent efforts resulted in a real-time tracking algorithm with a precision approaching 14cm. With this, we developed a gesture recognition algorithm able to perform segmentation of gestures and prediction on a spatio-temporal data series. Our new model, exploiting works on qualitative spatial reasoning, achieves recognition of 91%. Our goal is to ultimately use that knowledge for both human activity recognition and errors detection.

The intelligent home environment is a well-established example of the Ambient Intelligence application domain. A variety of sensors and actuators can be used to have the home environment adapt towards changing circumstances and user preferences. However, the complexity of how these intelligent home automation systems operate is often beyond the comprehension of non-technical users, and adding new technology to an existing infrastructure is often a burden. In this paper, we present a home automation framework designed based on smart widgets with a model driven methodology that raises the level of abstraction to configure home automation equipment. It aims to simplify user-level home automation management by mapping high-level home automation concepts onto a low-level composition and configuration of the automation building blocks with a reverse mapping to simplify the integration of new equipment into existing home automation systems. Experiments have shown that the mappings we proposed are sufficient to represent household appliances to the end user in a simple way and that new mappings can easily be added to our framework.

The limited available fossil fuels and the call for sustainable environment have brought about new technologies for the high efficiency in the use of fossil fuels and introduction of renewable energy. Smart grid is an emerging technology that can fulfill such demands by incorporating advanced information and communications technology (ICT). The pervasive deployment of the advanced ICT, especially the smart metering, will generate big energy data in terms of volume, velocity, and variety. The generated big data can bring huge benefits to the better energy planning, efficient energy generation, and distribution. As such data involve end users' privacy and secure operation of the critical infrastructure, there will be new security issues. This paper is to survey and discuss new findings and developments in the existing big energy data analytics and security. Several taxonomies have been proposed to express the intriguing relationships of various variables in the field.

With the proliferation of Internet of Things (IoT) devices such as smartphones, sensors, cameras, and RFIDs, it is possible to collect massive amount of data for localization and tracking of people within commercial buildings. Enabled by such occupancy monitoring capabilities, there are extensive opportunities for improving the energy consumption of buildings via smart HVAC control. In this respect, the major challenges we envision are 1) to achieve occupancy monitoring in a minimally intrusive way, e.g., using the existing infrastructure in the buildings and not requiring installation of any apps in the users' smart devices, and 2) to develop effective data fusion techniques for improving occupancy monitoring accuracy using a multitude of sources. This paper surveys the existing works on occupancy monitoring and multi-modal data fusion techniques for smart commercial buildings. The goal is to lay down a framework for future research to exploit the spatio-temporal data obtained from one or more of various IoT devices such as temperature sensors, surveillance cameras, and RFID tags that may be already in use in the buildings. A comparative analysis of existing approaches and future predictions for research challenges are also provided.


Question: Specific appliance power consumption classification and disaggregation.

This paper provides a comprehensive review of Non-Intrusive Load Monitoring (NILM) methodologies and techniques aimed at addressing the energy disaggregation challenge. Given the escalating urbanization in developing nations and the concurrent surge in high-rise construction, coupled with increased energy consumption, there is a pressing need for conservation and efficient energy management strategies. These strategies necessitate real-time monitoring of energy consumption by individual household appliances. The global adoption of smart meters in smart grid systems has catalyzed the growth of NILM, enabling the estimation of appliance-specific power usage based on the aggregated power consumption data of a building. NILM offers a cost-effective solution for real-time monitoring of appliance energy use, empowering households to better understand their consumption patterns and actively participate in energy conservation efforts. The paper delves into an in-depth analysis of NILM systems and their associated methods for energy disaggregation. It also surveys the latest NILM algorithms and evaluates their performance using various metrics. Additionally, the paper discusses the existing benchmarking frameworks for comparing NILM algorithms and explores potential use-cases for NILM. An overview of publicly available datasets is presented, along with an identification of current challenges and future research opportunities in the field of NILM.

Demand response is a critical part of renewable integration and energy cost reduction goals across the world. Motivated by the need to reduce costs arising from electricity shortage and renewable energy fluctuations, we propose a novel multiarmed bandit mechanism for demand response (MAB-MDR) which makes monetary offers to strategic consumers who have unknown response characteristics, to incetivize reduction in demand. Our work is inspired by a novel connection we make to crowdsourcing mechanisms. The proposed mechanism incorporates realistic features of the demand response problem including time varying and quadratic cost function. The mechanism marries auctions, that allow users to report their preferences, with online algorithms, that allow distribution companies to learn user-specific parameters. We show that MAB-MDR is dominant strategy incentive compatible, individually rational, and achieves sublinear regret. Such mechanisms can be effectively deployed in smart grids using new information and control architecture innovations and lead to welcome savings in energy costs.

In the development of smart grid, state estimation in distribution systems will likely face more challenges than that at the transmission level. This paper addresses one of such challenges, namely, the absence of topology information, by developing a forecast-aided topology change detection method and an event-triggered recursive Bayesian estimator to identify the correct topology. Simulation studies with microgrid induced changes are presented to illustrate the effectiveness of the proposed algorithm.

IEC 1000-3-2 regulations impose a reduced harmonic content on any converter with an input power higher than 75 W. However, if the power architecture of the system is based on small on-board converters, and the total power is higher than 75 W, IEC regulations must be fulfilled although each individual converter need not comply with the regulations. In this paper, one of the different possible solutions is presented. Each on-board converter has an active input current shaper (AICS) in order to reduce the input current harmonic content of each converter and, hence, to comply with IEC 1000-3-2 regulations. Moreover, two different types of AICSs were compared: the conventional one and a new type of AICS based on a full-wave rectifier.

A range of algorithms was used to classify online retail customers of a UK company using historical transaction data. The predictive capabilities of the classifiers were assessed using linear regression, Lasso and regression trees. Unlike most related studies, classifications were based upon specific and marketing focused customer behaviours. Prediction accuracy on untrained customers was generally better than 80%. The models implemented (and compared) for classification were: Logistic Regression, Quadratic Discriminant Analysis, Linear SVM, RBF SVM, Gaussian Process, Decision Tree, Random Forest and Multi-layer Perceptron (Neural Network). Postcode data was then used to classify solely on demographics derived from the UK Land Registry and similar public data sources. Prediction accuracy remained better than 60%.

Gesture recognition is a well-establish topic of research that is widely adopted for a broad range of applications. For instance, it can be exploited for the command of a smart environment without any remote control unit or even for the recognition of human activities from a set of video cameras deployed in strategic position. Many researchers working on assistive smart home, such as our team, believe that the intrusiveness of that technology will prevent the future adoption and commercialization of smart homes. In this paper, we propose a novel gesture recognition algorithm that is solely based on passive RFID technology. This technology enables the localization of small tags that can be embedded in everyday life objects (a cup or a book, for instance) while remaining non intrusive. However, until now, this technology has been largely ignored by researchers on gesture recognition, mostly because it is easily disturbed by noise (metal, human, etc.) and offer limited precision. Despite these issues, the localization algorithms have improved over the years, and our recent efforts resulted in a real-time tracking algorithm with a precision approaching 14cm. With this, we developed a gesture recognition algorithm able to perform segmentation of gestures and prediction on a spatio-temporal data series. Our new model, exploiting works on qualitative spatial reasoning, achieves recognition of 91%. Our goal is to ultimately use that knowledge for both human activity recognition and errors detection.

With the proliferation of Internet of Things (IoT) devices such as smartphones, sensors, cameras, and RFIDs, it is possible to collect massive amount of data for localization and tracking of people within commercial buildings. Enabled by such occupancy monitoring capabilities, there are extensive opportunities for improving the energy consumption of buildings via smart HVAC control. In this respect, the major challenges we envision are 1) to achieve occupancy monitoring in a minimally intrusive way, e.g., using the existing infrastructure in the buildings and not requiring installation of any apps in the users' smart devices, and 2) to develop effective data fusion techniques for improving occupancy monitoring accuracy using a multitude of sources. This paper surveys the existing works on occupancy monitoring and multi-modal data fusion techniques for smart commercial buildings. The goal is to lay down a framework for future research to exploit the spatio-temporal data obtained from one or more of various IoT devices such as temperature sensors, surveillance cameras, and RFID tags that may be already in use in the buildings. A comparative analysis of existing approaches and future predictions for research challenges are also provided.

The most recent Wireless Sensor Networks technologies can provide viable solutions to perform automatic monitoring of the water grid, and smart metering of water consumptions. However, sensor nodes located along water pipes cannot access power grid facilities, to get the necessary energy imposed by their working conditions. In this sense, it is of basic importance to design the network architecture in such a way as to require the minimum possible power. This paper investigates the suitability of the Wireless Metering Bus protocol for possible adoption in future smart water grids, by evaluating its transmission performance, through simulations and experimental tests executed by means of prototype sensor nodes.

In four university student kitchens over twenty-one days, we captured participants' food preparation activity, quantified the greenhouse gas emissions and direct energy connected to the food and cooking, and talked to participants about their food practices. Grounded in this uniquely detailed micro-account, our findings inform sustainable design for cooking and eating at home and quantify the potential impacts. We outline the relation of the impacts to our participants' approaches to everyday food preparation, the organisation of their time, and the role of social meals. Our technique allows evaluation of opportunities for sustainable intervention design: at the appliance, in the digitally-mediated organisation of meals and inventory management, and more broadly in reflecting upon and reshaping diet.

Global electrical grids are verging on the largest technological transformation since the introduction of electricity into the home. The antiquated infrastructure that delivers power to our homes and businesses is being replaced with a collection of digital systems called the smart grid. This grid is the modernization of the existing electrical system that enhances customers' and utilities' ability to monitor, control, and predict energy use.

Base stations represent the main contributor to the energy consumption of a mobile cellular network. Since traffic load in mobile networks significantly varies during a working or weekend day, it is important to quantify the influence of these variations on the base station power consumption. Therefore, this paper investigates changes in the instantaneous power consumption of GSM (Global System for Mobile Communications) and UMTS (Universal Mobile Telecommunications System) base stations according to their respective traffic load. The real data in terms of the power consumption and traffic load have been obtained from continuous measurements performed on a fully operated base station site. Measurements show the existence of a direct relationship between base station traffic load and power consumption. According to this relationship, we develop a linear power consumption model for base stations of both technologies. This paper also gives an overview of the most important concepts which are being proposed to make cellular networks more energy-efficient.