Question: Recurrence networks - an innovative framework for nonlinear time series examination.

This paper introduces recurrence plots, a novel graphical tool for analyzing the time constancy of dynamical systems, providing insights not easily attainable through conventional methods. Recurrence plots serve as a diagnostic tool, validating assumptions typically made in the computation of dynamical parameters from time series data. These parameters include the information dimension, entropy, Liapunov exponents, and dimension spectrum, all of which are derived under the assumption that the time series originates from an autonomous dynamical system, where the evolution equations do not explicitly contain time, and the series is significantly longer than the system's characteristic times. The recurrence plot is constructed by plotting dots in a NxN square grid, where N is the number of points on the orbit of a d-dimensional dynamical system. A dot is placed at coordinates (i, j) when the point x(j) is within a predefined radius of x(i). The radius r(i) is chosen such that it encompasses a reasonable number of other orbit points, typically around 10 neighbors. The recurrence plot is not fully symmetric due to the varying radius, but it does exhibit a degree of symmetry since proximity in one direction implies proximity in the other. This plot reveals natural time correlation information and can distinguish between different typologies based on the system's behavior. For autonomous systems with short characteristic times relative to the time series length, the recurrence plot appears homogeneous at large scales, with a uniform grey pattern and a potentially complex texture at smaller scales. An example of this is provided by the Henon map. In contrast, time evolution with drift, where parameters vary slowly over time, results in recurrence plots with a rich structure, as seen in the Lorenz system with a linearly varying term. This drift causes the plot to be paler away from the diagonal and darker near it, reflecting progressive decorrelation at larger time intervals. Another typology, referred to as periodic, is exemplified by an experimental time series featuring slow oscillations superimposed on chaotic motion. This typology is not easily discernible in the original time series, highlighting the unique insights provided by recurrence plots. The human eye-brain system is adept at identifying subtle variations in overall darkness, which can be further analyzed through histograms of darkness as a function of i-j, as demonstrated in the paper. This method reveals the existence of drift unequivocally, showcasing the power of recurrence plots in detecting subtle changes in dynamical systems.

The open-source software RRDtool and Cricket provide a solution to the problem of collecting, storing, and visualizing service network time series data for the real-time monitoring task. However, simultaneously monitoring all service network time series of interest is an impossible task even for the accomplished network technician. The solution is to integrate a mathematical model for automatic aberrant behavior detection in time series into the monitoring software. While there are many such models one might choose, the primary goal should be a model compatible with real-time monitoring. At WebTV, the solution was to integrate a model based on exponential smoothing and Holt-Winters forecasting into the Cricket/RRDtool architecture. While perhaps not optimal, this solution is flexible, efficient, and effective as a tool for automatic aberrant behavior detection.

In this paper I try to describe both the role of mathematics in shaping our understanding of how neural networks operate, and the curious new mathematical concepts generated by our attempts to capture neural networks in equations. My target reader being the non-expert, I will present a biased selection of relatively simple examples of neural network tasks, models and calculations, rather than try to give a full encyclopedic review-like account of the many mathematical developments in this eld.

Probabilistic forecasting, i.e. estimating the probability distribution of a time series’ future given its past, is a key enabler for optimizing business processes. In retail businesses, for example, forecasting demand is crucial for having the right inventory available at the right time at the right place. In this paper we propose DeepAR, a methodology for producing accurate probabilistic forecasts, based on training an auto-regressive recurrent network model on a large number of related time series. We demonstrate how by applying deep learning techniques to forecasting, one can overcome many of the challenges faced by widely-used classical approaches to the problem. We show through extensive empirical evaluation on several real-world forecasting data sets that our methodology produces more accurate forecasts than other state-of-the-art methods, while requiring minimal manual work.

The long short-term memory (LSTM) network trained by gradient descent solves difficult problems which traditional recurrent neural networks in general cannot. We have recently observed that the decoupled extended Kalman filter training algorithm allows for even better performance, reducing significantly the number of training steps when compared to the original gradient descent training algorithm. In this paper we present a set of experiments which are unsolvable by classical recurrent networks but which are solved elegantly and robustly and quickly by LSTM combined with Kalman filters.

On a new type of almost periodic time scales, a class of BAM neural networks is considered. By employing a fixed point theorem and differential inequality techniques, some sufficient conditions ensuring the existence and global exponential stability of C1-almost periodic solutions for this class of networks with time-varying delays are established. Two examples are given to show the effectiveness of the proposed method and results.

Intelligent monitoring and control of critical infrastructure such as electric power grids, public water utilities and transportation systems produces massive volumes of time series data from heterogeneous sensor networks. Time Series Forecasting (TSF) is essential for system safety and security, and also for improving the efficiency and quality of service delivery. Being highly dependent on various external factors, the observed system behavior is usually stochastic, which makes the next value prediction a tricky and challenging task that usually needs customized methods. In this paper we propose a novel deep learning based framework for time series analysis and prediction by ensembling parametric and nonparametric methods. Our approach takes advantage of extracting features at different time scales, which improves accuracy without compromising reliability in comparison with the state-of-the-art methods. Our experimental evaluation using real-world SCADA data from a municipal water management system shows that our proposed method outperforms the baseline methods evaluated here.

ACKNOWLEDGEMENT The timely and successful completion of the book could hardly be possible without the helps and supports from a lot of individuals. I will take this opportunity to thank all of them who helped me either directly or indirectly during this important work. First of all I wish to express my sincere gratitude and due respect to my supervisor Dr. R.K. Agrawal, Associate Professor SC & SS, JNU. I am immensely grateful to him for his valuable guidance, continuous encouragements and positive supports which helped me a lot during the period of my work. I would like to appreciate him for always showing keen interest in my queries and providing important suggestions. I also express whole hearted thanks to my friends and classmates for their care and moral supports. The moments, I enjoyed with them during my M.Tech course will always remain as a happy memory throughout my life. I owe a lot to my mother for her constant love and support. She always encouraged me to have positive and independent thinking, which really matter in my life. I would like to thank her very much and share this moment of happiness with her. Last but not the least I am also thankful to entire faculty and staff of SC & SS for their unselfish help, I got whenever needed during the course of my work.ABSTRACT Time series modeling and forecasting has fundamental importance to various practical domains. Thus a lot of active research works is going on in this subject during several years. Many important models have been proposed in literature for improving the accuracy and effeciency of time series modeling and forecasting. The aim of this book is to present a concise description of some popular time series forecasting models used in practice, with their salient features. In this book, we have described three important classes of time series models, viz. the stochastic, neural networks and SVM based models, together with their inherent forecasting strengths and weaknesses. We have also discussed about the basic issues related to time series modeling, such as stationarity, parsimony, overfitting, etc. Our discussion about different time series models is supported by giving the experimental forecast results, performed on six real time series datasets. While fitting a model to a dataset, special care is taken to select the most parsimonious one. To evaluate forecast accuracy as well as to compare among different models …

A simple linear neuron model with constrained Hebbian-type synaptic modification is analyzed and a new class of unconstrained learning rules is derived. It is shown that the model neuron tends to extract the principal component from a stationary input vector sequence.

Assessing directed functional connectivity from time series data is a key challenge in neuroscience. One approach to this problem leverages a combination of Granger causality analysis and network theory. This article describes a freely available MATLAB toolbox--'Granger causal connectivity analysis' (GCCA)--which provides a core set of methods for performing this analysis on a variety of neuroscience data types including neuroelectric, neuromagnetic, functional MRI, and other neural signals. The toolbox includes core functions for Granger causality analysis of multivariate steady-state and event-related data, functions to preprocess data, assess statistical significance and validate results, and to compute and display network-level indices of causal connectivity including 'causal density' and 'causal flow'. The toolbox is deliberately small, enabling its easy assimilation into the repertoire of researchers. It is however readily extensible given proficiency with the MATLAB language.

In consideration of working memory as a means for goal-directed behavior in nonstationary environments, we argue that the dynamics of working memory should satisfy two opposing demands: long-term maintenance and quick transition. These two characteristics are contradictory within the linear domain. We propose the near-saddle-node bifurcation behavior of a sigmoidal unit with a self-connection as a candidate of the dynamical mechanism that satisfies both of these demands. It is shown in evolutionary programming experiments that the near-saddle-node bifurcation behavior can be found in recurrent networks optimized for a task that requires efficient use of working memory. The result suggests that the near-saddle-node bifurcation behavior may be a functional necessity for survival in nonstationary environments.


Question: Recurrence networks - an innovative framework for nonlinear time series examination.

This paper offers a critical assessment of centrality measures in social networks, aiming to clarify conceptual ambiguities and enhance interpretability. It begins by revisiting the foundational intuitions underlying structural centrality measures, critically evaluating their alignment with these intuitions and their interpretative value. Through this process, the paper identifies and distinguishes three core conceptual dimensions of centrality. For each dimension, a trio of measures is proposed: an absolute measure to gauge the centrality of individual positions within a network, a relative measure to compare centrality across different positions, and a centralized measure to quantify the overall concentration of centrality in the network. The paper concludes with a discussion on the implications of these refined measures for the empirical investigation of small group dynamics, underscoring their potential to enrich understanding of social network structures and behaviors.

Recurrent neural networks (RNNs) are a powerful model for sequential data. End-to-end training methods such as Connectionist Temporal Classification make it possible to train RNNs for sequence labelling problems where the input-output alignment is unknown. The combination of these methods with the Long Short-term Memory RNN architecture has proved particularly fruitful, delivering state-of-the-art results in cursive handwriting recognition. However RNN performance in speech recognition has so far been disappointing, with better results returned by deep feedforward networks. This paper investigates deep recurrent neural networks, which combine the multiple levels of representation that have proved so effective in deep networks with the flexible use of long range context that empowers RNNs. When trained end-to-end with suitable regularisation, we find that deep Long Short-term Memory RNNs achieve a test set error of 17.7% on the TIMIT phoneme recognition benchmark, which to our knowledge is the best recorded score.

This paper compares the performance of two recurrent neural network models learning the grammar of simple English sentences and introduces a method for creating useful negative training examples from a set of grammatical sentences. The learning task is cast as a classification problem as opposed to a prediction problem; networks are taught to classify a given sequence as grammatical or ungrammatical. The two neural network models compared are the simple architecture proposed by Elman in [1], and the Long Short-Term Memory (LSTM) network from [4]. For comparison, a Naive Bayes (NB) classifier is also trained with bigrams to accomplish this task. Only the Elman network learns the training data better than NB, and neither network generalizes better than NB on sentences longer than the training sentences. The LSTM network does not generalize as well as the Elman network on a simple regular grammar, but shows comparable ability to generalize knowledge of English Grammar.

The modeling and identification of a mechanical system is the most important issue for many control systems in order to realize the desired control specifications. In particular, the friction characteristics often deteriorate the control performance, such as in the fast and precise positioning performance in industrial robots, the force estimation accuracy based on a disturbance observer, and the posture control performance of an inverted pendulum robot. Rolling friction tends to cause overshoot, undershoot, or limit cycles of the target value in positioning systems. In previous research, some model structures for rolling friction have been proposed to express the hysteresis characteristics in order to overcome these control issues. However, it is difficult to identify the correct parameters for precise modeling. In this paper, the modeling of rolling friction based on a Recurrent Neural Network (RNN) using Long Short-Term Memory (LSTM) is proposed to precisely express the rolling friction characteristics. The initial value design of the RNN during supervised learning is also presented to achieve a better model. The effectiveness of the proposed approach is verified by comparison with conventional friction models using an actual experimental setup.

We study the use of feed-forward convolutional neural networks for the unsupervised problem of mining recurrent temporal patterns mixed in multivariate time series. Traditional convolutional autoencoders lack interpretability for two main reasons: the number of patterns corresponds to the manually-fixed number of convolution filters, and the patterns are often redundant and correlated. To recover clean patterns, we introduce different elements in the architecture, including an adaptive rectified linear unit function that improves patterns interpretability, and a group-lasso regularizer that helps automatically finding the relevant number of patterns. We illustrate the necessity of these elements on synthetic data and real data in the context of activity mining in videos.

Long Short-Term Memory (LSTM) recurrent neural networks are renowned for being uninterpretable "black boxes". In the medical domain where LSTMs have shown promise, this is specifically concerning because it is imperative to understand the decisions made by machine learning models in such acute situations. This study employs techniques used in the Convolutional Neural Network domain to elucidate the operations that LSTMs perform on time series. The visualization techniques include input saliency by means of occlusion and derivatives, class mode visualization, and temporal outputs. Moreover, we demonstrate that LSTMs appear to extract features similar to those extracted by wavelets. It was found that deriving the inputs for saliency is a poor approximation and occlusion is a better approach. Moreover, analyzing LSTMs on different sets of data provide novel interpretations.

A, et al. Nonlinear analysis of electroencephalogram at rest and during cognitive tasks in patients with schizophrenia. One-dimensional electroencephalography (EEG) data were transformed into multidimensional phase space. The phase space concept is crucial for nonlinear dynamics analysis. In a hypothetical system governed by n variables, the phase space is n-dimensional. Each state of the system corresponds to a point in phase space, with n coordinates that are the values assumed by the governing variables for this specific state. If the system is observed for a period of time, the sequence of the point in phase space converges into a subspace, called system's attractor. The technique of delay coordinates is carried out to reconstruct the attractor's trajectory. To unfold the projection back to a multivariate state space, a representation of the original system, the delay coordinates y(t) = [xj(t), xj(t + T), … xj(t + [d-1] T)] from a single time series xj and the embedding procedure were performed. In this formula, y(t) is a point of the trajectory in the phase space at time t, x (t is iT) are the coordinates in the phase space corresponding to the time-delayed values of the time series, T is the time delay between the points of the time series considered, and d is the embedding dimension. 1,2 The choice of the optimal time delay T and embedding dimension d are important to the success of reconstruction with finite data. For the time delay, the first local minimum of the average mutual information between the set of measurements v(t) and v(t + T) was used in the present study. For the embedding dimension d the minimum (optimal) embedding was computed in the reconstruction procedure. 3 The attractor's reconstruction is necessary for the analysis of mathematical quantities characterizing the attractor itself. One important metric property of the attractor is the correlation dimension (D2), which estimates the degrees of freedom, such as the number of independent variables necessary to describe the dynamic of the original system. The larger the D2 values of the attractor, the more complicated the behavior of the nonlinear system. The D2 is thus a measure of the complexity of the process being investigated, and it characterizes the distribution of points in phase space. 4 The Gassberger–Procaccia algorithm 5 was computed to evaluate D2 values of the attractors from EEG data. With this algorithm, D2 is derived by determining the relative number …

We present an Artificial Neural Network (ANN) approach to predict stock market indices, particularly with respect to the forecast of their trend movements up or down. Exploiting different Neural Networks architectures, we provide numerical analysis of concrete financial time series. In particular, after a brief résumé of the existing literature on the subject, we consider the Multi-layer Perceptron (MLP), the Convolutional Neural Networks (CNN), and the Long Short-Term Memory (LSTM) recurrent neural networks techniques. We focus on the importance of choosing the correct input features, along with their preprocessing, for the specific learning algorithm one wants to use. Eventually, we consider the S&P500 historical time series, predicting trend on the basis of data from the past days, and proposing a novel approach based on combination of wavelets and CNN, which outperforms the basic neural networks ones. We show, that neural networks are able to predict financial time series movements even trained only on plain time series data and propose more ways to improve results. Key–Words: Artificial neural networks, Multi-layer neural network, Convolutional neural network, Long shortterm memory, Recurrent neural network, Deep Learning, Stock markets analysis, Time series analysis, financial forecasting

In recent years, neural networks have been used to generate symbolic melodies. However, the long-term structure in the melody has posed great difficulty for designing a good model. In this paper, we present a hierarchical recurrent neural network for melody generation, which consists of three Long-Short-Term-Memory (LSTM) subnetworks working in a coarse-to-fine manner along time. Specifically, the three subnetworks generate bar profiles, beat profiles and notes in turn, and the output of the high-level subnetworks are fed into the low-level subnetworks, serving as guidance for generating the finer time-scale melody components in low-level subnetworks. Two human behavior experiments demonstrate the advantage of this structure over the single-layer LSTM which attempts to learn all hidden structures in melodies. Compared with the state-of-the-art models MidiNet (Yang, Chou, and Yang 2017) and MusicVAE (Roberts et al. 2018), the hierarchical recurrent neural network produces better melodies evaluated by humans. Introduction Automatic music generation using neural networks has attracted much attention. There are two classes of music generation approaches, symbolic music generation (Hadjeres, Pachet, and Nielsen 2017)(Waite et al. 2016)(Yang, Chou, and Yang 2017) and audio music generation (van den Oord et al. )(Mehri et al. 2016). In this study, we focus on symbolic melody generation, which requires learning from sheet music. Many music genres such as pop music consist of melody and harmony. Since usually beautiful harmonies can be ensured by using legitimate chord progressions which have been summarized by musicians, we only focus on melody generation, similar to some recent studies (Waite et al. 2016)(Yang, Chou, and Yang 2017)(Colombo, Seeholzer, and Gerstner 2017)(Roberts et al. 2018). This greatly simplifies the melody generation problem. Melody is a linear succession of musical notes along time. It has both short time scale such as notes and long time scale such as phrases and movements, which makes the melody generation a challenging task. Existing methods generate pitches and rhythm simultaneously (Waite et al. 2016) or sequentially (Chu, Urtasun, and Fidler 2016) using Recurrent Neural Networks (RNNs), but they usually work on the note scale without explicitly modeling the larger time-scale components such as rhythmic patterns. It is difficult for them to learn long-term dependency or structure in melody. Theoretically, an RNN can learn the temporal structure of any length in the input sequence, but in reality, as the sequence gets longer it is very hard to learn long-term structure. Different RNNs have different learning capability, e.g., LSTM (Hochreiter and Schmidhuber 1997) performs much better than the simple Elman network. But any model has a limit for the length of learnable structure, and this limit depends on the complexity of the sequence to be learned. To enhance the learning capability of an RNN, one approach is to invent a new structure. In this work we take another approach: increase the granularity of the input. Since each symbol in the sequence corresponds to longer segment than the original representation, the same model would learn longer temporal structure. To implement this idea, we propose a Hierarchical Recurrent Neural Network (HRNN) for learning melody. It consists of three LSTM-based sequence generators — Bar Layer, Beat Layer and Note Layer. The Bar Layer and Beat Layer are trained to generate bar profiles and beat profiles, which are designed to represent the high-level temporal features of melody. The Note Layer is trained to generate melody conditioned on the bar profile sequence and beat profile sequence output by the Bar Layer and Beat Layer. By learning on different time scales, the HRNN can grasp the general regular patterns of human composed melodies in different granularities, and generate melody with realistic long-term structures. This method follows the general idea of granular computing (Bargiela and Pedrycz 2012), in which different resolutions of knowledge or information is extracted and represented for problem solving. With the shorter profile sequences to guide the generation of note sequence, the difficulty of generating note sequence with wellorganized structure is alleviated. Related Work Melody Generation with Neural Networks There is a long history of generating melody with RNNs. A recurrent autopredictive connectionist network called CONCERT is used to compose music (Mozer 1994). With a set of composition rules as constraints to evaluate ar X iv :1 71 2. 05 27 4v 2 [ cs .S D ] 5 S ep 2 01 8 melodies, an evolving neural network is employed to create melodies (Chen and Miikkulainen 2001). As an important form of RNN, LSTM (Hochreiter and Schmidhuber 1997) is used to capture the global music structure and improve the quality of the generated music (Eck and Schmidhuber 2002). Boulanger-Lewandowski, Bengio, and Vincent explore complex polyphonic music generation with an RNNRBM model (Boulanger-Lewandowski, Bengio, and Vincent 2012). Lookback RNN and Attention RNN are proposed to tackle the problem of creating melody’s long-term structure (Waite et al. 2016). The Lookback RNN introduces a handcrafted lookback feature that makes the model repeat sequences easier while the Attention RNN leverages an attention mechanism to learn longer-term structures. Inspired by convolution, two variants of RNN are employed to attain transposition invariance (Johnson 2017). To model the relation between rhythm and melody flow, a melody is divided into pitch sequence and duration sequence and these two sequences are processed in parallel (Colombo et al. 2016). This approach is further extended in (Colombo, Seeholzer, and Gerstner 2017). A hierarchical VAE is employed to learn the distribution of melody pieces in (Roberts et al. 2018), the decoder of which is similar to our model. The major difference is that the higher layer of its decoder uses the automatically learned representation of bars, while our higher layers use predefined representation of bars and beats which makes the learning problem easier. Generative Adversarial Networks (GANs) have also been used to generate melodies. For example, RNN-based GAN (Mogren 2016) and CNN-based GAN (Yang, Chou, and Yang 2017) are employed to generate melodies, respectively. However, the generated melodies also lack realistic long-term structures. Some models are proposed to generate multi-track music. A 4-layer LSTM is employed to produce the key, press, chord and drum of pop music seperately (Chu, Urtasun, and Fidler 2016). With pseudo-Gibbs sampling, a model can generate highly convincing chorales in the style of Bach (Colombo, Seeholzer, and Gerstner 2017). Three GANs for symbolic-domain multi-track music generation were proposed (Dong et al. 2018). An end-to-end melody and arrangement generation framework XiaoIce Band was proposed to generate a melody track with accompany tracks with RNN (Zhu et al. 2018). Hierarchical and Multiple Time Scales Networks The idea of hierarchical or multiple time scales has been used in neural network design, especially in the area of natural language processing. The Multiple Timescale Recurrent Neural Network (MTRNN) realizes the self-organization of a functional hierarchy with two types of neurons “fast” unit and “slow” unit (Yamashita and Tani 2008). Then it is shown that the MTRNN can acquire the capabilities to recognize, generate, and correct sentences in a hierarchical way: characters grouped into words, and words into sentences (Hinoshita et al. 2011). An LSTM auto-encoder is trained to preserve and reconstruct paragraphs by hierarchically building embeddings of words, sentences and paragraphs (Li, Luong, and Jurafsky 2015). To process inputs at multiple time Time signature Bar Key signature

The newly inaugurated Research Resource for Complex Physiologic Signals, which was created under the auspices of the National Center for Research Resources of the National Institutes of Health, is intended to stimulate current research and new investigations in the study of cardiovascular and other complex biomedical signals. The resource has 3 interdependent components. PhysioBank is a large and growing archive of well-characterized digital recordings of physiological signals and related data for use by the biomedical research community. It currently includes databases of multiparameter cardiopulmonary, neural, and other biomedical signals from healthy subjects and from patients with a variety of conditions with major public health implications, including life-threatening arrhythmias, congestive heart failure, sleep apnea, neurological disorders, and aging. PhysioToolkit is a library of open-source software for physiological signal processing and analysis, the detection of physiologically significant events using both classic techniques and novel methods based on statistical physics and nonlinear dynamics, the interactive display and characterization of signals, the creation of new databases, the simulation of physiological and other signals, the quantitative evaluation and comparison of analysis methods, and the analysis of nonstationary processes. PhysioNet is an on-line forum for the dissemination and exchange of recorded biomedical signals and open-source software for analyzing them. It provides facilities for the cooperative analysis of data and the evaluation of proposed new algorithms. In addition to providing free electronic access to PhysioBank data and PhysioToolkit software via the World Wide Web (http://www.physionet. org), PhysioNet offers services and training via on-line tutorials to assist users with varying levels of expertise.

The topology of social networks can be understood as being inherently dynamic, with edges having a distinct position in time. Most characterizations of dynamic networks discretize time by converting temporal information into a sequence of network “snapshots” for further analysis. Here we study a highly resolved data set of a dynamic proximity network of 66 individuals. We show that the topology of this network evolves over a very broad distribution of time scales, that its behavior is characterized by strong periodicities driven by external calendar cycles, and that the conversion of inherently continuous-time data into a sequence of snapshots can produce highly biased estimates of network structure. We suggest that dynamic social networks exhibit a natural time scale ∆nat, and that the best conversion of such dynamic data to a discrete sequence of networks is done at this natural rate.


Question: Recurrence networks - an innovative framework for nonlinear time series examination.

The emergence of scaling in random networks has introduced a novel paradigm for analyzing nonlinear time series data, marking a significant advancement in the field of complex systems. This paradigm shifts the focus from traditional linear models to the intricate patterns that emerge from the interactions within large, interconnected networks. The study of scaling in random networks, which has gained considerable attention since its inception, is particularly relevant for understanding the behavior of nonlinear time series. These networks, characterized by their power-law degree distributions, offer a new lens through which to examine the dynamics of complex systems. Unlike conventional approaches, which often struggle to capture the nuanced behaviors of nonlinear systems, the scaling properties of random networks provide a framework for identifying underlying patterns and predicting future trends. This method has been successfully applied across various domains, from biological networks to social media analysis, demonstrating its versatility and potential for revolutionizing how we interpret and forecast nonlinear phenomena. The exploration of scaling in random networks not only enriches our understanding of complex systems but also opens up new avenues for research and practical applications in fields such as data science, economics, and ecology.

The long short-term memory (LSTM) network trained by gradient descent solves difficult problems which traditional recurrent neural networks in general cannot. We have recently observed that the decoupled extended Kalman filter training algorithm allows for even better performance, reducing significantly the number of training steps when compared to the original gradient descent training algorithm. In this paper we present a set of experiments which are unsolvable by classical recurrent networks but which are solved elegantly and robustly and quickly by LSTM combined with Kalman filters.

Unlike financial forecasting, a type of mechanical trading technique called Trend Following (TF) doesn’t predict any market movement; instead it identifies a trend at early time of the day, and trades automatically afterwards by a pre-defined strategy regardless of the moving market directions during run time. Trend following trading has a long and successful history among speculators. The traditional TF trading method is by human judgment in setting the rules (aka the strategy). Subsequently the TF strategy is executed in pure objective operational manner. Finding the correct strategy at the beginning is crucial in TF. This usually involves human intervention in first identifying a trend, and configuring when to place an order and close it out, when certain conditions are met. In this paper, we presented a new type of TF, namely Trend Recalling algorithm that operates in a totally automated manner. It works by partially matching the current trend with one of the proven successful patterns from the past. Our experiments based on real stock market data show that this algorithm has an edge over the other trend following methods in profitability. The new algorithm is also compared to time-series forecasting type of stock trading, and it can even outperform the best forecasting type in a

Ladder logic diagram (LLD) as the interfacing programming language of programmable logic controllers (PLCs) is utilized in modern discrete event control systems. However, LLD is hard to debug and maintain in practice. This is due to many factors such as non-structured nature of LLD, the LLD programmers' background, and the huge sizes of real world LLD. In this paper, we introduce a recurrent neural network (RNN) based technique for PLC program diagnosis. A manufacturing control system example has been presented to illustrate the applicability of the proposed algorithm. This method could be very advantageous in reducing the complexity in PLC control programs diagnosis because of the ease of use of the RNN compared to debugging the LLD code.

Over the last decade, we have observed a revolution in brain structural and functional Connectomics. On one hand, we have an ever-more detailed characterization of the brain's white matter structural connectome. On the other, we have a repertoire of consistent functional networks that form and dissipate over time during rest. Despite the evident spatial similarities between structural and functional connectivity, understanding how different time-evolving functional networks spontaneously emerge from a single structural network requires analyzing the problem from the perspective of complex network dynamics and dynamical system's theory. In that direction, bottom-up computational models are useful tools to test theoretical scenarios and depict the mechanisms at the genesis of resting-state activity. Here, we provide an overview of the different mechanistic scenarios proposed over the last decade via computational models. Importantly, we highlight the need of incorporating additional model constraints considering the properties observed at finer temporal scales with MEG and the dynamical properties of FC in order to refresh the list of candidate scenarios.

Recurrent neural network (RNN) models are widely used for processing sequential data governed by a latent tree structure. Previous work shows that RNN models (especially Long Short-Term Memory (LSTM) based models) could learn to exploit the underlying tree structure. However, its performance consistently lags behind that of tree-based models. This work proposes a new inductive bias Ordered Neurons, which enforces an order of updating frequencies between hidden state neurons. We show that the ordered neurons could explicitly integrate the latent tree structure into recurrent models. To this end, we propose a new RNN unit: ON-LSTM, which achieve good performances on four different tasks: language modeling, unsupervised parsing, targeted syntactic evaluation, and logical inference1.

Pyramidal neurons represent the majority of excitatory neurons in the neocortex. Each pyramidal neuron receives input from thousands of excitatory synapses that are segregated onto dendritic branches. The dendrites themselves are segregated into apical, basal, and proximal integration zones, which have different properties. It is a mystery how pyramidal neurons integrate the input from thousands of synapses, what role the different dendrites play in this integration, and what kind of network behavior this enables in cortical tissue. It has been previously proposed that non-linear properties of dendrites enable cortical neurons to recognize multiple independent patterns. In this paper we extend this idea in multiple ways. First we show that a neuron with several thousand synapses segregated on active dendrites can recognize hundreds of independent patterns of cellular activity even in the presence of large amounts of noise and pattern variation. We then propose a neuron model where patterns detected on proximal dendrites lead to action potentials, defining the classic receptive field of the neuron, and patterns detected on basal and apical dendrites act as predictions by slightly depolarizing the neuron without generating an action potential. By this mechanism, a neuron can predict its activation in hundreds of independent contexts. We then present a network model based on neurons with these properties that learns time-based sequences. The network relies on fast local inhibition to preferentially activate neurons that are slightly depolarized. Through simulation we show that the network scales well and operates robustly over a wide range of parameters as long as the network uses a sparse distributed code of cellular activations. We contrast the properties of the new network model with several other neural network models to illustrate the relative capabilities of each. We conclude that pyramidal neurons with thousands of synapses, active dendrites, and multiple integration zones create a robust and powerful sequence memory. Given the prevalence and similarity of excitatory neurons throughout the neocortex and the importance of sequence memory in inference and behavior, we propose that this form of sequence memory may be a universal property of neocortical tissue.

In this paper, we study the problem of activity recognition and abnormal behaviour detection for elderly people with dementia. Very few studies have attempted to address this problem presumably because of the lack of experimental data in the context of dementia care. In particular, the paper investigates three variants of Recurrent Neural Networks (RNNs): Vanilla RNNs (VRNN), Long Short Term RNNs (LSTM) and Gated Recurrent Unit RNNs (GRU). Here activity recognition is considered as a sequence labelling problem, while abnormal behaviour is flagged based on the deviation from normal patterns. To provide an adequate discussion of the performance of RNNs in this context, we compare them against the state-of-art methods such as Support Vector Machines (SVMs), Naı̈ve Bayes (NB), Hidden Markov Models (HMMs), Hidden Semi-Markov Models (HSMM) and Conditional Random Fields (CRFs). The results obtained indicate that RNNs are competitive with those state-of-art methods. Moreover, the paper presents a methodology for generating synthetic data reflecting on some behaviours of people with dementia given the difficulty of obtaining real-world data. c 6 t r . ublis e ls i r . . i ilit f t f re ce r ra hairs.

A family of probabilistic time series models is developed to analyze the time evolution of topics in large document collections. The approach is to use state space models on the natural parameters of the multinomial distributions that represent the topics. Variational approximations based on Kalman filters and nonparametric wavelet regression are developed to carry out approximate posterior inference over the latent topics. In addition to giving quantitative, predictive models of a sequential corpus, dynamic topic models provide a qualitative window into the contents of a large document collection. The models are demonstrated by analyzing the OCR'ed archives of the journal Science from 1880 through 2000.

Inspired by the information theoretic idea of minimum description length, we add a term to the back propagation cost function that penalizes network complexity. We give the details of the procedure, called weight-elimination, describe its dynamics, and clarify the meaning of the parameters involved. From a Bayesian perspective, the complexity term can be usefully interpreted as an assumption about prior distribution of the weights. We use this procedure to predict the sunspot time series and the notoriously noisy series of currency exchange rates.

We define predictive information Ipred(T) as the mutual information between the past and the future of a time series. Three qualitatively different behaviors are found in the limit of large observation times T: Ipred(T) can remain finite, grow logarithmically, or grow as a fractional power law. If the time series allows us to learn a model with a finite number of parameters, then Ipred(T) grows logarithmically with a coefficient that counts the dimensionality of the model space. In contrast, power-law growth is associated, for example, with the learning of infinite parameter (or non-parametric) models such as continuous functions with smoothness constraints. There are connections between the predictive information and measures of complexity that have been defined both in learning theory and the analysis of physical systems through statistical mechanics and dynamical systems theory. Furthermore, in the same way that entropy provides the unique measure of available information consistent with some simple and plausible conditions, we argue that the divergent part of Ipred(T) provides the unique measure for the complexity of dynamics underlying a time series. Finally, we discuss how these ideas may be useful in problems in physics, statistics, and biology.


Question: Recurrence networks - an innovative framework for nonlinear time series examination.

This paper presents a method for visualizing graphs using force-directed placement, which simulates physical systems to create visually appealing and structurally meaningful representations of graphs. A graph, represented as G = (V,E), consists of a set V of vertices and a set E of edges connecting these vertices. Graphs are typically visualized by plotting vertices as points in a two-dimensional plane and edges as straight lines or curves connecting these points. Various styles of graph representation exist, tailored to different types of graphs or specific visualization goals. Our focus is on undirected graphs, which we aim to depict using straight edges, adhering to general aesthetic principles.

Our algorithm employs a simplified simulation of physical systems to generate two-dimensional images of undirected graphs. This approach is guided by established aesthetic criteria that enhance the readability and visual appeal of the graph layout. The criteria include minimizing edge crossings, distributing vertices uniformly, and maintaining symmetry and compactness in the layout. By simulating forces between nodes—repulsive forces between vertices and attractive forces along edges—the algorithm produces layouts that naturally align with these aesthetic standards, resulting in graphs that are not only visually pleasing but also easier to interpret. This force-directed placement method is particularly effective for undirected graphs, offering a novel and intuitive way to visualize complex networks while adhering to fundamental principles of graph aesthetics.

In this paper we first point out a fatal drawback that the widely used Granger causality (GC) needs to estimate the autoregressive model, which is equivalent to taking a series of backward recursive operations which are infeasible in many irreversible chemical reaction models. Thus, new causality (NC) proposed by Hu et al. (2011) is theoretically shown to be more sensitive to reveal true causality than GC. We then apply GC and NC to motor imagery (MI) which is an important mental process in cognitive neuroscience and psychology and has received growing attention for a long time. We study causality flow during MI using scalp electroencephalograms from nine subjects in Brain-computer interface competition IV held in 2008. We are interested in three regions: Cz (central area of the cerebral cortex), C3 (left area of the cerebral cortex), and C4 (right area of the cerebral cortex) which are considered to be optimal locations for recognizing MI states in the literature. Our results show that: 1) there is strong directional connectivity from Cz to C3/C4 during left- and right-hand MIs based on GC and NC; 2) during left-hand MI, there is directional connectivity from C4 to C3 based on GC and NC; 3) during right-hand MI, there is strong directional connectivity from C3 to C4 which is much clearly revealed by NC than by GC, i.e., NC largely improves the classification rate; and 4) NC is demonstrated to be much more sensitive to reveal causal influence between different brain regions than GC.

A continuous Deep Belief Network (cDBN) with two hidden layers is proposed in this 18 paper, focusing on the problem of weak feature learning ability when dealing with continuous data. 19 In cDBN, the input data is trained in an unsupervised way by using continuous version of transfer 20 functions, the contrastive divergence is designed in hidden layer training process to raise 21 convergence speed, an improved dropout strategy is then implemented in unsupervised training to 22 realize features learning by de-cooperating between the units, and then the network is fine-tuned 23 using back propagation algorithm. Besides, hyper-parameters are analysed through stability 24 analysis to assure the network can find the optimal. Finally, the experiments on Lorenz chaos series, 25 CATS benchmark and other real world like CO2 and waste water parameters forecasting show that 26 cDBN has the advantage of higher accuracy, simpler structure and faster convergence speed than 27 other methods. 28

In this paper I try to describe both the role of mathematics in shaping our understanding of how neural networks operate, and the curious new mathematical concepts generated by our attempts to capture neural networks in equations. My target reader being the non-expert, I will present a biased selection of relatively simple examples of neural network tasks, models and calculations, rather than try to give a full encyclopedic review-like account of the many mathematical developments in this eld.

Unlike financial forecasting, a type of mechanical trading technique called Trend Following (TF) doesn’t predict any market movement; instead it identifies a trend at early time of the day, and trades automatically afterwards by a pre-defined strategy regardless of the moving market directions during run time. Trend following trading has a long and successful history among speculators. The traditional TF trading method is by human judgment in setting the rules (aka the strategy). Subsequently the TF strategy is executed in pure objective operational manner. Finding the correct strategy at the beginning is crucial in TF. This usually involves human intervention in first identifying a trend, and configuring when to place an order and close it out, when certain conditions are met. In this paper, we presented a new type of TF, namely Trend Recalling algorithm that operates in a totally automated manner. It works by partially matching the current trend with one of the proven successful patterns from the past. Our experiments based on real stock market data show that this algorithm has an edge over the other trend following methods in profitability. The new algorithm is also compared to time-series forecasting type of stock trading, and it can even outperform the best forecasting type in a

Replicator neural networks self-organize by using their inputs as desired outputs; they internally form a compressed representation for the input data. A theorem shows that a class of replicator networks can, through the minimization of mean squared reconstruction error (for instance, by training on raw data examples), carry out optimal data compression for arbitrary data vector sources. Data manifolds, a new general model of data sources, are then introduced and a second theorem shows that, in a practically important limiting case, optimal-compression replicator networks operate by creating an essentially unique natural coordinate system for the manifold.

This paper aims to give a broad coverage of central concepts and principles involved in automated causal inference and emerging approaches to causal discovery from i.i.d data and from time series. After reviewing concepts including manipulations, causal models, sample predictive modeling, causal predictive modeling, and structural equation models, we present the constraint-based approach to causal discovery, which relies on the conditional independence relationships in the data, and discuss the assumptions underlying its validity. We then focus on causal discovery based on structural equations models, in which a key issue is the identifiability of the causal structure implied by appropriately defined structural equation models: in the two-variable case, under what conditions (and why) is the causal direction between the two variables identifiable? We show that the independence between the error term and causes, together with appropriate structural constraints on the structural equation, makes it possible. Next, we report some recent advances in causal discovery from time series. Assuming that the causal relations are linear with nonGaussian noise, we mention two problems which are traditionally difficult to solve, namely causal discovery from subsampled data and that in the presence of confounding time series. Finally, we list a number of open questions in the field of causal discovery and inference.

Over the last decade, we have observed a revolution in brain structural and functional Connectomics. On one hand, we have an ever-more detailed characterization of the brain's white matter structural connectome. On the other, we have a repertoire of consistent functional networks that form and dissipate over time during rest. Despite the evident spatial similarities between structural and functional connectivity, understanding how different time-evolving functional networks spontaneously emerge from a single structural network requires analyzing the problem from the perspective of complex network dynamics and dynamical system's theory. In that direction, bottom-up computational models are useful tools to test theoretical scenarios and depict the mechanisms at the genesis of resting-state activity. Here, we provide an overview of the different mechanistic scenarios proposed over the last decade via computational models. Importantly, we highlight the need of incorporating additional model constraints considering the properties observed at finer temporal scales with MEG and the dynamical properties of FC in order to refresh the list of candidate scenarios.

We present a new stochastic learning machine for speech separation based on the variational recurrent neural network (VRNN). This VRNN is constructed from the perspectives of generative stochastic network and variational auto-encoder. The idea is to faithfully characterize the randomness of hidden state of a recurrent neural network through variational learning. The neural parameters under this latent variable model are estimated by maximizing the variational lower bound of log marginal likelihood. An inference network driven by the variational distribution is trained from a set of mixed signals and the associated source targets. A novel supervised VRNN is developed for speech separation. The proposed VRNN provides a stochastic point of view which accommodates the uncertainty in hidden states and facilitates the analysis of model construction. The masking function is further employed in network outputs for speech separation. The benefit of using VRNN is demonstrated by the experiments on monaural speech separation.

This paper shows how Long Short-term Memory recurrent neural networks can be used to generate complex sequences with long-range structure, simply by predicting one data point at a time. The approach is demonstrated for text (where the data are discrete) and online handwriting (where the data are real-valued). It is then extended to handwriting synthesis by allowing the network to condition its predictions on a text sequence. The resulting system is able to generate highly realistic cursive handwriting in a wide variety of styles.

In this chapter we investigate computation from a dynamical systems perspective. A dynamical system is described in terms of its abstract state space, the system’s current state within its state space, and a rule that determines its motion through its state space. In a classical computational system, that rule is given explicitly by the computer program; in a physical system, that rule is the underlying physical law governing the behaviour of the system. So a dynamical systems approach to computation allows us to take a unified view of computation in classical discrete systems and in systems performing non-classical computation. In particular, it gives a route to a computational interpretation of physical embodied systems exploiting the natural dynamics of their material substrates. We start with autonomous (closed) dynamical systems: those whose dynamics is not an explicit function of time, in particular, those with no inputs from an external environment. We begin with computationally conventional discrete systems, examining their computational abilities from a dynamical systems perspective. The aim here is both to introduce the necessary dynamical systems concepts, and to demonstrate how classical computation can be viewed from this perspective. We then move on to continuous dynamical systems, such as those inherent in the complex dynamics of matter, and show how these too can be interpreted computationally, and see how the material embodiment can give such computation “for free”, without the need to explicitly implement the dynamics. We next broaden the outlook to open (non-autonomous) dynamical systems, where the dynamics is a function of time, in the form of inputs from an external environment, and which may be in a closely coupled feedback loop with that environment. We finally look at constructive, or developmental, dynamical systems, where the structure of the state space is changing during the computation. This includes vari-


Question: Software.zhishi.schema: A software programming classification system derived from Stackoverflow data.

This paper delves into the semantic portrayal of verbs within computational frameworks and its implications for lexical selection in the realm of machine translation (MT). By scrutinizing two distinct sets of English and Chinese verbs, it becomes evident that lexical selection is contingent upon the interpretation of the sentence structure and the constraints imposed on the arguments of the verb. A fresh representation methodology is proposed, which stands in contrast to the conventional use of selection restrictions prevalent in transfer-based MT systems. Our approach aligns closely with knowledge-based MT (KBMT) strategies, envisioning itself as a modular component that can seamlessly integrate into current MT systems. Through illustrative examples and empirical findings, we demonstrate that, under this novel scheme, even less-than-perfect matches can yield accurate lexical selections, thereby enhancing the overall efficiency and accuracy of machine translation processes.

As a traditional dynamic language, Python is increasingly used in various software engineering tasks. However, due to its flexibility and dynamism, Python is a particularly challenging language to write code in and maintain. Consequently, Python programs contain code smells which indicate potential comprehension and maintenance problems. With the aim of supporting refactoring strategies to enhance maintainability, this paper describes how to detect code smells in Python programs. We introduce 11 Python smells and describe the detection strategy. We also implement a smell detection tool named Pysmell and use it to identify code smells in five real world Python systems. The results show that Pysmell can detect 285 code smell instances in total with the average precision of 97.7%. It reveals that Large Class and Large Method are most prevalent. Our experiment also implies Python programs may be suffering code smells further.

The traditional modular structure defined by the package hierarchy suffers from the dominant decomposition problem and it is widely accepted that alternative forms of modularization are necessary to increase developer’s productivity. In this paper, we propose an alternative form to understand and assess package modularity based on co-change clusters, which are highly inter-related classes considering cochange relations. We evaluate how co-change clusters relate to the package decomposition of four real-world systems. The results show that the projection of co-change clusters to packages follows different patterns in each system. Therefore, we claim that modular views based on co-change clusters can improve developers’ understanding on how well-modularized are their systems, considering that modularity is the ability to confine changes and evolve components in parallel.

In this paper, we conducted a comparative study of how social organization takes place in a wasp colony and OSS developer communities. Both these systems display similar global organization patterns, such as hierarchies and clear labor divisions. As our analysis shows, both systems also define interacting agent networks with similar common features that reflect limited information sharing among agents. As far as we know, this is the first research study analyzing the patterns and functional significance of these systems' weighted-interaction networks. By illuminating the extent to which self-organization is responsible for patterns such as hierarchical structure, we can gain insight into the origins of organization in OSS communities.

This paper gives an overview about the development of the field of Knowledge Engineering over the last 15 years. We discuss the paradigm shift from a transfer view to a modeling view and describe two approaches which considerably shaped research in Knowledge Engineering: Role-limiting Methods and Generic Tasks. To illustrate various concepts and methods which evolved in the last years we describe three modeling frameworks: CommonKADS, MIKE, and PROTÉGÉ-II. This description is supplemented by discussing some important methodological developments in more detail: specification languages for knowledge-based systems, problem-solving methods, and ontologies. We conclude with outlining the relationship of Knowledge Engineering to Software Engineering, Information Integration and Knowledge Management.

Programmers frequently search for source code to reuse using keyword searches. The search effectiveness in facilitating reuse, however, depends on the programmer's ability to specify a query that captures how the desired code may have been implemented. Further, the results often include many irrelevant matches that must be filtered manually. More semantic search approaches could address these limitations, yet existing approaches are either not flexible enough to find approximate matches or require the programmer to define complex specifications as queries.
 We propose a novel approach to semantic code search that addresses several of these limitations and is designed for queries that can be described using a concrete input/output example. In this approach, programmers write lightweight specifications as inputs and expected output examples. Unlike existing approaches to semantic search, we use an SMT solver to identify programs or program fragments in a repository, which have been automatically transformed into constraints using symbolic analysis, that match the programmer-provided specification.
 We instantiated and evaluated this approach in subsets of three languages, the Java String library, Yahoo! Pipes mashup language, and SQL select statements, exploring its generality, utility, and trade-offs. The results indicate that this approach is effective at finding relevant code, can be used on its own or to filter results from keyword searches to increase search precision, and is adaptable to find approximate matches and then guide modifications to match the user specifications when exact matches do not already exist. These gains in precision and flexibility come at the cost of performance, for which underlying factors and mitigation strategies are identified.

JSON documents are raising as a common format for data representation due to the increasing popularity of NoSQL document-oriented databases. One of the reasons for such a popularity is their ability to handle large volumes of data at the absence of an explicit data schema. However, schema information is sometimes essential for applications during data retrieval, integration and analysis tasks, for example. Given this context, this paper presents an approach that extracts a schema from a JSON or Extended JSON document collection stored in a NoSQL document-oriented database or other document repository. Aggregation operations are considered in order to obtain a schema for each distinct structure in the collection, and a hierarchical data structure is proposed to group these schemas in order to generate a global schema in JSON Schema format. Experiments conducted on actual datasets, like DBPedia and Foursquare, demonstrate that the accuracy of the generated schemas is equivalent or even superior than related work.

The problem of classification has been widely studied in the data mining, machine learning, database, and information retrieval communities with applications in a number of diverse domains, such as target marketing, medical diagnosis, news group filtering, and document organization. In this paper we will provide a survey of a wide variety of text classification

A FUZZY LOGIC-BASED TEXT CLASSIFICATION METHOD FOR SOCIAL MEDIA

Support tickets from customers contain much hidden information. Unsupervised machine learning methods are able to discover this hidden information. In this paper we propose the categorization of support tickets using clustering methods in combination with topic models. Furthermore label generation techniques are used to generate meaningful names for these categories. The results are compared with related

This paper describes a new language resource of events and semantic roles that characterize real-world situations. Narrative schemas contain sets of related events (edit and publish), a temporal ordering of the events (edit before publish), and the semantic roles of the participants (authors publish books). This type of world knowledge was central to early research in natural language understanding. Scripts were one of the main formalisms, representing common sequences of events that occur in the world. Unfortunately, most of this knowledge was hand-coded and time consuming to create. Current machine learning techniques, as well as a new approach to learning through coreference chains, has allowed us to automatically extract rich event structure from open domain text in the form of narrative schemas. The narrative schema resource described in this paper contains approximately 5000 unique events combined into schemas of varying sizes. We describe the resource, how it is learned, and a new evaluation of the coverage of these schemas over unseen documents.


Question: Software.zhishi.schema: A software programming classification system derived from Stackoverflow data.

This paper introduces a methodology to significantly improve text categorization systems by integrating encyclopedic knowledge from Wikipedia, thereby addressing the brittleness bottleneck that plagues conventional information retrieval systems. Unlike humans, who categorize text by interpreting it within the broad context of their extensive knowledge and experiences, traditional systems are often limited to analyzing the text as a collection of individual words. This approach restricts their ability to learn from the nuances present in the training data. For example, when encountering the phrase "Wal-Mart supply chain goes real time," how can a text categorization algorithm deduce that Wal-Mart utilizes RFID technology for inventory management? Similarly, when reading that "Ciprofloxacin is a quinolone," how can a machine infer that it is an antibiotic produced by Bayer? Our solution involves leveraging Wikipedia, the largest and most comprehensive encyclopedia available, to enrich the representation of documents. Each Wikipedia article is associated with a specific concept, and we use machine learning algorithms to map documents into this rich feature space, encompassing both words and related Wikipedia concepts. Our empirical findings demonstrate that incorporating this knowledge-intensive representation into text categorization significantly enhances performance across a variety of datasets, marking a qualitative leap forward in the field.

Software evolves with continuous source-code changes. These code changes usually need to be understood by software engineers when performing their daily development and maintenance tasks. However, despite its high importance, such change-understanding practice has not been systematically studied. Such lack of empirical knowledge hinders attempts to evaluate this fundamental practice and improve the corresponding tool support.
 To address this issue, in this paper, we present a large-scale quantitative and qualitative study at Microsoft. The study investigates the role of understanding code changes during software-development process, explores engineers' information needs for understanding changes and their requirements for the corresponding tool support. The study results reinforce our beliefs that understanding code changes is an indispensable task performed by engineers in software-development process. A number of insufficiencies in the current practice also emerge from the study results. For example, it is difficult to acquire important information needs such as a change's completeness, consistency, and especially the risk imposed by it on other software components. In addition, for understanding a composite change, it is valuable to decompose it into sub-changes that are aligned with individual development issues; however, currently such decomposition lacks tool support.

Software development in highly variable domains constrained by tight regulations and with many business concepts involved results in hard to deliver and maintain applications, due to the complexity of dealing with the large number of concepts provided by the different parties and system involved in the process. One way to tackle these problems is thru combining software product lines and model-driven software development supported by ontologies. Software product lines and model-driven approaches would promote reuse on the software artifacts and, if supported by an ontological layer, those artifacts would be domain-validated. We intend to create a new conceptual framework for software development with domain validated models in highly variable domains. To define such a framework we will propose a model that relates several dimensions and areas of software development thru time and abstraction levels. This model would guarantee to the software house traceability of components, domain validated artifacts, easy to maintain and reusable components, due to the relations and mappings we propose to establish in the conceptual framework, between the software artifacts and the ontology.

A technical standard is an established norm or requirement in regard to technical systems. It is usually a formal document that establishes uniform engineering or technical criteria, methods, processes and practices. Every year, more than 10K standards are released by standardization organizations, such as International Standardization Organization (ISO), American National Standards Institute (ANSI), and China National Institute of Standardization (CNIS). Thereinto, China publishes more than 4K national and industrial standards per year. Confronted with the large amount of standards, how to manage and make full use of these standards efficiently and effectively has become a major issue to be addressed. In this paper, we introduce a mining and visualizing system for large-scale Chinese technical standards from spatial and temporal perspectives. Specifically, we propose an s-index metric to measure the contribution of a drafting unit. Moreover, we develop a multiple dimensional mining and visualizing analysis system for standardization experts and users to explore regional differences and temporal trends of standards in China. Our system has run on 95K Chinese standards spanning from 2001 to 2016 and provided online service. The extensive experiments and case study demonstrate the effectiveness of our proposed system.

Since 2009, DevOps, the combination of development and operation, has been adopted within organizations in industry, such as Netflix, Flickr, and Fotopedia. Configuration management tools have been used to support DevOps. However, in this paper we investigate which Knowledge, Skills, and Abilities (KSA) have been employed in developing and deploying modern web applications and how these KSAs support DevOps. By applying a qualitative analysis approach, namely grounded theory, to three web application development projects, we discover that the KSAs for both Software Development and IT Operator practitioners support the four perspectives of DevOps: collaboration culture, automation, measurement, and sharing.

Ontology is increasingly seen as a key factor for enabling interoperability across heterogeneous systems and semantic web applications. Ontology mapping is required for combining distributed and heterogeneous ontologies. Developing such ontology mapping has been a core issue of recent ontology research. This paper presents ontology mapping categories, describes the characteristics of each category, compares these characteristics, and surveys tools, systems, and related work based on each category of ontology mapping. We believe this paper provides readers with a comprehensive understanding of ontology mapping and points to various research topics about the specific roles of ontology mapping.

0950-5849/$ see front matter 2010 Elsevier B.V. A doi:10.1016/j.infsof.2010.12.001 ⇑ Address: School of Computing, Blekinge Institute 372 25, Sweden. E-mail addresses: kai.petersen@bth.se, kai.petersen URLs: http://www.bth.se/besq, http://www.ericsso Context: Software productivity measurement is essential in order to control and improve the performance of software development. For example, by identifying role models (e.g. projects, individuals, tasks) when comparing productivity data. The prediction is of relevance to determine whether corrective actions are needed, and to discover which alternative improvement action would yield the best results. Objective: In this study we identify studies for software productivity prediction and measurement. Based on the identified studies we first create a classification scheme and map the studies into the scheme (systematic map). Thereafter, a detailed analysis and synthesis of the studies is conducted. Method: As a research method for systematically identifying and aggregating the evidence of productivity measurement and prediction approaches systematic mapping and systematic review have been used. Results: In total 38 studies have been identified, resulting in a classification scheme for empirical research on software productivity. The mapping allowed to identify the rigor of the evidence with respect to the different productivity approaches. In the detailed analysis the results were tabulated and synthesized to provide recommendations to practitioners. Conclusion: Risks with simple ratio-based measurement approaches were shown. In response to the problems data envelopment analysis seems to be a strong approach to capture multivariate productivity measures, and allows to identify reference projects to which inefficient projects should be compared. Regarding simulation no general prediction model can be identified. Simulation and statistical process control are promising methods for software productivity prediction. Overall, further evidence is needed to make stronger claims and recommendations. In particular, the discussion of validity threats should become standard, and models need to be compared with each other. 2010 Elsevier B.V. All rights reserved.

This thesis discusses the problems and the methods of finding relevant information in large collections of documents. The contribution of this thesis to this problem is to develop better content analysis methods which can be used to describe document content with

0306-4573/$ see front matter Published by Elsevi doi:10.1016/j.ipm.2008.07.005 * Corresponding author. Tel.: +1 601 266 4035; fa E-mail addresses: Yuelin.Li@usm.edu (Y. Li), nick 1 Tel.: +1 732 932 7500x8271; fax: +1 732 932 69 The nature of the task that leads a person to engage in information interaction, as well as of information seeking and searching tasks, have been shown to influence individuals’ information behavior. Classifying tasks in a domain has been viewed as a departure point of studies on the relationship between tasks and human information behavior. However, previous task classification schemes either classify tasks with respect to the requirements of specific studies or merely classify a certain category of task. Such approaches do not lead to a holistic picture of task since a task involves different aspects. Therefore, the present study aims to develop a faceted classification of task, which can incorporate work tasks and information search tasks into the same classification scheme and characterize tasks in such a way as to help people make predictions of information behavior. For this purpose, previous task classification schemes and their underlying facets are reviewed and discussed. Analysis identifies essential facets and categorizes them into Generic facets of task and Common attributes of task. Generic facets of task include Source of task, Task doer, Time, Action, Product, and Goal. Common attributes of task includes Task characteristics and User’s perception of task. Corresponding sub-facets and values are identified as well. In this fashion, a faceted classification of task is established which could be used to describe users’ work tasks and information search tasks. This faceted classification provides a framework to further explore the relationships among work tasks, search tasks, and interactive information retrieval and advance adaptive IR systems design. Published by Elsevier Ltd.

Software engineering researchers have long been interested in where and why bugs occur in code, and in predicting where they might turn up next. Historical bug-occurence data has been key to this research. Bug tracking systems, and code version histories, record when, how and by whom bugs were fixed; from these sources, datasets that relate file changes to bug fixes can be extracted. These historical datasets can be used to test hypotheses concerning processes of bug introduction, and also to build statistical bug prediction models. Unfortunately, processes and humans are imperfect, and only a fraction of bug fixes are actually labelled in source code version histories, and thus become available for study in the extracted datasets. The question naturally arises, are the bug fixes recorded in these historical datasets a fair representation of the full population of bug fixes? In this paper, we investigate historical data from several software projects, and find strong evidence of systematic bias. We then investigate the potential effects of "unfair, imbalanced" datasets on the performance of prediction techniques. We draw the lesson that bias is a critical problem that threatens both the effectiveness of processes that rely on biased datasets to build prediction models and the generalizability of hypotheses tested on biased data.

We identify top management leadership, a sophisticated management infrastructure, process management efficacy, and stakeholder participation as important elements of a quality-oriented organizational system for software development. A model interrelating these constructs and quality performance is proposed. Data collected through a national survey of IS executives in Fortune 1000 companies and government agencies was used to Robert Zmud was the accepting senior editor for this paper. test the model using a Partial Least Squares analysis methodology. Our results suggest that software quality goals are best attained when top management creates a management infrastructure that promotes improvements in process design and encourages stakeholders to evolve the design of the development processes. Our results also suggest that all elements of the organizational system need to be developed in order to attain quality goals and that piecemeal adoption of select quality management practices are unlikely to be effective. Implications of this research for IS theory and practice are discussed.


Question: Software.zhishi.schema: A software programming classification system derived from Stackoverflow data.

This paper delves into the core aspects of lexical semantics research, encompassing methodology, the scope of descriptive coverage, the adequacy of representation, and the computational utility of such representations. In exploring these themes, I identify key challenges within the lexical semantics community and propose effective strategies for tackling them. I introduce a technique for the dissection of lexical categories and outline a theoretical framework for lexical semantics that incorporates concepts of cocompositionality and type coercion, alongside a multi-tiered semantic description system that distributes the semantic load more evenly across the lexicon. I contend that the process of lexical decomposition is feasible when executed in a generative manner, eschewing the assumption of a static set of primitives in favor of a fixed number of generative mechanisms that construct semantic expressions. I develop a Qualia Structure theory, a lexical item representation language, which significantly reduces lexical ambiguity within the lexicon while accounting for the systematic polysemy that characterizes word usage. Additionally, I explore the integration of individual lexical structures into the broader lexical knowledge base through a theory of lexical inheritance, which establishes the essential principles for the global organization of the lexicon, facilitating its seamless incorporation into a comprehensive conceptual framework.

**Question**: A Review of Methods for the Analysis of Temporal Patterns in Time Series Data

**Documents**: A review of the methods for the analysis of nonlinear time series
Time series analysis is a branch of statistics that deals with the analysis of data collected over time. This review paper provides an overview of the methods for the analysis of nonlinear time series. The paper is organized as follows: first, we provide a brief introduction to the basic concepts of time series analysis. Then, we review the methods for the analysis of nonlinear time series, including the methods for the detection of nonlinearities, the methods for the estimation of nonlinear models, and the methods for the forecasting of nonlinear time series. We also discuss the methods for the analysis of multivariate nonlinear time series and the methods for the analysis of time series with missing data. Finally, we provide a discussion of the limitations of the methods for the analysis of nonlinear time series and the directions for future research. The paper is intended to serve as a reference for researchers and practitioners in the field of time series analysis.

**Similar Documents**: This paper offers a comprehensive review of techniques employed in the analysis of nonlinear time series, a specialized area within statistics that focuses on interpreting data collected sequentially over time. Initially, we present an overview of fundamental time series analysis concepts. Subsequently, we delve into a detailed examination of methodologies for analyzing

We present the novel high-level visualization taxonomy. Our taxonomy classifies visualization algorithms rather than data. Algorithms are categorized based on the assumptions they make about the data being visualized; we call this set of assumptions the design model. Because our taxonomy is based on design models, it is more flexible than existing taxonomies and considers the user's conceptual model, emphasizing the human aspect of visualization. Design models are classified according to whether they are discrete or continuous and by how much the algorithm designer chooses display attributes such as spatialization, timing, colour, and transparency. This novel approach provides an alternative view of the visualization field that helps explain how traditional divisions (e.g., information and scientific visualization) relates and overlap, and that may inspire research ideas in hybrid visualization areas

0306-4573/$ see front matter Published by Elsevi doi:10.1016/j.ipm.2008.07.005 * Corresponding author. Tel.: +1 601 266 4035; fa E-mail addresses: Yuelin.Li@usm.edu (Y. Li), nick 1 Tel.: +1 732 932 7500x8271; fax: +1 732 932 69 The nature of the task that leads a person to engage in information interaction, as well as of information seeking and searching tasks, have been shown to influence individuals’ information behavior. Classifying tasks in a domain has been viewed as a departure point of studies on the relationship between tasks and human information behavior. However, previous task classification schemes either classify tasks with respect to the requirements of specific studies or merely classify a certain category of task. Such approaches do not lead to a holistic picture of task since a task involves different aspects. Therefore, the present study aims to develop a faceted classification of task, which can incorporate work tasks and information search tasks into the same classification scheme and characterize tasks in such a way as to help people make predictions of information behavior. For this purpose, previous task classification schemes and their underlying facets are reviewed and discussed. Analysis identifies essential facets and categorizes them into Generic facets of task and Common attributes of task. Generic facets of task include Source of task, Task doer, Time, Action, Product, and Goal. Common attributes of task includes Task characteristics and User’s perception of task. Corresponding sub-facets and values are identified as well. In this fashion, a faceted classification of task is established which could be used to describe users’ work tasks and information search tasks. This faceted classification provides a framework to further explore the relationships among work tasks, search tasks, and interactive information retrieval and advance adaptive IR systems design. Published by Elsevier Ltd.

Open Source Software (OSS) has become the subject of much commercial interest of late. Certainly, OSS seems to hold much promise in addressing the core issues of the software crisis, namely that of software taking too long to develop, exceeding its budget, and not working very well. Indeed, there have been several examples of significant OSS success stories—the Linux operating system, the Apache web server, the BIND domain name resolution utility, to name but a few. However, little by way of rigorous academic research on OSS has been conducted to date. In this study, a framework was derived from two previous frameworks which have been very influential in the IS field, namely that of Zachman’s IS architecture (ISA) and Checkland’s CATWOE framework from Soft Systems Methodology (SSM). The resulting framework is used to analyze the OSS approach in detail. The potential future of OSS research is also discussed.

Synthesizing the evidence from a set of studies that spans many countries and years, and that incorporates a wide variety of research methods and theoretical perspectives, is probably the single most challenging task of performing a systematic review. In this paper, we perform a tertiary review to assess the types and methods of research synthesis in systematic reviews in software engineering. Almost half of the 31 studies included in our review did not contain any synthesis; of the ones that did, two thirds performed a narrative or a thematic synthesis. The results show that, despite the focus on systematic reviews, there is, currently, limited attention to research synthesis in software engineering. This needs to change and a repertoire of synthesis methods needs to be an integral part of systematic reviews to increase their significance and utility for research and practice.

Using the 138,751 patents filed in 2006 under the Patent Cooperation Treaty, coclassification analysis is pursued on the basis of threeand four-digit codes in the International Patent Classification (IPC, 8 edition). The co-classifications among the patents enable us to analyze and visualize the relations among technologies at different levels of aggregation. The hypothesis that classifications might be considered as the organizers of patents into classes, and that therefore co-classification patterns—more than co-citation patterns—might be useful for mapping, is not corroborated. The classifications hang weakly together, even at the four-digit level; at the country level, more specificity can be made visible. However, countries are not the appropriate units of analysis because patent portfolios are largely similar in many advanced countries in terms of the classes attributed. Instead of classes, one may wish to explore the mapping of title words as a better approach to visualize the intellectual organization of patents.

Community Question Answering (CQA) websites provide a rapidly growing source of information in many areas. This rapid growth, while offering new opportunities, puts forward new challenges. In most CQA implementations there is little effort in directing new questions to the right group of experts. This means that experts are not provided with questions matching their expertise, and therefore new matching questions may be missed and not receive a proper answer. We focus on finding experts for a newly posted question. We investigate the suitability of two statistical topic models for solving this issue and compare these methods against more traditional Information Retrieval approaches. We show that for a dataset constructed from the Stackoverflow website, these topic models outperform other methods in retrieving a candidate set of best experts for a question. We also show that the Segmented Topic Model gives consistently better performance compared to the Latent Dirichlet Allocation Model.

Systematic literature reviews (SLRs) have been gaining a significant amount of attention from Software Engineering researchers since 2004. SLRs are considered to be a new research methodology in Software Engineering, which allow evidence to be gathered with regard to the usefulness or effectiveness of the technology proposed in Software Engineering for the development and maintenance of software products. This is demonstrated by the growing number of publications related to SLRs that have appeared in recent years. While some tools exist that can support some or all of the activities of the SLR processes defined in (Kitchenham & Charters, 2007), these are not free. The objective of this paper is to present the SLR-Tool, which is a free tool and is available on the following website: http://alarcosj.esi.uclm.es/SLRTool/, to be used by researchers from any discipline, and not only Software Engineering. SLR-Tool not only supports the process of performing SLRs proposed in (Kitchenham & Charters, 2007), but also provides additional functionalities such as: refining searches within the documents by applying text mining techniques; defining a classification schema in order to facilitate data synthesis; exporting the results obtained to the format of tables and charts; and exporting the references from the primary studies to the formats used in bibliographic packages such as EndNote, BibTeX or Ris. This tool has, to date, been used by members of the Alarcos Research Group and PhD students, and their perception of it is that it is both highly necessary and useful. Our purpose now is to circulate the use of SLR-Tool throughout the entire research community in order to obtain feedback from other users.

Deep learning has made significant breakthroughs in various fields of artificial intelligence. Advantages of deep learning include the ability to capture highly complicated features, weak involvement of human engineering, etc. However, it is still virtually impossible to use deep learning to analyze programs since deep architectures cannot be trained effectively with pure back propagation. In this pioneering paper, we propose the “coding criterion” to build program vector representations, which are the premise of deep learning for program analysis. Our representation learning approach directly makes deep learning a reality in this new field. We evaluate the learned vector representations both qualitatively and quantitatively. We conclude, based on the experiments, the coding criterion is successful in building program representations. To evaluate whether deep learning is beneficial for program analysis, we feed the representations to deep neural networks, and achieve higher accuracy in the program classification task than “shallow” methods, such as logistic regression and the support vector machine. This result confirms the feasibility of deep learning to analyze programs. It also gives primary evidence of its success in this new field. We believe deep learning will become an outstanding technique for program analysis in the near future.

This paper presents the results of the first study of the uniqueness of source code. We define the uniqueness of a unit of source code with respect to the entire body of written software, which we approximate with a corpus of 420 million lines of source code. Our high-level methodology consists of examining a collection of 6,000 software projects and measuring the degree to which each project can be `assembled' solely from portions of this corpus, thus providing a precise measure of `uniqueness' that we call syntactic redundancy. We parameterized our study over a variety of variables, the most important of which being the level of granularity at which we view source code. Our suite of experiments together consumed approximately four months of CPU time, providing quantitative answers to the following questions: at what levels of granularity is software unique, and at a given level of granularity, how unique is software? While we believe these questions to be of intrinsic interest, we discuss possible applications to genetic programming and developer productivity tools.

What constitutes an enterprise architecture framework is a contested subject. The contents of present enterprise architecture frameworks thus differ substantially. This paper aims to alleviate the confusion regarding which framework contains what by proposing a meta framework for enterprise architecture frameworks. By using this meta framework, decision makers are able to express their requirements on what their enterprise architecture framework must contain and also to evaluate whether the existing frameworks meets these requirements. An example classification of common EA frameworks illustrates the approach.