Question: Generating artificial social media data through simulation techniques.

This paper delves into the recent advancements in utilizing deep neural networks (DNNs) for acoustic modeling within speech recognition systems, a domain traditionally dominated by hidden Markov models (HMMs) and Gaussian mixture models (GMMs). The conventional approach involves HMMs to manage the temporal fluctuations in speech, while GMMs are employed to gauge the suitability of each HMM state for a set of coefficients representing the acoustic input. A novel method proposes the employment of feedforward neural networks, specifically deep architectures with numerous hidden layers, to assess the fit between input frames and HMM states. These DNNs, when trained with innovative techniques, have demonstrated superior performance over GMMs across a spectrum of speech recognition benchmarks, often achieving substantial gains. This paper synthesizes the collective insights and experiences of four research teams that have achieved notable successes in leveraging DNNs for acoustic modeling in speech recognition tasks.

**Question**: A Comparative Study of Different Evolutionary Algorithms for Solving the Traveling Salesman Problem

**Documents**: An Improved Ant Colony Optimization Algorithm for the Traveling Salesman Problem
Ant Colony Optimization (ACO) is a population-based metaheuristic that can be used to solve hard optimization problems. In this paper, we present an improved ACO algorithm for the traveling salesman problem (TSP). The proposed algorithm incorporates a new pheromone update strategy and a local search procedure to improve the solution quality. The new pheromone update strategy is based on the best-so-far solution, while the local search procedure is a 2-opt heuristic that is applied to the solutions generated by the ACO algorithm. The proposed algorithm is tested on a set of benchmark TSP instances, and the results are compared with those obtained by other ACO algorithms. The experimental results show that the proposed algorithm outperforms the other ACO algorithms in terms of solution quality and convergence speed.

**Similar Documents**: This paper introduces an enhanced version of the Ant Colony Optimization (ACO) algorithm specifically tailored for solving the Traveling Salesman Problem (TSP). ACO, as a metaheuristic technique, is renowned for its capability to tackle complex optimization challenges. The proposed enhancement to the ACO algorithm for TSP incorporates a refined pheromone update mechanism and an integrated local search strategy aimed at enhancing solution accuracy. The novel pheromone update scheme is anchored on the best-so-far solution, whereas the local search technique is a 2-opt heuristic, which is applied to the solutions generated by the ACO

Timeline generation is an important research task which can help users to have a quick understanding of the overall evolution of one given topic. Previous methods simply split the time span into fixed, equal time intervals without studying the role of the evolutionary patterns of the underlying topic in timeline generation. In addition, few of these methods take users’ collective interests into considerations to generate timelines. We consider utilizing social media attention to address these two problems due to the facts: 1) social media is an important pool of real users’ collective interests; 2) the information cascades generated in it might be good indicators for boundaries of topic phases. Employing Twitter as a basis, we propose to incorporate topic phases and user’s collective interests which are learnt from social media into a unified timeline generation algorithm.We construct both one informativeness-oriented and three interestingness-oriented evaluation sets over five topics.We demonstrate that it is very effective to generate both informative and interesting timelines. In addition, our idea naturally leads to a novel presentation of timelines, i.e., phase based timelines, which can potentially improve user experience.

As a consequence of the social revolution we faced on the Web, news and information we daily enjoy may come from different and diverse sources which are not necessarily the traditional ones such as newspapers, either in their paper or online version, television, radio, etc. Everyone on the Web is allowed to produce and share news which can soon become viral if they follow the new media channels represented by social networks. This freedom in producing and sharing news comes with a counter-effect: the proliferation of fake news. Unfortunately, they can be very effective and may influence people and, more generally, the public opinion. We propose a combined approach of natural language and image processing that takes into account the semantics encoded within both text and images coming with news together with contextual information that may help in the classification of a news as fake or not.

Social Network has become a place where people from every corner of the world has established a virtual civilization. In this virtual community, people used to share their views, express their feelings, photos, videos, blogs, etc. Social Networking Sites like Facebook, Twitters, etc. has given a platform to share innumerable contents with just a click of a button. However, there is no restriction applied by them for the uploaded content. These uploaded content may contains abusive words, explicit images which may be unsuitable for social platforms. As such there is no defined mechanism for restricting offensive contents from publishing on social sites. To solve this problem we have used our proposed approach. In our approach we are developing a social network prototype for implementing our approach for automatic filtering of offensive content in social network. Many popular social networking sites today don't have proper mechanism for restricting offensive contents. They use reporting methods in which user report if the content is abuse. This requires substantial human efforts and time. In this paper, we applied pattern matching algorithm for offensive keyword detection from social networking comments and prevent it from publishing on social platform. Apart from conventional method of reporting abusive contents by users our approach does not requires any human intervention and thus restrict offensive words by detecting and preventing it automatically.

Nowadays, review sites are more and more confronted with the spread of misinformation, i.e., opinion spam, which aims at promoting or damaging some target businesses, by misleading either human readers, or automated opinion mining and sentiment analysis systems. For this reason, in the last years, several data-driven approaches have been proposed to assess the credibility of user-generated content diffused through social media in the form of on-line reviews. Distinct approaches often consider different subsets of characteristics, i.e., features, connected to both reviews and reviewers, as well as to the network structure linking distinct entities on the review-site in exam. This article aims at providing an analysis of the main review- and reviewer-centric features that have been proposed up to now in the literature to detect fake reviews, in particular from those approaches that employ supervised machine learning techniques. These solutions provide in general better results with respect to purely unsupervised approaches, which are often based on graph-based methods that consider relational ties in review sites. Furthermore, this work proposes and evaluates some additional new features that can be suitable to classify genuine and fake reviews. For this purpose, a supervised classifier based on Random Forests have been implemented, by considering both well-known and new features, and a large-scale labeled dataset from which all these features have been extracted. The good results obtained show the effectiveness of new features to detect in particular singleton fake reviews, and in general the utility of this study.

Exponential growth of social media consumption demands eective user interests-social contents modeling for more personalized recommendation and social media summarization. However, due to the heterogeneous nature of social contents, traditional approaches lack the ability of capturing the hidden semantic correlations across these multi-modal data, which leads to semantic gaps between social content understanding and user interests. To eectively bridge the semantic gaps, we propose a novel deep learning framework for user interests-social contents modeling. We rst mine and parse data, i.e. textual content, visual content, social context and social relation, from heterogeneous social media feeds. en, we design a two-branch network to map the social contents and users into a same latent space. Particularly, the network is trained by a largemargin objective that combines a cross-instance distance constraint with a within-instance semantic-preserving constraint in an endto-end manner. At last, a Deep Semantic-Preserving Embedding (D-Sempre) is learned, and the ranking results can be given by calculating distances between social contents and users. To demonstrate the eectiveness of D-Sempre in user interests-social contents modeling, we construct a Twier dataset and conduct extensive experiments on it. As a result, D-Sempre eectively integrates the multimodal data from heterogeneous social media feeds and captures the hidden semantic correlations between users’ interests and social contents.

Recently, there has been tremendous interest in the phenomenon of influence propagation in social networks. The studies in this area assume they have as input to their problems a social graph with edges labeled with probabilities of influence between users. However, the question of where these probabilities come from or how they can be computed from real social network data has been largely ignored until now. Thus it is interesting to ask whether from a social graph and a log of actions by its users, one can build models of influence. This is the main problem attacked in this paper. In addition to proposing models and algorithms for learning the model parameters and for testing the learned models to make predictions, we also develop techniques for predicting the time by which a user may be expected to perform an action. We validate our ideas and techniques using the Flickr data set consisting of a social graph with 1.3M nodes, 40M edges, and an action log consisting of 35M tuples referring to 300K distinct actions. Beyond showing that there is genuine influence happening in a real social network, we show that our techniques have excellent prediction performance.

The SNOW 2014 Data Challenge aimed at creating a public benchmark and evaluation resource for the problem of topic detection in streams of social content. In particular, given a set of tweets spanning a time interval of interest, the Challenge required the extraction of the most significant news topics in short timeslots within the selected interval. Here, we provide details with respect to the Challenge definition, the data collection and evaluation process, and the results achieved by the 11 teams that participated in it, along with a concise retrospective analysis of the main conclusions and arising issues.

Online social networks, such as Twitter, have soared in popularity and in turn have become attractive targets of spam. In fact, spammers have evolved their strategies to stay ahead of Twitter's anti-spam measures in this short period of time. In this paper, we investigate the strategies Twitter spammers employ to reach relevant target audiences. Due to their targeted approaches to send spam, we see evidence of a large number of the spam accounts forming relationships with other Twitter users, thereby becoming deeply embedded in the social network.
 We analyze nearly 20 million tweets from about 7 million Twitter accounts over a period of five days. We identify a set of 14,230 spam accounts that manage to live longer than the other 73% of other spam accounts in our data set. We characterize their behavior, types of tweets they use, and how they target their audience. We find that though spam campaigns changed little from a recent work by Thomas et al., spammer strategies evolved much in the same short time span, causing us to sometimes find contradictory spammer behavior from what was noted in Thomas et al.'s work. Specifically, we identify four major strategies used by 2/3rd of the spammers in our data. The most popular of these was one where spammers targeted their own followers. The availability of various kinds of services that help garner followers only increases the popularity of this strategy. The evolution in spammer strategies we observed in our work suggests that studies like ours should be undertaken frequently to keep up with spammer evolution.

The emergence of online social networks in the past few years has generated an enormous amount of information about potentially any subject. Valuable data containing users' opinions and thoughts are available on those repositories and several sentiment analysis techniques have been proposed that address the problem of understanding the opinion orientation of the users' postings. In this paper, we take a different perspective to the problem through a user centric approach, which uses a graph to model users (and their postings) and applies link mining techniques to infer opinions of users. Preliminary experiments on a Twitter corpus have shown promising results.

*This work is supported by a grant from National Science Foundation (IIS-1111092) and a strategic research grant from City University of Hong Kong (7002770) *The work was done when the first author was visiting University of Illinois at Chicago. Abstract • The Web has seen a tremendous rise in social media. • People’s experiences and opinions expressed in Social Media (e.g., Twitter, Facebook) contain rich social signals (e.g., about consumer sentiment, stock market, economic conditions, etc.). • The goal of this paper is to exploit public sentiment for stock market prediction. • Topic based sentiment is an important factor in stock prediction because it reflects public opinions about current issues/topics. • We employ a Bayesian non-parametric topic-based sentiment time series approach learned from streaming tweets to predict the S&P100 Stock Index.


Question: Generating artificial data for social media platforms.

This paper explores the intricacies of automated documentation generation, specifically within the context of the PLANDoc system, a collaborative project between Columbia University and Bellcore. PLANDoc is designed to chronicle the work of planning engineers engaged in the analysis of telephone routes. It achieves this by analyzing the engineers' interactions with a specialized network planning software and subsequently generating concise, 1-2 page summaries. The development of PLANDoc was significantly influenced by a thorough user needs analysis, which highlighted the necessity for a sublanguage specification. This specification enabled the system to accurately identify input messages and to effectively generate various sentence paraphrases that accurately reflect the input data. A key feature of PLANDoc is its sophisticated use of conjunction in conjunction with paraphrasing techniques. This approach facilitates the creation of summaries that are both succinct—eliminating the repetition of redundant information—and coherent—avoiding the monotonous repetition of similar phrasing. Through this method, PLANDoc ensures that the generated documentation is not only informative but also engaging and easy to read.

Timeline generation is an important research task which can help users to have a quick understanding of the overall evolution of one given topic. Previous methods simply split the time span into fixed, equal time intervals without studying the role of the evolutionary patterns of the underlying topic in timeline generation. In addition, few of these methods take users’ collective interests into considerations to generate timelines. We consider utilizing social media attention to address these two problems due to the facts: 1) social media is an important pool of real users’ collective interests; 2) the information cascades generated in it might be good indicators for boundaries of topic phases. Employing Twitter as a basis, we propose to incorporate topic phases and user’s collective interests which are learnt from social media into a unified timeline generation algorithm.We construct both one informativeness-oriented and three interestingness-oriented evaluation sets over five topics.We demonstrate that it is very effective to generate both informative and interesting timelines. In addition, our idea naturally leads to a novel presentation of timelines, i.e., phase based timelines, which can potentially improve user experience.

Using semantic technologies for mining and intelligent information access to social media is a challenging, emerging research area. Traditional search methods are no longer able to address the more complex information seeking behaviour in media streams, which has evolved towards sense making, learning, investigation, and social search. Unlike carefully authored news text and longer web context, social media streams pose a number of new challenges, due to their large-scale, short, noisy, contextdependent, and dynamic nature. This paper defines five key research questions in this new application area, examined through a survey of state-of-the-art approaches for mining semantics from social media streams; user, network, and behaviour modelling; and intelligent, semanticbased information access. The survey includes key methods not just from the Semantic Web research field, but also from the related areas of natural language processing and user modelling. In conclusion, key outstanding challenges are discussed and new directions for research are proposed.

With the explosion of social media sites and proliferation of digital computing devices and Internet access, massive amounts of public data is being generated on a daily basis. Efficient techniques/ algorithms to analyze this massive amount of data can provide near real-time information about emerging trends and provide early warning in case of an imminent emergency (such as the outbreak of a viral disease). In addition, careful mining of these data can reveal many useful indicators of socioeconomic and political events, which can help in establishing effective public policies. The focus of this study is to review the application of big data analytics for the purpose of human development. The emerging ability to use big data techniques for development (BD4D) promises to revolutionalize healthcare, education, and agriculture; facilitate the alleviation of poverty; and help to deal with humanitarian crises and violent conflicts. Besides all the benefits, the large-scale deployment of BD4D is beset with several challenges due to the massive size, fast-changing and diverse nature of big data. The most pressing concerns relate to efficient data acquisition and sharing, establishing of context (e.g., geolocation and time) and veracity of a dataset, and ensuring appropriate privacy. In this study, we provide a review of existing BD4D work to study the impact of big data on the development of society. In addition to reviewing the important works, we also highlight important challenges and open issues.

With the recent rise in popularity and scale of social media, a growing need exists for systems that can extract useful information from huge amounts of data. We address the issue of detecting influenza epidemics. First, the proposed system extracts influenza related tweets using Twitter API. Then, only tweets that mention actual influenza patients are extracted by the support vector machine (SVM) based classifier. The experiment results demonstrate the feasibility of the proposed approach (0.89 correlation to the gold standard). Especially at the outbreak and early spread (early epidemic stage), the proposed method shows high correlation (0.97 correlation), which outperforms the state-of-the-art methods. This paper describes that Twitter texts reflect the real world, and that NLP techniques can be applied to extract only tweets that contain useful information.

One of the key features of social networks is that users are able to share information, and through cascades of sharing information, this information may reach a large number of individuals. The high availability of user-provided contents on online social media facilitates people aggregation around shared beliefs, interests, worldviews and narratives. With lack of means to verify information, social media has been accused of becoming a hot bed for sharing of misinformation. Facebook, as one of the largest social networking services, has been facing widespread criticism on how its newsfeed algorithm is designed thus amplifying dissemination of misinformation. In late 2016, Facebook revealed plans to address fake news on Facebook newsfeeds. In this work, we study the methods Facebook has proposed to combat the spread of misinformation and compare it with our previously proposed approach called ‘Right-click Authenticate’. By analyzing the Business Process Modeling and Notation of both approaches, this paper suggests some key weaknesses and improvements social media companies need to consider when tackling the spread of misinformation online.

Purpose: The present case study investigates the usefulness of altmetrics for measuring the broad impact of research. Methods: This case study is based on a sample of 1,082 PLOS (the Public Library of Science) journal articles recommended in F1000. The dataset includes altmetrics which were provided by PLOS. The F1000 dataset contains tags on papers which were assigned by experts to characterise them. Findings: Results from the Facebook and Twitter models show higher predicted numbers of counts for "good for teaching" papers than for those papers where the tag is not set. Further model estimations show that saves by Mendeley users are particularly to be expected when a paper introduces a new practical/ theoretical technique (tag: "technical advance"). The tag "New finding" is statistically significant in the model with which the Facebook counts are evaluated. Conclusions: The "good for teaching" is assigned to papers which could be of interest to a wider circle of readers than the peers in a specialized area. Thus, the results of the current study indicate that Facebook and Twitter, but not Figshare or Mendeley, might provide an indication of which papers are of interest to a broader circle of readers (and not only for the peers in a specialist area), and could therefore be useful for the measurement of the social impact of research.

Privacy of social network data is a growing concern which threatens to limit access to this valuable data source. Analysis of the graph structure of social networks can provide valuable information for revenue generation and social science research, but unfortunately, ensuring this analysis does not violate individual privacy is difficult. Simply anonymizing graphs or even releasing only aggregate results of analysis may not provide sufficient protection. Differential privacy is an alternative privacy model, popular in data-mining over tabular data, which uses noise to obscure individuals' contributions to aggregate results and offers a very strong mathematical guarantee that individuals' presence in the data-set is hidden. Analyses that were previously vulnerable to identification of individuals and extraction of private data may be safely released under differential-privacy guarantees. We review two existing standards for adapting differential privacy to network data and analyse the feasibility of several common social-network analysis techniques under these standards. Additionally, we propose out-link privacy, a novel standard for differential privacy over network data, and introduce two powerful out-link private algorithms for common network analysis techniques that were infeasible to privatize under previous differential privacy standards.

Nowadays, striking growth of online social media has led to easier and faster spreading of rumors on cyber space, in addition to tradition ways. In this paper, rumor detection on Persian Twitter community has been addressed for the first time by exploring and analyzing the significances of two categories of rumor features: Structural and Content-based features. While applying both feature sets leads to precision more than 80%, precision around 70% using only structural features has been obtained. Moreover, we show how features of users tending to produce and spread rumors are effective in rumor detection process. Also, the experiments have led to learning of a language model of collected rumors. Finally, all features have been ranked and the most discriminative ones have been discussed.

In this paper we introduce a novel computational technique of extraction of personality traits (HEXACO) of employees from Enterprise Social Media posts. We deal with challenges such as not being able to use existing survey instruments for scoring and not being able to directly use existing psychological studies on written text due to lack of overlapping words between the existing dictionary and words used in Enterprise Social Media. Using our approach we are able to infer personality traits (HEXACO) from posts and find better coverage and usage of the extended dictionary.

Abstract—Nowadays social media information, such as news, links, images, or VDOs, is shared extensively. However, the effectiveness of disseminating information through social media lacks in quality: less fact checking, more biases, and several rumors. Many researchers have investigated about credibility on Twitter, but there is no the research report about credibility information on Facebook. This paper proposes features for measuring credibility on Facebook information. We developed the system for credibility on Facebook. First, we have developed FB credibility evaluator for measuring credibility of each post by manual human’s labelling. We then collected the training data for creating a model using Support Vector Machine (SVM). Secondly, we developed a chrome extension of FB credibility for Facebook users to evaluate the credibility of each post. Based on the usage analysis of our FB credibility chrome extension, about 81% of users’ responses agree with suggested credibility automatically computed by the proposed system.


Question: An empirical comparison of multimodal feature fusion techniques for CNN-based gait recognition.

This study introduces a novel Gait Recognition framework employing 3D Convolutional Neural Networks (CNNs) to analyze multi-view gait patterns, effectively capturing spatio-temporal characteristics. The proposed method utilizes a unique input representation, combining grayscale imagery with optical flow, to mitigate the effects of color variations. Evaluated across three diverse datasets, featuring changes in attire, walking velocity, and viewing angles, the 3D CNN architecture demonstrates superior generalization capabilities, particularly in handling significant view angle shifts. Contrary to conventional Gait Recognition systems, this approach showcases competitive, if not superior, performance, especially under conditions with substantial view discrepancies.

With the exponential growth of web meta-data, exploiting multimodal online sources via standard search engine has become a trend in visual recognition as it effectively alleviates the shortage of training data. However, the web meta-data such as text data is usually not as cooperative as expected due to its unstructured nature. To address this problem, this paper investigates the numerical representation of web text data. We firstly adopt convolutional neural network (CNN) for web text modeling on top of word vectors. Combined with CNN for image, we present a multimodal fusion to maximize the discriminative power of visual and textual modality data for decision level and feature level simultaneously. Experimental results show that the proposed framework achieves significant improvement in large-scale image classification on Pascal VOC-2007 and VOC-2012 datasets.

Achieving high classification accuracy remains a challenge for human action recognition approaches that are based on convolutional neural networks (CNN). CNN-based action recognition methods on resource-constrained edge systems are currently unable to train on whole videos due to infeasible computational and memory requirements. On the other hand, approaches that utilize video-level supervision with sparse-sampling designs run the risk of learning local features shared with multiple similar classes. Additionally, features captured by motion estimation algorithms for temporal stream CNN's are already reduced in dimension, increasing the possibility of label mismatch in methods that rely on short-term features. To address the aforementioned points, we design a novel temporal representation to capture a 1) long-term interval of motion and 2) integrate the trajectory of motion captured therein. We compare our approach with current motion representations and demonstrate its efficacy for examples containing local features with high inter-class similarity. We implement our representation as part of two and three-stream CNN's and conduct experiments on one of the popular and most difficult action recognition datasets: HMDB51. Our results show that, for methods employing sparse-sampling designs, our approach surpasses the current state-of-the-art approaches achieving 71.76% on HMDB51.

Convolutional Neural Networks have been highly successful in performing a host of computer vision tasks such as object recognition, object detection, image segmentation and texture synthesis. In 2015, Gatys et. al [7] show how the style of a painter can be extracted from an image of the painting and applied to another normal photograph, thus recreating the photo in the style of the painter. The method has been successfully applied to a wide range of images and has since spawned multiple applications and mobile apps. In this paper, the neural style transfer algorithm is applied to fashion so as to synthesize new custom clothes. We construct an approach to personalize and generate new custom clothes based on a user’s preference and by learning the user’s fashion choices from a limited set of clothes from their closet. The approach is evaluated by analyzing the generated images of clothes and how well they align with the user’s fashion style.

The caption retrieval task can be defined as follows: given a set of images I and a set of describing sentences S, for each image i in I we ought to find the sentence in S that best describes i. The most commonly applied method to solve this problem is to build a multimodal space and to map each image and each sentence to that space, so that they can be compared easily. A non-conventional model called Word2VisualVec has been proposed recently: instead of mapping images and sentences to a multimodal space, they mapped sentences directly to a space of visual features. Advances in the computation of visual features let us infer that such an approach is promising. In this paper, we propose a new Recurrent Neural Network model following that unconventional approach based on Gated Recurrent Capsules (GRCs), designed as an extension of Gated Recurrent Units (GRUs). We show that GRCs outperform GRUs on the caption retrieval task. We also state that GRCs present a great potential for other applications.

While the accuracy of feature measurements heavily depends on changing environmental conditions, studying the consequences of this fact in pattern recognition tasks has received relatively little attention to date. In this paper, we explicitly take feature measurement uncertainty into account and show how multimodal classification and learning rules should be adjusted to compensate for its effects. Our approach is particularly fruitful in multimodal fusion scenarios, such as audiovisual speech recognition, where multiple streams of complementary time-evolving features are integrated. For such applications, provided that the measurement noise uncertainty for each feature stream can be estimated, the proposed framework leads to highly adaptive multimodal fusion rules which are easy and efficient to implement. Our technique is widely applicable and can be transparently integrated with either synchronous or asynchronous multimodal sequence integration architectures. We further show that multimodal fusion methods relying on stream weights can naturally emerge from our scheme under certain assumptions; this connection provides valuable insights into the adaptivity properties of our multimodal uncertainty compensation approach. We show how these ideas can be practically applied for audiovisual speech recognition. In this context, we propose improved techniques for person-independent visual feature extraction and uncertainty estimation with active appearance models, and also discuss how enhanced audio features along with their uncertainty estimates can be effectively computed. We demonstrate the efficacy of our approach in audiovisual speech recognition experiments on the CUAVE database using either synchronous or asynchronous multimodal integration models.

In this work, we propose a novel latent fingerprint enhancement method based on FingerNet inspired by recent development of Convolutional Neural Network (CNN). Although CNN is achieving superior performance in many computer vision tasks from low-level image processing to high-level semantic understanding, limited attention has been paid in fingerprint community. The proposed FingerNet has three major parts: one common convolution part shared by two different deconvolution parts, which are the enhancement branch and the orientation branch. The convolution part is to extract fingerprint features particularly for enhancement purpose. The enhancement deconvolution branch is employed to remove structured noise and enhance fingerprints as its task. The orientation deconvolution branch performs the task of guiding enhancement through a multi-task learning strategy. The network is trained in the manner of pixels-to-pixels and end-to-end learning, that can directly enhance latent fingerprint as the output. We also study some implementation details such as single-task learning, multi-task learning, and the residual learning. Experimental results of the FingerNet system on latent fingerprint dataset NIST SD27 demonstrate effectiveness and robustness of the proposed method.

Deep Neural Networks (DNNs) have shown to outperform traditional methods in various visual recognition tasks including Facial Expression Recognition (FER). In spite of efforts made to improve the accuracy of FER systems using DNN, existing methods still are not generalizable enough in practical applications. This paper proposes a 3D Convolutional Neural Network method for FER in videos. This new network architecture consists of 3D Inception-ResNet layers followed by an LSTM unit that together extracts the spatial relations within facial images as well as the temporal relations between different frames in the video. Facial landmark points are also used as inputs to our network which emphasize on the importance of facial components rather than the facial regions that may not contribute significantly to generating facial expressions. Our proposed method is evaluated using four publicly available databases in subject-independent and cross-database tasks and outperforms state-of-the-art methods.

Deep learning is popular as an end-to-end framework extracting the prominent features and performing the classification also. In this paper, we extensively investigate deep networks as an alternate to feature encoding technique of lowlevel descriptors for emotion recognition on the benchmark EmoDB dataset. Fusion performance with such obtained encoded features with other available features is also investigated. Highest performance to date in the literature is observed.

This paper investigates the potential of physiological signals as reliable channels for multi-subject emotion recognition. A three-stage decision framework is proposed for recognizing four emotions of multiple subjects. The decision framework consists of three stages: (1) in the initial stage, identifying a subject group that a test subject can be mapped to; (2) in the second stage, identifying an emotion pool that an instance of the test subject can be assigned to; and (3) in the final stage, generating the predicted emotion from the given emotion pool for the test instance. In comparison with a series of alternative methods, the high accuracy of 70.04% achieved by our proposed method clearly demonstrates the potential of the three-stage decision method in multi-subject emotion recognition.

Ground reaction forces generated during normal walking have recently been used to identify and/or classify individuals based upon the pattern of the forces observed over time. One feature that can be extracted from vertical ground reaction forces is body mass. This single feature has identifying power comparable to other studies that use multiple and more complex features. This study contributes to understanding the role of body mass in identification by (1) quantifying the accuracy and precision with which body mass can be obtained using vertical ground reaction forces, (2) quantifying the distribution of body mass across a population larger than has previously been studied in relation to gait analysis, and (3) quantifying the expected identification capabilities of systems using body mass as a weak biometric. Our results show that body mass can be measured in a fraction of a second with less than a 1 kilogram standard deviation of error.


Question: An empirical comparison of multimodal feature fusion for CNN-based gait recognition.

This paper introduces the TUM Gait from Audio, Image, and Depth (GAID) database, a comprehensive resource designed for multimodal recognition of individuals and traits through gait analysis. Gait recognition, the process of identifying people by their walking style, has garnered significant attention in recent years. Traditionally, gait recognition studies have relied solely on data obtained from RGB video streams. The GAID database, however, offers a more holistic approach by incorporating RGB video, depth information, and audio, thus enabling a multimodal analysis of gait. With a dataset comprising 305 individuals across three variations, the GAID database is among the largest of its kind. To explore the impact of temporal variations, a subset of 32 subjects was recorded twice. The paper establishes standardized experimental protocols for person identification and the evaluation of soft biometric traits such as age, gender, height, and shoe type. For each experiment, baseline results are provided across all modalities, highlighting the advantages of multimodal fusion in enhancing gait recognition accuracy.

The ability to accurately identify human activities is essential for developing automatic rehabilitation and sports training systems. In this paper, large-scale exercise motion data obtained from a forearm-worn wearable sensor are classified with a convolutional neural network (CNN). Time-series data consisting of accelerometer and orientation measurements are formatted as images, allowing the CNN to automatically extract discriminative features. A comparative study on the effects of image formatting and different CNN architectures is also presented. The best performing configuration classifies 50 gym exercises with 92.1% accuracy.

This study proposes a modified convolutional neural network (CNN) algorithm that is based on dropout and the stochastic gradient descent (SGD) optimizer (MCNN-DS), after analyzing the problems of CNNs in extracting the convolution features, to improve the feature recognition rate and reduce the time-cost of CNNs. The MCNN-DS has a quadratic CNN structure and adopts the rectified linear unit as the activation function to avoid the gradient problem and accelerate convergence. To address the overfitting problem, the algorithm uses an SGD optimizer, which is implemented by inserting a dropout layer into the all-connected and output layers, to minimize cross entropy. This study used the datasets MNIST, HCL2000, and EnglishHand as the benchmark data, analyzed the performance of the SGD optimizer under different learning parameters, and found that the proposed algorithm exhibited good recognition performance when the learning rate was set to [0.05, 0.07]. The performances of WCNN, MLP-CNN, SVM-ELM, and MCNN-DS were compared. Statistical results showed the following: (1) For the benchmark MNIST, the MCNN-DS exhibited a high recognition rate of 99.97%, and the time-cost of the proposed algorithm was merely 21.95% of MLP-CNN, and 10.02% of SVM-ELM; (2) Compared with SVM-ELM, the average improvement in the recognition rate of MCNN-DS was 2.35% for the benchmark HCL2000, and the time-cost of MCNN-DS was only 15.41%; (3) For the EnglishHand test set, the lowest recognition rate of the algorithm was 84.93%, the highest recognition rate was 95.29%, and the average recognition rate was 89.77%.

While the accuracy of feature measurements heavily depends on changing environmental conditions, studying the consequences of this fact in pattern recognition tasks has received relatively little attention to date. In this paper, we explicitly take feature measurement uncertainty into account and show how multimodal classification and learning rules should be adjusted to compensate for its effects. Our approach is particularly fruitful in multimodal fusion scenarios, such as audiovisual speech recognition, where multiple streams of complementary time-evolving features are integrated. For such applications, provided that the measurement noise uncertainty for each feature stream can be estimated, the proposed framework leads to highly adaptive multimodal fusion rules which are easy and efficient to implement. Our technique is widely applicable and can be transparently integrated with either synchronous or asynchronous multimodal sequence integration architectures. We further show that multimodal fusion methods relying on stream weights can naturally emerge from our scheme under certain assumptions; this connection provides valuable insights into the adaptivity properties of our multimodal uncertainty compensation approach. We show how these ideas can be practically applied for audiovisual speech recognition. In this context, we propose improved techniques for person-independent visual feature extraction and uncertainty estimation with active appearance models, and also discuss how enhanced audio features along with their uncertainty estimates can be effectively computed. We demonstrate the efficacy of our approach in audiovisual speech recognition experiments on the CUAVE database using either synchronous or asynchronous multimodal integration models.

Matching people across nonoverlapping camera views at different locations and different times, known as person reidentification, is both a hard and important problem for associating behavior of people observed in a large distributed space over a prolonged period of time. Person reidentification is fundamentally challenging because of the large visual appearance changes caused by variations in view angle, lighting, background clutter, and occlusion. To address these challenges, most previous approaches aim to model and extract distinctive and reliable visual features. However, seeking an optimal and robust similarity measure that quantifies a wide range of features against realistic viewing conditions from a distance is still an open and unsolved problem for person reidentification. In this paper, we formulate person reidentification as a relative distance comparison (RDC) learning problem in order to learn the optimal similarity measure between a pair of person images. This approach avoids treating all features indiscriminately and does not assume the existence of some universally distinctive and reliable features. To that end, a novel relative distance comparison model is introduced. The model is formulated to maximize the likelihood of a pair of true matches having a relatively smaller distance than that of a wrong match pair in a soft discriminant manner. Moreover, in order to maintain the tractability of the model in large scale learning, we further develop an ensemble RDC model. Extensive experiments on three publicly available benchmarking datasets are carried out to demonstrate the clear superiority of the proposed RDC models over related popular person reidentification techniques. The results also show that the new RDC models are more robust against visual appearance changes and less susceptible to model overfitting compared to other related existing models.

There is a topic of interest for many researchers about human activities recognition in recent years. In this paper, we propose the research to recognize human activities by deep learning (DL) algorithm. We collected the data from participants performing activities in order to evaluate the human activities recognition results. After pre-training the deep network, the fine-tuning process begins. Compared with Hidden Markov Model (HMM) and naïve Bayes Classifier (NBC), the experiment results show that the proposed deep learning algorithm is an effective way for recognizing human activities in smart home.

In this paper, we present our system design for audio visual multi-modal depression recognition. To improve the estimation accuracy of the Beck Depression Inventory (BDI) score, besides the Low Level Descriptors (LLD) features and the Local Gabor Binary Pattern-Three Orthogonal Planes (LGBP-TOP) features provided by the 2014 Audio/Visual Emotion Challenge and Workshop (AVEC2014), we extract extra features to capture key behavioural changes associated with depression. From audio we extract the speaking rate, and from video, the head pose features, the Space-Temporal Interesting Point (STIP) features, and local kinematic features via the Divergence-Curl-Shear descriptors. These features describe body movements, and spatio-temporal changes within the image sequence. We also consider global dynamic features, obtained using motion history histogram (MHH), bag of words (BOW) features and vector of local aggregated descriptors (VLAD). To capture the complementary information within the used features, we evaluate two fusion systems - the feature fusion scheme, and the model fusion scheme via local linear regression (LLR). Experiments are carried out on the training set and development set of the Depression Recognition Sub-Challenge (DSC) of AVEC2014, we obtain root mean square error (RMSE) of 7.6697, and mean absolute error (MAE) of 6.1683 on the development set, which are better or comparable with the state of the art results of the AVEC2014 challenge.

Recognizing human activities from video sequences or still images is a challenging task due to problems, such as background clutter, partial occlusion, changes in scale, viewpoint, lighting, and appearance. Many applications, including video surveillance systems, human-computer interaction, and robotics for human behavior characterization, require a multiple activity recognition system. In this work, we provide a detailed review of recent and state-of-the-art research advances in the field of human activity classification. We propose a categorization of human activity methodologies and discuss their advantages and limitations. In particular, we divide human activity classification methods into two large categories according to whether they use data from different modalities or not. Then, each of these categories is further analyzed into sub-categories, which reflect how they model human activities and what type of activities they are interested in. Moreover, we provide a comprehensive analysis of the existing, publicly available human activity classification datasets and examine the requirements for an ideal human activity recognition dataset. Finally, we report the characteristics of future research directions and present some open issues on human activity recognition.

Facial gender and smile classification in unconstrained environment is challenging due to the invertible and large variations of face images. In this paper, we propose a deep model composed of GNet and SNet for these two tasks. We leverage the multi-task learning and the general-to-specific fine-tuning scheme to enhance the performance of our model. Our strategies exploit the inherent correlation between face identity, smile, gender and other face attributes to relieve the problem of over-fitting on small training set and improve the classification performance. We also propose the tasks-aware face cropping scheme to extract attribute-specific regions. The experimental results on the ChaLearn 16 FotW dataset for gender and smile classification demonstrate the effectiveness of our proposed methods.

Recently, Wi-Fi has caught tremendous attention for its ubiquity. Motivated by Wi-Fi’s low cost and privacy preservation, researchers have been putting lots of investigation into its potential on action recognition and even person identification. In this paper, for bringing a new modality for multimedia community, we offer an comprehensive overview on these two topics in Wi-Fi. Also, through looking at these two topics from an unprecedented perspective, we could achieve generality instead of designing specific ad-hoc features for each scenario. Observing the great resemblance of Channel State Information (CSI, a finegrained information captured from the received Wi-Fi signal) to texture, we propose a brand-new framework based on computer vision methods. To minimize the effect of location dependency embedded in CSI, we propose a novel de-noising method based on Singular Value Decomposition (SVD) to eliminate the background energy and effectively extract the channel information of signals reflected by human bodies. From the experiments conducted, we demonstrate the feasibility and efficacy of the proposed methods. Also, we conclude the factors that would affect the performance and highlight a few potential issues that require further deliberations.

Multibiometric systems are being increasingly de- ployed in many large-scale biometric applications (e.g., FBI-IAFIS, UIDAI system in India) because they have several advantages such as lower error rates and larger population coverage compared to unibiometric systems. However, multibiometric systems require storage of multiple biometric templates (e.g., fingerprint, iris, and face) for each user, which results in increased risk to user privacy and system security. One method to protect individual templates is to store only the secure sketch generated from the corresponding template using a biometric cryptosystem. This requires storage of multiple sketches. In this paper, we propose a feature-level fusion framework to simultaneously protect multiple templates of a user as a single secure sketch. Our main contributions include: (1) practical implementation of the proposed feature-level fusion framework using two well-known biometric cryptosystems, namery,fuzzy vault and fuzzy commitment, and (2) detailed analysis of the trade-off between matching accuracy and security in the proposed multibiometric cryptosystems based on two different databases (one real and one virtual multimodal database), each containing the three most popular biometric modalities, namely, fingerprint, iris, and face. Experimental results show that both the multibiometric cryptosystems proposed here have higher security and matching performance compared to their unibiometric counterparts.


Question: An empirical comparison of multimodal feature fusion techniques for CNN-based gait recognition.

This paper delves into the impact of convolutional network depth on accuracy within the context of large-scale image recognition. Our primary contribution lies in a comprehensive evaluation of networks with increasing depth, highlighting that significant enhancements in performance can be achieved by expanding the depth to 16–19 weight layers. This insight was pivotal in our submission for the ImageNet Challenge 2014, where our team secured top positions in both the localization and classification categories. Furthermore, we demonstrate the robust generalization of our representations to other datasets, achieving cutting-edge results. To support ongoing research into the application of deep visual representations in computer vision, we have made our two most effective ConvNet models publicly accessible.

**Question**: Application of Deep Learning in Natural Language Processing: A Comprehensive Review

**Documents**: Deep Learning for Natural Language Processing: A Survey
This survey paper provides an overview of deep learning techniques applied to natural language processing (NLP) tasks. We cover a wide range of NLP applications, from syntactic parsing to sentiment analysis, and discuss how deep learning models, such as recurrent neural networks (RNNs), convolutional neural networks (CNNs), and attention mechanisms, have been used to address these tasks. We also review recent advances in pre-trained language models, like BERT and GPT, which have significantly improved the state-of-the-art in many NLP benchmarks. Finally, we discuss the challenges and future directions in deep learning for NLP, including the need for more interpretable models, better handling of low-resource languages, and the integration of deep learning with symbolic methods.

**Similar Documents**: This review paper offers an extensive exploration of deep learning methodologies in the realm of natural language processing (NLP). It spans various NLP tasks, from syntactic parsing to sentiment analysis, detailing how sophisticated deep learning architectures—such as recurrent neural networks (RNNs), convolutional neural networks (CNNs), and attention mechanisms—have been employed to tackle these challenges. The paper also examines the latest breakthroughs in pre-trained language models, exemplified by BERT and GPT, which have substantially elevated performance benchmarks across numerous NLP tasks. Moreover, it addresses the prevailing hurdles and future trajectories in deep learning for NLP, including the quest for more transparent models, improved support for low-resource languages, and the amalgamation of deep learning with symbolic processing techniques.

This paper presents a biometric user authentication based on a person’s gait. Unlike most previous gait recognition approaches, which are based on machine vision techniques, in our approach gait patterns are extracted from a physical device attached to the lower leg. From the output of the device accelerations in three directions: vertical, forward-backward, and sideways motion of the lower leg are obtained. A combination of these accelerations is used for authentication. Applying two different methods, histogram similarity and cycle length, equal error rates (EER) of 5% and 9% were achieved, respectively.

With the presence of ubiquitous devices in our daily lives, effectively capturing and managing user attention becomes a critical device requirement. While gaze-tracking is typically employed to determine the user’s focus of attention, gaze-lock detection to sense eyecontact with a device is proposed in [16]. This work proposes eye contact detection using deep neural networks, and makes the following contributions: (1) With a convolutional neural network (CNN) architecture, we achieve superior eye-contact detection performance as compared to [16] with minimal data pre-processing ; our algorithm is furthermore validated on multiple datasets, (2) Gaze-lock detection is improved by combining head pose and eye-gaze information consistent with social attention literature, and (3) We demonstrate gaze-locking on an Android mobile platform via CNN model compression.

In this paper, we study the problem of activity recognition and abnormal behaviour detection for elderly people with dementia. Very few studies have attempted to address this problem presumably because of the lack of experimental data in the context of dementia care. In particular, the paper investigates three variants of Recurrent Neural Networks (RNNs): Vanilla RNNs (VRNN), Long Short Term RNNs (LSTM) and Gated Recurrent Unit RNNs (GRU). Here activity recognition is considered as a sequence labelling problem, while abnormal behaviour is flagged based on the deviation from normal patterns. To provide an adequate discussion of the performance of RNNs in this context, we compare them against the state-of-art methods such as Support Vector Machines (SVMs), Naı̈ve Bayes (NB), Hidden Markov Models (HMMs), Hidden Semi-Markov Models (HSMM) and Conditional Random Fields (CRFs). The results obtained indicate that RNNs are competitive with those state-of-art methods. Moreover, the paper presents a methodology for generating synthetic data reflecting on some behaviours of people with dementia given the difficulty of obtaining real-world data. c 6 t r . ublis e ls i r . . i ilit f t f re ce r ra hairs.

Designing systems able to interact with students in a natural manner is a complex and far from solved problem. A key aspect of natural interaction is the ability to understand and appropriately respond to human emotions. This paper details our response to the continuous Audio/Visual Emotion Challenge (AVEC'12) whose goal is to predict four affective signals describing human emotions. The proposed method uses Fourier spectra to extract multi-scale dynamic descriptions of signals characterizing face appearance, head movements and voice. We perform a kernel regression with very few representative samples selected via a supervised weighted-distance-based clustering, that leads to a high generalization power. We also propose a particularly fast regressor-level fusion framework to merge systems based on different modalities. Experiments have proven the efficiency of each key point of the proposed method and our results on challenge data were the highest among 10 international research teams. Key-words Affective computing, Dynamic features, Multimodal fusion, Feature selection, Facial expressions. ACM Classification

Recognizing human activities from video sequences or still images is a challenging task due to problems, such as background clutter, partial occlusion, changes in scale, viewpoint, lighting, and appearance. Many applications, including video surveillance systems, human-computer interaction, and robotics for human behavior characterization, require a multiple activity recognition system. In this work, we provide a detailed review of recent and state-of-the-art research advances in the field of human activity classification. We propose a categorization of human activity methodologies and discuss their advantages and limitations. In particular, we divide human activity classification methods into two large categories according to whether they use data from different modalities or not. Then, each of these categories is further analyzed into sub-categories, which reflect how they model human activities and what type of activities they are interested in. Moreover, we provide a comprehensive analysis of the existing, publicly available human activity classification datasets and examine the requirements for an ideal human activity recognition dataset. Finally, we report the characteristics of future research directions and present some open issues on human activity recognition.

Reliable prediction of affective states in real world scenarios is very challenging and a significant amount of ongoing research is targeted towards improvement of existing systems. Major problems include the unreliability of labels, variations of the same affective states amongst different persons and in different modalities as well as the presence of sensor noise in the signals. This work presents a framework for adaptive fusion of input modalities incorporating variable degrees of certainty on different levels. Using a strategy that starts with ensembles of weak learners, gradually, level by level, the discriminative power of the system is improved by adaptively weighting favorable decisions, while concurrently dismissing unfavorable ones. For the final decision fusion the proposed system leverages a trained Kalman filter. Besides its ability to deal with missing and uncertain values, in its nature, the Kalman filter is a time series predictor and thus a suitable choice to match input signals to a reference time series in the form of ground truth labels. In the case of affect recognition, the proposed system exhibits superior performance in comparison to competing systems on the analysed dataset.

State-of-the-art person re-identification methods seek robust person matching through combining various feature types. Often, these features are implicitly assigned with a single vector of global weights, which are assumed to be universally good for all individuals, independent to their different appearances. In this study, we show that certain features play more important role than others under different circumstances. Consequently, we propose a novel unsupervised approach for learning a bottom-up feature importance, so features extracted from different individuals are weighted adaptively driven by their unique and inherent appearance attributes. Extensive experiments on two public datasets demonstrate that attribute-sensitive feature importance facilitates more accurate person matching when it is fused together with global weights obtained using existing methods.

We propose a novel approach to enhance the discriminability of Convolutional Neural Networks (CNN). The key idea is to build a tree structure that could progressively learn fine-grained features to distinguish a subset of classes, by learning features only among these classes. Such features are expected to be more discriminative, compared to features learned for all the classes. We develop a new algorithm to effectively learn the tree structure among a large number of classes. Experiments on large-scale image classification tasks demonstrate that our method could boost the performance of a given basic CNN model. Our method is quite general, hence it can potentially be used in combination with many other deep learning models.

In this paper, we study the problem of video-based person re-identification. This is more challenging, and of greater practical interest, than conventional image-based person re-identification. To address this problem, we propose the use of convolutional Long Short Term Memory (LSTM) based networks to learn a video-based representation for person re-identification. To this end, we propose to jointly leverage deep Convolutional Neural Networks (CNNs) and LSTM networks. Given sequential video frames of a person, the spatial information encoded in the frames is first extracted by a set of CNNs. An encoderdecoder framework derived from LSTMs is employed to encode the resulting temporal of CNN outputs. This approach leads to a refined feature representation that is able to explicitly model the video as an ordered sequence, while preserving the spatial information. Comparative experiments demonstrate that our approach achieves the state-of-the-art performance for video-based person re-identification on iLIDS-VID and PRID 2011, the two primary public datasets for this purpose.

In this paper, we present a convolutional neural network (CNN) approach for the face verification task. We propose a “Siamese” architecture of two CNNs, with each CNN reduced to only four layers by fusing convolutional and subsampling layers. Network training is performed using the stochastic gradient descent algorithm with annealed global learning rate. Generalization ability of network is investigated via unique pairing of face images, and testing is done on AT&T face database. Experimental work shows that the proposed CNN system can classify a pair of 46×46 pixel face images in 0.6 milliseconds, which is significantly faster compared to equivalent network architecture with cascade of convolutional and subsampling layers. The verification accuracy achieved is 3.33% EER (equal error rate). Learning converges within 20 epochs, and the proposed technique can verify a test subject unseen in training. This work shows the viability of the “Siamese” CNN for face verification applications, and further improvements to the architecture are under construction to enhance its performance.


Question: An empirical comparison of multimodal feature fusion techniques for CNN-based gait recognition.

This paper delves into the intricacies of optical flow estimation, exploring the factors behind the steady enhancement in algorithm accuracy, as evident from the Middlebury optical flow benchmark. Despite the evolution in accuracy, the fundamental formulation remains rooted in the foundational work of Horn and Schunck. Through rigorous analysis, we uncover how contemporary optimization methods, implementation techniques, and the objective function's design contribute to this advancement. Surprisingly, we find that the classical flow formulations, when coupled with modern optimization and implementation practices, can match recent performance benchmarks. A pivotal discovery is the impact of median filtering of intermediate flow fields during optimization, which, although critical for improved performance, paradoxically leads to higher energy solutions. To elucidate the underlying principles, we introduce a novel objective function that mathematically encapsulates the median filtering heuristic. This objective function introduces a nonlocal term that robustly aggregates flow estimates across expansive spatial neighborhoods, enhancing the algorithm's resilience. By further refining this nonlocal term to incorporate knowledge about flow and image boundaries, we develop a method that achieves top-tier performance on the Middlebury benchmark, underscoring the significance of these insights in the realm of optical flow estimation.

This paper proposed a multimodal fusion between brain and peripheral signals for emotion detection. The input signals were electroencephalogram, galvanic skin resistance, temperature, blood pressure and respiration, which can reflect the influence of emotion on the central nervous system and autonomic nervous system respectively. The acquisition protocol is based on a subset of pictures which correspond to three specific areas of valance-arousal emotional space (positively excited, negatively excited, and calm). The features extracted from input signals, and to improve the results, correlation dimension as a strong nonlinear feature is used for brain signals. The performance of the Quadratic Discriminant Classifier has been evaluated on different feature sets: peripheral signals, EEG's, and both. In comparison among the results of different feature sets, EEG signals seem to perform better than other physiological signals, and the results confirm the interest of using brain signals as peripherals in emotion assessment. According to the improvement in EEG results compare in each raw of the table, it seems that nonlinear features would lead to better understanding of how emotional activities work.

The goal of this paper is to implement an efficient FPGA-based hardware architectures for the design of fast artificial vision systems. The proposed architecture is capable of performing classification operations of a Convolutional Neural Network (CNN) in realtime. To show the effectiveness of the architecture, some design examples such as hand posture recognition, character recognition, and face recognition are provided. Experimental results show that the proposed architecture is well suited for embedded artificial computer vision systems requiring high portability, high computational speed, and accurate classification.

This paper presents an approach to people identification using gait based on floor pressure data. By using a large area high resolution pressure sensing floor, we were able to obtain 3D trajectories of the center of foot pressures over a footstep which contain both the 1D pressure profile and 2D position trajectories of the COP. Based on the 3D COP trajectories a set of features are then extracted and used for people identification together with other features such as stride length and cadence. The Fisher linear discriminant is used as the classifier. Encouraging results have been obtained using the proposed method with an average recognition rate of 94% and false alarm rate of 3% using pair-wise footstep data from 10 subjects.

We present a mobile vision system for multi-person tracking in busy environments. Specifically, the system integrates continuous visual odometry computation with tracking-by-detection in order to track pedestrians in spite of frequent occlusions and egomotion of the camera rig. To achieve reliable performance under real-world conditions, it has long been advocated to extract and combine as much visual information as possible. We propose a way to closely integrate the vision modules for visual odometry, pedestrian detection, depth estimation, and tracking. The integration naturally leads to several cognitive feedback loops between the modules. Among others, we propose a novel feedback connection from the object detector to visual odometry which utilizes the semantic knowledge of detection to stabilize localization. Feedback loops always carry the danger that erroneous feedback from one module is amplified and causes the entire system to become instable. We therefore incorporate automatic failure detection and recovery, allowing the system to continue when a module becomes unreliable. The approach is experimentally evaluated on several long and difficult video sequences from busy inner-city locations. Our results show that the proposed integration makes it possible to deliver stable tracking performance in scenes of previously infeasible complexity.

This poster presents an integrated framework to enable using standard non-sequential machine learning tools for accurate multi-modal activity recognition. Our framework contains simple pre- and post-classification strategies such as class-imbalance correction on the learning data using structure preserving oversampling, leveraging the sequential nature of sensory data using smoothing of the predicted label sequence and classifier fusion, respectively, for improved performance. Through evaluation on recent publicly-available OPPORTUNITY activity datasets comprising of a large amount of multi-dimensional, continuous-valued sensory data, we show that our proposed strategies are effective in improving the performance over common techniques such as One Nearest Neighbor (1NN) and Support Vector Machines (SVM). Our framework also shows better performance over sequential probabilistic models, such as Conditional Random Field (CRF) and Hidden Markov Models (HMM) and when these models are used as meta-learners.

Human action recognition has attracted much attentions because of its great potential applications. With the rapid development of computer performance and Internet, the methods of human action recognition based on deep learning become mainstream and develop at a breathless pace. This paper will give a novel reasonable taxonomy and a review of deep learning human action recognition methods based on color videos, skeleton sequences and depth maps. In addition, some datasets and effective tricks in action recognition deep learning methods will be introduced, and also the development trend is discussed.

An increasingly large amount of multimodal content is posted on social media websites such as YouTube and Facebook everyday. In order to cope with the growth of such so much multimodal data, there is an urgent need to develop an intelligent multi-modal analysis framework that can effectively extract information from multiple modalities. In this paper, we propose a novel multimodal information extraction agent, which infers and aggregates the semantic and affective information associated with user-generated multimodal data in contexts such as e-learning, e-health, automatic video content tagging and human-computer interaction. In particular, the developed intelligent agent adopts an ensemble feature extraction approach by exploiting the joint use of tri-modal (text, audio and video) features to enhance the multimodal information extraction process. In preliminary experiments using the eNTERFACE dataset, our proposed multi-modal system is shown to achieve an accuracy of 87.95%, outperforming the best state-of-the-art system by more than 10%, or in relative terms, a 56% reduction in error rate.

In this paper we address the automatic identification of deceit by using a multimodal approach. We collect deceptive and truthful responses using a multimodal setting where we acquire data using a microphone, a thermal camera, as well as physiological sensors. Among all available modalities, we focus on three modalities namely, language use, physiological response, and thermal sensing. To our knowledge, this is the first work to integrate these specific modalities to detect deceit. Several experiments are carried out in which we first select representative features for each modality, and then we analyze joint models that integrate several modalities. The experimental results show that the combination of features from different modalities significantly improves the detection of deceptive behaviors as compared to the use of one modality at a time. Moreover, the use of non-contact modalities proved to be comparable with and sometimes better than existing contact-based methods. The proposed method increases the efficiency of detecting deceit by avoiding human involvement in an attempt to move towards a completely automated non-invasive deception detection process.

State-of-the-art person re-identification methods seek robust person matching through combining various feature types. Often, these features are implicitly assigned with a single vector of global weights, which are assumed to be universally good for all individuals, independent to their different appearances. In this study, we show that certain features play more important role than others under different circumstances. Consequently, we propose a novel unsupervised approach for learning a bottom-up feature importance, so features extracted from different individuals are weighted adaptively driven by their unique and inherent appearance attributes. Extensive experiments on two public datasets demonstrate that attribute-sensitive feature importance facilitates more accurate person matching when it is fused together with global weights obtained using existing methods.

This paper proposes a method for detecting facial regions by combining a Gabor filter and a convolutional neural network. The first stage uses the Gabor filter which extracts intrinsic facial features. As a result of this transformation we obtain four subimages. The second stage of the method concerns the application of the convolutional neural network to these four images. The approach presented in this paper yields better classification performance in comparison to the results obtained by the convolutional neural network alone.


Question: An empirical comparison of multimodal feature fusion techniques for CNN-based gait recognition.

In the realm of recurrent neural networks (RNNs), learning to retain information over long periods has been a significant challenge, primarily due to the issue of decaying error backflow. This paper revisits Hochreiter's (1991) analysis of this problem and proposes a novel solution: Long Short-Term Memory (LSTM), a gradient-based method that efficiently addresses the issue of time-lag learning. By strategically truncating the gradient flow without compromising accuracy, LSTM introduces constant error carousels within special units to maintain a consistent error flow. Multiplicative gate units within these units learn to control access to this constant error flow, enabling the network to effectively manage memory over extended time steps. LSTM is characterized by its locality in both space and time, with computational complexity per time step and weight remaining constant. Our empirical evaluations using artificial data, which encompass local, distributed, real-valued, and noisy pattern representations, demonstrate the superiority of LSTM. Compared to real-time recurrent learning, backpropagation through time, recurrent cascade correlation, Elman networks, and neural sequence chunking, LSTM not only achieves a higher success rate but also learns at a significantly faster pace. Moreover, LSTM successfully tackles intricate, artificial long-time-lag tasks that have proven elusive to other recurrent network algorithms.

Robust visual tracking is a challenging computer vision problem, with many real-world applications. Most existing approaches employ hand-crafted appearance features, such as HOG or Color Names. Recently, deep RGB features extracted from convolutional neural networks have been successfully applied for tracking. Despite their success, these features only capture appearance information. On the other hand, motion cues provide discriminative and complementary information that can improve tracking performance. Contrary to visual tracking, deep motion features have been successfully applied for action recognition and video classification tasks. Typically, the motion features are learned by training a CNN on optical flow images extracted from large amounts of labeled videos. This paper presents an investigation of the impact of deep motion features in a tracking-by-detection framework. We further show that hand-crafted, deep RGB, and deep motion features contain complementary information. To the best of our knowledge, we are the first to propose fusing appearance information with deep motion features for visual tracking. Comprehensive experiments clearly suggest that our fusion approach with deep motion features outperforms standard methods relying on appearance information alone.

We provide a detailed review of the work on human action recognition over the past decade. We refer to “actions” as meaningful human motions. Starting with methods that are based on handcrafted representations, we review the impact of revamped deep neural networks on action recognition. We follow a systematic taxonomy of action recognition approaches to present a coherent discussion over their improvements and fall-backs.

One of the major challenges in person Re-Identification (ReID) is the inconsistent visual appearance of a person. Current works on visual feature and distance metric learning have achieved significant achievements, but still suffer from the limited robustness to pose variations, viewpoint changes, etc., and the high computational complexity. This makes person ReID among multiple cameras still challenging. This work is motivated to learn mid-level human attributes which are robust to visual appearance variations and could be used as efficient features for person matching. We propose a weakly supervised multi-type attribute learning framework which considers the contextual cues among attributes and progressively boosts the accuracy of attributes only using a limited number of labeled data. Specifically, this framework involves a threestage training. A deep Convolutional Neural Network (dCNN) is first trained on an independent dataset labeled with attributes. Then it is fine-tuned on another dataset only labeled with person IDs using our defined triplet loss. Finally, the updated dCNN predicts attribute labels for the target dataset, which is combined with the independent dataset for the final round of fine-tuning. The predicted attributes, namely deep attributes exhibit promising generalization ability across different datasets. By directly using the deep attributes with simple Cosine distance, we have obtained competitive accuracy on four person ReID datasets. Experiments also show that a simple distance metric learning modular further boosts our method, making it outperform many recent ∗Corresponding author Email address: slzhang.jdl@pku.edu.cn (Shiliang Zhang) Preprint submitted to Pattern Recognition September 11, 2017

Despite the steady progress in video analysis led by the adoption of convolutional neural networks (CNNs), the relative improvement has been less drastic as that in 2D static image classification. Three main challenges exist including spatial (image) feature representation, temporal information representation, and model/computation complexity. It was recently shown by Carreira and Zisserman that 3D CNNs, inflated from 2D networks and pretrained on ImageNet, could be a promising way for spatial and temporal representation learning. However, as for model/computation complexity, 3D CNNs are much more expensive than 2D CNNs and prone to overfit. We seek a balance between speed and accuracy by building an effective and efficient video classification system through systematic exploration of critical network design choices. In particular, we show that it is possible to replace many of the 3D convolutions by low-cost 2D convolutions. Rather surprisingly, best result (in both speed and accuracy) is achieved when replacing the 3D convolutions at the bottom of the network, suggesting that temporal representation learning on high-level “semantic” features is more useful. Our conclusion generalizes to datasets with very different properties. When combined with several other cost-effective designs including separable spatial/temporal convolution and feature gating, our system results in an effective video classification system that that produces very competitive results on several action classification benchmarks (Kinetics, Something-something, UCF101 and HMDB), as well as two action detection (localization) benchmarks (JHMDB and UCF101-24).

Cross-media hashing, which conducts cross-media retrieval by embedding data from different modalities into a common low-dimensional hamming space, has attracted intensive attention in recent years. This is motivated by the facts a) the multi-modal data is widespread, e.g., the web images on Flickr are associated with tags, and b) hashing is an effective technique towards large-scale high-dimensional data processing, which is exactly the situation of cross-media retrieval. Inspired by recent advances in deep learning, we propose a cross-media hashing approach based on multi-modal neural networks. By restricting in the learning objective a) the hash codes for relevant cross-media data being similar, and b) the hash codes being discriminative for predicting the class labels, the learned Hamming space is expected to well capture the cross-media semantic relationships and to be semantically discriminative. The experiments on two real-world data sets show that our approach achieves superior cross-media retrieval performance compared with the state-of-the-art methods.

Recently, Wi-Fi has caught tremendous attention for its ubiquity. Motivated by Wi-Fi’s low cost and privacy preservation, researchers have been putting lots of investigation into its potential on action recognition and even person identification. In this paper, for bringing a new modality for multimedia community, we offer an comprehensive overview on these two topics in Wi-Fi. Also, through looking at these two topics from an unprecedented perspective, we could achieve generality instead of designing specific ad-hoc features for each scenario. Observing the great resemblance of Channel State Information (CSI, a finegrained information captured from the received Wi-Fi signal) to texture, we propose a brand-new framework based on computer vision methods. To minimize the effect of location dependency embedded in CSI, we propose a novel de-noising method based on Singular Value Decomposition (SVD) to eliminate the background energy and effectively extract the channel information of signals reflected by human bodies. From the experiments conducted, we demonstrate the feasibility and efficacy of the proposed methods. Also, we conclude the factors that would affect the performance and highlight a few potential issues that require further deliberations.

Research in human action recognition has accelerated significantly since the introduction of powerful machine learning tools such as Convolutional Neural Networks (CNNs). However, effective and efficient methods for incorporation of temporal information into CNNs are still being actively explored in the recent literature. Motivated by the popular recurrent attention models in the research area of natural language processing, we propose the Attention-aware Temporal Weighted CNN (ATW CNN) for action recognition in videos, which embeds a visual attention model into a temporal weighted multi-stream CNN. This attention model is simply implemented as temporal weighting yet it effectively boosts the recognition performance of video representations. Besides, each stream in the proposed ATW CNN framework is capable of end-to-end training, with both network parameters and temporal weights optimized by stochastic gradient descent (SGD) with back-propagation. Our experimental results on the UCF-101 and HMDB-51 datasets showed that the proposed attention mechanism contributes substantially to the performance gains with the more discriminative snippets by focusing on more relevant video segments.

In this paper, we introduce the DEEP-SEE FACE framework, an assistive device designed to improve cognition, interaction, and communication of visually impaired (VI) people in social encounters. The proposed approach jointly exploits computer vision algorithms (region proposal networks, ATLAS tracking and global, and low-level image descriptors) and deep convolutional neural networks in order to detect, track, and recognize, in real-time, various persons existent in the video streams. The major contribution of the paper concerns a global, fixed-size face representation that takes into the account of various video frames while remaining independent of the length of the image sequence. To this purpose, we introduce an effective weight adaptation scheme that is able to determine the relevance assigned to each face instance, depending on the frame degree of motion/camera blur, scale variation, and compression artifacts. Another relevant contribution involves a hard negative mining stage that helps us differentiating between known and unknown face identities. The experimental results, carried out on a large-scale data set, validate the proposed methodology with an average accuracy and recognition rates superior to 92%. When tested in real life, indoor/outdoor scenarios, the DEEP-SEE FACE prototype proves to be effective and easy to use, allowing the VI people to access visual information during social events.

Recent advances with Convolutional Networks (ConvNets) have shifted the bottleneck for many computer vision tasks to annotated data collection. In this paper, we present a geometry-driven approach to automatically collect annotations for human pose prediction tasks. Starting from a generic ConvNet for 2D human pose, and assuming a multi-view setup, we describe an automatic way to collect accurate 3D human pose annotations. We capitalize on constraints offered by the 3D geometry of the camera setup and the 3D structure of the human body to probabilistically combine per view 2D ConvNet predictions into a globally optimal 3D pose. This 3D pose is used as the basis for harvesting annotations. The benefit of the annotations produced automatically with our approach is demonstrated in two challenging settings: (i) fine-tuning a generic ConvNet-based 2D pose predictor to capture the discriminative aspects of a subjects appearance (i.e.,personalization), and (ii) training a ConvNet from scratch for single view 3D human pose prediction without leveraging 3D pose groundtruth. The proposed multi-view pose estimator achieves state-of-the-art results on standard benchmarks, demonstrating the effectiveness of our method in exploiting the available multi-view information.

3-D convolutional neural networks (3-D-convNets) have been very recently proposed for action recognition in videos, and promising results are achieved. However, existing 3-D-convNets has two “artificial” requirements that may reduce the quality of video analysis: 1) It requires a fixed-sized (e.g., 112  $\times$ 112) input video; and 2) most of the 3-D-convNets require a fixed-length input (i.e., video shots with fixed number of frames). To tackle these issues, we propose an end-to-end pipeline named Two-stream 3-D-convNet Fusion, which can recognize human actions in videos of arbitrary size and length using multiple features. Specifically, we decompose a video into spatial and temporal shots. By taking a sequence of shots as input, each stream is implemented using a spatial temporal pyramid pooling (STPP) convNet with a long short-term memory (LSTM) or CNN-E model, softmax scores of which are combined by a late fusion. We devise the STPP convNet to extract equal-dimensional descriptions for each variable-size shot, and we adopt the LSTM/CNN-E model to learn a global description for the input video using these time-varying descriptions. With these advantages, our method should improve all 3-D CNN-based video analysis methods. We empirically evaluate our method for action recognition in videos and the experimental results show that our method outperforms the state-of-the-art methods (both 2-D and 3-D based) on three standard benchmark datasets (UCF101, HMDB51 and ACT datasets).