Question: A neural network-based method for evaluating the quality of document images.

This paper delves into the domain of representation learning, examining its critical role in the efficacy of machine learning algorithms. We posit that the variability in performance can be attributed to how representations either entangle or reveal the explanatory factors underlying the data. While domain-specific knowledge can inform the design of representations, a general approach involves leveraging learning algorithms with generic priors, driven by the pursuit of artificial intelligence. The paper offers a comprehensive review of recent advancements in unsupervised feature learning and deep learning, encompassing developments in probabilistic models, autoencoders, manifold learning, and deep neural networks. This exploration raises fundamental questions regarding the optimal objectives for representation learning, the computational methods for deriving representations, and the geometric interplay between representation learning, density estimation, and manifold learning. These inquiries highlight the need for further research to fully comprehend and harness the potential of representation learning in enhancing machine learning outcomes.

**Question**: An evolutionary approach to feature selection for deep learning models

**Documents**: Deep learning for sound event detection: A review
This paper provides a comprehensive review of the state-of-the-art research in sound event detection (SED) using deep learning (DL) techniques. The focus is on the use of DL for the automatic recognition of sound events in audio recordings, which is a challenging task due to the variability and complexity of the acoustic environment. The review covers various aspects of DL-based SED, including feature extraction, network architectures, data augmentation, and evaluation metrics. It also discusses the limitations and future directions of DL in SED, such as the need for more annotated data, the interpretability of DL models, and the integration of prior knowledge into DL frameworks.

**Similar Documents**: This paper offers an extensive overview of the latest advancements in sound event detection (SED) employing deep learning (DL) methodologies. Central to the discussion is the application of DL for the automated identification of sound events within audio recordings, a task fraught with challenges due to the acoustic environment's inherent variability and complexity. The review encompasses a wide range of DL-based SED topics, from feature extraction techniques to innovative network architectures, data augmentation strategies, and performance evaluation criteria. Moreover, it critically evaluates the current limitations of DL in SED, such as the scarcity of annotated data, the opacity of DL models, and the potential for integrating domain-specific knowledge into DL systems. These insights point to promising research avenues aimed at enhancing DL's capabilities in SED applications.

The objective of the paper is to recognize handwritten samples of Roman numerals using Tesseract open source Optical Character Recognition (OCR) engine. Tesseract is trained with data samples of different persons to generate one user-independent language model, representing the handwritten Roman digit-set. The system is trained with 1226 digit samples collected form the different users. The performance is tested on two different datasets, one consisting of samples collected from the known users (those who prepared the training data samples) and the other consisting of handwritten data samples of unknown users. The overall recognition accuracy is obtained as 92.1% and 86.59% on these test datasets respectively.

Industries can improve their business efficiency by analyzing and extracting relevant knowledge from large numbers of documents. Knowledge extraction manually from large volume of documents is labor intensive, unscalable and challenging. Consequently there have been a number of attempts to develop intelligent systems to automatically extract relevant knowledge from OCR documents. Moreover, the automatic system can improve the capability of search engine by providing application-specific domain knowledge. However, extracting the efficient information from OCR documents is challenging due to highly unstructured format [1, 11, 18, 26]. In this paper, we propose an efficient framework for a knowledge extraction system that takes keywords based queries and automatically extracts their most relevant knowledge from OCR documents by using text mining techniques. The framework can provide relevance ranking of knowledge to a given query. We tested the proposed framework on corpus of documents at GE Power where document consists of more than hundred pages in PDF.

Document digitization is faster and more affordable using digital cameras than scanners. On the other hand, if we aim at extending the basic digital camera functionalities for such a purpose, post-processings will be of first importance, at least to improve the text legibility. In this paper, we address the specific problem of the virtual flattening of curved documents, as for example the pages of an opened book lying on its spine. In order to compute the document shape, we use the shape from shading technique and discuss why, in some cases, it is more suitable than other 3D single-view reconstruction techniques. We extend the seminal work by Wada et al. (Proceedings of the IAPR Workshop on machine vision and applications, Tokyo, Japan, pp. 591–594, 1992) and consecutive papers, reformulating the problem in terms of perspective shape from shading. Finally, we design a complete post-processing algorithm and test it on real images. Even if the documents are much curved, it is shown that the restored images are almost identical to scanned images of the flattened documents.

Categorization of documents is challenging, as the number o f discriminating words can be very large. We present a nearest neighbor classification scheme for text categoriz ation in which the importance of discriminating words is learned using mutual information and weight adjustment tec hniques. The nearest neighbors for a particular document are then computed based on the matching words and their weigh ts. We evaluate our scheme on both synthetic and real world documents. Our experiments with synthetic data sets s how that this scheme is robust under different emulated conditions. Empirical results on real world documents demo nstrate that this scheme outperforms state of the art classification algorithms such as C4.5, RIPPER, Rainbow, an d PEBLS.

This paper compares the performance of several classi er algorithms on a standard database of handwritten digits. We consider not only raw accuracy, but also rejection, training time, recognition time, and memory requirements.

The process to take a Braille document image and convert its content into its equivalent natural language characters, is called Optical Braille Recognition (OBR). It involves two main consecutive steps: Braille cell recognition and Braille cell transcription. Braille cell recognition contains few steps including: Image acquisition, Image De-skewing, Image pre-processing, dot recognition, cell recognition and segmentation. Image transcription aims to convert the segmented Braille cell, into its equivalent natural language characters. In this survey we aim to study the earlier works done by other researchers on both Braille cell recognition and transcription.

Currently defect detection in additive manufacturing is predominately done by traditional image processing, approximation, and statistical methods. Two important aspects of defect detection are edge detection and porosity detection. Both problems appear to be good candidates to apply machine learning. This project describes the implementation of neural networks as an alternative method for defect detection. Results from current methods and the neural networks are compared for both speed and accuracy.

We propose a multi-task end-to-end optimized deep neural network (MEON) for blind image quality assessment (BIQA). MEON consists of two sub-networks—a distortion identification network and a quality prediction network—sharing the early layers. Unlike traditional methods used for training multi-task networks, our training process is performed in two steps. In the first step, we train a distortion type identification sub-network, for which large-scale training samples are readily available. In the second step, starting from the pre-trained early layers and the outputs of the first sub-network, we train a quality prediction sub-network using a variant of the stochastic gradient descent method. Different from most deep neural networks, we choose biologically inspired generalized divisive normalization (GDN) instead of rectified linear unit as the activation function. We empirically demonstrate that GDN is effective at reducing model parameters/layers while achieving similar quality prediction performance. With modest model complexity, the proposed MEON index achieves state-of-the-art performance on four publicly available benchmarks. Moreover, we demonstrate the strong competitiveness of MEON against state-of-the-art BIQA models using the group maximum differentiation competition methodology.

We address the problem of verifying neural-based perception systems implemented by convolutional neural networks. We define a notion of local robustness based on affine and photometric transformations. We show the notion cannot be captured by previously employed notions of robustness. The method proposed is based on reachability analysis for feed-forward neural networks and relies on MILP encodings of both the CNNs and transformations under question. We present an implementation and discuss the experimental results obtained for a CNN trained from the MNIST data set.

With the advent of the Internet, billions of images are now freely available online and constitute a dense sampling of the visual world. Using a variety of non-parametric methods, we explore this world with the aid of a large dataset of 79,302,017 images collected from the Internet. Motivated by psychophysical results showing the remarkable tolerance of the human visual system to degradations in image resolution, the images in the dataset are stored as 32 x 32 color images. Each image is loosely labeled with one of the 75,062 non-abstract nouns in English, as listed in the Wordnet lexical database. Hence the image database gives a comprehensive coverage of all object categories and scenes. The semantic information from Wordnet can be used in conjunction with nearest-neighbor methods to perform object classification over a range of semantic levels minimizing the effects of labeling noise. For certain classes that are particularly prevalent in the dataset, such as people, we are able to demonstrate a recognition performance comparable to class-specific Viola-Jones style detectors.


Question: A neural network-based method for evaluating the quality of document images.

This paper presents an innovative algorithm termed matching pursuit, designed to decompose signals into linear combinations of waveforms selected from a comprehensive, albeit redundant, dictionary of functions. This algorithm uniquely selects waveforms that closely match the intrinsic structures of the signal, thereby offering an adaptive signal representation methodology. Specifically, when the dictionary is comprised of Gabor functions, the matching pursuit algorithm facilitates an adaptive time-frequency transformation. A key contribution is the derivation of a signal energy distribution in the time-frequency plane, which notably avoids interference terms, a common pitfall in Wigner and Cohen class distributions. The algorithm's ability to isolate signal components that exhibit coherence relative to the chosen dictionary is highlighted. An example application is provided, demonstrating the extraction of patterns from noisy signals. The paper also includes a comparative analysis between the matching pursuit decomposition and signal expansion over an optimized wavepacket orthonormal basis, selected using Coifman and Wickerhauser's algorithm, showcasing the distinct advantages of the proposed method.

We combine three methods which significantly improve the OCR accuracy of OCR models trained on early printed books: (1) The pretraining method utilizes the information stored in already existing models trained on a variety of typesets (mixed models) instead of starting the training from scratch. (2) Performing cross fold training on a single set of ground truth data (line images and their transcriptions) with a single OCR engine (OCRopus) produces a committee whose members then vote for the best outcome by also taking the top-N alternatives and their intrinsic confidence values into account. (3) Following the principle of maximal disagreement we select additional training lines which the voters disagree most on, expecting them to offer the highest information gain for a subsequent training (active learning). Evaluations on six early printed books yielded the following results: On average the combination of pretraining and voting improved the character accuracy by 46% when training five folds starting from the same mixed model. This number rose to 53% when using different models for pretraining, underlining the importance of diverse voters. Incorporating active learning improved the obtained results by another 16% on average (evaluated on three of the six books). Overall, the proposed methods lead to an average error rate of 2.5% when training on only 60 lines. Using a substantial ground truth pool of 1,000 lines brought the error rate down even further to less than 1% on average.

Nonparametric Bayesian Models for Joint Analysis of Imagery and Text

Images features extraction is very important for the grading process of flue-cured tobacco leaves. In this paper, a machine vision techniques base system is proposed for the automatic inspection of flue-cured tobacco leaves. Machine vision techniques are used in this system to solve problems of features extraction and analysis of tobacco leaves, which include features of color, size, shape andsurface texture. The experimental results show that this system is a viable way for the features extraction of tobacco leaves, and can be used for the automatic classification of tobacco leaves.

This paper addresses automatic interpretation of images of outdoor scenes. The method allows instances of objects from a number of generic classes to be identi ed: vegetation, buildings, vehicles, roads, etc., thereby enabling image databases to be queried on scene content. The feature set is based, in part, on psychophysical principles and includes measures of colour, texture and shape. Using a large database of ground-truth labelled images, a neural network is trained as a pattern classi er. The method is demonstrated on a large test set to provide highly accurate image interpretations, with over 90% of the image area labelled correctly. Scene Understanding Object Recognition Classi cation Image Segmentation Neural Networks

A neural network model for a mechanism of visual pattern recognition is proposed in this paper. The network is self-organized by “learning without a teacher”, and acquires an ability to recognize stimulus patterns based on the geometrical similarity (Gestalt) of their shapes without affected by their positions. This network is given a nickname “neocognitron”. After completion of self-organization, the network has a structure similar to the hierarchy model of the visual nervous system proposed by Hubel and Wiesel. The network consits of an input layer (photoreceptor array) followed by a cascade connection of a number of modular structures, each of which is composed of two layers of cells connected in a cascade. The first layer of each module consists of “S-cells”, which show characteristics similar to simple cells or lower order hypercomplex cells, and the second layer consists of “C-cells” similar to complex cells or higher order hypercomplex cells. The afferent synapses to each S-cell have plasticity and are modifiable. The network has an ability of unsupervised learning: We do not need any “teacher” during the process of self-organization, and it is only needed to present a set of stimulus patterns repeatedly to the input layer of the network. The network has been simulated on a digital computer. After repetitive presentation of a set of stimulus patterns, each stimulus pattern has become to elicit an output only from one of the C-cell of the last layer, and conversely, this C-cell has become selectively responsive only to that stimulus pattern. That is, none of the C-cells of the last layer responds to more than one stimulus pattern. The response of the C-cells of the last layer is not affected by the pattern's position at all. Neither is it affected by a small change in shape nor in size of the stimulus pattern.

In this paper we investigate the use of a multimodal feature learning approach, using neural network based models such as Skip-gram and Denoising Autoencoders, to address sentiment analysis of micro-blogging content, such as Twitter short messages, that are composed by a short text and, possibly, an image. The approach used in this work is motivated by the recent advances in: i) training language models based on neural networks that have proved to be extremely efficient when dealing with web-scale text corpora, and have shown very good performances when dealing with syntactic and semantic word similarities; ii) unsupervised learning, with neural networks, of robust visual features, that are recoverable from partial observations that may be due to occlusions or noisy and heavily modified images. We propose a novel architecture that incorporates these neural networks, testing it on several standard Twitter datasets, and showing that the approach is efficient and obtains good classification results.

In this paper we present for the first time, the development of a new system for the off-line optical recognition of the characters used in the Orthodox Hellenic Byzantine Music Notation, that has been established since 1814. We describe the structure of the new system and propose algorithms for the recognition of the 71 distinct character classes, based on Wavelets, 4-projections and other structural and statistical features. Using a Nearest Neighbor classifier, combined with a post classification schema and a tree-structured classification philosophy, an accuracy of 99.4 % was achieved, in a database of about 18,000 byzantine character patterns that have been developed for the needs of the system. Optical music recognition Off-line character recognition, Byzantine Music, Byzantine Music Notation, Wavelets, Projections, Neural Networks Contour processing, Nearest Neighbor Classifier Byzantine Music Data Base

Several standard text-categorization techniques were applied to the problem of automated essay grading. Bayesian independence classifiers and knearest-neighbor classifiers were trained to assign scores to manually-graded essays. These scores were combined with several other summary text measures using linear regression. The classifiers and regression equations were then applied to a new set of essays. The classifiers worked very well. The agreement between the automated grader and the final manual grade was as good as the agreement between human graders.

This paper proposes a pencil sketch generating system that can create various pencil sketching styles for images, to satisfy different needs of users. The pencil sketch generating process includes process of three parts: line drawing, tone adjustment, and texture rendering. Some operations needed are first proposed, and various pencil sketching styles that generated by the proposed system are illustrated.

The primary problem dealt with in this paper is the following. Given some description of a visual object, find that object in an actual photograph. Part of the solution to this problem is the specification of a descriptive scheme, and a metric on which to base the decision of "goodness" of matching or detection.


Question: RTIC-C: A big data framework for large-scale traffic information analysis.

This paper presents a novel framework for real-time traffic sign recognition from video sequences, employing class-specific discriminative features to enhance accuracy and responsiveness. The proposed method, developed over a period from August 2007 to May 2009, has been meticulously refined to ensure robust performance under varying environmental conditions. By focusing on distinctive features unique to each traffic sign class, the system is able to efficiently identify and classify signs in real-time, significantly improving upon previous approaches. This innovation is particularly critical for applications requiring immediate interpretation of traffic signals, such as autonomous driving and advanced driver assistance systems. The effectiveness of the proposed method is substantiated through comprehensive testing, demonstrating its capability to accurately recognize traffic signs in diverse and challenging scenarios.

More and more real-time applications need to handle dynamic continuous queries over streaming data of high density. Conventional data and query indexing approaches generally do not apply for excessive costs in either maintenance or space. Aiming at these problems, this study first proposes a new indexing structure by fusing an adaptive cell and KDB-tree, namely CKDB-tree. A cell-tree indexing approach has been developed on the basis of the CKDB-tree that supports dynamic continuous queries. The approach significantly reduces the space costs and scales well with the increasing data size. Towards providing a scalable solution to filtering massive steaming data, this study has explored the feasibility to utilize the contemporary general-purpose computing on the graphics processing unit (GPGPU). The CKDB-tree-based approach has been extended to operate on both the CPU (host) and the GPU (device). The GPGPU-aided approach performs query indexing on the host while perform streaming data filtering on the device in a massively parallel manner. The two heterogeneous tasks execute in parallel and the latency of streaming data transfer between the host and the device is hidden. The experimental results indicate that (1) CKDB-tree can reduce the space cost comparing to the cell-based indexing structure by 60 percent on average, (2) the approach upon the CKDB-tree outperforms the traditional counterparts upon the KDB-tree by 66, 75 and 79 percent in average for uniform, skewed and hyper-skewed data in terms of update costs, and (3) the GPGPU-aided approach greatly improves the approach upon the CKDB-tree with the support of only a single Kepler GPU, and it provides real-time filtering of streaming data with 2.5M data tuples per second. The massively parallel computing technology exhibits great potentials in streaming data monitoring.

The emergence of Big Data applications provides new challenges in data management such as processing and movement of masses of data. Volunteer computing has proven itself as a distributed paradigm that can fully support Big Data generation. This paradigm uses a large number of heterogeneous and unreliable Internet-connected hosts to provide Peta-scale computing power for scientific projects. With the increase in data size and number of devices that can potentially join a volunteer computing project, the host bandwidth can become a main hindrance to the analysis of the data generated by these projects, especially if the analysis is a concurrent approach based on either in-situ or in-transit processing. In this paper, we propose a bandwidth model for volunteer computing projects based on the real trace data taken from the Docking@Home project with more than 280,000 hosts over a 5-year period. We validate the proposed statistical model using model-based and simulation-based techniques. Our modeling provides us with valuable insights on the concurrent integration of data generation with in-situ and in-transit analysis in the volunteer computing paradigm.

Intelligent Transportation Systems (ITS) have attracted the attention of researchers and the general public alike as a means to alleviate traffic congestion. Recently, the maturity of wireless technology has enabled a cost-efficient way to achieve ITS by detecting vehicles using Vehicle to Infrastructure (V2I) communications. Traditional ITS algorithms, in most cases, assume that every vehicle is observed, such as by a camera or a loop detector, but a V2I implementation would detect only those vehicles with wireless communications capability. We examine a family of transportation systems, which we will refer to as ‘Partially Detected Intelligent Transportation Systems’. An algorithm that can act well under a small detection rate is highly desirable due to gradual penetration rates of the underlying wireless technologies such as Dedicated Short Range Communications (DSRC) technology. Artificial Intelligence (AI) techniques for Reinforcement Learning (RL) are suitable tools for finding such an algorithm due to utilizing varied inputs and not requiring explicit analytic understanding or modeling of the underlying system dynamics. In this paper, we report a RL algorithm for partially observable ITS based on DSRC. The performance of this system is studied under different car flows, detection rates, and topologies of the road network. Our system is able to efficiently reduce the average waiting time of vehicles at an intersection, even with a low detection rate.

The limited available fossil fuels and the call for sustainable environment have brought about new technologies for the high efficiency in the use of fossil fuels and introduction of renewable energy. Smart grid is an emerging technology that can fulfill such demands by incorporating advanced information and communications technology (ICT). The pervasive deployment of the advanced ICT, especially the smart metering, will generate big energy data in terms of volume, velocity, and variety. The generated big data can bring huge benefits to the better energy planning, efficient energy generation, and distribution. As such data involve end users' privacy and secure operation of the critical infrastructure, there will be new security issues. This paper is to survey and discuss new findings and developments in the existing big energy data analytics and security. Several taxonomies have been proposed to express the intriguing relationships of various variables in the field.

As mobile devices with positioning capabilities continue to proliferate, data management for so-called trajectory databases that capture the historical movements of populations of moving objects becomes important. This paper considers the querying of such databases for convoys, a convoy being a group of objects that have traveled together for some time. More specifically, this paper formalizes the concept of a convoy query using density-based notions, in order to capture groups of arbitrary extents and shapes. Convoy discovery is relevant for reallife applications in throughput planning of trucks and carpooling of vehicles. Although there has been extensive research on trajectories in the literature, none of this can be applied to retrieve correctly exact convoy result sets. Motivated by this, we develop three efficient algorithms for convoy discovery that adopt the wellknown filter-refinement framework. In the filter step, we apply linesimplification techniques on the trajectories and establish distance bounds between the simplified trajectories. This permits efficient convoy discovery over the simplified trajectories without missing any actual convoys. In the refinement step, the candidate convoys are further processed to obtain the actual convoys. Our comprehensive empirical study offers insight into the properties of the paper’s proposals and demonstrates that the proposals are effective and efficient on real-world trajectory data.

Urban land use information plays an essential role in a wide variety of urban planning and environmental monitoring processes. During the past few decades, with the rapid technological development of remote sensing (RS), geographic information systems (GIS) and geospatial big data, numerous methods have been developed to identify urban land use at a fine scale. Points-of-interest (POIs) have been widely used to extract information pertaining to urban land use types and functional zones. However, it is difficult to quantify the relationship between spatial distributions of POIs and regional land use types due to a lack of reliable models. Previous methods may ignore abundant spatial features that can be extracted from POIs. In this study, we establish an innovative framework that detects urban land use distributions at the scale of traffic analysis zones (TAZs) by integrating Baidu POIs and a Word2Vec model. This framework was implemented using a Google open-source model of a deep-learning language in 2013. First, data for the Pearl River Delta (PRD) are transformed into a TAZ-POI corpus using a greedy algorithmby considering the spatial distributions of TAZs and inner POIs. Then, high-dimensional characteristic vectors of POIs and TAZs are extracted using the Word2Vec model. Finally, to validate the reliability of the POI/TAZ vectors, we implement a K-Means-based clustering model to analyze correlations between the POI/TAZ vectors and deploy TAZ vectors to identify urban land use types using a random forest algorithm (RFA) model. Compared with some state-of-the-art probabilistic topic models (PTMs), the proposed method can efficiently obtain the highest accuracy (OA = 0.8728, kappa = 0.8399). Moreover, the results can be used to help urban planners to monitor dynamic urban land use and evaluate the impact of urban planning schemes. ARTICLE HISTORY Received 21 March 2016 Accepted 28 September 2016

Data-driven applications are required to adapt to the everincreasing volume, velocity and veracity of data generated by a variety of sources including the Web and Internet of Things devices. At the same time, an event-driven computational paradigm is emerging as the core of modern systems designed for both database queries, data analytics and on-demand applications. MapReduce has been generalized to Map Collective and shown to be very effective in machine learning. However one often uses a dataflow computing model, which has been adopted by most major big data processing runtimes. The HPC community has also developed several asynchronous many tasks (AMT) systems according to the dataflow model. From a different point of view, the services community is moving to an increasingly event-driven model where (micro)services are composed of small functions driven by events in the form of Function as a Service(Faas) and serverless computing. Such designs allow the applications to scale quickly as well as be cost effective in cloud environments. An event-driven runtime designed for data processing consists of well-understood components such as communication, scheduling, and fault tolerance. One can make different design decisions for these components that will determine the type of applications a system can support efficiently. We find that modern systems are designed in a monolithic approach with a fixed set of choices that cannot be changed easily afterwards. Because of these design choices their functionality is limited to specific sets of applications. In this paper we study existing systems (candidate event-driven runtimes), the design choices they have made for each component, and how this affects the type of applications they can support. Further we propose a loosely coupled component-based approach for designing a big data toolkit where each component can have different implementations to support various applications. We believe such a polymorphic design would allow services and data analytics to be integrated seamlessly and expand from edge to cloud to high performance computing environments.

While big data workflows haven been proposed recently as the next-generation data-centric workflow paradigm to process and analyze data of ever increasing in scale, complexity, and rate of acquisition, a scalable distributed data model is still missing that abstracts and automates data distribution, parallelism, and scalable processing. In the meanwhile, although NoSQL has emerged as a new category of data models, they are optimized for storing and querying of large datasets, not for ad-hoc data analysis where data placement and data movement are necessary for optimized workflow execution. In this paper, we propose a NoSQL data model that: 1) supports high-performance MapReduce-style workflows that automate data partitioning and data-parallelism execution. In contrast to the traditional MapReduce framework, our MapReduce-style workflows are fully composable with other workflows enabling dataflow applications with a richer structure, 2) automates virtual machine provisioning and deprovisioning on demand according to the sizes of input datasets, 3) enables a flexible framework for workflow executors that take advantage of the proposed NoSQL data model to improve the performance of workflow execution. Our case studies and experiments show the competitive advantages of our proposed data model. The proposed NoSQL data model is implemented in a new release of DATAVIEW, one of the most usable big data workflow systems in the community.

The concept of using big data (BD) for wireless communication network optimization is no longer new. However, previous work has primarily focused on long-term policies in the network, such as network planning and management. Apart from this, the source of the data collected for analysis/model training is mostly limited to the core network (CN). In this article, we introduce a novel data-driven intelligent radio access network (RAN) architecture that is hierarchical and distributed and operates in real time. We also identify the required data and respective workflows that facilitate intelligent network optimizations. It is our strong belief that the wireless BD (WBD) and machine-learning/artificial-intelligence (AI)-based methodology applies to all layers of the communication system. To demonstrate the superior performance gains of our proposed methodology, two use cases are analyzed with system-level simulations; one is the neural-network-aided optimization for Transmission Control Protocol (TCP), and the other is prediction-based proactive mobility management.

This paper presents an intelligent traffic control system to pass emergency vehicles smoothly. Each individual vehicle is equipped with special radio frequency identification (RFID) tag (placed at a strategic location), which makes it impossible to remove or destroy. We use RFID reader, NSK EDK-125-TTL, and PIC16F877A system-on-chip to read the RFID tags attached to the vehicle. It counts number of vehicles that passes on a particular path during a specified duration. It also determines the network congestion, and hence the green light duration for that path. If the RFID-tag-read belongs to the stolen vehicle, then a message is sent using GSM SIM300 to the police control room. In addition, when an ambulance is approaching the junction, it will communicate to the traffic controller in the junction to turn ON the green light. This module uses ZigBee modules on CC2500 and PIC16F877A system-on-chip for wireless communications between the ambulance and traffic controller. The prototype was tested under different combinations of inputs in our wireless communication laboratory and experimental results were found as expected.


Question: RTIC-C: A big data framework for large-scale traffic information analysis.

This paper delves into the fundamental concepts of algorithms, specifically focusing on the class of problems solvable in polynomial time. To explore this, we first need a concrete understanding of what constitutes a "problem." We define an abstract problem, denoted as Q, as a binary relation between a set I of problem instances and a set S of problem solutions. For instance, consider the SHORTEST-PATH problem: an instance would be a graph along with two vertices, while a solution would be a sequence of vertices that form the shortest path between the two given vertices. This path may not be unique, allowing for multiple solutions to a single problem instance. However, our definition of an abstract problem is more expansive than what is typically required. The theory of NP-completeness, for instance, narrows the focus to decision problems, which have binary solutions (yes or no). We can represent such problems as functions mapping instances from set I to the solution set {0, 1}. Taking the SHORTEST-PATH problem as an example, a related decision problem could be determining whether a path exists between two vertices with a specific number of edges or fewer. If such a path exists, the function returns 1 (yes); otherwise, it returns 0 (no). Beyond decision problems, there are optimization problems that aim to minimize or maximize certain values. Although optimization problems are distinct, they can often be transformed into decision problems of comparable difficulty. This transformation is a common technique in algorithmic analysis. For an in-depth exploration of the Turing-machine model and further algorithmic concepts, refer to Hopcroft and Ullman [156] or Lewis and Papadimitriou [204].

In the context of RTIC-C, a big data system designed for massive traffic information mining, understanding the principles of algorithms, especially those that can be solved in polynomial time, is crucial. The system must efficiently process vast amounts of data, making the choice of algorithms and problem formulation critical to its performance. The transformation of complex optimization problems into decision problems, as discussed earlier, can be particularly beneficial in optimizing traffic flow and information processing in real-time scenarios. The system's ability to quickly determine the existence of certain traffic patterns or conditions, akin to the decision problem examples, can significantly enhance its effectiveness in managing and analyzing traffic data.

The advanced sensing and imaging capability of today's sensor networks enables real time monitoring in a large area. In order to provide continuous monitoring and prompt situational awareness, an abstract-level autonomous information processing framework is developed that is able to detect various categories of abnormal traffic events with unsupervised learning. The framework is based on cogent confabulation model, which performs statistical inference in a manner inspired by human neocortex system. It enables detection and recognition of abnormal target vehicles within the context of surrounding traffic activities and previous events using likelihood-ratio test. A neuromorphic architecture is proposed which accelerates the computation for real-time detection by leveraging memristor crossbar arrays.

Developers use cloud computing platforms to process a large quantity of data in parallel when developing big data analytics. Debugging the massive parallel computations that run in today's datacenters is time consuming and error-prone. To address this challenge, we design a set of interactive, real-time debugging primitives for big data processing in Apache Spark, the next generation data-intensive scalable cloud computing platform. This requires rethinking the notion of step-through debugging in a traditional debugger such as gdb, because pausing the entire computation across distributed worker nodes causes significant delay and naively inspecting millions of records using a watchpoint is too time consuming for an end user.
 First, BigDebug's simulated breakpoints and on-demand watchpoints allow users to selectively examine distributed, intermediate data on the cloud with little overhead. Second, a user can also pinpoint a crash-inducing record and selectively resume relevant sub-computations after a quick fix. Third, a user can determine the root causes of errors (or delays) at the level of individual records through a fine-grained data provenance capability. Our evaluation shows that BigDebug scales to terabytes and its record-level tracing incurs less than 25% overhead on average. It determines crash culprits orders of magnitude more accurately and provides up to 100% time saving compared to the baseline replay debugger. The results show that BigDebug supports debugging at interactive speeds with minimal performance impact.

Big data, which is maturing over time, has a high impact in analyzing scenarios and coming up with decision making strategies pertaining to any sector. Consequently the number of applications are also increasing where in the past, they were limited to the confines of company sales departments. In lieu, this paper is primarily focused on applying Data Sciences within the education sector. We will be discussing the overall application to address higher-education issues. We will also present a framework for capturing, analyzing it and concluding on data in order to benefit decision-makers, thus ultimately manifesting in informing design and finalizing policy.

0968-090X/$ see front matter 2011 Published b doi:10.1016/j.trc.2011.12.004 ⇑ Corresponding author. Tel.: +44 20 7679 7214. E-mail address: n.lathia@cs.ucl.ac.uk (N. Lathia). The increasing availability of sensor data in urban areas now offers the opportunity to perform continuous evaluations of transport systems and measure the effects of policy changes, in an empirical, large-scale, and non-invasive way. In this paper, we study one such example: the effect of changing the user-access policy in the London Barclays Cycle Hire scheme. When the scheme was launched in July 2010, users were required to apply for a key to access to the system. By December 2010, this policy was overridden in order to allow for ‘‘casual’’ usage, so that anyone in possession of a debit or credit card could gain access. While the transport authority measured the policy shift’s success by the increased number of trips, we set out to investigate how the change affected the system’s usage throughout the city. We present an extensive analysis of station data collected from the scheme’s web site both preand post-policy change, showing how differences in both global and local behaviour can be measured, and how the policy change correlates with a variety of effects observed around the city. We find that, as expected, quicker access to the system correlates with greater week end usage; it also reinforces the week-day commuting trend. In both the preand post-change periods, the geographic distribution of activity at individual stations forms concentric circles around central London. However, upon policy change, a number of stations undergo a complete usage change, now exhibiting an opposite trend with respect to that which they had prior to the policy change. 2011 Published by Elsevier Ltd.

Non-intrusive video-detection for traffic flow observation and surveillance is the primary alternative to conventional inductive loop detectors. Video Image Detection Systems (VIDS) can derive traffic parameters by means of image processing and pattern recognition methods. Existing VIDS emulate the inductive loops. We propose a trajectory based recognition algorithm to expand the common approach and to obtain new types of information (e.g. queue length or erratic movements). Different views of the same area by more than one camera sensor is necessary, because of the typical limitations of single camera systems, resulting from the occlusion effect of other cars, trees and traffic signs. A distributed cooperative multi-camera system enables a significant enlargement of the observation area. The trajectories are derived from multi-target tracking. The fusion of object data from different cameras is done using a tracking method. This approach opens up opportunities to identify and specify traffic objects, their location, speed and other characteristic object information. The system provides new derived and consolidated information about traffic participants. Thus, this approach is also beneficial for a description of individual traffic participants.

In modern cities, more and more vehicles, such as taxis, have been equipped with GPS devices for localization and navigation. Gathering and analyzing these large-scale real-world digital traces have provided us an unprecedented opportunity to understand the city dynamics and reveal the hidden social and economic “realities”. One innovative pervasive application is to provide correct driving strategies to taxi drivers according to time and location. In this paper, we aim to discover both efficient and inefficient passenger-finding strategies from a large-scale taxi GPS dataset, which was collected from 5350 taxis for one year in a large city of China. By representing the passenger-finding strategies in a Time-Location-Strategy feature triplet and constructing a train/test dataset containing both top- and ordinary-performance taxi features, we adopt a powerful feature selection tool, L1-Norm SVM, to select the most salient feature patterns determining the taxi performance. We find that the selected patterns can well interpret the empirical study results derived from raw data analysis and even reveal interesting hidden “facts”. Moreover, the taxi performance predictor built on the selected features can achieve a prediction accuracy of 85.3% on a new test dataset, and it also outperforms the one based on all the features, which implies that the selected features are indeed the right indicators of the passenger-finding strategies.

Big Data Analysis (BDA) has attracted considerable interest and curiosity from scientists of various fields recently. As big size and complexity of big data, it is pivotal to uncover hidden patterns, bursts of activity, correlations and laws of it. Complex network analysis could be effective method for this purpose, because of its powerful data organization and visualization ability. Besides the general complex software (Gephi), some bibliometrics software (Citespace and SCI2) could also be used for this analysis, due to their powerful data process and visualization functions. This paper presents a comparison of Citespace, SCI2 and Gephi from the following three aspects: (1) Data preprocessing. Citespace is time-consuming and laborious on merging synonyms. SCI2 is efficient and suitable for massive data. Both of them could remove duplicate records. However, Gephi lacks these. (2) Network extraction. Citespace pays more attention on temporal analysis, but SCI2 and Gephi pay less attention on it. Besides, Citespace and SCI2 could provide pruning algorithms highlight the main structure of social network, but Gephi lacks this. (3) Visualization. In Citespace, co-occurrence network could present time, frequency and betweenness centrality simultaneously; cluster view could label the clusters with phrases. However, SCI2 and Gephi provide more various layout algorithms to present network. Besides, they have a better edit-ability than Citespace on generated network.

Unmanned Aerial Vehicles (UAVs) are becoming an attractive solution for road traffic monitoring because of their mobility, low cost, and broad view range. Up to now, existing traffic monitoring systems based on UAVs only use one UAV with fixed trajectory to extract information about vehicles. In this paper, we propose a road traffic monitoring system using multiple UAVs. We develop a method to generate adaptive UAVs trajectories, which is based on the tracking of moving points in the UAV field of view. Also we generate UAVs trajectories using mobility models that are usually used to model vehicles mobility. UAVs monitor the traffic on a city road, they are responsible for collecting and sending, in real time, vehicle information to a traffic processing center for traffic regulation purposes. We show that the performance of our system is better than the performance of the fixed UAV trajectory traffic monitoring system in terms of coverage rates and events detection rates.

Data-driven applications are required to adapt to the everincreasing volume, velocity and veracity of data generated by a variety of sources including the Web and Internet of Things devices. At the same time, an event-driven computational paradigm is emerging as the core of modern systems designed for both database queries, data analytics and on-demand applications. MapReduce has been generalized to Map Collective and shown to be very effective in machine learning. However one often uses a dataflow computing model, which has been adopted by most major big data processing runtimes. The HPC community has also developed several asynchronous many tasks (AMT) systems according to the dataflow model. From a different point of view, the services community is moving to an increasingly event-driven model where (micro)services are composed of small functions driven by events in the form of Function as a Service(Faas) and serverless computing. Such designs allow the applications to scale quickly as well as be cost effective in cloud environments. An event-driven runtime designed for data processing consists of well-understood components such as communication, scheduling, and fault tolerance. One can make different design decisions for these components that will determine the type of applications a system can support efficiently. We find that modern systems are designed in a monolithic approach with a fixed set of choices that cannot be changed easily afterwards. Because of these design choices their functionality is limited to specific sets of applications. In this paper we study existing systems (candidate event-driven runtimes), the design choices they have made for each component, and how this affects the type of applications they can support. Further we propose a loosely coupled component-based approach for designing a big data toolkit where each component can have different implementations to support various applications. We believe such a polymorphic design would allow services and data analytics to be integrated seamlessly and expand from edge to cloud to high performance computing environments.

a Department of Computer Science and Technology, Tongji University, Shanghai 201804, PR China b Department of Electrical and Computer Engineering, University of Alberta, Edmonton, AB T6G 2G7, Canada c Key Laboratory of Embedded System and Service Computing, Ministry of Education, Tongji University, Shanghai 201804, PR China d System Research Institute, Polish Academy of Sciences, Warsaw, Poland e School of Software, Jiangxi Agricultural University, Nanchang 330013, PR China


Question: RTIC-C: A big data framework for extensive traffic information extraction.

This paper introduces the Branch-and-Bound Skyline (BBS) algorithm, an innovative approach for efficient computation of skylines in d-dimensional datasets. The skyline, comprising points not dominated by any other on all dimensions, has gained significant interest in the database community, particularly for progressive methods that swiftly yield initial results without the need to scan the entire database. Existing algorithms, despite their merits, have notable limitations that hinder their practical application. BBS, rooted in nearest-neighbor search principles, stands out for its I/O optimality, accessing only nodes that potentially harbor skyline points, thus minimizing data access overhead. The algorithm's simplicity and versatility make it adaptable to various progressive processing scenarios, including user preference handling and support for arbitrary dimensionality. Moreover, the paper explores intriguing variations of skyline computation, illustrating BBS's applicability in enhancing processing efficiency for these extended scenarios.

**Question**: RTIC-C: A Big Data System for Massive Traffic Information Mining

**Documents**: A Scalable System for Real-Time Traffic Information Collection and Analysis

In the era of big data, managing and analyzing massive traffic information poses significant challenges due to the sheer volume and velocity of data generated. This paper presents RTIC-C, a scalable and robust system designed for real-time traffic information collection and analysis. RTIC-C leverages advanced data processing techniques and machine learning algorithms to handle the high throughput of data, ensuring timely and accurate traffic information for various stakeholders, including commuters, transportation authorities, and urban planners. The system's architecture is built to support high scalability and reliability, making it suitable for deployment in large-scale urban environments. RTIC-C's effectiveness is demonstrated through case studies and performance evaluations, highlighting its capability to process big data and deliver actionable insights for traffic management and planning.

A wide variety of applications for road safety and traffic efficiency are intended to answer the urgent call for smarter, greener, and safer mobility. Although IEEE 802.11p is considered the de facto standard for on-the-road communications, stakeholders have recently started to investigate the usability of LTE to support vehicular applications. In this article, related work and running standardization activities are scanned and critically discussed; strengths and weaknesses of LTE as an enabling technology for vehicular communications are analyzed; and open issues and critical design choices are highlighted to serve as guidelines for future research in this hot topic.

As mobile devices with positioning capabilities continue to proliferate, data management for so-called trajectory databases that capture the historical movements of populations of moving objects becomes important. This paper considers the querying of such databases for convoys, a convoy being a group of objects that have traveled together for some time. More specifically, this paper formalizes the concept of a convoy query using density-based notions, in order to capture groups of arbitrary extents and shapes. Convoy discovery is relevant for reallife applications in throughput planning of trucks and carpooling of vehicles. Although there has been extensive research on trajectories in the literature, none of this can be applied to retrieve correctly exact convoy result sets. Motivated by this, we develop three efficient algorithms for convoy discovery that adopt the wellknown filter-refinement framework. In the filter step, we apply linesimplification techniques on the trajectories and establish distance bounds between the simplified trajectories. This permits efficient convoy discovery over the simplified trajectories without missing any actual convoys. In the refinement step, the candidate convoys are further processed to obtain the actual convoys. Our comprehensive empirical study offers insight into the properties of the paper’s proposals and demonstrates that the proposals are effective and efficient on real-world trajectory data.

Data-driven applications are required to adapt to the everincreasing volume, velocity and veracity of data generated by a variety of sources including the Web and Internet of Things devices. At the same time, an event-driven computational paradigm is emerging as the core of modern systems designed for both database queries, data analytics and on-demand applications. MapReduce has been generalized to Map Collective and shown to be very effective in machine learning. However one often uses a dataflow computing model, which has been adopted by most major big data processing runtimes. The HPC community has also developed several asynchronous many tasks (AMT) systems according to the dataflow model. From a different point of view, the services community is moving to an increasingly event-driven model where (micro)services are composed of small functions driven by events in the form of Function as a Service(Faas) and serverless computing. Such designs allow the applications to scale quickly as well as be cost effective in cloud environments. An event-driven runtime designed for data processing consists of well-understood components such as communication, scheduling, and fault tolerance. One can make different design decisions for these components that will determine the type of applications a system can support efficiently. We find that modern systems are designed in a monolithic approach with a fixed set of choices that cannot be changed easily afterwards. Because of these design choices their functionality is limited to specific sets of applications. In this paper we study existing systems (candidate event-driven runtimes), the design choices they have made for each component, and how this affects the type of applications they can support. Further we propose a loosely coupled component-based approach for designing a big data toolkit where each component can have different implementations to support various applications. We believe such a polymorphic design would allow services and data analytics to be integrated seamlessly and expand from edge to cloud to high performance computing environments.

Intelligent Transportation Systems (ITS) have attracted the attention of researchers and the general public alike as a means to alleviate traffic congestion. Recently, the maturity of wireless technology has enabled a cost-efficient way to achieve ITS by detecting vehicles using Vehicle to Infrastructure (V2I) communications. Traditional ITS algorithms, in most cases, assume that every vehicle is observed, such as by a camera or a loop detector, but a V2I implementation would detect only those vehicles with wireless communications capability. We examine a family of transportation systems, which we will refer to as ‘Partially Detected Intelligent Transportation Systems’. An algorithm that can act well under a small detection rate is highly desirable due to gradual penetration rates of the underlying wireless technologies such as Dedicated Short Range Communications (DSRC) technology. Artificial Intelligence (AI) techniques for Reinforcement Learning (RL) are suitable tools for finding such an algorithm due to utilizing varied inputs and not requiring explicit analytic understanding or modeling of the underlying system dynamics. In this paper, we report a RL algorithm for partially observable ITS based on DSRC. The performance of this system is studied under different car flows, detection rates, and topologies of the road network. Our system is able to efficiently reduce the average waiting time of vehicles at an intersection, even with a low detection rate.

Big Data Analysis (BDA) has attracted considerable interest and curiosity from scientists of various fields recently. As big size and complexity of big data, it is pivotal to uncover hidden patterns, bursts of activity, correlations and laws of it. Complex network analysis could be effective method for this purpose, because of its powerful data organization and visualization ability. Besides the general complex software (Gephi), some bibliometrics software (Citespace and SCI2) could also be used for this analysis, due to their powerful data process and visualization functions. This paper presents a comparison of Citespace, SCI2 and Gephi from the following three aspects: (1) Data preprocessing. Citespace is time-consuming and laborious on merging synonyms. SCI2 is efficient and suitable for massive data. Both of them could remove duplicate records. However, Gephi lacks these. (2) Network extraction. Citespace pays more attention on temporal analysis, but SCI2 and Gephi pay less attention on it. Besides, Citespace and SCI2 could provide pruning algorithms highlight the main structure of social network, but Gephi lacks this. (3) Visualization. In Citespace, co-occurrence network could present time, frequency and betweenness centrality simultaneously; cluster view could label the clusters with phrases. However, SCI2 and Gephi provide more various layout algorithms to present network. Besides, they have a better edit-ability than Citespace on generated network.

With the rise of location-aware IoT devices, there is an increased desire to process and manage the stationary and moving trajectory data generated by these real-time sensors. There has been a corresponding evolution of distributed database and compute technology to handle the increasing data load. Here we describe challenges in managing this kind of data and survey the technologies that address those challenges. We evaluate two distributed database technologies against a simulated high volume sensor data source and discuss the performance of predominant moving trajectory access patterns.

Unbounded, unordered, global-scale datasets are increasingly common in day-to-day business (e.g. Web logs, mobile usage statistics, and sensor networks). At the same time, consumers of these datasets have evolved sophisticated requirements, such as event-time ordering and windowing by features of the data themselves, in addition to an insatiable hunger for faster answers. Meanwhile, practicality dictates that one can never fully optimize along all dimensions of correctness, latency, and cost for these types of input. As a result, data processing practitioners are left with the quandary of how to reconcile the tensions between these seemingly competing propositions, often resulting in disparate implementations and systems. We propose that a fundamental shift of approach is necessary to deal with these evolved requirements in modern data processing. We as a field must stop trying to groom unbounded datasets into finite pools of information that eventually become complete, and instead live and breathe under the assumption that we will never know if or when we have seen all of our data, only that new data will arrive, old data may be retracted, and the only way to make this problem tractable is via principled abstractions that allow the practitioner the choice of appropriate tradeoffs along the axes of interest: correctness, latency, and cost. In this paper, we present one such approach, the Dataflow Model, along with a detailed examination of the semantics it enables, an overview of the core principles that guided its design, and a validation of the model itself via the real-world experiences that led to its development. We use the term “Dataflow Model” to describe the processing model of Google Cloud Dataflow [20], which is based upon technology from FlumeJava [12] and MillWheel [2]. This work is licensed under the Creative Commons AttributionNonCommercial-NoDerivs 3.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/. Obtain permission prior to any use beyond those covered by the license. Contact copyright holder by emailing info@vldb.org. Articles from this volume were invited to present their results at the 41st International Conference on Very Large Data Bases, August 31st September 4th 2015, Kohala Coast, Hawaii. Proceedings of the VLDB Endowment, Vol. 8, No. 12 Copyright 2015 VLDB Endowment 2150-8097/15/08.

In many graph applications, the structure of the graph changes dynamically over time and may require real time analysis. However, constructing a large graph is expensive, and most studies for large graphs have not focused on a dynamic graph data structure, but rather a static one. To address this issue, we propose DegAwareRHH, a high performance dynamic graph data store designed for scaling out to store large, scale-free graphs by leveraging compact hash tables with high data locality. We extend DegAwareRHH for multiple processes and distributed memory, and perform dynamic graph construction on large scale-free graphs using emerging 'Big Data HPC' systems such as the Catalyst cluster at LLNL. We demonstrate that DegAwareRHH processes a request stream 206.5x faster than a state-of-the-art shared-memory dynamic graph processing framework, when both implementations use 24 threads/processes to construct a graph with 1 billion edge insertion requests and 54 million edge deletion requests. DegAwareRHH also achieves a processing rate of over 2 billion edge insertion requests per second using 128 compute nodes to construct a large-scale web graph, containing 128 billion edges, the largest open-source real graph dataset to our knowledge.

Intelligent Transportation Systems (ITS) enable new services within Smart Cities. Efficient Waste Collection is considered a fundamental service for Smart Cities. Internet of Things (IoT) can be applied both in ITS and Smart cities forming an advanced platform for novel applications. Surveillance systems can be used as an assistive technology for high Quality of Service (QoS) in waste collection. Specifically, IoT components: (i) RFIDs, (ii) sensors, (iii) cameras, and (iv) actuators are incorporated into ITS and surveillance systems for efficient waste collection. In this paper we propose an advanced Decision Support System (DSS) for efficient waste collection in Smart Cities. The system incorporates a model for data sharing between truck drivers on real time in order to perform waste collection and dynamic route optimization. The system handles the case of ineffective waste collection in inaccessible areas within the Smart City. Surveillance cameras are incorporated for capturing the problematic areas and provide evidence to the authorities. The waste collection system aims to provide high quality of service to the citizens of a Smart City.

Video-based traffic sign detection, tracking, and recognition is one of the important components for the intelligent transport systems. Extensive research has shown that pretty good performance can be obtained on public data sets by various state-of-the-art approaches, especially the deep learning methods. However, deep learning methods require extensive computing resources. In addition, these approaches mostly concentrate on single image detection and recognition task, which is not applicable in real-world applications. Different from previous research, we introduce a unified incremental computational framework for traffic sign detection, tracking, and recognition task using the mono-camera mounted on a moving vehicle under non-stationary environments. The main contributions of this paper are threefold: 1) to enhance detection performance by utilizing the contextual information, this paper innovatively utilizes the spatial distribution prior of the traffic signs; 2) to improve the tracking performance and localization accuracy under non-stationary environments, a new efficient incremental framework containing off-line detector, online detector, and motion model predictor together is designed for traffic sign detection and tracking simultaneously; and 3) to get a more stable classification output, a scale-based intra-frame fusion method is proposed. We evaluate our method on two public data sets and the performance has shown that the proposed system can obtain results comparable with the deep learning method with less computing resource in a near-real-time manner.


Question: RTIC-C: A big data framework for extensive traffic information extraction.

This paper delves into the analysis of trade-offs between various computational aspects in hash coding, focusing on the problem of sequentially checking messages for membership in a predetermined set. Two innovative hash coding techniques are introduced and contrasted with a conventional method. The computational elements under scrutiny include the hash area's size (space), the time needed to ascertain that a message is not part of the given set (reject time), and an acceptable error rate.
The primary goal of the new methods is to minimize the space required for hash-coded information, which is typically larger in conventional approaches. This reduction in space is achieved by acknowledging that a negligible error rate might be acceptable in specific scenarios, particularly those involving vast amounts of data where a core-resident hash area is impractical using conventional techniques.
In scenarios with extensive data, performance can be enhanced by employing a smaller core-resident hash area in conjunction with the novel methods. When needed, a secondary, albeit time-consuming, test can be used to correct the minor errors introduced by the new methods. An illustrative example is provided to highlight potential domains where these techniques could be beneficial.
The analysis of the fundamental problem reveals that permitting a small number of messages to be incorrectly identified as part of the given set enables the use of a significantly smaller hash area without increasing reject time. This approach is particularly advantageous in big data systems, such as traffic information mining, where space and time efficiency are critical.

Increasing number of social media content shared across the globe in real time creates a fascinating area of research. For most events, social media users act as collective sensors by sharing important information about events of any scale. Due to its real time nature, social media is much quicker to respond to such events relative to conventional news media. This paper proposes an event detection system which provides 5W1H (What, Where, When, Why, Who, How) analysis for each detected event. We make use of a myriad of techniques such as anomaly detection, named entity recognition, automatic summary generation, user link analysis. Our experimental results for the system indicate a faster event detection performance compared to conventional news sources. Event analysis results are also in line with the corresponding news articles about detected events.

Intelligent Transportation Systems (ITS) enable new services within Smart Cities. Efficient Waste Collection is considered a fundamental service for Smart Cities. Internet of Things (IoT) can be applied both in ITS and Smart cities forming an advanced platform for novel applications. Surveillance systems can be used as an assistive technology for high Quality of Service (QoS) in waste collection. Specifically, IoT components: (i) RFIDs, (ii) sensors, (iii) cameras, and (iv) actuators are incorporated into ITS and surveillance systems for efficient waste collection. In this paper we propose an advanced Decision Support System (DSS) for efficient waste collection in Smart Cities. The system incorporates a model for data sharing between truck drivers on real time in order to perform waste collection and dynamic route optimization. The system handles the case of ineffective waste collection in inaccessible areas within the Smart City. Surveillance cameras are incorporated for capturing the problematic areas and provide evidence to the authorities. The waste collection system aims to provide high quality of service to the citizens of a Smart City.

This paper presents the City Browser, a tool developed to analyze the complexities underlying human mobility at the city scale. The tool uses data generated from mobile phones as a proxy to provide several insights with regards to the commuting patterns of the population within the bounds of a city. The three major components of the browser are the data warehouse, modules and algorithm, and the visualization interface. The modules and algorithm component utilizes Call Detail Records (CDRs) stored within the data warehouse to infer mobility patterns that are then communicated through the visualization interface. The modules and algorithm component consists of four modules: the spatial-temporal decomposition module, the home/work capturing module, the community detection module, and the flow estimation module. The visualization interface manages the output of each module to provide a comprehensive view of a city’s mobility dynamics over varying time scales. A case study is presented on the city of Riyadh in Saudi Arabia, where the browser was developed to better understand city mobility patterns.

People flow at a citywide level is in a mixed state with several basic patterns (e.g. commuting, working, commercial), and it is therefore difficult to extract useful information from such a mixture of patterns directly. In this paper, we proposed a novel tensor factorization approach to modeling city dynamics in a basic life pattern space (CitySpectral Space). To obtain the CitySpectrum, we utilized Non-negative Tensor Factorization (NTF) to decompose a people flow tensor into basic life pattern tensors, described by three bases i.e. the intensity variation among different regions, the time-of-day and the sample days. We apply our approach to a big mobile phone GPS log dataset (containing 1.6 million users) to model the fluctuation in people flow before and after the Great East Japan Earthquake from a CitySpectral perspective. In addition, our framework is extensible to a variety of auxiliary spatial-temporal data. We parametrize a people flow with a spatial distribution of the Points of Interest (POIs) to quantitatively analyze the relationship between human mobility and POI distribution. Based on the parametric people flow, we propose a spectral approach for a site-selection recommendation and people flow simulation in another similar area using POI distribution.

Identifying trustworthy information in the presence of noisy data contributed by numerous unvetted sources from online social media (e.g., Twitter, Facebook, and Instagram) has been a crucial task in the era of big data. This task, referred to as truth discovery, targets at identifying the reliability of the sources and the truthfulness of claims they make without knowing either a priori. In this work, we identified three important challenges that have not been well addressed in the current truth discovery literature. The first one is “misinformation spread” where a significant number of sources are contributing to false claims, making the identification of truthful claims difficult. For example, on Twitter, rumors, scams, and influence bots are common examples of sources colluding, either intentionally or unintentionally, to spread misinformation and obscure the truth. The second challenge is “data sparsity” or the “long-tail phenomenon” where a majority of sources only contribute a small number of claims, providing insufficient evidence to determine those sources’ trustworthiness. For example, in the Twitter datasets that we collected during real-world events, more than 90% of sources only contributed to a single claim. Third, many current solutions are not scalable to large-scale social sensing events because of the centralized nature of their truth discovery algorithms. In this paper, we develop a Scalable and Robust Truth Discovery (SRTD) scheme to address the above three challenges. In particular, the SRTD scheme jointly quantifies both the reliability of sources and the credibility of claims using a principled approach. We further develop a distributed framework to implement the proposed truth discovery scheme using Work Queue in an HTCondor system. The evaluation results on three real-world datasets show that the SRTD scheme significantly outperforms the state-of-the-art truth discovery methods in terms of both effectiveness and efficiency.

Available online 18 July 2010 Information overload is a severe problem for human operators of large-scale control systems as, for example, encountered in the domain of road traffic management. Operators of such systems are at risk to lack situation awareness, because existing systems focus on themere presentation of the available information on graphical user interfaces—thus endangering the timely and correct identification, resolution, and prevention of critical situations. In recent years, ontology-based approaches to situation awareness featuring a semantically richer knowledge model have emerged. However, current approaches are either highly domain-specific or have, in case they are domain-independent, shortcomings regarding their reusability. In this paper, we present our experience gained from the development of BeAware!, a framework for ontology-driven information systems aiming at increasing anoperator's situation awareness. In contrast to existing domain-independent approaches, BeAware!'s ontology introduces the concept of spatio-temporal primitive relations betweenobserved real-world objects thereby improving the reusability of the framework. To show its applicability, a prototype of BeAware! has been implemented in thedomain of road trafficmanagement. An overviewof this prototype and lessons learned for the development of ontology-driven information systems complete our contribution. © 2010 Elsevier B.V. All rights reserved.

Today, we live in a digital universe in which information and technology are not only around us but also play important roles in dictating the quality of our lives. As we delve deeper into this digital universe, we're witnessing explosive growth in the variety, velocity, and volume of data1,2 being transmitted over the Internet. A zetabyte of data passed through the Internet in the past year; IDC predicts that this digital universe will explode to an unimaginable eight Zbytes by 2015. These data are and will be generated mainly from Internet search, social media, mobile devices, the Internet of Things, business transactions, next-generation radio astronomy telescopes, high-energy physics synchrotron, and content distribution. Government and business organizations are now overflowing with data, easily aggregating to terabytes or even petabytes of information.

In modern cities, more and more vehicles, such as taxis, have been equipped with GPS devices for localization and navigation. Gathering and analyzing these large-scale real-world digital traces have provided us an unprecedented opportunity to understand the city dynamics and reveal the hidden social and economic “realities”. One innovative pervasive application is to provide correct driving strategies to taxi drivers according to time and location. In this paper, we aim to discover both efficient and inefficient passenger-finding strategies from a large-scale taxi GPS dataset, which was collected from 5350 taxis for one year in a large city of China. By representing the passenger-finding strategies in a Time-Location-Strategy feature triplet and constructing a train/test dataset containing both top- and ordinary-performance taxi features, we adopt a powerful feature selection tool, L1-Norm SVM, to select the most salient feature patterns determining the taxi performance. We find that the selected patterns can well interpret the empirical study results derived from raw data analysis and even reveal interesting hidden “facts”. Moreover, the taxi performance predictor built on the selected features can achieve a prediction accuracy of 85.3% on a new test dataset, and it also outperforms the one based on all the features, which implies that the selected features are indeed the right indicators of the passenger-finding strategies.

In recent years, mobile devices (e.g., smartphones and tablets) have met an increasing commercial success and have become a fundamental element of the everyday life for billions of people all around the world. Mobile devices are used not only for traditional communication activities (e.g., voice calls and messages) but also for more advanced tasks made possible by an enormous amount of multi-purpose applications (e.g., finance, gaming, and shopping). As a result, those devices generate a significant network traffic (a consistent part of the overall Internet traffic). For this reason, the research community has been investigating security and privacy issues that are related to the network traffic generated by mobile devices, which could be analyzed to obtain information useful for a variety of goals (ranging from fine-grained user profiling to device security and network optimization). In this paper, we review the works that contributed to the state of the art of network traffic analysis targeting mobile devices. In particular, we present a systematic classification of the works in the literature according to three criteria: 1) the goal of the analysis; 2) the point where the network traffic is captured; and 3) the targeted mobile platforms. In this survey, we consider points of capturing such as Wi-Fi access points, software simulation, and inside real mobile devices or emulators. For the surveyed works, we review and compare analysis techniques, validation methods, and achieved results. We also discuss possible countermeasures, challenges, and possible directions for future research on mobile traffic analysis and other emerging domains (e.g., Internet of Things). We believe our survey will be a reference work for researchers and practitioners in this research field.

The growing number of remote sensing systems and ship reporting technologies (e.g. Automatic Identification System, Long Range Identification and Tracking, radar tracking, Earth Observation) are generating an overwhelming amount of spatio-temporal and geographically distributed data related to vessels and their movements. Research on reliable data mining techniques has proven essential to the discovery of knowledge from such increasingly available information on ship traffic at sea. Data driven knowledge discovery has very recently demonstrated its value in fields that go beyond the original maritime safety and security remits of such data. They include, but are not limited to, fisheries management, maritime spatial planning, gridding ship emissions, mapping activities at sea, risk assessment of offshore platforms, and trade indicators. The extraction of useful information from maritime Big Data is thus a key element in providing operational authorities, policy-makers and scientists with supporting tools to understand what is happening at sea and improve maritime knowledge. This work will provide a survey of the recent JRC research activities relevant to automatic anomaly detection and knowledge discovery in the maritime domain. Data mining, data analytics and predictive analysis examples are introduced using real data. In addition, this paper presents approaches to detect anomalies in reporting messages and unexpected behaviours at sea.


Question: RTIC-C: A big data platform for extensive traffic information extraction.

This paper advocates for the integration of a novel Skyline operation into database systems to facilitate the extraction of significant data points from vast datasets. A data point is deemed significant if no other point outperforms it in all dimensions. For instance, a hotel in Nassau might be an appealing choice for a traveler if there isn't another hotel that is both less expensive and closer to the beach. The paper explores how SQL can be enhanced to support Skyline queries, presents and evaluates various algorithms for executing the Skyline operation, and illustrates how this operation can be synergized with other database functions, such as joins.

In the context of RTIC-C, a system designed for the extensive mining of traffic information from big data, the concept of the Skyline operation could be particularly valuable. By applying the Skyline operator, the system could efficiently filter out the most relevant and non-dominated data points from the massive traffic dataset. This could aid in identifying critical traffic patterns, optimal routes, and significant anomalies without being overwhelmed by less relevant data. The Skyline operation, when combined with other database operations within RTIC-C, could enhance the system's capability to process and analyze big data, making it more effective in extracting meaningful insights from traffic information.

In many graph applications, the structure of the graph changes dynamically over time and may require real time analysis. However, constructing a large graph is expensive, and most studies for large graphs have not focused on a dynamic graph data structure, but rather a static one. To address this issue, we propose DegAwareRHH, a high performance dynamic graph data store designed for scaling out to store large, scale-free graphs by leveraging compact hash tables with high data locality. We extend DegAwareRHH for multiple processes and distributed memory, and perform dynamic graph construction on large scale-free graphs using emerging 'Big Data HPC' systems such as the Catalyst cluster at LLNL. We demonstrate that DegAwareRHH processes a request stream 206.5x faster than a state-of-the-art shared-memory dynamic graph processing framework, when both implementations use 24 threads/processes to construct a graph with 1 billion edge insertion requests and 54 million edge deletion requests. DegAwareRHH also achieves a processing rate of over 2 billion edge insertion requests per second using 128 compute nodes to construct a large-scale web graph, containing 128 billion edges, the largest open-source real graph dataset to our knowledge.

Updating road markings is one of the routine tasks of transportation agencies. Compared with traditional road inventory mapping techniques, vehicle-borne mobile light detection and ranging (LiDAR) systems can undertake the job safely and efficiently. However, current hurdles include software and computing challenges when handling huge volumes of highly dense and irregularly distributed 3-D mobile LiDAR point clouds. This paper presents the development and implementation aspects of an automated object extraction strategy for rapid and accurate road marking inventory. The proposed road marking extraction method is based on 2-D georeferenced feature (GRF) images, which are interpolated from 3-D road surface points through a modified inverse distance weighted (IDW) interpolation. Weighted neighboring difference histogram (WNDH)-based dynamic thresholding and multiscale tensor voting (MSTV) are proposed to segment and extract road markings from the noisy corrupted GRF images. The results obtained using 3-D point clouds acquired by a RIEGL VMX-450 mobile LiDAR system in a subtropical urban environment are encouraging.

Purpose – Intensive traffic often occurs in web-enabled business processes hosted by travel industry and government portals. An extreme case for intensive traffic is flash crowd situations when the number of web users spike within a short time due to unexpected events caused by political unrest or extreme weather conditions. As a result, the servers hosting these business processes can no longer handle overwhelming service requests. To alleviate this problem, process engineers usually analyze audit trail data collected from the application server and reengineer their business processes to withstand unexpected surge in the visitors. However, such analysis can only reveal the performance of the application server from the internal perspective. This paper aims to investigate this issue. Design/methodology/approach – This paper proposes an approach for analyzing key performance indicators of traffic intensive web-enabled business processes from audit trail data, web server logs, and stress testing logs. Findings – The key performance indicators identified in the study’s approach can be used to understand the behavior of traffic intensive web-enabled business processes and the underlying factors that affect the stability of the web server. Originality/value – The proposed analysis also provides an internal as well as an external view of the performance. Moreover, the calculated key performance indicators can be used by the process engineers for locating potential bottlenecks, reengineering business processes, and implementing contingency measures for traffic intensive situations.

Developers use cloud computing platforms to process a large quantity of data in parallel when developing big data analytics. Debugging the massive parallel computations that run in today's datacenters is time consuming and error-prone. To address this challenge, we design a set of interactive, real-time debugging primitives for big data processing in Apache Spark, the next generation data-intensive scalable cloud computing platform. This requires rethinking the notion of step-through debugging in a traditional debugger such as gdb, because pausing the entire computation across distributed worker nodes causes significant delay and naively inspecting millions of records using a watchpoint is too time consuming for an end user.
 First, BigDebug's simulated breakpoints and on-demand watchpoints allow users to selectively examine distributed, intermediate data on the cloud with little overhead. Second, a user can also pinpoint a crash-inducing record and selectively resume relevant sub-computations after a quick fix. Third, a user can determine the root causes of errors (or delays) at the level of individual records through a fine-grained data provenance capability. Our evaluation shows that BigDebug scales to terabytes and its record-level tracing incurs less than 25% overhead on average. It determines crash culprits orders of magnitude more accurately and provides up to 100% time saving compared to the baseline replay debugger. The results show that BigDebug supports debugging at interactive speeds with minimal performance impact.

Social media play a decisive role in communicating and spreading information during global events. In particular, real-time microblogging platforms such as Twitter have become prevalent. Researchers have used microblogging for a number of tasks, including past events analysis, predictions, and information retrieval. Nevertheless, little attention has been given to quantitative data extraction. In this paper, we address two questions: can we develop a mechanism to extract quantitative data from a collection of tweets, and can we use the salient findings to describe an event? To answer the first question, we introduce Raimond, a virtual text curator, specialized in quantitative data extraction from Twitter. To address the second question, we use our system on three events and evaluate its output using a crowdsourcing strategy. We demonstrate the e↵ectiveness of our approach with a number of real world examples.

Wireless vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communication holds great promise for significantly reducing the human and financial costs of vehicle collisions. A common characteristic of this communication is the broadcast of a device's core state information at regular intervals (e.g. a vehicle's speed and location, or a traffic signal's state and timing). The aggregate of these uncoordinated broadcasts will lead to channel congestion under dense traffic scenarios, with a resulting reduction in the effectiveness of the collision avoidance applications making use of the transmitted information. Active congestion control using distributed techniques is a topic of great interest for establishing the scalability of this technology for deployment. This paper defines a new congestion control algorithm that can be applied to the message rate of devices in this vehicular environment. While other published approaches rely on binary control, the LInear MEssage Rate Integrated Control (LIMERIC) algorithm takes advantage of full precision control inputs that are available on the wireless channel. The result is provable convergence to fair and efficient channel utilization in the deterministic environment, under simple criteria for setting adaptive parameters. This "perfect" convergence avoids the limit cycle behavior inherent to binary control. We also discuss several practical aspects associated with implementing LIMERIC, including: guidelines for the choice of system parameters to obtain desired utilization outcomes, a gain saturation technique that maintains robust stability under all conditions, convergence with asynchronous updates, and the implications of measurement noise for statistical properties of convergence. The paper illustrates key analytical results using MATLAB numerical results, and employs standard NS-2 simulations to demonstrate the performance of LIMERIC in several high density scenarios.

High speed computer and telephone networks carry large amounts of data and signalling traffic. The engineers who build and maintain these networks use a combination of hardware and software tools to monitor the stream of network traffic. Some of these tools operate directly on the live network; others record data on magnetic tape for later offline analysis by software. Most analysis tasks require tens to hundreds of gigabytes of data. Traffic analysis applications include protocol performance analysis, conformance testing, error monitoring and fraud detection.

Twitter, a popular microblogging service, has received much attention recently. An important characteristic of Twitter is its real-time nature. For example, when an earthquake occurs, people make many Twitter posts (tweets) related to the earthquake, which enables detection of earthquake occurrence promptly, simply by observing the tweets. As described in this paper, we investigate the real-time interaction of events such as earthquakes in Twitter and propose an algorithm to monitor tweets and to detect a target event. To detect a target event, we devise a classifier of tweets based on features such as the keywords in a tweet, the number of words, and their context. Subsequently, we produce a probabilistic spatiotemporal model for the target event that can find the center and the trajectory of the event location. We consider each Twitter user as a sensor and apply Kalman filtering and particle filtering, which are widely used for location estimation in ubiquitous/pervasive computing. The particle filter works better than other comparable methods for estimating the centers of earthquakes and the trajectories of typhoons. As an application, we construct an earthquake reporting system in Japan. Because of the numerous earthquakes and the large number of Twitter users throughout the country, we can detect an earthquake with high probability (96% of earthquakes of Japan Meteorological Agency (JMA) seismic intensity scale 3 or more are detected) merely by monitoring tweets. Our system detects earthquakes promptly and sends e-mails to registered users. Notification is delivered much faster than the announcements that are broadcast by the JMA.

Taxi-calling apps are gaining increasing popularity for their efficiency in dispatching idle taxis to passengers in need. To precisely balance the supply and the demand of taxis, online taxicab platforms need to predict the Unit Original Taxi Demand (UOTD), which refers to the number of taxi-calling requirements submitted per unit time (e.g., every hour) and per unit region (e.g., each POI). Predicting UOTD is non-trivial for large-scale industrial online taxicab platforms because both accuracy and flexibility are essential. Complex non-linear models such as GBRT and deep learning are generally accurate, yet require labor-intensive model redesign after scenario changes (e.g., extra constraints due to new regulations). To accurately predict UOTD while remaining flexible to scenario changes, we propose LinUOTD, a unified linear regression model with more than 200 million dimensions of features. The simple model structure eliminates the need of repeated model redesign, while the high-dimensional features contribute to accurate UOTD prediction. We further design a series of optimization techniques for efficient model training and updating. Evaluations on two large-scale datasets from an industrial online taxicab platform verify that LinUOTD outperforms popular non-linear models in accuracy. We envision our experiences to adopt simple linear models with high-dimensional features in UOTD prediction as a pilot study and can shed insights upon other industrial large-scale spatio-temporal prediction problems.

Intelligent Transportation Systems (ITS) have attracted the attention of researchers and the general public alike as a means to alleviate traffic congestion. Recently, the maturity of wireless technology has enabled a cost-efficient way to achieve ITS by detecting vehicles using Vehicle to Infrastructure (V2I) communications. Traditional ITS algorithms, in most cases, assume that every vehicle is observed, such as by a camera or a loop detector, but a V2I implementation would detect only those vehicles with wireless communications capability. We examine a family of transportation systems, which we will refer to as ‘Partially Detected Intelligent Transportation Systems’. An algorithm that can act well under a small detection rate is highly desirable due to gradual penetration rates of the underlying wireless technologies such as Dedicated Short Range Communications (DSRC) technology. Artificial Intelligence (AI) techniques for Reinforcement Learning (RL) are suitable tools for finding such an algorithm due to utilizing varied inputs and not requiring explicit analytic understanding or modeling of the underlying system dynamics. In this paper, we report a RL algorithm for partially observable ITS based on DSRC. The performance of this system is studied under different car flows, detection rates, and topologies of the road network. Our system is able to efficiently reduce the average waiting time of vehicles at an intersection, even with a low detection rate.