Question: A minimally supervised method for number normalization.

This paper explores a groundbreaking technique, Long Short-Term Memory (LSTM), designed to address the challenges of learning over long time intervals in recurrent neural networks. The problem of insufficient error backflow, leading to slow learning, is a well-documented issue in recurrent backpropagation. Hochreiter's analysis of this problem in 1991 highlights the decaying error backflow as a significant barrier. To overcome this, we introduce LSTM, a gradient-based method that allows for efficient learning over extended time periods. By truncating the gradient where it does not compromise learning, LSTM introduces constant error carousels within special units to maintain a constant error flow. These units are equipped with multiplicative gate units that learn to control access to this constant error flow, thereby enabling LSTM to bridge time lags of over 1000 discrete-time steps. The beauty of LSTM lies in its locality in both space and time, with computational complexity per time step and weight being linear, denoted as O(1). Our experiments with artificial data, featuring local, distributed, real-valued, and noisy pattern representations, demonstrate LSTM's superior performance. Compared to methods like real-time recurrent learning, backpropagation through time, recurrent cascade correlation, Elman nets, and neural sequence chunking, LSTM not only leads to a higher success rate but also learns significantly faster. Additionally, LSTM has proven capable of solving complex, artificial long-time-lag tasks that were previously unsolvable by other recurrent network algorithms.

In this report, we explore various methods that are applied to solve NER. In section 1, we introduce the named entity problem. In section 2, various named entity recognition methods are discussed in three three broad categories of machine learning paradigm and explore few learning techniques in them. In the first part, we discuss various supervised techniques. Subsequently we move to semi-supervised and unsupervised techniques. In the end we discuss about the method from deep learning to solve NER.

Children's speech processing is more challenging than that of adults due to lacking of large scale children's speech corpora. With the developing of the physical speech organ, high inter speaker and intra speaker variabilities are observed in children's speech. On the other hand, data collection on children is difficult as children usually have short attention span and their language proficiency is limited. In this paper, we propose to improve children's automatic speech recognition performance with transfer learning technique. We compare two transfer learning approaches in enhancing children's speech recognition performance with adults' data. The first method is to perform acoustic model adaptation on the pre-trained adult model. The second is to train acoustic model with deep neural network based multi-task learning approach: the adults' and children's acoustic characteristics are learnt jointly in the shared hidden layers, while the output layers are optimized with different speaker groups. Our experiment results show that both transfer learning approaches are effective in transferring rich phonetic and acoustic information from adults' model to children model. The multi-task learning approach outperforms the acoustic adaptation approach. We further show that the speakers' acoustic characteristics in languages can also benefit the target language under the multi-task learning framework.

Automatic information retrieval systems have to deal with documents of varying lengths in a text collection. Document length normalization is used to fairly retrieve documents of all lengths. In this study, we ohserve that a normalization scheme that retrieves documents of all lengths with similar chances as their likelihood of relevance will outperform another scheme which retrieves documents with chances very different from their likelihood of relevance. We show that the retrievaf probabilities for a particular normalization method deviate systematically from the relevance probabilities across different collections. We present pivoted normalization, a technique that can be used to modify any normalization function thereby reducing the gap between the relevance and the retrieval probabilities. Training pivoted normalization on one collection, we can successfully use it on other (new) text collections, yielding a robust, collectzorz independent normalization technique. We use the idea of pivoting with the well known cosine normalization function. We point out some shortcomings of the cosine function andpresent two new normalization functions--pivoted unique normalization and piuotert byte size normalization.

The objective of the paper is to recognize handwritten samples of Roman numerals using Tesseract open source Optical Character Recognition (OCR) engine. Tesseract is trained with data samples of different persons to generate one user-independent language model, representing the handwritten Roman digit-set. The system is trained with 1226 digit samples collected form the different users. The performance is tested on two different datasets, one consisting of samples collected from the known users (those who prepared the training data samples) and the other consisting of handwritten data samples of unknown users. The overall recognition accuracy is obtained as 92.1% and 86.59% on these test datasets respectively.

Gaussian mixture models with universal backgrounds (UBMs) have become the standard method for speaker recognition. Typically, a speaker model is constructed by MAP adaptation of the means of the UBM. A GMM supervector is constructed by stacking the means of the adapted mixture components. A recent discovery is that latent factor analysis of this GMM supervector is an effective method for variability compensation. We consider this GMM supervector in the context of support vector machines. We construct a support vector machine kernel using the GMM supervector. We show similarities based on this kernel between the method of SVM nuisance attribute projection (NAP) and the recent results in latent factor analysis. Experiments on a NIST SRE 2005 corpus demonstrate the effectiveness of the new technique

Our inspiration comes from Nell (Never Ending Language Learning), a computer program running at Carnegie Mellon University to extract structured information from unstructured web pages. We consider the problem of semi-supervised learning approach to extract category instances (e.g. country(USA), city(New York)) from web pages, starting with a handful of labeled training examples of each category or relation, plus hundreds of millions of unlabeled web documents. Semisupervised approaches using a small number of labeled examples together with many unlabeled examples are often unreliable as they frequently produce an internally consistent, but nevertheless, incorrect set of extractions. We believe that this problem can be overcome by simultaneously learning independent classifiers in a new approach named Coupled Bayesian Sets algorithm, based on Bayesian Sets, for many different categories and relations (in the presence of an ontology defining constraints that couple the training of these classifiers). Experimental results show that simultaneously learning a coupled collection of classifiers for random 11 categories resulted in much more accurate extractions than training classifiers through original Bayesian Sets algorithm, Naive Bayes, BaS-all and Coupled Pattern Learner (the category extractor used in NELL).

This paper introduces a new corpus of read English speech, suitable for training and evaluating speech recognition systems. The LibriSpeech corpus is derived from audiobooks that are part of the LibriVox project, and contains 1000 hours of speech sampled at 16 kHz. We have made the corpus freely available for download, along with separately prepared language-model training data and pre-built language models. We show that acoustic models trained on LibriSpeech give lower error rate on the Wall Street Journal (WSJ) test sets than models trained on WSJ itself. We are also releasing Kaldi scripts that make it easy to build these systems.

Answering questions correctly from standardized eighth-grade science tests is itself a test of machine intelligence.

This paper concerns limited domain TTS system based on the concatenative method, and presents an algorithm capable to extract the minimal domainoriented text corpus from the real data of the given domain, while still reaching the maximum coverage of the domain. The proposed approach ensures that the least amount of texts are extracted, containing the most common phrases and (possibly) all the words from the domain. At the same time, it ensures that appropriate phrase overlapping is kept, allowing to find smooth concatenation in the overlapped regions to reach high quality synthesized speech. In addition, several recommendations allowing a speaker to record the corpus more fluently and comfortably are presented and discussed. The corpus building is tested and evaluated on several domains differing in size and nature, and the authors present the results of the algorithm and demonstrate the advantages of using the domain oriented corpus for speech synthesis.

We present a paradigm for the automatic assessment of pronunciation quality by machine. In this scoring paradigm, both native and nonnative speech data is collected and a database of human-expert ratings is created to enable the development of a variety of machine scores. We ®rst discuss issues related to the design of speech databases and the reliability of human ratings. We then address pronunciation evaluation as a prediction problem, trying to predict the grade a human expert would assign to a particular skill. Using the speech and the expert-ratings databases, we build statistical models and introduce dierent machine scores that can be used as predictor variables. We validate these machine scores on the Voice Interactive Language Training System (VILTS) corpus, evaluating the pronunciation of American speakers speaking French and we show that certain machine scores, like the log-posterior and the normalized duration, achieve a correlation with the targeted human grades that is comparable to the human-to-human correlation when a sucient amount of speech data is available. Ó 2000 Elsevier Science B.V. All rights reserved.


Question: A minimally supervised method for number normalization.

This paper delves into the potential of deep recurrent neural networks (RNNs) for speech recognition tasks, highlighting their capability in processing sequential data. RNNs, when paired with end-to-end training methodologies like Connectionist Temporal Classification, can be effectively trained for sequence labeling problems, even when the input-output alignment is uncertain. The Long Short-term Memory (LSTM) architecture, a variant of RNNs, has demonstrated remarkable success in cursive handwriting recognition. However, RNNs have previously underperformed in speech recognition tasks compared to deep feedforward networks. In this study, we explore deep RNNs, which merge the hierarchical representation of deep networks with the temporal context handling ability of RNNs. Our findings reveal that, when trained end-to-end with appropriate regularization, deep LSTM RNNs can achieve a test set error rate of 17.7% on the TIMIT phoneme recognition benchmark, marking the best-known score to date.

**Question**: An Integrated Framework for Privacy-Preserving Data Mining

**Documents**: A privacy-preserving data mining framework based on a hybrid approach
In recent years, data mining has become an important tool for extracting useful information from large data sets. However, this process can also lead to privacy concerns, as sensitive information may be revealed during the mining process. To address these concerns, we propose a privacy-preserving data mining framework based on a hybrid approach. Our framework combines two main techniques: data perturbation and secure multi-party computation. Data perturbation is used to modify the original data set in a way that preserves privacy while still maintaining the utility of the data for mining. Secure multi-party computation is used to perform the mining process on the perturbed data set in a way that ensures the privacy of the individual data points. The hybrid approach allows us to achieve a balance between privacy and utility, making it suitable for a wide range of data mining applications. We evaluate our framework using several real-world data sets and show that it can effectively preserve privacy while still achieving good performance in terms of mining accuracy.

**Similar Documents**: This paper presents an integrated framework for privacy-preserving data mining, designed to address the dual challenge of extracting valuable insights from large datasets while safeguarding individual privacy. The proposed framework employs a dual-strategy approach, combining data perturbation and secure multi-party computation techniques. Data perturbation modifies the original dataset to ensure privacy without compromising the data's utility for mining purposes. Secure multi-party computation is then utilized to conduct the mining process on the perturbed data,

Mel Frequency Cepstral Coefficients (MFCC) and Perceptual Linear Prediction (PLP) are the most popular acoustic features used in speech recognition. Often it depends on the task, which of the two methods leads to a better performance. In this work we develop acoustic features that combine the advantages of MFCC and PLP. Based on the observation that the techniques have many similarities, we revise the processing steps of PLP. In particular, the filter-bank, the equal-loudness pre-emphasis and the input for the linear prediction are improved. It is shown for a broadcast news transcription task and a corpus of children’s speech that the new variant of PLP performs better than both MFCC and conventional PLP for a wide range of clean and noisy acoustic conditions.

In automatic speech recognition on mobile devices, very often what a user says strongly depends on the particular context he or she is in. The n-grams relevant to the context are often not known in advance. The context can depend on, for example, particular dialog state, options presented to the user, conversation topic, location, etc. Speech recognition of sentences that include these n-grams can be challenging, as they are often not well represented in a language model (LM) or even include out-of-vocabulary (OOV) words. In this paper, we propose a solution for using contextual information to improve speech recognition accuracy. We utilize an on-the-fly rescoring mechanism to adjust the LM weights of a small set of n-grams relevant to the particular context during speech decoding. Our solution handles out of vocabulary words. It also addresses efficient combination of multiple sources of context and it even allows biasing class based language models. We show significant speech recognition accuracy improvements on several datasets, using various types of contexts, without negatively impacting the overall system. The improvements are obtained in both offline and live experiments.

The objective of the paper is to recognize handwritten samples of Roman numerals using Tesseract open source Optical Character Recognition (OCR) engine. Tesseract is trained with data samples of different persons to generate one user-independent language model, representing the handwritten Roman digit-set. The system is trained with 1226 digit samples collected form the different users. The performance is tested on two different datasets, one consisting of samples collected from the known users (those who prepared the training data samples) and the other consisting of handwritten data samples of unknown users. The overall recognition accuracy is obtained as 92.1% and 86.59% on these test datasets respectively.

We present an evaluation of “off-theshelf” language identification systems as applied to microblog messages from Twitter. A key challenge is the lack of an adequate corpus of messages annotated for language that reflects the linguistic diversity present on Twitter. We overcome this through a “mostly-automated” approach to gathering language-labeled Twitter messages for evaluating language identification. We present the method to construct this dataset, as well as empirical results over existing datasets and off-theshelf language identifiers. We also test techniques that have been proposed in the literature to boost language identification performance over Twitter messages. We find that simple voting over three specific systems consistently outperforms any specific system, and achieves state-of-the-art accuracy on the task.

Text-independent speaker verification (SV) is currently in the process of embracing DNN modeling in every stage of SV system. Slowly, the DNN-based approaches such as end-to-end modelling and systems based on DNN embeddings start to be competitive even in challenging and diverse channel conditions of recent NIST SREs. Domain adaptation and the need for a large amount of training data are still a challenge for current discriminative systems and (unlike with generative models), we see significant gains from data augmentation, simulation and other techniques designed to overcome lack of training data. We present an analysis of a SV system based on DNN embeddings (x-vectors) and focus on robustness across diverse data domains such as standard telephone and microphone conversations, both in clean, noisy and reverberant environments. We also evaluate the system on challenging far-field data created by re-transmitting a subset of NIST SRE 2008 and 2010 microphone interviews. We compare our results with the stateof-the-art i-vector system. In general, we were able to achieve better performance with the DNN-based systems, but most importantly, we have confirmed the robustness of such systems across multiple data domains.

Text documents often contain valuable structured data that is hidden Yin regular English sentences. This data is best exploited infavailable as arelational table that we could use for answering precise queries or running data mining tasks.We explore a technique for extracting such tables from document collections that requires only a handful of training examples from users. These examples are used to generate extraction patterns, that in turn result in new tuples being extracted from the document collection.We build on this idea and present our Snowball system. Snowball introduces novel strategies for generating patterns and extracting tuples from plain-text documents.At each iteration of the extraction process, Snowball evaluates the quality of these patterns and tuples without human intervention,and keeps only the most reliable ones for the next iteration. In this paper we also develop a scalable evaluation methodology and metrics for our task, and present a thorough experimental evaluation of Snowball and comparable techniques over a collection of more than 300,000 newspaper documents.

Direct acoustics-to-word (A2W) models in the end-to-end paradigm have received increasing attention compared to conventional subword based automatic speech recognition models using phones, characters, or context-dependent hidden Markov model states. This is because A2W models recognize words from speech without any decoder, pronunciation lexicon, or externally-trained language model, making training and decoding with such models simple. Prior work has shown that A2W models require orders of magnitude more training data in order to perform comparably to conventional models. Our work also showed this accuracy gap when using the English Switchboard-Fisher data set. This paper describes a recipe to train an A2W model that closes this gap and is at-par with state-of-the-art sub-word based models. We achieve a word error rate of 8.8.8%/13.9% on the Hub5-2000 Switchboard/CallHome test sets without any decoder or language model. We find that model initialization, training data order, and regularization have the most impact on the A2W model performance. Next, we present a joint word-character A2W model that learns to first spell the word and then recognize it. This model provides a rich output to the user instead of simple word hypotheses, making it especially useful in the case of words unseen or rarely-seen during training.

Most Text to Speech (TTS) systems today assume that the input text is in a single language and is written in the same language that the text needs to be synthesized in. However, in bilingual and multilingual communities, code mixing or code switching occurs in speech, in which speakers switch between languages in the same utterance. Due to the popularity of social media, we now see code-mixing even in text in these multilingual communities. TTS systems capable of synthesizing such text need to be able to handle text that is written in multiple languages and scripts. Code-mixed text poses many challenges to TTS systems, such as language identification, spelling normalization and pronunciation modeling. In this work, we describe a preliminary framework for synthesizing code-mixed text. We carry out experiments on synthesizing code-mixed Hindi and English text. We find that there is a significant user preference for TTS systems that can correctly identify and pronounce words in different languages.

An ASR system usually does not predict any punctuation or capitalization. Lack of punctuation causes problems in result presentation and confuses both the human reader and off-the-shelf natural language processing algorithms. To overcome these limitations, we train two variants of Deep Neural Network (DNN) sequence labelling models a Bidirectional Long Short-Term Memory (BLSTM) and a Convolutional Neural Network (CNN), to predict the punctuation. The models are trained on the Fisher corpus which includes punctuation annotation. In our experiments, we combine time-aligned and punctuated Fisher corpus transcripts using a sequence alignment algorithm. The neural networks are trained on Common Web Crawl GloVe embedding of the words in Fisher transcripts aligned with conversation side indicators and word time infomation. The CNNs yield a better precision and BLSTMs tend to have better recall. While BLSTMs make fewer mistakes overall, the punctuation predicted by the CNN is more accurate especially in the case of question marks. Our results constitute significant evidence that the distribution of words in time, as well as pre-trained embeddings, can be useful in the punctuation prediction task.

Indonesia has a lot of tribe, so that there are a lot of dialects. Speech classification is difficult if the database uses speech signals from various people who have different characteristics because of gender and dialect. The different characteristics will influence frequency, intonation, amplitude, and period of the speech. It makes the system must be trained for the various templates reference of speech signal. Therefore, this study has been developed for Indonesian speech classification. This study designs the solution of the different characteristics for Indonesian speech classification. The solution combines Fuzzy on Hidden Markov Models. The new design of fuzzy Hidden Markov Models will be proposed in this study. The models will consist of Fuzzy C-Means Clustering which will be designed to substitute the vector quantization process and a new forward and backward method to handle the membership degree of data. The result shows FHMM is better than HMM and the improvement was 3.33 %.


Question: Exploring intrinsic video techniques and their practical applications.

This article focuses on the challenge of dividing an image into distinct regions, a process known as image segmentation. We introduce a metric to assess the likelihood of a boundary existing between two regions, utilizing a graph-based representation of the image. Based on this metric, we devise an efficient segmentation algorithm that, despite making locally optimal decisions, results in segmentations that uphold global consistency. The algorithm's performance is demonstrated on image segmentation tasks, using two distinct types of local neighborhoods for graph construction. Our results are showcased through the segmentation of both real and synthetic images, highlighting the algorithm's efficiency and practicality. Notably, the algorithm's runtime is nearly linear with respect to the number of graph edges, making it particularly fast in real-world applications. A key feature of our approach is its capacity to maintain fine details in regions with low variability, while effectively ignoring finer details in areas with high variability, thus enhancing the overall quality of the segmentation.

The use of AV features alone is insufficient to induce high-level semantics. This article proposes a framework that utilizes both internal AV features and various types of external information sources for event detection in team sports video. Three schemes are also proposed to tackle the asynchronism between the fusion of AV and external information. The framework is extensible as it can provide increasing functionalities given more detailed external information and domain knowledge. By demonstrating its effectiveness on soccer and American football, we believe that with the availability of appropriate domain knowledge, the framework is applicable to other team sports.

Exposure bracketing for high dynamic range (HDR) imaging involves capturing several images of the scene at different exposures. If either the camera or the scene moves during capture, the captured images must be registered. Large exposure differences between bracketed images lead to inaccurate registration, resulting in artifacts such as ghosting (multiple copies of scene objects) and blur. We present two techniques, one for image capture (Fibonacci exposure bracketing) and one for image registration (generalized registration), to prevent such motion-related artifacts. Fibonacci bracketing involves capturing a sequence of images such that each exposure time is the sum of the previous N(N > 1) exposures. Generalized registration involves estimating motion between sums of contiguous sets of frames, instead of between individual frames. Together, the two techniques ensure that motion is always estimated between frames of the same total exposure time. This results in HDR images and videos which have both a large dynamic range and minimal motion-related artifacts. We show, by results for several real-world indoor and outdoor scenes, that the proposed approach significantly outperforms several existing bracketing schemes.

JackIn is a new human-human communication framework for connecting two or more people. With first-person view video streaming from a person (called Body) wearing a transparent head-mounted display and a head-mounted camera, the other person (called Ghost) participates in the shared first-person view. With JackIn, people's activities can be shared and assistance or guidance can be given through other peoples expertise. This can be applied to daily activities such as cooking lessons, shopping navigation, education in craft-work or electrical work, and sharing experiences of sporting and live events. For a better viewing experience with frist-person view, we developed the out-of-body view in which first-person images are integrated to construct a scene around a Body, and a Ghost can virtually control the viewpoint to look around the space surrounding the Body. We also developed a tele-pointing gesture interface. We conducted an experiment to evaluate how effective this framework is and found that Ghosts can understand the spatial situation of the Body.

With the advent of perceptual loss functions, new possibilities in super-resolution have emerged, and we currently have models that successfully generate near-photorealistic high-resolution images from their low-resolution observations. Up to now, however, such approaches have been exclusively limited to single image super-resolution. The application of perceptual loss functions on video processing still entails several challenges, mostly related to the lack of temporal consistency of the generated images, i.e., flickering artifacts. In this work, we present a novel adversarial recurrent network for video upscaling that is able to produce realistic textures in a temporally consistent way. The proposed architecture naturally leverages information from previous frames due to its recurrent architecture, i.e. the input to the generator is composed of the low-resolution image and, additionally, the warped output of the network at the previous step. We also propose an additional loss function to further reinforce temporal consistency in the generated sequences. The experimental validation of our algorithm shows the effectiveness of our approach which obtains competitive samples in terms of perceptual quality with improved temporal consistency.

Accurate segmentation of humans from live videos is an important problem to be solved in developing immersive video experience. We propose to extract the human segmentation information from color and depth cues in a video using multiple modeling techniques. The prior information from human skeleton data is also fused along with the depth and color models to obtain the final segmentation inside a graph-cut framework. The proposed method runs real time on live videos using single CPU and is shown to be quantitatively outperforming the methods that directly fuse color and depth data.

In this paper we introduce a 3-dimensional (3D) SIFT descriptor for video or 3D imagery such as MRI data. We also show how this new descriptor is able to better represent the 3D nature of video data in the application of action recognition. This paper will show how 3D SIFT is able to outperform previously used description methods in an elegant and efficient manner. We use a bag of words approach to represent videos, and present a method to discover relationships between spatio-temporal words in order to better describe the video data.

A fundamental problem in computer vision is that of inferring the intrinsic, 3D structure of the world from flat, 2D images of that world. Traditional methods for recovering scene properties such as shape, reflectance, or illumination rely on multiple observations of the same scene to overconstrain the problem. Recovering these same properties from a single image seems almost impossible in comparison-there are an infinite number of shapes, paint, and lights that exactly reproduce a single image. However, certain explanations are more likely than others: surfaces tend to be smooth, paint tends to be uniform, and illumination tends to be natural. We therefore pose this problem as one of statistical inference, and define an optimization problem that searches for the most likely explanation of a single image. Our technique can be viewed as a superset of several classic computer vision problems (shape-from-shading, intrinsic images, color constancy, illumination estimation, etc) and outperforms all previous solutions to those constituent problems.

Photometric stereo is one technique for 3D shape determination that has been implemented in a variety of experimental settings and that has produced consistently good results. The idea is to use intensity values recorded from multiple images obtained from the same viewpoint but under different conditions of illumination. The resulting radiometric constraint makes it possible to obtain local estimates of both surface orientation and surface curvature without requiring global smoothness assumptions and without requiring prior image segmentation. This paper moves photometric stereo one step closer to practical viability by describing an experimental setting in which surface gradient estimation is achieved on full frame video data at near video frame rates (i.e., 15Hz). The implementation uses commercially available hardware. Reflectance is modeled empirically using measurements obtained from a calibration sphere. Estimation of the gradient, (p, q), requires only simple table lookup. Curvature estimation uses, in addition, the reflectance map, R(p, q). The required lookup table and reflectance maps are derived during calibration. Because reflectance is modeled empirically, no prior physical model of the reflectance characteristics of the objects to be analyzed is assumed. At the same time, if a good physical model is available, it can be retrofit to the method for implementation purposes. Photometric stereo is subject to error in the presence of cast shadows and interreflection. No purely local technique can succeed since these phenomena are inherently non-local. Nevertheless, this paper demonstrates that one can exploit the redundancy in three light source photometric stereo to, in most cases, locally detect the presence of cast shadows and interreflection. Detection is facilitated by explicitly including a local confidence estimate in the lookup table used for gradient estimation.

Recent years have seen tremendous progress in still-image segmentation; however the naı̈ve application of these state-of-the-art algorithms to every video frame requires considerable computation and ignores the temporal continuity inherent in video. We propose a video recognition framework that relies on two key observations: 1) while pixels may change rapidly from frame to frame, the semantic content of a scene evolves more slowly, and 2) execution can be viewed as an aspect of architecture, yielding purpose-fit computation schedules for networks. We define a novel family of “clockwork” convnets driven by fixed or adaptive clock signals that schedule the processing of different layers at different update rates according to their semantic stability. We design a pipeline schedule to reduce latency for real-time recognition and a fixed-rate schedule to reduce overall computation. Finally, we extend clockwork scheduling to adaptive video processing by incorporating data-driven clocks that can be tuned on unlabeled video. The accuracy and efficiency of clockwork convnets are evaluated on the Youtube-Objects, NYUD, and Cityscapes video datasets.

We present a system for capturing ink strokes written with ordinary pen and paper using a fast camera with a frame rate comparable to a stylus digitizer. From the video frames, ink strokes are extracted and used as input to an online handwriting recognition engine. A key component in our system is a pen up/down detection model for detecting the contact of the pen-tip with the paper in the video frames. The proposed model consists of feature representation with convolutional neural networks and classification with a recurrent neural network. We also use a high speed tracker with kernelized correlation filters to track the pen-tip. For training and evaluation, we collected labeled video data of users writing English and Japanese phrases from public datasets, and we report on character accuracy scores for different frame rates in the two languages.


Question: Intrinsic video: methods and applications

A technique for transferring cinematic color grading to amateur video clips

In professional film production, achieving a distinctive visual style often involves meticulous color grading by expert colorists. This complex process, however, is typically beyond the reach of amateur videographers who lack the time and expertise to apply similar color adjustments to their own videos. This paper introduces a method to automate the transfer of a cinematic color palette from a model video clip to an amateur video sequence. By estimating a frame-by-frame color transformation that aligns the color distributions of the two videos, the method aims to replicate the model's visual style. Directly applying this transformation can result in visual artifacts like color bleeding and flickering. To address this, we propose a differential-geometry-based approach that ensures smooth and coherent color transitions by minimizing the curvature of the color transformation curve, akin to curvature flow models in image processing. Moreover, our method automatically selects a series of keyframes that accurately represent the interpolated color transformation, providing a basis for further manual refinement of the color grade. We demonstrate the effectiveness of our technique in transferring a variety of color palettes across different visual styles and input video clips, showcasing its potential to enhance the aesthetic quality of amateur video content.

We present a system that transforms a monocular video of a soccer game into a moving 3D reconstruction, in which the players and field can be rendered interactively with a 3D viewer or through an Augmented Reality device. At the heart of our paper is an approach to estimate the depth map of each player, using a CNN that is trained on 3D player data extracted from soccer video games. We compare with state of the art body pose and depth estimation techniques, and show results on both synthetic ground truth benchmarks, and real YouTube soccer footage.

This paper proposes a quasi-dense reconstruction from uncalibrated sequence. The main innovation is that all geometry is computed based on re-sampled quasi-dense correspondences rather than the standard sparse points of interest. It not only produces more accurate and robust reconstruction due to highly redundant and well spread input data, but also fills the gap of insufficiency of sparse reconstruction for visualization application. The computational engine is the quasi-dense 2-view and the quasi-dense 3-view algorithms developed in this paper. Experiments on real sequences demonstrate the superior performance of quasi-dense w.r.t. sparse reconstruction both in accuracy and robustness.

With nearly one billion online videos viewed everyday, an emerging new frontier in computer vision research is recognition and search in video. While much effort has been devoted to the collection and annotation of large scalable static image datasets containing thousands of image categories, human action datasets lag far behind. Current action recognition databases contain on the order of ten different action categories collected under fairly controlled conditions. State-of-the-art performance on these datasets is now near ceiling and thus there is a need for the design and creation of new benchmarks. To address this issue we collected the largest action video database to-date with 51 action categories, which in total contain around 7,000 manually annotated clips extracted from a variety of sources ranging from digitized movies to YouTube. We use this database to evaluate the performance of two representative computer vision systems for action recognition and explore the robustness of these methods under various conditions such as camera motion, viewpoint, video quality and occlusion.

Existing optical flow datasets are limited in size and variability due to the difficulty of capturing dense ground truth. In this paper, we tackle this problem by tracking pixels through densely sampled space-time volumes recorded with a high-speed video camera. Our model exploits the linearity of small motions and reasons about occlusions from multiple frames. Using our technique, we are able to establish accurate reference flow fields outside the laboratory in natural environments. Besides, we show how our predictions can be used to augment the input images with realistic motion blur. We demonstrate the quality of the produced flow fields on synthetic and real-world datasets. Finally, we collect a novel challenging optical flow dataset by applying our technique on data from a high-speed camera and analyze the performance of the state-of-the-art in optical flow under various levels of motion blur.

An unmanned aerial vehicle (UAV) has many applications in a variety of fields. Detection and tracking of a specific road in UAV videos play an important role in automatic UAV navigation, traffic monitoring, and ground-vehicle tracking, and also is very helpful for constructing road networks for modeling and simulation. In this paper, an efficient road detection and tracking framework in UAV videos is proposed. In particular, a graph-cut-based detection approach is given to accurately extract a specified road region during the initialization stage and in the middle of tracking process, and a fast homography-based road-tracking scheme is developed to automatically track road areas. The high efficiency of our framework is attributed to two aspects: the road detection is performed only when it is necessary and most work in locating the road is rapidly done via very fast homography-based tracking. Experiments are conducted on UAV videos of real road scenes we captured and downloaded from the Internet. The promising results indicate the effectiveness of our proposed framework, with the precision of 98.4% and processing 34 frames per second for 1046 × 595 videos on average.

In most professional cinema productions, the color palette of the movie is painstakingly adjusted by a team of skilled colorists -- through a process referred to as color grading -- to achieve a certain visual look. The time and expertise required to grade a video makes it difficult for amateurs to manipulate the colors of their own video clips. In this work, we present a method that allows a user to transfer the color palette of a model video clip to their own video sequence. We estimate a per-frame color transform that maps the color distributions in the input video sequence to that of the model video clip. Applying this transformation naively leads to artifacts such as bleeding and flickering. Instead, we propose a novel differential-geometry-based scheme that interpolates these transformations in a manner that minimizes their curvature, similarly to curvature flows. In addition, we automatically determine a set of keyframes that best represent this interpolated transformation curve, and can be used subsequently, to manually refine the color grade. We show how our method can successfully transfer color palettes between videos for a range of visual styles and a number of input video clips.

This paper describes an extension of the high efficiency video coding (HEVC) standard for coding of multi-view video and depth data. In addition to the known concept of disparity-compensated prediction, inter-view motion parameter, and inter-view residual prediction for coding of the dependent video views are developed and integrated. Furthermore, for depth coding, new intra coding modes, a modified motion compensation and motion vector coding as well as the concept of motion parameter inheritance are part of the HEVC extension. A novel encoder control uses view synthesis optimization, which guarantees that high quality intermediate views can be generated based on the decoded data. The bitstream format supports the extraction of partial bitstreams, so that conventional 2D video, stereo video, and the full multi-view video plus depth format can be decoded from a single bitstream. Objective and subjective results are presented, demonstrating that the proposed approach provides 50% bit rate savings in comparison with HEVC simulcast and 20% in comparison with a straightforward multi-view extension of HEVC without the newly developed coding tools.

We present a model for intrinsic decomposition of RGB-D images. Our approach analyzes a single RGB-D image and estimates albedo and shading fields that explain the input. To disambiguate the problem, our model estimates a number of components that jointly account for the reconstructed shading. By decomposing the shading field, we can build in assumptions about image formation that help distinguish reflectance variation from shading. These assumptions are expressed as simple nonlocal regularizers. We evaluate the model on real-world images and on a challenging synthetic dataset. The experimental results demonstrate that the presented approach outperforms prior models for intrinsic decomposition of RGB-D images.

This paper presents a camera tracking system for the spatial stabilization of Augmented Reality (AR) media. Our approach integrates both artificial landmarks (fiducials) and natural features for camera tracking. Artificial landmarks are used for system initialization and computation of initial camera pose. Robust and extendible tracking is achieved by dynamically calibrating the 3D positions of a priori uncalibrated natural features. Analysis and experimental results demonstrate the effectiveness of this approach for presenting stabilized AR media in long camera-motion video sequences.

An unsupervised video object segmentation algorithm, which discovers a primary object in a video sequence automatically, is proposed in this work. We introduce three energies in terms of foreground and background probability distributions: Markov, spatiotemporal, and antagonistic energies. Then, we minimize a hybrid of the three energies to separate a primary object from its background. However, the hybrid energy is nonconvex. Therefore, we develop the alternate convex optimization (ACO) scheme, which decomposes the nonconvex optimization into two quadratic programs. Moreover, we propose the forward-backward strategy, which performs the segmentation sequentially from the first to the last frames and then vice versa, to exploit temporal correlations. Experimental results on extensive datasets demonstrate that the proposed ACO algorithm outperforms the state-of-the-art techniques significantly.


Question: Exploring intrinsic video techniques and their diverse applications.

This paper explores an advanced method for estimating optical flow with high accuracy, integrating a unique energy functional that merges three key principles: brightness constancy, gradient constancy, and a discontinuity-preserving spatio-temporal smoothness constraint. To accommodate substantial displacements, the methodology deliberately eschews linearizations in the data terms, ensuring a more precise representation of motion. A consistent numerical scheme is introduced, featuring nested fixed point iterations, which effectively implements a coarse-to-fine warping strategy. This theoretical grounding for warping, previously utilized on an experimental basis, is now supported by rigorous analysis. Comparative assessments reveal that this novel technique significantly outperforms conventional optical flow estimation methods, boasting reduced angular errors. Moreover, the method exhibits remarkable stability against parameter fluctuations and exceptional resilience under noisy conditions. These findings were presented at the 8th European Conference on Computer Vision in 2004, where the paper was honored with the Longuet-Higgins Best Paper Award, highlighting its significant contributions to the field of computer vision.

The paper presents a system for automatic, geo-registered, real-time 3D reconstruction from video of urban scenes. The system collects video streams, as well as GPS and inertia measurements in order to place the reconstructed models in geo-registered coordinates. It is designed using current state of the art real-time modules for all processing steps. It employs commodity graphics hardware and standard CPU’s to achieve real-time performance. We present the main considerations in designing the system and the steps of the processing pipeline. Our system extends existing algorithms to meet the robustness and variability necessary to operate out of the lab. To account for the large dynamic range of outdoor videos the processing pipeline estimates global camera gain changes in the feature tracking stage and efficiently compensates for these in stereo estimation without impacting the real-time performance. The required accuracy for many applications is achieved with a two-step stereo reconstruction process exploiting the redundancy across frames. We show results on real video sequences comprising hundreds of thousands of frames.

This paper develops a novel framework for efficient large-scale video retrieval. We aim to find video according to higher level similarities, which is beyond the scope of traditional near duplicate search. Following the popular hashing technique we employ compact binary codes to facilitate nearest neighbor search. Unlike the previous methods which capitalize on only one type of hash code for retrieval, this paper combines heterogeneous hash codes to effectively describe the diverse and multi-scale visual contents in videos. Our method integrates feature pooling and hashing in a single framework. In the pooling stage, we cast video frames into a set of pre-specified components, which capture a variety of semantics of video contents. In the hashing stage, we represent each video component as a compact hash code, and combine multiple hash codes into hash tables for effective search. To speed up the retrieval while retaining most informative codes, we propose a graph-based influence maximization method to bridge the pooling and hashing stages. We show that the influence maximization problem is submodular, which allows a greedy optimization method to achieve a nearly optimal solution. Our method works very efficiently, retrieving thousands of video clips from TRECVID dataset in about 0.001 second. For a larger scale synthetic dataset with 1M samples, it uses less than 1 second in response to 100 queries. Our method is extensively evaluated in both unsupervised and supervised scenarios, and the results on TRECVID Multimedia Event Detection and Columbia Consumer Video datasets demonstrate the success of our proposed technique.

The goal of augmented reality is to provide the user with a view of the surroundings enriched by virtual objects. Practically all augmented reality systems rely on standard real-time rendering methods for displaying graphical objects. Although such conventional computer graphics algorithms are fast, they often fail to produce sufficiently realistic renderings. Therefore, virtual models can easily be distinguished from the real environment. We have recently proposed a novel approach for generating augmented reality images [4]. Our method is based on the idea of applying stylization techniques for adapting the visual realism of both the camera image and the virtual graphical objects. Since both the camera image and the virtual objects are stylized in a corresponding way, they appear very similar. Here, we present a new method for the stylization of augmented reality images. This approach generates a painterly brush stroke rendering. The resulting stylized augmented reality video frames look similar to paintings created in the pointillism style. We describe the implementation of the camera image filter and the non-photorealistic rendererfor virtual objects. These components have been newly designed or adapted for this purpose. They are fast enough for generating augmented reality images in near real-time (more than 14 frames per second).

In this paper, novel models for fire and smoke detection using image processing is provided. The models use different colour models for both fire and smoke. The colour models are extracted using a statistical analysis of samples extracted from different type of video sequences and images. The extracted models can be used in complete fire/smoke detection system which combines colour information with motion analysis.

We present an automated framework for visual assessment of the expertise level of surgeons using the OSATS (Objective Structured Assessment of Technical Skills) criteria. Video analysis techniques for extracting motion quality via frequency coefficients are introduced. The framework is tested on videos of medical students with different expertise levels performing basic surgical tasks in a surgical training lab setting. We demonstrate that transforming the sequential time data into frequency components effectively extracts the useful information differentiating between different skill levels of the surgeons. The results show significant performance improvements using DFT and DCT coefficients over known state-of-the-art techniques.

Taking photographs under low light conditions with a hand-held camera is problematic. A long exposure time can cause motion blur due to the camera shaking and a short exposure time gives a noisy image. We consider the new technical possibility offered by cameras that take image bursts. Each image of the burst is sharp but noisy. In this preliminary investigation, we explore a strategy to efficiently denoise multi-images or video. The proposed algorithm is a complex image processing chain involving accurate registration, video equalization, noise estimation and the use of state-of-the-art denoising methods. Yet, we show that this complex chain may become risk free thanks to a key feature: the noise model can be estimated accurately from the image burst. Preliminary tests will be presented. On the technical side, the method can already be used to estimate a non parametric camera noise model from any image burst.

We present a method for space-time completion of large space-time "holes" in video sequences of complex dynamic scenes. The missing portions are filled-in by sampling spatio-temporal patches from the available parts of the video, while enforcing global spatio-temporal consistency between all patches in and around the hole. This is obtained by posing the task of video completion and synthesis as a global optimization problem with a well-defined objective function. The consistent completion of static scene parts simultaneously with dynamic behaviors leads to realistic looking video sequences. Space-time video completion is useful for a variety of tasks, including, but not limited to: (i) Sophisticated video removal (of undesired static or dynamic objects) by completing the appropriate static or dynamic background information, (ii) Correction of missing/corrupted video frames in old movies, and (iii) Synthesis of new video frames to add a visual story, modify it, or generate a new one. Some examples of these are shown in the paper.

With nearly one billion online videos viewed everyday, an emerging new frontier in computer vision research is recognition and search in video. While much effort has been devoted to the collection and annotation of large scalable static image datasets containing thousands of image categories, human action datasets lag far behind. Current action recognition databases contain on the order of ten different action categories collected under fairly controlled conditions. State-of-the-art performance on these datasets is now near ceiling and thus there is a need for the design and creation of new benchmarks. To address this issue we collected the largest action video database to-date with 51 action categories, which in total contain around 7,000 manually annotated clips extracted from a variety of sources ranging from digitized movies to YouTube. We use this database to evaluate the performance of two representative computer vision systems for action recognition and explore the robustness of these methods under various conditions such as camera motion, viewpoint, video quality and occlusion.

We present a system that accepts as input a continuous stream of TV broadcast images from sports events. The system automatically detects a predetermined billboard in the scene, and replaces it with a user defined pattern, with no cooperation from the billboards or camera operators. The replacement is performed seamlessly so that a viewer should not be able to detect the substitution. This allows the targeting of advertising to the appropriate audience, which is especially useful for international events. This requires several modules using state of the art computer graphics, image processing and computer vision technology. The system relies on modular design, and on a pipeline architecture, in which the search and track modules propagate their results in symbolic form throughout the pipeline buffer, and the replacement is performed at the exit of the pipe only, therefore relying on accumulated information. Also, images are processed only once. This allows the system to make replacement decisions based on complete sequences, thus avoiding mid-sequence on screen billboard changes. We present the algorithms and the overall system architecture, and discuss further applications of the technology.

The field of perceptual quality assessment has gone through a wide range of developments and it is still growing. In particular, the area of no-reference (NR) image and video quality assessment has progressed rapidly during the last decade. In this article, we present a classification and review of latest published research work in the area of NR image and video quality assessment. The NR methods of visual quality assessment considered for review are structured into categories and subcategories based on the types of methodologies used for the underlying processing employed for quality estimation. Overall, the classification has been done into three categories, namely, pixel-based methods, bitstream-based methods, and hybrid methods of the aforementioned two categories. We believe that the review presented in this article will be helpful for practitioners as well as for researchers to keep abreast of the recent developments in the area of NR image and video quality assessment. This article can be used for various purposes such as gaining a structured overview of the field and to carry out performance comparisons for the state-of-the-art


Question: Intrinsic video: techniques and applications

This paper presents a novel optical flow data set extracted from the open-source 3D animated short film Sintel, designed to address the limitations of existing optical flow benchmarks. Real-world scenes with natural motion pose challenges for measuring ground truth optical flow, leading to data sets that are often small, lacking in complexity, and diversity. This new data set, however, offers features absent in the widely used Middlebury flow evaluation, such as extended sequences, extensive motions, specular reflections, motion and defocus blur, and atmospheric effects. The open-source nature of the graphics data used in Sintel enables the rendering of scenes under various complexity levels, facilitating a comprehensive evaluation of existing optical flow algorithms. A comparison of several recent optical flow methods reveals that those highly ranked on the Middlebury evaluation struggle with the more intricate data set, highlighting the need for further advancements in optical flow estimation techniques. To ensure the validity of using synthetic data, the study also examines the image and flow statistics of Sintel in relation to real films and videos, finding them to be remarkably similar. The data set, evaluation metrics, and website are made publicly accessible to encourage further research and algorithm development in the field of optical flow.

We present a model for intrinsic decomposition of RGB-D images. Our approach analyzes a single RGB-D image and estimates albedo and shading fields that explain the input. To disambiguate the problem, our model estimates a number of components that jointly account for the reconstructed shading. By decomposing the shading field, we can build in assumptions about image formation that help distinguish reflectance variation from shading. These assumptions are expressed as simple nonlocal regularizers. We evaluate the model on real-world images and on a challenging synthetic dataset. The experimental results demonstrate that the presented approach outperforms prior models for intrinsic decomposition of RGB-D images.

When rescue robots navigate in a rough terrain, significant vibration of the video occurs unavoidably and a video stabilization system is proposed in this paper to reduce the disturbance on the visual system of the present rescue robots. The Kalman filter is applied to estimate the motion vector of the robot and the false estimation occurrence can be thus greatly reduced by applying analysis of correlation and variance of the motion vector estimation. With a hierarchical searching algorithm implemented on the TI DSP 6437, the frame rate can be improved from 10 fps to 28 fps to realize the real-time video stabilization and furthermore, the cooperative mission of multiple rescue robots has been achieved by applying the developed visual servo technique. The proposed virtual bounded motion control algorithm further leads the robots approaching the target precisely with cooperation of multiple rescue robots. The demo video is available at: http://lab816.cn.nctu.edu.tw/DemoVideo/. video stabilization

We propose a function-based temporal pooling method that captures the latent structure of the video sequence data - e.g., how frame-level features evolve over time in a video. We show how the parameters of a function that has been fit to the video data can serve as a robust new video representation. As a specific example, we learn a pooling function via ranking machines. By learning to rank the frame-level features of a video in chronological order, we obtain a new representation that captures the video-wide temporal dynamics of a video, suitable for action recognition. Other than ranking functions, we explore different parametric models that could also explain the temporal changes in videos. The proposed functional pooling methods, and rank pooling in particular, is easy to interpret and implement, fast to compute and effective in recognizing a wide variety of actions. We evaluate our method on various benchmarks for generic action, fine-grained action and gesture recognition. Results show that rank pooling brings an absolute improvement of 7-10 average pooling baseline. At the same time, rank pooling is compatible with and complementary to several appearance and local motion based methods and features, such as improved trajectories and deep learning features.

Active vision systems are usually limited to either partial or static scene reconstructions. In this paper, we propose to acquire the entire 3D shape of a dynamic scene. This is performed using a multiple projectors and cameras system, that allows to recover the entire shape of the object within a single scan at each frame. Like previous approaches, a static and simple pattern is used to avoid interferences of multiple patterns projected on the same object. In this paper, we extend the technique to capture a dense entire shape of a moving object with accuracy and high video frame rate. To achieve this, we mainly propose two additional steps; one is checking the consistency between the multiple cameras and projectors, and the other is an algorithm for light sectioning based on a plane parameter optimization. In addition, we also propose efficient noise reduction and mesh generation algorithm which are necessary for practical applications. In the experiments, we show that we can successfully reconstruct dense entire shapes of moving objects. Results are illustrated on real data from a system composed of six projectors and six cameras that was actually built.

This paper presents a classified review of soccer video analysis works. The existing approaches in the aspects of highlight event detection, video summarization and retrieval based on video stream, ball and player tracking for provision of match statistics, technical and tactical analysis and application of different sources in soccer video analysis have been surveyed. In addition, some major existing commercial softwares developed for video analysis are introduced and compared. With regard to the existing challenge for automatic and realtime provision of video analysis, different computer vision approaches are discussed and compared. Audio, video and text feature extraction methods have been investigated and the future trends for improvement of the reviewed systems have been introduced in terms of response time optimization, increase of precision and eliminating the need of human intervention for video analysis.

In this paper we introduce a new salient object segmentation method, which is based on combining a saliency measure with a conditional random field (CRF) model. The proposed saliency measure is formulated using a statistical framework and local feature contrast in illumination, color, and motion information. The resulting saliency map is then used in a CRF model to define an energy minimization based segmentation approach, which aims to recover well-defined salient objects. The method is efficiently implemented by using the integral histogram approach and graph cut solvers. Compared to previous approaches the introduced method is among the few which are applicable to both still images and videos including motion cues. The experiments show that our approach outperforms the current state-of-the-art methods in both qualitative and quantitative terms.

Video and images acquired by a visual system are seriously degraded under hazy and foggy weather, which will affect the detection, tracking, and recognition of targets. Thus, restoring the true scene from such a foggy video or image is of significance. The main goal of this paper was to summarize current video and image defogging algorithms. We first presented a review of the detection and classification method of a foggy image. Then, we summarized existing image defogging algorithms, including image restoration algorithms, image contrast enhancement algorithms, and fusion-based defogging algorithms. We also presented current video defogging algorithms. We summarized objective image quality assessment methods that have been widely used for the comparison of different defogging algorithms, followed by an experimental comparison of various classical image defogging algorithms. Finally, we presented the problems of video and image defogging which need to be further studied. The code of all algorithms will be available at <;uri xlink:href="http://www.yongxu.org/lunwen.html" xlink:type="simple">http://www.yongxu.org/lunwen.html<;/uri>.

Accurate segmentation of humans from live videos is an important problem to be solved in developing immersive video experience. We propose to extract the human segmentation information from color and depth cues in a video using multiple modeling techniques. The prior information from human skeleton data is also fused along with the depth and color models to obtain the final segmentation inside a graph-cut framework. The proposed method runs real time on live videos using single CPU and is shown to be quantitatively outperforming the methods that directly fuse color and depth data.

A method for smoke detection in video is proposed. It is assumed the camera monitoring the scene is stationary. Since the smoke is semi-transparent, edges of image frames start loosing their sharpness and this leads to a decrease in the high frequency content of the image. To determine the smoke in the field of view of the camera, the background of the scene is estimated and decrease of high frequency energy of the scene is monitored using the spatial wavelet transforms of the current and the background images. Edges of the scene are especially important because they produce local extrema in the wavelet domain. A decrease in values of local extrema is also an indicator of smoke. In addition, scene becomes grayish when there is smoke and this leads to a decrease in chrominance values of pixels. Periodic behavior in smoke boundaries and convexity of smoke regions are also analyzed. All of these clues are combined to reach a final decision.

Near-duplicate video retrieval (NDVR) has recently attracted much research attention due to the exponential growth of online videos. It has many applications, such as copyright protection, automatic video tagging and online video monitoring. Many existing approaches use only a single feature to represent a video for NDVR. However, a single feature is often insufficient to characterize the video content. Moreover, while the accuracy is the main concern in previous literatures, the scalability of NDVR algorithms for large scale video datasets has been rarely addressed. In this paper, we present a novel approach-Multiple Feature Hashing (MFH) to tackle both the accuracy and the scalability issues of NDVR. MFH preserves the local structural information of each individual feature and also globally considers the local structures for all the features to learn a group of hash functions to map the video keyframes into the Hamming space and generate a series of binary codes to represent the video dataset. We evaluate our approach on a public video dataset and a large scale video dataset consisting of 132,647 videos collected from YouTube by ourselves. This dataset has been released (http://itee.uq.edu.au/shenht/UQ_VIDEO/). The experimental results show that the proposed method outperforms the state-of-the-art techniques in both accuracy and efficiency.


Question: Exploring intrinsic video and its diverse applications.

This paper explores the computation of optical flow, a critical challenge due to the inherent underdetermination at a single point in an image sequence, where one measurement must inform two velocity components. To overcome this limitation, a novel approach is introduced, premised on the assumption that the velocity of the brightness pattern across the image is largely smooth and continuous. An iterative algorithm is detailed, demonstrating its efficacy through successful optical flow computation in various synthetic image sequences. Notably, the method is resilient to spatial and temporal coarseness in image quantization, as well as robust against brightness level quantization and additive noise interference. The paper also critically examines scenarios where the smoothness assumption is breached at singular points or along lines within the image, illustrating the algorithm's performance under such conditions.

The paper describes an approach to IV that involves spatializing text content for enhanced visual browsing and analysis. The application arena is large text document corpora such as digital libraries, regulations and procedures, archived reports, etc. The basic idea is that text content from these sources may be transformed to a spatial representation that preserves informational characteristics from the documents. The spatial representation may then be visually browsed and analyzed in ways that avoid language processing and that reduce the analysts mental workload. The result is an interaction with text that more nearly resembles perception and action with the natural world than with the abstractions of written language.

We present QuickCut, an interactive video editing tool designed to help authors efficiently edit narrated video. QuickCut takes an audio recording of the narration voiceover and a collection of raw video footage as input. Users then review the raw footage and provide spoken annotations describing the relevant actions and objects in the scene. QuickCut time-aligns a transcript of the annotations with the raw footage and a transcript of the narration to the voiceover. These aligned transcripts enable authors to quickly match story events in the narration with semantically relevant video segments and form alignment constraints between them. Given a set of such constraints, QuickCut applies dynamic programming optimization to choose frame-level cut points between the video segments while maintaining alignments with the narration and adhering to low-level film editing guidelines. We demonstrate QuickCut's effectiveness by using it to generate a variety of short (less than 2 minutes) narrated videos. Each result required between 14 and 52 minutes of user time to edit (i.e. between 8 and 31 minutes for each minute of output video), which is far less than typical authoring times with existing video editing workflows.

The aim of this paper is to address recognition of natural human actions in diverse and realistic video settings. This challenging but important subject has mostly been ignored in the past due to several problems one of which is the lack of realistic and annotated video datasets. Our first contribution is to address this limitation and to investigate the use of movie scripts for automatic annotation of human actions in videos. We evaluate alternative methods for action retrieval from scripts and show benefits of a text-based classifier. Using the retrieved action samples for visual learning, we next turn to the problem of action classification in video. We present a new method for video classification that builds upon and extends several recent ideas including local space-time features, space-time pyramids and multi-channel non-linear SVMs. The method is shown to improve state-of-the-art results on the standard KTH action dataset by achieving 91.8% accuracy. Given the inherent problem of noisy labels in automatic annotation, we particularly investigate and show high tolerance of our method to annotation errors in the training set. We finally apply the method to learning and classifying challenging action classes in movies and show promising results.

Tone-mapping constitutes a key component within the field of high dynamic range (HDR) imaging. Its importance is manifested in the vast amount of tone-mapping methods that can be found in the literature, which are the result of an active development in the area for more than two decades. Although these can accommodate most requirements for display of HDR images, new challenges arose with the advent of HDR video, calling for additional considerations in the design of tone-mapping operators (TMOs). Today, a range of TMOs exist that do support video material. We are now reaching a point where most camera captured HDR videos can be prepared in high quality without visible artifacts, for the constraints of a standard display device. In this report, we set out to summarize and categorize the research in tone-mapping as of today, distilling the most important trends and characteristics of the tone reproduction pipeline. While this gives a wide overview over the area, we then specifically focus on tone-mapping of HDR video and the problems this medium entails. First, we formulate the major challenges a video TMO needs to address. Then, we provide a description and categorization of each of the existing video TMOs. Finally, by constructing a set of quantitative measures, we evaluate the performance of a number of the operators, in order to give a hint on which can be expected to render the least amount of artifacts. This serves as a comprehensive reference, categorization and comparative assessment of the state-of-the-art in tone-mapping for HDR video.

This paper presents a fire-alarming method based on video processing. We propose a system that uses color and motion information extracted from video sequences to detect fire. Flame can be recognized according to its color which is a primary element of fire images. Thus choosing a suitable color model is the key to detect flames from fire images. An effective fire detection criterion based on color model is proposed in this paper by intensive experiments and trainings. The detection criterion is used to make a raw localization of fire regions first. However, color alone is not enough for fire detection. To identify a real burning fire, in addition to chromatic features, dynamic features are usually adopted to distinguish other fire aliases. In this paper, both the growth of fire region and the invariability of flame are utilized to further detect the fire regions as a complement of the detection criterion. The effectiveness of the proposed fire-alarming method is demonstrated by the experiments implemented on a large number of scenes.

This paper presents an interactive thumbnail-based video browser for mobile devices such as smartphones featuring a touch screen. Developed as part of on-going research and supported by user studies, it introduces HiStory, a hierarchical storyboard design offering an interface metaphor that is familiar and intuitive yet supports fast and effective completion of Known-Item-Search tasks by rapidly providing an overview of a video's content with varying degrees of granularity.

We propose an algorithm for detection and tracking of small and fast moving objects, like a ping pong ball or a cricket ball, in sports video sequences. For detection, the proposed method uses only motion as a cue; moreover it does not use any texture information. Our method is able to detect the object with very low contrast and negligible texture content. Along with detection, we also propose a tracking algorithm using the multiple filter bank approach. Thus we provide a complete solution. The tracking algorithm is able to track maneuvering as well as non-maneuvering movements of the object without using any a priori information about the target dynamics

In many computer vision applications, machines will need to reason beyond the present, and predict the future. This task is challenging because it requires leveraging extensive commonsense knowledge of the world that is difficult to write down. We believe that a promising resource for efficiently obtaining this knowledge is through the massive amounts of readily available unlabeled video. In this paper, we present a large scale framework that capitalizes on temporal structure in unlabeled video to learn to anticipate both actions and objects in the future. The key idea behind our approach is that we can train deep networks to predict the visual representation of images in the future. We experimentally validate this idea on two challenging “in the wild” video datasets, and our results suggest that learning with unlabeled videos significantly helps forecast actions and anticipate objects.

A series of methods have been proposed to reconstruct an image from compressively sensed random measurement, Patch Vectorization CS Measurement

Nowadays, almost any web site that provides means for sharing user-generated multimedia content, like Flickr, Facebook, YouTube and Vimeo, has tagging functionalities to let users annotate the material that they want to share. The tags are then used to retrieve the uploaded content, and to ease browsing and exploration of these collections, e.g. using tag clouds. However, while tagging a single image is straightforward, and sites like Flickr and Facebook allow also to tag easily portions of the uploaded photos, tagging a video sequence is more cumbersome, so that users just tend to tag the overall content of a video. Moreover, the tagging process is completely manual, and often users tend to spend as few time as possible to annotate the material, resulting in a sparse annotation of the visual content. A semi-automatic process, that helps the users to tag a video sequence would improve the quality of annotations and thus the overall user experience. While research on image tagging has received a considerable attention in the latest years, there are still very few works that address the problem of automatically assigning tags to videos, locating them temporally within the video sequence. In this paper we present a system for video tag suggestion and temporal localization based on collective knowledge and visual similarity of frames. The algorithm suggests new tags that can be associated to a given keyframe exploiting the tags associated to videos and images uploaded to social sites like YouTube and Flickr and visual features.


Question: Pricing methods for IT services: a value-oriented perspective.

This paper investigates a cognitive shortcut, the availability heuristic, through which individuals assess the frequency of categories or the likelihood of occurrences based on how effortlessly pertinent examples spring to mind. Typically, availability mirrors ecological frequency, yet it can also be swayed by additional elements. Consequently, the dependence on the availability heuristic gives rise to predictable prejudices. These biases are illustrated in the estimated frequency of word categories, of combinatorial results, and of recurrent events. The occurrence of illusory correlation is elucidated as an availability bias. The paper also delves into the impact of the accessibility of events and scenarios on subjective probability assessments.

In relation to the question about pricing strategies for information technology services, the availability heuristic could influence how customers perceive the value of such services. For instance, if a service provider's recent marketing efforts have made their services more readily available in the customer's mind, this could lead to an overestimation of the service's frequency or probability of usage, potentially affecting perceived value and willingness to pay. Understanding these biases can assist in crafting value-based pricing strategies that align more closely with customer perceptions and expectations.

Information Technology Infrastructure Library, ITIL framework, is a set of comprehensive publications providing descriptive guidance on the management of IT processes, functions, roles and responsibilities related to Information Technology Service Management. However, and in spite of its repercussion and popularity, the ITIL framework does not suggest an implementation order for their processes. This decision constitutes the first challenge that an organization must overcome when starting an ITIL implementation, the enterprise size being one of the leading factors to be considered in the decision making process. In the scenario of Small and Medium Enterprises dedicated to producing software, this paper is devoted to investigating which processes are the most selected to start the implementation of ITIL in these organizations. This is done by means of two different instruments. Firstly, a systematic literature review on the topic and secondly, a survey conducted among experts and practitioners. Results show in both cases that Incident Management Process should be the first process when implementing ITIL framework.

Nowadays, almost all the enterprises in the world fall in Small Medium Enterprises (SMEs) category even though SMEs has different definitions in different countries. At the same time, there is no exemplar standard and /or framework for information technology (IT) governance for SMEs. This paper presents the main issues in implementing IT governance in SMEs. Firstly, it explains the definition of SMEs and their characteristics, secondly it discusses IT governance definition and its framework. Finally, some issues and approaches for ITG implementation in SMEs are described, and end with conclusion and future work.

Function point metrics are the most accurate and effective metrics yet developed for software sizing and also for studying software productivity, quality, costs, risks, and economic value. Unlike the older "lines of code" metric function points can be used to study requirements, design, and in fact all software activities from development through maintenance. In the future function point metrics can easily become a universal metric used for all software applications and for all software contracts in all countries. The government of Brazil already requires function points for all software contracts, and South Korea and Italy may soon follow. However, there are some logistical problems with function point metrics that need to be understood and overcome in order for function point metrics to become the primary metric for software economic analysis. Manual function point counting is too slow and costly to be used on large software projects above 10,000 function points in size. Also, application size is not constant but grows at about 2% per calendar month during development and 8% or more per calendar year for as long as software is in active use. This paper discusses a method of high-speed function point counting that can size any application in less than two minutes, and which can predict application growth during development and for five years after release. This new method is based on pattern matching and is covered by U.S. utility patent application and hence is patent pending.

T he rise and fall of the dot-com-driven Internet economy shouldn't distract us from seeing that the business environment continues to change at a dramatically increasing pace. To thrive in this turbulent environment, we must confront the business need for relentless innovation and forge the future workforce culture. Agile software development approaches such as Extreme Programming , Crystal methods, Lean Development, Scrum, Adaptive Software Development (ASD), and others view change from a perspective that mirrors today's turbulent business and technology environment. In a recent study of more than 200 software development projects, QSM Associates' Michael Mah reported that the researchers couldn't find nearly half of the projects' original plans to measure against. Why? Conforming to plan was no longer the primary goal; instead, satisfying customers—at the time of delivery , not at project initiation—took precedence. In many projects we review, major changes in the requirements, scope, and technology that are outside the development team's control often occur within a project's life span. Accepting that Barry Boehm's life cycle cost differentials theory—the cost of change grows through the software's development life cycle—remains valid, the question today is not how to stop change early in a project but how to better handle inevitable changes throughout its life cycle. Traditional approaches assumed that if we just tried hard enough, we could anticipate the complete set of requirements early and reduce cost by eliminating change. Today, eliminating change early means being unresponsive to business con-ditions—in other words, business failure. Similarly, traditional process manage-ment—by continuous measurement, error identification, and process refine-ments—strove to drive variations out of processes. This approach assumes that variations are the result of errors. Today, while process problems certainly cause some errors, external environmental changes cause critical variations. Because we cannot eliminate these changes, driving down the cost of responding to them is the only viable strategy. Rather than eliminating rework, the new strategy is to reduce its cost. However, in not just accommodating change, but embracing it, we also must be careful to retain quality. Expectations have grown over the years. The market demands and expects innovative, high-quality software that meets its needs— and soon. Agile methods are a response to this expectation. Their strategy is to reduce the cost of change throughout a project. Extreme Programming (XP), for example, calls for the software development team to • produce the first delivery in weeks, to achieve an early win and rapid …

O UTSOURCING is the contracting of various systems to outside information systems (IS) vendors. Ever since the Eastman Kodak--IBM partnership was reported in 1989 [15], outsourcing has emerged and has been recognized as a key method of managing IS. In previous studies of outsourcing [4, 5, 12, 15, 17, 20], two gaps are noticeable. The first relates to the fact that requirements for outsourcing are not uniform, and managers have different approaches to the process. Yet most previous research studies have phrased their inquiries unidimensionally in terms of the extent or degree of outsourcing: either as a binary variable or as size of contract in terms of the percentage of total IS budget [5, 15]. Neither unidimensional gauge is sufficiently expressive; neither allows representation of diverse patterns of outsourcing. Second, even though the IS industry had not used the term “outsourcing” explicitly in the past, outsourcing is not a new concept and has existed for many years in one form or another. Even though firms have repetitively used outside vendors for years, researchers have neither considered prior relationships in their studies of IS outsourcing nor studied the intentions of client firms to continue the partnership with the outsourcing vendors in the future. In this study, IS outsourcing decisions are investigated at two levels. The first level deals with the initial outsourcing decision of client firms. The second level pertains to the intention to continue the relationships with current outsourcing vendors in the future. The following three research questions are explored at these two levels (in contrast to the singlelevel approach taken by most researchers): • What are the dimensions of outsourcing decisions? Two dimensions, extent of substitution by vendors and strategic impact of IS applications, are proposed in order to conceptualize the diverse types of outsourcing relationships between clients and vendors. Based on these two dimensions, four types of outsourcing relationships are proposed. • What are the determinants that affect the dimensions of outsourcing decisions at the first level? The concepts derived from incomplete contracts [2, 9] and transactions cost economics [23, 24] theory are used with information technology (IT) organizational contexts and processes as a foundation to study the determinants of the two dimensions of outsourcing decisions. Information Syst

The first € price and the £ and $ price are net prices, subject to local VAT. Prices indicated with * include VAT for books; the €(D) includes 7% for Germany, the €(A) includes 10% for Austria. Prices indicated with ** include VAT for electronic products; 19% for Germany, 20% for Austria. All prices exclusive of carriage charges. Prices and other details are subject to change without notice. All errors and omissions excepted. C. Bishop Pattern Recognition and Machine Learning

The Context Toolkit (Dey, Salber, and Abowd 2001 [this special issue]) is only one of many possible architectures for supporting context-aware applications. In this essay, we look at the trade-offs involved with a service infrastructure approach to context-aware computing. We describe the advantages that a service infrastructure for contextawareness has over other approaches, outline some of the core technical challenges that must be addressed before such an infrastructure can be built, and point out promising research directions for overcoming these challenges.

The resource-based view of the firm (RBV) hypothesizes that the exploitation of valuable, rare resources and capabilities contributes to a firm’s competitive advantage, which in turn contributes to its performance. Despite this notion, few empirical studies test these hypotheses at the conceptual level. In response to this gap, this study empirically examines the relationships between value, rareness, competitive advantage, and performance. The results suggest that value and rareness are related to competitive advantage, that competitive advantage is related to performance, and that competitive advantage mediates the rareness-performance relationship. These findings have important academic and practitioner implications which are then discussed. Copyright  2008 John Wiley & Sons, Ltd.

This keynote paper: presents a 21st century vision of computing; identifies various computing paradigms promising to deliver the vision of computing utilities; defines Cloud computing and provides the architecture for creating market-oriented Clouds by leveraging technologies such as VMs; provides thoughts on market-based resource management strategies that encompass both customer-driven service management and computational risk management to sustain SLA-oriented resource allocation; presents some representative Cloud platforms especially those developed in industries along with our current work towards realising market-oriented resource allocation of Clouds by leveraging the 3rd generation Aneka enterprise Grid technology; reveals our early thoughts on interconnecting Clouds for dynamically creating an atmospheric computing environment along with pointers to future community research; and concludes with the need for convergence of competing IT paradigms for delivering our 21st century vision.

Cloud services have emerged as an innovative IT provisioning model in the recent years. However, after their usage severe considerations have emerged with regard to their varying performance due to multitenancy and resource sharing issues. These issues make it very difficult to provide any kind of performance estimation during application design or deployment time. The aim of this paper is to present a mechanism and process for measuring the performance of various Cloud services and describing this information in machine understandable format. The framework is responsible for organizing the execution and can support multiple Cloud providers. Furthermore we present approaches for measuring service performance with the usage of specialized metrics for ranking the services according to a weighted combination of cost, performance and workload.