Question: Design of a low-power, high-speed CMOS comparator for application in A/D converter technology.

This paper describes the design and implementation of a high-speed, low-power CMOS comparator circuit, specifically tailored for application in analog-to-digital converters (ADCs). The comparator circuit is composed of a preamplifier, a latch stage, and a dynamic latch serving as an output sampler. The output sampler is innovatively designed with a full transmission gate (TG) and two inverters, which significantly reduces power consumption in high-speed comparator circuits. Simulations indicate that the charge injection from the TG constructively enhances the sampled signal, amplifying it with a gain of 1.15. When this gain is coupled with the high gain from the inverters, the sampled signals are amplified to approach the rail voltages, enhancing the comparator's performance. The comparator has been designed and manufactured using a standard 0.35 μm digital CMOS technology. Experimental results demonstrate a sampling frequency of 1 GHz, with a 16 mV resolution for a 1 V input signal range, all while consuming only 2 mW of power from a 3.3 V supply. This design is scalable, suitable for reduction to smaller feature sizes and operation at lower supply voltages.

A novel sensing current protection enhanced (SCPE) technique and high-voltage (HV) generating systems are adopted to greatly enhance the advantages of the twin-bit cell. The SCPE technique provides the tradeoff of sensing margin loss between “1” and “0” bits sensing case to realize fast read access. The read speed is improved by 7.7%. HV generating systems with parallel-series-transform and capacitance-shared techniques are proposed to satisfy the complicated requirements of output capability and current drivability with lower area penalty. The HV periphery area decreases by 71%. A 1.5-V 280-KBytes embedded NOR flash memory IP has been fabricated in HHGrace (Shanghai Huahong Grace Semiconductor Manufacturing Corporation) 90-nm 4 poly 4 metal CMOS process. The complicated operation voltages and access time of 23.5 ns have been achieved at 1.5 V. The die size of the proposed IP is 0.437 mm<inline-formula> <tex-math notation="LaTeX">$^{2}$ </tex-math></inline-formula> and the area size of charge pump has been reduced to 0.006 mm<inline-formula> <tex-math notation="LaTeX">$^{2}$ </tex-math></inline-formula> on average.

In this paper, we present the design of a new very-high output impedance CMOS current mirror with enhanced output voltage compliance. The proposed current mirror uses MOS current dividers to sample the output current and a feedback action is used to force it to be equal to the input current; yielding very high impedance with a very large output voltage range. The proposed implementation yields an increase of the output impedance by a factor of about gmro compared with that of the super-Wilson current mirror, thus offering a potential solution to mitigate the effect of the low output impedance of ultra-deep submicron CMOS transistors used in sub 1-V current mirrors and current sources. A NMOS version of the proposed current mirror circuit was implemented using STMicroelectronics 1-V 90-nm CMOS process and simulated using Spectre to validate its performance. The output current is mirrored with a transfer error lower than 1% down to an output voltage as low as 80 mV for an input current of 5 muA, and 111 mV when the input current is increased to 50 muA.

An ultra low-power, precise voltage reference using a switched-capacitor technique in 0.35-μm CMOS is presented in this paper. The temperature dependence of the carrier mobility and channel length modulation effect can be effectively minimized by using 3.3 and 5 V N-type transistors to operate in the saturation and subthreshold regions, respectively. In place of resistors, a precise reference voltage with flexible trimming capability is achieved by using capacitors. When the supply voltage is 1 V and the temperature is 80°C, the supply current is 250 nA. The line sensitivity is 0.76%/V; the PSRR is -41 dB at 100 Hz and -17 dB at 10 MHz. Moreover, the occupied die area is 0.049 mm2.

This paper describes a simple architecture and low power consumption undervoltage lockout (UVLO) circuit with hysteretic threshold. The UVLO circuit monitors the supply voltage and determines whether or not the supply voltage satisfies a predetermined condition. The under voltage lockout circuit is designed based on CSMC 0.5um CMOS technology, utilizing a relatively few amount of circuitry. It is realized with a current source inverter. The threshold voltage is determined by the W/L ratio of current source inverter and resistor in reference generator. The hysteresis is realized by using a feedback circuit to overcome the bad disturbance and noise rejection of the single threshold. Hysteretic threshold range is 40mV. The quiescent current is about 1uA at 3V supply voltage,while the power of circuit consumes only 3uW.

“Some Design Aspects of a Two-Stage Rail-to-Rail CMOS Op Amp' by Sander L. J. Gierkink, Peter J. Holzmann, Remco J. Wiegerink and Roelof F. Wassenaar, Analog Integrated Circuits and Signal Processing, Vol. 21, No. 2, Nov. 1999, pp 143-152. “Compact Low-Voltage Power-Efficient Operational Amplifier Cells for VLSI by Klaas-Jan de Langen and Johan H. Huijsing, IEEE Journal of Solid State Circuits, vol. 33, No. 10, Oct. 1998, pp. 1482–1496. “Design Aspects of a Rail-to-Rail CMOS Op Amp' by Glierkink et al., Mesa Research institute, ECT-97-36, pp. 23-28.

A low-power CMOS voltage reference was developed using a 0.35 m standard CMOS process technology. The device consists of MOSFET circuits operated in the subthreshold region and uses no resistors. It generates two voltages having opposite temperature coefficients and adds them to produce an output voltage with a near-zero temperature coefficient. The resulting voltage is equal to the extrapolated threshold voltage of a MOSFET at absolute zero temperature, which was about 745 mV for the MOSFETs we used. The temperature coefficient of the voltage was 7 ppm/ C at best and 15 ppm/ C on average, in a range from 20 to 80 C. The line sensitivity was 20 ppm/V in a supply voltage range of 1.4–3 V, and the power supply rejection ratio (PSRR) was 45 dB at 100 Hz. The power dissipation was 0.3 W at 80 C. The chip area was 0.05 mm . Our device would be suitable for use in subthreshold-operated, power-aware LSIs.

This paper presents an approach to the design of a high-precision CMOS voltage reference. The proposed circuit is designed for TSMC 0.35 m standard CMOS process. We design the ̄rstorder temperature compensation bandgap voltage reference circuit. The proposed post-simulated circuit delivers an output voltage of 0.596V and achieves the reported temperature coe±cient (TC) of 3.96 ppm/ C within the temperature range from 60 C to 130 C when the supply voltage is 1.8V. When simulated in a smaller temperature range from 40 C to 80 C, the circuit achieves the lowest reported TC of 2.09 ppm/ C. The reference current is 16.586 A. This circuit provides good performances in a wide range of temperature with very small TC.

This paper describes a transmitter driver based on a CMOS inverter with a resistive feedback. By employing the proposed driver topology, the pre-driver can be greatly simplified, resulting in a remarkable reduction of the overall driver power consumption. It also offers another advantage that the implementation of equalization is straightforward, compared with a conventional voltage-mode driver. Furthermore, the output impedance remains relatively constant while the data is being transmitted, resulting in good signal integrity. For evaluation of the driver performance, a fully functional 20 Gb/s transmitter is implemented, including a PRBS generator, a serializer, and a half-rate clock generator. In order to enhance the overall speed of the digital circuits for 20 Gb/s data transmission, the resistive feedback is applied to the time-critical inverters, which enables shorter rise/fall times. The prototype chip is fabricated in a 65 nm CMOS technology. The implemented driver circuit operates up to the data rate of 20 Gb/s, exhibiting an energy efficiency of 0.4 pJ/b for the output swing of 250 mVpp,diff.

This paper presents an energy-efficient SAR ADC which adopts reduced MSB cycling step with dual sampling of the analog signal. By sampling and holding the analog signal asymmetrically at both input sides of comparator, the MSB cycling step can be hidden by hold mode. Benefits from this technique, not only the total capacitance of DAC is reduced by half, but also the average switching energy is reduced by 68% compared with conventional SAR ADC. Moreover, switching energy distribution is more uniform over entire output code compared with previous works.

This paper introduces a new low voltage sub-nanosecond monolithic pulsed current driver for light detection and ranging (LIDAR) applications. Unique architecture based on a controlled current source and Vernier activation sequence, combined with a monolithic implementation that allows operation with low input voltage levels, high-resolution pulse width and sub-nanosecond rise and fall times. An on-chip low voltage pulsed driver sub-nanosecond prototype has been implemented in a TS 0.18-μm 5V-gated power management process. It incorporates an integrated wide range sesnseFET based current sensor and a rail-to-rail comparator for current regulation. To characterize the avalanche capabilities of the integrated lateral MOSFET power devices required for the driver IC, a separate line of investigation has been carried out. Several lateral diffused MOS (LDMOS) power devices have been custom designed and experimentally evaluated for a life-cycle performance characterization. Post-layout analysis of the power driver IC is in a good agreement with the theoretical predictions. For a 5V input voltage, rise and fall times of the laser pulse light output are on the order of hundreds of picoseconds, with currents up to 5A. To validate the concept of high-resolution pulse width generation and short fall time, a discrete prototype has been constructed and experimentally tested.


Question: Enhanced keyword spotting utilizing keyword/garbage model improvements.

This paper introduces an enhanced keyword spotting system designed for minimal memory usage, efficient processing, and exceptional accuracy. To accomplish this, we adopt a streamlined methodology centered around deep neural networks. The proposed system trains a deep neural network to accurately forecast the keyword(s) or constituent subword components of the keyword(s), complemented by a posterior handling mechanism that generates a definitive confidence score. The outcomes demonstrate a 45% relative enhancement in keyword recognition accuracy compared to a state-of-the-art Hidden Markov Model-based system. Furthermore, the system's robustness in noisy environments is evident, with a 39% relative improvement in performance under babble noise conditions, showcasing its superior adaptability and effectiveness in real-world scenarios.

Recently Long Short-Term Memory (LSTM) Recurrent Neural Networks (RNN) acoustic models have demonstrated superior performance over deep neural networks (DNN) models in speech recognition and many other tasks. Although a lot of work have been reported on DNN model adaptation, very little has been done on LSTM model adaptation. In this paper we present our extensive studies of speaker adaptation of LSTM-RNN models for speech recognition. We investigated different adaptation methods combined with KL-divergence based regularization, where and which network component to adapt, supervised versus unsupervised adaptation and asymptotic analysis. We made a few distinct and important observations. In a large vocabulary speech recognition task, by adapting only 2.5% of the LSTM model parameters using 50 utterances per speaker, we obtained 12.6% WERR on the dev set and 9.1% WERR on the evaluation set over a strong LSTM baseline model.

Exploring Deep Learning Methods for discovering features in speech signals. Navdeep Jaitly Doctor of Philosophy Graduate Department of Computer Science University of Toronto 2014 This thesis makes three main contributions to the area of speech recognition with Deep Neural Network Hidden Markov Models (DNN-HMMs). Firstly, we explore the effectiveness of features learnt from speech databases using Deep Learning for speech recognition. This contrasts with prior works that have largely confined themselves to using traditional features such as Mel Cepstral Coefficients and Mel log filter banks for speech recognition. We start by showing that features learnt on raw signals using Gaussian-ReLU Restricted Boltzmann Machines can achieve accuracy close to that achieved with the best traditional features. These features are, however, learnt using a generative model that ignores domain knowledge. We develop methods to discover features that are endowed with meaningful semantics that are relevant to the domain using capsules. To this end, we extend previous work on transforming autoencoders and propose a new autoencoder with a domain-specific decoder to learn capsules from speech databases. We show that capsule instantiation parameters can be combined with Mel log filter banks to produce improvements in phone recognition on TIMIT. On WSJ the word error rate does not improve, even though we get strong gains in classification accuracy. We speculate this may be because of the mismatched objectives of word error rate over an utterance and frame error rate on the sub-phonetic class for a frame. Secondly, we develop a method for data augmentation in speech datasets. Such methods result in strong gains in object recognition, but have largely been ignored in speech recognition. Our data augmentation encourages the learning of invariance to vocal tract length of speakers. The method is shown to improve the phone error rate on TIMIT and the word error rate on a 14 hour subset of WSJ. Lastly, we develop a method for learning and using a longer range model of targets, conditioned on the input. This method predicts the labels for multiple frames together and uses a geometric average of these predictions during decoding. It produces state of the art results on phone recognition with TIMIT and also produces significant gains on WSJ.

Text-independent speaker verification (SV) is currently in the process of embracing DNN modeling in every stage of SV system. Slowly, the DNN-based approaches such as end-to-end modelling and systems based on DNN embeddings start to be competitive even in challenging and diverse channel conditions of recent NIST SREs. Domain adaptation and the need for a large amount of training data are still a challenge for current discriminative systems and (unlike with generative models), we see significant gains from data augmentation, simulation and other techniques designed to overcome lack of training data. We present an analysis of a SV system based on DNN embeddings (x-vectors) and focus on robustness across diverse data domains such as standard telephone and microphone conversations, both in clean, noisy and reverberant environments. We also evaluate the system on challenging far-field data created by re-transmitting a subset of NIST SRE 2008 and 2010 microphone interviews. We compare our results with the stateof-the-art i-vector system. In general, we were able to achieve better performance with the DNN-based systems, but most importantly, we have confirmed the robustness of such systems across multiple data domains.

We address the problem of query segmentation: given a keyword query, the task is to group the keywords into phrases, if possible. Previous approaches to the problem achieve reasonable segmentation performance but are tested only against a small corpus of manually segmented queries. In addition, many of the previous approaches are fairly intricate as they use expensive features and are difficult to be reimplemented.
 The main contribution of this paper is a new method for query segmentation that is easy to implement, fast, and that comes with a segmentation accuracy comparable to current state-of-the-art techniques. Our method uses only raw web n-gram frequencies and Wikipedia titles that are stored in a hash table. At the same time, we introduce a new evaluation corpus for query segmentation. With about 50,000 human-annotated queries, it is two orders of magnitude larger than the corpus being used up to now.

Performance variabilit y in speech and speaker recognition systems can be attributed to many factors. One major factor, which is often acknowledged but seldom analyzed, is inherent differences in the recognizabilit y of different speakers. In speaker recognition systems such differences are characterized by the use of animal names for different types of speakers, including sheep, goats, lambs and wolves, depending on their behavior with respect to automatic recognition systems. In this paper we propose statistical tests for the existence of these animals and apply these tests to hunt for such animals using results from the 1998 NIST speaker recognition evaluation.

Spoken language recognition refers to the automatic process through which we determine or verify the identity of the language spoken in a speech sample. We study a computational framework that allows such a decision to be made in a quantitative manner. In recent decades, we have made tremendous progress in spoken language recognition, which benefited from technological breakthroughs in related areas, such as signal processing, pattern recognition, cognitive science, and machine learning. In this paper, we attempt to provide an introductory tutorial on the fundamentals of the theory and the state-of-the-art solutions, from both phonological and computational aspects. We also give a comprehensive review of current trends and future research directions using the language recognition evaluation (LRE) formulated by the National Institute of Standards and Technology (NIST) as the case studies.

We report automatic speech recognition accuracy for individual words using eleven surface electromyographic (sEMG) recording locations on the face and neck during three speaking modes: vocalized, mouthed, and mentally rehearsed. An HMM based recognition system was trained and tested on a 65 word vocabulary produced by 9 American English speakers in all three speaking modes. Our results indicate high sEMG-based recognition accuracy for the vocalized and mouthed speaking modes (mean rates of 92.1% and 86.7% respectively), but an inability to conduct recognition on mentally rehearsed speech due to a lack of sufficient sEMG activity.

This paper presents a novel audio fingerprinting method that is highly robust to a variety of audio distortions. It is based on an unconventional audio fingerprint generation scheme. The robustness is achieved by generating different versions of the spectrogram matrix of the audio signal by using a threshold based on the average of the spectral values to prune this matrix. We transform each version of this pruned spectrogram matrix into a 2-D binary image. Multiple versions of these 2-D images suppress noise to a varying degree. This varying degree of noise suppression improves likelihood of one of the images matching a reference image. To speed up matching, we convert each image into an n-dimensional vector, and perform a nearest neighbor search based on this n-dimensional vector. We give results with two different feature parameters and their combination. We test this method on TRECVID 2010 content-based copy detection evaluation dataset, and we validate the performance on TRECVID 2009 dataset also. Experimental results show the effectiveness of these features even when the audio is distorted. We compare the proposed method to two state-of-the-art audio copy detection systems, namely NN-based and Shazam systems. Our method by far outperforms Shazam system for all audio transformations (or distortions) in terms of detection performance, number of missed queries and localization accuracy. Compared to NN-based system, our approach reduces minimal Normalized Detection Cost Rate (min NDCR) by 23 % and improves localization accuracy by 24 %.

Long short-term memory (LSTM) based acoustic modeling methods have recently been shown to give state-of-the-art performance on some speech recognition tasks. To achieve a further performance improvement, in this research, deep extensions on LSTM are investigated considering that deep hierarchical model has turned out to be more efficient than a shallow one. Motivated by previous research on constructing deep recurrent neural networks (RNNs), alternative deep LSTM architectures are proposed and empirically evaluated on a large vocabulary conversational telephone speech recognition task. Meanwhile, regarding to multi-GPU devices, the training process for LSTM networks is introduced and discussed. Experimental results demonstrate that the deep LSTM networks benefit from the depth and yield the state-of-the-art performance on this task.

In this paper we discuss the recent evolution of spoken dialog systems in commercial deployments. Yet based on a simple finite state machine design paradigm, dialog systems reached today a higher level of complexity. The availability of massive amounts of data during deployment led to the development of continuous optimization strategy pushing the design and development of spoken dialog applications from an art to science. At the same time new methods for evaluating the subjective caller experience are available. Finally we describe the inevitable evolution for spoken dialog applications from speech only to multimodal interaction.


Question: Enhanced keyword spotting utilizing keyword/garbage model refinements.

This paper investigates the use of Convolutional Neural Networks (CNNs) for efficient keyword spotting (KWS) systems with minimal computational footprint. CNNs are particularly advantageous for KWS tasks due to their superior performance relative to Deep Neural Networks (DNNs), despite requiring significantly fewer parameters. Our research focuses on two scenarios: one in which the number of multiplications in the KWS system is restricted, and another in which the number of parameters is constrained. We introduce novel CNN architectures tailored to the specific requirements of each scenario. Our findings indicate that these CNN-based systems outperform DNNs by achieving a 27-44% relative reduction in false reject rate, all while adhering to the stringent limitations imposed by each application. This highlights the potential of CNNs in enhancing the efficiency and effectiveness of small-footprint keyword spotting systems.

Attention-based sequence-to-sequence models for speech recognition jointly train an acoustic model, language model (LM), and alignment mechanism using a single neural network and require only parallel audio-text pairs. Thus, the language model component of the end-to-end model is only trained on transcribed audio-text pairs, which leads to performance degradation especially on rare words. While there have been a variety of work that look at incorporating an external LM trained on text-only data into the end-to-end framework, none of them have taken into account the characteristic error distribution made by the model. In this paper, we propose a novel approach to utilizing text-only data, by training a spelling correction (SC) model to explicitly correct those errors. On the LibriSpeech dataset, we demonstrate that the proposed model results in an 18.6% relative improvement in WER over the baseline model when directly correcting top ASR hypothesis, and a 29.0% relative improvement when further rescoring an expanded n-best list using an external LM.

Efforts to develop new countermeasures in order to protect automatic speaker verification from spoofing have intensified over recent years. The ASVspoof 2015 initiative showed that there is great potential to detect spoofing attacks, but also that the detection of previously unforeseen spoofing attacks remains challenging. This paper argues that there is more to be gained from the study of features rather than classifiers and introduces a new feature for spoofing detection based on the constant Q transform, a perceptually-inspired time-frequency analysis tool popular in the study of music. Experimental results obtained using the standard ASVspoof 2015 database show that, when coupled with a standard Gaussian mixture model-based classifier, the proposed constant Q cepstral coefficients (CQCCs) outperform all previously reported results by a significant margin. In particular, those for a subset of unknown spoofing attacks (for which no matched training data was used) is 0.46%, a relative improvement of 72% over the best, previously reported results.

This paper examines whether visual speech information can be effective within audio-masking-based speaker separation to improve the quality and intelligibility of the target speech. Two visual-only methods of generating an audio mask for speaker separation are first developed. These use a deep neural network to map the visual speech features to an audio feature space from which both visually derived binary masks and visually derived ratio masks are estimated, before application to the speech mixture. Second, an audio ratio masking method forms a baseline approach for speaker separation which is extended to exploit visual speech information to form audio-visual ratio masks. Speech quality and intelligibility tests are carried out on the visual-only, audio-only, and audio-visual masking methods of speaker separation at mixing levels from $-$ 10 to +10 dB. These reveal substantial improvements in the target speech when applying the visual-only and audio-only masks, but with highest performance occurring when combining audio and visual information to create the audio-visual masks.

Current speech enhancement techniques operate on the spectral domain and/or exploit some higher-level feature. The majority of them tackle a limited number of noise conditions and rely on first-order statistics. To circumvent these issues, deep networks are being increasingly used, thanks to their ability to learn complex functions from large example sets. In this work, we propose the use of generative adversarial networks for speech enhancement. In contrast to current techniques, we operate at the waveform level, training the model end-to-end, and incorporate 28 speakers and 40 different noise conditions into the same model, such that model parameters are shared across them. We evaluate the proposed model using an independent, unseen test set with two speakers and 20 alternative noise conditions. The enhanced samples confirm the viability of the proposed model, and both objective and subjective evaluations confirm the effectiveness of it. With that, we open the exploration of generative architectures for speech enhancement, which may progressively incorporate further speech-centric design choices to improve their performance.

This paper proposes a segmental conditional random field framework for large vocabulary continuous speech recognition. Fundamental to this approach is the use of acoustic detectors as the basic input, and the automatic construction of a versatile set of segment-level features. The detector streams operate at multiple time scales (frame, phone, multi-phone, syllable or word) and are combined at the word level in the CRF training and decoding processes. A key aspect of our approach is that features are defined at the word level, and are naturally geared to explain long span phenomena such as formant trajectories, duration, and syllable stress patterns. Generalization to unseen words is possible through the use of decomposable consistency features [1], [2], and our framework allows for the joint or separate discriminative training of the acoustic and language models. An initial evaluation of this framework with voice search data from the Bing Mobile (BM) application results in a 2% absolute improvement over an HMM baseline.

The use of Deep Belief Networks (DBN) to pretrain Neural Networks has recently led to a resurgence in the use of Artificial Neural Network Hidden Markov Model (ANN/HMM) hybrid systems for Automatic Speech Recognition (ASR). In this paper we report results of a DBN-pretrained context-dependent ANN/HMM system trained on two datasets that are much larger than any reported previously with DBN-pretrained ANN/HMM systems 5870 hours of Voice Search and 1400 hours of YouTube data. On the first dataset, the pretrained ANN/HMM system outperforms the best Gaussian Mixture Model Hidden Markov Model (GMM/HMM) baseline, built with a much larger dataset by 3.7% absolute WER, while on the second dataset, it outperforms the GMM/HMM baseline by 4.7% absolute. Maximum Mutual Information (MMI) fine tuning and model combination using Segmental Conditional Random Fields (SCARF) give additional gains of 0.1% and 0.4% on the first dataset and 0.5% and 0.9% absolute on the second dataset.

This paper presents a study of the noise suppression technique proposed by Ephraim and Malah. This technique has been used recently for the restoration of degraded audio recordings because it is free of the frequently encountered ‘musical noise’ artifact. It is demonstrated how this artifact is actually eliminated without bringing distortion to the recorded signal even if the noise is only poorly stationary.

The number of natural language queries submitted to search engines is increasing as search environments get diversified. However, legacy search engines are still optimized for short keyword queries. Thus, the use of natural language queries at legacy search engines degrades the retrieval performance of the engines. This paper proposes a novel method to translate a natural language query into a keyword query relevant to the natural language query for retrieving better search results without change of the engines. The proposed method formulates the translation as a generation task. That is, the method generates a keyword query from a natural language query by preserving the semantics of the natural language query. A recurrent neural network encoder-decoder architecture is adopted as a generator of keyword queries from natural language queries. In addition, an attention mechanism is also used to cope with long natural language queries.

In this paper, we present an algorithm for extracting keywords representing the asserted main point in a document, without relying on external devices such as natural language processing tools or a document corpus. Our algorithm KeyGraph is based on the segmentation of a graph, representing the co-occurrence between terms in a document, into clusters. Each cluster corresponds to a concept on which author's idea is based, and top ranked terms by a statistic based on each term's relationship to these clusters are selected as keywords. This strategy comes from considering that a document is constructed like a building for expressing new ideas based on traditional concepts. The experimental results show that thus extracted terms match author's point quite accurately, even though KeyGraph does not use each term's average frequency in a corpus, i.e., KeyGraph is a contentsensitive, domain independent device of indexing.

Whispered speech is a natural mode of speech in which voicing is absent – its acoustics differ significantly from normally spoken speech or so-called neutral speech, such that it is challenging to use only neutral speech to build speech processing and automatic recognition systems that can deal effectively with whisper. At the same time, humans can naturally produce and perceive whispered speech without explicit training. Tonal languages such as Mandarin present an interesting dilemma – tone is primarily encoded by pitch tracks which are absent during whispered speech, but humans can still tell tones apart. How humans manage to process whispered speech well without explicit training on it, whereas machine algorithms fail, is presently an unresolved question which could prove fruitful with study. This, however, is hindered by the lack of suitable, systematically collected corpora. We present iWhisper-Mandarin, a 25-hour parallel corpus of neutral and whispered Mandarin, designed to support research in linguistics and speech technology. We demonstrate and verify that earlier techniques applied to whispered speech from non-tonal languages also work with Mandarin, and present some preliminary studies on voice activity detection and whispered Mandarin speech recognition.


Question: Enhanced keyword detection utilizing keyword/garbage model refinement.

In the realm of unsupervised feature learning and deep learning, recent advancements have highlighted the significant impact of training large models on performance improvement. This paper addresses the challenge of training deep networks with an expansive parameter space, comprising billions of parameters, utilizing tens of thousands of CPU cores. To tackle this issue, we have engineered a software framework, DistBelief, designed to harness the power of computing clusters with thousands of machines for the training of extensive models.

Within the DistBelief framework, we have devised two algorithms tailored for large-scale distributed training: (i) Downpour SGD, an asynchronous stochastic gradient descent procedure that facilitates a high number of model replicas, and (ii) Sandblaster, a versatile framework supporting a range of distributed batch optimization methods, including a distributed implementation of L-BFGS. Both Downpour SGD and Sandblaster L-BFGS enhance the scale and velocity of deep network training, enabling the training of a deep network 30 times larger than previously documented in the literature. This achievement is underscored by state-of-the-art performance on ImageNet, a comprehensive visual object recognition task featuring 16 million images and 21,000 categories.

Moreover, we demonstrate that these methodologies significantly expedite the training of a comparably smaller deep network for a commercial speech recognition application. While our focus is on the application of these methods to the training of large neural networks, the underlying algorithms are universally applicable to any gradient-based machine learning algorithm.

In the context of keyword spotting, these advancements in distributed deep networks could potentially revolutionize the approach to keyword/garbage model-based spotting, enabling the processing of vast datasets and the development of more sophisticated models. The enhanced scale and speed of deep network training, facilitated by Downpour SGD and Sandblaster L-BFGS, could translate into more accurate and responsive keyword spotting systems, especially in real-time applications where large volumes of data need to be processed efficiently. The adaptability of these algorithms to various gradient-based machine learning techniques further broadens their applicability in enhancing keyword spotting mechanisms.

The use of hidden Markov models (HMMs) in continuous speech recognition is reviewed. Markov models are presented as a generalization of their predecessor technology, dynamic programming. A unified view is offered in which both linguistic decoding and acoustic matching are integrated into a single, optimal network search framework. Advances in recognition architectures are discussed. The fundamentals of Viterbi beam search, the dominant search algorithm used today in speed recognition, are presented. Approaches to estimating the probabilities associated with an HMM model are examined. The HMM-supervised training paradigm is examined. Several examples of successful HMM-based speech recognition systems are reviewed.<<ETX>>

ÐThis paper describes a hidden Markov model-based approach designed to recognize off-line unconstrained handwritten words for large vocabularies. After preprocessing, a word image is segmented into letters or pseudoletters and represented by two feature sequences of equal length, each consisting of an alternating sequence of shape-symbols and segmentationsymbols, which are both explicitly modeled. The word model is made up of the concatenation of appropriate letter models consisting of elementary HMMs and an HMM-based interpolation technique is used to optimally combine the two feature sets. Two rejection mechanisms are considered depending on whether or not the word image is guaranteed to belong to the lexicon. Experiments carried out on real-life data show that the proposed approach can be successfully used for handwritten word recognition. Index TermsÐHandwriting modeling, preprocessing, segmentation, feature extraction, hidden Markov models, word recognition, rejection.

This paper presents a model to convert natural Bengali language to text. The proposed model requires the usage of the open sourced framework Sphinx 4 which is written in Java and provides the required procedural coding tools to develop an acoustic model for a custom language like Bengali. Our main objective was to ensure that the system was adequately trained on a word by word basis from various speakers so that it could recognize new speakers fluently. We used a free digital audio workstation (DAW) called Audacity to manipulate the collected recording data via continuous frequency profiling techniques to reduce the Signal-to-Noise-Ratio (SNR), vocal leveling, normalization and syllable splitting as well as merging which ensure an error free 1:1-word mapping of each utterance with its mirror transcription file text. To evaluate the performance of proposed model, we utilize an audio dataset of recorded speech data from 10 individual speakers consisting of both males and females using custom transcript files that we wrote. Experimental results demonstrate that the proposed model exhibits average 71.7% accuracy for our tested dataset.

Keywords in the ACM Digital Library and IEEE Xplore digital library and in NSF grants anticipate future CS research.

Accessing the wealth of structured data available on the Data Web is still a key challenge for lay users. Keyword search is the most convenient way for users to access information (e.g., from data repositories). In this paper we introduce a novel approach for determining the correct resources for user-supplied keyword queries based on a hidden Markov model. In our approach the user-supplied query is modeled as the observed data and the background knowledge is used for parameter estimation. Instead of learning parameter estimation from training data, we leverage the semantic relationships between data items for computing the parameter estimations. In order to maximize accuracy and usability, query segmentation and resource disambiguation are mutually tightly interwoven. First, an initial set of potential segmentations is obtained leveraging the underlying knowledge base; then the final correct set of segments is determined after the most likely resource mapping was computed using a scoring function. While linguistic methods like named entity, multi-word unit recognition and POS-tagging fail in the case of an incomplete sentences (e.g. for keyword-based queries), we will show that our statistical approach is robust with regard to query expression variance. Our experimental results when employing the hidden Markov model for resource identification in keyword queries reveal very promising results.

Search engine logs are an emerging new type of data that offers interesting opportunities for data mining. Existing work on mining such data has mostly attempted to discover knowledge at the level of queries (e.g., query clusters). In this paper, we propose to mine search engine logs for patterns at the level of terms through analyzing the relations of terms inside a query. We define two novel term association patterns (i.e., context-sensitive term substitutions and term additions) and propose new methods for mining such patterns from search engine logs. These two patterns can be used to address the mis-specification and under-specification problems of ineffective queries. Experiment results on real search engine logs show that the mined context-sensitive term substitutions can be used to effectively reword queries and improve their accuracy, while the mined context-sensitive term addition patterns can be used to support query refinement in a more effective way.

To test if simultaneous spectral and temporal processing is required to extract robust features for automatic speech recognition (ASR), the robust spectro-temporal two-dimensional-Gabor filter bank (GBFB) front-end from Schädler, Meyer, and Kollmeier [J. Acoust. Soc. Am. 131, 4134-4151 (2012)] was de-composed into a spectral one-dimensional-Gabor filter bank and a temporal one-dimensional-Gabor filter bank. A feature set that is extracted with these separate spectral and temporal modulation filter banks was introduced, the separate Gabor filter bank (SGBFB) features, and evaluated on the CHiME (Computational Hearing in Multisource Environments) keywords-in-noise recognition task. From the perspective of robust ASR, the results showed that spectral and temporal processing can be performed independently and are not required to interact with each other. Using SGBFB features permitted the signal-to-noise ratio (SNR) to be lowered by 1.2 dB while still performing as well as the GBFB-based reference system, which corresponds to a relative improvement of the word error rate by 12.8%. Additionally, the real time factor of the spectro-temporal processing could be reduced by more than an order of magnitude. Compared to human listeners, the SNR needed to be 13 dB higher when using Mel-frequency cepstral coefficient features, 11 dB higher when using GBFB features, and 9 dB higher when using SGBFB features to achieve the same recognition performance.

We present our winning solution for the WSDM Cup 2017 triple scoring task. We devise an ensemble of four base scorers, so as to leverage the power of both text and knowledge bases for that task. Then we further refine the outputs of the ensemble by trigger word detection, achieving even better predictive accuracy. The code is available at https://github.com/wsdm-cup-2017/bokchoy.

Automatic key phrase extraction has many important applications including but not limited to summarization, cataloging/indexing, feature extraction for clustering and classification, and data mining. This paper presents the KP-Miner system, and demonstrates through experimentation and comparison with existing systems that the it is effective in extracting keyphrases from both English and Arabic documents of varied length. Unlike other existing keyphrase extraction systems, the KP-Miner system has the advantage of being configurable as the rules and heuristics adopted by the system are related to the general nature of documents and keyphrase. This implies that the users of this system can use their understanding of the document(s) being input into the system, to fine tune it to their particular needs.

We propose a novel regularized adaptation technique for context dependent deep neural network hidden Markov models (CD-DNN-HMMs). The CD-DNN-HMM has a large output layer and many large hidden layers, each with thousands of neurons. The huge number of parameters in the CD-DNN-HMM makes adaptation a challenging task, esp. when the adaptation set is small. The technique developed in this paper adapts the model conservatively by forcing the senone distribution estimated from the adapted model to be close to that from the unadapted model. This constraint is realized by adding Kullback-Leibler divergence (KLD) regularization to the adaptation criterion. We show that applying this regularization is equivalent to changing the target distribution in the conventional backpropagation algorithm. Experiments on Xbox voice search, short message dictation, and Switchboard and lecture speech transcription tasks demonstrate that the proposed adaptation technique can provide 2%-30% relative error reduction against the already very strong speaker independent CD-DNN-HMM systems using different adaptation sets under both supervised and unsupervised adaptation setups.


Question: Enhanced keyword detection utilizing keyword/garbage model refinements.

This study evaluates the effectiveness of various learning methods for recognizing generic visual categories, while maintaining invariance to changes in pose, lighting, and background clutter. To conduct this assessment, a comprehensive dataset was assembled, featuring stereo image pairs of 50 uniformly colored toys, each photographed under 36 different azimuths, 9 elevations, and 6 lighting conditions, resulting in a total of 194,400 individual images. The toys were categorized into five generic classes: four-legged animals, human figures, airplanes, trucks, and cars, with 10 instances per category. For the training phase, five instances from each category were selected, and the remaining five were reserved for testing purposes. The training and testing processes utilized low-resolution grayscale images of the objects, which were subjected to varying degrees of variability and surrounding clutter. Three prominent learning methods were employed for comparison: nearest neighbor techniques, support vector machines (SVM), and convolutional networks. These methods were tested both on raw pixel data and on features derived from principal component analysis (PCA). The test error rates for identifying unseen object instances placed on uniform backgrounds were approximately 13% for SVM and 7% for convolutional networks. In the context of a segmentation/recognition task involving highly cluttered images, SVM was found to be impractical, whereas convolutional networks demonstrated an error rate of 16/7%. A real-time version of the system was successfully implemented, achieving a detection and classification rate of natural scenes at around 10 frames per second.

In essence, the paper explores the utility of different learning methods for the recognition of generic visual categories under varying conditions, showcasing the superiority of convolutional networks in handling cluttered scenes and achieving real-time performance.

We describe an experiment in applying text classification algorithms to Dutch texts. Four well-known learning algorithms, Rocchio’s algorithm, the Simple Bayesian Classifier (SBC), the Sleeping Experts (SE) and Winnow were implemented. They were tested on a corpus of articles from the Dutch newspaper NRC, pre-classified into four categories. The algorithms are compared on learning speed and error rate. We also investigated the effect of discarding terms, using either a dynamic stoplist or the Winnow heuristic.

Publications in the life sciences are characterized by a large technical vocabulary, with many lexical and semantic variations for expressing the same concept. Towards addressing the problem of relevance in biomedical literature search, we introduce a deep learning model for the relevance of a document’s text to a keyword style query. Limited by a relatively small amount of training data, the model uses pre-trained word embeddings. With these, the model first computes a variable-length Delta matrix between the query and document, representing a difference between the two texts, which is then passed through a deep convolution stage followed by a deep feed-forward network to compute a relevance score. This results in a fast model suitable for use in an online search engine. The model is robust and outperforms comparable state-of-the-art deep learning approaches.

In this paper, we present a new scene text detection algorithm based on two machine learning classifiers: one allows us to generate candidate word regions and the other filters out nontext ones. To be precise, we extract connected components (CCs) in images by using the maximally stable extremal region algorithm. These extracted CCs are partitioned into clusters so that we can generate candidate regions. Unlike conventional methods relying on heuristic rules in clustering, we train an AdaBoost classifier that determines the adjacency relationship and cluster CCs by using their pairwise relations. Then we normalize candidate word regions and determine whether each region contains text or not. Since the scale, skew, and color of each candidate can be estimated from CCs, we develop a text/nontext classifier for normalized images. This classifier is based on multilayer perceptrons and we can control recall and precision rates with a single free parameter. Finally, we extend our approach to exploit multichannel information. Experimental results on ICDAR 2005 and 2011 robust reading competition datasets show that our method yields the state-of-the-art performance both in speed and accuracy.

This paper describes connectionist techniques for recogni ti n of Broadcast News. The fundamental difference between connectionist systems and more convent ional mixture-of-Gaussian systems is that connectionist models directly estimate posterior probabi lities as opposed to likelihoods. Access to posterior probabilities has enabled us to develop a number of no vel approaches to confidence estimation, pronunciation modelling and search. In addition we have inv stigated a new feature extraction technique based on the modulation-filtered spectrogram, and methods f or combining multiple information sources. We have incorporated all of these techniques into a system fo r the transcription of Broadcast News, and we present results on the 1998 DARPA Hub-4E Broadcast News ev aluation data.

The experimental evidence accumulated over the past 20 years indicates that text indexing systems based on the assignment of appropriately weighted single terms produce retrieval results that are superior to those obtainable with other more elaborate text representations. These results depend crucially on the choice of effective termweighting systems. This article summarizes the insights gained in automatic term weighting, and provides baseline single-term-indexing models with which other more elaborate content analysis procedures can be compared. 1. AUTOMATIC TEXT ANALYSIS In the late 195Os, Luhn [l] first suggested that automatic text retrieval systems could be designed based on a comparison of content identifiers attached both to the stored texts and to the users’ information queries. Typically, certain words extracted from the texts of documents and queries would be used for content identification; alternatively, the content representations could be chosen manually by trained indexers familiar with the subject areas under consideration and with the contents of the document collections. In either case, the documents would be represented by term vectors of the form D= (ti,tj,...ytp) (1) where each tk identifies a content term assigned to some sample document D. Analogously, the information requests, or queries, would be represented either in vector form, or in the form of Boolean statements. Thus, a typical query Q might be formulated as Q = (qa,qbr.. . ,4r) (2)

The naive Bayes classiier, currently experiencing a renaissance in machine learning, has long been a core technique in information retrieval. We review some of the variations of naive Bayes models used for text retrieval and classiication, focusing on the distributional assumptions made about word occurrences in documents.

The description of a novel type of rn-gram language model is given. The model offers, via a nonlinear recursive procedure, a computation and space efficient solution to the problem of estimating probabilities from sparse data. This solution compares favorably to other proposed methods. While the method has been developed for and successfully implemented in the IBM Real Time Speech Recognizers, its generality makes it applicable in other areas where the problem of estimating probabilities from sparse data arises. Sparseness of data is an inherent property of any real text, and it is a problem that one always encounters while collecting frequency statistics on words and word sequences (m-grams) from a text of finite size. This means that even for a very large data collection, the maximum likelihood estimation method does not allow us to adequately estimate probabilities of rare but nevertheless possible word sequences-many sequences occur only once (“singletons”); many more do not occur at all. Inadequacy of the maximum likelihood estimator and the necessity to estimate the probabilities of m-grams which did not occur in the text constitute the essence of the problem. The main idea of the proposed solution to the problem is to reduce unreliable probability estimates given by the observed frequencies and redistribute the “freed” probability “mass” among m-grams which never occurred in the text. The reduction is achieved by replacing maximum likelihood estimates for m-grams having low counts with renormalized Turing’s estimates [l], and the redistribution is done via the recursive utilization of lower level conditional distributions. We found Turing’s method attractive because of its simplicity and its characterization as the optimal empirical Bayes’ estimator of a multinomial probability. Robbins in [2] introduces the empirical Bayes’ methodology and Nadas in [3] gives various derivations of the Turing’s formula. Let N be a sample text size and let n, be the number of words (m-grams) which occurred in the text exactly r times, so that N = C rn,. (1) Turing’s estimate PT for a probability of a word (m-gram) which occurred in the sample r times is r

The Development Workflow There are two major stages within isolated word recognition: a training stage and a testing stage. Training involves “teaching” the system by building its dictionary, an acoustic model for each word that the system needs to recognize. In our example, the dictionary comprises the digits ‘zero’ to ‘nine’. In the testing stage we use acoustic models of these digits to recognize isolated words using a classification algorithm. The development workflow consists of three steps: Speech acquisition • Speech analysis • User interface development •

Recording university lectures through lecture capture systems is increasingly common, generating large amounts of audio and video data. Transcribing recordings greatly enhances their usefulness by making them easy to search. However, the number of recordings accumulates rapidly, rendering manual transcription impractical. Automatic transcription, on the other hand, suffers from low levels of accuracy, partly due to the special language of academic disciplines, which standard language models do not cover. This paper looks into the use of Wikipedia to dynamically adapt language models for scholarly speech. We propose Ranked Word Correct Rate as a new metric better aligned with the goals of improving transcript search ability and specialist word recognition. The study shows that, while overall transcription accuracy may remain low, targeted language modelling can substantially improve search ability, an important goal in its own right.

Speech recognition in adverse conditions: A review Sven L. Mattys a , Matthew H. Davis b , Ann R. Bradlow c & Sophie K. Scott d a Department of Psychology, University of York, York, UK b Medical Research Council, Cognition and Brain Sciences Unit, Cambridge, UK c Department of Linguistics, Northwestern University, Evanston, IL, USA d Institute of Cognitive Neuroscience, University College London, London, UK


Question: Enhanced keyword spotting utilizing keyword/garbage model refinements.

This paper explores the application of convolutional networks in enhancing keyword spotting techniques, particularly focusing on their utility in processing images, speech, and time series data. Convolutional networks, renowned for their ability to extract hierarchical features from input data, have shown remarkable performance in various domains, including computer vision, speech recognition, and signal processing. In the context of keyword spotting, these networks can be leveraged to create robust models that distinguish between keywords and non-keyword (garbage) inputs. By training on large datasets of images, audio clips, and time series signals, convolutional networks can learn to recognize patterns specific to keywords, enabling efficient and accurate spotting in diverse environments. This approach not only improves the overall accuracy of keyword spotting systems but also enhances their robustness to background noise and speaker variability. Through comprehensive evaluation, the effectiveness of convolutional networks in keyword spotting is demonstrated, highlighting their potential to revolutionize the field with their versatile and powerful pattern recognition capabilities.

Most keyword search engines return directly matching keyword phrases. However, publishers cannot anticipate all possible ways in which users would search for the items in their documents. In fact, many times, there may be no direct keyword match between a keyword search phrase and descriptions of relevant items that are perfect matches for the search. We present an automated, high precision-based information retrieval solution to boost item find-ability by bridging the semantic gap between item information and popular keyword search phrases. Our solution achieves an average of 80% F-measure for various boosted matches for keyword search phrases in various categories.

Given an image, how do we automatically assign keywords to it? In this paper, we propose a novel, graph-based approach (GCap) which outperforms previously reported methods for automatic image captioning. Moreover, it is fast and scales well, with its training and testing time linear to the data set size. We report auto-captioning experiments on the "standard" Corel image database of 680 MBytes, where GCap outperforms recent, successful auto-captioning methods by up to 10 percentage points in captioning accuracy (50% relative improvement).

In this paper, we apply a bidirectional Long Short-Term Memory with a Conditional Random Field to the task of disfluency detection. Long-range dependencies is one of the core problems for disfluency detection. Our model handles long-range dependencies by both using the Long Short-Term Memory and hand-crafted discrete features. Experiments show that utilizing the hand-crafted discrete features significantly improves the model’s performance by achieving the state-of-the-art score of 87.1% on the Switchboard corpus.

The Kalman filter is a very interesting signal processing tool, which is widely used in many practical applications. In this paper, we study the Kalman filter in the context of echo cancellation. The contribution of this work is threefold. First, we derive a different form of the Kalman filter by considering, at each iteration, a block of time samples instead of one time sample as it is the case in the conventional approach. Second, we show how this general Kalman filter (GKF) is connected with some of the most popular adaptive filters for echo cancellation, i.e., the normalized least-mean-square (NLMS) algorithm, the affine projection algorithm (APA) and its proportionate version (PAPA). Third, a simplified Kalman filter is developed in order to reduce the computational load of the GKF; this algorithm behaves like a variable step-size adaptive filter. Simulation results indicate the good performance of the proposed algorithms, which can be attractive choices for echo cancellation.

Accurate, real-time Automatic Speech Recognition (ASR) requires huge memory storage and computational power. The main bottleneck in state-of-the-art ASR systems is the Viterbi search on a Weighted Finite State Transducer (WFST). The WFST is a graph-based model created by composing an Acoustic Model (AM) and a Language Model (LM) offline. Offline composition simplifies the implementation of a speech recognizer as only one WFST has to be searched. However, the size of the composed WFST is huge, typically larger than a Gigabyte, resulting in a large memory footprint and memory bandwidth requirements.
 In this paper, we take a completely different approach and propose a hardware accelerator for speech recognition that composes the AM and LM graphs on-the-fly. In our ASR system, the fully-composed WFST is never generated in main memory. On the contrary, only the subset required for decoding each input speech fragment is dynamically generated from the AM and LM models. In addition to the direct benefits of this on-the-fly composition, the resulting approach is more amenable to further reduction in storage requirements through compression techniques.
 The resulting accelerator, called UNFOLD, performs the decoding in real-time using the compressed AM and LM models, and reduces the size of the datasets from more than one Gigabyte to less than 40 Megabytes, which can be very important in small form factor mobile and wearable devices.
 Besides, UNFOLD improves energy-efficiency by orders of magnitude with respect to CPUs and GPUs. Compared to a state-of-the-art Viterbi search accelerators, the proposed ASR system outperforms by providing 31x reduction in memory footprint and 28% energy savings on average.

In this paper, we present an algorithm for extracting keywords representing the asserted main point in a document, without relying on external devices such as natural language processing tools or a document corpus. Our algorithm KeyGraph is based on the segmentation of a graph, representing the co-occurrence between terms in a document, into clusters. Each cluster corresponds to a concept on which author's idea is based, and top ranked terms by a statistic based on each term's relationship to these clusters are selected as keywords. This strategy comes from considering that a document is constructed like a building for expressing new ideas based on traditional concepts. The experimental results show that thus extracted terms match author's point quite accurately, even though KeyGraph does not use each term's average frequency in a corpus, i.e., KeyGraph is a contentsensitive, domain independent device of indexing.

Formulation of speech separation as a supervised learning problem has shown considerable promise. In its simplest form, a supervised learning algorithm, typically a deep neural network, is trained to learn a mapping from noisy features to a time-frequency representation of the target of interest. Traditionally, the ideal binary mask (IBM) is used as the target because of its simplicity and large speech intelligibility gains. The supervised learning framework, however, is not restricted to the use of binary targets. In this study, we evaluate and compare separation results by using different training targets, including the IBM, the target binary mask, the ideal ratio mask (IRM), the short-time Fourier transform spectral magnitude and its corresponding mask (FFT-MASK), and the Gammatone frequency power spectrum. Our results in various test conditions reveal that the two ratio mask targets, the IRM and the FFT-MASK, outperform the other targets in terms of objective intelligibility and quality metrics. In addition, we find that masking based targets, in general, are significantly better than spectral envelope based targets. We also present comparisons with recent methods in non-negative matrix factorization and speech enhancement, which show clear performance advantages of supervised speech separation.

This paper is focused on the application of the Language Identification (LID) technology for intelligent vehicles. We cope with short sentences or words spoken in moving cars in four languages: English, Spanish, German, and Finnish. As the response time of the LID system is crucial for user acceptance in this particular task, speech signals of different durations with total average of 3.8s are analyzed. In this paper, the authors propose the use of Deep Neural Networks (DNN) to model effectively the i-vector space of languages. Both raw i-vectors and session variability compensated i-vectors are evaluated as input vectors to DNNs. The performance of the proposed DNN architecture is compared with both conventional GMM-UBM and i-vector/LDA systems considering the effect of durations of signals. It is shown that the signals with durations between 2 and 3s meet the requirements of this application, i.e., high accuracy and fast decision, in which the proposed DNN architecture outperforms GMM-UBM and i-vector/LDA systems by 37% and 28%, respectively.

We present a novel deep Recurrent Neural Network (RNN) model for acoustic modelling in Automatic Speech Recognition (ASR). We term our contribution as a TC-DNN-BLSTM-DNN model, the model combines a Deep Neural Network (DNN) with Time Convolution (TC), followed by a Bidirectional LongShort Term Memory (BLSTM), and a final DNN. The first DNN acts as a feature processor to our model, the BLSTM then generates a context from the sequence acoustic signal, and the final DNN takes the context and models the posterior probabilities of the acoustic states. We achieve a 3.47 WER on the Wall Street Journal (WSJ) eval92 task or more than 8% relative improvement over the baseline DNN models

In automatic speech recognition (ASR) what a user says depends on the particular context she is in. Typically, this context is represented as a set of word n-grams. In this work, we present a novel, all-neural, end-to-end (E2E) ASR system that utilizes such context. Our approach, which we refer to as Contextual Listen, Attend and Spell (CLAS) jointly-optimizes the ASR components along with embeddings of the context n-grams. During inference, the CLAS system can be presented with context phrases which might contain-of-vocabulary (OOV) terms not seen during training. We compare our proposed system to a more traditional contextualization approach, which performs shallow-fusion between independently trained LAS and contextual n-gram models during beam search. Across a number of tasks, we find that the proposed CLAS system outperforms the baseline method by as much as 68% relative WER, indicating the advantage of joint optimization over individually trained components.


Question: Graphcut-based texture synthesis for enhancing single-image superresolution.

This paper presents a novel technique for texture synthesis using a non-parametric sampling approach. The synthesis algorithm progressively expands a new image starting from an initial seed, incrementing it pixel by pixel. Under the assumption of a Markov random field model, the conditional probability of a pixel, given its previously synthesized neighbors, is approximated by querying the original texture sample to locate analogous neighborhoods. The level of stochasticity is governed by a single parameter that is perceptually meaningful, ensuring that the maximum amount of local structure is conserved. This method yields high-quality outcomes for a diverse array of synthetic and real-world textures, demonstrating its versatility and effectiveness in texture replication.

The rapid development of computer hardware and Internet technology makes large scale data dependent models computationally tractable, and opens a bright avenue for annotating images through innovative machine learning algorithms. Semisupervised learning (SSL) therefore received intensive attention in recent years and was successfully deployed in image annotation. One representative work in SSL is Laplacian regularization (LR), which smoothes the conditional distribution for classification along the manifold encoded in the graph Laplacian, however, it is observed that LR biases the classification function toward a constant function that possibly results in poor generalization. In addition, LR is developed to handle uniformly distributed data (or single-view data), although instances or objects, such as images and videos, are usually represented by multiview features, such as color, shape, and texture. In this paper, we present multiview Hessian regularization (mHR) to address the above two problems in LR-based image annotation. In particular, mHR optimally combines multiple HR, each of which is obtained from a particular view of instances, and steers the classification function that varies linearly along the data manifold. We apply mHR to kernel least squares and support vector machines as two examples for image annotation. Extensive experiments on the PASCAL VOC'07 dataset validate the effectiveness of mHR by comparing it with baseline algorithms, including LR and HR.

Textbooks have a direct bearing on the quality of education imparted to the students. Therefore, it is of paramount importance that the educational content of textbooks should provide rich learning experience to the students. Recent studies on understanding learning behavior suggest that the incorporation of digital visual material can greatly enhance learning. However, textbooks used in many developing regions are largely text-oriented and lack good visual material. We propose techniques for finding images from the web that are most relevant for augmenting a section of the textbook, while respecting the constraint that the same image is not repeated in different sections of the same chapter. We devise a rigorous formulation of the image assignment problem and present a polynomial time algorithm for solving the problem optimally. We also present two image mining algorithms that utilize orthogonal signals and hence obtain different sets of relevant images. Finally, we provide an ensembling algorithm for combining the assignments. To empirically evaluate our techniques, we use a corpus of high school textbooks in use in India. Our user study utilizing the Amazon Mechanical Turk platform indicates that the proposed techniques are able to obtain images that can help increase the understanding of the textbook material.

In this paper, we propose a new model called deep network cascade (DNC) to gradually upscale low-resolution images layer by layer, each layer with a small scale factor. DNC is a cascade of multiple stacked collaborative local auto-encoders. In each layer of the cascade, non-local self-similarity search is first performed to enhance high-frequency texture details of the partitioned patches in the input image. The enhanced image patches are then input into a collaborative local auto-encoder (CLA) to suppress the noises as well as collaborate the compatibility of the overlapping patches. By closing the loop on non-local self-similarity search and CLA in a cascade layer, we can refine the super-resolution result, which is further fed into next layer until the required image scale. Experiments on image super-resolution demonstrate that the proposed DNC can gradually upscale a low-resolution image with the increase of network layers and achieve more promising results in visual quality as well as quantitative performance.

We introduce BranchGAN , a novel training method that enables unconditioned generative adversarial networks (GANs) to learn image manifolds at multiple scales. The key novel feature of BranchGAN is that it is trained in multiple branches, progressively covering both the breadth and depth of the network, as resolutions of the training images increase to reveal finer-scale features. Specifically, each noise vector, as input to the generator network, is explicitly split into several sub-vectors, each corresponding to, and is trained to learn, image representations at a particular scale. During training, we progressively “defreeze” the sub-vectors, one at a time, as a new set of higher-resolution images is employed for training and more network layers are added. A consequence of such an explicit sub-vector designation is that we can directly manipulate and even combine latent (sub-vector) codes which model different feature scales. Experiments demonstrate the effectiveness of our training method in scale-disentangled learning of image manifolds and synthesis, without any extra labels and without compromising quality of the synthesized high-resolution images. We further demonstrate three applications enabled or improved by BranchGAN.

Video, like images, should support content aware resizing. We present video retargeting using an improved seam carving operator. Instead of removing 1D seams from 2D images we remove 2D seam manifolds from 3D space-time volumes. To achieve this we replace the dynamic programming method of seam carving with graph cuts that are suitable for 3D volumes. In the new formulation, a seam is given by a minimal cut in the graph and we show how to construct a graph such that the resulting cut is a valid seam. That is, the cut is monotonic and connected. In addition, we present a novel energy criterion that improves the visual quality of the retargeted images and videos. The original seam carving operator is focused on removing seams with the least amount of energy, ignoring energy that is introduced into the images and video by applying the operator. To counter this, the new criterion is looking forward in time - removing seams that introduce the least amount of energy into the retargeted result. We show how to encode the improved criterion into graph cuts (for images and video) as well as dynamic programming (for images). We apply our technique to images and videos and present results of various applications.

Image processing is one of the most important areas of signal processing. Image processing is applicable for the improvement of pictorial improvement for human perception and autonomous machine application also the efficient storage and transmission. In this paper guided filter is proposed which can be applicable to both images as well as to video. This guided filter has many advantages over the previous version of filter like mean filter, median filter, sobel filter, bilateral filter, etc. Among of those bilateral filter is more popular. But this bilateral filter has certain limitations like gradient reversal effect this introduces false edges in the image. Proposed guided filter can smooth image/video and also preserved the edge that is information at the edges of image/video but this filter has better behaviors near edges. Also the fast implementation of guided filter is possible. This filter also has algorithm implementation regardless of kernel size. Guided filter gives better output than that of bilateral filter after filtering this bilateral filter gives output like cartoon this also a problem with this. The guided filter is both effective and efficient in many applications of computer vision and computer graphics, etc. Filtering techniques are used in image and video processing applications for various technologies.

We develop a framework for learning generic, expressive image priors that capture the statistics of natural scenes and can be used for a variety of machine vision tasks. The approach extends traditional Markov random field (MRF) models by learning potential functions over extended pixel neighborhoods. Field potentials are modeled using a Products-of-Experts framework that exploits nonlinear functions of many linear filter responses. In contrast to previous MRF approaches all parameters, including the linear filters themselves, are learned from training data. We demonstrate the capabilities of this Field of Experts model with two example applications, image denoising and image inpainting, which are implemented using a simple, approximate inference scheme. While the model is trained on a generic image database and is not tuned toward a specific application, we obtain results that compete with and even outperform specialized techniques.

Super-resolution reconstruction based on deep learning has come a long way since the first proposed method in 2015. Numerous methods have been developed for this task using deep learning approaches. Among these methods, residual deep learning algorithms have shown better performance. Although all early proposed deep learning based super-resolution frameworks used bicubic upsampled versions of low resolution images as the main input, most of the current ones use the low resolution images directly by adding up-sampling layers to their networks. In this work, we propose a new method by using both low resolution and bicubic upsampled images as the inputs to our network. The final results confirm that decreasing the depth of the network in lower resolution space and adding the bicubic path lead to almost similar results to those of the deeper networks in terms of PSNR and SSIM, yet making the network computationally inexpensive and more efficient.

We present in this paper a new method for contrast enhancement based on the curvelet transform. The curvelet transform represents edges better than wavelets, and is therefore well-suited for multiscale edge enhancement. We compare this approach with enhancement based on the wavelet transform, and the Multiscale Retinex. In a range of examples, we use edge detection and segmentation, among other processing applications, to provide for quantitative comparative evaluation. Our findings are that curvelet based enhancement out-performs other enhancement methods on noisy images, but on noiseless or near noiseless images curvelet based enhancement is not remarkably better than wavelet based enhancement.

This paper presents a novel image fusion method, suitable for pan-sharpening of multispectral (MS) bands, based on multi-resolution analysis (MRA). The low-resolution MS bands are sharpened by injecting high-pass directional details extracted from the high-resolution panchromatic (Pan) image by means of the curvelet transform, which is a non-separable MRA, whose basis function are directional edges with progressively increasing resolution. The advantage with respect to conventional separable MRA, either decimated or not, is twofold: directional detail coefficients matching image edges may be preliminarily soft-thresholded to achieve denoising better than in the separable wavelet domain; modeling of the relationships between high-resolution detail coefficients of MS bands and of the Pan image is more fitting, being carried out in a directional wavelet domain. Experiments carried out on a very-high resolution MS + Pan QuickBird image show that the proposed curvelet method quantitatively outperforms state-of-the art image fusion methods, in terms of geometric, radiometric, and spectral fidelity.


Question: Graphcut-based texture synthesis for enhancing single-image superresolution.

This paper introduces an efficient algorithm for texture synthesis, which is crucial for various applications in computer graphics, vision, and image processing. Despite its significance, developing an algorithm that is both time-efficient and capable of producing high-quality textures has remained a challenge. Our proposed method offers a solution to this issue by delivering high-quality results at a significantly faster rate than existing techniques. The algorithm is straightforward to implement and requires only a texture sample as input. It generates textures that are equal to or surpass the quality of those produced by previous methods, yet it operates at least a hundred times faster. This breakthrough allows us to apply texture synthesis to scenarios where it was previously deemed impractical. Specifically, we have utilized our algorithm for constrained synthesis in image editing and for generating textures that change over time. The algorithm is based on Markov Random Field texture models and employs a deterministic searching process for texture generation. To expedite this process, we employ tree-structured vector quantization, enhancing the algorithm's efficiency and applicability.

We propose a two-class classification model for grouping. Human segmented natural images are used as positive examples. Negative examples of grouping are constructed by randomly matching human segmentations and images. In a preprocessing stage an image is oversegmented into superpixels. We define a variety of features derived from the classical Gestalt cues, including contour, texture, brightness and good continuation. Information-theoretic analysis is applied to evaluate the power of these grouping cues. We train a linear classifier to combine these features. To demonstrate the power of the classification model, a simple algorithm is used to randomly search for good segmentations. Results are shown on a wide range of images.

Raw depth images captured by consumer depth cameras suffer from noisy and missing values. Despite the success of CNN-based image processing on color image restoration, similar approaches for depth enhancement have not been much addressed yet because of the lack of raw-clean pairwise dataset. In this paper, we propose a pairwise depth image dataset generation method using dense 3D surface reconstruction with a filtering method to remove low quality pairs. We also present a multi-scale Laplacian pyramid based neural network and structure preserving loss functions to progressively reduce the noise and holes from coarse to fine scales. Experimental results show that our network trained with our pairwise dataset can enhance the input depth images to become comparable with 3D reconstructions obtained from depth streams, and can accelerate the convergence of dense 3D reconstruction results.

Inpainting is the art of modifying an image in a form that is not detectable by an ordinary observer. There are numerous and very different approaches to tackle the inpainting problem, though as explained in this paper, the most successful algorithms are based upon one or two of the following three basic techniques: copy-and-paste texture synthesis, geometric partial differential equations (PDEs), and coherence among neighboring pixels. We combine these three building blocks in a variational model, and provide a working algorithm for image inpainting trying to approximate the minimum of the proposed energy functional. Our experiments show that the combination of all three terms of the proposed energy works better than taking each term separately, and the results obtained are within the state-of-the-art.

Many algorithms for image processing and pattern recognition have recently been implemented on GPU (graphic processing unit) for faster computational times. However, the implementation using GPU encounters two problems. First, the programmer should master the fundamentals of the graphics shading languages that require the prior knowledge on computer graphics. Second, in a job which needs much cooperation between CPU and GPU, which is usual in image processings and pattern recognitions contrary to the graphics area, CPU should generate raw feature data for GPU processing as much as possible to effectively utilize GPU performance. This paper proposes more quick and efficient implementation of neural networks on both GPU and multi-core CPU. We use CUDA (compute unified device architecture) that can be easily programmed due to its simple C language-like style instead of GPU to solve the first problem. Moreover, OpenMP (Open Multi-Processing) is used to concurrently process multiple data with single instruction on multi-core CPU, which results ineffectively utilizing the memories of GPU. In the experiments, we implemented neural networks-based text detection system using the proposed architecture, and the computational times showed about 15 times faster than implementation using CPU and about 4 times faster than implementation on only GPU without OpenMP.

Learning good image priors is of utmost importance for the study of vision, computer vision and image processing applications. Learning priors and optimizing over whole images can lead to tremendous computational challenges. In contrast, when we work with small image patches, it is possible to learn priors and perform patch restoration very efficiently. This raises three questions - do priors that give high likelihood to the data also lead to good performance in restoration? Can we use such patch based priors to restore a full image? Can we learn better patch priors? In this work we answer these questions. We compare the likelihood of several patch models and show that priors that give high likelihood to data perform better in patch restoration. Motivated by this result, we propose a generic framework which allows for whole image restoration using any patch based prior for which a MAP (or approximate MAP) estimate can be calculated. We show how to derive an appropriate cost function, how to optimize it and how to use it to restore whole images. Finally, we present a generic, surprisingly simple Gaussian Mixture prior, learned from a set of natural images. When used with the proposed framework, this Gaussian Mixture Model outperforms all other generic prior methods for image denoising, deblurring and inpainting.

Single image reflection separation is an ill-posed problem since two scenes, a transmitted scene and a reflected scene, need to be inferred from a single observation. To make the problem tractable, in this work we assume that categories of two scenes are known. It allows us to address the problem by generating both scenes that belong to the categories while their contents are constrained to match with the observed image. A novel network architecture is proposed to render realistic images of both scenes based on adversarial learning. The network can be trained in a weakly supervised manner, i.e., it learns to separate an observed image without corresponding ground truth images of transmission and reflection scenes which are difficult to collect in practice. Experimental results on real and synthetic datasets demonstrate that the proposed algorithm performs favorably against existing methods.

We propose in this paper to unify two different approaches to image restoration: On the one hand, learning a basis set (dictionary) adapted to sparse signal descriptions has proven to be very effective in image reconstruction and classification tasks. On the other hand, explicitly exploiting the self-similarities of natural images has led to the successful non-local means approach to image restoration. We propose simultaneous sparse coding as a framework for combining these two approaches in a natural manner. This is achieved by jointly decomposing groups of similar signals on subsets of the learned dictionary. Experimental results in image denoising and demosaicking tasks with synthetic and real noise show that the proposed method outperforms the state of the art, making it possible to effectively restore raw images from digital cameras at a reasonable speed and memory cost.

This research work proposes a new method for garment retexturing using a single static image along with depth information obtained using the Microsoft Kinect 2 camera. First the garment is segmented out from the image and texture domain coordinates are computed for each pixel of the shirt using 3D information. After that shading is applied on the new colours from the texture image by applying linear stretching of the luminance of the segmented garment. The proposed method is colour and pattern invariant and results in to visually realistic retexturing. The proposed method has been tested on various images and it is shown that it generally performs better and produces more realistic results compared to the state-of-the-art methods. The proposed method can be an application for the virtual fitting room.

In this paper we propose a new and efficient texture feature extraction method: the Segmentation-based Fractal Texture Analysis, or SFTA. The extraction algorithm consists in decomposing the input image into a set of binary images from which the fractal dimensions of the resulting regions are computed in order to describe segmented texture patterns. The decomposition of the input image is achieved by the Two-Threshold Binary Decomposition (TTBD) algorithm, which we also propose in this work. We evaluated SFTA for the tasks of content-based image retrieval (CBIR) and image classification, comparing its performance to that of other widely employed feature extraction methods such as Haralick and Gabor filter banks. SFTA achieved higher precision and accuracy for CBIR and image classification. Additionally, SFTA was at least 3.7 times faster than Gabor and 1.6 times faster than Haralick with respect to feature extraction time.

Texturing 3D models using casual images has gained importance in the last decade, with the advent of huge databases of images. We present a novel approach for performing this task, which manages to account for the 3D geometry of the photographed object. Our method overcomes the limitation of both the constrained-parameterization approach, which does not account for the photography effects, and the photogrammetric approach, which cannot handle arbitrary images. The key idea of our algorithm is to formulate the mapping estimation as a Moving-Least-Squares problem for recovering local camera parameters at each vertex. The algorithm is realized in a FlexiStickers application, which enables fast interactive texture mapping using a small number of constraints.