Question: Categorize sentences from various viewpoints using a Category Expert Attention Network.

This work investigates the application of convolutional neural networks (CNNs) in conjunction with pre-trained word vectors for the classification of sentences. Through a series of experiments, we demonstrate that a basic CNN model, requiring minimal hyperparameter tuning and utilizing static word vectors, yields outstanding results across various benchmarks. The performance can be further enhanced by fine-tuning the word vectors specifically for the task at hand. To accommodate both static and task-specific vectors, we introduce a straightforward modification to the CNN architecture. This enhanced CNN model outperforms the current state of the art in 4 out of 7 classification tasks, encompassing sentiment analysis and question categorization. Our findings underscore the efficacy of CNNs in sentence classification when combined with pre-trained word vectors, particularly when the vectors are fine-tuned for the specific task.

Languages such as Chinese and Japanese have a significantly large number (several thousands) of alphabets as compared to other languages, and each of their sentences consists of several concatenated words with wide varieties of inflected forms; thus appropriate word segmentation is quite difficult. Therefore, recently proposed sophisticated language-processing methods designed for languages such as English cannot be applied. In this paper, we address those issues and propose a new and efficient document classification technique for such languages. The proposed method is characterized into a new “image-based character embedding” method and character-level convolutional neural networks method with “wildcard training.” The first method encodes each character based on its pictorial structures and preserves them. Further, the second method treats some of the input characters as wildcards in the classification stage and functions as efficient data augmentation. We confirmed that our proposed method showed superior performance when compared conventional methods for Japanese document classification problems.

We describe SLING, a framework for parsing natural language into semantic frames. SLING supports general transition-based, neural-network parsing with bidirectional LSTM input encoding and a Transition Based Recurrent Unit (TBRU) for output decoding. The parsing model is trained end-to-end using only the text tokens as input. The transition system has been designed to output frame graphs directly without any intervening symbolic representation. The SLING framework includes an efficient and scalable frame store implementation as well as a neural network JIT compiler for fast inference during parsing. SLING is implemented in C++ and it is available for download on GitHub.

In this work we address the task of computerassisted assessment of short student answers. We combine several graph alignment features with lexical semantic similarity measures using machine learning techniques and show that the student answers can be more accurately graded than if the semantic measures were used in isolation. We also present a first attempt to align the dependency graphs of the student and the instructor answers in order to make use of a structural component in the automatic grading of student answers.

In this paper we present a detailed scheme for annotating expressions of opinions, beliefs, emotions, sentiment and speculation (private states) in the news and other discourse. We explore inter-annotator agreement for individual private state expressions, and show that these low-level annotations are useful for producing higher-level subjective sentence annotations.

This article introduces a sentiment analysis approach that adopts the way humans read, interpret, and extract sentiment from text. Our motivation builds on the assumption that human interpretation should lead to the most accurate assessment of sentiment in text. We call this automated process Human Reading for Sentiment (HRS). Previous research in sentiment analysis has produced many frameworks that can fit one or more of the HRS aspects; however, none of these methods has addressed them all in one approach. HRS provides a meta-framework for developing new sentiment analysis methods or improving existing ones. The proposed framework provides a theoretical lens for zooming in and evaluating aspects of any sentiment analysis method to identify gaps for improvements towards matching the human reading process. Key steps in HRS include the automation of humans low-level and high-level cognitive text processing. This methodology paves the way towards the integration of psychology with computational linguistics and machine learning to employ models of pragmatics and discourse analysis for sentiment analysis. HRS is tested with two state-of-the-art methods; one is based on feature engineering, and the other is based on deep learning. HRS highlighted the gaps in both methods and showed improvements for both.

An obstacle to the development of many natural language processing products is the vast amount of training examples necessary to get satisfactory results. The generation of these examples is often a tedious and timeconsuming task. This paper this paper proposes a method to transform the sentiment of sentences in order to limit the work necessary to generate more training data. This means that one sentence can be transformed to an opposite sentiment sentence and should reduce by half the work required in the generation of text. The proposed pipeline consists of a sentiment classifier with an attention mechanism to highlight the short phrases that determine the sentiment of a sentence. Then, these phrases are changed to phrases of the opposite sentiment using a baseline model and an autoencoder approach. Experiments are run on both the separate parts of the pipeline as well as on the end-to-end model. The sentiment classifier is tested on its accuracy and is found to perform adequately. The autoencoder is tested on how well it is able to change the sentiment of an encoded phrase and it was found that such a task is possible. We use human evaluation to judge the performance of the full (end-to-end) pipeline and that reveals that a model using word vectors outperforms the encoder model. Numerical evaluation shows that a success rate of 54.7% is achieved on the sentiment change.

Extractive text summarization aims to create a condensed version of one or more source documents by selecting the most informative sentences. Research in text summarization has therefore often focused on measures of the usefulness of sentences for a summary. We present an approach to sentence extraction that maps sentences to nodes of a hierarchical ontology. By considering ontology attributes we are able to improve the semantic representation of a sentence's information content. The classifier that maps sentences to the taxonomy is trained using search engines and is therefore very flexible and not bound to a specific domain. In our experiments, we train an SVM classifier to identify summary sentences using ontology-based sentence features. Our experimental results show that the ontology-based extraction of sentences outperforms baseline classifiers, leading to higher Rouge scores of summary extracts.

As the amount of unstructured text data that humanity produces overall and on the Internet grows, so does the need to intelligently to process it and extract different types of knowledge from it. Convolutional neural networks (CNNs) and recurrent neural networks (RNNs) have been applied to natural language processing systems with comparative, remarkable results. The CNN is a noble approach to extract higher level features that are invariant to local translation. However, it requires stacking multiple convolutional layers in order to capture long-term dependencies, due to the locality of the convolutional and pooling layers. In this paper, we describe a joint CNN and RNN framework to overcome this problem. Briefly, we use an unsupervised neural language model to train initial word embeddings that are further tuned by our deep learning network, then, the pre-trained parameters of the network are used to initialize the model. At a final stage, the proposed framework combines former information with a set of feature maps learned by a convolutional layer with long-term dependencies learned via long-short-term memory. Empirically, we show that our approach, with slight hyperparameter tuning and static vectors, achieves outstanding results on multiple sentiment analysis benchmarks. Our approach outperforms several existing approaches in term of accuracy; our results are also competitive with the state-of-the-art results on the Stanford Large Movie Review data set with 93.3% accuracy, and the Stanford Sentiment Treebank data set with 48.8% fine-grained and 89.2% binary accuracy, respectively. Our approach has a significant role in reducing the number of parameters and constructing the convolutional layer followed by the recurrent layer as a substitute for the pooling layer. Our results show that we were able to reduce the loss of detailed, local information and capture long-term dependencies with an efficient framework that has fewer parameters and a high level of performance.

This paper develops a model that addresses sentence embedding, a hot topic in current natural language processing research, using recurrent neural networks (RNN) with Long Short-Term Memory (LSTM) cells. The proposed LSTM-RNN model sequentially takes each word in a sentence, extracts its information, and embeds it into a semantic vector. Due to its ability to capture long term memory, the LSTM-RNN accumulates increasingly richer information as it goes through the sentence, and when it reaches the last word, the hidden layer of the network provides a semantic representation of the whole sentence. In this paper, the LSTM-RNN is trained in a weakly supervised manner on user click-through data logged by a commercial web search engine. Visualization and analysis are performed to understand how the embedding process works. The model is found to automatically attenuate the unimportant words and detect the salient keywords in the sentence. Furthermore, these detected keywords are found to automatically activate different cells of the LSTM-RNN, where words belonging to a similar topic activate the same cell. As a semantic representation of the sentence, the embedding vector can be used in many different applications. These automatic keyword detection and topic allocation abilities enabled by the LSTM-RNN allow the network to perform document retrieval, a difficult language processing task, where the similarity between the query and documents can be measured by the distance between their corresponding sentence embedding vectors computed by the LSTM-RNN. On a web search task, the LSTM-RNN embedding is shown to significantly outperform several existing state of the art methods. We emphasize that the proposed model generates sentence embedding vectors that are specially useful for web document retrieval tasks. A comparison with a well known general sentence embedding method, the Paragraph Vector, is performed. The results show that the proposed method in this paper significantly outperforms Paragraph Vector method for web document retrieval task.


Question: Categorize sentences from various viewpoints using a category expert attention network.

This study conducts an exhaustive evaluation of various models aimed at learning distributed representations of phrases and sentences from unlabelled data, an area less explored compared to word representations in contemporary NLP research. Our findings highlight that the choice of model is significantly influenced by the specific application at hand. For representations intended for use within supervised learning frameworks, we recommend employing deeper and more intricate models. Conversely, for the creation of representation spaces that can be effectively deciphered using rudimentary spatial distance metrics, simpler log-bilinear models outperform their more complex counterparts. Additionally, we introduce two novel unsupervised representation-learning objectives. These objectives are crafted to strike a balance between computational efficiency, adaptability across domains, and overall performance, thereby optimizing the learning process for a broader range of applications.

Attention mechanism has been a key component in Recurrent Neural Networks (RNNs) based sequence to sequence learning framework, which has been adopted in many text understanding tasks, such as neural machine translation and abstractive summarization. In these tasks, the attention mechanism models how important each part of the source sentence is to generate a target side word. To compute such importance scores, the attention mechanism summarizes the source side information in the encoder RNN hidden states (i.e., ht), and then builds a context vector for a target side word upon a subsequence representation of the source sentence, since ht actually summarizes the information of the subsequence containing the first t-th words in the source sentence. We in this paper, show that an additional attention mechanism called word attention, that builds itself upon word level representations, significantly enhances the performance of sequence to sequence learning. Our word attention can enrich the source side contextual representation by directly promoting the clean word level information in each step. Furthermore, we propose to use contextual gates to dynamically combine the subsequence level and word level contextual information. Experimental results on abstractive summarization and neural machine translation show that word attention significantly improve over strong baselines. In particular, we achieve the state-of-the-art result on WMT’14 English-French translation task with 12M training data.

Neural Machine Translation (NMT) based on the encoder-decoder architecture has recently achieved the state-of-the-art performance. Researchers have proven that extending word level attention to phrase level attention by incorporating source-side phrase structure can enhance the attention model and achieve promising improvement. However, word dependencies that can be crucial to correctly understand a source sentence are not always in a consecutive fashion (i.e. phrase structure), sometimes they can be in long distance. Phrase structures are not the best way to explicitly model long distance dependencies. In this paper we propose a simple but effective method to incorporate source-side long distance dependencies into NMT. Our method based on dependency trees enriches each source state with global dependency structures, which can better capture the inherent syntactic structure of source sentences. Experiments on Chinese-English and English-Japanese translation tasks show that our proposed method outperforms state-of-the-art SMT and NMT baselines.

Representation learning is a fundamental problem in natural language processing. This paper studies how to learn a structured representation for text classification. Unlike most existing representation models that either use no structure or rely on pre-specified structures, we propose a reinforcement learning (RL) method to learn sentence representation by discovering optimized structures automatically. We demonstrate two attempts to build structured representation: Information Distilled LSTM (ID-LSTM) and Hierarchically Structured LSTM (HS-LSTM). ID-LSTM selects only important, taskrelevant words, and HS-LSTM discovers phrase structures in a sentence. Structure discovery in the two representation models is formulated as a sequential decision problem: current decision of structure discovery affects following decisions, which can be addressed by policy gradient RL. Results show that our method can learn task-friendly representations by identifying important words or task-relevant structures without explicit structure annotations, and thus yields competitive performance.

Recursive structure is commonly found in the inputs of different modalities such as natural scene images or natural language sentences. Discovering this recursive structure helps us to not only identify the units that an image or sentence contains but also how they interact to form a whole. We introduce a max-margin structure prediction architecture based on recursive neural networks that can successfully recover such structure both in complex scene images as well as sentences. The same algorithm can be used both to provide a competitive syntactic parser for natural language sentences from the Penn Treebank and to outperform alternative approaches for semantic scene segmentation, annotation and classification. For segmentation and annotation our algorithm obtains a new level of state-of-theart performance on the Stanford background dataset (78.1%). The features from the image parse tree outperform Gist descriptors for scene classification by 4%.

We present the components of the ColumbiaGWU contribution to the 2016 TAC KBP BeSt Evaluation.

Community question answering aims at choosing the most appropriate answer for a given question, which is important in many NLP applications. Previous neural network-based methods consider several different aspects of information through calculating attentions. These different kinds of attentions are always simply summed up and can be seen as a “single view”, causing severe information loss. To overcome this problem, we propose a Multi-View Fusion Neural Network, where each attention component generates a “view” of the QA pair and a fusion RNN integrates the generated views to form a more holistic representation. In this fusion RNN method, a filter gate collects important information of input and directly adds it to the output, which borrows the idea of residual networks. Experimental results on the WikiQA and SemEval-2016 CQA datasets demonstrate that our proposed model outperforms the state-of-the-art methods.

Answer selection in community question answering (cQA) is a challenging task in natural language processing. The difficulty lies in that it not only needs the consideration of semantic matching between question answer pairs but also requires a serious modeling of contextual factors. In this letter, we propose an attentive deep neural network architecture so as to learn the deterministic information for answer selection. The architecture can support various input formats through the organization of convolutional neural networks, attention-based long short-term memory, and conditional random fields. Experiments are carried out on the SemEval-2015 cQA dataset. We attain 58.35% on macroaveraged $\mathrm{F_1}$, which outperforms the Top-1 system in the shared task by 1.16% and improves the state-of-the-art deep-neural-network-based method by 2.21%.

While Named Entity extraction is useful in many natural language applications, the coarse categories that most NE extractors work with prove insufficient for complex applications such as Question Answering and Ontology generation. We examine one coarse category of named entities, persons, and describe a method for automatically classifying person instances into eight finergrained subcategories. We present a supervised learning method that considers the local context surrounding the entity as well as more global semantic information derived from topic signatures and WordNet. We reinforce this method with an algorithm that takes advantage of the presence of entities in multiple contexts.

Type-level word embeddings use the same set of parameters to represent all instances of a word regardless of its context, ignoring the inherent lexical ambiguity in language. Instead, we embed semantic concepts (or synsets) as defined in WordNet and represent a word token in a particular context by estimating a distribution over relevant semantic concepts. We use the new, context-sensitive embeddings in a model for predicting prepositional phrase (PP) attachments and jointly learn the concept embeddings and model parameters. We show that using context-sensitive embeddings improves the accuracy of the PP attachment model by 5.4% absolute points, which amounts to a 34.4% relative reduction in errors.

Convolutional Neural Networks (CNNs) have recently achieved remarkably strong performance on sentence classification tasks (Kim, 2014; Kalchbrenner et al., 2014). However, these models require practitioners to specify the exact model architecture and accompanying hyperparameters, e.g., the choice of filter region size, regularization parameters, and so on. It is currently unknown how sensitive model performance is to changes in these configurations for the task of sentence classification. We thus conduct a sensitivity analysis of one-layer CNNs to explore the effect of architecture components on model performance; our aim is to distinguish between important and comparatively inconsequential design decisions for sentence classification. We focus on one-layer CNNs (to the exclusion of more complex models) due to their comparative simplicity and strong empirical performance (Kim, 2014). We derive practical advice from our extensive empirical results for those interested in getting the most out of CNNs for sentence classification. One important observation borne out by our experimental results is that researchers should report performance variances, as these can be substantial due to stochastic initialization and inference.


Question: A real-time inter-frame histogram construction method for SPAD image sensors.

This paper details the development and evaluation of a multifunctional 64 × 32 CMOS single-photon avalanche diode (SPAD) array, designed for high-speed 2-D imaging and 3-D ranging applications. Fabricated using a high-voltage 0.35-μm CMOS process, the chip features 2048 pixels, each equipped with a low-noise 30-μm SPAD capable of operating at 100 cps at 5-V excess bias, a prompt avalanche sensing circuit, and digital processing components. The array is versatile, delivering two-dimensional intensity data through photon counting in both free-running (down to 10-μs integration time) and time-gated modes, as well as facilitating smart light demodulation with in-pixel background suppression. This feature allows for phase-resolved imaging, enabling the extraction of three-dimensional depth-resolved images or decay lifetime maps by measuring the phase shift between a modulated excitation light and the reflected photons. The pixel-level memories enable fully parallel processing and global-shutter readout, eliminating motion artifacts and partial exposure issues. The array is capable of capturing high-speed optical events at an impressive frame rate of up to 100 000 fps, even at the single-photon level. The low-noise SPADs provide a high dynamic range of up to 110 dB at 100 fps, with a peak photon detection efficiency of nearly 50% at 410 nm. The SPAD imager supports various operating modes, making it suitable for both time-domain applications such as fluorescence lifetime imaging (FLIM) and fluorescence correlation spectroscopy, as well as frequency-domain FLIM and lock-in 3-D ranging, ideal for automotive vision and lidar systems.

We present our studies on small-sized pixel structures for high-resolution CMOS image sensors. To minimize the number of pixel components, single-transistor pixel and 2T pixel architecture were proposed. To deal with crosstalk between pixels, MOS capacitor deep trench isolation (CDTI) was integrated. CDTI-integrated pixel allows better achievements in dark current and full-well capacity in comparison with the configuration integrating oxide-filled deep trench isolation (DTI). To improve quantum efficiency (QE) and minimize optical crosstalk, back-side illumination (BSI) was developed. Also, vertical photodiode was proposed to maximize its charge-collection region. To take advantages of these structures/technologies, we developed two pixel options (P-type and N-type) combining CDTI or DTI, BSI and vertical photodiode. All the presented pixel structures were designed in 1.4µm-pitch sensor arrays, fabricated and tested.

This paper presents a novel technique for camera tampering detection. It is implemented in real-time and was developed for use in surveillance and security applications. This method identifies camera tampering by detecting large differences between older frames of video and more recent frames. A buffer of incoming video frames is kept and three different measures of image dissimilarity are used to compare the frames. After normalization, a set of conditions is tested to decide if camera tampering has occurred. The effects of adjusting the internal parameters of the algorithm are examined. The performance of this method is shown to be extremely favorable in real-world settings.

The paper considers two problems associated with the detection and classification of motion in image sequences obtained from a static camera. Motion is detected by differencing a reference and the "current" image frame, and therefore requires a suitable reference image and the selection of an appropriate detection threshold. Several threshold selection methods are investigated, and an algorithm based on hysteresis thresholding is shown to give acceptably good results over a number of test image sets. The second part of the paper examines the problem of detecting shadow regions within the image which are associated with the object motion. This is based on the notion of a shadow as a semi-transparent region in the image which retains a (reduced contrast) representation of the underlying surface pattern, texture or grey value. The method uses a region growing algorithm which uses a growing criterion based on a fixed attenuation of the photometric gain over the shadow region, in comparison to the reference image.

This paper presents a novel and efficient texture-based method for modeling the background and detecting moving objects from a video sequence. Each pixel is modeled as a group of adaptive local binary pattern histograms that are calculated over a circular region around the pixel. The approach provides us with many advantages compared to the state-of-the-art. Experimental results clearly justify our model.

The paper presents an early fire-alarm raising method based on video processing. The basic idea of the proposed of fire-detection is to adopt a RGB (red, green, blue) model based chromatic and disorder measurement for extracting fire-pixels and smoke-pixels. The decision function of fire-pixels is mainly deduced by the intensity and saturation of R component. The extracted fire-pixels will be verified if it is a real fire by both dynamics of growth and disorder, and further smoke. Based on iterative checking on the growing ratio of flames, a fire-alarm is given when the alarm-raising condition is met. Experimental results show that the developed technique can achieve fully automatic surveillance of fire accident with a lower false alarm rate and thus is very attractive for the important military, social security, commercial applications, and so on, at a general cost.

The paper presents a system for automatic, geo-registered, real-time 3D reconstruction from video of urban scenes. The system collects video streams, as well as GPS and inertia measurements in order to place the reconstructed models in geo-registered coordinates. It is designed using current state of the art real-time modules for all processing steps. It employs commodity graphics hardware and standard CPU’s to achieve real-time performance. We present the main considerations in designing the system and the steps of the processing pipeline. Our system extends existing algorithms to meet the robustness and variability necessary to operate out of the lab. To account for the large dynamic range of outdoor videos the processing pipeline estimates global camera gain changes in the feature tracking stage and efficiently compensates for these in stereo estimation without impacting the real-time performance. The required accuracy for many applications is achieved with a two-step stereo reconstruction process exploiting the redundancy across frames. We show results on real video sequences comprising hundreds of thousands of frames.

This paper proposes a method for transferring the RGB color spectrum to near-infrared (NIR) images using deep multi-scale convolutional neural networks. A direct and integrated transfer between NIR and RGB pixels is trained. The trained model does not require any user guidance or a reference image database in the recall phase to produce images with a natural appearance. To preserve the rich details of the NIR image, its high frequency features are transferred to the estimated RGB image. The presented approach is trained and evaluated on a real-world dataset containing a large amount of road scene images in summer. The dataset was captured by a multi-CCD NIR/RGB camera, which ensures a perfect pixel to pixel registration.

In this paper, a moving object detection algorithm is combined with a background estimate and a Bing (Binary Norm Gradient) object is proposed in video surveillance. A simple background estimation method is used to detect rough images of a group of moving foreground objects. The foreground setting in the foreground will estimate another set of candidate object windows, and the target (pedestrian / vehicle) from the intersection area comes from the first two steps. In addition, the time cost is reduced by the estimated area. Experiments on outdoor datasets show that the proposed method can not only achieve high detection rate (DR), but also reduce false alarm rate (FAR) and time cost.

This paper describes a machine learning approach for visual object detection which is capable of processing images extremely rapidly and achieving high detection rates. This work is distinguished by three key contributions. The first is the introduction of a new image representation called the “Integral Image” which allows the features used by our detector to be computed very quickly. The second is a learning algorithm, based on AdaBoost, which selects a small number of critical visual features from a larger set and yields extremely efficient classifiers[6]. The third contribution is a method for combining increasingly more complex classifiers in a “cascade” which allows background regions of the image to be quickly discarded while spending more computation on promising object-like regions. The cascade can be viewed as an object specific focus-of-attention mechanism which unlike previous approaches provides statistical guarantees that discarded regions are unlikely to contain the object of interest. In the domain of face detection the system yields detection rates comparable to the best previous systems. Used in real-time applications, the detector runs at 15 frames per second without resorting to image differencing or skin color detection.

We analyze the computer vision task of pixel-level background subtraction. We present recursive equations that are used to constantly update the parameters of a Gaussian mixture model and to simultaneously select the appropriate number of components for each pixel. We also present a simple non-parametric adaptive density estimation method. The two methods are compared with each other and with some previously proposed algorithms. 2005 Elsevier B.V. All rights reserved.


Question: A real-time inter-frame histogram construction method for single-photon avalanche diode (SPAD) image sensors.

In the realm of high-performance single-photon imaging systems, CMOS single photon avalanche diodes (SPADs) are emerging as a critical technology. These systems are integral to applications such as fluorescence lifetime imaging, time-of-flight positron emission tomography, and Raman Spectroscopy. The adoption of CMOS SPADs is driven by their miniaturization, cost-effectiveness, and superior performance, which are facilitated by their integration with advanced mixed-signal circuits in deep-submicron (DSM) technology. The core component, the SPAD, boasts single-photon sensitivity and sub-nanosecond time resolution, complemented by DSM CMOS circuitry that excels in high-speed, low-power processing. This paper explores recent advancements in DSM CMOS SPAD detectors, circuits, and arrays, delving into scalability, miniaturization, and performance trade-offs. It also scrutinizes the design aspects, research hurdles, and prospective avenues for CMOS SPAD image sensors, underscoring their pivotal role in next-generation imaging systems.

A skip and fill algorithm is developed to digitally self-calibrate pipelined analog-to-digital converters (ADC’s) in real time. The proposed digital calibration technique is applicable to capacitor-ratioed multiplying digital-to-analog converters (MDAC’s) commonly used in multi-step or pipelined ADC’s. This background calibration process can replace, in effect, a trimming procedure usually done in the factory with a hidden electronic calibration. Unlike other self-calibration techniques working in the foreground, the proposed technique is based on the concept of skipping conversion cycles randomly but filling in data later by nonlinear interpolation. This opens up the feasibility of digitally implementing calibration hardware and simplifying the task of self-calibrating multi-step or pipelined ADC’s. The proposed method improves the performance of the inherently fast ADC’s by maintaining simple system architectures. To measure errors resulting from capacitor mismatch, op amp dc gain, offset, and switch feedthrough in real time, the calibration test signal is injected in place of the input signal using a split-reference injection technique. Ultimately, the missing signal within 2/3 of the Nyquist bandwidth is recovered with 16-bit accuracy using a 44-th order polynomial interpolation, behaving essentially as an FIR filter. This work was supported by the National Science Foundation under Grant MIP-9312671.

Positron emission tomography (PET) is a nuclear functional imaging technique that produces a three-dimensional image of functional organs in the body. PET requires high resolution, fast and low power multichannel analog to digital converter (ADC). A typical multichannel ADC for PET scanner architecture consists of several blocks. Most of the blocks can be designed by using fast, low power D flip-flops. A preset-able true single phase clocked (TSPC) D flip-flop shows numerous glitches (noise) at the output due to unnecessary toggling at the intermediate nodes. Preset-able modified TSPC (MTSPC) D flip-flop have been proposed as an alternative solution to alleviate this problem. However, the MTSPC D flip-flop requires one extra PMOS to suspend toggling of the intermediate nodes. In this work, we designed a 7-bit preset-able gray code counter by using the proposed D flip-flop. This work involves UMC 180 nm CMOS technology for preset-able 7-bit gray code counter where we achieved 1 GHz maximum operation frequency with most significant bit (MSB) delay 0.96 ns, power consumption 244.2 μW (micro watt) and power delay product (PDP) 0.23 pJ (Pico joule) from 1.8 V power supply.

The retinex is a human perception-based image processing algorithm which provides color constancy and dynamic range compression. We have previously reported on a single-scale retinex (SSR) and shown that it can either achieve color/lightness rendition or dynamic range compression, but not both simultaneously. We now present a multi-scale retinex (MSR) which overcomes this limitation for most scenes. Both color rendition and dynamic range compression are successfully accomplished except for some \patho-logical" scenes that have very strong spectral characteristics in a single band.

The development of new Simultaneous Localization and Mapping (SLAM) techniques is quickly advancing in research communities and rapidly transitioning into commercial products. Creating accurate and high-quality SLAM maps relies on a robust initialization process. However, the robustness and usability of SLAM initialization for end-users has often been disregarded. This paper presents and evaluates a novel tracking system for 6DOF pose tracking between a single keyframe and the current camera frame, without any prior scene knowledge. Our system is particularly suitable for SLAM initialization, since it allows 6DOF pose tracking in the intermediate frames before a wide-enough baseline between two keyframes has formed. We investigate how our tracking system can be used to interactively guide users in performing an optimal motion for SLAM initialization. However, our findings from a pilot study indicate that the need for such motion can be completely hidden from the user and outsourced to our tracking system. Results from a second user study show that letting our tracking system create a SLAM map as soon as possible is a viable and usable solution. Our work provides important insight for SLAM systems, showing how our novel tracking system can be integrated with a user interface to support fast, robust and user-friendly SLAM initialization.

In this paper, we present work in progress towards the generation of a ground truth data set for High Dynamic Range (HDR) stereo matching. The development and evaluation of novel stereo matching algorithms that are tailored to the characteristics of HDR images would greatly benefit from the availability of such reference data. We describe our laboratory setup and processing steps for acquiring multi-exposed stereo images along with a corresponding reference disparity map computed by a structured light approach. We discuss the special requirements and challenges which HDR test scenes impose on the computation of the reference disparities and show some preliminary results.

A 12-bit SAR ADC architecture with dual reference shifting and interpolation technique has been proposed and implemented with 8-way time interleaving in 65nm CMOS. The proposed technique converts 4 bits per SAR conversion cycle with reduced overhead, which is a key to achieve both high speed and resolution while maintaining low power consumption. The measured peak SNDR is 72dB and remains above 65.3dB at 1-GHz input frequency at sample rate of 1.6 GS/s. It achieves a record power efficiency of 17.8fJ/conv-step among the recently published high-speed/resolution ADCs.

Pedestrian detection will play a major role in future driver assistance and autonomous driving. One powerful algorithm in this field uses HOG features to describe the specific properties of pedestrians in images. To determine their locations, features are extracted and classified window-wise from different scales of an input image. The results of the classification are finally merged to remove overlapping detections. The real-time execution of this method requires specific FPGA- or ASIC-architectures. Recent work focused on accelerating the feature extraction and classification. Although merging is an important step in the algorithm, it is only rarely considered in hardware implementations. A reason for that could be its complexity and irregularity that is not trivial to implement in hardware. In this paper, we present a new bottom-up FPGA architecture that maps the full HOG-based algorithm for pedestrian detection including feature extraction, SVM classification, and multi-scale processing in combination with merging. For that purpose, we also propose a new hardware-optimized merging method. The resulting architecture is highly efficient. Additionally, we present an FPGA-based full real-time and multi-scale pedestrian detection demonstration system.

This paper relates first experiences using a state-of-the-art, time-of-flight sensor that is able to deliver 3D images. The properties and capabilities of the sensor make it a potential powerful tool for applications within mobile robotics especially for real-time tasks, as the sensor features a frame rate of up to 30 frames per second. Its capabilities in terms of basic obstacle avoidance and local path-planning are evaluated and compared to the performance of a standard laser scanner.

We provide a novel hardware and software system for micro air vehicles (MAV) that allows high-speed, low-latency onboard image processing. It uses up to four cameras in parallel on a miniature rotary wing platform. The MAV navigates based on onboard processed computer vision in GPS-denied in- and outdoor environments. It can process in parallel images and inertial measurement information from multiple cameras for multiple purposes (localization, pattern recognition, obstacle avoidance) by distributing the images on a central, low-latency image hub. Furthermore the system can utilize low-bandwith radio links for communication and is designed and optimized to scale to swarm use. Experimental results show successful flight with a range of onboard computer vision algorithms, including localization, obstacle avoidance and pattern recognition.

Online 3D reconstruction of real-world scenes has been attracting increasing interests from both the academia and industry, especially with the consumer-level depth cameras becoming widely available. Recent most online reconstruction systems take live depth data from a moving Kinect camera and incrementally fuse them to a single high-quality 3D model in real time. Although most real-world scenes have static environment, the daily objects in a scene often move dynamically, which are non-trivial to reconstruct especially when the camera is also not still. To solve this problem, we propose a single depth camera-based real-time approach for simultaneous reconstruction of dynamic object and static environment, and provide solutions for its key issues. In particular, we first introduce a robust optimization scheme which takes advantage of raycasted maps to segment moving object and background from the live depth map. The corresponding depth data are then fused to the volumes, respectively. These volumes are raycasted to extract views of the implicit surface which can be used as a consistent reference frame for the next iteration of segmentation and tracking. Particularly, in order to handle fast motion of dynamic object and handheld camera in the fusion stage, we propose a sequential 6D pose prediction method which largely increases the registration robustness and avoids registration failures occurred in conventional methods. Experimental results show that our approach can reconstruct moving object as well as static environment with rich details, and outperform conventional methods in multiple aspects.


Question: A real-time inter-frame histogram construction method for SPAD image sensors.

This work details the development of a novel camera prototype designed for capturing 2D and 3D images in environments with minimal light, utilizing a single-photon avalanche-diode (SPAD) image sensor for direct time-of-flight (d-ToF) measurements. The sensor comprises a 64x64 pixel array, each equipped with an in-pixel time-to-digital converter (TDC) to facilitate high-speed frame acquisition. Advanced circuit design methodologies have been integrated to ensure robust 3D imaging under challenging conditions characterized by low sensitivity and high levels of uncorrelated noise, including dark counts and background illumination. Notably, the design includes a pioneering time-gated front-end for the SPAD detector, a reverse start-stop mechanism, and real-time image reconstruction at an impressive Ikfps. To our knowledge, this is the inaugural ToF camera based on a SPAD sensor that has been successfully fabricated and validated for 3D image reconstruction using a standard CMOS process, without the need for specialized opto-flavor or high voltage options. The camera boasts a depth resolution of 1cm, operating effectively at illumination power levels as low as 0.1nW/mm2, down from less than 6nW/mm2.

The essence of the research revolves around the innovation of a real-time inter-frame histogram builder for SPAD image sensors, which is a crucial component in the development of a high-speed, low-light imaging system. This system, centered around a SPAD image sensor, is capable of direct time-of-flight measurement, enabling it to capture 3D images with exceptional clarity under low-light conditions. The sensor's architecture, featuring a 64x64 pixel array with in-pixel time-to-digital converters, allows for high frame rate acquisition, making it ideal for real-time applications. Moreover, the incorporation of advanced circuit design techniques ensures that the camera can perform effectively in scenarios with high levels of uncorrelated noise, such as dark counts and background illumination. The introduction of a time-gated front-end for the SPAD detector, a reverse start-stop scheme, and real-time image reconstruction at a rate of Ikfps, highlights the system's capability to operate under challenging environmental conditions. This pioneering camera, fabricated using a standard CMOS process without the need for specialized opto-flavor or high voltage options, achieves a depth resolution of 1cm at illumination power as low as 0.1nW/mm2, demonstrating its potential for applications in various fields requiring high-precision, low

This paper introduces a new compositional framework for classifying color correction methods according to their two main computational units. The framework was used to dissect fifteen among the best color correction algorithms and the computational units so derived, with the addition of four new units specifically designed for this work, were then reassembled in a combinatorial way to originate about one hundred distinct color correction methods, most of which never considered before. The above color correction methods were tested on three different existing datasets, including both real and artificial color transformations, plus a novel dataset of real image pairs categorized according to the kind of color alterations induced by specific acquisition setups. Differently from previous evaluations, special emphasis was given to effectiveness in real world applications, such as image mosaicing and stitching, where robustness with respect to strong image misalignments and light scattering effects is required. Experimental evidence is provided for the first time in terms of the most recent perceptual image quality metrics, which are known to be the closest to human judgment. Comparative results show that combinations of the new computational units are the most effective for real stitching scenarios, regardless of the specific source of color alteration. On the other hand, in the case of accurate image alignment and artificial color alterations, the best performing methods either use one of the new computational units, or are made up of fresh combinations of existing units.

Reliable real-time detection of traffic lights is a major concern for the task of autonomous driving. As deep convolutional networks have proven to be a powerful tool in visual object detection, we propose DeepTLR, a camera-based system for real-time detection and classification of traffic lights. Detection and state classification are realized using a single deep convolutional network. DeepTLR does not use any prior knowledge about traffic light locations. Also the detection is executed frame by frame without using temporal information. It is able to detect traffic lights on the whole camera image without any presegmentation. This is achieved by classifying each fine-grained pixel region of the input image and performing a bounding box regression on regions of each class. We show that our algorithm is able to run on frame-rates required for real-time applications while reaching notable results.

We present a real-time face tracker in this paper The system has achieved a rate of 30% frameshecond using an HP-9000 workstation with a framegrabber and a Canon VC-CI camera. It can track a person 'sface while the person moves freely (e.g., walks, jumps, sits down and stands up) in a room. Three types of models have been employed in developing the system. First, we present a stochastic model to characterize skin-color distributions of human faces. The information provided by the model is sufJicient for tracking a human face in various poses and views. This model is adaptable to different people and different lighting conditions in real-time. Second, a motion model e's used to estimate image motion and to predict search window. Third, a camera model is used toprediet and to compensate for camera motion. The system can be applied to tele-conferencing and many HCI applications including lip-reading and gaze tracking. The principle in developing this system can be extended to other tracking problems such as tracking the human hand.

This paper proposes and validates a real-time on-road vehicle detection system, which uses a single camera for the purpose of intelligent driver assistance. A three-step vehicle detection framework is presented to detect and track the target vehicle within an image. In the first step, probable vehicle locations are hypothesized using pattern recognition. The vehicle candidates are then verified in the hypothesis verification step. In this step, lane detection is used to filter vehicle candidates that are not within the lane region of interest. In the final step tracking and online learning are implemented to optimize the detection algorithm during misdetection and temporary occlusion. Good detection performance and accuracy was observed in highway driving environments with minimal shadows.

We present a fast and precise vision-based software intended for multiple robot localization. The core component of the proposed localization system is an efficient method for black and white circular pattern detection. The method is robust to variable lighting conditions, achieves sub-pixel precision, and its computational complexity is independent of the processed image size. With off-the-shelf computational equipment and low-cost camera, its core algorithm is able to process hundreds of images per second while tracking hundreds of objects with millimeter precision. We propose a mathematical model of the method that allows to calculate its precision, area of coverage, and processing speed from the camera's intrinsic parameters and hardware's processing capacity. The correctness of the presented model and performance of the algorithm in real-world conditions are verified in several experiments. Apart from the method description, we also publish its source code; so, it can be used as an enabling technology for various mobile robotics problems.

Computational techniques involving contrast enhancement and noise filtering on two-dimensional image arrays are developed based on their local mean and variance. These algorithms are nonrecursive and do not require the use of any kind of transform. They share the same characteristics in that each pixel is processed independently. Consequently, this approach has an obvious advantage when used in real-time digital image processing applications and where a parallel processor can be used. For both the additive and multiplicative cases, the a priori mean and variance of each pixel is derived from its local mean and variance. Then, the minimum mean-square error estimator in its simplest form is applied to obtain the noise filtering algorithms. For multiplicative noise a statistical optimal linear approximation is made. Experimental results show that such an assumption yields a very effective filtering algorithm. Examples on images containing 256 × 256 pixels are given. Results show that in most cases the techniques developed in this paper are readily adaptable to real-time image processing.

This is Part II of the paper, "Gray-Level Grouping (GLG): an Automatic Method for Optimized Image Contrast Enhancement". Part I of this paper introduced a new automatic contrast enhancement technique: gray-level grouping (GLG). GLG is a general and powerful technique, which can be conveniently applied to a broad variety of low-contrast images and outperforms conventional contrast enhancement techniques. However, the basic GLG method still has limitations and cannot enhance certain classes of low-contrast images well, e.g., images with a noisy background. The basic GLG also cannot fulfill certain special application purposes, e.g., enhancing only part of an image which corresponds to a certain segment of the image histogram. In order to break through these limitations, this paper introduces an extension of the basic GLG algorithm, selective gray-level grouping (SGLG), which groups the histogram components in different segments of the grayscale using different criteria and, hence, is able to enhance different parts of the histogram to various extents. This paper also introduces two new preprocessing methods to eliminate background noise in noisy low-contrast images so that such images can be properly enhanced by the (S)GLG technique. The extension of (S)GLG to color images is also discussed in this paper. SGLG and its variations extend the capability of the basic GLG to a larger variety of low-contrast images, and can fulfill special application requirements. SGLG and its variations not only produce results superior to conventional contrast enhancement techniques, but are also fully automatic under most circumstances, and are applicable to a broad variety of images.

Some state-of-the-art video stabilization methods can achieve quite good visual effect, but they always cost a lot of time. On the other hand, current real-time video stabilization methods cannot generate satisfactory results. In this paper, we propose a novel trajectory-based video stabilization method which can generate high-quality results in real time. Our method runs very fast, because many techniques are proposed for acceleration. In the trajectory smoothing step, trajectories are extracted, pre-processed, and smoothed. A video splitting algorithm is proposed for pre-processing, and binomial filtering is used for smoothing. Both of them are simple and fast. In the frame warping step, we calculate a spatially varying warp that is directed by a grid mesh for each frame. Instead of solving time consuming global optimization problems, the transformation matrix of each grid is calculated using nearby trajectories in our method, leading to very high speed. We implement our method and run it on a variety of videos. Experiments show that while the stabilization effect is comparable with state-of-the-art methods, our algorithm can run in real time.

Paul P. K. Lee, Russell C. Gee*, R. Michael Guidash, T-H. Lee, and Eric R. Fossum* Microelectronics Technology Division, Eastman Kodak Company 1669 Lake Avenue, Rochester, New York 14650-2008 Tel: (716) 477-2869, Fax: (716) 477-4947, Internet: ppklee@Kodak.COM *Jet Propulsion Laboratory, California Institute of Technology 4800 Oak Grove Drive, Pasadena, CA 91109 Abstract This paper describes the integration of the active pixel sensor (APS) architecture normally fabricated in conventional complementary metal oxide semiconductor (CMOS) technology with a pinned photodiode (PPD) device using a mixed process technology. This new technology allows mix and match of CMOS and high performance charged coupled device (CCD) modules. The PPD [1] becomes the photoactive element in an XYaddressable area array with each pixel containing active devices for the transfer, readout and reset functions. It is a standard photo-sensitive element, available in a high performance true two-phase CCD technology developed previously for CCD-based image sensors [2]. An n-well 2-μm CMOS technology was combined with the CCD process to provide the best features from both. A Design of Experiment approach was used with Technology Computer Aided Design (TCAD) tools to develop and optimize the new mixed process technology without sacrificing any CCD performance while minimizing impact to the CMOS device characteristics [3]. By replacing polysilicon photocapacitor or photogate in conventional APS with the pinned photodiode, deficiencies in poor blue response and high dark current are minimized [4].

A typical pedestrian protection system requires sophisticated hardware and robust detection algorithms. To solve these problems the existing systems use hybrid sensors where mono and stereo vision merged with active sensors. One of the most assuring pedestrian detection sensors is far infrared range camera. The classical pedestrian detection approach based on Histogram of oriented gradients is not robust enough to be applied in devices which consumers can trust. An application of deep neural network-based approach is able to perform with significantly higher accuracy. However, the deep learning approach requires a high number of labeled data examples. The investigation presented in this paper aimed the acceleration of pedestrian labeling in far-infrared image sequences. In order to accelerate pedestrian labeling in far-infrared camera videos, we have integrated the YOLOv3 object detector into labeling software. The verification of the pre-labeled results was around eleven times faster than manual labeling of every single frame.


Question: A real-time inter-frame histogram construction method for SPAD image sensors.

This paper introduces an integrated time-correlated single-photon-counting (TCSPC) sensor that significantly reduces pile-up mechanisms by an order of magnitude through an ultra-parallel approach to photon detection and time-resolving hardware. TCSPC is a photon-efficient method widely used in applications like time-of-flight (ToF) range measurement, ToF 3D imaging, scanning optical microscopy, diffuse optical tomography, and Raman sensing. Conventional TCSPC systems often struggle with high-intensity optical waveforms due to limited detection and conversion channels, leading to pile-up distortion. However, this novel sensor design employs an ultra-parallel TDC architecture that merges the time-code generation and histogram conversion processes into a single, direct-to-histogram conversion. This results in a sensor capable of 71.4ps resolution, with a dynamic range exceeding 18.85ns, and a throughput of 14GS/s. The sensor can handle 1.7Gphotons/s and produce 21k histograms/s, each capturing a total of 1.7kphotons within a 1μs exposure period, showcasing its ability to process high-dynamic-range optical waveforms efficiently. The integration of this advanced TDC architecture on a single chip represents a significant advancement in real-time inter-frame histogram building for single-photon avalanche diode (SPAD) image sensors.

This paper describes a machine learning approach for visual object detection which is capable of processing images extremely rapidly and achieving high detection rates. This work is distinguished by three key contributions. The first is the introduction of a new image representation called the “Integral Image” which allows the features used by our detector to be computed very quickly. The second is a learning algorithm, based on AdaBoost, which selects a small number of critical visual features from a larger set and yields extremely efficient classifiers[6]. The third contribution is a method for combining increasingly more complex classifiers in a “cascade” which allows background regions of the image to be quickly discarded while spending more computation on promising object-like regions. The cascade can be viewed as an object specific focus-of-attention mechanism which unlike previous approaches provides statistical guarantees that discarded regions are unlikely to contain the object of interest. In the domain of face detection the system yields detection rates comparable to the best previous systems. Used in real-time applications, the detector runs at 15 frames per second without resorting to image differencing or skin color detection.

Image captured under various environment conditions with low contrast results in visual deterioration. To enhance the contrast of images a number of techniques have been proposed over the years. Histogram equalization is one of those methods which could be used for this purpose. For the contrast enhancement purpose histogram equalization (HE) attempts reduction in the number of gray levels. Contrast enhancement by histogram equalization method involve mapping of gray levels on the basis of changeover function that is derived from the probability distribution of gray levels embedded in input image. For this, the neighbouring gray levels in image having low probability density values are combined into single gray level value. Whereas for neighbouring gray levels having high probability density value the gap between them is increased, HE often makes over-enhancement of images with frequent gray level region resulting in loss of contrast for less frequent area of the images. Thus, the HE technique is not able to preserve image brightness. So wherever the preservation of image brightness is required this method is not preferred. This paper review various histogram equalization techniques used in preserving image brightness as well as the contrast enhancement is presented. The key approach for this aim is to use image segmentation. The criteria used are decomposition of images and the presentation of the basic difference between these techniques. For comparative study of performance of these techniques, absolute mean brightness error (AMBE) is used to determine degree of brightness preservation while peak signal to noise ratio (PSNR) have been used to fine\ degree of contrast enhancement.

■ Compact, efficient and easy to program digital signal processors execute algorithms once too computationally expensive for real-time applications. ■ Modeling and simulation programs more accurately predict the effects that physical ocean parameters have on the performance of imaging systems under different geometric configurations. ■ Image processing algorithms that handle data from multiple synchronous sources and that can extract and match feature points from each such source derive accurate 3-D scene information. ■ Digital compression schemes provide high-quality standardizations for increased data transfer rates (i.e. streaming video) and reduced storage requirements. This paper reports developments over the past three years in the following topics: ■ Image formation and image processing methods; ■ Extended range imaging techniques;

The most prevailing approach now for parking lot vacancy detecting system is to use sensor-based techniques. The main impediments to the camera-based system in applying to parking lots on rooftop and outside building are the glaring sun light and dark shadows in the daytime, and low-light intensity and back-lighting in the nighttime. To date, no camera-based detecting systems for outdoor parking lots have been in practical use. A few engineering firms provide the camera-based system, which is only for underground and indoor parking lots. This paper reports on the new camera based system called ParkLotD for detecting vacancy/occupancy in parking lots. ParkLotD uses a classifier based on fuzzy c-means (FCM) clustering and hyper-parameter tuning by particle swarm optimization (PSO). The test result of the detection error rate for the indoor multi-story parking lot has improved by an order of magnitude compared to the current system based on the edge detection approach. ParkLotD demonstrates high detection performance and enables the camera-based system to achieve the practical use in outdoor parking lots.

Road traffic density has always been a concern in large cities around the world, and many approaches were developed to assist in solving congestions related to slow traffic flow. This work proposes a congestion rate estimation approach that relies on real-time video scenes of road traffic, and was implemented and evaluated on eight different hotspots covering 33 different urban roads. The approach relies on road scene morphology for estimation of vehicles average speed along with measuring the overall video scenes randomness acting as a frame texture analysis indicator. Experimental results shows the feasibility of the proposed approach in reliably estimating traffic density and in providing an early warning to drivers on road conditions, thereby mitigating the negative effect of slow traffic flow on their daily lives.

An algorithm is presented for the rapid solution of the phase of the complete wave function whose intensity in the diffraction and imaging planes of an imaging system are known. A proof is given showing that a defined error between the estimated function and the correct function must decrease as the algorithm iterates. The problem of uniqueness is discussed and results are presented demonstrating the power of the method.

DTAM is a system for real-time camera tracking and reconstruction which relies not on feature extraction but dense, every pixel methods. As a single hand-held RGB camera flies over a static scene, we estimate detailed textured depth maps at selected keyframes to produce a surface patchwork with millions of vertices. We use the hundreds of images available in a video stream to improve the quality of a simple photometric data term, and minimise a global spatially regularised energy functional in a novel non-convex optimisation framework. Interleaved, we track the camera's 6DOF motion precisely by frame-rate whole image alignment against the entire dense model. Our algorithms are highly parallelisable throughout and DTAM achieves real-time performance using current commodity GPU hardware. We demonstrate that a dense model permits superior tracking performance under rapid motion compared to a state of the art method using features; and also show the additional usefulness of the dense model for real-time scene interaction in a physics-enhanced augmented reality application.

Digital photography has made it possible to quickly and easily take a pair of images of low-light environments: one with flash to capture detail and one without flash to capture ambient illumination. We present a variety of applications that analyze and combine the strengths of such flash/no-flash image pairs. Our applications include denoising and detail transfer (to merge the ambient qualities of the no-flash image with the high-frequency flash detail), white-balancing (to change the color tone of the ambient image), continuous flash (to interactively adjust flash intensity), and red-eye removal (to repair artifacts in the flash image). We demonstrate how these applications can synthesize new images that are of higher quality than either of the originals.

We integrate the cascade-of-rejectors approach with the Histograms of Oriented Gradients (HoG) features to achieve a fast and accurate human detection system. The features used in our system are HoGs of variable-size blocks that capture salient features of humans automatically. Using AdaBoost for feature selection, we identify the appropriate set of blocks, from a large set of possible blocks. In our system, we use the integral image representation and a rejection cascade which significantly speed up the computation. For a 320 × 280 image, the system can process 5 to 30 frames per second depending on the density in which we scan the image, while maintaining an accuracy level similar to existing methods.

In recent years, neural network accelerators have been shown to achieve both high energy efficiency and high performance for a broad application scope within the important category of recognition and mining applications. Still, both the energy efficiency and performance of such accelerators remain limited by memory accesses. In this paper, we focus on image applications, arguably the most important category among recognition and mining applications. The neural networks which are state-of-the-art for these applications are convolutional neural networks (CNNs), and they have an important property: weights are shared among many neurons, considerably reducing the neural network memory footprint. This property allows to entirely map a CNN within an SRAM, eliminating all DRAM accesses for weights. By further hoisting this accelerator next to the image sensor, it is possible to eliminate all remaining DRAM accesses, i.e., for inputs and outputs. In this paper, we propose such a CNN accelerator, placed next to a CMOS or CCD sensor. The absence of DRAM accesses combined with a careful exploitation of the specific data access patterns within CNNs allows us to design an accelerator which is highly energy-efficient. We present a single-core implementation down to the layout at 65 nm, with a modest footprint of 5.94mm $^{\boldsymbol {2}}$  and consuming only 336mW, but still about  $\boldsymbol {30\times }$  faster than high-end GPUs. For visual processing with higher resolution and frame-rate requirements, we further present a multicore implementation with elevated performance.


Question: A real-time inter-frame histogram construction method for SPAD image sensors.

A cutting-edge Time to Digital Converter (TDC) with direct histogram output, capable of handling multiple events, has been developed and integrated into a 65nm Xilinx Virtex 5 FPGA. This delay-line based design offers a temporal precision of 16.3 picoseconds across a 2.86 nanosecond dynamic range. Notably, the device boasts a conversion rate of 6.17 Gsamples/s and a sampling rate of 61.7 Gsamples/s—both of which are the highest reported values in the field. The system exhibits an impressive linearity with a -0.9/+3 LSB DNL and -1.5/+5 LSB INL. The TDC's efficacy is demonstrated in a direct time-of-flight optical ranging application, where it achieved an error of 12mm over a distance of 350mm. This advancement represents a significant leap in real-time histogram building capabilities for SPAD image sensors, showcasing the TDC's potential in enhancing imaging and ranging applications.

This paper presents a new coded structured light pattern which permits to solve the correspondence problem by a single shot and without using geometrical constraints. The pattern is composed by the projection of a grid made by coloured slits in such a way that each slit with its two neighbours appears only once in the pattern. The technique proposed permits a rapid and robust 3D scene measurement, even with moving objects. Ó 1998 Elsevier Science B.V. All rights reserved.

In this paper, we present a moving object detection system named Flux Tensor with Split Gaussian models (FTSG) that exploits the benefits of fusing a motion computation method based on spatio-temporal tensor formulation, a novel foreground and background modeling scheme, and a multi-cue appearance comparison. This hybrid system can handle challenges such as shadows, illumination changes, dynamic background, stopped and removed objects. Extensive testing performed on the CVPR 2014 Change Detection benchmark dataset shows that FTSG outperforms state-of-the-art methods.

This paper proposes an approach to real-time dense localisation and mapping that aims at unifying two different representations commonly used to define dense models. On one hand, much research has looked at 3D dense model representations using voxel grids in 3D. On the other hand, image-based key-frame representations for dense environment mapping have been developed. Both techniques have their relative advantages and disadvantages which will be analysed in this paper. In particular each representation's space-size requirements, their effective resolution, the computation efficiency, their accuracy and robustness will be compared. This paper then proposes a new model which unifies various concepts and exhibits the main advantages of each approach within a common framework. One of the main results of the proposed approach is its ability to perform large scale reconstruction accurately at the scale of mapping a building.

Road traffic density has always been a concern in large cities around the world, and many approaches were developed to assist in solving congestions related to slow traffic flow. This work proposes a congestion rate estimation approach that relies on real-time video scenes of road traffic, and was implemented and evaluated on eight different hotspots covering 33 different urban roads. The approach relies on road scene morphology for estimation of vehicles average speed along with measuring the overall video scenes randomness acting as a frame texture analysis indicator. Experimental results shows the feasibility of the proposed approach in reliably estimating traffic density and in providing an early warning to drivers on road conditions, thereby mitigating the negative effect of slow traffic flow on their daily lives.

Recently, a new ICT approach to agriculture called “Smart Agriculture” has been received great attention to support farmers' decision-making for good final yield on various kinds of field conditions. For this purpose, this paper presents two image sensing methods that enable an automatic observation to capture flowers and seedpods of soybeans in real fields. The developed image sensing methods are considered as sensors in an agricultural cyber-physical system in which big data on the growth status of agricultural plants and environmental information (e.g., weather, temperature, humidity, solar radiation, soil condition, etc.) are analyzed to mine useful rules for appropriate cultivation. The proposed image sensing methods are constructed by combining several image processing and machine learning techniques. The flower detection is realized based on a coarse-to-fine approach where candidate areas of flowers are first detected by SLIC and hue information, and the acceptance of flowers is decided by CNN. In the seedpod detection, candidates of seedpod regions are first detected by the Viola-Jones object detection method, and we also use CNN to make a final decision on the acceptance of detected seedpods. The performance of the proposed image sensing methods is evaluated for a data set of soybean images that were taken from a crowd of soybeans in real agricultural fields in Hokkaido, Japan.

How to make vision system work robustly under dynamic light conditions is still a challenging research focus in computer/robot vision community. In this paper, a novel camera parameters auto-adjusting technique based on image entropy is proposed. Firstly image entropy is defined and its relationship with camera parameters is verified by experiments. Then how to optimize the camera parameters based on image entropy is proposed to make robot vision adaptive to the different light conditions. The algorithm is tested by using the omnidirectional vision in indoor RoboCup Middle Size League environment and the perspective camera in outdoor ordinary environment, and the results show that the method is effective and color constancy to some extent can be achieved.

A simple block-based compressed-sensing reconstruction for still images is adapted to video. Incorporating reconstruction from a residual arising from motion estimation and compensation, the proposed technique alternatively reconstructs frames of the video sequence and their corresponding motion fields in an iterative fashion. Experimental results reveal that the proposed technique achieves significantly higher quality than a straightforward reconstruction that applies a still-image reconstruction independently frame by frame, a 3D reconstruction that exploits temporal correlation between frames merely in the form of a motion-agnostic 3D transform, and a similar, yet non-iterative, motion-compensated residual reconstruction.

This paper presents the vision-based road detection system currently installed onto the MOB-LAB land vehicle. Based on a geometrical transform and on a fast morphological processing, the system is capable to detect road markings even in extremely severe shadow conditions on at and structured roads. The use of a special-purpose massively architecture (PAPRICA) allows to achieve a processing rate of about 17 Hz.

A 12-bit SAR ADC architecture with dual reference shifting and interpolation technique has been proposed and implemented with 8-way time interleaving in 65nm CMOS. The proposed technique converts 4 bits per SAR conversion cycle with reduced overhead, which is a key to achieve both high speed and resolution while maintaining low power consumption. The measured peak SNDR is 72dB and remains above 65.3dB at 1-GHz input frequency at sample rate of 1.6 GS/s. It achieves a record power efficiency of 17.8fJ/conv-step among the recently published high-speed/resolution ADCs.

RGB-D (Red Green Blue and Depth) sensors are devices that can provide color and depth information from a scene at the same time. Recently, they have been widely used in many solutions due to their commercial growth from the entertainment market to many diverse areas (e.g., robotics, CAD, etc.). In the research community, these devices have had good uptake due to their acceptable levelofaccuracyformanyapplicationsandtheirlowcost,butinsomecases,theyworkatthelimitof their sensitivity, near to the minimum feature size that can be perceived. For this reason, calibration processes are critical in order to increase their accuracy and enable them to meet the requirements of such kinds of applications. To the best of our knowledge, there is not a comparative study of calibration algorithms evaluating its results in multiple RGB-D sensors. Speciﬁcally, in this paper, a comparison of the three most used calibration methods have been applied to three different RGB-D sensors based on structured light and time-of-ﬂight. The comparison of methods has been carried out by a set of experiments to evaluate the accuracy of depth measurements. Additionally, an object reconstruction application has been used as example of an application for which the sensor works at the limit of its sensitivity. The obtained results of reconstruction have been evaluated through visual inspection and quantitative measurements.


Question: A fully integrated voltage boost converter with maximum power point tracking for low-voltage energy harvesters, achieving 73.6% efficiency and a minimum input of 0.21V.

This paper presents an innovative charge pump circuit design that prioritizes gate-oxide reliability, specifically tailored for low-voltage CMOS processes. The design incorporates dual pumping branches and ensures that the charge transfer switches can be fully activated and deactivated, thereby achieving superior pumping efficiency compared to conventional designs. Notably, the circuit's design prevents any device from experiencing gate-source or gate-drain voltages that exceed the standard operating power supply voltage (VDD), safeguarding against potential reliability issues. To substantiate the effectiveness of the proposed charge pump circuit, two test chips were fabricated using a 0.35um 3.3-V CMOS process. Experimental results demonstrate that the four-stage charge pump circuit, with each pumping capacitor at 2 pF, can deliver an output voltage of approximately 8.8 V under a 3.3-V power supply condition. This output is primarily constrained by the junction breakdown voltage of the parasitic pn-junction in the given process. The proposed circuit is particularly advantageous for low-voltage CMOS applications due to its enhanced pumping efficiency and the absence of overstress on the gate oxide of the devices, thereby ensuring long-term reliability and performance.

In the transportation field, the empty weight of the vehicle has to be as light as possible in order to have the highest energetic efficiency. For onboard storage electrical systems, DC/DC converters are usually used to control the power flows in the vehicle. This paper focuses on two different DC/DC converter applications: high current/low voltage on the one hand and low current/high voltage on the other hand. For all power mobile applications, a standard dc/dc converter is not realizable due to mass and volume constraints. As the inductor is usually the biggest and heaviest component in the system, it is important to develop a new structure allowing the reduction of its size and weight. The chosen technology is based on interleaved channels operating in discontinuous mode, also using reverse conduction of power MOSFET transistors. For both of the cases described above, a prototype has been built composed of 8 and 4 channels respectively. The first converter is used for power assistance (250 A/18 V) using the MOS reverse conduction property. In this particular case, efficiency is up to 96.3 %. The second is developed for a mobile solar application (3.8 A/270 V). Efficiencies are up to 98.4 %

This study addresses a grid-connected photovoltaic (PV) generation system. In order to make the PV generation system more flexible and expandable, the backstage power circuit is composed of a high step-up converter and a pulsewidth-modulation (PWM) inverter. In the dc-dc power conversion, the high step-up converter is introduced to improve the conversion efficiency of conventional boost converters and to allow the parallel operation of low-voltage PV modules. Moreover, an adaptive total sliding-mode control system is designed for the current control of the PWM inverter to maintain the output current with a higher power factor and less variation under load changes. In addition, an adaptive step-perturbation method is proposed to achieve the objective of maximum power point tracking, and an active sun tracking scheme without any light sensors is investigated to make PV plates face the sun directly in order to capture maximum irradiation and enhance system efficiency. Experimental results are given to verify the validity of the high step-up converter, the PWM inverter control, the ASP method, and the active sun tracker for a grid-connected PV generation system.

This paper proposes a single-phase, single-stage buck–boost inverter for photovoltaic (PV) systems. The presented topology has one common terminal in input and output ports which eliminates common mode leakage current problem in the grid-connected PV applications. Although it uses four switches, its operation is bimodal and only two switches receive high-frequency pulse width modulation signals in each mode. Its principle of operation is described in detail with the help of equivalent circuits. Its dynamic model is presented, based on which a bimodal controller is designed. Experimental results, in stand-alone and grid-connected mode, obtained with a 300-W laboratory prototype are presented to validate its performance.

The following closed-loop transfer functions of the boost converter operating in continuous-conduction mode (CCM) supplying a complex impedance load are derived and analyzed: input-to-output voltage Mvcl and reference-to-output Tcl. The load of the boost dc-dc converter is composed of a series-connected resistance and inductance. The dynamic characteristics of the closed-loop boost converter with a third-order double-lead integral compensator are evaluated for different load inductances. The theoretically predicted results are validated through switching-circuit simulations using a suitable converter design example.

In telecom applications, the vision for a total power conversion efficiency from the mains to the output of PoL converters of 95% demands for an optimization of every conversion step, i.e. the PFC rectifier front-end should show an outstanding efficiency in the range of 99%. For recently discussed 400 V DC distribution bus voltages a buck-type PFC rectifier is a logical solution. In this paper, an efficiency-optimized, nearly 99% efficient, 5 kW three-phase buck-type PFC rectifier with 400 V output is presented. Methods for calculating losses of all components are described, and are used to optimize the converter design for efficiency at full load. Special attention is paid to semiconductor losses, which are shown to be dominant, with the parasitic device capacitance losses being a significant component. A prototype of the proposed rectifier is constructed which verifies the accuracy of the models used for loss calculation and optimization.

In this paper, the 500MHz band low power rectenna with high sensitivity characteristic is described. The rectenna is consisting of the folded dipole antenna with a 1.6kΩ impedance and the Cockcroft-Walton type rectifier for improvement of the rectification efficiency and the output dc voltage. Furthermore an L-Type LPF is added to reduce the negative influence due to the junction capacitance of the SBD. The developed rectennas (stage number: 1, 2) achieve the rectification efficiency of 48.9 % (stage number: 1) and 43.1 % (stage number: 2) at -15dBm of the input power. In addition to above, the top level efficiency of 8.7 % (stage number:1) at the input power of -30dBm can be achieved. In the Tokyo are, the output dc voltage of 0.22 V can be measured at the 25 km point from the antenna tower.

Enabling technologies for wireless sensor networks have gained considerable attention in research communities over the past few years. It is highly desirable, even necessary in certain situations, for wireless sensor nodes to be self-powered. With this goal in mind, a vibration based piezoelectric generator has been developed as an enabling technology for wireless sensor networks. The focus of this paper is to discuss the modeling, design, and optimization of a piezoelectric generator based on a two-layer bending element. An analytical model of the generator has been developed and validated. In addition to providing intuitive design insight, the model has been used as the basis for design optimization. Designs of 1 cm3 in size generated using the model have demonstrated a power output of 375 μW from a vibration source of 2.5 m s−2 at 120 Hz. Furthermore, a 1 cm3 generator has been used to power a custom designed 1.9 GHz radio transmitter from the same vibration source. (Some figures in this article are in colour only in the electronic version)

The self-oscillating flyback converter is a popular circuit for cost-sensitive applications due to its simplicity and low component count. This paper presents self-oscillating fly-back converter for mobile batteries charging applications. The converter is designed to step-down the high voltage input to an acceptable level from 220 VAC to 7VDC. By adding indictor and capacitor the power factor and input currant increased, and became 500mA, where the old input currant was 250mA, Diode was connected between Emitter and collector, so the output voltage was improved, and has less ripple, in addition to the soft switching was obtained, simulation model is developed by using MULTISIM software. The operation principle of this converter is described using some operating waveforms based on simulation results and a detailed circuit operation analysis. Analytical results are verified by laboratory experiment. With an input of 220 VAC, 500mA and an output of 7V DC, the efficiency of circuit was75%.

In this paper, a new current source gate drive circuit is proposed for high efficiency synchronous buck VRMs. The proposed circuit achieves quick turn-on and turn-off transition times to reduce switching loss and conduction loss in power MOSFETS. The driver circuit consists of two sets of four control switches and two very small inductors (typically 50 nH-300 nH each at 1 MHz). It drives both the control MOSFET and synchronous MOSFET in synchronous buck VRMs. An analysis, design procedure, optimization procedure and experimental results are presented for the proposed circuit. Experimental results demonstrate an efficiency of 86.6% at 15 A load and 81.9% at 30 A load for 12 V input and 1.3 V output at 1 MHz.

A systematic approach is proposed for the derivation of nonisolated three-port converter (TPC) topologies based on dual-input converters (DIC) and dual-output converters (DOC), which serves as an interface for a renewable source, a storage battery, and a load simultaneously. The power flow in a TPC is analyzed and compared with that in a DIC or a DOC, respectively. Beginning with building the power flow paths of a TPC from a DIC or a DOC, the general principles and detailed procedures for the generation and optimization of TPC topologies are presented. Based on these works, a family of nonisolated TPC topologies is developed. The derived TPCs feature single-stage power conversion between any two of the three ports, and result in high integration and high efficiency. One of the proposed TPCs, named Boost-TPC, is taken as an example for verifying the performance of its circuit topology with related operational methods. Pulsewidth modulation and power management methods, used in this circuit, are analyzed in detail. Experiments have been carried out on a 1-kW prototype of the Boost-TPC, which demonstrate the feasibility and effectiveness of the proposed topology derivation method.