Question: The emergence of emotion-sensitive conversational agents: challenges in digital feelings.

**Similar Documents**: Ethical Considerations in Artificial Intelligence

In the rapidly evolving field of artificial intelligence (AI), the integration of ethical principles has become a critical concern. As AI systems increasingly influence various aspects of human life, from decision-making to emotional interactions, it is imperative to address the ethical implications of these technologies. This document invites readers to contribute to the ongoing dialogue on AI ethics by submitting typo and error reports, as well as any other feedback, to hibbard@wisc.edu. The aim is to foster a community-driven discourse that ensures AI development aligns with ethical standards and societal values, particularly in the context of emotion-aware conversational agents and the potential threats they pose to digital emotions. By engaging with these issues, we can work towards creating AI technologies that not only advance human capabilities but also respect and enhance the emotional well-being of individuals.

This article introduces a sentiment analysis approach that adopts the way humans read, interpret, and extract sentiment from text. Our motivation builds on the assumption that human interpretation should lead to the most accurate assessment of sentiment in text. We call this automated process Human Reading for Sentiment (HRS). Previous research in sentiment analysis has produced many frameworks that can fit one or more of the HRS aspects; however, none of these methods has addressed them all in one approach. HRS provides a meta-framework for developing new sentiment analysis methods or improving existing ones. The proposed framework provides a theoretical lens for zooming in and evaluating aspects of any sentiment analysis method to identify gaps for improvements towards matching the human reading process. Key steps in HRS include the automation of humans low-level and high-level cognitive text processing. This methodology paves the way towards the integration of psychology with computational linguistics and machine learning to employ models of pragmatics and discourse analysis for sentiment analysis. HRS is tested with two state-of-the-art methods; one is based on feature engineering, and the other is based on deep learning. HRS highlighted the gaps in both methods and showed improvements for both.

This paper reports on an ongoing effort to enable the rapid development of multi-party virtual human negotiation scenarios. We present a case study in which a new scenario supporting negotiation between two human role players and two virtual humans was developed over a period of 12 weeks. We discuss the methodology and development process that were employed, from storyline design through role play and iterative development of the virtual humans’ semantic and task representations and natural language processing capabilities. We analyze the effort, expertise, and time required for each development step, and discuss opportunities to further streamline the development process.

Dialogue systems are often regarded as being tedious and inflexible. We believe that one reason is rigid and inadaptable system utterances. A good dialogue system should automatically choose a formulation that reflects the user’s expectations. However, current dialogue system development environments only allow the definition of questions with unchangeable formulations. In this paper we present a new approach to the generation of system questions by only defining basic concepts. This is the basis for realising adaptive, user-tailored, and human-like system questions in dialogue systems.

Detection of human emotions is an essential part of affect-aware human-computer interaction (HCI). In daily conversations, the preferred way of describing affects is by using categorical emotion labels (e.g., sad, anger, surprise). In categorical emotion classification, multiple descriptors (with different degrees of relevance) can be assigned to a sample. Perceptual evaluations have relied on primary and secondary emotions to capture the ambiguous nature of spontaneous recordings. Primary emotion is the most relevant category felt by the evaluator. Secondary emotions capture other emotional cues also conveyed in the stimulus. In most cases, the labels collected from the secondary emotions are discarded, since assigning a single class label to a sample is preferred from an application perspective. In this work, we take advantage of both types of annotations to improve the performance of emotion classification. We collect the labels from all the annotations available for a sample and generate primary and secondary emotion labels. A classifier is then trained using multitask learning with both primary and secondary emotions. We experimentally show that considering secondary emotion labels during the learning process leads to relative improvements of 7.9% in F1-score for an 8-class emotion classification task.

With people increasingly using emoticons in written text on the Web in order to express, stress, or disambiguate their sentiment, it is crucial for automated sentiment analysis tools to correctly account for such graphical cues for sentiment. We analyze how emoticons typically convey sentiment and we subsequently propose and evaluate a novel method for exploiting this with a manually created emoticon sentiment lexicon in a lexicon-based polarity classification method. We evaluate our approach on 2,080 Dutch tweets and forum messages, which all contain emoticons. We validate our findings on 10,069 English reviews of apps, some of which contain emoticons. We find that accounting for the sentiment conveyed by emoticons on a paragraph level – and, to a lesser extent, on a sentence level – significantly improves polarity classification performance. Whenever emoticons are used, their associated sentiment tends to dominate the sentiment conveyed by textual cues and forms a good proxy for the polarity of text.

Facial expressions are an important way through which humans interact socially. Building a system capable of automatically recognizing facial expressions from images and video has been an intense field of study in recent years. Interpreting such expressions remains challenging and much research is needed about the way they relate to human affect. This paper presents a general overview of automatic RGB, 3D, thermal and multimodal facial expression analysis. We define a new taxonomy for the field, encompassing all steps from face detection to facial expression recognition, and describe and classify the state of the art methods accordingly. We also present the important datasets and the bench-marking of most influential methods. We conclude with a general discussion about trends, important questions and future lines of research.

Approach • Social information independent of physical analog s – Text (e.g., emote) – abstract graphical representations (e.g., chat cir cles) • Interested in Abstract Approach a. Creates and deploys working systems b. Lack of attention Babble Prototype

In the modern Era of technology, Chatbots is the next big thing in the era of conversational services. Chatbots is a virtual person who can effectively talk to any human being using interactive textual skills. Currently, there are many cloud base Chatbots services which are available for the development and improvement of the chatbot sector such as IBM Watson, Microsoft bot, AWS Lambda, Heroku and many others. A virtual person is based on machine learning and Artificial Intelligence (AI) concepts and due to dynamic nature, there is a drawback in the design and development of these chatbots as they have built-in AI, NLP, programming and conversion services. This paper gives an overview of cloud-based chatbots technologies along with programming of chatbots and challenges of programming in current and future Era of chatbot.

Building a text corpus suitable to be used in corpus-based speech synthesis is a time-consuming process that usually requires some human intervention to select the desired phonetic content and the necessary variety of prosodic contexts. If an emotional text-to-speech (TTS) system is desired, the complexity of the corpus generation process increases. This paper presents a study aiming to validate or reject the use of a semantically neutral text corpus for the recording of both neutral and emotional (acted) speech. The use of this kind of texts would eliminate the need to include semantically emotional texts into the corpus. The study has been performed for Basque language. It has been made by performing subjective and objective comparisons between the prosodic characteristics of recorded emotional speech using both semantically neutral and emotional texts. At the same time, the performed experiments allow for an evaluation of the capability of prosody to carry emotional information in Basque language. Prosody manipulation is the most common processing tool used in concatenative TTS. Experiments of automatic recognition of the emotions considered in this paper (the "Big Six emotions") show that prosody is an important emotional indicator, but cannot be the only manipulated parameter in an emotional TTS system-at least not for all the emotions. Resynthesis experiments transferring prosody from emotional to neutral speech have also been performed. They corroborate the results and support the use of a neutral-semantic-content text in databases for emotional speech synthesis

1. ABSTRACT Animated agents based either on real video, cartoon-style drawings or even model-based 3D graphics offer great promise for computer-based presentations as they make presentations more lively and appealing and allow for the emulation of conversation styles known from human-human communication. In this paper, we describe a life-like interface agent which presents multimedia material to the user following the directives of a script. The overall behavior of the presentation agent is partly determined by such a script, and partly by the agent's self-behavior. In our approach, the agent's behavior is defined in a declarative specification language. Behavior specifications are used to automatically generate a control module for an agent display system. The first part of the paper describes the generation process which involves AI planning and a two-step compilation. Since the manual creation of presentation scripts is tedious and error-prone, we also address the automated generation of presentation scripts which may be forwarded to the interface agent. The second part of the paper presents an approach for multimedia presentation design which combines hierarchical planning with temporal reasoning.


Question: The emergence of emotionally aware conversational agents: Challenges in digital sentiments.

This paper introduces an innovative neural conversational model designed to advance the field of natural language understanding and machine intelligence. Unlike conventional methods that are often confined to specialized domains and necessitate intricate, handcrafted rules, our approach utilizes the cutting-edge sequence-to-sequence framework for a more versatile solution. The model's core functionality involves predicting the subsequent sentence in a dialogue, based on the preceding sentence or sentences. A key advantage is its end-to-end training capability, significantly reducing the reliance on manual rule development. Through extensive training on a large conversational dataset, the model demonstrates an ability to engage in basic conversations. Despite potential misalignments in the objective function, preliminary findings indicate that the model performs impressively. It can assimilate knowledge from domain-specific datasets, as well as from vast, general-domain datasets, such as movie subtitles, which may contain noise. In a domain-specific IT helpdesk scenario, the model is capable of resolving technical issues through dialogue. In more open-domain contexts, such as movie transcripts, it can exhibit rudimentary common sense reasoning. As anticipated, however, the model occasionally struggles with maintaining consistency throughout conversations.

A huge number of videos are posted every day on social media platforms such as Facebook and YouTube. This makes the Internet an unlimited source of information. In the coming decades, coping with such information and mining useful knowledge from it will be an increasingly difficult task. In this paper, we propose a novel methodology for multimodal sentiment analysis, which consists in harvesting sentiments from Web videos by demonstrating a model that uses audio, visual and textual modalities as sources of information. We used both featureand decision-level fusion methods to merge affective information extracted from multiple modalities. A thorough comparison with existing works in this area is carried out throughout the paper, which demonstrates the novelty of our approach. Preliminary comparative experiments with the YouTube dataset show that the proposed multimodal system achieves an accuracy of nearly 80%, outperforming all state-of-the-art systems by more than 20%. & 2015 Elsevier B.V. All rights reserved.

We propose detecting and responding to humor in spoken dialogue by extracting language and audio cues and subsequently feeding these features into a combined recurrent neural network (RNN) and logistic regression model. In this paper, we parse Switchboard phone conversations to build a corpus of punchlines and unfunny lines where punchlines precede laughter tokens in Switchboard transcripts. We create a combined RNN and logistic regression model that uses both acoustic and language cues to predict whether a conversational agent should respond to an utterance with laughter. Our model achieves an F1-score of 63.2 and accuracy of 73.9. This model outperforms our logistic language model (F1-score 56.6) and RNN acoustic model (59.4) as well as the final RNN model of D. Bertero, 2016 (52.9). Using our final model, we create a “laughbot” that audibly responds to a user with laughter when their utterance is classified as a punchline. A conversational agent outfitted with a humorrecognition system such as the one we present in this paper would be valuable as these agents gain utility in everyday life. Keywords—Chatbots; spoken natural language processing; deep learning; machine learning

The discipline where sentiment/ opinion/ emotion has been identified and classified in human written text is well known as sentiment analysis. A typical computational approach to sentiment analysis starts with prior polarity lexicons where entries are tagged with their prior out of context polarity as human beings perceive using their cognitive knowledge. Till date, all research efforts found in sentiment lexicon literature deal mostly with English texts. In this article, we propose multiple computational techniques like, WordNet based, dictionary based, corpus based or generative approaches for generating SentiWordNet(s) for Indian languages. Currently, SentiWordNet(s) are being developed for three Indian languages: Bengali, Hindi and Telugu. An online intuitive game has been developed to create and validate the developed SentiWordNet(s) by involving Internet population. A number of automatic, semi-automatic and manual validations and evaluation methodologies have been adopted to measure the coverage and credibility of the developed SentiWordNet(s).

Literacy learning learning how to read and write, begins long before children enter school. One of the key skills to reading and writing is the ability to represent thoughts symbolically and share them in language with an audience who may not necessarily share the same temporal and spatial context for the story. Children learn and practice these important language skills everyday, telling stories with the peers and adults around them. In particular, storytelling in the context of peer collaboration provides a key environment for children to learn language skills important for literacy. In light of this, we designed Sam, an embodied conversational agent who tells stories collaboratively with children. Sam was designed to look like a peer for preschool children, but to tell stories in a developmentally advanced way: modeling narrative skills important for literacy. Results demonstrated that children who played with the virtual peer told stories that more closely resembled the virtual peer’s linguistically advanced stories: using more quoted speech and temporal and spatial expressions. In addition, children listened to Sam's stories carefully, assisting her and suggesting improvements. The potential benefits of having technology play a social role in young children’s literacy learning is discussed.

CHAPTER 18 Lexicons for Sentiment and Affect Extraction " [W]e write, not with the fingers, but with the whole person. The nerve which controls the pen winds itself about every fibre of our being, threads the heart, pierces the liver. " Virginia Woolf, Orlando " She runs the gamut of emotions from A to B. " Dorothy Parker, reviewing Hepburn's performance in Little Women In this chapter we turn to tools for interpreting affective meaning, extending our affective study of sentiment analysis in Chapter 7. We use the word 'affective', following the tradition in affective computing (Picard, 1995) to mean emotion, sentiment, personality , mood, and attitudes. Affective meaning is closely related to subjectivity, subjectivity the study of a speaker or writer's evaluations, opinions, emotions, and speculations (Wiebe et al., 1999). How should affective meaning be defined? One influential typology of affec-tive states comes from Scherer (2000), who defines each class of affective states by factors like its cognition realization and time course: Emotion: Relatively brief episode of response to the evaluation of an external or internal event as being of major significance. Mood: Diffuse affect state, most pronounced as change in subjective feeling, of low intensity but relatively long duration, often without apparent cause. Interpersonal stance: Affective stance taken toward another person in a specific interaction, colouring the interpersonal exchange in that situation. Figure 18.1 The Scherer typology of affective states, with descriptions from Scherer (2000).

The work described in this paper represents the study and the attempt to make a contribution to one of the most stimulating and promising sectors in the field of emotion recognition, which is health care management. Multidisciplinary studies in artificial intelligence, augmented reality and psychology stressed out the importance of emotions in communication and awareness. The intent is to recognize human emotions, processing images streamed in real-time from a mobile device. The adopted techniques involve the use of open source libraries of visual recognition and machine learning approaches based on convolutional neural networks (CNN).

With people increasingly using emoticons in written text on the Web in order to express, stress, or disambiguate their sentiment, it is crucial for automated sentiment analysis tools to correctly account for such graphical cues for sentiment. We analyze how emoticons typically convey sentiment and we subsequently propose and evaluate a novel method for exploiting this with a manually created emoticon sentiment lexicon in a lexicon-based polarity classification method. We evaluate our approach on 2,080 Dutch tweets and forum messages, which all contain emoticons. We validate our findings on 10,069 English reviews of apps, some of which contain emoticons. We find that accounting for the sentiment conveyed by emoticons on a paragraph level – and, to a lesser extent, on a sentence level – significantly improves polarity classification performance. Whenever emoticons are used, their associated sentiment tends to dominate the sentiment conveyed by textual cues and forms a good proxy for the polarity of text.

We present a shared task on automatically determining sentiment intensity of a word or a phrase. The words and phrases are taken from three domains: general English, English Twitter, and Arabic Twitter. The phrases include those composed of negators, modals, and degree adverbs as well as phrases formed by words with opposing polarities. For each of the three domains, we assembled the datasets that include multi-word phrases and their constituent words, both manually annotated for real-valued sentiment intensity scores. The three datasets were presented as the test sets for three separate tasks (each focusing on a specific domain). Five teams submitted nine system outputs for the three tasks. All datasets created for this shared task are freely available to the research community.

Human often wants to listen to music that fits best his current emotion. A grasp of emotions in songs might be a great help for us to effectively discover music. In this paper, we aimed at automatically classifying moods of songs based on lyrics and metadata, and proposed several methods for supervised learning of classifiers. In future, we plan to use automatically identified moods of songs as metadata in our music search engine. Mood categories in a famous contest about Audio Music Mood Classification (MIREX 2007) are applied for our system. The training data is collected from a LiveJournal blog site in which each blog entry is tagged with a mood and a song. Then three kinds of machine learning algorithms are applied for training classifiers: SVM, Naive Bayes and Graph-based methods. The experiments showed that artist, sentiment words, putting more weight for words in chorus and title parts are effective for mood classification. Graph-based method promises a good improvement if we have rich relationship information among songs.

Across the globe, people are voicing their opinion in social media and various other online fora. Given such data, modern deep learning-based sentiment analysis methods excel at determining the sentiment polarity of what is being said about companies, products, etc. Unfortunately, such deep methods require significant training data, while for many languages, resources and training data are scarce. In this work, we present a cross-lingual propagation algorithm that yields sentiment embedding vectors for numerous languages. We then rely on a dual-channel convolutional neural architecture to incorporate them into the network. This allows us to achieve gains in deep sentiment analysis across a range of languages and domains.


Question: Utilizing contour lines for the identification and localization of junctions in natural scenes

This paper presents a comprehensive approach for automatic scale selection in feature detection, addressing the critical issue of scale variation in object representation within images. Building upon the foundational work of Witkin (1983) and Koenderink (1984), which introduced the concept of scale-space representation to encapsulate image structures across multiple scales, this study advances the field by proposing a methodology to identify the most relevant scales for further analysis. The traditional scale-space theory, however, lacked a systematic approach for selecting appropriate scales for local features. This article fills this gap by outlining a framework that generates hypotheses regarding significant scale levels in image data. The core principle involves detecting local extrema over scales of γ-normalized derivatives, which are indicative of structures of interest. This principle is leveraged as a pivotal mechanism in algorithms designed for automatic scale selection, dynamically adjusting the scales of processing to match the local image structure. The validity of this approach is substantiated through a thorough theoretical analysis of its behavior under varying input pattern scales and by integrating it with various early visual processing modules. This integration is demonstrated through experiments conducted on both real-world and synthetic datasets. Additionally, the paper offers a detailed evaluation of how different feature detectors, when coupled with the proposed scale selection mechanism, perform when applied to model patterns. This includes an in-depth exploration of its application to blob detection, junction detection, edge detection, ridge detection, and local frequency estimation. Given the frequent limitations posed by low-level vision modules in computer vision systems, the paper argues that the incorporation of automatic scale selection mechanisms is indispensable for the development of autonomous vision systems capable of analyzing complex, unfamiliar environments.

Text detection and localization in natural scene images is important for content-based image analysis. This problem is challenging due to the complex background, the non-uniform illumination, the variations of text font, size and line orientation. In this paper, we present a hybrid approach to robustly detect and localize texts in natural scene images. A text region detector is designed to estimate the text existing confidence and scale information in image pyramid, which help segment candidate text components by local binarization. To efficiently filter out the non-text components, a conditional random field (CRF) model considering unary component properties and binary contextual component relationships with supervised parameter learning is proposed. Finally, text components are grouped into text lines/words with a learning-based energy minimization method. Since all the three stages are learning-based, there are very few parameters requiring manual tuning. Experimental results evaluated on the ICDAR 2005 competition dataset show that our approach yields higher precision and recall performance compared with state-of-the-art methods. We also evaluated our approach on a multilingual image dataset with promising results.

This paper presents an approach to object discovery in a give n unlabeled image set, based on mining repetitive spatial config urations of image contours. Contours that similarly deform from one image to a n ther are viewed as collaborating, or, otherwise, conflicting. This is captu red by a graph over all pairs of matching contours, whose maximum a posteriori mult icoloring assignment is taken to represent the shapes of discovered objects. Multicoloring is conducted by our new Coordinate Ascent Swendsen-Wang cut (CASW ). CASW uses the Metropolis-Hastings (MH) reversible jumps to probabil istically sample graph edges, and color nodes. CASW extends SW cut by introducing a r e ularization in the posterior of multicoloring assignments that prevent s the MH jumps to arrive at trivial solutions. Also, CASW seeks to learn paramet ers of the posterior via maximizing a lower bound of the MH acceptance rate. This s peeds up multicoloring iterations, and facilitates MH jumps from local mi nima. On benchmark datasets, we outperform all existing approaches to unsuper vised object discovery.

This paper describes a hierarchical estimation framework for the computation of diverse representations of motion information. The key features of the resulting framework (or family of algorithms) a,re a global model that constrains the overall structure of the motion estimated, a local rnodel that is used in the estimation process, and a coa,rse-fine refinement strategy. Four specific motion models: affine flow, planar surface flow, rigid body motion, and general optical flow, are described along with their application to specific examPles.

This letter presents a two-step method for urban main road extraction from high-resolution remotely sensed imagery by integrating spectral-spatial classification and shape features. In the first step, spectral-spatial classification segments the imagery into two classes, i.e., the road class and the nonroad class, using path openings and closings. The local homogeneity of the gray values obtained by local Geary's C is then fused with the road class. In the second step, the road class is refined by using shape features. The experimental results indicated that the proposed method was able to achieve a comparatively good performance in urban main road extraction.

In this paper, we propose a new approach that we call the ldquofluid vector flowrdquo (FVF) active contour model to address problems of insufficient capture range and poor convergence for concavities. With the ability to capture a large range and extract concave shapes, FVF demonstrates improvements over techniques like gradient vector flow, boundary vector flow, and magnetostatic active contour on three sets of experiments: synthetic images, pediatric head MRI images, and brain tumor MRI images from the Internet brain segmentation repository.

We sketch a workflow for cognitively plausible recognition of vague geographical concepts, such as a city centre. Our approach imitates a pedestrian strolling through the streets, and comparing his/her internal cognitive model of a city centre with the stimulus from the external world to decide whether he/she is in the city centre or outside. The cognitive model of a British city centre is elicited through an online questionnaire survey and used to delineate referents of city centre from point of interest data. We first compute a measure of ‘city centre-ness’ at each location within a city, and then merge the area of high city centre-ness to a contiguous region. The process is illustrated on the example of the City of Bristol, and the computed city centre area for Bristol is evaluated by comparison to reference areas derived from alternative sources. The evaluation suggests that our approach performs well and produces a representation of a city centre that is near to people’s conceptualisation. The benefits of our work are better (and user-driven) descriptions of complex geographical concepts. We see such models as a prerequisite for generalisation over large changes in detail, and for very specific purposes.

This paper considers the objectives of accurate stereo matching, especially at object boundaries, robustness against recording or illumination changes and efficiency of the calculation. These objectives lead to the proposed semi-global matching method that performs pixelwise matching based on mutual information and the approximation of a global smoothness constraint. Occlusions are detected and disparities determined with sub-pixel accuracy. Additionally, an extension for multi-baseline stereo images is presented. There are two novel contributions. Firstly, a hierarchical calculation of mutual information based matching is shown, which is almost as fast as intensity based matching. Secondly, an approximation of a global cost calculation is proposed that can be performed in a time that is linear to the number of pixels and disparities. The implementation requires just 1 second on typical images.

In this paper we present a comprehensive and critical survey of face detection algorithms. Face detection is a necessary first-step in face recognition systems, with the purpose of localizing and extracting the face region from the background. It also has several applications in areas such as content-based image retrieval, video coding, video conferencing, crowd surveillance, and intelligent human–computer interfaces. However, it was not until recently that the face detection problem received considerable attention among researchers. The human face is a dynamic object and has a high degree of variability in its apperance, which makes face detection a difficult problem in computer vision. A wide variety of techniques have been proposed, ranging from simple edge-based algorithms to composite high-level approaches utilizing advanced pattern recognition methods. The algorithms presented in this paper are classified as either feature-based or image-based and are discussed in terms of their technical approach and performance. Due to the lack of standardized tests, we do not provide a comprehensive comparative evaluation, but in cases where results are reported on common datasets, comparisons are presented. We also give a presentation of some proposed applications and possible application areas. c © 2001 Academic Press

In this paper, a fast and flexible algorithm for computing watersheds in digital grayscale images is introduced. A review of watersheds and related notion is first presented, and the major methods to determine watersheds are discussed. The present algorithm is based on an immersion process analogy, in which the flooding of the water in the picture is efficiently simulated using a queue of pixels. It is described in detail and provided in a pseudo C language. We prove the accuracy of this algorithm is superior to that of the existing implementations. Furthermore, it is shown that its adaptation to any kind of digital grid and its generalization to n-dimensional images and even to graphs are straightforward. In addition, its strongest point is that it is faster than any other watershed algorithm. Applications of this algorithm with regard to picture segmentation are presented for MR imagery and for digital elevation models. An example of 3-D watershed is also provided. Lastly, some ideas are given on how to solve complex segmentation tasks using watersheds on graphs. Zndex Terms-Algorithm, digital image, FIFO structure, graph, grid, mathematical morphology, picture segmentation, watersheds.

Moving vehicles are detected and tracked automatically in monocular image sequences from road traffic scenes recorded by a stationary camera. In order to exploit the a priori knowledge about shape and motion of vehicles in traffic scenes, a parameterized vehicle model is used for an intraframe matching process and a recursive estimator based on a motion model is used for motion estimation. An interpretation cycle supports the intraframe matching process with a state MAP-update step. Initial model hypotheses are generated using an image segmentation component which clusters coherently moving image features into candidate representations of images of a moving vehicle. The inclusion of an illumination model allows taking shadow edges of the vehicle into account during the matching process. Only such an elaborate combination of various techniques has enabled us to track vehicles under complex illumination conditions and over long (over 400 frames) monocular image sequences. Results on various real-world road traffic scenes are presented and open problems as well as future work are outlined.


Question: Employing contour techniques for identifying and localizing junctions in natural images

This paper delves into a comprehensive strategy for identifying and tracking features within a sequence of images captured by a moving camera, aimed at constructing a three-dimensional understanding of the real world. The focus is on natural scenes that are rich in complexity and diversity, featuring a broad spectrum of objects such as roads, buildings, trees, and bushes, as exemplified in Figure 1. The primary challenge lies in the vast variety of elements present in these scenes, which renders top-down recognition techniques ineffective. To tackle this issue, we are developing a computer vision system that hinges on the analysis of motion in monocular image sequences. This involves the extraction and continuous tracking of image features. By doing so, we can derive representations of the three-dimensional counterparts of these features, enabling a more nuanced interpretation of the physical environment. This approach is particularly significant for navigating through unstructured environments, where the visual landscape is constantly changing, and a robust, dynamic understanding of the surroundings is essential for autonomous systems.

In compositing applications, objects depicted in images frequently have to be separated from their background, so that they can be placed in a new environment. Alpha mattes are important tools aiding the selection of objects, but cannot normally be created in a fully automatic way. We present an algorithm that requires as input two images---one where the object is in focus, and one where the background is in focus---and then automatically produces an alpha matte indicating which pixels belong to the object. This algorithm is inspired by human visual processing and involves nonlinear response compression, center-surround mechanisms as well as a filling-in stage. The output can then be refined with standard computer vision techniques.

Recognizing and localizing queried objects in range images plays an important role for robotic manipulation and navigation. Even though it has been steadily studied, it is still a challenging task for scenes with occlusion and clutter.

In this paper, we propose a Scale and Rotation Invariant Implicit Shape Model (SRIISM), and develop a local feature matching based system using the model to accurately locate and identify large numbers of object instances in an image. Due to repeated instances and cluttered background, conventional methods for multiple object instance identification suffer from poor identification results. In the proposed SRIISM, we model the joint distribution of object centers, scale, and orientation computed from local feature matches in Hough voting, which is not only invariant to scale changes and rotation of objects, but also robust to false feature matches. In the multiple object instance identification system using SRIISM, we apply a fast 4D bin search method in Hough space with complexity O(n), where n is the number of feature matches, in order to segment and locate each instance. Futhermore, we apply maximum likelihood estimation (MLE) for accurate object pose detection. In the evaluation, we created datasets simulating various industrial applications such as pick-and-place and inventory management. Experiment results on the datasets show that our method outperforms conventional methods in both accuracy (5%-30% gain) and speed (2x speed up).

Many algorithms for both identifying and reconstructing a 3-D object are based on the 2-D silhouettes of the object. In general, identifying a nonconvex object using a silhouettebased approach implies neglecting some features of its surface as identification clues. The same features cannot be reconstructed by volume intersection techniques using multiple silhouettes of the object. This paper addresses the problem of finding which parts of a nonconvex object are relevant for silhouette-based image understanding. For this purpose, the geometric concept of visual hull of a 3-D object is introduced. The visual hull of a 3-D object S is the closest approximation of S that can be obtained with the volume intersection approach. An equivalent statement, relative to object identification, is that the visual hull of S is the maximal object silhouette-equivalent to S, i.e., which can be substituted for S without affecting any silhouette. Only the parts of the surface of S that also lie on the surface of the visual hull can be reconstructed or identified using silhouette-based algorithms. The visual hull of an object depends not only on the object itself but also on the region allowed to the viewpoint. Two main viewing regions can be considered, resulting in the external and internal visual hull. In the former case the viewing region is related to the convex hull of S, in the latter it is bounded by S itself. The internal visual hull also admits an interpretation not related to silhouettes: the features of the surface of S that is not coincident with the surface of the internal visual hull cannot be observed from any viewpoint lying outside the convex hull. After a general discussion of the visual hull and its properties, algorithms for computing the internal and external visual hulls of 2-D objects and 3-D planar face objects are presented and their complexity analyzed. In general, the visual hull of a 3-D planar face object turns out to be bounded by planar and curved patches. A precise statement of the concept of visual hull appears to be novel, as is the problem of its computation.

Convolutional neural networks (CNNs) are usually built by stacking convolutional operations layer-by-layer. Although CNN has shown strong capability to extract semantics from raw pixels, its capacity to capture spatial relationships of pixels across rows and columns of an image is not fully explored. These relationships are important to learn semantic objects with strong shape priors but weak appearance coherences, such as traffic lanes, which are often occluded or not even painted on the road surface as shown in Fig. 1 (a). In this paper, we propose Spatial CNN (SCNN), which generalizes traditional deep layer-by-layer convolutions to slice-byslice convolutions within feature maps, thus enabling message passings between pixels across rows and columns in a layer. Such SCNN is particular suitable for long continuous shape structure or large objects, with strong spatial relationship but less appearance clues, such as traffic lanes, poles, and wall. We apply SCNN on a newly released very challenging traffic lane detection dataset and Cityscapse dataset. The results show that SCNN could learn the spatial relationship for structure output and significantly improves the performance. We show that SCNN outperforms the recurrent neural network (RNN) based ReNet and MRF+CNN (MRFNet) in the lane detection dataset by 8.7% and 4.6% respectively. Moreover, our SCNN won the 1st place on the TuSimple Benchmark Lane Detection Challenge, with an accuracy of 96.53%.

Vessel enhancement and segmentation aim at (binary) per-pixel segmentation considering certain local features as probabilistic vessel indicators. We propose a new methodology to combine any local probability map with local directional vessel information. The resulting global vessel segmentation is represented as a set of discrete streamlines populating the vascular structures and providing additional connectivity and geometric shape information. The streamlines are computed by numerical integration of the directional vector field that is obtained from the eigenanalysis of the local Hessian indicating the local vessel direction. The streamline representation allows for sophisticated post-processing techniques using the additional information to refine the segmentation result with respect to the requirements of the particular application such as image registration. We propose different post-processing techniques for hierarchical segmentation, centerline extraction, and catheter removal to be used for X-ray angiograms. We further demonstrate how the global approach is able to significantly improve the segmentation compared to conventional local Hessian-based approaches.

An increasing safety and reducing road accidents, thereby saving lives are one of great interest in the context of Advanced Driver Assistance Systems. Apparently, among the complex and challenging tasks of future road vehicles is road lane detection or road boundaries detection. It is based on lane detection (which includes the localization of the road, the determination of the relative position between vehicle and road, and the analysis of the vehiclepsilas heading direction). One of the principal approaches to detect road boundaries and lanes using vision system on the vehicle. However, lane detection is a difficult problem because of the varying road conditions that one can encounter while driving. In this paper, a vision-based lane detection approach capable of reaching real time operation with robustness to lighting change and shadows is presented. The system acquires the front view using a camera mounted on the vehicle then applying few processes in order to detect the lanes. Using a pair of hyperbolas which are fitting to the edges of the lane, those lanes are extracted using Hough transform. The proposed lane detection system can be applied on both painted and unpainted road as well as curved and straight road in different weather conditions. This approach was tested and the experimental results show that the proposed scheme was robust and fast enough for real time requirements. Eventually, a critical overview of the methods were discussed, their potential for future deployment were assist.

This paper addresses automatic interpretation of images of outdoor scenes. The method allows instances of objects from a number of generic classes to be identi ed: vegetation, buildings, vehicles, roads, etc., thereby enabling image databases to be queried on scene content. The feature set is based, in part, on psychophysical principles and includes measures of colour, texture and shape. Using a large database of ground-truth labelled images, a neural network is trained as a pattern classi er. The method is demonstrated on a large test set to provide highly accurate image interpretations, with over 90% of the image area labelled correctly. Scene Understanding Object Recognition Classi cation Image Segmentation Neural Networks

This document is meant to serve as an addendum to [1], published at BMVC 2009. The purpose of this addendum is twofold: (1) to respond to feedback we’ve received since publication and (2) to describe a number of changes, especially to the non-maximal suppression, that further improve performance. The performance of our updated detection increases 5% to over 91% detection rate at 1 false positive per image on the INRIA dataset, and similarly on the Caltech Pedestrian Dataset, while overall system runtime for multiscale detection decreases by 1/3 to just under 1.5s per 640×480 image. We begin by rectifying an important omission to the related work. Levi and Weiss had an innovative application of integral images to multiple image channels quite early on, demonstrating good results on face detection from few training examples [4]. This work appears to be the earliest such use of integral images, indeed the authors even describe a precursor to integral histograms. Many thanks to Mark Everingham for sending us this reference.

In this work we show that Deep Convolutional Neural Networks can outperform humans on the task of boundary detection, as measured on the standard Berkeley Segmentation Dataset. Our detector is fully integrated in the popular Caffe framework and processes a 320x420 image in less than a second. Our contributions consist firstly in combining a careful design of the loss for boundary detection training, a multi-resolution architecture and training with external data to improve the detection accuracy of the current state of the art, from an optimal dataset scale F-measure of 0.780 to 0.808 while human performance is at 0.803. We further improve performance to 0.813 by combining deep learning with grouping, integrating the Normalized Cuts technique within a deep network. We also examine the potential of our boundary detector in conjunction with the higher level tasks of object proposal generation and semantic segmentation for both tasks our detector yields clear improvements over state-of-the-art systems.


Question: Employing contour lines for the identification and localization of junctions in natural images

This document presents a wavelet-based method for detecting and analyzing singularities in signals, with a focus on identifying irregular structures and transient phenomena that carry significant information. The concept of Lipschitz exponents is introduced as a means to mathematically characterize these singularities. Theorems are discussed that estimate the local Lipschitz exponents of functions through the examination of the wavelet transform's evolution across scales. It is shown that the local maxima of the wavelet transform modulus effectively pinpoint the positions of these irregular structures and provide a means to compute their Lipschitz exponents numerically. A distinct behavior is observed in the wavelet transform of singularities with rapid oscillations, and a method is presented to measure the local frequency of these oscillations from the maxima of the wavelet transform modulus. Numerical evidence supports the idea that one- and two-dimensional signals can be accurately approximated using the local maxima of their wavelet transform modulus. As a practical application, an algorithm is developed to eliminate white noise from signals by analyzing the evolution of the wavelet transform maxima across scales. In the context of two-dimensional signals, such as images, the wavelet transform maxima highlight the location of edges. This denoising algorithm is further adapted for image enhancement, demonstrating its effectiveness in improving image quality.

Automated blood vessel segmentation of retinal images offers huge potential benefits for medical diagnosis of different ocular diseases. In this paper, 2D Matched Filters (MF) are applied to fundus retinal images to detect vessels which are enhanced by Contrast Limited Adaptive Histogram Equalization (CLAHE) method. Due to the Gaussian nature of blood vessel profile, the MF with Gaussian kernel often misclassifies non-vascular structures (e.g., step, ramp or other transients) as vessels. To avoid such false detection, this paper introduces Laplacian of Gaussian (LoG) filters in the vessel segmentation process. The inherent zero-crossing property of LoG filter is used in the algorithm, along with the MF, in order to extract vessels reliably from the retinal images. The proposed method is validated against three publicly available databases, STARE, DRIVE and HRF. Simulation results show that the proposed method is able to segment vessels accurately from the three database images with an average accuracy that is competitive to the existing methodologies.

Corner points are formed from two or more edges and edges usually define the boundary between two different objects or parts of the same objects. In this novel we discuss the theory of the Harris corner detection and indicate its disadvantage. Then it proposes an improved algorithm of Harris detection algorithm based on the neighboring point eliminating method. It reduces the time of the detection, and makes the corners distributing more homogenous so that avoids too many corners stay together. Experimental results show that the algorithm can detect corner more equality distributing, and can be used in some fact applications such as image registration well. Key words-Corner point; Harris corner; neighboring point; homogenous distributing 1. Introduce Corner point is a maximum curvature point of the two-dimensional image brightness change or the edge of sharp curves in a image, which haves widely used in such as the camera calibration, virtual scene reconstruction, motion estimation, image registration and more computer vision tasks. These points keep important features of image and can also be effective in reducing the amount of data information, allow real-time processing possible. Now, there are two corner detection methods: based on the image edge method and based on the image gray scale method. The former always need to encode the image edge, which is heavily dependent on image segmentation and edge extraction, have large difficulty and calculate, and if the detected area haves some change, it will lead to the failure of the operation, therefore, this method is not used widely. The latter method detect corner by calculating the curvature and gradient, it avoid these disadvantage. In currently, it haves Moravec operator, Harris operator, Susan operator and so on. Experimental shows that Harris corner detection method haves the best result in above mentioned. But in fact applications, Harris algorithm must to give a proper threshold T, and the algorithm based on the provided threshold to detect the ideal corner. T always depend on the image, specifically it is difficult to determine under the different colors. And it gets the ideal T after user blind set T more times. In addition, the points which have the larger eigenvalues are concentrated in some regions cause the detect corners are non-uniform distribution; and if T dropped, even though the overall distribution of the corner towards reasonable, but it also leads to the points close together, have the clustering phenomenon, it will affect the later analysis and processing. 2. Harris Corner Detection Algorithm 2.1 Detection Theory Harris corner detection algorithm is based on the point feature extraction of signal. It makes the window ω (usually rectangular area) to move infinitesimal displacement to any direction, and the variation of gray can be defined as: − = + + 2 , , , , ] [ v u v y u x v u y x I I w E + + = v u v u y x O yY xX w , 2 2 2 , )] , ( [ 2 2 2 By Cxy Ax + + = T y x M y x ) , ( ) , ( = (1) Where X and Y are one-order gray-level gradient, they can be got by convolution with 978-1-4244-4131-0/09/$25.00 ©2009 IEEE

Driving an automobile autonomously on rural roads requires knowledge about the geometry of the road. Furthermore, knowledge about the meaning of each lane of the road is needed in order to decide which lane should be taken and if the vehicle can do a lane change. This paper addresses the problem of extracting additional information about lanes. The information is extracted from the types of road-markings. The type of lane border markings is estimated in order to find out if a lane change is allowed. Arrows, which are painted on the road, are extracted and classified in order to determine the meaning of a lane such as a turn off lane.

Rectangular shape detection has a wide range of applications, such as license plate detection, vehicle detection, and building detection. In this paper, we propose a geometric framework for rectangular shape detection based on the channel-scale space of RGB images. The framework consists of algorithms developed to address three issues of a candidate shape (i.e., a connected component of edge points), including: 1) outliers; 2) open shape; and 3) fragmentation. Furthermore, we propose an interestness measure for rectangular shapes by integrating imbalanced points (one type of interest points). Our experimental study shows the promise of the proposed framework.

Updating road markings is one of the routine tasks of transportation agencies. Compared with traditional road inventory mapping techniques, vehicle-borne mobile light detection and ranging (LiDAR) systems can undertake the job safely and efficiently. However, current hurdles include software and computing challenges when handling huge volumes of highly dense and irregularly distributed 3-D mobile LiDAR point clouds. This paper presents the development and implementation aspects of an automated object extraction strategy for rapid and accurate road marking inventory. The proposed road marking extraction method is based on 2-D georeferenced feature (GRF) images, which are interpolated from 3-D road surface points through a modified inverse distance weighted (IDW) interpolation. Weighted neighboring difference histogram (WNDH)-based dynamic thresholding and multiscale tensor voting (MSTV) are proposed to segment and extract road markings from the noisy corrupted GRF images. The results obtained using 3-D point clouds acquired by a RIEGL VMX-450 mobile LiDAR system in a subtropical urban environment are encouraging.

Ansar, the method by Ansar and Daniilidis (2003), implementation from Xu et al. (2016). Mirzaei, the method by Mirzaei and Roumeliotis (2011). RPnL, the method by Zhang et al. (2013). ASPnL, the method by Xu et al. (2016). LPnL Bar LS, the method by Xu et al. (2016). LPnL Bar ENull, the method by Xu et al. (2016). DLT-Lines, the method by Hartley and Zisserman (2004, p. 180), our implementation. DLT-Plücker-Lines, the method by Přibyl et al. (2015), our implementation. DLT-Combined-Lines, the proposed method.

Image saliency detection is very useful in many computer vision tasks while it still remains a challenging problem. In this paper, we propose a new computational saliency detection model which is implemented with a coarse to fine strategy under the Bayesian framework. First, saliency points are applied to get a coarse location of the saliency region. And then, based on the rough region, we compute a prior map for the Bayesian model to achieve the final saliency map. Experimental results on a public available dataset show the effectiveness of the proposed prior map and the strength of our saliency map compared with several previous method.

To facilitate scene understanding and robot navigation in a modern urban area, we design a multilayer feature graph (MFG) based on two views from an on-board camera. The nodes of an MFG are features such as scale invariant feature transformation (SIFT) feature points, line segments, lines, and planes while edges of the MFG represent different geometric relationships such as adjacency, parallelism, collinearity, and coplanarity. MFG also connects the features in two views and the corresponding 3D coordinate system. Building on SIFT feature points and line segments, MFG is constructed using feature fusion which incrementally, iteratively, and extensively verifies the aforementioned geometric relationships using random sample consensus (RANSAC) framework. Physical experiments show that MFG can be successfully constructed in urban area and the construction method is demonstrated to be very robust in identifying feature correspondence.

This paper presents an approach to object discovery in a give n unlabeled image set, based on mining repetitive spatial config urations of image contours. Contours that similarly deform from one image to a n ther are viewed as collaborating, or, otherwise, conflicting. This is captu red by a graph over all pairs of matching contours, whose maximum a posteriori mult icoloring assignment is taken to represent the shapes of discovered objects. Multicoloring is conducted by our new Coordinate Ascent Swendsen-Wang cut (CASW ). CASW uses the Metropolis-Hastings (MH) reversible jumps to probabil istically sample graph edges, and color nodes. CASW extends SW cut by introducing a r e ularization in the posterior of multicoloring assignments that prevent s the MH jumps to arrive at trivial solutions. Also, CASW seeks to learn paramet ers of the posterior via maximizing a lower bound of the MH acceptance rate. This s peeds up multicoloring iterations, and facilitates MH jumps from local mi nima. On benchmark datasets, we outperform all existing approaches to unsuper vised object discovery.

This paper proposes an algorithm for solving subsets of typical (canonical) jigsaw puzzles. This algorithm combines shape and image matching with a cyclic ‘‘growth’’ process that tries to place pieces in their correct positions. First, the jigsaw pieces are extracted from the input image. Then, the corner points of the jigsaw pieces are detected. Next, piece classification and recognition are performed based on jigsaw piece models. Connection relationships between pieces are calculated and finally recovered by using boundary shape matching and image merging. We tested this algorithm by employing real-world images containing dozens of jigsaw pieces. The experiment s results show this algorithm is efficient and effective. 2003 Elsevier Science B.V. All rights reserved.


Question: Employing contour techniques for the identification and localization of junctions within natural images.

This paper introduces a computational model of visual attention grounded in the principle of selective tuning, offering a novel approach to the challenges of selection, information flow through the visual hierarchy, and task-oriented attentional focus. The core proposition is that attention serves to enhance the efficiency of the inherent search mechanism in visual processing. This is achieved through the selective adjustment of the visual processing network, facilitated by a top-down cascade of winner-take-all mechanisms integrated within the visual processing pyramid. The paper delves into a detailed comparison with leading computational models of attention and the pertinent neurobiological findings. The model has been realized in a practical implementation, with several demonstrations of its operational capabilities provided. While primarily formulated as a hypothesis for primate visual attention, the model also surpasses current computational attention solutions in machine vision and is ideally suited to addressing the attention problem in robotic vision systems. Specifically, it excels in detecting and localizing junctions in natural images by leveraging contour analysis, thereby enhancing the model's relevance to real-world applications in computer vision.

A computer vision system has been implemented that can recognize threedimensional objects from unknown viewpoints in single gray-scale images. Unlike most other approaches, the recognition is accomplished without any attempt to reconstruct depth information bottom-up from the visual input. Instead, three other mechanisms are used that can bridge the gap between the two-dimensional image and knowledge of three-dimensional objects. First, a process of perceptual organization is used to form groupings and structures in the image that are likely to be invariant over a wide range of viewpoints. Second, a probabilistic ranking method is used to reduce the size of the search space during model based matching. Finally, a process of spatial correspondence brings the projections of three-dimensional models into direct correspondence with the image by solving for unknown viewpoint and model parameters. A high level of robustness in the presence of occlusion and missing data can be achieved through full application of a viewpoint consistency constraint. It is argued that similar mechanisms and constraints form the basis for recognition in human vision. This paper has been published in Artificial Intelligence, 31, 3 (March 1987), pp. 355–395.

The ability to automatically detect ‘visually interesting’ regions in an image has many practical applications especially in the design of active machine vision systems. This paper describes a data-driven approach that uses eye tracking in tandem with principal component analysis to extract lowlevel image features that attract human gaze. Data analysis on an ensemble of image patches extracted at the observer’s point of gaze revealed features that resemble derivatives of the 2D Gaussian operator. Dissimilarities between human and random fixations are investigated by comparing the features extracted at the point of gaze to the general image structure obtained by random sampling in monte-carlo simulations. Finally, a simple application where these features are used to predict fixations is illustrated.

This is the first in a pair of papers in which we present image-processing-based procedures for the measurement of fluvial gravels. The spatial and temporal resolution of surface grain-size characterization is constrained by the time-consuming and costly nature of traditional measurement techniques. Several groups have developed image-processing-based procedures, but none have demonstrated the transferability of these techniques between sites with different lithological, clast form and textural characteristics. Here we focus on image-processing procedures for identifying and measuring image objects (i.e. grains); the second paper examines the application of such procedures to the measurement of fluvially deposited gravels. Four image-segmentation procedures are developed, each having several internal parameters, giving a total of 416 permutations. These are executed on 39 images from three field sites at which the clasts have contrasting physical properties. The performance of each procedure is evaluated against a sample of manually digitized grains in the same images, by comparing three derived statistics. The results demonstrate that it is relatively straightforward to develop procedures that satisfactorily identify objects in any single image or a set of images with similar sedimentary characteristics. However, the optimal procedure is that which gives consistently good results across sites with dissimilar sedimentary characteristics. We show that neighborhood-based operations are the most powerful, and a morphological bottom-hat transform with a double threshold is optimal. It is demonstrated that its performance approaches that of the procedures giving the best results for individual sites. Overall, it out-performs previously published, or improvements to previously published, methods.

Training a generic objectness measure to produce a small set of candidate object windows, has been shown to speed up the classical sliding window object detection paradigm. We observe that generic objects with well-defined closed boundary can be discriminated by looking at the norm of gradients, with a suitable resizing of their corresponding image windows in to a small fixed size. Based on this observation and computational reasons, we propose to resize the window to 8 × 8 and use the norm of the gradients as a simple 64D feature to describe it, for explicitly training a generic objectness measure. We further show how the binarized version of this feature, namely binarized normed gradients (BING), can be used for efficient objectness estimation, which requires only a few atomic operations (e.g. ADD, BITWISE SHIFT, etc.). Experiments on the challenging PASCAL VOC 2007 dataset show that our method efficiently (300fps on a single laptop CPU) generates a small set of category-independent, high quality object windows, yielding 96.2% object detection rate (DR) with 1, 000 proposals. Increasing the numbers of proposals and color spaces for computing BING features, our performance can be further improved to 99.5% DR.

Corner points are formed from two or more edges and edges usually define the boundary between two different objects or parts of the same objects. In this novel we discuss the theory of the Harris corner detection and indicate its disadvantage. Then it proposes an improved algorithm of Harris detection algorithm based on the neighboring point eliminating method. It reduces the time of the detection, and makes the corners distributing more homogenous so that avoids too many corners stay together. Experimental results show that the algorithm can detect corner more equality distributing, and can be used in some fact applications such as image registration well. Key words-Corner point; Harris corner; neighboring point; homogenous distributing 1. Introduce Corner point is a maximum curvature point of the two-dimensional image brightness change or the edge of sharp curves in a image, which haves widely used in such as the camera calibration, virtual scene reconstruction, motion estimation, image registration and more computer vision tasks. These points keep important features of image and can also be effective in reducing the amount of data information, allow real-time processing possible. Now, there are two corner detection methods: based on the image edge method and based on the image gray scale method. The former always need to encode the image edge, which is heavily dependent on image segmentation and edge extraction, have large difficulty and calculate, and if the detected area haves some change, it will lead to the failure of the operation, therefore, this method is not used widely. The latter method detect corner by calculating the curvature and gradient, it avoid these disadvantage. In currently, it haves Moravec operator, Harris operator, Susan operator and so on. Experimental shows that Harris corner detection method haves the best result in above mentioned. But in fact applications, Harris algorithm must to give a proper threshold T, and the algorithm based on the provided threshold to detect the ideal corner. T always depend on the image, specifically it is difficult to determine under the different colors. And it gets the ideal T after user blind set T more times. In addition, the points which have the larger eigenvalues are concentrated in some regions cause the detect corners are non-uniform distribution; and if T dropped, even though the overall distribution of the corner towards reasonable, but it also leads to the points close together, have the clustering phenomenon, it will affect the later analysis and processing. 2. Harris Corner Detection Algorithm 2.1 Detection Theory Harris corner detection algorithm is based on the point feature extraction of signal. It makes the window ω (usually rectangular area) to move infinitesimal displacement to any direction, and the variation of gray can be defined as: − = + + 2 , , , , ] [ v u v y u x v u y x I I w E + + = v u v u y x O yY xX w , 2 2 2 , )] , ( [ 2 2 2 By Cxy Ax + + = T y x M y x ) , ( ) , ( = (1) Where X and Y are one-order gray-level gradient, they can be got by convolution with 978-1-4244-4131-0/09/$25.00 ©2009 IEEE

The goal of this work is to detect a human figure image and localize his joints and limbs along with their associated pixel masks. In this work we attempt to tackle this problem in a general setting. The dataset we use is a collection of sports news photographs of baseball players, varying dramatically in pose and clothing. The approach that we take is to use segmentation to guide our recognition algorithm to salient bits of the image. We use this segmentation approach to build limb and torso detectors, the outputs of which are assembled into human figures. We present quantitative results on torso localization, in addition to shortlisted full body configurations.

The Hough transformation can detect straight lines in an edge-enhanced picture, however its extension to recover ellipses requires too long a computing time. This correspondence proposes a modified method which utilizes two properties of an ellipse in such a way that it iteratively searches for clusters in two different parameter spaces to find almost complete ellipses, then evaluates their parameters by the least mean squares method.

The current OCR cannot segment words and characters from video images due to complex background as well as low resolution of video images. To have better accuracy, this paper presents a new gradient based method for words and character segmentation from text line of any orientation in video frames for recognition. We propose a Max-Min clustering concept to obtain text cluster from the normalized absolute gradient feature matrix of the video text line image. Union of the text cluster with the output of Canny operation of the input video text line is proposed to restore missing text candidates. Then a run length algorithm is applied on the text candidate image for identifying word gaps. We propose a new idea for segmenting characters from the restored word image based on the fact that the text height difference at the character boundary column is smaller than that of the other columns of the word image. We have conducted experiments on a large dataset at two levels (word and character level) in terms of recall, precision and f-measure. Our experimental setup involves 3527 characters of English and Chinese, and this dataset is selected from TRECVID database of 2005 and 2006.

We propose a unified approach for bottom-up hierarchical image segmentation and object candidate generation for recognition, called Multiscale Combinatorial Grouping (MCG). For this purpose, we first develop a fast normalized cuts algorithm. We then propose a high-performance hierarchical segmenter that makes effective use of multiscale information. Finally, we propose a grouping strategy that combines our multiscale regions into highly-accurate object candidates by exploring efficiently their combinatorial space. We conduct extensive experiments on both the BSDS500 and on the PASCAL 2012 segmentation datasets, showing that MCG produces state-of-the-art contours, hierarchical regions and object candidates.

In this paper, we present an interactive system for users to easily colorize the natural images of complex scenes. In our system, colorization procedure is explicitly separated into two stages: Color labeling and Color mapping. Pixels that should roughly share similar colors are grouped into coherent regions in the color labeling stage, and the color mapping stage is then introduced to further fine-tune the colors in each coherent region. To handle textures commonly seen in natural images, we propose a new color labeling scheme that groups not only neighboring pixels with similar intensity but also remote pixels with similar texture. Motivated by the insight into the complementary nature possessed by the highly contrastive locations and the smooth locations, we employ a smoothness map to guide the incorporation of intensity-continuity and texture-similarity constraints in the design of our labeling algorithm. Within each coherent region obtained from the color labeling stage, the color mapping is applied to generate vivid colorization effect by assigning colors to a few pixels in the region. A set of intuitive interface tools is designed for labeling, coloring and modifying the result. We demonstrate compelling results of colorizing natural images using our system, with only a modest amount of user input.


Question: Employing contour analysis for the identification and localization of junctions within natural images.

This document presents an innovative method for creating a three-dimensional representation from a series of images captured by a moving camera. Feature points are identified and tracked across the image sequence, enabling the precise calculation of their three-dimensional positions through the application of Kalman Filters. Additionally, the camera's motion relative to the scene (ego-motion) is accurately estimated. This approach integrates spatial information in three dimensions from a two-dimensional image sequence, offering a robust and accurate solution for 3D scene reconstruction and camera motion analysis.

This paper describes a hierarchical estimation framework for the computation of diverse representations of motion information. The key features of the resulting framework (or family of algorithms) a,re a global model that constrains the overall structure of the motion estimated, a local rnodel that is used in the estimation process, and a coa,rse-fine refinement strategy. Four specific motion models: affine flow, planar surface flow, rigid body motion, and general optical flow, are described along with their application to specific examPles.

The problem we are addressing in Alvey Project MMI149 is that of using computer vision to understand the unconstrained 3D world, in which the viewed scenes will in general contain too wide a diversity of objects for topdown recognition techniques to work. For example, we desire to obtain an understanding of natural scenes, containing roads, buildings, trees, bushes, etc., as typified by the two frames from a sequence illustrated in Figure 1. The solution to this problem that we are pursuing is to use a computer vision system based upon motion analysis of a monocular image sequence from a mobile camera. By extraction and tracking of image features, representations of the 3D analogues of these features can be constructed.

A method is presented for automated segmentation of vessels in two-dimensional color images of the retina. This method can be used in computer analyses of retinal images, e.g., in automated screening for diabetic retinopathy. The system is based on extraction of image ridges, which coincide approximately with vessel centerlines. The ridges are used to compose primitives in the form of line elements. With the line elements an image is partitioned into patches by assigning each image pixel to the closest line element. Every line element constitutes a local coordinate frame for its corresponding patch. For every pixel, feature vectors are computed that make use of properties of the patches and the line elements. The feature vectors are classified using a kNN-classifier and sequential forward feature selection. The algorithm was tested on a database consisting of 40 manually labeled images. The method achieves an area under the receiver operating characteristic curve of 0.952. The method is compared with two recently published rule-based methods of Hoover et al. and Jiang et al. . The results show that our method is significantly better than the two rule-based methods (p<0.01). The accuracy of our method is 0.944 versus 0.947 for a second observer.

This paper considers the objectives of accurate stereo matching, especially at object boundaries, robustness against recording or illumination changes and efficiency of the calculation. These objectives lead to the proposed semi-global matching method that performs pixelwise matching based on mutual information and the approximation of a global smoothness constraint. Occlusions are detected and disparities determined with sub-pixel accuracy. Additionally, an extension for multi-baseline stereo images is presented. There are two novel contributions. Firstly, a hierarchical calculation of mutual information based matching is shown, which is almost as fast as intensity based matching. Secondly, an approximation of a global cost calculation is proposed that can be performed in a time that is linear to the number of pixels and disparities. The implementation requires just 1 second on typical images.

We present a robust and real time approach to lane marker detection in urban streets. It is based on generating a top view of the road, filtering using selective oriented Gaussian filters, using RANSAC line fitting to give initial guesses to a new and fast RANSAC algorithm for fitting Bezier Splines, which is then followed by a post-processing step. Our algorithm can detect all lanes in still images of the street in various conditions, while operating at a rate of 50 Hz and achieving comparable results to previous techniques.

A lane-detection system is an important component of many intelligent transportation systems. We present a robust lane-detection-and-tracking algorithm to deal with challenging scenarios such as a lane curvature, worn lane markings, lane changes, and emerging, ending, merging, and splitting lanes. We first present a comparative study to find a good real-time lane-marking classifier. Once detection is done, the lane markings are grouped into lane-boundary hypotheses. We group left and right lane boundaries separately to effectively handle merging and splitting lanes. A fast and robust algorithm, based on random-sample consensus and particle filtering, is proposed to generate a large number of hypotheses in real time. The generated hypotheses are evaluated and grouped based on a probabilistic framework. The suggested framework effectively combines a likelihood-based object-recognition algorithm with a Markov-style process (tracking) and can also be applied to general-part-based object-tracking problems. An experimental result on local streets and highways shows that the suggested algorithm is very reliable.

We present a discrete approach for boundary extraction from 3D image data. The proposed technique is based on the duality between the Voronoi graph computed accross the digital boundary and the Delaunay triangulation. The originality of the approach is that algorithms perform only integer arithmetic and the method does not suffer from standard round problems and numerical instabilities in the case of floating point computations. This method has been applied both on segmented anatomical structures and on manufactured objects presenting corners and edges. The experimental results show that the method allows to produce a polygonal boundary representation which is guaranteed to be a 2-manifold. This representation is successfully transformed into a triangular quality mesh which meets all topological and geometrical requirements of applications such as augmented reality or simulation.

In this work we show that Deep Convolutional Neural Networks can outperform humans on the task of boundary detection, as measured on the standard Berkeley Segmentation Dataset. Our detector is fully integrated in the popular Caffe framework and processes a 320x420 image in less than a second. Our contributions consist firstly in combining a careful design of the loss for boundary detection training, a multi-resolution architecture and training with external data to improve the detection accuracy of the current state of the art, from an optimal dataset scale F-measure of 0.780 to 0.808 while human performance is at 0.803. We further improve performance to 0.813 by combining deep learning with grouping, integrating the Normalized Cuts technique within a deep network. We also examine the potential of our boundary detector in conjunction with the higher level tasks of object proposal generation and semantic segmentation for both tasks our detector yields clear improvements over state-of-the-art systems.

-This paper presents a texture segmentation algorithm inspired by the multi-channel filtering theory for visual information processing in the early stages of human visual system. The channels are characterized by a bank of Gabor filters that nearly uniformly covers the spatial-frequency domain, and a systematic filter selection scheme is proposed, which is based on reconstruction of the input image from the filtered images. Texture features are obtained by subjecting each (selected) filtered image to a nonlinear transformation and computing a measure of "energy" in a window around each pixel. A square-error clustering algorithm is then used to integrate the feature images and produce a segmentation. A simple procedure to incorporate spatial information in the clustering process is proposed. A relative index is used to estimate the "'true" number of texture categories. Texture segmentation Multi-channel filtering Clustering Clustering index Gabor filters Wavelet transform I . I N T R O D U C T I O N Image segmentation is a difficult yet very important task in many image analysis or computer vision applications. Differences in the mean gray level or in color in small neighborhoods alone are not always sufficient for image segmentation. Rather, one has to rely on differences in the spatial arrangement of gray values of neighboring pixels-that is, on differences in texture. The problem of segmenting an image based on textural cues is referred to as the texture segmentation problem. Texture segmentation involves identifying regions with "uniform" textures in a given image. Appropriate measures of texture are needed in order to decide whether a given region has uniform texture. Sklansky (o has suggested the following definition of texture which is appropriate in the segmentation context: "A region in an image has a constant texture if a set of local statistics or other local properties of the picture are constant, slowly varying, or approximately periodic". Texture, therefore, has both local and global connotations--i t is characterized by invariance of certain local measures or properties over an image region. The diversity of natural and artificial textures makes it impossible to give a universal definition of texture. A large number of techniques for analyzing image texture has been proposed in the past two decades/2,3) In this paper, we focus on a particular approach to texture analysis which is referred to as ° This work was supported in part by the National Science Foundation infrastructure grant CDA-8806599, and by a grant from E. I. Du Pont De Nemours & Company Inc. the multi-channel filtering approach. This approach is inspired by a multi-channel filtering theory for processing visual information in the early stages of the human visual system. First proposed by Campbell and Robson (4) the theory holds that the visual system decomposes the retinal image into a number of filtered images, each of which contains intensity variations over a narrow range of frequency (size) and orientation. The psychophysical experiments that suggested such a decomposition used various grating patterns as stimuli and were based on adaptation techniques. I~l Subsequent psychophysiological experiments provided additional evidence supporting the theory. De Valois et al. ,(5) for example, recorded the response of simple cells in the visual cortex of the Macaque monkey to sinusoidal gratings with different frequencies and orientations. It was observed that each cell responds to a narrow range of frequency and orientation only. Therefore, it appears that there are mechanisms in the visual cortex of mammals that are tuned to combinations of frequency and orientation in a narrow range. These mechanisms are often referred to as channels, and are appropriately interpreted as band-pass filters. The multi-channel filtering approach to texture analysis is intuitively appealing because it allows us to exploit differences in dominant sizes and orientations of different textures. Today, the need for a multi-resolution approach to texture analysis is well recognized. While other approaches to texture analysis have had to be extended to accommodate this paradigm, the multi-channel filtering approach, is inherently multi-resolutional. Another important

Moving vehicles are detected and tracked automatically in monocular image sequences from road traffic scenes recorded by a stationary camera. In order to exploit the a priori knowledge about shape and motion of vehicles in traffic scenes, a parameterized vehicle model is used for an intraframe matching process and a recursive estimator based on a motion model is used for motion estimation. An interpretation cycle supports the intraframe matching process with a state MAP-update step. Initial model hypotheses are generated using an image segmentation component which clusters coherently moving image features into candidate representations of images of a moving vehicle. The inclusion of an illumination model allows taking shadow edges of the vehicle into account during the matching process. Only such an elaborate combination of various techniques has enabled us to track vehicles under complex illumination conditions and over long (over 400 frames) monocular image sequences. Results on various real-world road traffic scenes are presented and open problems as well as future work are outlined.


Question: Hierarchical character-word models for the identification of languages.

This paper explores the application of multi-task learning to optimize the performance of slot filling models for natural language understanding across various tasks or domains. The central challenge is to minimize the training data required for model adaptation to new tasks. The proposed multi-task framework achieves superior results with reduced data volumes by harnessing insights from related tasks. It is designed to accommodate an open vocabulary, enabling models to extend their capabilities to words not encountered during training, a critical feature when training data is scarce. The efficacy of the domain adaptation and open vocabulary strategies is showcased through an evaluation on a recently compiled crowd-sourced dataset, encompassing four distinct domains.

**Question**: A Comparative Study of Deep Learning Techniques for Network Intrusion Detection

**Documents**: A Hybrid Deep Learning Approach for Intrusion Detection in SDN
Intrusion Detection Systems (IDSs) have been widely used to secure networks by detecting anomalies and malicious activities. With the emergence of Software-Defined Networking (SDN), there is a need for more efficient and intelligent IDSs that can adapt to the new network paradigm. This paper proposes a hybrid deep learning approach for intrusion detection in SDN. The proposed approach combines the strengths of Convolutional Neural Networks (CNNs) and Long Short-Term Memory (LSTM) networks to create a more effective IDS. The CNN is used to extract features from network traffic data, while the LSTM is used to capture the temporal dependencies in the data. The proposed approach is evaluated on a standard dataset and compared with other state-of-the-art methods. The results show that the proposed approach outperforms the other methods in terms of detection accuracy and false positive rate.

**Similar Documents**: This paper introduces a novel hybrid deep learning framework designed to enhance Intrusion Detection Systems (IDSs) within Software-Defined Networking (SDN) environments. Given the critical role IDSs play in safeguarding networks against anomalies and cyber threats, the transition to SDN necessitates more sophisticated and adaptable detection mechanisms. The proposed solution integrates the capabilities of Convolutional Neural Networks (CNNs) and Long Short-Term Memory (LSTM) networks to bolster the effectiveness of SDN-based IDSs. Specifically, the CNN is tasked with extracting relevant features from network traffic data, while the LSTM component is responsible for discerning the temporal patterns inherent in the data. To assess the performance of this hybrid approach, it is rigorously tested on a canonical dataset and its outcomes are juxtaposed against those of leading-edge methodologies. The comparative analysis reveals that the proposed framework excels in detection accuracy and exhibits a

This paper reports on investigations using two techniques for language model text data augmentation for low-resourced automatic speech recognition and keyword search. Lowresourced languages are characterized by limited training materials, which typically results in high out-of-vocabulary (OOV) rates and poor language model estimates. One technique makes use of recurrent neural networks (RNNs) using word or subword units. Word-based RNNs keep the same system vocabulary, so they cannot reduce the OOV, whereas subword units can reduce the OOV but generate many false combinations. A complementary technique is based on automatic machine translation, which requires parallel texts and is able to add words to the vocabulary. These methods were assessed on 10 languages in the context of the Babel program and NIST OpenKWS evaluation. Although improvements vary across languages with both methods, small gains were generally observed in terms of word error rate reduction and improved keyword search performance.

We describe efforts to adapt the Tesseract open source OCR engine for multiple scripts and languages. Effort has been concentrated on enabling generic multi-lingual operation such that negligible customization is required for a new language beyond providing a corpus of text. Although change was required to various modules, including physical layout analysis, and linguistic post-processing, no change was required to the character classifier beyond changing a few limits. The Tesseract classifier has adapted easily to Simplified Chinese. Test results on English, a mixture of European languages, and Russian, taken from a random sample of books, show a reasonably consistent word error rate between 3.72% and 5.78%, and Simplified Chinese has a character error rate of only 3.77%.

Direct acoustics-to-word (A2W) models in the end-to-end paradigm have received increasing attention compared to conventional subword based automatic speech recognition models using phones, characters, or context-dependent hidden Markov model states. This is because A2W models recognize words from speech without any decoder, pronunciation lexicon, or externally-trained language model, making training and decoding with such models simple. Prior work has shown that A2W models require orders of magnitude more training data in order to perform comparably to conventional models. Our work also showed this accuracy gap when using the English Switchboard-Fisher data set. This paper describes a recipe to train an A2W model that closes this gap and is at-par with state-of-the-art sub-word based models. We achieve a word error rate of 8.8.8%/13.9% on the Hub5-2000 Switchboard/CallHome test sets without any decoder or language model. We find that model initialization, training data order, and regularization have the most impact on the A2W model performance. Next, we present a joint word-character A2W model that learns to first spell the word and then recognize it. This model provides a rich output to the user instead of simple word hypotheses, making it especially useful in the case of words unseen or rarely-seen during training.

We present a method for authorship discrimination that is based on the frequency of bigrams of syntactic labels that arise from partial parsing of the text. We show that this method, alone or combined with other classification features, achieves a high accuracy on discrimination of the work of Anne and Charlotte Brontë, which is very difficult to do by traditional methods. Moreover, high accuracies are achieved even on fragments of text little more than 200 words long. .................................................................................................................................................................................

This paper presents a model to convert natural Bengali language to text. The proposed model requires the usage of the open sourced framework Sphinx 4 which is written in Java and provides the required procedural coding tools to develop an acoustic model for a custom language like Bengali. Our main objective was to ensure that the system was adequately trained on a word by word basis from various speakers so that it could recognize new speakers fluently. We used a free digital audio workstation (DAW) called Audacity to manipulate the collected recording data via continuous frequency profiling techniques to reduce the Signal-to-Noise-Ratio (SNR), vocal leveling, normalization and syllable splitting as well as merging which ensure an error free 1:1-word mapping of each utterance with its mirror transcription file text. To evaluate the performance of proposed model, we utilize an audio dataset of recorded speech data from 10 individual speakers consisting of both males and females using custom transcript files that we wrote. Experimental results demonstrate that the proposed model exhibits average 71.7% accuracy for our tested dataset.

Today, parallel corpus-based systems dominate the transliteration landscape. But the resource-scarce languages do not enjoy the luxury of large parallel transliteration corpus. For these languages, rule-based transliteration is the only viable option. In this article, we show that by properly harnessing the monolingual resources in conjunction with manually created rule base, one can achieve reasonable transliteration performance. We achieve this performance by exploiting the power of Character Sequence Modeling (CSM), which requires only monolingual resources. We present the results of our rule-based system for Hindi to English, English to Hindi, and Persian to English transliteration tasks. We also perform extrinsic evaluation of transliteration systems in the context of Cross Lingual Information Retrieval. Another important contribution of our work is to explain the widely varying accuracy numbers reported in transliteration literature, in terms of the entropy of the language pairs and the datasets involved.

In this paper, we explore a novel bilingual word alignment approach based on DNN (Deep Neural Network), which has been proven to be very effective in various machine learning tasks (Collobert et al., 2011). We describe in detail how we adapt and extend the CD-DNNHMM (Dahl et al., 2012) method introduced in speech recognition to the HMMbased word alignment model, in which bilingual word embedding is discriminatively learnt to capture lexical translation information, and surrounding words are leveraged to model context information in bilingual sentences. While being capable to model the rich bilingual correspondence, our method generates a very compact model with much fewer parameters. Experiments on a large scale EnglishChinese word alignment task show that the proposed method outperforms the HMM and IBM model 4 baselines by 2 points in F-score.

We propose a new phrase-based translation model and decoding algorithm that enables us to evaluate and compare several, previously proposed phrase-based translation models. Within our framework, we carry out a large number of experiments to understand better and explain why phrase-based models outperform word-based models. Our empirical results, which hold for all examined language pairs, suggest that the highest levels of performance can be obtained through relatively simple means: heuristic learning of phrase translations from word-based alignments and lexical weighting of phrase translations. Surprisingly, learning phrases longer than three words and learning phrases from high-accuracy wordlevel alignment models does not have a strong impact on performance. Learning only syntactically motivated phrases degrades the performance of our systems.

Automatic abstractive text summarization is an important and challenging research topic of natural language processing. Among many widely used languages, the Chinese language has a special property that a Chinese character contains rich information comparable to a word. Existing Chinese text summarization methods, either adopt totally character-based or word-based representations, fail to fully exploit the information carried by both representations. To accurately capture the essence of articles, we propose a hybrid word-character approach (HWC) which preserves the advantages of both wordbased and character-based representations. We evaluate the advantage of the proposed HWC approach by applying it to two existing methods, and discover that it generates state-of-the-art performance with a margin of 24 ROUGE points on a widely used dataset LCSTS. In addition, we find an issue contained in the LCSTS dataset and offer a script to remove overlapping pairs (a summary and a short text) to create a clean dataset for the community. The proposed HWC approach also generates the best performance on the new, clean LCSTS dataset.

Previous work demonstrated that Web counts can be used to approximate bigram counts, suggesting that Web-based frequencies should be useful for a wide variety of Natural Language Processing (NLP) tasks. However, only a limited number of tasks have so far been tested using Web-scale data sets. The present article overcomes this limitation by systematically investigating the performance of Web-based models for several NLP tasks, covering both syntax and semantics, both generation and analysis, and a wider range of n-grams and parts of speech than have been previously explored. For the majority of our tasks, we find that simple, unsupervised models perform better when n-gram counts are obtained from the Web rather than from a large corpus. In some cases, performance can be improved further by using backoff or interpolation techniques that combine Web counts and corpus counts. However, unsupervised Web-based models generally fail to outperform supervised state-of-the-art models trained on smaller corpora. We argue that Web-based models should therefore be used as a baseline for, rather than an alternative to, standard supervised models.