Question: Focusing on the descriptions produced by image captioning models.

Introducing the Microsoft COCO: Common Objects in Context Dataset
This paper unveils the Microsoft COCO (Common Objects in Context) dataset, designed to push the boundaries of object recognition technology by integrating the task of object recognition within the larger framework of scene understanding. The dataset is compiled from images of intricate, everyday scenes, featuring common objects in their natural settings. To facilitate accurate object localization, objects are meticulously labeled through per-instance segmentation. The dataset encompasses 91 object categories, all of which would be familiar to a four-year-old child, reflecting a broad spectrum of everyday items. With a total of 2.5 million labeled instances spread across 328,000 images, the creation of this dataset was a massive undertaking, relying heavily on crowd-sourced labor facilitated through innovative user interfaces for category detection, instance spotting, and instance segmentation. A comprehensive statistical analysis of the dataset is provided, contrasting it with other prominent datasets like PASCAL, ImageNet, and SUN. Additionally, baseline performance metrics for both bounding box and segmentation detection are established using a Deformable Parts Model, serving as a benchmark for future research in object recognition within complex scenes.

Attention is an increasingly popular mechanism used in a wide range of neural architectures. Because of the fast-paced advances in this domain, a systematic overview of attention is still missing. In this article, we define a unified model for attention architectures for natural language processing, with a focus on architectures designed to work with vector representation of the textual data. We discuss the dimensions along which proposals differ, the possible uses of attention, and chart the major research activities and open challenges in the area.

We present an approach that exploits hierarchical Recurrent Neural Networks (RNNs) to tackle the video captioning problem, i.e., generating one or multiple sentences to describe a realistic video. Our hierarchical framework contains a sentence generator and a paragraph generator. The sentence generator produces one simple short sentence that describes a specific short video interval. It exploits both temporal-and spatial-attention mechanisms to selectively focus on visual elements during generation. The paragraph generator captures the inter-sentence dependency by taking as input the sentential embedding produced by the sentence generator, combining it with the paragraph history, and outputting the new initial state for the sentence generator. We evaluate our approach on two large-scale benchmark datasets: YouTubeClips and TACoS-MultiLevel. The experiments demonstrate that our approach significantly outperforms the current state-of-the-art methods with BLEU@4 scores 0.499 and 0.305 respectively.

An overview of the state-of-the-art on semantics extraction from images is presented. In this survey, we present the relevant approaches in terms of content representation as well as in terms of knowledge representation. Knowledge can be represented in either implicit or explicit fashion while the image is represented in different levels, namely, low-level, intermediate and semantic level. For each combination of knowledge and image representation, a detailed discussion is addressed that leads to fruitful conclusions for the impact of each approach. 1 Semantics extraction basic pipeline Semantics extraction refers to digital data interpretation from a human point of view. In the case that the digital data correspond to images, this usually entails an appearance-based inference using color, texture and/or shape information along with a type of context inference (or representation) that can combine and transform these machine-extracted evidence into what we call a scene description. Following Biederman et al. [1] definitions of context in a visual scene, we can derive three types of context for real-world scene annotation problems: (i) semantic context which encodes the probability of a certain category to be present in a scene (e.g category “streets” has high probability to coexist with category “building” in the same scene ); (ii) spatial context which encodes the spatial relations of categories (e.g sky is usually above grass in a scene) and (iii) scale context which encodes the relative object size (category “human” is expected to occupy a small region in a scene which includes the“building” category). The research goals in semantics extraction are mostly a function of the granularity of the semantics in question. The goal could be the extraction of a single or multiple semantics of the entire image (e.g. indoor/outdoor setting), or the extraction of the semantics for different objects in an image. In the latter case, the semantics could be generic (e.g. a vehicle) or specific (e.g. a motorbike). Those goals make it clear that semantics extraction is not a new research area. Depending on the goal, the task of semantics extraction can be considered as a categorization, classification, recognition and understanding task that all share in common the effort for solving the semantic gap. As stated in [2], “Despite intensive recent research, the automatic establishment of a correspondence between

Sequence-to-sequence models have shown promising improvements on the temporal task of video captioning, but they optimize word-level cross-entropy loss during training. First, using policy gradient and mixed-loss methods for reinforcement learning, we directly optimize sentence-level task-based metrics (as rewards), achieving significant improvements over the baseline, based on both automatic metrics and human evaluation on multiple datasets. Next, we propose a novel entailment-enhanced reward (CIDEnt) that corrects phrase-matching based metrics (such as CIDEr) to only allow for logically-implied partial matches and avoid contradictions, achieving further significant improvements over the CIDEr-reward model. Overall, our CIDEnt-reward model achieves the new state-of-the-art on the MSR-VTT dataset.

This paper explores the current state of the task of generating easily understandable information from data for people using natural language, which is currently addressed by two independent research fields: the natural language generation field — and, more specifically, the data-to-text sub-field — and the linguistic descriptions of data field. Both approaches are explained in a detailed description which includes: i) a methodological revision of both fields including basic concepts and definitions, models and evaluation procedures; ii) the most relevant systems, use cases and real applications described in the literature. Some reflections about the current state and future trends of each field are also provided, followed by several remarks that conclude by hinting at some potential points of mutual interest and convergence between both fields. © 2015 Elsevier B.V. All rights reserved.

We propose an iterative crowd-enabled active learning algorithm for building high-precision visual classifiers from unlabeled images. Our method employs domain experts to identify a small number of examples of a specific visual event. These expert-labeled examples seed a classifier, which is then iteratively trained by active querying of a non-expert crowd. These non-experts actively refine the classifiers at every iteration by answering simple binary questions about the classifiers’ detections. The advantage of this approach is that experts efficiently shepherd an unsophisticated crowd into training a classifier capable of fine-grained distinctions. This obviates the need to label an entire dataset to obtain high-precision classifiers. We find these classifiers are advantageous for creating a large vocabulary of visual attributes for specialized taxonomies. We demonstrate our crowd active learning pipeline by creating classifiers for attributes related to North American birds and fashion.

Models that learn to represent textual and knowledge base relations in the same continuous latent space are able to perform joint inferences among the two kinds of relations and obtain high accuracy on knowledge base completion (Riedel et al., 2013). In this paper we propose a model that captures the compositional structure of textual relations, and jointly optimizes entity, knowledge base, and textual relation representations. The proposed model significantly improves performance over a model that does not share parameters among textual relations with common sub-structure.

Given a pre-trained CNN without any testing samples, this paper proposes a simple yet effective method to diagnose feature representations of the CNN. We aim to discover representation flaws caused by potential dataset bias. More specifically, when the CNN is trained to estimate image attributes, we mine latent relationships between representations of different attributes inside the CNN. Then, we compare the mined attribute relationships with ground-truth attribute relationships to discover the CNN’s blind spots and failure modes due to dataset bias. In fact, representation flaws caused by dataset bias cannot be examined by conventional evaluation strategies based on testing images, because testing images may also have a similar bias. Experiments have demonstrated the effectiveness of our method.

With the rapid proliferation of smartphones and tablet computers, search has moved beyond text to other modalities like images and voice. For many applications like Fashion, visual search offers a compelling interface that can capture stylistic visual elements beyond color and pattern that cannot be as easily described using text. However, extracting and matching such attributes remains an extremely challenging task due to high variability and deformability of clothing items. In this paper, we propose a fine-grained learning model and multimedia retrieval framework to address this problem. First, an attribute vocabulary is constructed using human annotations obtained on a novel finegrained clothing dataset. This vocabulary is then used to train a fine-grained visual recognition system for clothing styles. We report benchmark recognition and retrieval results on Women’s Fashion Coat Dataset and illustrate potential mobile applications for attribute-based multimedia retrieval of clothing items and image annotation.

We extend the SKIP-GRAM model of Mikolov et al. (2013a) by taking visual information into account. Like SKIP-GRAM, our multimodal models (MMSKIP-GRAM) build vector-based word representations by learning to predict linguistic contexts in text corpora. However, for a restricted set of words, the models are also exposed to visual representations of the objects they denote (extracted from natural images), and must predict linguistic and visual features jointly. The MMSKIP-GRAM models achieve good performance on a variety of semantic benchmarks. Moreover, since they propagate visual information to all words, we use them to improve image labeling and retrieval in the zero-shot setup, where the test concepts are never seen during model training. Finally, the MMSKIP-GRAM models discover intriguing visual properties of abstract words, paving the way to realistic implementations of embodied theories of meaning.


Question: Focusing on the descriptions produced by image captioning models.

Innovative Attention Mechanisms for Image Captioning

This paper introduces an advanced attention-based model for automatic image captioning, termed "Innovative Attention Mechanisms". Our approach innovatively models the interdependencies among image segments, caption words, and the state of a recurrent neural network (RNN) language model through three distinct pairwise interactions. Diverging from conventional attention-based methods that solely link image segments to the RNN state, our model facilitates a direct connection between caption words and image segments. During the training phase, these connections are deduced from image-level captions, paralleling the training strategy employed in weakly-supervised object detection. These connections prove beneficial by pinpointing the relevant regions during the testing phase, thereby enhancing caption accuracy. We also explore and contrast various methods for generating attention areas: CNN activation grids, object proposals, and spatial transformer networks applied in a convolutional manner. Spatial transformers emerge as the most effective, enabling image-specific attention areas and permitting joint training with the rest of the network. Integrating our attention mechanism with spatial transformer attention areas results in cutting-edge performance on the MSCOCO dataset, showcasing the model's superiority in image captioning tasks.

Natural language generation lies at the core of generative dialogue systems and conversational agents. We describe an ensemble neural language generator, and present several novel methods for data representation and augmentation that yield improved results in our model. We test the model on three datasets in the restaurant, TV and laptop domains, and report both objective and subjective evaluations of our best model. Using a range of automatic metrics, as well as human evaluators, we show that our approach achieves better results than state-of-the-art models on the same datasets.

We propose a novel attention network, which accurately attends to target objects of various scales and shapes in images through multiple stages. The proposed network enables multiple layers to estimate attention in a convolutional neural network (CNN). The hierarchical attention model gradually suppresses irrelevant regions in an input image using a progressive attentive process over multiple CNN layers. The attentive process in each layer determines whether to pass or suppress feature maps for use in the next convolution. We employ local contexts to estimate attention probability at each location since it is difficult to infer accurate attention by observing a feature vector from a single location only. The experiments on synthetic and real datasets show that the proposed attention network outperforms traditional attention methods in various attribute prediction tasks.

Questions that require counting a variety of objects in images remain a major challenge in visual question answering (VQA). The most common approaches to VQA involve either classifying answers based on fixed length representations of both the image and question or summing fractional counts estimated from each section of the image. In contrast, we treat counting as a sequential decision process and force our model to make discrete choices of what to count. Specifically, the model sequentially selects from detected objects and learns interactions between objects that influence subsequent selections. A distinction of our approach is its intuitive and interpretable output, as discrete counts are automatically grounded in the image. Furthermore, our method outperforms the state of the art architecture for VQA on multiple metrics that evaluate counting.

Together with the development of more accurate methods in Computer Vision and Natural Language Understanding, holistic architectures that answer on questions about the content of real-world images have emerged. In this tutorial, we build a neural-based approach to answer questions about images. We base our tutorial on two datasets: (mostly on) DAQUAR, and (a bit on) VQA. With small tweaks the models that we present here can achieve a competitive performance on both datasets, in fact, they are among the best methods that use a combination of LSTM with a global, full frame CNN representation of an image. We hope that after reading this tutorial, the reader will be able to use Deep Learning frameworks, such as Keras and introduced Kraino, to build various architectures that will lead to a further performance improvement on this challenging task. 1 Preface In this tutorial1 we build a few architectures that can answer questions about images. The architectures are based on our two papers on this topic: Malinowski et al. [2015] and Malinowski et al. [2016]; and more broadly, on our project towards a Visual Turing Test2. In particular, an encoder-decoder perspective of Malinowski et al. [2016] allows us to effectively experiment with various design choices. For the sake of simplicity, we only consider a classification-based approach to answer questions about images, although an approach that generate answers word-by-word is also studied in the community [Malinowski et al., 2015]. In the tutorial, we mainly focus on the DAQUAR dataset [Malinowski and Fritz, 2014a], but a few possible directions to apply learnt techniques to VQA [Antol et al., 2015] are also pointed. First, we will get familiar with the task of answering questions about images, and a dataset that implements the task (due to a small size, we mainly use DAQUAR as it better serves an educational purpose that we aim at this tutorial). Next, we build a few blind models that answer questions about images without actually seeing such images. Such models already exhibit a reasonable performance as they can effectively learn various biases that exist in a dataset, which we also interpret as learning a common sense knowledge [Malinowski et al., 2015, 2016]. Subsequently, we build a few language+vision models that answer questions based on both a textual and a visual inputs. Finally, we leave the tutorial with a few possible research directions. Technical aspects The tutorial is originally written using Python Notebook, which the reader is welcome to download and use through the tutorial. Instructions necessary to run the Notebook version of this tutorial are provided in the following: https://github.com/mateuszmalinowski/visual_turing_test-tutorial. In this tutorial, we heavily use a Python code, and therefore it is expected the reader either already knows this language, or can quickly learn it. However, we made an effort to make this tutorial approachable to a wider audience. We use Kraino that is a framework prepared for this tutorial in order to simplify the development of the question answering architectures. Under the hood, it uses Theano4 [Bastien et al., 2012] and Keras5 [Chollet, 2015] – two frameworks to build Deep Learning models. We also use various CNNs representations extracted from images that can be downloaded as explained at the beginning of our Notebook tutorial. We also highlight exercises that a curious reader may attempt to solve. This tutorial was presented for the first time during the 2nd Summer School on Integrating Vision and Language: Deep Learning. http://mpii.de/visual_turing_test https://github.com/mateuszmalinowski/visual_turing_test-tutorial/blob/master/visual_ turing_test.ipynb http://deeplearning.net/software/theano/ https://keras.io 1 ar X iv :1 61 0. 01 07 6v 1 [ cs .C V ] 4 O ct 2 01 6

We consider the ROC story cloze task (Mostafazadeh et al., 2016) and present several findings. We develop a model that uses hierarchical recurrent networks with attention to encode the sentences in the story and score candidate endings. By discarding the large training set and only training on the validation set, we achieve an accuracy of 74.7%. Even when we discard the story plots (sentences before the ending) and only train to choose the better of two endings, we can still reach 72.5%. We then analyze this “ending-only” task setting. We estimate human accuracy to be 78% and find several types of clues that lead to this high accuracy, including those related to sentiment, negation, and general ending likelihood regardless of the story context.

There is an increased interest in artificially intelligent technology that surrounds us and takes decisions on our behalf. This creates the need for such technology to be able to communicate with humans and understand natural language and non-verbal behaviour that may carry information about our complex physical world. Artificial agents today still have little knowledge about the physical space that surrounds us and about the objects or concepts within our attention. We are still lacking computational methods in understanding the context of human conversation that involves objects and locations around us. Can we use multimodal cues from human perception of the real world as an example of language learning for robots? Can artificial agents and robots learn about the physical world by observing how humans interact with it and how they refer to it and attend during their conversations? This PhD project’s focus is on combining spoken language and non-verbal behaviour extracted by multi-party dialogue in order to increase context awareness and spatial understanding for artificial agents.

As described in the main paper, we collected a dataset D = (x(n), e(n), a(n), π(n))500 n=1. Environment Complexity. Our environments are 3D scenarios consisting of complex objects such as fridge, microwave and television with many states. These objects can be in different spatial relations with respect to other objects. For example, “bag of chips” can be found behind the television. Figure 1 shows some sample environments from our dataset. For example, an object of category television consists of 6 channels, volume level and power status. An object can have different values of states in different environment and different environment consists of different set of objects and their placement. For example, television might be powered on in one environment and closed in another, microwave might have an object inside it or not in different environment, etc. Moreover, there are often more than one object of the same category. For example, our environment typically have two books, five couches, four pillows etc. Objects of the same category can have different appearance. For example, a book can have the cover of a math book or a Guinness book of world record; resulting in complex object descriptions such as in “throw the sticky stuff in the bowl”. They can also have the same appearance making people use spatial relations or other means while describing them such as in “get me the cup next to the microwave”. This dataset is significantly more challenging compared to the 2D navigation dataset or GUI actions in windows dataset considered earlier.

This paper aims to highlight vision related tasks centered around “car”, which has been largely neglected by vision community in comparison to other objects. We show that there are still many interesting car-related problems and applications, which are not yet well explored and researched. To facilitate future car-related research, in this paper we present our on-going effort in collecting a large-scale dataset, “CompCars”, that covers not only different car views, but also their different internal and external parts, and rich attributes. Importantly, the dataset is constructed with a cross-modality nature, containing a surveillance-nature set and a web-nature set. We further demonstrate a few important applications exploiting the dataset, namely car model classification, car model verification, and attribute prediction. We also discuss specific challenges of the car-related problems and other potential applications that worth further investigations. The latest dataset can be downloaded at http://mmlab.ie.cuhk.edu.hk/ datasets/comp_cars/index.html.

With the rapid proliferation of smartphones and tablet computers, search has moved beyond text to other modalities like images and voice. For many applications like Fashion, visual search offers a compelling interface that can capture stylistic visual elements beyond color and pattern that cannot be as easily described using text. However, extracting and matching such attributes remains an extremely challenging task due to high variability and deformability of clothing items. In this paper, we propose a fine-grained learning model and multimedia retrieval framework to address this problem. First, an attribute vocabulary is constructed using human annotations obtained on a novel finegrained clothing dataset. This vocabulary is then used to train a fine-grained visual recognition system for clothing styles. We report benchmark recognition and retrieval results on Women’s Fashion Coat Dataset and illustrate potential mobile applications for attribute-based multimedia retrieval of clothing items and image annotation.

Generic generation and manipulation of text is challenging and has limited success compared to recent deep generative modeling in visual domain. This paper aims at generating plausible natural language sentences, whose attributes are dynamically controlled by learning disentangled latent representations with designated semantics. We propose a new neural generative model which combines variational auto-encoders and holistic attribute discriminators for effective imposition of semantic structures. With differentiable approximation to discrete text samples, explicit constraints on independent attribute controls, and efficient collaborative learning of generator and discriminators, our model learns highly interpretable representations from even only word annotations, and produces realistic sentences with desired attributes. Quantitative evaluation validates the accuracy of sentence and attribute generation.


Question: Focusing on the descriptions produced by image captioning models.

This paper introduces BLEU (Bilingual Evaluation Understudy), a metric designed for the automatic evaluation of machine translation quality. Unlike previous methods that relied heavily on human judgment, BLEU provides an objective and efficient way to assess translations by comparing them against a set of reference translations. The metric is based on the premise that a good translation should contain a high proportion of n-grams (sequences of n words) found in the reference translations. BLEU calculates a score between 0 and 1, with higher scores indicating better translation quality. This approach has been widely adopted in the field of natural language processing, particularly for evaluating the performance of machine translation systems. By offering a standardized method for measuring translation accuracy, BLEU has facilitated the development and comparison of various machine translation algorithms, ultimately contributing to advancements in automatic language processing technologies.

**Question**: A Comparative Study of Deep Learning Models for Text Classification

**Documents**: Convolutional Neural Networks for Sentence Classification

**Similar Documents**: This study explores the application of convolutional neural networks (CNNs) in sentence classification tasks. CNNs, originally developed for image processing, have proven to be highly effective in capturing the spatial hierarchy of visual patterns. In this paper, we adapt CNNs to text classification by treating sentences as one-dimensional "images" of words. The network architecture employs a series of convolutional layers followed by max-pooling operations to extract high-level features from the input sentences. These features are then fed into a fully connected layer for classification. We demonstrate the effectiveness of our approach by comparing it with traditional machine learning algorithms and other deep learning models on various text classification datasets. The results show that CNNs achieve superior performance, highlighting their potential for tackling complex natural language processing tasks. This work contributes to the growing body of research on deep learning for text analysis and opens up new avenues for improving the accuracy of text classification systems.

The majority of NLG evaluation relies on automatic metrics, such as BLEU. In this paper, we motivate the need for novel, systemand data-independent automatic evaluation methods: We investigate a wide range of metrics, including state-of-the-art word-based and novel grammar-based ones, and demonstrate that they only weakly reflect human judgements of system outputs as generated by data-driven, end-to-end NLG. We also show that metric performance is dataand system-specific. Nevertheless, our results also suggest that automatic metrics perform reliably at system-level and can support system development by finding cases where a system performs poorly.

Accelerated by the tremendous increase in Internet bandwidth and storage space, video data has been generated, published and spread explosively, becoming an indispensable part of today's big data. In this paper, we focus on reviewing two lines of research aiming to stimulate the comprehension of videos with deep learning: video classification and video captioning. While video classification concentrates on automatically labeling video clips based on their semantic contents like human actions or complex events, video captioning attempts to generate a complete and natural sentence, enriching the single label as in video classification, to capture the most informative dynamics in videos. In addition, we also provide a review of popular benchmarks and competitions, which are critical for evaluating the technical progress of this vibrant field.

This work introduces a model that can recognize objects in images even if no training data is available for the objects. The only necessary knowledge about the unseen categories comes from unsupervised large text corpora. In our zero-shot framework distributional information in language can be seen as spanning a semantic basis for understanding what objects look like. Most previous zero-shot learning models can only differentiate between unseen classes. In contrast, our model can both obtain state of the art performance on classes that have thousands of training images and obtain reasonable performance on unseen classes. This is achieved by first using outlier detection in the semantic space and then two separate recognition models. Furthermore, our model does not require any manually defined semantic features for either words or images.

Neural network architectures with memory and attention mechanisms exhibit certain reasoning capabilities required for question answering. One such architecture, the dynamic memory network (DMN), obtained high accuracy on a variety of language tasks. However, it was not shown whether the architecture achieves strong results for question answering when supporting facts are not marked during training or whether it could be applied to other modalities such as images. Based on an analysis of the DMN, we propose several improvements to its memory and input modules. Together with these changes we introduce a novel input module for images in order to be able to answer visual questions. Our new DMN+ model improves the state of the art on both the Visual Question Answering dataset and the bAbI-10k text question-answering dataset without supporting fact supervision.

People often try to find an image using a short query and images are usually indexed using short annotations. Matching the query vocabulary with the indexing vocabulary is a difficult problem when little text is available. Textual user generated content in Web 2.0 platforms contains a wealth of data that can help solve this problem. Here we describe how to use Wikipedia and Flickr content to improve this match. The initial query is launched in Flickr and we create a query model based on co-occurring terms. We also calculate nearby concepts using Wikipedia and use these to expand the query. The final results are obtained by ranking the results for the expanded query using the similarity between their annotation and the Flickr model. Evaluation of these expansion and ranking techniques, over the Image CLEF 2010 Wikipedia Collection containing 237,434 images and their multilingual textual annotations, shows that a consistent improvement compared to state of the art methods.

Obtaining effective mid-level representations has become an increasingly important task in computer vision. In this paper, we propose a fully automatic algorithm which harvests visual concepts from a large number of Internet images (more than a quarter of a million) using text-based queries. Existing approaches to visual concept learning from Internet images either rely on strong supervision with detailed manual annotations or learn image-level classifiers only. Here, we take the advantage of having massive well organized Google and Bing image data, visual concepts (around 14, 000) are automatically exploited from images using word-based queries. Using the learned visual concepts, we show state-of-the-art performances on a variety of benchmark datasets, which demonstrate the effectiveness of the learned mid-level representations: being able to generalize well to general natural images. Our method shows significant improvement over the competing systems in image classification, including those with strong supervision.

Many of the existing methods for learning joint embedding of images and text use only supervised information from paired images and its textual attributes. Taking advantage of the recent success of unsupervised learning in deep neural networks, we propose an end-to-end learning framework that is able to extract more robust multi-modal representations across domains. The proposed method combines representation learning models (i.e., auto-encoders) together with cross-domain learning criteria (i.e., Maximum Mean Discrepancy loss) to learn joint embeddings for semantic and visual features. A novel technique of unsupervised-data adaptation inference is introduced to construct more comprehensive embeddings for both labeled and unlabeled data. We evaluate our method on Animals with Attributes and Caltech-UCSD Birds 200-2011 dataset with a wide range of applications, including zero and few-shot image recognition and retrieval, from inductive to transductive settings. Empirically, we show that our frame-work improves over the current state of the art on many of the considered tasks.

What just happened in artificial intelligence and how it is being misunderstood.

Here we present DeepGaze II, a model that predicts where people look in images. The model uses the features from the VGG-19 deep neural network trained to identify objects in images. Contrary to other saliency models that use deep features, here we use the VGG features for saliency prediction with no additional fine-tuning (rather, a few readout layers are trained on top of the VGG features to predict saliency). The model is therefore a strong test of transfer learning. After conservative crossvalidation, DeepGaze II explains about 87% of the explainable information gain in the patterns of fixations and achieves top performance in area under the curve metrics on the MIT300 hold-out benchmark. These results corroborate the finding from DeepGaze I (which explained 56% of the explainable information gain), that deep features trained on object recognition provide a versatile feature space for performing related visual tasks. We explore the factors that contribute to this success and present several informative image examples. A web service is available to compute model predictions at http://deepgaze.bethgelab.org.

Analysis and detection of complex events in videos require a semantic representation of the video content. Existing video semantic representation methods typically require users to pre-define an exhaustive concept lexicon and manually annotate the presence of the concepts in each video, which is infeasible for real-world video event detection problems. In this paper, we propose an automatic semantic concept discovery scheme by exploiting Internet images and their associated tags. Given a target event and its textual descriptions, we crawl a collection of images and their associated tags by performing text based image search using the noun and verb pairs extracted from the event textual descriptions. The system first identifies the candidate concepts for an event by measuring whether a tag is a meaningful word and visually detectable. Then a concept visual model is built for each candidate concept using a SVM classifier with probabilistic output. Finally, the concept models are applied to generate concept based video representations. We use the TRECVID Multimedia Event Detection (MED) 2013 as our video test set and crawl 400K Flickr images to automatically discover 2, 000 visual concepts. We show significant performance gains of the proposed concept discovery method over different video event detection tasks including supervised event modeling over concept space and semantic based zero-shot retrieval without training examples. Importantly, we show the proposed method of automatic concept discovery outperforms other well-known concept library construction approaches such as Classemes and ImageNet by a large margin (228%) in zero-shot event retrieval. Finally, subjective evaluation by humans also confirms clear superiority of the proposed method in discovering concepts for event representation.


Question: Focusing on the descriptions produced by image captioning models.

This paper presents a novel technique for enhancing the generation of image descriptions through deep visual-semantic alignments. Unlike conventional approaches, our method focuses on creating a robust connection between the visual content of images and their semantic representations. By aligning visual features with corresponding semantic concepts, our model can generate more accurate and contextually relevant captions for images. This alignment process involves a deep learning framework that learns to associate specific visual patterns with semantic meanings, thereby improving the descriptive capabilities of image captioning models. The effectiveness of our technique is demonstrated through extensive experiments on benchmark datasets, where it outperforms state-of-the-art methods in terms of generating detailed and coherent image descriptions. By paying closer attention to the visual-semantic alignments, our method bridges the gap between visual understanding and language generation, leading to a significant enhancement in the quality of image captions produced by our model.

Over the last years, the robotics community has made substantial progress in detection and 3D pose estimation of known and unknown objects. However, the question of how to identify objects based on language descriptions has not been investigated in detail. While the computer vision community recently started to investigate the use of attributes for object recognition, these approaches do not consider the task settings typically observed in robotics, where a combination of appearance attributes and object names might be used in referral language to identify specific objects in a scene. In this paper, we introduce an approach for identifying objects based on natural language containing appearance and name attributes. To learn rich RGB-D features needed for attribute classification, we extend recently introduced sparse coding techniques so as to automatically learn attribute-dependent features. We introduce a large data set of attribute descriptions of objects in the RGB-D object dataset. Experiments on this data set demonstrate the strong performance of our approach to language based object identification. We also show that our attribute-dependent features provide significantly better generalization to previously unseen attribute values, thereby enabling more rapid learning of new attribute values.

We are creating multimedia contents everyday and everywhere. While automatic content generation has played a fundamental challenge to multimedia community for decades, recent advances of deep learning have made this problem feasible. For example, the Generative Adversarial Networks (GANs) is a rewarding approach to synthesize images. Nevertheless, it is not trivial when capitalizing on GANs to generate videos. The difficulty originates from the intrinsic structure where a video is a sequence of visually coherent and semantically dependent frames. This motivates us to explore semantic and temporal coherence in designing GANs to generate videos. In this paper, we present a novel Temporal GANs conditioning on Captions, namely TGANs-C, in which the input to the generator network is a concatenation of a latent noise vector and caption embedding, and then is transformed into a frame sequence with 3D spatio-temporal convolutions. Unlike the naive discriminator which only judges pairs as fake or real, our discriminator additionally notes whether the video matches the correct caption. In particular, the discriminator network consists of three discriminators: video discriminator classifying realistic videos from generated ones and optimizes video-caption matching, frame discriminator discriminating between real and fake frames and aligning frames with the conditioning caption, and motion discriminator emphasizing the philosophy that the adjacent frames in the generated videos should be smoothly connected as in real ones. We qualitatively demonstrate the capability of our TGANs-C to generate plausible videos conditioning on the given captions on two synthetic datasets (SBMG and TBMG) and one real-world dataset (MSVD). Moreover, quantitative experiments on MSVD are performed to validate our proposal via Generative Adversarial Metric and human study.

With the rapid proliferation of smartphones and tablet computers, search has moved beyond text to other modalities like images and voice. For many applications like Fashion, visual search offers a compelling interface that can capture stylistic visual elements beyond color and pattern that cannot be as easily described using text. However, extracting and matching such attributes remains an extremely challenging task due to high variability and deformability of clothing items. In this paper, we propose a fine-grained learning model and multimedia retrieval framework to address this problem. First, an attribute vocabulary is constructed using human annotations obtained on a novel finegrained clothing dataset. This vocabulary is then used to train a fine-grained visual recognition system for clothing styles. We report benchmark recognition and retrieval results on Women’s Fashion Coat Dataset and illustrate potential mobile applications for attribute-based multimedia retrieval of clothing items and image annotation.

As a fundamental task in the natural language processing field, language modeling aims to estimate the distribution of the word sequences. However, the most existing algorithms have focused on the main texts while often ignoring the vastly-accessible semantic attributes of the documents, e.g., titles, authors, sentiments and tags. To address this issue, we propose Semantic Attribute Modulated (SAM) language modeling, a novel language modeling framework that incorporates the various semantic attributes. Attributes are selected automatically with an attribute attention mechanism. We build three text datasets with a diversity of semantic attributes. On the three text datasets, we empirically examine the language model perplexities of several typical corpora, and then demonstrate the superiority of our model with the different combinations of the attributes. Extensive qualitative results, including word semantic analysis, attention values and an interesting lyric generation, further demonstrate the effectiveness of our SAMmethod.

Cross-view retrieval, which focuses on searching images as response to text queries or vice versa, has received increasing attention recently. Crossview hashing is to efficiently solve the cross-view retrieval problem with binary hash codes. Most existing works on cross-view hashing exploit multiview embedding method to tackle this problem, which inevitably causes the information loss in both image and text domains. Inspired by the Generative Adversarial Nets (GANs), this paper presents a new model that is able to Turn Crossview Hashing into single-view hashing (TUCH), thus enabling the information of image to be preserved as much as possible. TUCH is a novel deep architecture that integrates a language model network T for text feature extraction, a generator network G to generate fake images from text feature and a hashing network H for learning hashing functions to generate compact binary codes. Our architecture effectively unifies joint generative adversarial learning and cross-view hashing. Extensive empirical evidence shows that our TUCH approach achieves state-of-the-art results, especially on text to image retrieval, based on imagesentences datasets, i.e. standard IAPRTC-12 and large-scale Microsoft COCO.

Build accurate DNN models requires training on large labeled, context specific datasets, especially those matching the target scenario. We believe advances in wireless localization, working in unison with cameras, can produce automated annotation of targets on images and videos captured in the wild. Using pedestrian and vehicle detection as examples, we demonstrate the feasibility, benefits, and challenges of an automatic image annotation system. Our work calls for new technical development on passive localization, mobile data analytics, and error-resilient ML models, as well as design issues in user privacy policies.

We propose a framework of word-vector adaptation, which makes vectors of visually similar concepts close to each other. Here, word vectors are real-valued vector representation of words, e.g., word2vec representation. Our basic idea is to assume that each concept has some hypernyms that are important to determine its visual features. For example, for a concept Swallow with hypernyms Bird, Animal and Entity, we believe Bird is the most important since birds have common visual features with their feathers etc. Adapted word vectors are obtained for each word by taking a weighted sum of a given original word vector and its hypernym word vectors. Our weight optimization makes vectors of visually similar concepts close to each other, by giving a large weight for such important hypernyms. We apply the adapted word vectors to zero-shot learning on the TRECVID 2014 semantic indexing dataset. We achieved 0.083 of Mean Average Precision, which is the best performance without using TRECVID training data to the best of our knowledge.

This paper presents the results of a computational experiment designed to investigate the extent to which metrics of image fidelity can be improved through knowledge of where humans tend to fixate in images. Five common metrics of image fidelity were augmented using two sets of fixation data, one set obtained under task-free viewing conditions and another set obtained when viewers were asked to judge image quality. The augmented metrics were then compared to subjective ratings of the images. The results show that most metrics can be improved using eye fixation data, but a greater improvement was found using fixations obtained in the task-free condition (task-free viewing).

Given a pre-trained CNN without any testing samples, this paper proposes a simple yet effective method to diagnose feature representations of the CNN. We aim to discover representation flaws caused by potential dataset bias. More specifically, when the CNN is trained to estimate image attributes, we mine latent relationships between representations of different attributes inside the CNN. Then, we compare the mined attribute relationships with ground-truth attribute relationships to discover the CNN’s blind spots and failure modes due to dataset bias. In fact, representation flaws caused by dataset bias cannot be examined by conventional evaluation strategies based on testing images, because testing images may also have a similar bias. Experiments have demonstrated the effectiveness of our method.

• Neural network language model and distributed representation for words (Vector representation) • Capture syntactic and remantic regularities in language • Outperform state-of-the-art


Question: Technological advancements facilitating the Internet of Health Things.

In the burgeoning landscape of the Internet of Things (IoT) and wearable technology, compact sensor-laden devices are increasingly gathering data from their environment, processing it, and transmitting it to remote servers for analysis. While seemingly innocuous, these emerging technologies have sparked significant concerns regarding security and privacy. This paper delves into the potential risks and impacts associated with the compromise of such devices. By focusing on the design process of IoT and wearable devices, we highlight prevalent design methodologies and their implications for security and privacy. As case studies, we examine the Google Nest Thermostat and the Nike+ Fuelband to illustrate how current industry trends, which often treat security as an afterthought or an additional feature, can influence the final product and potentially jeopardize user security and privacy. Subsequently, we explore enhancements to the design flow that facilitate the seamless integration of security measures, markedly diverging from conventional practices.

This exploration not only underscores the critical need for integrating robust security protocols from the outset of device development but also emphasizes the importance of considering privacy implications throughout the design process. By adopting a proactive approach to security and privacy, manufacturers can ensure that IoT and wearable devices are not only innovative and functional but also safe and respectful of user privacy. This shift in mindset is essential as these devices become increasingly ubiquitous in our daily lives, collecting sensitive personal data that could be vulnerable to exploitation if not adequately protected.

The current Internet of Things (IoT) solutions require support at different network layers, from higher level applications to lower level media-based support. This paper presents some of the main application requirements for IoT, characterizing architecture, Quality of Service (QoS) features, security mechanisms, discovery service resources and web integration options and the protocols that can be used to provide them (e.g. CoAP, XMPP, DDS, MQTT-SN, AMQP). As examples of lower-level requirements and protocols, several wireless network characteristics (e.g. ZigBee, Z-Wave, BLE, LoRaWAN, SigFox, IEEE 802.11af, NB-IoT) are presented. The variety of possible applications scenarios and the heterogeneity of enabling technologies combined with a large number of sensors and devices, suggests the need for simulation and modeling tactics to describe how the previous requirements can be met. As a potential solution, the creation of simulation models and the usage of the OMNET++ simulation tool to enable meaningful IoT simulation is discussed. The analysis of the behavior of IoT applications is proposed for two use cases: Wireless Sensor Networks (WSN) for home and industrial automation, and Low Power Wide Area (LPWA) networks for smart meters, smart buildings, and smart cities.

Healthcare professionals spend much of their time wandering between patients and offices, while the supportive technology stays stationary. Therefore, mobile applications has adapted for healthcare industry. In spite of the advancement and variety of available mobile based applications, there is an eminent need to investigate the current position of the acceptance of those mobile health applications that are tailored towards the tracking patients condition, share patients information and access. Consequently, in this study Technology Acceptance Model has designed to investigate the user acceptance of mobile technology application within healthcare industry. The purpose of this study is to design a quantitative approach based on the technology acceptance model questionnaire as its primary research methodology. It utilized a quantitative approach based a Technology Acceptance Model (TAM) to evaluate the system mobile tracking Model. The related constructs for evaluation are: Perceived of Usefulness, Perceived Ease of Use, User Satisfaction and Attribute of Usability. All these constructs are modified to suit the context of the study. Moreover, this study outlines the details of each construct and its relevance toward the research issue. The outcome of the study represents series of approaches that will apply for checking the suitability of a mobile tracking on patient progress application for health care industry and how well it achieves the aims and objectives of the design.

The imminent arrival of the Internet of Things (IoT), which consists of a vast number of devices with heterogeneous characteristics, means that future networks need a new architecture to accommodate the expected increase in data generation. Software defined networking (SDN) and network virtualization (NV) are two technologies that promise to cost-effectively provide the scale and versatility necessary for IoT services. In this paper, we survey the state of the art on the application of SDN and NV to IoT. To the best of our knowledge, we are the first to provide a comprehensive description of every possible IoT implementation aspect for the two technologies. We start by outlining the ways of combining SDN and NV. Subsequently, we present how the two technologies can be used in the mobile and cellular context, with emphasis on forthcoming 5G networks. Afterward, we move to the study of wireless sensor networks, arguably the current foremost example of an IoT network. Finally, we review some general SDN-NV-enabled IoT architectures, along with real-life deployments and use-cases. We conclude by giving directions for future research on this topic.

This paper presents challenges and issues in smart buildings and the Internet of Things (IoT), which we identified in years of research in real buildings. To tackle these challenges, a decentralized service-oriented architecture based on a message-oriented middleware has been implemented for the domain of smart buildings. It uses a network-transparent IoT message bus and provides the means for composing applications from auxiliary services, which facilitate device abstraction, protocol adaption, modularity, and maintainability. We demonstrate the flexibility of our architecture by describing how three distinct applications---privacy-enhancing energy data visualization, automated building energy management, and a generic user interface---can be integrated and operated simultaneously in our real smart building laboratory. We compare the advantages of our architecture to conventional ones and provide a best-practice solution for the Intranet of Things and Energy in smart buildings.

The Internet of Things (IoT) is an impending technology that captured the industries and research interests in a brief period. Presently, the number of Internet-connected devices are estimated at billions and by 2020 many studies expect to have around 50 billion devices that are connected to the Internet. The IoT devices have produced a significant amount of data that could be threatening to the security and directing the flow of data in the network. One of the alternative solutions for problems caused by IoT is to have some kind of solutions that are based on programming and primary control. Software Define Network (SDN) offers a programmable and primary control for the basic network without variability of the current architecture of the network. This paper discusses some solutions for various domains for the combined SDN based on IoT and current directions in studies. The relative analysis of these solutions prepares a simple and brief view of those directions.

Internet of things capability and alliance: entrepreneurial orientation, market orientation and product and process innovation Xiaoyu Yu Bang Nguyen Yi Chen Article information: To cite this document: Xiaoyu Yu Bang Nguyen Yi Chen , (2016),"Internet of things capability and alliance: entrepreneurial orientation, market orientation and product and process innovation", Internet Research, Vol. 26 Iss 2 pp. Permanent link to this document: http://dx.doi.org/10.1108/IntR-10-2014-0265

Internet of Things (IoT) are being adopted for industrial and manufacturing applications such as manufacturing automation, remote machine diagnostics, prognostic health management of industrial machines and supply chain management. CloudBased Manufacturing is a recent on-demand model of manufacturing that is leveraging IoT technologies. While Cloud-Based Manufacturing enables on-demand access to manufacturing resources, a trusted intermediary is required for transactions between the users who wish to avail manufacturing services. We present a decentralized, peer-to-peer platform called BPIIoT for Industrial Internet of Things based on the Block chain technology. With the use of Blockchain technology, the BPIIoT platform enables peers in a decentralized, trustless, peer-to-peer network to interact with each other without the need for a trusted intermediary.

Sensor networks have the potential to greatly impact many aspects of medical care. By outfitting patients with wireless, wearable vital sign sensors, collecting detailed real-time data on physiological status can be greatly simplified. However, there is a significant gap between existing sensor network systems and the needs of medical care. In particular, medical sensor networks must support multicast routing topologies, node mobility, a wide range of data rates and high degrees of reliability, and security. This paper describes our experiences with developing a combined hardware and software platform for medical sensor networks, called CodeBlue. CodeBlue provides protocols for device discovery and publish/subscribe multihop routing, as well as a simple query interface that is tailored for medical monitoring. We have developed several medical sensors based on the popular MicaZ and Telos mote designs, including a pulse oximeter, EKG and motion-activity sensor. We also describe a new, miniaturized sensor mote designed for medical use. We present initial results for the CodeBlue prototype demonstrating the integration of our medical sensors with the publish/subscribe routing substrate. We have experimentally validated the prototype on our 30-node sensor network testbed, demonstrating its scalability and robustness as the number of simultaneous queries, data rates, and transmitting sensors are varied. We also study the effect of node mobility, fairness across multiple simultaneous paths, and patterns of packet loss, confirming the system’s ability to maintain stable routes despite variations in node location and

What can you glean from using inexpensive, off-the-shelf parts to create Internet of Things (IoT) use cases? As it turns out, a lot. The fast productization of IoT technologies is leaving users vulnerable to security and privacy risks.

The Internet of Things (IoT) has connected an incredible diversity of devices in novel ways, which has enabled exciting new services and opportunities. Unfortunately, IoT systems also present several important challenges to developers. This paper proposes a vision for how we may build IoT systems in the future by reconceiving IoT's fundamental unit of construction not as a "thing", but rather as a widely and finely distributed "microservice" already familiar to web service engineering circles. Since IoT systems are quite different from more established uses of microservice architectures, success of the approach depends on adaptations that enable them to met the key challenges that IoT systems present. We argue that a microservice approach to building IoT systems can combine in a mutually enforcing way with patterns for microservices, API gateways, distribution of services, uniform service discovery, containers, and access control. The approach is illustrated using two case studies of IoT systems in personal health management and connected autonomous vehicles. Our hope is that the vision of a microservices approach will help focus research that can fill in current gaps preventing more effective, interoperable, and secure IoT services and solutions in a wide variety of contexts.


Question: Innovative technologies facilitating the Internet of Health Things ecosystem.

In the realm of Internet of Things (IoT) and smart city frameworks, long-range communication technologies operating in unlicensed bands are emerging as pivotal components. Traditionally, IoT connectivity has relied on two primary methods: multihop mesh networks utilizing short-range communication in the unlicensed spectrum, and long-range cellular technologies like 2G/GSM/GPRS in licensed frequency bands. However, these conventional models are being redefined by the advent of low-rate, long-range transmission technologies in the unlicensed sub-gigahertz bands. These technologies, designed to create star-topology access networks known as low-power wide-area networks (LPWANs), are reshaping the IoT landscape. This paper explores this innovative approach to IoT connectivity, highlighting its superior efficiency, effectiveness, and architectural design benefits over established methods, particularly in the context of smart city applications. The focus on unlicensed bands not only reduces deployment costs but also enhances the scalability and robustness of IoT networks, making them more suitable for the diverse demands of smart city environments.

This article discusses the challenges for Database Management in the Internet of Things. We provide scenarios to illustrate the new world that will be produced by the Internet of Things, where physical objects are fully integrated into the information highway. We discuss the different types of data that will be part of the Internet of Things. These include identification, positional, environmental, historical, and descriptive data. We consider the challenges brought by the need to manage vast quantities of data across heterogeneous systems. In particular, we consider the areas of querying, indexing, process modeling, transaction handling, and integration of heterogeneous systems. We refer to the earlier work that might provide solutions for these challenges. Finally we discuss a road map for the Internet of Things and respective technical priorities.

The Internet of Things (IoT) conceives a world where everyday objects are able to join the Internet and exchange data as well as process, store, collect them from the surrounding environment, and actively intervene on it. An unprecedented number of services may be envisioned by exploiting the Internet of Things. Fog Computing, which is also known as Edge Computing, was proposed in 2012 as the ideal paradigm to support the resource-constrained IoT devices in data processing and information delivery. Indeed, the Fog, which does not replace the centralized Cloud but cooperates with it, distributes Cloud Computing technologies and principles anywhere along the Cloud-to-Things continuum and particularly at the network edge. The Fog proximity to the IoT devices allows for several advantages that must be continuously guaranteed, also when end devices move from one place to another. In this paper, we aim at examining in depth what it means to provide mobility support in a Fog environment and at investigating what are the main challenges to be faced. Besides, in order to highlight the importance of this topic in everyday life, we provide the reader with three scenarios where there is an integration between the IoT and Fog Computing, and in which mobility support is essential. We finally point out the main research directions in the field.

Excellence in IT is both a driver and a key enabler of the digital transformation. The digital transformation changes the way we live, work, learn, communicate, and collaborate. The Internet of Things (IoT) fundamentally influences today's digital strategies with disruptive business operating models and fast changing markets. New business information systems are integrating emerging Internet of Things infrastructures and components. With the huge diversity of Internet of Things technologies and products organizations have to leverage and extend previous Enterprise Architecture efforts to enable business value by integrating Internet of Things architectures. Both architecture engineering and management of current information systems and business models are complex and currently integrating beside the Internet of Things synergistic subjects, like Enterprise Architecture in context with services & cloud computing, semantic-based decision support through ontologies and knowledge-based systems, big data management, as well as mobility and collaboration networks. To provide adequate decision support for complex business/IT environments, we have to make transparent the impact of business and IT changes over the integral landscape of affected architectural capabilities, like directly and transitively impacted IoT-objects, business categories, processes, applications, services, platforms and infrastructures. The paper describes a new metamodel-based approach for integrating Internet of Things architectural objects, which are semi-automatically federated into a holistic Digital Enterprise Architecture environment.

This paper discusses novel communication network topologies and components and describes an evolutionary path of bringing Ethernet into automotive applications with focus on electric mobility. For next generation in-vehicle networking, the automotive industry identified Ethernet as a promising candidate besides CAN and FlexRay. Ethernet is an IEEE standard and is broadly used in consumer and industry domains. It will bring a number of changes for the design and management of in-vehicle networks and provides significant re-use of components, software, and tools. Ethernet is intended to connect inside the vehicle high-speed communication requiring sub-systems like Advanced Driver Assistant Systems (ADAS), navigation and positioning, multimedia, and connectivity systems. For hybrid (HEVs) or electric vehicles (EVs), Ethernet will be a powerful part of the communication architecture layer that enables the link between the vehicle electronics and the Internet where the vehicle is a part of a typical Internet of Things (IoT) application. Using Ethernet for vehicle connectivity will effectively manage the huge amount of data to be transferred between the outside world and the vehicle through vehicle-to-x (V2V and V2I or V2I+I) communication systems and cloud-based services for advanced energy management solutions. Ethernet is an enabling technology for introducing advanced features into the automotive domain and needs further optimizations in terms of scalability, cost, power, and electrical robustness in order to be adopted and widely used by the industry.

The exponentially increasing number of internet of things (IoT) devices and the data generated by these devices introduces the spectrum crisis at the already crowded ISM 2.4 GHz band. To address this issue and enable more flexible and concurrent communications among IoT devices, we propose B2W2, a novel communication framework that enables N-way concurrent communication among WiFi and Bluetooth Low Energy (BLE) devices. Specifically, we demonstrate that it is possible to enable the BLE to WiFi cross-technology communication while supporting the concurrent BLE to BLE and WiFi to WiFi communications. We conducted extensive experiments under different real-world settings and results show that its throughput is more than 85X times higher than the most recently reported cross-technology communication system [22], which only supports one-way communication (i.e., broadcasting) at any specific time.

Wireless communication networks are highly prone to security threats. The major applications of wireless communication networks are in military, business, healthcare, retail, and transportations. These systems use wired, cellular, or adhoc networks. Wireless sensor networks, actuator networks, and vehicular networks have received a great attention in society and industry. In recent years, the Internet of Things (IoT) has received considerable research attention. The IoT is considered as future of the internet. In future, IoT will play a vital role and will change our living styles, standards, as well as business models. The usage of IoT in different applications is expected to rise rapidly in the coming years. The IoT allows billions of devices, peoples, and services to connect with others and exchange information. Due to the increased usage of IoT devices, the IoT networks are prone to various security attacks. The deployment of efficient security and privacy protocols in IoT networks is extremely needed to ensure confidentiality, authentication, access control, and integrity, among others. In this paper, an extensive comprehensive study on security and privacy issues in IoT networks is provided. Keywords—Internet of Things (IoT); security issues in IoT; security; privacy

The world population is rapidly aging and becoming a burden to health systems around the world. In this work we present a conceptual framework to encourage the research community to develop more comprehensive and adaptive ICT solutions for prevention and rehabilitation of chronic conditions in the daily life of the aging population and beyond health facilities. We first present an overview of current international standards in human functioning and disability, and how chronic conditions are interconnected in older age. We then describe innovative mobile and sensor technologies, predictive data analysis in healthcare, and game-based prevention and rehabilitation techniques. We then set forth a multidisciplinary approach for the personalized prevention and rehabilitation of chronic conditions using unobtrusive and pervasive sensors, interactive activities, and predictive analytics, which also eases the tasks of health-related researchers, caregivers and providers. Our proposal represents a conceptual basis for future research, in which much remains to be done in terms of standardization of technologies and health terminology, as well as data protection and privacy legislation.

Advances in semiconductor technology have resulted in the creation of miniature medical embedded systems that can wirelessly monitor the vital signs of patients. These lightweight medical systems can aid providers in large disasters who become overwhelmed with the large number of patients, limited resources, and insufficient information. In a mass casualty incident, small embedded medical systems facilitate patient care, resource allocation, and real-time communication in the advanced health and disaster aid network (AID-N). We present the design of electronic triage tags on lightweight, embedded systems with limited memory and computational power. These electronic triage tags use noninvasive, biomedical sensors (pulse oximeter, electrocardiogram, and blood pressure cuff) to continuously monitor the vital signs of a patient and deliver pertinent information to first responders. This electronic triage system facilitates the seamless collection and dissemination of data from the incident site to key members of the distributed emergency response community. The real-time collection of data through a mesh network in a mass casualty drill was shown to approximately triple the number of times patients that were triaged compared with the traditional paper triage system.

Wireless body area network has gained much interest and became emerging technology at health service facilities due to its wide range of utility and vital role to improve the human health. In this research paper, we are conducting a comprehensive survey of wireless Body Area Network (WBAN) and also introducing a virtual doctor server (VDS) in existing WBAN architecture. Existing architecture of WBAN consists of: wireless sensor, wireless actuator node , wireless central unit and wireless Personal Device (PD). Personal Digital Assistant (PDA) or smart phone can be used as PD. Based on the existing architecture mentioned above, we propose a design concept for a virtual doctor server (VDS) to support various patient health care services. VDS will keep the historical data about the patient, generate the daily tips and advices for him, call the doctor or emergency squad if required and can provide first aid assistance instructions on patient or any of his close relative's PDA's.

The Internet of Things (IoT) could be described as the pervasive and global network which aids and provides a system for the monitoring and control of the physical world through the collection, processing and analysis of generated data by IoT sensor devices. It is projected that by 2020 the number of connected devices is estimated to grow exponentially to 50 billion. The main drivers for this growth are our everyday devices such as cars, refrigerators, fans, lights, mobile phones and other operational technologies including the manufacturing infrastructures which are now becoming connected systems across the world. It is apparent that security will pose a fundamental enabling factor for the successful deployment and use of most IoT applications and in particular secure routing among IoT sensor nodes thus, mechanisms need to be designed to provide secure routing communications for devices enabled by the IoT technology. This survey analyzes existing routing protocols and mechanisms to secure routing communications in IoT, as well as the open research issues. We further analyze how existing approaches ensure secure routing in IoT, their weaknesses, threats to secure routing in IoT and the open challenges and strategies for future research work for a better secure IoT routing.


Question: Technological innovations facilitating the Internet of Health Things ecosystem.

This document introduces the Identity Authentication and Capability Based Access Control (IACAC) framework, designed specifically for the Internet of Things (IoT) ecosystem. With the rapid expansion of IoT applications across various sectors, ensuring secure communication between devices through robust authentication and access control mechanisms has become paramount. The inherent characteristics of IoT networks, such as device mobility, dynamic topology, and the physical insecurity of low-power devices, pose significant security challenges. The IACAC model is devised to address these vulnerabilities by offering a lightweight yet resilient solution suitable for resource-limited and distributed IoT environments.

A key feature of IACAC is its integration of identity authentication and capability-based access control, which collectively fortify IoT devices against man-in-the-middle, replay, and denial of service (DoS) attacks. By introducing the concept of capability for access control, the model ensures that only authorized entities can interact with IoT devices, thereby safeguarding the network from unauthorized access and malicious activities.

The paper conducts a comprehensive evaluation of the IACAC protocol, including a performance analysis that compares computational time with existing solutions. The results of this analysis, alongside a review of related studies, serve to validate the effectiveness and security of the IACAC model. Additionally, the protocol's security is assessed using a verification tool, with outcomes confirming its resilience against the aforementioned attacks.

Moreover, the document delves into the broader challenges faced by IoT networks and models security threats through practical use cases. This provides a realistic perspective on the potential risks and the necessity of deploying robust security measures like IACAC to ensure the integrity and reliability of IoT systems.

The Internet of Things (IoT) opens opportunities for wearable devices, home appliances, and software to share and communicate information on the Internet. Given that the shared data contains a large amount of private information, preserving information security on the shared data is an important issue that cannot be neglected. In this paper, we begin with general information security background of IoT and continue on with information security related challenges that IoT will encountered. Finally, we will also point out research directions that could be the future work for the solutions to the security challenges that IoT encounters.

The Internet of Things envisions integration, coordination, communication, and collaboration of real-world objects in order to perform daily tasks in a more intelligent and efficient manner. To comprehend this vision, this article studies the design of a large-scale IoT system for smart grid application, which constitutes a large number of home users and has the requirement of fast response time. In particular, we focus on the messaging protocol of a universal IoT home gateway, where our cloud enabled system consists of a back-end server, a unified home gateway (UHG) at the end users, and a user interface for mobile devices. We discuss the features of such an IoT system to support a large-scale deployment with a UHG and real-time residential smart grid applications. Based on the requirements, we design an IoT system using XMPP and implemented in a testbed for energy management applications. To show the effectiveness of the designed testbed, we present some results using the proposed IoT architecture.

Patient tracking is an important component toward creating smart cities. In this demo we use Bluetooth Low Energy beacons and single board computers to track patients in the emerging field of smart care facilities. Our model utilizes a fixed scanner moving transmitter method for wireless tracking of patients through the facility. The data collected by all scanners is stored within a central database that is designed to be efficiently queried. We show how inexpensive components in conjunction with free open source software can be used to implement a patient tracking system. We focus on the pipeline between acquisition and display of the location data. Additionally, we also discuss the manipulation of the data required for usability, and optional filtering operations that improve accuracy.

In the Web of Things, standard web technologies and protocols are used to represent and communicate with physical and virtual things. One challenge toward this vision is integrating things with different characteristics, protocols, interfaces and constraints while maintaining the simplicity and flexibility required for a variety of applications. In this paper we present the Thing Broker, a core platform for Web of Things that provides RESTFul interfaces to things using a Twitter-based set of abstractions and communication model. We present the key abstractions, a reference implementation and explain how a typical WoT application can be created using Thing Broker. We finish with a preliminary evaluation and draw some lessons from our experiences.

The Internet of Things (IoT) being a promising technology of the future is expected to connect billions of devices. The increased number of communication is expected to generate mountains of data and the security of data can be a threat. The devices in the architecture are essentially smaller in size and low powered. Conventional encryption algorithms are generally computationally expensive due to their complexity and requires many rounds to encrypt, essentially wasting the constrained energy of the gadgets. Less complex algorithm, however, may compromise the desired integrity. In this paper we propose a lightweight encryption algorithm named as Secure IoT (SIT). It is a 64-bit block cipher and requires 64-bit key to encrypt the data. The architecture of the algorithm is a mixture of feistel and a uniform substitution-permutation network. Simulations result shows the algorithm provides substantial security in just five encryption rounds. The hardware implementation of the algorithm is done on a low cost 8-bit micro-controller and the results of code size, memory utilization and encryption/decryption execution cycles are compared with benchmark encryption algorithms. The MATLAB code for relevant simulations is available online at https://goo.gl/Uw7E0W. Keywords—IoT; Security; Encryption; Wireless Sensor Network WSN; Khazad

This paper puts the Internet of Things in a wider context: How it relates to the Future Internet overall, and where the business value lies so that it will become interesting for enterprises to invest in it. Real-World Awareness and Business Process Decomposition are the two major paradigms regarding future business value. The major application domains where the Internet of Things will play an important role and where there are concrete business opportunities are highlighted. But there are also many technical challenges that need to be addressed. These are listed and it is shown how they are tackled by existing research projects with industrial participation.

Advances in sensing and computing capabilities are making it possible to embed increasing computing power in small devices. This has enabled the sensing devices not just to passively capture data at very high resolution but also to take sophisticated actions in response. Combined with advances in communication, this is resulting in an ecosystem of highly interconnected devices referred to as the Internet of Things IoT. In conjunction, the advances in machine learning have allowed building models on this ever increasing amounts of data. Consequently, devices all the way from heavy assets such as aircraft engines to wearables such as health monitors can all now not only generate massive amounts of data but can draw back on aggregate analytics to “improve” their performance over time. Big data analytics has been identified as a key enabler for the IoT. In this chapter, we discuss various avenues of the IoT where big data analytics either is already making a significant impact or is on the cusp of doing so. We also discuss social implications and areas of concern.

The vision of the Internet of Things (IoT) relies on the provisioning of real-world services, which are provided by smart objects that are directly related to the physical world. A structured, machine-processible approach to provision such real-world services is needed to make heterogeneous physical objects accessible on a large scale and to integrate them with the digital world. The incorporation of observation and measurement data obtained from the physical objects with the Web data, using information processing and knowledge engineering methods, enables the construction of ”intelligent and interconnected things”. The current research mostly focuses on the communication and networking aspects between the devices that are used for sensing amd measurement of the real world objects. There is, however, relatively less effort concentrated on creating dynamic infrastructures to support integration of the data into the Web and provide unified access to such data on service and application levels. This paper presents a semantic modelling and linked data approach to create an information framework for IoT. The paper describes a platform to publish instances of the IoT related resources and entities and to link them to existing resources on the Web. The developed platform supports publication of extensible and interoperable descriptions in the form of linked data.

The Internet of Things (IoT) could be described as the pervasive and global network which aids and provides a system for the monitoring and control of the physical world through the collection, processing and analysis of generated data by IoT sensor devices. It is projected that by 2020 the number of connected devices is estimated to grow exponentially to 50 billion. The main drivers for this growth are our everyday devices such as cars, refrigerators, fans, lights, mobile phones and other operational technologies including the manufacturing infrastructures which are now becoming connected systems across the world. It is apparent that security will pose a fundamental enabling factor for the successful deployment and use of most IoT applications and in particular secure routing among IoT sensor nodes thus, mechanisms need to be designed to provide secure routing communications for devices enabled by the IoT technology. This survey analyzes existing routing protocols and mechanisms to secure routing communications in IoT, as well as the open research issues. We further analyze how existing approaches ensure secure routing in IoT, their weaknesses, threats to secure routing in IoT and the open challenges and strategies for future research work for a better secure IoT routing.

Cloud computing is ever stronger converging with the Internet of Things (IoT) offering novel techniques for IoT infrastructure virtualization and its management on the cloud. However, system designers and operations managers face numerous challenges to realize IoT cloud systems in practice, mainly due to the complexity involved with provisioning large-scale IoT cloud systems and diversity of their requirements in terms of IoT resources consumption, customization of IoT capabilities and runtime governance. In this paper, we introduce the concept of software-defined IoT units - a novel approach to IoT cloud computing that encapsulates fine-grained IoT resources and IoT capabilities in well-defined APIs in order to provide a unified view on accessing, configuring and operating IoT cloud systems. Our software-defined IoT units are the fundamental building blocks of software-defined IoT cloud systems. We present our framework for dynamic, on-demand provisioning and deploying such software-defined IoT cloud systems. By automating provisioning processes and supporting managed configuration models, our framework simplifies provisioning and enables flexible runtime customizations of software-defined IoT cloud systems. We demonstrate its advantages on a real-world IoT cloud system for managing electric fleet vehicles.