Question: An extensive overview of recommendation methods based on neighborhood approaches.

This paper introduces ItemRank, a novel scoring algorithm for recommender systems that leverages a random-walk approach to rank items based on expected user preferences. Recommender systems play a critical role in assisting consumers in discovering products of interest by analyzing past user interactions. ItemRank stands out by employing a random-walk methodology to generate personalized suggestions, focusing on recommending top-ranked items to users who are likely to be interested. To validate the effectiveness of ItemRank, we conducted experiments using the widely recognized MovieLens data set, a comprehensive collection of user interactions with movies that serves as a benchmark for evaluating recommender system algorithms (as seen in works such as [Fouss et al., 2005; Sarwar et al., 2002]). We compared ItemRank against leading ranking techniques, specifically those outlined in [Fouss et al., 2005], and our findings revealed that ItemRank outperforms these algorithms in terms of recommendation accuracy. Moreover, ItemRank boasts lower computational complexity and memory usage compared to other advanced algorithms, making it a more efficient choice for practical implementation in recommender systems.

The popularity of online recommender systems has soared; they are deployed in numerous websites and gather tremendous amounts of user data that are necessary for the recommendation purposes. This data, however, may pose a severe threat to user privacy, if accessed by untrusted parties or used inappropriately. Hence, it is of paramount importance for recommender system designers and service providers to find a sweet spot, which allows them to generate accurate recommendations and guarantee the privacy of their users. In this chapter we overview the state of the art in privacy enhanced recommendations. We analyze the risks to user privacy imposed by recommender systems, survey the existing solutions, and discuss the privacy implications for the users of the recommenders. We conclude that a considerable effort is still required to develop practical recommendation solutions that provide adequate privacy guarantees, while at the same time facilitating the delivery of high-quality recommendations to their users. Arik Friedman NICTA, Australia, e-mail: arik.friedman@nicta.com.au Bart Knijnenburg UC Irvine, USA, e-mail: bart.k@uci.edu Kris Vanhecke iMinds Ghent University, Belgium, e-mail: kris.vanhecke@intec.ugent.be Luc Martens iMinds Ghent University, Belgium, e-mail: luc.martens@intec.ugent.be Shlomo Berkovsky CSIRO, Australia, e-mail: shlomo.berkovsky@csiro.au 1 Accepted to be included in the 2nd edition of the Recommender Systems handbook. Please do not share publicly, and consult the authors before citing this chapter.

In this paper, we propose a recommendation technique that not only can recommend items of interest to the user as traditional recommendation systems do but also specific aspects of consumption of the items to further enhance the user experience with those items. For example, it can recommend the user to go to a specific restaurant (item) and also order some specific foods there, e.g., seafood (an aspect of consumption). Our method is called Sentiment Utility Logistic Model (SULM). As its name suggests, SULM uses sentiment analysis of user reviews. It first predicts the sentiment that the user may have about the item based on what he/she might express about the aspects of the item and then identifies the most valuable aspects of the user's potential experience with that item. Furthermore, the method can recommend items together with those most important aspects over which the user has control and can potentially select them, such as the time to go to a restaurant, e.g. lunch vs. dinner, and what to order there, e.g., seafood. We tested the proposed method on three applications (restaurant, hotel, and beauty & spa) and experimentally showed that those users who followed our recommendations of the most valuable aspects while consuming the items, had better experiences, as defined by the overall rating.

Along with the rapid expansion of digital music formats, managing and searching for songs has become significant. Though music information retrieval (MIR) techniques have been made successfully in last ten years, the development of music recommender systems is still at a very early stage. Therefore, this paper surveys a general framework and state-of-art approaches in recommending music. Two popular algorithms: collaborative filtering (CF) and content-based model (CBM), have been found to perform well. Due to the relatively poor experience in finding songs in long tail and the powerful emotional meanings in music, two user-centric approaches: context-based model and emotion-based model, have been paid increasing attention. In this paper, three key components in music recommender user modelling, item profiling, and match algorithms are discussed. Six recommendation models and four potential issues towards user experience, are explained. However, subjective music recommendation system has not been fully investigated. To this end, we propose a motivation-based model using the empirical studies of human behaviour, sports education, music psychology.

When a Web application with a built-in recommender offers a social networking component which enables its users to form a trust network, it can generate more personalized recommendations by combining user ratings with information from the trust network. These are the so-called trust-enhanced recommendation systems. While research on the incorporation of trust for recommendations is thriving, the potential of explicitly stated distrust remains almost unexplored. In this article, we introduce a distrust-enhanced recommendation algorithm which has its roots in Golbeck's trust-based weighted mean. Through experiments on a set of reviews from Epinions.com, we show that our new algorithm outperforms its standard trust-only counterpart with respect to accuracy, thereby demonstrating the positive effect that explicit distrust can have on trust-based recommendations.

Traditional recommender systems assume that all the users are independent, and they usually face the cold start and data sparse problems. To alleviate these problems, social recommender systems use social relations as an additional input to improve recommendation accuracy. Social recommendation follows the intuition that people with social relationships share some kinds of preference towards items. Current social recommendation methods commonly apply the Matrix Factorization (MF) model to incorporate social information into the recommendation process. As an alternative model to MF, we propose a novel social recommendation approach based on Euclidean Embedding (SREE) in this paper. The idea is to embed users and items in a unified Euclidean space, where users are close to both their desired items and social friends. Experimental results conducted on two real-world data sets illustrate that our proposed approach outperforms the state-of-the-art methods in terms of recommendation accuracy.

With the increasing popularity of location-based services, such as tour guide and location-based social network, we now have accumulated many location data on the Web. In this paper, we show that, by using the location data based on GPS and users' comments at various locations, we can discover interesting locations and possible activities that can be performed there for recommendations. Our research is highlighted in the following location-related queries in our daily life: 1) if we want to do something such as sightseeing or food-hunting in a large city such as Beijing, where should we go? 2) If we have already visited some places such as the Bird's Nest building in Beijing's Olympic park, what else can we do there? By using our system, for the first question, we can recommend her to visit a list of interesting locations such as Tiananmen Square, Bird's Nest, etc. For the second question, if the user visits Bird's Nest, we can recommend her to not only do sightseeing but also to experience its outdoor exercise facilities or try some nice food nearby. To achieve this goal, we first model the users' location and activity histories that we take as input. We then mine knowledge, such as the location features and activity-activity correlations from the geographical databases and the Web, to gather additional inputs. Finally, we apply a collective matrix factorization method to mine interesting locations and activities, and use them to recommend to the users where they can visit if they want to perform some specific activities and what they can do if they visit some specific places. We empirically evaluated our system using a large GPS dataset collected by 162 users over a period of 2.5 years in the real-world. We extensively evaluated our system and showed that our system can outperform several state-of-the-art baselines.

Providing model-generated explanations in recommender systems is important to user experience. State-of-the-art recommendation algorithms — especially the collaborative filtering (CF)-based approaches with shallow or deep models — usually work with various unstructured information sources for recommendation, such as textual reviews, visual images, and various implicit or explicit feedbacks. Though structured knowledge bases were considered in content-based approaches, they have been largely ignored recently due to the availability of vast amounts of data and the learning power of many complex models. However, structured knowledge bases exhibit unique advantages in personalized recommendation systems. When the explicit knowledge about users and items is considered for recommendation, the system could provide highly customized recommendations based on users’ historical behaviors and the knowledge is helpful for providing informed explanations regarding the recommended items. A great challenge for using knowledge bases for recommendation is how to integrate large-scale structured and unstructured data, while taking advantage of collaborative filtering for highly accurate performance. Recent achievements in knowledge-base embedding (KBE) sheds light on this problem, which makes it possible to learn user and item representations while preserving the structure of their relationship with external knowledge for explanation. In this work, we propose to explain knowledge-base embeddings for explainable recommendation. Specifically, we propose a knowledge-base representation learning framework to embed heterogeneous entities for recommendation, and based on the embedded knowledge base, a soft matching algorithm is proposed to generate personalized explanations for the recommended items. Experimental results on real-world e-commerce datasets verified the superior recommendation performance and the explainability power of our approach compared with state-of-the-art baselines.

Most recommender systems recommend a list of items. The user examines the list, from the first item to the last, and often chooses the first attractive item and does not examine the rest. This type of user behavior can be modeled by the cascade model. In this work, we study cascading bandits, an online learning variant of the cascade model where the goal is to recommend K most attractive items from a large set of L candidate items. We propose two algorithms for solving this problem, which are based on the idea of linear generalization. The key idea in our solutions is that we learn a predictor of the attraction probabilities of items from their features, as opposing to learning the attraction probability of each item independently as in the existing work. This results in practical learning algorithms whose regret does not depend on the number of items L. We bound the regret of one algorithm and comprehensively evaluate the other on a range of recommendation problems. The algorithm performs well and outperforms all baselines.

Representing relationships as translations in vector space lives at the heart of many neural embedding models such as word embeddings and knowledge graph embeddings. In this work, we study the connections of this translational principle with collaborative ltering algorithms. We propose Translational Recommender Networks (TransRec), a new aentive neural architecture that utilizes the translational principle to model the relationships between user and item pairs. Our model employs a neural aention mechanism over a Latent Relational Aentive Memory (LRAM) module to learn the latent relations between user-item pairs that best explains the interaction. By exploiting adaptive user-item specic translations in vector space, our model also alleviates the geometric inexibility problem of other metric learning algorithms while enabling greater modeling capability and ne-grained ing of users and items in vector space. e proposed architecture not only demonstrates the state-of-the-art performance across multiple recommendation benchmarks but also boasts of improved interpretability. alitative studies over the LRAM module shows evidence that our proposed model is able to infer and encode explicit sentiment, temporal and aribute information despite being only trained on implicit feedback. As such, this ascertains the ability of TransRec to uncover hidden relational structure within implicit datasets.

I present a hybrid matrix factorisation model representing users and items as linear combinations of their content features’ latent factors. The model outperforms both collaborative and content-based models in cold-start or sparse interaction data scenarios (using both user and item metadata), and performs at least as well as a pure collaborative matrix factorisation model where interaction data is abundant. Additionally, feature embeddings produced by the model encode semantic information in a way reminiscent of word embedding approaches, making them useful for a range of related tasks such as tag recommendations.


Question: An extensive review of neighborhood-based recommendation techniques.

This paper explores the application of linear algebra, specifically the singular value decomposition (SVD), in enhancing intelligent information retrieval from scientific databases. Traditional retrieval methods often rely on a lexical match between words in user queries and those found in or assigned to documents. However, due to the vast diversity in language and terminology used to describe similar content, these lexical methods can be both incomplete and imprecise. By leveraging the SVD to analyze large, sparse term-by-document matrices, it is possible to uncover the implicit higher-order structure in the association of terms with documents. Terms and documents are then represented by the largest singular vectors, enabling a more sophisticated matching against user queries. This retrieval method, termed Latent Semantic Indexing (LSI), capitalizes on the subspace to represent significant associative relationships between terms and documents that are not immediately apparent within individual documents. LSI is a fully automated and intelligent indexing technique, offering a broad applicability and a promising avenue for improving user access to various textual materials, as well as documents and services described textually. The paper also discusses the computational requirements for managing databases encoded using LSI and presents an overview of current and future applications of this innovative indexing method.

Point-of-interest (POI) recommendation that suggests new places for users to visit arises with the popularity of location-based social networks (LBSNs). Due to the importance of POI recommendation in LBSNs, it has attracted much academic and industrial interest. In this paper, we offer a systematic review of this field, summarizing the contributions of individual efforts and exploring their relations. We discuss the new properties and challenges in POI recommendation, compared with traditional recommendation problems, e.g., movie recommendation. Then, we present a comprehensive review in three aspects: influential factors for POI recommendation, methodologies employed for POI recommendation, and different tasks in POI recommendation. Specifically, we propose three taxonomies to classify POI recommendation systems. First, we categorize the systems by the influential factors check-in characteristics, including the geographical information, social relationship, temporal influence, and content indications. Second, we categorize the systems by the methodology, including systems modeled by fused methods and joint methods. Third, we categorize the systems as general POI recommendation and successive POI recommendation by subtle differences in the recommendation task whether to be bias to the recent check-in. For each category, we summarize the contributions and system features, and highlight the representative work. Moreover, we discuss the available data sets and the popular metrics. Finally, we point out the possible future directions in this area and conclude this survey.

Compared with explicit feedback data, implicit feedback data is easier to be collected and more widespread. However, implicit feedback data is also more difficult to be analyzed due to the lack of negative examples. Bayesian personalized ranking (BPR) is a well-known algorithm for personalized recommendation on implicit feedback data due to its high performance. However, it (1) treats all the unobserved feedback the same as negative examples which may be just caused by unseen, (2) treats all the observed feedback the same as positive examples which may be just caused by noisy action, and (3) assumes the preferences of users are independent which is difficult to achieve in reality. To solve all these problems, we propose a novel personalized recommendation algorithm called collaborative pairwise learning to rank (CPLR), which considers the influence between users on the preferences for both items with observed feedback and items without. To take these information into consideration, we try to optimize a generalized AUC instead of the standard AUC used in BPR. CPLR can be seen as a generalized BPR. Besides BPR, several extension algorithms of BPR, like social BPR (SBPR) and group preference based BPR (GBPR), are special cases of CPLR. Extensive experiments demonstrate the promise of our approach in comparison with traditional collaborative filtering methods and state-of-the-art pairwise learning to rank algorithms. Compared with the performance of baseline algorithms on five real-world data sets, the improvements of CPLR are over 17%, 23% and 22% for Pre@5, MAP and NDCG, respectively. © 2018 Published by Elsevier B.V.

Recent online services rely heavily on automatic personalization to recommend relevant content to a large number of users. This requires systems to scale promptly to accommodate the stream of new users visiting the online services for the first time. In this work, we propose a content-based recommendation system to address both the recommendation quality and the system scalability. We propose to use a rich feature set to represent users, according to their web browsing history and search queries. We use a Deep Learning approach to map users and items to a latent space where the similarity between users and their preferred items is maximized. We extend the model to jointly learn from features of items from different domains and user features by introducing a multi-view Deep Learning model. We show how to make this rich-feature based user representation scalable by reducing the dimension of the inputs and the amount of training data. The rich user feature representation allows the model to learn relevant user behavior patterns and give useful recommendations for users who do not have any interaction with the service, given that they have adequate search and browsing history. The combination of different domains into a single model for learning helps improve the recommendation quality across all the domains, as well as having a more compact and a semantically richer user latent feature vector. We experiment with our approach on three real-world recommendation systems acquired from different sources of Microsoft products: Windows Apps recommendation, News recommendation, and Movie/TV recommendation. Results indicate that our approach is significantly better than the state-of-the-art algorithms (up to 49% enhancement on existing users and 115% enhancement on new users). In addition, experiments on a publicly open data set also indicate the superiority of our method in comparison with transitional generative topic models, for modeling cross-domain recommender systems. Scalability analysis show that our multi-view DNN model can easily scale to encompass millions of users and billions of item entries. Experimental results also confirm that combining features from all domains produces much better performance than building separate models for each domain.

This paper proposes an item concept embedding (ICE) framework to model item concepts via textual information. Specifically, in the proposed framework there are two stages: graph construction and embedding learning. In the first stage, we propose a generalized network construction method to build a network involving heterogeneous nodes and a mixture of both homogeneous and heterogeneous relations. The second stage leverages the concept of neighborhood proximity to learn the embeddings of both items and words. With the proposed carefully designed ICE networks, the resulting embedding facilitates both homogeneous and heterogeneous retrieval, including item-to-item and word-to-item retrieval. Moreover, as a distributed embedding approach, the proposed ICE approach not only generates related retrieval results but also delivers more diverse results than traditional keyword-matching-based approaches. As our experiments on two real-world datasets show, ICE encodes useful textual information and thus outperforms traditional methods in various item classification and retrieval tasks.

Weak semantic techniques rely on the integration of Semantic Web techniques with social annotations and aim to embrace the strengths of both. In this article, we propose a novel weak semantic technique for query expansion. Traditional query expansion techniques are based on the computation of two-dimensional co-occurrence matrices. Our approach proposes the use of three-dimensional matrices, where the added dimension is represented by semantic classes (i.e., categories comprising all the terms that share a semantic property) related to the folksonomy extracted from social bookmarking services, such as delicious and StumbleUpon. The results of an indepth experimental evaluation performed on both artificial datasets and real users show that our approach outperforms traditional techniques, such as relevance feedback and personalized PageRank, so confirming the validity and usefulness of the categorization of the user needs and preferences in semantic classes. We also present the results of a questionnaire aimed to know the users opinion regarding the system. As one drawback of several query expansion techniques is their high computational costs, we also provide a complexity analysis of our system, in order to show its capability of operating in real time.

The Web has accumulated a rich source of information, such as text, image, rating, etc, which represent different aspects of user preferences. However, the heterogeneous nature of this information makes it difficult for recommender systems to leverage in a unified framework to boost the performance. Recently, the rapid development of representation learning techniques provides an approach to this problem. By translating the various information sources into a unified representation space, it becomes possible to integrate heterogeneous information for informed recommendation.
 In this work, we propose a Joint Representation Learning (JRL) framework for top-N recommendation. In this framework, each type of information source (review text, product image, numerical rating, etc) is adopted to learn the corresponding user and item representations based on available (deep) representation learning architectures. Representations from different sources are integrated with an extra layer to obtain the joint representations for users and items. In the end, both the per-source and the joint representations are trained as a whole using pair-wise learning to rank for top-N recommendation. We analyze how information propagates among different information sources in a gradient-descent learning paradigm, based on which we further propose an extendable version of the JRL framework (eJRL), which is rigorously extendable to new information sources to avoid model re-training in practice.
 By representing users and items into embeddings offline, and using a simple vector multiplication for ranking score calculation online, our framework also has the advantage of fast online prediction compared with other deep learning approaches to recommendation that learn a complex prediction network for online calculation.

As a result of the importance of academic collaboration at smart conferences, various researchers have utilized recommender systems to generate effective recommendations for participants. Recent research has shown that the personality traits of users can be used as innovative entities for effective recommendations. Nevertheless, subjective perceptions involving the personality of participants at smart conferences are quite rare and have not gained much attention. Inspired by the personality and social characteristics of users, we present an algorithm called Socially and Personality Aware Recommendation of Participants (SPARP). Our recommendation methodology hybridizes the computations of similar interpersonal relationships and personality traits among participants. SPARP models the personality and social characteristic profiles of participants at a smart conference. By combining the aforementioned recommendation entities, SPARP then recommends participants to each other for effective collaborations. We evaluate SPARP using a relevant data set. Experimental results confirm that SPARP is reliable and outperforms other state-of-the-art methods.

Recommender Systems are commonly used in web applications to support users in finding items of their interest. We here propose to use Recommender Systems in Internet of Things scenarios to support human decision making in the physical world. For instance, users' choices (visit actions) for points of interests (POIs) while they explore a sensor enabled city can be tracked and used to generate recommendations for not yet visited POIs. In this PhD research, we propose a novel learning approach for generating an explainable human behaviour model and relevant recommendations in sensor enabled areas. Moreover, we propose techniques for simulating user behaviour and analyse the collective dynamics of a population of users.

The rapid growth of location-based social networks (LBSNs) has greatlyenriched people’s urban experience through social media and attracted increasingnumber of users in recent years. Typical location-based social networking sites allowusers to “check in” at a physical place and share the location with their onlinefriends, and therefore bridge the gap between the real world and online social networks.The purpose is that on the finalextraction of information, the results are worked for SocialRecommendation. However, the recommender systems presentsome failures in the filtering of the results and the way theyaresuggested to users.The method used here intendsto create an application thatperforms the Data Mining with textual information linked togeo-location data. The availability of large amounts of geographical and social data on LBSNsprovides an unprecedented opportunity to study human mobile behaviour throughdata analysis in a spatial-temporal-social context, enabling a variety of locationbasedservices, from mobile marketing to disaster relief.Data mining (sometimes called data or knowledge discovery) is the process of analyzing data from different perspectives and summarizing it into useful information information that can be used to increase revenue, cuts costs, or both.

Research on recommender systems is a challenging task, as is building and operating such systems. Major challenges include non-reproducible research results, dealing with noisy data, and answering many questions such as how many recommendations to display, how often, and, of course, how to generate recommendations most effectively. In the past six years, we built three research-article recommender systems for digital libraries and reference managers, and conducted research on these systems. In this paper, we share some experiences we made during that time. Among others, we discuss the required skills to build recommender systems, and why the literature provides little help in identifying promising recommendation approaches. We explain the challenge in creating a randomization engine to run A/B tests, and how low data quality impacts the calculation of bibliometrics. We further discuss why several of our experiments delivered disappointing results, and provide statistics on how many researchers showed interest in our recommendation dataset.


Question: Applying pre-trained models for fine-grained image categorization in the fashion domain.

This paper presents a novel deep learning approach to generate binary hash codes for efficient image retrieval, particularly in the context of large-scale datasets. Motivated by the recent progress in convolutional neural networks (CNNs), we have developed a framework that leverages the power of CNNs to learn binary codes for rapid image retrieval. The key insight is that when data labels are accessible, binary codes can be effectively learned by introducing a hidden layer to represent the latent concepts that are crucial for classifying the images. This strategy not only facilitates the learning of hash codes but also enables the acquisition of image representations directly from the CNN. Unlike conventional supervised methods that necessitate pairwise inputs for binary code learning, our method employs a pointwise approach, making it highly adaptable to extensive datasets. Our experiments on the CIFAR-10 and MNIST datasets have shown that our technique surpasses several leading hashing algorithms in terms of performance. Moreover, we have conducted extensive tests on a large-scale dataset consisting of 1 million clothing images, further validating the scalability and effectiveness of our method in the fashion domain.

Large Convolutional Network models have recently demonstrated impressive classification performance on the ImageNet benchmark (Krizhevsky et al., 2012). However there is no clear understanding of why they perform so well, or how they might be improved. In this paper we address both issues. We introduce a novel visualization technique that gives insight into the function of intermediate feature layers and the operation of the classifier. Used in a diagnostic role, these visualizations allow us to find model architectures that outperform Krizhevsky et al. on the ImageNet classification benchmark. We also perform an ablation study to discover the performance contribution from different model layers. We show our ImageNet model generalizes well to other datasets: when the softmax classifier is retrained, it convincingly beats the current state-of-the-art results on Caltech-101 and Caltech-256 datasets.

This paper investigates unified feature learning and classifier training approaches for image aesthetics assessment . Existing methods built upon handcrafted or generic image features and developed machine learning and statistical modeling techniques utilizing training examples. We adopt a novel deep neural network approach to allow unified feature learning and classifier training to estimate image aesthetics. In particular, we develop a double-column deep convolutional neural network to support heterogeneous inputs, i.e., global and local views, in order to capture both global and local characteristics of images . In addition, we employ the style and semantic attributes of images to further boost the aesthetics categorization performance . Experimental results show that our approach produces significantly better results than the earlier reported results on the AVA dataset for both the generic image aesthetics and content -based image aesthetics. Moreover, we introduce a 1.5-million image dataset (IAD) for image aesthetics assessment and we further boost the performance on the AVA test set by training the proposed deep neural networks on the IAD dataset.

Scene classification is increasingly popular due to its extensive usage in many real-world applications such as object detection, image retrieval, and so on. Traditionally, the low-level hand-crafted image representations are adopted to describe the scene images. However, they usually fail to detect semantic features of visual concepts, especially in handling complex scenes. In this paper, we propose a novel high-level image representation which utilizes image attributes as features for scene classification. More specifically, the attributes of each image are firstly extracted by a deep convolution neural network (CNN), which is trained to be a multi-label classifier by minimizing an element-wise logistic loss function. The process of generating attributes can reduce the “semantic gap” between the low-level feature representation and the high level scene meaning. Based on the attributes, we then build a system to discover semantically meaningful descriptions of the scene classes. Extensive experiments on four large-scale scene classification datasets show that our proposed algorithm considerably outperforms other state-of-the-art methods.

Most recent garment capturing techniques rely on acquiring multiple views of clothing, which may not always be readily available, especially in the case of pre-existing photographs from the web. As an alternative, we propose a method that is able to compute a rich and realistic 3D model of a human body and its outfits from a single photograph with little human interaction. Our algorithm is not only able to capture the global shape and geometry of the clothing, it can also extract small but important details of cloth, such as occluded wrinkles and folds. Unlike previous methods using full 3D information (i.e. depth, multi-view images, or sampled 3D geometry), our approach achieves detailed garment recovery from a single-view image by using statistical, geometric, and physical priors and a combination of parameter estimation, semantic parsing, shape recovery, and physicsbased cloth simulation. We demonstrate the effectiveness of our algorithm by re-purposing the reconstructed garments for virtual try-on and garment transfer applications, as well as cloth animation for digital characters.

Food recognition is an emerging topic in the field of computer vision. The recent interest of the research community in this area is justified by the rise in popularity of food diary applications, where the users take note of their food intake for self-monitoring or to provide useful statistics to dietitians. However, manually annotating food intake can be a tedious task, thus explaining the need of a system that automatically recognizes food, and possibly its amount, from pictures acquired by mobile devices. In this work we propose an approach to food recognition which combines the strengths of different state-of-the-art classifiers, namely Convolutional Neural Networks, Extreme Learning Machines and Neural Trees. We show that the proposed architecture can achieve good results even with low computational power, as in the case of mobile devices.

Fine-grained car recognition aims to recognize the category information of a car, such as car make, car model, or even the year of manufacture. A number of recent studies have shown that a deep convolutional neural network (DCNN) trained on a large-scale data set can achieve impressive results at a range of generic object classification tasks. In this paper, we propose a spatially weighted pooling (SWP) strategy, which considerably improves the robustness and effectiveness of the feature representation of most dominant DCNNs. More specifically, the SWP is a novel pooling layer, which contains a predefined number of spatially weighted masks or pooling channels. The SWP pools the extracted features of DCNNs with the guidance of its learnt masks, which measures the importance of the spatial units in terms of discriminative power. As the existing methods that apply uniform grid pooling on the convolutional feature maps of DCNNs, the proposed method can extract the convolutional features and generate the pooling channels from a single DCNN. Thus minimal modification is needed in terms of implementation. Moreover, the parameters of the SWP layer can be learned in the end-to-end training process of the DCNN. By applying our method to several fine-grained car recognition data sets, we demonstrate that the proposed method can achieve better performance than recent approaches in the literature. We advance the state-of-the-art results by improving the accuracy from 92.6% to 93.1% on the Stanford Cars-196 data set and 91.2% to 97.6% on the recent CompCars data set. We have also tested the proposed method on two additional large-scale data sets with impressive results observed.

Fine-grained categorization refers to the task of classifying objects that belong to the same basic-level class (e.g. different bird species) and share similar shape or visual appearances. Most of the state-of-the-art basic-level object classification algorithms have difficulties in this challenging problem. One reason for this can be attributed to the popular codebook-based image representation, often resulting in loss of subtle image information that are critical for fine-grained classification. Another way to address this problem is to introduce human annotations of object attributes or key points, a tedious process that is also difficult to generalize to new tasks. In this work, we propose a codebook-free and annotation-free approach for fine-grained image categorization. Instead of using vector-quantized codewords, we obtain an image representation by running a high throughput template matching process using a large number of randomly generated image templates. We then propose a novel bagging-based algorithm to build a final classifier by aggregating a set of discriminative yet largely uncorrelated classifiers. Experimental results show that our method outperforms state-of-the-art classification approaches on the Caltech-UCSD Birds dataset.

We present a method for training multi-label, massively multi-class image classification models, that is faster and more accurate than supervision via a sigmoid cross-entropy loss (logistic regression). Our method consists in embedding high-dimensional sparse labels onto a lower-dimensional dense sphere of unit-normed vectors, and treating the classification problem as a cosine proximity regression problem on this sphere. We test our method on a dataset of 300 million high-resolution images with 17,000 labels, where it yields considerably faster convergence, as well as a 7% higher mean average precision compared to logistic regression.

We evaluate whether features extracted from the activation of a deep convolutional network trained in a fully supervised fashion on a large, fixed set of object recognition tasks can be repurposed to novel generic tasks. Our generic tasks may differ significantly from the originally trained tasks and there may be insufficient labeled or unlabeled data to conventionally train or adapt a deep architecture to the new tasks. We investigate and visualize the semantic clustering of deep convolutional features with respect to a variety of such tasks, including scene recognition, domain adaptation, and fine-grained recognition challenges. We compare the efficacy of relying on various network levels to define a fixed feature, and report novel results that significantly outperform the state-of-the-art on several important vision challenges. We are releasing DeCAF, an open-source implementation of these deep convolutional activation features, along with all associated network parameters to enable vision researchers to be able to conduct experimentation with deep representations across a range of visual concept learning paradigms.

Fine-grained classification is a recognition task where subtle differences distinguish between different classes. To tackle this classification problem, part-based classification methods are mostly used. Partbased methods learn an algorithm to detect parts of the observed object and extract local part features for the detected part regions. In this paper we show that not all extracted part features are always useful for the classification. Furthermore, given a part selection algorithm that actively selects parts for the classification we estimate the upper bound for the fine-grained recognition performance. This upper bound lies way above the current state-of-the-art recognition performances which shows the need for such an active part selection method. Though we do not present such an active part selection algorithm in this work, we propose a novel method that is required by active part selection and enables sequential part-based classification. This method uses a support vector machine (SVM) ensemble and allows to classify an image based on arbitrary number of part features. Additionally, the training time of our method does not increase with the amount of possible part features. This fact allows to extend the SVM ensemble with an active part selection component that operates on a large amount of part feature proposals without suffering from increasing training time.


Question: Applying pre-trained models to fine-grained image classification in the fashion domain.

This paper presents a technique called Batch Normalization (BN) designed to address the challenges associated with training deep neural networks. The main issue stems from the changing distribution of each layer's inputs during the training phase, as the parameters of preceding layers undergo updates. This phenomenon, referred to as internal covariate shift, leads to slower training, necessitating lower learning rates and meticulous initialization, particularly when dealing with nonlinearities that saturate. To tackle this problem, we propose normalizing the inputs of each layer. The novelty of our approach lies in integrating normalization into the model's architecture and executing it on a per mini-batch basis during training. Batch Normalization empowers us to employ significantly higher learning rates and be less stringent about initialization, in certain scenarios even negating the requirement for Dropout. When applied to a cutting-edge image classification model, Batch Normalization achieves comparable accuracy with a remarkable 14 times reduction in the number of training steps required. Furthermore, it outperforms the original model by a considerable margin. By leveraging an ensemble of batch-normalized networks, we have surpassed the best previously published results on ImageNet classification, achieving a top-5 test error rate of 4.82%, which exceeds the accuracy of human evaluators.

**Question**: Investigating the Impact of Social Media on Mental Health: A Case Study of Teenagers

**Documents**: The Role of Social Media in Adolescent Mental Health: A Review of the Literature
The use of social media among adolescents has become ubiquitous, with potential implications for mental health. This review synthesizes the existing literature on the relationship between social media usage and mental health outcomes in adolescents. Studies suggest a complex interplay between social media use and mental health, with both positive and negative effects reported. Positive impacts include increased social connectedness, emotional support, and opportunities for self-expression. Negative effects include increased risk of cyberbullying, body image concerns, and sleep disturbances. The review highlights the need for further research to better understand the mechanisms linking social media use and mental health, as well as to identify protective and risk factors. Recommendations for future research include longitudinal studies to establish causality, and the development of guidelines for healthy social media use among adolescents.

**Similar Documents**: A Comprehensive Review: Social Media's Influence on Adolescent Mental Health
Adolescents' engagement with social media platforms has become nearly universal, sparking concern over its potential effects on mental well-being. This comprehensive review aims to consolidate findings from the literature regarding the correlation between social media consumption and mental health outcomes in teenagers. The reviewed studies

A Aesthetics & Attribute Database (AADB) Fusing Attributes and Content for Aesthetics Ranking Demo, code and model can be download through project webpage http://www.ics.uci.edu/~skong2/aesthetics.html References: [8] He, K., Zhang, X., Ren, S., Sun, J., ECCV, 2014 [15] Lu, X., Lin, Z., Jin, H., Yang, J., Wang, J., IEEE Trans. on Multimedia, 2015 [16] Lu, X., Lin, Z., Jin, H., Yang, J., Wang, J.Z., ACMMM, 2014 [17] Lu, X., Lin, Z., Shen, X., Mech, R., Wang, J.Z., ICCV, 2015 [23] Murray, N., Marchesotti, L., Perronnin, F., CVPR, 2012 Acknowledgements: This work was supported by Adobe gift fund, NSF grants DBI-1262547 and IIS1253538. Experimental Results We use Spearman's rho rank correlation ( ) to measure ranking performance . By thresholding the rating scores, we achieve state-of-the-art classification accuracy on AVA despite never training with a classification loss. We first train a simple model with Euclidean loss for numerical rating of photo aesthetics (a) fine-tuning with rank loss Based on the regression net, we apply rank loss to fine-tune the network

We report on a series of experiments with convolutional neural networks (CNN) trained on top of pre-trained word vectors for sentence-level classification tasks. We show that a simple CNN with little hyperparameter tuning and static vectors achieves excellent results on multiple benchmarks. Learning task-specific vectors through fine-tuning offers further gains in performance. We additionally propose a simple modification to the architecture to allow for the use of both task-specific and static vectors. The CNN models discussed herein improve upon the state of the art on 4 out of 7 tasks, which include sentiment analysis and question classification.

Transfer learning is established as an effective technology in computer vision for leveraging rich labeled data in the source domain to build an accurate classifier for the target domain. However, most prior methods have not simultaneously reduced the difference in both the marginal distribution and conditional distribution between domains. In this paper, we put forward a novel transfer learning approach, referred to as Joint Distribution Adaptation (JDA). Specifically, JDA aims to jointly adapt both the marginal distribution and conditional distribution in a principled dimensionality reduction procedure, and construct new feature representation that is effective and robust for substantial distribution difference. Extensive experiments verify that JDA can significantly outperform several state-of-the-art methods on four types of cross-domain image classification problems.

Approximate nearest neighbor search is an efficient strategy for large-scale image retrieval. Encouraged by the recent advances in convolutional neural networks (CNNs), we propose an effective deep learning framework to generate binary hash codes for fast image retrieval. Our idea is that when the data labels are available, binary codes can be learned by employing a hidden layer for representing the latent concepts that dominate the class labels. The utilization of the CNN also allows for learning image representations. Unlike other supervised methods that require pair-wised inputs for binary code learning, our method learns hash codes and image representations in a point-wised manner, making it suitable for large-scale datasets. Experimental results show that our method outperforms several state-of-the-art hashing algorithms on the CIFAR-10 and MNIST datasets. We further demonstrate its scalability and efficacy on a large-scale dataset of 1 million clothing images.

Fine-grained categorization refers to the task of classifying objects that belong to the same basic-level class (e.g. different bird species) and share similar shape or visual appearances. Most of the state-of-the-art basic-level object classification algorithms have difficulties in this challenging problem. One reason for this can be attributed to the popular codebook-based image representation, often resulting in loss of subtle image information that are critical for fine-grained classification. Another way to address this problem is to introduce human annotations of object attributes or key points, a tedious process that is also difficult to generalize to new tasks. In this work, we propose a codebook-free and annotation-free approach for fine-grained image categorization. Instead of using vector-quantized codewords, we obtain an image representation by running a high throughput template matching process using a large number of randomly generated image templates. We then propose a novel bagging-based algorithm to build a final classifier by aggregating a set of discriminative yet largely uncorrelated classifiers. Experimental results show that our method outperforms state-of-the-art classification approaches on the Caltech-UCSD Birds dataset.

The exponential increase in the availability of online reviews and recommendations makes sentiment classification an interesting topic in academic and industrial research. Reviews can span so many different domains that it is difficult to gather annotated training data for all of them. Hence, this paper studies the problem of domain adaptation for sentiment classifiers, hereby a system is trained on labeled reviews from one source domain but is meant to be deployed on another. We propose a deep learning approach which learns to extract a meaningful representation for each review in an unsupervised fashion. Sentiment classifiers trained with this high-level feature representation clearly outperform state-of-the-art methods on a benchmark composed of reviews of 4 types of Amazon products. Furthermore, this method scales well and allowed us to successfully perform domain adaptation on a larger industrial-strength dataset of 22 domains.

While 3D object representations are being revived in the context of multi-view object class detection and scene understanding, they have not yet attained wide-spread use in fine-grained categorization. State-of-the-art approaches achieve remarkable performance when training data is plentiful, but they are typically tied to flat, 2D representations that model objects as a collection of unconnected views, limiting their ability to generalize across viewpoints. In this paper, we therefore lift two state-of-the-art 2D object representations to 3D, on the level of both local feature appearance and location. In extensive experiments on existing and newly proposed datasets, we show our 3D object representations outperform their state-of-the-art 2D counterparts for fine-grained categorization and demonstrate their efficacy for estimating 3D geometry from images via ultra-wide baseline matching and 3D reconstruction.

In this paper, we report the feature obtained from the Deep Convolutional Neural Network boosts food recognition accuracy greatly by integrating it with conventional hand-crafted image features, Fisher Vectors with HoG and Color patches. In the experiments, we have achieved 72.26% as the top-1 accuracy and 92.00% as the top-5 accuracy for the 100-class food dataset, UEC-FOOD100, which outperforms the best classification accuracy of this dataset reported so far, 59.6%, greatly.

Medical images are often used in clinical diagnosis. However, interpreting the insights gained from them is often a time-consuming task even for experts. For this reason, there is a need for methods that can automatically approximate the mapping from medical images to condensed textual descriptions. For identifying the presence of relevant biomedical concepts in medical images for the ImageCLEF 2017 Caption concept detection subtask we propose the use of a pretrained residual deep neural network. Specifically, a 50-layered resNet was used and retrained on the medical images. The proposed method achieved F1 Score 0.1583 on the test data.

The applicability of computer vision to real paintings and artworks has been rarely investigated, even though a vast heritage would greatly benefit from techniques which can understand and process data from the artistic domain. This is partially due to the small amount of annotated artistic data, which is not even comparable to that of natural images captured by cameras. In this paper, we propose a semantic-aware architecture which can translate artworks to photo-realistic visualizations, thus reducing the gap between visual features of artistic and realistic data. Our architecture can generate natural images by retrieving and learning details from real photos through a similarity matching strategy which leverages a weakly-supervised semantic understanding of the scene. Experimental results show that the proposed technique leads to increased realism and to a reduction in domain shift, which improves the performance of pre-trained architectures for classification, detection, and segmentation. Code will be made publicly available.


Question: Applying pre-trained models to achieve fine-grained image classification in the fashion domain.

This paper advocates for a paradigm shift in object recognition, moving from mere naming to comprehensive description. This approach not only enables the identification of familiar objects but also facilitates the articulation of unusual features in well-known objects (e.g., "spotty dog" instead of simply "dog") and provides descriptive details about unfamiliar entities ("hairy and four-legged" rather than "unknown"). Moreover, it allows for the recognition of new objects with minimal or no visual examples. The core of this new recognition strategy revolves around inferring attributes, which can be either semantic (e.g., "spotty") or discriminative (e.g., "a feature present in dogs but absent in sheep"). A significant challenge in learning attributes is achieving generalization across various object categories, not just across instances within a single category. To address this, we introduce an innovative feature selection technique designed to facilitate the learning of attributes that generalize effectively across different categories. Our assertions are substantiated through extensive evaluation, which not only highlights the limitations of the conventional naming-based recognition paradigm but also showcases the enhanced capabilities offered by our attribute-centric framework, particularly in the context of fashion field image classification. By leveraging pre-trained models and fine-tuning them with attribute-based learning, we demonstrate improved performance in recognizing fine-grained details within fashion items, such as specific patterns, materials, and styles. This approach not only enhances the accuracy of image classification but also opens up new possibilities for personalized fashion recommendations and search functionalities.

The advance of deep learning has made huge changes in computer vision and produced various off-the-shelf trained models. Particularly, Convolutional Neural Network (CNN) has been widely used to build image classification model which allow researchers transfer the pre-trained learning model for other classifications. We propose a transfer learning method to detect breast cancer using histopathology images based on Google's Inception v3 model which were initially trained for the classification of non-medical images. The pilot study shows the feasibility of transfer learning in the detection of breast cancer with AUC of 0.93.

We propose simple and effective models for the image annotation that make use of Convolutional Neural Network (CNN) features extracted from an image and word embedding vectors to represent their associated tags. Our first set of models is based on the Canonical Correlation Analysis (CCA) framework that helps in modeling both views - visual features (CNN feature) and textual features (word embedding vectors) of the data. Results on all three variants of the CCA models, namely linear CCA, kernel CCA and CCA with k-nearest neighbor (CCA-KNN) clustering, are reported. The best results are obtained using CCA-KNN which outperforms previous results on the Corel-5k and the ESP-Game datasets and achieves comparable results on the IAPRTC-12 dataset. In our experiments we evaluate CNN features in the existing models which bring out the advantages of it over dozens of handcrafted features. We also demonstrate that word embedding vectors perform better than binary vectors as a representation of the tags associated with an image. In addition we compare the CCA model to a simple CNN based linear regression model, which allows the CNN layers to be trained using back-propagation.

We evaluate whether features extracted from the activation of a deep convolutional network trained in a fully supervised fashion on a large, fixed set of object recognition tasks can be repurposed to novel generic tasks. Our generic tasks may differ significantly from the originally trained tasks and there may be insufficient labeled or unlabeled data to conventionally train or adapt a deep architecture to the new tasks. We investigate and visualize the semantic clustering of deep convolutional features with respect to a variety of such tasks, including scene recognition, domain adaptation, and fine-grained recognition challenges. We compare the efficacy of relying on various network levels to define a fixed feature, and report novel results that significantly outperform the state-of-the-art on several important vision challenges. We are releasing DeCAF, an open-source implementation of these deep convolutional activation features, along with all associated network parameters to enable vision researchers to be able to conduct experimentation with deep representations across a range of visual concept learning paradigms.

Here we present DeepGaze II, a model that predicts where people look in images. The model uses the features from the VGG-19 deep neural network trained to identify objects in images. Contrary to other saliency models that use deep features, here we use the VGG features for saliency prediction with no additional fine-tuning (rather, a few readout layers are trained on top of the VGG features to predict saliency). The model is therefore a strong test of transfer learning. After conservative crossvalidation, DeepGaze II explains about 87% of the explainable information gain in the patterns of fixations and achieves top performance in area under the curve metrics on the MIT300 hold-out benchmark. These results corroborate the finding from DeepGaze I (which explained 56% of the explainable information gain), that deep features trained on object recognition provide a versatile feature space for performing related visual tasks. We explore the factors that contribute to this success and present several informative image examples. A web service is available to compute model predictions at http://deepgaze.bethgelab.org.

The success and impact of activity recognition algorithms largely depends on the availability of the labeled training samples and adaptability of activity recognition models across various domains. In a new environment, the pre-trained activity recognition models face challenges in presence of sensing bias- ness, device heterogeneities, and inherent variabilities in human behaviors and activities. Activity Recognition (AR) system built in one environment does not scale well in another environment, if it has to learn new activities and the annotated activity samples are scarce. Indeed building a new activity recognition model and training the model with large annotated samples often help overcome this challenging problem. However, collecting annotated samples is cost-sensitive and learning activity model at wild is computationally expensive. In this work, we propose an activity recognition framework, UnTran that utilizes source domains' pre-trained autoencoder enabled activity model that transfers two layers of this network to generate a common feature space for both source and target domain activities. We postulate a hybrid AR framework that helps fuse the decisions from a trained model in source domain and two activity models (raw and deep-feature based activity model) in target domain reducing the demand of annotated activity samples to help recognize unseen activities. We evaluated our framework with three real-world data traces consisting of 41 users and 26 activities in total. Our proposed UnTran AR framework achieves ≈ 75% F1 score in recognizing unseen new activities using only 10% labeled activity data in the target domain. UnTran attains ≈ 98% F1 score while recognizing seen activities in presence of only 2-3% of labeled activity samples.

We address image classification on a large-scale, i.e. when a large number of images and classes are involved. First, we study classification accuracy as a function of the image signature dimensionality and the training set size. We show experimentally that the larger the training set, the higher the impact of the dimensionality on the accuracy. In other words, high-dimensional signatures are important to obtain state-of-the-art results on large datasets. Second, we tackle the problem of data compression on very large signatures (on the order of 105 dimensions) using two lossy compression strategies: a dimensionality reduction technique known as the hash kernel and an encoding technique based on product quantizers. We explain how the gain in storage can be traded against a loss in accuracy and/or an increase in CPU cost. We report results on two large databases — ImageNet and a dataset of lM Flickr images — showing that we can reduce the storage of our signatures by a factor 64 to 128 with little loss in accuracy. Integrating the decompression in the classifier learning yields an efficient and scalable training algorithm. On ILSVRC2010 we report a 74.3% accuracy at top-5, which corresponds to a 2.5% absolute improvement with respect to the state-of-the-art. On a subset of 10K classes of ImageNet we report a top-1 accuracy of 16.7%, a relative improvement of 160% with respect to the state-of-the-art.

This paper describes the participation of the ECOUAN team in the LifeCLEF 2015 challenge. We used a deep learning approach in which the complete system was learned without hand-engineered components. We pre-trained a convolutional neural network using 1.8 million images and used a fine-tuning strategy to transfer learned recognition capabilities from general domains to the specific challenge of Plant Identification task. The classification accuracy obtained by our method outperformed the best result obtained in 2014. Our group obtained the 4th position among all teams and the 10th position among 18 runs.

Zero-Shot Learning (ZSL) is generally achieved via aligning the semantic relationships between the visual features and the corresponding class semantic descriptions. However, using the global features to represent fine-grained images may lead to sub-optimal results since they neglect the discriminative differences of local regions. Besides, different regions contain distinct discriminative information. The important regions should contribute more to the prediction. To this end, we propose a novel stacked semantics-guided attention (SGA) model to obtain semantic relevant features by using individual class semantic features to progressively guide the visual features to generate an attention map for weighting the importance of different local regions. Feeding both the integrated visual features and the class semantic features into a multi-class classification architecture, the proposed framework can be trained end-to-end. Extensive experimental results on CUB and NABird datasets show that the proposed approach has a consistent improvement on both fine-grained zero-shot classification and retrieval tasks.

Learning to recognize of object classes is one of the most important functionalities of vision. It is estimated that humans are able to learn tens of thousands of visual categori es in their life. Given the photometric and geometric variabilities displayed by objects as well as the high degree of intra-clas s variabilities, we hypothesize that humans achieve such a fe at by using knowledge and information cumulated throughout the learning process. In recent years, a handful of pioneering p apers have applied various forms of knowledge transfer algorithms to the problem of learning object classes. We first review some o f these papers by loosely grouping them into three categories : transfer through prior parameters, transfer through shared features or parts, and transfer through contextual information. In the second half of the paper, we detail a recent algorithm proposed by the author. This incremental learning scheme us es information from object classes previously learned in the f orm of prior models to train a new object class model. Training images can be presented in an incremental way. We present experimental results tested with this model on a large numbe r of object categories.

Recent works have demonstrated that Convolutional Neural Networks (CNNs) achieve state-of-the-art results in several computer vision tasks. CNNs have also shown their ability to provide effective descriptors for image retrieval. In this paper, we focus on CNN feature extraction for instance-level image search. We started by studying in depth several methods proposed to improve the Regional Maximal Activation (RMAC) approach. Then, we selected some of these advances and introduced a new approach that combines multi-scale and multi-layer feature extraction with feature selection. We also propose an approach for local RMAC descriptor extraction based on class activation maps. Our parameter-free approach provides short descriptors and achieves state-of-the-art performance without the need of CNN finetuning or additional data in any way. In order to demonstrate the effectiveness of our approach, we conducted extensive experiments on four well known instance-level image retrieval benchmarks (the INRIA Holidays dataset, the University of Kentucky Benchmark, Oxford5k and Paris6k).


Question: Applying pre-trained models for fine-grained image classification in the fashion domain.

This paper introduces Style Finder, a cutting-edge system for recognizing and retrieving fine-grained clothing styles. As smartphones and tablets gain widespread popularity, search methodologies have expanded beyond text to include images and voice. In the realm of fashion, visual search presents a particularly engaging interface, capable of capturing nuanced visual attributes that transcend mere color and pattern, and which are often difficult to articulate through text. The extraction and alignment of such features, however, are complicated by the inherent variability and deformability of garments. To tackle this challenge, we propose a sophisticated fine-grained learning model and multimedia retrieval framework. Our approach begins with the creation of an attribute vocabulary, derived from meticulous human annotations on a specially curated fine-grained clothing dataset. This vocabulary serves as the foundation for training a high-precision visual recognition system, specifically tailored to discern clothing styles. We showcase the system's efficacy through benchmark recognition and retrieval outcomes on the Women's Fashion Coat Dataset, demonstrating its potential for enhancing mobile applications. These applications include attribute-based multimedia retrieval for clothing items and image annotation, offering a glimpse into the future of fashion technology.

We address image classification on a large-scale, i.e. when a large number of images and classes are involved. First, we study classification accuracy as a function of the image signature dimensionality and the training set size. We show experimentally that the larger the training set, the higher the impact of the dimensionality on the accuracy. In other words, high-dimensional signatures are important to obtain state-of-the-art results on large datasets. Second, we tackle the problem of data compression on very large signatures (on the order of 105 dimensions) using two lossy compression strategies: a dimensionality reduction technique known as the hash kernel and an encoding technique based on product quantizers. We explain how the gain in storage can be traded against a loss in accuracy and/or an increase in CPU cost. We report results on two large databases — ImageNet and a dataset of lM Flickr images — showing that we can reduce the storage of our signatures by a factor 64 to 128 with little loss in accuracy. Integrating the decompression in the classifier learning yields an efficient and scalable training algorithm. On ILSVRC2010 we report a 74.3% accuracy at top-5, which corresponds to a 2.5% absolute improvement with respect to the state-of-the-art. On a subset of 10K classes of ImageNet we report a top-1 accuracy of 16.7%, a relative improvement of 160% with respect to the state-of-the-art.

This paper explores a simple and efficient baseline for text classification. Our experiments show that our fast text classifier fastText is often on par with deep learning classifiers in terms of accuracy, and many orders of magnitude faster for training and evaluation. We can train fastText on more than one billion words in less than ten minutes using a standard multicore CPU, and classify half a million sentences among 312K classes in less than a minute.

BACKGROUND AND OBJECTIVES
Highly accurate classification of biomedical images is an essential task in the clinical diagnosis of numerous medical diseases identified from those images. Traditional image classification methods combined with hand-crafted image feature descriptors and various classifiers are not able to effectively improve the accuracy rate and meet the high requirements of classification of biomedical images. The same also holds true for artificial neural network models directly trained with limited biomedical images used as training data or directly used as a black box to extract the deep features based on another distant dataset. In this study, we propose a highly reliable and accurate end-to-end classifier for all kinds of biomedical images via deep learning and transfer learning.

METHODS
We first apply domain transferred deep convolutional neural network for building a deep model; and then develop an overall deep learning architecture based on the raw pixels of original biomedical images using supervised training. In our model, we do not need the manual design of the feature space, seek an effective feature vector classifier or segment specific detection object and image patches, which are the main technological difficulties in the adoption of traditional image classification methods. Moreover, we do not need to be concerned with whether there are large training sets of annotated biomedical images, affordable parallel computing resources featuring GPUs or long times to wait for training a perfect deep model, which are the main problems to train deep neural networks for biomedical image classification as observed in recent works.

RESULTS
With the utilization of a simple data augmentation method and fast convergence speed, our algorithm can achieve the best accuracy rate and outstanding classification ability for biomedical images. We have evaluated our classifier on several well-known public biomedical datasets and compared it with several state-of-the-art approaches.

CONCLUSIONS
We propose a robust automated end-to-end classifier for biomedical images based on a domain transferred deep convolutional neural network model that shows a highly reliable and accurate performance which has been confirmed on several public biomedical image datasets.

Person re-identification (Re-ID) poses an inevitable challenge to deep learning: how to learn a robust deep model with millions of parameters on a small training set of few or no labels. In this paper, two deep transfer learning methods are proposed to address the training data sparsity problem, respectively from the supervised and unsupervised settings. First, a two-stepped fine-tuning strategy with proxy classifier learning is developed to transfer knowledge from auxiliary datasets. Second, given an unlabelled Re-Iddataset, an unsupervised deep transfer learning model is proposed based on a co-training strategy. Extensive experiments show that the proposed models achieve a good performance of deep Re-ID models.

This letter presents a novel approach for logo retrieval. Considering the fact that logo only occupies a small portion of an image, we apply Faster R-CNN to detect logo proposals first, and then use a two-step pooling strategy with adaptive weight to obtain an accurate global signature. The adaptive weighted pooling method can effectively balance the recall and precision of proposals by incorporating the probability of each proposal being a logo. Experimental results show that the proposed method interprets the similarity between query and database image more accurately and achieves state of the art performance.

Images can convey rich emotions to viewers. Recent research on image emotion analysis mainly focused on affective image classification, trying to find features that can classify emotions better. We concentrate on affective image retrieval and investigate the performance of different features on different kinds of images in a multi-graph learning framework. Firstly, we extract commonly used features of different levels for each image. Generic features and features derived from elements-of-art are extracted as low-level features. Attributes and interpretable principles-of-art based features are viewed as mid-level features, while semantic concepts described by adjective noun pairs and facial expressions are extracted as high-level features. Secondly, we construct single graph for each kind of feature to test the retrieval performance. Finally, we combine the multiple graphs together in a regularization framework to learn the optimized weights of each graph to efficiently explore the complementation of different features. Extensive experiments are conducted on five datasets and the results demonstrate the effectiveness of the proposed method.

Given a grayscale photograph as input, this paper attacks the problem of hallucinating a plausible color version of the photograph. This problem is clearly underconstrained, so previous approaches have either relied on significant user interaction or resulted in desaturated colorizations. We propose a fully automatic approach that produces vibrant and realistic colorizations. We embrace the underlying uncertainty of the problem by posing it as a classification task and use class-rebalancing at training time to increase the diversity of colors in the result. The system is implemented as a feed-forward pass in a CNN at test time and is trained on over a million color images. We evaluate our algorithm using a “colorization Turing test,” asking human participants to choose between a generated and ground truth color image. Our method successfully fools humans on 32% of the trials, significantly higher than previous methods. Moreover, we show that colorization can be a powerful pretext task for self-supervised feature learning, acting as a cross-channel encoder. This approach results in state-of-the-art performance on several feature learning benchmarks.

Artist recognition is not easy for human beings if the artworks are general and 1 share many similarities, but modern machine learning or deep learning models may 2 help people identify the artwork authors. In this project, we trained Convolutional 3 Neural Networks (CNN) identifying Japanese anime artists. Our dataset was 4 crawled from Pixiv, a well-known Japanese website containing lots of fine anime 5 artworks. Specifically, the dataset has 9 authors and each one has 200 paintings. 6 We trained different kinds of networks, from a very basic Conv2D net to more 7 advanced networks, in order to carry out the identification. It turned out that 8 traditional neural networks can in some way identify different authors, but the 9 accuracy could be improved to a better level by tuning hyperparameters or using 10 more advanced networks such as ResNet and VGGNet. 11

The number of digital images is growing extremely rapidly, and so is the need for their classification. But, as more images of pre-defined categories become available, they also become more diverse and cover finer semantic differences. Ultimately, the categories themselves need to be divided into subcategories to account for that semantic refinement. Image classification in general has improved significantly over the last few years, but it still requires a massive amount of manually annotated data. Subdividing categories into subcategories multiples the number of labels, aggravating the annotation problem. Hence, we can expect the annotations to be refined only for a subset of the already labeled data, and exploit coarser labeled data to improve classification. In this work, we investigate how coarse category labels can be used to improve the classification of subcategories. To this end, we adopt the framework of Random Forests and propose a regularized objective function that takes into account relations between categories and subcategories. Compared to approaches that disregard the extra coarse labeled data, we achieve a relative improvement in subcategory classification accuracy of up to 22% in our large-scale image classification experiments.

In principle, zero-shot learning makes it possible to train a recognition model simply by specifying the category’s attributes. For example, with classifiers for generic attributes like striped and four-legged, one can construct a classifier for the zebra category by enumerating which properties it possesses—even without providing zebra training images. In practice, however, the standard zero-shot paradigm suffers because attribute predictions in novel images are hard to get right. We propose a novel random forest approach to train zero-shot models that explicitly accounts for the unreliability of attribute predictions. By leveraging statistics about each attribute’s error tendencies, our method obtains more robust discriminative models for the unseen classes. We further devise extensions to handle the few-shot scenario and unreliable attribute descriptions. On three datasets, we demonstrate the benefit for visual category learning with zero or few training examples, a critical domain for rare categories or categories defined on the fly.


Question: Applying pre-trained models for fine-grained image classification in the fashion domain.

This paper presents an innovative descriptor for images that facilitates the development of efficient and compact classifiers for object category recognition with high accuracy. The descriptor is derived from the output of numerous weakly trained object category classifiers applied to the image. These trained categories are chosen from a visual concept ontology, but the goal is not to explicitly decompose the scene based on these categories. Instead, it acknowledges that existing object category classifiers often capture additional image features beyond the category itself; these features can collectively represent visual classes that are not directly related to the semantic meanings of the constituent categories. The key benefit of this descriptor is its ability to enable object-category queries against image databases using efficient classifiers like linear support vector machines, which are computationally lightweight during testing. Furthermore, this approach can accommodate queries for novel categories. Despite reducing the representation to a mere 200 bytes per image, the classification accuracy for object category recognition remains competitive with cutting-edge methods (36% accuracy compared to 42%), but with significantly lower computational requirements.

Question: Using Pre-Trained Models for Fine-Grained Image Classification in Fashion Field

Documents: Transfer Learning for Fashion Image Classification Using Pre-Trained Models
In the realm of fashion image classification, transfer learning has emerged as a powerful technique to enhance accuracy and efficiency. This paper explores the application of pre-trained models to tackle the challenge of fine-grained image classification within the fashion domain. The pre-trained models, initially developed for large-scale image classification tasks such as ImageNet, are adapted to recognize specific fashion items with high precision. The process involves fine-tuning the pre-trained models on a fashion-focused dataset to ensure that they can accurately differentiate between subtle variations in fashion items. This approach not only reduces the need for extensive training on large datasets but also accelerates the learning process. The results demonstrate that pre-trained models, when fine-tuned for fashion image classification, can achieve high levels of accuracy, comparable to those of models trained from scratch, but with significantly less computational effort and time. This technique opens up new possibilities for efficient and accurate fashion item recognition in various applications, including e-commerce, fashion recommendation systems, and style analysis.

Food recognition is an emerging topic in the field of computer vision. The recent interest of the research community in this area is justified by the rise in popularity of food diary applications, where the users take note of their food intake for self-monitoring or to provide useful statistics to dietitians. However, manually annotating food intake can be a tedious task, thus explaining the need of a system that automatically recognizes food, and possibly its amount, from pictures acquired by mobile devices. In this work we propose an approach to food recognition which combines the strengths of different state-of-the-art classifiers, namely Convolutional Neural Networks, Extreme Learning Machines and Neural Trees. We show that the proposed architecture can achieve good results even with low computational power, as in the case of mobile devices.

Most existing zero-shot learning approaches exploit transfer learning via an intermediate-level semantic representation such as visual attributes or semantic word vectors. Such a semantic representation is shared between an annotated auxiliary dataset and a target dataset with no annotation. A projection from a low-level feature space to the semantic space is learned from the auxiliary dataset and is applied without adaptation to the target dataset. In this paper we identify an inherent limitation with this approach. That is, due to having disjoint and potentially unrelated classes, the projection functions learned from the auxiliary dataset/domain are biased when applied directly to the target dataset/domain. We call this problem the projection domain shift problem and propose a novel framework, transductive multi-view embedding, to solve it. It is ‘transductive’ in that unlabelled target data points are explored for projection adaptation, and ‘multi-view’ in that both lowlevel feature (view) and multiple semantic representations (views) are embedded to rectify the projection shift. We demonstrate through extensive experiments that our framework (1) rectifies the projection shift between the auxiliary and target domains, (2) exploits the complementarity of multiple semantic representations, (3) achieves state-of-the-art recognition results on image and video benchmark datasets, and (4) enables novel cross-view annotation tasks.

Learning similarity functions between image pairs with deep neural networks yields highly correlated activations of large embeddings. In this work, we show how to improve the robustness of embeddings by exploiting independence in ensembles. We divide the last embedding layer of a deep network into an embedding ensemble and formulate training this ensemble as an online gradient boosting problem. Each learner receives a reweighted training sample from the previous learners. This leverages large embedding sizes more effectively by significantly reducing correlation of the embedding and consequently increases retrieval accuracy of the embedding. Our method does not introduce any additional parameters and works with any differentiable loss function. We evaluate our metric learning method on image retrieval tasks and show that it improves over state-ofthe- art methods on the CUB-200-2011, Cars-196, Stanford Online Products, In-Shop Clothes Retrieval and VehicleID datasets by a significant margin.

In this paper we evaluated deep-learning frameworks based on Convolutional Neural Networks for the accurate classification of multispectral remote sensing data. Certain state-of-the-art models have been tested on the publicly available SAT-4 and SAT-6 high resolution satellite multispectral datasets. In particular, the performed benchmark included the AlexNet, AlexNet-small and VGG models which had been trained and applied to both datasets exploiting all the available spectral information. Deep Belief Networks, Autoencoders and other semi-supervised frameworks have been, also, compared. The high level features that were calculated from the tested models managed to classify the different land cover classes with significantly high accuracy rates i.e., above 99.9%. The experimental results demonstrate the great potentials of advanced deep-learning frameworks for the supervised classification of high resolution multispectral remote sensing data.

We construct multi-modal concept representations by concatenating a skip-gram linguistic representation vector with a visual concept representation vector computed using the feature extraction layers of a deep convolutional neural network (CNN) trained on a large labeled object recognition dataset. This transfer learning approach brings a clear performance gain over features based on the traditional bag-of-visual-word approach. Experimental results are reported on the WordSim353 and MEN semantic relatedness evaluation tasks. We use visual features computed using either ImageNet or ESP Game images.

We evaluate whether features extracted from the activation of a deep convolutional network trained in a fully supervised fashion on a large, fixed set of object recognition tasks can be repurposed to novel generic tasks. Our generic tasks may differ significantly from the originally trained tasks and there may be insufficient labeled or unlabeled data to conventionally train or adapt a deep architecture to the new tasks. We investigate and visualize the semantic clustering of deep convolutional features with respect to a variety of such tasks, including scene recognition, domain adaptation, and fine-grained recognition challenges. We compare the efficacy of relying on various network levels to define a fixed feature, and report novel results that significantly outperform the state-of-the-art on several important vision challenges. We are releasing DeCAF, an open-source implementation of these deep convolutional activation features, along with all associated network parameters to enable vision researchers to be able to conduct experimentation with deep representations across a range of visual concept learning paradigms.

The number of digital images is growing extremely rapidly, and so is the need for their classification. But, as more images of pre-defined categories become available, they also become more diverse and cover finer semantic differences. Ultimately, the categories themselves need to be divided into subcategories to account for that semantic refinement. Image classification in general has improved significantly over the last few years, but it still requires a massive amount of manually annotated data. Subdividing categories into subcategories multiples the number of labels, aggravating the annotation problem. Hence, we can expect the annotations to be refined only for a subset of the already labeled data, and exploit coarser labeled data to improve classification. In this work, we investigate how coarse category labels can be used to improve the classification of subcategories. To this end, we adopt the framework of Random Forests and propose a regularized objective function that takes into account relations between categories and subcategories. Compared to approaches that disregard the extra coarse labeled data, we achieve a relative improvement in subcategory classification accuracy of up to 22% in our large-scale image classification experiments.

Zero-shot learning consists in learning how to recognise new concepts by just having a description of them. Many sophisticated approaches have been proposed to address the challenges this problem comprises. In this paper we describe a zero-shot learning approach that can be implemented in just one line of code, yet it is able to outperform state of the art approaches on standard datasets. The approach is based on a more general framework which models the relationships between features, attributes, and classes as a two linear layers network, where the weights of the top layer are not learned but are given by the environment. We further provide a learning bound on the generalisation error of this kind of approaches, by casting them as domain adaptation methods. In experiments carried out on three standard real datasets, we found that our approach is able to perform significantly better than the state of art on all of them, obtaining a ratio of improvement up to 17%.

The crowd-sourced Naturewatch GBIF dataset is used to obtain a species classification dataset containing approximately 1.2 million photos of nearly 20 thousand different species of biological organisms observed in their natural habitat. We present a general hierarchical species identification system based on deep convolutional neural networks trained on the NatureWatch dataset. The dataset contains images taken under a wide variety of conditions and is heavily imbalanced, with most species associated with only few images. We apply multi-view classification as a way to lend more influence to high frequency details, hierarchical fine-tuning to help with class imbalance and provide regularisation, and automatic specificity control for optimising classification depth. Our system achieves 55.8% accuracy when identifying individual species and around 90% accuracy at an average taxonomy depth of 5.1—equivalent to the taxonomic rank of ”family”—when applying automatic specificity control.

Transfer learning allows practitioners to recognize and apply knowledge learned in previous tasks (source task) to new tasks or new domains (target task), which share some commonality. The two important factors impacting the performance of transfer learning models are: (a) the size of the target dataset, and (b) the similarity in distribution between source and target domains. Thus far, there has been little investigation into just how important these factors are. In this paper, we investigate the impact of target dataset size and source/target domain similarity on model performance through a series of experiments. We find that more data is always beneficial, and model performance improves linearly with the log of data size, until we are out of data. As source/target domains differ, more data is required and fine tuning will render better performance than feature extraction. When source/target domains are similar and data size is small, fine tuning and feature extraction renders equivalent performance. Our hope is that by beginning this quantitative investigation on the effect of data volume and domain similarity in transfer learning we might inspire others to explore the significance of data in developing more accurate statistical models.