Question: ARMDN: An associative and recurrent mixture density network approach for eRetail demand prediction.

This paper explores a comprehensive evaluation of data transformations in statistical analysis, where the traditional assumption is that observations (y1, y2, ..., yn) adhere to a normal distribution with a constant variance and can be explained by a linear model parameterized by θ. However, this study adopts a more flexible approach, positing that a normal, homoscedastic, linear model may only be applicable post a suitable transformation of the y values. The inference process for both the transformation and the linear model parameters is carried out by calculating the likelihood function and deriving the posterior distribution. The paper meticulously disentangles the contributions of normality, homoscedasticity, and additivity to the transformation process. It also critically examines the relationship between the proposed methods and previous transformation-finding techniques. The efficacy of these methods is substantiated through illustrative examples. This methodology can be likened to a sophisticated toolkit for data analysts, offering a nuanced approach to model fitting that accounts for data's inherent complexities through transformations. This is particularly relevant in scenarios where data may not naturally conform to the assumptions of a linear model, thus requiring a transformation to achieve model adequacy. The approach can be applied to various fields, including economics, where transformations might be necessary to stabilize variance or achieve linearity in relationships between variables. By considering the role of normality, homoscedasticity, and additivity in transformations, this study provides a robust framework for data analysis that can enhance the accuracy and reliability of statistical models.

Definition: Convolution-Based Memory Models are a mathematical model of neural storage of complex data structures using distributed representations. Data structures stored range from lists of pairs through sequences, trees and networks.

This paper investigates the method of forecasting stock price difference on artificially generated price series data using neuro-fuzzy systems and neural networks. As trading profits is more important to an investor than statistical performance, this paper proposes a novel rough set-based neuro-fuzzy stock trading decision model called stock trading using rough set-based pseudo outer-product (RSPOP) which synergizes the price difference forecast method with a forecast bottleneck free trading decision model. The proposed stock trading with forecast model uses the pseudo outer-product based fuzzy neural network using the compositional rule of inference [POPFNN-CRI(S)] with fuzzy rules identified using the RSPOP algorithm as the underlying predictor model and simple moving average trading rules in the stock trading decision model. Experimental results using the proposed stock trading with RSPOP forecast model on real world stock market data are presented. Trading profits in terms of portfolio end values obtained are benchmarked against stock trading with dynamic evolving neural-fuzzy inference system (DENFIS) forecast model, the stock trading without forecast model and the stock trading with ideal forecast model. Experimental results showed that the proposed model identified rules with greater interpretability and yielded significantly higher profits than the stock trading with DENFIS forecast model and the stock trading without forecast model

In this paper, we present a unied end-to-end approach to build a large scale Visual Search and Recommendation system for ecommerce. Previous works have targeted these problems in isolation. We believe a more eective and elegant solution could be obtained by tackling them together. We propose a unied Deep Convolutional Neural Network architecture, called VisNet 1, to learn embeddings to capture the notion of visual similarity, across several semantic granularities. We demonstrate the superiority of our approach for the task of image retrieval, by comparing against the state-of-the-art on the Exact Street2Shop [14] dataset. We then share the design decisions and trade-os made while deploying the model to power Visual Recommendations across a catalog of 50M products, supporting 2K queries a second at Flipkart, India’s largest e-commerce company. e deployment of our solution has yielded a signicant business impact, as measured by the conversion-rate.

Anomaly detection in time series is an important task with several practical applications. The common approach of training one model in an offline manner using historical data is likely to fail under dynamically changing and non-stationary environments where the definition of normal behavior changes over time making the model irrelevant and ineffective. In this paper, we describe a temporal model based on Recurrent Neural Networks (RNNs) for time series anomaly detection to address challenges posed by sudden or regular changes in normal behavior. The model is trained incrementally as new data becomes available, and is capable of adapting to the changes in the data distribution. RNN is used to make multi-step predictions of the time series, and the prediction errors are used to update the RNN model as well as detect anomalies and change points. Large prediction error is used to indicate anomalous behavior or a change (drift) in normal behavior. Further, the prediction errors are also used to update the RNN model in such a way that short term anomalies or outliers do not lead to a drastic change in the model parameters whereas high prediction errors over a period of time lead to significant updates in the model parameters such that the model rapidly adapts to the new norm. We demonstrate the efficacy of the proposed approach on a diverse set of synthetic, publicly available and proprietary real-world datasets.

In recent years, deep neural networks (DNN) have demonstrated significant business impact in large scale analysis and classification tasks such as speech recognition, visual object detection, pattern extraction, etc. Training of large DNNs, however, is universally considered as time consuming and computationally intensive task that demands datacenter-scale computational resources recruited for many days. Here we propose a concept of resistive processing unit (RPU) devices that can potentially accelerate DNN training by orders of magnitude while using much less power. The proposed RPU device can store and update the weight values locally thus minimizing data movement during training and allowing to fully exploit the locality and the parallelism of the training algorithm. We evaluate the effect of various RPU device features/non-idealities and system parameters on performance in order to derive the device and system level specifications for implementation of an accelerator chip for DNN training in a realistic CMOS-compatible technology. For large DNNs with about 1 billion weights this massively parallel RPU architecture can achieve acceleration factors of 30, 000 × compared to state-of-the-art microprocessors while providing power efficiency of 84, 000 GigaOps∕s∕W. Problems that currently require days of training on a datacenter-size cluster with thousands of machines can be addressed within hours on a single RPU accelerator. A system consisting of a cluster of RPU accelerators will be able to tackle Big Data problems with trillions of parameters that is impossible to address today like, for example, natural speech recognition and translation between all world languages, real-time analytics on large streams of business and scientific data, integration, and analysis of multimodal sensory data flows from a massive number of IoT (Internet of Things) sensors.

Many applications in speech, robotics, finance, and biology deal with sequential data, where ordering matters and recurrent structures are common. However, this structure cannot be easily captured by standard kernel functions. To model such structure, we propose expressive closed-form kernel functions for Gaussian processes. The resulting model, GP-LSTM, fully encapsulates the inductive biases of long short-term memory (LSTM) recurrent networks, while retaining the non-parametric probabilistic advantages of Gaussian processes. We learn the properties of the proposed kernels by optimizing the Gaussian process marginal likelihood using a new provably convergent semi-stochastic gradient procedure, and exploit the structure of these kernels for scalable training and prediction. This approach provides a practical representation for Bayesian LSTMs. We demonstrate state-of-the-art performance on several benchmarks, and thoroughly investigate a consequential autonomous driving application, where the predictive uncertainties provided by GP-LSTM are uniquely valuable.

Amazon sales-rank gives a relative estimate of a product item’s popularity among other items in the same category. An early prediction of the Amazon sales-rank of a product would imply an early guess of its sales-popularity relative to the other products on Amazon, which is one of the largest e-commerce hub across the globe. Traditional methods suggest use of product review related features, e.g., volume of reviews, text content of the reviews etc. for the purpose of prediction. In contrast, we propose in this paper for the first time a network-assisted approach to construct suitable features for prediction. In particular, we build a co-purchase network treating the individual products as nodes, with edges in between if two products are bought with one another. The way a product is positioned in this network (e.g., its centrality, clustering coefficient etc.) turns out to be a strong indicator of its sales-rank. This network-assisted approach has two distinct advantages over the traditional baseline method based on review analysis – (i) it works even if the product has no reviews (relevant especially in the early stages of the product launch) and (ii) it is notably more discriminative in classifying a popular (i.e., low sales-rank) product from an unpopular (i.e., high sales-rank) one. Based on this observation, we build a supervised model to early classify a popular product from an unpopular one. We report our results on two different product categories (CDs and cell phones) and obtain remarkably better classification accuracy compared to the baseline scheme. When the top 100 (700) products based on sales-rank are labelled as popular and the bottom 100 (700) are labelled as unpopular, the classification accuracy of our method is 89.85% (82.1%) for CDs and 84.11% (84.8%) for cell phones compared to 46.37% (68.75%) and 83.17% (71.95%) respectively from the baseline method.

As the worlds of commerce and the Internet technology become more inextricably linked, a large number of user consumption series become available for online market intelligence analysis. A critical demand along this line is to predict the future product adoption state of each user, which enables a wide range of applications such as targeted marketing. Nevertheless, previous works only aimed at predicting if a user would adopt a particular product or not with a binary buy-or-not representation. The problem of tracking and predicting users’ adoption rates, i.e., the frequency and regularity of using each product over time, is still under-explored. To this end, we present a comprehensive study of product adoption rate prediction in a competitive market. This task is nontrivial as there are three major challenges in modeling users’ complex adoption states: the heterogeneous data sources around users, the unique user preference and the competitive product selection. To deal with these challenges, we first introduce a flexible factor-based decision function to capture the change of users’ product adoption rate over time, where various factors that may influence users’ decisions from heterogeneous data sources can be leveraged. Using this factor-based decision function, we then provide two corresponding models to learn the parameters of the decision function with both generalized and personalized assumptions of users’ preferences. We further study how to leverage the competition among different products and simultaneously learn product competition and users’ preferences with both generalized and personalized assumptions. Finally, extensive experiments on two real-world datasets show the superiority of our proposed models.

Neural network architectures with memory and attention mechanisms exhibit certain reasoning capabilities required for question answering. One such architecture, the dynamic memory network (DMN), obtained high accuracy on a variety of language tasks. However, it was not shown whether the architecture achieves strong results for question answering when supporting facts are not marked during training or whether it could be applied to other modalities such as images. Based on an analysis of the DMN, we propose several improvements to its memory and input modules. Together with these changes we introduce a novel input module for images in order to be able to answer visual questions. Our new DMN+ model improves the state of the art on both the Visual Question Answering dataset and the bAbI-10k text question-answering dataset without supporting fact supervision.

The long short-term memory (LSTM) network trained by gradient descent solves difficult problems which traditional recurrent neural networks in general cannot. We have recently observed that the decoupled extended Kalman filter training algorithm allows for even better performance, reducing significantly the number of training steps when compared to the original gradient descent training algorithm. In this paper we present a set of experiments which are unsolvable by classical recurrent networks but which are solved elegantly and robustly and quickly by LSTM combined with Kalman filters.


Question: ARMDN: An associative and recurrent mixture density network approach for eRetail demand forecasting.

In this report, we explore the application of deep learning methodologies to time series modeling, specifically focusing on demand forecasting in the context of energy supply. Given the critical importance of accurate demand forecasting for electricity providers—owing to their limited storage capacity for energy production—improving forecast accuracy can significantly enhance operational efficiency. In a sector as vast as the U.S. energy market, valued at over $1 trillion and accounting for nearly 9% of the country's GDP, even slight improvements in forecasting can yield substantial economic benefits. This is particularly true when considering the potential for enhanced utilization of existing energy production capabilities. The patterns within energy load data, characterized by large datasets and inherent non-linearities, present a prime opportunity for the application of advanced machine learning techniques. Utilizing the dataset from the "Global Energy Forecasting Competition 2012 Load Forecasting" hosted on Kaggle, our project aimed to leverage deep learning architectures for predicting energy loads across various network grid regions. The dataset consisted of hourly demand data spanning four and a half years from 20 distinct geographic zones, complemented by hourly temperature readings from 11 different areas. Our analysis predominantly centered on short-term load forecasting, a critical aspect for real-time operational planning and scheduling. Through this endeavor, we sought to demonstrate the efficacy of deep learning models in addressing the complex patterns within energy demand data, thereby contributing to more efficient energy management strategies.

Traffic forecasting is a vital part of intelligent transportation systems. It becomes particularly challenging due to short-term (e.g., accidents, constructions) and long-term (e.g., peak-hour, seasonal, weather) traffic patterns. While most of the previously proposed techniques focus on normal condition forecasting, a single framework for extreme condition traffic forecasting does not exist. To address this need, we propose to take a deep learning approach. We build a deep neural network based on long short term memory (LSTM) units. We apply Deep LSTM to forecast peak-hour traffic and manage to identify unique characteristics of the traffic data. We further improve the model for postaccident forecasting with Mixture Deep LSTM model. It jointly models the normal condition traffic and the pattern of accidents. We evaluate our model on a realworld large-scale traffic dataset in Los Angeles. When trained end-to-end with suitable regularization, our approach achieves 30%-50% improvement over baselines. We also demonstrate a novel technique to interpret the model with signal stimulation. We note interesting observations from the trained neural network.

This article expands on research that has been done to develop a recurrent neural network (RNN) capable of predicting aircraft engine vibrations using long short-term memory (LSTM) neurons. LSTM RNNs can provide a more generalizable and robust method for prediction over analytical calculations of engine vibration, as analytical calculations must be solved iteratively based on specific empirical engine parameters, making this approach ungeneralizable across multiple engines. In initial work, multiple LSTM RNN architectures were proposed, evaluated and compared. This research improves the performance of the most effective LSTM network design proposed in the previous work by using a promising neuroevolution method based on ant colony optimization (ACO) to develop and enhance the LSTM cell structure of the network. A parallelized version of the ACO neuroevolution algorithm has been developed and the evolved LSTM RNNs were compared to the previously used fixed topology. The evolved networks were trained on a large database of flight data records obtained from an airline containing flights that suffered from excessive vibration. Results were obtained using MPI (Message Passing Interface) on a high performance computing (HPC) cluster, evolving 1000 different LSTM cell structures using 168 cores over 4 days. The new evolved LSTM cells showed an improvement of 1.35%, reducing prediction error from 5.51% to 4.17% when predicting excessive engine vibrations 10 seconds in the future, while at the same time dramatically reducing the number of weights from 21,170 to 11,810.

In statistics and signal processing, a time series is a sequence of data points, measured typically at successive times, spaced apart at uniform time intervals. Time series prediction is the use of a model to predict future events based on known past events; to predict future data points before they are measured. Solutions in such cases can be provided by non-parametric regression methods, of which each neural network based predictor is a class. As a learning method of time series data with neural network, Elman type Recurrent Neural Network has been known. In this paper, we propose the multi RNN. In order to verify the effectiveness of our proposed method, we experimented by the simple artificial data and the heart pulse wave data.

Recommendation systems are widely used to recommend products to the end users that are most appropriate. Online book selling websites now-a-days are competing with each other by many means. Recommendation system is one of the stronger tools to increase profit and retaining buyer. The book recommendation system must recommend books that are of buyer's interest. This paper presents book recommendation system based on combined features of content filtering, collaborative filtering and association rule mining.

With the rapid development of urban traffic, forecasting the flows of crowd plays an increasingly important role in traffic management and public safety. However, it is very challenging as it is affected by many complex factors, including spatio-temporal dependencies of regions and other external factors such as weather and holiday. In this paper, we proposed a deep-learning-based approach, named STRCNs, to forecast both inflow and outflow of crowds in every region of a city. STRCNs combines Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM) network structures to capture spatio-temporal dependencies, simultaneously. More particularly, our model can be decomposed into four components: Closeness captures the changes of instantaneous flows; Daily influence detects the changes of daily influence flows regularly; Weekly influence reacts weekly patterns of influence flows and External influence gets the influence of external factors. For the first three properties (Closeness, Daily influence and Weekly influence), we give a branch of recurrent convolutional network units to learn both spatial and temporal dependencies in crowd flows. External factors are fed into a two-layers fully connected neural network. STRCNs assigns different weights to different branches, and then merges the outputs of the four parts together. Experimental results on two data sets (MobileBJ and TaxiBJ) demonstrate that STRCNs outperforms classical time series and other deep-learning-based prediction methods.

ACKNOWLEDGEMENT The timely and successful completion of the book could hardly be possible without the helps and supports from a lot of individuals. I will take this opportunity to thank all of them who helped me either directly or indirectly during this important work. First of all I wish to express my sincere gratitude and due respect to my supervisor Dr. R.K. Agrawal, Associate Professor SC & SS, JNU. I am immensely grateful to him for his valuable guidance, continuous encouragements and positive supports which helped me a lot during the period of my work. I would like to appreciate him for always showing keen interest in my queries and providing important suggestions. I also express whole hearted thanks to my friends and classmates for their care and moral supports. The moments, I enjoyed with them during my M.Tech course will always remain as a happy memory throughout my life. I owe a lot to my mother for her constant love and support. She always encouraged me to have positive and independent thinking, which really matter in my life. I would like to thank her very much and share this moment of happiness with her. Last but not the least I am also thankful to entire faculty and staff of SC & SS for their unselfish help, I got whenever needed during the course of my work.ABSTRACT Time series modeling and forecasting has fundamental importance to various practical domains. Thus a lot of active research works is going on in this subject during several years. Many important models have been proposed in literature for improving the accuracy and effeciency of time series modeling and forecasting. The aim of this book is to present a concise description of some popular time series forecasting models used in practice, with their salient features. In this book, we have described three important classes of time series models, viz. the stochastic, neural networks and SVM based models, together with their inherent forecasting strengths and weaknesses. We have also discussed about the basic issues related to time series modeling, such as stationarity, parsimony, overfitting, etc. Our discussion about different time series models is supported by giving the experimental forecast results, performed on six real time series datasets. While fitting a model to a dataset, special care is taken to select the most parsimonious one. To evaluate forecast accuracy as well as to compare among different models …

In this paper, we examine the use of GARCH models, Neural Network Regression (NNR), Recurrent Neural Network (RNN) regression and model combinations for forecasting and trading currency volatility, with an application to the GBP/USD and USD/JPY exchange rates. Both the results of the NNR/RNN models and the model combination results are benchmarked against the simpler GARCH alternative. The idea of developing a nonlinear nonparametric approach to forecast FX volatility, identify mispriced options and subsequently develop a trading strategy based upon this process is intuitively appealing. Using daily data from December 1993 through April 1999, we develop alternative FX volatility forecasting models. These models are then tested out-of-sample over the period April 1999-May 2000, not only in terms of forecasting accuracy, but also in terms of trading efficiency: In order to do so, we apply a realistic volatility trading strategy using FX option straddles once mispriced options have been identified. Allowing for transaction costs, most trading strategies retained produce positive returns. RNN models appear as the best single modelling approach yet, somewhat surprisingly, model combination which has the best overall performance in terms of forecasting accuracy, fails to improve the RNN-based volatility trading results. Another conclusion from our results is that, for the period and currencies considered, the currency option market was inefficient and/or the pricing formulae applied by market participants were inadequate. * Christian Dunis is Girobank Professor of Banking and Finance at Liverpool Business School and Director of CIBEF (E-mail: cdunis@globalnet.co.uk). The opinions expressed herein are not those of Girobank. ** Xuehuan Huang is an Associate Researcher with CIBEF. *** CIBEF – Centre for International Banking, Economics and Finance, JMU, John Foster Building, 98 Mount Pleasant, Liverpool L3 5UZ. We are grateful to Professor Ken Holden and Professor John Thompson of Liverpool Business School for helpful comments on an earlier version of this paper.

This paper compares two approaches for predicting air temperature from historical pressure, humidity, and temperature data gathered from meteorological sensors in Northwestern Nevada. We describe our data and our representation and compare a standard neural network against a deep learning network. Our empirical results indicate that a deep neural network with Stacked Denoising Auto-Encoders (SDAE) outperforms a standard multilayer feed forward network on this noisy time series prediction task. In addition, predicting air temperature from historical air temperature data alone can be improved by employing related weather variables like barometric pressure, humidity and wind speed data in the training process.

This paper presents a nonparametric regression model of categorical time series in the setting of conditional tensor factorization and Bayes network. The underlying algorithms are developed to provide a flexible and parsimonious representation for fusion of correlated information from heterogeneous sources, which can be used to improve the performance of prediction tasks and infer the causal relationship between key variables. The proposed method is first illustrated by numerical simulation and then validated with two real-world datasets: (1) experimental data, collected from a swirl-stabilized lean-premixed laboratory-scale combustor, for detection of thermoacoustic instabilities and (2) publicly available economics data for causal inference-making.

Classifying products precisely and efficiently is a major challenge in modern e-commerce. The high traffic of new products uploaded daily and the dynamic nature of the categories raise the need for machine learning models that can reduce the cost and time of human editors. In this paper, we propose a decision level fusion approach for multi-modal product classification based on text and image neural network classifiers. We train input specific state-of-the-art deep neural networks for each input source, show the potential of forging them together into a multi-modal architecture and train a novel policy network that learns to choose between them. Finally, we demonstrate that our multi-modal network improves classification accuracy over both networks on a real-world largescale product classification dataset that we collected from Walmart.com. While we focus on image-text fusion that characterizes e-commerce businesses, our algorithms can be easily applied to other modalities such as audio, video, physical sensors, etc.


Question: The hidden aspect of cannula injections: exploring the mechanism of arterial wall perforations and emboli formation.

This paper delves into the critical issue of blindness resulting from filler injections, a severe complication that has gained attention as the use of soft tissue fillers becomes more widespread. The primary objective is to comprehensively review the existing literature on blindness cases following filler injections, emphasizing the pertinent aspects of vascular anatomy and discussing effective prevention and treatment methods. A thorough literature search was conducted to compile all documented instances of vision impairment caused by filler injections. The analysis revealed that 98 cases of vision changes were directly attributed to filler use, with high-risk sites for complications being predominantly the glabella (38.8%), nasal region (25.5%), nasolabial fold (13.3%), and forehead (12.2%). Among the filler types, autologous fat (47.9%) was found to be the most frequent cause of this complication, followed by hyaluronic acid (23.5%). Symptoms typically manifested as immediate vision loss accompanied by pain, and in most instances, vision loss was irreversible. Central nervous system complications occurred in 23.5% of the cases. The review did not identify any consistently effective treatment for blindness resulting from filler injections. Despite the rarity of blindness from fillers, it is imperative for medical practitioners performing these procedures to have a solid understanding of vascular anatomy and to be well-versed in prevention and management strategies to mitigate the risk of such severe complications.

The early reaction to the injection of silicone, collagen, and lyophilized heterologous fetal connective and cartilage tissues into the limiting zone deep dermis-superficial subcutaneous tissue was histologically examined in the pig and compared with clinical results. The inflammatory reaction to lyophilized heterologous fetal tissue is considerably more intense than that to collagen and silicone and lasts for several weeks. Therefore, it is not recommended for soft tissue filling in the face. Admitting an inferior antigenicity of fetal tissues, the authors suggest that enzymatically denaturalized collagen should be manufactured from heterologous fetal connective tissue, to be then further tested. The reaction of tissue to silicone and collagen is minimal. Silicone is preferred for dermal injections since in clinical experience it remains in the site of injection much longer. For subdermal injections, however, collagen is preferred. Based on experience with over 600 patients since 1958, the first author continues using liquid silicone. The lack of complications is probably a result of the fact that only small amounts (milliliters) of silicone were used in wrinkles or small depressions in the dermal layer and that from the beginning injection into the subcutaneous tissue was avoided. Since 1988 a new technique for the treatment of wrinkles and skin depressions with injections of dermal miniautografts has been used with satisfactory results.

Recently particulate systems like nanoparticles have been used as a physical approach to alter and improve the pharmacokinetic and pharmacodynamic properties of various types of drug molecules. They have been used in vivo to protect the drug entity in the systemic circulation, restrict access of the drug to the chosen sites and to deliver the drug at a controlled and sustained rate to the site of action.Drug delivery research is clearly moving from the microto the nanosize scale. Nanotechnology is therefore emerging as a field in medicine that is expected to elicit significant therapeutic benefits. The development of effective nanodelivery systems capable of carrying a drug specifically and safely to a desired site of action is one of the most challenging tasks of pharmaceutical formulation investigators. They are attempting to reformulate and add new indications to the existing blockbuster drugs to maintain positive scientific outcomes and therapeutic breakthroughs. The nanodelivery systems mainly include nanoemulsions, lipid or polymeric Nanoparticles and liposomes.For the past few years, there has been a considerable research on the basis of Novel drug delivery system, using particulate vesicle systems as such drug carriers for small and large molecules. Nanoparticles have been improving the therapeutic effect of drugs and minimize the side effects. Basically, Nanoparticles have been prepared by using various techniques as such dispersion of preformed polymers, polymerization of monomers and ionic gelation or co-acervation of hydrophilic polymer.

BACKGROUND
One of the main problems especially in operating room and monitoring devices is measurement of Blood Pressure (BP) by sphygmomanometer cuff. Objective :In this study we designed a new method to measure BP changes continuously for detecting information between cuff inflation times by using vital signals in monitoring devices. This will be achieved by extraction of the time difference between each cardiac cycle and a relative pulse wave.

METHODS
Finger pulse and ECG signals in lead I were recorded by a monitoring device. The output of monitoring device wasinserted in a computer by serial network communication. A software interface (Microsoft Visual C#.NET ) was used to display and process the signals in the computer. Time difference between each cardiac cycle and pulse signal was calculated throughout R wave detection in ECG and peak of pulse signal by the software. The relation between time difference in two waves and BP was determined then the coefficients of equation were obtained in different physical situations. The results of estimating BP were compared with the results of sphygmomanometer method and the error rate was calculated.

RESULTS
In this study, 25 subjects participated among them 15 were male and 10 were female. The results showed that BP was linearly related to time difference. Average of coefficient correlation was 0.9±0.03 for systolic and 0.82±0.04 for diastolic blood pressure. The highest error percentage was calculated 8% for male and 11% for female group. Significant difference was observed between the different physical situation and arm movement changes. The relationship between time difference and age was estimated in a linear relationship with a correlation coefficient of 0.76.

CONCLUSION
By determining linear relation values with high accuracy, BP can be measured with insignificant error. Therefore it can be suggested as a new method to measure the blood pressure continuously.

OBJECTIVES
In international studies, only a few data are available on subcutaneous penile vein thrombosis. The pathogenesis is unknown, and no general recommendation exists regarding therapy.

METHODS
A total of 25 patients with the clinical picture of a "superficial penile vein thrombosis" were treated at our policlinic. All patients had noted sudden and almost painless indurations on the penile dorsal surface. The extent of the thrombosis varied. Detailed anamnesis, ultrasonography, and routine laboratory tests were performed for all patients, knowing that primary therapy was conservative.

RESULTS
No patient indicated any pain. Some reported a feeling of tension in the area of the thrombosis. In all patients, the thrombosis occurred in the dorsal penis shaft. It was close to the sulcus coronarius in 21 patients, near the penis root in 3, and in the entire penis shaft in 1 patient. The length of the thrombotic vein was between 2 and 4 cm. The ultrasound results were similar for all patients. The primary treatment was conservative for all patients. Recovery was achieved in more than 92% of cases (23 of 25 patients) using conservative therapy, which consisted of local dressing with heparin ointment (10,000 IU) and oral application of an antiphlogistic for 14 days. In 2 cases, thrombectomy was necessary.

CONCLUSIONS
Extended imaging diagnosis does not improve the evaluation of the extent of a superficial penile vein thrombosis. Conservative primary therapy consisting of heparin ointment and oral application of antiphlogistics is sufficient. If the thrombosis persists after conservative therapy, surgery is indicated.


Question: The sinister aspect of cannula injections: exploring the occurrence of arterial wall perforations and emboli.

This paper presents a comprehensive study evaluating the efficacy of filler rhinoplasty through anthropometric analysis, a technique that has not been previously reported. The primary objective was to objectively demonstrate the effectiveness of filler rhinoplasty by analyzing the changes in anthropometric measurements post-procedure.

A total of 242 patients who underwent hyaluronic acid filler rhinoplasty and revisited the clinic within 2 months were included in the study. The analysis was based on the specific injection site, the volume of filler injected, and the subsequent changes in anthropometric parameters.

The results revealed that 112 patients (46.3%) underwent nasal dorsum augmentation, 8 patients (3.3%) received filler for tip rotation, and 122 patients (50.4%) had whole nose augmentation. The average volume of filler injected was 1 ± 0.4 mL for nasal dorsum augmentation, 0.9 ± 0.3 mL for tip rotation, and 1.6 ± 0.5 mL for whole nose augmentation. Upon follow-up, significant changes were observed in anthropometric measurements. Specifically, the radix height increased by 78.3%, the nasofrontal angle increased by 5.7 ± 4.1°, and the nasolabial angle increased by 9.4 ± 4.5°. Conversely, the modified nasofacial angle decreased by 1.9 ± 2.9°. It is noteworthy that three cases (1.2%) experienced vascular complications.

The study concludes that filler rhinoplasty is a simple and effective treatment option, achieving outcomes comparable to surgical augmentation rhinoplasty. Among the various anthropometric measurements analyzed, the nasal radix height proved to be the most useful for evaluating dorsum augmentation, whereas the nasolabial angle was the most indicative of nasal tip rotation. This highlights the utility of anthropometric analysis in objectively assessing the results of filler rhinoplasty.

Since more than 30 years, intrauterine contraceptive devices (IUCD) have been used for a contraceptive opportunity. Although they are termed to be a safe and effective method for contraception, they also have some type of complications and uterine perforation, septic abortion, pelvic abscess are the serious complications of these devices. The incidence of uterine perforation is very low, but in the literature nearly 100 cases were reported about the extra uterine localization of IUCD. Migration may occur to the adjacent organs. We here in describe a case of a 31 year-old woman who had an IUCD with stone formation in the bladder. In the literature all of the cases were reported as IUCD migration, but although it seems technically impossible, IUCD placement into the bladder should also be considered in misplaced IUCDs.

The transdermal route is an excellent site for drug delivery due to the avoidance of gastric degradation and hepatic metabolism, in addition to easy accessibility. Although offering numerous attractive advantages, many available transdermal systems are not able to deliver drugs and other compounds as desired. The use of hypodermic needles, associated with phobia, pain and accidental needle-sticks has been used to overcome the delivery limitation of macromolecular compounds. The means to overcome the disadvantages of hypodermic needles has led to the development of microneedles for transdermal delivery. However, since the initial stages of microneedle fabrication, recent research has been conducted integrating various fabrication techniques for generating sophisticated microneedle devices for transdermal delivery including progress on their commercialization. A concerted effort has been made within this review to highlight the current advances of microneedles, and to provide an update of pharmaceutical research in the field of microneedle-assisted transdermal drug delivery systems.

Dr. Emsen recently published a case report describing a patient with accessory axillary breast tissue treated with ultrasound-guided liposuction [3]. We recently treated a similar case of bilateral axillary breast tissue, and we offer our contribution to the debate (Figs. 1, 2). Accessory axillary breast tissues are common embriogenic alterations found in 1% to 6% of women that often manifest bilaterally [1,2]. Ectopic breast tissue is at risk for benign and malignant breast disease with reported diagnoses of fibrocystic disease, mastitis, fibroadenoma, atypical hyperplasia, and carcinoma [4 6]. In a review of 82 ectopic breast cancer cases, Marshall et al. [5] found an increased incidence of cancer in aberrant breast tissue. Ectopic cancers located in the axilla seemed to present with more extensive disease at an earlier age, suggesting that aberrant breast tissue may be at increased risk for malignant change. According to this view, as for any ectopic mass, tissue diagnosis is required and mandatory [4]. We recently treated a 43-year-old woman who was asymptomatic until her second pregnancy (2004), when two bilateral masses appeared in axillae and began to increase in size. Six months after parturition, a clear fluid was continuously secreted throughout few small pores on the skin. Ultrasounddiagnosed breast tissue and excision with histologic examination confirmed the radiologic findings. In this case, we preoperatively contemplated, as Dr. Emsen did, less invasive and aesthetic methods such as liposuction. However, a collegial discussion with our pathologist and an analysis of literature data suggested en bloc removal for a more precise analysis not only of cellular, but also of histologic characteristics of suspected mammary ducts. We believe that liposuction, more traumatic on tissues than simple excision, does not allow for this. Furthermore, although liposuction leaves minimal incision scars, the Fig. 1. A 43-year-old woman. Right axilla with accessory breast tissue.


Question: The ominous aspect of cannula injections: exploring the occurrence of arterial wall perforations and emboli.

This study aimed to explore the detailed anatomy of the angular artery (AA) and its implications for clinical procedures, particularly in facial cosmetic surgery. The objectives were to identify the morphological characteristics and provide a clearer anatomical definition of the AA to facilitate safer and more effective surgical practices.

A total of 31 hemifaces from 17 Korean cadavers and 26 hemifaces from 13 Thai cadavers were meticulously dissected to analyze the AA. The results revealed four distinct patterns of AA topography:

1. Persistent Pattern: The AA runs along the lateral side of the nose (11%).
2. Detouring Pattern: The AA courses through the cheek and tear trough area (18%).
3. Alternative Pattern: The AA passes through the medial canthal area via a branch of the ophthalmic artery (22.8%).
4. Latent Pattern: The AA is not present (26.3%).

Understanding these variations in AA anatomy is crucial for surgeons planning to inject facial fillers. This knowledge can help prevent complications such as arterial wall perforations and emboli, which can occur during cannula injections. By being aware of the different AA courses, surgeons can adopt safer injection techniques, thereby reducing the risk of adverse events and enhancing the overall outcomes of cosmetic procedures.

The biologic behavior of a facial filler determines its advantages and disadvantages. The purpose of this article is to look at the relevant biology as part of a logical basis for making treatment decisions. Historical perspectives and biologic characteristics such as local tissue reaction (including phagocytosis and granulomatous inflammation) cross-linking, particle concentration, immunogenicity, biofilm formation, gel hardness, and collagen neogenesis are considered. Bovine collagen is the most immunogenic facial filler. Porcine and bioengineered human collagen implants have very low immunogenicity, but allergic reactions and elevations of IgG are possible. Cross-linking and concentration affect the longevity of collagen and hyaluronic acid fillers. Gel hardness affects how a hyaluronic acid filler flows through the syringe and needle. Calcium hydroxylapatite, poly-L-lactic acid, and polymethylmethacrylate fillers have been shown to stimulate collagen neogenesis. It appears that any facial filler can form a granuloma. Bacterial biofilms may play a role in the activation of quiescent granulomas. Various authors interpret the definition and significance of a granuloma differently.


Question: The hidden aspect of cannula injections: exploring the occurrence of arterial wall perforations and emboli.

We report a case of ischemic oculomotor nerve palsy and skin necrosis resulting from vascular embolization following the administration of hyaluronic acid filler in the glabellar region. Despite the widespread use of hyaluronic acid for soft tissue augmentation, severe complications can arise from direct vascular embolization. Immediately post-injection, the patient experienced a sudden onset of blepharoptosis, exotropia, and diplopia, accompanied by the gradual development of skin necrosis. With the administration of steroid therapy, the symptoms and signs of oculomotor nerve palsy showed continuous improvement. The skin defects were managed with intensive wound care, leading to healing with minimal scarring. This case underscores the necessity for meticulous technique by experienced practitioners when performing percutaneous filler injections in periorbital areas, as well as the need for immediate consideration of embolization as a potential cause when symptoms manifest.

For small to moderate-sized defects in the head and neck region, local flaps have been the mainstay of reconstruction for years. However, in certain instances, additional flap translation is required be it advancement, transposition or rotation. In such cases, the local flap concept is combined with perforator flap know-how, allowing larger loco-regional flaps to be raised to reconstruct relatively larger defects, even in cosmetically-expensive areas. In our cohort of fifteen patients', we have utilised detailed microanatomy of the facial artery perforators to reconstruct such defects with good results.

Image segmentation for detection of vessel walls is necessary for quantitative assessment of vessel diseases by intravascular ultrasound. A new segmentation method based on gradient vector flow (GVF) snake model is proposed in this paper. The main characteristics of the proposed method include two aspects: one is that nonlinear filtering is performed on GVF field to reduce the critical points, change the morphological structure of the parallel curves and extend the capture range; the other is that balloon snake is combined with the model. Thus, the improved GVF and balloon snake can be automatically initialized and overcome the problem caused by local energy minima. Results of 20 in vivo cases validated the accuracy and stability of the segmentation method for intravascular ultrasound images.


Question: The hidden aspect of cannula injections: exploring the occurrence of arterial wall perforations and emboli.

This comprehensive review explores the potential complications associated with the use of injectable dermal fillers, a procedure that has gained widespread popularity among clinicians globally. Despite the high rate of patient satisfaction and successful outcomes, adverse events, ranging from minor to severe, have been documented following the administration of various dermal filler types. The author meticulously outlines typical procedural errors and provides detailed descriptions of the clinical manifestations of both frequent and uncommon complications. Additionally, the review offers practical, step-by-step treatment guidelines for addressing the consequences of these complications, designed to be easily integrated into clinical practice.

Arterial stiffness index is one of the biomechanical indices of vascular healthiness. These indexes are based on detailed pulse waveform analysis which is presented here. After photoplethysmographyic (PPG) pulse wave measurement, we decompose the pulse waveform for the estimation and determination of arterial elasticity. Firstly, it is electro-optically measured PPG signal and by electromechanical film (EMFi) measured signal that are analyzed and investigated by dividing each wave into five logarithmic normal function components. For both the PPG and EMFi waveform we can find very easily a good fit between the original and overlapped and summed wave components. Each wave component is assumed to resemble certain phenomenon in the arteries and certain indexes can be calculated for example based on the mutual timing of the components. Several studies have demonstrated that these kinds of indexes calculated based on actual biomechanical processed can predict future cardiovascular events. Many dynamic factors, e.g., arterial stiffness, depend on fixed structural features of the arterial wall. For more accurate description, arterial stiffness is estimated based on pulse wave decomposition analysis in the radial measured by EMFi and PPG method and tibial arterial walls measured by PPG method parallelly. Elucidation of the precise relationship between endothelial function and arterial stiffness can be done through biomechanics. However, arterial wall elasticity awaits still further biomechanical studies with clinical relations and the influence of arterial flexibility, resistance and ageing inside of the radial pulse waveform.

With the gradual increase of cases using fillers, cases of patients treated by non-medical professionals or inexperienced physicians resulting in complications are also increasing. We herein report 2 patients who experienced acute complications after receiving filler injections and were successfully treated with adipose-derived stem cell (ADSCs) therapy. Case 1 was a 23-year-old female patient who received a filler (Restylane) injection in her forehead, glabella, and nose by a non-medical professional. The day after her injection, inflammation was observed with a 3×3 cm skin necrosis. Case 2 was a 30-year-old woman who received a filler injection of hyaluronic acid gel (Juvederm) on her nasal dorsum and tip at a private clinic. She developed erythema and swelling in the filler-injected area A solution containing ADSCs harvested from each patient's abdominal subcutaneous tissue was injected into the lesion at the subcutaneous and dermis levels. The wounds healed without additional treatment. With continuous follow-up, both patients experienced only fine linear scars 6 months postoperatively. By using adipose-derived stem cells, we successfully treated the acute complications of skin necrosis after the filler injection, resulting in much less scarring, and more satisfactory results were achieved not only in wound healing, but also in esthetics.


Question: Neural machine translation with augmented memory capabilities.

This paper offers a comprehensive analysis and comparison of different statistical and heuristic models for calculating word alignments in translation tasks. The study encompasses the five alignment models detailed in the seminal work by Brown, Della Pietra, Della Pietra, and Mercer (1993), alongside the hidden Markov model, various smoothing techniques, and enhanced models. These statistical approaches are juxtaposed against two heuristic models grounded in the Dice coefficient. The paper explores multiple strategies for merging word alignments to achieve symmetrization of directed statistical alignment models, crucial for improving translation accuracy. The evaluation methodology hinges on assessing the quality of the Viterbi alignment against a manually created reference alignment. This rigorous assessment is conducted across two language pairs: German-English using the Verbmobil dataset, and French-English using the Hansards corpus. The analysis delves into the impact of various design choices within the statistical alignment system, with evaluations carried out on training datasets of differing sizes. A key finding is that advanced alignment models, featuring a first-order dependence and incorporating a fertility model, outperform basic heuristic models significantly. In the appendix, the authors provide an efficient training algorithm for the alignment models discussed, contributing to the methodological toolkit for natural language processing tasks.

How should one apply deep learning to tasks such as morphological reinflection, which stochastically edit one string to get another? A recent approach to such sequence-to-sequence tasks is to compress the input string into a vector that is then used to generate the output string, using recurrent neural networks. In contrast, we propose to keep the traditional architecture, which uses a finite-state transducer to score all possible output strings, but to augment the scoring function with the help of recurrent networks. A stack of bidirectional LSTMs reads the input string from leftto-right and right-to-left, in order to summarize the input context in which a transducer arc is applied. We combine these learned features with the transducer to define a probability distribution over aligned output strings, in the form of a weighted finite-state automaton. This reduces hand-engineering of features, allows learned features to examine unbounded context in the input string, and still permits exact inference through dynamic programming. We illustrate our method on the tasks of morphological reinflection and lemmatization.

Modeling attention in neural multi-source sequence-to-sequence learning remains a relatively unexplored area, despite its usefulness in tasks that incorporate multiple source languages or modalities. We propose two novel approaches to combine the outputs of attention mechanisms over each source sequence, flat and hierarchical. We compare the proposed methods with existing techniques and present results of systematic evaluation of those methods on the WMT16 Multimodal Translation and Automatic Post-editing tasks. We show that the proposed methods achieve competitive results on both tasks.

We present a simple regularization technique for Recurrent Neural Networks (RNNs) with Long Short-Term Memory (LSTM) units. Dropout, the most successful technique for regularizing neural networks, does not work well with RNNs and LSTMs. In this paper, we show how to correctly apply dropout to LSTMs, and show that it substantially reduces overfitting on a variety of tasks. These tasks include language modeling, speech recognition, and machine translation.

We propose a selective encoding model to extend the sequence-to-sequence framework for abstractive sentence summarization. It consists of a sentence encoder, a selective gate network, and an attention equipped decoder. The sentence encoder and decoder are built with recurrent neural networks. The selective gate network constructs a second level sentence representation by controlling the information flow from encoder to decoder. The second level representation is tailored for sentence summarization task, which leads to better performance. We evaluate our model on the English Gigaword, DUC 2004 and MSR abstractive sentence summarization datasets. The experimental results show that the proposed selective encoding model outperforms the state-ofthe-art baseline models.

We present a simple method to incorporate syntactic information about the target language in a neural machine translation system by translating into linearized, lexicalized constituency trees. Experiments on the WMT16 German-English news translation task shown improved BLEU scores when compared to a syntax-agnostic NMT baseline trained on the same dataset. An analysis of the translations from the syntax-aware system shows that it performs more reordering during translation in comparison to the baseline. A smallscale human evaluation also showed an advantage to the syntax-aware system. ”accepted...” header

Recurrent neural network-based sequence-to-sequence attentional models have proven effective in abstractive text summarization. In this paper, we model abstractive text summarization using a dual encoding model. Different from the previous works only using a single encoder, the proposed method employs a dual encoder including the primary and the secondary encoders. Specifically, the primary encoder conducts coarse encoding in a regular way, while the secondary encoder models the importance of words and generates more fine encoding based on the input raw text and the previously generated output text summarization. The two level encodings are combined and fed into the decoder to generate more diverse summary that can decrease repetition phenomenon for long sequence generation. The experimental results on two challenging datasets (i.e., CNN/DailyMail and DUC 2004) demonstrate that our dual encoding model performs against existing methods.

Being able to model long-term dependencies in sequential data, such as text, has been among the long-standing challenges of recurrent neural networks (RNNs). This issue is strictly related to the absence of explicit planning in current RNN architectures, more explicitly, the network is trained to predict only the next token given previous ones. In this paper, we introduce a simple way of biasing the RNNs towards planning behavior. Particularly, we introduce an additional neural network which is trained to generate the sequence in reverse order, and we require closeness between the states of the forward RNN and backward RNN that predict the same token. At each step, the states of the forward RNN are required to match the future information contained in the backward states. We hypothesize that the approach eases modeling of long-term dependencies thus helping in generating more globally consistent samples. The model trained with conditional generation achieved 4% relative improvement (CER of 7.3 compared to a baseline of 7.6).

Recurrent neural network language models (RNNLMs) have recently demonstrated state-of-the-art performance across a variety of tasks. In this paper, we improve their performance by providing a contextual real-valued input vector in association with each word. This vector is used to convey contextual information about the sentence being modeled. By performing Latent Dirichlet Allocation using a block of preceding text, we achieve a topic-conditioned RNNLM. This approach has the key advantage of avoiding the data fragmentation associated with building multiple topic models on different data subsets. We report perplexity results on the Penn Treebank data, where we achieve a new state-of-the-art. We further apply the model to the Wall Street Journal speech recognition task, where we observe improvements in word-error-rate.

Chinese input recommendation plays an important role in alleviating human cost in typing Chinese words, especially in the scenario of mobile applications. The fundamental problem is to predict the conditional probability of the next word given the sequence of previous words. Therefore, statistical language models, i.e. n-grams based models, have been extensively used on this task in real application. However, the characteristics of extremely different typing behaviors usually lead to serious sparsity problem, even n-gram with smoothing will fail. A reasonable approach to tackle this problem is to use the recently proposed neural models, such as probabilistic neural language model, recurrent neural network and word2vec. They can leverage more semantically similar words for estimating the probability. However, there is no conclusion on which approach of the two will work better in real application. In this paper, we conduct an extensive empirical study to show the differences between statistical and neural language models. The experimental results show that the two different approach have individual advantages, and a hybrid approach will bring a significant improvement.

We investigate the use of hierarchical phrase-based SMT lattices in end-to-end neural machine translation (NMT). Weight pushing transforms the Hiero scores for complete translation hypotheses, with the full translation grammar score and full ngram language model score, into posteriors compatible with NMT predictive probabilities. With a slightly modified NMT beam-search decoder we find gains over both Hiero and NMT decoding alone, with practical advantages in extending NMT to very large input and output vocabularies.


Question: Neural machine translation with memory augmentation.

This research introduces an enhanced attentional neural translation framework that integrates structural alignment biases from word-based models. Despite the remarkable performance of neural encoder-decoder models in machine translation, they often lack the complexity and inductive biases found in conventional translation systems. To address this, we augment the attentional neural translation model with structural biases, encompassing positional bias, Markov conditioning, fertility, and alignment direction agreement. These enhancements are designed to refine the model's translation capabilities, particularly in handling challenging languages with limited resources. Our empirical results demonstrate significant improvements over both the baseline attentional model and the standard phrase-based model across various language pairs, underscoring the effectiveness of incorporating structural alignment biases in low-resource translation settings.

Statistical machine translation (SMT) performance suffers when models are trained on only small amounts of parallel data. The learned models typically have both low accuracy (incorrect translations and feature scores) and low coverage (high out-of-vocabulary rates). In this work, we use an additional data resource, comparable corpora, to improve both. Beginning with a small bitext and corresponding phrase-based SMT model, we improve coverage by using bilingual lexicon induction techniques to learn new translations from comparable corpora. Then, we supplement the model’s feature space with translation scores estimated over comparable corpora in order to improve accuracy. We observe improvements between 0.5 and 1.7 BLEU translating Tamil, Telugu, Bengali, Malayalam, Hindi, and Urdu into English.

In this paper, we propose a sequential neural encoder with latent structured description SNELSD for modeling sentences. This model introduces latent chunk-level representations into conventional sequential neural encoders, i.e., recurrent neural networks with long short-term memory LSTM units, to consider the compositionality of languages in semantic modeling. An SNELSD model has a hierarchical structure that includes a detection layer and a description layer. The detection layer predicts the boundaries of latent word chunks in an input sentence and derives a chunk-level vector for each word. The description layer utilizes modified LSTM units to process these chunk-level vectors in a recurrent manner and produces sequential encoding outputs. These output vectors are further concatenated with word vectors or the outputs of a chain LSTM encoder to obtain the final sentence representation. All the model parameters are learned in an end-to-end manner without a dependency on additional text chunking or syntax parsing. A natural language inference task and a sentiment analysis task are adopted to evaluate the performance of our proposed model. The experimental results demonstrate the effectiveness of the proposed SNELSD model on exploring task-dependent chunking patterns during the semantic modeling of sentences. Furthermore, the proposed method achieves better performance than conventional chain LSTMs and tree-structured LSTMs on both tasks.

This paper reports on investigations using two techniques for language model text data augmentation for low-resourced automatic speech recognition and keyword search. Lowresourced languages are characterized by limited training materials, which typically results in high out-of-vocabulary (OOV) rates and poor language model estimates. One technique makes use of recurrent neural networks (RNNs) using word or subword units. Word-based RNNs keep the same system vocabulary, so they cannot reduce the OOV, whereas subword units can reduce the OOV but generate many false combinations. A complementary technique is based on automatic machine translation, which requires parallel texts and is able to add words to the vocabulary. These methods were assessed on 10 languages in the context of the Babel program and NIST OpenKWS evaluation. Although improvements vary across languages with both methods, small gains were generally observed in terms of word error rate reduction and improved keyword search performance.

Dictionaries and phrase tables are the basis of modern statistical machine translation systems. This paper develops a method that can automate the process of generating and extending dictionaries and phrase tables. Our method can translate missing word and phrase entries by learning language structures based on large monolingual data and mapping between languages from small bilingual data. It uses distributed representation of words and learns a linear mapping between vector spaces of languages. Despite its simplicity, our method is surprisingly effective: we can achieve almost 90% precision@5 for translation of words between English and Spanish. This method makes little assumption about the languages, so it can be used to extend and refine dictionaries and translation tables for any language pairs.

We describe a simple neural language model that relies only on character-level inputs. Predictions are still made at the word-level. Our model employs a convolutional neural network (CNN) and a highway network over characters, whose output is given to a long shortterm memory (LSTM) recurrent neural network language model (RNN-LM). On the English Penn Treebank the model is on par with the existing state-ofthe-art despite having 60% fewer parameters. On languages with rich morphology (Czech, German, French, Spanish, Russian), the model consistently outperforms a Kneser-Ney baseline and word-level/morpheme-level LSTM baselines, again with far fewer parameters. Our results suggest that on many languages, character inputs are sufficient for language modeling.

Inspired by recent work in machine translation and object detection, we introduce an attention based model that automatically learns to describe the content of images. We describe how we can train this model in a deterministic manner using standard backpropagation techniques and stochastically by maximizing a variational lower bound. We also show through visualization how the model is able to automatically learn to fix its gaze on salient objects while generating the corresponding words in the output sequence. We validate the use of attention with state-of-theart performance on three benchmark datasets: Flickr9k, Flickr30k and MS COCO.

Previous neural machine translation models used some heuristic search algorithms (e.g., beam search) in order to avoid solving the maximum a posteriori problem over translation sentences at test phase. In this paper, we propose the GumbelGreedy Decoding which trains a generative network to predict translation under a trained model. We solve such a problem using the Gumbel-Softmax reparameterization, which makes our generative network differentiable and trainable through standard stochastic gradient methods. We empirically demonstrate that our proposed model is effective for generating sequences of discrete words.

Attention is an increasingly popular mechanism used in a wide range of neural architectures. Because of the fast-paced advances in this domain, a systematic overview of attention is still missing. In this article, we define a unified model for attention architectures for natural language processing, with a focus on architectures designed to work with vector representation of the textual data. We discuss the dimensions along which proposals differ, the possible uses of attention, and chart the major research activities and open challenges in the area.

Andor, D., Alberti, C., Weiss, D., Severyn, A., Presta, A., Ganchev, K., Petrov, S., and Collins, M. (2016). Globally normalized transition-based neural networks. In Association for Computational Linguistics. Ballesteros, M., Goldberg, Y., Dyer, C., and Smith, N. A. (2016). Training with exploration improves a greedy stack-LSTM parser. Proceedings of the conference on empirical methods in natural language processing. Chen, D. and Manning, C. D. (2014). A fast and accurate dependency parser using neural networks. In Proceedings of the conference on empirical methods in natural language processing, pages 740–750. Cheng, H., Fang, H., He, X., Gao, J., and Deng, L. (2016). Bi-directional attention with agreement for dependency parsing. arXiv preprint arXiv:1608.02076. Hashimoto, K., Xiong, C., Tsuruoka, Y., and Socher, R. (2016). A joint many-task model: Growing a neural network for multiple nlp tasks. arXiv preprint arXiv:1611.01587. Kiperwasser, E. and Goldberg, Y. (2016). Simple and accurate dependency parsing using bidirectional LSTM feature representations. Transactions of the Association for Computational Linguistics, 4:313–327. Kuncoro, A., Ballesteros, M., Kong, L., Dyer, C., Neubig, G., and Smith, N. A. (2016). What do recurrent neural network grammars learn about syntax? CoRR, abs/1611.05774. McDonald, R. T. and Pereira, F. C. (2006). Online learning of approximate dependency parsing algorithms. In EACL. Nivre, J., Hall, J., and Nilsson, J. (2006). Maltparser: A data-driven parser-generator for dependency parsing. In Proceedings of LREC, volume 6, pages 2216–2219.

We present a simple method to incorporate syntactic information about the target language in a neural machine translation system by translating into linearized, lexicalized constituency trees. Experiments on the WMT16 German-English news translation task shown improved BLEU scores when compared to a syntax-agnostic NMT baseline trained on the same dataset. An analysis of the translations from the syntax-aware system shows that it performs more reordering during translation in comparison to the baseline. A smallscale human evaluation also showed an advantage to the syntax-aware system. ”accepted...” header


Question: Neural machine translation with memory augmentation.

This paper presents an innovative framework, termed Memory-augmented Neural Networks (MANN), which enhances the functionality of neural networks by integrating them with external memory components. This integration enables the network to interact with the memory through attention mechanisms, reminiscent of the interaction between a processor and memory in traditional computing architectures like the Turing Machine or Von Neumann model. Crucially, the MANN system is fully differentiable, facilitating its training via gradient descent for efficient optimization. Initial evaluations indicate that MANNs are capable of deducing rudimentary algorithms, including tasks like copying, sorting, and associative recall, solely from provided input and output examples. This approach showcases the potential for neural networks to learn and execute more complex tasks by leveraging external memory, thereby expanding their capabilities beyond what is achievable with standard architectures.

Teaching machines to read natural language documents remains an elusive challenge. Machine reading systems can be tested on their ability to answer questions posed on the contents of documents that they have seen, but until now large scale training and test datasets have been missing for this type of evaluation. In this work we define a new methodology that resolves this bottleneck and provides large scale supervised reading comprehension data. This allows us to develop a class of attention based deep neural networks that learn to read real documents and answer complex questions with minimal prior knowledge of language structure.

Neural machine translation (NMT) models are able to partially learn syntactic information from sequential lexical information. Still, some complex syntactic phenomena such as prepositional phrase attachment are poorly modeled. This work aims to answer two questions: 1) Does explicitly modeling source or target language syntax help NMT? 2) Is tight integration of words and syntax better than multitask training? We introduce syntactic information in the form of CCG supertags either in the source as an extra feature in the embedding, or in the target, by interleaving the target supertags with the word sequence. Our results on WMT data show that explicitly modeling syntax improves machine translation quality for English↔German, a high-resource pair, and for English↔Romanian, a lowresource pair and also several syntactic phenomena including prepositional phrase attachment. Furthermore, a tight coupling of words and syntax improves translation quality more than multitask training.

We investigate the use of hierarchical phrase-based SMT lattices in end-to-end neural machine translation (NMT). Weight pushing transforms the Hiero scores for complete translation hypotheses, with the full translation grammar score and full ngram language model score, into posteriors compatible with NMT predictive probabilities. With a slightly modified NMT beam-search decoder we find gains over both Hiero and NMT decoding alone, with practical advantages in extending NMT to very large input and output vocabularies.

Conversational machine comprehension requires a deep understanding of the conversation history. To enable traditional, single-turn models to encode the history comprehensively, we introduce FLOW, a mechanism that can incorporate intermediate representations generated during the process of answering previous questions, through an alternating parallel processing structure. Compared to shallow approaches that concatenate previous questions/answers as input, FLOW integrates the latent semantics of the conversation history more deeply. Our model, FLOWQA, shows superior performance on two recently proposed conversational challenges (+7.2% F1 on CoQA and +4.0% on QuAC). The effectiveness of FLOW also shows in other tasks. By reducing sequential instruction understanding to conversational machine comprehension, FLOWQA outperforms the best models on all three domains in SCONE, with +1.8% to +4.4% improvement in accuracy.

Attention is an increasingly popular mechanism used in a wide range of neural architectures. Because of the fast-paced advances in this domain, a systematic overview of attention is still missing. In this article, we define a unified model for attention architectures for natural language processing, with a focus on architectures designed to work with vector representation of the textual data. We discuss the dimensions along which proposals differ, the possible uses of attention, and chart the major research activities and open challenges in the area.

Machine reading using differentiable reasoning models has recently shown remarkable progress. In this context, End-to-End trainable Memory Networks (MemN2N) have demonstrated promising performance on simple natural language based reasoning tasks such as factual reasoning and basic deduction. However, other tasks, namely multi-fact questionanswering, positional reasoning or dialog related tasks, remain challenging particularly due to the necessity of more complex interactions between the memory and controller modules composing this family of models. In this paper, we introduce a novel end-to-end memory access regulation mechanism inspired by the current progress on the connection short-cutting principle in the field of computer vision. Concretely, we develop a Gated End-toEnd trainable Memory Network architecture (GMemN2N). From the machine learning perspective, this new capability is learned in an end-to-end fashion without the use of any additional supervision signal which is, as far as our knowledge goes, the first of its kind. Our experiments show significant improvements on the most challenging tasks in the 20 bAbI dataset, without the use of any domain knowledge. Then, we show improvements on the Dialog bAbI tasks including the real human-bot conversion-based Dialog State Tracking Challenge (DSTC-2) dataset. On these two datasets, our model sets the new state of the art. ∗work done as an Intern at XRCE

In this work, we propose a novel decoding approach for neural machine translation (NMT) based on continuous optimisation. The resulting optimisation problem can then be tackled using a whole range of continuous optimisation algorithms which have been developed and used in the literature mainly for training. Our approach is general and can be applied to other sequence-to-sequence neural models as well. We make use of this powerful decoding approach to intersect an underlying NMT with a language model, to intersect left-to-right and right-to-left NMT models, and to decode with soft constraints involving coverage and fertility of the source sentence words. The experimental results show the promise of the proposed framework.

We present an efficient algorithm to estimate large modified Kneser-Ney models including interpolation. Streaming and sorting enables the algorithm to scale to much larger models by using a fixed amount of RAM and variable amount of disk. Using one machine with 140 GB RAM for 2.8 days, we built an unpruned model on 126 billion tokens. Machine translation experiments with this model show improvement of 0.8 BLEU point over constrained systems for the 2013 Workshop on Machine Translation task in three language pairs. Our algorithm is also faster for small models: we estimated a model on 302 million tokens using 7.7% of the RAM and 14.0% of the wall time taken by SRILM. The code is open source as part of KenLM.

We study the topmost weight matrix of neural network language models. We show that this matrix constitutes a valid word embedding. When training language models, we recommend tying the input embedding and this output embedding. We analyze the resulting update rules and show that the tied embedding evolves in a more similar way to the output embedding than to the input embedding in the untied model. We also offer a new method of regularizing the output embedding. Our methods lead to a significant reduction in perplexity, as we are able to show on a variety of neural network language models. Finally, we show that weight tying can reduce the size of neural translation models to less than half of their original size without harming their performance.

Andor, D., Alberti, C., Weiss, D., Severyn, A., Presta, A., Ganchev, K., Petrov, S., and Collins, M. (2016). Globally normalized transition-based neural networks. In Association for Computational Linguistics. Ballesteros, M., Goldberg, Y., Dyer, C., and Smith, N. A. (2016). Training with exploration improves a greedy stack-LSTM parser. Proceedings of the conference on empirical methods in natural language processing. Chen, D. and Manning, C. D. (2014). A fast and accurate dependency parser using neural networks. In Proceedings of the conference on empirical methods in natural language processing, pages 740–750. Cheng, H., Fang, H., He, X., Gao, J., and Deng, L. (2016). Bi-directional attention with agreement for dependency parsing. arXiv preprint arXiv:1608.02076. Hashimoto, K., Xiong, C., Tsuruoka, Y., and Socher, R. (2016). A joint many-task model: Growing a neural network for multiple nlp tasks. arXiv preprint arXiv:1611.01587. Kiperwasser, E. and Goldberg, Y. (2016). Simple and accurate dependency parsing using bidirectional LSTM feature representations. Transactions of the Association for Computational Linguistics, 4:313–327. Kuncoro, A., Ballesteros, M., Kong, L., Dyer, C., Neubig, G., and Smith, N. A. (2016). What do recurrent neural network grammars learn about syntax? CoRR, abs/1611.05774. McDonald, R. T. and Pereira, F. C. (2006). Online learning of approximate dependency parsing algorithms. In EACL. Nivre, J., Hall, J., and Nilsson, J. (2006). Maltparser: A data-driven parser-generator for dependency parsing. In Proceedings of LREC, volume 6, pages 2216–2219.


Question: Neural machine translation enhanced with memory augmentation.

Overview of Deep Learning Techniques in Machine Translation
Deep learning techniques, particularly deep neural networks (DNNs), have become integral to machine translation (MT) systems. This paper provides a comprehensive review of how DNNs are employed in different components of MT, highlighting their impact on enhancing translation quality and efficiency. DNNs have revolutionized MT by enabling more accurate and contextually relevant translations, as well as improving the handling of rare words and morphological variants. The paper discusses the evolution of DNNs in MT, from early feedforward architectures to more sophisticated recurrent and transformer-based models. It also explores the integration of DNNs with other techniques, such as attention mechanisms and memory augmentation, to further refine translation outcomes. The review covers recent advancements and challenges in the field, underscoring the ongoing research efforts to optimize DNNs for MT applications.

We propose to exploit the potential of multiple word clusterings in class-based recurrent neural network (RNN) language models for ensemble RNN language modeling. By varying the clustering criteria and the space of word embedding, different word clusterings are obtained to define different word/class factorizations. For each such word/class factorization, several base RNNLMs are learned, and the word prediction probabilities of the base RNNLMs are then combined to form an ensemble prediction. We use a greedy backward model selection procedure to select a subset of models and combine these models for word prediction. The proposed ensemble language modeling method has been evaluated on Penn Treebank test set as well as Wall Street Journal (WSJ) Eval 92 and 93 test sets, where it improved test set perplexity and word error rate over the state-of-the-art single RNNLMs as well as multiple RNNLMs produced by varying RNN learning conditions.

In this paper, we apply a bidirectional Long Short-Term Memory with a Conditional Random Field to the task of disfluency detection. Long-range dependencies is one of the core problems for disfluency detection. Our model handles long-range dependencies by both using the Long Short-Term Memory and hand-crafted discrete features. Experiments show that utilizing the hand-crafted discrete features significantly improves the model’s performance by achieving the state-of-the-art score of 87.1% on the Switchboard corpus.

In this study, we extend recurrent neural network-based language models (RNNLMs) by explicitly integrating morphological and syntactic factors (or features). Our proposed RNNLM is called a factored RNNLM that is expected to enhance RNNLMs. A number of experiments are carried out on top of state-of-the-art LVCSR system that show the factored RNNLM improves the performance measured by perplexity and word error rate. In the IWSLT TED test data sets, absolute word error rate reductions over RNNLM and n-gram LM are 0.4∼0.8 points.

Deep Neural Networks have been shown to succeed at a range of natural language tasks such as machine translation and text summarization. While tasks on source code (ie, formal languages) have been considered recently, most work in this area does not attempt to capitalize on the unique opportunities offered by its known syntax and structure. In this work, we introduce SmartPaste, a first task that requires to use such information. The task is a variant of the program repair problem that requires to adapt a given (pasted) snippet of code to surrounding, existing source code. As first solutions, we design a set of deep neural models that learn to represent the context of each variable location and variable usage in a data flow-sensitive way. Our evaluation suggests that our models can learn to solve the SmartPaste task in many cases, achieving 58.6% accuracy, while learning meaningful representation of variable usages.

Neural Machine translation has shown promising results in recent years. In order to control the computational complexity, NMT has to employ a small vocabulary, and massive rare words outside the vocabulary are all replaced with a single unk symbol. Besides the inability to translate rare words, this kind of simple approach leads to much increased ambiguity of the sentences since meaningless unks break the structure of sentences, and thus hurts the translation and reordering of the in-vocabulary words. To tackle this problem, we propose a novel substitution-translation-restoration method. In substitution step, the rare words in a testing sentence are replaced with similar in-vocabulary words based on a similarity model learnt from monolingual data. In translation and restoration steps, the sentence will be translated with a model trained on new bilingual data with rare words replaced, and finally the translations of the replaced words will be substituted by that of original ones. Experiments on Chinese-to-English translation demonstrate that our proposed method can achieve more than 4 BLEU points over the attention-based NMT. When compared to the recently proposed method handling rare words in NMT, our method can also obtain an improvement by nearly 3 BLEU points.

Neural network-based Open-ended conversational agents automatically generate responses based on predictive models learned from a large number of pairs of utterances. The generated responses are typically acceptable as a sentence but are often dull, generic, and certainly devoid of any emotion. In this paper we present neural models that learn to express a given emotion in the generated response. We propose four models and evaluate them against 3 baselines. An encoder-decoder framework-based model with multiple attention layers provides the best overall performance in terms of expressing the required emotion. While it does not outperform other models on all emotions, it presents promising results in most cases.

Research on question answering with knowledge base has recently seen an increasing use of deep architectures. In this extended abstract, we study the application of the neural machine translation paradigm for question parsing. We employ a sequence-to-sequence model to learn graph patterns in the SPARQL graph query language and their compositions. Instead of inducing the programs through question-answer pairs, we expect a semi-supervised approach, where alignments between questions and queries are built through templates. We argue that the coverage of language utterances can be expanded using late notable works in natural language generation.

In this paper we present a cross-lingual extension of a neural tensor network model for knowledge base completion. We exploit multilingual synsets from BabelNet to translate English triples to other languages and then augment the reference knowledge base with cross-lingual triples. We project monolingual embeddings of different languages to a shared multilingual space and use them for network initialization (i.e., as initial concept embeddings). We then train the network with triples from the cross-lingually augmented knowledge base. Results on WordNet link prediction show that leveraging cross-lingual information yields significant gains over exploiting only monolingual triples.

ive Summarization of Reddit Posts with Multi-level Memory Networks Byeongchang Kim Hyunwoo Kim Gunhee Kim Department of Computer Science and Engineering & Center for Superintelligence Seoul National University, Seoul, Korea {byeongchang.kim,hyunwoo.kim}@vision.snu.ac.kr gunhee@snu.ac.kr http://vision.snu.ac.kr/projects/reddit-tifu

Citation Recommendation is useful for an author to find out the papers or books that can support the materials she is writing about. It is a challengeable problem since the vocabulary used in the content of papers and in the citation contexts are usually quite different. To address this problem, we propose to use translation model, which can bridge the gap between two heterogeneous languages. We conduct an experiment and find the translation model can provide much better candidates of citations than the state-of-the-art methods.