
\section{Introduction}


Hyperparameter tuning in modern machine learning systems is critical for building effective models. However, the process--in particular for deep learning architectures--is often unintuitive and time consuming. Due to their effectiveness and wide availability, deep learning architectures are nonetheless popular as developers are willing to struggle with the tuning process. Standardized platforms (e.g., Tensorflow, Keras, PyTorch, etc.) have allowed developers to easily release their models through open-source platforms such as GitXiv and Github. The lack of clarity on which settings to use (and where and when) presents an added challenge to `model consumers.' These down-stream developers must contend with finding ways to \textit{retune} or \textit{fine-tune} models to extend, retrain, or perform new evaluation with new data.

Take for example word embedding models such as word2vec~\cite{mikolov2007language}, GloVe~\cite{pennington2014glove}, or Doc2Vec~\cite{le2014distributed}. These models capture the syntactic (and possibly semantic) relationships between terms by embedding words in a high-dimensional vector space by learning the weights for each vector dimension. The magnitude and direction of the vectors \textit{between} can be used to find related terms (nearest neighbors). Additionally, that can be used for other semantic labeling tasks, such as identifying synonyms or other relationships (e.g., hypernyms, antonyms, etc.). Embedding models, unfortunately, are not always easy to train. Even relatively simple ones, such as word2vec, easily have a dozen hyperparameters that control model construction. Everything from window size (how much text to consider around terms) to learning rate (alpha) to vector size (in what dimensional space) to architecture type (`cbow' or 'skip-gram') to sampling types can be modified. It is difficult to understand which hyperparameter 'knobs' to adjust (and how) in order to achieve a good outcome. Additionally, publicly released models are often trained and evaluated on standard corpora (e.g., Wikipedia and SEMEVAL~\cite{semeval}). Real-world use on new corpora requires identifying both the right ground-truth and the right hyperparameters for that corpora.

\begin{figure}
  \centering
  \includegraphics[width=\linewidth]{figs/fig-lamvi2-screenshot.png}
  \caption{Screenshot of LAMVI-2\xspace. A parallel coordinates visualization (A) provides access to models tuned with different hyperparameters (left) and evaluated on different metrics (right). The central heatmap supports model inspection (rows). In this case we see the distances of various terms (columns) to the query word ``legitimate'' encoded through color. A linked t-SNE visualization (B) provides a dimension-reduced view of the embedding. The user can interact with the visualization to understand the trade-offs between hyperparameters and metric, isolate models that perform well, identify outliers (models and instances), and provide examples for evaluation/supervision.}
	\label{fig:lamvi2-screenshot}
\end{figure}


Automated hyperparameter optimization is one common solution.  The simplest variant, grid-search, progressively varies each parameter to find a good model. More complex schemes use other search mechanisms~\cite{ bergstra2012random, NIPS2011_4443,  greff2016lstm, NIPS2012_4522}. An optimizer will attempt to identify the best hyperparameter set given a specific criteria (e.g., to get the best accuracy, use this configuration). Optimization suffers from two problems. First, each `run' may be expensive so constraining the hyper-parameter search is often useful (e.g., identifying a range). Second, not all real world problems have a single optimization metric and many require trade-offs. For example, one may trade-off precision and recall, or increased model size with increased accuracy, but at the cost of space resources (important on devices such as cell phones). In some situations a generic model may be constructed with multiple uses in mind, and maximum performance across all applications can not be simultaneously achieved. Finally, optimization algorithms often assume `stable' test data. A common scenario is that in adapting a model to a new scenario, the developer will find problems in the training or test data and will add data or labels as part of tuning.

Because of these limitations and, debugging and tuning models often requires human interaction. To support human-in-the-loop model tuning, we introduce LAMVI-2\xspace (LAnguage Model Visual Inspector 2.0). LAMVI-2\xspace is a visualization-driven tool specifically focused on debugging neural embedding models (Figure~\ref{fig:lamvi2-screenshot}). In implementing a variety of prototypes and through task analysis, we identified key decision points in model construction and tuning. Rather than a broad solution, LAMVI-2\xspace uses visual channels to target key features of the data/model that are useful in making decisions. The system allows the developer to focus on hyperparameter interaction and to identify how these parameters impact the model at both a fine level (e.g., specific term embeddings and distances) and in more broad measures of model performance. Through LAMVI-2\xspace, the developer can construct new model instances and add these to the comparison set. Through the interface, developers can find the best hyperparameters for their model by identifying complex correlations and interactions (e.g., increasing parameter X improves metric Y, but only up to the value Z). Outliers, that indicate possible problems in training or evaluation data, are also apparent in the interface. By integrating a labeling system directly into the interface, LAMVI-2\xspace users can also act to `correct' or `extend' model training and evaluation based on their observations.

While previous solutions, include our own work, focused on inspecting \textit{single} trained models LAMVI-2\xspace is specifically constructed for \textit{multiple}-model scenarios use cases. This is achieved by combining multiple visual encodings and interaction patterns for tuning-related tasks. As can be seen in Figure~\ref{fig:lamvi2-screenshot}, we 'anchor' LAMVI-2\xspace on a parallel coordinates style visualization (showing parameters and output metrics) and link it to heatmaps (capturing term distances) and a familiar `projection view' (a t-SNE projection commonly used by model developers). Interactions include adding model instances, filtering models based on multiple criteria, reordering (to find trends and outliers in model instances) and browsing and search features to isolate problems and new labels. LAMVI-2\xspace broadly allows the user to identify relationships and correlations between parameters and output metrics \textit{and} to inspect the difference between the embeddings produced by each model instance. 

We demonstrate the utility of LAMVI-2\xspace through a case study that replicates the early word2vec work as well as a performing a realistic user study using a real-world task. Participants were able to more rapidly and effectively tune a model using LAMVI-2\xspace over default tools. As critically, we found that they were also better able to better understand the relationship of different hyperparameters.

In this work we contribute a LAMVI-2\xspace, a novel visualization system for tuning hyperparameters of word embedding models. We believe that this approach can be adapted to other machine learning tasks where tuning is necessary.  Additionally, we demonstrate the usefulness on LAMVI-2\xspace on a realistic task. 

\section{Related Work}
Work related to our project broadly falls into two broad lines of research and development (with some significant recent overlaps). The first line emerges from the machine learning and database communities on (non-visual) model management and tuning (e.g., ~\cite{modeltracker}). The second line of work is from the visualization and HCI communities on data-specific (i.e., text) visualizations and interactive learning.

\subsection{Hyperparameter Spaces}

\begin{figure}
  \begin{center}
    \includegraphics[width=0.5\textwidth]{figs/3dplot}
  \end{center}
\caption[hyperparameter space]{3D plot of two hyperparameters.}
\label{fig:3d}
\end{figure}

However, some authors have adopted the use of static~\cite{Young2015} and interactive parallel coordinates~\cite{vizier} to illustrate the selection of hyperparameters. We take these as evidence that LAMVI-2\xspace visualizations can be effective for explaining the relationship between hyperparameters. By augmenting generic parallel coordinates with domain-specific views, LAMVI-2\xspace supports finer grain inspection of models. Many visualizations of hyperparameter spaces utilize static 3D models (see Figure~\ref{fig:3d}). 



\subsection{Text and Machine Learning}
There is a large amount of work on use of visualization to inspect text, some with a machine learning focus. For example, AntConc provides concordances and other visual toolkits to support corpus linguistic analysis, such as word frequencies and collocation inspection~\cite{anthony2011antconc}. Kulesza et al. use simple bar graphs of feature importance to provide answers to the user's \textit{why}-questions regarding text message classification in an email client application~\cite{kulesza2011oriented}.  

Chuang et al. use association matrices and alignment charts to investigate latent topic coverage of a large collection of model variants~\cite{chuang2012termite,chuang2013topic}.
LDAvis employs interactive visualizations to facilitate the user to interpret the content and inter-relationships of different latent topics learned by a topic model~\cite{sievert2014ldavis}. Other examples related to topic visualization include \cite{cui2014hierarchical,liu2009interactive}. Alexander and Gleicher~\cite{alexander16} built a visualization system to compare different topic models. The tool visualized (in)stability of different topic-document connections but did not integrate views into the hyperparameter space and metrics to enable end-user driven optimizations.

These tools provide inspection capabilities for different text-related modeling tasks. They do not however, directly address the model tuning problem. In most cases, each visualization provides access to at most one model instance at a time.

\subsection{Visual Inspection of Neural Networks}

Visualization support for machine learning engineering, and in particular neural nets, has seen significant recent attention (surveyed in~\cite{hohman2018visual}). While model construction~\cite{8019861}, debugging and model selection have been identified as key tasks, hyperparameter tuning has seen less attention. While the attention on building and comparing new architectures is reasonable given the proliferation of new techniques, tuning of the type we describe here is critical to real-world deployment and adaptation of existing architectures to new contexts.

Model comparison is a critical step in model selection. However, many current approaches focus on comparison of the model output or internal `model states' rather than on the parameters that caused a certain behavior. Some exceptions include work on understanding the impact of input data on performance~\cite{Kulesza:2015:PED:2678025.2701399,Patel:2008:ISM:1357054.1357160}.

Using visualization techniques to improve understandability of neural network (NN) models has also drawn a great deal of attention. However, many NN visualization projects are focused on computer vision~\cite{yosinski2015understanding,samek2015evaluating}. There are also several interactive visualization projects developed for educational purposes, such as the recent Tensorflow Playground (\url{http://playground.tensorflow.org/}) and distill.pub~\cite{olah2018the}. More elaborate neural network architectures, such as convolutional neural networks, have also seen progress in using visualization to understand network behavior~\cite{DBLP:journals/corr/SimonyanVZ13, DBLP:journals/corr/ZeilerF13}. 

Compared to images, videos, or quantitative multidimensional datasets, words and sentences lack natural visual representation and can be harder to interpret. The required transformation of text to data that can be visually encoded presents unique challenges. To improve understandability of natural language representations learned by neural network models, existing techniques for visualizing high-dimensional data are often borrowed, such as principal component analysis (PCA)~\cite{jolliffe2002principal}, multi-dimensional scaling~\cite{kruskal1964multidimensional}, t-SNE~\cite{van2008visualizing}, and other multi-view analytics solutions~\cite{conceptvector}. While useful for inspecting particular embeddings, these encodings/systems do not address the problem of hyperparamter comparison and tuning. 

Notable work on visualizing specific text-based deep learning include systems by Karpathy et al. that employ multiple views of activation levels to illustrate how recurrent neural networks capture patterns in text sequences, such as long-range dependencies~\cite{karpathy2015visualizing}. Others have demonstrated the use of visualizations to investigate different NN features for both text and images (\cite{ming1understanding,bilal,datacube}), finding patterns in hidden states (\cite{DBLP:journals/corr/StrobeltGHPR16,deepeyes,liushi}) or analyzing classifier output (\cite{activis}). These approaches may be integrated into our own if LAMVI-2\xspace is adapted to other neural architectures or ML models.

\subsection{Interactive Learning}
While most machine learning visualizations (including those focused on neural networks) target individual models, a few systems seek to provide inter-model comparisons. ModelTracker~\cite{modeltracker}, for example, allows the user to track the evolution of a model in terms of accuracy and provides interactive learning features. The tool allows the user to inspect  model-specific classification metrics and add new labels. This notion of interactive learning~\cite{amershi10, amershi2014power, LIU201748} inspired features in LAMVI-2\xspace to add labels through the visualization.

\section{Task Analysis}

Based on our experience with early protoypes (WEVI\xspace, LAMVI-1\xspace) and in producing educational material for word2vec users~\cite{rong2014word2vec} we identified key tasks and guidelines for hyperparameter tuning that motivate our design. We describe our prototypes and tasks below.

\subsection{Prototypes: WEVI\xspace and LAMVI-1\xspace}
[Author note: we have anonymized the references and demo in our video but the `live' versions are not anonymous.]

\label{sec-lamvi:v1v2}

\begin{figure*}[!htbp]
\begin{center}
              \includegraphics[width=.8\textwidth]{figs/fig-wevi}
\caption[Screenshot of WEVI\xspace]{The WEVI\xspace interface}
\label{fig:wevi-screenshot}
\end{center}
\end{figure*}

\begin{figure*}[!htbp]
\begin{center}
              \includegraphics[width=.8\textwidth]{figs/fig-screenshot}
\caption[Screenshot of LAMVI-1\xspace]{The LAMVI-1\xspace interface}
\label{fig:lamvi1-screenshot}
\end{center}
\end{figure*}
  
LAMVI-2\xspace is the third in a line of visualization systems developed for analysis of word embeddings.  The first, WEVI\xspace~\cite{wevi}, was produced as a teaching tool to work alongside a (now-popular) tutorial on word2vec hyperparameter setting~\cite{rong2014word2vec} (see Figure~\ref{fig:wevi-screenshot}. While popular, WEVI\xspace was focused on extremely small language models and did not support model comparison. To test the impact of hyperparameters, the student would make a change and observe the effect on the neural network weights during training. Though useful for understanding how training happened and embeddings could be used to find close terms, the tool did not support realistic tasks. Nonetheless, we observed that end-users would iteratively vary the hyperparameters to understand their effect on the model.

While not intended for real-world use, WEVI\xspace did clarify for us that students, and likely developers, were not interested in vector weights for individual words. Rather, the relationship of words to each other was critical in determining whether a particular embedding was good. This focus on word \textit{relationships} has become a key guideline for our subsequent implementations.

LAMVI-1\xspace~\cite{lamvidemo,rong2016visual} was built to support real-world developer practice. The LAMVI-1\xspace interface (see Figure~\ref{fig:lamvi1-screenshot}) provided similar controls (panel A) to WEVI\xspace, though it used a full implementation of word2vec (allowing hyperparameters like window size to be controlled). Based on our experience with WEVI\xspace, we allowed LAMVI-1\xspace users to focus their analysis to specific ``anchor'' terms (in the example the input text is Jane Austen's ``Pride and Prejudice'' and the anchor term is ``looked'') and the relationship of this anchor to other terms. LAMVI-1\xspace allowed end-user to create compound vectors (e.g., \textit{elizabeth} $+$ \textit{bennet}) as well as displaying a `tracking list.' Tracked terms were those the end-user believed should (or explicitly should not) be related to the target term (a form of ground-truth labels). During training, LAMVI-1\xspace would display the distance of the two terms (anchor and tracking) in a ranked list that simultaneously allowed the developer to identify convergence. Similarly to WEVI\xspace, the LAMVI-1\xspace end-user could initiate training and observe the various visualizations dynamically change based on hyperparameter configurations.

LAMVI-1\xspace solved a number of problems with WEVI\xspace. By explicitly tracking the interesting aspects of the model--the \textit{interactions} between terms, rather than the terms themselves--the tool supported both easier debugging and also allowed the end-user to dynamically build ground-truth labels based on their observations of the data. In using LAMVI-1\xspace for our own projects and observing others' use, it became clear that one of the most common use-cases, comparing models, was not supported directly. Users of LAMVI-1\xspace would make changes to the hyperparameters to retrain the model but would quickly lose track of the behavior of previous configurations. LAMVI-2\xspace was specifically designed with this problem in mind.

\subsection{Design Guidelines}
Our experience with WEVI\xspace and LAMVI-1\xspace, as well as in model building allowed us to identify three main decision points: \textbf{(D1) decisions on which models to test}; \textbf{(D2) decisions on how models are evaluated}; and \textbf{(D3) decisions on which models to use}. Each decision requires the execution of multiple analytical tasks.

\noindent
\textbf{D1: Model Construction}--Given the large hyperparameter space and cost of learning and testing, a key decision point in model construction is where resources are best expended. This decision is strongly connected to determining which model to \textit{use} (D3) but may involve a broader set of tasks.

\begin{itemize}[noitemsep]
    \item \textbf{T1}: Identify unexplored hyperparameter settings. To avoid overfitting, the developer should be able to identify which portions of the hyperparameter space are unexplored.
    \item \textbf{T2}: Identify the correlation and interaction between hyperparameters and output metrics. A key type of analytical determination is the identification and mental modeling of those interactions that display trends. Specifically, the developer must know if tuning the hyperparameter value(s) \textit{in a specific direction} will produce better results.  
    \item \textbf{T3}: Identify satisficing hyperparameter setting ranges. As hyperparameters often lead to tradeoffs, a developer should be able to identify those models that satisfice their need (given evaluation metrics) and focus on those for construction and selection.
\end{itemize}


\noindent
\textbf{D2: Model Evaluation}--Model are principally selected, or trained, based on different metrics (i.e., pre-established benchmarks). These can include temporal performance measures (e.g., speed of training or execution), accuracy metrics (based on model-specific ground truth or an evaluation of the model's contribution to downstream applications), and other model features (e.g., memory footprint, power consumption, etc.). It is our goal for developers to be able to identify satisficing ranges for multiple performance metrics (T3). Additionally, because ground truth test-sets are often specific to the training data or generic (e.g.,~\cite{semeval}), they often require adaptation. When applying the model to new training data, the developer often needs to change the ground-truth. Put another way, the developer should not only be able to define which performance metrics to apply (with their given ground-truth labels), but also dynamically augment ground-truth labels.

\begin{itemize}[noitemsep]
    \item \textbf{T4}: Identify, and mentally-model tradeoffs in hyperparameters given different performance metrics.
    \item \textbf{T5}: Identify satisficing-models given global performance metrics.
    \item \textbf{T6}: Compare low-level model differences (i.e., word or word-cluster specific) to identify new examples to incorporate into performance metrics.
\end{itemize}

While we focus in this work on text embeddings, it is worth acknowledging that many of our ground-truth labels are `second order.' That is, there is not `correct' vector representation for any particular word. Rather, the user might want a pair of words to be similar ('close'), or some order on nearest neighbors, or may want a certain group of words to form a coherent cluster. Conversely, the user might have negative examples and want to exclude certain words from a cluster. The forms of ground-truth can vary significantly depending on task. The visualization interface must make it easy for the user to explore the possible candidates for establishing such ground-truth items (T6), and to manage them by creating, updating, and deleting individual items.

\noindent
\textbf{D3: Model Selection}--A final step in our process is to identify the model to deploy. This involves a number of the tasks described above (T2, T3, T5, and T6). In the act of isolating `good' models, and having understood their trade-offs and limitations, a decision can be made about the `best' model. This step also allows the developer to document their choice in a way that others can understand the decision.

\begin{itemize}[noitemsep]
    \item \textbf{T7}: Document model choice/selection for external audiences.
\end{itemize}
\section{LAMVI-2}
\label{sec-lamvi:system}

To support the tasks described above, LAMVI-2\xspace is composed of a number of interlinked views and interaction patterns. We briefly decompose the main elements (Figure~\ref{fig:lamvi2-screenshot}) in the context of the high-level decision, and low-level analytical tasks.

\subsection{Parallel Coordinates (panel A)}
Understanding hyperparameters and metrics is inherently a multivariate data analysis problem. There are quantitative, ordinal, and categorical hyperparameters, and similarly quantitaitve, ordinal and categorical metrics. The analytical tasks that involve comparing (T1-T5) inherently require multiple comparisons between these variables. Additionally, many key decisions in our workflow require filtering on multiple dimensions simultaneously (T6). The interactivity that is afforded by parallel coordinates fits well for the goal of identifying satisficing models. 

Parallel coordinate plots~\cite{inselberg1985plane} naturally fit as good mechanism for encoding multiple dimensions onto a 2-D plot. There are multiple reasons to adopt this form beyond the expressiveness of the encoding given our task. First, parallel coordinate plots are becoming familiar to end-users. Despite the learning curve, they are effective representations and are already being adapted in the context of model comparison. For example, Google's Vizier~\cite{vizier} and the open-source Ray Tune software~\cite{raytune} offer parallel coordinate plots. This use is somewhat unsurprising as one of the original case studies~\cite{detective} describes the use of parallel coordinates to trade-off parameters (of chip production system) against different performance measures. While there are likely better static representations, interactive documentation (T7) can be achieved by providing a pre-populated version of LAMVI-2\xspace. 

This view is attractive for our context as the encoding supports rapid identification of positive/negative relations (T2, T4) between data dimensions (based on slope). While a limitation of the technique is that, at any given time, each dimension can have at least two neighboring dimensions, but most parallel coordinate implementations (including our own) allow coordinates to be re-ordered. Since one of our requirements above is to facilitate efficient identification of relationships between model hyperparameters and performance scores, we opt to use parallel coordinate plots to depict both kinds of variables. \textit{To begin,} hyperparameter dimensions are placed on the left and output/performance dimensions are placed on the right (with the heatmap in the center). This reflects the natural ordering of parameter to model to outcome but elements may be reordered if necessary. Unexplored hyperparameter spaces (T1) are salient as `empty' areas in the parameter axes.

Our implementation supports the standard parallel coordinates interaction patterns.  By brushing along a specific axis, the end user can \textbf{filter} the data to narrow down the number of lines being highlighted to a smaller set whose crossing points with the axis are within the brushed range. To focus, the brushed region can be rescaled or moved through cursor interactions. Double clicking an an axis label with cause the data to \textbf{flip} for that dimension. This feature may help the user reduce the number of line crossings between axes and also allows for consistent mental models. Other features such as colors and bundling can be enabled, but are not done so by default. Together, these features enable the filtering necessary for identifying good models (T3, T5).

While LAMVI-2\xspace can support any number of loaded models, realistically we do not expect more than 30-40 (and that depends significantly on model training costs). In part, an appeal of LAMVI-2\xspace is that we can `seed' the system with random points in the hyperparameter space and allow the end-user to narrow this space by visual inspection.

\subsection{Embedding Explorer (panel C)}
Embedding words in a 2D space is a natural way to inspect models. In particular, for task T6, inspecting the words in this way allows the developer to identify possible issues with the model. More specifically, approaches such as t-SNE~\cite{van2008visualizing,wattenberg2016how} and PCA reduce multi-dimensional vectors to two dimensions, allow the developer to inspect the relationship of terms both in magnitude and distance. For better or worse, such representations have become standard for both interactive and static visualizations (e.g., Google's Tensorboard: https://www.tensorflow.org/programmers\_guide/embedding). The benefit of projections is that the viewer is that related terms are placed near each other in the low-dimension projections. Gestalt properties of this projection make it easy to find patterns and detect outliers. For neural-net based embeddings, a further (argued) benefit is that the  \textit{relative directions} of terms has `meaning.' 

Traditional use of 2D projections involves the placement of all the words in the model or some large top-$k$. Unfortunately, with many words, the visualizations can become difficult to navigate. More critically, there are more opportunities with large differences for words to be placed near each other as a consequence of the dimensionality-reduction and not due to `true' distance~\cite{chuang12}. Projecting all the terms has the appeal that it follows the `overview first, filter, details on demand' mantra. However, it is unclear to us that the particular `overview' provided benefit for the tasks we outlined. 

While LAMVI-2\xspace can project all terms in the model, we employ a different default strategy. Instead of a view of the entire set of words, the LAMVI-2\xspace Embedding Explorer shows a projection anchored on a specific query 'term' and a set of its nearest neighbors. As different models in the parallel coordinate view are selected or loaded, the projection updates. LAMVI-2\xspace uses t-SNE for projection. The interactive explorer supports basic pan-and-zoom functionality to explore the projection. The 'focus' term is made salient by using a larger font size (and is orange). Terms that are labeled (e.g., antonyms or synonyms) are also made salient by color coding and increased font size. The developer can easily find term pairs that are unlabeled as well as the clustering of labeled examples (e.g., are the antonyms all near each other?). In the event that these labeled terms are not part of the k-nearest neighbors (i.e., an antonym \textit{could} be very far), LAMVI-2\xspace will add those terms to the projection.

In LAMVI-2\xspace, the developer can perform tasks such as filtering, sorting, creating new models, or switching to old ones. Terms in the embedded view are animated so the developer can observe the impact of the interaction on the spatial relationship of the terms. A secondary benefit of focusing on a small set of terms is that these animations can be visually tracked more easily. Terms that are no longer `near' to the query are animated out, while those that now meet the criteria are animated in. Label placement also dynamically changes based on the vector representations of the selected model. The t-SNE optimization algorithm randomly initializes the embedding vectors and often the labels temporarily shrink to the center of the space, making it difficult to track terms as they move. To counter this, we first perform 150 iterations of the t-SNE algorithm in the background to find a better starting state for the animation.

As we describe below, the t-SNE projection is in some sense a `dual-encoding' of term distances (though it uniquely displays direction). We have found that a useful interaction pattern is to support a focus on word pairs or sets (the smallest 'unit' of useful analysis for word embedding). By clicking to highlight two or more words, and switch focus between models, the developer can track the relationship of a small set of terms between model variants. Because highlights are preserved between views, the user can both visually track where the words are relocated and acquire additional spatial awareness of changes in the embedding space across different models. This allows them to identify patterns that are consistent (building a mental-model of the models' behavior) as well as finding models that are `outliers.'
. 

\subsection{Nearest-Neighbor Heatmap (panel B)}
A unique feature of LAMVI-2\xspace is the Nearest-Neighbor Heatmap (panel B). The heatmap is connected both to the parallel coordinates plot (models are rows) and the t-SNE projection (words are columns). Words are both visually linked (through a line) as well as through linked brushing. Mousing over a word will highlight the appropriate column.

The heatmap is specifically targeted at task T6, comparison of low-level model differences. The encoding allows us to compactly display data with a high information-density summary~\cite{munzner2014visualization}. Each cell corresponds to the distance between a `query' and a sampling of related terms.  For example, in Figure~\ref{fig:lamvi2-screenshot}, the current query term is ``legitimate'' (also displayed as the first column).  The second column corresponds to the term ``dubious.''   

As the number of nearest neighbors (i.e., columns) and models increases it becomes difficult to identify patterns and outliers in the data.  We use a combination of visual encodings and interaction techniques to help the user navigate this information. Patterns of particular interest are those columns where models agree on the distance between the query term and the other word.  For example, the models seem to agree that `valid' is similar to 'legitimate.'  This manifests as a column of a single color. A second pattern of interest are those when there is either a single outlier in a column or a bifurcation, where some models place the term 'close' (blue) and some 'far' (red). For example, five of the seven models place 'moderate' (4th column) far and two place it close. Other patterns delineate outliers and clusters at the model level. By looking at multiple rows, the end-user can find `clusters' of models that generate similar embeddings and those that are distinct or anomalous.

To increase the number of columns that can be simultaneously display we designed a two-mode display for columns inspired by Tablelens~\cite{rao1994table}. This increased the amount of data displayed but still supported the pattern finding task. Under the ``normal'' mode, columns in the matrix are compressed to a few pixels (for example the column for 'valid'), minimizing space but still allowing the viewer to identify patterns across all models. Under the ``zoomed-in'' mode, the columns in the matrix are displayed with a larger width with labels displayed below the bottom row. By default, the top-K nearest neighbors of the query word will have their corresponding columns zoomed-in (K starts at 15) . As the user interacts with the other parts of the interface, pertinent columns will be adjusted to zoom-in. This focus switching mechanism optimizes the efficiency of space usage while allowing the user to quickly navigate through the most relevant columns. Other heat-map `modes' can put columns with inter-model agreement in the 'normal' (low-pixel) and low-agreement terms in the zoomed-in state.


Although the heatmap is integrated into the parallel coordinates display, the vertical ordering of the rows does not necessarily correspond to the same high-low ordering. While this breaks the encoding, it also allows us to dynamically reorder both the rows and columns of the heatmap to emphasize patterns. Line segments connect the heatmap rows to the parallel dimensions coordinate on the left and right ensure that the viewer can `follow' a particular model's encoding.  When models are filtered in the parallel coordinate dimensions, the corresponding rows are highlighted (those retained by the filter) or are `grayed out.' In our experience, we have not found the `interruption' of the heatmap in the middle of the parallel coordinates plot to be disruptive (visually or semantically). However, we note again that  parallel coordinates axes that are classically 'dimensions' of the encoding can be grouped together (i.e., hyperparameters and metrics), placing the heatmap to the side. This may be a desirable configuration for some end-users or for specific model types.

LAMVI-2\xspace supports a variety of \textbf{clusters} and \textbf{orderings}. As the primary focus for our heatmap is for the identification of model disagreement and clusters, the clustering of models can simplify the process greatly. Figure~\ref{fig:lamvi2-four-matrix} illustrates four different configurations for the same data.  The default sort order places rows in the order the models were loaded.  To minimize edge-crossings, the heatmap can also be sorted by model hyperparameters (Figure~\ref{fig:lamvi2-four-matrix}c and d). In this case, LAMVI-2\xspace takes into account the value of the hyperparameter dimension immediately before (in this case 'retro') or the performance metric immediately after ('triples') in deciding the order of the rows. Because the coordinates can be reordered, the user can move a metric dimension next to the heatmap and order by this value. This will cause the rows of the heatmap to reorder to correspond with the metric and allow the user to inspect the properties of 'good' versus 'bad' models.

\begin{figure*}[t]
\begin{center}
\centerline{\includegraphics[width=.8\textwidth]{figs/fig-four-matrix}}
\caption[Four sorting modes of LAMVI-2\xspace heatmap.]{Four sorting modes of the LAMVI-2\xspace heatmap: (a) default mode -- models are sorted by loading order; (b) cluster-sort mode -- models and words are sorted according to the dendrogram of hierarchical clustering; (c) hyperparameter-sort mode -- models are sorted by the right-most hyperparameter in the parallel coordinate plot; (d) metric-sort mode -- models are sorted by the left-most performance metric.}
\label{fig:lamvi2-four-matrix}
\end{center}
\vskip -0.2in
\end{figure*}

A more sophisticated order is created through hierarchical clustering. We achieve this by running a hierarchical clustering algorithm on the heatmap data (row-wise). The order for the rows corresponds to the order used to visualize the dendrogram. The effect is that models that have similar distance properties are placed near each other. As illustrated in Figure~\ref{fig:lamvi2-four-matrix}b, patterns become apparent (e.g., the agreement in the last three rows) or that the top row is an outlier. Clustering in this way has the further advantage of making salient specific cells that are different (i.e., within-cluster outliers).

\begin{figure*}[t]
\begin{center}
\centerline{\includegraphics[width=.8\textwidth]{figs/fig-heatmap-filter}}
\caption[Filtering of rows in LAMVI-2\xspace heatmap via parallel coordinate brushing.]{Heatmap rows can be filtered and highlighted by brushing axes on the parallel coordinate plot. This allows a focused comparison among models that meet a user-specified filtering criterion.}
\label{fig:heatmap-filter}
\end{center}
\end{figure*}

We retain the \textbf{filtering} interaction pattern of the parallel coordinates inside of the heatmap (see Figure~\ref{fig:heatmap-filter}). Whenever the user applies filtering or brushing to one of the dimensions in the parallel coordinate plot, the corresponding rows in the heatmap will be highlighted. The non-highlighted rows are still visible but will be applied with a semi-transparent mask. Conversely, the end-user can select rows in the heatmap to filter one or more models. 

\subsection{SPLOM}
A fourth panel (not pictured) displays a Scatter Plot Matrix (SPLOM). The grid displays pairwise correlations between all hyperparameters and metrics (each point is a model instance). Because it is sometimes difficult to quantify correlations between dimensions in parallel coordinates directly, the SPLOM view provides a more accurate and comprehensive encoding of this data (further supporting T1-T4).  The SPLOM is interactive and linked to the other displays. As the end-user filters (by drawing selection rectangles), models are dynamically filtered from all the views.

\subsection{Inspecting Models and Training}
Recall that a key goal for LAMVI-2\xspace is to facilitate the creation of training data (T6). The end-user may issue queries through a text-input dialog. This will refresh the embedding view to include the query string and nearby terms. When the user is interested in the similarity and neighborhoods of a \textit{pair} of words simultaneously, she may directly query the two words together. This allows her to more easily mark good cases and bad cases as more words are displayed (neighbors of both terms) that likely match the relationship of the query pair.

Our current implementation also supports inspecting the emergence of linguistic regularity captured by the model. The user may enter queries like ``king --queen woman'' and observe how the desired candidate, ``man'', evolves. The user may also inspect the activation levels of the hidden units given all three words as context. The distances between query and neighbors is shown automatically in the heatmap.

\begin{figure*}[t]
\vskip 0.2in
\begin{center}
\centerline{\includegraphics[width=.8\textwidth]{figs/fig-tooltip}}
\caption[Creating user-customized ground-truth in LAMVI-2.]{Multifunctional tooltip in LAMVI-2 heatmap: (a) compact tooltip shows the similarity of the word pair under focus; (b) expanded tooltip allows the user to assign a label to the word pair, creating user-customized ground truth.}
\label{fig:lamvi2-tooltip}
\end{center}
\vskip -0.2in
\end{figure*}

Clicking on a cell in the heatmap will show a dialog box by which pairs can be `labeled' (see Figure~\ref{fig:lamvi2-tooltip}). Though we can support multiple labels, the default prototype categories include \textit{synonyms} and \textit{antonyms}. As labels are created, these are dynamically fed as labels to downstream evaluation `modules.' 

A particular workflow that can be executed is the identification of models that correlate with outliers. When the user explores the embedding and discovers words that are inappropriately placed in the neighborhood of a query she can click on the word label to automatically zoom into the associated column in the heatmap. The user can scan down the entire column and see if the score is consistently spuriously high. She can then filter on those rows to see if certain hyperparameters result in the undesired score.

\subsection{Implementation Details}
The back-end of the LAMVI-2\xspace system is written in Python and was designed with extensability in mind. The front-end was programmed using D3~\cite{bostock2011d3}. A model developer can configure and evaluate models programmatically. These models can then be loaded into the interface. We currently provide basic command line tools that will accept a high-level configuration (e.g., one indicating the space of hyperparameters to explore) and builds a model for each point in this space. A web-based tool allows for generation of a JSON configuration file that can be fed to the command-line pipeline.  The tool is currently configured for multi-scale grid search as well as random search~\cite{bergstra2012random}, but can be readily adapted to other hyperparameter optimizers~\cite{NIPS2011_4443,NIPS2012_4522}. Labeled examples created by the developer are stored in a text-based label file. The entire `state' of a particular experimental run can be preserved and provided to other developers as documentation (T7).

\section{Case Study}
\label{sec-lamvi:casestudy}

To understand how LAMVI-2\xspace could be used in a realistic situation we replicated the work by Mikolov et al.~\cite{mikolov2013efficient} (as implemented by Rehurek for the GenSim package using the \textit{text8} corpus~\cite{rehurek}) using our tools. In part, we selected this case as both the the original Mikolov et al. paper and the implementation by Rehurek report their parameters without specific discussion of trade-offs and optimality. We demonstrate both that it is possible to identify the `optimal' hyperparameters but also that we can more fully describe the effect of these parameters on performance. An advantage of using LAMVI-2\xspace is that it becomes easier for the author to identify and report (task T7) on patterns in the data that may provide guidance for others using new neural architectures. As the architectures, and associated hyperparameter spaces, become more complex, such guidance is critical.

To build a set of comparative models we used hyperparameter ranges that were expansive, but included the `optimal' values reported in the studies above. In the interest of replication, merging with other models and retrofitting were not used during training. Hyper-parameter ranges were: Word Vector Size: 100--400, increments of 50; Skip-Gram architecture: Both with/without; Hierarchical Softmax: Both with/without; Window size: 3--7, increments of 2; Negative samples: 5--20, increments of 5. 

\begin{figure*}[t]
\begin{center}
\centerline{\includegraphics[width=\textwidth]{figs/fig-LAMVI-size.png}}
\caption[Inspecting size hyper-parameter in LAMVI-2\xspace]{A view of the LAMVI-2\xspace interface for the word2vec case study. The end-user has targeted the word 'sufficient' and has filtered to those models containing skip grams, a window of 5, and 15 negative samples.}
\label{fig:LAMVI-size}
\end{center}
\end{figure*}

Figure~\ref{fig:LAMVI-size} displays the LAMVI-2\xspace interface with the different models in this space active. The analogy metric is displayed on the right as is training time. When filtering on the maximal accuracy we achieve results comparable to Rehurek, including roughly 28\% accuracy on the analogy task for the model with settings \{size: 200, skip-gram, negative sampling w/ 15 samples, window size: 5\}. More critically, we observe trade-offs that were not described in the original paper, but now understood in the outside literature. For example, the use of both skip-gram and hierarchical sampling results in increased accuracy at the cost of a significant increase in training time. 

We find that these optimal parameters are slightly different than those reported by Mikolov (dimension: 300, window size of 7). This may be partially due to the use of \textit{text8} but demonstrates the need to perform algorithmic tuning when changing corpora. Additionally, given the time to train different sets of models, performance improvements can be achieved through more fine-grained tuning of hyperparameters. However, we do find that the granularity of the adjustments correlates with the granularity of the improvement. The SPLOM visualization further shows decreasing returns for more training time relative to analogy performance.


\section{Evaluation}
\label{sec-lamvi:experiment}

Evaluating tools for deep learning models can be difficult, as they require significant participant expertise. Realistic scenarios are similarly complex, as a local evaluation of the embedding may not be sufficient. For example, one might create a classifier for synonyms and use this in a downstream information extraction (IE) system. An optimization in the sub-module (the synonym classifier) may not mean optimal behavior in the IE system. To simulate this, we created a two-stage task for polarity classification of IMDB reviews (i.e., are reviews positive ore negative). Because we may not know the polarity of every word in the document, one approach is to find synonyms to known positive terms and label these as positive and identify antonyms and label them as negative (conversely, synonyms of negative polarity terms can be labeled negative).  This scenario naturally induces two metrics: the accuracy in identifying synonyms/antonyms and the accuracy in classifying reviews.  Our experiment focuses on how well (and quickly) participants can achieve an optimal hyperparameter-configuration using LAMVI-2\xspace versus baseline ``command line'' style tools.

\subsection{Task and Model}
For our task, we used the Cornell IMDB movie review corpus~\cite{pang2002thumbs} as the training corpus (100k reviews, 50k of which have positive/negative sentiment labels). The input feature to this classifier is the average word embedding (i.e., the centroid of all words) in that document.  The intuition is that if the embedding is `good', positive synonyms will be grouped together and negative synonyms will be grouped together in space. If the document has more positive terms, the average vector will be in a `positive' location.  We train a logistic regression classifier for this ``downstream'' application.   We calculate $f_A$, as the accuracy score for the sentiment classifier using 50k documents (equal partition, positive/negative). The participant's goal is to provide the classifier with a good embedding. 

As described above, we may only have sentiment scores for \textit{some} of the words in the corpus.  By creating an embedding that places synonyms of a given polarity together, it becomes possible to provide scores for all words.  We challenge our participants to tune a word2vec skip-gram model~\cite{mikolov2013distributed} to provide such an embedding. We provide the participants with `synonym-antonym triples'--three words where one word is an `anchor' (e.g., \textit{ghastly}), one is a synonym (\textit{awful}), and one is an antonym (e.g., \textit{delightful}).  We also provide a few examples (specifically five) to the study participant (we call this the 'training set') with the expectation that they add additional examples.  Thirty-five example triples are withheld from the participant (the 'test set') and we evaluate whatever embedding configuration they pick against this set. The 40 triples (5 train, 35 test) were generated by consulting external lexicons (SentiWordNet~\cite{baccianella2010sentiwordnet}, \textit{Oxford American Writer's Thesaurus}~\cite{lindberg2012oxford}, and an online thesaurus dictionary.\footnote{\url{http://www.thesaurus.com/}.}). We specifically selected words with obvious polarity (top 10\% of SentiWordNet) so it was clear from our examples whether the sense was positive or negative. Only words that occurred between 150 and 250 in the IMDB dataset were used (to ensure sufficient data).

The \textit{triples} score, $f_T$, measures how well a model distinguishes synonyms from antonyms. It is calculated as the average difference between the similarity difference of a set of synonym-antonym triples:
\begin{equation}
f_T = \frac{1}{N}\sum_i^N(\cos(q_{i,A},q_{i,B})-\cos(q_{i,A},q_{i,C}))
\end{equation}
where $q_{i,A}$ and $q_{i,B}$ are the vectors of a pair of synonyms and $q_{i,A}$ and $q_{i,C}$ are the vectors of a pair of antonyms.

We use the average of two metric scores, triples and accuracy, to measure the performance of the model. For both $f_T$ and $f_A$, the user can access the corresponding scores assessed on the training set in the interface (i.e., for each parameter configuration a model was generated, and for each model accuracy was calculated for both the triple task and sentiment analysis task and provided to the participant). The final evaluation of the model was to be assessed on the test set.

\subsubsection{Parameter Space}
To control the scope of the experiment, we set some of the parameters to be non-adjustable by the user (specifically, we fixed the frequency downsampling parameter to be $1\times10^{-4}$, 5 iterations, and 5 negative samples). We allowed the user to select whether to have the vectors pretrained on an external corpus (the `lockf' parameter and whether to use an external lexicon to retrofit the vectors (the 'retro' parameter). Pretraining provides a `starting' point for training, for example based on Wikipedia, and is used when the domain corpus is too small. Retrofitting happens after the model training and is shown to be able to improve the quality of the vectors~\cite{faruqui2014retrofitting}. In our case, the paraphrase database (PPDB)~\cite{ganitkevitch2013ppdb} is used as the semantic lexicon for retrofitting. The database contains 8 million English lexical paraphrases (e.g., \emph{jailed} and \emph{imprisoned}). 

The parameters that the participant could vary included word vector \textbf{size}; sliding \textbf{window} length; whether vectors pretrained on Wikipedia should be used (\textbf{lockf}, which could be varied between 0 and 1 to determine the weight of the pretrained vectors on the final vectors, or -1 to be disabled); the initial learning rate for gradient descent (\textbf{alpha}); \textbf{retro}fitting, indicating whether to use an external dictionary to augment the vectors after training (ranging between 0 and 2, where a larger number indicates less external influence and a -1 disabling this step). While these parameters may appear overwhelming, we emphasize that they are realistic decision points encountered by developers.

\subsection{Procedure}
Participants answered a brief questionnaire about their experience and familiarity with machine learning and word embedding models. Each participant was randomly placed into a baseline condition or one in which they used LAMVI-2\xspace. 

The \textit{baseline} emulated the command line tools currently available, and allowed the participant to enter model parameters and see the evalution metrics. The baseline also allowed the participant to enter `queries' into their model to see similar terms. Baseline participants could add triples to their own `test' set. While this is a `weak' baseline in that we did not offer any visualization tools, the command-line tools were familiar to many of our participants and were part of the `standard workflow.' Participants in the LAMVI-2\xspace condition were given access to the full interface (with the exception of the SPLOM view that had limited use given the constraints below).

Models covering a broad parameter range (capturing all likely selections by the participant) were pre-generated.  Given the time constraint of the experiment, this would allow a participant to load a range of models without waiting for training.  Two models were preloaded into the experimental interface. Participants could load at most 13 more models at a time (for 15 total). This was done to create some `cost' to loading additional models as otherwise the participant could simply load all models and select the optimal one. In total, 374 models were pre-trained for the experiment.

After a three minute introduction using a prepared script, the participant was allowed five minutes to experiment with the interface and ask questions. Logging of interactions began and recorded which models were loaded, filtering operations, search, etc. The participant selected one model as the final submission. Finally, Each participant answered a questionnaire with questions taken from the system usability scale (SUS) as well as a few open-ended interview questions.

\subsection{Results}
Twelve subjects were recruited and received \$15 for participating. All were graduate students in computer science or a relevant major at a large public university with some machine learning experience. From our pre-survey, 25\% of the subjects reported to be beginner level with regard to machine learning, and 75\% self-identified as intermediate to expert level. One third of the subjects had learned about word embeddings but had not used them in practice; one third used them as a black-box component in a project; and one third had done at least a research project on word embedding models.

Table~\ref{table:lamvi2-eval} summarizes the results for the comparison between LAMVI-2\xspace and the baseline system. On both the training set and the test, the average performance of the models in groups with the baseline and the LAMVI-2\xspace system are reported. LAMVI-2\xspace users consistently outperformed the baseline system under the metrics of both $f_T$ and $f_A$.

\begin{table}[htbp]
\centering
\caption[LAMVI-2 Evaluation Results]{LAMVI-2 Evaluation Results}
\label{table:lamvi2-eval}
\begin{tabular}{c|c|c|c|}
\cline{2-4}
\multirow{2}{*}{} & \multicolumn{3}{c|}{Training Set} \\ \cline{2-4} 
 & triples $f_T$ & accuracy $f_A$ & average \\ \hline
\multicolumn{1}{|c|}{Random} & 0.202 & 0.751 & 0.477  \\ \hline
\multicolumn{1}{|c|}{Baseline} & 0.277 & 0.795 & 0.536 \\ \hline
\multicolumn{1}{|c|}{LAMVI-2\xspace} & \textbf{0.295} & \textbf{0.814} & \textbf{0.554}\\ \hline
\multirow{2}{*}{} & \multicolumn{3}{c|}{Test Set} \\ \cline{2-4} \hline
\multicolumn{1}{|c|}{Random} &  0.159 & 0.745 & 0.452 \\ \hline
\multicolumn{1}{|c|}{Baseline} &  0.203 & 0.791 & 0.497 \\ \hline
\multicolumn{1}{|c|}{LAMVI-2\xspace} &  \textbf{0.223} & \textbf{0.810} & \textbf{0.517} \\ \hline


\end{tabular}
\end{table}

\begin{figure*}[htbp]
\begin{center}
\centerline{\includegraphics[width=\textwidth]{figs/fig-curbest-models}}
\caption[Model performance improvement process.]{Model performance improvement against the number of models loaded.}
\label{fig:curbest-models}
\end{center}
\end{figure*}

Figure~\ref{fig:curbest-models} compares LAMVI-2\xspace with the baseline system in terms of the performance score ($\frac{f_T+f_A}{2}$) for the current best performing model at each interaction step (i.e., how many models were tested). We note that LAMVI-2\xspace participants were able to improve model performance at a faster rate. 

Participants using LAMVI-2\xspace generated 16 synonyms and 13 antonyms (on average) using the query functionality. We compared the performance of the user's best-selected model on each of their own synonym/antonym selections (simulating ground-truth construction) with the performance of their model choice on our hold-out set. The Pearson correlation across all participants (averaged) .599 and ranged from .509 to .774 (all statistically significant at $p<0.0001$). This indicates that participants `flagged' synonym-antonym triples that exhibit similar similarity/dissimilarity patterns as our preselected triples in the held-out dataset. Additionally, a higher correlation ($r$) generally means that the final user-selected model has a better task performance overall.

\subsection{Additional Observations}

\textit{Navigation Strategies}--Participants in the LAMVI-2\xspace condition developed specific strategies to guide themselves to navigate through the parameter space. They noted that the parallel coordinates plot helped with this search, especially for the parameters that they were unfamiliar with.  For example, during the post-study interview, user P3 mentioned that ``I have a search algorithm. I start with a default setting, and vary each parameter at a time, and see if I can get the biggest improvement. Then I fix the one with the biggest improvement, and vary other parameters one by one.'' User P7 said, ``I first applied the prior knowledge that I have. Then I try the medium values for the options that I don't know, and then test the extreme values deviated from the medium values.'' Similar to these two users, most other users developed a variant of ``greedy search'' or ``gradient descent'' algorithm for parameter searching. 

For the LAMVI-2\xspace participants the parallel coordinate plot was important to these strategies. P5 commented, ``It was tremendously helpful to use the parallel coordinates to tune down the scope of searching. I first look right to the pick the performance I want, and look left to confirm the intuition. I confirmed my intuition that lockf and alpha and retro should be locked to a specific value, and window and size can be fine-tuned.''

\textit{Outlier Model Detection}--The nearest-neighbor heatmap appeared useful for helping identify outlier \textit{models}. Whenever the user loads a new model, the system automatically uses the new loaded model to find the nearest neighbor words to the query and updates the heatmap. When the user loads a model that results in significant disagreement with previously loaded models, there is a dramatic change on the heatmap which is very salient. As user P3 mentioned, ``When I turn the lockf up, I suddenly see the heatmap change all over, and this helps me know that something bad happens. So I am able to quickly recognize that and revert the change.''

\textit{Confirming Intuitions}--While all our participants were able to make use of the LAMVI-2\xspace interface we found that those with high familiarity with embedding models used it in part to confirm intuitions. For example, P1, P2, and P7 believed the learning rate (from prior experience) should not be very big. By targeting \textit{alpha} they were able to obtain substantial improvement on both $f_T$ and $f_A$. Participants P2 and P5 mentioned that they knew \textit{window} cannot not be too big or too small and focused on tuning that parameter towards the middle region (4--8) and noted the gradual improvement on the performance (specially with $f_A$). Several participants in the LAMVI-2\xspace group reported that being able to use the parallel coordinates plot helped them confirm such intuitions quickly.

\textit{Requested Features}--Multiple participants requested the ability to remove loaded models. This was surprising to us as we thought end-users would like to see models that did not work so they could avoid that area of the parameter space.  However, participants noted that these discarded models often introduced clutter especially in later stages of the workflow when they had narrowed down their search space and were focusing on debugging specific models (e.g., when they were using the query functionality). In the experiment, multiple parameters had non-linear effects on the performance. For example, the two extremes of \textit{lockf} and \textit{retro} had similar effects, while the middle-range values of the two parameters cause a linear change on the performance metric. P5 commented that this non-linear pattern was hard to identify with the parallel coordinates without any external knowledge. Augmented parallel coordinates, such as those with overlaid histograms, may alleviate this issue. The SPLOM view, which participants did not have access to, may also be useful.

\textit{Interface issues}--Most participants in our study were unfamiliar with parallel coordinates and found the brushing interactions to be unclear. We suspect that this is a common issue when learning parallel coordinates, but training and adding visual hints to the interface may help. P3 and P11 found the interactions of flagging word pairs as synonyms and/or antonyms to be cumbersome. Currently, to flag a word pair, the user has to find the target word in the overhead embedding explorer, and follows the connector to the corresponding column in the heatmap and click, and then in a popped-up window select the associated option. This interaction pattern can be simplified by allowing the user directly click on words in the embedding explorer to flag them.



\section{Discussion and Limitations}
\label{sec-lamvi:discuss}


We constructed LAMVI-2\xspace for extension. However, we focused our implementation in a number of ways for the first iteration.  One specific issue is that only one `architecture' at a time is supported. There are many other embedding models including GloVe~\cite{pennington2014glove}, DeepWalk~\cite{perozzi2014deepwalk}, and LINE~\cite{tang2015line}. Not all utilize the same parameter space and an interesting question is how to allow the end-user to compare models which use very different parameters. A simple solution may be to allow parameters values to be ``null'' for models to which the parameter does not apply. Other, more complex, parallel coordinates configurations might also be an alternative~\cite{claessen11}. We also note that while we have adopted the dimensionality-reduction style visualization for the embeddings these sometimes lead to `tea leaf reading.' Alternative representations may yield more interpretable results~\cite{chuang12}.

LAMVI-2\xspace is currently focused on \textit{word} embeddings. There are many other model types that might benefit from the exploratory features of LAMVI-2\xspace, but various adaptations to the visualizations would be necessary. LAMVI-2\xspace may be most readily adaptable for other embedding types (e.g., document/paragraph or bimodal~\cite{allamanis2015bimodal,le2014distributed}) as we can show other kinds of items in the t-SNE/PCA views. The workflow of inspecting vector similarity, vector interaction, and tracking the ranks of watched candidate items could be the same.  Other text-related architectures, such as those for memory networks~\cite{karpathy2015visualizing} that ``generate'' sentences would require other changes to the visualization panels. Similarly, image or audio related architectures would also require data-specific adaptations but we believe model comparison through the parallel coordinate views may still be useful in these contexts. In these cases the heatmap view may be adapted to show nearest neighbor images, salience maps, or pixel masks.

\section{Conclusion}
\label{sec-lamvi:conclude}

In this paper we describe LAMVI-2\xspace, a visual tool for comparing word embedding models. LAMVI-2\xspace is specifically focused on three key decision points for machine learning model developers: (1) determining good hyperparameters for model testing, (2) find good examples for evaluations, and (3) selecting optimal ('satisficing') models. LAMVI-2\xspace is designed to support these decisions through a focus on hyperparameters and their relationship to the model internals and performance metrics. LAMVI-2\xspace combines different visual panels in an integrated view that supports these tasks. By extending a parallel coordinates display, LAMVI-2\xspace allows the user to `pivot' between tasks to identify satisfactory hyperparameters. 

Unlike automated hyperparameter optimization algorithms, the interface allows the user to make decisions about how different metrics should be traded off. Additionally, the tool allows the end-user to isolate patterns in the embedding space and focus their attention on desirable models.  We demonstrate that LAMVI-2\xspace can be used to gain insight into the relationship between hyperparameters.  Using a realistic experiment we also show how LAMVI-2\xspace users can more rapidly identify these optimal configurations.

See \url{http://lamvi.info} for live demos.

\section*{Acknowledgements}
We dedicate this paper to Xin Rong. It is based on an unpublished chapter from Xin Rong's doctoral dissertation. It is an important contribution that we felt should be shared.