\section{Background}
\label{sec:background}


There are three key concepts that support the contributions of this paper: performance analysis, calling context profiles, and a performance tool using such profiles (i.e., Hatchet). In this section we provide an overview of the three concepts.


\subsection{Performance Analysis}

{\em Performance} describes how well an application executes on computing resources, typically in terms of metrics such as time-to-completion or efficiency of resource use. For critical applications run on High Performance Computing (HPC) resources,  also known as supercomputers, poor performance limits the problem size or level of detail that can feasibly be computed by an application. 

Performance analysis is a key step in optimizing an application's performance. Developers collect performance data to support their analysis. The most simple performance metric is the total execution time of a given application. More detailed metrics can be used to capture performance of specific regions in the application code (e.g., functions). 



\subsection{Calling Context Profiles}

A common form of performance data is a {\em profile}. A profile accumulates metrics associated with regions. Different kinds of profiles aggregate or disambiguate regions. For example, a {\em flat} profile will accumulate metrics for each function. A more detailed profile may accumulate metrics for each {\em call site}---a particular function call at a line of code---or at a {\em calling context}---the chain of calls leading to a particular function call. The calling context is also referred to as a {\em call path}.

Calling contexts can be structured into a calling context tree (CCT), which is a prefix tree of all calling contexts. Each node in the CCT represents an invocation of a particular code region with a unique calling context. Each calling context is represented as a path from the root of the tree to a given node. CCTs are a popular structure for collecting profile data because they allow performance analysts to examine how the behavior of a code region changes based on how that region is invoked.

Two common metrics in CCTs are inclusive time and exclusive time. Inclusive time refers to the total time required to complete a function call---including the time for all the functions the original invokes. In other terms, it is the total time of the entire calling context subtree. Exclusive time is the time for the function without its subtree.





\subsection{Hatchet}

Hatchet~\cite{bhatele2019hatchet} is an open-source library for CCT analysis. It provides a canonical data model that can be used to read, represent, and index data produced by several HPC profilers. In Hatchet, the CCT is stored as a {\em GraphFrame} object, which build on pandas~\cite{reback2020pandas}, thereby allowing performance analysts to leverage modern data analysis ecosystems.










Hatchet provides numerous operators for GraphFrame data. For tree comparisons, Hatchet supports creating new GraphFrames representing the intersection or union of multiple trees. These operations can create derived metrics calculated by arbitrary functions of the metrics from the original trees, such as {\em speedup} (i.e., the ratio of timing data between two GraphFrames for each node) and {\em imbalance} (i.e., the variance across trees).

Hatchet also features a call path query language~\cite{brink_usability_nodate} that enable users to study GraphFrame data, including CCTs, through the creation and deployment of complex filters. Patterns of call paths, also called query paths, comprise of a list of abstract graph nodes and are fed into the filters to extract knowledge on performance. 











\section{Conclusion}
\label{sec:conclusion}

We conducted a design study investigating a hybrid visualization-scripting workflow in the context of performance analysis of calling context trees. Following design study methodology and criteria for rigor, we carefully considered the division of visualization and scripting tasks and used that analysis to design a notebook-embedded interactive visualization. We evaluated the resulting notebook-embedded visualization with seven performance analysts, validating the design for our identified tasks and demonstrating interest in further use of the overall tool and workflow. The variety of queries expressed by our participants and their interest retrieving queries from the visualization further support the benefits of this workflow.


We reflect on the ramifications of the notebook-embedded visualization-scripting workflow, comment on the merits and limitations of the visualization we developed, and provide guidance for future visualization-scripting designs and design projects. Our work suggests the visualization-scripting workflow embodied by the notebook-embedded visualization shows promise and reveals the need for further design work regarding how users will perceive and convey transitions between both visualization and scripting states.



\section{Interactive, Embedded CCT Visualization Design}
\label{sec:cctvis}

\begin{figure*}
    \centering
    \includegraphics[width=\textwidth]{figs/tree_with_letters.png}
    \caption{Embedded CCT Visualization. The main view (a) is a pannable, zoomable node-link diagram with two node encodings---color and size. Selected nodes (b) are shown with a thick border. Their details are shown in a floating table (c). Features like mass-pruning, changing encoding metrics, and exporting queries are available through the menus (d). }
    \label{fig:vislayout}
\end{figure*}


We designed the visualization through an iterative process with weekly meetings sharing design documents and prototypes. Our design was informed through our interviews with frontline analysts, feedback from the Hatchet project manager and team, and our resulting task analysis. In addition to these design inputs, there were two other constraints we sought to fulfill. First, {\bf [C1]}---the visualization would be embedded in Jupyter notebooks as this is an intended environment by the Hatchet team and on in which they distribute examples of their new features. Second, {\bf [C2]}---the visualization needed to be intuitive as the Hatchet project manager had observed target users would not take time to learn an unfamiliar visualization. 

We explain the key elements of our interactive, embedded calling context tree visualization. \autoref{fig:vislayout} shows an overview of the tool. 


\subsection{Tree View and Bivariate Encoding}
\label{sec:maintree}

We chose a node-link depiction as the main representation for its support of our tasks and for our users' familiarity. Multiple subtasks were identified that involved path or structure, such as [T1] - call path tracing, [T2.2] - tree structure comparison, and [T4.1, T4.3] - aggregating or eliding tree substructures. Node-link diagrams have been shown to be a good idiom for these network tasks~\cite{ghoniem-2004,lee2006,keller-2006}. We knew the target audience would be familiar with node-link diagrams as they were prevalent in the related work and used in the original Hatchet publication. Thus, we expected a node-link diagram would require little training to understand, supporting our intuitive constraint [C2].

We chose to orient the node-link diagram with the root on the left and expanding rightwards to make use of Jupyter cell aspect ratios. Users can interactively pan and zoom the tree with mouse/trackpad interactions. If the input data contains multiple trees, they are shown stacked vertically.

To support the metric analysis tasks [T4], we encode two metrics, a primary metric with node color and a secondary metric with node size. Users can change the metrics as well as switch between a diverging color map and a single hue ramp, both which can be inverted, through the menus. Legends for both encodings appear under the menu bar.

Bivariate support was a priority feature during the design as often other metrics are evaluated in the context of execution time. Early designs used color for a single metric, due to its distinguishability and preattentive nature to support outlier detection. Color is often used in other HPC visualization~\cite{adhianto2010hpctoolkit, malony2014performance}, so we expected familiarity from our users as well. We considered a bivariate colormap to put metrics on equal footing, but ultimately chose size as the second channel so metrics could be decoded individually, stand out individually, and intersections would still be salient. 

\subsection{Node Details, Selection, and Labeling}
\label{sec:labeling}

\begin{figure}
    \centering
    \includegraphics[width=\columnwidth]{figs/collapsed.png}
    \caption{Aggregate nodes are depicted with arrows indicating a subtree that can be expanded on click.}
    \label{fig:pruning}
\end{figure}

\begin{figure*}
    \centering
    \includegraphics[width=\linewidth]{figs/text_figure.png}
    \caption{Here we show the difference our labeling scheme makes in the comprehensibility of the visualization. We compare a na\"ive approach where everything is labeled (a) against a basic simplification technique where we remove internal node labels (b), and the final technique we utilized that combined collision detection and heuristics (c).}
    \label{fig:labeling}
\end{figure*}



 While only two metrics are directly encoded on the nodes, users can access more attributes through node selection. Details of selected nodes are shown in a floating table, designed to look similar to the pandas output. Nodes may be selected individually on click or mass selected using a brush. 
 
 Directly labeling all nodes leads to a lot of occlusion, especially in larger datasets. To mitigate this problem, we omit labels on internal nodes and use a heuristic strategy to remove overlapping labels. Node names are still available via hover tooltip.
 
 Our overlap removal is a sliding algorithm that first holds text positions in a y-coordinate ordered array. We then check collisions against all text-boxes within the distance of the box's height. When a collision is detected, we remove the label of the node closer to the root. If the labels are at the same level, we remove the one with the smaller primary metric. The number of nearby text-boxes to check for collisions is typically under 30, even for deep trees with tightly laid out leaves, so the runtime of this approach is near linear.

 \begin{figure}[htp]
    \centering
    \includegraphics[width=.8\columnwidth]{figs/histogram.png}
    \caption{Mass prune interface. The histogram shows binned counts of nodes with specific ``speedup'' metric values. The top histogram shows prunable nodes. The bottom shows counts of nodes that cannot be pruned because it would disconnect the tree. Two handles (pink) allow interactively adjusting the prune.}
    \label{fig:mass_prune}
\end{figure}

\subsection{Tree Simplification (Pruning)}
\label{sec:pruning}


Calling context trees can contain thousands of nodes, not all of which are of interest to a given analysis. However, analysts want access to all of them should their analysts require them and also for initial overviews. Thus, supporting the tree simplification tasks ([T4]), we added several interactive features for reducing the size of the tree shown.

We note these tree elision functionalities are {\em pruning} operations and not filtering because they only elide nodes from the leaves towards the root (i.e., subtrees) to maintain the tree structure and therefore the context of the shown nodes. 

We have a manual prune in support of T4.1 (subtree elision) and a mass prune in support of T4.2 (elision based on metrics). Elided subtrees are signified with a circle with a black arrow, indicating that they can be expanded (\autoref{fig:pruning}). Their metrics are the average of their subtree. 

With {\bf Manual Pruning}, double-clicking on a node will collapse (or uncollapse) its subtree. The {\bf Mass Prune} feature elides subtrees based on a metric value range, set through a floating interface shown in \autoref{fig:mass_prune} which is available through the menus. The interface shows the distribution of the primary metric as a butterfly histogram. The top histogram shows the distribution of prunable nodes. The bottom shows the distribution of nodes that are internal, i.e., they will not be pruned because doing so would disconnect the tree.

The two pink handles can be adjusted to set the range that should be shown. The tree updates interactively when the user releases a handle. By default, the tree is shown with the mass prune functionality set to elide prunable nodes with 0.0 as their primary metric value.

We deprioritized design for T4.3 (aggregation along internal paths and across subtrees) due to lack of specific motivating data. We included a prototype of the feature in the most simple case (aggregation along an internal path) in our early evaluations, but stopped including it due to time after gathering initial data for future design.

\subsection{Notebook Embedded Features}
\label{sec:embeddeddesign}

To move scripting results to the visualization, users need only to pass the GraphFrame they want visualized through the visualization's Jupyter magic function described in \autoref{sec:implementation}. Users can flexibly derive new data via Hatchet or general Python scripting in the regular Jupyter cells which is how we expected them to implement more complex and hard to predict filtering tasks ([T4.4]) and precise metric comparisons and searches ([T2.1, T3.1, T3.3]).

We pass state information from the visualization back to the scripting cells by exporting a query in the Hatchet query language. This supports multiple goals of [T5]. First, users can save and later apply the output query to the same GraphFrame to retrieve the pruned tree of interest. They can also apply it to a GraphFrame of a different execution of the same program to similarly prune the tree. Second, the visualization can act as a graphical interface for creating Hatchet queries, helping users learn the query language.

We designed the query export to be on-request, rather than automatically available at all times, at the request of the Hatchet project manager. They were concerned an automatic approach would be confusing. (See our reflections in \autoref{sec:reflections} for further discussion on deliberate versus automatic workflow behavior.) Once exported through the visualization menus, the query can be placed in any Python variable and then either applied or saved to disk.


\subsection{Implementation}
\label{sec:implementation}

The calling context tree visualization was implemented in JavaScript using D3~\cite{bostock-d3-2011}. Javascript code was built and combined with HTML and CSS files using webpack. The loading of the visualization into Jupyter cells and the data transfer between Python and Javascript and vice versa code was done through our Roundtrip~\cite{scullyAllisonPoster, Roundtrip} library. 

To best support the transition from scripting to visual analysis, we wanted our API for creating a visualization to be as lightweight and simple as possible. We also wished to convey to users that the functionality is Jupyter specific, so we provide users with an Jupyter magic interface by defining our call as \texttt{\%cct}. Jupyter magic functions are denoted with the ``\%" character and often modify the behavior of a notebook. This magic function takes one argument, a Hatchet GraphFrame. 
\section{Evaluation}
\label{sec:eval}

We conducted formal evaluation sessions with seven HPC experts to evaluate the efficacy of the visualization design and the scripting-visualization workflow. 

\subsection{Evaluation Design}
\label{sec:evaldesign}

The evaluation sessions were 60 minutes long and conducted over video-conferencing software. Each session consisted of an initial briefing, overview of visualization features (25 minutes), three tasks for the users to complete (15-20 minutes), semi-structured interview questions (10-15 minutes), and a debriefing. During the overview, a facilitator shared their screen and introduced a tutorial notebook for the project and then demonstrated the features on a small example dataset. Participants had access to these notebooks and were encouraged to follow along, a choice which we found to be effective for explaining the visualization functionality, catching gaps in understanding quickly, and eliciting feedback during the tutorial phase. After the tutorial, the participant shared their screen and performed tasks on the evaluation dataset. 

The evaluation design document and evaluation script are available in the supplemental materials. We piloted the evaluation with two HPC experts, a PhD student and a research professional.

\subsubsection{Evaluation Datasets} 
For the tutorial, we used a small Caliper-generated dataset collected during an execution of Lulesh~\cite{LULESH}. For the evaluation tasks, we used HPCToolkit-generated datasets collected from two Kripke~\cite{kripke} executions, run on 64 and 128 cores respectively, resulting in trees of 1500 and 2700 nodes. Krikpe is a particle transport simulation code. Although most of the participants were more familiar with targeted Caliper-generated datasets, we chose HPCToolkit-generated examples to evaluate the visualization with larger CCTs. 

\subsubsection{Participants} 
We recruited seven participants (\autoref{tab:participants}). P1 and P2 were Hatchet users that we previously interviewed early in the design study. P3, P4, and P5 were all professionals at the same organization spread across three different software teams. P6 and P7 were PhD students focusing on HPC. Of the participants, only P4 said they were familiar with Kripke. P1 stated they were not familiar, though they were aware of it.

Of the participants, P1, P2, and P5 are active Hatchet users, P3, P4, and P6 have some familiarity, and P7 had not used Hatchet before. P5 is an active Jupyter user. P1, P2, P4, and P6 used Jupyter regularly but in a limited enough fashion to need reminders of how to expand and run cells. P3 and P7 had only cursory knowledge of Jupyter. 

\begin{table}[h]
    \centering
    \caption{Participant Familiarity with Hatchet and Jupyter}
    \scriptsize
    \label{tab:participants}
\begin{tabular}{ c | c | c | c | c | c | c | c}
 & P1 & P2 & P3 & P4 & P5 & P6 & P7 \\
 \hline
 Hatchet & High & High & Some & Some & High & Some & None  \\  
 Jupyter & Some & Some & Little & Some & High & Some & Little     
\end{tabular}
\end{table}

\subsubsection{Evaluation Tasks} 
Participants were asked to perform three tasks, the third only being asked if time permitted. The first task asked participants to familiarize themselves with the dataset and was intended to allow us to observe insights generated in open exploration. The second task guided the participants to load the second dataset, calculate a speedup metric between the datasets, and then identify targets for optimization. The final task asked participants to export the Hatchet query described by the visualization, apply it using Python, and then visualize the results. It was designed to introduce participants to the visualization-to-scripting workflow. We list these tasks and their corresponding task analysis items below:

\begin{enumerate}[label=E\arabic*.]
    \itemsep=0.25ex
        \item Open Ended Exploration [T1.1, T1.2, T3.1, T4.2, T4.3]
        \item Identify A Candidate for Optimization [T2.1, T3.1, T3.2]
        \item Export and Apply Query [T5.1, T5.3]
\end{enumerate}
    
We designed these tasks through iterative documentation and discussion between primary investigators with the goal of evaluating how well the embedded visualization supported the operations identified in the task analysis, including ones designed for the notebook-embedded workflow. 

\subsubsection{Semi-structured interview} 
After the tasks, to elicit feedback regarding the visualization and notebook-embedded workflow, we prompted participants with the following questions:
    
\begin{enumerate}[label=Q\arabic*.]
    \itemsep=0.25ex
        \item What if any changes do you suggest should be made?
        \item What if any features/facets do you suggest should be kept?
        \item Have you used other CCT visualizers before? What are your thoughts of this visualization in comparison to those?
\end{enumerate}



\subsection{Evaluation Task Results}
\label{sec:evalresults}

We summarize our observations during the evaluation tasks.

\subsubsection{E1: Open-ended Exploration} 
Participants were asked to explore the dataset provided using any and all tools available to them in the notebook. 
During this task, participants were asked to think aloud, explain their analysis process, describe the dataset, and highlight any interesting data points or portions of the CCT they came across.

All participants used the tree visualization as the primary or sole method of data exploration. Almost all participants began this task by hovering over nodes to view the names associated with nodes until they found something meaningful to them. 

Most participants (P1-P5, P7) noted large subtrees with functions related to the MPI or Libsim, two parallel libraries, or the C++ method \texttt{memset}. They noted they preferred to see core Kripke functions rather than library functions and began removing them from the view. There were a variety of strategies used. P1, P3, P5, P6, and P7 used the Mass Prune feature liberally. P1, P3, and P7 then further adjusted the view by either manually collapsing or re-expanding subtrees. P2 and P4 relied more heavily on manually collapsing subtrees. After tree simplification, all participants identified key functions used in the core processing loop of the Kripke program. 

In addition to tree transformations, participants explored other visualization functionality during this time. Participants P2-P6 changed the encodings from the defaults. P5 noted they were only interested in inclusive time and mapped both channels to that metric. The other participants swapped the encodings. P3, P4, P6, and P7 explicitly sought out nodes based on the intensity of their encoding.
    
\subsubsection{E2: Speedup and Targets for Optimization}
Participants were asked to load a second Kripke profile into a Hatchet GraphFrame using Python. We guided them to apply a function to calculate a new GraphFrame with a speedup metric comparing the two datasets. With the new speedup GraphFrame, we asked them to identify a function that would be a good candidate for optimization. This task roughly approximates an analysis common in HPC: evaluating performance scaling when allocated more resources.


Many participants (P1, P3, P5-P7) repeated their approach of mass pruning as a first step but found the results were different compared to what they saw in the initial evaluation task. Because ``speedup'' was now the primary metric, participants were pruning over a different distribution. Due to the heavy skew of this distribution, mass pruning was not as effective as in E1. 

P6 and P7 changed the primary encoding to exclusive time to prune to the nodes that take more time and then changed color back to speedup to continue their analysis. P1-P4 manually collapsed subtrees belonging to external libraries to focus on the core Kripke calls, with P2 and P3 then changing encodings so small speedups were shown in red.

Noting that there were a few outliers skewing the mass prune distribution, P5 returned to the Python cells to remove the few extreme values from the GraphFrame programmatically. They then returned to the visualization and used the mass prune on the filtered set.

All but P5 offered a suitable candidate for optimization. P5 vocalized several insights regarding which parts of the program showed poor scaling and why they might do so, but said they would require more time to give a better answer. P2, P3, P4, P6, P7 suggested multiple core Kripke functions which had high exclusive times and low speedup. P2 and P6 cited encoding-based reasons for their conclusion, stating that these nodes stood out as ``big and red'' after they changed encodings. 

\subsubsection{E3: Saving and Retrieving Queries} 
Participants were asked to save the transformations they made to the tree in the previous two tasks by exporting a query back to the notebook from the visualization. They were then asked to filter their most recently visualized graph frame and visualize the filtered frame. This task was designed to exercise the integrated workflow aspect of the visualization so participants could give informed feedback.

All five participants who we asked to perform this task (P1, P3, P5-P7) did so successfully. P2 and P4 were not asked as their sessions were shorter than anticipated due to issues outside the evaluation design. 


\subsection{Analysis}
\label{sec:evalanalysis}

Two authors who attended the evaluation sessions coded the transcripts and evaluation notes. They then met to discuss, combine, and organize codes, producing a final document of hierarchically organized themes (see supplemental material). We discuss the emergent themes regarding tasks and insights, analysis strategies, feature utility and intuitiveness, issues and suggestions for improvement, and observations regarding the notebook-embedded workflow.

\subsubsection{Evaluation Task Success and Insight Discovery} 
Participants successfully completed every task they were asked, except for P5 on E2 who made several insights regarding various nodes with poor performance, some of which others gave as their answers, but said they would need more time to come up with a better answer. 

In addition to the well-defined task, we observed participants vocalize several other insights during both the familiarity task (E1) and the speedup task (E2). We discuss a sample here. In E1, several identified particular hot spots (P1, P5, P6) or hot paths (P3, P7). P1 and P4 identified that the version of Kripke used a parallel library called RAJA~\cite{RAJA}. The RAJA nodes were not clearly marked as such due to limitations in data collection, but the participants were able to recognize it from the structure. P5 hypothesized that this implementation of Kripke likely overused the \texttt{memset} function call, suggesting an alternative that could be more performant.

In E2, P1 inferred from the function names where poor performance might be due to load balancing. P2 discussed the range of speedups and P4 identified some speedups as satisfactory. P5 suggested an observed slowdown in \texttt{MPI\_Barrier} was consistent when more parallel processes need to synchronize.

The task successes and additional insights demonstrate that participants were able to parse large, cluttered trees, build mental models of the underlying structure of the code, and identify likely problem areas. This suggests our target audience is able to perform the analysis tasks we aimed for in our visualization design on realistically-sized datasets.

\subsubsection{Variety of Strategies} 
Participants demonstrated a variety of strategies in analyzing the CCT. P1, P3, P5, and P6 used the mass prune feature heavily to find focus areas of the tree and then would tweak the results with the manual collapsing and uncollapsing of subtrees. In contrast, P2 and P7 preferred panning, zooming, and manual collapsing. These two participants had more experience with commercial indented tree+table visualizations, which may have influenced their strategies. 

We also observed a variety off strategies for node selection. While most used the single and multi-click features, P5 used the brush to select multiple nodes at the same tree level. 

\subsubsection{Utility and Intuitive Design} We asked participants which features should be kept. Additionally, participants stated positive sentiments towards features unprompted. Each feature was mentioned at least once with the exception of node composition. We sample prevalent mentions.

P1, P5, and P6 said the visualization had everything they needed. P1, P3, and P5 found the mass prune functionality especially useful. P2 and P4 appreciated the bivariate encoding, with P4 saying that size made it easy to see where the most time was spent. P3 and P4 also liked the ability to compare trees.

P1, P2, P3, P6, and P7 expressed that they found the interactions with the visualization intuitive. P2 stated, ``I knew how to use it for the most part before you even explained it. . .''  

Three participants (P2, P3, P5) expressed a desire to work with this visualization further with P2 and P5 were especially interested in seeing how it would work with their own data.

\subsubsection{Issues and Suggestions for Improvement}
We asked participants what should be changed. Participant also made feature requests and reported issues unprompted.
The most common requests included undo features (P1-P4, P6),  the ability to add annotations (P4, P7), and the ability to prune by brushing (P4, P7). There were several one-off quality of life issues, such as increasing the tooltip font size and having the menus close when clicked off of. There were also a variety of requests for other metrics or to mass prune by other criteria. We discuss these in \autoref{sec:evalembedded}. 

In terms of negative sentiments, P3 and P5 expressed frustration with the skewed histogram, wishing to dynamically change the range. P2 found the mass prune handles to be ``unintuitive.'' P1 and P3 expressed some discomfort with the notebook-embedded workflow, which we also discuss in \autoref{sec:evalembedded}.

\subsubsection{Notebook-Embedded Workflow} 
\label{sec:evalembedded}
We observed that most participants  worked through the tasks which required scripting or switching between the context of code and visualization without significant frustration or friction, though we note these opportunities were limited by the length of the evaluation. In particular, while all participants suggested a complex query that we made the conscious choice should be handled by scripting, only one (P5) was prepared to implement it.

It is important to note that the complex queries suggested by the participants were not all the same. The most prevalent was to filter out MPI or LibSim libraries (P2, P3, P4, P5). Individual suggestions were pruning RAJA-related calls, pruning system calls, pruning libraries, pruning libraries qualified on metrics, pruning based on ``what we control,'' pruning by regular expression, pruning based on depth, and pruning leaves. Even in the most prevalent case, pruning MPI calls, is not as simple as a name match and may not be necessary for other datasets. This wide variety of desired, complicated queries suggests the need for the flexibility inherent to scripting. This observation is consistent with the wide variety of strategies observed across individuals when doing performance analysis.

We probed users further about whether their query should be a feature of scripting or visualization. P2 said that pruning a tree through scripting ``would work with me.'' P5 emphatically preferred scripting, explaining that debugging regular expressions would be easier. P4 expressed an interest in both, but prioritized scripting support before interactive visualization. 

In addition to queries, three participants (P3, P5, P6) inquired about the possibility of adding metrics while working on the speedup task. P5 suggested ``percentage of time,'' which is something Hatchet can do in a few lines of code. Adding metrics is another task supported by the integrated scripting environment.

Not all participants were comfortable with our embedded approach. P3 described the workflow as ``not helpful.'' P1 said the combination of scripting and visualization, two things they think of as separate, was ``unintuitive.'' However, P1 qualified their comment, noting the workflow has a need, but they were unsure how to implement it. They suggested improved messaging for the query save feature. Most participants did not comment on the speedup calculation or the query save workflow, suggesting the interface was not particularly notable to them.

P1, P4, and P5 liked the capability of retrieving a query from the visualization both for saving and retrieving the subset, but for other reasons as well. P1 was interested in retrieving the subselection as a table for further analysis. P5 said they did not have a personal use, but ``could see [their] colleagues using this to build queries.''


\subsubsection{Comparisons to Alternative Visualizations} 
Several participants (P2, P4-P7) acknowledged experience with alternative calling context tree visualizations when asked, though some had to be prompted by the facilitators regarding their indented tree experience, as they did consider them a visualization on their own. P2 and P7 had used commercial products with indented trees. P4 uses a commercially-backed Graphviz~\cite{Ellson03graphvizand} call graph visualization. P4-P7 had used Hatchet's console-based ASCII indented tree. P5 also used a web-based flamegraph.

P4 noted the commerically-backed tool has ``more power'' with ``more resources put into them,'' but noted our visualization could show more nodes at once and was more interactive. P6 also mentioned interactivity and said our visualization was ``more effective'' than the console-based indented tree.  While P2 declined to compare because theirs was in a commercial debugger used for different tasks, they also noted the interactivity was intuitive. 

P4 further stated they liked that our tool supported comparing multiple datasets, with P4 noting it was something the other tools did not. This interest in comparison was echoed by P5.

P7 found our node-link tree more natural than the file system-like indented tree they had experienced and noted it was easier to track hot paths with our encodings. 
 
\subsection{Limitations and Threats to Validity}
\label{sec:threats}

The generalizability of our evaluation is limited by the modest number of participants. We sought participants who would be representative users---those who had expertise in HPC performance analysis and familiarity with Hatchet. This limited our participant pool. Furthermore, all of the participants knew at least one of the authors, which may have biased their comments and interview responses towards expressing positive sentiments, e.g., P2 declining to compare our visualization to a commercial tool.

Though Jupyter is a delivery mechanism for Hatchet and other tools in our collaborator's organization, familiarity with Jupyter varied in our participant pool, with only one participant being a frequent user. The lack of familiarity may have affected user feedback. Also, some users may not have focused solely on the visualization over the entire notebook during the evaluation. Deployment over time is needed to determine if the integrated visualization is adopted.

We designed evaluation sessions to last an hour as this is a typical schedule slot duration for our professional users. This limited the kinds of tasks we could include. Even though most participants had some familiarity with Hatchet, they were not so familiar with the API that we would expect them to write Hatchet queries in open-ended analysis on the fly, though one, P5, did. The time constraints limited our ability to further explore the scripting and visualization workflow.
\section{Introduction}


Interactive visualizations offer powerful support for deriving novel insights from large and complex data. However, these same visualizations are often entirely GUI-based, limiting the queries and data manipulations to what the tool developers had already foreseen. Scripting offers enormous flexibility in specifying queries, but its output often lacks context or is unintuitive. To remedy this tension, we investigate a closed-loop visualization-scripting workflow through a design study supporting performance analysis in high performance computing.

Performance analysis is an intrinsically exploratory activity. There are some agreed upon strategies and commonly-used software but the concept of ``poor'' performance is ill-defined and the space of potential problems is vast. Accordingly, performance analysis remains a creative endeavor. Often a performance problem can be hard to articulate or identify without iterating through a loop of collecting, transforming, and visualizing profiling data. If initial expectations are not verified, analysts need to re-examine their datasets, poke and prod at previously ignored facets, and derive new metrics which add fresh perspectives to stubborn data. As computing experts, performance analysts have strong scripting capabilities which are typically not leveraged by GUI-only performance visualization tools.


Following visualization design study methodology~\cite{sedlmair2012design}, we investigate a visualization-scripting workflow for analyzing calling context trees, a common type of data in performance analysis. We conduct interviews with front-line performance analysts and gather task-related data through weekly meetings with collaborators. From these interactions we produce a formal task analysis that characterizes performance analysis tasks as supported primarily by scripting, visualization, or both. 

Using this task analysis, we design a notebook-embedded interactive calling context tree (CCT) visualization that espouses the hybrid workflow and proposed task organization.  We designed the visualization to fit in this workflow rather than standing alone, prioritizing intuitiveness of use for lightweight exploration. Furthermore, while visualizations generated in notebooks are typically an endpoint in notebook-based analysis, our CCT visualization supports the transfer of data and state between Python scripting cells and the interactive visualization context. This means that the data in the notebook can reflect changes made to the data in the visualization and vice-versa. 


We evaluate our design with seven performance analysis experts. We find that participants completed our evaluation tasks successfully and derived insights about program performance. Through our evaluation we also gathered feedback about the notebook-embedded nature of our tool and its visualization-scripting paradigm. Though overall our design fulfilled its goals and was well-received, we note yet unresolved design concerns regarding managing state synchronization between scripting cells and the interactive visualization. We suggest further examination of these design choices.

The primary contributions of this work are:

\begin{itemize}
    \itemsep=0.25ex
    \item A notebook-embedded-aware task analysis for performance analysis with calling context trees (\autoref{sec:task_anl})
    \item A validated design for a scalable, notebook-embedded calling context tree visualization (\autoref{sec:cctvis}), and
    \item Reflections and guidance regarding designing notebook-embedded hybrid scripting/visualization workflows. (\autoref{sec:reflections})
\end{itemize}

We discuss related work in \autoref{sec:related}. We then discuss relevant background on performance analysis and calling context trees in \autoref{sec:background} and our design process in \autoref{sec:designprocess}. We present our evaluation of the resulting notebook-embedded visualization in \autoref{sec:eval} and conclude in \autoref{sec:conclusion}.


\section{Evaluation}
\label{sec:eval}
In order to evaluate how effectively our visualization met goals set by the task analysis we designed a formal evaluation. We executed this evaluation with stakeholders and high performance computing experts. In this section we explain the evaluation design, participants, results and analysis of those results. 

\subsection{Participants}
    Due to the specialized nature of this software, we were limited to selecting participants for this software evaluation who had significant expertise with performance analysis, generally and HPC performance analysis specifically. Furthermore, since this visualization is embedded in a Jupyter notebook we desired participants who have some familiarity with Python and Jupyter. 
    
    With this narrow criteria we were able to acquire 7 participants. Two (P1,P2) are the primary users of hatchet. They were interviewed in the early stages of this software development and had their needs represented by a project manager who met with the visualization researchers every week. The next three (P3-P5) are all professional performance analysts. Of those, P5 is an active user of Hatchet, while P3 and P4 do not actively use Hatchet but have been exposed to it through tutorials. The final two participants are PhD students who primarily do performance analysis research on HPC systems, one of whom has used Hatchet before (P6) and one who has not (P7).
    
    Fluency with Jupyter was mixed among these participants. P3 and P7 were the least experienced having only a cursory knowledge of Jupyter and little hands on experience. P1, P2, P4, and P6 were more familiar and used Jupyter regularly but in limited ways. They needed to be informed how to expand out individual cells and hotkeys for running code cells. P5 was very experienced and did not have to be reminded of any basic Jupyter functionality.

\subsection{Evaluation Design}
    The evaluation was designed as an hour to hour and fifteen minute long session comprised of three parts: visualization tutorial, evaluation tasks, and semi-structured interview. For the evaluation, users were provided with two Jupyter notebooks (a reference notebook and a working space), one Caliper-collected profile of the open-source Lulesh software, two HPCToolkit-collected profiles of the open source Kripke program and an install of the CCT Vis software. 
    
    For the tutorial, the facilitator first acquainted the participant with the reference notebook containing basic usage instructions for Hatchet and the Calling Context Tree Visualization. This notebook could be used by the participant for reference at any time during the tutorial or main tasks. Following this, the rest of the tutorial was interactive: The facilitator shared their screen asked the participant to follow along in their own notebook as they load up the Lulesh dataset, visualize it and explain the functionality of the visualization step-by-step.
    
    In the main phase of the evaluation, the users are asked to perform two tasks along with an optional third, if time permitted. The evaluation tasks and their associated tasks from the analysis are the following:
    \begin{enumerate}[noitemsep,topsep=0pt]
        \item Open Ended Exploration [T1.1, T1.2, T3.1, T4.2, T4.3]
        \item Calculate Speedup and Identify A Candidate for Optimization [T2.1, T3.1, T3.2]
        \item Export Query, Filter and Re-Visualize [T5.1, T5.3]
    \end{enumerate}
    
    These evaluation tasks were crafted in an iterative process refined through documentation and discussions between the primary investigators. They were explicitly designed to provide opportunities for users to perform the tasks highlighted in the task analysis on which our visual designs were based. In the supplemental materials we provide the design document used in developing these tasks.
    
    In the open ended interview we asked three questions of our participants:
      \begin{enumerate}[noitemsep,topsep=0pt]
        \item What if any changes do you suggest should be made?
        \item What if any features/facets do you suggest should be kept?
        \item Have you used other CCT visualizers before? What are your thoughts of this visualization in comparison to those?
    \end{enumerate}
    These questions were chosen to elicit feedback on any aspect of their experience with the visualization or the notebook-embedded workflow.
    
\subsection{Data}
    The Lulesh profile was used for the tutorial due to its small size. This made it easy to highlight the functionality of the visualization without overwhelming participants. The two Kripke profiles used were run on 64 and 128 cores respectively and were chosen for the main experiment due to its larger size and higher incidence of noisy data. Although most of our participants were more familiar with Caliper datasets we wanted to evaluate how effectively this visualization could help users cut through noisy data. We also wanted to verify the performance of this visualization with a reasonably large dataset in the hands of real users.
    
\subsection{Piloting} 
We piloted this evaluation with two HPC experts: one PhD Student and one researcher. These two Pilots were used to verify how well our tasks and interview questions probed for the insights we desired. We also used the Pilots to verify our assumptions about how long the evaluation session would take. The pilots went mostly as expected so there was little iteration on the evaluation design between our pilot and full evaluations.

\subsection{Evaluation Results}
\noindent\textbf{E1:} For the first task, users were asked to explore the dataset provided using any and all tools available to them in this notebook. These include the built in console-based visualization provided by the Hatchet library and the tabular outputs provided by the dataframe half of the Hatchet graphframe. During this task participants were asked to think aloud, explain their analysis process, describe the dataset, and highlight any interesting data points or portions of the code they came across.

For this task, all seven participants opted to use the tree visualization as the primary or sole method of data exploration. Almost all participants began this task by hovering over nodes to view the names associated with each node in a scattershot fashion until they found something meaningful to them. For the majority of participants (P1-P5, P7), exploration led them immediately and frequently into subtrees of call paths and functions related to the MPI, Libsim or Memset libraries. For users seeking to understand the structure of the Kripke program itself, these calls are seen as noise. 

Participants P1, P3, P5, P6 and P7 overcame this roadblock of noisy data by using the Mass Prune functionality before making other transformations to the tree. P2 and P4 instead opted to make more deliberate reductions by manually collapsing sub-trees. Among those who used the mass prune functionality, P1, P3 and P7 opted to further adjust the mass pruned trees with manual pruning or re-expansion of collapsed trees. After tree simplification, all participants were able to identify several large-runtime, key functions used in the core processing loop of the Kripke program. Before identifying these nodes however, P2 was the sole participant who noted that he had "difficulty" finding the core Kripke functions.

External to directly transforming the tree itself, many users also explored additional functionality provided by the visualization during this time. Participants P2-P6 all opted to change the default encodings on the tree. Where color was mapped to the exclusive time a function call took to run and inclusive time was mapped to size, many opted for the inverse encodings. P5 was only interested in inclusive time so he mapped both encodings to that metric. P3, P4, P6, and P7 explicitly sought out nodes based on the intensity of their encoding.
    
\noindent\textbf{E2:} For the second task, participants were asked to load a second Kripke profile, collected from a run on more CPUs than the initial profile, into a Hatchet graphframe. They were also provided a function that calculates speedup over two graphframes and returns a new graphframe with the metrics of the smaller profile and the newly derived speedup metrics. We then asked participants to calculate speedup between the two datasets and, using any tools available to them, see if they could identify a function which may be a good candidate for optimization. This task roughly approximates a common HPC performance analysis approach called a "scaling study," where analysts evaluate how well a program scales to up to being allocated more resources for processing.

Compared to the prior task, users were now roughly understood the dataset they would be working with. Compared to the last dataset however, the default encodings had changed. Color was now associated with speedup and size was associated with exclusive time. Inclusive time was no longer visualized by default, but was still present on the dataset.

Many participants (P1, P3, P5-P7) repeated their approach of mass pruning as a first step but found the results were different compared to what they saw in the initial evaluation task. Because "speedup" was now the primary metric, participants were pruning over a different distribution. Due to the heavy skew of this distribution, mass pruning was not as effective as in E1 as most nodes were placed into a single bucket, however two participants (P6,P7) opted to change the primary encoding back to exclusive time, retrieved their original tree of only time-expensive nodes and returned color to speedup.

For P1 and P3 a more manual approach was taken after the mass-prune to collapse noisy sub-trees. This mirrored the approach taken by P2 and P4 in this task. These participants primarily identified sub-trees which belonged to external libraries and collapsed them from view. They each produced a similar structured tree focused around core Kripke function calls. P2 and P3 concluded their analysis by inverting the color scheme so that small speedup values were highlighted in red.

P5 took a unique approach when stymied by the low-resolution of the mass prune histogram: he returned to the notebook. Knowing that there was a filter function on the Hatchet graphframe itself, P5 saw an opportunity to filter his data programmatically. He wrote a lambda function to remove all nodes with a speedup greater than 1 and re-visualized this pared down dataset. Seeing the opportunity for more fine-tuned control over this reduction and freedom to explore, P5 returned to the notebook and wrote a second filter after examining the visualization.

All but P5 were able to identify a suitable candidate for optimization. Most (P2, P3, P4, P6, P7) found multiple, citing pointing out the same four core Kripke functions which had high exclusive runtimes and low speedup relative to the scale of resources. P2 and P6 cited encoding based reasons for their conclusion, stating that these nodes stood out as "big and red". 


\noindent\textbf{E3:}In the third task participants were asked to save the transformations they made to the tree in the previous two tasks by exporting a query back to the notebook from the visualization. They were then asked to filter their most recently visualized graph frame and re visualize it. Although less free-form than the previous two tasks, the inclusion of this task allowed us to receive feedback on the observed and perceived fluency of the workflow critical to long-term adoption of our visualization.

Five participants were able to perform this task (P1, P3, P5-P7). This task was less open ended from prior tasks so methodologies undertaken among participants were the same. Among those participants all completed it successfully. Furthermore, this task generated good discussion on the design of this particular interaction with the notebook.
    

\subsection{Analysis}
    Transcripts of the evaluation and semi-structured interview we produced automatically by the virtual meeting software we used for these sessions. These transcripts, along with notes produced by the second visualization researcher in attendance, were coded by both researchers. These codes were aligned and organized into themes which we discuss here.

\noindent\textbf{Visualization Effectiveness:} The participants completed every task they took part in successfully with the sole exception of P5 on evaluation task 2. It should be noted here that this task was not failed due to the participant giving up, but rather due to a lack of time. His workflow exemplified a very realistic and exploratory approach to performance analysis which works much better with the longer time scales afforded analysts grappling with difficult issues and likely would have yielded fruit if given more time.

Considering the overall success of participants in accomplishing these tasks, we assert that this is positive evidence of this visualization's effectiveness in supporting exploratory performance analysis. Participants were able to parse large, cluttered trees, build mental models of the underlying structure of the code and identify likely problem areas.  Furthermore, since these tasks were performed successfully on a reasonably large calling context trees with over 1000 nodes, we also assert that this visualization seems to meet our desired goal of scalability for some realistic datasets. No individual made mention of the performance of this visualization and no problematic slowdown or lag that was tied to the visualization itself was observed during the evaluation session. 

In addition to our broad assessment that this visualization was effective at enabling participants to perform the tasks we outlined as part of our task analysis, we also observed that this visualization proved effective in enabling participants to extract insights from the particular dataset we used for the evaluation. This does not prove that it would be effective for all datasets but hints at a transferability of insight enabling.

In the first evaluation task, P1-P4 all recognized the core Kripke functions not only as datapoints of interest but as the actual functions in the code. They synthesized their background knowledge of Kripke with the layout of the tree in the visualization. P1 and P4 in particular identified that this particular run of Kripke used a parallel library called Raja. P5 in this task also noted that this implementation of Kripke likely overused a C++ function call called memset.

In the second evaluation task, P1-P5 extracted more insights about the data than the optimization target we requested. For example, P4 and P5 identified the same nodes which had particularly high speedup and were interested in digging in deeper to understand why it was so high. P1 inferred that there was likely a load balancing problem driving poor speedup based on data labels. By filtering the data programmatically, P5 was able to identify that many function calls had less than 50\% speedup and were slowing down as the program scaled up. He also verified his understanding of the data by verbalizing that "MPI Barrier" slows down, which would be consistent with expectations as it has to wait longer when there are more cpus it needs to sync up.

Participant sentiments about the effectiveness of this visualization were overall positive. P1, P5 and P6 indicated that it has everything they needed. P1, P3 and P5 found the mass prune functionality especially useful using superlative expressions like "super useful" or "super valuable". P1, P2, P3, P6 and P7 expressed that they found the interactions with the visualization intuitive. P2 in particular said: "I knew how to use it for the most part before you even explained it. . ." P2 and P4 appreciated the bivariate encoding, with P4 saying that size made it easy to see where the most time was spent. Three participants (P2,P3 and P5) expressed a desire to work with this visualization further with P2 and P5 especially excited to see how it would work with their own data.

Critical statements were raised throughout the evaluation and in response to the first interview question. Many were focused on quality of life improvements and feature requests. Some of the most common were the request for a single action undo(P1-P4, P6), as well as a request that the drop down menu collapse when they click away from the menu bar. There were also requests from every participant for the mass prune to work with a diversity of node attributes. We discuss these in our assessment of the notebook-embedded workflow.

As far as negative sentiments went there was no general negative feelings expressed to the visualization and its design. P1 and P3 expressed some dislike of the notebook-embedded workflow, finding it "unintuitive" and "not-helpful" respectively. Specific to the mass prune functionality, P2 found the sliders on the mass prune to be "unintuitive". P3 and P5 expressed frustration at the poor resolution of the histogram, wishing it could be "zoomed in on," with the same data points spread across more buckets. 

\noindent\textbf{Notebook-Embedded Workflow:} The embedding of our visualization within the exploratory scripting workflow was an intentional and informed decision. Based on our understanding of performance analysis as an inherently exploratory process, we designed this visualization to support exploratory workflows as best as possible by removing barriers of context switching and data export which most visualizations require. Although not definitive, we argue that these evaluations demonstrated the utility and viability of embedding visualizations in scripting notebooks. These evaluations further show the interest among performance analysts for tools which support this workflow.

In terms of concrete observations, we saw that overall participants worked through the portions of tasks which required scripting or switching between the context of code and visualization without significant frustration or friction. Although P1 described this combination "of two things [scripting and vis]" as "unintuitive," and P3 described this workflow as "not helpful," no other participants shared a similar negative sentiment. Any other explicit commentary on this approach was decidedly positive, if present at all. Most participants took the syntax for the query retrieval as logical and gave very little thought to the implementation details. This leads us to believe that this swapping between scripting and vis in the second and third task felt natural to most participants despite not previously being exposed to this workflow. 

In particular P1 qualified his critical comment by expressing that he appreciates the need for this workflow and concedes that the formatting and text in the query retrieval popup may be where the unintuitive feeling comes from. P1, P4 and P5 expressed a particular interest in this idea of retrieving a query from the visualization with the intent of not only saving the state of the visualization with it, but also using it to align their data with the changes they made to the visualization. P1 in particular described it as a good approach to get down to the tabular data for a handful of functions. P5 said that he did not have a personal use case for this but said that he "could see [his] colleagues using this to build queries".

Outside of direct commentary on this approach, nearly every participant ran into a problem which demonstrates the need and utility of ready access to scripting alongside visualization. This problem was generally running into the noisy nodes which represented a barrier to their effective analysis. Although all participants overcame this barrier with some combination of manual and/or mass pruning, every participant inquired as to whether there was functionality to prune away subtrees based on a wide variety of abstract criteria which would not be possible to effectively capture in the point and click interface of an interactive visualization. 

For some, it was a desire to prune based on their mental model. P1 requested a desire to prune away all "Raja" nodes, which is a library that added some psuedo-function calls to the tree. Note that these Raja nodes are not labeled as such in the dataset and they can only be recognized as such by synthesizing observations of the visualized data with prior understanding of how parallel code is written. P5 expressed a desire to prune away everything but "what we control", referring to the core functions which an engineer could edit or optimize, compared to libraries which they do not have access to.P7 expressed a similar desire to remove libraries all together and P3 desired to remove "system calls". These requests are very abstract and do not map well to the data in a way that can be automatically detected by the visualization software. However, in some instances, patterns describing groups of nodes can be recognized by individuals and applied to a graphframe using the much more expressive scripting functionality afforded by Jupyter and Hatchet's query language.

In other instances, participants expressed slightly more concrete attributes to prune on. P2, P3, P3, P5 expressed a desire to prune based on a particular name like "libpsm2" or "libmpi." P4 and P6 also expressed an interest in pruning by tree depth, although the former expressed it as a desire to "prune leaves". Both of these functionalities would be more easy to automate than the above, however Hatchet already supports them and by virtue of being a library in a programming language, catches errors better and represents a familiar workflow to new users.

In addition to expressing a desire for this diverse array of pruning criteria, some participants articulated whether they would prefer to have these advanced features on the scripting side or vis side. P2 and P5 preferred scripting over point and click functionality for this with P2 citing that pruning a tree through scripting "would work with me" and P5 explaining that debugging regular expressions would be easier in scripting over vis. P4 expressed an interest in both, saying that he would like scripting to filter before visualizing and would also like more robust tools for filtering in the visualization.

Finally, as part of the evaluations, three participants (P3, P5, P6) touched on the deeper implications of our second evaluation task by inquiring about the possibility of adding metrics. Their requests we general in the area of support for more metrics or in P5's case, requesting to get a specific derived metric: "percentage of time." Because of the flexible design of this visualization, we were able to inform participants that if a quantitative metric exists on the dataset or can be derived and attached to the graphframe our visualization supports it. And, given the expressiveness of the scripting possible in Jupyter notebooks, this means that we can visualize many types of metrics derived on the fly by experts.


\noindent\textbf{Comparisons to Alternative Visualizations:} Most participants (P2, P4-P7) cited experience with alternative calling context tree visualizations. P2 cited a tree visualization in ARM DDT, but declined to compare it substantially to our visualization due to the fact that it was inside of a debugger and used for different tasks. He did mention however that it used the same interaction for collapsing and expanding sub trees that we did. 

P4 expressed that he had worked with a dot-based call graph visualization called gperftools and the console based intended tree provided by the Hatchet library. Although stating that other tools are "more powerful" with "more resources put into them," he noted that our visualization made up for that with scalability, interactivity and script based support for comparing multiple datasets. 

P5 had worked with the Hatchet console-based indented tree and some trace viewers like the flamegraph found in web browser profiliers. Like P4 he also liked that this visualization supported comparing multiple datasets. 

P6 had only worked with the console-based indented tree and found this one to be much more interactive. He also asserted that, by comparison, our is more "effective."

Finally, P7 had no experience with a console-based indented tree, however they had worked with an indented tree+table visualization in Intel Advisor. By comparison to Intel Advisor, P7 found our tree more natural than Advisor's file-system like layout. he also expressed that he found it much easier to track a hot path with our color and size encodings. 


\section{Design Process}

We followed the design study methodology of Sedlmair et al.~\cite{sedlmair2012design}, informed by the criteria for rigor proposed by Meyer and Dykes~\cite{meyer2019criteria}. We iterated frequently at every stage of the process and strove to make informed decisions about the designs we proposed, including but not limited to our choice of a node and link layout for the tree and what channels we used for our bivariate encoding. 

We were continuously reflexive in our approach to the design of this system. Even after development was underway we reconsidered and reflected on our biases by assessing alternative encoding rules, interaction paradigms and designs. Frequent discussions with project stakeholders and between the visualization researchers themselves highlighted areas of misunderstanding and incorrect assumptions. Throughout this process we collected abundant design and collaborative artifacts from meetings and personal reflections. 

\subsection{Team and Roles}
\label{sec:roles}


\subsection{Design, Data, and Task Analysis}
Our core research team for this project is comprised of two visualization experts and a Hatchet project manager with whom they met with weekly. More infrequently, these visualization experts would also meet with software developers working on the Hatchet project. The visualization experts met with and formally interviewed two performance analysts preceding the principal design and software development phase of this visualisation. These two performance analysts represent typical front-line analysts who would use this software. Due to the limited availability of these analysts, in future meetings, the project manager acted as their representative, expressing their needs and giving feedback from their perspective

Design and development of this visualization and its prototypes has been active since 2020. In the weekly meetings held since its inception, the visualization experts shared progress updates, discussed design and development directions, and generated abundant artifacts in the form of notes, design drawings and recorded demos. 

\subsection{Data}
\label{sec:data}
\note{Revise this after background section is done}
The calling context tree visualization takes a Graphframe object produced by the Hatchet library as its primary input. This Hatchet Graphframe can be generated by reading in a wide variety of commonly used profiling datasets (gProf, cProf, TAU). Primarily however, this visualization was designed primarily to support the calling context trees created by two performance analysis tools: Caliper and HPCToolkit.

Caliper is a open source tool for the annotation-based profiling of HPC software \cite{caliper}. Since it is annotation based, users will place macros in their code denoting blocks which they want to collect performance metrics on. This produces small, easy-to-understand calling context trees, but requires the labor overhead of manually annotating code before profiling. It is the profiler most commonly used by our collaborators for whom this visualization was developed.

HPCToolkit, by contrast, is a suite of tools that includes a sampling-based profiler\cite{adhianto2010hpctoolkit}. As a sampling-based profiler, HPCToolkit does not require any manual annotations and instead samples the state of the program at fixed intervals as it is running. It constructs its calling context tree based on the structure of the program at sampling time. Unfortunately, this automatic approach catches much more than the core functions which may be of interest to analysts. Furthermore, it can produce very large trees which can be difficult to visualize using traditional means. This profiling software is very widely used and easy to work with, thus justifying our desired support for it.

\subsection{Task Analysis}
\label{sec:task_anl}

During design and development we identified five high level tasks along with numerous subtasks. We describe them here and provide our justification for including them. Furthermore, because this visualization was developed to work fundamentally inside of a notebook using a \textit{Graphical-Scripting} paradigm, we note alongside each subtask if it should be handled by the scripting side of our software stack or the visual side. Quotes used as justification were obtained from meetings with our primary institutional collaborator (IC) or from formal semi-structured interviews with front-line analysts (PA1, PA2).


\noindent\textbf{[T1] Call Path Tracing}

\textbf{Description:} Call path tracing describes the activity of starting at a parent function call and tracing the hierarchy of calls which leads to a particular child or starting as a child and leading up to an ancestor.  

\textbf{Justification:} ``If you did this filtering you've lost the common parent node of the tree, so it made it very hard to orient yourself about where you were.” - PA1 ``. . .trees are really helpful, because it gives me basically a call stack. And that makes it much easier for me to understand like where stuff is going. . . ” - PA2

\textbf{Subtasks:}
    \begin{enumerate}[noitemsep,topsep=0pt]
        \item Tracing the path from a leaf up to the root or an inner, ancestor node to a child node. Used for orientation. (Visual)
        \item Identify ancestors and descendants of nodes (call stack). Used for analysis. (Visual)
    \end{enumerate}


\noindent\textbf{[T2] Tree Comparison}

\textbf{Description:} This task encapsulates user's expressed desires to compare two trees from similar runs of the same program. These runs will often differ in terms of scale (eg. how many cpus, threads and machines was each run using) or in terms of implementation details (eg. using different implementations of MPI for the same program). For programs run with different size configurations, users wish to compare metrics between functions shared by both profiles. For different implementations, users want to examine the different structures of each profile; see where they overlap and where they diverge.

\textbf{Justification:} The activity of comparing different scales of runs is one of the most common forms of Performance Analysis in the HPC space. 

\textbf{Subtasks}
    \begin{enumerate}[noitemsep,topsep=0pt]
        \item Compare metrics on individual nodes between scaling runs (Script)
        \item Compare tree structures with different software implementations (Visual)
    \end{enumerate}


\noindent\textbf{[T3] Outlier Identification}

\textbf{Description:}   Outlier identification describes any action of looking at a dataset and identifying data points which are abnormal relative to the distribution of all other points in the dataset. In performance analysis the specific outliers being sought are dependent on the metric being observed. For a measured metric like the \texttt{exclusive time} it takes a particular function call to execute, the highest outlier would be interesting because it potentially indicates a poor performing function. For a derived metric, like the \texttt{speedup} gained by running software on more CPUs, a lower outlier would be interesting because it shows a function which did not scale up to use the additional processing resources for some reason.

\textbf{Justification:} 
“ . . .because sometimes outliers will really work with the color scale. So like if something is 100 times worse than everything else, then you know the rest of the colors are yellow meaning, you know the same thing.” - PA1
“This is an example, I have explained to you guys before where PA2 here is basically making his own multivariate analysis, right? He is looking at what is the speed up or slow down right, so if a. . .function slows down by a lot, the second question he wants to know is, does he care, you know if this is something tiny.” - IC on PA2's work


\textbf{Subtasks}
    \begin{enumerate}[noitemsep,topsep=0pt]
        \item Find call sites with sufficiently large metrics (Script, Visual)
        \item Find meaningful outliers through bivariate analysis (Visual)
        \item Analyze how well individual functions scale with more processing units (Script)
    \end{enumerate}


\noindent\textbf{[T4] Tree Simplification}

\textbf{Description:} Tree simplification describes a bundle of subtasks which involve filtering remove adjunct subtrees and nodes or or adjusting a tree's structure to combine subtrees and nodes to make it fit the mental model of the analyst.  

\textbf{Justification:} Many HPC profiling tools measure the execution of a program by sampling code at specified rates and collecting contextual information about the calling context of functions at that sampling time. Compared to other tools, this approach allows users to measure their code's performance without modifying their code to specify regions of interest. Unfortunately, this approach also creates very large calling context trees containing many nodes irrelevant to a user's analysis. Users accordingly expressed a desire to get to the relevant parts of their trees quickly and remove all the function calls and nodes which did conform to their mental model of the call stack.


\textbf{Subtasks}
    \begin{enumerate}[noitemsep,topsep=0pt]
    \item Mass filtering on complex querying criteria. For example, "Remove all nodes which are part of the MPI library" (Scripting)
    \item Manually remove subtrees which are not relevant to current analysis (Visual)
    \item Simplify trees by defining basic thresholds of inclusivity and exclusivity (Visual)
    \item Simplify tree structure by combining subtrees and internal nodes (Visual)
    \end{enumerate}

\noindent\textbf{[T5] Save and Recover Modifications to Data}

\textbf{Description:} This task describes the activities users can take to reproduce the changes they have made to the visualization across different analysis sessions or when sharing their notebook with other users. It involves exporting, saving and recovering state based information describing the modified structure of a tree.

    \textbf{Justification:} This task was derived from users' desires to pick-up work which they left off across multiple days of analysis without having to re-execute a tedious series of point-and-click reductions of a tree. Furthermore, users expressed an interest in potentially utilizing their analysis notebooks as transferable artifacts, but for that to work, it would require them to also transfer the working state of their trees. Finally, our inclusion of this feature conforms to Shneiderman's "extract" task suggested in \cite{shneiderman2003eyes}.

\textbf{Subtasks}
    \begin{enumerate}[noitemsep,topsep=0pt]
        \item Extracting the state of the visualization after changing the structure (Visual, Scripting)
        \item Storing the visualization state for future notebook runs(Scripting)
        \item Recovering and applying the visualization state to re-loaded data in new runs of the notebook (Scripting)
    \end{enumerate}

\subsection{Addressing Rigor}
It can be difficult to demonstrate that scientific research using exclusively qualitative methods is rigorous when positivist conceptions of rigor, most specifically reproducibility, dominate our general understanding of the topic. So, rather than argue that the research we lay out in this paper would meet that standard, we argue through an interpretive lens that we met a standard for rigorous research research according to Meyer and Dykes' Criteria for Rigor in Visualization Design Study \cite{meyer2019criteria}. In this work, Meyer and Dykes lay out 6 criteria for rigor and explain that rigorous research may strive to meet all of these criteria but it is not necessary that all are met. Here we reflect on how our research applies to these criteria.

Starting with informed, we came into the research with collective significant experience in the domain of information visualization in HPC Performance Analysis. We bolstered this foundation with research into existing calling context tree visualizations and standard conventions used in this and other domains. It was thanks to this informed background that we supported interactions like "double clicking to collapse a subtree" and chose to provide a bivariate encoding on nodes.

We consider our process to be deeply reflexive as we generated abundant notes and pages of discussion around the design and implementation of this visualization. Additionally, through our constant contact with collaborators embedded in the High Performance Computing (HPC) domain we were able to get immediate and constant feedback on the assumptions and biases which may have negatively impacted the design, otherwise. Finally, by sharing a preliminary version of this work with the HPC community in the form of a research poster\cite{scullyAllisonPoster}, we got significant feedback which impacted the direction and scope of our design.

Our approach to design produced abundant artifacts in the form of 27 written records of meetings across 21 pages, 5 pages of pen-and-paper designs, 5 pages of task analysis, 5 pages of evaluation design,  two hour-long thick descriptions of the workflows and needs of HPC performance analysts from front-line visualization users, 2 power point presentations on this visualization, and a research poster. This list is not inclusive but roughly describes the abundant and diverse artifacts that have been produced from and have informed this research.

As for plausibility and resonance, we cannot objectively claim to meet these criteria as they refer to content in this paper itself. However, we outline our arguments for the effectiveness of our visualization in Section \ref{sec:eval} and refrain from making implausible or unsupported claims to the best of our ability. We also try to write this paper in a clear, easy to understand way which we hope will resonate with researchers reading this and promote their understanding of this research.

Finally, throughout this paper we attempt to be as transparent as possible when describing our design choices and the results of our evaluation. We highlight threats to validity and limitations within their own sections to convey very clearly that this work is not conclusive and rather seeks to expand the communities understanding of notebook-embedded visualizations and calling context trees.


VISUAL DESIGN STUFF

The "Calling Context Tree View" is the main view of our visualization. Our users primarily needed to only view metrics on nodes in the context of the overall program with little need to frequently cross reference with other artifacts like source code or tabular layouts of the data. This meant that we could dedicate significant space to the tree alone and eschew complex dashboard layouts. Accordingly this view is comprised of two main components: the tree and legends.

For the design of the tree itself we considered various hierarchical layouts, including flame graphs, sunburst charts and indented trees as theses are the most common representations of call trees found in the performance analysis domain. Ultimately however, we opted to use a node-link representation as it fits our user's mental models most closely. By choosing this representation we hoped that this would reduce the amount of "startup" time required to start understanding the visualization and extracting insights.

The nodes of the tree use two encoding channels to show the intersection of two metrics associated with each function call, each on their own scales. The "primary metric", associated with other adjustments to the tree, is encoded by a 6-step blue-to-red diverging color-map and the "secondary metric" is encoded by radius, increasing from 4 to 25 pixels. We show the approximate sizes and colors associated with a range of values with the two legends in the top left. The tree itself also supports panning and zooming.

The inclusion of this bivariate encoding came at the direct request of project stakeholders. One in particular was attempting bivariate analysis using Hatchet's built in console-based indented tree and tabular layouts but could not find a means to highlight relevant intersections of these two variables. Our initial tree design used hue as the sole encoding channel due to it's well documented distinguishability and effectiveness when used to encode a relatively small number of categories \cite{munzner2014visualization}. Furthermore, using colormaps to show the relative magnitude of metrics in performance dataset visualizations is common practice in the HPC field \cite{adhianto2010hpctoolkit, malony2014performance}

For the secondary channel, area was decided upon due to its relatively high effectiveness and expressiveness in conveying the relative magnitude of metrics on each node. More effective channels than area would not fit this visualization as they were already used (position, to encode the structure of our tree) or did not make sense with circles like angle or length. We did consider a 2D color map in initial designs but decided that two completely separate channels would be easier to decode and show intersections more naturally.

\noindent\textbf{Mitigating Text Collision: } Early prototypes of the calling context tree saw text labels occluding all other marks in the tree when a real calling context dataset of reasonable size was loaded (>1000 nodes). We mitigated this problem by expanding the size of the tree to give significantly more white space between nodes and removed labels on internal nodes. Unfortunately, this solution still resulted in text overlap when many sibling leafs would stack upon one another in neat rows or when a shallow leaf label would collide with a deeper leaf label that shared the same alignment on the y axis.

In order to mitigate this problem with a more generalized solution, we considered some heuristics which would allow us to detect collisions quickly but found too many edge cases with each of them. Accordingly we opted for a sliding algorithm whereby we ordered the text positions in an array ordered by y-coordinate and checked the collisions against the \textit{k} text-boxes below our current text box. When a collision was detected, we removed either the text box labeling the node containing the minimum "primary metric" (in the case of siblings) or the label of the shallower node if they were at different levels. The runtime of this algorithm is $O(n*k)$.

\subsection{Pruning}

\begin{figure}
    \centering
    \includegraphics{figs/prettier_popup_alt_no_i.png}
    \caption{Not final}
    \label{fig:mass_prune}
\end{figure}

As we described in Section \ref{sec:data}, we tested and designed our visualization around datasets collected from two tools: Caliper and HPCToolkit. The former works by users explicitly marking what functions they want to have profiled, producing small calling context trees. The latter works by sampling running code at fixed intervals and reconstructing the calling context of the code at the time it was sampled. This produces much larger calling context trees with large subtrees containing library functions which analysts are not interested in. So, in order to accommodate analysis of these bloated datasets we provide significant functionality for reducing the size of trees.

We call this process of tree reduction "pruning" not only to evoke imagery of managing trees found in nature but also because we see it as distinctly different than raw filtering. We define pruning as a selective filter which removes nodes from the leaves upward but never internal nodes. This means that we can only remove sub trees and siblings of unpruned leaves. We implemented this limited type of filtering because users expressed to us very clearly that they do not want to work with orphaned trees and always wanted the broader context associated with a particular function call.

To accommodate both fine-grained and quick, approximate tree reductions we provide two mechanisms for pruning: manual and mass.

\noindent\textbf{Manual Pruning: } A user can manually prune away a subtree by double clicking on an internal node. The subtree then collapses into the double-clicked node and is replaced with a surrogate node. This surrogate node contains the same metric categories as regular nodes in the tree but its associated data are averages over the metrics in the nodes comprising the collapsed sub tree. The surrogate node uses the same "circle" mark as regular nodes and is encoded with color and size on the same scales as normal nodes, using these aggregate metrics in the place of the real metrics. A surrogate node also shows that it is eliding a subtree with the inclusion of a black arrow to its right. Any surrogate node can be re-expanded by double clicking on it again.

\noindent\textbf{Mass Pruning: } To prune away many subtrees all at once, we provide a mass pruning tool. This tool is contained within a popup which users can activate from the dropdown menu at the top of the visualization window (Figure \ref{fig:mass_prune}). In this popup, users are given a scented slider where they can select which nodes they would like to exclude by dragging the tabs on the bottom of the popup over a distribution of the current "primary metric" across all nodes. 

This distribution is shown by two histograms, one violet and growing upwards and the other gray and growing downwards. The violet bars indicate internal and leaf nodes which can be removed by moving the slider to exclude them. The gray distribution indicates nodes which have a metric value of 0.0 and cannot be removed because they are ancestors to the nodes on the violet side. The tree view updates automatically a soon as a user has finished dragging and releases the left mouse button.

\subsection{Composition of Nodes}
In addition to pruning, project stakeholders expressed a desire to simplify the tree in other ways, as we explained in \textbf{[T4]}. Namely, they wished to combine parents with children and similar subtrees. As the latter would produce a graph, we did not consider implementing it at this phase of development as graphs are much more complex visualizations which would require significant design changes at an advanced stage of development. Combining parents with children by contrast is a more reasonable request, however as it would mostly require rewiring the tree assuming we limit how nodes are combined.

This functionality was implemented before evaluation of the software using a similar interaction mode to manual pruning: \texttt{ctrl+double click}. Double clicking a node this way combines it with its parent. The subsumed nodes children are then made the children of the parent node. In evaluations unfortunately, this interaction produced buggy behavior and had to be removed from the list of tasks we asked our participants to perform. We will elaborate on this further in \ref{sec:reflections}.

\subsection{Supporting Graphical-Scripting}
\textbf{Exporting Visualization State:} Within our collaborators' institution, Jupyter Notebooks are used as both an analysis tool by performance analysts and also as a platform for internal communication of research findings. To support both of these use cases we provided functionality to export a query from the visualization describing the current state of the visualized tree. This query, when applied to the original data, removes nodes which were pruned away in the visualization making re-visualizations of the data pruned by default.

This query is a string and is stored in a user-specified python variable using Roundtrip's Javscript-to-Python conversion functionality. It can be written out to disk using simple python code and recovered with a file read. This long term save capability means users can recover the working state of their data and visualization even if they close their notebooks over the course of multiple day or weeks-long analysis session, without needing to manually repeat pruning steps which they undertook to arrive at this state. Furthermore, for communicating research findings and collaboration, users can pass along this query with their notebook when exchanging it with colleagues allowing them to recover the visualization state.

\noindent\textbf{Supporting the Data Exploration:} By embedding our visualization in an a scripting environment like a Jupyter notebook, we enable users to rapidly investigate their data with very little overhead. In particular, this model supports the visualization of newly derived data on-the-fly with no context switching. If an mathematical operation can be supported by the Hatchet data model, it can also be visualized. This means that users can take some insight from their visualized data, and engage with it more thoroughly by combining datasets and deriving new metrics which the visualization was not directly designed for. The number of potentially useful metrics in the domain of performance analysis is more than can be reasonably supported by a point a click interface. However we can support the expressiveness of scripting by giving users a simple interface for loading structured datasets: our magic python function.


\subsection{Implementation}


\note{import library, \%cct, argument is graphframe}
To best support the transition from scripting to visual analysis, we wanted our api for creating a visualization to be as lightweight and simple as possible. To instantiate a visualization, users must import it from the hatchet library by calling \texttt{import vis.}


 

\section{Reflections}
\label{sec:reflections}
We reflect upon our design study and what we learned regarding designing in a notebook embedded context and CCT visualization design.

\subsection{Notebook-Embedded Visualization}
The embedding of our visualization within the exploratory scripting workflow was an intentional and informed decision, based on both the constraints of our collaborators---using notebooks as a delivery mechanism---and our observations that exploratory performance analysis benefits from the flexibility of scripting. Based on our understanding of performance analysis as an inherently exploratory process, we designed the visualization assuming the exploratory workflow of modifying the dataset with Hatchet scripts, doing visual tasks, and then either modifying the dataset again or exporting a subset of the data found during visual exploration. We discuss our lessons learned regarding our notebook-embedded design process and evaluation.

\vspace{1ex}

\noindent\textbf{Identifying visual or scripting strengths in our task analysis helped us focus and prioritize our design.} Our early interviews with frontline analysts sought to understand the entire performance analysis process, without focusing on CCT visualization in particular. By embedding ourselves with the Hatchet team, we were further able to hear feedback regarding not just the visual needs, but how Hatchet scripting is used in workflows and what requests there were of the library as a whole. From this, we developed our task analysis with both scripting and visual tasks. We further reflected upon which mode of exploration, scripting or interactive visualization, served the task best. 

By referring to our task analysis, we also avoided adding complex features for a single user, as those were better served under a class of scripting operations. This is especially important as our access to frontline analysts was limited in both time and number of people. It can be tempting to add features for one person, but as our evaluation showed, the query features were quite diverse among participants. We plan to further analysis these queries to determine if more abstract query features, like our metric mass prune, will be helpful, while leaving the specific ones to scripting.

Feedback from participants during our evaluation largely validated our task assignment choices, though we found evidence for more tasks that spanned both visualization and scripting. Even in these cases, participants' prioritizations followed our task analysis. 

\vspace{1ex}

\noindent\textbf{There were benefits and drawbacks to using Jupyter as a platform for our HPC audience. It is unclear how visualizations should be delivered in this domain.} Jupyter notebooks are a delivery mechanism for the Hatchet team within their organization. The notebooks with new Hatchet feature examples are uploaded to a performance analysis portal where developers discover them and if helpful to their analysis, use them. The online portal also has web-based visualizations supporting other analysis tasks. During our evaluation, one participant (P4) noted the importance of this online availability, because notebooks sent other ways would be more easily lost. 

However, as demonstrated in our evaluation, most of our intended audience do not use Jupyter regularly. Only one participant (P5) knew the common hotkeys. Thus, while the Jupyter delivery mechanism of scripting and visualization fits within the organization, it is still unclear whether the mechanism best supports the users. Some had used file system style visualizations embedded in desktop debugger applications or printed in ASCII to a terminal, but noted the limits in interactivity and the less natural idiom. Some had used the web-based visualizations in the portal, but these do not support scripting.  

\vspace{1ex}

\noindent\textbf{We heard mixed feedback regarding how automatic or deliberate synchronization between scripting state and visualization state should be.} There are two key datasets that can be transferred between the scripting cells and the visualization, the GraphFrame input to the visualization, and the Hatchet query describing the subset being shown by the visualization. Automatic synchronization ensures these two datasets are always the same between the scripting cells and visualization. Deliberate synchronization only pushes data between scripting and visualization when the user takes a deliberate action, such as (re-)running a code cell or performing a particular action in the visualization. Our visualization can be run in either mode.

We expected that automatic synchronization would be preferred due to its fluidity and decreased number of manual operations. However, the Hatchet project manager, who is also an experienced HPC performance analyst, was concerned that users would find the automatic approach mysterious and that state they wanted to save could be overwritten. The project manager argued it would only be one more quick operation when switching contexts, followed by several more similar ones for the interaction, anyway.

We conducted our evaluation in deliberate mode. While Participant P1 expressed discomfort at the combination of scripting and visualization as a whole, he noted improvements to the interface could help. P6 inquired about the workflow of adjusting the source data for a visualization. When we informed him that functionality for automatically updating the visualization existed, but was not supported in the evaluation, he expressed mild enthusiasm for the functionality.

These tensions suggest further study regarding the deliberate or automatic design choice and further study regarding how to signify synchronization (or lack there of) between contexts.

\subsection{CCT Visualization Design}

We reflect upon the visual design of our calling context tree, including encoding choices, and transferrability.

\vspace{1ex}

\noindent\textbf{We were able to leverage both visualization knowledge and prior work on CCTs, but there were still hurdles that may limit adoption of the design in HPC tools.} We made a conscious decision to aim for simple, intuitive visualization, building on our experience with difficulties in adoption of more unique approaches and allowing us to focus on the design concerns of the notebook-embeddded workflow. Though we were able to use existing software (e.g., a modification of d3js's Reingold-Tilford~\cite{reingold} layout) and visualization principles (e.g., saliency and separability of color and size), we found we still needed significant effort to support the multiple forms of tree navigation and simplification as well as labeling concerns. We suspect tree navigation and labeling implementation needs may be why the file-system-style indented tree remains used in HPC tools for more than file navigation.

This choice for an simple visualization was further informed by Seidlmair, Meyer and Munzner's assertion that ``The goal of the design cycle is satisfying rather than optimizing''~\cite{sedlmair2012design}. In the context of a visualization embedded in a scripting workflow, we found that efficiently filling in the gaps not met by scripting alone in a usable way satisfied much better than attempting to design something complex which could not meet all contingencies and may not get used. 

\vspace{1ex}

\noindent\textbf{Many participants changed encodings in common ways, swapping two variables or inverting, suggesting derivable cultural norms.}
During the design of our software we acknowledged that a na\"ive bivariate encoding of any metric may not be enough for effective analysis. Although it would take a more thorough investigation we posited that hard-coded ``intelligent defaults'' for metric encodings may be helpful, for example, making speedup use inverted color by default  or always mapping time to node size. 

We did not formally investigate these special encodings, but in our evaluations we observed that between the two default GraphFrame metrics (time (exclusive) and inclusive time), inclusive time appeared more salient when encoded by color than size. When shown with color, it highlighted a more visually literal ``hotpath'' of the most active nodes, starting with red nodes at the root fading off into yellow and light blue. We also noted that the intersection of ``color'' for ``speedup'' and ``size'' for ``time'' highlighted potential optimization candidates in a highly visceral way, where the most problematic nodes were ``big and red.''

There are hundreds of performance metrics which can be affixed to a calling context tree, so our observations here are very limited. It does however provoke a line of inquiry into what makes for a good mapping between performance metrics and encodings and whether we can define or prioritize common performance metrics such that our tool and others can apply them to performance visualizations generally.


\vspace{1ex}

\noindent\textbf{We observed broader domain interest in our bivariate tree design, suggesting transferrability.}
Although not especially novel to the visualization community, we found there was significant domain interest in bivariate encodings for tree visualization within the HPC and related communities. While presenting preliminary results in poster form at a domain conference~\cite{scullyAllisonPoster}, the lead author was approached by an HPC professional, a professional working in data provenance, and a machine learning professional. In these encounters, the professionals noted they were not analyzing calling context trees, but were interested because our visualization showed two metrics in tandem. These encounters extended into larger conversations about how this visual design for hierarchical data could work for their specific needs. Although not definitive, we believe that this indicates that this particular design is transferable across multiple computing domains working with hierarchical data. 



\section{Related Work}
\label{sec:related}

We discuss related work on the topics of integrating scripting with interactive visualization, calling context tree visualizations, and general multivariate hierarchical approaches.

\subsection{Integrating Scripting and Graphical Interfaces}

Previous works~\cite{hartmann2008design,bigelow2016iterating,hempel2019sketch} have sought to balance the trade off between the flexibility of scripting and the ease-of-use of graphical user interfaces. Hanpuku~\cite{bigelow2016iterating} enables visualization authoring through an iterative edit loop between D3~\cite{bostock-d3-2011} and Adobe Illustrator. \textsc{Sketch-N-Sketch}~\cite{hempel2019sketch} supports SVG authoring through both direct manipulation and Javascript code, concurrently updating both the graphical output and and source code in an integrated system. 

Computational notebooks such as Jupyter~\cite{jupyter}, R-Markdown~\cite{rmarkdown-manual,rmarkdown-book,rmarkdown-cookbook}, and Observable~\cite{Observable} provide a literate~\cite{Knuth1984} environment combining executable code, the output of that code including visualizations, and formatted narrative text and documentation. Notebooks can package code and results with explanation, making them amenable to sharing as well as a convenient starting points for further iterative analysis.

In Jupyter in particular, code is input into individually executable {\em cells}, which when run may produce output, including visualizations. While these visualizations may be used in a user's analysis process, there is often a manual step in which users express their findings again in code due to the separation between the visualization and code cells. Several libraries~\cite{Roundtrip,b2,mage} have sought to better integrate scripting and graphical interfaces in Jupyter. B2~\cite{b2} tracks data queries from both code and visualization to integrate scripted data queries and visual data queries. The \texttt{mage}~\cite{mage} library allows creation of interactive widgets linked with code to support both graphical and scripting interfaces. 

These prior works have focused on design and implementation of paradigms that permit users to benefit from both scripting and graphical interfaces. Our work focuses on designing interactive visualizations assuming this general paradigm in a complex data exploration scenario. 

\subsection{Visualizing Calling Context Trees}
  Calling Context Trees (CCTS) are frequently visualized use node-link diagrams~\cite{Ellson03graphvizand, weidendorfer2004tool, DeRose2007, Ahn2009STAT, Lin2010, nguyen2016vipact}, indented trees~\cite{geimer2010further, adhianto2010hpctoolkit, bell2003paraprof}, or icicle plots (``flame graphs'') and sunbursts~\cite{Adamoli2010Trevis, moret2010exploring, Bezemer2015FlameGraph}. Across all these idioms, node coloring is frequently used to encode an attribute. Icicle plots and sunbursts permit length and arc length respectively to encode attributes. Indented tree have been associated with a table (i.e., tree+table idiom)~\cite{adhianto2010hpctoolkit} to show several attributes. DeRose et al.~\cite{DeRose2007} used node height, width, and embedded bar charts to show a collection of attributes in a node-link approach. To leverage familiarity, we chose the prevalent node-link approach, albeit with a different bivariate encoding to support our constraints and common tasks. We discuss our design in \autoref{sec:cctvis}. 
  
  Some performance analysis tasks~\cite{bergel2017visual} require analyzing large collections of similar CCTs, e.g., when they are represented per parallel CPU, resulting in aggregative approaches. VIPACT~\cite{nguyen2016vipact} shows distributions of per-CPU data as gradient halos on nodes. Xie et al.~\cite{xie2018ccts} prioritizes locating anomalous CCTs in execution traces through an embedding approach. CallFlow~\cite{nguyen2019visualizing, kesavan2021scalable} prioritizes showing the distribution of attributes and aggregates call sites by their module, using a Sankey diagram with embedded histograms and other distribution indicators as well as linked statistical charts to represent the resulting structure and encode performance metrics. Comparison across per-CPU trees was not a goal discussed in our interviews with target users, and thus we focus on tasks with pre-aggregated per-program trees.
       
\subsection{Multivariate Encodings of Other Hierarchical Data}
 
    Tree+table~\cite{nobre_juniper_2018} and glyph-based node-link diagrams~\cite{dong_pansytree_2020,darzi_functree2_2019} have been used in other contexts. Like with CCTs, as node-link diagrams grow in size, the area available on each node shrinks, making even one node-based variable difficult to convey. Other approaches to multivariate encoding for hierarchical data eschew node-link visual representations \cite{zheng_visualization_2021,keim_visual_nodate, wittenburg_equal-height_2018, zhou_multi-level_2020, tekusova_visualizing_2008,noauthor_closer_nodate} due to scalability concerns. Please see the surveys by Schulz~\cite{Schulz2011TreeVis} and Nobre et al.~\cite{Nobre2019Multivariate} for further discussion. We prioritized familiarity with node-link approaches in our visualization, dealing with space constraints through interactivity elision and pruning. 


\section{Design Process}
\label{sec:designprocess}

We followed the iterative design study methodology of Sedlmair et al.~\cite{sedlmair2012design}, informed by the criteria for rigor proposed by Meyer and Dykes~\cite{meyer2019criteria}. We describe our process in these frameworks, our resulting data and task analyses, and our processes for rigor.

Our core research team includes two visualization experts and a Hatchet project manager who met weekly regarding the project. We also met at least monthly with the larger Hatchet development team. The visualization team interviewed two regular Hatchet users, identified as {\em frontline analysts} during the design phase. The Hatchet project manager, who met more frequently with the frontline analysts as they were in the same organization, communicated needs and requests from them and also gathered example use cases and notebooks. 

\subsection{Data}
\label{sec:data}

Our data is a calling context tree as represented by a Hatchet GraphFrame object. Thus, we have a tree network structure where each node has associated quantitative attributes. The set of attributes is fixed across the entire tree. There may be any number of attributes. Typically, both inclusive and exclusive time will be part of the attribute list.

The size of the tree depends on the profiling tool which originally collected the data. The examples we use here were collected in one of two schemes. Caliper~\cite{caliper} collects data on functions that are manually marked, typically resulting in small trees focused on a narrow region an analyst sought to explore. HPCToolkit~\cite{adhianto2010hpctoolkit} collects data without human annotation and thus covers a much more complete portion of the code, resulting in a large and detailed tree. Although specific tree sizes vary depending on the program being analyzed, they do scale up with application parallelization. As an example, the Kripke datasets we used for our evaluation were approximately 1500 nodes and 2700 nodes, respectively. For massively parallel runs these can scale up to tens of thousands of nodes.

Though both tools collect CCTs per parallel process, our interviews with frontline analysts did not discuss viewing the distribution across those processes. Rather, they started their Hatchet analyses with an aggregated per-run tree. We thus focused on that aggregated data. 

\subsection{Task Analysis}
\label{sec:task_anl}

Our task analysis is informed by semi-structured interviews with two active Hatchet users, artifacts collected from them, feature requests communicated through the Hatchet project manager, and ongoing meetings with the Hatchet team. The supplemental materials include quotes and memos supporting each task.

We were aware from project inception that the target delivery mechanism for the visualization was Jupyter notebooks. Development teams at our collaborators' organization are made aware of new features through sample notebooks available through a common web portal. Thus, when analyzing tasks, we considered whether the visualization or scripting capabilities should support them. \autoref{fig:taskanalysis} shows an overview of tasks and their contexts.

\vspace{1ex}

\noindent\textbf{[T1] Call Path Tracing.} Users wanted to follow the chain of function calls leading to or from nodes of interest. This provides context to a node of interest, orienting users with respect to code, and allows them to analyze the call stack.

\vspace{0.5ex}

\noindent\textit{Subtasks}
    \begin{enumerate}[topsep=0pt]
        \itemsep=0.25ex
        \item Trace the path from a node towards the root. Often use for orientation. (Visual)
        \item Identify ancestors and descendants of nodes. Used for analysis. (Visual)
    \end{enumerate}

\vspace{1ex}

\noindent\textbf{[T2] Tree Comparison.} Performance is often analyzed by comparison between two executions of the same program run with different resources, underlying libraries, or versions. Users want to see how node attributes differed between runs and how the structure of the tree changed between software updates.

\vspace{0.5ex}

\noindent\textit{Subtasks}
    \begin{enumerate}[topsep=0pt]
        \itemsep=0.25ex
        \item Compare node metrics between scaling runs (Script)
        \item Compare tree structures across different implementations (Visual)
    \end{enumerate}


\vspace{1ex}

\noindent\textbf{[T3] Metric Analysis.} Users want to know the distribution of attributes across nodes and which nodes have extreme or abnormal attributes. For example, if the speedup metric for a node is unexpectedly small, users may want to investigate it further. Our discussions with stakeholders underscored a common need for bivariate analysis: users wanted to know individual metrics like speedup in the context of how long a node executed. A node with poor speedup might not matter much if the node's execution time is already short.


\vspace{0.5ex}

\noindent\textit{Subtasks}
    \begin{enumerate}[topsep=0pt]
        \itemsep=0.25ex
        \item Find nodes with extreme single metric values (Script, Visual)
        \item Find meaningful outliers through bivariate analysis (Visual)
        \item Analyze how individual functions scale with more processing units (Script)
    \end{enumerate}

\vspace{1ex}

\noindent\textbf{[T4] Tree Simplification.} Users want to simplify the tree to focus on subtrees of interest for a particular analysis or to elide nodes that represent code they cannot change. In some cases, these nodes represent details that do not match their level of abstraction in thinking about the code. Users would like to aggregate these to maintain context without details. The concept of ``interest'' changes depending on the particular analysis. Users can describe a wide variety of queries and filters to retrieve or remove nodes.


\vspace{0.5ex}

\noindent\textit{Subtasks}
    \begin{enumerate}[topsep=0pt]
    \itemsep=0.25ex
    \item Elide subtrees not relevant to current analysis (Visual)
    \item Elide nodes based on a metric (Visual)
    \item Aggregate subtrees or internal paths based on structure and function identity (Visual)
    \item Filter based on some other complex criteria. For example, ``Remove all nodes in a specific library'' (Scripting)
    \end{enumerate}
    
    
\vspace{1ex}

\noindent\textbf{[T5] Save, Transfer, \& Recover Modifications.} Users wish to reproduce the modifications they made to the tree, for use across different analysis sessions or when sharing with others. They may make changes either visually or through scripting and then transfer those changes between the two contexts.

\vspace{0.5ex}

\noindent\textit{Subtasks}
    \begin{enumerate}[topsep=0pt]
        \itemsep=0.25ex
        \item Extracting the tree state from the visualization (Visual, Scripting)
        \item Store the tree state for future use (Scripting)
        \item Recover the tree state (Scripting)
    \end{enumerate}


\subsection{Addressing Rigor}
\label{sec:rigor}

Meyer and Dykes' describe six criteria for rigor in a design study. We describe how we have structured our study to fulfill these criteria. 

The {\sc Informed} criterion addresses the preparedness for the research. Our team has significant experience in both visualization and performance analysis, studied related work in CCTs and design methodologies, and collected diverse use examples through our collaborators.

The {\sc Reflexive} criterion urges researchers to reflect upon their biases and actions. We generated abundant notes and pages of discussions around design and implementation. We had weekly contact with collaborators regarding the project. We also shared a preliminary version of the work~\cite{scullyAllisonPoster} with the HPC community and gathered feedback that informed the design.

The {\sc Abundant} criterion suggests numerous artifacts and perspectives, long-term collaborations, and rapid prototyping. Our long-term collaboration starts in Fall 2019, with regular meetings starting in Summer 2020, though the CCT visualization was not a focus until Summer 2021. We generated 19 pages of meeting records, five pages of pen-and-paper-designs, five pages of task analysis, five pages of evaluation design, transcripts of two hour-long interviews, two slide presentations, and a research poster. In support of the {\sc Transparent} criterion, we have made many of these available in supplemental material or open-source repositories. We also aim to highlight limitations where they apply throughout this document.

We cannot claim to meet the {\sc Plausibility} and {\sc Resonance} criteria, as those require community validation. We however support {\sc Plausiblity} through evidence of our {\sc Abundant} and {\sc Reflexive} methods. We discuss transferability in \autoref{sec:reflections}.
