\section{Details of Language Translations}\label{app:lang-eval}

The tables below describe the details of all 18\xspace language translations. Technical information regarding running experiments and evaluating generated programs can be found at the MultiPL-E website and in our code repository. Here we address language-specific decisions that are relevant to the prompt translation task. Specifically, we outline the following details: 
\begin{enumerate}
    \item The language version used as a reference for creating the value-to-value translation. 
    \item The stop tokens used for signaling the end of program generation. Across languages these reflect terms that begin and/or start code blocks (variations of \texttt{\textbackslash n\}} are common).
    \item Details about prompt creation. This sections highlight the choice of comments and any necessary preamble information (e.g., the opening tag \texttt{<?php} in PHP)
    \item Information about mapping values and/or types from Python. Notes here describe places where case-by-case decisions need to be made, or a language's limitations required not converting a subset of values and/or types. Omitted discussions represents straightforward conversions (e.g., integers in Perl). 

\end{enumerate}



\input{lang-tables/bash-lang-table}

\input{lang-tables/cpp-lang-table}

\input{lang-tables/csharp}

\input{lang-tables/d-lang-table}

\input{lang-tables/go-lang-table}

\input{lang-tables/java-lang-table}

\input{lang-tables/javascript-lang-table}

\input{lang-tables/julia-lang-table}

\input{lang-tables/lua-lang-table}

\input{lang-tables/perl-lang-table}

\input{lang-tables/php-lang-table}

\input{lang-tables/py-lang-table}

\input{lang-tables/r-lang-table}

\input{lang-tables/racket-lang-table}

\input{lang-tables/ruby-lang-table}

\input{lang-tables/rust-lang-table}

\input{lang-tables/scala-lang-table}

\input{lang-tables/swift-lang-table}

\input{lang-tables/typescript-lang-table}

\section{Datasheet}

The datasheet below is derived from the following work \cite{gebru2021datasheets}:
\begin{quote}
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. "Datasheets for datasets." Communications of the ACM, 2021, 86-92.
\end{quote}

\paragraph{Motivation}

\begin{itemize}

\item \textbf{For what purpose was the dataset created?}

The dataset was originally created to evaluate the performance of the large language language model. It was translated from Python to other programming languages to allow for evaluation of large language models on other programming languages.

\item \textbf{Who created the dataset?}

It was originally created by \citet{chen2021evaluating} and modified by the authors of this paper.

\item \textbf{Who funded the creation of the dataset?}
\ifanon
\emph{Omitted for review.}
\else
This work was partially supported by the National Science Foundation.
\fi

\end{itemize}

\paragraph{Composition}

\begin{itemize}

\item \textbf{What do the instances that comprise the dataset represent?}

The instances of the dataset represent programming problems in 18 programming languages.

\item \textbf{How many instances are there in total?}

There are 2,898 instances (the modified set of 161 Python problems multiplied by 18 programming languages).

\item \textbf{Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set?}

The dataset cleans the original dataset and excludes 3 of 164 problems as described in \cref{limitations}.

\item \textbf{What data does each instance consist of?}
Each instance is a programming problem with a problem description in natural language, a function signature, and unit tests.

\item \textbf{Is there a label or target associated with each instance?}

Each instance is numbered and labeled by the name of the function it tests and the language it is written in.

\item \textbf{Is the dataset self-contained, or does it link to or otherwise rely on external resources?}

The dataset is self-contained.

\end{itemize}

\paragraph{Collection process}

\begin{itemize}

\item \textbf{How was the data associated with each instance acquired?}

The original Python dataset was manually cleaned. The versions for other programming languages and prompt variations were produced by a suite of compilers.




\item \textbf{Over what timeframe was the data collected?} 

May--August 2022

\item \textbf{Were any ethical review processes conducted?}

Not applicable. The dataset adapts a open source dataset released under the terms of the MIT license.

\end{itemize}

\paragraph{Preprocessing/cleaning/labeling}

\begin{itemize}

\item \textbf{Was any preprocessing/cleaning/labeling of the data done?}

We added missing type annotations, formatted examples to use docstrings consistently, and changed random tests into unit tests in two problems.

\item \textbf{Was the "raw" data saved in addition to the preprocessed/cleaned/labeled data?}

The raw data is available at \url{https://github.com/openai/human-eval}.

\item {Is the software that was used to preprocess/clean/label the data available?}

The cleaning process described above was manual.

\end{itemize}

\paragraph{Uses}

\begin{itemize}

\item \textbf{Has the dataset been used for any tasks already?}

The dataset has been used for the NL2Code task and for comparing the performance of two LLMs of code.

\item \textbf{Is there a repository that links to any or all papers or systems that use the dataset?}

\ifanon
\emph{Omitted for review}
\else
\url{https://github.com/nuprl/MultiPL-E}
\fi

\item \textbf{What other tasks could the dataset be used for?} 

The dataset could be used to evaluate other LLMs of code, or potentially to improve their performance.



\end{itemize}

\paragraph{Distribution}

\begin{itemize}

\item \textbf{Will the dataset be distributed to third parties outside of the entity? }

Yes. 

\item \textbf{How will the dataset be distributed?}

The dataset is publicly available at
\ifanon
\emph{Omitted for review}
\else
\url{https://github.com/nuprl/MultiPL-E}
\fi

\item \textbf{When will the dataset be distributed?}

Immediately.

\item \textbf{Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)?}

No.

\item \textbf{Have any third parties imposed IP-based or other restrictions on the data associated with the instances?}

No.

\item \textbf{Do any export controls or other regulatory restrictions apply to the dataset or to individual instances?}

No.

\end{itemize}

\paragraph{Maintenance}

\begin{itemize}

\item \textbf{Who will be supporting/hosting/maintaining the dataset?}

The original authors.

\item \textbf{How can the owner/curator/manager of the dataset be contacted?}

See the dataset website.

\item \textbf{Is there an erratum?}

No. Any identified and confirmed errors will be acknowledged as part of the repository. 

\item \textbf{Will the dataset be updated (for example, to correct labeling errors, add new instances, delete instances)?}

Yes.

\item \textbf{Will older versions of the dataset continue to be supported/hosted/maintained?}

Yes.

\item \textbf{If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so?}

Yes, as described in the paper and website.

\end{itemize}

\section{Complete Statistical Findings}\label{app:mem}

We use binomial mixed-effects models fitted with the lme4 library in R for significance testing. A binomial distribution is appropriate because our outcomes consist of proportions of successes and failures; we use the number of completions (200, except in rare failure cases) as weights. 

We fit models to the Codex pass@1 completion rates in all experiments reported below. We treat problem number as a random effect to account for variability inherent to per-problem differences. For comparisons that do not break down effects by language, we also include language as a random effect. We include random slopes and intercepts for random effects except where noted.

Values that are statistically significant with a threshold of $p=0.5$ are displayed in \textbf{bold}.

\subsection{Mixed-Effects Results from Section \ref{subsec:multilingual}} 

To quantify the differences in performance among programming languages, a model with a fixed effect of programming language and random effects for problem number was fitted to the Codex pass@1 data.

Dummy coding was used with Python as the reference level; slopes for each language indicate differences between the pass@1 rate for Python and that language.

Table \ref{tab:mem_lang} shows the full estimates found by the model.

\begin{table}[H]
    \centering
    \begin{tabular}{|llll|}\hline
Fixed effects&$\widehat{\beta}$&$z$&$p$\\\hline
Intercept&-0.48 (+/- 0.4)&-1.1&0.27\\
Bash&-2.59 (+/- 0.3)&-7.7&< \textbf{0.0001}\\
C++&0.10 (+/- 0.4)&0.3&0.77\\    
C\#&-4.09 (+/- 0.6)&-7.2&< \textbf{0.0001}\\
D&-4.79 (+/- 0.5)&-9.7&< \textbf{0.0001}\\
Go&-2.61 (+/- 0.4)&-6.5&< \textbf{0.0001}\\
Java&-1.28 (+/- 0.3)&-3.9&< \textbf{0.0001}\\
Julia&-1.91 (+/- 0.4)&-5.2&< \textbf{0.0001}\\
JavaScript&-0.27 (+/- 0.3)&-0.8&0.43\\    
Lua&-1.04 (+/- 0.4)&-2.8&\textbf{0.005}\\ 
Perl&-2.0 (+/- 0.4)&-5.3&< \textbf{0.0001}\\
PHP&-0.30 (+/- 0.4)&-0.8&0.40\\  
R&-3.69 (+/- 0.4)&-8.5&< \textbf{0.0001}\\
Ruby&-0.68 (+/- 0.3)&-2.3&\textbf{0.024}\\ 
Racket&-3.78 (+/- 0.4)&-9.8&< \textbf{0.0001}\\
Rust&-1.07 (+/- 0.3)&-3.4&< \textbf{0.0001}\\
Scala&-0.52 (+/- 0.3)&-1.6&0.10\\   
Swift&-1.8 (+/- 0.3)&-5.7&< \textbf{0.0001}\\
TypeScript&-0.27 (+/- 0.3)&-0.9&0.39\\\hline
    \end{tabular}
    \caption{Mixed-effects results for Codex language comparison}\label{tab:mem_lang}
\end{table}

A similar model was fit to the InCoder pass@1 data, but without random slopes, because the very low pass rates for many problems makes the random effects estimates unstable. Table \ref{tab:mem_lang_incoder} shows the full estimates found by the model.

\begin{table}[H]
    \centering
    \begin{tabular}{|llll|}\hline
Fixed effects&$\widehat{\beta}$&$z$&$p$\\\hline
Intercept&-4.35 (+/- 0.3)&-14.5&< \textbf{0.0001}\\
Bash&-3.3 (+/- 0.03)&-131.1&< \textbf{0.0001}\\
C++&-0.99 (+/- 0.02)&-57.1&< \textbf{0.0001}\\
C\#&-1.94 (+/- 0.02)&-100.9&< \textbf{0.0001}\\
D&-3.97 (+/- 0.03)&-139.5&< \textbf{0.0001}\\
Go&-1.62 (+/- 0.02)&-86.3&< \textbf{0.0001}\\
Java&-1.29 (+/- 0.02)&-72.5&< \textbf{0.0001}\\
Julia&-4.90 (+/- 0.04)&-132.6&< \textbf{0.0001}\\
JavaScript&-0.97 (+/- 0.02)&-56.1&< \textbf{0.0001}\\
Lua&-2.21 (+/- 0.02)&-111.2&< \textbf{0.0001}\\
Perl&-2.43 (+/- 0.02)&-118.2&< \textbf{0.0001}\\
PHP&-1.76 (+/- 0.02)&-93.6&< \textbf{0.0001}\\
R&-2.80 (+/- 0.02)&-128.2&< \textbf{0.0001}\\
Ruby&-1.87 (+/- 0.02)&-98.4&< \textbf{0.0001}\\
Racket&-3.41 (+/- 0.02)&-138.2&< \textbf{0.0001}\\
Rust&-2.74 (+/- 0.02)&-125.3&< \textbf{0.0001}\\
Scala&-1.92 (+/- 0.02)&-100.3&< \textbf{0.0001}\\
Swift&-2.05 (+/- 0.02)&-105.3&< \textbf{0.0001}\\
TypeScript&-1.11 (+/- 0.02)&-63.6&< \textbf{0.0001}\\\hline
    \end{tabular}
    \caption{Mixed-effects results for InCoder language comparison}\label{tab:mem_lang_incoder}
\end{table}


\subsection{Mixed-Effects Results for Section \ref{subsec:ablation}} 

A mixed-effects model was fit to the InCoder pass@1 rates to explore how the translation components affect its performance.  This model compared InCoder pass@1 rates for four experiments: Doctest-Only Translation, Full Translation, No Translation, and Remove Doctests. Experiment was treated as a fixed-effect, with Python and Doctest-Only Translation as the reference levels. Random effects for language were included; random effects for problem were not included, as the extremely low pass rates for many problems caused instability in estimating them. Table \ref{tab:mem_ablation_small_incoder} shows the full estimates found by the model.

\begin{table}[H]
    \centering
    \begin{tabular}{|llll|}\hline
Fixed effects&$\widehat{\beta}$&$z$&$p$\\\hline
(Intercept)&-2.97 (+/- 0.2)&-15.6&< \textbf{0.001}\\
Remove&0.20 (+/- 0.07)&2.8&\textbf{0.005}\\ 
No Translation&0.03 (+/- 0.03)&1.0&0.32\\    
Full Translation&0.02 (+/- 0.02)&1.3&0.20\\\hline  
    \end{tabular}
    \caption{Mixed-effects results for the InCoder ablation study}\label{tab:mem_ablation_small_incoder}
\end{table}

A similar mixed-effects model was fit to understand the impact of translating natural language terms and doctests on Codex performance. This model compared Codex pass@1 rates for four experiments: Doctest-Only Translation, Full Translation, No Translation, and Remove Doctests. Experiment was treated as a fixed-effect, with Python and Doctest-Only Translation as the reference levels. Random effects for problem and language were included. Table \ref{tab:mem_ablation_small} shows the full estimates found by the model.

\begin{table}[H]
    \centering
    \begin{tabular}{|llll|}\hline
Fixed effects&$\widehat{\beta}$&$z$&$p$\\\hline
(Intercept)&-1.24 (+/- 0.3)&-4.4&< \textbf{0.0001}\\
Full Translation&0.04 (+/- 0.02)&2.2&\textbf{0.03}\\  
No Translation&-0.08 (+/- 0.1)&-1.3&0.2\\    
Remove&-0.35 (+/- 0.1)&-3.8&< \textbf{0.0001}\\\hline
    \end{tabular}
    \caption{Mixed-effects results for the Codex ablation study}\label{tab:mem_ablation_small}
\end{table}

A second model was fitted for Codex treating both Language and Experiment as fixed-effects, with interaction terms included. For this model, we include only random intercepts but not random slopes for Problem, because of the large number of effects the model must estimate. Tables \ref{tab:mem_ablation} and \ref{tab:mem_ablation_interaction} show the full estimates found by the model.

\begin{table}[H]
    \centering
    \begin{tabular}{|llll|}\hline
Fixed effects&$\widehat{\beta}$&$z$&$p$\\\hline\hline
(Intercept)&-0.44 (+/- 0.2)&-2.2&\textbf{0.03}\\  
Full Translation&-0.006 (+/- 0.02)&-0.3&0.78\\
No Translation&-0.07 (+/- 0.02)&-3.1&\textbf{0.002}\\ 
Remove&-0.14 (+/- 0.02)&-6.6&< \textbf{0.0001}\\\hline\hline
Bash&-1.75 (+/- 0.02)&-77.8&< \textbf{0.0001}\\
C++&0.17 (+/- 0.02)&8.1&< \textbf{0.0001}\\
C\#&-1.45 (+/- 0.02)&-65.7&< \textbf{0.0001}\\
D&-1.97 (+/- 0.02)&-86.5&< \textbf{0.0001}\\
Go&-1.14 (+/- 0.02)&-52.5&< \textbf{0.0001}\\
Java&-0.63 (+/- 0.02)&-29.2&< \textbf{0.0001}\\
Julia&-0.87 (+/- 0.02)&-40.4&< \textbf{0.0001}\\
JavaScript&0.15 (+/- 0.02)&7.2&< \textbf{0.0001}\\
Lua&-0.48 (+/- 0.02)&-22.7&< \textbf{0.0001}\\
Perl&-1.10 (+/- 0.02)&-51.0&< \textbf{0.0001}\\
PHP&-0.006 (+/- 0.02)&-0.3&0.76\\    
R&-2.02 (+/- 0.02)&-88.8&< \textbf{0.0001}\\
Ruby&-0.31 (+/- 0.02)&-14.4&< \textbf{0.0001}\\
Racket&-2.36 (+/- 0.02)&-100.3&< \textbf{0.0001}\\
Rust&-0.30 (+/- 0.02)&-14.0&< \textbf{0.0001}\\
Scala&-0.24 (+/- 0.02)&-11.0&< \textbf{0.0001}\\
Swift&-0.62 (+/- 0.02)&-28.9&< \textbf{0.0001}\\
TypeScript&0.10 (+/- 0.02)&4.8&< \textbf{0.0001}\\\hline
\end{tabular}
    \caption{Mixed-effects results for the Codex ablation study by language, main effects}\label{tab:mem_ablation}
\end{table}

\begin{table}[H]
    \centering
 \begin{tabular}{|llll|}\hline
Fixed effects&$\widehat{\beta}$&$z$&$p$\\\hline\hline
Full Translation*Bash&-0.02 (+/- 0.03)&-0.5&0.58\\    
No Translation*Bash&-0.47 (+/- 0.03)&-14.4&< \textbf{0.0001}\\
Remove*Bash&-0.59 (+/- 0.03)&-17.9&< \textbf{0.0001}\\\hline
Full Translation*C++&-0.03 (+/- 0.03)&-0.9&0.35\\    
No Translation*C++&-0.15 (+/- 0.03)&-5.0&< \textbf{0.0001}\\
Remove*C++&-0.11 (+/- 0.03)&-3.7&\textbf{0.0002}\\\hline
Full Translation*C\#&-0.02 (+/- 0.03)&-0.6&0.58\\    
No Translation*C\#&0.05 (+/- 0.03)&1.6&0.10\\    
Remove*C\#&0.2 (+/- 0.03)&6.9&< \textbf{0.0001}\\\hline
Full Translation*D&0.02 (+/- 0.03)&0.5&0.59\\    
No Translation*D&0.20 (+/- 0.03)&6.4&< \textbf{0.0001}\\
Remove*D&0.10 (+/- 0.03)&3.2&\textbf{0.001}\\ \hline
Full Translation*Go&-0.03 (+/- 0.03)&-0.8&0.41\\    
No Translation*Go&-0.03 (+/- 0.03)&-1.0&0.32\\    
Remove*Go&0.05 (+/- 0.03)&1.8&0.08\\  \hline
Full Translation*Java&0.12 (+/- 0.03)&3.9&< \textbf{0.0001}\\
No Translation*Java&0.14 (+/- 0.03)&4.5&< \textbf{0.0001}\\
Remove*Java&0.12 (+/- 0.03)&4.0&< \textbf{0.0001}\\\hline
Full Translation*Julia&0.05 (+/- 0.03)&1.8&0.07\\  
No Translation*Julia&0.05 (+/- 0.03)&1.7&0.10\\  
Remove*Julia&-0.09 (+/- 0.03)&-2.9&\textbf{0.004}\\ \hline
Full Translation*JavaScript&0.01 (+/- 0.03)&0.4&0.66\\    
No Translation*JavaScript&0.08 (+/- 0.03)&2.6&\textbf{0.01}\\  
Remove*JavaScript&-0.09 (+/-0.03)&-2.8&\textbf{0.005}\\\hline 
Full Translation*Lua&0.06 (+/- 0.03)&1.9&0.06\\  
No Translation*Lua&-0.04 (+/- 0.03)&-1.3&0.19\\    
Remove*Lua&-0.12 (+/- 0.03)&-3.8&\textbf{0.0001}\\\hline
Full Translation*Perl&0.23 (+/- 0.03)&7.5&< \textbf{0.0001}\\
No Translation*Perl&-0.25 (+/- 0.03)&-8.1&< \textbf{0.0001}\\
Remove*Perl&-0.21 (+/- 0.03)&-6.9&< \textbf{0.0001}\\\hline
Full Translation*PHP&0.06 (+/- 0.03)&1.8&0.07\\  
No Translation*PHP&0.02 (+/- 0.03)&0.6&0.58\\    
Remove*PHP&-0.26 (+/- 0.03)&-8.6&< \textbf{0.0001}\\\hline
Full Translation*R&0.22 (+/- 0.03)&7.0&< \textbf{0.0001}\\
No Translation*R&0.26 (+/- 0.03)&8.1&< \textbf{0.0001}\\
Remove*R&-0.11 (+/- 0.03)&-3.5&{0.0004}\\\hline
Full Translation*Ruby&0.012 (+/- 0.03)&0.4&0.68\\    
No Translation*Ruby&0.02 (+/- 0.03)&0.5&0.58\\    
Remove*Ruby&-0.19 (+/- 0.03)&-6.1&< \textbf{0.0001}\\\hline
Full Translation*Racket&0.04 (+/- 0.03)&1.2&0.23\\    
No Translation*Racket&-0.07 (+/- 0.03)&-2.0&0.05\\  
Remove*Racket&-0.21 (+/- 0.03)&-6.1&< \textbf{0.0001}\\\hline
Full Translation*Rust&0.03 (+/- 0.03)&1.0&0.31\\    
No Translation*Rust&-0.04 (+/- 0.03)&-1.5&0.14\\    
Remove*Rust&-0.34 (+/- 0.03)&-11.2&< \textbf{0.0001}\\\hline
Full Translation*Scala&0.01 (+/- 0.03)&0.5&0.64\\    
No Translation*Scala&0.17 (+/- 0.03)&5.5&< \textbf{0.0001}\\
Remove*Scala&0.03 (+/- 0.03)&1.1&0.26\\\hline    
Full Translation*Swift&-0.01 (+/- 0.03)&-0.5&0.63\\    
No Translation*Swift&-0.39 (+/- 0.03)&-13.0&< \textbf{0.0001}\\
Remove*Swift&-0.24 (+/- 0.03)&-8.0&< \textbf{0.0001}\\\hline
Full Translation*TypeScript&0.05 (+/- 0.03)&1.6&0.11\\    
No Translation*TypeScript&0.11 (+/- 0.03)&3.8&\textbf{0.0002}\\
Remove*TypeScript&-0.25 (+/- 0.03)&-8.4&< \textbf{0.0001}\\\hline\hline
\end{tabular}
    \caption{Mixed-effects results for the Codex ablation study by language, interaction effects}\label{tab:mem_ablation_interaction}
\end{table}

\subsection{Mixed-Effects Results for Section \ref{subsec:eval-freq} and Section \ref{subsec:static-types}} 

A mixed-effects model treating Frequency and Static Type-checking as fixed-effects, with random effects for language and problem, was fit to the data. Interaction terms were included for Typed with each frequency category. Table \ref{tab:mem_freq_and_types} shows the full estimates found by the model.

\begin{table}[H]
    \centering
    \begin{tabular}{|llll|}\hline
Fixed effects&$\widehat{\beta}$&$z$&$p$\\\hline
(Intercept)&-0.47 (+/- 0.3)&-1.5&0.14\\    
Low&-1.95 (+/- 0.4)&-5.0&< \textbf{0.001}\\
Medium&-0.31 (+/- 0.3)&-1.2&0.22\\    
Niche&-1.7 (+/- 0.6)&-3.0&\textbf{0.002}\\ 
Typed&-0.3 (+/- 0.3)&-1.0&0.33\\    
Low*Typed&1.49 (+/- 0.5)&3.0&\textbf{0.003}\\ 
Medium*Typed&-1.49 (+/- 0.5)&-3.1&\textbf{0.002}\\ 
Niche*Typed&-0.31 (+/- 0.8)&-0.4&0.70\\\hline
    \end{tabular}
    \caption{Mixed-effects results for language frequency and static type-checking comparison}\label{tab:mem_freq_and_types}
\end{table}

\subsection{Mixed-Effects Results from Section \ref{subsec:static-types}} 

A mixed-effects model testing the effect of removing Python type annotations was fit treating Annotations as a fixed-effect and problem as a random effect. Table \ref{tab:mem_python_types} shows the full estimates found by the model.

\begin{figure}[H]
    \centering
    \includegraphics[width=0.5\textwidth]{images/pyexpt.pdf}
    \caption{Impact of Python type annotations on Codex performance}
    \label{fig:pyexpt}
\end{figure}

\begin{table}[H]
    \centering
    \begin{tabular}{|llll|}\hline
Fixed effects&$\widehat{\beta}$&$z$&$p$\\\hline
Intercept&-0.26 (+/- 0.5)&-0.5&0.60\\
Annotations&-0.21 (+/- ).2)&-1.2&0.22\\\hline
    \end{tabular}
    \caption{Mixed-effects results for Python type annotation experiments}\label{tab:mem_python_types}
\end{table}

A mixed-effects model testing the effect of weakening TypeScript annotations to Any and running without static type-checking was fit. There were three fixed-effects: Any, comparing TypeScript with precise types to TypeScript with all Any types; JS, comparing TypeScript with annotations to JavaScript; and NoCheck, comparing TypeScript with and without static type-checking. 
Table \ref{tab:mem_script} shows the full estimates found by the model.

\begin{figure}[H]
    \centering
    \includegraphics[width=0.75\textwidth]{images/tsexpt.pdf}
    \caption{Impact of type-checking and precise type annotations on TypeScript performance}
    \label{fig:tsexpt}
\end{figure}

\begin{table}[H]
    \centering
    \begin{tabular}{|llll|}\hline
Fixed effects&$\widehat{\beta}$&$z$&$p$\\\hline
Intercept&-0.24 (+/- 0.4)&-0.6&0.56\\
JavaScript&-0.03 (+/- 0.03)&-1.2&0.23\\
Any Types&-0.38 (+/- 0.03)&-13.3&< \textbf{0.001}\\
NoCheck&0.04 (+/- 0.03)&1.5&0.14\\\hline
    \end{tabular}
    \caption{Mixed-effects results for TypeScript experiments}\label{tab:mem_script}
\end{table}

\subsection{Mixed-Effects Results from Section \ref{subsec:prompt-engineering}} 

Tables \ref{tab:mem_comments_php} and \ref{tab:mem_comments_rkt} shows the results of singe-line versus multi-line comments for PHP and Racket. Separate models were run for each language, with multi-line as a fixed effect and problem number as a random effect.

\begin{figure}[H]
    \centering
    \includegraphics[width=0.5\textwidth]{images/commentexpt.pdf}
    \caption{Impact of comment style on Codex performance for PHP and Racket}
    \label{fig:multiexpt}
\end{figure}


\begin{table}[H]
    \centering
    \begin{tabular}{|llll|}\hline
Fixed effects&$\widehat{\beta}$&$z$&$p$\\\hline
Intercept&-0.46 (+/- 0.4)&-1.2&0.22\\
Multi-line&-0.43 (+/-0.1)&-3.3&\textbf{0.001}\\\hline
    \end{tabular}
    \caption{Mixed-effect model estimates for PHP comment experiment}\label{tab:mem_comments_php}
\end{table}

\begin{table}[H]
    \centering
    \begin{tabular}{|llll|}\hline
Fixed effects&$\widehat{\beta}$&$z$&$p$\\\hline
Intercept&-4.62 (+/- 0.4)&-10.9&< \textbf{0.0001}\\
Multi-line&1.26 (+/-0.2)&6.4&< \textbf{0.0001}\\\hline
    \end{tabular}
    \caption{Mixed-effect model estimates for Racket comment experiment}\label{tab:mem_comments_rkt}
\end{table}

Table \ref{tab:mem_perl} shows the results of comparing Perl with and without an argument-naming line after the function signature. Argument-naming was treated as a fixed effect and problem number as a random effect.

\begin{figure}[H]
    \centering
    \includegraphics[width=0.5\textwidth]{images/perlexpt.pdf}
    \caption{Impact of argument-naming line on Codex performance for Perl}
    \label{fig:perlexpt}
\end{figure}


\begin{table}[H]
    \centering
    \begin{tabular}{|llll|}\hline
Fixed effects&$\widehat{\beta}$&$z$&$p$\\\hline
Intercept&-3.03 (+/- 0.4)&-7.9&<\textbf{0.0001}\\
Argument-naming&0.81 (+/-0.2)&3.6&\textbf{0.0008}\\\hline
    \end{tabular}
    \caption{Mixed-effect model estimates for Perl experiment}\label{tab:mem_perl}
\end{table}

Table \ref{tab:mem_bash} shows the results of comparing Bash with and without encoding-specifying comments. Comments and NL Translation were treated as fixed effects and problem number as a random effect; an interaction term for Comments and NL Translation was also included.

\begin{figure}[H]
    \centering
    \includegraphics[width=0.5\textwidth]{images/bashexpt.pdf}
    \caption{Impact of encoding comments and NL translation on Codex performance for Bash}
    \label{fig:bashexpt}
\end{figure}


\begin{table}[H]
    \centering
    \begin{tabular}{|llll|}\hline
Fixed effects&$\widehat{\beta}$&$z$&$p$\\\hline
Intercept&-3.09 (+/- 0.3)&-9.9&< \textbf{0.001}\\
Comments&0.01 (+/- 0.1)&0.08&0.94\\
Rewording&-0.04 (+/- 0.03)&-1.3&0.19\\
Comments*Rewording&0.08 (+/-0.4)&1.8&0.07\\\hline
    \end{tabular}
    \caption{Mixed-effect model estimates for Bash experiment}\label{tab:mem_bash}
\end{table}


\subsection{Mixed-Effects Results for Language Feature} 

We categorize problems into groups based on which Python language features they use: dictionaries, tuples, booleans, lists, or none of the above. We base these categorizations on the Python type annotations for each problem. Problems were coded 1 Tuple, List, Bool, and Dictionary if they contain a type annotation for the respective feature, and 0 otherwise. Figure \ref{fig:features} shows the performance by language on each type of problem. There are only 3 problems in the dictionary category, so these results should be interpreted with caution.

\begin{figure}[H]
    \centering
    \includegraphics[width=0.75\textwidth]{images/reworded_davinci_lang_features_pass1.pdf}
    \caption{Impact of programming language features on Codex pass@1 performance by language}
    \label{fig:features}
\end{figure}

We fit a mixed-effects model to understand how Codex pass@1 rates are affected by the language features used in the problem, using Tuple, List, Bool, and Dictionary as fixed-effects, with random effects for problem and language. Table \ref{tab:mem_features_small} shows the full estimates found by the model.

\begin{table}[H]
    \centering
    \begin{tabular}{|llll|}\hline
Fixed effects&$\widehat{\beta}$&$z$&$p$\\\hline
(Intercept)&-1.19 (+/- 0.3)&-3.5&< \textbf{0.001}\\
List&-0.15 (+/- 0.4)&-0.3&0.73\\    
Bool&0.10 (+/- 0.6)&0.2&0.86\\    
Tuple&-0.73 (+/- 0.9)&-0.8&0.40\\    
Dictionary&-3.27 (+/- 1.8)&-1.8&0.07\\\hline
    \end{tabular}
    \caption{Mixed-effects results for the impact of language features}\label{tab:mem_features_small}
\end{table}


\section{Characterization of Code Generation Errors}\label{app:errors}

This section provides details regarding our error evaluation study as overviewed in Section \ref{subsec:errors}. First we discuss the process of categorizing errors in a multi-language context. Then we provide the full set of themes, errors, and counts across the four studied languages: Python (\textsc{High}, untyped), C\# (\textsc{Medium}, typed), Swift (\textsc{Low}, typed), and Racket (\textsc{Niche}, untyped).  Finally, we showcase full code examples of a variety of errors.

\subsection{Notes on Process \& Findings}


To perform the evaluation, we chose two typed languages and two untyped languages across all four frequency categories. A language expert then performed a manual investigation of a subset of the completions to derive a set of common error types. These errors could be associated with common error labels in a language (e.g., \texttt{NameError} in Python) or an observed phenomenon (e.g., \texttt{UseofDeprecatedIdentifiers} in Swift). Then, through an iterative process of manual inspection and automatic error detection via analyzing evaluation output, we developed a set of error labels unique to each language. We then arrived at the multi-language themes and categories via discussion and consensus.

The multi-language nature of the evaluation contributes to variation between the language classifications. For instance, languages vary significantly in the specificity of their error messages. Consider the theme of \texttt{TimeoutOrInfiniteRecursion}: Python has a specific error message \texttt{RecursionError} when it encounters an infinite recursive loop, whereas Racket will simply evaluate indefinitely. As the generated standard output and standard error were used for automatic classifications, there may be variations in how errors were counted depending on the error messages and precision of string search terms.

Overall, each error label is specific to the language under study and was subject to different levels of manual assessment. Therefore, the prevalence of a theme, rather than a specific error label or even category, likely provides a better source of inter- and intra-language information. Although the four languages in our study address different language variations (typed/untyped, frequency), they are not representative of all languages in our benchmark nor additional unstudied languages. Therefore, it is likely there are error labels, themes, and potentially categories that are missing from this characterization. Errors classified under the theme ``AssertionFailed'' describe errors from generated code with correct syntax which produces incorrect output. Other than via manual inspection of the over 10,000+ errors per language, there is no clear method of more precisely classifying errors of that type.

\subsection{Complete Error Themes}

In Tables \ref{tab:py_errors} - \ref{tab:rkt_errors} below, rows with the \colorbox{black!15}{gray} background are the most frequent error in that category for the specific language. Items in \emph{italics} are errors directly referenced in Section \ref{subsec:errors}. There are around 32,000 completions for each language for our full translation. Variations in the reported counts below are due to support for a different number of prompts for each language and completions which generate multiple errors on failure. In the later case, we count all present errors.

\input{errors/python-error-table}

\input{errors/csharp-error-table}

\input{errors/swift-error-table}

\input{errors/racket-error-table}



\subsection{Error Examples Related to Section \ref{subsec:errors}}


\begin{figure}[H]
    \centering
\begin{lstlisting}[numbers=none]
class Problem {
    // Translated docstring
    public static string Intersection(Tuple<long, long> interval1, Tuple<long, long> interval2) {
        long start = Math.Max(interval1.Item1, interval2.Item1);
        long end = Math.Min(interval1.Item2, interval2.Item2);
        if (start > end) {
            return "NO";
        }
        long length = end - start + 1;
        return IsPrime(length) ? "YES" : "NO";
    }
\end{lstlisting}
    \caption{\emph{C\#}: Exhibits a local context error as it relies on a non-existent \texttt{IsPrime} method (Prompt \#127, Theme: ``UndefinedIdentifier'', Code: ``UndefinedIdentifier'')}
    \label{fig:local-context}
\end{figure}


\begin{figure}[H]
    \centering
\begin{lstlisting}[numbers=none]
#lang racket

;; You are given a non-empty list of positive integers. Return the 
greatest integer that is greater than 
;; zero, and has a frequency greater than or equal to the value of the
integer itself. 
;; The frequency of an integer is the number of times it appears in the list.
;; If no such a value exist, return -1.
;; Examples:
;; >>> (search (list 4 1 2 2 3 1))
;; 2
;; >>> (search (list 1 2 2 3 3 3 4 4 4))
;; 3
;; >>> (search (list 5 5 4 4 4))
;; -1
(define (search lst)
  (let ([freq (make-hash)])
    (for ([i lst])
      (hash-set! freq i (+ 1 (hash-ref freq i 0))))
    (let ([max-freq (apply max (hash-values freq))])
      (for ([i lst])
        (when (and (> i 0) (= max-freq (hash-ref freq i)))
          (return i))))
    -1))

\end{lstlisting}
    \caption{\emph{Racket}: A completion which uses the keyword \texttt{return}. \texttt{return} is not a keyword in Racket, but is a common keyword in Python. (Prompt \#69, Theme: ``UndefinedIdentifier'', Error: ``unboundIdentifier'' )}
    \label{fig:pythonic}
\end{figure}



\begin{figure}[H]
    \centering
\begin{lstlisting}[numbers=none]
def same_chars(s0: str, s1: str) -> bool:
    """
    docstring
    """
    return Counter(s0) == Counter(s1)
\end{lstlisting}
    \caption{\emph{Python}: This code expects a call to \texttt{import collections} to facilitate the use of \texttt{Counter} dictionary subclass (Prompt \# 54, Theme: ``UndefinedIdentifier'', Error: ``NameError'') }
    \label{fig:import-error}
\end{figure}



\begin{figure}[H]
    \centering
\begin{lstlisting}[numbers=none]
def find_closest_elements(numbers: List[float]) -> Tuple[float, float]:
    """ From a supplied list of numbers (of length at least two) select and return two
    that are the closest to each other and return them in 
    order (smaller number, larger number).
    >>> find_closest_elements([1.0, 2.0, 3.0, 4.0, 5.0, 2.2])
    (2.0, 2.2)
    >>> find_closest_elements([1.0, 2.0, 3.0, 4.0, 5.0, 2.0])
    (2.0, 2.0)
    """
    # YOUR CODE HERE
    raise NotImplementedError()
\end{lstlisting}
    \caption{\emph{Python}: The full completion for this program contains only a call to \texttt{raise}, so this code will always produce an error. (Prompt \#20, Theme: ``ExceptionInGeneratedCode'', Error: ``NotImplementedError'') }
    \label{fig:exception-purpose}
\end{figure}

\begin{figure}[H]
    \centering
\begin{lstlisting}[numbers=none]
#lang racket 

;; You are given a string s.
;; if s[i] is a letter, reverse its case from lower to upper or vise versa, 
;; otherwise keep it as it is.
;; If the string contains no letters, reverse the string.
;; The function should return the resulted string.
;; Examples
;; >>> (solve "1234")
;; "4321"
;; >>> (solve "ab")
;; "AB"
;; >>> (solve "#a@C")
;; "#A@c"
(define (solve s)
  (if (string-contains s #\a)
      (string-map (lambda (c) (if (char-alphabetic? c)
                                  (char-upcase c)
                                  (char-downcase c)))
                  s)
      (string-reverse s)))
```

### [Reverse the order of words in a string]
(https://www.codewars.com/kata/reverse-the-order-of-words-in-a-string)

```racket
#lang racket
\end{lstlisting}
    \caption{\emph{Racket}: This snippet is an example of Markdown being generated instead of Racket (it is word wrapped for presentation here). (Prompt \#161, Theme: ``GenerateAnotherLanguage'', Error: ``GeneratingWebpageOrMarkdown'')}
    \label{fig:racket-lang}
\end{figure}




\begin{figure}[H]
    \centering
\begin{lstlisting}[numbers=none]
/// This function takes an array l and returns an array l' such that
/// l' is identical to l in the indicies that are not divisible by three, 
while its values at the indicies that are divisible by three are equal
/// to the values of the corresponding indicies of l, but sorted.
/// >>> sort_third(l: [1, 2, 3])
/// [1, 2, 3]
/// >>> sort_third(l: [5, 6, 3, 4, 8, 9, 2])
/// [2, 6, 3, 4, 8, 9, 5]
func sort_third(l: [Int]) -> [Int] {
    var l = l
    for i in stride(from: 2, to: l.count, by: 3) {
        l[i] = l[i].min(l[i - 1])
        l[i - 1] = l[i].max(l[i - 1])
    }
    return l
}
\end{lstlisting}
    \caption{\emph{Swift}: This code produces the type error \texttt{error: value of optional type 'Int?' must be unwrapped to a value of type 'Int'}
    (Prompt \# 33, Theme: ``InvalidTypeConversion'', Error:``ShouldHaveUnwrappedOptional'')}
    \label{fig:type-system}
\end{figure}

\begin{figure}[H]
    \centering
\begin{lstlisting}[numbers=none]
/// Given a positive integer n, return the count of the numbers of n-digit
/// positive integers that start or end with 1.
func starts_one_ends(n: Int) -> Int {
    if n == 1 {
        return 2
    }
    if n == 2 {
        return 4
    }
    return starts_one_ends(n - 1) + starts_one_ends(n - 2)
}
\end{lstlisting}
    \caption{\emph{Swift}: A correct recursive call to \texttt{starts\_one\_ends} in Swift would be \texttt{starts\_one\_ends(n: n - 1)}. (Prompt \#83, Theme: ``LanguageSpecific'', Error: ``MissingArgumentLabel'')}
    \label{fig:label-swift}
\end{figure}

\subsection{Additional Error Examples from our Evaluation}


\begin{figure}[H]
    \centering
\begin{lstlisting}[numbers=none]
/// Translated docstring
func match_parens(lst: [String]) -> String {
    var lst = lst
    var str = ""
    var count = 0
    for i in 0..<lst.count {
        str += lst[i]
    }
    for i in 0..<str.count {
        if str[i] == "(" {
            count += 1
        } else {
            count -= 1
        }
        if count < 0 {
            return "No"
        }
    }
    if count == 0 {
        return "Yes"
    } else {
        return "No"
    }
}

\end{lstlisting}
    \caption{\emph{Swift}: This is an example of incorrect string indexing in Swift - you cannot call \texttt{str[i]} where \texttt{i} is an \texttt{Integer} (Prompt \#119, Theme: ``IncorrectAPIMethodCall'', Error: ``SubscriptStringWithInt'')}
    \label{fig:bad-swift}
\end{figure}

\begin{figure}[H]
    \centering
\begin{lstlisting}
def car_race_collision(n: int) -> int:
    """
    Imagine a road that's a perfectly straight infinitely long line.
    n cars are driving left to right;  simultaneously, a different set of n cars
    are driving right to left.   The two sets of cars start out being very far from
    each other.  All cars move in the same speed.  Two cars are said to collide
    when a car that's moving left to right hits a car that's moving right to left.
    However, the cars are infinitely sturdy and strong; as a result, they continue moving
    in their trajectory as if they did not collide.

    This function outputs the number of such collisions.
    """
    # The idea is to sort the cars by their starting positions.
    # Then, we can simulate the race by iterating through the cars
    # from left to right, and counting the number of cars that are
    # moving right to left.
    #
    # The time complexity is O(n log n) because of the sorting.
    # The space complexity is O(n) because of the sorting.
    #
    # The time complexity can be improved to O(n) by using a
    # counting sort.  The space complexity can be improved to
    # O(1) by using a bucket sort.
    left_to_right = [0] * n
    right_to_left = [0] * n
    for i in range(n):
        left_to_right[i] = int(input())
        right_to_left[i] = int(input())
    left_to_right.sort()
    right_to_left.sort()
    collisions = 0
    left_to_right_index = 0
    right_to_left_index = 0
    while left_to_right_index < n and right_to_left_index < n:
        if left_to_right[left_to_right_index] < right_to_left[right_to_left_index]:
            collisions += right_to_left_index
            left_to_right_index += 1
        else:
            right_to_left_index 
\end{lstlisting}
    \caption{\emph{Python}: This code produces an \texttt{EOFError} due to the calls to \texttt{input()} inside the \texttt{for} loop. The canonical example does not use \texttt{input()}.  Our evaluation procedures do not handle user input, hence the error. (Prompt \#41, Theme: ``LanguageSpecific'', Error: ``EOFError'') }
    \label{fig:python-eof}
\end{figure}


\lstdefinestyle{customStyle}{
basicstyle=\footnotesize\ttfamily,
keywordstyle=\bfseries,
showstringspaces=false,
morekeywords={include, printf},
commentstyle={},
escapeinside=QQ,
escapebegin=\begin{otherlanguage*}{bulgarian}\normalfont,
escapeend=\end{otherlanguage*},
}

\lstset{style=customStyle}
\begin{figure}[H]
    \centering
\begin{lstlisting}
;; You are given a list of integers.
;; You need to find the largest prime value and return the sum of its digits.
;; Examples:
;; >>> (skjkasdkd (list 0 3 2 1 3 5 7 4 5 5 5 2 181 32 4 32 3 2 32 324 4 3))
;; 10
;; >>> (skjkasdkd (list 1 0 1 8 2 4597 2 1 3 40 1 2 1 2 4 2 5 1))
;; 25
;; >>> (skjkasdkd (list 1 3 1 32 5107 34 83278 109 163 23 2323 32 30 1 9 3))
;; 13
;; >>> (skjkasdkd (list 0 724 32 71 99 32 6 0 5 91 83 0 5 6))
;; 11
;; >>> (skjkasdkd (list 0 81 12 3 1 21))
;; 3
;; >>> (skjkasdkd (list 0 8 1 2 1 7))
;; 7
(define (skjkasdkd lst)
  (define (is-prime? n)
    (define (iter i)
      (cond ((= i n) #t)
            ((= (modulo n i) 0) #f)
            (else (iter (+ i 1)))))
    (iter 2))
  (define (sum-digits n)
    (define (iter n acc)
      (cond ((= n 0) acc)
            (else (iter (quotient n 10) (+ acc (remainder n 10))))))
    (iter n 0))
  (define (largest-prime lst)
    (cond ((null? lst) 0)
          ((is-prime? (car lst)) (max (car lst) (largest-prime (cdr lst))))
          (else (largest-prime (cdr lst)))))
 (sum-digits (largest-prime lst)))
```
QQ
## QЗадачаQ 2

QДа се дефинира функцияQ `(skjkasdkd lst)`, Qкоято приема списък от цели числа и връща списък от цели числа, които са прости и по-големи от най-голямото просто число в дадения списък.Q

QПримери:Q
QQ

```racket
>>> (skjkasdkd (list 0 3 2 1 3 5 7 4 5 5 5 2 181 32 4 32 3 2 32 324 4 3))
'(181 191 193 197 199 211 223 227 229 233 239 241 251 257 263 
269 271 277 281 283 293 307 311 313 317 331 337 347 349 353 
359 367 373 379 383 389 397 401 409 419 421 431 433

\end{lstlisting}
    \caption{\emph{Racket}: Codex-Davinci can produce comments/code in languages other than English. The Bulgarian text above translates as the following via Google Translate: ``\#\# Task 2, Define a function `(skjkasdkd lst)` that accepts a list of integers and returns a list of integers that are prime and greater than the largest prime in the given list., Examples:'' (Prompt \#119, Theme: ``GenerateAnotherLanguage'', Error: ``GeneratingWebpageOrMarkdown'')}
    \label{fig:bad-rkt}
\end{figure}


\subsection{Additional Error Examples}

This section contains error examples which we found through manual inspection of Codex-Davinci output outside of our structured analysis.


\lstdefinelanguage{JavaScript}{
sensitive=false,
morekeywords={let,function,return,undefined,null,typeof,true,false,ref,deref,
              var,new,break,while,try,catch,finally,throw,err,if,else,for,do,
              with,delete,new,instanceof,this,},
keywordstyle=\ttfamily,
identifierstyle=\ttfamily,
comment=[l]{//},
commentstyle=\ttfamily,
string=[d]{"},
stringstyle=\ttfamily,
mathescape=true,
extendedchars=true,
basicstyle=\footnotesize\ttfamily,
showstringspaces=false,
numbers=none,
firstnumber=0,
numberstyle=\tiny,
stepnumber=5,
numbersep=5pt,
upquote=true,
columns=fixed,
flexiblecolumns=true,
}

\begin{figure}[H]
\centering
\begin{lstlisting}[language=JavaScript,numbers=none]
function sum_squares(lst){
    return sum(map(lambda n: n ** 2, filter(lambda n: n
}
\end{lstlisting}
    \caption{This JavaScript completion is generated for our full translation at $T=0.8$ for prompt \#142. The complete function body is executable, functional-style Python.}
    \label{fig:javascript-python}
\end{figure}


\begin{figure}[H]
\centering
\begin{lstlisting}[language=JavaScript,numbers=none]
local function will_it_fly(q, w)
   return is_balanced(q) and sum(q) <= w
end
\end{lstlisting}
    \caption{This Lua completion is generated by our full translation at $T=0.2$ for prompt \#72. \texttt{is\_balanced} is not defined in the local context and is likely an expected helper function. The call to \texttt{sum} is Python-like, as \texttt{sum} is a Python built-in method.}
    \label{fig:lua-python}
\end{figure}











\section{Related work}

\subsection{Early work (neural)}

\subsubsection{\citet{feng2020codebert}: CodeBERT}

\subsubsection{\citet{clement-etal-2020-pymt5}: PyMT5}

\subsection{Language models that test code generation}

\subsubsection{\citet{raffel2019exploring}: Colossal Clean Crawled Corpus}

\subsubsection{\citet{thepile}: The Pile}

\subsubsection{\citet{brown2020language}: GPT-3}

\subsubsection{GPT-J}

\subsubsection{GPT-Neo}



\subsubsection{GPT-Neo-X}

\subsection{Code generation models}

\subsubsection{\citet{chen2021evaluating}: Codex} 

Fine-tuned GPT-3 model on 159 GB of deduped, filtered code. Code lexer is based on GPT-3 text tokenizer, extended with a set of tokens for representing different whitespace lengths.

Samples tokens until they encounter one of 5 stop sequences: `$\backslash$nclass', `$\backslash$ndef', `$\backslash$n\#', `$\backslash$nif', or
`$\backslash$nprint'. Uses nucleus sampling with p=0.95 for all samples. Finds the optimal temperature for pass@1 is 0,2, while pass@100 is 0.8.

Evaluates on HumanEval using pass@k metrics. They take 200 samples for each pass@k metric, regardless of k. 

Also report Codex-S, which is fine-tuned on problems from competitive programming websites (10,000) and from repos with CI (40,000).

Some error patterns they notice: long docstrings are hard; invoking undefined functions/variables; binding.

\subsubsection{\citet{fried2022incoder}: InCoder}

Trained on a causal masking objective: a sequence is replaced by a sentinel token and moved to the end of the sequence, training it to predict completions for missing regions. This also allows to it generate: you just don't add sentinel tokens.

Trained on 159 GB of data, 52 GB of it in Python. 57 GB are from StackOverflow (via The Pile) and the rest from GitHub. 

Allows tokens to extend over whitespace.

Trains a Transformer language model with 6.7B parameters.

Creates two evaluation datasets from HumanEval: a single line infilling and a multi-line infilling. 

Evaluates against prior work using HumanEval and MBPP. For HumanEval, samples 200 candidates and calculates pass@1 with temp=0.2, and pass@10 and pass@100 with temp=0.8. Uses top-p sampling with p=0.95 for all. 

Also tests on other tasks: type inference, variable name prediction, docstring generation. Evaluation datasets mostly taken from CodeXGLUE.

\subsubsection{\cite{xu2022systematic}: PolyCoder}

Claims to be the first language ``trained exclusively on code from multiple programming languages.'' This is dubious.

GPT-2 model with 2.7B parameters trained on most popular Github repos for 12 languages, taking only 25,000 repos per language.

Outperforms Codex in perplexity on C. In every other measure, less good.

Uses the same evaluation metrics as \citet{chen2021evaluating}, except weirdly they only sample 100 times. \molly{and they don't mention why!}


\subsection{Code generation tasks}

NL2Code: generating code from docstring / natural language description.

Docstring generation: generate a docstring from a function.

\citet{tufano2020unit} generate unit tests from code.

\citet{lu2021codexglue} proposes 10 tasks.

\subsection{NL2Code benchmarks}

\subsubsection{\citet{chen2021evaluating}: HumanEval}. Proposes a benchmark dataset of 164 original programming problems with unit tests. The problems cover algorithms, simple mathematics, and language comprehension. Has an average of 7.7 tests per problem.

\subsubsection{\citet{hendrycks2021measuring}: APPS}. 5000 training and 5000 test examples of coding problems, each with a set of unit tests. Most involve full programs rather than generating individual functions. Average number of test cases per problem: 21.2.

Taken from websites where programmers share programming challenge problems. Problems are grouped by difficulty into introductory level, interview level, and competition level.

Two formats: call-based format where the programmer is given the beginning of a function, and standard input format, where the program must output its answers to the STDOUT stream (like ACM programming contest).

Computes the average number of tests passed. 

\molly{cites other dataset collections that they see as working on a similar problem: SPoC (below), Iyer \cite{iyer2018mapping} (Java), Spider (EMNLP) (SQL), \citet{raychev2016probabilistic} (JavaScript \& Python), }

\subsubsection{\citet{austin2021program}: MBPP and MathQA-Python}

Mostly Basic Programming Problems (MBPP): 974 Python functions, paired with natural language descriptions and test cases. Designed not to involve libraries or print statements. Each has 3 test cases. Most natural language descriptions are only 1 line.

MathQA-Python: 23,914 pairs of Python programs and natural language descriptions. The programs are in a strange format, since the prompts are taken from mathematical word problems with multiple choice answers.

Each problem contains three I/O examples as assert statements in the prompt.

They include the test cases in the prompt. They also report results from including 0, 1, 2, or 3 test cases.

\subsubsection{\citet{kulal2019spoc}: SPoC}

C++ code generation dataset. Uses crowdsourced descriptions of a line of code rather than documentation.

\subsection{Other code generation benchmarks}

In addition to code generation benchmarks that pair natural language comments with code, there are benchmarks that pair programming-related natural language questions with code snippets. These benchmarks include CoNaLa \citet{yin2018learning}, which mines matching natural language questions and code snippets from StackOverflow; \citet{husain2019codesearchnet}... 

\subsubsection{\citet{yin2018learning}: CoNaLa}

A dataset of natural language questions about code mined from StackOverflow. Pairs natural language intent with code snippet. Contains both a manually annotated subset and a larger automatically mined corpus.

\subsubsection{\citet{wang2022mconala}: MCoNaLa}

Translation of CoNaLa into Spanish, Japanese, and Russian.

\citet{husain2019codesearchnet}: benchmark for retrieving relevant code from natural language query.

\citet{lu2021codexglue}: benchmarks for 10 tasks.

\citet{villmow-etal-2021-contest}: benchmark for unit test generation.

\subsection{NL2Code evaluation}

\citet{xu2022in-ide}

\subsubsection{\citet{xu2022systematic}}

Evaluates Codex, GPT-J, GPT-Neo, GPT-NeoX-20B, Code-
Parrot and proposes a new model, PolyCoder. Evaluates on HumanEval, and also looks at perplexity in 12 languages: C, C\#, C++, Go, Java, JavaScript, PHP, Python, Ruby, Rust, Scala, TypeScript.

\subsubsection{\citet{austin2021program}}

Evaluates Transformer models on MBPP and MathQA-Python. The largest has 137B parameters.

Runs a fine-tuning experiment as well as a prompt engineering experiment where they prepend some solved problems to the front of the prompt. 

Evaluates using pass@80 with temperature of 0.5. Also measures the fraction of samples that pass all tests.

Breaks errors into multiple categories: syntax errors, type errors, runtime errors, and tests failed.

Also investigates whether the largest model can predict the output from a function call. It generally cannot.

\subsection{Metric citations}

Pass@k comes from \citet{kulal2019spoc}.

\citet{hendrycks2021measuring} use average number of test cases passed, in addition to pass@1, pass@5, and pass@100.

\citet{kulal2019spoc} evaluates using unit tests.

\citet{chen2021evaluating} show that BLEU is not a reliable indicator of functional correctness. 

\citet{ren2020codebleu} show that BLEU is not a reliable way of evaluating code.

\citet{austin2021program} also show that BLEU score is not reliable.

\citet{chen2021evaluating} provide a numerically stable implementation of pass@k.


\subsection{PL-to-PL translation}
Also know as answering the question of ``why is this easier than has been expected'', ``ML HAS been used to learn language to language translations'' and ``PL-to-PL is a hard problem, in general''

\citet{karaivanov2014phrase} use phrase-based statistical machine translation to do PL-to-PL translation. Their approach adds in understanding of the underlying PL grammar to guide the prefix generation process. They provide an example of C\# to Java.

\citet{allamanis2018survey} tells us what was done before 2018. Makes important statements about what this problem looks like from the PL/SE side.

























\section{Introduction}

Advances in scaling large language models (LLMs) have improved not only their ability to generate natural language text, but also to generate code in programming languages. LLMs exposed to code can generate test cases, documentation, and even full programs from natural language descriptions~\citep{gpt-neoX,chen2021evaluating,fried2022incoder,nijkamp2022conversational,xu2022systematic}.
However, relatively little is known about how well code generation models generalize across programming languages. Most evaluation has been done on Python, since it is the best-represented language in most training datasets \citep{yin2018learning,chen2021evaluating,austin2021program,hendrycks2021measuring}. 

In this paper, we present MultiPL-E\xspace, the first parallel multi-language\footnote{In this paper, ``multi-language'' refers to multiple \emph{programming languages.}} benchmark for evaluating the code generation
performance of LLMs for 18\xspace{} programming languages. 
We focus on the natural-language-to-code (NL2Code) task: given a natural language description, generate the corresponding function in code.
A key feature of this task is that its evaluation is straightforward: we can run \emph{unit tests} to determine if the generated function behaves correctly.
There are a handful of NL2Code benchmarks that use unit tests in this manner~\citep{kulal2019spoc,hendrycks2021measuring,austin2021program}.
But, these benchmarks only evaluate performance on a single language.

MultiPL-E\xspace{} uses a suite of compilers to translate the Python benchmarks from \citet{chen2021evaluating} into parallel benchmarks in 18\xspace{} languages. 
For the first time, MultiPL-E\xspace{} provides a way to evaluate code generation models on a consistent set of benchmark problems across many languages.
The 18\xspace{} languages capture a broad spectrum of language features, application areas, and popularity, allowing us to explore the impact of these factors on model performance.

The MultiPL-E\xspace{} benchmark and associated tools are open source and easy to extend. For the NL2Code task, each compiler translates Python unit tests, doctests, and function signatures to its target language. Because these program regions do not contain arbitrary Python code, each MultiPL-E\xspace{} compiler is much simpler than a full-fledged compiler. Our framework makes it easy to add new benchmark programs and to extend the benchmark to additional languages.

We evaluate the performance of Codex~\citep{chen2021evaluating}  and InCoder~\citep{fried2022incoder} on MultiPL-E\xspace. Our analysis presents several insights into the effectiveness of LLMs for code generation, including that (1)~Codex performs best on JavaScript and equally well on C++, Scala, and TypeScript as on Python; (2)~model perplexity is not strongly correlated with the correctness of generated code; (3)~type annotations have limited impact on model performance for gradually typed languages; (4) model performance is correlated with language popularity, but some niche languages perform as well as more popular languages; and (5)~performance is sensitive to prompt design for both niche and popular languages.

\paragraph{Contributions} Our key contributions are:

\begin{itemize}
    \item MultiPL-E\xspace: a parallel benchmark for NL2Code in 18\xspace{} languages encompassing a variety of programming paradigms, language features, and popularity levels;
    \item An easily extensible system for translating NL2Code problems, including doctests, unit tests, and Python-oriented terminology, into new programming languages;
    \item A multi-language parallel evaluation of two models, Codex \citep{chen2021evaluating} and InCoder \cite{fried2022incoder};
    \item Explorations of language frequency effects, the impact of type annotations, and prompt translation sensitivity on NL2Code performance, along with a fine-grained analysis of NL2Code errors in four languages. 
\end{itemize}

\ifanon
\else
Our code and data is available at \url{github.com/nuprl/MultiPL-E}
\fi

\section{Evaluating Code Generation}\label{sec:task}

Code generation has long been a task of interest: there is extensive work on program synthesis~\cite{alur:sygus,chaudhuri:neurosymbolic} using both symbolic and neuro-symbolic approaches. More recently, LLMs trained for text generation have demonstrated the ability to perform program completion~\citep{brown2020language,gptj,gpt-neoX}; and several LLMs have been trained or fine-tuned on programming language text~\citep{feng2020codebert,clement-etal-2020-pymt5,chen2021evaluating,nijkamp2022conversational,fried2022incoder,xu2022systematic}. 

Unlike traditional program synthesis techniques, neural language models are able to condition on and generate both natural language and programming language text. Moreover, LLMs offer the promise of synthesizing knowledge gleaned from code in multiple programming languages. \textit{Just as language models exposed to multiple natural languages are able to generalize across languages, might not multi-language models of code do the same?}

Although this kind of multi-language generalization is an intriguing possibility, little is known about how well code generation models perform across programming languages. We make progress towards answering this question by evaluating the multi-language code generation abilities of two state-of-the-art models: Codex and InCoder.

\subsection{Large Language Models for Code}

Both models use a Tranformer architecture~\citep{vaswani2017attention} and are trained with a language modeling objective on a mixture of natural language and code.

\paragraph{InCoder} InCoder \citep{fried2022incoder} is a 6.7B parameter language model trained using a causal masking objective~\citep{aghajanyan2022cm3}. It supports both code infilling and code completion; we test only the latter. InCoder was trained on 159 GB of deduped, filtered code from Github (around a third in Python) and 57GB from StackOverflow.

\paragraph{Codex} Codex is a GPT-3 language model fine-tuned on code. \citet{chen2021evaluating} describe a 12B parameter version of Codex fine-tuned on 159 GB of deduped, filtered Python code from Github. We use the more recent \texttt{codex-davinci-002} model, which is trained on multiple languages, but details of its training set and size are not public~\citep{codexblog}.

\subsection{The Natural Language to Code Task}

\begin{figure}[t]
\begin{subfigure}{\columnwidth}
\caption{The prompt given to the NL2Code model:}
\label{prompt-example}
\begin{lstlisting}[
  language=Python,
  linebackgroundcolor={%
  \ifnum\value{lstnumber}=1\color{signaturecolor}\fi
  \ifnum\value{lstnumber}=10\color{doctestcolor}\fi  
  \ifnum\value{lstnumber}=11\color{doctestcolor}\fi
  \ifnum\value{lstnumber}=12\color{doctestcolor}\fi
}]
def lsi(lst):
  """Create a function that returns a 
  tuple (a, b), where 'a' is the largest
  of negative integers, and 'b' is the
  smallest of positive integers in a 
  list. If there is no negative or 
  positive integers, return them as 
  None.
  Examples:
    lsi([2, 4, 1, 3, 5, 7]) == (None, 1)
    lsi([]) == (None, None)
    lsi([0]) == (None, None)
  """
\end{lstlisting}
\end{subfigure}

\tikz\draw [dash dot] (0,0) -- (8,0);

\begin{subfigure}{\columnwidth}
\vspace{1em}
\caption{A completion generated by Codex:}
\label{generated-code-example}
\begin{lstlisting}[
  language=Python
]
  if len(lst) == 0:
    return (None, None)
  else:
    neg_nums = [x for x in lst if x<0]
    pos_nums = [x for x in lst if x>0]
  if len(neg_nums) == 0:
    neg_nums = None
  if len(pos_nums) == 0:
    pos_nums = None
  return (max(neg_nums), min(pos_nums))
\end{lstlisting}

\end{subfigure}\\

\tikz\draw [dash dot] (0,0) -- (8,0);

\begin{subfigure}{\columnwidth}
\vspace{1em}
\caption{3 of the 12 unit tests (the generated solution fails two):}
\label{unit-tests-example}
\begin{lstlisting}[language=Python, mathescape=true]
$\makebox[0.5em]{\textcolor{red}X}$ assert lsi([2, 4, 1, 3, 5, 7]) == (None, 1)
$\makebox[0.5em]{\textcolor{red}X}$ assert lsi([2, 4, 1, 3, 7, 0]) == (None, 1)
$\makebox[0.5em]{\textcolor{green}\checkmark}$ assert lsi([1, 3, 4, 5, 6, -2]) == (-2, 1)
\end{lstlisting}

\end{subfigure}
\caption{Problem 136 of 164 of the HumanEval benchmark. We shorten the name
\texttt{largest\_smallest\_integers} for brevity. Top: the prompt for the model, with the function signature, natural language description, and doctests. Middle: a Codex-generated solution. Bottom: unit tests.}
\label{fig:prompt_examples}
\end{figure}

\noindent
Code-aware language models have been applied to a variety of tasks, including test generation \citep{tufano2020unit}, docstring generation \citep{lu2021codexglue}, code search \citep{feng2020codebert,ahmed2022multilingual}, type inference~\cite{wei:lambdanet,hellendoorn:dl-ti,pradel:typewriter}, and more \cite{drori:codex-univ-math}. 
We focus on the \textbf{natural-language-to-code} task (NL2Code): given the description of a function in natural language, complete the function body.

\Cref{prompt-example} shows an example prompt from the HumanEval benchmark dataset for NL2Code~\citep{chen2021evaluating}. Each prompt contains several sources of information for the model: the function signature (its name and parameters); a brief natural language description of the intended function; and, optionally, examples in the form of Python doctests. The model's task is to generate the body of the function given this input; the unit tests are then used to test whether the generated function is correct. The model does not see the unit tests. 

Note that the model does not receive an explicit cue about the target language, but each of the three prompt regions provide implicit cues: the syntax of the function signature, the terminology used in the natural language description, and the syntax of the doctests all suggest that the target is Python. Consequently, to translate this prompt to a new programming language, we should target all three regions of the prompt.

\subsection{Evaluating NL2Code}

Existing benchmarks for evaluating natural-language-to-code generation models have two main limitations: (1)~they evaluate only a single language \citep{kulal2019spoc,chen2021evaluating,hendrycks2021measuring,austin2021program}, or (2)~they do not test the correctness of generated code \citep{yin2018learning,xu2022systematic}.\footnote{Textual similarity metrics (e.g., BLEU) have been shown to correlate weakly with code correctness.~\citep{ren2020codebleu,austin2021program,chen2021evaluating}.} For instance, although \citet{xu2022systematic} measure model perplexity on several programming languages, they only test code correctness for Python.

Unlike natural languages, programming languages have a well-defined notion of \emph{semantic equivalence}: two functions are semantically equivalent if and only if they behave identically on all inputs. Equivalence-checking is undecidable; however, we can use \textit{unit tests} to approximate equivalence between programs by verifying a sample of program inputs.

In the context of NL2Code, we judge a generated function correct if it passes a suite of unit tests included with the benchmark.
\Cref{generated-code-example} shows just one solution generated by Codex for the example prompt. This solution is incorrect because it fails some of the unit tests (\cref{unit-tests-example}).

We posit that a multi-language NL2Code benchmark should have \emph{a parallel suite of benchmark problems in multiple programming languages} that can be solved and tested consistently across languages. MultiPL-E\xspace{} \emph{compiles} NL2Code benchmarks for Python into parallel benchmarks for 18\xspace{} other programming languages. Writing 18\xspace{} full-language compilers would require extraordinary effort. However, a MultiPL-E\xspace{} compiler only needs to translate function signatures, comments, and unit tests.

There are a number of single-language NL2Code benchmarks \citep{yin2018learning,kulal2019spoc,hendrycks2021measuring}. We select HumanEval \citep{austin2021program} as a source benchmark for several reasons: (1)~it is a diverse collection of 164 problems; (2)~all problems have tests to check correctness; (3)~all problems are functions that receive and return first-order values, which lend themselves to rigorous testing; (4)~it is a challenging benchmark: the best model evaluated by \citet{fried2022incoder} achieves only a 36\% pass rate on Python. 


\section{The MultiPL-E\xspace{} Benchmarking Approach}\label{sec:dataset}

\begin{table}[t]
    \centering
    \begin{tabular}{|l|c|l|l|l|}\hline
         PL &Typed? & GitHub \% & TIOBE & Category \\\hline
              Bash&$\times$&-&43&\textsc{Niche}\xspace\\
         C++&\checkmark&7.0&4&\textsc{High}\xspace\\
          C\#&\checkmark&3.1&5&\textsc{Medium}\xspace\\
           D&\checkmark&-&35&\textsc{Niche}\xspace\\
         Go&\checkmark&7.9&12&\textsc{Medium}\xspace\\
         Java&\checkmark&13.1&3&\textsc{High}\xspace\\
         JavaScript&$\times$&14.3&7&\textsc{High}\xspace\\
          Julia&$\times$&0.1&28&\textsc{Niche}\xspace\\
          Lua&$\times$&0.2&25&\textsc{Niche}\xspace\\
         Perl&$\times$&0.3&17&\textsc{Low}\xspace\\
        PHP&$\times$&5.3&11&\textsc{Medium}\xspace\\
         R&$\times$&0.05&19&\textsc{Low}\xspace\\
         Racket&$\times$&-&-&\textsc{Niche}\xspace\\
       Ruby&$\times$&6.2&15&\textsc{Medium}\xspace\\
        Rust&\checkmark&1.1&22&\textsc{Low}\xspace\\
        Scala&\checkmark&1.7&32&\textsc{Low}\xspace\\
        Swift&\checkmark&0.7&10&\textsc{Low}\xspace\\
        TypeScript&\checkmark&9.1&33&\textsc{High}\xspace\\\hline
    \end{tabular}
    \caption{MultiPL-E\xspace languages by frequency, as calculated by GitHut 2.0 and the TIOBE Programming Community index.} \label{tab:pls}
\end{table}

MultiPL-E\xspace is a multi-language, parallel NL2Code benchmark that includes a diverse set of programming languages. Table~\ref{tab:pls} gives an overview of the 18 MultiPL-E\xspace languages.
We categorize the languages into four frequency classes (\textsc{Niche}\xspace, \textsc{Low}\xspace, \textsc{Medium}\xspace, or \textsc{High}\xspace) based on a weighting of TIOBE rank and GitHub frequency. 
There are eight languages in MultiPL-E\xspace{} that have never been used before to measure NL2Code performance, including newer languages (Julia and Swift), older scripting languages (Bash and Perl), and languages used for specific applications (Lua and R). The broad range of languages in MultiPL-E\xspace shows the generality of our compilation approach and allows us to explore how language frequency affects performance (\cref{subsec:eval-freq}).

A key feature of MultiPL-E\xspace{} is that it is easy to extend with new models, benchmarks, and languages. To support new languages and benchmarks without manual (and error-prone) effort, we build 18\xspace{} \emph{compilers} to translate NL2Code benchmarks written in Python.
Writing one of these compilers is straightforward when the target language is similar to Python, but requires care for typed languages and even some untyped languages, notably Perl and R.
The remainder of this section presents the design of these compilers.

\subsection{Compiling Python Benchmarks}
\label{compilers-lol}


A MultiPL-E\xspace{} compiler is significantly easier to build than a complete compiler: to translate a benchmark problem, we only need to compile function signatures and unit tests
(not arbitrary statements and expressions).
Our compilers preserve comments, since they contain the natural language description for the NL2Code task; however, we automatically rephrase them to replace Python-specific terminology.


\paragraph{Compiling Unit Tests}

MultiPL-E\xspace{} supports any unit test where the input and output to the test is a \emph{first-order value}. In Python, these include constants and data structures such as lists, tuples, and dictionaries, but exclude values such as \texttt{lambda} expressions.\footnote{We do not support testing higher-order functions, but support generated code that uses higher-order functions.} HumanEval unit tests apply the model-generated function to a first-order value, and compare the result with an expected first-order value.

\begin{figure}
\begin{subfigure}{\columnwidth}
\caption{Original Python assertion.}
\begin{lstlisting}[language=python]
assert lsi([0]) == (None, None)
\end{lstlisting}
\end{subfigure}

\vspace{0.5em}
\tikz\draw [dash dot] (0,0) -- (8,0);
\vspace{0.5em}

\begin{subfigure}{\columnwidth}
\caption{Equivalent R.}
\begin{lstlisting}{r}
if(!identical(lsi(c(0)), c(NULL, NULL))){
    quit('no', 1)}
\end{lstlisting}
\end{subfigure}

\vspace{0.5em}
\tikz\draw [dash dot] (0,0) -- (8,0);
\vspace{0.5em}

\begin{subfigure}{\columnwidth}
\caption{Equivalent JavaScript.}
  
\begin{lstlisting}
assert.deepEqual(lsi([0]), [void 0, void 0]);
\end{lstlisting}
\end{subfigure}
    
\caption{Example of a translated assertion.}
\label{fig:unit-tests}
\end{figure}

Each MultiPL-E\xspace{} compiler has a recursive function that compiles Python values to the target language's values, with the recursive step handling nested values.
Even for an untyped target, this value-to-value compilation requires care, because not all Python value types have perfect analogues in every target.
For example, we compile both tuples and lists to JavaScript arrays, since JavaScript lacks a canonical tuple type.
We also support untyped targets where the compilation strategy is less obvious. For example, when the target is R, it may appear natural to compile Python lists to R lists: both kinds of lists can be nested and allow heterogenous values.
However, R's vector type is much more commonly used (data frames are made of vectors).
Unfortunately, vectors must be homogeneous and cannot be nested, so not all Python lists can be translated to vectors.
For example, an argument typed \lstinline|List[Int]| can be translated to a vector, but a nested list cannot. In order to more closely match the token distribution of idiomatic R code seen during training, our R compiler uses type-oriented compilation techniques even though R is untyped.

The final step of compiling tests is to choose an appropriate test for equality.
The meaning of equality operators varies across programming languages.
Python's \lstinline|==| operator checks \emph{deep equality}, i.e., item-by-item equality within data structures. Deep equality is the appropriate choice for unit tests. In some languages, we need to import equality-testing functions
from testing libraries, as in the JavaScript example shown in \Cref{fig:unit-tests}.

\paragraph{Compiling Function Signatures}


Compiling a function signature to an untyped language is straightforward, but requires care when the target is typed. Most typed languages require argument and return type annotations. Fortunately, a large subset of the HumanEval benchmarks employ Python's optional type annotations; we add type annotations to the others.\footnote{\Cref{subsec:static-types} shows that this does not affect Python pass rates.}
The MultiPL-E\xspace compilers for typed languages translate these types to target language types.

Many typed languages require type annotations in data structures, which appear in unit tests. For example, C++ vectors require an annotation specifying their element type, and numbers  in Rust (sometimes) require a type suffix. We perform limited local type inference to calculate these types from the type of the function signature to ensure that the unit tests always compile successfully.
The HumanEval benchmarks have five problems that employ Python types that cannot be expressed in all conventional typed languages, such as \texttt{Any}, which is the type of all values.
We fail to compile these 5 problems to most typed languages.

A further problem arises when the target language has  discriminated unions, which require us to inject values into constructors in the target language. For example, we compile a Python number \lstinline|n| to \lstinline|Some(n)| in Rust,  only if the context is typed \lstinline|Optional[Int]| (similarly in Swift and Scala).

\begin{figure}[t]
\begin{subfigure}{\columnwidth}
\caption{Original Python docstring from HumanEval \#95.}
\footnotesize
\texttt{
Given a \colorbox{red!40}{dictionary}, return \colorbox{pink!20}{True} if all keys are strings in lower case or all keys are strings in upper case, else return \colorbox{pink!20}{False}. The function should return \colorbox{pink!20}{False} is the given \colorbox{red!40}{dictionary} is empty.}
\end{subfigure}
\vspace{0.5em}
\begin{subfigure}{\columnwidth}
\caption{Terminology translated to Perl.}
\footnotesize
\texttt{
Given a \colorbox{red!40}{hash}, return \colorbox{pink!20}{1} if all keys are strings in lower case or all keys are strings in upper case, else return \colorbox{pink!20}{""}. The function should return \colorbox{pink!20}{""} is the given \colorbox{red!40}{hash} is empty.}
\end{subfigure}

\caption{A Python docstring and its Perl translation. }\label{fig:lang-in-prompts}
\end{figure}

\paragraph{Translating Doctests}

Python \emph{doctests} are a standard format for examples in documentation.
While many of the HumanEval prompts include examples, not all of them are validly formatted doctests. We standardize examples to the Python doctest format ("\textgreater{}\textgreater{}\textgreater" prepended). We apply value-to-value compilation to the doctests as we do for unit tests. However, since not all languages have an equivalent doctest format, we keep the Python format for all target languages.

\subsubsection{Translating Python Terminology in Prompts}

Different programming languages use different terminology to refer to the same concept. For example,
a Python \emph{list} is closest to a JavaScript \emph{array} or a Rust \emph{vector}.
To mitigate the impact of these differences, we identify Python-specific terminology in the natural language portion of the prompt, and translate it to the most natural equivalent for the target language.
Figure~\ref{fig:lang-in-prompts} shows an example of a prompt translated from Python to Perl. Notably, Perl not only lacks Booleans, but uses 1 for true and the empty string for false.


\begin{figure*}[t]
    \centering
    \includegraphics[width=\textwidth]{images/all_models_temps_reworded.pdf}
    \caption{From right to left: InCoder pass@1, pass@10, pass@100; Codex pass@1, pass@10, pass@100 on all languages.}
    \label{fig:total_results}
\end{figure*}

\subsection{Limitations of Our Approach}
\label{limitations}

A handful of HumanEval benchmarks cannot be easily translated using the MultiPL-E\xspace{} approach. Of the 164 original benchmarks:
(1)~we exclude 3 that have Python helper functions in their prompt;
(2)~we modify 2 benchmarks to use unit tests instead of randomized testing; and (3) for certain typed languages, we fail to compile up to 5 benchmarks with untranslatable types. (\Cref{subsec:replication}) shows that these changes do not lead to significantly different results for Python.

Our approach can be generalized to additional programming languages, so long as the target language has natural analogues for the Python data types used in the benchmarks. We do not include two previously studied languages, C \cite{xu2022systematic} and SQL \cite{yu-etal-2018-spider} because they do not meet this criterion.

\section{Evaluation}

We use MultiPL-E\xspace to evaluate two NL2Code models: InCoder (6.7B) and Codex (\texttt{code-davinci-002}). For each language, we calculate pass@$k$ using the methodology employed by 
 \citet{chen2021evaluating} and subsequent work.
 Intuitively, pass@1 is the likelihood of the model producing a completion that passes \emph{all unit tests}, pass@10 is the likelihood of any one of 10 completions passing all unit tests, and so on.
 We calculate pass@1 with temperature $0.2$, and use temperature $0.8$ for pass@10 and pass@100.
 
 We fit mixed-effects models to evaluate the statistical significance of the differences between groups that we report below~\citep{lme4}.
 Appendix~\ref{app:mem} has a full description of each model with its estimate table.
 
\subsection{Multi-Language Performance of NL2Code}\label{subsec:multilingual}

\Cref{fig:total_results} shows the mean pass@$k$ rate for each model on each MultiPL-E\xspace language. We find reliable differences between Codex pass@1 rates for Python and all but 4 languages: C++, JavaScript, Scala, and TypeScript. InCoder performs significantly better on Python than all other languages (all $p<0.001$).

\paragraph{Python Results and Replication}
\label{subsec:replication}

Our InCoder Python results replicate the previously reported performance of InCoder on HumanEval~\citep{fried2022incoder}. This shows that the few standardization changes we made to the benchmarks do not affect model performance.

We evaluate a more recent Codex model (\texttt{code- davinci-002}) than previous work and observe a large improvement on Python: a pass@1 rate of 45.9\%, compared to 28.8\% reported earlier~\cite{chen2021evaluating}.
This is notably better than the pass@1 rates reported for the two best models evaluated in \citet{fried2022incoder}: CodeGen~\citep{nijkamp2022conversational} (29.3\%) and PaLM-Code~\citep{chowdhery2022palm} (36\%), which we lack the resources to run.

\paragraph{Codex Performs Best on JavaScript}

Codex's performance on JavaScript is better than its performance on Python, though the difference is not significant (+2.3\%; $p=0.43$). Codex achieves a pass@1 rate higher than 40\% on C++, Java, TypeScript, PHP, Ruby, Rust, Scala, and Lua. 

InCoder performs more weakly than Codex. Like Codex, it performs better on more frequently-used languages (Python) than less popular ones (\cref{subsec:eval-freq}).

The Codex training set is not public; it is possible that the latest model has been trained on solutions to the HumanEval benchmarks in Python, and this could be inflating its performance.
However, MultiPL-E\xspace{} is a new dataset for 18\xspace{} other languages. That Codex matches or exceeds its Python performance on these new languages suggests a negligible impact of any train/test overlap.

\paragraph{Perplexity and Code Correctness Do Not Correlate}

\citet{xu2022systematic} report Codex perplexity scores for 10 of our 18 languages. Figure \ref{fig:perplexity} plots Codex pass@1 scores against these scores. We do not observe a strong correlation between the measures. Notably, perplexity is highest for JavaScript and TypeScript, while we find that Codex performs best on these languages. This suggests that perplexity may not be a reliable evaluation metric for NL2Code. One caveat is that \citet{xu2022systematic} likely evaluate an older Codex model, since they report substantially lower pass rates for Python.

\begin{figure}[t]
    \centering
    \includegraphics[width=0.45\textwidth]{images/perplexity.pdf}
    \caption{Codex pass@1 rates versus perplexity scores reported in \citet{xu2022systematic}.}
    \label{fig:perplexity}
\end{figure}

\subsection{Ablation Study}\label{subsec:ablation}

 \begin{figure}[t]
\centering
\includegraphics[width=0.45\textwidth]{images/all_expts_pass1.pdf}
\caption{Ablation study of translation components. From right to left, pass@1 with original prompts; translated doctests; translated text and doctests; and doctests removed.}\label{fig:ablation}
\end{figure}

Our compilers target multiple distinct regions of the prompt for each problem. We explore the impact of each  component in an ablation study with four experiments: 

\begin{itemize}
    \item \textbf{Original Prompt}: does not translate doctests or natural language terminology (e.g. prompts as in HumanEval);
    \item \textbf{Test-only Translation}: translates doctests but not Python-specific terminology;
    \item \textbf{Full Translation}: translates unit tests, doctests, and Python-terminology in the prompt; and
    \item \textbf{No Doctests}: removes doctests and does not translate natural language terminology.
\end{itemize}

Figure \ref{fig:ablation} shows pass@1 results for each variation. For Codex, translating doctests and Python-specific terminology has little impact on better-performing languages. However, translating these components seems more important for challenging languages like Bash and Perl. Overall, we find significant differences between the \textbf{Full Translation} and \textbf{Test-Only Translation} experiments ($p=0.03$), and between \textbf{No Doctests} and \textbf{Test-Only Translation} ($p<0.001$). This suggests that doctests are useful to Codex, but that their format is not important.

Although removing doctests weakens Codex performance, puzzlingly, it appears to help InCoder performance ($p=0.005$). Overall, we observe little benefit to translating doctests or Python terminology for InCoder.

\section{Factors in Code Generation Success}

This section explores our results in more depth. We examine how success rate is affected by language popularity, language features, and choices in prompt translation. We also study the kinds of errors that arise in NL2Code across several languages.

\begin{figure}[t]
\centering
\includegraphics[width=0.45\textwidth]{images/lang_freq_pass1_reworded.pdf}
\caption{Model performance by language frequency and type-checking. Languages to the left of dashed line are untyped; languages to the right are  typed.}\label{fig:lang-frequency}
\end{figure}


\subsection{Programming Language Frequency}\label{subsec:eval-freq}

\Cref{fig:lang-frequency} groups languages by frequency and plots the pass@1 rates for both models. Both models perform best on high frequency languages. However, Codex performs as well on certain \textsc{Low} and \textsc{Niche} languages as on \textsc{Medium} languages: Lua is the 9th-best language in our dataset, although it only appears in 0.2\% of GitHub activity and is not in the TIOBE Top-20.
We find reliable differences in Codex pass@1 rates between \textsc{Low} and \textsc{Niche} languages when compared to the \textsc{High} category ($p<0.001$; $p=0.002$), but not between the \textsc{Medium} and \textsc{High} categories ($p=0.22$).

\subsection{Type Annotations}\label{subsec:static-types}

One may conjecture that type annotations improve model performance by constraining the code generation search space. Or, perhaps, they might hurt performance by complicating the task. In \cref{fig:lang-frequency}, the dashed line in each category separates languages with type annotations (left) from languages without (right). We observe no overall effect of type annotations on Codex pass@1 rates ($p=0.33$).

To explore the impact of type annotations at a more fine-grained level, we run a series of follow-up experiments on Python, which allows optional type annotations, and TypeScript, a gradually typed cousin of JavaScript. Gradual typing allows us to weaken type annotations and the TypeScript compiler can even be configured to ignore all type errors.

\paragraph{Precise type annotations improve TypeScript performance}

TypeScript has an ``Any'' type, which is compatible with all types. We run Codex on a variation of the TypeScript prompts where all types in the function signature are translated to ``Any''. We find that the loss of precise types hurts performance on TypeScript (-2.5\%; $p<0.001$).

\paragraph{Type annotations \emph{do not} improve Python performance}

We run a similar experiment with Codex and Python, where we remove all the type annotations from the prompts. We find that this has no significant effect on Codex's pass@1 rate for Python ($p=0.23$). 

We interpret these results as evidence that type annotations do not guide search in general, since they do not improve Python performance, but that informative types are necessary for languages where type annotations are standard, perhaps in order to fit the token distribution of high-quality typed code seen in training. 

\paragraph{TypeScript type errors correlate with runtime errors}

Type-checking can reject programs that would in fact run without error. We run the Codex-generated TypeScript programs without first checking for type errors. We observe no significant difference in pass@1 rates ($p=0.14$), suggesting that typed programs are rejected for genuine errors.

\subsection{Sensitivity to Compilation Choices}\label{subsec:prompt-engineering}

Each MultiPL-E\xspace{} compiler makes small choices about how to translate prompts that could have an impact on performance. We explore some of these choices below.

\paragraph{Comment style affects performance}

Most programming languages have several comment styles (e.g., single-line vs. multi-line). To investigate their impact, we consider PHP (\textsc{Medium}) and Racket (\textsc{Niche}). Our original prompts use single-line comments for both PHP and Racket, following conventional style. We run a set of experiments with Codex where we use multi-line comments. This improves the pass@1 rate for Racket (+1.9\%, $p<0.001$), but decreases it for PHP (-3.1\% , $p=0.001$). 

\paragraph{Naming arguments improves performance for Perl}

Functions in Perl do not have formal named arguments. Instead, all arguments are passed in a special array. Our compiler to Perl produces a prompt that pops elements off the special array and names them, with the expectation that this would improve model performance. We run a follow-up experiment where we omit this, so the model has to infer everything about arguments from the natural language description and examples. 
This significantly lowers Codex's pass@1 rate (-8\%; $p<0.001$).

\paragraph{}

Our results show that NL2Code performance can be sensitive to prompt engineering choices for both high and low frequency languages. 

\subsection{Characterization of NL2Code Errors}\label{subsec:errors} 

NL2Code systems generate many failing programs---programs that produce errors or fail to pass unit tests---than programs which run successfully. This section presents a detailed evaluation of errors present in the Codex-generated functions for 4 languages: Python, C\#, Swift, and Racket. See Appendix \ref{app:errors} for a full categorization.

We first identified specific error labels for each language and then grouped them into themes (e.g. ``NullReference''). We produced five general error categories: \textsc{Run-time}, \textsc{Static}, \textsc{Type}, \textsc{Language}, and \textsc{Model}. We group similar error sources together across languages, even if they occur in different contexts: for example, a function call with a value of the wrong type may fail at compile-time or run-time or depending on the language's type system.

The most common \textsc{Static} theme across all languages is ``UndefinedIdentifier'', which contains errors related to referencing non-existent terms. These errors can be caused in many ways -- calls to functions not in the local context, 
use of Python-like keywords, 
or calls to methods from external libraries that were not imported.

Some errors in the \textsc{Runtime} category mimic those we expect from software engineers (e.g., index-out-of-range errors). However, others are unlike human mistakes. Notable themes in the latter group (\textsc{Model}) include generating code that throws exceptions on purpose and generating code in an entirely different language (e.g., Markdown, not Racket).

Finally, the category \textsc{Language} includes multiple themes related to specifics of the target language itself. The ``LanguageSpecific'' theme contains idiosyncratic errors such as the requirement of labeled arguments in Swift. ``DoesNotKnowSyntax'' includes errors in Racket caused by incorrectly generated core language constructs.  


\section{Conclusion}

We propose MultiPL-E\xspace, the first parallel multi-language benchmark for natural-language-to-code generation. We write compilers to translate the HumanEval benchmark suite of Python programs into 18\xspace{} programming languages that span a spectrum of language features and popularity. 

We present the first multi-language code correctness evaluation of two state-of-the-art NL2Code models: Codex and InCoder. We show that Codex performs best on JavaScript and does as well as Python on four other languages (C++, JavaScript, Scala, and TypeScript). Our results highlight the importance of testing: we do not find a strong correlation between perplexity and code correctness.
In our detailed by-language analysis, we find a predictable effect of language frequency, but draw mixed conclusions about the impact of type annotations. Our detailed error analysis highlights common patterns in four languages, finding model errors that are both like and unlike those of human programmers.

Our publicly available benchmark is also easy to extend to new  problems and languages. We hope it will help evaluate and develop future work on multi-language LLMs of code.

\ifanon

\else

\section{Acknowledgments}

We thank Steven Holtzen for loaning us his GPUs. This work was partially supported by the National Science Foundation grant CCF-2052696.

\fi



