
\section{Introduction}


Machine reading comprehension (MRC) is one of the fundamental tasks in natural language understanding, which requires a machine to read a document to correctly answer questions based on the context. MRC  has attracted significant efforts from both academia and industry with continuous development of MRC datasets~\cite{CNNDM,MSMARCO,RACE,SQuAD2,CoQA,DREAM}, which also keeps pushing the frontier of MRC models and learning algorithms~\cite{BERT,Roberta,Albert,GPT3} to eventually bridge the gap between AI systems and human readers. 

In addition to the advances in English MRC, researchers have also made substantial progresses in Chinese MRC challenges with many high-quality Chinese MRC datasets released. These MRC datasets focus on a variety of domains of Chinese understanding, such as fact extraction~\cite{CMRC2017,CMRC2018}, dialogue understanding~\cite{C3}, common sense~\cite{DuReader}, idiom selection~\cite{ChID} and exam questions used in language proficiency tests~\cite{C3}.

However, all these datasets provide particularly limited challenges for the purpose of building MRC models with the same language proficiency as \emph{native} Chinese speakers. There are 3 major dataset limitations. First, the length of reading materials are \textbf{\emph{short}}. For example, {{\textrm{$\mathrm{C^3_M}$}}}~\cite{C3}, a multiple-choice MRC dataset with the longest documents, has merely 180 characters per document on average. Even in the cloze-based datasets, the longest average document length is just around 500 characters. Second, the questions are \textbf{\emph{not sufficiently difficult}}. Most existing datasets are either extractive or domain-specific (e.g., focusing on idiom or simple facts). Although {{\textrm{$\mathrm{C^3}$}}}~\cite{C3} provides exam-based free-form multiple-choice questions, they are designed for \emph{non-native} speakers and therefore do not require native-level reasoning capabilities and common sense knowledge to answer the questions. More importantly, none of existing datasets consider reading comprehension on \textbf{\emph{classical Chinese}} documents, such as classical literature and poetry. Classical Chinese, as a writing style used in almost all formal writing until early 20th century~\cite{wiki}, plays a critical role in Chinese culture and has led to numerous idioms and proverbs. Even today, classical literature and poetry are still widely taught and examined in China's education system.

We developed a new general-form multiple-choice Chinese MRC dataset, \emph{Native Chinese Reader ({\textrm{NCR}})}, towards building a \emph{native-level} Chinese comprehension system. 
The {{\textrm{NCR}}} dataset contains 8390 documents with over 20K questions collected using the exam questions for the Chinese course in China's high schools, which are designed to evaluate the language proficiency of \emph{native} Chinese youth. Therefore, {{\textrm{NCR}}} naturally overcomes the limitations of existing datasets with sufficiently challenging questions and long documents in an average length of 1024 characters over both modern and classical Chinese writing styles (see Table~\ref{tab:comparison}).

We provided in-depth analysis of {{\textrm{NCR}}} and implemented baselines using popular pretrained models. To further examine the limit of current MRC methods, we additionally launched a  online competition using {{\textrm{NCR}}}. 
The best model we obtained achieves an average test accuracy of 59\%, which is far below human evaluation result of 79\% accuracy. This suggests a significant gap between current MRC model capabilities and the native-level Chinese language proficiency. We hope that {{\textrm{NCR}}} could serve as a milestone for the community to benefit future breakthroughs in Chinese natural language understanding.


\section{Related Work}




\paragraph{English Datasets:}
\input{21_english}


\paragraph{Chinese Datasets:}
\input{22_chinese}







\section{Native Chinese Reader ({\textrm{NCR}}) Dataset}
In this section, we provide detailed analysis of the documents and questions in our {{\textrm{NCR}}} dataset, including overall statistics, document styles, major challenges as well as studies on question types. 


\subsection{Task Format and Collection Methodology}
\label{collection}
In {{\textrm{NCR}}}, each document is associated with a series of multiple-choice questions. Each question has 2 to 4 options, of which exactly one is correct. The task is to select the correct option based on the document and the question.
Both questions and options are expressed in natural language covering a wide range of question types (more details discussed below).

All the questions and documents are collected from online open-access high-school education materials. 
After data cleaning, 8315 documents followed by \xss{20284} questions are obtained. We randomly split the dataset at the document level, with 6315 for training, 1000 for validation and 1000 for testing. 
Furthermore, to make sure our test set has sufficient novel questions that never appear online, we also invited a few high-school Chinese teachers to manually generate 193 questions for a total of 73 additional documents to augment the test set.
Finally, {{\textrm{NCR}}} consists of 6315 documents with 15419 questions for training, 1000 documents with 2443 questions for validation and 1073 documents with \xss{2615} questions for testing.
\begin{table}[ht]
    \caption{The overall statistics of different Chinese multi-choice MRC datasets. ChID and CMRC2019 are cloze-based without questions. \textbf{*} means statistics are collected over validation and test set only.}
    \label{tab:length_count}
    \small
    \centering
    \begin{tabular}{l|c c c c c}
    \hline
        \textbf{Datasets} & \tabincell{c}{Len. of Doc.\\Avg./Max.} & \tabincell{c}{Len. of Que.\\Avg./Max.} & \tabincell{c}{Len. of Opt.\\Avg./Max.} &
        \tabincell{c}{\#Opt. per Que.\\Min./Avg./Max.} &
        \tabincell{c}{\#Que. per Doc.\\Min./Avg./Max.}
        \\
    \hline
        \textbf{ChID}  & 159.1 / 581 & N/A & 4 / 4 & 7 / 7.0 / 7& 1 / 1.2 / 12 \\
        \textbf{CMRC 2019} & 557.3 / 688 & N/A & 13.8 / 29 & 5 / \textbf{10.6} / 15 & 0 / \textbf{9.9} / 15\\
        \textbf{C3-M} & 180.2 / 1,274 & 13.5 / 57 & 6.5 / 45 & 2 / 3.7 / 4 & 1 / 1.9 / 6 \\
        \textbf{C3-D} & 76.3 / 1,540 & 10.9 / 34 & 4.4 / 31 & 3 / 3.8 / 4 & 1 / 1.2 / 6\\
        \textbf{C3} & 116.9 / 1,540 & 12.2 / 57 & 5.5 / 45 & 2 / 3.8 / 4 & 1 / 1.5 / 6 \\
    \hline
   
   
        \textbf{NCR} \scriptsize{Classical only}* & 521.5 / 1,258 & 25.7 / 178 & 36.8 / 130 & 2 / 4.0 / 4 & 1 / 2.2 / 5  \\
        \textbf{NCR} \scriptsize{Modern only}* & 1207.8 / 4,640 & 24.4 / 276 & 44.1 / 152 & 2 / 4.0 / 4 &  1 / 2.5 / 5 \\
        \textbf{NCR} \scriptsize{All} & \textbf{1023.7 / 4,640} & \textbf{24.5 / 352} & \textbf{43.0 / 256} & 2 / 4.0 / 4 & 1 / 2.4 / 5 \\
    \hline
    \end{tabular}
\end{table}
\subsection{Dataset Statistics}
\label{statistics}
We summarize the \xss{high-level} statistics of our {{\textrm{NCR}}} dataset and other related multi-choice Chinese MCR datasets in Table~\ref{tab:length_count}. 
In addition, we also measure the statistics of classical and modern documents from the validation and test set, where we can observe that modern Chinese articles are more than twice longer than classical Chinese literature.
Comparing with other Chinese MRC datasets, {{\textrm{NCR}}} is an order of magnitude longer, even including those very concise classical Chinese documents.
Besides documents, {{\textrm{NCR}}} also contains much longer questions and answer options. Particularly for the option length, {{\textrm{NCR}}} is almost an order of magnitude longer except the CMRC2019 dataset. We remark that CMRC2019 is a cloze-style dataset with a completely different question style from {{\textrm{NCR}}}: CMRC2019 options are original document texts while the reader only needs to match the options to the corresponding blank in the document. Overall, {{\textrm{NCR}}} has substantially longer articles, questions and options 
with diverse document styles, which suggests a much higher comprehension difficulty than existing datasets.




\begin{table}[ht]
    \caption{Statistics of document length over {{\textrm{NCR}}} validation set and test set. Classical Chinese articles (including poetry) are much shorter than modern Chinese articles.
    }
    \label{tab:num_clsssical}
    \centering
    \begin{tabular}{c|c|ccc}
    \hline
Style & count & min & avg & max\\
    \hline
    Modern & 1493 &  47 & 1208 & 4640 \\
    Classical & 580 & 24 & 522 & 1258 \\
    Poetry & 63 & 24 & 156 & 668 \\
   
      
   
       
       
   
        
   
       
       
    \hline
    \end{tabular}
\end{table}
\input{32_table_peotry}
\subsection{Document Style and Challenges}
\label{article_style}

We manually annotated the writing styles of the documents in validation set and test set with summarized statistics in Table~\ref{tab:num_clsssical}. Almost a quarter of the documents are in classical Chinese. We remark that most documents are collected from online open-access resources. This indicates that classical Chinese indeed plays a critical role in China's Chinese class, which, however, is often ignored in previous Chinese MRC studies. Table~\ref{tab:short_examples} presents two example documents, one in classical Chinese (\textbf{D1}) and one in modern Chinese (\textbf{D2}), with associated questions. In the following content, we will discuss the major challenges in {{\textrm{NCR}}} with respect to different document writing styles.




\paragraph{Classical Chinese: }




Classical Chinese literature is substantially more difficult than modern Chinese documents due to its conciseness and flexible grammar. 
Most classical Chinese words are expressed in a single character and therefore are not restrictively categorized into parts of speech: nouns can be used as verbs, adjectives can be used as nouns, and so on. For example, the character \begin{CJK}{UTF8}{gbsn}``东''\end{CJK} only means ``east'' in modern Chinese. However, in the classical Chinese sentence, \begin{CJK}{UTF8}{gbsn}``顺流而东也''\end{CJK}(advance eastward along the river), it actually means ``advance eastward''. 
Classical Chinese also has distinguishing sentence patterns from nowadays, such as changing the order of characters and often dropping subjects and objects when a reference to them is understood. 




Furthermore, an important sub-category in classical Chinese is \emph{poetry}, which is typified by certain traditional poetic forms and rhythms. About 10\% of the classical documents in {{\textrm{NCR}}} are poetry. Table~\ref{tab:short_examples}, \textbf{D1} shows a famous classical Chinese poem from Song dynasty, which is particularly short and abstract in order to satisfy the poetic form of \begin{CJK}{UTF8}{gbsn}``相见欢''\end{CJK} (Xiang-Jian-Huan). This poem describes a scene where the poet stands on a tower staring at the moon. However, in order to correctly understand the sentiment and meaning of the poem, the reader needs to leverage imagery and symbolism in classical Chinese culture (e.g., moon and autumn mean sadness) as well as the personal background of the poet (e.g., Li Yu was a captured emperor)







\paragraph{Modern Chinese: }

For the modern Chinese documents in {{\textrm{NCR}}}, in addition to the challenge due to longer average length, the associated questions also focus more on the high-level metaphors and the underlying thoughts, which often require non-trivial reasoning with historical and cultural knowledge. Table~\ref{tab:short_examples}~\textbf{D2} shows an excerpt from a long article (the full document in {{\textrm{NCR}}} has about 2000 characters) written by a famous Chinese author, \begin{CJK}{UTF8}{gbsn}鲁迅\end{CJK} (Lu Xun). The article describes a scene where the author unexpectedly met one of his old friends not seen for a long time and had a meal together. The associated question (Q3) primarily asks about the high-level thoughts expressed by the author, which has to be inferred from the entire article and requires the readers to have strong knowledge of the author's personal experiences and the background of the era.

\xss{In addition to human annotation, we found that sentences in classical documents are usually shorter than in modern documents, which can be used as a simple criterion to categorize writing style. In detail, we split each document into sentences, compute the proportion of sentences with a length greater than 10, and denote it as $p_{(s>10)}$. 
We plot the histogram of $p_{(s>10)}$ in Figure~\ref{fig.dev+test} and~\ref{fig:training}. 
We remark that in validation and test sets, 98\% of the classical documents have $p_{(s>10)}<0.2$ while 96\% of the modern documents have $p_{(s>10)}\ge 0.2$. This suggests an approximate yet effective categorization criterion, i.e., $p_{(s>10)}<0.2$, for classifying document style over the training set.
}

\begin{figure}[t]
    \centering
    \begin{minipage}[t]{0.45\linewidth}
    \centering
    \includegraphics[width=1\linewidth]{figures/length_ratio.png}
  \caption{\xss{The histogram of $p_{(s>10)}$ over the validation and test sets. The blue bars correspond to classical documents, while the orange bars represent modern documents. We can observe that 0.2 is an effective classification boundary to distinguish classical and modern documents.}}
  \label{fig.dev+test}
    \end{minipage}
    \hspace{2mm}
    \begin{minipage}[t]{0.45\linewidth}
    \centering
    \includegraphics[width=1\linewidth]{figures/length_ratio_training.png}
  \caption{\xss{The histogram of $p_{(s>10)}$ over the training set. Although the writing style of these documents are not annotated, we could still observe two modalities with a separation boundary at $0.2$, which we will use as a rough criterion to distinguish modern and classical documents in the training set.}}
    \label{fig:training}
    \end{minipage}
\end{figure}



\subsection{Question Type}
\label{sec:question_type}
To perform fine-grained analysis of the questions in {{\textrm{NCR}}}, we conduct human annotations for a sampled batch of 300 questions from the test set. The label of each question is on the consensus of 3 annotators. 
The questions are categorized into 5 different categories:

\textbf{Matching} questions ask about a fact that has been explicitly described in the document. The correct answer can be directly obtained from a short span or a single sentence from the document. Note that different options can refer to different spans. 

\textbf{Semantic} questions ask about the semantic meanings of words or characters in a sentence, including antonym, synonymy, rhetoric and word segmentation. Q1 in Table~\ref{tab:examples} belongs to this category. We find that semantic questions are usually associated with classical Chinese documents.

\textbf{Summary} questions require the readers to understand all the facts stated throughout the entire document in order to choose a desired option, which presents a correct or incorrect fact summary.

\textbf{Reasoning} questions require the reader to perform non-trivial reasoning to infer a conclusion  not explicitly stated in the document. A reasoning question in {{\textrm{NCR}}} often requires the reader to strong background knowledge and common sense. Q3 in Table~\ref{tab:examples} and Q1 in Table~\ref{tab:reasoning_questions} belong to this category.

\textbf{Sentiment} questions ask about the implicit sentiment that the author expressed in the document. Sentiment questions in {{\textrm{NCR}}} typically require knowledge of imagery, symbolism and even the author's sociopolitical perspective.  Q2 in Table~\ref{tab:examples} belongs to this category.





\begin{table}[b]
   
    \caption{Distribution of question types in {{\textrm{NCR}}} and {{\textrm{$\mathrm{C^3}$}}}.}
    \label{tab:reasoning_type}
    \centering
    \begin{tabular}{c|c c c c c}
    \hline
      Type   & {{\textrm{NCR}}} & $\mathrm{C^3_M}$ & $\mathrm{C^3_D}$ & $\mathrm{C^3}$ \\
    \hline
        Matching & 0.33\% & 47.0\% &48.6\% & 47.8\% \\
        Semantic & 20.0\% & 1.7\% &2.3\% & 2.0\% \\
        Summary & 38.3\% & 15.3\%& 4.3\% & 9.8\% \\
        Reasoning & 28.3\% & 29.3\% & 36.7\% & 33.0\% \\
        Sentiment & 13.0\% & 6.7\% & 8.0 \% & 7.4\% \\
    \hline
   
   
    \end{tabular}
\end{table}

The annotations are summarized in Table~\ref{tab:reasoning_type}. We can observe that {{\textrm{NCR}}} has very few matching questions, which indicates that most {{\textrm{NCR}}} questions require non-trivial comprehension of the documents. 

As a comparison, we also sampled a total of 600 questions from {{\textrm{$\mathrm{C^3}$}}} dataset, another exam-question-based MRC dataset, with 300 from {{\textrm{$\mathrm{C^3_M}$}}} and 300 from {{\textrm{$\mathrm{C^3_D}$}}} respectively, and annotated the sampled questions with the same standard and annotation process.
The statistics are summarized in Table~\ref{tab:reasoning_type}. We can observe that {{\textrm{$\mathrm{C^3}$}}} contains a large portion of matching questions and much fewer summary and sentiment questions. Despite the fact that {{\textrm{NCR}}} and {{\textrm{$\mathrm{C^3}$}}} has about the same percentage of reasoning questions, we remark that reasoning questions in {{\textrm{NCR}}} are significantly harder than those in {{\textrm{$\mathrm{C^3}$}}}. This is not only because the documents in {{\textrm{NCR}}} are longer (so that fact extraction will be harder) but also because the reasoning questions in {{\textrm{NCR}}} typically require reasoning over a combination of document-level facts and background knowledge of both Chinese history and culture. To better illustrate the difference between {{\textrm{NCR}}} and {{\textrm{$\mathrm{C^3}$}}}, we select two example reasoning questions  in Table~\ref{tab:reasoning_questions}, with one from {{\textrm{$\mathrm{C^3}$}}} and one from {{\textrm{NCR}}} respective.


\begin{table}[ht]
    \caption{Examples of reasoning questions from NCR (top) and {{\textrm{$\mathrm{C^3}$}}} (bottom) with Chinese (left) and English translation (right). We defer the NCR document to Table~2 in Appendix~C. $\star$ denotes the correct option. In order to correctly answer Q1 from {{\textrm{NCR}}}, the reader not only needs to comprehend the texts in D1 describing the scene where Jun-tu meets the author's mother but also needs to understand the cultural meaning of  (madam).}
    \label{tab:reasoning_questions}
    \centering
    \small
    \begin{tabular}{m{0.8cm} | m{5cm} m{7cm}}
    \toprule
        \multirow{13}{*}{NCR}
        & D1: \begin{CJK}{UTF8}{gkai} 故乡（节选）鲁迅 \end{CJK}& D1: \emph{My old home} (excerpt) \textbf{Lu Xun}\\
        \cline{2-3}
        &Q1: \begin{CJK}{UTF8}{gkai}选出与本选段中中年闰土形象分析不恰当的一项\end{CJK} & Q1: Select the incorrect analysis of middle-aged Jun-tu's image. \\
        &\begin{CJK}{UTF8}{gkai}{A. 他称“我”母亲为“老太太”，表现了他有意讨好“我”母亲。}$\star$\end{CJK} & \textbf{A.He called my mother the ``madam'', which shows his intention to please my mother.} $\star$ \\
        &\begin{CJK}{UTF8}{gkai}B.他称自己少年时的好友为“老爷”，说明了他受封建等级观念影响很深。 \end{CJK} & B. He called his former good friend ``master'', which shows that he was deeply influenced by the feudal concept of hierarchy.  \\
        &\begin{CJK}{UTF8}{gkai}C.从他的对话中可以看出他的生活景况非常不好，他是当时下层人民形象的缩影。\end{CJK} &C. From his dialogue, we can see that the situation of his life is very bad, and he is the epitome of the image of the lower class at that time.\\
        &\begin{CJK}{UTF8}{gkai}D.宏儿和水生就像当年的“我”和闰土一样，彼此之间没有隔阂。\end{CJK} &D. Hung-eth and the Shui-sheng are just like me and Jun-tu at that time, they are not estranged from each other.\\
    \bottomrule
    \toprule
    \multirow{4}{*}{{\textrm{$\mathrm{C^3}$}}}
    &D2: \begin{CJK}{UTF8}{gkai} 男：还能不能再便宜点儿？” 女：“已经给您打五折了，先生！” \end{CJK}& D2. Man: Can you make it a little cheaper? " Woman: "I've given you a 50\% discount, sir!" \\
    \cline{2-3}
    &Q2: \begin{CJK}{UTF8}{gkai} 他们最可能是什么关系？ \end{CJK} & Q2. What is the most likely relationship between them? \\
    &\begin{tabular}{m{5.3cm}}
        \begin{CJK}{UTF8}{gkai} A.夫妻\quad C.老师和学生\end{CJK} \\ 
        \begin{CJK}{UTF8}{gkai} B.同事 \end{CJK}\quad  \begin{CJK}{UTF8}{gkai} \textsf{D.售货员和顾客} $\star$ \end{CJK}
    \end{tabular}
 & 
    \begin{tabular}{m{8cm}}
    A. Husband and wife \quad C. Teacher and student  \\ 
    B. Colleagues  \quad    \textbf{D. Salesperson and customer} $\star$
    \end{tabular}\\
    \bottomrule
    \end{tabular}
\end{table}







\section{Approach}
We use Chinese pre-trained BERT as the baseline. In this section, we will introduce BERT and itsand the t for this dataset.
\subsection{BERT}
\label{BERT}
The full name of BERT is Bidirectional Encoder Representation from Transformers, which is a pre-trained language model using Transformers encoder as the core. It is no longer like the previous language model for one-way language model training or shallow splicing of language models in two directions. It builds two pre-training tasks, Masked Language Model (MLM) and Next Sentence Prediction (NSP), which can be used in Semi-supervised learning is performed in a large number of unlabeled corpora, which can generate language representations with contextual information. In the representation of the input sequence, BERT adds [CLS] at the top of each sequence, and adds [SEP] at the end of the first sentence and the latter sentence. At the same time, in order to integrate the semantics, position, and upper and lower sentences of the token, Each token is mapped to token embedding, position embedding, and segmentation embedding, and the sum of the three embeddings is used as the input vector of the model. In downstream tasks, you only need to modify the output layer of the pre-trained BERT and fine-tune the model to achieve good results. Since the launch of BERT, BERT and its improved models have swept the lists of major NLPs.

\subsection{Improved model of BERT}
\label{BERT_family}
In the improved BERT model, RoBERTa removed the NSP in BERT and performed a series of parameter optimizations. ERNIE optimized the BERT mask mechanism to perform entity-level and phrase-level masks. Cui and others of Harbin Institute of Technology IFLYTEK introduced whole word masking (wwm) into the MLM task of BERT. First, the article was divided into word sequences with a Chinese word segmentation tool, and then the whole word was masked. In addition, the team also used more and larger corpus for pre-training, which proved to be effective in many downstream tasks. Later, Cui and others proposed MacBERT, replaced NSP with the Sentence Order Prediction (SOP) task in Albert, and made the following modifications to MLM: 1) wwm and n-gram mask were used. For the n-gram mask strategy, from The ratio of unigram to 4-gram is 40\%, 30\%, 20\%, 10\%, 2) In order to solve the problem that the [MASK] token used by the pre-training MLM will not appear in the fine-tuning stage, they will [MASK] token Replace with words with similar meanings to the original words. The choice of synonyms is based on Word2Vec. If no similar words are found, replace them with random words.
\subsection{Input and output processing}
\label{input_output}
The baseline model of this article is also based on pre-trained BERT, and the following modifications are made to the input layer and output layer. 
The input of this task consists of three parts, which are articles, questions, and options. There are 2-4 options. To enable batch processing, we fill the questions of 2 and 3 options into 4 options. For the convenience of presentation, the following notation is introduced below, the document is denoted as D, the question is denoted as Q, and the answer options are denoted as A1,A2,A3,A4. For the i-th option, we construct an input sequence of [CLS] D [SEP] Q [SEP] Ai [SEP], where D is Segment 1 input by BERT, and Q and Ai are combined as Segment 2. It can be seen from Table ~\ref{tab:statistics} that quite a lot of text length exceeds 512, the maximum length limit of BERT, so we adopt a strategy of intercepting the input sequence. First, we keep the parts of Q and A intact. For D, we assume that the beginning and end of the article usually convey more important information, so we cut off the middle part of D so that the total length of the sequence is maintained at 512.
We selected several commonly used Chinese pre-trained model as the baseline, including Google's BERT-chinese, BERT-multilingual, Baidu's ERNIE, HFL's BERT-wwm, BERT-wwm-ext, and MacBERT. These 6 pre-training models have the same model structure, all of which are 12-layer of Transformer Encoder. Each layer has 12 attention heads and the hidden state is 768. We also applied two large models, Roberta-large and MacBERT-large, both of which have 24 layers and 16 attention heads.
However, due to the different training data, data volume, and pre-training methods used by these pre-training BERTs, the training results are also different. In section ~\label{input_output}, we will compare the results of the 8 baseline models.
BERT will generate a 768-dimensional hidden state for each token of the input sequence. We take the output state of [CLS] token in the 12th layer and map it to a one-dimensional logit through a single-layer neural network. The logit means to what extent Ai is the correct option. In the same way, we can get the logit of the other 3 options, and the probability of the 4 options can be obtained by transforming the 4 logit by softmax. The goal of training is to minimize the four-class cross-entropy between the label probability and the ground truth. In the inferring stage, we just selects the option with the highest probability as the prediction result.

\section{Experiment}
\label{experiment setting}

In this section, we conduct quantitative study as well as human evaluation on our {{\textrm{NCR}}} dataset. 

\subsection{Baseline Methods}
\paragraph{Trivial Baselines:} We consider random guess and deterministic choice as trivial baselines. Deterministic choice always selects the same option ID.

\paragraph{Fine-Tuning of Pretrained Model: } We utilize the MRC model architecture from the BERT paper~\cite{BERT} and perform fine-tuning with {{\textrm{NCR}}}. We consider the fine-tuned performance of 7 popular Chinese pretrained models, including BERT-Chinese~\cite{BERT}, ERNIE~\cite{ERNIE}, BERT-wwm, BERT-wwm-ext, RoBERTa-large-Chinese~\cite{WWM}, MacBERT-base and MacBERT-large~\cite{MacBERT}.
We also investigate the effectiveness of data augmentation by additionally collecting 6000 documents and 13K exam questions for China's primary-school Chinese course. We combine these primary-school exam questions and the {{\textrm{NCR}}} training data as an augmented dataset to further boost the final performance. 
All the model and training details can be found in appendix~B.

\paragraph{Competition:}
To examine the limit of current MRC methods, we organized a 3-month-long online competition using {{\textrm{NCR}}} with training and validation set released. Participants are allowed to use any open-access pretrained model or any open-access \emph{unlabeled} data. Use of any external MRC \emph{supervision} is forbidden, since a portion of the test questions are possibly accessible online. This aims to prevent human annotations overlapping with our held-out data for a fair competition. There are a total of 141 participating teams and the best submission model with the highest test accuracy is taken as the \emph{competition} model. The team is from \xss{an} industry lab. They first pre-trained an XLNet-based model~\cite{XLNET} on a company-collected large corpus\footnote{Unfortunately, the company disagreed to release their internal pretraining data but the final trained model will be released at our project website.}. For each question, they use an information retrieval tool Okapi BM25~\cite{robertson1976relevance, robertson2009probabilistic} to extract the most relevant parts from the document and then run this pre-trained model for answer selection based on the extracted texts.






\paragraph{Human Evaluation:}
We randomly sample 50 documents with 120 questions from the annotated subset of test data in {{\textrm{NCR}}} and send these questions to 30 sophomore college students. All the students are native Chinese speakers majored in computer science, who have not \xss{taken} any Chinese course during the recent 2 years after college admission. Therefore, we believe they have reasonable reading comprehension capabilities close to typical China's high school students.  Each question is completed by at least 3 students to get an accurate performance estimation.



\subsection{Results}
\label{sec:result}

\subsubsection{Overall Performance}
\label{sec:overall_perf}
The performance of different baseline methods as well as human volunteers are summarized in Table~\ref{tab:results}. Pretrained models are substantially better than trivial baselines. Particularly, the MacBERT-large model produces the highest fine-tuning test accuracy of \xss{0.4780} while the use of external data augmentation further boost the test performance to \xss{0.5021}, which suggest the effectiveness of data augmentation. 
The best model comes from the participants of the online competition. The best competition model achieves a test accuracy of \xss{0.5985}, which is much higher than the best fine-tuning model that authors obtained. 
However, the human volunteers achieves an average accuracy of \xss{0.7917}, which results in a 20\% performance margin over the best baseline MRC model. \xss{Human performance is averaged per question over three annotators. To measure the inter-agreement, we calculate the agreement ratios. 60.83\% of the questions have the same answer from all the 3 students, and 96.67\% of the questions have the same answers from at least 2 students.}
\input{51_table_overall}
\input{52_table_article_style}
\input{55_table_modern_only}
\input{53_table_document_len}

\subsubsection{Fine-Grained Analysis}
We measure the performance of the MRC models, i.e., the best fine-tuned model (\textbf{\emph{FT}}) and the competition model (\textbf{\emph{CMP}}), and human on the test questions w.r.t. different factors, including writing style, document length and question type. 
We remark that the model accuracy are measured over the entire test set except the study on question type, which are over the annotated subset only.
\paragraph{Writing style:}
Table~\ref{tab:article_type_result} illustrates the performance of MRC models and human on different document styles. The performance of AI significantly drops on classical Chinese documents, particularly on poetry. By contrast, we observe an opposite phenomena for humans, who perform the best on poetry, the most abstract form of classical Chinese literature. \xss{We remark that in China's Chinese exams, questions on modern and classical texts may often have different examination focuses.
Questions on classical Chinese are more biased towards understanding the meaning of characters, words, and sentences (see Table~\ref{tab:short_examples}, Q1), which may not be intrinsically difficult for native Chinese students who have been well trained. While modern Chinese questions are often more general and require in-depth understanding of the entire document (see Table~\ref{tab:short_examples}, Q3), which can be challenging for humans. 
This is because Chinese documents are much longer and reading and remembering facts under long documents can easily make a human distracted.
}

\xss{We also investigate whether a AI model purely trained on modern Chinese can directly transfer to classical texts. 
Since we do not have ground truth annotations on training set, we following the filtering process in section~\ref{article_style} as a rough categorization, which yields 4507 documents.
We fine-tune the MacBERT-large on this filtered training set (without data augmentation) and show the testing results  in Table~\ref{tab:modern_only}.
We can observe that the performance on classical documents drops significantly with classical documents filtered out, while the performance on modern documents remains unchanged.
Hence, we argue that classical Chinese training data can be critical.
}
This also suggests an important direction for improving Chinese pre-trained models. 




\paragraph{Document Length:}
Table.~\ref{tab:document_len_result} summarizes the performance of human and AI on documents with various lengths. 
For classical Chinese documents, the most challenging documents are those shortest ones, which are most likely poetry.
For modern articles, human performance drops for particularly long articles. For AI, the hierarchically-structured competition model performs the worst on relatively short articles while the fine-tuning model has the most difficulties in documents with a moderate length, i.e., from 300 to 600 characters. This suggests possible enhancements on model architecture.
Moreover, although the performance gap between human and machine becomes less significant on particularly long documents, the accuracy of AI systems remains unsatisfying in general.
We also want to remark that even those relative short documents in {{\textrm{NCR}}} are substantially longer and more sophisticated than existing datasets of the similar question types like {\textrm{$\mathrm{C^3}$}}.



\input{54_table_question_type}
\paragraph{Question Type:}
We also compare the performance of human and AI on different question types in Table.~\ref{tab:reasoning_type_result}. To our surprise, sentiment questions, which are the most challenging for humans, yield the smallest performance margin. While the largest gap is on semantic questions, which we believe the easiest for human. This indicates that a pretrained model is capable of capturing high-level sentiment information but still lacks word/character-level reasoning abilities. In addition, we also observed that the hierarchical competition model performs much worse than the fine-tuned model on sentiment questions, which suggests that running a retrieval model first may result in a loss of document-level global information which can be critical for sentiment analysis.
This raises an open challenge for building more effective hierarchical models for processing long texts.















\section{Conclusion}

We present a novel Chinese MRC dataset, \emph{Native Chinese Reader ({\textrm{NCR}})}, towards building \emph{native-level} Chinese MRC models. Experiments on {{\textrm{NCR}}} indicate a significant gap between current MRC methods and human performance, which suggests great opportunities for future research, and, hopefully, pushes the frontier of Chinese natural language understanding.

\textbf{Remark:} Our dataset primarily consists of open-access exam questions or generated ones with teacher permission. All the documents are all public teaching materials. The released models are permitted by the online competition participants. Annotations and human evaluation results are completed by PhD students and interns that are all paid according to our institute regulation. Hence, we believe that our project will not lead to any legal or ethical issues. 


\section{Model Details}
\label{sec:model_details}

Our our code can be found at our project website and are released under the MIT license.


\input{41_input_output}

All baseline models are trained on 8 Tesla V100 GPU with 32G memory. We set the same hyper-parameters for all base models and large models, separately. For base models, we set epoch as 10, batch size as 64, learning rate as 5e-6. For large models, we set epoch as 10, batch size as 32, learning rate as 2e-6. Regarding data augmentation, we first use the primary school dataset for 5 epoch and then turn to NCR dataset for 5 epochs.  All the codes are implemented based on Hugging Face transformers~\cite{HuggingFace}. It takes about 5 minutes to run one epoch for base model and 20 minutes for large model. 

The best competition model applied a similar structure. However, they cut the document into several chunks and utilize an information retrieval tool Okapi BM25 to extract the most relevant chunk according to the question as to the actual passage, which reduce the super long document to a reasonable length. The document segment, question and the options are then input to XLNet-based classifier to predict the correct answer.
Compared to our baseline model, their IR tool can effectively keep the most informative segment of the document while reducing to a reasonable sequence length. 

\section{Surface pattern in options}
\begin{CJK}{UTF8}{gkai}
\xss{We investigate some special patterns and predict the answer based on these patterns. In Table~\ref{tab:pattern}, we focus on several absolute quantifiers including ``只能'' (``can only''), ``必然'', ``必定'', ``必须'', (``must''), ``只可能'', (``may only''), ``绝对'' (``absolute''). For each pattern, We keep the questions where only one option contains this pattern or only one option doesn't contain this pattern and choose the special option as the predicted answer. The results are aggregated from the whole dataset with a total of 20477 questions. ``combination’’ represents a combination pattern that including all the aforementioned patterns. We can observe that there are not many questions meeting the requirements, and the accuracy is indeed higher than random but also still very low.}
\begin{table}[ht]
    \centering
    \caption{\xss{Predict the answer based on special patterns. The results are aggregated from the whole dataset with a total of 20477 questions. ``combination’’ represents a combination pattern that including all the aforementioned patterns.}}
    \label{tab:pattern}
    \begin{tabular}{c|ccccccc}
    \toprule
        Pattern & 只能 & 必然 & 必定 & 必须 & 只可能 & 绝对 & combination \\
    \midrule
        \# Question & 291 & 308 & 52 & 658 & 5 & 101 & 1296 \\ 
        Accuracy & 0.3470 & 0.2825 & 0.3077 & 0.2903 & 0 & 0.2673 & 0.2994 \\
    \bottomrule
    \end{tabular}
\end{table}
\end{CJK}

\section{Additional Examples}
\label{sec:additional_examples}
Limited by space, we show some additional examples in this section. The example in Table~\ref{tab:classical_chinese} is a classical Chinese with a question of sentiment. It described a dialogue between the author and his friend, which reflects the author's inner contradictions and complex feelings. The author quoted predecessors' (\begin{CJK}{UTF8}{gkai}曹操 Cao Cao\end{CJK}) verses and the allusions to \textbf{the Battle of Chibi} (\begin{CJK}{UTF8}{gkai}赤壁之战\end{CJK}) to express his feelings. The question requires the readers to analyze the document from different perspectives, including the author's sentiment, writing techniques, and rhyme.

In Table~\ref{tab:hometown}, we present the document of question in Table~6 of the main paper with its English translation, which is a excerpt from Lu Xun's famous article \emph{My old home} (sometimes it is translated to \emph{Hometown})

\input{91_table_classical}

\input{92_table_hometown}





\section{Motivation}

\subsection{For what purpose was the dataset created? Was there a specific task in mind? Was there a specific gap that needed to be filled? Please provide a description. } we seek to build a native-level Chinese comprehension system, we collect documents with questions from the exam questions for the Chinese course in China’s high schools, which are designed to evaluate the language proficiency of native Chinese youth. These questions are also not easy for native Chinese speakers and aim to push the frontier of building native-level Chinese MRC models.

\subsection{Who created this dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)? }
It was created by Haihua Institute for Frontier Information Technology.

\subsection{Who funded the creation of the dataset? If there is an associated grant, please provide the name of the grantor and the grant name and number. }
Haihua Institute for Frontier Information Technology.

\section{Composition}

\subsection{What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? Are there multiple types of instances (e.g., movies, users, and ratings; people and interactions between them; nodes and edges)? Please provide a description. }

Each instance contains a document and a list of multiple-choice questions related to this document. And each question is comprised of a question text and 2~4 options, of which exactly one is correct, and the correct answer is also included.

\begin{table}[h!]
    \centering
    \caption{Fields and description in {\textrm{NCR}}}
    \begin{tabular}{m{4cm} | m{9cm}}
    \hline
        Field & Description \\
    \hline
        ID & Document ID \\
        Content & Document text \\
        Questions & A list of quetions \\
        Questions:Question & Question text \\
        Qusetions:Answer & The ground truth answer \\\
        Questions:Q\_id	& Question id\\
    \hline
        Type (Annotated in validation/test set) & Writing style of document, 00 for moder-style (without poetry), 11 for classical-style (without poetry), 22 for classical poetry, 33 for modern poetry \\
    \hline
    \end{tabular}
    \label{tab:my_label}
\end{table}

\subsection{How many instances are there in total (of each type, if appropriate)?}
NCR consists of 6315  documents with 15419 questions for training, 1000 documents with 2443 questions for validation and 1073 documents with 2615 questions for testing.

\subsection{Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set? If the dataset is a sample, then what is the larger set? Is the sample representative of the larger set (e.g., geographic coverage)? If so, please describe how this representativeness was validated/verified. If it is not representative of the larger set, please describe why not (e.g., to cover a more diverse range of instances, because instances were withheld or unavailable). }
The dataset is a sample of instances from the larger set containing all exam questions for the Chinese course in China’s high schools. It is impossible to collect all the questions since there are too many questions online and many new questions will be generated every day.

\subsection{What data does each instance consist of? “Raw” data (e.g., unprocessed text or images)or features? In either case, please provide a description. }

Data is all in the form of text.

\subsection{Is there a label or target associated with each instance? If so, please provide a description. }
We provide each question with the correct answer. For data in the validation and test set, we also provide the writing style of the document.

\subsection{Is any information missing from individual instances? If so, please provide a description, explaining why this information is missing (e.g., because it was unavailable). This does not include intentionally removed information, but might include, e.g., redacted text. }
There is no information missing.

\subsection{Are relationships between individual instances made explicit (e.g., users’ movie ratings, social network links)? If so, please describe how these relationships are made explicit.}
A document is associated with several questions, these questions share the same document.

\subsection{Are there recommended data splits (e.g., training, development/validation, testing)? If so, please provide a description of these splits, explaining the rationale behind them.}
We randomly split the dataset collected online at the document level, with 6315 for training, 1000 for validation and 1000 for testing. To make sure our test set has sufficient novel questions that never appear online, we also invited a few high-school Chinese teachers to manually generate 193 questions for a total of 73 additional documents to augment the test set. Finally NCR consists of 6315  documents with 15419 questions for training, 1000 documents with 2443 questions for validation and 1073 documents with 2615 questions for testing.  

\subsection{Are there any errors, sources of noise, or redundancies in the dataset? If so, please provide a description.}
There are no errors, sources of noise, or redundancies in the dataset because we filter out these data.

\subsection{Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)? If it links to or relies on external resources, a) are there guarantees that they will exist, and remain constant, over time; b) are there official archival versions of the complete dataset (i.e., including the external resources as they existed at the time the dataset was created); c) are there any restrictions (e.g., licenses, fees) associated with any of the external resources that might apply to a future user? Please provide descriptions of all external resources and any restrictions associated with them, as well as links or other access points, as appropriate.}
The dataset is self-contained.

\subsection{Does the dataset contain data that might be considered confidentiality, data that includes the content of individuals' non-public communications)? If so, please provide a description.}
No, all data is public.

\subsection{Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? If so, please describe why.}
The dataset does not contain any data that may be offensive, insulting, threatening, or might otherwise cause anxiety.

\subsection{Does the dataset relate to people? If not, you may skip the remaining questions in this section.}
This dataset does not relate to people.

\section{Collection}
\subsection{How was the data associated with each instance acquired? Was the data directly observable (e.g., raw text, movie ratings), reported by subjects (e.g., survey responses), or indirectly inferred/derived from other data (e.g., part-of-speech tags, model-based guesses for age or language)? If data was reported by subjects or indirectly inferred/derived from other data, was the data validated/verified? If so, please describe how. }
A document is associated with several questions, these questions share the same document, which is directly observable.

\subsection{What mechanisms or procedures were used to collect the data (e.g., hardware apparatus or sensor, manual human curation, software program, software API)? How were these mechanisms or procedures validated?}
The data was collected by manual human curation. And the collected data was validated by another person to ensure the quality.

\subsection{Who was involved in the data collection process (e.g., students, crowdworkers, contractors) and how were they compensated (e.g., how much were crowdworkers paid)?}
\begin{CJK}{UTF8}{gkai}
We contracted out the data collection process to 海天瑞声 ( SpeechOcean, http://en.speechocean.com/)
\end{CJK}

\subsection{Over what timeframe was the data collected? Does this timeframe match the creation time frame of the data associated with the instances (e.g., recent crawl of old news articles)? If not, please describe the timeframe in which the data associated with the instances was created.}
The data collection process lasted for around 40 days. Since the dataset contains many articles in classical Chinese, which can be date back to thousands of years ago.

\subsection{Were any ethical review processes conducted (e.g., by an institutional review board)? If so, please provide a description of these review processes, including the outcomes, as well as a link or other access point to any supporting documentation. }
No ethical review processes were conducted.

\section{Processing}
\subsection{Was any preprocessing/cleaning/labeling of the data done (e.g., discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? If so, please provide a description. If not, you may skip the remainder of the questions in this section.}
We clean the data by filter out some questions which are base on the format. For example, some questions are about the marked or boldface words.

\subsection{Was the “raw” data saved in addition to the preprocessed/cleaned/labeled data (e.g., to support unanticipated future uses)? If so, please provide a link or other access point to the “raw” data. }
No.

\subsection{Is the software used to preprocess/clean/label the instances available? If so, please provide a link or other access point. }
No.

\section{Uses}
\subsection{Has the dataset been used for any tasks already? If so, please provide a description.}
To examine the limit of current MRC methods, we organized a 3-month-long online competition using NCR with a training and validation set released.  Participants are allowed to use any open-access pre-trained model or any open-access unlabeled data.  Use of any external MRC supervision is forbidden, since a portion of the test questions are possibly accessible online. This aims to prevent human annotations overlapping with our held-out data for a fair competition. There are a total of 141 participating teams and the best submission model with the highest test accuracy is taken as the competition model. The team is from an industry lab. They first pre-trained an XLNet-based model on a company-collected large corpus. For each question, they use an information retrieval tool Okapi BM25 to extract the most relevant parts from the document and then run this pre-trained model for answer selection based on the extracted texts. The final model with the highest accuracy is released: \url{https://github.com/xssstory/NCR_competition_model}

\subsection{Is there a repository that links to any or all papers or systems that use the dataset?}
The competition website: \url{https://www.biendata.xyz/competition/haihua_2021/}

Github for the competition model with the highest accuracy: \url{https://github.com/xssstory/NCR_competition_model}

Github for baselines:
\url{https://github.com/xssstory/NCR_baseline}

\subsection{What (other) tasks could the dataset be used for?}
This dataset may also be used for a question-answering generation task. 
\subsection{Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses? For example, is there anything that a future user might need to know to avoid uses that could result in unfair treatment of individuals or groups (e.g., stereotyping, quality of service issues) or other undesirable harms (e.g., financial harms, legal risks) If so, please provide a description. Is there anything a future user could do to mitigate these undesirable harms?}
Users should keep in mind that the questions and their answer comes from different teachers. 

\subsection{Are there tasks for which the dataset should not be used? If so, please provide a description.}
Unknown.

\section{Distribution}
\subsection{Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created? }
Yes, the dataset is public now.

\subsection{How will the dataset will be distributed (e.g., tarball on website, API, GitHub)? Does the dataset have a digital object identififier (DOI)? }
The dataset is available at \url{https://github.com/xssstory/NCR_competition_model} and \url{https://drive.google.com/drive/folders/1Ci-KLHKk-yP-y5fWX4_cU8bA2fL_q76e?usp=sharing}

\subsection{When will the dataset be distributed?}
It is available now.

\subsection{Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)? If so, please describe this license and/or ToU, and provide a link or other access point to, or otherwise reproduce, any relevant licensing terms or ToU, as well as any fees associated with these restrictions.}
This dataset is released under the CC BY-SA 4.0 license for general research purposes.

\subsection{Have any third parties imposed IP-based or other restrictions on the data associated with the instances? If so, please describe these restrictions, and provide a link or other access point to, or otherwise reproduce, any relevant licensing terms, as well as any fees associated with these restrictions.}
No.

\subsection{Do any export controls or other regulatory restrictions apply to the dataset or to individual instances? If so, please describe these restrictions, and provide a link or other access point to, or otherwise reproduce, any supporting documentation.}
No

\section{Maintance}
\subsection{Who is supporting/hosting/maintaining the dataset?}
The dataset will be hosted on GitHub and google drive and will be maintained by Shusheng Xu and Yichen Liu.

\subsection{How can the owner/curator/manager of the dataset be contacted (e.g., email address)?}
The maintainers can be contacted at xuss20@mails.tsinghua.edu.cn and y17043@nyu.edu

\subsection{Is there an erratum? If so, please provide a link or other access point.}
No.

\subsection{Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)? If so, please describe how often, by whom, and how updates will be communicated to users (e.g., mailing list, GitHub)?}
The dataset will be updated if necessary, updates will be communicated via the project website \url{https://sites.google.com/view/native-chinese-reader/}  Github at \url{https://github.com/xssstory/NCR_competition_model} and \url{https://github.com/xssstory/NCR_baseline}

\subsection{If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (e.g., were individuals in question told that their data would be retained for a fixed period of time and then deleted)? If so, please describe these limits and explain how they will be enforced.}
N/A

\subsection{Will older versions of the dataset continue to be supported/hosted/maintained? If so, please describe how. If not, please describe how its will be communicated to users.}
Yes, it will be communicated to users via the project website \url{https://sites.google.com/view/native-chinese-reader/}  Github at \url{https://github.com/xssstory/NCR_competition_model} and \url{https://github.com/xssstory/NCR_baseline}

\subsection{If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so? If so, please provide a description. Will these contributions be validated/verified? If so, please describe how. If not, why not? Is there a process for communicating/distributing these contributions to other users? If so, please provide a description. }
This is not supported now.

\end{document}

\section{Datasheet}
\subsection{Motivation}
\subsubsection*{For what purpose was the dataset created? Was there a specific task in mind? Was there a specific gap that needed to be filled? Please provide a description. } we seek to build a native-level Chinese comprehension system, we collect documents with questions from the exam questions for the Chinese course in China’s high schools, which are designed to evaluate the language proficiency of native Chinese youth. These questions are also not easy for native Chinese speakers and aim to push the frontier of building native-level Chinese MRC models.

\subsubsection*{Who created this dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)? }
It was created by Haihua Institute for Frontier Information Technology.

\subsubsection*{Who funded the creation of the dataset? If there is an associated grant, please provide the name of the grantor and the grant name and number. }
Haihua Institute for Frontier Information Technology.

\subsection{Composition}

\subsubsection*{What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? Are there multiple types of instances (e.g., movies, users, and ratings; people and interactions between them; nodes and edges)? Please provide a description. }

Each instance contains a document and a list of multiple-choice questions related to this document. And each question is comprised of a question text and 2~4 options, of which exactly one is correct, and the correct answer is also included.

\begin{table}[h!]
    \centering
    \caption{Fields and description in {\textrm{NCR}}}
    \begin{tabular}{m{4cm} | m{9cm}}
    \hline
        Field & Description \\
    \hline
        ID & Document ID \\
        Content & Document text \\
        Questions & A list of quetions \\
        Questions:Question & Question text \\
        Qusetions:Answer & The ground truth answer \\\
        Questions:Q\_id	& Question id\\
    \hline
        Type (Annotated in validation/test set) & Writing style of document, 00 for moder-style (without poetry), 11 for classical-style (without poetry), 22 for classical poetry, 33 for modern poetry \\
    \hline
    \end{tabular}
    \label{tab:my_label}
\end{table}

\subsubsection*{How many instances are there in total (of each type, if appropriate)?}
NCR consists of 6315  documents with 15419 questions for training, 1000 documents with 2443 questions for validation and 1073 documents with 2615 questions for testing.

\subsubsection*{Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set? If the dataset is a sample, then what is the larger set? Is the sample representative of the larger set (e.g., geographic coverage)? If so, please describe how this representativeness was validated/verified. If it is not representative of the larger set, please describe why not (e.g., to cover a more diverse range of instances, because instances were withheld or unavailable). }
The dataset is a sample of instances from the larger set containing all exam questions for the Chinese course in China’s high schools. It is impossible to collect all the questions since there are too many questions online and many new questions will be generated every day.

\subsubsection*{What data does each instance consist of? “Raw” data (e.g., unprocessed text or images)or features? In either case, please provide a description. }

Data is all in the form of text.

\subsubsection*{Is there a label or target associated with each instance? If so, please provide a description. }
We provide each question with the correct answer. For data in the validation and test set, we also provide the writing style of the document.

\subsubsection*{Is any information missing from individual instances? If so, please provide a description, explaining why this information is missing (e.g., because it was unavailable). This does not include intentionally removed information, but might include, e.g., redacted text. }
There is no information missing.

\subsubsection*{Are relationships between individual instances made explicit (e.g., users’ movie ratings, social network links)? If so, please describe how these relationships are made explicit.}
A document is associated with several questions, these questions share the same document.

\subsubsection*{Are there recommended data splits (e.g., training, development/validation, testing)? If so, please provide a description of these splits, explaining the rationale behind them.}
We randomly split the dataset collected online at the document level, with 6315 for training, 1000 for validation and 1000 for testing. To make sure our test set has sufficient novel questions that never appear online, we also invited a few high-school Chinese teachers to manually generate 193 questions for a total of 73 additional documents to augment the test set. Finally NCR consists of 6315  documents with 15419 questions for training, 1000 documents with 2443 questions for validation and 1073 documents with 2615 questions for testing.  

\subsubsection*{Are there any errors, sources of noise, or redundancies in the dataset? If so, please provide a description.}
There are no errors, sources of noise, or redundancies in the dataset because we filter out these data.

\subsubsection*{Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)? If it links to or relies on external resources, a) are there guarantees that they will exist, and remain constant, over time; b) are there official archival versions of the complete dataset (i.e., including the external resources as they existed at the time the dataset was created); c) are there any restrictions (e.g., licenses, fees) associated with any of the external resources that might apply to a future user? Please provide descriptions of all external resources and any restrictions associated with them, as well as links or other access points, as appropriate.}
The dataset is self-contained.

\subsubsection*{Does the dataset contain data that might be considered confidentiality, data that includes the content of individuals' non-public communications)? If so, please provide a description.}
No, all data is public.

\subsubsection*{Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? If so, please describe why.}
The dataset does not contain any data that may be offensive, insulting, threatening, or might otherwise cause anxiety.

\subsubsection*{Does the dataset relate to people? If not, you may skip the remaining questions in this subsection.}
This dataset does not relate to people.

\subsection{Collection}
\subsubsection*{How was the data associated with each instance acquired? Was the data directly observable (e.g., raw text, movie ratings), reported by subjects (e.g., survey responses), or indirectly inferred/derived from other data (e.g., part-of-speech tags, model-based guesses for age or language)? If data was reported by subjects or indirectly inferred/derived from other data, was the data validated/verified? If so, please describe how. }
A document is associated with several questions, these questions share the same document, which is directly observable.

\subsubsection*{What mechanisms or procedures were used to collect the data (e.g., hardware apparatus or sensor, manual human curation, software program, software API)? How were these mechanisms or procedures validated?}
The data was collected by manual human curation. And the collected data was validated by another person to ensure the quality.

\subsubsection*{Who was involved in the data collection process (e.g., students, crowdworkers, contractors) and how were they compensated (e.g., how much were crowdworkers paid)?}
\begin{CJK}{UTF8}{gkai}
We contracted out the data collection process to 海天瑞声 ( SpeechOcean, http://en.speechocean.com/)
\end{CJK}

\subsubsection*{Over what timeframe was the data collected? Does this timeframe match the creation time frame of the data associated with the instances (e.g., recent crawl of old news articles)? If not, please describe the timeframe in which the data associated with the instances was created.}
The data collection process lasted for around 40 days. Since the dataset contains many articles in classical Chinese, which can be date back to thousands of years ago.

\subsubsection*{Were any ethical review processes conducted (e.g., by an institutional review board)? If so, please provide a description of these review processes, including the outcomes, as well as a link or other access point to any supporting documentation. }
No ethical review processes were conducted.

\subsection{Processing}
\subsubsection*{Was any preprocessing/cleaning/labeling of the data done (e.g., discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? If so, please provide a description. If not, you may skip the remainder of the questions in this subsection.}
We clean the data by filter out some questions which are base on the format. For example, some questions are about the marked or boldface words.

\subsubsection*{Was the “raw” data saved in addition to the preprocessed/cleaned/labeled data (e.g., to support unanticipated future uses)? If so, please provide a link or other access point to the “raw” data. }
No.

\subsubsection*{Is the software used to preprocess/clean/label the instances available? If so, please provide a link or other access point. }
No.

\subsection{Uses}
\subsubsection*{Has the dataset been used for any tasks already? If so, please provide a description.}
To examine the limit of current MRC methods, we organized a 3-month-long online competition using NCR with a training and validation set released.  Participants are allowed to use any open-access pre-trained model or any open-access unlabeled data.  Use of any external MRC supervision is forbidden, since a portion of the test questions are possibly accessible online. This aims to prevent human annotations overlapping with our held-out data for a fair competition. There are a total of 141 participating teams and the best submission model with the highest test accuracy is taken as the competition model. The team is from an industry lab. They first pre-trained an XLNet-based model on a company-collected large corpus. For each question, they use an information retrieval tool Okapi BM25 to extract the most relevant parts from the document and then run this pre-trained model for answer selection based on the extracted texts. The final model with the highest accuracy is released: \url{https://github.com/xssstory/NCR_competition_model}

\subsubsection*{Is there a repository that links to any or all papers or systems that use the dataset?}
The competition website: \url{https://www.biendata.xyz/competition/haihua_2021/}

Github for the competition model with the highest accuracy: \url{https://github.com/xssstory/NCR_competition_model}

Github for baselines:
\url{https://github.com/xssstory/NCR_baseline}

\subsubsection*{What (other) tasks could the dataset be used for?}
This dataset may also be used for a question-answering generation task. 
\subsubsection*{Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses? For example, is there anything that a future user might need to know to avoid uses that could result in unfair treatment of individuals or groups (e.g., stereotyping, quality of service issues) or other undesirable harms (e.g., financial harms, legal risks) If so, please provide a description. Is there anything a future user could do to mitigate these undesirable harms?}
Users should keep in mind that the questions and their answer comes from different teachers. 

\subsubsection*{Are there tasks for which the dataset should not be used? If so, please provide a description.}
Unknown.

\subsection{Distribution}
\subsubsection*{Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created? }
Yes, the dataset is public now.

\subsubsection*{How will the dataset will be distributed (e.g., tarball on website, API, GitHub)? Does the dataset have a digital object identififier (DOI)? }
The dataset is available at \url{https://github.com/xssstory/NCR_competition_model} and \url{https://drive.google.com/drive/folders/1Ci-KLHKk-yP-y5fWX4_cU8bA2fL_q76e?usp=sharing}

\subsubsection*{When will the dataset be distributed?}
It is available now.

\subsubsection*{Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)? If so, please describe this license and/or ToU, and provide a link or other access point to, or otherwise reproduce, any relevant licensing terms or ToU, as well as any fees associated with these restrictions.}
This dataset is released under the CC BY-SA 4.0 license for general research purposes.

\subsubsection*{Have any third parties imposed IP-based or other restrictions on the data associated with the instances? If so, please describe these restrictions, and provide a link or other access point to, or otherwise reproduce, any relevant licensing terms, as well as any fees associated with these restrictions.}
No.

\subsubsection*{Do any export controls or other regulatory restrictions apply to the dataset or to individual instances? If so, please describe these restrictions, and provide a link or other access point to, or otherwise reproduce, any supporting documentation.}
No

\subsection{Maintance}
\subsubsection*{Who is supporting/hosting/maintaining the dataset?}
The dataset will be hosted on GitHub and google drive and will be maintained by Shusheng Xu and Yichen Liu.

\subsubsection*{How can the owner/curator/manager of the dataset be contacted (e.g., email address)?}
The maintainers can be contacted at xuss20@mails.tsinghua.edu.cn and y17043@nyu.edu

\subsubsection*{Is there an erratum? If so, please provide a link or other access point.}
No.

\subsubsection*{Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)? If so, please describe how often, by whom, and how updates will be communicated to users (e.g., mailing list, GitHub)?}
The dataset will be updated if necessary, updates will be communicated via the project website \url{https://sites.google.com/view/native-chinese-reader/}  Github at \url{https://github.com/xssstory/NCR_competition_model} and \url{https://github.com/xssstory/NCR_baseline}

\subsubsection*{If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (e.g., were individuals in question told that their data would be retained for a fixed period of time and then deleted)? If so, please describe these limits and explain how they will be enforced.}
N/A

\subsubsection*{Will older versions of the dataset continue to be supported/hosted/maintained? If so, please describe how. If not, please describe how its will be communicated to users.}
Yes, it will be communicated to users via the project website \url{https://sites.google.com/view/native-chinese-reader/}  Github at \url{https://github.com/xssstory/NCR_competition_model} and \url{https://github.com/xssstory/NCR_baseline}

\subsubsection*{If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so? If so, please provide a description. Will these contributions be validated/verified? If so, please describe how. If not, why not? Is there a process for communicating/distributing these contributions to other users? If so, please provide a description. }
This is not supported now.
\section{Introduction}








































\end{document}

\subsection*{Acknowledgements}
Yi Wu is supported by 2030 Innovation Megaprojects of China (Programme on New Generation Artificial Intelligence) Grant No. 2021AAA0150000. We would also like to thank the anonymous reviewers for their insightful feedbacks.

\section{Project Statement}
\textbf{Dataset: } All the materials are collected from online education materials or generated by qualified high-school teachers. All the documents are manually verified by the authors so that they all follow the China's Chinese education standard, which does not contain any sensitive or inappropriate content for high school students. Language of the dataset is written by Mandarin Chinese in the simplified script per China's education system regulation, including the classical Chinese literature.


\textbf{Accessibility: } The dataset, related models as well as our human annotations are all kept at Google drive with links at our project website. All of our authors promise to keep the best efforts to keep them accessible. Our project website also has a detailed introduction to the online competition we organized. We include the requirement that all the participating teams should release their model in our competition agreement, so the release of models are permitted by all the participants. 

\textbf{Human Accuracy: } Note that each problem requires non-trivial reasoning and reading a long article. To ensure the students treat the problems seriously, we leave this task as a bonus homework in our deep learning course for second-year college students at Tsinghua university. We also make sure that each student will be assigned no more than 10 articles for a reasonable workload. We also make sure each problem is at least answered by 3 students to get an accurate estimate. 

\textbf{Licence:} This dataset is released under the CC BY-SA 4.0 license for general research purpose.

\clearpage
\input{90_appendix}
