
\section{Introduction}
\label{sec:sec1}

A movie review is a subjective evaluation written by a moviegoer, usually consisting of two kinds of data: 1) textual reviews and 2) ratings. Textual reviews are qualitative evaluations; ratings are quantitative evaluations expressed by scores. Fig.~\ref{fig:fig1} shows a sample movie review from IMDb\footnote{https://www.imdb.com/}, a leading website managing movie reviews along with other information. The example shows the rating and the textual review for the movie. We note that these two types of information are closely related: the rating may be considered a numerical summarization reflecting the textual review.

\begin{figure}[!h]
\centering {
  \includegraphics[width=13cm]{img/fig1.png}
}
\caption{Sample movie review}
\label{fig:fig1}
\end{figure}

Movie reviews have a significant impact on the decision making of potential moviegoers~\cite{chintagunta2010effects}. However, a certain group of movie reviewers may generate indistinguishable and/or intentionally malicious reviews so as to bias the overall movie rating. While such groups ought to lack credibility, many well-known existing movie review management systems including Metacritic~\footnote{https://www.metacritic.com/} and Rotten Tomatoes~\footnote{https://www.rottentomatoes.com} do not consider the credibility of movie reviews in calculating overall movie ratings.

To enhance the evaluation of ratings in movie review systems, it is necessary to make the credibility of movie reviews a factor in the overall movie rating. For this, we need a method to classify movie reviews according to their credibility. However, this is challenging because even experts cannot clearly judge movie review credibility and the number of movie reviews is very large, which requires much time and effort for manual annotation. Table~\ref{tab:tab6} shows an example of different annotation results for a real review from Rotten Tomatoes\footnote{https://www.rottentomatoes.com/}.







\begin{table}
\center{
\caption{An example of varying annotation results}
\setlength{\tabcolsep}{3pt}
\begin{tabular}{|c|c|c|c|}
\hline
\textbf{Textual reviews} & \textbf{Ratings}  & \textbf{Annotation1}  & \textbf{Annotation2}  \\ \hline\hline 
\makecell{Really bad. I'm a huge fan of the Marvel \\movies in general, but this one stands out \\ as a big disappointment.} & 4 & Distrusted & Trusted \\ \hline

\end{tabular}

\label{tab:tab6}
}
\end{table}

In this paper, we propose a weakly supervised learning method for judging the credibility of movie reviews. It annotates entire data sets based on a rule-based function instead of manual annotation by human beings, thus annotating at a very fast speed. The most influential factor in the accuracy of weakly supervised learning is a criterion for classification. In this paper, we use a simple and clear criterion based on the historical movie ratings of movie reviewers. Fig.~\ref{fig:fig2} shows actual examples of movie reviewers submitting indistinguishable reviews for multiple movies, which are consequently considered untrustworthy. Fig.~\ref{fig:fig2-a} is an example where a movie reviewer has given the maximum 10 points to all the movies they have reviewed. We indicate that indistinguishable overall positive reviews with similar expressions are written regardless of the specific movies reviewed. Fig.~\ref{fig:fig2-b} shows another example where a movie reviewer has given the minimum 0 points and overall negative reviews to all the movies reviewed. Hence, we conclude by evaluating their historical reviews that reviews that will be written by those movie reviewers also have no discrimination.


\begin{figure}[!h]
\begin{subfigure}[t]{0.47\linewidth}
  \centering
 
  \includegraphics[width=.99\textwidth]{img/fig2-a.png}  
  \caption{A reviewer giving the maximum points}
  \label{fig:fig2-a}
\end{subfigure}
\begin{subfigure}[t]{0.47\linewidth}
  \centering
 
  \includegraphics[width=.99\textwidth]{img/fig2-b.png}  
  \caption{A reviewer giving the minimum points}
  \label{fig:fig2-b}
\end{subfigure}%
\caption{Examples of movie reviewers without discrimination}
\label{fig:fig2}
\end{figure}

The contributions of the paper is summarized as follows. 

\begin{itemize}
    \item We propose a weakly learning method for judging the credibility of movie reviews. The proposed method enables fast annotation according to predefined rules. To reduce accuracy limitations in weakly supervised learning, we use a simple but effective criterion to classify movie reviews clearly according to their credibility, which we call \emph{historical credibility}.
    \item The proposed learning method is \emph{efficient}. Specifically, we show that the proposed method can annotate 8,000 movie reviews in only 0.712 seconds, which occupies about 0.145\% of the entire training process when we use Word2Vec as the text representation model. This is a notable result if the annotation results are effective because the usual (human) annotation approach to classifying movie reviews based on credibility takes much time and effort even for experts.
    \item The proposed learning model is \emph{effective}. Through extensive experiments, we measure the accuracy of the proposed learning model and that of the chosen comparison method on 40,000 movie reviews collected for five different movies. The comparison method uses the helpfulness votes of other reviewers as the criterion to judge the credibility of movie reviews, a criterion that has been used in much research. Our finding is that the proposed learning model shows an improvement in accuracy of the learning model over the comparison method by 1.57\% $\sim$ 4.54\% depending on the text representation model, machine learning technique, and movie genres chosen.
\end{itemize}

The paper is organized as follows. In Section~\ref{sec:sec2}, we describe related work. In Section~\ref{sec:sec3}, we describe text representation models and machine learning techniques that will be used in the proposed learning method. In Section~\ref{sec:sec4}, we propose a weakly supervised learning method using historical credibility as a criterion for judging the credibility of movie reviews. In Section~\ref{sec:sec5}, we explain how the experimental results show the efficiency and accuracy of the proposed method. In Section~\ref{sec:sec6}, we further discuss the proposed learning method. Section~\ref{sec:sec7} concludes the paper.

\section{Related Work}
\label{sec:sec2}

\noindent {\bf Research on weakly supervised learning:} Weakly supervised learning allows us to annotate large-scale data at a very fast rate, at the cost of potentially incurring some inaccurate annotation. Typical annotation methods for weakly supervised learning can be classified into three categories~\cite{dong2018data}: 1) crowdsourcing, 2) distant supervision, and 3) labeling functions. Crowdsourcing is a form of supervised learning, but annotation of data is performed by several annotators who are not experts~\cite{raykar2010learning}. Distant supervision extracts the structure from a data set using large-scale existing databases; the extracted structure is used for annotation~\cite{mintz2009distant}. Labeling functions are constructed from defined rules for annotating data~\cite{ratner2017snorkel, ratner2016data}. This allows annotation to be done quickly, but the results include some noise. The weakly supervised learning used in this paper employs labeling functions because we annotate the entire set of movie reviews based on the predefined criterion, which is obtained from analysis of historical movie ratings.

Recently, weakly supervised learning has been adopted in many studies. For instance, Lee et al. have applied weakly supervised learning~\cite{lee2018sentiment} to sentiment classification, using weakly supervised learning based on a convolutional neural network to identify keywords for classifying positive and negative sentences. Lin et al. have used weakly supervised learning for understanding the sentiments of users from social media content containing different data models such as texts and images~\cite{lin2018multi}. 


\noindent {\bf Research on movie reviews:} For movie reviews, most research has focused on sentiment analysis of textual reviews, because movie reviews depend on the sentiment experienced after watching the movie~\cite{serrano2015sentiment}. Topal et al. have constructed an emotion map according to emotions expressed after watching the movie and have recommended movies to moviegoers based on this emotion map~\cite{topal2016movie}. Manek et al. have performed sentiment classification for movie reviews using a Gini index based on feature selection and a support vector machine (SVM) classifier~\cite{manek2017aspect}. Chakraborty  et al. have compared the performance of two clustering algorithms, namely k-means and k-means++, for emotion classification of movie reviews by using Word2Vec as a classifier of the textual review~\cite{chakraborty2018comparative}. Alsaqer et al. have improved the accuracy of movie review summarization based on sentiment analysis~\cite{alsaqer2017movie}. He et al. have proposed a self-training method using some labeled features extracted from an existing sentiment lexicon for sentiment analysis~\cite{he2011self}. Elmurngi et al. have detected fake movie reviews based on sentiment analysis using supervised learning methods such as SVM and naïve Bayes~\cite{elmurngi2018fake}.

\noindent {\bf Research on credibility in online review systems:} To the best of our knowledge, there are no public methods that consider the credibility of movie reviews. Many well-known existing movie scoring systems including Metacritic and Rotten Tomatoes do not consider movie review credibility in calculating the overall movie rating. IMDb is known to consider movie reviewers’ credibility; however, since the specific methods are not disclosed, we cannot evaluate their effectiveness. 

On the other hand, much effort has been invested in research on review credibility in other online review systems. It is well known that judging the credibility of online reviews is important because they affect consumers’ decision making~\cite{lee2011helpful, mudambi2010makes, ham2019subjective}. Liu et al. have evaluated the effectiveness of reviews of travel products based on two aspects~\cite{liu2015makes}: 1) reviewer information such as profiles and 2) reviews including some quantitative information (i.e., ratings and length of reviews). As a result, they have shown that qualitative judgments are the most influential factors for review credibility. Ghose et al. have explored multiple aspects of textual reviews, namely subjectivity levels, readability, and spelling errors, to understand the helpfulness of reviews in terms of economic and social outcomes~\cite{ghose2010estimating}. Hochmeister et al. have compared destination experts (i.e., the most active members) in TripAdvisor with general reviewers and have shown that destination experts have more influence in the TripAdvisor community~\cite{hochmeister2013destination}. Barbado et al. have proposed a method for detecting fake online reviews in the domain of consumer electronics businesses (i.e., studying Yelp businesses)~\cite{barbado2019framework}, labeling reviews in the data set used as trustworthy or fake. Reyes et al. have studied five chosen factors in online reviews that can affect the decision making of customers by exploring TripAdvisor reviews~\cite{reyes2019impact}.

Some research results have shown that helpfulness votes from other reviewers are a critical measure for evaluating review credibility. Fang et al. have conducted empirical analysis to explore two factors affecting the value of reviews: 1) the reviews themselves and 2) reviewer characteristics~\cite{fang2016analysis}, using the helpfulness vote as the criterion for determining review value. Shan et al. have shown that the reviewer reputation generated by peer ratings has great influence on credibility evaluation~\cite{shan2016credible}. Yang et al. have examined six factors (specifically, reviewer location, reviewer level, reviewer helpfulness vote, review rating, review length, and review photo) and have shown that the review rating and reviewer helpfulness vote are the most influential factors~\cite{yang2017exploring}. Gang et al. have examined the relationship between emotions and review helpfulness~\cite{ren2019examining}. For this, they have defined three kinds of emotion embedded in online reviews (specifically, anger, fear, and sadness) and have analyzed the different effects of each emotion on perceived review helpfulness.

\section{Preliminaries}
\label{sec:sec3}

\subsection{Text representation models}
\label{sec:sec3-1}

In this paper, we build a learning model to classify movie reviews based on their credibility. To annotate movie reviews based on textual content, we need text representation models. We use two kinds of bag-of-words models: 1) TF-IDF and 2) Word2Vec. TF-IDF is a representative model for representing the text~\cite{jones2004statistical}. Word2Vec is a vector space model that embeds a corpus of text into a vector space~\cite{mikolov2013distributed}. The purpose of this paper is not to propose a specific novel text representation model. Instead, we choose some representative text representation models to show the effectiveness of the proposed learning method; we could equally apply any other models in place of TF-IDF and Word2Vec to the proposed method.

\subsubsection{TF-IDF}

TF-IDF represents the relative importance of words in a document based on the entire set of documents~\cite{wu2008interpreting}. Eq.~(\ref{equ1-1}) is the formula for obtaining TF-IDF, which is calculated by multiplying TF (text frequency) and IDF (inverse document frequency). Essentially, the result of TF-IDF increases as the frequency of given words in a document increases and the number of documents containing given words decreases. Eq.~(\ref{equ1-2}) and Eq.~(\ref{equ1-3}) are formulas for obtaining TF and IDF, respectively. TF is the frequency of a particular word in the document. Specifically, $f(i,j)$ is the frequency of a word $i$ in a document $j$ while $\max  \left\{ f(w,d):w\in d \right\}$ is the maximum of $f(i,j)$ of the words in the document. IDF is the total number of documents $|D|$ divided by the number of documents containing a word $t$. IDF uses a logarithm to prevent the result from increasing too much as $|D|$ increases~\cite{chowdhury2010introduction}. 

\begin{eqnarray}
\mathrm{TF-IDF_{i,j}} = TF_{i,j} \times IDF_{i}
\label{equ1-1}
\end{eqnarray}

\begin{eqnarray}
\mathrm{TF_{i, j}} = 0.5 + \frac{0.5 \times f(i, j)}{\max  \left\{ f(w,d):w\in d \right\} }
\label{equ1-2}
\end{eqnarray}

\begin{eqnarray}
\mathrm{IDF_{i}} =  \log\left( \frac{\mid D\mid}{count \left\{ d:t\in d  \right\}   }\right)
\label{equ1-3}
\end{eqnarray}

\subsubsection{Word2Vec}

Word2Vec is a word embedding model that maps words contained in the document into a vector space, where it tries to preserve contextual meanings~\cite{mikolov2013distributed}. Word2Vec supports two kinds of architectures~\cite{ling2015two}: 1) continuous bag-of-words~(simply, CBOW) and 2) continuous skip-gram, as shown in Fig.~\ref{fig:word2vec}. The continuous CBOW architecture tries to predict the center word from the context words. For example, in a given sentence, ``This movie \_ is wonderful,'' this architecture tries to find the most likely word to fill in the blank from the context. As a result, it shows better performance when we predict frequently used words. Conversely, the continuous skip-gram architecture tries to predict the context from a given center word. For example, for the given center word ``soundtrack,'' this architecture tries to predict the surrounding context.

\begin{figure}[!h]
\begin{subfigure}[b]{.47\linewidth}
  \centering
  \includegraphics[width=0.95\linewidth]{img/cbow.png}  
  \caption{Continuous bag-of-words~(CBOW)}
  \label{fig:cbox}
\end{subfigure}
\begin{subfigure}[b]{.48\linewidth}
  \centering
  \includegraphics[width=0.97\linewidth]{img/skipgram.png}  
  \caption{Continuous skip-gram}
  \label{fig:skipgram}
\end{subfigure}
\caption{Two architectures for Word2Vec~\cite{ling2015two}}
\label{fig:word2vec}
\end{figure}

\subsection{Machine learning techniques}
\label{sec:sec3-2}

For the classification of movie reviews, we apply machine learning techniques to the textual reviews. A variety of text classifiers using machine learning techniques have been proposed~\cite{ikonomakis2005text}: decision trees, naïve Bayes, rule induction, neural networks, nearest neighbors, and support vector machines~(SVMs). In this paper, we apply two techniques that have been studied for movie reviews~\cite{pang2002thumbs}: 1) naïve Bayes and 2) SVM. (As previously noted, we could apply any other machine learning techniques instead of naïve Bayes and SVM for the proposed method.)


\subsubsection{Naïve Bayes classifier} 

The naïve Bayes classifier has been widely used for document classification because it is quite simple but shows high accuracy~\cite{ch1997bayesian}. The name ``naïve Bayes’’ refers to a classifier based on Bayes' theorem. The important characteristic of the naïve Bayes classifier is that the features used for the classifier are stochastically independent of each other.

When a set of classes $C$ = \{${\mathbf c}_1$, ${\mathbf c}_2$, ..., ${\mathbf c}_n$\} is given, the naïve Bayes classifier predicts the ${\mathbf c}_i$ that is most likely to include a given document $d$ based on Bayes' theorem. Eq.~(\ref{equl-4}) shows Bayes' theorem, which gives the probability that a given document $d$ is included in a category ${\mathbf c}_i$. $P(d)$ is the probability of choosing a document $d$ from all the documents. $P$(${\mathbf c}_i$) is the number of documents in a category ${\mathbf c}_i$ divided by the total number of documents. $P$($d$ $\mid$ ${\mathbf c}_i$) is, for a given category ${\mathbf c}_i$, the probability of choosing document $d$ in that category. When a set of classes \{${\mathbf c}_1$, ${\mathbf c}_2$\} is given, we can predict the category in which a document $d$ is included by comparing $P$(${\mathbf c}_1\mid d$) and $P$(${\mathbf c}_2\mid d$).

\begin{eqnarray}
\mathrm{P(c_{i}\mid d)}=\frac{P(c_{i} ,d)}{P(d)} =  \frac{P(c_{i})P(d\mid c_{i})}{P(d)}
\label{equl-4}
\end{eqnarray}







\subsubsection{Support vector machines} 

A support vector machine~(SVM) finds the optimal hyperplane for a linear classification on data on a multidimensional space~\cite{vapnik2013nature}. Here, we can have multiple possible hyperplanes that can classify the data. The SVM finds a hyperplane having the maximal margin. Fig.~\ref{fig:fig3} shows an example of SVM-based classification. Here, when we compare candidate hyperplanes ${\mathbf B}_1$ and ${\mathbf B}_2$ in terms of the margin between the classified categories, ${\mathbf B}_1$ is chosen as the result hyperplane because its margin is greater than that of ${\mathbf B}_2$. The maximal margins concept aims to minimize incorrect classifications.

\begin{figure}[!h]
\centering{
\includegraphics[width=0.5\linewidth]{img/fig3.png}
}
\caption{An example of the SVM classifier}
\label{fig:fig3}
\end{figure}


We can extend SVM to the case of nonlinear classification. For this, the kernel function has been introduced~\cite{wilk2016comparative}, defining additional functions for describing new dimensions required in nonlinear classification. Fig.~\ref{fig:fig4} shows the effect of nonlinear classification using the SVM kernel function. Fig.~\ref{fig:fig4-a} shows objects that are not classified by linear classification in SVM; Fig.~\ref{fig:fig4-b} shows that they can be classified by nonlinear classification using a kernel function. Kernel functions commonly used for the SVM classifier are the polynomial kernel, the Gaussian kernel, and the radial basis function~\cite{wilk2016comparative}.


\begin{figure}[!h]
\begin{subfigure}[b]{.48\linewidth}
  \centering
  \includegraphics[width=0.95\linewidth]{img/fig4_a.png}  
  \caption{Linear classification}
  \label{fig:fig4-a}
\end{subfigure}
\begin{subfigure}[b]{.47\linewidth}
  \centering
  \includegraphics[width=0.95\linewidth]{img/fig4_b.png}  
  \caption{Nonlinear classification}
  \label{fig:fig4-b}
\end{subfigure}
\caption{The effect of nonlinear classification using a kernel function in SVM}
\label{fig:fig4}
\end{figure}

\section{The Proposed Method}
\label{sec:sec4}

In this section, we propose a new learning method for judging the credibility of movie reviews. Our research objectives are summarized as follows: the proposed learning model 1) should be efficient at dealing with large-scale movie review data sets and 2) should provide accurate results based on a clear criterion for judging the credibility of movie reviews. To satisfy the first objective, we propose a method based on weakly supervised learning, which allows fast annotation by a predefined rule. To satisfy the second objective, we propose a new criterion for judging the credibility of movie reviews, based essentially on historical rating records. In Section~\ref{sec:sec4-1}, we will present the overall framework based on weakly supervised learning. In Section~\ref{sec:sec4-2}, we will formally define a new criterion for judging credibility. 

\subsection{The overall framework}
\label{sec:sec4-1}

Fig.~\ref{fig:fig5} shows the overall framework for classifying movie reviews based on credibility. It consists of 1) a learning phase and 2) an evaluation phase. In the learning phase, we first analyze historical review ratings for each reviewer so as to determine movie review credibility. In Section~\ref{sec:sec4-2}, we present a new criterion for determining movie review credibility. Next, we annotate the entire movie review as ``trusted'' or ``distrusted'' based on the proposed criterion. Here, we note that the annotation of the entire data set is performed at a very fast speed because we can annotate movie reviews based on the defined rule using a clear criterion without manual effort. Finally, we build a learning model by obtaining a vector of keywords from textual reviews using two text representation models, namely Word2Vec and TF-IDF, and by applying two machine learning techniques, namely naïve Bayes classifier and SVM, to the annotated data sets. In the evaluation phase, we can use the learning model to judge the credibility of new movie reviews based on the textual review information. Here, thanks to the learning model, we can even judge the credibility of movie reviews that are written by first-time reviewers.

\begin{figure}[!h]
\begin{subfigure}[b]{1\linewidth}
  \centering
  \includegraphics[width=0.97\linewidth]{img/fig5-a.png}  
  \caption{Learning phase}
  \label{fig:fig5-a}
\end{subfigure}
\vspace*{0.3cm}
\begin{subfigure}[b]{1\linewidth}
  \centering
  \includegraphics[width=0.97\linewidth]{img/fig5-b.png}  
  \caption{Evaluation phase}
  \label{fig:fig5-b}
\end{subfigure}
\caption{A weakly supervised learning method for classifying movie reviews based on credibility}
\label{fig:fig5}
\end{figure}

 
\subsection{Historical credibility for weakly supervised learning}
\label{sec:sec4-2}

In this section we present a simple criterion, which we call \emph{historical credibility}, for weakly supervised learning. We need a criterion that can clearly classify movie reviews according to their credibility, utilizing the historical movie ratings of each movie reviewer. The basic idea is to identify movie reviewers who have given indistinguishable ratings and to define all movie reviews written by them as distrusted reviews. We define historical credibility as in Definition~1. 

\begin{definition}
\emph{Historical credibility} is a criterion that evaluates review credibility based on the historical review ratings for each reviewer, defined as follows: all reviews are classified as either trusted or distrusted. The \emph{distrusted reviews} are defined as those written by a reviewer $R$ satisfying the following two conditions: 1) the standard deviation of all past movie ratings $X_R$ is 0 and 2) the average over $X_R$ is either the lowest score (e.g., 1 point) or the highest score (e.g., 10 points). All other reviews are defined as the \emph{trusted reviews}. Here, we exclude a single review written by a reviewer because it is classified as distrusted by the definition but cannot be judged.\hfill
\end{definition}

As a comparison criterion for the proposed historical credibility, we choose the helpfulness vote. In general review systems, the helpfulness vote has been used as a meaningful measure for determining the value of the review, as explained in Section~\ref{sec:sec2}. Thus, Shan et al. and Yang et al. have shown that the helpfulness vote has a great influence on credibility evaluation~\cite{shan2016credible, yang2017exploring}. Further, Fang et al. have used the helpfulness vote as a criterion to evaluate the influence of various review factors on review credibility~\cite{fang2016analysis}. Similarly, with movie reviews, the helpfulness vote can be an effective criterion in determining review value.

Table~\ref{tab:tab1} shows annotation methods for historical credibility as the proposed criterion and the helpfulness vote as the comparison criterion. For historical credibility, we define distrusted reviews by Definition~1 and all other reviews as trusted. For the helpfulness vote, we define each movie review as trusted when the number of helpful votes for the review exceeds the number of unhelpful votes. 

\begin{table}
\fontsize{9}{11}{11}\selectfont
\centering
\caption{Data annotation based on credibility criteria
}
\label{tab:tab1}
\begin{tabular}{|c|p{4cm}|p{4cm}|}
\hline
\textbf{Movie reviews}                                               
& \textbf{Historical credibility}
& \textbf{Helpfulness vote~\cite{shan2016credible, yang2017exploring, fang2016analysis}}                              
                                                                                             
\\ \hline \hline
Distrusted review          
& Review that is defined as distrusted by Definition~1                                           
& Review with fewer helpful than unhelpful votes, or equal numbers of each kind of vote                                  
\\ \hline
Trusted review             
& Review that is defined as trusted by Definition~1                               
& Review with more helpful than unhelpful votes                                                                                                                                                                 
\\ \hline
\end{tabular}
\end{table}

\section{Performance Evaluation}
\label{sec:sec5}

\subsection{Data collection}
\label{sec:sec5-1}

In this section, we describe the method we used to collect movie reviews and present the results of the data collection. We use BeautifulSoup and Selenium to implement a crawler to collect movie reviews from a movie website. The target movie website is Naver Movie\footnote{https://movie.naver.com/.}, the largest movie website in Korea\footnote{As of October 17, 2019, Naver was managing 12,539,719 ratings and 2,086,503 textual reviews.}. From this website, we can collect all the historical movie reviews for a specific user; this is an essential condition for verifying the concept proposed in this paper. In addition, we collect the number of helpful or unhelpful votes for each movie review, as required for the comparison method. Only if these conditions are satisfied can we apply our concept into data sets crawled from any other website carrying movie reviews.

We consider various movie genres, comparing learning for movie reviews within the same genre and across different genres. For this purpose, reviews of a total of five movies from three different genres were collected. We collect movie reviews having both movie ratings and textual reviews. The movie ratings range from 1 through 10 points; the textual reviews are described in up to 140 characters. We also collect all the historical movie ratings for the movie reviewers included for the five movies chosen. The total number of historical movie ratings for these movie reviewers was 87,125. Table~\ref{tab:table2} shows the characteristics of the movie reviews collected. We collected 40,000 movie reviews, 8,000 for each movie. Here, 32,000 reviews are used for training and 8,000 are used for testing. 

\begin{table}[]
\centering
\caption{Characteristics of collected movie reviews}
\label{tab:table2}
\begin{tabular}{|c|c|c|c|c|}
\hline
\textbf{Genres}        & \textbf{Movies}                                               & \textbf{Release date} & \textbf{\begin{tabular}[c]{@{}c@{}}Training \\ data\end{tabular}} & \textbf{\begin{tabular}[c]{@{}c@{}}Testing\\ data\end{tabular}} \\ \hline \hline
\multirow{3}{*}{Drama} & Assassination                                                          & 22 Jul. 2015          & 6,400                                                                      & 1,600                                                                     \\ \cline{2-5} 
                       & \begin{tabular}[c]{@{}c@{}}I can speak\end{tabular} & 21 Sep. 2017          & 6,400                                                                      & 1,600                                                                     \\ \cline{2-5} 
                       & The Spy Gone North                                                 & 08 Aug. 2018          & 6,400                                                                      & 1,600                                                                     \\ \hline
Comedy                 & \begin{tabular}[c]{@{}c@{}}Intimate Strangers\end{tabular}                                           & 31 Oct. 2018          & 6,400                                                                      & 1,600                                                                     \\ \hline
Action                 & Confidential Assignment                                                   & 18 Jan. 2017          & 6,400                                                                      & 1,600                                                                     \\ \hline
\hline
\multicolumn{3}{|c|}{Total}                                                                                    & 32,000                                                                      & 8,000                                                                    \\ \hline
\end{tabular}
\end{table}

\subsection{Experimental methods}
\label{sec:sec5-2}

In the experiments, we measure the elapsed learning time for movie reviews to check the efficiency of annotation in the proposed learning method. In addition, we measure the accuracy of the proposed learning method using historical credibility and that of the comparison method using the helpfulness vote as the classification criterion. Here, we measure the accuracy of combinations of the text representation models (TF-IDF and Word2Vec) and machine learning techniques (naïve Bayes and SVM). For TF-IDF and Word2Vec, we first need to extract keywords from the movie reviews. For this, we use a Korean keyword extractor \footnote{https://pypi.org/project/python-mecab-ko/} and select the top 20 keywords based on frequency for each movie review. In TF-IDF, we apply a logarithm to TF to smooth its significance because the same proper nouns, such as actor/actress and movie names, tend to be repeated in movie reviews. In Word2Vec, we use a skip-gram to consider the relevance of given keywords such as actor/actress and movie names in the movie reviews. We use a Gaussian kernel function for SVM, which has been widely used to classify textual data on high dimensions~\cite{keerthi2003asymptotic}. For the experiment, the entire data set is annotated according to the annotation methods presented in Section~4.2. 

The experimental results consist of three parts:

\begin{enumerate}
    \item Elapsed time for the proposed method
    \item Accuracy comparison between the proposed method and the helpfulness vote
    \item Variation in accuracy of the proposed method
\end{enumerate}

In the first part, we show the efficiency of the proposed learning method. In the second part, we use the following experimental variables: 1) textual representation models, 2) machine learning techniques, and 3) variety of movie genres. In the third part, we use the following experimental variables: 1) number of movies, 2) machine learning techniques, and 3) textual representation models. 



\subsection{Experimental results}
\label{sec:sec5-3}


\subsubsection*{Elapsed time for the proposed method}

Table~\ref{tab:tab4} shows the average elapsed learning time for the five chosen movies using the proposed learning method using a combination of text representation models and machine learning techniques. Here, to measure the elapsed time, we use 8,000 movie reviews for each movie. We note that annotation of 8,000 movie reviews requires only 0.712 seconds on average. In particular, we note that when we use Word2Vec as the text representation model, which shows better performance than TF-IDF as will be shown in Section 5.3.3, the portion of the entire training process occupied by annotation is only about 0.145\% $\sim$ 0.280\%. By comparison, manual annotation to judge the credibility of 8,000 movie reviews requires much more time in absolute terms; the helpfulness vote approach also requires much time to collect enough votes after the movie reviews are written.

\begin{table}
\fontsize{9}{10}\selectfont
\center{
\caption{ Elapsed learning time for movie reviews (seconds)}
\setlength{\tabcolsep}{3pt}
\begin{tabular}{|c|c|c|c|c|}

\hline
Combination & \textbf{TF-IDF + NB} & \textbf{W2V + NB}  & \textbf{TF-IDF + SVM} & \textbf{W2V + SVM} \\
\hline
\hline Annotation & \multicolumn{4}{|c|}{0.712}  \\ 
\hline Training & 0.122 & 195.822 & 3.533 & 288.114 \\
\hline Testing & 0.040 & 57.615 & 0.027 & 200.070 \\
\hline
\hline  \textbf{Total} & 0.874 & 254.149 & 4.273 & 488.896 \\

\hline
\end{tabular}
\label{tab:tab4}
}
\end{table}

\subsubsection{Accuracy comparison between the proposed method and the helpfulness vote}

Fig.~\ref{fig:fig6} compares the accuracy of the methods by credibility criteria, namely the proposed historical credibility and the helpfulness vote. Here, we measure the accuracy for each combination of text representation models, namely TF-IDF or Word2Vec, and machine learning techniques, namely naïve Bayes or SVM. The results indicate that the accuracy of the proposed historical credibility is comparable to or even better than that of the helpfulness vote for all combinations of text representation models and machine learning techniques.

Fig.~\ref{fig:fig6-a} shows the accuracy when we use TF-IDF as the text representation model and naïve Bayes as the machine learning technique. The result shows that mean accuracy for the helpfulness vote is 0.506, while that for historical credibility is 0.526. That is, the proposed historical credibility shows better accuracy than the helpfulness vote by a margin of 3.92\%. Fig.~\ref{fig:fig6-b} shows the accuracy when we use TF-IDF with SVM as the machine learning technique. The result shows that the proposed historical credibility has an accuracy improved over the helpfulness vote by 4.41\%. Fig.~\ref{fig:fig6-c} shows the accuracy when we use Word2Vec as the text representation model and naïve Bayes as the machine learning technique. The result shows that the proposed historical credibility has an accuracy improved over the helpfulness vote by 1.55\%. Fig.~\ref{fig:fig6-d} shows that the accuracy of the proposed historical credibility when we use Word2Vec and SVM is improved over the accuracy of the helpfulness vote by 2.49\%. 

\begin{figure}[!h] 
  \begin{subfigure}[b]{0.5\linewidth}
    \centering
    \includegraphics[width=0.95\linewidth]{img/fig6_a.png} 
    \caption{Naïve Bayes with TF-IDF} 
    \label{fig:fig6-a} 
    \vspace{4ex}
  \end{subfigure
  \begin{subfigure}[b]{0.5\linewidth}
    \centering
    \includegraphics[width=0.95\linewidth]{img/fig6_b.png} 
    \caption{SVM with TF-IDF} 
    \label{fig:fig6-b} 
    \vspace{4ex}
  \end{subfigure} 
  \begin{subfigure}[b]{0.5\linewidth}
    \centering
    \includegraphics[width=0.95\linewidth]{img/fig6_c.png} 
    \caption{Naïve Bayes with Word2Vec} 
    \label{fig:fig6-c} 
  \end{subfigure
  \begin{subfigure}[b]{0.5\linewidth}
    \centering
    \includegraphics[width=0.95\linewidth]{img/fig6_d.png} 
    \caption{SVM with Word2Vec} 
    \label{fig:fig6-d} 
  \end{subfigure} 
  \caption{Accuracy comparison by combination of text representation models and machine learning techniques}
  \label{fig:fig6} 
\end{figure}

Fig.~\ref{fig:fig9} compares the accuracy of the methods by credibility criteria, namely historical credibility and helpfulness vote, and by variety of movie genres. Here, we use three movies having the same genre, drama, and then having different genres: drama, comedy, and action. The result shows that in improvement in accuracy of the proposed historical credibility over that of the helpfulness vote in both cases. Specifically, the proposed historical credibility shows accuracy improvements of 1.57\% in the case of the same genre and 4.54\% in the case of the three different genres. 

\begin{figure}[!h]
\begin{subfigure}{0.47\linewidth}
  \centering
  \includegraphics[width=.99\linewidth]{img/fig9_a.png}  
  \caption{The same movie genre: drama}
  \label{fig:fig9-a}
\end{subfigure}
\quad
\begin{subfigure}{0.47\linewidth}
  \centering
  \includegraphics[width=.99\linewidth]{img/fig9_b.png}  
  \caption{ Different movie genres: drama, comedy, and action}
  \label{fig:fig9-b}
\end{subfigure}

\caption{Accuracy comparison by variety of movie genres}
\label{fig:fig9}
\end{figure}

\subsubsection*{Variation in accuracy of the proposed method}

Fig.~\ref{fig:fig7} shows the accuracy of the proposed historical credibility by text representation model, namely TF-IDF and Word2Vec. Here, we select three movies from different genres: Assassination, Intimate Strangers, and Confidential Assignment. Fig.~\ref{fig:fig7-a} shows the accuracy of TF-IDF and that of Word2Vec when we keep the machine learning technique fixed as naïve Bayes; Fig.~\ref{fig:fig7-b} shows the respective accuracy with the machine learning technique fixed as SVM. Both results show better accuracy when using Word2Vec than when using TF-IDF. Specifically, for the naïve Bayes learning technique, mean accuracy using TF-IDF is 0.526 compared with 0.552 using Word2Vec; in the SVM technique, the mean accuracy using TF-IDF is 0.529 compared with 0.551 using Word2Vec.

\begin{figure}[!h]
\begin{subfigure}{0.47\linewidth}
  \centering
  \includegraphics[width=.98\linewidth]{img/fig7_a.png}  
  \caption{Naïve Bayes}
  \label{fig:fig7-a}
\end{subfigure}
\quad
\begin{subfigure}{0.47\linewidth}
  \centering
  \includegraphics[width=.98\linewidth]{img/fig7_b.png}  
  \caption{SVM}
  \label{fig:fig7-b}
\end{subfigure}

\caption{Accuracy variation by text representation model}
\label{fig:fig7}

\end{figure}

Fig.~\ref{fig:fig8} shows the accuracy by machine learning technique, namely naïve Bayes and SVM. Here, we select the same three movies from different genres as before. Fig.~\ref{fig:fig8-a} shows the accuracy of naïve Bayes and that of SVM when we keep the text representation model fixed as TF-IDF. Fig.~\ref{fig:fig8-b} shows the respective accuracy with the text representation model fixed as Word2Vec. The results show slightly better overall accuracy when using SVM than when using naïve Bayes. Specifically, in TF-IDF, mean accuracy using naïve Bayes is 0.526 compared with 0.529 using SVM. In Word2Vec, mean accuracy using naïve Bayes is 0.551 compared with 0.551 using SVM.

\begin{figure}[!h]
\begin{subfigure}{0.47\linewidth}
  \centering
  \includegraphics[width=.99\linewidth]{img/fig8_a.png}  
  \caption{TF-IDF}
  \label{fig:fig8-a}
\end{subfigure}
\quad
\begin{subfigure}{0.47\linewidth}
  \centering
  \includegraphics[width=.99\linewidth]{img/fig8_b.png}  
  \caption{Word2Vec}
  \label{fig:fig8-b}
\end{subfigure}

\caption{Accuracy variation by machine learning technique}
\label{fig:fig8}
\end{figure}

\section{Discussions}
\label{sec:sec6}

\noindent {\bf Language-independent method: } In this paper, we use movie reviews collected from the largest Korean movie review website, Naver Movie, because it satisfies two requirements to show the effectiveness of the proposed method: 1) it maintains all the historical movie reviews written by each user, as required for the proposed historical credibility method; 2) it maintains helpful and unhelpful votes for each review, as required for the comparison method based on the helpfulness vote. We can apply and test the proposed method to movie reviews collected from other websites only if they satisfy the two requirements above. We note that the proposed method is language-independent. The accuracy of the learning model might vary according to the language used in movie reviews. In this paper, we have focused on the relative accuracy difference between methods, not on the absolute value of the accuracy for the target language. This relative difference should be maintained in different languages.

\noindent {\bf Variation in accuracy according to text representation model and machine learning technique: } In this paper, we aim to show the effectiveness of the proposed learning method, rather than the superiority of a specific text representation model or a specific machine learning technique. For this reason, we choose two representative text representation models and two representative machine learning techniques. However, we could apply any other text representation model and/or machine learning technique to the proposed method.
 

\section{Conclusions}
\label{sec:sec7}

In this paper, we have proposed a weakly supervised learning method for judging the credibility of movie reviews. The proposed learning method has two advantages. First, it is efficient because it allows fast annotation according to predefined rules. Second, it is effective because it uses a simple but effective criterion to judge movie review credibility based on historical ratings. 

For the experiments, we collected 40,000 actual movie reviews. We have shown that the proposed learning method is efficient because it can annotate 8,000 movie reviews in only 0.712 seconds, which occupies only about 0.145\% of the entire training process when we use Word2Vec as the text representation model. We have also measured the accuracy of the proposed learning method. By comparing accuracy between the proposed learning method and the comparison method based on the helpfulness vote, which has been used as an effective criterion in many studies, we have shown that the accuracy of the proposed method is better than that of the comparison method by 1.57\% $\sim$ 4.54\% depending on the text representation model, machine learning technique, and movie genres used.
