\section{Introduction}
Social media's growing popularity amongst children and young adults raises serious concerns for their safety. The threat of online sexual grooming is an increasing problem in the digital age \citep{greene2020experiences}. In 2021 alone, UK police forces recorded over 5000 offences relating to sexual communications with a child, with social media apps such as Instagram and Snapchat popular amongst online predators \citep{nspcc}. Minors who fall victim to offenders suffer considerably, with many abusers seeking to establish physical contact offline \citep{shelton2016online}. To assist law enforcement, academics have been trying to identify in advance those predators that steer the relationship towards physical encounters \citep{briggs2011exploratory, winters2017sexual, o2003typology, williams2013identifying}. This important work predominantly relies on time-consuming human-effort to manually annotate conversations involving child sex offenders. In this work, we explore the extent to which automated annotation can mimic an expert human annotator.




Automatically detecting predatory behavior is a challenging task. Offenders use a variety of subtle behaviors to manipulate the flow of the conversation. Predators may use flattery to build trust \citep{barber2021exposing}, make threats or bribe a child as a coercion tactic \citep{joleby2021offender}. Whilst human experts can identify these contextual psychological behaviors in text, their implicitness can be problematic for machines \citep{buckingham2020extreme}. In computational social sciences, dictionary-based approaches \citep{tausczik2010psychological} are often used to identify psychological characteristics such as neuroticism \citep{bogdanova2014exploring}. However, these methods heavily rely on the included vocabulary, causing a large numbers of false positives for some behaviors \citep{kaur2021abusive} while overlooking others \citep{broome2020psycho}. Prior work utilising machine learning has focused on identifying predators from a mixed corpus of illicit and everyday conversations \citep{inches2012overview,pendar2007toward,miah2011detection,EBRAHIMI201633,gupta2012characterizing}. While valuable in its own right, this line of research does not offer significant value to grooming experts as it identifies grooming behavior after-the-fact \citep{razi2021human} and does not link or contribute to psychological insight.  



This work is novel in its use of supervised and deep learning methods to analyse an expert-annotated corpus of real conversations between sexual predators and decoys pretending to be early teens. We report results and highlight the performance possible without incurring the cost of expert-labelling.
The paper is organised as follows: the methods and experimental setup are described in Section~\ref{sec:method}; Section~\ref{sec:results} then summarises results (with further details provided in the appendices) before Section~\ref{sec:concl} concludes the paper.

\section{Method and Experimental Setup}
\label{sec:method}

\begin{table}[h]
\centering
\begin{tabular}{lcc}
\hline
\textbf{Region} & \textbf{Num. Msgs} & \textbf{\% of Corpus}\\
\hline
Train & 4712 & 70\% \\
Test & 1355 & 20\% \\
Validation & 704 & 10\%\\
\hline
\end{tabular}
\caption{The number of messages included for training, testing, and validation regions.}
\label{tab:data-splits}
\end{table}


\begin{table*}[h]
\centering
\small{
\begin{tabular}{lccccc}
\hline
\textbf{Category} & \textbf{Coverage (\%)}& \textbf{Model}& \textbf{Precision}& \textbf{Recall} & \textbf{F1} \\
\hline
Communication/Coordination & 73.1 & NLI (1) & $84.0$ ($\pm0.6$) & $87.1$ ($\pm0.5$) & $85.5$ ($\pm0.2$)\\
Rapport Building & 15.2 & NLI (5) & $83.0$ ($\pm9.3$)  & $80.1$ ($\pm15.7$)  & $81.3$ ($\pm12.6$) \\
Control & 20.8 & NLI (5) & $61.5$ ($\pm1.7$) & $59.5$ ($\pm1.4$) & $60.4$ ($\pm0.2$)\\
Challenge & 4.5 & NLI (5) & $36.1$ ($\pm1.7$) & $24.2$ ($\pm4.1$) & $28.9$ ($\pm3.5$)\\
Negotiation & 20.9 & NLI (5) & $62.9$ ($\pm2.9$) & $66.7$ ($\pm1.7$) & $64.7$ ($\pm2.3$)\\
Use of Emotions & 16.4 & NLI (5) & $58.2$ ($\pm1.4$) & $53.1$ ($\pm1.65$) & $55.6$ ($\pm1.6$)\\
Testing Boundaries & 31.2 & NLI (5) & $69.6$ ($\pm0.8$) & $76.1$ ($\pm2.7$) & $72.7$ ($\pm0.9$)\\
Use of Sexual Topics & 18.3 & NLI (5) & $64.5$ ($\pm1.6$) & $69.3$ ($\pm3.6$) & $66.8$ ($\pm2.1$)\\
Mitigation & 3.0 & NLI (3) &   $62.2$ ($\pm10.7$) & $38.4$ ($\pm7.0$) & $47.5$ ($\pm8.5$)\\
Encouragement & 8.0 & NLI (1) & $43.2$ ($\pm4.8$) & $22.0$ ($\pm1.6$) & $29.1$ ($\pm2.0$)\\
Risk Management & 4.6 & NLI (1) & $63.1$ ($\pm4.0$) & $49.5$ ($\pm4.1$) & $55.4$ ($\pm3.5$)\\
\hline
\end{tabular}}
\caption{Eleven offender behavior categories derived and labelled by domain-experts, alongside their prevalence within the corpus. We report performance for the best performing model (based on F1) for each category. NLI (5) denotes the 5-message-input Natural Language Inference model, and similarly for NLI (3) and NLI (1). }
\label{tab:main-results}
\end{table*}


\paragraph{Dataset and Labelling.}
The \citet{pjwebsite} website  is an online repository of real, chat-based conversations between adults who were later convicted of grooming offences and decoys posing as children. Twenty-four chats, comprising of 12,942 messages, were labelled by a domain-expert with a background in forensic psychology. Chats were annotated in concordance with a theory of child grooming known as ``self-regulation'' \citep{elliott2017self} -- the notion that online predation contains a \textit{potentiality phase}, where the predator attempts to form a positive relationship with the victim; and a \textit{disclosure phase}, where the predator becomes more explicitly goal-oriented. 
The following behaviors were identified and applied to the offender messages: (1) communication/coordination, (2) rapport building, (3) control, (4) challenges, (5) negotiation, (6) use of emotions, (7) testing boundaries, (8) use of sexual topics, (9) mitigation, (10) encouragement, (11) risk management. A qualitative description of these behaviors is given in Section~\ref{app-label-desc} of the Appendix. Overall 6,772 messages sent by the offender were labelled, with each message assigned a `Yes' or `No' for each of the above categories, based on the annotator's judgment. Due to the subjective nature of this assessment, labelling performed by different expert annotators may not be in complete agreement. 

We construct binary classification tasks and predict whether each message is an example of each behavior category. Offender messages were split into training, testing, and validation regions (see Table~\ref{tab:data-splits} for information on data splits), and were stratified to ensure equal distribution of behaviors per region. To increase model confidence, we cross validated each experiment three times through random re-sampling. In our results, we report the average ($\pm SD$) scores for precision, recall and F1 as evaluation metrics. 



\paragraph{Supervised Machine Learning.} Offender messages were tokenized, part-of-speech (POS) tagged, and lemmatized with \texttt{spaCy}. We removed stop-words (i.e., the, a, am, at, be, is), and used the frequency counts of the remaining unigrams as input features. Consistent with previous work \citep{miah2011detection, bogdanova2014exploring}, we compared the performance of four classifiers: Random Forest, Logistic Regression, Support Vector Machines, and Naive Bayes. For each algorithm, we optimized hyperparameters\footnote{Hyperparameters and the range of values explored are included in Section~\ref{hyp-app} of the Appendix.} using the GridSearch class in \texttt{Scikit-learn} \citep{pedregosa2011scikit} with 3-fold cross validation. 


\paragraph{Deep Learning.}
To make predictions using transformer-based deep learning, we formulate the problem as a natural language inference (NLI) task. In NLI, the objective is to determine whether two sentences logically complement or contradict one another \citep{bowman2015large}. For our purposes, we compare each message with a standardized sentence crafted from each behavior label (i.e., ``This message is an example of \textit{building rapport}''). Each message/label sentence pair is then used as an input to the deep learning model, where a softmax activation function calculates the probability the two sentences are logically related. For training\footnote{We used a Tesla P100 GPU for training.}, we use \texttt{RoBERTa-large} \citep{liu2019roberta} with an implementation built in Pytorch. This model has been fine-tuned for NLI tasks using the Multi-Genre Natural Language Inference corpus \citep{N18-1101}. We ran both zero-shot and few-shot training setups. For the zero-shot model, predictions were made on the test set without any domain-specific training. In few-shot settings, we experiment with different amounts of positive training examples up to the full training set. As per \citep{wang2021entailment}, models were trained for 10 epochs with batch size 32 and a learning rate of $10^{-5}$.

\paragraph{Multi-Message Input.}
To explore whether the surrounding messages increase the contextual understanding of the transformer, we repeat full-shot experiments but expand the message window to include multiple prior messages sent by both speakers. To do this, we concatenate multiple messages into a single input which is then passed to the transformer. We compare the performance of NLI models with an input of 1, 3, and 5 messages.

\paragraph{Human-in-the-Loop Validation.}

To establish how comparable the model's output is to labelling performed by a human annotator, we scored the predictions made by the best performing model for $\sim10\%$ of the corpus ($645$ predator messages). Each message was scored on a 1-3 scale of agreement (1=disagree, 2=uncertain, 3=agree). Sections of chats were chosen at random. To mitigate potential bias, annotations generated during the initial labelling were shielded from view.

We use Cohen's $\kappa$ \citep{cohen1960coefficient} to measure pairwise agreement between the two human raters and the predictions generated by the best performing model for each behavior. We report the mean of these scores as an index of overall agreement. 





\section{Results}
\label{sec:results}


Transfomer-based deep learning models achieved best performance across all categories, with the 5 message input NLI model performing best for most categories. Evaluation metrics for the highest performing model for each behavior are shown in Table~\ref{tab:main-results}. The performance of all models on all categories is reported in Section~\ref{full-results} of the Appendix. Eight of the eleven behaviors obtain a maximum F1 score above 50\%, with the most prevalent category, \textit{communication/coordination}, performing best with F1=$86\%$. Automatically labelling \textit{encouragement} and \textit{challenge} was least successful, with a maximum F1 score of $29\%$ for both. 
For all eleven categories, a transformer-based model, trained on the full training set, was more successful than the traditional supervised machine learning algorithms. Averaging across behaviors, transformers improved performance of single message classification from $33\%$ to $53\%$ compared to the best performing supervised model. In 8 out of 11 cases, the supervised models, trained on the full training set, outperformed the zero-shot transformer. However, in 10 out of 11 cases, only 50 labelled instances were required by the transformer to exceed the best performing bag-of-words models. As many of the behaviors rely on contextual understanding, we increased the input size from a single message to 3 and 5 messages. The first contains two messages sent by the offender and one by the decoy, and the latter contains three by the offender and two by the decoy. Increasing the window size indeed improved performance for all behaviors\footnote{The majority of messages ($73\%$) were labelled as \textit{communication/coordination}. As a result, this behavior was not tested in a multi-message setting as grouping messages meant that all inputs would be positively labelled.} except for \textit{encouragement} and \textit{risk management}. On average, while 3-message classification did not improve single-message performance, 5-message classification improved on single-message F1 from $50\%$ to $56\%$.


\begin{figure}[h]
\centerline{\includegraphics[scale=0.55]{train_size_change_singlecol.pdf}}
\caption{Model performance (F1) on the eleven behavioral categories for single message classification, as the number of positive training examples increases (up to full training set). Error bars are standard  deviations.}
\label{fewshot-perf-change}
\end{figure}

Figure~\ref{fewshot-perf-change} shows the change in F1 for single message classification as the number of training samples increases. Whilst predictions for all behaviors improved with additional training, large variation in performance between the categories remains even for a fixed amount of positive training examples. 
To better understand this variation, we manually examined the predictions made by the best performing single message model for $\sim 10\%$ of the corpus. We observed that the transformer has learned some rules that caused a number of false positives. For example, the \textit{testing boundaries} model correctly predicted a number of questions aimed at assessing the victim's willingness to engage, e.g., "what are you looking for?", but also included several logistical questions, e.g., "how about i come by at 8pm?". The \textit{rapport building} model correctly recognised complements and sweet talk as positive examples, but missed more everyday examples of rapport building, e.g., "how was your spring break?". Some aspects of \textit{control} required coverage over longer ranges than we have focused on. For example, persistently asking the same question is a sub-category of \textit{control} that was often misclassified. In this case, increasing the the message window to 3 messages was not enough to improve performance, however, the larger 5 message window did notably improve performance, with the F1 increasing from $54\%$ to $60\%$ due to better recall. In trying to predict \textit{encouragement}, the worst performing category, the transformer appeared to overfit on short verbal nods (i.e., "kool" and "sure", which appeared in the corpus either as evidence of encouragement or simple linguistic fillers) which increased the false positive rate. Over-reliance on this simple rule would explain why the model did not utilise the additional information provided in the multi-message setting. \textit{Risk management} achieved reasonable performance (F1=$55\%$) despite being one of the rarest behaviors in the corpus. This was largely a consequence of the transformer recognising instances where the offender attempts to establish the presence of a parent, e.g., ``when are they getting home?''.


In addition to drawing qualitative observations on the ability of the transformer-based model to recognise different offender behaviors, we quantify how comparable the best performing single-message model is to a human annotator. We compare the labelled classes between the original labels (Rater 1), with the predictions made by the best performing models (Model), and the labels verified during post-validation (Rater 2). We report a mean $\kappa$ score of 0.68, indicating `substantial agreement' between raters \citep{mchugh2012interrater}. Pairwise agreement is reported in Table~\ref{tab:post-val}. Agreement was highest between the two human raters ($\kappa=0.8$). The average agreement between each rater and the predictions generated by the model was $\kappa=0.62$ (see Section~\ref{app:agree} in the Appendix for pairwise agreement per behavior). 


\begin{table}[h]
\centering
\begin{tabular}{lccc}
\hline
 & \textbf{Rater 1} & \textbf{Rater 2} & \textbf{Model}\\
\hline
\textbf{Rater 1} & - & 0.8 (0.78) & 0.59 \\
\textbf{Rater 2} &  & - & 0.65 (0.66)\\
\hline
\end{tabular}
\caption{Pairwise agreement (Cohen's $\kappa$) between raters over all behaviors. A score of 1.0 indicates perfect agreement. Value in parentheses indicate Rater 2 agreement when `uncertain' validation ratings are included.}
\label{tab:post-val}
\end{table}


\section{Conclusion}
\label{sec:concl}

Manually labeling the 24 chat logs used in this work took over 600 hours. Given that the full Perverted-Justice corpus contains 850 chat logs, it would be infeasible to label the entire corpus without the help of automated methods. We find that a transformer-based deep learning approach yields promising results when applied to the detection of online predatory behavior. However, even with training, the agreement between the model and a human annotator is not comparable to the agreement between two human annotators. The model's success in predicting behaviors varies significantly between the categories. Performance is better for more common behaviors, however, we show that the variation is not only due to the number of positive examples in the training data. Extending the prediction task to include multiple messages, increasing the contextual window, boosted detection for certain behaviors. The F1 score for \textit{rapport building}, for example, increased from $61\%$ for single messages to $81\%$ when classification was based on five-messages. Performing post-validation on the automatic classifications allowed us to gain qualitative insight about the model's performance, which may be used to design better prompts and improve performance further. Overall, our results are an encouraging step towards building an automated, psychology-informed model to detect online sexual exploitation.






