\section{Introduction}
\todo[inline]{Add example tweets}
Social media platforms have become key portals for sharing information during crisis situations~\cite{DBLP:journals/pacmhci/ZadeSRK0S18}. 
	A major difference between social media and traditional sources is the possibility of receiving feedback from affected people in real-time. 
	Relief organizations such as the \emph{Red Cross} can benefit from this two-way communication channel to inform people and gain insights from situational updates received from actors affected by the crisis.  
	Hence, extracting crisis information from social media can be of great help to leverage situational awareness and deliver faster responses.

	Previous works~\cite{to2017identifying,stowe2018improving} have focused on

	binary classification of social media text into two categories (e.g., relevant or irrelevant).
	A fine-grained classification, however, would improve the automation of social media data processing.
	For example, by providing the classified tweets to the matching experts.
	The study by Olteanu et.al~\cite{olteanu2015expect} was one of the firsts to investigate tweets generated during different crisis events. 
	The authors concluded that crisis-related tweets tend to contain informative and crucial information types\footnote{In the context of this paper. We use the terms labels, classes and information types synonymously.}, such as affected individuals, infrastructures and utilities, donations and volunteers, caution and advice, sympathy, and emotional support. 
	This level of categorization is valuable for crisis responders to filter critical information quickly and consequently deliver disaster responses.  Table~\ref{tab:informationTypes} shows that disaster-related tweets can be categorized into multi-level information types, which allows the extraction of \textit{actionable} information.
	We consider the same definition of '\textit{actionable}' as formalized in~\cite{DBLP:journals/pacmhci/ZadeSRK0S18}. \textit{Actionable} posts are defined as the ones that would generate an immediate alert for an individual (i.e. stakeholders) responsible for that information type

	(e.g., \textit{SearchAndRescue}, \textit{MovePeople}), in contrast to the \textit{non-actionable} tweets that
	belong to the classes like \textit{Volunteer} or \textit{FirstPartyObservation}.






	
	Short text (e.g., tweets) categorization is known to be a challenging Natural Language Processing (NLP) task~\cite{song2014short}.  


	This is because short text 1) does not possess sufficient word contextual information; 2) is inherently noisy (i.e. contains misspellings, acronyms, emojis, etc.), hence cleaning the text before further processing is essential to capture better features and semantics.

	In the case of Twitter data, the classification task becomes even more challenging because a tweet could belong to multiple information categories simultaneously, thereby making it a multi-label classification problem. 


	To tackle these challenges, previous studies have made significant use of recent results from machine learning and NLP, such as transfer learning from pre-trained models (e.g., ELMo, BERT)~\cite{sun2019fine}.
	While pre-trained models are first trained on a large amount of generic data to capture rich representations and can be fine-tuned into downstream tasks,
	they are not always able to capture the structural information and relationship between the sequence of words in a text. 
	

	In this paper, we aim to identify \textit{actionable or critical} information that might be relevant for disaster relief
	from disaster-related tweets and support disaster mitigation.

	Our approach uses BERT's contextualized sentence-level embeddings to capture the semantics
	of a tweet.
	Furthermore, we employ a Graph Convolutional Network (GCN) to apprehend correlations between entities in tweets and their corresponding labels. 
	In this way, our approach allows each label to derive information from the correlated entities and other labels that were assigned to the same tweet. 

	Therefore, our system integrates the contextualized representation of tweets with the semantic correlations between \textit{named-entities} and labels.
	Finally, the last component in our approach (Relation Network) employs a neural model as a learnable distance metric to compute the similarity between the BERT-vectorized tweets and the label vectors, which are obtained using the Text-GCN approach.

	The main contributions of this paper can be summarized as follows:
	\begin{itemize}
		\item We present a deep neural framework (dubbed I-AID) to identify crisis information from disaster-related tweets. 
		\item Our approach leverages transfer learning from pre-trained language models to embed tweets into low-dimensional vectors. Then, it uses a GCN to capture correlations between entities and labels in a graph-structural manner. 
		\item We employ a \textit{learnable} distance metric to learn the similarity between a tweet's vector and the labels' vectors in a data-driven way. 
		\item
		Our experimental results suggest that our method achieves superior performance over different baseline models (see Section~\ref{sec:baselines}) with up to $+20\%$ in macro-averaged F1-score. 
	\end{itemize}
	
All our implementations are open-source and available at the project website.\footnote{Link is removed for anonymous submission} 

	
\begin{table*}[h!]
	\centering 
	\small
	\caption{Crisis Information Types\label{tab:informationTypes}}
	\begin{tabularx}{0.80\textwidth}{@{}p{2.5cm}p{3.5cm}p{7cm}@{}}
		\toprule
		\textbf{Intent Type }& \textbf{Information Type} & \multicolumn{1}{c}{\textbf{Description}} \\ \midrule
		
		\multirow{3}{*} {REQUEST}& GoodServices & Request for a particular service or physical good \\
		& SearchAndRescue & Request a rescue \\
		& InformationWanted & Request information \\ \midrule
		\multirow{11}{*}{REPORT} & Weather & Weather report \\
		& FirstPartyObservation & Eye-witness account \\
		& ThirdPartyObservation & Information from someone else\\
		& EmergingThreats & Problem that cause loss or damage \\
		& ServiceAvailable & Someone is providing a service \\
		& SignificantEventChange & New occurrence that officers to respond to\\
		& MultimediaShare & Shared images or video \\
		& Factoid & Relate some facts, typically numerical \\
		& Official & Report by a government or public representative \\
		& CleanUp & Report of the clean up after the event \\
		& Hashtags & Report with hashtags correspond to each event \\ \midrule
		\multirow{3}{*}{CALLTOACTION} & Volunteer & Call for volunteers to help in response efforts \\
		& Donations & Call for donation of goods or money \\
		& MovePeople & Call to leave an area or go to another area \\ \midrule
	\end{tabularx}
\end{table*}

	

\section{Related Work}\label{sec:relatedworks}
The objective of this work is to categorize disaster-related tweets into multiple information types. Therefore, our work is related to \textit{Extracting Disaster-related Information from Social Media and Short-text, Multi-label Categorization}.
	
	\subsection{Extracting Disaster-related Information from Social Media}
	Several research studies have demonstrated the role of social media as a primary source of information during disasters~\cite{cresci2015crisis}. 

	While some works~\cite{PalshikarAP18} have focused on distinguishing disaster-related information from tweets,
	others~\cite{olteanu2015expect,mccreadie2019trec} have introduced annotation schemes to classify disaster-related tweets into more fine-grained categories to account for the attitude, information source, and decision-making behavior of people tweeting before, during, and after disasters.
	In order to advance the state of social media crisis monitoring solutions, several initiatives~\cite{imran2016lrec}
	have been rolled out in recent years. One of them is the Incident Streams (TREC-IS) track~\cite{mccreadie2019trec} of the Text REtrieval Conference, which commenced in 2018.
	The track aims to categorize disaster-related tweets into multiple information types.  Since in this work we study the TREC-IS datasets, we consider \textit{actionable} information to be the same as that defined by the authors in the challenge. 
	
	\subsection{Multi-label Text Classification}
	\todo[inline]{Shorten}
	Several approaches have been proposed to tackle the problem of short-text classification.
	On a high level, to represent text, these approaches either 
	use: 1) explicit, hand-engineered features~\cite{sriram2010short}.
	or 2) neural networks to model the dependencies within the text (implicit features)~\cite{wang2017combining}. 
	With the end-to-end deep learning approaches gaining traction and with the introduction of attention mechanisms~\cite{vaswani2017attention}, there has been a significant advancement in text-classification tasks. Recent works~\cite{miyazaki2019label} have also focused on alleviating the tedious process of finding good text-based features and for tackling the multi-label classification problem. Further, pre-trained contextualized word-embeddings, particularly BERT, have proven to be revolutionary for text representation and classification. 
	Previous studies~\cite{miyazaki2019label} have leveraged this multi-layer, multi-head self-attention mechanism of BERT, however, since it only captures the local contextual information, it falls short. To circumvent this and  apprehend the global relations among words in a vocabulary, graph-based approaches (Graph Neural Networks
	~\cite{kipf2016semi,yao2019graph} have been found promising. 
	
	Recent studies~\cite{DBLP:conf/ecir/LuDN20,DBLP:conf/icaart/PalSS20} have exploited the advantages of BERT and GCN, while the combination of them
	has previously been applied to the multi-label classification task~\cite{DBLP:conf/icaart/PalSS20}, to the best of our knowledge, it has not been tailored to
	multi-label, short-text (tweet) classification. 

	In VGCN-BERT~\cite{DBLP:conf/ecir/LuDN20}, a GCN is used to capture the correlation between words at the vocabulary level (global information). That is, given a vocabulary, GCN
	would relate the meaning of 'new' to 'innovation' and 'exciting', similar to the context-independent word embeddings like Word2Vec~\cite{DBLP:journals/corr/abs-1301-3781}. 
	For an input sentence, the (local) contextual information is captured using the BERT embeddings, while the global information pertaining to the words in the sentence is extracted from the graph embeddings and concatenated with the sentential BERT embeddings. 
	The two representations of BERT and GCN then interact via the self-attention mechanism to perform a classification task. 
	In contrast, in our approach, we employ GCNs to explicitly infuse the correlation-information of the entities in a tweet and their corresponding labels with the tweet's contextualized BERT-representation. 

	In addition, our approach differs from both MAGNET~\cite{DBLP:conf/icaart/PalSS20} and VGCN-BERT~\cite{DBLP:conf/ecir/LuDN20} in the way it computes the similarity between a tweet's representation and the class labels.
	While the above-mentioned approaches use either a fixed, linear distance metric (dot-product function) or self-attention to measure the similarities, our approach benefits from a deeper end-to-end neural architecture to learn this distance function. 
	In particular, we employ meta-learning to learn
	the mapping between the input features and multi-label output in a data-driven way.
	In the subsequent sections, we will further elaborate on our approach. 

	

	
	\subsection{Meta Learning}

	Meta learning (a.k.a. \textit{learning-to-learn} paradigm) refers to the process of improving a learning algorithm over multiple learning episodes. In contrast to the conventional ML approaches, which improve model predictions over multiple data instances, the meta-learning framework treats tasks as training examples to solve a new task  ~\cite{DBLP:journals/corr/abs-2004-05439}.
	In our study, we employ a specific branch of meta learning called metric learning.  Metric-learning learns a distance function between data samples so that the test instances get classified by comparing them to the \texttt{K} labeled examples. The "distance function" consists of 1) an embedding function, which encodes all instances into a vector space and 2) a similarity metric, such as cosine similarity or Euclidean distance, to calculate how close two instances are in the space~\cite{DBLP:journals/corr/abs-2007-09604}. Over the years, the following approaches have been developed to perform this task: Siamese~\cite{koch2015siamese}, Matching~\cite{vinyals2016matching}, Prototypical~\cite{snell2017prototypical}, and Relation Network~\cite{sung2018learning}.
	While the embedding function in all of these approaches is a deep neural net, they differ in terms of the similarity function. Unlike its predecessors, which rely on a fixed similarity metric (such as cosine, euclidean etc.),  Relation Network employs a flexible function approximator to learn similarity and focuses on learning a good similarity metric in a data driven way. Thus, this eliminates the need to manually choose the right metric (Euclidean, cosine, Mahalanobis). By jointly deep learning the embedding with a non-linear similarity metric jointly, Relation Network can better identify matching/mismatching pairs~\cite{garcia2018fewshot}. Hence, in our study, we use the Relation Network for learning the similarity metric. 

\section{The Approach}\label{sec:approach}
\todo[inline]{add updates of textGAT}
We begin this section by giving a formal specification of the problem of tweets classification. Then, we discuss the details of each component in our approach in Section~\ref{sec:I-AID Architecture}. Figure~\ref{fig:I-AID} illustrates an overview of our approach and how components work together.
	
	\subsection{Problem Formulation}\label{sec:problem_formulation}
	Let $T$ be a set of tweets and $\Lambda=\{\lambda_1, \lambda_2,\cdots ,\lambda_k\}$ be a set of predefined labels (also called information types, see Table~\ref{tab:informationTypes}). 
	We formulate the problem of \textit{identifying crisis information from tweets} as a multi-label classification task, where a tweet $t$ can be assigned one or more labels from $\Lambda$ simultaneously. 
	Our task is to learn a function $\mathcal{H}: T \to \{0,1\}^k$ that maps tweets $T$ to relevant labels from $\Lambda$. We assume a supervised learning setting, where 
	a training data $\mathcal{D}=\big\{ (t^{(i)},y^{(i)})\times \{0,1\}^k \big\}_{i=1}^N$ consists of $N$ tweets. Hence, each tweet $t^{(i)}$ is labelled with a set of corresponding labels $y^{(i)}$, where $y^{(i)}_j = 1$ means that $t^{(i)}$ belongs to the class $\lambda_j$. Conversely, $y^{(i)}_j = 0$ means that $t^{(i)}$ does not belong to the class $\lambda_j$. Our approach learns the function $\mathcal{H}$ by using three neural networks. It first transforms tweets $t^{(i)}$ into a d-dimensional vector $x^{(i)}$ using a pretrained BERT model. In parallel, it learns label embeddings $l$ using a graph convolution network. These are then concatenated with the tweet embedding $x^{(i)}$. Finally, these vectors are fed to our last component (Relation Network) to identify relevant labels for $t^{(i)}$. 
	
	\subsection{The I-AID Architecture}\label{sec:I-AID Architecture}
	

	\begin{figure*}[t!]
		\centering
		\captionsetup{justification=centering}
		\includegraphics[width=0.8\linewidth]{Figures/I-AID.pdf}
		\caption{Illustration of our proposed I-AID System.}
		\label{fig:I-AID}
	\end{figure*}
	
	\subsubsection{BERT-Encoder.}
	This is the first component in our system which transforms an input tweet into a vector representation ($Z$) of it's contextual meaning. As shown in Figure~\ref{fig:I-AID}, the BERT-Encoder takes tweet $t^{(i)}$ with $m$ tokens $[w^{(i)}_1,w^{(i)}_2,\dots ,w^{(i)}_m]$ and outputs vector $Z$. We employ a BERT-base architecture with $12$ encoder blocks, $768$ hidden dimensions, and $12$ attention heads. We refer the readers to the original BERT paper~\cite{devlin2018bert} for a detailed description of its architecture and input representation. 
	Furthermore, a special preprocessing is performed for BERT input. A \texttt{[CLS]} token is appended to the beginning of the tweet and another token \texttt{[SEP]} is inserted after each sentence as an indicator of sentence boundary. Each token $w_i$ is assigned three kinds of embeddings (token, segmentation, and position). These three embeddings are summed to a single output vector \texttt{[CLS]} that captures the meaning of an input tweet.




	
	\subsubsection{Text-Graph Neural Network (TextGAT)}
Traditional methods (e.g., word2vec) can properly capture features from a text. However, these methods ignore the structural information and relationship between words in a text corpus~\cite{peng2018large}.
 The recently proposed graph networks~\cite{yao2019graph} aim to tackle this challenge by modeling text as a graph where words are nodes and relations between them (e.g, words co-occurrence or collocation) are edges. In our work, we construct a heterogeneous graph network $G = (V, A)$ where nodes $V$ are words, \textit{named-entities} and labels in the dataset $\mathcal{D}$. We spot \textit{named-entities} in tweets $t^{(i)} \in \mathcal{D}$ using spaCy\footnote{https://spacy.io/api/entityrecognizer}. Furthermore, relations between nodes are modeled using an adjacency matrix $A$.
	
Each node $v_i \in V$ is represented as a low-dimensional vector. We initialize nodes vectors with a pre-trained word2vec model (Glove embedding). 
As shown in Figure~\ref{fig:I-AID}, TextGAT component has two graph attention layers (GAT). Each GAT layer takes nodes features $F\in \mathbb{R}^d$ as an input and computes new nodes features $\hat{F}$ w.r.t the importance of neighbors (i.e., attention score) for a given node $v_i$.

	\begin{equation}
	    a_{ij}=attention~(\textbf{W}h_i,\textbf{W}h_j)
	\end{equation}
	
	The attention scores are then normalized using a softmax function:
	
	\begin{equation}
	\alpha_{ij}=\frac{exp(a_{ij})}{\sum_{k\in N_i}exp(a_{ik})}
	\end{equation}
	
	In order to obtain sufficient representation ability to transform the input features into higher level semantics, the linear transformation is learned. 
	To that end, as an initial step, a shared linear transformation, parametrized by a weight matrix $\textbf{W}$. 
	We then perform self-attention on the nodes, and a shared attention mechanism \textit{attention} is adopted to compute attention coefficients.
	
	
	
	In this way, TextGAT combines structural information of a node's similarity relations (i.e., entity-entity, entity-label) and semantic information of a word vector representation.
	
	\subsubsection{Relation Network.}\label{relation-network} Finally, our last component aims to learn a similarity metric in a data-driven way. 

	To do so, we employ a neural model as a learnable, non-linear distance function that \textit{learns} how to match similarity (i.e., relation) between the tweet's vector and each label during the training phase.  
	By using a learnable distance function in our Relation Network, we intend to learn an efficient metric to facilitate the mapping between a tweet and the multiple labels that could be assigned to it. Thus, by employing the Relation Network, we eliminate the need to manually choose the right distance metric (e.g. cosine, euclidean, or manhattan).

	Relation Network takes the concatenated vector of BERT-Encoder output with the label vectors as an input. 

	
	For multi-label classification purposes, we use binary cross-entropy, as given in Equation~\ref{eq:binary_cross_entropy}, as a loss function.
	Since our goal is to fetch all the relevant labels, we use sigmoid activation, which computes the probability of each label independently over all possible labels ($\Lambda$), instead of the softmax function that only considers the label with maximum probability.


	\begin{equation}
	\label{eq:binary_cross_entropy}
	\arg\min\frac{1}{k}\sum_{i=1}^{k}\left[y^{(i)}\log(\hat{y}^{(i)})+(1-y^{(i)})\log(1-\hat{y}^{(i})\right]
	\end{equation}
	
	Where $y^{(i)}$ and $\hat{y}^{(i)}$ are the predicted and ground-truth labels of tweet $i$ respectively $k$ is the number of all possible labels.  

\section{Experiments}\label{sec:experiment}
In our evaluation, we address the following research questions: 
	\begin{itemize}
	    \item[-] \textbf{RQ1: }
	    How does our approach perform (\textit{compare with state-of-the-art multi-label models}) on short-text classification (e.g., tweets)?
	   
	    \item[-] \textbf{RQ2: } How effective is our approach in identifying tweets with \textit{actionable} information?
	    \item[-] \textbf{RQ3: } How do each component in our approach affect the performance? (i.e., Ablation Study)
	\end{itemize}
		
	
\begin{table}[ht]
\centering
\captionsetup{justification=centering}
\caption{Overview of the Datasets.}
\begin{tabular}[t]{lcccc}
\toprule
\textbf{Datasets} & \textbf{\# Train} & \textbf{\# Valid} & \textbf{\# Test} & \textbf{\# Classes}\\
\midrule
TREC-IS &  27,467 & 6,867 &  8,584 & 25      \\
COVID19-Tweets &  4,844 & 1,211 & 1,514  & 12   \\
\bottomrule
\end{tabular}
\label{tab:datasets}
\end{table}%
	
	
\subsection{Datasets}
We conduct a set of experiments on two public datasets. 
The datasets are collections of disaster-related tweets provided by TREC~\cite{mccreadie2019trec}. Table~\ref{tab:datasets} gives an overview about dataset statistics: number of tweets used to train, validate, and test the models, in addition to total number of classes.  
In particular, we split each dataset with a ratio ($80\%-20\%$); where we use $80\%$ of tweets to train the models and $20\%$ for test. During training, we use $20\%$ of the training data to validate the system. We briefly summarize each dataset as follows:

\begin{itemize}
    \item \textbf{TREC-IS: }
    This dataset contains approximately $35$K tweets collected during $33$ unique disasters between $2012-2019$ (e.g., \textit{wildfires, earthquakes, floods, hurricanes, bombings, and shootings}). 
	The tweets are labeled into $25$ information types by human experts and volunteers. 
	
	\item \textbf{COVID19-Tweets: } This dataset contains a collection of tweets about the COVID-19 outbreak in different affected regions. In total, the data has $7,590$ tweets labeled with one or more of the full $12$ information type labels (the same as for TREC-IS dataset). 
\end{itemize}

Figure~\ref{fig:tweets-informationTypes} shows the distribution of tweets per information type in both datasets. Apparently, the datasets are highly imbalanced w.r.t information types distribution across tweets. For example in TREC-IS dataset, there are more than 4,000 tweets are mapped to the information types of \textit{Sentiment}, \textit{News}, and \textit{ThirdPartyObservation}. In contrast, the information types \textit{CleanUP, InformationWanted}, and \textit{MovePeople} have significantly fewer tweets. In Covid19-Tweets, the tweets distribution is extremely imbalanced, most of tweets are categorized into \textit{Irrelevant} or \textit{ContextualInformation, Advice, News}. This skewing distribution in information types across tweets made the task of classification more challenging. 

\begin{figure*}[t!]
    \centering
    \begin{subfigure}[t]{0.5\textwidth}
        \centering
        \small
        \captionsetup{justification=centering}
        \includegraphics[width=\columnwidth]{Figures/trecis-labels.pdf}
        \caption{TREC-IS Dataset}
    \end{subfigure}%
    ~ 
    \begin{subfigure}[t]{0.5\textwidth}
        \centering
        \small
        \captionsetup{justification=centering}

        \includegraphics[width=\columnwidth]{Figures/covidtweets-labels.pdf}
        \caption{Covid19 Tweets Dataset}
    \end{subfigure}
    \caption{Tweets Distribution across Information Types}
    \label{fig:tweets-informationTypes}
\end{figure*}





	\subsection{Baselines} \label{sec:baselines}
	We consider a set of state-of-the-art models in \textit{multi-label classification}\footnote{\scriptsize Note that, we consider relevant baselines that were implemented for multi-label classification. The current version of VGCN-BERT can not be directly used for multi-label classification, therefore we didn't include it in the baselines.}, as baselines in our evaluation. We briefly describe each baseline as follows:

	
\begin{itemize}
		\item \textbf{TextCNN}~\cite{kim2014convolutional}: This approach uses a convolutional neural network to construct text representation. 
	
	
		\item \textbf{HAN}~\cite{yang2016hierarchical}: This model uses a hierarchical attention-neural-network to encode text with word-level attention on each sentence.
		
		\item \textbf{BiLSTM}~\cite{zhou2016attention}: A bi-directional LSTM model that parses the text from left-to-right and right-to-left, then uses the final hidden state as a feature representation for the whole text.
		\item \textbf{BERT}~\cite{devlin2018bert}: This a BERT-base model with a transformer architecture (multi-layer, multi-head and self-attention) that represent tweet as a contextualized embedding vector. 
		\item \textbf{TextGAT}~\cite{velivckovic2017graph}: This model uses graph attention network (GAT) to represent corpus as a heterogeneous graph, then learn word and tweet embeddings with jointly. 
	
	
		\item \textbf{MAGNET}~\cite{DBLP:conf/icaart/PalSS20}: This approach employs BERT embeddings to represent tweets and GCN for labels classifiers. Then, it employs to use a dot-product function to compute similarities between tweet vector and label vectors.
		
	\end{itemize}
	
\subsection{Implementation and Preprocessing}\label{sec:implementation}
We use the open-source implementations for the models (TextCNN, HAN, BiLSTM, and TextGAT) provided by the corresponding authors on GitHub. For BERT model, we use the implementation from Huggingface\footnote{\url{https://huggingface.co/}} library. Moreover, we implemented the code for MAGNET model as it has not been open-sourced to date. Hyper-parameters are set in the baseline methods same as  mentioned in their original papers. In our model, we tune the hyper-parameters via grid search method to find optimal values for best performances. In particular, our model achieves its best performance with the following values: training-epochs to $200$ with batch-size of $128$ and Adam optimizer~\cite{kingma2014adam} with a learning-rate of $2\mathrm{e}^{-5}$. To avoid overfitting, We add a dropout layer with a rate of $0.25$ and apply an early-stopping technique during model's training. Our implementation of I-AID model is open-sourced and available on the project website\footnote{Links are removed due to anonymous submission}.

\paragraph{Data Preprocessing:} Given that datasets are disaster-related tweets, we perform proper preprocessing steps. In particular, we perform the following preprocessing steps: (1) We use the NLTK's TweetTokenize API to tokenize tweets and retain the text content. (2) Stop-words, URLs, usernames, and Unicode-characters were removed. (3) Extra white-spaces, repeated full stops, question marks, and exclamation marks were removed. (4) Emojis\footnote{\url{https://pypi.org/project/emoji/}} were converted to text using the emoji python library. (4) Finally, spaCy library is used \footnote{\url{https://github.com/explosion/spaCy}}, to extract \textit{named-entities} from tweets.

	
\subsection{Evaluation Metrics}\label{section:evaluation_metrics}

	We employ appropriate evaluation metrics for multi-label classification scenario. Particularly, We use a weighted-averaged F1-score, hamming loss, and jaccard score to evaluate the system's performance in multi-label classification tasks: 

	\begin{itemize}
	
	 \item{\textbf{Weighted-average F1-score}}:
	 F1-score is the harmonic mean of precision and recall scores. We use a \textit{weighted-averaged} that calculates F1 score for each label independently, then adds them together uses a weight that depends on the number of true labels of each class.
	 \begin{equation}
	     F1= 2 \times \frac{precision\times recall}{precision+recall}
	 \end{equation}
	 
	\item \textbf{Hamming Loss.}: To estimate the error rate in classification, we use the \textit{hamming loss} function~\cite{schapire1999improved} 
	that estimates the fraction of incorrectly predicted labels out of all predicted labels. Hence, the smaller the value, the better the performance.  
	
	\begin{equation}
	{hamming\:Loss\:}(y^{(i)}, \hat{y}^{(i)})=\frac{1}{N}\sum_{i=1}^{N}\frac{1}{k} |y^{(i)}\oplus\hat{y}^{(i)}|
	\end{equation}
 	where $N$ is the dataset size, $k$ is the total number of labels (i.e. $|\Lambda|$ ), $\oplus$ denotes the XOR operator and $y^{(i)}$ and $\hat{y}^{(i)}$ are the ground-truth and predict labels of tweet $i$ respectively.
 	
	 \item \textbf{Jaccard Score.} To assess the system's accuracy, we use the Jaccard score to compute the similarity between predicted labels $\hat{y}^{(i)}$ and ground truth labels $y^{(i)}$. 
	 Jaccard score computes the percentage of common labels in two sets out of total labels as
	\begin{equation}
	{jaccard\:}(y^{(i)}, \hat{y}^{(i)})=\frac{|y^{(i)} \cap \hat{y}^{(i)}|}{ |y^{(i)} \cup \hat{y}^{(i)}|}  
	\end{equation}
	Where $y_i$ and $\hat{y}_i$ are the ground-truth and predict labels sets of tweet $i$. $\cap$ and $\cup$ denote intersection and union set operations respectively.
	\end{itemize}
	
	\subsubsection{Evaluating Actionable Information: Accumulated Alert Worth}
	Now, we aim to evaluate the efficacy of our system in identifying tweets with actionable information, i.e. system should trigger an alert if an input tweet includes critical information (e.g., requests for search and rescue or reports of emerging threats). Recently, TREC-IS~\cite{mccreadie2020incident} introduce an ad-hoc metric called \emph{AAW} to evaluate systems performance in detecting actionable/critical information in tweets. The AAW score ranges from $-1$ to $1$, where positive value indicates high critical information in a tweet and negative score for less critical. We refer the readers to find more details about AAW metric in the original paper~\cite{mccreadie2019trec}. Here, we summarize AAW metric as follows: 
	
	\begin{equation}
	    hPW(t)=
	    \begin{cases}
	\alpha+((1-\alpha)\cdot (\beta(t)+\hat{\beta}(t)) & \text{if } p_t^s>=0.7\\
	-1 &  \text{otherwise} 
	\end{cases}   
	\end{equation}
	
	
	\begin{equation}
	lPW(t)=    
	\begin{cases}
	argmax(-log(\frac{\delta}{2}+1), -1) & \text{if } p_t^s>=0.7\\
	\beta(t)+\hat{\beta}(t) &  \text{otherwise} 
	\end{cases}   
	\end{equation}
	
	\begin{equation}
	\label{eq:AAW}
	AAW=
	\frac{1}{2}\sum_{t\in \mathcal{T}}
	\begin{cases}
	\frac{1}{|\mathcal{T}_{h}|} \cdot hPW(t) & \text{if } t \in \mathcal{T}_{h}\\
	\frac{1}{|\mathcal{T}_{l}|} \cdot lPW(t) &  \text{otherwise} 
	\end{cases}   
	\end{equation}
	where $\mathcal{T}_{h}$ and $\mathcal{T}_{l}$ denote sets of tweets with high and low priorities respectively. 
	hPW(t) is a scoring function for tweets that should generate alerts and
	lPW(t) is a scoring function for tweets that should not generate alerts.
	

\subsection{Discussion}
\subsubsection{Performance Comparison (RQ1)}

		\begin{table*}[t!]
		\centering
		\caption{Evaluation results of our approach and baselines on two datasets (TREC-IS and COVID19-Tweets) under weighted-average (Precision, Recall and F1), Hamming Loss and Jaccard score. Best results are in bold.}
		\begin{tabularx}{0.85\linewidth}{@{}p{2.2cm}p{2.1cm}p{1.5cm}p{1cm}p{1cm}p{1cm}p{1.5cm}p{1.1cm}XXXXXX@{}}
			\toprule
			\multirow{2}[2]{*}{\textbf{Datasets}} & \multirow{2}[2]{*}{\textbf{Metrics}}& \multicolumn{6}{c}{\textbf{Baselines}} & \multirow{2}[2]{*}{\textbf{I-AID}} \\ 
			\cmidrule(lr){3-8}
			{\textbf{}}&  & TextCNN & HAN & BiLSTM & BERT & TextGAT & MAGNET\\
			\cmidrule(r){1-1} \cmidrule(lr){2-2} \cmidrule(lr){3-8} \cmidrule(l){9-9}
			\multirow{5}{*}{TREC-IS} & 
			Precision & 0.29 & 0.31 & 0.27 & 0.42  & 0.18  & 0.50 & \textbf{0.57}\\
			& Recall & 0.55 & 0.55 & 0.65 & \textbf{0.67} & 0.58 & 0.60 & 0.62 \\
			& F1-score & 0.25  & 0.37 & 0.31 & 0.50  &  0.26 & 0.53 & \textbf{0.59}  \\
		
			& Jaccard score & 0.18 & 0.28 & 0.19 & 0.34 &  0.18 &  0.38 & \textbf{0.43} \\
			& Hamming Loss & 0.24 & 0.15 & 0.26 & 0.11 &  0.24 &  0.09 & \textbf{0.07} \\
			
			\cmidrule(r){1-1} \cmidrule(lr){2-2} \cmidrule(lr){3-8} \cmidrule(l){9-9}
			\multirow{5}{*}{COVID19-Tweets} & 
			Precision & 0.39 & 0.33 & 0.35 & 0.37 & 0.29 & 0.40 & \textbf{0.45}  \\
			& Recall & 0.74 & 0.60 & 0.72 & 0.71  & 0.62 & \textbf{0.76} & 0.71  \\
			& F1-Score & 0.47  &  0.40 & 0.43  &  0.47 &  0.36 & 0.51 & \textbf{0.55}   \\
		
			& Jaccard Score &  0.33 & 0.28 & 0.21 & 0.37 & 0.18 & 0.40 & \textbf{0.43} \\
			& Hamming Loss &  0.11 & \textbf{0.04} & 0.07 & 0.10 & 0.17 &  0.12 & 0.08 \\
			\bottomrule
		\end{tabularx}
		\label{tab:results}
	\end{table*}
	
			
	
	We evaluate the performance of our approach and baselines methods 
	using different metrics for multi-label classification task. To ensure a fair evaluation, we use the same \textit{train} dataset during models training and \textit{test} dataset for evaluation. Table~\ref{tab:results} reports our evaluation results for each model on two datasets (TREC-IS and Covid19-Tweets). We consider Weighted-averaged F1-score as the primary metric to compare and rank systems performances. Weighted-average F1 takes into account the average performance of each system across all information types. 
	Across all the datasets, our approach (I-AID) achieved significant performance improvements over the other baselines under several metrics. In particular, our approach outperforms the weighted-averaged F1-score of MAGNET, the state-of-the-art baseline of multi-label classification, by $+6\%$ on TREC-IS and $+4\%$ on Covid19-Tweets. On the other hand, we evaluate the performance of our approach per each information types individually.
	
	Comparing the results from all the metrics under different settings (i.e., with different datasets), while our model achieved sub-optimal results under precision and recall metrics, it mostly performed better than the baselines. To this end, our results indicate that I-AID performs fairly well when categorizing disaster-related tweets into multiple information types, this is because of three facts:  
	(1) We constructed a multi-model framework that leverages contextualized embedding from the BERT model to capture contextual information within a tweet. 
	(2) Our approach enriches semantics of the tweet representations by injecting label information and integrating additional structural information between entities in tweets and labels (i.e., entity-entity and entity-label) via GCN.
	(3) Finally, we avoid the manual selection of similarity metric by automatically learning the most efficient distance function in our \emph{Relation Network} (as discussed in Section~\ref{relation-network}) in a data-driven way.
	
    
\subsubsection{Actionable Information In Tweets (RQ2)}
Asking ourselves $Q_2$ to understand the behavior of our approach in identifying tweets with critical/actionable information, we use an ad-hoc metric proposed by TREC to evaluate system's ability in this behavior. There are two ways to define an \textit{actionable} tweet: (1) In terms of high priority information, commonly marked as critical by human assessors. (2) In terms of information type, for instance, a tweet with the labels \textit{MovePeople} or \textit{CleanUP} is considered more actionable than \textit{News} or \textit{Multimediashare}.
In table~\ref{tab:AAW}, we report the evaluation results of our approach and all the relevant approaches under AAW metric (Equation~\ref{eq:AAW}) on the test dataset from TREC-IS (edition 2019\cite{mccreadie2019trec} RUN B) for this task. As shown in Table~\ref{tab:AAW}, the top 6 rows show the evaluation results for the baseline approaches in multi-label classification. The rest of Table~\ref{tab:AAW} contains the best results from all systems that participated in TREC-IS 2019~\cite{mccreadie2019trec}. 
Apparently, our model (I-AID) outperforms significantly all baseline approaches. Our approach achieves improvement in high-priority AWW by $+0.26\%$ compared with MAGNET model and by $+32\%$ compared with \textit{nyu-smap}---the best-achieved result in TREC-IS 2019. Comparing to the \textit{median} scores, our approach achieve outperforming results in high-prioty by $+28\%$ and by $+30\%$ in overall AWW. Remarkably, our approach is the first to achieve a positive AAW on high-priority tweets. Although, we outperform the state of the art in both classification and AAW, the results of our evaluation suggest that a significant amount of research is still necessary to spot high-priority tweets in a satisfactory manner. 
	
	\begin{table}[t!]
		\centering
		\small
	
	
		\caption{Performance evaluation under AAW metric on the test dataset from TREC-IS (RUN B). A higher AAW value indicates better prediction.
		\begin{tabularx}{0.95\linewidth}{@{}p{3.5cm}p{2.5cm}p{1.3cm}@{}}
			\toprule
			\multirow{2}[2]{*}{\textbf{Systems}} & \multicolumn{2}{l}{\textbf{Accumulated Alert Worth (AAW)}} \\ \cmidrule(r){2-3} 
			& High Priority             &       All      \\ \midrule
			
			TextCNN                                     &         -0.9764           &    -0.4884       \\ 
			HAN                                         &       -0.7816             &    -0.4600      \\
			Bi-LSTM                                      &      -0.8760              &    -0.4482     
			\\
			BERT (UPB\_BERT)  &         -0.9680         &       -0.4882     \\  
			TextGAT     &   -0.9794      &   -0.4897        \\
			MAGNET          &    -0.9436         &              -0.4726
			\\ 
			
			\midrule
			Median                                     &         -0.9197           &    -0.4609       \\
			BJUTDMS-run2                                &      -0.9942                  &   -0.4971 \\
			IRIT         &         -0.9942      &                   -0.4971  \\
			irlabISIBase &        -0.2337      &                -0.4935     \\
			UCDbaseline                                 &       -0.7856             &    -0.4131      \\
			nyu-smap &                  -0.1213 &               -0.1973 \\ 
			
			SC-KRun28482low &                   -0.9905 &                     -0.4955\\
			xgboost &                   -0.9942 &                                 -0.4972\\
			UCDrunEL2           &        -0.8556   &  -0.4382 \\
		
			cmu-rf-autothre &       -0.8481     &       -0.4456 \\
			\midrule
			I-AID                              &     \textbf{0.2044} &    \textbf{-0.1509}\\
			\bottomrule
		\end{tabularx}
		\label{tab:AAW}
	\end{table}
\subsubsection{Ablation Study (RQ3)}
Our system has two main components, namely BERT-Encoder and TextGAT. We perform an ablation study that evaluates the performance of each component individually. We try to investigate the influence of each component on the overall performance of I-AID system. To do so, we employ our system with the BERT-Encoder only to classify tweets into multiple information types with the same experimental setting as discussed earlier in Section\ref{sec:implementation}. In the same manner, we employ the TextGAT component as another baseline. Table~\ref{tab:results} shows the evaluation results of each component under different metrics. We can see that the BERT-Encoder can achieve better performance than TextGAT. In particular, on TREC-IS dataset, BERT-Encoder reach $\%50$ F1-score compared with $\%0.26$ by TextGAT. Our results indicate that BERT-Encoder can learn rich representation features from short text (e.g tweets). We notice that only using a BERT-based model can achieve superior performance compared to traditional baselines in text classification tasks. 

\section{Conclusion}\label{sec:conclusion}
We propose a multi-model framework for identifying actionable information from crisis tweets. Our system combines three components: BERT Encoder, TextGAT, and Relation Network. BERT Encoder is used to obtain local information, while the TextGAT module aims to find correlations between named-entities in tweets and their corresponding labels. Finally, we use the Relation Network as a final output layer to learn how important each label is in response to the content of tweets and to output a set of relevant labels. 
Our experimental findings are: 
1) Combining local information captured by BERT-Encoder and global information by Text-GCN is beneficial for rich representation in short-text and significantly advances multi-label classification. 
2) Leveraging transfer-learning from pre-trained language models can efficiently handle sparsity and noise in social media data. 
3) Benchmarking multi-label classification is a challenging task, which requires proper evaluation metrics for fine-grained evaluation.Note that, for an almost-balanced dataset, I-AID achieved its best weighted-averaged F-score of $0.59$ on TREC-IS dataset. This clearly indicates the sensitivity of our approach to the dataset's balancing. Dealing with unbalanced classes remains a future extension to our approach, 
In our future work, we plan to use natural language generation models to address this problem. 
\balance
\bibliographystyle{ACM-Reference-Format}

\section{Introduction}
\begin{figure}
    \centering
    \includegraphics[width=\linewidth]{Figures/tweet-multilabel.pdf}
    \caption{Example of Multi-label Tweet Classification with assigned labels: \textit{FirstPartyObservation}, \textit{EmergingThreats} and \textit{News}.}
    \label{fig:tweetexample}
\end{figure}
Social media has become a key portal for sharing information during emergencies~\cite{DBLP:journals/pacmhci/ZadeSRK0S18}. 
The major difference between social media and traditional news' sources is the possibility of receiving feedback from affected people in real-time. 
Relief organizations can benefit from this two-way communication channel to inform people and gain insights from situational updates received from affected people.
Hence, extracting crisis information from social media can\todo{try avoid using can, may, will, ... as it give the feeling of doubt. Better if you say something like "Recent studies highlight the need to filter information on social media into fine-grained content categories.(add ref.)" (from your abstract) } substantially leverage situational awareness and deliver faster responses. 

Previous works~\cite{to2017identifying,stowe2018improving} addressed information extraction on social media as a binary text classification of two categories (e.g., relevant or irrelevant). 
However, there is a lack of efficient systems to date that can filter informative messages into deeper fine-grained categories. 
The study by \emph{Olteanu et.al}~\cite{olteanu2015expect} was one of the firsts to investigate fine-grained multi-label tweets classification during crisis events. 
The authors concluded that crisis-related tweets tend to contain informative and crucial information types\footnote{In the context of this paper. 
We use the terms labels, classes and information types synonymously.}, such as \texttt{affected individuals, infrastructures and utilities, donations} and \texttt{volunteers, caution and advice, sympathy}, and \texttt{emotional support}. 
As an example, Figure~\ref{fig:tweetexample} shows a tweet can be categorized into more than one information type simultaneously.
This level of categorization is valuable for crisis responders to filter critical information quickly and consequently deliver disaster responses.  
Moreover, Table~\ref{tab:informationTypes} shows that disaster-related tweets can be categorized into multi-level information types, which allows extracting tweets with  \textit{actionable} information. 
We here consider the same definition of '\textit{actionable}' as formalized in~\cite{DBLP:journals/pacmhci/ZadeSRK0S18}.
\textit{Actionable} posts are defined as the ones that would generate an immediate alert for individuals (i.e., stakeholders) responsible for that information type (e.g., \texttt{SearchAndRescue}, \texttt{MovePeople}), in contrast to the \textit{non-actionable} tweets that  belong to the classes like \texttt{Volunteer} or \texttt{FirstPartyObservation}.

On the other hand, short text (e.g., tweets) categorization is known to be a challenging Natural Language Processing (NLP) task~\cite{song2014short}.
This is because short text 
1) does not possess sufficient contextual information;
2) is inherently noisy (i.e. contains misspellings, acronyms, emojis, etc.).
In the case of multi-label classification, the task becomes even more challenging because a tweet could belong to multiple information categories simultaneously.
	
In this paper, we aim to categorize disaster-related tweets into fine-grained information types, and identify \textit{actionable} or \textit{critical} information that might be relevant for disaster relief and support disaster mitigation. 
Our approach contains three components: 
First, we use a BERT encoder to capture the semantics of a tweet and represent into a low-dimensional vector. 
Second, we employ a graph attention network (GAT) to apprehend correlations between tweets tokens (e.g., words or entities) and the corresponding labels of tweets.
Finally, we use a Relation Network as a learnable distance metric to compute the similarity between a tweet's vector -obtained from BERT encoder- and labels vectors -obtained from GAT- in a data-driven way.
To this end, our system integrates the contextualized representation of tweets with the semantic correlations between tweets and their labels. 
The main contributions of this paper can be summarized as follows:
	\begin{itemize}
		\item We propose a multi-model approach (dubbed I-AID) to categorize disaster-related tweets into multi-label types and identify critical information in their context. 
		\item Our approach leverages a contextualized representation from a pre-trained language model (BERT) to capture tweet's semantics. 
		In addition, our approach capture structural information between tweets tokens and their corresponding labels in a graph-structural manner. 
		\item We employ a \textit{learnable} distance metric to learn the similarity between a tweet's vector and the labels' vectors in a data-driven way. 
		\item We conduct several experiments to evaluate the performance of our approach and state-of-art baselines in multi-label text classification.
	\end{itemize}
	
	The rest of this paper is structured as follows: In Section~\ref{sec:relatedworks} we discuss previous works on the classification of crisis information on social media. In Section~\ref{sec:approach} we describe the preliminaries and architecture of our proposed approach. Finally, Section~\ref{sec:conclusion} concludes the paper. All our implementations are open-source and available at the project website.\footnote{Link is removed for anonymous submission} 

	
\begin{table*}[h!]
	\centering 
	\small
	\caption{Crisis Information Types\label{tab:informationTypes}}
	\begin{tabularx}{0.80\textwidth}{@{}p{2.5cm}p{3.5cm}p{7cm}@{}}
		\toprule
		\textbf{Intent Type }& \textbf{Information Type} & \multicolumn{1}{c}{\textbf{Description}} \\ \midrule
		
		\multirow{3}{*} {REQUEST}& GoodServices & Request for a particular service or physical good \\
		& SearchAndRescue & Request a rescue \\
		& InformationWanted & Request information \\ \midrule
		\multirow{11}{*}{REPORT} & Weather & Weather report \\
		& FirstPartyObservation & Eye-witness account \\
		& ThirdPartyObservation & Information from someone else\\
		& EmergingThreats & Problem that cause loss or damage \\
		& ServiceAvailable & Someone is providing a service \\
		& SignificantEventChange & New occurrence that officers to respond to\\
		& MultimediaShare & Shared images or video \\
		& Factoid & Relate some facts, typically numerical \\
		& Official & Report by a government or public representative \\
		& CleanUp & Report of the clean up after the event \\
		& Hashtags & Report with hashtags correspond to each event \\ \midrule
		\multirow{3}{*}{CALLTOACTION} & Volunteer & Call for volunteers to help in response efforts \\
		& Donations & Call for donation of goods or money \\
		& MovePeople & Call to leave an area or go to another area \\ \midrule
	\end{tabularx}
\end{table*}

	

\section{Related Work}\label{sec:relatedworks}
The objective of this work is to categorize disaster-related tweets into multiple information types. Therefore, our work is related to \textit{Extracting Disaster-related Information from Social Media and Short-text, Multi-label Categorization}.
	
	\subsection{Extracting Disaster-related Information from Social Media}
	Several research studies have demonstrated the role of social media as a primary source of information during disasters~\cite{cresci2015crisis}. 

	While some works~\cite{PalshikarAP18} have focused on distinguishing disaster-related information from tweets,
	others~\cite{olteanu2015expect,mccreadie2019trec} have introduced annotation schemes to classify disaster-related tweets into more fine-grained categories to account for the attitude, information source, and decision-making behavior of people tweeting before, during, and after disasters.
	In order to advance the state of social media crisis monitoring solutions, several initiatives~\cite{imran2016lrec}
	have been rolled out in recent years. One of them is the Incident Streams (TREC-IS) track~\cite{mccreadie2019trec} of the Text REtrieval Conference, which commenced in 2018.
	The track aims to categorize disaster-related tweets into multiple information types.  Since in this work we study the TREC-IS datasets, we consider \textit{actionable} information to be the same as that defined by the authors in the challenge. 
	
	\subsection{Multi-label Text Classification}

	While earlier works in text classification~\cite{sriram2010short} viewed feature engineering and model training as different subtasks,  
 with the end-to-end deep learning approaches~\cite{miyazaki2019label} gaining traction and 
	with the introduction of attention mechanism~\cite{vaswani2017attention}, there has been a significant advancement in the field.
Pre-trained contextualized word-embeddings, particularly BERT, are becoming increasingly popular for text classification~\cite{miyazaki2019label}.

	However, since BERT only captures the local contextual information, it falls short. To circumvent this and  apprehend the global relations among words in a vocabulary, graph-based approaches (Graph Neural Networks
	~\cite{kipf2016semi,yao2019graph} have been found promising. 
	
	Recent studies~\cite{DBLP:conf/ecir/LuDN20,DBLP:conf/icaart/PalSS20} have exploited the advantages of BERT and GCN.

	In VGCN-BERT~\cite{DBLP:conf/ecir/LuDN20}, a GCN is used to capture the correlation between words at the vocabulary level (global information). That is, given a vocabulary, GCN
	would relate the meaning of 'new' to 'innovation' and 'exciting', similar to the context-independent word embeddings like Word2Vec~\cite{DBLP:journals/corr/abs-1301-3781}. 
	For an input sentence, the (local) contextual information is captured using the BERT embeddings, while the global information pertaining to the words in the sentence is extracted from the graph embeddings and concatenated with the sentential BERT embeddings. 
	The two representations of BERT and GCN then interact via the self-attention mechanism to perform a classification task. 
	In contrast, in our approach, we employ GAT (Graph Attention Network) to explicitly infuse the correlation-information of the entities in a tweet and their corresponding labels with the tweet's contextualized BERT-representation. 

	In addition, our approach differs from both MAGNET~\cite{DBLP:conf/icaart/PalSS20} and VGCN-BERT~\cite{DBLP:conf/ecir/LuDN20} in the way it computes the similarity between a tweet's representation and the class labels.
	While the above-mentioned approaches use either a fixed, linear distance metric (dot-product function) or self-attention to measure the similarities, our approach benefits from a deeper end-to-end neural architecture to learn this distance function. 
	In particular, we employ meta-learning to learn
	the mapping between the input features and multi-label output in a data-driven way.
	In the subsequent sections, we will further elaborate on our approach. 

	

	
	\subsection{Meta Learning}

	Meta learning (a.k.a. \textit{learning-to-learn} paradigm) refers to the process of improving a learning algorithm over multiple learning episodes. In contrast to the conventional ML approaches, which improve model predictions over multiple data instances, the meta-learning framework treats tasks as training examples to solve a new task  ~\cite{DBLP:journals/corr/abs-2004-05439}.
	In our study, we employ a specific branch of meta learning called metric learning.  Metric-learning learns a distance function between data samples so that the test instances get classified by comparing them to the \texttt{K} labeled examples. The "distance function" consists of 1) an embedding function, which encodes all instances into a vector space and 2) a similarity metric, such as cosine similarity or Euclidean distance, to calculate how close two instances are in the space~\cite{DBLP:journals/corr/abs-2007-09604}. Over the years, the following approaches have been developed to perform this task: Siamese~\cite{koch2015siamese}, Matching~\cite{vinyals2016matching}, Prototypical~\cite{snell2017prototypical}, and Relation Network~\cite{sung2018learning}.
	While the embedding function in all of these approaches is a deep neural net, they differ in terms of the similarity function. Unlike its predecessors, which rely on a fixed similarity metric (such as cosine, euclidean etc.),  Relation Network employs a flexible function approximator to learn similarity and focuses on learning a good similarity metric in a data driven way. Thus, this eliminates the need to manually choose the right metric (Euclidean, cosine, Mahalanobis). By jointly deep learning the embedding with a non-linear similarity metric jointly, Relation Network can better identify matching/mismatching pairs~\cite{garcia2018fewshot}. Hence, in our study, we use the Relation Network for learning the similarity metric. 

\section{Our Approach}\label{sec:approach}

\begin{table}[t!]
\small
\centering
\caption{A list of symbols used in this paper.}
\begin{tabularx}{0.95\columnwidth}{@{}p{0.8cm}p{6.5cm}XX@{}}
\toprule
\textbf{Symbol} & \textbf{Description} \\ \hline
 $T$ & a set of Tweets \\
 $\lambda$ & a set of labels can be assigned to a tweet \\ 
 $N$ & The number of nodes in a graph \\
 $h_i$ & The feature vector of node $v_i$\\
 $F$ & The dimension of node vector \\
 $\beta$ & The feature vector learned by BERT-Encoder\\
 $\mathcal{L}$ & The learned features matrix for labels $\Lambda$ \\
 $Z$ & The concatenated vector between BERT-Encoder output and Labels vectors \\
 \bottomrule
\end{tabularx}
\end{table}
We begin this section by giving a formal specification of the problem of tweets classification. Then, we discuss the details of each component in our approach in Section~\ref{sec:I-AID Architecture}. Figure~\ref{fig:I-AID} illustrates an overview of our approach and how components work together.
	
	\subsection{Problem Formulation}\label{sec:problem_formulation}
	Let $T$ be a set of tweets and $\Lambda=\{\lambda_1, \lambda_2,\cdots ,\lambda_k\}$ be a set of predefined labels (also called information types, see Table~\ref{tab:informationTypes}). 
	We formulate the problem of \textit{identifying crisis information from tweets} as a multi-label classification task, where a tweet $t$ can be assigned one or more labels from $\Lambda$ simultaneously. 
	Our task is to learn a function $\mathcal{M}: T \to \{0,1\}^k$ that maps tweets $T$ to relevant labels from $\Lambda$. We assume a supervised learning setting, where 
	a training data $\mathcal{D}=\big\{ (t^{(i)},y^{(i)})\times \{0,1\}^k \big\}_{i=1}^N$ consists of $N$ tweets. Hence, each tweet $t^{(i)}$ is labelled with a set of corresponding labels $y^{(i)}$, where $y^{(i)}_j = 1$ means that $t^{(i)}$ belongs to the class $\lambda_j$. Conversely, $y^{(i)}_j = 0$ means that $t^{(i)}$ does not belong to the class $\lambda_j$. Our approach learns the function $\mathcal{H}$ by using three neural networks. It first transforms tweets $t^{(i)}$ into a d-dimensional vector $x^{(i)}$ using a pretrained BERT model. In parallel, it learns label embeddings $l$ using a graph convolution network. These are then concatenated with the tweet embedding $x^{(i)}$. Finally, these vectors are fed to our last component (Relation Network) to identify relevant labels for $t^{(i)}$. 
	
	\subsection{The I-AID Architecture}\label{sec:I-AID Architecture}
	

	\begin{figure*}[t!]
		\centering
		\captionsetup{justification=centering}
		\includegraphics[width=0.8\linewidth]{Figures/I-AID.pdf}
		\caption{Illustration of our proposed I-AID System.}
		\label{fig:I-AID}
	\end{figure*}
	
	\subsubsection{BERT-Encoder.}
	This is the first component in our system which transforms an input tweet into a vector representation ($Z$) of it's contextual meaning. As shown in Figure~\ref{fig:I-AID}, the BERT-Encoder takes tweet $t^{(i)}$ with $m$ tokens $[w^{(i)}_1,w^{(i)}_2,\dots ,w^{(i)}_m]$ and outputs vector $Z$. We employ a BERT-base architecture with $12$ encoder blocks, $768$ hidden dimensions, and $12$ attention heads. We refer the readers to the original BERT paper~\cite{devlin2018bert} for a detailed description of its architecture and input representation. 
	Furthermore, a special preprocessing is performed for BERT input. A \texttt{[CLS]} token is appended to the beginning of the tweet and another token \texttt{[SEP]} is inserted after each sentence as an indicator of sentence boundary. Each token $w_i$ is assigned three kinds of embeddings (token, segmentation, and position). These three embeddings are summed to a single output vector \texttt{[CLS]} that captures the meaning of an input tweet.




	
\subsubsection{Text-Graph Neural Network (TextGAT)}
Traditional methods (e.g., word2vec) can properly capture features from a text. However, these methods ignore the structural information and relationship between words in a text corpus~\cite{peng2018large}. The recently proposed graph networks~\cite{yao2019graph} aim to tackle this challenge by modeling text as a graph where words are nodes and relations between them are edges.
In our work, we build a graph $G=(\mathcal{V}, \mathcal{E})$ where $\mathcal{V}$ and $\mathcal{E}$ represent nodes set and their edges respectively. Each node $v_i \in \mathcal{V}$ can be a \textit{word, named-entity\footnote{We spot named-entities in tweets using spaCy entity recongnizer\url{https://spacy.io/api/entityrecognizer}} or label} from the dataset $D$. We represent nodes using a feature matrix $\mathbf{H}=\{h_1, h_2, \cdots ,h_N\}$ where $N$ is the number of nodes and $F$ is the feature dimension. 
As shown in Figure~\ref{fig:I-AID}, TextGAT component has two graph attention layers (GAT). Each layer takes nodes features $\mathbf{H}$ as an input and performs an \textit{attention} operation~\cite{velivckovic2017graph} to learn new feature $\mathbf{\hat{H}}=\{\hat{h_1}, \hat{h_2}, \cdots ,\hat{h_N}\}$ for each node based on it's neighbours importance (i.e., \textit{attention from its neighbours}). Hence, we employ a shared attention mechanism $att: \mathbb{R}^{\hat{F}} \times \mathbb{R}^{\hat{F}} \longrightarrow \mathbb{R}$ over all nodes. The graph attention operated on the node representation can be written as: 
\begin{equation}\label{eq:att}
	    a_{ij}=att~(\textbf{W}x_i,\textbf{W}x_j)
\end{equation}

Where $att$ is a single-layer feedforward network, parametrized by a weight matrix $\mathbf{W}\in \mathbb{R}^{\hat{F}\times F}$ is applied to every node. Finally, we use a softmax function to normalize the attention scores as shown in Eq.~\ref{eq:softmax}.
	\begin{equation}\label{eq:softmax}
	\alpha_{ij}=\frac{exp(a_{ij})}{\sum_{k\in N_i}exp(a_{ik})}
	\end{equation}

To this end, TextGAT learns structural information between nodes based on the relative importance of neighbouring. The learned representations of labels are then extracted and concatenated with the tweet's vector as an input for the last component. 
	
\subsubsection{Relation Network.}\label{relation-network} In this component, we aim to learn a similarity metric in a data-driven way (\textit{learnable}) between the tweet's vector $X$ and labels vectors $\lambda$.
Furthermore, we employ a neural network as a learnable, non-linear distance function that \textit{learns} how to match similarity (i.e., relation) between the tweet's vector and each label. Relation Network takes as an input the concatenated matrix $Z= X \sqcup \Lambda$ of BERT-Encoder output with the label vectors. Since our task is multi-label classification, we use a binary cross-entropy as a loss function, as in Eq.~\ref{eq:binary_cross_entropy}. Then, we use a sigmoid function in the output layer to compute the probability of each label independently over all possible labels ($\Lambda$), in contrast to a softmax function which only considers the label with the highest probability. Finally, a set of relevant labels are returned as a final output of our approach.
\begin{equation}
	\label{eq:binary_cross_entropy}
	\arg\min\frac{1}{k}\sum_{i=1}^{k}\left[y^{(i)}\log(\hat{y}^{(i)})+(1-y^{(i)})\log(1-\hat{y}^{(i})\right]
	\end{equation}
	
	Where $y^{(i)}$ and $\hat{y}^{(i)}$ are the predicted and ground-truth labels of tweet $i$ respectively $k$ is the number of all possible labels. 



\section{Experiments}\label{sec:experiment}
In this section, we conduct several experiments to answer the following research questions:
	\begin{itemize}
	    \item[-] \textbf{RQ1: }
	    How does our approach perform (\textit{compare with state-of-the-art multi-label models}) on short-text classification (e.g., tweets)?
	   
	    \item[-] \textbf{RQ2: } How effective is our approach in identifying tweets with \textit{actionable} information?
	    \item[-] \textbf{RQ3: } How do each component in our approach affect the performance? (i.e., Ablation Study)
	\end{itemize}
		
	
\begin{table}[ht]
\centering
\captionsetup{justification=centering}
\caption{Overview of the Datasets.}
\begin{tabular}[t]{lcccc}
\toprule
\textbf{Datasets} & \textbf{\# Train} & \textbf{\# Valid} & \textbf{\# Test} & \textbf{\# Classes}\\
\midrule
TREC-IS &  27,467 & 6,867 &  8,584 & 25      \\
COVID19-Tweets &  4,844 & 1,211 & 1,514  & 12   \\
\bottomrule
\end{tabular}
\label{tab:datasets}
\end{table}%
	
	
\subsection{Datasets}
We conduct a set of experiments on two public datasets. 
The datasets are collections of disaster-related tweets provided by TREC~\cite{mccreadie2019trec}. Table~\ref{tab:datasets} gives an overview about dataset statistics: number of tweets used to train, validate, and test the models, in addition to total number of classes.  
In particular, we split each dataset with a ratio ($80\%-20\%$); where we use $80\%$ of tweets to train the models and $20\%$ for test. During training, we use $20\%$ of the training data to validate the system. We briefly summarize each dataset as follows:

\begin{itemize}
    \item \textbf{TREC-IS: }
    This dataset contains approximately $35$K tweets collected during $33$ unique disasters between $2012-2019$ (e.g., \textit{wildfires, earthquakes, floods, hurricanes, bombings, and shootings}). 
	The tweets are labeled into $25$ information types by human experts and volunteers. 
	
	\item \textbf{COVID19-Tweets: } This dataset contains a collection of tweets about the COVID-19 outbreak in different affected regions. In total, the data has $7,590$ tweets labeled with one or more of the full $12$ information type labels (the same as for TREC-IS dataset). 
\end{itemize}

Figure~\ref{fig:tweets-informationTypes} shows the distribution of tweets per information type in both datasets. Apparently, the datasets are highly imbalanced w.r.t information types distribution across tweets. For example in TREC-IS dataset, there are more than 4,000 tweets are mapped to the information types of \textit{Sentiment}, \textit{News}, and \textit{ThirdPartyObservation}. In contrast, the information types \textit{CleanUP, InformationWanted}, and \textit{MovePeople} have significantly fewer tweets. In Covid19-Tweets, the tweets distribution is extremely imbalanced, most of tweets are categorized into \textit{Irrelevant} or \textit{ContextualInformation, Advice, News}. This skewing distribution in information types across tweets made the task of classification more challenging. 

\begin{figure*}[t!]
    \centering
    \begin{subfigure}[t]{0.5\textwidth}
        \centering
        \small
        \captionsetup{justification=centering}
        \includegraphics[width=\columnwidth]{Figures/trecis-labels.pdf}
        \caption{TREC-IS Dataset}
    \end{subfigure}%
    ~ 
    \begin{subfigure}[t]{0.5\textwidth}
        \centering
        \small
        \captionsetup{justification=centering}

        \includegraphics[width=\columnwidth]{Figures/covidtweets-labels.pdf}
        \caption{Covid19 Tweets Dataset}
    \end{subfigure}
    \caption{Tweets Distribution across Information Types}
    \label{fig:tweets-informationTypes}
\end{figure*}





	\subsection{Baselines} \label{sec:baselines}
	We consider a set of state-of-the-art models in \textit{multi-label classification}\footnote{\scriptsize Note that, we consider relevant baselines that were implemented for multi-label classification. The current version of VGCN-BERT can not be directly used for multi-label classification, therefore we didn't include it in the baselines.}, as baselines in our evaluation. We briefly describe each baseline as follows:

	
\begin{itemize}
		\item \textbf{TextCNN}~\cite{kim2014convolutional}: This approach uses a convolutional neural network to construct text representation. 
	
	
		\item \textbf{HAN}~\cite{yang2016hierarchical}: This model uses a hierarchical attention-neural-network to encode text with word-level attention on each sentence.
		
		\item \textbf{BiLSTM}~\cite{zhou2016attention}: A bi-directional LSTM model that parses the text from left-to-right and right-to-left, then uses the final hidden state as a feature representation for the whole text.
		\item \textbf{BERT}~\cite{devlin2018bert}: This a BERT-base model with a transformer architecture (multi-layer, multi-head and self-attention) that represent tweet as a contextualized embedding vector. 
		\item \textbf{TextGAT}~\cite{velivckovic2017graph}: This model uses graph attention network (GAT) to represent corpus as a heterogeneous graph, then learn word and tweet embeddings with jointly. 
	
	
		\item \textbf{MAGNET}~\cite{DBLP:conf/icaart/PalSS20}: This approach employs BERT embeddings to represent tweets and GCN for labels classifiers. Then, it employs to use a dot-product function to compute similarities between tweet vector and label vectors.
		
	\end{itemize}
	
\subsection{Implementation and Preprocessing}\label{sec:implementation}
We use the open-source implementations for the models (TextCNN, HAN, BiLSTM, and TextGAT) provided by the corresponding authors on GitHub. For BERT model, we use the implementation from Huggingface\footnote{\url{https://huggingface.co/}} library. Moreover, we implemented the code for MAGNET model as it has not been open-sourced to date. Hyper-parameters are set in the baseline methods same as  mentioned in their original papers. In our model, we tune the hyper-parameters via grid search method to find optimal values for best performances. In particular, our model achieves its best performance with the following values: training-epochs to $200$ with batch-size of $128$ and Adam optimizer~\cite{kingma2014adam} with a learning-rate of $2\mathrm{e}^{-5}$. To avoid overfitting, We add a dropout layer with a rate of $0.25$ and apply an early-stopping technique during model's training. Our implementation of I-AID model is open-sourced and available on the project website\footnote{Links are removed due to anonymous submission}.

\paragraph{Data Preprocessing:} Given that datasets are disaster-related tweets, we perform proper preprocessing steps. In particular, we perform the following preprocessing steps: (1) We use the NLTK's TweetTokenize API to tokenize tweets and retain the text content. (2) Stop-words, URLs, usernames, and Unicode-characters were removed. (3) Extra white-spaces, repeated full stops, question marks, and exclamation marks were removed. (4) Emojis\footnote{\url{https://pypi.org/project/emoji/}} were converted to text using the emoji python library. (4) Finally, spaCy library is used \footnote{\url{https://github.com/explosion/spaCy}}, to extract \textit{named-entities} from tweets.

	
\subsection{Evaluation Metrics}\label{section:evaluation_metrics}

	We employ appropriate evaluation metrics for multi-label classification scenario. Particularly, We use a weighted-averaged F1-score, hamming loss, and jaccard score to evaluate the system's performance in multi-label classification tasks: 

	\begin{itemize}
	
	 \item{\textbf{Weighted-average F1-score}}:
	 F1-score is the harmonic mean of precision and recall scores. We use a \textit{weighted-averaged} that calculates F1 score for each label independently, then adds them together uses a weight that depends on the number of true labels of each class.
	 \begin{equation}
	     F1= 2 \times \frac{precision\times recall}{precision+recall}
	 \end{equation}
	 
	\item \textbf{Hamming Loss.}: To estimate the error rate in classification, we use the \textit{hamming loss} function~\cite{schapire1999improved} 
	that estimates the fraction of incorrectly predicted labels out of all predicted labels. Hence, the smaller the value, the better the performance.  
	
	\begin{equation}
	{hamming\:Loss\:}(y^{(i)}, \hat{y}^{(i)})=\frac{1}{N}\sum_{i=1}^{N}\frac{1}{k} |y^{(i)}\oplus\hat{y}^{(i)}|
	\end{equation}
 	where $N$ is the dataset size, $k$ is the total number of labels (i.e. $|\Lambda|$ ), $\oplus$ denotes the XOR operator and $y^{(i)}$ and $\hat{y}^{(i)}$ are the ground-truth and predict labels of tweet $i$ respectively.
 	
	 \item \textbf{Jaccard Score.} To assess the system's accuracy, we use the Jaccard score to compute the similarity between predicted labels $\hat{y}^{(i)}$ and ground truth labels $y^{(i)}$. 
	 Jaccard score computes the percentage of common labels in two sets out of total labels as
	\begin{equation}
	{jaccard\:}(y^{(i)}, \hat{y}^{(i)})=\frac{|y^{(i)} \cap \hat{y}^{(i)}|}{ |y^{(i)} \cup \hat{y}^{(i)}|}  
	\end{equation}
	Where $y_i$ and $\hat{y}_i$ are the ground-truth and predict labels sets of tweet $i$. $\cap$ and $\cup$ denote intersection and union set operations respectively.
	\end{itemize}
	
	\subsubsection{Evaluating Actionable Information: Accumulated Alert Worth}
	Now, we aim to evaluate the efficacy of our system in identifying tweets with actionable information, i.e. system should trigger an alert if an input tweet includes critical information (e.g., requests for search and rescue or reports of emerging threats). Recently, TREC-IS~\cite{mccreadie2020incident} introduce an ad-hoc metric called \emph{AAW} to evaluate systems performance in detecting actionable/critical information in tweets. The AAW score ranges from $-1$ to $1$, where positive value indicates high critical information in a tweet and negative score for less critical. We refer the readers to find more details about AAW metric in the original paper~\cite{mccreadie2019trec}. Here, we summarize AAW metric as follows: 
	
	\begin{equation}
	    hPW(t)=
	    \begin{cases}
	\alpha+((1-\alpha)\cdot (\beta(t)+\hat{\beta}(t)) & \text{if } p_t^s>=0.7\\
	-1 &  \text{otherwise} 
	\end{cases}   
	\end{equation}
	
	
	\begin{equation}
	lPW(t)=    
	\begin{cases}
	argmax(-log(\frac{\delta}{2}+1), -1) & \text{if } p_t^s>=0.7\\
	\beta(t)+\hat{\beta}(t) &  \text{otherwise} 
	\end{cases}   
	\end{equation}
	
	\begin{equation}
	\label{eq:AAW}
	AAW=
	\frac{1}{2}\sum_{t\in \mathcal{T}}
	\begin{cases}
	\frac{1}{|\mathcal{T}_{h}|} \cdot hPW(t) & \text{if } t \in \mathcal{T}_{h}\\
	\frac{1}{|\mathcal{T}_{l}|} \cdot lPW(t) &  \text{otherwise} 
	\end{cases}   
	\end{equation}
	where $\mathcal{T}_{h}$ and $\mathcal{T}_{l}$ denote sets of tweets with high and low priorities respectively. 
	hPW(t) is a scoring function for tweets that should generate alerts and
	lPW(t) is a scoring function for tweets that should not generate alerts.
	

\subsection{Discussion}
\subsubsection{Performance Comparison (RQ1)}

		\begin{table*}[t!]
		\centering
		\caption{Evaluation results of our approach and baselines on two datasets (TREC-IS and COVID19-Tweets) under weighted-average (Precision, Recall and F1), Hamming Loss and Jaccard score. Best results are in bold.}
		\begin{tabularx}{0.85\linewidth}{@{}p{2.2cm}p{2.1cm}p{1.5cm}p{1cm}p{1cm}p{1cm}p{1.5cm}p{1.1cm}XXXXXX@{}}
			\toprule
			\multirow{2}[2]{*}{\textbf{Datasets}} & \multirow{2}[2]{*}{\textbf{Metrics}}& \multicolumn{6}{c}{\textbf{Baselines}} & \multirow{2}[2]{*}{\textbf{I-AID}} \\ 
			\cmidrule(lr){3-8}
			{\textbf{}}&  & TextCNN & HAN & BiLSTM & BERT & TextGAT & MAGNET\\
			\cmidrule(r){1-1} \cmidrule(lr){2-2} \cmidrule(lr){3-8} \cmidrule(l){9-9}
			\multirow{5}{*}{TREC-IS} & 
			Precision & 0.29 & 0.31 & 0.27 & 0.42  & 0.18  & 0.50 & \textbf{0.57}\\
			& Recall & 0.55 & 0.55 & 0.65 & \textbf{0.67} & 0.58 & 0.60 & 0.62 \\
			& F1-score & 0.25  & 0.37 & 0.31 & 0.50  &  0.26 & 0.53 & \textbf{0.59}  \\
		
			& Jaccard score & 0.18 & 0.28 & 0.19 & 0.34 &  0.18 &  0.38 & \textbf{0.43} \\
			& Hamming Loss & 0.24 & 0.15 & 0.26 & 0.11 &  0.24 &  0.09 & \textbf{0.07} \\
			
			\cmidrule(r){1-1} \cmidrule(lr){2-2} \cmidrule(lr){3-8} \cmidrule(l){9-9}
			\multirow{5}{*}{COVID19-Tweets} & 
			Precision & 0.39 & 0.33 & 0.35 & 0.37 & 0.29 & 0.40 & \textbf{0.45}  \\
			& Recall & 0.74 & 0.60 & 0.72 & 0.71  & 0.62 & \textbf{0.76} & 0.71  \\
			& F1-Score & 0.47  &  0.40 & 0.43  &  0.47 &  0.36 & 0.51 & \textbf{0.55}   \\
		
			& Jaccard Score &  0.33 & 0.28 & 0.21 & 0.37 & 0.18 & 0.40 & \textbf{0.43} \\
			& Hamming Loss &  0.11 & \textbf{0.04} & 0.07 & 0.10 & 0.17 &  0.12 & 0.08 \\
			\bottomrule
		\end{tabularx}
		\label{tab:results}
	\end{table*}
	
	
	
	
	We evaluate the performance of our approach and baselines methods 
	using different metrics for multi-label classification task. To ensure a fair evaluation, we use the same \textit{train} dataset during models training and \textit{test} dataset for evaluation. Table~\ref{tab:results} reports our evaluation results for each model on two datasets (TREC-IS and Covid19-Tweets). We consider Weighted-averaged F1-score as the primary metric to compare and rank systems performances. Weighted-average F1 takes into account the average performance of each system across all information types. 
	Across all the datasets, our approach (I-AID) achieve outperformed results over the other baselines under several metrics. In particular, our approach outperforms the weighted-averaged F1-score of MAGNET, the state-of-the-art baseline of multi-label classification, by $+6\%$ on TREC-IS and $+4\%$ on Covid19-Tweets. On the other hand, we evaluate the performance of our approach per each information types individually.
	
	Comparing the results from all the metrics under different settings (i.e., with different datasets), while our model achieved sub-optimal results under precision and recall metrics, it mostly performed better than the baselines. To this end, our results indicate that I-AID performs fairly well when categorizing disaster-related tweets into multiple information types, this is because of three facts:  
	(1) We constructed a multi-model framework that leverages contextualized embedding from the BERT model to capture contextual information within a tweet. 
	(2) Our approach enriches semantics of the tweet representations by injecting label information and integrating additional structural information between entities in tweets and labels (i.e., entity-entity and entity-label) via GCN.
	(3) Finally, we avoid the manual selection of similarity metric by automatically learning the most efficient distance function in our \emph{Relation Network} (as discussed in Section~\ref{relation-network}) in a data-driven way. By using a learnable distance function, we intend to learn an efficient metric to facilitate the mapping between a tweet and the multiple labels that could be assigned to it.
	
    
\subsubsection{Actionable Information In Tweets (RQ2)}
Asking ourselves $Q_2$ to understand the behavior of our approach in identifying tweets with critical/actionable information, we use an ad-hoc metric proposed by TREC to evaluate system's ability in this behavior. There are two ways to define an \textit{actionable} tweet: (1) In terms of high priority information, commonly marked as critical by human assessors. (2) In terms of information type, for instance, a tweet with the labels \textit{MovePeople} or \textit{CleanUP} is considered more actionable than \textit{News} or \textit{Multimediashare}.
In table~\ref{tab:AAW}, we report the evaluation results of our approach and all the relevant approaches under AAW metric (Equation~\ref{eq:AAW}) on the test dataset from TREC-IS (edition 2019\cite{mccreadie2019trec} RUN B) for this task. As shown in Table~\ref{tab:AAW}, the top 6 rows show the evaluation results for the baseline approaches in multi-label classification. The rest of Table~\ref{tab:AAW} contains the best results from all systems that participated in TREC-IS 2019~\cite{mccreadie2019trec}. 
Apparently, our model (I-AID) outperforms substantially all baseline approaches. Our approach achieves improvement in high-priority AWW by $+0.26\%$ compared with MAGNET model and by $+32\%$ compared with \textit{nyu-smap}---the best-achieved result in TREC-IS 2019. Comparing to the \textit{median} scores, our approach achieve outperforming results in high-prioty by $+28\%$ and by $+30\%$ in overall AWW. Remarkably, our approach is the first to achieve a positive AAW on high-priority tweets. Although, we outperform the state of the art in both classification and AAW, the results of our evaluation suggest that a significant amount of research is still necessary to spot high-priority tweets in a satisfactory manner. 
	
	\begin{table}[t!]
		\centering
		\small
	
	
		\caption{Performance evaluation under AAW metric on the test dataset from TREC-IS (RUN B). A higher AAW value indicates better prediction.
		\begin{tabularx}{0.95\linewidth}{@{}p{3.5cm}p{2.5cm}p{1.3cm}@{}}
			\toprule
			\multirow{2}[2]{*}{\textbf{Systems}} & \multicolumn{2}{l}{\textbf{Accumulated Alert Worth (AAW)}} \\ \cmidrule(r){2-3} 
			& High Priority             &       All      \\ \midrule
			
			TextCNN                                     &         -0.9764           &    -0.4884       \\ 
			HAN                                         &       -0.7816             &    -0.4600      \\
			Bi-LSTM                                      &      -0.8760              &    -0.4482     
			\\
			BERT (UPB\_BERT)  &         -0.9680         &       -0.4882     \\  
			TextGAT     &   -0.9794      &   -0.4897        \\
			MAGNET          &    -0.9436         &              -0.4726
			\\ 
			
			\midrule
			Median                                     &         -0.9197           &    -0.4609       \\
			BJUTDMS-run2                                &      -0.9942                  &   -0.4971 \\
			IRIT         &         -0.9942      &                   -0.4971  \\
			irlabISIBase &        -0.2337      &                -0.4935     \\
			UCDbaseline                                 &       -0.7856             &    -0.4131      \\
			nyu-smap &                  -0.1213 &               -0.1973 \\ 
			
			SC-KRun28482low &                   -0.9905 &                     -0.4955\\
			xgboost &                   -0.9942 &                                 -0.4972\\
			UCDrunEL2           &        -0.8556   &  -0.4382 \\
		
			cmu-rf-autothre &       -0.8481     &       -0.4456 \\
			\midrule
			I-AID                              &     \textbf{0.2044} &    \textbf{-0.1509}\\
			\bottomrule
		\end{tabularx}
		\label{tab:AAW}
	\end{table}
\subsubsection{Ablation Study (RQ3)}
Our system has two main components, namely BERT-Encoder and TextGAT. We perform an ablation study that evaluates the performance of each component individually. We try to investigate the influence of each component on the overall performance of I-AID system. To do so, we employ our system with the BERT-Encoder only to classify tweets into multiple information types with the same experimental setting as discussed earlier in Section\ref{sec:implementation}. In the same manner, we employ the TextGAT component as another baseline. Table~\ref{tab:results} shows the evaluation results of each component under different metrics. We can see that the BERT-Encoder can achieve better performance than TextGAT. In particular, on TREC-IS dataset, BERT-Encoder reach $\%50$ F1-score compared with $\%0.26$ by TextGAT. Our results indicate that BERT-Encoder can learn rich representation features from short text (e.g tweets). We notice that only using a BERT-based model can achieve superior performance compared to traditional baselines in text classification tasks. 

\section{Conclusion}\label{sec:conclusion}
We propose a multi-model framework for identifying actionable information from crisis tweets. Our system combines three components: BERT Encoder, TextGAT, and Relation Network. BERT Encoder is used to obtain local information, while the TextGAT module aims to find correlations between named-entities in tweets and their corresponding labels. Finally, we use the Relation Network as a final output layer to learn how important each label is in response to the content of tweets and to output a set of relevant labels. 
Our experimental findings are: 
1) Combining local information captured by BERT-Encoder and global information by Text-GCN is beneficial for rich representation in short-text and significantly advances multi-label classification. 
2) Leveraging transfer-learning from pre-trained language models can efficiently handle sparsity and noise in social media data. 
3) Benchmarking multi-label classification is a challenging task, which requires proper evaluation metrics for fine-grained evaluation.Note that, for an almost-balanced dataset, I-AID achieved its best weighted-averaged F-score of $0.59$ on TREC-IS dataset. This clearly indicates the sensitivity of our approach to the dataset's balancing. Dealing with unbalanced classes remains a future extension to our approach, 
In our future work, we plan to use natural language generation models to address this problem. 
\balance
\bibliographystyle{ACM-Reference-Format}

\section{Introduction}

Social media platforms have become key portals for sharing information during crisis situations. Hence, extracting crisis information from social media is a major process to leverage situational awareness and deliver faster responses~\cite{nazer2017intelligent}. 
Previous studies~\cite{to2017identifying,stowe2018improving} focused on an \textit{abstract-level} of categorizing information (e.g.,  relevant/irrelevant) from a massive social media posts.  

However, there is a lack of efficient tools that can filter informative disaster-related tweets in a deeper and fine-grained categories (e.g., finding affected people, ask for donations, analysing infrastructure damage). In a primary
study by Olteanu et.al~\cite{olteanu2015expect} explored a collection of tweets during different crisis events. 
The authors found that crisis-related tweets tend to comprise one of the following general information types such as affected individuals, infrastructures and utilities, donations and volunteers, caution and advice, sympathy and emotional support. 

This level of information categorization is useful for crisis managers as well as first responders to quickly filter crisis information and deliver disaster responses. 
As shown in Table~\ref{tab:informationTypes}, disaster-related tweets can be categorized into multi-level information types (i.e., labels) so as to extract actionable information and crisis mapping. 
On the other hand, short text (e.g., tweets) categorization as multi-label classification problem (MLC) is a challenging task -\textit{tweet can be classified into one or more types (i.e., class/label)}-, due to tweets characteristics such as misspellings, limited use of standard grammar, abbreviations, slang, non-standard terminology, emojis, and noise. 

 Different research studies have made significant efforts at tackling these challenges using \textit{transfer-learning} and pre-trained language models (e.g., EMLO, BERT, GPT-2) \cite{zaherafine,sun2019fine}. Where, these models are first trained on a large amount of unlabeled data to capture rich representations. Then, they are applied to downstream tasks such as text classification, word sense disambiguation or sentiment analysis.
 
 In this paper, we aim to extract \textit{actionable information} from disaster-related tweets to support such as disaster decisions such as move people, ask for donations or volunteers.
 In particular, our approach leverages \textit{transfer-learning} from BERT model to capture  semantic meaning of a tweet and represent as a low-dimensional vector. In addition, we propose a graph convolutional network (GCN) to capture correlations between entities recognized in tweets and information types.
 Experiments on a realistic dataset (TREC-IS) of disaster-related tweets show that our approach apparently outperforms existing state-of-the-art methods.  The main contributions of this paper can be summarized as follows:
\begin{itemize}
	\item We present a BERT-GCN model (dubbed I-AID) to identify crisis information from disaster-related tweets. 
	\item We leverage \textit{transfer-learning} from BERT pretrained model to embed tweets into low-dimensional vectors. Then, we use GCN to capture correlations between entities and labels in a graph-structural manner. 
	\item We conduct extensive experiments to measure the performance categorizing disaster-related tweets into multiple information types. 
	Our results suggest that our method consistently achieves superior performance over different baseline models (see Section~\ref{sec:baselines} by up to $5\%$ in F1-score. 
\end{itemize}





All our implementations are open-source and available on the project website.\footnote{Link is removed for anonymous submission

The rest of this paper is structured as follows: Section \ref{sec:relatedworks} discusses the previous works in the classification crisis information on social media. Section \ref{sec:approach} describes the preliminaries and architecture of our proposed approach.  Finally, Section \ref{sec:experiment} shows the experimental setup, result analysis and discussion of our experiments. Section \ref{sec:conclusion} concludes the paper.

\begin{table*}[t!]
	\centering
	\caption{Actionable Information Types\label{tab:informationTypes}}
	\begin{tabular}{@{}lll@{}}
		\toprule
		\textbf{Intent Type }& \textbf{Information Type} & \multicolumn{1}{c}{\textbf{Description}} \\ \midrule
		
		\multirow{3}{*} {REQUEST}& GoodServices & Asking for a particular service or physical good \\
		& SearchAndRescue & Requesting a rescue \\
		& InformationWanted & Requesting information \\ \midrule
		\multirow{11}{*}{REPORT} & Weather & Providing a weather report \\
		& FirstPartyObservation &Giving an eye-witness account \\
		& ThirdPartyObservation & Reporting an information from someone else\\
		& EmergingThreats & Reporting a problem that cause loss or damage \\
		& ServiceAvailable & Reporting that that someone is providing a service \\
		& SignificantEventChange & Reporting a new occurrence that officers to respond\\
		& MultimediaShare & Sharing images or video \\
		& Factoid & Relating some facts, typically numerical \\
		& Offical & Report by a government or public representative \\
		& CleanUp & Report of the clean up after the event \\
		& Hashtags & Report with hashtags correspond to each event \\ \midrule
		\multirow{3}{*}{CALLTOACTION} & Volunteer & Asking people to volunteer to help in response effort \\
		& Donations & Asking people to donate goods or money \\
		& MovePeople & Asking people to leave an area or go to another area \\ \midrule
	\end{tabular}
\end{table*}

\section{Related Work}\label{sec:relatedworks}
The aim of this work is to categorize disaster-related tweets into multiple information types. Thus, our work is related to \textit{short text categorization, multilabel classification, pretrained embedding models}. In the following, we briefly discuss the state of the art in each of these areas in turn. 

\subsection{Short Text Classification}
A lot of work has been done to learn from the short-textual nature of tweets. Wang et al. ~\cite{wang2017combining} conceptualized tweets using a knowledge base and then by enriching them with word and sub-word information, proposed a CNN-based framework for classification. Along the same lines, Burel et al.~\cite{burel2017semantic} extracted conceptual information (from a knowledge base) about named entities in text 
and by employing a deep CNN model, identified the information category of crisis-related tweets. To alleviate the tedious process of finding good text-based features, Kundu et al.~\cite{KunduSD18} trained an end-to-end deep learning based model to classify tweets into different actionable classes. They used a 5-layer network; the first hidden layer composed of Bidirectional LSTM to encode the contextual information, followed by 3 more layers of vanilla LSTM. In their experiments against FIRE2016 and FIRE2017 datasets, this model outperformed the tf-idf and CNN based models. Caragea et al.~\cite{caragea2016identifying} proposed a CNN for text classification to identify informative
tweets during disaster events. Their experiments against a subset of CrisisLexT26 dataset show that Convolutional Neural Networks outperform other supervised learning algorithms like Support Vector Machines and Artificial Neural Networks, which use a combination of unigrams, bigrams and trigrams as input features. A similar work ,\cite{NguyenAJSIM17}, also concludes CNNs to be the most effective in classifying tweets of new event, when no event-specific data is available, using available out-of-event data. 
\subsection{Social Sensing}
Several research studies demonstrate the role of social media as a primary source of information during disasters. In particular, social media (e.g Twitter) has been used to leverage situational awareness and crisis mapping~\cite{yin2015using,cameron2012emergency,cresci2015crisis}. For example, to distinguish disaster-related information from others on social media, Palshikar et al.~\cite{DBLP:journals/isf/PalshikarAP18} employed weak supervision to learn a bag of words model from news corpus of various disasters. The word model was then augmented using a corpus of unlabelled tweets, followed by a method of learning word-weights online to dynamically improve the model in run-time setting. This approach could effectively classify completely unseen stream of tweets. Stowe et al.~\cite{stowe2016identifying} proposed an annotation schema to classify disaster-related tweets into six fine-grained categories to account for the attitude, information source and decision-making behavior of people tweeting before, during and after disasters. They also trained feature-rich binary classifiers for each of the six categories independently and performed an ablation study to understand which features contributed to the relevance and fine-grained categorization of tweets.
Going a step ahead, ~\cite{ToriumiB16} developed a real-time information sharing system using a clustering-based tweet classification approach to aid in emanating awareness during disasters. The system considered tweets that were retweeted more than 100 times and leveraged this to create a retweet network to connect topic-similar tweets. Extending the work of ~\cite{ZhangWGH16} for extracting keypharses from tweets, Chowdhury et al.~\cite{ChowdhuryCC19} proposed an approach for keyphrase extraction from disaster-related tweets. By utilizing contextual word embeddings, POS-tags, phonetics, and phonological features, they improved the model's performance on both general Twitter data and disaster-related tweets.  

On the other hand, for improving awareness pertaining to socio-economic recovery post disasters, Shibuya et al.~\cite{ShibuyaT19} considered an increased demand for used cars as an indicator of socio-economic recovery, and correlated public sentiment on social media with market data to gather insights. To help curb rumors and fake news during disasters, ~\cite{BarnwalGKBG19} proposed a hybrid approach based on classification and ranking for identifying fact-checkable tweets during disasters. To understand the similarities and differences among different crisis events and study the impact of certain tweet characteristics on classification based on disaster-informativeness, a systematic analysis of the CrisisLexT26~\cite{olteanu2015expect} dataset was performed by Graf et al.~\cite{GrafRSPK18}. Based on this analysis, they trained a cross-domain classifier, which could classify tweets across disaster events of different types, with a set of 29 features representing the temporal, linguistic and source characteristics of a tweet.  

\subsection{End-to-End Learning Models}
\todo[inline]{Kind of features extract from tweets. Kind of deep learning models}
Our system is an end-to-end neural architectures for learning semantic representation of data.

Our primary research aims to leverage multi-label based classifiers that can identify low-level information types (e.g. donation, volunteer, evacuation) from disaster-related tweets. Exploiting the Label Embedding method for text classification~\cite{ZhangXCWJ18}, Miyazaki et al.~\cite{MiyazakiMTOG19} apprehend the hierarchical structure of labels present in the TREC-IS 2018 dataset as a key feature. They vectorize the tweet text and the corresponding higher and lower-level label using pre-trained models such as BERT and ELMo. These embeddings are then fed to a BiLSTM network that outputs three matrices for text, lower-level label, and higher-level label respectively. These matrices are then used for calculating label attention and hierarchical attention vectors, which are then passed to a 2-layer feed-forward network for classification of the tweet into one of the 24 information categories. 


\section{The Approach}\label{sec:approach}
In this section, we present the architecture of our proposed approach in Figure~\ref{fig:i-Aid}. 
We begin our approach by formulating the problem in Section~\ref{sec:problem_formulation}, then discuss the details of each component separately in the following subsections.

\subsection{Preliminaries}\label{sec:problem_formulation}
\subsubsection{Multi-label Classification}
We formulated the problem of identifying crisis information from disaster-related tweets as MLC task, where tweet $t$ can be assigned multiple types (i.e., class, label or category).  
Let $T=\{t_1, t_2,\cdots ,t_n\}$ be a set of disaster-related tweets and $C=\{c_1, c_2,\cdots ,c_k\}$ is a finite set of information types. 


Assume that we have a set of $n$ training samples $\big\{ (x_i,y_i)\big\}_{i=1}^N$.
Where $x_i$ corresponds to the learned features (see Section~\ref{section:tweets_representation}) from input tweet $t_i$ and 
$y_i$ is a set of corresponding label combinations. We define $y_i$ as binary vector $\{y_{i,1},y_{i,2},\cdots ,y_{i,j}\}$, where $y_i\in \{0,1\}$ represents relevance ($1$) or non-relevance ($0$) to label $c_i$. 

Our goal is to learn a function $f(x_i)\rightarrow y_i$ that maps the learned features $x$ to the relevant labels $y=[y_1, y_2,\cdots ,y_m]$. In multi-label classification task, a tweet can be assigned to multiple labels simultaneously. To address this property, we adapt a fine-tuned BERT classifier with an extra \textit{dense} layer, where we use a $sigmoid$ activation function and binary cross entropy as a loss function in Equation.~\ref{equation:binary_cross}.

In contrast to \textit{softmax} activation function -\textit{which returns the label with maximum probability}- a \textit{sigmoid} function returns the probabilities for all labels. In our experiments, we pick up top $k$ labels as a final label combinations (i.e., $y_i$). Where $k$ is a hyper-parameter and is tuned during our experiments~(see Section~\ref{sec:hyper-parameters}).

\subsubsection{Data Preprocessing}\label{sec:preprocessing}
Learning a meaningful and effective representation of text is a critical prerequisite for different NLP tasks. Such text from tweets are inherently noisy, cleaning the text before further processing helps to generate better features and semantics.
We used \textit{tweettokenize} API to preprocess tweets and create uniform representation, in particular we perform the following preprocessing steps.
\begin{itemize}
	\item Stop-words, URLs, usernames and Unicode-characters are removed.
	\item Extra white-spaces, repeated full stops, question marks and exclamation marks are removed.
	\item Emojis were converted to text using the python library \texttt{emoji\footnote{\url{https://pypi.org/project/emoji/}}}
	\item Lemmatization, restoring language vocabulary to general form (can express complete semantics) by \texttt{WordNetLemmatizer}\footnote{\url{https://www.nltk.org/_modules/nltk/stem/wordnet.html}}. 
	\item Finally all tweet tokens are converted to lower-case.
	
\end{itemize}

The preprocessing of the tweet \texttt{"My heart goes out to all those affected by Typhoon Haiyan. You can help by donating to the Phi\-l\-ip\-pi\-ne RED CROSS here (link: \url{http://www.redcross.org}) \url{redcross.org}"} will remove the URL at the end of the tweet and generate the following tokenized word vector:
[\texttt{my, heart, goes, out, to, all, those, affected, by, {typhoon}, {haiyan}, you, can, help, by, donating, to, the, {philippine}, {red}, {cross}, here}].

\subsection{Text Representation}\label{section:tweets_representation}

Learning a good representation of text plays a central role in many NLP applications. 
Recently, vector-based representation (e.g. word2vec) has achieved state-of-the-art performances in different tasks. 
However, this embedding-based representations are generic and obtained from general domain corpus. In addition, 
entities in text are \todo[inline]{Entity embedding..}. 
On the other hand, text from tweets is usually noisy and incomplete. 
Twitter users often tend to spell words based on their pronunciations. Thus, while spellings of a word may differ in their lexical format, they tend to have similar phonetics. 
Based on these insights, we propose a joint embedding of tweet2vec, knowledge graph and phonetics to alleviate the aforementioned problems and represent tweet into a combined feature vector. 
\begin{itemize}

	\item{\textbf{Sentence Embedding}}.
	While word2vec models capture semantic relationship between words, they lose this ability at the sentence level. Sentence representation are usually expressed as the average sum of words vectors in a sentence.
	This bag-of-word2vec approach has major a drawback that different meaning sentences can have identical representations as long as the same words are used and ignore word sequences.
	To alleviate this problem, we propose to embed the whole sentence into a fixed-length, lower dimensional vector representation (sent2vec) using encoder-decoder neural model. This vector capture rich, semantic and latent information that can be train classifiers machine learning tasks.
	
	\item{\textbf{Entity Embedding}}
	A key characteristic of embedding models is their ability to capture semantic relationships. Leveraging this idea, entity-based embedding models have been recently proposed to map entities (i.e, concepts), their types and relationship into a lower-dimensional vector. 
	More robust concept and entity embeddings can be learned from knowledge graphs (e.g. ConceptNet) where \textit{entities} and their relationship are embed into a common semantic space. 
	We identify entities ($e$) in tweets using Spacy API~\footnote{\url{https://spacy.io/}}, then retrieve their corresponding semantic vector from the knowledge graph of ConceptNet.
	
	


	
	
\end{itemize}
\subsection{The I-AID Framework}
We propose a deep neural multi-model with auto-encoders and BERT-based architectures to address the problem of multi-label classification of disaster-related tweets. The basic idea is to learn tweets representation using auto-encoders (i.e. transformers, encoder-decoder representation), in addition to semantics-based and phonetics embedding features extracted from tweets content. The overall architecture of our proposed system is represented in Figure~\ref{fig:i-Aid}, it has three main components:
\begin{itemize}
	\item \textbf{Encoder-Decoder}: It maps input data $(x,y)$ and encoders into into a low-dimensional latent vector ($\mathcal{Z}$. \textbf{Decoder}: It takes a latent vector ($\mathcal{Z}$) and learns to reconstruct back the original tweet text.
	\item \textbf{Fine-tuned BERT Classifier}: It uses the learned vector ($Z$) from auto-encoder, semantics embedding from knowledge graph and phonetics embedding features to classify tweet into multi-label information types.
\end{itemize}

\begin{figure*}
	\centering
	\captionsetup{justification=centering}
	\includegraphics[width=\textwidth]{Figures/I-AID.pdf}
	\caption{The Architecture of I-AID System}
	\label{fig:i-Aid}
\end{figure*}
\subsection{The Encoder-Decoder Module}
The \textit{encoder}-\textit{decoder} model (i.e, auto-encoder) has been shown effective in the field of machine translation. Given an input sequence of tweet $X$, we aim to learn a low-dimensional vector $\mathbf{Z}$ to represent \textit{latent feature} using encoder-decoder architecture. 
\begin{center}
	\texttt{ENC}: $X=\{x_1,x_2,\dots,x_n\}\rightarrow \mathbf{Z}\in\mathbb{R}^d$
\end{center}

The encoding process (\texttt{ENC}) maps input sequence of tokens $x_i$ into a latent vector $\mathbf{Z}\in\mathbb{R}^d$ with $d$ fixed length. 
We employ two bidirectional LSTM layers which process $X$ from both forward and backward directions:
\begin{center}
	$\overleftarrow{h}_t$=LSTM$\left(x_t, \overleftarrow{h}_{t+1}\right)$
	\\
	$\overrightarrow{h}_t$=LSTM$\left(x_t,\overrightarrow{h}_{t+t}\right)$
\end{center}
Where \texttt{LSTM} donates is the long-term short memory unit function which compute hidden vector $h$ by iterating Equations~\ref{eq:bilstm}. Finally, we get the encoded vector $\mathbf{Z}=\left[\overleftarrow{h}_t.\overrightarrow{h}_t\right]$ by concatenating the forward and backward hidden vectors at time $t$.
\begin{equation}\label{eq:bilstm}
\begin{array}{c}
{f_{t}=\sigma\left(x_{t} W_{f}+h_{t-1} U_{f}+b_{f}\right)} \\ {i_{t}=\sigma\left(x_{t} W_{i}+h_{t-1} U_{i}+b_{i}\right)} \\ {o_{t}=\sigma\left(x_{t} W_{o}+h_{t-1} U_{o}+b_{o}\right)} \\ {g_{t}=\tanh \left(x_{t} W_{c}+h_{t-1} U_{c}+b_{c}\right)} \\ {c_{t}=f_{t} \odot c_{t-1}+i_{t} \odot g_{t}} \\ {h_{t}=o_{t} \odot \tanh \left(c_{t}\right)}\end{array}
\end{equation}
where $x_{t}\in\mathbb{R}^{d}$, $h_{t}$, $c_{t}\in\mathbb{R}^{h}$, $W_{f}$, $W_{i}$, $W_{o}$, $W_{c}\in\mathbb{R}^{d x h}$, $U_{f}$, $U_{i}$, $U_{o}$, $U_{c}\in\mathbb{R}^{h x h}$, $b_{f}$, $b_{i}$, $b_{o}$, $b_{c} \in \mathbb{R}^{h}$ ($h$ represents the number of hidden units, and $d$ represents the dimension of the input), $\sigma$ is usually the sigmoid activation function, and $\odot$ is the dot product. 

On the other hand, the \textit{decoder} network employ two bi-directional LSTM layers and learns to to reconstruct back $\hat{X}$ from latent vector $\mathbf{Z}=\left[h_1,h_2\dots, h_d\right]$.
\begin{center}
	\texttt{DEC}: $\mathbf{Z}\in\mathbb{R}^d\rightarrow\hat{X}=\{\hat{x}_1,\hat{x}_2,\dots,\hat{x}_n\}$
\end{center}
To this end, we obtain a fixed-length vector $\mathbf{Z}$ for input $X$ representation using end-to-end training of the \textit{encoder} and \textit{decoder} networks. To minimize the reconstruction loss, we use mean sequence error (MSE) as a loss function in Equation~\ref{eq:mse} to compute losses in between source input $X$ and reconstructed $\hat{X}$ .
\begin{equation}\label{eq:mse}
MSE=\frac{1}{n}\sum_{i=1}^{n}(x_i-\hat{x}_i)^2
\end{equation}


\subsection{BERT Encoder}

BERT~\cite{devlin2018bert} is a bidirectional model of stacked transformers~\cite{vaswani2017attention} that is designed to be easily applied with downstream NLP tasks with fine-tuned manner. 
Since it is trained on general-domain datasets (Wikipedia and BookCorpus). To adapt BERT model in our task as MLC problem, we first investigate the base BERT model, which has 12 layers, 12 attention heads and 786 hidden units. 
Then, we added extra layer which is jointly trained with the \textit{encoder-decoder} model. 
In particular, we add a dense layer on top of BERT, with \textit{sigmoid} activation function. Our fine-tuned BERT classifier takes a combination of the \textit{learned features} as the input tweet representation. 
Second, we minimize the binary cross-entropy loss function 
\begin{equation}\label{equation:binary_cross}
\pazocal{L}(y, \hat{y})=-\big(y\log\hat{y}+(1-y)\log(1-\hat{y})\big)
\end{equation}

\section{Experiments}\label{sec:experiment}
We conducted several experiments to verify the effectiveness of our proposed approach. The goal of our evaluation is to answer the following questions:
\begin{itemize}

	\item[$Q_1.$] To which extent, our approach can identify \textit{actionable information} from disaster-related tweets compared with the state-of-the-art baselines?

	\item[$Q_2.$] How to leverage transfer learning from pretrained language models (e.g, BERT, GPT-2) to generate tweets for imbalanced information types?
\end{itemize}

\subsection{Dataset}
\todo{add statistical analysis of the dataset}
We used the TREC-IS offered by~\cite{mccreadie2019trec} to evaluate the performance of our approach and baselines in identifying actionable information. The TREC-IS dataset contains a various collection of tweets posted during $15$ crisis events (e.g. Wildfire, Earthquake, Flood, Typhoon/Hurricane, Bombing, Shooting). The tweets are labelled into $25$ information types by crowd-sourcing workers and volunteers. 
We randomly split the TREC-IS dataset into subsets $80\%$ for training and $20\%$ for validating the models. Then, we used the test dataset used in TREC-IS 2019B to obtain the experimental results according to different evaluation metrics (See Section~\ref{section:evaluation_metrics}). 

Figure~\ref{fig:tweets-informationTypes} shows number of tweets per each information types (i.e classes) across all events. Apparently, the tweets distribution per information types are highly imbalanced. For instance, classes as \texttt{Sentiment, News} and \texttt{ThirdPartyObservation} have more than $4000$ tweets, in contrast to \texttt{CleanUP, InformationWanted} and \texttt{MovePeople} which have very less tweets.
Training a machine learning model with imbalanced class is a challenging task. To address challenge, we propose to generate extra tweets for lower classes using the state-of-art text augmentation model (GPT-2). In particular, we fine-tune GPT-2 model by feeding tweets from a lower class (e.g. InformationWanted) to generate $1000$ augmented tweets. At the end, we obtained $11k$ extra tweets generated for $11$ lower classes as shown in Figure ??.


\begin{figure}[h!]
	\centering
	\captionsetup{justification=centering}
	\includegraphics[width=0.7\linewidth]{Figures/tweets_per_InformationTypes.pdf}
	\caption{Tweets per Information Types on TREC-IS dataset}
	\label{fig:tweets-informationTypes}
\end{figure}

\subsection{Baselines} \label{sec:baselines}
we compared our proposed approach (i-AID) with different state-of-the-art models in text classification:
\begin{itemize}
	\item \textbf{text-CNN}~\cite{kim2014convolutional}: this model uses a convolutional neural network to model text representation and capture different granularity of text features with multiple convolution filters. 
	\item \textbf{Bi-LSTM}~\cite{zhou2016attention}: A bi-directional LSTM model which parses the text from left-to-right and right-to-left, then uses the final hidden state as a feature representation for the whole text.
	\item \textbf{HAN}~\cite{yang2016hierarchical}: this model use a hierarchical attention neural network framework on textual dataset. In particular, HAN model encodes text with word-level attentions on each sentence and sentence-level attentions on each document. 
	\item \textbf{FastText}~\cite{joulin2016fasttext}: A simple and efficient approach for text classification which treats (word/n)-grams as document embeddings, then feeds into a linear classifier. 
	\item \textbf{Median}: We compared our approach with the median scores obtained from TREC-IS 2019B challenge (include $32$ submitted approaches). 




\end{itemize}








\subsection{Evaluation Metrics}\label{section:evaluation_metrics}
Evaluation metrics for multi-label classification performance are inherently different from those used in multi-class (or binary) classification, due to the inherent differences of the classification problem. 
In the following subsections, we discuss in detail each performance metric used in our evaluation:

\begin{itemize}
	\item \textbf{Accumulated Alert Worth (Alerting)}: the TREC-IS introduced a new metric called Accumulated Alert Worth (AAW) to evaluate the performance of systems based on alerting. In~\cite{trec-paper}, the authors explained in detail how AAW metric evaluates the system performance in generating alerts for disaster-related tweets that contains high or critical information. 
	
	\item \textbf{F1-Marco for Actionable Information}: 
	Macro-F1 measures support for all classes better than accuracy, especially with imbalanced class distributions. This measure evaluate the performance of categorizing tweets when only considering the actionable information types, namely: \texttt{GoodsServices, SearchAndRescue, MovePeople, EmergingThreats, NewSubEvent} and \texttt{ServiceAvailable}. This metric aims to capture whether systems are able to find the \textit{actionable information} within disaster-related tweets.
	
	\item \textbf{Classification Error}: Hamming Loss Function, Let $N$ donates the size of dataset, $C$ is the number of possible classes, $Y_i$ and $\hat{Y}_i$ donates the target and predict classes of data sample $i$ and $\oplus$ donates XOR operator. Then we define the Hamming Loss function as follows: 
	\begin{equation}
	Hamm=\frac{1}{N}\sum_{i=1}^{N}\frac{1}{C} |Y_{i}\oplus\hat{Y}_{i}|
	\end{equation}
\end{itemize}


\subsection{Hyper-parameter Tuning}\label{sec:hyper-parameters}

In our experiments, we tuned the hyper-parameters in our experiments. 
We used grid search to find the best hyper-parameters for our models.
Moreover, we fine-tune the BERT model on 4 TITAN XpGPUs and set the batch size to 24 to ensure that the GPU memory is fully utilized. The dropout probability is always kept at $0.1$. We use Adam with $\beta_1$= 0.9 and $\beta_2$= 0.999. We empirically set the max number of the epoch to 4 and save the best model on the validation set for testing.

\subsection{Discussion and Results Analysis}
\paragraph{\textbf{To answer} $Q_1$}
Table~\ref{tab:AAW} shows the evaluation results based on Alerting performance (AAW) in comparsion to the baselines and median scores on the TREC-IS test dataset. It is observed that our proposed model (I-AID) considerably outperforms the state-of-art baseline by $42\%$ in high priority AAW and $63\%$ in overall AWW. 

Remarkably, 
\paragraph{\textbf{To answer} $Q_2$}

\begin{table}[]
	\centering
	\label{tab:AAW}
	\caption{Performance Evaluation based on Alerting Metric in TREC-IS test dataset}
	\begin{tabular}{|l|l|l|}
		\hline
		\multicolumn{1}{|c|}{\multirow{2}{*}{\textbf{RUNS}}} & \multicolumn{2}{l|}{\textbf{Accumlated Alert Worth}} \\ \cline{2-3} 
		\multicolumn{1}{|c|}{}                      & High Priority             &       All      \\ \hline
		Median                                     &         -0.9197           &    -0.4609       \\ \hline
		textCNN                                     &         -0.9764           &    -0.4884       \\ \hline
		FastText                                    &       -0.7975             &    -0.4217      \\ \hline
		HAN                                         &       -0.7816             &    -0.4600      \\ \hline
		Bi-LSTM                                      &      -0.8760              &    -0.4482      \\ \hline
		UCDbaseline                                 &       -0.7856             &    -0.4131      \\ \hline
		\textbf{I-AID}                              &      \textbf{ 0.2044  }   &    \textbf{0.0468}\\ \hline
	\end{tabular}
\end{table}

\begin{figure}[ht!]
	\centering
	\captionsetup{justification=centering}
	\includegraphics[width=0.7\linewidth]{Figures/F1_per_InformationTypes.pdf}
	\caption{F1-Macro scores per Information Types}
	\label{fig:F1-informationTypes}
\end{figure}




\begin{table*}[h!]
	
	\centering
	\caption{Performance evaluation -\textit{macro-averaged}- of baselines and our system (i-AID).}
	\begin{tabularx}{0.9\textwidth}{@{}XXXXXXX@{}}
		\toprule
		\multirow{2}[2]{*}{\textbf{TREC Dataset}} & \multirow{2}[2]{*}{\textbf{Metrics}}& \multicolumn{4}{c}{\textbf{Baselines}} & \multirow{2}[2]{*}{\textbf{i-AID}} \\ 
		\cmidrule(lr){3-6}
		&  & \textbf{textCNN} & \textbf{FastText} & \textbf{HAN} & \textbf{Bi-LSTM} &  \\
		\cmidrule(r){1-1} \cmidrule(lr){2-2} \cmidrule(lr){3-6} \cmidrule(l){7-7}
		\multirow{5}{*}{Unbalanced} & Precision & 0.116 & 0.155 & 0.076 & \textbf{0.170} & 0.100 \\
		& Recall & 0.331 & 0.199 & 0.319  & 0.373 &  \textbf{0.958}\\
	
		& $F_1$-\textit{all} &  0.134 &  0.138 & 0.118 & \textbf{0.208} &  0.162\\
		& $F_1$-\textit{actionable} & 0.1508 & 0.00 &  0.151 & 0.173 & \textbf{0.194}\\
	
		\cmidrule(r){1-1} \cmidrule(lr){2-2} \cmidrule(lr){3-6} \cmidrule(l){7-7}
		\multirow{5}{*}{Balanced} & Precision & 0.08 & 1.0 & 0.11 & 0.68/0.56 &  \\
		& Recall & 0.64 & 0.05  & 0.11 & 0.68/0.51 &  \\
	
		& $F_1$-\textit{all} &  &  &  &  &  \\
		& $F_1$-\textit{actionable} & 0.34 & 0.09 & 0.09 & 0.68/0.53  &  \\ \bottomrule
	\end{tabularx}
\end{table*}

\section{Conclusion}\label{sec:conclusion}
In this paper, we conduct extensive experiments to investigate the different approaches to fine-tuning BERT for identifying actionable information from disaster-related tweets. In this paper, we propose a novel model that takes advantages of both explicit and implicit representations of short-text. We enrich the features of short texts by conceptualizing them with the help of a well-known knowledge graph.

There  are some experimental findings: 1) The top layer of BERT is more useful for text  classification; 2) With an appropriate layer-wise decreasing learning rate, BERT can overcome the catastrophic for getting problem; 3) Within-task and in domain further pre-training can significantly boost its performance; 4) A preceding multi-task fine-tuning is also helpful to the single-task fine-tuning, but its benefit is smaller than further pre-training; 5) BERT can improve the task with small-size data. With the above findings, we achieve state-of-
the-art performances on eight widely studied text classification datasets.

In the future work, we will employ few-shot learning technique to handle the data imbalances.



\bibliographystyle{plain}
\balance

\section{Introduction}
	Social media platforms have become key portals for sharing information during crisis situations~\cite{DBLP:journals/pacmhci/ZadeSRK0S18}. 
	A major difference between social media and traditional sources is the possibility of receiving feedback from affected people in real-time. 
	Relief organizations such as the \emph{Red Cross} can benefit from this two-way communication channel to inform people and gain insights from situational updates received from actors affected by the crisis.  
	Hence, extracting crisis information from social media can be of great help to leverage situational awareness and deliver faster responses.

	Previous works~\cite{to2017identifying,stowe2018improving} have focused on

	the binary classification of social media text into two categories (e.g., relevant or irrelevant).
	A fine-grained classification, however, would improve the automation of social media data processing.
	For example, by providing the classified tweets to the matching experts.
	The study by Olteanu et.al~\cite{olteanu2015expect} was one of the firsts to investigate tweets generated during different crisis events. 
	The authors concluded that crisis-related tweets tend to contain informative and crucial information types\footnote{In the context of this paper. We use the terms labels, classes and information types synonymously.}, such as affected individuals, infrastructures and utilities, donations and volunteers, caution and advice, sympathy, and emotional support. 
	This level of categorization is valuable for crisis responders to filter critical information quickly and consequently deliver disaster responses.  Table~\ref{tab:informationTypes} shows that disaster-related tweets can be categorized into multi-level information types, which allows the extraction of \textit{actionable} information.
	We consider the same definition of '\textit{actionable}' as formalized in~\cite{DBLP:journals/pacmhci/ZadeSRK0S18}. \textit{Actionable} posts are defined as the ones that would generate an immediate alert for an individual (i.e. stakeholders) responsible for that information type

	(e.g., \textit{SearchAndRescue}, \textit{MovePeople}), in contrast to the \textit{non-actionable} tweets that
	belong to the classes like \textit{Volunteer} or \textit{FirstPartyObservation}.






	
	Short text (e.g., tweets) categorization is known to be a challenging Natural Language Processing (NLP) task~\cite{song2014short}.  


	This is because short text 1) does not possess sufficient word contextual information; 2) is inherently noisy (i.e. contains misspellings, acronyms, emojis, etc.), hence cleaning the text before further processing is essential to capture better features and semantics.

	In the case of Twitter data, the classification task becomes even more challenging because a tweet could belong to multiple information categories simultaneously, thereby making it a multi-label classification problem. 


	To tackle these challenges, previous studies have made significant use of recent results from machine learning and NLP, such as transfer learning from pre-trained models (e.g., ELMo, BERT)~\cite{sun2019fine}.
	While pre-trained models are first trained on a large amount of generic data to capture rich representations and can be fine-tuned into downstream tasks,
	they are not always able to capture the structural information and relationship between the sequence of words in a text. 
	

	In this paper, we aim to identify \textit{actionable or critical} information that might be relevant for disaster relief
	from disaster-related tweets and support disaster mitigation.

	Our approach uses BERT's contextualized sentence-level embeddings to capture the semantics
	of a tweet.
	Furthermore, we employ a Graph Convolutional Network (GCN) to apprehend correlations between entities in tweets and their corresponding labels. 
	In this way, our approach allows each label to derive information from the correlated entities and other labels that were assigned to the same tweet. 

	Therefore, our system integrates the contextualized representation of tweets with the semantic correlations between \textit{named-entities} and labels.
	Finally, the last component in our approach (Relation Network) employs a neural model as a learnable distance metric to compute the similarity between the BERT-vectorized tweets and the label vectors, which are obtained using the Text-GCN approach.

	The main contributions of this paper can be summarized as follows:
	\begin{itemize}
		\item We present a deep neural framework (dubbed I-AID) to identify crisis information from disaster-related tweets. 
		\item Our approach leverages transfer learning from pre-trained language models to embed tweets into low-dimensional vectors. Then, it uses a GCN to capture correlations between entities and labels in a graph-structural manner. 
		\item We employ a \textit{learnable} distance metric to learn the similarity between a tweet's vector and the labels' vectors in a data-driven way. 
		\item
		Our experimental results suggest that our method achieves superior performance over different baseline models (see Section~\ref{sec:baselines}) with up to $+20\%$ in macro-averaged F1-score. 
	\end{itemize}
	
 	The rest of this paper is structured as follows:	In Section~\ref{sec:relatedworks} we discuss previous work on the classification of crisis information on social media. In Section~\ref{sec:approach} we describe the preliminaries and architecture of our proposed approach. 
	
	\begin{table*}[t!]
		\centering 
		\scriptsize
		\caption{Crisis Information Types\label{tab:informationTypes}}
		\begin{tabularx}{\textwidth}{@{}p{2.5cm}p{3.5cm}p{7cm}@{}}
			\toprule
			\textbf{Intent Type }& \textbf{Information Type} & \multicolumn{1}{c}{\textbf{Description}} \\ \midrule
			
			\multirow{3}{*} {REQUEST}& GoodServices & Request for a particular service or physical good \\
			& SearchAndRescue & Request a rescue \\
			& InformationWanted & Request information \\ \midrule
			\multirow{11}{*}{REPORT} & Weather & Weather report \\
			& FirstPartyObservation & Eye-witness account \\
			& ThirdPartyObservation & Information from someone else\\
			& EmergingThreats & Problem that cause loss or damage \\
			& ServiceAvailable & Someone is providing a service \\
			& SignificantEventChange & New occurrence that officers to respond to\\
			& MultimediaShare & Shared images or video \\
			& Factoid & Relate some facts, typically numerical \\
			& Official & Report by a government or public representative \\
			& CleanUp & Report of the clean up after the event \\
			& Hashtags & Report with hashtags correspond to each event \\ \midrule
			\multirow{3}{*}{CALLTOACTION} & Volunteer & Call for volunteers to help in response efforts \\
			& Donations & Call for donation of goods or money \\
			& MovePeople & Call to leave an area or go to another area \\ \midrule
		\end{tabularx}
	\end{table*}
	
	
	\section{Related Work}\label{sec:relatedworks}
	
	\subsection{Extracting Disaster-related Information from Social Media}
	Several research studies have demonstrated the role of social media as a primary source of information during disasters~\cite{cresci2015crisis}. 

	While some works~\cite{PalshikarAP18} have focused on distinguishing disaster-related information from tweets,
	others~\cite{olteanu2015expect,mccreadie2019trec} have introduced annotation schemes to classify disaster-related tweets into more fine-grained categories to account for the attitude, information source, and decision-making behavior of people tweeting before, during, and after disasters.
	In order to advance the state of social media crisis monitoring solutions, several initiatives~\cite{imran2016lrec}
	have been rolled out in recent years. One of them is the Incident Streams (TREC-IS) track~\cite{mccreadie2019trec} of the Text REtrieval Conference, which commenced in 2018.
	The track aims to categorize disaster-related tweets into multiple information types.  Since in this work we study the TREC-IS datasets, we consider \textit{actionable} information to be the same as that defined by the authors in the challenge. 
	
	\subsection{Multi-label Text Classification}
	Several approaches have been proposed to tackle the problem of short-text classification.
	On a high level, to represent text, these approaches either 
	use: 1) explicit, hand-engineered features~\cite{sriram2010short}.
	or 2) neural networks to model the dependencies within the text (implicit features)~\cite{wang2017combining}. 
	Recently, end-to-end deep learning approaches gaining traction and with the introduction of attention mechanisms~\cite{vaswani2017attention}, there has been a significant advancement in text-classification tasks. Recent works~\cite{miyazaki2019label} have also focused on alleviating the tedious process of finding good text-based features and for tackling the multi-label classification problem. Further, pre-trained contextualized word-embeddings, particularly BERT, have proven to be revolutionary for text representation and classification. 
	Previous study~\cite{miyazaki2019label} have leveraged this multi-layer, multi-head self-attention mechanism of BERT, however, since it only captures the local contextual information, it falls short. To circumvent this and  apprehend the global relations among words in a vocabulary, graph-based approaches (Graph Neural Networks
	~\cite{kipf2016semi,yao2019graph} have been found promising. 
	
	Recent studies~\cite{DBLP:conf/ecir/LuDN20,DBLP:conf/icaart/PalSS20} have exploited the advantages of BERT and GCN, while the combination of them
	has previously been applied to the multi-label classification task~\cite{DBLP:conf/icaart/PalSS20}, to the best of our knowledge, it has not been tailored to
	multi-label, short-text (tweet) classification. 

	In VGCN-BERT~\cite{DBLP:conf/ecir/LuDN20}, a GCN is used to capture the correlation between words at the vocabulary level (global information). That is, given a vocabulary, GCN
	would relate the meaning of 'new' to 'innovation' and 'exciting', similar to the context-independent word embeddings like Word2Vec~\cite{DBLP:journals/corr/abs-1301-3781}. 
	For an input sentence, the (local) contextual information is captured using the BERT embeddings, while the global information pertaining to the words in the sentence is extracted from the graph embeddings and concatenated with the sentential BERT embeddings. 
	The two representations of BERT and GCN then interact via the self-attention mechanism to perform a classification task. 
	In contrast, in our approach, we employ GCNs to explicitly infuse the correlation-information of the entities in a tweet and their corresponding labels with the tweet's contextualized BERT-representation. 

	In addition, our approach differs from both MAGNET~\cite{DBLP:conf/icaart/PalSS20} and VGCN-BERT~\cite{DBLP:conf/ecir/LuDN20} in the way it computes the similarity between a tweet's representation and the class labels.
	While the above-mentioned approaches use either a fixed, linear distance metric (dot-product function) or self-attention to measure the similarities, our approach benefits from a deeper end-to-end neural architecture to learn this distance function. 
	In particular, we employ meta-learning to learn
	the mapping between the input features and multi-label output in a data-driven way.
	In the subsequent sections, we will further elaborate on our approach. 


	

	
\section{The Approach}\label{sec:approach}
	
	\subsection{Problem Formulation}\label{sec:problem_formulation}
	Let $T$ be a set of tweets and $\Lambda=\{\lambda_1, \lambda_2,\cdots ,\lambda_k\}$ be a set of predefined labels (also called information types, see Table~\ref{tab:informationTypes}). 
	We formulate the problem of \textit{identifying crisis information from tweets} as a multi-label classification task, where a tweet $t$ can be assigned one or more labels from $\Lambda$ simultaneously.  
	Our task is to learn a function $h: T \to \{0,1\}^k$ that maps tweets to all relevant labels from $\Lambda$. 
	We assume a supervised learning setting. Let $\Sigma$ be the vocabulary of the English language. 
	The training data $\mathcal{D}=\big\{ (t^{(i)},y^{(i)}) \in \Sigma^n \times \{0,1\}^k \big\}_{i=1}^N$ consists of $N$ tweets. 
	Each tweet $t^{(i)}$ in the training set is labelled with a vector of corresponding labels $y^{(i)}$, where $y^{(i)}_j = 1$ means that $t^{(i)}$ belongs to the class $\lambda_j$. Conversely, $y^{(i)}_j = 0$ means that $t^{(i)}$ does not belong to the class $\lambda_j$. Our approach learns the function $h$ by using three neural networks. It first transforms any tweet $t^{(i)}$ into a d-dimensional vector $x^{(i)}$ using a pretrained BERT model. In parallel, it learns label embeddings $l$ using a GCN. These are then concatenated with the tweet embedding $x^{(i)}$. Finally, these vectors are fed to our last component (Relation Network) to determine the correct classification for $t^{(i)}$. 
	
	\subsection{The I-AID Architecture}
	

	\begin{figure*}[t!]
		\centering
		\captionsetup{justification=centering}
		\includegraphics[width=0.95\textwidth]{Figures/I-AID.pdf}
		\caption{The architecture of our proposed I-AID System.}
		\label{fig:I-AID}
	\end{figure*}
	
	\subsubsection{BERT-Encoder.}
	BERT~\cite{devlin2018bert} is a bidirectional model of stacked transformers and designed to be easily applied in different NLP tasks.

   



	We use a BERT-based encoder to represent an input tweet as a contextualized vector. 
	In particular, we employ the BERT-base architecture with $12$ encoder blocks, $768$ hidden dimensions, and $12$ attention heads. We refer the readers to the original BERT paper~\cite{devlin2018bert} for a detailed description of its architecture and input representation. 
	As shown in Figure~\ref{fig:I-AID}, our BERT-Encoder takes tweet $t^{(i)}$ with $m$ tokens $[w^{(i)}_1,w^{(i)}_2,\dots ,w^{(i)}_m]$. A special preprocessing is performed for BERT: \texttt{[CLS]} token is appended to the beginning of the tweet and another token \texttt{(SEP)} is inserted after each sentence as an indicator of sentence boundary. Each token $w_i$ is assigned three kinds of embeddings: token embedding, segmentation embedding, and position embedding. These three embeddings are summed to a single output vector \texttt{[CLS]} which captures the overall meaning of the input tweet.
	\subsubsection{Text Graph Convolutional Network (Text-GCN).}
	Traditional methods (e.g., word2vec) can model the features of a text. However, they ignore the structural information and relationship between the sequence of words in a text~\cite{peng2018large}. 
	The recently proposed graph networks~\cite{yao2019graph} aim to tackle this challenge by representing words as nodes in a graph and model relations (such as co-occurrence or collocation) among them as edges. 
	In this work, we build a graph convolution network (GCN)~\cite{kipf2016semi} from the available training data $\mathcal{D}$. First, we spot \textit{named-entities} in the tweets $t^{(i)} \in \mathcal{D}$
	Then, we compute initial vector representations of nodes using a pre-trained word2vec model.
	Initial vector representations for the labels $\lambda_j$ are retrieved in the same manner. We then compute the \textit{named-entity - named-entity} and \textit{named-entity - label} co-occurrence frequencies. Based on this data, we construct an undirected graph $G = (V, A)$ as follows: the set of nodes $V$ contains all \textit{named-entities} and all labels found in the training data. 
	As we represent all the nodes using embeddings, we can regard $V$ as a $|V| \times d'$-matrix, where $d'$ is the size of the embeddings used for the nodes of $G$. Each entry $a_ij$ of adjacency matrix $A$ of the graph $G$ (dimensions=$|V|\times|V|$) is equal to the co-occurrence frequency of the node $v_i$ with the node $v_j$ within tweets multiplied by the cosine similarity of their embeddings. We write $P$ to mean the degree matrix of $A$ with entries $p_{ii} = \sum\limits_{j} a_{ij}$. 
	
	As shown in Figure~\ref{fig:I-AID}, our Text-GCN component has two graph convolution layers $\pazocal{L}^{(1)}$ and $\pazocal{L}^{(2)}$. Each layer performs a convolution operation on the graph $G$ and computes a new representation for each node by aggregating the embedding vectors from the said node's neighbors. The new $z$-dimensional node feature matrix $\pazocal{L}^{(1)}\in\mathbb{R}^{n\times z}$ is computed as:
	
	\begin{equation}
	\pazocal{L}^{(1)}=\texttt{GCN}\big(\sigma(\hat{A}XW_0)\big)
	\end{equation}
	
	where $\hat{A}=P^{-\frac{1}{2}}AP^{-\frac{1}{2}}$ is a normalized adjacency matrix $A$. 
	$W_0\in\mathbb{R}^{m\times k}$ is a weighted matrix in the first GCN layer. $\sigma$ is the activation function like \texttt{ReLU}. 
	The second GCN layer $\pazocal{L}^{(2)}$ incorporates further information from larger neighborhoods by operating another convolution operation in the graph:
	\begin{equation}
	\pazocal{L}^{(2)}=\texttt{GCN}\big(\sigma(\hat{A}L^1W_1)\big)
	\end{equation}
	
	In this way, our Text-GCN component combines structural information of a node's similarity relations (i.e., entity-entity, entity-label) and semantic information of a word vector representation.
	
	\subsubsection{Relation Network.}\label{relation-network} Finally, our last component aims to learn a similarity metric in a data-driven way. 

	To do so, we employ a neural model as a learnable, non-linear distance function that \textit{learns} how to match similarity (i.e., relation) between the tweet's vector and each label during the training phase.  
	By using a learnable distance function in our Relation Network, we intend to learn an efficient metric to facilitate the mapping between a tweet and the multiple labels that could be assigned to it. Thus, by employing the Relation Network, we eliminate the need to manually choose the right distance metric (e.g. cosine, euclidean, or manhattan).

	Relation Network takes the concatenated vector of BERT-Encoder output with the label vectors as an input. 

	
	For multi-label classification purposes, we use binary cross-entropy, as given in Equation~\ref{eq:binary_cross_entropy}, as a loss function.
	Since our goal is to fetch all the relevant labels, we use sigmoid activation, which computes the probability of each label independently over all possible labels ($\Lambda$), instead of the softmax function that only considers the label with maximum probability.


	\begin{equation}
	\label{eq:binary_cross_entropy}
	\arg\min\frac{1}{k}\sum_{i=1}^{k}\left[y^{(i)}\log(\hat{y}^{(i)})+(1-y^{(i)})\log(1-\hat{y}^{(i})\right]
	\end{equation}
	
	Where $y^{(i)}$ and $\hat{y}^{(i)}$ are the predicted and ground-truth labels of tweet $i$ respectively $k$ is the number of all possible labels.  
	\section{Experiments}\label{sec:experiment}
	We conducted several experiments to verify the effectiveness of our approach on two public datasets. Our goal is to answer the following research questions: 
	\begin{itemize}
	    \item $\boldsymbol{Q_1:}$ 
	    How does our approach perform compare with state-of-the-art multi-label models on short-text classification (e.g. tweets)?
	   
	    \item $\boldsymbol{Q_2:}$ How effective is our approach in identifying tweets with \textit{actionable} information?
	    \item $\boldsymbol{Q_3:}$ How do each component in our approach affect the performance?
	   
	\end{itemize}
		
				
	\begin{table}[t]
		\centering
  		\scriptsize
		\captionsetup{justification=centering}
		\caption{Overview of TREC datasets.}
		\begin{tabularx}{.9\textwidth}{@{}p{3cm}p{1.5cm}p{1.4cm}p{1.7cm}p{3cm}@{}}
			\toprule
			\textbf{Datasets} & \textbf{Train} & \textbf{Test} & \textbf{\#Classes} & \textbf{Avg. tweets/class} \\ \midrule
			TREC-Full &      27,414   &  6,853      & 25        &          3,000    \\
			TREC-High &    13,150    &   3,288     & 12        &        2,000           \\
			TREC-Balanced  &     20,398 &   5,100     & 6         &          7,000    \\ \bottomrule
		\end{tabularx}
		\label{tab:datasets}
	\end{table}
	
	\subsection{Datasets}
	We employed our experiments on two public datasets of disaster-related tweets, provided by TREC~\cite{mccreadie2019trec}, we briefly summarize each dataset as follows:
	\begin{itemize}
	    \item \textbf{TREC-IS}: This dataset contains approximately $35$K tweets collected during $33$ unique disasters between $2012-2018$ (e.g., \textit{wildfires, earthquakes, floods, hurricanes, bombings, and shootings}). 
	The tweets are labeled into $25$ information types by crowdsourcing workers and volunteers. As shown in Figure~\ref{fig:tweets-informationTypes}, the distribution of tweets per information type is highly imbalanced. More than 4,000 tweets are mapped to the information types of \textit{Sentiment}, \textit{News}, and \textit{ThirdPartyObservation.} In contrast, the information types \textit{CleanUP, InformationWanted}, and \textit{MovePeople} have significantly fewer tweets. 
	    \item \textbf{COVID19-Tweets}: This dataset contains a collection of tweets about the COVID-19 outbreak in different affected regions. In total, the data has $7590$ tweets labeled with one or more of the full 25 information type labels (the same as for TREC-IS dataset). 
	   
	\end{itemize}
	

	\begin{figure}[t!]
		\centering
		\small
		\captionsetup{justification=centering}
		\includegraphics[width=0.8\textwidth]{Figures/tweets_per_InformationTypes.pdf}
		\caption{Information Type Distribution per Tweets in TREC-IS dataset.}
		\label{fig:tweets-informationTypes}
	\end{figure}
Table~\ref{tab:datasets} gives an overview of the datasets, including the number of tweets used in training and testing, number of classes, and average number of tweets per class (called information type in the dataset). In each experiment, we split each dataset with a ratio ($80\%-20\%$); where we used $80\%$ of tweets to train the models and $20\%$ for evaluation.
		



	\subsection{Baselines} \label{sec:baselines}
	We compared our approach with the following state-of-the-art baselines for \textit{multi-label text classification:

	\begin{itemize}
		\item \textbf{TextCNN}~\cite{kim2014convolutional}: This approach uses a convolutional neural network to construct text representation. 
	
		\item \textbf{FastText}~\cite{joulin2016bag}: A simple and efficient baseline approach for text classification which processes text as n-grams features and feeds to a linear classifier. 
		\item \textbf{HAN}~\cite{yang2016hierarchical}: This model uses a hierarchical attention-neural-network to encode text with word-level attention on each sentence.
		
		\item \textbf{BiLSTM}~\cite{zhou2016attention}: A bi-directional LSTM model that parses the text from left-to-right and right-to-left, then uses the final hidden state as a feature representation.
		\item \textbf{BERT}~\cite{devlin2018bert}:
		We used a BERT-base model with 
		a transformer architecture (multi-layer, multi-head and self-attention) to represents a tweet into a contextualized embedding vector.
		\item \textbf{Text-GCN}:
		\item \textbf{VGCN-BERT}:
	
		\item \textbf{MAGNET}~\cite{DBLP:conf/icaart/PalSS20}: This approach employs BERT embeddings to represent tweets and GCN for labels classifiers. Then, it employs to use a dot-product function to compute similarities between tweet vector and label vectors.
	
	\end{itemize}
	
	\subsection{Preprocessing and Evaluation Setup}
	For the baseline models: TextCNN, HAN, and BiLSTM, we used the open-source implementations provided by the corresponding authors.  
	For FastText and BERT, we used the implementations from their respective libraries (\textit{fastText} and \textit{Huggingface}).
	Further, we implemented the code of MAGNET model as it has not been open-sourced to date. 
	The baseline methods are set with the hyper-parameters mentioned in the original papers. 

	In our model, we set training-epochs to $200$ with batch-size of $128$ and Adam~\cite{kingma2014adam} 
	optimizer with learning-rate of $2\mathrm{e}{-5}$.
	To avoid overfitting during training, we employed Earlystopping mechanism and a dropout rate of $0.25$. 




	
	\paragraph{Data Preprocessing:} we used the NLTK's TweetTokenize API to preprocess tweets and retain the text content.
	We performed the following preprocessing steps:
	(1) Stop-words, URLs, usernames, and Unicode-characters were removed.
	(2) Extra white-spaces, repeated full stops, question marks, and exclamation marks were removed.
	(3) Emoji
	were converted to text using the emoji python library.
	(4) Finally, SpaCy library is used 
	to extract \textit{named-entities} from tweets.
All our implementations are open-source and available at the project website\footnote{Link is removed due to anonymous review.}
	
	\subsection{Evaluation Metrics}\label{section:evaluation_metrics}

	We adopted macro-averaged F1-score, hamming loss, and Jaccard index to assess the performance of systems in multi-label classification scenario~\cite{chen2017ensemble,yang2018sgm}. 

	










	\begin{itemize}
	    \item \textbf{Jaccard Score.} We used the Jaccard score to compute the similarity between predicted labels $\hat{y}^{(i)}$ and ground truth labels $y^{(i)}$.  





	    
	    \item \textbf{Hamming Loss.}:
	We employed the \textit{hamming loss}
	to estimate the fraction of incorrectly predicted labels out of all predicted labels.  Hence, the smaller the value, the better the performance.  




	

	
	\item\textbf{Accumulated Alert Worth}.
	This is a new metric dubbed \emph{AAW} introduced by TREC-IS~\cite{mccreadie2020incident}  to benchmark the efficiency of detecting crisis information in disaster-related tweets ((i.e., tweets with critical information, see~\cite{mccreadie2019trec} for details)). 
	It is formally defined as:
	\begin{equation}
	\label{eq:AAW}
	AAW=
	\frac{1}{2}\sum_{t\in T}
	\begin{cases}
	\frac{1}{|T_{high/critical}|} \cdot highPriorityWorth(t) & \text{if } t \in T_{high/critical}\\
	\frac{1}{|T_{low/medium}|} \cdot lowPriorityWorth(t) &  \text{otherwise} 
	\end{cases}   
	\end{equation}
	where $T_{high/critical}$ is the set of tweets with high/critical priority,
	$T_{low/medium}$ is the set of tweets with low/medium priority,
	$highPriorityWorth(t)$ is a scoring function for tweets that should generate alerts and
	$lowPriorityWorth(t)$ is a scoring function for tweets that should not generate alerts.



	


	\end{itemize}




\subsection{Discussion}
\subsubsection{Performance Comparison (RQ1)}

	\begin{table*}[t!]
		\centering
		\scriptsize
		\caption{Performance evaluation of baselines and our approach (I-AID) on three datasets from TREC-IS disasters tweets. Best results are in bold.}
		\begin{tabularx}{\textwidth}{@{}p{2cm}p{2.1cm}p{1.2cm}p{1.1cm}p{0.9cm}p{1.1cm}p{0.8cm}p{1.3cm}XXXXXX@{}}
			\toprule
			{\textbf{TREC}} & \multirow{2}[2]{*}{\textbf{Metrics}}& \multicolumn{6}{c}{\textbf{Baselines}} & \multirow{2}[2]{*}{\textbf{I-AID}} \\ 
			\cmidrule(lr){3-8}
			{\textbf{Dataset}}&  & TextCNN & FastText & HAN & BiLSTM & BERT & MAGNET\\
			\cmidrule(r){1-1} \cmidrule(lr){2-2} \cmidrule(lr){3-8} \cmidrule(l){9-9}
			\multirow{5}{*}{TREC-Full} & 
			Precision & 0.158 & 0.332 & 0.099 & 0.212  & 0.124  & 0.279 & \textbf{0.515}\\
			& Recall & 0.439 & 0.101 & \textbf{0.625} & 0.379 & 0.412 & 0.392 & 0.530 \\
			& $F_1$-\textit{score} & 0.205  & 0.123 & 0.167 & 0.219  &  0.177 & 0.317 & \textbf{0.517}  \\
		
			& Jaccard score & 0.237 & 0.280 & 0.138 & 0.242 &  0.221 &  0.373 & \textbf{0.474} \\
			& Hamming Loss & 0.164 & 0.093 & 0.592 & 0.233 &  0.283 &  0.124 & \textbf{0.083} \\
			
			\cmidrule(r){1-1} \cmidrule(lr){2-2} \cmidrule(lr){3-8} \cmidrule(l){9-9}
			\multirow{5}{*}{TREC-High} & 
			Precision & 0.298 & 0.312 & 0.140 & 0.214 & 0.145 & 0.273 & \textbf{0.594}  \\
			& Recall & 0.455 & 0.194 & 0.579 & 0.412  & 0.366 &  0.493 & \textbf{0.625}  \\
			& $F_1$-\textit{Score} & 0.248  &  0.195 & 0.205  &  0.226 &  0.204 & 0.335 & \textbf{0.602}   \\
		
			& Jaccard Score &  0.413 &  0.538 &  0.235 & 0.324 & 0.331 & 0.469 & \textbf{0.717} \\
			& Hamming Loss &  0.140 & 0.092 & 0.415 & 0.222 & 0.516 &  0.161 & \textbf{0.063} \\
			
			\cmidrule(r){1-1} \cmidrule(lr){2-2} \cmidrule(lr){3-8} \cmidrule(l){9-9}
			\multirow{5}{*}{TREC-Balanced} & 
			Precision & 0.425 & \textbf{0.685} & 0.282 & 0.360  & 0.377 & 0.625 & 0.675\\
			& Recall & 0.711 & 0.427 & 0.652 & 0.805 & \textbf{0.756} & 0.583 & 0.732 \\
		
			& $F_1$-\textit{Score} & 0.486 &  0.511 & 0.377 & 0.487 &  0.497 & 0.597 & \textbf{0.702} \\
		
			& Jaccard Score &  0.356 &  0.451 & 0.295 & 0.311 &   0.381 &0.500 & \textbf{0.602} \\
			& Hamming Loss & 0.385 & 0.208 & 0.512 &  0.573 &  0.418 &  0.209 & \textbf{0.172} \\
			
			\bottomrule
		\end{tabularx}
		\label{tab:results}
	\end{table*}
	\begin{figure}[t!]
		\centering
		\captionsetup{justification=centering} 
		\includegraphics[width=0.8\textwidth]{Figures/F1_per_InformationTypes.pdf}
		\caption{Averaged macro F1-scores per information type achieved by I-AID}
		\label{fig:F1-informationTypes}
	\end{figure}
	
	We evaluated the performance of our system against the baselines
	under different metrics, as shown in Table~\ref{tab:results}. 
	To ensure a fair evaluation, we used the same datasets in the training and testing of all models. 
	Table~\ref{tab:results} shows the results on test datasets for all models under six evaluation metrics (\textit{precision, recall, F1-score, accuracy, Jaccard score, and hamming loss}). 
	We consider the macro-averaged F1-score as the primary metric to compare performances, as it takes into account the average performance of each system across all labels in classification tasks. 
	Across all the datasets, our approach (I-AID) achieved significant performance improvements over the other baselines under several metrics. In particular, our approach outperforms the macro-averaged F1-score of MAGNET, the state-of-the-art baseline of multi-label classification, by $+20\%$. 
	As shown in Figure~\ref{fig:F1-informationTypes},
	our approach achieved high
	F1-scores across all the information types except for the highly imbalanced ones (e.g.,\textit{ CleanUP, Discussion, MovePeople)}. 
	Comparing the results from all the metrics under different settings (i.e., with different datasets),
	while our model achieved sub-optimal results under precision and recall metrics, it mostly performed better than the baselines.  
	To this end, our results indicate that I-AID performs fairly well when categorizing disaster-related tweets into multiple information types, this is because of three facts:  
	(1) We constructed a multi-model framework that leverages contextualized embedding from the BERT model to capture contextual information within a tweet. 
	(2) Our approach enriches semantics of the tweet representations by injecting label information and integrating additional structural information between entities in tweets and labels (i.e., entity-entity and entity-label) via GCN. 
	(3)
	Finally, we avoid the manual selection of similarity metric by automatically learning the most efficient distance function in our \emph{Relation Network} (as discussed in Section~\ref{relation-network}) in a data-driven way.

    
    \subsubsection{Actionable Information In Tweets (RQ2)}
    
    To answer $Q_2$ and better understand the behavior of our approach in finding tweets with crisis information, we benchmarked the performance of all the relevant approaches using AAW metric (Equation~\ref{eq:AAW}) which demonstrates the system's ability in detecting tweets with critical or actionable information. 
	There are two ways to define an \textit{actionable} tweet: 
	(1) In terms of high priority information, commonly marked as critical by human assessors. 
	(2) In terms of information type, for instance, a tweet with the labels \textit{MovePeople} or \textit{CleanUP} is considered more actionable than \textit{News} or \textit{Multimediashare}. Table~\ref{tab:AAW} shows the evaluation results of I-AID compared with the baselines on the test dataset (the top 4 rows). 
	The rest of Table~\ref{tab:AAW} contains the best results from all systems that participated in TREC-IS 2019~\cite{mccreadie2019trec}.
	Our model (I-AID) significantly outperforms all baselines.
	In particular, I-AID achieved improvement by $+32\%$ in high priority AWW compared with the best-achieved result in TREC-IS by \textit{nyu-smap}. 
	Remarkably, our approach is the first to achieve a positive AAW on high-priority tweets. 
	Although we outperform the state of the art in both classification and AAW, the results of our evaluation suggest that a significant amount of research is still necessary to spot high-priority tweets in a satisfactory manner. 
	
	\begin{table}[t!]
		\centering
		\scriptsize
	
	
		\caption{Performance evaluation under AAW metric on the test dataset from TREC-IS 2018. A higher AAW value indicates better prediction.
		\begin{tabularx}{0.75\textwidth}{@{}p{3.5cm}p{2.5cm}p{1.3cm}@{}}
			\toprule
			\multirow{2}[2]{*}{\textbf{Systems}} & \multicolumn{2}{l}{\textbf{Accumulated Alert Worth (AAW)}} \\ \cmidrule(r){2-3} 
			& High Priority             &       All      \\ \midrule
			
			TextCNN                                     &         -0.9764           &    -0.4884       \\ 
			FastText                                    &       -0.7975             &    -0.4217      \\
			HAN                                         &       -0.7816             &    -0.4600      \\
			Bi-LSTM                                      &      -0.8760              &    -0.4482     
			\\
			BERT (UPB\_BERT)  &         -0.9680         &       -0.4882     \\  
			MAGNET          &    -0.9436         &              -0.4726
			\\ 
			I-AID                              &     \textbf{0.2044} &    \textbf{-0.1509}\\
			\midrule
			Median                                     &         -0.9197           &    -0.4609       \\
			BJUTDMS-run2                                &      -0.9942                  &   -0.4971 \\
			IRIT         &         -0.9942      &                   -0.4971  \\
			irlabISIBase &        -0.2337      &                -0.4935     \\
			UCDbaseline                                 &       -0.7856             &    -0.4131      \\
			nyu-smap &                  -0.1213 &               -0.1973 \\ 
			
			SC-KRun28482low &                   -0.9905 &                     -0.4955\\
			xgboost &                   -0.9942 &                                 -0.4972\\
			UCDrunEL2           &        -0.8556   &  -0.4382 \\
		
			cmu-rf-autothre &       -0.8481     &       -0.4456 \\
			
			\bottomrule
		\end{tabularx}
		\label{tab:AAW}
	\end{table}
	
\subsubsection{Ablation Study (RQ3)}

\section{Conclusion}\label{sec:conclusion}

We proposed a multi-model framework (dubbed I-AID) for identifying actionable information from crisis tweets. 
Our system combines three components: BERT Encoder, Text-GCN, and Relation Network. 
BERT Encoder is used to obtain local information, while the Text-GCN module aims to find correlations between named-entities in tweets and their corresponding labels. 
Finally, we use the Relation Network as a final output layer to learn how important each label is in response to the content of tweets and to output a set of relevant labels. 
Our experimental findings are: 
1) Combining local information captured by BERT-Encoder and global information by Text-GCN is beneficial for rich representation in short-text and significantly advances multi-label classification. 
2) Leveraging transfer-learning from pre-trained language models can efficiently handle sparsity and noise in social media data. 
3) Benchmarking multi-label classification is a challenging task, which requires proper evaluation metrics for fine-grained evaluation.
Note that, for an almost-balanced dataset, 
I-AID achieved its best macro-averaged F-score of $0.70$. This clearly indicates the sensitivity of our approach to the dataset's balancing. 
In our future work, we plan to use natural language generation models to address this problem. 
\bibliographystyle{plain}

\section{Introduction}
\begin{figure}[t!]
	\centering
	\includegraphics[width=0.7\linewidth]{Figures/sampletweet.png}
	\caption{Example of Tweets Categorization}
	\label{fig:tweet_example}
\end{figure}
Social media platforms become key portals for sharing information during crisis situations.
Hence, extracting useful information from social media is a major process to leverage situational awareness and deliver faster crisis responses. 
Several approaches can help social media users, emergency managers and relief organizations. Previous studies focused on an \textit{abstract-level} of filtering information (e.g relevant/irrelevant) from a massive social media posts. However, there is a lack of efficient tools that can filter informative disaster-related tweets in a deeper and fine-grained categories (e.g. finding affected people, ask for donations, analysing infrastructure damage).

In a primary study, Olteanu \textit{et.al}~\cite{olteanu2015expect} studied a collection of tweets during different crisis events, the authors found that crisis-related tweets tend to bare one of the following general information categories: \emph{affected individuals, infrastructures and utilities, donations and volunteer, caution and advice, sympathy and emotional support, other useful information}. This level of information categorization is useful for crisis managers as well as first responders to quickly deliver disaster responses. As shown in Table~\ref{tab:informationTypes}, disaster-related tweets can be categorized into multi-level information types (i.e. labels) to extract actionable information and crisis mapping. In this work, we study how to leverage social media posts (e.g., tweets) during disasters to improve situational awareness. In particular, we aim to extract \textit{actionable information} from disaster-related tweets to support such as disaster decisions such as move people, ask for donations or volunteer. 

Short text categorization is gaining more and more attention due to the availability of data (e.g. tweets). Although, traditional text classification methods perform well on long text (e.g, document, articles). However short text (e.g. tweets) have particular challenges like its messy nature, misspellings, grammatical errors, abbreviations, slang, unusual terms, emojis, and noise. Researchers have recently made some great efforts at tackling these challenges using pretrained embedding models (e.g. EMLO, BERT, GPT-2) and \textit{transfer-learning}. In this work, we propose to leverage \textit{transfer-learning} from pretrained language models (BERT) and other language resources as knowledge graph for conceptualized representation of text. 
These pretrained models are first trained on large amount of unlabeled data to capture rich representations, and then applied to downstream tasks such as text classification, word sense disambiguation or sentiment analysis.

In this work, we propose a semantics-based BERT model (dubbed as CrisisBERT) to classify crisis tweets into multiple information types. Our Crisis model represent features from tweet content into three folds: (1) Contextualized word embedding from BERT model, (2) Leverage entity embedding from knowledge graph and (3) Phonological representation using phonetics embedding. Finally, we employ two bidirectional LSTM layers at the output to assign relevant information types for each tweet. 
The main contributions in this paper can be summarized as follows:
\begin{itemize}
	\item We present a semantics-aware BERT model to identify actionable information in disaster-related tweets. 
	\item Our proposed model (\textit{dub} crisisBERT) is an end-to-end based framework that learns combined feature representation of a tweet from BERT embedding, knowledge graph and phonetics embedding. 
	\item The use of \textit{phonological representations} to encode a sentence. 
	\item We conduct extensive experiments to show the superiority of the proposed ensemble learning method. Our method significantly outperforms different baseline models (see Section~\ref{sec:baselines} up to $5\%$.
\end{itemize}

All our implementations are open-source and available from the project website\footnote{
	\url{https://github.com/dice-group/TREC-IS19}}.

The remaining of this paper is structured as follows: Section discusses the previous works in the classification crisis information on social media. Section  and describes the preliminaries and architectures of our proposed approach.  Finally, Section shows the experimental setup, result analysis and discussion of our experiments. Section concludes the paper.



\section{Related Work}\label{sec:relatedworks}
The aim of this work is to categorize disaster-related tweets into multiple information types. Thus, our work is related to \textit{short text categorization, multilabel classification, pretrained embedding models}. In the following, we briefly discuss the state of the art in each of these areas in turn. 

\subsection{Short Text Classification}
A lot of work has been done to learn from the short-textual nature of tweets. Wang et al. ~\cite{wang2017combining} conceptualized tweets using a knowledge base and then by enriching them with word and sub-word information, proposed a CNN-based framework for classification. Along the same lines, Burel et al.~\cite{burel2017semantic} extracted conceptual information (from a knowledge base) about named entities in text 
and by employing a deep CNN model, identified the information category of crisis-related tweets. To alleviate the tedious process of finding good text-based features, Kundu et al.~\cite{KunduSD18} trained an end-to-end deep learning based model to classify tweets into different actionable classes. They used a 5-layer network; the first hidden layer composed of Bidirectional LSTM to encode the contextual information, followed by 3 more layers of vanilla LSTM. In their experiments against FIRE2016 and FIRE2017 datasets, this model outperformed the tf-idf and CNN based models. Caragea et al.~\cite{caragea2016identifying} proposed a CNN for text classification to identify informative
tweets during disaster events. Their experiments against a subset of CrisisLexT26 dataset show that Convolutional Neural Networks outperform other supervised learning algorithms like Support Vector Machines and Artificial Neural Networks, which use a combination of unigrams, bigrams and trigrams as input features. A similar work ,\cite{NguyenAJSIM17}, also concludes CNNs to be the most effective in classifying tweets of new event, when no event-specific data is available, using available out-of-event data. 
\subsection{Social Sensing}
Several research studies demonstrate the role of social media as a primary source of information during disasters. In particular, social media (e.g Twitter) has been used to leverage situational awareness and crisis mapping~\cite{yin2015using,cameron2012emergency,cresci2015crisis}. For example, to distinguish disaster-related information from others on social media, Palshikar et al.~\cite{DBLP:journals/isf/PalshikarAP18} employed weak supervision to learn a bag of words model from news corpus of various disasters. The word model was then augmented using a corpus of unlabelled tweets, followed by a method of learning word-weights online to dynamically improve the model in run-time setting. This approach could effectively classify completely unseen stream of tweets. Stowe et al.~\cite{stowe2016identifying} proposed an annotation schema to classify disaster-related tweets into six fine-grained categories to account for the attitude, information source and decision-making behavior of people tweeting before, during and after disasters. They also trained feature-rich binary classifiers for each of the six categories independently and performed an ablation study to understand which features contributed to the relevance and fine-grained categorization of tweets.
Going a step ahead, ~\cite{ToriumiB16} developed a real-time information sharing system using a clustering-based tweet classification approach to aid in emanating awareness during disasters. The system considered tweets that were retweeted more than 100 times and leveraged this to create a retweet network to connect topic-similar tweets. Extending the work of ~\cite{ZhangWGH16} for extracting keypharses from tweets, Chowdhury et al.~\cite{ChowdhuryCC19} proposed an approach for keyphrase extraction from disaster-related tweets. By utilizing contextual word embeddings, POS-tags, phonetics, and phonological features, they improved the model's performance on both general Twitter data and disaster-related tweets.  

On the other hand, for improving awareness pertaining to socio-economic recovery post disasters, Shibuya et al.~\cite{ShibuyaT19} considered an increased demand for used cars as an indicator of socio-economic recovery, and correlated public sentiment on social media with market data to gather insights. To help curb rumors and fake news during disasters, ~\cite{BarnwalGKBG19} proposed a hybrid approach based on classification and ranking for identifying fact-checkable tweets during disasters. To understand the similarities and differences among different crisis events and study the impact of certain tweet characteristics on classification based on disaster-informativeness, a systematic analysis of the CrisisLexT26~\cite{olteanu2015expect} dataset was performed by Graf et al.~\cite{GrafRSPK18}. Based on this analysis, they trained a cross-domain classifier, which could classify tweets across disaster events of different types, with a set of 29 features representing the temporal, linguistic and source characteristics of a tweet.  

\subsection{End-to-End Learning Models}
\todo[inline]{Kind of features extract from tweets. Kind of deep learning models}
Our system is an end-to-end neural architectures for learning semantic representation of data.

Our primary research aims to leverage multi-label based classifiers that can identify low-level information types (e.g. donation, volunteer, evacuation) from disaster-related tweets. Exploiting the Label Embedding method for text classification~\cite{ZhangXCWJ18}, Miyazaki et al.~\cite{MiyazakiMTOG19} apprehend the hierarchical structure of labels present in the TREC-IS 2018 dataset as a key feature. They vectorize the tweet text and the corresponding higher and lower-level label using pre-trained models such as BERT and ELMo. These embeddings are then fed to a BiLSTM network that outputs three matrices for text, lower-level label, and higher-level label respectively. These matrices are then used for calculating label attention and hierarchical attention vectors, which are then passed to a 2-layer feed-forward network for classification of the tweet into one of the 24 information categories. 


\section{The Approach}\label{sec:approach}
In this section, we present the architecture of our proposed approach in Figure~\ref{fig:i-Aid}. 
We begin our approach by formulating the problem in Section~\ref{sec:problem_formulation}, then discuss the details of each component.

\subsection{Problem Definition}\label{sec:problem_formulation}
We formulated the problem of identifying relevant information from disaster-related tweets as multi-label classification task, where each tweet $t$ can be assigned multiple information types (i.e., class, label or category) based on tweet context. 
The rest of this section, we present some base definitions, then we define the task of identifying actionable information from disaster-related tweets.
Let $T=\{t_1, t_2,\cdots ,t_n\}$ be a set of disaster-related tweets and $C=\{c_1, c_2,\cdots ,c_k\}$ is a finite set of information types (also called classes or labels). 
Assume that we have a set of $n$ training samples $\big\{ (x_i,y_i)\big\}_{i=1}^N$.
Where $x_i$ corresponds to the learned features (see Section~\ref{section:tweets_representation}) from input tweet $t_i$ and 
$y_i$ is a set of corresponding label combinations. We define $y_i$ as binary vector $\{y_{i,1},y_{i,2},\cdots ,y_{i,j}\}$, where $y_i\in \{0,1\}$ represents relevance ($1$) or non-relevance ($0$) to label $c_i$. 

Our goal is to learn a function $f(x_i)\rightarrow y_i$ that maps the learned features $x$ to the relevant labels $y=[y_1, y_2,\cdots ,y_m]$. In multi-label classification task, a tweet can be assigned to multiple labels simultaneously. To address this property, we adapt a fine-tuned BERT classifier with an extra \textit{dense} layer, where we use a $sigmoid$ activation function and binary cross entropy as a loss function in Equation.~\ref{equation:binary_cross}.

In contrast to \textit{softmax} activation function -\textit{which returns the label with maximum probability}- a \textit{sigmoid} function returns the probabilities for all labels. In our experiments, we pick up top $k$ labels as a final label combinations (i.e., $y_i$). Where $k$ is a hyper-parameter and is tuned during our experiments~(see Section~\ref{sec:hyper-parameters}).

\subsection{Data Preprocessing}\label{sec:preprocessing}
Learning a meaningful and effective representation of text is a critical prerequisite for different NLP tasks. Such text from tweets are inherently noisy, cleaning the text before further processing helps to generate better features and semantics.
We used \textit{tweettokenize} API to preprocess tweets and create uniform representation, in particular we perform the following preprocessing steps.
\begin{itemize}
	\item Stop-words, URLs, usernames and Unicode-characters are removed.
	\item Extra white-spaces, repeated full stops, question marks and exclamation marks are removed.
	\item Emojis were converted to text using the python library \texttt{emoji\footnote{\url{https://pypi.org/project/emoji/}}}
	\item Lemmatization, restoring language vocabulary to general form (can express complete semantics) by \texttt{WordNetLemmatizer}\footnote{\url{https://www.nltk.org/_modules/nltk/stem/wordnet.html}}. 
	\item Finally all tweet tokens are converted to lower-case.
	
\end{itemize}

The preprocessing of the tweet \texttt{"My heart goes out to all those affected by Typhoon Haiyan. You can help by donating to the Phi\-l\-ip\-pi\-ne RED CROSS here (link: \url{http://www.redcross.org}) \url{redcross.org}"} will remove the URL at the end of the tweet and generate the following tokenized word vector:
[\texttt{my, heart, goes, out, to, all, those, affected, by, {typhoon}, {haiyan}, you, can, help, by, donating, to, the, {philippine}, {red}, {cross}, here}].

\subsection{Text Representation}\label{section:tweets_representation}

Learning a good representation of text plays a central role in many NLP applications. 
Recently, vector-based representation (e.g. word2vec) has achieved state-of-the-art performances in different tasks. 
However, this embedding-based representations are generic and obtained from general domain corpus. In addition, 
entities in text are \todo[inline]{Entity embedding..}. 
On the other hand, text from tweets is usually noisy and incomplete. 
Twitter users often tend to spell words based on their pronunciations. Thus, while spellings of a word may differ in their lexical format, they tend to have similar phonetics. 
Based on these insights, we propose a joint embedding of tweet2vec, knowledge graph and phonetics to alleviate the aforementioned problems and represent tweet into a combined feature vector. 
\begin{itemize}

	\item{\textbf{Sentence Embedding}}.
	While word2vec models capture semantic relationship between words, they lose this ability at the sentence level. Sentence representation are usually expressed as the average sum of words vectors in a sentence.
	This bag-of-word2vec approach has major a drawback that different meaning sentences can have identical representations as long as the same words are used and ignore word sequences.
	To alleviate this problem, we propose to embed the whole sentence into a fixed-length, lower dimensional vector representation (sent2vec) using encoder-decoder neural model. This vector capture rich, semantic and latent information that can be train classifiers machine learning tasks.
	
	\item{\textbf{Entity Embedding}}
	A key characteristic of embedding models is their ability to capture semantic relationships. Leveraging this idea, entity-based embedding models have been recently proposed to map entities (i.e, concepts), their types and relationship into a lower-dimensional vector. 
	More robust concept and entity embeddings can be learned from knowledge graphs (e.g. ConceptNet) where \textit{entities} and their relationship are embed into a common semantic space. 
	We identify entities ($e$) in tweets using Spacy API~\footnote{\url{https://spacy.io/}}, then retrieve their corresponding semantic vector from the knowledge graph of ConceptNet.
	
	


	
	\item{\textbf{Phonetics Embedding}} 
	Tweets often include short-hands, slang and arbitrary capitalization which increases overall nosiness.
	For example suppose the following tweets: 
	\begin{center}
		Tweet $A$: 'u hav to b KIDDDDING me'\\
		Tweet $B$: 'you have to be kidding me'
	\end{center}
	Although both sentences have same meaning, Tweet $A$ has multiple typos and considered as nosiness and cleaned during preprocessing (see Section~\ref{sec:preprocessing}). 
	To alleviate this problem, we proposed to employ phonetics embedding to model this kind of noisness in tweets. 
	Following the approach~\cite{aguilar2019modeling} that model the noisiness in tweets and create more normalized representations of words at character-level using their phonological features. Both sentences have same phonetics 
	\begin{center}
		Phonetics representation of Tweet A and B: $/ju hÊv t@ bi kIdIN mi/$    
	\end{center}
	
	
	In particular, we use \texttt{Epitran}\footnote{\url{https://github.com/dmort27/epitran}} library to map \textit{graphemes} to \textit{phonemes} with their corresponding international phonetic alphabetic (IPA). 
	Then, we use \texttt{PanPhon}\footnote{\url{https://github.com/dmort27/panphon}} library to convert each IPA phoneme into their phonological vector representation. 
	These features provide articulatory information such as the way the mouth and nasal areas are involved in the elaboration of sounds while people speak.
	
	
	

	
	Joint Embedding:
	
\end{itemize}
\subsection{The I-AID Framework}
We propose a deep neural multi-model with auto-encoders and BERT-based architectures to address the problem of multi-label classification of disaster-related tweets. The basic idea is to learn tweets representation using auto-encoders (i.e. transformers, encoder-decoder representation), in addition to semantics-based and phonetics embedding features extracted from tweets content. The overall architecture of our proposed system is represented in Figure~\ref{fig:i-Aid}, it has three main components:
\begin{itemize}
	\item \textbf{Encoder-Decoder}: It maps input data $(x,y)$ and encoders into into a low-dimensional latent vector ($\mathcal{Z}$. \textbf{Decoder}: It takes a latent vector ($\mathcal{Z}$) and learns to reconstruct back the original tweet text.
	\item \textbf{Fine-tuned BERT Classifier}: It uses the learned vector ($Z$) from auto-encoder, semantics embedding from knowledge graph and phonetics embedding features to classify tweet into multi-label information types.
\end{itemize}

\begin{figure*}
	\centering
	\captionsetup{justification=centering}
	\includegraphics[width=\textwidth]{Figures/IAID.pdf}
	\caption{The Architecture of I-AID System}
	\label{fig:i-Aid}
\end{figure*}
\subsection{The Encoder-Decoder Module}
The \textit{encoder}-\textit{decoder} model (i.e, auto-encoder) has been shown effective in the field of machine translation. Given an input sequence of tweet $X$, we aim to learn a low-dimensional vector $\mathbf{Z}$ to represent \textit{latent feature} using encoder-decoder architecture. 
\begin{center}
	\texttt{ENC}: $X=\{x_1,x_2,\dots,x_n\}\rightarrow \mathbf{Z}\in\mathbb{R}^d$
\end{center}

The encoding process (\texttt{ENC}) maps input sequence of tokens $x_i$ into a latent vector $\mathbf{Z}\in\mathbb{R}^d$ with $d$ fixed length. 
We employ two bidirectional LSTM layers which process $X$ from both forward and backward directions:
\begin{center}
	$\overleftarrow{h}_t$=LSTM$\left(x_t, \overleftarrow{h}_{t+1}\right)$
	\\
	$\overrightarrow{h}_t$=LSTM$\left(x_t,\overrightarrow{h}_{t+t}\right)$
\end{center}
Where \texttt{LSTM} donates is the long-term short memory unit function which compute hidden vector $h$ by iterating Equations~\ref{eq:bilstm}. Finally, we get the encoded vector $\mathbf{Z}=\left[\overleftarrow{h}_t.\overrightarrow{h}_t\right]$ by concatenating the forward and backward hidden vectors at time $t$.
\begin{equation}\label{eq:bilstm}
\begin{array}{c}
{f_{t}=\sigma\left(x_{t} W_{f}+h_{t-1} U_{f}+b_{f}\right)} \\ {i_{t}=\sigma\left(x_{t} W_{i}+h_{t-1} U_{i}+b_{i}\right)} \\ {o_{t}=\sigma\left(x_{t} W_{o}+h_{t-1} U_{o}+b_{o}\right)} \\ {g_{t}=\tanh \left(x_{t} W_{c}+h_{t-1} U_{c}+b_{c}\right)} \\ {c_{t}=f_{t} \odot c_{t-1}+i_{t} \odot g_{t}} \\ {h_{t}=o_{t} \odot \tanh \left(c_{t}\right)}\end{array}
\end{equation}
where $x_{t}\in\mathbb{R}^{d}$, $h_{t}$, $c_{t}\in\mathbb{R}^{h}$, $W_{f}$, $W_{i}$, $W_{o}$, $W_{c}\in\mathbb{R}^{d x h}$, $U_{f}$, $U_{i}$, $U_{o}$, $U_{c}\in\mathbb{R}^{h x h}$, $b_{f}$, $b_{i}$, $b_{o}$, $b_{c} \in \mathbb{R}^{h}$ ($h$ represents the number of hidden units, and $d$ represents the dimension of the input), $\sigma$ is usually the sigmoid activation function, and $\odot$ is the dot product. 

On the other hand, the \textit{decoder} network employ two bi-directional LSTM layers and learns to to reconstruct back $\hat{X}$ from latent vector $\mathbf{Z}=\left[h_1,h_2\dots, h_d\right]$.
\begin{center}
	\texttt{DEC}: $\mathbf{Z}\in\mathbb{R}^d\rightarrow\hat{X}=\{\hat{x}_1,\hat{x}_2,\dots,\hat{x}_n\}$
\end{center}
To this end, we obtain a fixed-length vector $\mathbf{Z}$ for input $X$ representation using end-to-end training of the \textit{encoder} and \textit{decoder} networks. To minimize the reconstruction loss, we use mean sequence error (MSE) as a loss function in Equation~\ref{eq:mse} to compute losses in between source input $X$ and reconstructed $\hat{X}$ .
\begin{equation}\label{eq:mse}
MSE=\frac{1}{n}\sum_{i=1}^{n}(x_i-\hat{x}_i)^2
\end{equation}


\subsection{BERT Encoder}

BERT~\cite{devlin2018bert} is a bidirectional model of stacked transformers~\cite{vaswani2017attention} that is designed to be easily applied with downstream NLP tasks with fine-tuned manner. 
Since it is trained on general-domain datasets (Wikipedia and BookCorpus). To adapt BERT model in our task as MLC problem, we first investigate the base BERT model, which has 12 layers, 12 attention heads and 786 hidden units. 
Then, we added extra layer which is jointly trained with the \textit{encoder-decoder} model. 
In particular, we add a dense layer on top of BERT, with \textit{sigmoid} activation function. Our fine-tuned BERT classifier takes a combination of the \textit{learned features} as the input tweet representation. 
Second, we minimize the binary cross-entropy loss function 
\begin{equation}\label{equation:binary_cross}
\pazocal{L}(y, \hat{y})=-\big(y\log\hat{y}+(1-y)\log(1-\hat{y})\big)
\end{equation}

\section{Experiments}\label{sec:experiment}
We conducted several experiments to verify the effectiveness of our proposed approach. The goal of our evaluation is to answer the following questions:
\begin{itemize}

	\item[$Q_1.$] To which extent, our approach can identify \textit{actionable information} from disaster-related tweets compared with the state-of-the-art baselines?

	\item[$Q_2.$] How to leverage transfer learning from pretrained language models (e.g, BERT, GPT-2) to generate tweets for imbalanced information types?
\end{itemize}

\subsection{Dataset}
\todo{add statistical analysis of the dataset}
We used the TREC-IS offered by~\cite{mccreadie2019trec} to evaluate the performance of our approach and baselines in identifying actionable information. The TREC-IS dataset contains a various collection of tweets posted during $15$ crisis events (e.g. Wildfire, Earthquake, Flood, Typhoon/Hurricane, Bombing, Shooting). The tweets are labelled into $25$ information types by crowd-sourcing workers and volunteers. 
We randomly split the TREC-IS dataset into subsets $80\%$ for training and $20\%$ for validating the models. Then, we used the test dataset used in TREC-IS 2019B to obtain the experimental results according to different evaluation metrics (See Section~\ref{section:evaluation_metrics}). 

Figure~\ref{fig:tweets-informationTypes} shows number of tweets per each information types (i.e classes) across all events. Apparently, the tweets distribution per information types are highly imbalanced. For instance, classes as \texttt{Sentiment, News} and \texttt{ThirdPartyObservation} have more than $4000$ tweets, in contrast to \texttt{CleanUP, InformationWanted} and \texttt{MovePeople} which have very less tweets.
Training a machine learning model with imbalanced class is a challenging task. To address challenge, we propose to generate extra tweets for lower classes using the state-of-art text augmentation model (GPT-2). In particular, we fine-tune GPT-2 model by feeding tweets from a lower class (e.g. InformationWanted) to generate $1000$ augmented tweets. At the end, we obtained $11k$ extra tweets generated for $11$ lower classes as shown in Figure ??.


\begin{figure}[h!]
	\centering
	\captionsetup{justification=centering}
	\includegraphics[width=0.7\linewidth]{Figures/tweetsperInformationTypes.pdf}
	\caption{Tweets per Information Types on TREC-IS dataset}
	\label{fig:tweets-informationTypes}
\end{figure}

\subsection{Baselines} \label{sec:baselines}
we compared our proposed approach (i-AID) with different state-of-the-art models in text classification:
\begin{itemize}
	\item \textbf{text-CNN}~\cite{kim2014convolutional}: this model uses a convolutional neural network to model text representation and capture different granularity of text features with multiple convolution filters. 
	\item \textbf{Bi-LSTM}~\cite{zhou2016attention}: A bi-directional LSTM model which parses the text from left-to-right and right-to-left, then uses the final hidden state as a feature representation for the whole text.
	\item \textbf{HAN}~\cite{yang2016hierarchical}: this model use a hierarchical attention neural network framework on textual dataset. In particular, HAN model encodes text with word-level attentions on each sentence and sentence-level attentions on each document. 
	\item \textbf{FastText}~\cite{joulin2016fasttext}: A simple and efficient approach for text classification which treats (word/n)-grams as document embeddings, then feeds into a linear classifier. 
	\item \textbf{Median}: We compared our approach with the median scores obtained from TREC-IS 2019B challenge (include $32$ submitted approaches). 




\end{itemize}








\subsection{Evaluation Metrics}\label{section:evaluation_metrics}
Evaluation metrics for multi-label classification performance are inherently different from those used in multi-class (or binary) classification, due to the inherent differences of the classification problem. 
In the following subsections, we discuss in detail each performance metric used in our evaluation:

\begin{itemize}
	\item \textbf{Accumulated Alert Worth (Alerting)}: the TREC-IS introduced a new metric called Accumulated Alert Worth (AAW) to evaluate the performance of systems based on alerting. In~\cite{trec-paper}, the authors explained in detail how AAW metric evaluates the system performance in generating alerts for disaster-related tweets that contains high or critical information. 
	
	\item \textbf{F1-Marco for Actionable Information}: 
	Macro-F1 measures support for all classes better than accuracy, especially with imbalanced class distributions. This measure evaluate the performance of categorizing tweets when only considering the actionable information types, namely: \texttt{GoodsServices, SearchAndRescue, MovePeople, EmergingThreats, NewSubEvent} and \texttt{ServiceAvailable}. This metric aims to capture whether systems are able to find the \textit{actionable information} within disaster-related tweets.
	
	\item \textbf{Classification Error}: Hamming Loss Function, Let $N$ donates the size of dataset, $C$ is the number of possible classes, $Y_i$ and $\hat{Y}_i$ donates the target and predict classes of data sample $i$ and $\oplus$ donates XOR operator. Then we define the Hamming Loss function as follows: 
	\begin{equation}
	Hamm=\frac{1}{N}\sum_{i=1}^{N}\frac{1}{C} |Y_{i}\oplus\hat{Y}_{i}|
	\end{equation}
\end{itemize}


\subsection{Hyper-parameter Tuning}\label{sec:hyper-parameters}

In our experiments, we tuned the hyper-parameters in our experiments. 
We used grid search to find the best hyper-parameters for our models.
Moreover, we fine-tune the BERT model on 4 TITAN XpGPUs and set the batch size to 24 to ensure that the GPU memory is fully utilized. The dropout probability is always kept at $0.1$. We use Adam with $\beta_1$= 0.9 and $\beta_2$= 0.999. We empirically set the max number of the epoch to 4 and save the best model on the validation set for testing.

\subsection{Discussion and Results Analysis}
\paragraph{\textbf{To answer} $Q_1$}
Table~\ref{tab:AAW} shows the evaluation results based on Alerting performance (AAW) in comparsion to the baselines and median scores on the TREC-IS test dataset. It is observed that our proposed model (I-AID) considerably outperforms the state-of-art baseline by $42\%$ in high priority AAW and $63\%$ in overall AWW. 

Remarkably, 
\paragraph{\textbf{To answer} $Q_2$}

\begin{table}[]
	\centering
	\label{tab:AAW}
	\caption{Performance Evaluation based on Alerting Metric in TREC-IS test dataset}
	\begin{tabular}{|l|l|l|}
		\hline
		\multicolumn{1}{|c|}{\multirow{2}{*}{\textbf{RUNS}}} & \multicolumn{2}{l|}{\textbf{Accumlated Alert Worth}} \\ \cline{2-3} 
		\multicolumn{1}{|c|}{}                      & High Priority             &       All      \\ \hline
		Median                                     &         -0.9197           &    -0.4609       \\ \hline
		textCNN                                     &         -0.9764           &    -0.4884       \\ \hline
		FastText                                    &       -0.7975             &    -0.4217      \\ \hline
		HAN                                         &       -0.7816             &    -0.4600      \\ \hline
		Bi-LSTM                                      &      -0.8760              &    -0.4482      \\ \hline
		UCDbaseline                                 &       -0.7856             &    -0.4131      \\ \hline
		\textbf{I-AID}                              &      \textbf{ 0.2044  }   &    \textbf{0.0468}\\ \hline
	\end{tabular}
\end{table}

\begin{figure}[ht!]
	\centering
	\captionsetup{justification=centering}
	\includegraphics[width=0.7\linewidth]{Figures/F1InformationTypes.pdf}
	\caption{F1-Macro scores per Information Types}
	\label{fig:F1-informationTypes}
\end{figure}




\begin{table*}[h!]
	
	\centering
	\caption{Performance evaluation -\textit{macro-averaged}- of baselines and our system (i-AID).}
	\begin{tabularx}{0.9\textwidth}{@{}XXXXXXX@{}}
		\toprule
		\multirow{2}[2]{*}{\textbf{TREC Dataset}} & \multirow{2}[2]{*}{\textbf{Metrics}}& \multicolumn{4}{c}{\textbf{Baselines}} & \multirow{2}[2]{*}{\textbf{i-AID}} \\ 
		\cmidrule(lr){3-6}
		&  & \textbf{textCNN} & \textbf{FastText} & \textbf{HAN} & \textbf{Bi-LSTM} &  \\
		\cmidrule(r){1-1} \cmidrule(lr){2-2} \cmidrule(lr){3-6} \cmidrule(l){7-7}
		\multirow{5}{*}{Unbalanced} & Precision & 0.116 & 0.155 & 0.076 & \textbf{0.170} & 0.100 \\
		& Recall & 0.331 & 0.199 & 0.319  & 0.373 &  \textbf{0.958}\\
	
		& $F_1$-\textit{all} &  0.134 &  0.138 & 0.118 & \textbf{0.208} &  0.162\\
		& $F_1$-\textit{actionable} & 0.1508 & 0.00 &  0.151 & 0.173 & \textbf{0.194}\\
	
		\cmidrule(r){1-1} \cmidrule(lr){2-2} \cmidrule(lr){3-6} \cmidrule(l){7-7}
		\multirow{5}{*}{Balanced} & Precision & 0.08 & 1.0 & 0.11 & 0.68/0.56 &  \\
		& Recall & 0.64 & 0.05  & 0.11 & 0.68/0.51 &  \\
	
		& $F_1$-\textit{all} &  &  &  &  &  \\
		& $F_1$-\textit{actionable} & 0.34 & 0.09 & 0.09 & 0.68/0.53  &  \\ \bottomrule
	\end{tabularx}
\end{table*}

\section{Conclusion}\label{sec:conclusion}
In this paper, we conduct extensive experiments to investigate the different approaches to fine-tuning BERT for identifying actionable information from disaster-related tweets. In this paper, we propose a novel model that takes advantages of both explicit and implicit representations of short-text. We enrich the features of short texts by conceptualizing them with the help of a well-known knowledge graph.

There  are some experimental findings: 1) The top layer of BERT is more useful for text  classification; 2) With an appropriate layer-wise decreasing learning rate, BERT can overcome the catastrophic for getting problem; 3) Within-task and in domain further pre-training can significantly boost its performance; 4) A preceding multi-task fine-tuning is also helpful to the single-task fine-tuning, but its benefit is smaller than further pre-training; 5) BERT can improve the task with small-size data. With the above findings, we achieve state-of-
the-art performances on eight widely studied text classification datasets.

In the future work, we will employ few-shot learning technique to handle the data imbalances.

\section*{Acknowledgment}
This work has been supported by the BMVI projects LIMBO (GA no. 19F2029C), OPAL (no. 19F2028A), SOLIDE (no. 13N14456) as well as the EuroStars project SAGE (no. E!10882).


\bibliographystyle{plain}
\balance

\section{Introduction}
\label{sec:introduction}
Social media has become a key medium for sharing information during emergencies~\cite{DBLP:journals/pacmhci/ZadeSRK0S18}. 
The major difference between social media and traditional news sources is the possibility of receiving feedback from affected people in real time. 
Relief organizations can benefit from this two-way communication channel to inform people and gain insights from situational updates received from affected people.
Hence, extracting crisis information from posts on social media (e.g., tweets) can substantially leverage situational awareness and result in faster responses. 

Most previous works~\cite{to2017identifying,stowe2018improving} addressed information extraction from social media as a binary text classification problem (e.g., with the labels \texttt{Relevant} and \texttt{Irrelevant}). 
However, there is a lack of efficient systems that can map relevant posts to more fine-grained labels as, for example, defined in~\cite{DBLP:conf/iscram/McCreadieBS19} (see Figure~\ref{fig:tweetexample}). 
Such fine-grained labels are particularly valuable for crisis responders as they filter critical information to deliver disaster responses quickly.
In particular, labeling disaster-related tweets using multiple labels allows the rapid detection of tweets with \textit{actionable} information. 
Table~\ref{tab:informationTypes} shows the list of information types (which we use as labels) defined by~\cite{DBLP:conf/iscram/McCreadieBS19}. 
We adopt the definition of \textit{actionable} tweets as formalized in~\cite{DBLP:journals/pacmhci/ZadeSRK0S18}.
\textit{Actionable} tweets are defined as the ones that would generate an immediate alert for individuals (i.e., stakeholders) responsible for the information type with which they are labeled (e.g., \texttt{SearchAndRescue}, \texttt{MovePeople}). 
This stands in contrast to  \textit{non-actionable} tweets that are labeled with labels such as \texttt{Hashtags} or \texttt{FirstPartyObservation} (see Table~\ref{tab:informationTypes}).

\Figure[t!] (topskip=0pt, botskip=0pt, midskip=0pt)[width=0.95\columnwidth]{Figures/tweet-multilabel.pdf}{Example of multi-label tweet classification with assigned labels: FirstPartyObservation, EmergingThreats and News.     \label{fig:tweetexample}}



On the other hand, categorizing tweets is known to be a challenging short text Natural Language Processing (NLP) task~\cite{song2014short}.
This is because tweets 
i) does not possess sufficient contextual information, and 
ii) is inherently noisy (e.g., contains misspellings, acronyms, emojis, etc.). 
Moreover, in the multi-label case, the classification task becomes even more challenging because a tweet can belong to one or more labels simultaneously.
	
In this paper, we aim to 
i) label disaster-related tweets with fine-grained information types so as to 
ii)  identify \textit{actionable} or \textit{critical} tweets that might be relevant for disaster relief and support disaster mitigation. 
Our approach contains three components: 
First, we use BERT as a sentence encoder to capture the semantics of tweets and to represent them as low-dimensional vectors. 
Second, we employ a graph attention network (GAT) to capture correlations between the words and entities in tweets and the labels of said tweets.
Finally, we use a \textit{Relation Network}~\cite{sung2018learning} as a learnable distance metric to compute the similarity between the vector representation of tweets (obtained from the BERT encoder) and the vector representation of labels (obtained from the GAT) in a supervised way. 
By these means, our system integrates a contextualized representation of tweets with correlations between tweets and their labels. 
The main contributions of this paper can be summarized as follows:
	\begin{itemize}
		\item We propose a multimodel
		approach (dubbed I-AID) to categorize disaster-related tweets into multiple information types. 
		\item Our approach leverages a contextualized representation from a pretrained language model (BERT) to capture the semantics of tweets.  
		In addition, our approach employs a GAT component to capture the structural information between the words and entities in tweets and their labels.
		\item We employ a \textit{learnable} distance metric, in a supervised way, to learn the similarity between a tweet's vector and the labels' vectors. 
		\item We conduct several experiments to evaluate the performance of our approach and state-of-the-art baselines in multi-label text classification. 
	\end{itemize}
	
The rest of this paper is structured as follows:
In Section~\ref{sec:relatedworks}, we discuss previous work on the classification of crisis information on social media. 
In Section~\ref{sec:approach}, we describe the preliminaries and architecture of our proposed approach. 
Finally, we discuss the experimental results in Section~\ref{sec:experiment} and conclude the paper in Section~\ref{sec:conclusion}.

	
\begin{table*}[h!]
	\centering 
	\small
	\caption{Crisis Information Types (i.e., Labels or Classes)\label{tab:informationTypes}}
	\begin{tabularx}{0.85\textwidth}{@{}p{2.1cm}p{3cm}p{7cm}@{}}
		\toprule
		\textbf{Intent Type }& \textbf{Information Type} & \multicolumn{1}{c}{\textbf{Description}} \\ \midrule
		
		\multirow{3}{*} {REQUEST}& GoodServices & Request for a particular service or physical good \\
		& SearchAndRescue & The user is requesting a rescue for themselves or others\\
		& InformationWanted & The user is requesting information \\ \midrule
		\multirow{11}{*}{REPORT} & Weather & Weather report \\
		& FirstPartyObservation & The user is giving an eyewitness account \\
		& ThirdPartyObservation & The user is reporting information from someone else\\
		& EmergingThreats & Problems that cause loss or damage \\
		& ServiceAvailable & Someone is providing a service \\
		& SignificantEventChange & New occurrence to which officers need to respond\\
		& MultimediaShare & Shared images or video \\
		& Factoid & The user is reporting some facts, typically numerical \\
		& Official & Report by a government or public representative \\
		& CleanUp & Report of the cleanup after an event \\
		& Hashtags & Report with hashtags correspond to each event \\ \midrule
		\multirow{3}{*}{CALLTOACTION} & Volunteer & Call for volunteers to help in response efforts \\
		& Donations & Call for donations of goods or money \\
		& MovePeople & Call to leave an area or go to another area \\ \midrule
		\multirow{6}{*}{OTHER} & PastNews & The post is reporting an event that has occurred \\
		& ContinuingNews & The user is providing/linking to a continuous event\\
		& Advice & Provide some advice to the public\\
		& Sentiment & The post is expressing some sentiments about an event\\
		& Discussion & Users are discussing an event\\
		& Irrelevant & The post is irrelevant\\
		\midrule
	\end{tabularx}
\end{table*}


\section{Related Work}\label{sec:relatedworks}
The objective of this work is to categorize disaster-related tweets into multiple information types.
Therefore, we relate our work to extract \textit{disaster-related information on social media}, \textit{multi-label text classification} and \textit{meta learning}. In the following, we briefly discuss the state of the art in each of these areas. 
	
	\subsection{Extracting Disaster-related Information from Social Media}
	Several studies demonstrate the role of social media as a primary source of information during disasters~\cite{cresci2015crisis}. 

	While some works~\cite{PalshikarAP18} focused on filtering relevant information from tweets, 
	others (e.g., \cite{olteanu2015expect,mccreadie2019trec}) proposed annotation schemes to classify tweets into fine-grained labels that consider the attitude, information source and decision-making behavior of people tweeting before, during and after disasters.
	To advance the state of social media crisis monitoring solutions, initiatives like~\cite{imran2016lrec} 
	have been rolled out in recent years.
	One of them is the Incident Streams (TREC-IS) track~\cite{mccreadie2019trec} of the Text REtrieval Conference, which commenced in 2018.
	The track aims to categorize disaster-related tweets into multiple information types. 
	In this work, we study the TREC-IS dataset and adopt the definition of \textit{actionable} information from the authors of the TREC-IS challenge. In addition, we employ their performance metric (called Accumulated Alert Worth~\cite{TRECIS-AAW}) to evaluate our system in identifying \textit{actionable} information in tweets.
	
	\subsection{multi-label Text Classification}

	Earlier works in text classification~\cite{sriram2010short} consider feature engineering and model training as different subtasks. 
 With the advent of end-to-end deep learning approaches~\cite{miyazaki2019label} and the attention mechanism~\cite{vaswani2017attention}, there has been a significant advancement in the field of multi-label text classification. 
Pretrained language models (e.g, BERT~\cite{devlin2018bert}) are becoming increasingly popular for text classification~\cite{miyazaki2019label}.

However, since BERT only captures the local contextual information, the BERT embeddings do not sufficiently capture the global information about the lexicon of a language.~\cite{DBLP:conf/ecir/LuDN20}
To circumvent this and comprehend the global relations among words in a vocabulary, graph-based approaches such as graph convolution network (GCN)~\cite{kipf2016semi} and graph attention network (GAT)~\cite{yao2019graph} have been promising.
	
Recent studies~\cite{DBLP:conf/ecir/LuDN20,DBLP:conf/icaart/PalSS20} have exploited the advantages of combining BERT and graph networks.

In VGCN-BERT~\cite{DBLP:conf/ecir/LuDN20}, a GCN is used to capture the correlation between words at the vocabulary level (i.e., global information). 
For instance, given a vocabulary, the GCN would relate the meaning of \textit{"new"} to \textit{"innovation"} and \textit{"exciting"}, similar to context-independent word embeddings like \textit{word2vec}~\cite{DBLP:journals/corr/abs-1301-3781}. For an input sentence, the \textit{local} contextual information is captured using BERT embeddings, while the global information pertaining to words in a sentence is extracted using graph embeddings and subsequently concatenated with BERT. The two representations of BERT and GCN then interact via the self-attention mechanism to perform the classification task.

In a similar work, Ankit Pal~\textit{et al.}~\cite{DBLP:conf/icaart/PalSS20} leverage the combination between BERT embeddings and GAT to learn feature representation for text in a multi-label classification task. 
Their proposed approach (dubbed MAGNET) employs two components: First, a BiLSTM network with BERT embedding is used to capture text representation into an embedding vector. In the second component, the authors use GAT to learn a feature vector for labels. In particular, their GAT models the correlation between words and the corresponding labels, then averages the labels' vectors into a single output vector. Finally, the authors use a dot-product function to compute the similarity between the input's vector from BiLSTM and the label's vector. 
In contrast, our approach differs from both MAGNET and VGCN-BERT in computing the similarity between a tweet's representation and its labels' vectors. 
We employ a GAT model to explicitly infuse the correlation information of the entities and labels of a tweet with the tweet's contextualized BERT representation. 
While MAGNET and VGCN-BERT use either a fixed and linear distance metric (dot-product function) or self-attention to measure the similarities, our approach benefits from a deeper end-to-end neural architecture to learn this distance function. In particular, we employ meta-learning to learn
the mapping between the input features and multi-label output in a supervised way.

	

	
	\subsection{Meta Learning}

	Meta learning (also called \textit{learning-to-learn} paradigm) refers to the process of improving a learning algorithm over multiple learning episodes.
	In contrast to conventional machine learning approaches, which improve model prediction over multiple data instances, the meta-learning framework treats tasks as training examples to solve a new task~\cite{DBLP:journals/corr/abs-2004-05439}.
	In our study, we employ a specific branch of meta learning called metric learning. 
	Metric learning learns a distance function between data samples so that the test instances get classified by comparing them to the labeled examples. 
	The distance function consists of 
	i) an embedding function, which encodes all instances into a vector space, and 
	ii) a similarity metric, such as cosine similarity or Euclidean distance, to calculate how close two instances are in the space~\cite{DBLP:journals/corr/abs-2007-09604}. 
	Recently, many approaches have been developed to perform this task, such as \textit{Siamese}~\cite{koch2015siamese}, \textit{Matching}~\cite{vinyals2016matching}, \textit{Prototypical}~\cite{snell2017prototypical}, and \textit{Relation Network}~\cite{sung2018learning}.
	While the embedding function in all of these approaches is a deep neural network, they differ in terms of the similarity function. 
	Unlike its predecessors, which rely on a fixed similarity metric (such as cosine, Euclidean, etc.),  Relation Network employs a flexible function approximator to learn similarity and focuses on learning a good similarity metric in a supervised way. 
	The use of function approximators eliminates the need to manually choose the right metric (e.g., Euclidean, cosine, Manhattan). 
	By jointly learning the embedding and a nonlinear similarity metric, Relation Network can better identify matching/mismatching pairs~\cite{garcia2018fewshot}. 
	For this purpose, we use the Relation Network in our work for learning the similarity metric. 


\section{Our Approach}\label{sec:approach}

\begin{table}[tb]
\small
\centering
\caption{A List of Symbols Used in This paper.}
\begin{tabularx}{\columnwidth}{@{}p{1.2cm}p{6.5cm}XX@{}}
\toprule
\textbf{Symbol} & \textbf{Description} \\ \hline
 $S$ & Number of tweets in the dataset.\\
 $w$ & Tweet tokens (e.g., word or entity). \\
 $y^{(i)}$ & Ground-truth multi-label assigned to a tweet $i$.\\
 $\hat{y}^{(i)}$ & Predicted multi-label assigned to a tweet $i$.\\ 
 $\lambda_i$ & A single label/information type for a tweet.\\ 
 \midrule
 $N$ & Number of nodes in a graph \\
 $\mathcal{V}$& Nodes of a graph \\
 $\mathcal{E}$ & Edges between nodes in a graph\\
 $A$ & Adjacency matrix of a graph \\
 $\tau^{(i)}$ & Embedding vector of tweet $i$ learned by BERT\\
 $h^{(i)}$ & Embedding vector of node $v^{(i)}$\\
 \midrule
 $F$ & Dimension of node vector \\
 $\iota^{(i)}$ & Embedding vector for label $\lambda^{(i)}$\\
 $Z$ & The concatenated vector of $\tau^{(i)}$ and  $\iota$ \\
 $\mathcal{L}$& Binary cross-entropy loss function\\
 $\alpha_{ij}$ & Attention score between nodes $v^{(i)}$ and $v^{(j)}$\\
 \midrule
 $hPW(t)$ & Scoring function for high priority tweets.\\
 $hPW(l)$ & Scoring function for low priority tweets.\\
 \bottomrule
\end{tabularx}
\end{table}

\Figure[t!] (topskip=0pt, botskip=0pt, midskip=0pt)[width=0.95\textwidth]{Figures/I-AID.pdf}
		{The I-AID architecture: BERT-Encoder embeds tweet $t^{(i)}$ into a feature vector $\tau^{(i)}$. TextGAT builds a graph $G$ from our dataset, employs graph  attention layers and output labels vectors $\iota$. Relation Network learns a distance metric between $\tau^{(i)}$ and $\iota$, then outputs predicted labels $\hat{y}^{(i)}$ for $t^{(i)}$.
		\label{fig:I-AID}
	}
	
We begin this section by giving a formal specification of the multi-label tweet classification problem. 
Afterward, we discuss the details of each component of our approach in Section~\ref{sec:I-AID Architecture}.
Figure~\ref{fig:I-AID} gives an overview of our approach and how its components work together. 
	
	\subsection{Problem Formulation}\label{sec:problem_formulation}
	Let $T$ be a set of tweets and $\Lambda=\{\lambda_1, \lambda_2,\cdots ,\lambda_k\}$ be a set of $k$ predefined labels (also called information types, see Table~\ref{tab:informationTypes}). 
	We formulate the problem of \textit{identifying crisis information from tweets} as a multi-label classification task, where a tweet $t$ can be assigned one or more labels from $\Lambda$ simultaneously. 
	Our task is to learn a multi-label classifier $\mathbf{M}: T \to \{0,1\}^k$ that maps tweets $T$ to relevant labels from $\Lambda$.
	We assume a supervised learning setting where 
	a training data $\mathcal{D}=\big\{ (t^{(i)},y^{(i)})\times \{0,1\}_{j}^k \big\}_{i=1}^S$ consists of $S$ tweets. 
	Hence, each tweet $t^{(i)}$ is labelled with a set of corresponding labels $y^{(i)}$, where $y^{(i)}_j = 1$ means that $t^{(i)}$ belongs to the class $\lambda_j$. 
	Conversely, $y^{(i)}_j = 0$ means that $t^{(i)}$ does not belong to the class $\lambda_j$. 
	The goal of our approach is to learn the function $\mathbf{M}$ by using three neural networks.
	First, we transform tweet $t^{(i)}$ into an embedding vector $\tau^{(i)}$ using a pretrained BERT model. 
	In parallel, our approach learns labels' embeddings $\iota$ using a graph attention network (GAT). 
	These are then concatenated with the tweet embedding $\tau^{(i)}$. 
	Finally, these vectors are fed to our last component (Relation Network) to identify relevant labels for $t^{(i)}$. 
	
	\subsection{The I-AID Architecture}\label{sec:I-AID Architecture}
	

	\subsubsection{BERT-Encoder}
	This is the first component in our system that transforms an input tweet into a vector representation $\tau$ of its contextual meaning. 
	As shown in Figure~\ref{fig:I-AID}, the BERT-Encoder takes tweet $t^{(i)}$ with $m$ tokens $[w^{(i)}_1,w^{(i)}_2,\dots ,w^{(i)}_m]$ and outputs the embedding vector $\tau^{(i)}$. We employ a BERT-base architecture with $12$ encoder blocks, $768$ hidden dimensions, and $12$ attention heads. We refer readers to the original BERT paper~\cite{devlin2018bert} for a detailed description of its architecture and input representation. 
	Furthermore, a special preprocessing is performed for BERT input. A \texttt{[CLS]} token is appended to the beginning of the tweet and another token \texttt{[SEP]} is inserted after each sentence as an indicator of sentence boundary. Each token $w^{(i)}$ is assigned three kinds of embeddings (token, segmentation, and position). These three embeddings are summed to a single output vector $\tau^{(i)}$ that captures the meaning of an input tweet.




	
\subsubsection{Text-Graph Neural Network (TextGAT)}
Traditional methods (e.g., word2vec~\cite{DBLP:journals/corr/abs-1301-3781}) can properly capture features from a text. 
However, these methods ignore the structural information and relationship between words in a text corpus~\cite{peng2018large}. 
The recently proposed graph networks~\cite{yao2019graph} aim to tackle this challenge by modeling text as a graph where words are nodes and relations between them are edges.
In our work, we build a graph $G=(\mathcal{V}, \mathcal{E})$ from the dataset $\mathcal{D}$, where $\mathcal{V}$ and $\mathcal{E}$ represent nodes set and their edges, respectively. Each node $v^{(i)} \in \mathcal{V}$ can be a \texttt{word, named-entity\footnote{We spot named-entities in tweets using spaCy entity recongnizer \url{https://spacy.io/api/entityrecognizer}}} or \texttt{label} (tweet's class or information type). 
We represent nodes using a feature matrix $\mathbf{H}=\{h^{(1)}, h^{(2)}, \cdots ,h^{(N)}\}$ where $h^{(i)} \in \mathbb{R}^F$ is the feature vector of node $v^{(i)}$ with $F$ dimension and $N$ is the number of nodes.
First, we initialize the nodes' representation $\mathbf{H}$ with pretrained embeddings from Glove embedding~\cite{pennington2014glove}.
Further, relations between nodes are modeled using an adjacency matrix $\mathbf{A} \in \mathbb{R}^{N\times N}$.

As shown in Figure~\ref{fig:I-AID}, TextGAT component has two graph attention layers.
Each layer takes nodes' features $\mathbf{H}$ as input and performs an \textit{attention} operation~\cite{velivckovic2017graph} to learn a new feature $\mathbf{\hat{H}}=\{\hat{h}^{(1)}, \hat{h}^{(2)}, \cdots ,\hat{h}^{(N)}\}$ for each node based on its neighbours' importance (i.e., \textit{attention from its neighbours}). 
Hence, we employ the shared attention mechanism $att: \mathbb{R}^{\hat{F}} \times \mathbb{R}^{\hat{F}} \longrightarrow \mathbb{R}$ over all nodes. 
The graph attention operated on the node representation can be written as: 
\begin{equation}\label{eq:att}
	    \alpha_{ij}=att~(\mathbf{W}v^{(i)},\mathbf{W}v^{(j)})
\end{equation}

where $att$ is a single-layer feedforward network, parametrized by a weight matrix $\mathbf{W}\in \mathbb{R}^{\hat{F}\times F}$ which is applied to every node. 
Finally, we use a softmax function to normalize the attention scores as shown in Eq.~\ref{eq:softmax}.
	\begin{equation}\label{eq:softmax}
	\alpha_{ij}=\frac{exp(\alpha_{ij})}{\sum_{k\in N_i}exp(\alpha_{ik})}
	\end{equation}

To this end, TextGAT learns the structural information between nodes based on the relative importance of neighbours. 
The learned representations of labels are then extracted and concatenated with the tweet's vector as input for the last component, as shown in Figure~\ref{fig:I-AID}. 
	
\subsubsection{Relation Network.}
\label{relation-network} 
In this component, we aim to learn a similarity metric in a supervised way (also called learning-to-learn or meta learning) between the tweet's vector $\tau^{(i)}$ and labels vectors $\iota$.
Furthermore, we employ a neural network as a learnable, nonlinear distance function that \textit{learns} how to match similarity (i.e., relation) between the tweet's vector and each label. Relation Network takes as input the concatenated matrix $Z= \tau^{(i)} \otimes \iota$ of BERT-Encoder output with the labels' vectors. Since our task is multi-label classification, we use the binary cross-entropy as a loss function in Eq.~\ref{eq:binary_cross_entropy}. Then we use a sigmoid function in the output layer to compute the probability of each label independently over all possible labels ($\Lambda$), in contrast to a softmax function which only considers the label with highest probability. Finally, a set of relevant labels is returned as a final output of our approach.
\begin{equation}
	\label{eq:binary_cross_entropy}
	\mathcal{L}=-\frac{1}{S}\sum_{i=1}^{S}\left[y^{(i)}\log(\hat{y}^{(i)})+(1-y^{(i)})\log(1-(\hat{y}^{(i)})\right]
	\end{equation}
	
	where $y^{(i)}$ and $\hat{y}^{(i)}$ are the predicted and ground-truth labels of tweet $i$ respectively. $S$ is size of tweets in the training dataset. 



\section{Experiments}\label{sec:experiment}
In this section, we report the evaluation results of our approach and baseline methods. 
We aim to answer the following research questions:
	\begin{itemize}
	    \item[$Q_1$:]
	    How does our approach perform compared with  state-of-the-art multi-label models in short text (e.g., tweets) classification?
	   
	    \item[$Q_2$:] How effective is our approach in identifying tweets with \textit{actionable} information?
	    \item[$Q_3$:] How does each component in our approach affect the overall performance (i.e., Ablation Study)?
	\end{itemize}
		
	
\begin{table}[ht]
\centering
\caption{Overview of the Datasets.}
\begin{tabular}[t]{lcccc}
\toprule
\textbf{Datasets} & \textbf{\# Train} & \textbf{\# Valid} & \textbf{\# Test} & \textbf{\# Classes}\\
\midrule
TREC-IS &  27,467 & 6,867 &  8,584 & 25      \\
COVID-19 Tweets &  4,844 & 1,211 & 1,514  & 12   \\
\bottomrule
\end{tabular}
\label{tab:datasets}
\end{table}%
	
	
\subsection{Datasets}
We conducted a set of experiments on two public datasets provided by TREC~\cite{mccreadie2019trec}. 
Table~\ref{tab:datasets} gives an overview of each dataset: the number of tweets used in training (\textbf{\# Train}), validating (\textbf{\# Valid}), and testing (\textbf{\# Test}) our approach and baselines, in addition to the total number of classes (\textbf{\# Classes}).  
In particular, we split each dataset with $80\%-20\%$ ratio, where we use $80\%$ of tweets for training and $20\%$ for testing. During the training phase, we use $20\%$ from the training data to validate the model. We briefly summarize each dataset as follows:

\begin{itemize}
    \item \textbf{TREC-IS: }
    This dataset contains approximately $35$K tweets collected during $33$ different disasters between 2012 and 2019 (e.g., \texttt{wildfires, earthquakes, hurricanes, bombings,} and \texttt{floods}). 
	The tweets are labeled with $25$ information types by human experts and volunteers. 
	
	\item \textbf{COVID-19 Tweets: } This dataset contains a collection of tweets about the COVID-19 outbreak in different affected regions. In total, the data has $7,590$ tweets labeled with one or more of the full $12$ information type labels (the same as for the TREC-IS dataset). 
\end{itemize}

Figure~\ref{fig:tweets-informationTypes} shows the distribution of tweets per information type in both datasets. 
Apparently, the datasets are highly imbalanced \textit{w.r.t.} tweets' distribution across information types. 
For example, in the TREC-IS dataset, there are more than $6,000$ tweets that are categorized into the information types \texttt{Hashtags}, \texttt{News}, \texttt{MultimediaShare}, and \texttt{Location}. In contrast, the information types \texttt{CleanUP, InformationWanted}, and \texttt{MovePeople} have significantly fewer tweets. Similarly in COVID-19 Tweets, the tweets' distribution is extremely imbalanced: most tweets are categorized into \texttt{Irrelevant, ContextualInformation, Advice,} or \texttt{News}. This skewing distribution in tweets renders multi-label classification more challenging. 


\begin{figure*}[t!]
\centering
    \subfigure[TREC-IS Dataset] 
    {
        \includegraphics[width=\columnwidth]{Figures/trecis-labels.pdf}
    }
    \hfill
     \subfigure[COVID-19 Tweets Dataset]
    {
        \includegraphics[width=\columnwidth]{Figures/COVIDtweets-labels.pdf}
    }   
\hfill    
\caption{Tweets' distribution across all information types in both datasets (TREC-IS and COVID-19 Tweets)}    
\label{fig:tweets-informationTypes}    
\end{figure*}
	




	\subsection{Baselines} \label{sec:baselines}
	We consider a set of state-of-the-art approaches in multi-label classification\footnote{\scriptsize Note that we consider the relevant baselines that were implemented for multi-label classification. 
	The current version of VGCN-BERT cannot be directly used for multi-label classification. Therefore, we did not include it in the baselines.} as baselines in our evaluation. We briefly describe each baseline as follows:

	
\begin{itemize}
		\item \textbf{TextCNN}~\cite{kim2014convolutional} uses a convolutional neural network to construct text representation. 
	
	
		\item \textbf{HAN}~\cite{yang2016hierarchical} uses a hierarchical attention neural network to encode text with word-level attention on each sentence.
		
		\item \textbf{BiLSTM}~\cite{zhou2016attention} is a bidirectional LSTM model that parses the text from left to right and right to left, then uses the final hidden state as a feature representation for the whole text.
	
	
		\item \textbf{MAGNET}~\cite{DBLP:conf/icaart/PalSS20} employs a bidirectional LSTM with BERT embeddings to represent tweets and GAT for labels classifiers. Then it uses a dot-product function to compute similarities between tweet vectors and labels' vectors.
		
	\end{itemize}
	
\subsection{Implementation and Preprocessing}\label{sec:implementation}
We use the open-source implementations for TextCNN, HAN, and  BiLSTM models provided by the corresponding authors in their GitHub repositories. Furthermore, we implemented the code for the MAGNET model as it has not been open-sourced to date. In our approach, we use the implementation of BERT-Encoder from the \texttt{Huggingface}\footnote{\url{https://huggingface.co/}} library. 

Hyperparameters in the baselines are set with same values as mentioned in their original papers. In our model, we tune  hyperparameters via the grid search method to find optimal values for best performances. Specifically, our model achieves its best performance with the following values: \texttt{training-epochs} to $200$ with \texttt{batch-size} of $128$ and \texttt{Adam} optimizer~\cite{kingma2014adam} with a \texttt{learning-rate} of $2\mathrm{e}^{-5}$. To avoid overfitting, we add a \texttt{dropout} layer with a rate of $0.25$ and apply an \texttt{early-stopping} technique during model's training. The implementation of the I-AID model is open-sourced and available on the project website\footnote{\url{https://github.com/dice-group/I-AID}}.

\paragraph{Data Preprocessing} Given that the evaluation datasets are tweets, we perform ad-hoc preprocessing steps to capture the tweets' semantics. 
In particular, we perform the following preprocessing steps: 
(1) We use the \texttt{NLTK's TweetTokenize}\footnote{\url{https://www.nltk.org/api/nltk.tokenize.html}} API to tokenize tweets and retain the text content. 
(2) Stop-words, URLs, usernames, and Unicode-characters were removed. 
(3) Extra white spaces, repeated full stops, question marks, and exclamation marks are removed. 
(4) Emojis are converted to text using the \texttt{emoji}\footnote{\url{https://pypi.org/project/emoji/}} python library.
Finally, (5) \texttt{spaCy}\footnote{\url{https://github.com/explosion/spaCy}} library is used, to extract \texttt{named-entities} from tweets.

	
\subsection{Evaluation Metrics}\label{section:evaluation_metrics}

	We consider standard evaluation metrics for a multi-label classification task. In particular, we use a \textit{weighted average F1 score}, \textit{hamming loss} and \textit{Jaccard index} to evaluate the system's performance: 

	\begin{itemize}
	
	 \item{\textbf{Weighted average F1 score}}:
	 F1 score is the harmonic mean of precision and recall scores. We use a \textit{weighted average} that calculates the F1 score for each label independently, then adds them together and uses a weight relative to the number of tweets in each label.
	 \begin{equation}
	    \text{F1}_{w.avg.}= 2 \sum_{i=1}^{k} \frac{|T_{\lambda_{i}}|}{|T|} \frac{precision_{\lambda_{i}}\times recall_{\lambda_{i}}}{precision_{\lambda_{i}} + recall_{\lambda_{i}}}
	  
	 \end{equation}
	 
	 where $|T_{\lambda_{i}}|$ denotes the number of tweets with label $\lambda_{i}$ and $|T|$ is the total number of tweets. Precision$_{\lambda_{i}}$ and recall$_{\lambda_{i}}$ are the values of precision and recall for $\lambda_{i}$.
	 
	\item \textbf{Hamming Loss}: To estimate the error rate in classification, we use the \textit{hamming loss} function~\cite{schapire1999improved} 
	that computes the fraction of incorrectly predicted labels out of all predicted labels. Hence, the smaller the value, the better the performance.  
	
	\begin{equation}
	{hamming\:Loss\:}(y^{(i)}, \hat{y}^{(i)})=\frac{1}{S}\sum_{i=1}^{S}\frac{1}{k} |y^{(i)}\oplus\hat{y}^{(i)}|
	\end{equation}
 	where $S$ is the dataset size, $k$ is the total number of labels (i.e., $|\Lambda|$ ), $\oplus$ denotes the XOR operator, and $y^{(i)}$ and $\hat{y}^{(i)}$ are the groundtruth and predict labels, respectively, of tweet $i$ .
 	
	 \item \textbf{Jaccard Index:} To assess the system's accuracy, we use the Jaccard index to evaluate the similarity between predicted labels $\hat{y}^{(i)}$ and groundtruth labels $y^{(i)}$. 
	 Jaccard index computes the percentage of common labels in two sets of all labels as:
	\begin{equation}
	{jaccard\:}(y^{(i)}, \hat{y}^{(i)})=\frac{|y^{(i)} \cap \hat{y}^{(i)}|}{ |y^{(i)} \cup \hat{y}^{(i)}|}  
	\end{equation}
	where $y_i$ and $\hat{y}_i$ are the groundtruth and predicted labels for tweet $i$. $\cap$ and $\cup$ denote intersection and union set operations, respectively.
	\end{itemize}
	
	\subsubsection{Evaluating Actionable Information}
	We aim to evaluate the efficacy of our system in identifying tweets with actionable information,
	i.e., the system should trigger an alert if an input tweet includes actionable information (e.g., requests for search and rescue or reports of emerging threats). For this purpose, TREC-IS~\cite{mccreadie2020incident} introduces a new evaluation metric called \textit{Accumulated Alert Worth} (AAW) to evaluate systems in detecting actionable information during crisis. 
	The AAW score ranges from $-1$ to $+1$, where a positive value indicates highly critical information in a tweet while a negative score indicates it is less critical. More details about the AAW metric can be found in~\cite{TRECIS-AAW}. Here, we summarize the AAW metric as follows: 
	\begin{equation}
	\label{eq:AAW}
	AAW=
	\frac{1}{2}\sum_{t\in T}
	\begin{cases}
	\frac{1}{|T_h|} \cdot hPW(t) & \text{if } t \in T_h\\
	\frac{1}{|T_l|} \cdot lPW(t) &  \text{otherwise} 
	\end{cases}   
	\end{equation}
	
	where $T_h$ and $T_l$ denote the sets of tweets with high and low priorities, respectively.
	$hPW(t)$ is a scoring function for tweets that should generate alerts and $lPW(t)$ is a scoring function for tweets that should not generate alerts.
	Formally,
	
	\begin{equation*}
	    hPW(t)=
	    \begin{cases}
	\alpha+((1-\alpha)\cdot (\varphi(t)+\hat{\varphi}(t)) & \text{if } p_t^s>=0.7\\
	-1 &  \text{otherwise} 
	\end{cases}   
	\end{equation*}
	
	\begin{equation*}
	lPW(t)=    
	\begin{cases}
	argmax(-log(\frac{\delta}{2}+1), -1) & \text{if } p_t^s>=0.7\\
	\varphi(t)+\hat{\varphi}(t) &  \text{otherwise} 
	\end{cases}   
	\end{equation*}

	 where $p_t^s$ is the priority score of a tweet by the system, and $\varphi(t)$ and $\hat{\varphi}(t)$ are actionable and non-actionable scores, respectively, for tweet $t$,. 
	

\subsection{Discussion}
\subsubsection{Performance Comparison ($Q_1$)}

		\begin{table*}[th!]
		\centering
		\caption{Evaluation results of our approach (I-AID) and baselines on two datasets: TREC-IS and COVID-19 Tweets using weighted average F1, Hamming Loss and Jaccard Index. Best results are in bold.}
		\begin{tabularx}{0.75\linewidth}{@{}p{2.2cm}p{2.1cm}p{1.5cm}p{1cm}p{1.5cm}p{1.1cm}XXXX@{}}
			\toprule
			\multirow{2}[2]{*}{\textbf{Datasets}} & \multirow{2}[2]{*}{\textbf{Metrics}}& \multicolumn{4}{c}{\textbf{Baselines}} & \multirow{2}[2]{*}{\textbf{I-AID}} \\ 
			\cmidrule(lr){3-6}
			{\textbf{}}&  & TextCNN & HAN & BiLSTM & MAGNET\\
			\cmidrule(r){1-1} \cmidrule(lr){2-2} \cmidrule(lr){3-6} \cmidrule(l){7-7}
			\multirow{3}{*}{TREC-IS} 
			& $\text{F1}_{w.avg.}$& 0.25  & 0.37 & 0.31 & 0.53 & \textbf{0.59}  \\
		
			& Jaccard Index & 0.18 & 0.28 & 0.19 & 0.38 & \textbf{0.43} \\
			& Hamming Loss & 0.24 & 0.15 & 0.26 & 0.09 & \textbf{0.07} \\
			
			\cmidrule(r){1-1} \cmidrule(lr){2-2} \cmidrule(lr){3-6} \cmidrule(l){7-7}
			\multirow{3}{*}{COVID-19 Tweets} 
			& $\text{F1}_{w.avg.}$ & 0.47  &  0.40 & 0.43  & 0.51 & \textbf{0.55}   \\
		
			& Jaccard Index &  0.33 & 0.28 & 0.21 & 0.40 & \textbf{0.43} \\
			& Hamming Loss &  0.11 & \textbf{0.04} & 0.07 &0.12 & 0.08 \\
			\bottomrule
		\end{tabularx}
		\label{tab:results}
	\end{table*}
	
	We use different metrics in multi-label classification to evaluate the performance of I-AID and baseline methods.  
	To ensure a fair evaluation, we use the same \textit{train} dataset for training all models and the \textit{test} dataset for evaluation. Table~\ref{tab:results} reports our evaluation results for each model on both datasets (TREC-IS and COVID-19 Tweets). 
	We consider the weighted average F1 score as the primary metric to compare and rank systems. Weighted average F1 takes into account the average performance of each system across all information types. 
	Overall, our approach (I-AID) achieves superior results to the other baselines under several metrics. In particular, our approach outperforms the weighted average F1 score of MAGNET---\textit{the state-of-the-art baseline in multi-label tweets classification}---by $+6\%$ on TREC-IS and $+4\%$ on COVID-19 Tweets. 
	
	We employ the Jaccard index and Hamming loss in further analysis to evaluate accuracy and error rate. 
	using Jaccard index, our approach outperforms all baseline methods in both datasets. In particular, I-AID achieves $43\%$ Jaccard index for both datasets compared with MAGNET's score of $38\%$ for the TREC-IS dataset and $40\%$ for COVID-19 Tweets. 
	On the other hand, our approach achieves suboptimal results using Hamming loss. For the TREC-IS dataset, I-AID achieves the best performance with rate $0.07\%$. While in COVID-19 Tweets, it achieves the second best score with $0.08\%$ compared with HAN model's score $0.04\%$. 
	
    Our experiments demonstrate that I-AID performs fairly well when categorizing disaster-related tweets into multiple information types. This is due to three facts:  
	i) we constructed a multimodel framework that leverages contextualized embeddings from the BERT model to capture contextual information in tweets. 
	ii) Our approach enriches the semantics of tweet representation by injecting label information and integrating additional structural information between tweets' tokens and labels using GAT.
	iii) Finally, we employ a Relation Network to \textit{learn} automatically similarities between tweets and labels. By using a learnable distance function, we learn an efficient metric in a supervised way to facilitate the mapping between a tweet and multi-label output. 
    
\subsubsection{Actionable Information In Tweets ($Q_2$)}
To answer $Q_2$, 
we use the AAW metric, proposed by TREC (Eq.~\ref{eq:AAW}), to evaluate the I-AID's ability to identify tweets with critical information. 
There are two ways to define an \textit{actionable} tweet~\cite{DBLP:journals/pacmhci/ZadeSRK0S18}: 
i) in terms of high priority information, commonly marked as critical by human assessors, and
ii) in terms of information type, for instance, a tweet with the labels \texttt{MovePeople} or \texttt{CleanUP} is considered more actionable than \texttt{News} or \texttt{Multimediashare}. In our evaluation, we consider the second definition of \textit{actionable} posts. 
The evaluation results of the AAW metric are presented in Table~\ref{tab:AAW}, where the top $6$ rows show the evaluation results for the baseline approaches in multi-label classification. 
The rest of Table~\ref{tab:AAW} shows the AAW results of the best approaches from the TREC-IS challenge (2019 edition~\cite{mccreadie2019trec} RUN B). The result of our approach (I-AID) is presented at the bottom of Table~\ref{tab:AAW}.

 
Our approach (I-AID) substantially outperforms all baseline approaches. 
In particular, in high priority AWW,  I-AID achieves an absolute improvement of $+26\%$ compared to the MAGNET model and $+32\%$ compared to \texttt{nyu-smap} (the best-achieved result in TREC-IS 2019). 
Furthermore, I-AID outperforms the Median score of TREC-IS participants by $+28\%$ in high priority and by $+30\%$ in overall AWW. 
Remarkably, our approach is the first to achieve a positive AAW score on high priority tweets. 
Although we outperform the state-of-the-art in both classification and AAW, the results of our evaluation suggest that a significant amount of research is still necessary to spot high priority tweets in a satisfactory manner. 
	
	\begin{table}[t!]
		\centering
		\small
	
	
		\caption{Performance evaluation using the AAW metric on the test dataset from TREC-IS (RUN B). A higher AAW value indicates better prediction.
		\begin{tabularx}{0.95\linewidth}{@{}p{3cm}p{2.5cm}p{1.3cm}@{}}
			\toprule
			\multirow{2}[2]{*}{\textbf{Systems}} & \multicolumn{2}{l}{\textbf{Accumulated Alert Worth (AAW)}} \\ \cmidrule(r){2-3} 
			& High Priority             &       All      \\ \midrule
			
			TextCNN                                     &         -0.9764           &    -0.4884       \\ 
			HAN                                         &       -0.7816             &    -0.4600      \\
			Bi-LSTM                                      &      -0.8760              &    -0.4482     
			\\
			BERT (UPB\_BERT)  &         -0.9680         &       -0.4882     \\  
			TextGAT     &   -0.9794      &   -0.4897        \\
			MAGNET          &    -0.9436         &              -0.4726
			\\ 
			
			\midrule
			Median                                     &         -0.9197           &    -0.4609       \\
			BJUTDMS-run2                                &      -0.9942                  &   -0.4971 \\
			IRIT         &         -0.9942      &                   -0.4971  \\
			irlabISIBase &        -0.2337      &                -0.4935     \\
			UCDbaseline                                 &       -0.7856             &    -0.4131      \\
			nyu-smap &                  -0.1213 &               -0.1973 \\ 
			
			SC-KRun28482low &                   -0.9905 &                     -0.4955\\
			xgboost &                   -0.9942 &                                 -0.4972\\
			UCDrunEL2           &        -0.8556   &  -0.4382 \\
		
			cmu-rf-autothre &       -0.8481     &       -0.4456 \\
			\midrule
			I-AID                              &     \textbf{0.2044} &    \textbf{-0.1509}\\
			\bottomrule
		\end{tabularx}
		\label{tab:AAW}
	\end{table}
\subsubsection{Ablation Study ($Q_3$)}

\begin{table}[h!]
\centering
\caption{Ablation Study of I-AID Model}
		\begin{tabularx}{\columnwidth}{@{}p{2cm}p{1.8cm}p{1cm}p{1cm}llll@{}}
			\toprule
			{\textbf{Datasets}} & {\textbf{Metrics}} &  \textbf{I-AID-BERT} & \textbf{I-AID-TGAT} & \textbf{I-AID}\\
		
			\midrule
			\multirow{3}{*}{TREC-IS} 
			& $F1_{w.avg.}$ & 0.50  & 0.26 & 0.59\\
			& Jaccard Index & 0.34 & 0.18 & 0.43\\
			& Hamming Loss & 0.11 & 0.24 & 0.07\\
		
			\midrule
			\multirow{3}{*}{COVID-19 Tweets} 
			& $F1_{w.avg.}$ & 0.47  &  0.36 & 0.55\\
			& Jaccard Index &  0.37 & 0.15 & 0.43\\
			& Hamming Loss &  0.10 & 0.17 & 0.05\\
			\bottomrule
		\end{tabularx}
		\label{tab:ablationstudy}
	\end{table}

As discussed in Section~\ref{sec:approach}, our approach employs two main components (namely, BERT-Encoder and TextGAT) for representing input tweets. 
We perform an ablation study to evaluate the performance of each component individually. 
To do so, we implement two more versions of I-AID: the I-AID-BERT and the I-AID-TGAT. 
In the I-AID-BERT we deploy our system with the BERT-Encoder only to classify tweets into multiple information types. 
In the same manner, we implement I-AID-TGAT with only the TextGAT component. 
Table~\ref{tab:ablationstudy} shows the evaluation results for each component in our ablation study. 
Evidently, the BERT-Encoder-based implementation of I-AID achieves better performance than the TextGAT version. 
In particular, on the TREC-IS dataset, I-AID-BERT reach $50\%$ F1 score compared with $26\%$ by I-AID-TGAT. 
These results prove that I-AID-BERT can learn rich representation features from short text better than I-AID-TGAT. 
Moreover, in our approach we demonstrate that leverageing BERT and GAT together in a multimodel framework improves the overall performance. On the TREC-IS datastet, the I-AID model achieves superior results by $+9\%$ in F1 score compared with the BERT-based model and by $+33\%$ compared with the TGAT-based model. Our experiment on COVID-19 Tweets leads to a similar conclusion. Our approach outperforms BERT-based and GAT-based systems in F1 scores by $+8\%$, $+19\%$, respectively. On the other hand, we observe that the GAT-based model achieves better performance with fewer output labels. In COVID-19 Tweets with $12$ labels, I-AID-TGAT achieves an improved F1 score of $+10\%$ compared with its performance in TREC-IS with $25$ labels.

\section{Conclusion}\label{sec:conclusion}
In this paper, we propose I-AID, a multimodel approach for multi-label tweets classification. 
Our system combines three components: BERT-Encoder, TextGAT, and Relation Network. 
The BERT-Encoder is used to obtain locality information, while the TextGAT component aims to find correlations between tweets' tokens and their corresponding labels. 
Finally, we use a Relation Network as a last component output to learn the relevance of each label \textit{w.r.t.} the tweet content. 
Our main findings are as follows: 
i) Combining local information captured by BERT-Encoder and global information by TextGAT is beneficial for rich representation in short text and significantly advances multi-label classification. 
ii) Leveraging transfer learning from pretrained language models can efficiently handle sparsity and noise in social media data. 
iii) Benchmarking multi-label classification is a challenging task that requires proper evaluation metrics for fine-grained evaluation. 
I-AID achieves its best weighted average F-score of $0.59$ on the TREC-IS dataset.
This result clearly indicates the sensitivity of our approach to the dataset's balancing. 
Dealing with unbalanced classes remains a future extension to our approach. We plan to use data augmentation and natural language generation to address this problem.

\bibliographystyle{IEEEtran}
