
\section{Introduction}
ACM's consolidated article template, introduced in 2017, provides a
consistent \LaTeX\ style for use across ACM publications, and
incorporates accessibility and metadata-extraction functionality
necessary for future Digital Library endeavors. Numerous ACM and
SIG-specific \LaTeX\ templates have been examined, and their unique
features incorporated into this single new template.

If you are new to publishing with ACM, this document is a valuable
guide to the process of preparing your work for publication. If you
have published with ACM before, this document provides insight and
instruction into more recent changes to the article template.

The ``\verb|acmart|'' document class can be used to prepare articles
for any ACM publication --- conference or journal, and for any stage
of publication, from review to final ``camera-ready'' copy, to the
author's own version, with {\itshape very} few changes to the source.

\section{Template Overview}
As noted in the introduction, the ``\verb|acmart|'' document class can
be used to prepare many different kinds of documentation --- a
double-blind initial submission of a full-length technical paper, a
two-page SIGGRAPH Emerging Technologies abstract, a ``camera-ready''
journal article, a SIGCHI Extended Abstract, and more --- all by
selecting the appropriate {\itshape template style} and {\itshape
  template parameters}.

This document will explain the major features of the document
class. For further information, the {\itshape \LaTeX\ User's Guide} is
available from
\url{https://www.acm.org/publications/proceedings-template}.

\subsection{Template Styles}

The primary parameter given to the ``\verb|acmart|'' document class is
the {\itshape template style} which corresponds to the kind of publication
or SIG publishing the work. This parameter is enclosed in square
brackets and is a part of the {\verb|documentclass|} command:
\begin{verbatim}
  \documentclass[STYLE]{acmart}
\end{verbatim}

Journals use one of three template styles. All but three ACM journals
use the {\verb|acmsmall|} template style:
\begin{itemize}
\item {\verb|acmsmall|}: The default journal template style.
\item {\verb|acmlarge|}: Used by JOCCH and TAP.
\item {\verb|acmtog|}: Used by TOG.
\end{itemize}

The majority of conference proceedings documentation will use the {\verb|acmconf|} template style.
\begin{itemize}
\item {\verb|acmconf|}: The default proceedings template style.
\item{\verb|sigchi|}: Used for SIGCHI conference articles.
\item{\verb|sigchi-a|}: Used for SIGCHI ``Extended Abstract'' articles.
\item{\verb|sigplan|}: Used for SIGPLAN conference articles.
\end{itemize}

\subsection{Template Parameters}

In addition to specifying the {\itshape template style} to be used in
formatting your work, there are a number of {\itshape template parameters}
which modify some part of the applied template style. A complete list
of these parameters can be found in the {\itshape \LaTeX\ User's Guide.}

Frequently-used parameters, or combinations of parameters, include:
\begin{itemize}
\item {\verb|anonymous,review|}: Suitable for a ``double-blind''
  conference submission. Anonymizes the work and includes line
  numbers. Use with the \verb|\acmSubmissionID| command to print the
  submission's unique ID on each page of the work.
\item{\verb|authorversion|}: Produces a version of the work suitable
  for posting by the author.
\item{\verb|screen|}: Produces colored hyperlinks.
\end{itemize}

This document uses the following string as the first command in the
source file:
\begin{verbatim}
\documentclass[sigconf]{acmart}
\end{verbatim}

\section{Modifications}

Modifying the template --- including but not limited to: adjusting
margins, typeface sizes, line spacing, paragraph and list definitions,
and the use of the \verb|\vspace| command to manually adjust the
vertical spacing between elements of your work --- is not allowed.

{\bfseries Your document will be returned to you for revision if
  modifications are discovered.}

\section{Typefaces}

The ``\verb|acmart|'' document class requires the use of the
``Libertine'' typeface family. Your \TeX\ installation should include
this set of packages. Please do not substitute other typefaces. The
``\verb|lmodern|'' and ``\verb|ltimes|'' packages should not be used,
as they will override the built-in typeface families.

\section{Title Information}

The title of your work should use capital letters appropriately -
\url{https://capitalizemytitle.com/} has useful rules for
capitalization. Use the {\verb|title|} command to define the title of
your work. If your work has a subtitle, define it with the
{\verb|subtitle|} command.  Do not insert line breaks in your title.

If your title is lengthy, you must define a short version to be used
in the page headers, to prevent overlapping text. The \verb|title|
command has a ``short title'' parameter:
\begin{verbatim}
  \title[short title]{full title}
\end{verbatim}

\section{Authors and Affiliations}

Each author must be defined separately for accurate metadata
identification. Multiple authors may share one affiliation. Authors'
names should not be abbreviated; use full first names wherever
possible. Include authors' e-mail addresses whenever possible.

Grouping authors' names or e-mail addresses, or providing an ``e-mail
alias,'' as shown below, is not acceptable:
\begin{verbatim}
  \author{Brooke Aster, David Mehldau}
  \email{dave,judy,steve@university.edu}
  \email{firstname.lastname@phillips.org}
\end{verbatim}

The \verb|authornote| and \verb|authornotemark| commands allow a note
to apply to multiple authors --- for example, if the first two authors
of an article contributed equally to the work.

If your author list is lengthy, you must define a shortened version of
the list of authors to be used in the page headers, to prevent
overlapping text. The following command should be placed just after
the last \verb|\author{}| definition:
\begin{verbatim}
  \renewcommand{\shortauthors}{McCartney, et al.}
\end{verbatim}
Omitting this command will force the use of a concatenated list of all
of the authors' names, which may result in overlapping text in the
page headers.

The article template's documentation, available at
\url{https://www.acm.org/publications/proceedings-template}, has a
complete explanation of these commands and tips for their effective
use.

Note that authors' addresses are mandatory for journal articles.

\section{Rights Information}

Authors of any work published by ACM will need to complete a rights
form. Depending on the kind of work, and the rights management choice
made by the author, this may be copyright transfer, permission,
license, or an OA (open access) agreement.

Regardless of the rights management choice, the author will receive a
copy of the completed rights form once it has been submitted. This
form contains \LaTeX\ commands that must be copied into the source
document. When the document source is compiled, these commands and
their parameters add formatted text to several areas of the final
document:
\begin{itemize}
\item the ``ACM Reference Format'' text on the first page.
\item the ``rights management'' text on the first page.
\item the conference information in the page header(s).
\end{itemize}

Rights information is unique to the work; if you are preparing several
works for an event, make sure to use the correct set of commands with
each of the works.

The ACM Reference Format text is required for all articles over one
page in length, and is optional for one-page articles (abstracts).

\section{CCS Concepts and User-Defined Keywords}

Two elements of the ``acmart'' document class provide powerful
taxonomic tools for you to help readers find your work in an online
search.

The ACM Computing Classification System ---
\url{https://www.acm.org/publications/class-2012} --- is a set of
classifiers and concepts that describe the computing
discipline. Authors can select entries from this classification
system, via \url{https://dl.acm.org/ccs/ccs.cfm}, and generate the
commands to be included in the \LaTeX\ source.

User-defined keywords are a comma-separated list of words and phrases
of the authors' choosing, providing a more flexible way of describing
the research being presented.

CCS concepts and user-defined keywords are required for for all
articles over two pages in length, and are optional for one- and
two-page articles (or abstracts).

\section{Sectioning Commands}

Your work should use standard \LaTeX\ sectioning commands:
\verb|section|, \verb|subsection|, \verb|subsubsection|, and
\verb|paragraph|. They should be numbered; do not remove the numbering
from the commands.

Simulating a sectioning command by setting the first word or words of
a paragraph in boldface or italicized text is {\bfseries not allowed.}

\section{Tables}

The ``\verb|acmart|'' document class includes the ``\verb|booktabs|''
package --- \url{https://ctan.org/pkg/booktabs} --- for preparing
high-quality tables.

Table captions are placed {\itshape above} the table.

Because tables cannot be split across pages, the best placement for
them is typically the top of the page nearest their initial cite.  To
ensure this proper ``floating'' placement of tables, use the
environment \textbf{table} to enclose the table's contents and the
table caption.  The contents of the table itself must go in the
\textbf{tabular} environment, to be aligned properly in rows and
columns, with the desired horizontal and vertical rules.  Again,
detailed instructions on \textbf{tabular} material are found in the
\textit{\LaTeX\ User's Guide}.

Immediately following this sentence is the point at which
Table~\ref{tab:freq} is included in the input file; compare the
placement of the table here with the table in the printed output of
this document.

\begin{table}
  \caption{Frequency of Special Characters}
  \label{tab:freq}
  \begin{tabular}{ccl}
    \toprule
    Non-English or Math&Frequency&Comments\\
    \midrule
    \O & 1 in 1,000& For Swedish names\\
    $\pi$ & 1 in 5& Common in math\\
    \$ & 4 in 5 & Used in business\\
    $\Psi^2_1$ & 1 in 40,000& Unexplained usage\\
  \bottomrule
\end{tabular}
\end{table}

To set a wider table, which takes up the whole width of the page's
live area, use the environment \textbf{table*} to enclose the table's
contents and the table caption.  As with a single-column table, this
wide table will ``float'' to a location deemed more
desirable. Immediately following this sentence is the point at which
Table~\ref{tab:commands} is included in the input file; again, it is
instructive to compare the placement of the table here with the table
in the printed output of this document.

\begin{table*}
  \caption{Some Typical Commands}
  \label{tab:commands}
  \begin{tabular}{ccl}
    \toprule
    Command &A Number & Comments\\
    \midrule
    \texttt{{\char'134}author} & 100& Author \\
    \texttt{{\char'134}table}& 300 & For tables\\
    \texttt{{\char'134}table*}& 400& For wider tables\\
    \bottomrule
  \end{tabular}
\end{table*}

Always use midrule to separate table header rows from data rows, and
use it only for this purpose. This enables assistive technologies to
recognise table headers and support their users in navigating tables
more easily.

\section{Math Equations}
You may want to display math equations in three distinct styles:
inline, numbered or non-numbered display.  Each of the three are
discussed in the next sections.

\subsection{Inline (In-text) Equations}
A formula that appears in the running text is called an inline or
in-text formula.  It is produced by the \textbf{math} environment,
which can be invoked with the usual
\texttt{{\char'134}begin\,\ldots{\char'134}end} construction or with
the short form \texttt{\$\,\ldots\$}. You can use any of the symbols
and structures, from $\alpha$ to $\omega$, available in
\LaTeX~\cite{Lamport:LaTeX}; this section will simply show a few
examples of in-text equations in context. Notice how this equation:
\begin{math}
  \lim_{n\rightarrow \infty}x=0
\end{math},
set here in in-line math style, looks slightly different when
set in display style.  (See next section).

\subsection{Display Equations}
A numbered display equation---one set off by vertical space from the
text and centered horizontally---is produced by the \textbf{equation}
environment. An unnumbered display equation is produced by the
\textbf{displaymath} environment.

Again, in either environment, you can use any of the symbols and
structures available in \LaTeX\@; this section will just give a couple
of examples of display equations in context.  First, consider the
equation, shown as an inline equation above:
\begin{equation}
  \lim_{n\rightarrow \infty}x=0
\end{equation}
Notice how it is formatted somewhat differently in
the \textbf{displaymath}
environment.  Now, we'll enter an unnumbered equation:
\begin{displaymath}
  \sum_{i=0}^{\infty} x + 1
\end{displaymath}
and follow it with another numbered equation:
\begin{equation}
  \sum_{i=0}^{\infty}x_i=\int_{0}^{\pi+2} f
\end{equation}
just to demonstrate \LaTeX's able handling of numbering.

\section{Figures}

The ``\verb|figure|'' environment should be used for figures. One or
more images can be placed within a figure. If your figure contains
third-party material, you must clearly identify it as such, as shown
in the example below.
\begin{figure}[h]
  \centering
  \includegraphics[width=\linewidth]{sample-franklin}
  \caption{1907 Franklin Model D roadster. Photograph by Harris \&
    Ewing, Inc. [Public domain], via Wikimedia
    Commons. (\url{https://goo.gl/VLCRBB}).}
  \Description{A woman and a girl in white dresses sit in an open car.}
\end{figure}

Your figures should contain a caption which describes the figure to
the reader.

Figure captions are placed {\itshape below} the figure.

Every figure should also have a figure description unless it is purely
decorative. These descriptions convey what’s in the image to someone
who cannot see it. They are also used by search engine crawlers for
indexing images, and when images cannot be loaded.

A figure description must be unformatted plain text less than 2000
characters long (including spaces).  {\bfseries Figure descriptions
  should not repeat the figure caption – their purpose is to capture
  important information that is not already provided in the caption or
  the main text of the paper.} For figures that convey important and
complex new information, a short text description may not be
adequate. More complex alternative descriptions can be placed in an
appendix and referenced in a short figure description. For example,
provide a data table capturing the information in a bar chart, or a
structured list representing a graph.  For additional information
regarding how best to write figure descriptions and why doing this is
so important, please see
\url{https://www.acm.org/publications/taps/describing-figures/}.

\subsection{The ``Teaser Figure''}

A ``teaser figure'' is an image, or set of images in one figure, that
are placed after all author and affiliation information, and before
the body of the article, spanning the page. If you wish to have such a
figure in your article, place the command immediately before the
\verb|\maketitle| command:
\begin{verbatim}
  \begin{teaserfigure}
    \includegraphics[width=\textwidth]{sampleteaser}
    \caption{figure caption}
    \Description{figure description}
  \end{teaserfigure}
\end{verbatim}

\section{Citations and Bibliographies}

The use of \BibTeX\ for the preparation and formatting of one's
references is strongly recommended. Authors' names should be complete
--- use full first names (``Donald E. Knuth'') not initials
(``D. E. Knuth'') --- and the salient identifying features of a
reference should be included: title, year, volume, number, pages,
article DOI, etc.

The bibliography is included in your source document with these two
commands, placed just before the \verb|\end{document}| command:
\begin{verbatim}
  \bibliographystyle{ACM-Reference-Format}
  

\section{The Challenge and Dataset}\label{sec:conclusions}

Booking.com is the world’s largest online travel agency. It is a platform where millions of travelers find accommodations for their trips, and millions of accommodation providers list their hotels, apartments, guest houses, and other lodgings. \cite{kiseleva2015go, bernardi2019150}. 

Booking.com recently organized the \textit{WSDM WebTour 2021 Challenge} \cite{booking2021challenge}. The training dataset consists of over a million anonymized hotel reservations based on real data. Each reservation is a part of a customer’s trip, which includes at least four consecutive reservations. The challenge's goal is to recommend the final city of each trip, and we evaluated models using an accuracy metric for the first four items suggested. For Accuracy@4, the metric value is one when the real city is one of the four main suggestions and zero otherwise.



\section{Conclusion}

In this study, we present our approach for the \textit{WSDM WebTour 2021 Challenge}. We conducted several experiments using different session-based models to recommend the next destination on a trip. We modified the existing NARM model to add contextual information to the session, space-time modeling, and approaches to reduce the negative effect of class imbalance in the training phase. Our results show that it is possible to enhance the performance of state-of-the-art models through simple changes. The improved NARM outperforms all baseline models, and that the implemented training approaches can be used in any session-based recommendations model. 
\section{Experiments}\label{sec:experiments}
In this section, we present the experimental settings and results.

\subsection{Dataset}

We randomly partition the training dataset by trip, using 90\% of data for training and 10\% for model validation. We filtered trips with less than four cities or more than 10 and trips with a duration of more than 22 days in the training split. These values came from the initial analysis and are outliers. Table \ref{tab:data_statistics} shows the statistics of the dataset used in our experiments.

\begin{table}[!h]
  \caption{Statistics of the dataset used in our experiments}
  \label{tab:data_statistics}
  \begin{tabular}{ccccl}
    \toprule
    Split & Trips & Users & Cities\\
    \midrule
    train &  195917 & 181480 & 38542 \\
    test  &  21769 & 21524 & 15488 \\
   
  \bottomrule
\end{tabular}
\end{table}

\subsection{Baselines}

To show the effectiveness of our approach, we choose some popular baseline models used for session-based recommendation problems.

\begin{itemize}
  \item \textbf{Popularity}: Popular predictor recommends the most popular city in last city's country on the current trip.
  \item \textbf{Item-KNN}: This baseline recommends the most similar city to the last city on current trip. The similarity is defined as the cosine distance between the trip vector of the city.  It is similar to the co-occurrence approach.  \cite{davidson2010youtube}
  \item \textbf{Caser}: This baseline is a convolutional neural network (CNN) approach for a sequential recommendation. The Convolutional Sequence Embedding Recommendation Model (Caser) embeds a sequence of recent items into an "image" in time and learns sequential patterns as local features. \cite{tang2018personalized}
  \item \textbf{SASRec}: SASRec is a self-attentive approach that captures users’ sequential behaviors and achieves state-of-the-art performance on sequential recommendation. \cite{kang2018self}
  \item \textbf{NARM}: Neural Attentive Session-based Recommendation is an encoder-decoder architecture with an attention mechanism to model the user's sequential behavior, which is then combined as a unified session representation later. \cite{li2017neural}
\end{itemize}

We choose the NARM model, which shows the best performance among the models listed above, such as our baseline model to improve during the competition. Therefore, our approach is an adaptation of the NARM model. Some approaches were modeled for the click prediction problem. For our experiments, we adapted the last layer of all deep learning models to produce a ranking list of all items,  $ y = [ y_{1}, y_{2}, y_{3}, ..., y_{n} ] $, that can occur on the current trip. Furthermore, we trained our model using the same loss function.

\subsection{Experimental Settings}

All our models were trained using similar parameters. We used 50-dimensional embeddings for all category features, and we normalized dense features using standard deviation. Optimization was carried out using Rectified Adam (RAdam), with a 0.001 learning rating and 0.01 weight decay, with mini-batch size fixed at 64. We truncated the trip history size using a fixed window of 10 time-steps with padding. The number of epochs was defined by early stopping using 10 steps, and we used cross-entropy as a loss function for most models or focal loss with 1 $\alpha$ and 3 $\gamma$. 

Finally, we use the MARS-Gym framework \cite{santana2020marsgym} to model, train, and evaluate all experiments described in this paper. All experiments are present and can be reviewed at \url{https://github.com/marlesson/booking_challenge}.

\subsection{Experimental Results}

Each user's embeddings are presented in the 2-D plane in Figure \ref{fig:user_emb}. We can see that the month is a good predictor of user behavior, and, in general, users who travel in the same month also share similarities. We can also notice that there is a mixed cluster in the center, maybe belonging to users who are likely to travel on different dates with greater frequency. Furthermore, we can use this representation for new users to improve a cold-start recommendation.

\begin{figure}[h]
  \centering
  \includegraphics[width=230pt]{images/user_emb_month.png}
  \caption{We used t-SNE to project 100-D into 2-D and produce this Figure. Each point represents a user, and the color shows the most frequent month that users traveling.}
  \Description{User embedding}
  \label{fig:user_emb}
\end{figure}

Each model's performance on the test dataset is summarized in Table \ref{tab:freq}. We can see that the Item-KNN method performs similarly to the Popularity method, which is a cold start method. Therefore, there is no benefit in adopting the Item-KNN method given the complexity posed by cold recommendations. 




\begin{table}[h!]
  \caption{Performance comparison of proposed method with baseline methods}
  \label{tab:freq}
  \begin{tabular}{lccc}
    \toprule
    Methods & ACC@4 (test)\\
    \midrule
    Popularity & 0.364   \\
    Item-KNN & 0.371   \\
   
    SASRec & 0.478   \\
    Caser & 0.484   \\
    NARM & 0.497   \\
    NARM V1 & 0.520   \\
    NARM V2 & 0.545   \\
   
   
   

\end{tabular}
\end{table}

SASRec and Caser are very different approaches, the former based on attention mechanism and later on horizontal and vertical convolutions. Both showed similar results, with Caser performing slightly better. However, the best base baseline model was NARM. We can see that the original NARM outperformed state-of-the-art baselines using the same input information. Therefore, we chose NARM in this study to improve the architecture and training phase. 

We evaluate two versions of NARM. In NARM V1, we use only improvements  that were applied in the training and regularization steps, with the same input data were used for other baseline models, but applying approaches \ref{sec:time_modeling}, \ref{sec:multi_task}, \ref{sec:loss_function}, \ref{sec:data_arg}. NARM V1 outperforms the original NARM with approximately $+4.62\%$ accuracy. This is a promising finding, as this indicates that these techniques can now be used for any other session-based models that are in need of improvement.

Finally, we apply all approaches present in this study to NARM V2, and it represents our final approach for the challenge. Figure \ref{fig:architecture} shows our final architecture. We found that NARM V2 outperforms the original NARM with approximately $+9.66\%$ accuracy. This supports the idea that when it is not possible to obtain an item or user metadata, session statistics can also be used as features. 



\section{Introduction}
Deep learning approaches for recommender systems have garnered significant attention owing to their potential for modeling long- and short-term user preferences.  Models using recurrent neural networks (RNNs), transformers, and attention mechanisms are handy for text, presenting encouraging results when used in session-based recommendation problems \cite{li2017neural, tan2016improved, volkovs2019robust, chen2019behavior, kang2018self, tang2018personalized}, where sequential information and short-term context are fundamental in recommending the next item of the session.

In general, session-based approaches use only the information from the item's interaction sequence in the session as a predictor of the next item. However, in some domains, contextual information within and between sessions and global information are essential to modeling user behavior. Recommending travel destinations has several particularities, such as the sequence of cities on a trip can contain noise, financial constraints may lead to diversions, and destinations are highly correlated to the moment in time and duration of the trip \cite{mizrachi2019combining, bernardi2019150, kiseleva2015go} .

The \textit{WSDM WebTour 2021 Challenge} \cite{booking2021challenge} organized by Booking.com, focuses on recommending travel destinations in a session. This study describes our approach for the \textit{WSDM WebTour 2021 Challenge} and proposes some approaches that can improve session-based models for recommender systems, especially with the highly time-dependent recommendation domains, noise information, and imbalanced classes. 

\section{Proposed Approaches}\label{sec:aproaches}


This section describes each approach used in the present study to improve our final model. First, we chose a state-of-the-art session-based model to improve (described in \ref{seq:architecture}). Further, we created new features from statistical information about users and cities and added time modeling to focus on the travel problem particularity (described in \ref{seq:feature_enginner} and \ref{sec:time_modeling}). Also, we have reduced the effect of the imbalance dataset using specific loss functions and multitask modeling (described in \ref{sec:loss_function} and \ref{sec:multi_task}). Finally, we improve the generalization of noise information through data augmentation (described in \ref{sec:data_arg}).

Each approach can be applied separately in other session-based models.


\subsection{Model Architecture}
\label{seq:architecture}

We use the architecture of the neural attentive recommendation machine (NARM) \cite{li2017neural} and adapt it for the traveling problem presented in this study. NARM uses an encoder-decoder architecture to address the session-based recommendations problems. According to the authors of the paper, the idea of NARM is to build a hidden representation of the current session using an RNN module. It converts the input click sequence $X = [x_1, x_2, ..., x_n]$ into a set of high-dimensional hidden representation with an attention signal that can be used to produce a ranking list of all items that can occur in the next step of the current session. 

Our approach uses categorical and dense features of the user, city, and trip combined with the trip history. The core of the NARM module is the same as the original paper, and we just changed the size of the inputs, the bottleneck with hidden representation, and the outputs.

The features are concatenated and follow two paths. The first group of features passes through an attention layer before being fed into the NARM module. It is an important step for relating different positions of the same input sequence. In the second path, the features bypass the feature bottleneck generated by the NARM module, which improves the decoder by providing it with more contextual information on the session.

Figure \ref{fig:architecture} shows the final architecture proposed with modifications described in this study. 


\begin{figure}[h]
  \centering
  \includegraphics[width=250pt]{images/Diagrama_NARM.pdf}
  \caption{The general architecture of the proposed model. Our approach uses categorical and dense features of the user, city combined with the trip history. The features follow two paths. First, the features pass through an attention layer before being fed into the NARM module and in the second path, the features bypass the feature bottleneck generated by the NARM module, which improves the decoder by providing it with more contextual information on the session.}

  \Description{The general architecture of the proposed model.}
  \label{fig:architecture}
\end{figure}

\subsection{Feature Engineer}
\label{seq:feature_enginner}

Our approach uses statistical features from users, cities, and trips combined with the trip's history information. For every trip, we extract more than 30 statistical features. Below we provide examples of the important features we use in the present study:

\textbf{User Statistics}
\begin{itemize}
    \item Number of trips
    \item Number of cities visited
    \item Average number of unique cities visited per trip
    \item Average trip size
    \item Average trip duration
    \item Most frequently traveled month
    \item Number of unique cities the user visited
    \item Number of unique countries the user visited
    \item Device used for booking
    \item Booker country
\end{itemize}

\textbf{City Statistics}
\begin{itemize}
  \item City interactions that have preceded the current trip
  \item City interactions that have preceded the current trip by user
  \item City interactions in the current trip
  \item Country interactions that have preceded the current trip
  \item Country interactions that have preceded the current trip by user
\end{itemize}

Furthermore, we employ user statistical features to create user embedding from an autoencoder trained on the same training split data. This approach proved helpful because only 0.7\% of the users have made more than one trip; the vast majority are new users. In addition to providing a dense representation of the user features, the autoencoder ensures that similar users are close to each other in the created vector space. Moreover, the autoencoder was trained separately from the recommendation model. Figure \ref{fig:user_emb} shows user embeddings colored by the most frequently traveled month. 

\subsection{Time Modeling}\label{sec:time_modeling}

In terms of tourism, a trip's location is merely a piece of information that can tell how good the place is for a trip. The date (moment in time) and duration of the trip are as crucial as, for example, sometimes, the traveler can only visit certain places only in specific seasons or specific time windows. Sometimes, it is not feasible to stay for a few days more.

Embeddings are highly useful for modeling categorical features such as the location, and RNNs are remarkably good for capturing sequential data's time-dependence. However, they are limited when the events have irregular time intervals, such as duration of stay on a specific city trip. Therefore, we need to incorporate time interval explicitly into the model as a time-space embedding. In this case, we use the approach presented in \cite{li2017time} concatenated with a start trip month embedding for time-space embedding.

Therefore, each city embedding presented in a trip follows through a time embedding layer. We incorporate time (start trip month) and duration (how long users stayed in a city) information. In the end, we obtain a space-time embedding. 

\subsection{Multi-Task Learning}\label{sec:multi_task}

The data about the cities in our dataset are imbalanced and have high dimensionality. In contrast, the data about countries are less imbalanced and have lesser dimensionality. Thus, we adapt our model to predict both to try to improve models' gradient signal. According to \cite{ruder2017overview}, multi-task learning acts as a regularizer by introducing an inductive bias into the model. As such, it reduces the risk of overfitting and the Rademacher complexity of the model, and thus, it has the ability to fit random noise.

We used two targets in the present study. The city of the next step interaction was the main target that we used to evaluate the model, and the city's country was used as the second target. The second target was used only for regularization and as inductive bias in the primary target. Both targets use the same loss function, and the final loss is a combination of the two.

\subsection{Loss Function}\label{sec:loss_function}

Focal loss \cite{lin2017focal} is very useful for training imbalanced datasets. It adds a weighted term in front of the cross-entropy loss to balance the gradient from positive and negative samples. Easily classified negatives comprise most of the loss and dominate the gradient; this effect can be reduced by using a focal loss strategy. We use focal loss for both targets, and the final loss is the average of each loss.

Formally, the focal loss is expressed as follows:
\begin{equation}
  FL(p) = -\alpha_t(1-p)^\gamma log(p)
\end{equation}
where $\gamma$ is adjusts the rate at which easy examples are down-weighted and $\alpha$ is a prefixed value between 0 and 1 to balance the positive-labeled samples and negative-labeled samples. We use Equation \ref{eq:loss} for calculation loss. 
\begin{equation}\label{eq:loss}
  L(p_c, p_o) = \beta FL(p_c) + (1-\beta) FL(p_o)
\end{equation} where $\beta$ is a balance parameter, and $p_c$ and $p_o$ are obtained from the target prediction.



\subsection{Data Augmentation}\label{sec:data_arg}

Data augmentation techniques have been widely used to enhance image \cite{shorten2019survey}, text \cite{wei2019eda}, or recommendations based models \cite{tan2016improved,de2015artificial}. The principal inputs of the model in the challenge are cities in the current session's history. These inputs can have noise, some users may take a long or short trip, and users can jump one or more cities on a popular route. We applied three different data augmentations steps to improve the accuracy and generalize the model. 

The first is a sequence preprocessing step proposed in \cite{tan2016improved,de2015artificial}, where we generate new samples using each trip's time-step. Given an input training trip $[c_1, c_2, c_3, ..., c_n]$, we generate the sequences and corresponding labels $([c_1], c_2), ([c_1, c_2], c_3), ([c_1, c_2, ..., c_{n-1}], c_{n})$ for training. We filter only trips with more than four cities. 

For each sample in training, we randomly choose a step in the trip to change. Given an input training trip $[c_1, c_2, c_3, c_4]$, we change it in three ways: remove a step as a dropout layer and generate sequences such as $[c_1, c_2,  c_4]$;  replace with a mask token (unknown) and generate sequences with the same size but with mask $[c_1, c_2,  c_3, c_{UKN}]$ ; and replace with a similar city, such as $[c_1, s_2,  c_3, c_4]$ where $s_2$ and $c_2$ are similar cities. The mask token will be the same as that used in production/inference mode when the model does not embed a specific city, and the similarity definition is the same as that we used for the Item-KNN model. 

We apply both methods to make the model less susceptible to overfitting or noise information. 

\section{Related Work}\label{sec:related_work}
