
\section{Limitations}


Social media does not necessarily represent offline events, 
and unlike the surveys we use for comparison, we cannot control for demographics.
We study texts written in English and platforms that attract a U.S. audience, as the authors are all most familiar with this setting.
We focus on one specific community out of many, and we do not expect that the findings in this paper will necessarily generalize outside of the U.S. or to other online spaces; 
indeed, one of our results indicates that different platforms display different patterns.
Finally, averaging over posts allows us to track patterns and make comparisons but can also reduce nuance.
Our work is best read in conjunction with ethnographic studies like \citet{homewood2017turned} and \citet{daley2014influences}, which highlight individual voices of birth control users.
\section{Introduction}


Birth control users include nearly two thirds of women in the U.S. between the ages of 15-49---almost 50 million women \citep{daniels2020current}.
In addition to providing a means of family planning, birth control methods can also be used to treat and manage many medical conditions, including acne, endometriosis, cancer risks, gender dysphoria, and irregular and painful menstruation \citep{schindler2013non, nahata2017gender}.
However, birth control methods are not one-size fits all, and the choices of whether to use birth control and which method to select are complicated by personal beliefs, cost and accessibility, and a wide array of side effects that are difficult to predict and identify \citep{manzer2022did,yoost2014understanding,polis2016typical}.


When navigating this decision, birth control users face a sensemaking challenge that involves dealing with social stigma, understanding contraceptive methods and potential side effects, and seeking normalcy after upsetting experiences seeking and using contraception.
This challenge leads many birth control users to the internet \citep{yee2010role,russo2013women}, where they can engage in community sensemaking activities, e.g., seeking and sharing information, reviews, advice, hypotheses, and stories \citep{carron2015help,yang2019seekers}.


Prior work has shown that these various sensemaking activities can vary in frequency across different online platforms \citep{choudhury2014seeking,rivas2020classification,zhang2021distress}.
Different healthcare conditions \citep{sannon2019really} and side effects \citep{choudhury2014seeking} can result in different behavior,
and recent work has demonstrated differences in the online sensemaking practices surrounding reproductive healthcare topics like pregnancy and vulvodynia \citep{young2019girl,andalibi2021sensemaking,chopra2021living}.
However, while birth control researchers have examined the frequencies of different method discussions over time \citep{nobles2019repeal,merz2020population}, a comparison across platforms of the difficult decision-making of birth control users and their sensemaking practices has so far been unexplored. 


We study birth control users' unique combination of sensemaking activities---deciding between methods, interpreting side effects, calculating risks, etc.---and we use computational text analysis methods to discover the intersections between these activities and online platforms.
Because different methods and side effects could lead to different activities, we first
explore \textit{which birth control methods and side effects are more likely to be discussed on different online platforms} (\textbf{RQ1}).
Using lexicons crafted for our online settings, we compare these discussion frequencies to the distribution of experienced side effects as reported via a nationally representative survey by \citet{nelson2018women}, indicating increased or decreased interest in discussing these concerns online.
Next, using topic modeling and hand-annotations, we discover and characterize \textit{the kinds of sensemaking activities that birth control users engage in online and how these differ by platform} (\textbf{RQ2}).
We contrast these activities with prior work on related online healthcare communities, and we highlight the particular strategies of birth control users.


\paragraph{Contributions.}
We identify and characterize how birth control users make sense of their experiences and options via three large English-language online platforms (Reddit, Twitter, and WebMD).
Their unique combination of sensemaking strategies included \textit{storytelling}, \textit{risk analysis}, \textit{timing and calculations}, \textit{causal reasoning}, \textit{method and hormone comparison}, and \textit{information and explanations}; in particular, \textit{storytelling} is used to prepare for and overcome painful insertion experiences.
We examine these patterns across birth control methods, side effects, and platforms, unlike prior work, and we compare our observations of side effect discussions to survey data, finding large variations and highlighting the importance of integrating different methods.
Throughout this study, we employ a mixture of lexicons (which we provide as a community resource\footnote{\url{https://github.com/maria-antoniak/birth-control-across-platforms}}), topic modeling, and hand annotation, demonstrating how a detailed and diverse analysis across online platforms can expand our understandings of social sensemaking, web and social media activity, and vital healthcare issues.


\section{Related Work}
\label{section-related-work}





\smallskip
\noindent\textbf{Sensemaking in online communities.}
As explained by \citet{weick2005organizing}, sensemaking necessarily involves communication and is an ``\textit{activity that talks events and organizations into existence},'' in part through retrospective storytelling.
It is a process that 
relies on collaborative problem solving \citep{pirolli2005sensemaking}, and more recently, it has been defined in closely related work by \citet{andalibi2021sensemaking} 
as ``\textit{how individuals make sense of complex phenomena by constructing mental models that draw on new or existing experiences, information, emotions, ideas, and memories}.''
In online healthcare communities, the individual works to make sense of their healthcare experience, and the community collectively gathers information, compares stories, and makes sense of a shared experience \citep{mamykina2015collective}. 
Sharing experiences can help narrators make sense of their stories \citep{tangherlini2000heroes} and can transfer important information to others without firsthand experience \citep{Bietti2019StorytellingAA}, leading users through a transformative process via the gathering and organizing of information \citep{genuis2017looking,patel2019feel}.
This process is contextual; each community can rely on different strategies \citep{young2019girl}.



\smallskip
\noindent\textbf{Healthcare activity across platforms.}
Specific platform affordances can facilitate different types of healthcare-related disclosure and interactions.
\citet{zhang2021distress} formulate a \textit{social media disclosure ecology} in which platform affordances like anonymity, persistence, and visibility control can predict, e.g., pandemic-related disclosures.
Health information seeking behavior can differ across platforms, with search engines more frequently used for serious and stigmatized conditions and Twitter more frequently used for symptoms \citep{choudhury2014seeking}, and community affordances such as threaded conversations and hashtags can influence participation decisions of those with invisible chronic health conditions \citep{sannon2019really}.
The distribution of content type can also differ by platform; e.g., prior work found that \textit{sharing experiences} is less frequent on social networks than forums, although these rates can depend on healthcare topic \citep{rivas2020classification}.




\smallskip
\noindent\textbf{Reproductive healthcare and online sensemaking.}
Studies of reproductive healthcare communities have emphasized some unique sensemaking practices.
For example, in a study of sensemaking practices after pregnancy loss, \citet{andalibi2021sensemaking} highlighted the importance of seeking emotional validation, rather than just information, when trying to re-discover normalcy.
\citet{young2019girl} similarly supported these dual information management and emotional needs in a study of a vulvodynia Facebook group,
and \citet{chopra2021living} discussed polycystic ovary syndrome and why personal tracking is an important communal activity that involves comparison to others' experiences.
Whether these themes hold for online birth control communities is not currently known.


Two recent studies have focused specifically on online discussions of birth control.
First, \citet{nobles2019repeal} used search queries to track interest in different birth control methods, finding that U.S. political events correlate strongly with increased information seeking behavior online.
Second, \citet{merz2020population} examined the prevalence and sentiment of tweets mentioning different birth control methods. 
The authors found that long-acting methods like the IUD were mentioned more often on Twitter, and the proportion of these tweets increased over time.
While most tweets expressing sentiment about contraception were negative, tweets about long-acting methods were more likely to express positive sentiment.


\section{Discussion}



 \paragraph{Methods and side effects across platforms.}
In response to our first research question, we find that birth control discussions on Twitter, WebMD, and Reddit substantially differ in their distributions of methods and side effect mentions.
We cannot claim that using a method or experiencing a side effect \textit{cause} people to choose a specific platform, but our observations add detail and sometimes contradict prior findings.
For example, in a study of general healthcare information seeking, \citet{choudhury2014seeking} found that people more often use search engines for serious and stigmatized conditions and more often use Twitter to discuss symptoms.
We do not include search engines in our study, but in our comparison across platforms, we find that Twitter has a higher frequency of severe side effect discussions for birth control, while WebMD and Reddit have higher frequencies of general side effect discussions.
Our focus on birth control might explain these variations, as many of the birth control side effects are themselves highly stigmatized (e.g., menstrual bleeding, vaginal discharge). 
The discussion of severe side effects on Twitter is likely related to Twitter's tendency toward sensational content, and could also explain the relative frequency of IUD discussion on Twitter; the IUD has been associated with both severe side effects and potential legal bans
\citep{nobles2019repeal}.
These findings emphasize the importance of considering platform setting when studying online patterns related to specific health conditions, medications, and side effects.



\paragraph{Online sensemaking practices of birth control users.}
In response to our second research question, our identification and comparison of topics that align with sensemaking themes reveals important activities specific to birth control and to the different platforms and methods.
Prior work has found that online communities focused on reproductive healthcare (e.g., pregnancy, vulvodynia) employ strategies related to validation \citep{andalibi2021sensemaking}, information management \citep{patel2019feel,young2019girl,andalibi2021sensemaking}, personal tracking \citep{chopra2021living}, and identifying causation \citep{patel2019feel}.
We found these themes again in our birth control communities, with variations; for example, personal tracking is also a prominent theme in birth control discussions, but it is focused on pill timings and calculations and on self-observations of side effects. 
Identifying causation is a common concern for those experiencing infertility \citep{patel2019feel}, and we find this theme again in our datasets but focused on side effects like weight gain.
This combination of themes reflects the unique problems facing birth control users, as they choose between ``least bad'' options, struggle to identify and treat side effects, determine risk of pregnancy given their circumstances, and avoid rare but alarming outcomes like stroke and heart attack.




\paragraph{Storytelling, pain, and the IUD.}
We would like to particularly highlight storytelling as an important sensemaking strategy for birth control users.
Storytelling is known to help communities work through trauma \citep{tangherlini2000heroes}, but birth control users employ storytelling specifically to address physical pain.
Pain is a major cross-cutting theme across the platforms; it is the most frequently and consistently discussed side effect, and it is most often mentioned alongside the IUD.
While \citet{barr2010managing} identifies pain as the second most common side effect (after bleeding changes) and \citet{dickerson2013satisfaction} identifies pain as the most common side effect for the IUD and second most common side effect for the implant, we find a much larger gap between pain and the next most frequent side effects in our analysis.
Pain is mentioned frequently in posts about the IUD and in posts whose probable topics are about seeking and sharing IUD insertion stories.
Pain is inherently a subjective experience that cannot be precisely communicated \citep{scarry1987body}, but sharing of personal experiences provides one way for a community to build a sense of what is \textit{normal} or \textit{to be expected}  \citep{patel2019feel,andalibi2021sensemaking}. 
We suggest that there is an urgent unmet need for (a) honest education and preparation before IUD insertions and (b) pain treatment options during this procedure.




\paragraph{Online discussions differ from survey reports.}
In comparison to the survey results in \citet{nelson2018women}, we find not only large differences in the frequency at which different side effects are discussed but also differences across the platforms.
It could be that the differences from the survey are due to the demographic distribution opting into the survey versus those opting to post online; future work surveying the demographic distribution of these users would address this question, but our results take a first step at measuring differences between these settings. 
We find, e,g., while mood changes are discussed at similar frequencies in comparison to the survey data across the platforms, other side effects like strokes, bloating, fatigue, bleeding, and dizziness might be discussed more, less, or at equal frequency ranks to the reports in \citet{nelson2018women}, depending on the platform.
Bloating is less frequently mentioned on all the platforms and for all the methods in comparison to the survey data, perhaps indicating a general lack of concern about this side effect in contrast to its reported frequency.
But bloating is much less frequent on Twitter for the pill and implant, which could also indicate that this particular setting is less suited to discussing this side effect, perhaps because of the embarrassing or intimate nature of this side effect.



\paragraph{Stigma, privacy, and contextual disclosures.}
Birth control can be a controversial, stigmatized, and intimate topic.
This can lead birth control users to seek out additional information privately.
For example, in a set of interviews of young Black and Hispanic women, \citet{yee2010role} found that a greater number reported seeking decision-marking support on the internet, citing its privacy, in comparison to other sources of information (e.g., talking to physicians, reading provided information).
Interpreting side effects, analyzing the risk of pregnancy, or normalizing a painful experience require disclosing personal details and stories.
Birth control users might analyze the risk and benefit of making these disclosures in certain settings and to certain audiences.
While social penetration theory \citep{altman1973social} posits that more disclosures are possible as social bonds deepen, prior work has also found that intimate language can be frequent among both close connections and strangers (but not in between) \citep{pei-jurgens-2020-quantifying}.
The variations we observe across methods, side effects, and sensemaking practices could be indicative of platform affordances for privacy, audience size, and anonymity, each of which can affect decisions to self-disclose.
For example, we found that side effect discussions are less frequent on Twitter, where users are facing a much larger and non-specialized audience, unlike the other platforms.
Giving users more platform-specific tools to control their audience size and membership \citep{,mondal2014understanding} could allow for more productive discussions of this sensitive topic.




\paragraph{Other factors influencing platform decisions.}
It is important to note that decisions to seek information online can correlate with demographic characteristics.
For example, in a survey of U.S. young adults, those with a sexual risk history (early sexual activity, involvement in an unintended pregnancy) less frequently reported using the internet as a source and more frequently reported seeking information from a doctor/nurse, and men more frequently reported using the internet than women \cite{khurana2015young}.
It is also possible that users follow a \textit{birth control journey}, where different needs at different points in one's journey can lead one to different platforms, as has been reported for other healthcare topics \citep{sannon2019really,andalibi2018announcing}.
These journeys can intersect with methods; for example, while the pill is a popular first method, many people report switching to the IUD as they gain more experience with birth control \citep{nelson2018women}.
This would mirror the journeys of those with invisible chronic illnesses who move from one platform to another as their needs evolve and as they grow more comfortable with self-disclosure \citep{sannon2019really}.
This is mirrored in intra-community research that models user trajectories in online cancer support groups, finding that users often transition from information-seeking to information-sharing roles over time \citep{yang2019seekers}. 




\paragraph{Recommendations.}
Our results show that patterns found in one platform are not necessarily replicable on other online platforms, even when grounded in the same health topic.
We recommend that social media researchers compare results across multiple platforms to increase confidence in shared patterns.
If relying on a single platform, its choice should be guided by an understanding of which platforms might be more less or conducive to discussions of the desired methods, side effects, or sensemaking activities.
Our work also highlights benefits of mixing computational tools like topic modeling with lexicon-based methods and hand-annotation, and how these methods can be used to characterize difficult-to-identify themes like sensemaking practices; we recommend taking this careful approach, putting unsupervised results in context with fine-grained measurements.


\section{Methods}


\subsection{Birth Control Method Lexicon}
\label{section:method-labeling}


To explore the prevalence across platforms of different birth control methods, we develop a lexicon to match each document to the primary method being discussed.
Due to data scarcity and space limitations, we do not consider methods that are not available on all three platforms (e.g., WebMD does not include the male condom), and of the remaining methods, we limit our analysis to the three most commonly discussed reversible, non-emergency methods, mirroring prior work such as \citet{merz2020population}.\footnote{Using the lexicons described above for Reddit, we find that the implant (8,578 posts) is discussed more than twice as often as the shot (3,529 posts) or barrier methods like the male condom (3,029 posts). This contrasts with reported rates of contraception use in the U.S., where the pill is used by 14.0\%, the IUD by 8.4\%, the male condom by 8.4\%, the implant by 2.0\%, and the shot by 2.0\% of women aged 15-49 \citep{daniels2020current}.}
Our goal is to measure \textit{prevalence of discussions} rather than \textit{prevalence of usage} because (1) we are interested in the level of interest and concern of birth control users regardless of actual usage and (2) other study designs (e.g., surveys) are better suited to studying usage rates.



We rely on sets of keywords to assign each text a primary birth control method.
Because of their different structures, each platform requires its own set of keywords and matching techniques.
All of these lexicons are made available for future research.
To estimate the lexicon performance, we checked 450 documents balanced across Reddit posts, Twitter posts, and Twitter replies.\footnote{We omitted Reddit comments because of our method detection strategy, which defaulted to the method in the parent post if no method was found in the comment; and we omitted WebMD reviews because our method detection strategy already used hand-labeling of medication tags.}
Our precision and recall scores were perfect (1.0).




\subsubsection{WebMD.}
The URLs for WebMD reviews include a \emph{drugname} parameter.
This parameter includes the medication name followed by its class. 
We were able to classify 99.2\% of the WebMD reviews by mapping these names to their corresponding method, and the remaining reviews were for methods outside our target three. 


\subsubsection{Reddit.}
\label{subsubsection:reddit-method-labeling}
To assign Reddit posts and comments to the birth control methods, we use a custom set of keywords.
In all cases, the text was assigned to the method that had the highest keyword count (of all the methods).\footnote{We could instead have assigned each Reddit post and comment to as many methods as were mentioned in its text, rather than assigning each post and comment only to the method mentioned most frequently. 
We chose the latter assignment because it better aligns with the WebMD assignments; WebMD reviews are explicitly assigned by the user to a single method (even if other methods are also mentioned).
We do not find that these different assignment techniques affect our distributional results.}
If the highest keyword count was not for one of our target three methods, or if multiple methods were mentioned with equal frequency, we discarded the text.
We follow a similar procedure for Reddit comments, but in cases where the comments do not contain a birth control keyword, we assign the comment to the birth control method of its parent post.


We developed the Reddit methods keyword set by (a) drawing terms from the WebMD keyword set, (b) drawing terms from the Twitter keyword set,
(c) supplementing the WebMD and Twitter sets with general terms that are overloaded on Twitter but usable on Reddit (e.g., ``pill''), and (d) iteratively running the assignments and examining the unassigned posts and comments.
We were able to assign all but 4.6\% of the Reddit posts and 32.6\% of the Reddit comments to a birth control method.
Manual examination of the unassigned posts reveals discussions of pregnancy scares, access (e.g., online appointments and prescriptions), and treatment of side effects with non-contraceptive medications.
The remainder of these unassigned posts discussed topics that are adjacent to birth control, e.g., insurance, menstruation, but are not explicitly connected to a birth control method.


\subsubsection{Twitter.}
We use a more limited keyword set, derived from a similar survey of birth control tweets by \citet{merz2020population}, to query for tweets about each of our target birth control methods.
Our focus was on precision rather than recall, as the Twitter API requires a keyword match to retrieve tweets on a particular topic; if we included keywords like ``pill'', our results would contain many false positives, unlike Reddit and WebMD where the topic is already constrained to birth control.
After gathering our initial Twitter dataset, we then apply the full Reddit keyword set, using the same methods described for Reddit above.
As with the Reddit data, we find that more texts can be assigned to only the pill, IUD, or implant than to a combination of methods; 12.7\% of posts and 17.6\% of the gathered tweets either mentioned multiple methods with equal frequency or most frequently mentioned a method not in our set of three target methods.





\subsection{Side Effects Lexicon}
\label{section:side-effects}


To measure the frequency of discussions about side effect birth control, we develop a lexicon of terms and patterns.
Because we do not use the side effects lexicon for data collection (unlike the methods lexicon), we can rely on one lexicon across our three datasets.
As with the methods lexicon, we focus specifically on measuring the \textit{prevalence of discussions} rather than the \textit{prevalence of experiences}, as these are better measured via surveys \citep{nelson2018women}.
The frequency of discussions could be influenced by the prevalence of experiences and the level of \textit{concern} birth control users on this platforms have about particular side effects, as well as platform setting and a user's individual goals.
We select patterns that match any discussion of the side effect, whether or not it is mentioned in the affirmative. 


We grounded our development of the side effects lexicon in prior work.
We matched the side effect categories from \citet{nelson2018women} as closely as possible; this study conducted a nationally representative survey of U.S. birth control experiences and reported prevalences of side effect experiences across different birth control methods.
In addition to these, we added lexicon categories for pain, skin changes, PMS, appetite changes, sexual partner feeling IUD strings, and heart attack.
We identified these additional categories and patterns from topic models trained on our datasets and from other work; for example, we also drew on work by \citet{barr2010managing} that discusses a variety of side effects and their known frequencies and associations with different birth control methods.
For each side effect, we then iteratively queried and made updates to the lexicon when we encountered false positives or false negatives.



\input{tables/lexicon_coverage}


\subsubsection{Lexicon evaluation.}

To evaluate our lexicon, for each side effect we randomly selected a set of matching and non-matching texts, balanced across platforms and match status, resulting in a set of 1,040 texts.
We then manually checked whether each text does or does not contain a discussion of the target side effect.
Across the side effects, we found precision of 0.98 and recall of 0.96.
We show the lexicon coverage in Table \ref{table:lexicon-coverage}.

\subsubsection{Comparing observations with survey responses.}

We compare the distributions of side effect mentions to prior work surveying people in the U.S. about their experiences with birth control side effects.
This allows us to compare the frequency of side effect experiences with the frequency of online discussions.
Differences between these distributions can indicate healthcare needs that users are addressing via the internet.
We compare to surveys (rather than controlled studies) because surveys better approximate the self-reported anecdotes and personal experiences shared online.
We first converted each distribution to a ranking, where lower ranks represent greater percents of discussions $r_{platforms}$ or experiences $r_{survey}$.
We then find the difference between the ranks for each side effect, $d_{ranks}$.
\begin{equation}
    d_{ranks} =  r_{survey} - r_{platforms}
\end{equation}
These ranks avoid issues in directly comparing percents, which might on average be higher or lower.
When $d_{ranks}$ is positive, it indicates more online discussion than expected given the frequency of reported side effect experiences. 




\begin{figure*}[t]
    \centering
    \includegraphics[width=0.49\textwidth]{figures/final/barplot.side_effects_distributions.pdf}
    \includegraphics[width=0.49\textwidth]{figures/final/barplot.bc_type_x_side_effect.pdf}
    \caption{Distribution of the side effect mentions across the platforms and methods. Bars represent the percents of documents for the specified platform or method mentioning \textit{any} side effect that also mention the \textit{specified} side effect. 
    See Table \ref{table:lexicon-coverage} for the denominator sizes. 
    Error bars indicate the standard deviation over 20 bootstrapped samples of the datasets.
    } 
    \label{figure:barplot-side-effects-distribution}
\end{figure*}




\begin{figure}[t]
    \centering

        \centering
        \includegraphics[width=0.49\linewidth]{figures/final/barplot.bc_type_dist.reddit_posts.normalized_by_year.pdf}
        \includegraphics[width=0.49\linewidth]{figures/final/barplot.bc_type_dist.twitter_posts.normalized_by_year.pdf}
        \includegraphics[width=0.49\linewidth]{figures/final/barplot.bc_type_dist.reddit_comments.normalized_by_year.pdf}
        \includegraphics[width=0.49\linewidth]{figures/final/barplot.bc_type_dist.twitter_replies.normalized_by_year.pdf}
        \includegraphics[width=0.49\linewidth]{figures/final/barplot.bc_type_dist.webmd_reviews.normalized_by_year.pdf}
    
    \caption{Document distributions of methods over time.}
    
    \label{figure:bc-type-distributions}
\end{figure}



\begin{figure}[!ht]

    \centering
    
    \includegraphics[width=0.1805\textwidth]{figures/final/barplot.side_effects_distributions.compare_to_nelson.ranks.pill.v.pdf}
    \includegraphics[width=0.135\textwidth]{figures/final/barplot.side_effects_distributions.compare_to_nelson.ranks.iud.v.pdf}
    \includegraphics[width=0.135\textwidth]{figures/final/barplot.side_effects_distributions.compare_to_nelson.ranks.implant.v.pdf}
    \\
    \includegraphics[width=0.25\textwidth]{figures/final/legend.png}
    
    \caption{Distribution of the side effect mentions across the platforms. Bars represent the \textbf{differences} between the rank reported in the survey results from \citet{nelson2018women} and the rank on the specified platform. Platform ranks are determined by first finding the frequency of side effect mentions; these are the percent of documents mentioning \textit{any} side effect that also mention the \textit{specified} side effect. See Table \ref{table:lexicon-coverage} for the denominator sizes. Bars to the \textbf{right} of the x-axis represent side effects that are mentioned \textbf{more frequently} on the platforms than are reported in the survey.}
    
    \label{figure:barplot-side-effects-distribution-comparison}
    
\end{figure}



\begin{figure*}[!ht]

    \centering
    \begin{subfigure}[b]{0.24\textwidth}
        \includegraphics[width=\textwidth]{figures/final/heatmap.topic_x_methods.percents.storytelling.pdf}
        \caption{Storytelling}
    \end{subfigure}
    \begin{subfigure}[b]{0.24\textwidth}
        \includegraphics[width=\textwidth]{figures/final/heatmap.topic_x_methods.percents.risk_analysis.pdf}
        \caption{Risk Analysis}
    \end{subfigure}
    \begin{subfigure}[b]{0.24\textwidth}
        \includegraphics[width=\textwidth]{figures/final/heatmap.topic_x_methods.percents.timing_and_calculations.pdf}
        \caption{Timing \& Calculations}
    \end{subfigure}
    \begin{subfigure}[b]{0.24\textwidth}
        \includegraphics[width=\textwidth]{figures/final/heatmap.topic_x_methods.percents.method_and_hormone_comparison.pdf} 
        \caption{Method \& Dose Comparison}
    \end{subfigure}
    

    \caption{Mean topic probabilities across methods for four of our sensemaking clusters. Rows are normalized to sum to one.}
    
    \label{figure:heatmaps-topics-methods}
    
\end{figure*}






\subsection{Sensemaking Topic Model}



We compare sensemaking activity distributions through an unsupervised, bottom-up analysis using topic modeling.
Topic modeling is an automatic method to identify prominent themes and discourses in a dataset \citep{blei2003latent} and is a popular unsupervised technique for the analysis of online health communities \citep{nobles2018std,abebe2020quantifying}.
While some sensemaking themes might be shared across settings, other themes might be more prevalent in certain platforms and side effects.
This process helps us to avoid biases in our data coding (which might overlook certain themes while over-representing others) by quickly revealing frequent themes in each dataset \citep{nelson2020computational}.



\subsubsection{Model training.}


We trained a latent Dirichlet allocation (LDA) topic model \citep{blei2003latent} on each of our datasets, by combining the Reddit posts and comments and the Twitter posts and replies for training, resulting in three models, using MALLET\footnote{\url{http://mallet.cs.umass.edu/}} for training.
LDA remains a highly consistent and reliable model \citep{harrando-etal-2021-apples,hoyle2022AreNT}, especially when trained via Gibbs sampling for smaller datasets.
We balanced each training set, sampling 8,000 documents for each method for each of the Reddit and Twitter datasets and 800 documents for each method for the WebMD dataset (see Figure \ref{figure:bc-type-distributions} for method distributions).
By balancing the training set, we avoid weighting the topics toward a certain method.


We removed a set of frequent stopwords,
numbers, punctuation, and duplicate documents from the training sets.
Removing stopwords and duplicate documents has been shown to improve the legibility of the final topics, whereas stemming can be harmful \citep{schofield-mimno-2016-comparing,schofield-etal-2017-pulling,schofield-etal-2017-quantifying}.
To avoid capitalization discrepancies,
we lowercase all text.
We experimented with different numbers of topics and found 35 to be interpretable across the datasets; at this number of topics, we observed topics that were neither too fine-grained (e.g., splitting a single theme across multiple topics) nor too high level (e.g., combining themes that should be separated), and that produced reasonable evaluation scores (see below).
However, we emphasize that there is no ``correct'' number of topics and that this method is used for exploration and interpretation. 





\subsubsection{Model evaluation.}
\label{topic-model-evaluation}


While we do not have space here to list the full sets of topics for each dataset, we provide the topics, most probable words, and example paraphrased documents in our code repository for manual examination.
We report human evaluation of our topics following the recommendations in \citet{hoyle2021automated}.
Using the \textit{word intrusion task} \citep{chang2009reading}, we show two expert annotators (the first two authors) and one non-expert annotator a set of the four most probable words plus an ``intruder'' word that has low probability for the current topic but high probability for another topic.
We report the proportion of topics for which the annotators identified the intruder.


We found that our Reddit and WebMD topics have performance much higher than a random baseline of 0.2 (annotator accuracies for Reddit: 0.71, 0.74, 0.77, WebMD: 0.54, 0.66, 0.77) while the Twitter topics have lower performance but are still substantially above the random baseline (Twitter: 0.46, 0.46, 0.51).
The first (non-expert) annotator consistently had lower scores.
The lower performance on Twitter is expected, as text processing methods are notoriously challenged by the short text lengths and non-standard language \citep{gimpel-etal-2011-part}, and the short tweets require contextual knowledge to interpret.
This vulnerability of the word intrusion task to esoteric topics is noted by \citet{hoyle2021automated}.


We also calculate the ``UMass'' Coherence: the log probability that a document containing at least one instance of a higher-ranked word also contains at least one instance of a lower-ranked word \citep{roder2015exploring}.
We find the highest mean coherence scores across topics for Reddit ($-416$), lower scores for WebMD ($-577$), and lowest scores for Twitter ($-712$).
We note the criticisms of this and other automatic metrics in \citet{hoyle2021automated}; in comparison to human evaluation, automatic metrics can exaggerate differences between models.



\subsubsection{Identifying sensemaking themes.}


Across the datasets, we find that many of the topics discuss birth control methods (including one method in their 10 most probable words), side effects, pregnancy, and access (costs, appointments).
We then identify a series of cross-cutting sensemaking topics.
These topics include discussions of information seeking/sharing, educational links and resources, how-to explanations, experience seeking/sharing, emotional support, and other sensemaking-related discussions.
Two researchers independently coded the topics as more or less related to sensemaking.
The researchers then conferred and agreed upon a final set of topics and assigned them to thematic clusters.
During this coding, we relied on the sensemaking definition from \citet{andalibi2021sensemaking}: ``\textit{sensemaking describes how individuals make sense of complex phenomena by constructing mental models that draw on new or existing experiences, information, emotions, ideas, and memories}.''
We also drew inspiration from prior work studying sensemaking in online healthcare communities, particularly those works also focused on reproductive healthcare (see \S Related Work).
We further explore and validate these sensemaking topics by hand-labeling a small subsection of the data with social support goals \citep{yang2019seekers}.
After coding 150 documents for each dataset, we measured the agreement between the annotators using Krippendorff $\alpha$, as each document could receive zero, one, or more labels.
Our agreement was acceptable, with a score across the labels of 0.74.\footnote{Lower scores are not surprising for subjective language labeling tasks \citep{artstein2008inter,godwin2016collecting}, and our scores are substantially higher than the agreement scores for a very similar classification task in \citet{rivas2020classification}.}



To compare the topics across the platforms, we aligned the topics from the different models using Jensen-Shannon divergence (JSD) for the word distributions associated with each pair of topics. 
JSD is frequently used in prior work to compare topic distributions \citep{hall-etal-2008-studying,fang2012mining}.
After manual examination of the ranked topic pairs, we categorized topics with JSD scores below 0.6 as aligned across the datasets, and those with scores above 0.8 we considered diverging.


\section{Data}
\label{section:data}


We collected data related to birth control from three prominent online platforms: posts and comments on \textit{r/BirthControl}, birth control reviews on \textit{WebMD}, and \textit{Twitter} posts and replies related to birth control.
These are all English-language subcommunities of larger websites, and while we have limited demographic information, we observe that the majority of location-specific posts (e.g., politics, insurance) are about the U.S.
Table \ref{table:data} summarizes these datasets.



\input{tables/data}



\paragraph{Reddit.}
r/BirthControl\footnote{\url{https://www.reddit.com/r/birthcontrol/}} is a user-created and user-moderated online forum dedicated to birth control.
Users are pseudonymous and range from one-time questioners to experienced question answerers. 
As of November 20, 2022, r/BirthControl had 107,537 members.
According to a Pew Research Center survey of U.S. residents, more men (15\%) than women (8\%) use Reddit, and more White (12\%) and Hispanic (14\%) than Black (4\%) survey respondents use Reddit \citep{perrin2019share},
but these platform-wide distributions are unlikely to be representative of r/BirthControl, for which more detailed demographics are unavailable.
Prior work on a related subreddit found that 81\% of the users identified themselves as white \citep{nobles2020examining}.


Posts tend to be longer than comments, there are more comments than posts in our dataset, and both posts and comments have increased in frequency over time.
For each document, we collected the title (for posts), text, date, and user-assigned tag (for posts).
We removed comments written by the parent post's author, and we also removed stickied comments (auto-generated or mod-written comments).
We did not include documents deleted by the time of collection.




\paragraph{WebMD.}
WebMD\footnote{\url{https://www.webmd.com/sex/birth-control/birth-control-pills}} is a healthcare website that has hosted news, information, prescription discounts, community forums, and user-written medication reviews.
We focused on the medication reviews as a contrast to the Reddit and Twitter datasets.
The majority of reviews are written by women (71\%), and unlike Twitter and Reddit, the website has declined in visitors and reviewers since 2009 \citep{yun2019decline}.
Unlike Twitter and Reddit, WebMD reviews do not allow users to interact directly, though reviews sometimes mention other reviews or the general trends observed in other reviews.


We gathered reviews by querying \emph{birth control} as a condition on the drugs and supplements portion of the site and selecting medications that had more than 5 reviews.\footnote{At the time of data collection, this search term yielded both preventative and emergency contraception, but the terms must be queried separately as of January 2021.}
For each WebMD review, we collect the birth control name, the date of the review, and the review text. 



\paragraph{Twitter.}
Twitter is a large social network where discussions range widely from personal to global topics.
Compared to the general public, Twitter users are more likely to be Democrats, and they also skew younger than users of YouTube, Facebook, or Instagram \citep{perrin2019share}.
The gender and racial distribution on Twitter are close to uniform; out of a set of U.S. survey respondents, 24\% of men and 21\% women report using Twitter, while 24\% of Black, 25\% of Hispanic, and  21\% of White respondents report using Twitter \citep{perrin2019share}.
On Twitter, birth control discussions take place in the context of many other discussion topics, and while users can group their conversations using hashtags, there are not separated communities like subreddits.
Moderation is organized by Twitter, rather by users as on Reddit.


Using the Twitter Academic API (v2), we collected all the English Twitter posts containing a set of keywords corresponding to our three target birth control methods.
Separately, we collected all the Twitter replies containing the same set of keywords.
Recent work has shown that this API returns reliable representations of the full tweet space \citep{Pfeffer2022ThisSS}.
See \S Methods below for the design of the Twitter-specific keywords.
We removed Twitter handles from tweet texts.
The high vocabulary size for Twitter posts (Table \ref{table:data}) partly reflects many unique URLs shared in these documents, which can be indicative of information-providing behavior.
There are many possible design decisions when gathering Twitter data, and we chose not to collect the full conversations (replies and parent posts of the keyword-containing posts and replies) to (a) limit the size of our collection, (b) target texts explicitly discussing birth control, and (c) more closely replicate prior work on birth control tweets \citep{merz2020population}.


\section{Broader Impacts, Ethics, and Limitations}
\label{section:ethics}


Our study was considered exempt under our institution's IRB.
However, while Reddit, Twitter, and WebMD posts and replies are ``public,'' they can contain highly personal information,
requiring a balance between potential harms and potential benefits to the community, as described in the guiding principles of the Belmont Report: \textit{respect for persons}, \textit{beneficence}, and \textit{justice}.\footnote{\url{https://www.hhs.gov/ohrp/regulations-and-policy}}
Considering possible harms, e.g., re-identification of those using stigmatized medications, we do not collect any user-specific information, and we do not infer medical conditions for individual users; instead, we rely on patterns averaged across many users.
We also anonymize and paraphrase any direct quotations.
To protect users' agency to edit or delete their data at its original source, we release our data collection lexicons but not copies of the collected data.
We balance these concerns and protective actions against the benefits of this research; among other benefits, our work highlights the unmet pain treatment needs of a population known to be discriminated against by physicians \citep{Samulowitz2018BraveMA} and examines the kinds of support needed by those facing difficult healthcare choices.
\section{Results}




\subsection{Results: Birth Control Methods Across Platforms}


We compare the rates of discussion of the different birth control methods across platforms and over time.
We find that the types of questions that users ask and the type of information they share differs based on birth control method.
We measure the rise and fall of discussions of different birth control methods over time, and we compare these distributions across platforms.
Importantly, these patterns do not necessarily indicate real-world increases or decreases in use of different methods, but they do show the concerns and interests of users on these platforms. 




Figure \ref{figure:bc-type-distributions} shows the frequency of documents for each medication method by platform over time. 
The pill is the most popular method in both \textit{Reddit Posts} and \textit{WebMD Reviews}, while the pill and IUD are tied in \textit{Reddit Comments}.
Discussions of the pill on WebMD are much more frequent that discussions of other birth control methods and do not show a rise in interest for the IUD, counter to both the other datasets and our original hypothesis.
On Twitter, the pill begins as the most popular but is replaced by the IUD in posts, while replies always center on the IUD.
Twitter replies also increase sharply after 2016, differing from the Twitter posts (so is unlikely to be a symptom of our keywords).
This suggests that the IUD generates more discussion on Twitter, especially post-2016, compared to the other birth control methods.
On Reddit, the number of posts discussing the pill and IUD are similar, with slightly more posts discussing the pill, though this difference is erased in the comments.
The implant is always the least discussed of the three methods.



\subsection{Results: Side Effects Across Platforms}


Figure \ref{figure:bc-type-distributions} shows the distributions of the side effects across the different platforms, where frequency proportions are calculated by dividing the number of texts mentioning the specified side effect by the total number of texts mentioning any side effect (i.e., $p(s_i|a)$ where $s_i$ is the specific side effect and $a$ is any side effect).
Discussions of different methods have changed dramatically over time on Twitter, with discussions of the IUD rising and the pill falling.
The pill remains most most mentioned in Reddit posts.


Figure \ref{figure:barplot-side-effects-distribution} shows the distribution of side effects across platforms.
We find that pain, cramps, menstrual bleeding irregularities, mood changes, and skin conditions are the most commonly discussed side effects.
In particular, \textit{pain is consistently and frequently discussed across the platforms} and is an outlier among the side effects.
Discussions of stroke are more frequent in Twitter posts than in the other datasets, but for all other side effects, Twitter has the lowest discussion frequencies (perhaps because of stigma around publicly sharing such information compared to the more frequently pseudonymous settings on Reddit and WebMD).
Except for pain, menstrual bleeding, and severe effects, WebMD has the highest frequency of side effects discussions compared to the other platforms.
In particular, WebMD has higher frequencies for discussions of mood changes, skin conditions, and headache discussions.
Reddit has has the highest frequency of discussions for menstrual bleeding irregularities.





Figure \ref{figure:barplot-side-effects-distribution} also compares the side effects by birth control method.
Across the birth control methods, we again find that pain is a frequently mentioned side effect, but it is most frequently mentioned in discussions that also mention the IUD rather than the pill or implant.
The implant is the only method shown to cause weight gain \citep{barr2010managing}, so this association is expected---but the discussions of weight gain for the other methods are less expected and could indicate that this potential side effect is a concern across methods.
Menstrual bleeding, mood changes, headache, and libido are least often discussed in with the IUD, as are general mentions of side effects,  while nausea, stroke, and skin conditions are most often discussed in discussions of the pill.






\subsection{Results: Comparison to Observed Distributions}


Figure \ref{figure:barplot-side-effects-distribution-comparison} shows the differences between our observed distributions of side effect mentions and the reported distributions of side effect experiences in \citet{nelson2018women} (described above).
We find large differences between how frequently side effects are discussed online compared with how frequently they are reported in the survey from \citet{nelson2018women}.
For example, strokes are rarely experienced according to \citet{nelson2018women}, but when mentioned with the pill, they are discussed more frequently on Twitter in comparison to other side effects; this is unsurprising, since sensational topics are frequently discussed on Twitter.
Mood changes are more likely to be discussed across the platforms for the implant in comparison to the survey data.
Dizziness is more likely to be discussed for the IUD than expected, while weight gain is universally discussed less frequently than expected.
These patterns indicate cases in which users turn to the internet more or less frequently than expected based on their self-reported experiences in the survey.




\subsection{Results: Sensemaking Across Platforms}


Our final set of sensemaking themes included: \textbf{storytelling} (e.g., \textit{implant insertion and removal} on Twitter), \textbf{risk analysis} (e.g., \textit{risks of serious side effects} on WebMD), \textbf{timing and calculations} (e.g., \textit{daily timing of pill} on WebMD), \textbf{method and dose comparison} (e.g., \textit{hormone dosages and comparison} on Reddit), \textbf{causal reasoning} (e.g., \textit{weight changes and causes} on Reddit), and \textbf{information and explanations} (e.g., \textit{research on the male pill} on Twitter).
We show four of these themes (others omitted for space) in Figure \ref{figure:heatmaps-topics-methods}, with the topics from each platform included in that theme and their probabilities for each birth control method.
We observe a greater number of storytelling topics for the IUD and implant than for the pill, and examination of these topics shows that insertion and removal experiences fuel this pattern.
Each method is associated with risks and is included in method comparisons, but the pill in particular is included in discussions of timing and calculations.



Using Jensen-Shannon divergence to compare topics across the platforms, we find that the most \textbf{aligned} ($JSD < 0.6$) topic categories were: \textit{weight changes}, \textit{general side effects}, \textit{IUD insertion experiences}, \textit{implant insertion experiences}, \textit{menstrual timing and cycles}, and \textit{bleeding changes}.
Each of our three datasets included at least one representative topic from these categories.


The most \textbf{diverging} ($JSD > 0.8$) topics were: \textit{causes and side effects of vaginal infections} and \textit{explanations of how birth control works} on Reddit; \textit{IUD jokes and random}, \textit{viral folk stories}, and \textit{implant news about unexpected experiences} on Twitter; and \textit{treating or causing other medical issues} and \textit{secondhand advice and experiences} on WebMD.
These topics were the most unique to their training dataset, without directly comparable topics in the other datasets.



