

\section{Introduction}\label{sec:intro}

While many loud gravitational-wave signals have been detected, much of the high precision science and new discoveries in the nascent field of gravitational-wave astronomy will benefit from noise-characterization and noise-mitigation techniques \cite{Davis:2021ecd,Robinet:2020lbf,LIGOScientific:2019hgc,TheLIGOScientific:2016zmo,Piccinni:2018akm,Pankow:2018qpo,Driggers:2018gii,McIver:2015pms}.

The data of gravitational-wave detectors is dominated by noise. This noise is by and large Gaussian with a stable spectrum, but $\lesssim 10\%$ of it may be infested by high-powered  short-lived disturbances (glitches) and by nearly monochromatic coherent spectral artefacts (lines) in a variety of amplitudes, from extremely large to extremely weak. 

Typically the short-lived glitches affect the sensitivity of short-lived signal searches while the coherent lines affect the sensitivity of searches for persistent signals.
But when a short-lived glitch is powerful enough, it can also temporarily degrade the sensitivity of searches for long-lived signals, by increasing the average noise-floor level in a broad frequency range for its duration.

Two noise-mitigating techniques are typically used to prepare the gravitational-wave data for searches: \textit{gating}, performed in the time-domain and \textit{line-cleaning}, performed in the frequency domain. Broadly speaking, the former is used to remove loud glitches and the latter to remove spectral lines. The latter is usually only used in searches for persistent signals or stochastic backgrounds \cite{Abbott:2021xxi,Steltner:2020hfd,Papa:2020vfz}

In this paper we illustrate a new gating application, \texttt{gatestrain}, which enables a more precise removal of glitches compared with other widely used and publicly available gating methods, discarding significantly less data. Furthermore our gating procedure does not rely on any single fixed threshold that establishes what data should be gated, but rather it adjusts the threshold based on the achieved noise reduction. These are important features when the glitches vary much from data-set to data-set, and within the same same data-set, because the method does not require time-intensive tuning of ad-hoc parameters.

We publish the gates found with our new method on the public O3 data of the Advanced LIGO detectors as well as the new gating application in the supplementary material \cite{suppMaterial}.

The paper is organised as follows. In Section \ref{sec:noise} we describe the noise disturbances in Advanced LIGO data, which are particularly detrimental to continuous-wave searches, and the typical mitigation techniques used to prepare the data before performing such searches. In Section \ref{sec:gatingMethod} we present the idea of time-domain gating and explain our new method \texttt{gatestrain}{}. The performance of \texttt{gatestrain}{} on Advanced LIGO data from the first, second and third observation runs (O1, O2 and O3a) is presented in Section \ref{sec:results}. In Section \ref{sec:discussion} we discuss our results.

\section{Noise and mitigation techniques}\label{sec:noise}

The present generation of gravitational-wave detectors operates in a noise-dominated regime. The noise is primarily Gaussian with two main types of deviations: short-lived non-Gaussian transients - {\it{glitches}} - and long-lived nearly monochromatic coherent artefacts - {\it{lines}}. 

Lines are instrumental or environmental disturbances manifesting as narrow spectral artefacts, sometimes visible as lines in the frequency domain of the raw data. These disturbances lead to false candidates in continuous-wave searches and in searches for stochastic backgrounds \cite{Abbott:2020mev,Zhang:2020rph}. A standard way to deal with lines has been to replace the affected frequency bins with Gaussian noise in the data input to the search, not allowing the excess power to ``spread" to many signal-frequency results. This method is called {\it{line cleaning}}. Line-cleaning relies on knowing where the spectral contamination occurs and hence on detector-characterization studies such as \cite{Covas:2018oik} that produce the so-called ``lines lists" that LIGO releases together with its data. 

Loud glitches impact the sensitivity of transient signal searches by contributing to the background distribution used to estimate the significance of any finding. But they also degrade persistent-signal searches in two ways: 1) a high-power glitch directly increases the noise floor in a broad frequency range and 2) a loud glitch invalidates one of the assumptions of the line-cleaning method, and introduces artefacts in the cleaned data. We will discuss this latter point in Section \ref{sec:lineCleaningWithGlitches}.

A typical mitigation technique for glitches is {\it{gating}}: the time-domain data affected by a glitch is simply removed. Gating is a standard step of the compact-binary coalescence search pipeline \texttt{pyCBC\,} \cite{Usman:2015kfa}, but continuous-wave search pipelines also use it \cite{Astone:2005fj,Piccinni:2018akm,Leaci:2010zz}. In fact the O1 data Einstein@Home search for continuous waves from Cassiopeia A, Vela~Jr.\ and G347.3 \cite{Ming:2019xse,Papa:2020vfz} used the gating on its data and specifically used the \texttt{pyCBC\,} gating module because of its ease of use and prompt availability.

\section{Time-domain gating}
\label{sec:gatingMethod}

The core idea of gating is to detect and remove glitches in the time-domain. The different applications differ mostly in how the glitch detection is done and what classifies as being part of the glitch. This has implications on the effectiveness of the gating and/or on how much ``tinkering" and tuning is necessary to achieve optimal gating in any specific data set. 

Typically gating in preparation for transient signal searches tends to be less aggressive in removing data than the gating in preparation for persistent signal searches, by using shorter gates and higher thresholds. With this approach a number of glitches survive, increasing the false alarm rate, but transient signals that may happen near glitches are not discarded together with the glitch. Persistent signals on the other hand are always-on, thus even with the most aggressive cleaning, only a small fraction of signal is removed. 

In this Section we describe our gating method that presents two novelties with respect to publicly available gating methods: 1) the adaptive determination of the gate duration 2) the self-adjusting amplitude threshold for gate-identification, based on a iterative data-quality check of the gated data.

In most gating methods employed in gravitational-wave searches, the glitch detection does not happen on the raw data. Our gating procedure uses the same initial signal-processing steps as \texttt{pyCBC\,}, up to the actual detection of glitches, when the two methods differ. The data is divided in chunks with durations on the order of few to tens of minutes. In \cite{Ming:2019xse,Papa:2020vfz} we used chunks that are $\SI{1800}{\second}$ long. For each chunk the following steps are taken:
\begin{itemize}
\item high-pass the data with an 8th-order Butterworth filter at $\SI{10}{\hertz}$. Since the released Advanced LIGO data is not to be used for astrophysical searches below 10 Hz, we refer to this as the input data.
\item let ${\mathcal{P}}_r(f)$ be the power spectral density noise floor (with units $[1/{\textrm{Hz}}]$) estimated from this data. Since the data is not stationary, in order to produce this ``reference" power spectral density we divide the chunk in O(100) segments, compute the noise spectrum on each of these and, bin per bin, take the median over all the realisations. 
\item take the Fourier-transform ${\tilde{h}} (f)$, whiten and obtain: $\tilde{h}_w(f) = {\frac {{\tilde{h}(f)}} { \sqrt{\mathcal{P}_r(f)} } }$
\item inverse-Fourier-transform and obtain ${h}_w(t)$ ($[\sqrt{\textrm{Hz}}]$)
\item gate the ${h}_w(t)$ time-series
\end{itemize}
This whitening process produces a ${h}_w(t)$ time-series that in Gaussian noise has a mean $\mu= 0$ and standard deviation $\sigma=1.0$ \cite{LIGOScientific:2019hgc}, with similar contributions from all frequencies. When a glitch happens, it is more visible in ${h}_w(t)$ than in the original $h(t)$, because its contribution is not ``hidden" by loud noise from the low frequencies. This is shown clearly in Figure~\ref{fig:exampleGatingTD}.

Periods that harbor glitches are identified in $h_w(t)$. The data in these periods is set to zero with a Tukey taper on either side of the period. With the expression gate, $g_i$, we refer to each set of neighbouring data points whose original value has been set to zero and their time-stamps: $g_i=(\{t\}_i,\{h^g_w\}_i)$.

The \texttt{pyCBC\,} gates are established based on two parameters: a threshold $H$ and a duration $\tau_{\textrm{dur}}$. All times $t_k$ are recorded where $\left| h_w(t_k)\right| \gt H$. The process of constructing the gates is strictly sequential. It starts with the first point $t_1$. This identifies the points lying within a distance $\tau_{\textrm{dur}} /2$ of $t_1$. The time of the largest $h_w$ amongst these, $t_1^\star$, is taken as the center of the first gate. All points $\in [t_1^\star-\tau_{\textrm{dur}} /2, t_1^\star+\tau_{\textrm{dur}} /2]$ are zeroed-out. A taper is also applied to either side of this interval to prevent discontinuities \cite{Usman:2015kfa}. The second point $t_2$ is the next $t_k$ that has not been affected by the gating at the previous iteration. The process repeats for $t_2$ as for $t_1$ and a second gate is identified. The process ends when there are no more $t_k$s. We note that gates may overlap.
The typical values for transient signal searches, e.g. in the first gravitational-wave transient catalogs \cite{LIGOScientific:2018mvr,Abbott:2020niy,Nitz:2018imz,Nitz:2019hdf,Nitz:2021uxj}, are $H=100\sqrt{\textrm{Hz}}~ (25  \sqrt{\textrm{Hz}})$, $\tau_{\textrm{dur}}=\SI{0.25}{\second}~(\SI{0.125}{\second})$ and a Tukey taper of $\SI{0.25}{\second}~(\SI{0.125}{\second})$ (for the latest catalogs). In \cite{Ming:2019xse} we used $H=50\sqrt{\textrm{Hz}}$, $\tau_{\textrm{dur}}=\SI{16}{\second}$ and a Tukey taper of $\SI{0.25}{\second}$. 

\begin{figure}[htbp]
	\centering
	\includegraphics[width = \columnwidth]{GatesDur.png}
	\caption{
		Cumulative histogram of measured glitch durations. Depending on run and detector $50\%$ to $80\%$ of glitches last less than one second, while less than a few percent last longer than O($\SI{10}{\second}$). The maximum glitch duration is a few tens of seconds for H1 in O1 and O2 data, and L1 in O1 data, and reaches nearly $\SI{300}{\second}$ in L1 O2 data. The glitch durations are measured with \texttt{gatestrain}{}.  Note that the x-axis is displayed in symlog, i.e. linear between $0-\SI{1}{\second}$ and logscale above.
	}
	\label{fig:motivateTDead}
\end{figure}


Fig. \ref{fig:motivateTDead} shows that glitches can last from fractions of a second to a few tens of seconds, with more than 60\% of the glitches in the O2 data lasting less than 1s. It also shows that there is a great variability in glitch duration, depending on the detector and on the run. For instance, in O1 $\sim 50\%$ of  glitches in either detectors last less than $\SI{1}{\second}$, whereas in O2 $\sim 50\%$ of the L1 glitches last less than  $\SI{0.3}{\second}$. 
This variability is hard to capture with simple glitch-detection schemes: for example for glitches having ``long tails" that do not make it above the single threshold, those tails remain undetected and are excluded from the gates.

We develop a more generic glitch-identification and -removal scheme, with a varying gate size, estimated on the data itself. This is particularly relevant when other  data, for example from environmental monitors around the detectors, is not available, as for the gravitational-wave data releases.

Our method uses three parameters: a high threshold $H^{\textrm{high}}$, a low threshold $H^{\textrm{low}}$ and a duration parameter $\tau_{\textrm{dur}}$.
The gates are constructed as follows: 
\begin{itemize}
\item All times $t^h_k$ are recorded where $\left|h_w(t^h_k)\right| \gt H^{\textrm{high}}$.  
\item All times $t^\ell_j$ are recorded where $\left|h_w(t^\ell_j)\right| \gt H^{\textrm{low}}$. 
\item The $t^\ell_j$ are then divided in groups such that for each member of the group there exists at least another member closer than $\tau_{\textrm{dur}}$. When there are no nearby points, a single-member group is created. 
\item We only keep those groups such that there exists at least a $t^h_k$ closer than $\tau_{\textrm{dur}}$ to at least one member of the group. 
\item For each of the surviving groups: all timestamps between the earliest and latest, plus a Tukey taper to either side, constitute a gate. 
\end{itemize}

The low threshold  is set as $H^{\textrm{low}}=n^{\ell} \sigma$, where $\sigma$ is an estimate of the standard deviation of well-behaved parts of the data. We have used 
the harmonic mean of the standard deviation of $h_w(t)$ from shorter duration chunks, say $\approx \SI{10}{\second}$ long, out of the the 30 minute segment under consideration. We use the harmonic mean so that $\sigma$ is not affected by the presence of disturbances. We set $n^{\ell}$ to be high enough that Gaussian noise fluctuations at such level are rare, typically $n^{\ell}  \approx 5.5$. 


The high threshold is crucial because whether a glitch is identified, hinges on there being $|h_w(t)|$ values above $H^{\textrm{high}}$. A too low $H^{\textrm{high}}$ leads to too many unnecessary gates and thus wasted data, while a too high $H^{\textrm{high}}$ leads to missed glitches. We use an iterative lowering of $H^{\textrm{high}}$ and evaluate the performance of the gating at each threshold. We stop lowering the threshold when it has reached a pre-set minimum value or when the measured performance is satisfactory.

As an indicator of the performance of the gating we take the quantity
\begin{align}
R = \frac{1}{N_f}\sum_{f_i}^{N_f} \frac{\mathcal{P}(f_i)}{\mathcal{P}_{r}(f_i)}, 
\end{align}
where $\mathcal{P}$ is the power spectral density from the gated data and $\mathcal{P}_{r}$ is the reference power spectral density described in Section \ref{sec:gatingMethod}\footnote{In \texttt{gatestrain}{} it is alternatively possible to specify an external file which holds the reference PSD. This could e.g. be calculated by taking the harmonic mean over the full run. Since the detector changes on various timescales, this method is not recommended.}. The sum is over frequency bins $f_i$.  Experience has shown that using a $\approx5\sim10\,\si{Hz}$ band between $\SI{25}{\Hz}$ and $\SI{40}{\Hz}$, depending on the run, without loud lines or disturbances is suitable to identify most glitches. The reason for this lies in the character of the LIGO data, with most glitches having spectral content at lower frequencies. A value of $R\approx1$ indicates that the gated time series is not / no more affected by glitches. Value of $R>1$ indicates the presence of glitches in the gated data. 

At the first iteration we use a high threshold,  $H^{\textrm{high}} \approx 50 ~\sigma$, gate the data and compute $R$. If this ratio exceeds a threshold $R^\text{th}$, we reduce $H^{\textrm{high}}$, gate the data and check $R$ again. We continue until either $H^{\textrm{high}}$ reaches a minimum value or $R$ becomes small enough. In the O1 and O2 data we found that decreasing the $H^{\textrm{high}}$ threshold by $10$ at each iteration, and setting the $R$ and the $H^{\textrm{high}}$ thresholds to $1.05$ and $20$, respectively, achieved stable and good performance. On O3 data the same choices gave very good performance. 
We note that a reasonable choice for the $H^{\textrm{high}}$ could be to set it equal to $H^{\textrm{low}}$, but in O1 and O2 data this leads to sacrificing a lot more data, for a very small decrease in noise level. For this reason we leave it as a free parameter.

\begin{figure*}[bthp]
	\centering
	\includegraphics[width=\textwidth]{exampleGatedStrain.png}
	\caption{Example of how the gating procedure works using three snippets of data spanning $\approx \SI{270}{\second}$. 
	The lower panel shows the original strain data ${h}(t)$ (blue) and the gated ${h_g}(t)$ (orange). The insets magnify the input strain to show where the glitch occurs. The three snippets of data present glitches of different size. The upper panel shows the absolute value of the whitened time series $|{h}_w(t)|$(red), which is the quantity used to detect glitches, as explained in the main text. The glitch-detection threshold are the (blue) horizontal lines, solid and dashed. 
	}
	\label{fig:exampleGatingTD}
\end{figure*}

\begin{figure}[htbp]
	\centering
	\includegraphics[width=\columnwidth]{exampleASD.png}
	\caption{Amplitude spectral density (ASD) of the data during the $\SI{1800}{\second}${} from which the snippets of Fig. \ref{fig:exampleGatingTD} are taken. We show the ASD of the data at different stages of the gating process. The noise floor of the original data (top curve, blue) is significantly higher than the reference ASD (dashed line). In the first iteration (purple) the glitch from the central data snippet is removed, resulting in a vastly improved ASD, but the comparison to the reference ASD shows potential for further improvement below $\sim\SI{100}{\hertz}$. In the second iteration the left data-snippet glitch of the previous figure is found and removed, and this lowers the noise floor the low-frequency region (orange). At this point gating ends because the reference power spectral density ${\mathcal{P}}_r$ is matched. The glitch in the right hand-side data snippet is not large enough to make a difference and it is left ungated. For reference, the bottom line (green) shows the ASD after removing this third glitch, which our procedure does not do.
	}
	\label{fig:exampleGatingASD}
\end{figure}

An example of this process with $\tau_{\textrm{dur}}=\SI{3}{\second}$ and a Tukey window of $\SI{0.25}{\second}$,  is shown in Fig.~\ref{fig:exampleGatingTD} (time-domain) and Fig.~\ref{fig:exampleGatingASD} (frequency-domain). Three glitches are clearly seen in the time-domain plot. In the first iteration, with the highest  $H^{\textrm{high}}$ threshold, only the middle peak is detected and removed. The resulting amplitude spectral density (ASD, equal to $\sqrt{{\mathcal{P}}}$)  is shown in purple in Figure~\ref{fig:exampleGatingASD}. The comparison with the reference ${\mathcal{P}}_r$ yields $R\ge 1.32$ and indicates that there could be more glitches, so the process continues with a lower values of $H^{\textrm{high}}$. In the second iteration $H^{\textrm{high}}=40$ and the second glitch is included. After removing the second glitch $R\leq 1.05$ and this concludes the gating procedure. The third peak is thus not gated as it has not enough impact on the sensitivity. For comparison the ASD after removal of the third peak is also shown. 

We note that very loud glitches can lead to a ringing of the whitening filter and thus an overestimation of the glitch duration : 
the full glitch is removed, but the method's efficiency in saving data is degraded.
\begin{table}[ht]
\begin{tabular}{|c|c|c|c|}
\hline
\hline
 Data & SFTs & Data [h] & Data [d]\\
\hline
\hline
 \TBstrut O1-H1& \OOneHtotalSFTs & 1562 & 65.1 \\ 
\hline
 \TBstrut O1-L1& \OOneLtotalSFTs & 1060 & 44.2 \\ 
\hline
 \TBstrut O2-H1& \OTwoHtotalSFTs & 2533 & 105.5 \\ 
\hline
 \TBstrut O2-L1& \OTwoLtotalSFTs & 2492 & 103.8 \\ 
 \hline
 \TBstrut O3a-H1& \num{5977} & \num{2988.5}  & \num{124.5} \\ 
 \hline
 \TBstrut O3a-L1& \num{6377} & \num{3188.5} & \num{132.9}\\
\hline
\hline
\end{tabular}
\caption{Data that we used the gating procedure on.}
\label{tab:DataUsedSummary}
\end{table}
\section{Results}\label{sec:results}

We consider public data from the O1, O2 and O3a Advanced LIGO runs \cite{o1data,o2data,Vallisneri:2014vxa}. We produce half-hour baseline (Short) Fourier transforms, SFTs \cite{SFTs}, as summarized in Table \ref{tab:DataUsedSummary}. We prepare different data sets, depending on the run, to compare the performance of our method with existing ones. Table~\ref{tab:TimeGatedgateAmount} summarizes how much data was gated by the different procedures.

\subsection{Gating the O1 and O2 LIGO data}
\label{sec:O1O2}

We prepare three different sets, one without gating, one with the \texttt{pyCBC\,}{} gating procedure used in \cite{Ming:2019xse}, with $\tau_{\textrm{dur}}$ conservatively set to $\SI{16}{\second}$, and one with our new gating \texttt{gatestrain}. 


Table \ref{tab:TimeGatedgateAmount} shows how many gates were used and how much time was zeroed-out by each procedure. While the number of gates of our procedure is similar or larger than the number of gates identified by \texttt{pyCBC\,}, overall our \texttt{gatestrain}{} removes much less data: $\sim4-9\%$ of what is removed by \texttt{pyCBC\,}. For instance of the \OOneHtotalSFTs{} O1-H1 SFTs, \OOneHgatedSFTs{} are affected by one or more glitches which are gated with a total of \OOneHgts{} gates and $\SI{\OOneHdurh}{\hour}$ of time-domain data lost. In comparison, \texttt{pyCBC\,} gating results in a slightly lower number of affected SFTs (\OOneHpygatedSFTs) but $\SI{\OOneHpydurh}{\hour}$ of lost data. 
For the L1 detector both methods find roughly $\sim180$ O1 SFTs to be affected by glitches, where \texttt{pyCBC\,} gating removes $\SI{\OOneLpydurh}{\hour}$ of time-domain data while \texttt{gatestrain}{} removes $\SI{\OOneLdurh}{\hour}$. 
\begin{table}
\begin{tabular}{|c|c|c|c|c|c|}
\hline
\hline
 \multirow{2}{*}{Data} &   \multirow{2}{*}{Method} &   \multirow{2}{*}{SFTs} &   \multirow{2}{*}{Gates} &  \multicolumn{2}{c|}{How much data zeroed-out}\\
 \TBstrut 	 	&  	 &  w/ gates &  &  [s] &  [h] \\
\hline
\hline
 \TBstrut O1-H1 & \texttt{pyCBC\,} & \num{\OOneHpygatedSFTs} & \num{\OOneHpygts} & \num{\OOneHpydur} & \num{\OOneHpydurh}\\ 
\hline
 \TBstrut O1-H1 & \texttt{gatestrain}{} & \num{\OOneHgatedSFTs} & \num{\OOneHgts} & \num{\OOneHdur} & \num{\OOneHdurh} \\ 
\hline
 \TBstrut O1-L1 & \texttt{pyCBC\,} & \num{\OOneLpygatedSFTs} & \num{\OOneLpygts} & \num{\OOneLpydur} & \num{\OOneLpydurh} \\ 
\hline
 \TBstrut O1-L1 & \texttt{gatestrain}{} & \num{\OOneLgatedSFTs} & \num{\OOneLgts} & \num{\OOneLdur} & \num{\OOneLdurh} \\ 
\hline
 \TBstrut O2-H1 & \texttt{pyCBC\,} & \num{\OTwoHpygatedSFTs} & \num{\OTwoHpygts} & \num{\OTwoHpydur} & \num{\OTwoHpydurh}\\ 
\hline
 \TBstrut O2-H1 & \texttt{gatestrain}{} & \num{\OTwoHgatedSFTs} & \num{\OTwoHgts} & \num{\OTwoHdur} & \num{\OTwoHdurh} \\ 
\hline
 \TBstrut O2-L1 & \texttt{pyCBC\,} & \num{\OTwoLpygatedSFTs} & \num{\OTwoLpygts} & \num{\OTwoLpydur} & \num{\OTwoLpydurh} \\ 
\hline
 \TBstrut O2-L1 & \texttt{gatestrain}{} & \num{\OTwoLgatedSFTs} & \num{\OTwoLgts} & \num{\OTwoLdur} & \num{\OTwoLdurh} \\
 \hline
\TBstrut O3a-H1 & \texttt{self-gating}{} & \num{5695} & \num{20205} & \num{141070.9} & \num{39.2} \\ 
\hline
\TBstrut O3a-H1 & \texttt{gatestrain}{} & \num{4885} & \num{11581} & \num{38236.8} & \num{10.6} \\ 
\hline
\TBstrut O3a-L1 & \texttt{self-gating}{} & \num{6366} & \num{49653} & \num{1441915.8} & \num{400.5} \\ 
\hline
\TBstrut O3a-L1 & \texttt{gatestrain}{} & \num{5825} & \num{21525} & \num{742985.6} & \num{206.4} \\
\hline
 \hline
\end{tabular}
\caption{Total amount and duration of gates for each detector / observing run produced by \texttt{gatestrain}{} and  \texttt{pyCBC\,} as used in \cite{Ming:2019xse} or LIGO's \texttt{self-gating}{}{} procedure used on O3 data \cite{RilesZweizig2021}. Since \texttt{pyCBC\,} gates may overlap, their total duration is less than the total number of gates times $\SI{16}{\second}$.}
\label{tab:TimeGatedgateAmount}
\end{table}

\begin{figure*
	\centering
	{\includegraphics[width=\textwidth]{{ASDs_v2}.png}}
	\caption{The upper rows show the average Amplitude Spectral Densities (ASDs) of H1/L1 data from the O1, O2 and O3a runs, before and after removal of glitches with our  \texttt{gatestrain}{}-method. The ASD is estimated as the square root of the arithmetic mean over all timestamps, bin per bin, of the power spectral density. For O1 and O2 we also plot the pyCBC-gated data in gold, and for O3 the ASD of the \texttt{self-gated}{}{} data \cite{RilesZweizig2021} in orange. \texttt{gatestrain}{} achieves a noise floor level comparable to other methods, apart for L1 O3a data between 20 Hz and 50 Hz. The relative difference of \texttt{gatestrain}{}-gated data and \texttt{pyCBC\,}{}-/ \texttt{self-gated}{}{} data  $\xi\left(f\right) = { \left(ASD_{\textrm{\texttt{gatestrain}}} - ASD_{\texttt{self-gated}{},\texttt{pyCBC\,}}\right) }/{ ASD_{\textrm{\texttt{self-gated}{},\texttt{pyCBC\,}}}}$ is shown in the lower row respectively.}
	\label{fig:ASDComparisons}
\end{figure*}


The noise floor of gated data is lower than that of ungated data. Fig. \ref{fig:ASDComparisons} compares the amplitude spectral density (ASD) of gated data and ungated data. 

The actual improvements differ between detectors and runs:
an improvement of a factor greater than $3$ is seen in O1-H1 data, in the highest sensitivity region in frequency, and of  $\sim4\%$ in O1-L1 data . In O2 data gating significantly decreases the noise floor below 60 Hz in both detectors, and for H1 yields an appreciable decrease in the 100-450 Hz range. 

We demonstrate the gain in sensitivity in continuous-wave searches with a Monte Carlo simulation where we consider $500$ simulated continuous-wave signals with frequency between $20 - \SI{1000}{\hertz}$ distributed log-uniformly. The amplitude of the signals is such that they are clearly visible in the search results. The signals are added to the real data in the time-domain. The data is then treated as it would be treated for a search, i.e. it is gated and Fourier-transformed in chunks to yield the SFTs. We perform a perfectly matched single-template $\mathcal{F}$-statistic search \cite{Jaranowski:1998qm} using these SFTs, from non-gated and gated data. We compare the results in Fig. \ref{fig:recoveries}. 
An overall positive effect of gating can be seen, with a relative increase in detection statistic of up to $33\%$ for H1 O1 data. Since gating lowers the noise level more in the low-frequency region, the signal-recovery improves more for low-frequency signals than for higher-frequency signals. The \texttt{pyCBC\,}-gating results are comparable to the  \texttt{gatestrain}{} results, so in Figure~\ref{fig:recoveries} we only show the \texttt{gatestrain}{} results.

\begin{figure}[!hbtp]
	\centering
	\includegraphics[width=\columnwidth]{InjectionRecoveries_Single.png}
	\caption{Relative differences in the detection statistic $2F$ of $500$ recovered simulated signals in \texttt{gatestrain}-gated versus non-gated (left panel) SFTs. It can be seen that gating has an overall positive effect which varies depending on detector and observation run. 
	}
	\label{fig:recoveries}
\end{figure}

\begin{figure}[h!tbp]
	\includegraphics[width=0.95\columnwidth]{GlitchSFTExample.png}
	\caption{Amplitude spectral density (ASD) of an SFT disturbed by a loud glitch before and after gating. The top panel shows the ASD of the {\it {original data}} before and after line cleaning. The lower panel shows the ASD of the {\it{gated data}} with and without line cleaning. The noise floor is greatly reduced thanks to the gating and then line cleaning removes the peaks without the discontinuities evident in the upper plot.}
	\label{fig:lineCleanGating}
\end{figure}

\subsection{Gating and line-cleaning in the presence of glitches}
\label{sec:lineCleaningWithGlitches}
Gating also mitigates artefacts introduced by glitches at the frequencies cleaned-out in the frequency domain. The line-cleaning procedure used in many continuous-wave searches substitutes the data at frequency bins that have been flagged to harbour disturbances, with Gaussian noise. In these bins fake SFT data is created with a standard deviation consistent with the noise level estimated based on the real data, in nearby-frequency bins. If the data in these nearby bins is quite Gaussian, the fake noise will look like a realisation of noise from the nearby bins. But if the nearby noise has significant non-gaussian contributions, the fake noise will not look at all like the noise in the nearby bins, and in the presence of loud glitches, it will be higher. The gating removes these non-gaussian contributions and, with them, this type of problem. This is illustrated in Figure \ref{fig:lineCleanGating} that shows how the noise floor of the cleaned data is greatly reduced and that the gating before the line-cleaning allows for the lines to be removed without producing other spectral artefacts.

\subsection{Gating the O3 LIGO data}
\label{sec:O3}

The first six months of the O3 data (O3a) were publicly released shortly before the initial submission of this paper. This data presents multiple families of glitches that have required substantial effort by LIGO in order to work-around with an ad-hoc gating procedure \cite{APS2021}. The basic algorithm is called \texttt{self-gating}{}{} and it is described in \cite{RilesZweizig2021}. The resulting gates are released with \cite{RilesZweizig2021}. 

We apply our \texttt{gatestrain}{} to the O3a public data with minimal changes in parameters with respect to the O1/O2 data, in order to allow for a slightly more aggressive gating. This is justified because the O3 data is significantly more glitchy than the data from the two previous Advanced LIGO runs. At every iteration we change $H^{\textrm{high}}$ by 11, rather than 10; we set the smallest $H^{\textrm{high}}$ threshold to be $H^{\textrm{low}}$; we reduce the threshold for the gating result-check $R^\text{th}: 1.05 \rightarrow 1.01$.

Table \ref{tab:TimeGatedgateAmount} shows how many SFTs are affected by gates, and how many gates the \texttt{self-gating}{}{} method and our method produce. It also shows how much data is zeroed-out as a result of these gates. There is a caveat: \cite{RilesZweizig2021} exclude SFTs with a total gate duration longer than $\SI{30}{\second}$. The data zeroed-out in this way is included in the count given in the last two columns of Table~\ref{tab:TimeGatedgateAmount}. In order to make a fair comparison we also adopt this criterium and zero-out the entire SFT when our gate duration is longer than $\SI{30}{\second}$. We include this data in the zeroed-out count in the last columns of Table~\ref{tab:TimeGatedgateAmount} for the \texttt{gatestrain}{} method. With this convention \texttt{gatestrain}{} preserves $\num{222} \,\si{\hour}$  of data that the \texttt{self-gating}{}{} gates instead remove.

\texttt{gatestrain}{} achieves a more precise removal: Only $\num{14}~(\num{402})$ SFTs in H1 (L1) are excluded due to long gate duration when using \texttt{gatestrain}, while \cite{RilesZweizig2021} exclude $\num{61}~( \num{773})$ in H1 (L1), respectively. On ``good data", i.e. on SFTs that are free of long-duration gates, the gain is less dramatic, but it is still significant: \texttt{gatestrain}{} removes $\num{3.4}\,\si{\hour} ~(\num{10.1}\,\si{\hour}) $ in H1 (L1), whereas \cite{RilesZweizig2021} exclude $\num{8.2}\,\si{\hour} ~ (\num{13.5}\,\si{\hour}) $.

In the frequency range $\approx$ 20-50 Hz, the amplitude spectral density of the L1 data gated with \texttt{self-gating}{}{} is lower by $\leq 9\%$ with respect to the L1 data gated with our method. Elsewhere the performance of the two methods is comparable, as shown in Figure~\ref{fig:ASDComparisons}.

We also recover the fifteen isolated continuous-wave hardware injections below 2 kHz \cite{O3HardwareInjections} using the $\mathcal{F}$-statistic with comparable efficiency in both data sets. 
Like \cite{RilesZweizig2021} we too could not recover the signal at $\SI{12.34}{\hertz}$ due to the high noise in this frequency range.  The relative gain in detection statistic with respect to \cite{RilesZweizig2021} is a few percent.


We publish our O3a gates in the Supplemental Materials and at \cite{suppMaterial}. 


\section{Discussion}
\label{sec:discussion}

In this paper we present a new method to remove non-Gaussian noise transients in an overall fairly well-behaved noise background. The method extends the gating implementation by \cite{Usman:2015kfa} with two main novelties: 
i) the duration of glitches is measured and only the data affected by the glitch is removed ii) the amplitude threshold that defines glitch-data is not fixed, but rather it is adaptive and is iteratively changed during the gating procedure.

As shown in Fig. \ref{fig:motivateTDead}, there is no single typical glitch duration. Glitches come in various sizes and durations, and the glitch populations change from one run to the next. While the \texttt{pyCBC\,} method of \cite{Usman:2015kfa} is proven perfectly adequate for compact-binary-coalescence searches in O1 and O2, for continuous-wave searches our \texttt{gatestrain}{} method keeps more data untouched. For both H1 and L1 the number of gated SFTs increases slightly compared to the $\SI{16}{\second}$ fixed-gate-duration of \texttt{pyCBC\,} (as used in \cite{Ming:2019xse}). On the other hand, \texttt{gatestrain}{} removes less than $10\%$ of the data that \texttt{pyCBC\,} removes.

We note that the \texttt{pyCBC\,} gate duration $\tau_{\textrm{dur}}=\SI{16}{\second}$ that we used in \cite{Ming:2019xse} is cautiously long and thus unsurprisingly more data is lost. In Appendix \ref{sec:app3s16sgateDur} we show what happens with $\tau_{\textrm{dur}}=\SI{3}{\second}$ and $\tau_{\textrm{dur}}=\SI{0.25}{\second}$. While shorter \texttt{pyCBC\,} gate durations lead to a decrease in the amount of lost data, the noise-level decrease may be adversely affected.

Below $\sim\SI{1}{\kilo\hertz}$ the recovery of hardware and software injections shows SNR improvements compared to not gating and consistent with what is observed with \texttt{pyCBC\,} gating. No negative effect are recorded apart from regions of loud lines, e.g. violin modes, calibrations lines and power mains. These are cleaned anyway. 

Unlike \texttt{pyCBC\,} which is written in Python, \texttt{gatestrain}{} is written in C. It is developed as part of LALSuite and leverages LALSuite functions and methods; it can be found in the LALSuite fork \cite{suppMaterial} under the name of \texttt{lalapps\textunderscore gateStrain\textunderscore v1}. 
It takes $\sim 30-\SI{40}{\second}$ for an instance of \texttt{gatestrain}{}  to produce a gated SFT in the frequency range $10$ to $\SI{2000}{\hertz}$, lasting 1800s. The input data are gravitational-wave frame (gwf) files of the public data release. The outputs are gated SFTs or gated gravitational-wave frame (gwf) files and optionally ungated SFTs .

Although other gating methods exist \cite{Leaci:2010zz}, they are utilized within specific frameworks, the software is not publicly available and the input data is not the standard gravitational-wave frame (gwf) format. Our \texttt{lalapps\textunderscore gateStrain\textunderscore v1}{} works within the general LIGO Algorithm Library framework, and could be in fact be merged in the official LALSuite repository.

We present the results in the context of continuous gravitational-wave searches, however the method is valid for other searches, e.g. transient searches and stochastic background searches. We analyze the data around the eleven compact-binary coalescence gravitational-wave events of the GWTC-1 catalog. Not one was gated with out-of-the-box \texttt{gatestrain}-method. The glitch near GW170817 was automatically detected starting $\SI{1.09}{\second}$ before the event, $\SI{7.41}{\milli\second}$ off of LIGO's gate mid-time . Our \texttt{gatestrain}{} applied a $\SI{92.65}{\milli\second}$ gate with $\SI{0.25}{\second}$ taper to each side in comparison to LIGO's gate with $\SI{0.2}{\second}$ duration and $\SI{0.5}{\second}$ taper \cite{TheLIGOScientific:2017qsa, Vallisneri:2014vxa}. 
The loud GW150914 signal produces a peak in $h_w\left(t\right) \sim 1.5\sqrt{Hz}$ which is well below the lowest $H^{\textrm{high}}$ threshold of $\approx 6\sqrt{Hz}$. The weakest GW150914-like signal that would trigger our gating in O1 data is more than four times stronger than GW150914. This means that even though with the threshold settings described, the gating is very unlikely to remove a signal, as the detectors become more sensitive, and depending on the type of search carried out, the threshold levels need to be evaluated.
On the other hand, as the rate of detectable short-duration signals increases, an efficient method that automatically excises the disturbed portions of the data, becomes even more important.

Borne out of the desire to generalize the methodology that we had used on O1 data in continuous wave searches, and successfully applied to O2 data, our gating procedure successfully gates the O3 data, achieving a much smaller data loss than the LIGO gates with the same spectral noise improvement. Our procedure removes less than half the data compared to the ad-hoc \texttt{self-gating}{}{} procedure of \cite{RilesZweizig2021}.

Since the gated data remains a small fraction of the total data set, the impact of the more efficient gating on the detection statistic is small and the very loud hardware-injected fake signals present in the LIGO data for validation purposes, are recovered with comparable values of the detection statistic in both \texttt{gatestrain}{} data and \texttt{self-gated}{}{} data. The benefits of our method for gating is that it does not require ad-hoc time-consuming studies and careful tuning for every new data set and every new family of glitches that appears. 

Thanks to its adaptive algorithm, with practically no tuning, we were able to determine the O3a gates in less than a week. We make our tool available together with the O3a gates in the supplemental material \cite{suppMaterial}, for others to employ in their analyses of LIGO O3 data. \cite{suppMaterial} will be updated with the O3b gates, as soon as that data becomes public.

\acknowledgments
We thank Alex Nitz, Tito Dal Canton and Badri Krishnan for useful discussions on the \texttt{pyCBC\,} gating. We thank the anonymous referee for helpful comments and remarks on the manuscript. This work has utilised the ATLAS cluster computing at MPI for Gravitational Physics Hannnover.
This research has made use of data, software and/or web tools obtained from the Gravitational Wave Open Science Center (https://www.gw-openscience.org), a service of LIGO Laboratory, the LIGO Scientific Collaboration and the Virgo Collaboration. LIGO is funded by the U.S. National Science Foundation. Virgo is funded by the French Centre National de Recherche Scientifique (CNRS), the Italian Istituto Nazionale della Fisica Nucleare (INFN) and the Dutch Nikhef, with contributions by Polish and Hungarian institutes.




