\section*{ S.1 Values of Parameters of Low-Energy Effective Model}

In this supplemental material, we show the values of the parameters of the {\it ab initio} low-energy effective model. 
Table S~\ref{W_FeSe} (\ref{W_FeTe}) is the three-dimensional effective interaction of the maximally localized Wannier functions ({MLWFs}) originating from the Fe $3d$ orbitals of FeSe (FeTe)~\cite{S_hirayama13}.
We obtain the two-dimensional effective interaction by uniformly subtracting 0.6 (0.4) eV from the three-dimensional effective interaction of FeSe (FeTe)
following Refs.~\citen{S_nakamura10} and \citen{S_nakamurap}.
Table S~\ref{t_LDA} is the transfer integral of the {MLWFs} of FeSe and FeTe {obtained by} the local-density-approximation (LDA) {calculation}.
We also show the transfer integral of the {MLWFs} in the constrained GW with the self-interaction correction (cGW-SIC) model in Table S~\ref{t_cGW-SIC}.
Figure S~\ref{FeSe_band} is the corresponding band structure of FeSe.
The on-site potential of each model is shown in  Fig. S~\ref{ton}.
The detail of the cGW method is explained in Refs.~\citen{S_hirayama13} and \citen{S_aryasetiawan09}. 


\begin{figure}[tb]
\centering  
\includegraphics[width=0.45\textwidth ]{17078Fig1SM.eps}  
\caption{
(Color online) Electronic band structures of the Fe $3d$ {MLWFs} of FeSe in the LDA [(red) solid line]
 and the cGW-SIC [(green) dashed line].
The Fermi energy is set to zero.
}
\label{FeSe_band}
\end{figure} 

\begin{figure}[ptb]
\begin{center} 
\includegraphics[width=0.45\textwidth ]{17078Fig2SM.eps} 
\end{center} 
\caption{(Color online) On-site potential of Wannier orbitals for FeSe and FeTe {obtained by} the LDA and cGW-SIC {calculations}.
}
\label{ton}
\end{figure} 

\begin{figure}[tb]
\centering 
\includegraphics[clip,width=0.4\textwidth ]{17078Fig3SM.eps} 
\caption{
(Color online) Ground state energy per site of AF in the mVMC for FeSe at ambient pressure and at 4.0 GPa with $t^{\text{cGW-SIC}}$ (in meV).
System size is $N_{\text{s}}=8\times 8$.
}
\label{AF_FeSe_energy_4GPa}
\end{figure} 

\section*{ S.2 Magnetism of FeSe under Pressure}

We calculate the magnetism of FeSe under pressure as well as that at ambient pressure. 
Figure S~\ref{AF_FeSe_energy_4GPa} shows the energy per site {by} the cGW-SIC {calculation for} FeSe at 4.0 GPa in the experimental geometry~\cite{S_margadonna09}.
{Although the AFS-type short-ranged fluctuations were observed~\cite{S_rahn15,S_wangX} at ambient pressure, the} unique energetic degeneracy at ambient pressure is lifted under pressure, and we predict that the AFH phase becomes {slightly more} stable {than the AFS}.
The experimental verification would be desired.



\begin{table*}[htb] 
\caption{
Bare and effective three-dimensional Coulomb interactions between two electrons for all the combinations of Fe $3d$ orbitals in FeSe (in eV).
Here, $v$ and $J_{v}$ represent the bare on-site and exchange Coulomb interactions, respectively. 
The static limit
of the effective on-site and exchange Coulomb interactions are denoted by $U$ and $J$, respectively. 
}
\ 
\label{W_FeSe} 
\scalebox{0.85}[0.85]{
\begin{tabular}{c|ccccc|ccccc} 
\hline \hline \\ [-8pt]
FeSe           &      &      &     $v$     &      &               &      &      &      $U$    &      &          \\ [+1pt]
\hline \\ [-8pt]
               & $XY$ & $YZ$ & $3Z^{2}-R^{2}$ & $ZX$ & $X^{2}-Y^{2}$ & $XY$ & $YZ$ & $3Z^{2}-R^{2}$ & $ZX$ & $X^{2}-Y^{2}$ \\ 
\hline \\ [-8pt] 
$XY$           & 18.65 & 16.50 & 17.28 & 16.50 & 16.51  & 4.51 & 3.19 & 3.21 & 3.19 & 3.48 \\
$YZ$           & 16.50 & 16.97 & 17.05 & 15.70 & 15.30  & 3.19 & 4.11 & 3.53 & 3.02 & 2.98 \\
$3Z^{2}-R^{2}$ & 17.28 & 17.05 & 19.09 & 17.05 & 16.00  & 3.21 & 3.53 & 4.68 & 3.53 & 3.01 \\
$ZX$           & 16.50 & 15.70 & 17.05 & 16.97 & 15.30  & 3.19 & 3.02 & 3.53 & 4.11 & 2.98 \\
$X^{2}-Y^{2}$  & 16.51 & 15.30 & 16.00 & 15.30 & 15.88  & 3.48 & 2.98 & 3.01 & 2.98 & 3.77 \\
\hline \hline \\ [-8pt]
               &      &      &     $J_{v}$     &      &               &      &      &      $J$    &      &          \\ [+1pt]
\hline \\ [-8pt]
               & $XY$ & $YZ$ & $3Z^{2}-R^{2}$ & $ZX$ & $X^{2}-Y^{2}$ & $XY$ & $YZ$ & $3Z^{2}-R^{2}$ & $ZX$ & $X^{2}-Y^{2}$ \\ 
\hline \\ [-8pt] 
$XY$           &      & 0.66 & 0.79 & 0.66 & 0.34 &      & 0.57 & 0.69 & 0.57 & 0.32  \\
$YZ$           & 0.66 &      & 0.46 & 0.56 & 0.61 & 0.57 &      & 0.42 & 0.48 & 0.53  \\
$3Z^{2}-R^{2}$ & 0.79 & 0.46 &      & 0.46 & 0.75 & 0.69 & 0.42 &      & 0.42 & 0.62  \\
$ZX$           & 0.58 & 0.56 & 0.46 &      & 0.61 & 0.57 & 0.48 & 0.42 &      & 0.53  \\
$X^{2}-Y^{2}$  & 0.34 & 0.61 & 0.75 & 0.61 &      & 0.32 & 0.53 & 0.62 & 0.53 &       \\
\hline
\hline 
\end{tabular}
}
\end{table*}



\begin{table*}[htb] 
\caption{
Bare and effective three-dimensional Coulomb interactions between two electrons for all the combinations of Fe $3d$ orbitals in FeTe (in eV).
Here, $v$ and $J_{v}$ represent the bare on-site and exchange Coulomb interactions, respectively. 
The static limit
of the effective on-site and exchange Coulomb interactions are denoted by $U$ and $J$, respectively. 
}
\ 
\label{W_FeTe} 
\scalebox{0.85}[0.85]{
\begin{tabular}{c|ccccc|ccccc} 
\hline \hline \\ [-8pt]
FeTe           &      &      &     $v$     &      &               &      &      &      $U$    &      &          \\ [+1pt]
\hline \\ [-8pt]
               & $XY$ & $YZ$ & $3Z^{2}-R^{2}$ & $ZX$ & $X^{2}-Y^{2}$ & $XY$ & $YZ$ & $3Z^{2}-R^{2}$ & $ZX$ & $X^{2}-Y^{2}$ \\ 
\hline \\ [-8pt] 
$XY$           & 17.08 & 15.05 & 16.20 & 15.05 & 16.26  & 3.46 & 2.30 & 2.36 & 2.30 & 2.80 \\
$YZ$           & 15.05 & 15.36 & 15.87 & 14.24 & 14.94  & 2.30 & 3.08 & 2.62 & 2.15 & 2.30 \\
$3Z^{2}-R^{2}$ & 16.20 & 15.87 & 18.25 & 15.87 & 16.08  & 2.36 & 2.62 & 3.73 & 2.62 & 2.35 \\
$ZX$           & 15.05 & 14.24 & 15.87 & 15.36 & 14.94  & 2.30 & 2.15 & 2.62 & 3.08 & 2.30 \\
$X^{2}-Y^{2}$  & 16.32 & 14.94 & 16.08 & 14.94 & 16.77  & 2.82 & 2.30 & 2.36 & 2.30 & 3.39\\
\hline \hline \\ [-8pt]
               &      &      &     $J_{v}$     &      &               &      &      &      $J$    &      &          \\ [+1pt]
\hline \\ [-8pt]
               & $XY$ & $YZ$ & $3Z^{2}-R^{2}$ & $ZX$ & $X^{2}-Y^{2}$ & $XY$ & $YZ$ & $3Z^{2}-R^{2}$ & $ZX$ & $X^{2}-Y^{2}$ \\ 
\hline \\ [-8pt] 
$XY$           &      & 0.59 & 0.73 & 0.59 & 0.33 &      & 0.49 & 0.62 & 0.49 & 0.31 \\
$YZ$           & 0.59 &      & 0.42 & 0.49 & 0.59 & 0.49 &      & 0.37 & 0.40 & 0.49 \\
$3Z^{2}-R^{2}$ & 0.73 & 0.42 &      & 0.42 & 0.74 & 0.62 & 0.37 &      & 0.37 & 0.62 \\
$ZX$           & 0.59 & 0.49 & 0.42 &      & 0.59 & 0.49 & 0.40 & 0.37 &      & 0.49 \\
$X^{2}-Y^{2}$  & 0.33 & 0.59 & 0.74 & 0.59 &      & 0.31 & 0.49 & 0.62 & 0.49 &      \\
\hline
\hline 
\end{tabular}
}
\end{table*}






\begin{table*}[ptb] 
\caption{
Transfer integral for the $3d$ orbitals of the Fe sites in the FeSe and FeTe, $t^{\text{LDA}}_{mn}(R_x, R_y, R_z)$, 
where $t^{\text{LDA}}$ is the expectation value of the Kohn-Sham Hamiltonian for the Wannier function : $t^{\text{LDA}}=\langle \phi | \mathcal{H}^{\text{LDA}} |\phi \rangle$, $m$ and $n$ denote symmetries of the $3d$ orbitals, and the axis of $(R_x, R_y, R_z)$ is taken along the Fe-Se/Te directions.
Units are given in meV.  
} 
\
\scalebox{0.75}[0.75]{ 
\begin{tabular}{c|rrrrrrr|rrr} 
\hline \hline \\ [-4pt]
  FeSe    \\ [+2pt] 
\hline \\ [-4pt]
$(m, n)$ $\backslash$ $\bm{R}$ 
& \big[$0,0,0$\big] 
& \big[$\frac{1}{2},-\frac{1}{2},0$\big] 
& \big[$1,0,0$\big] 
& \big[$1,-1,0$\big] 
& \big[$\frac{3}{2},-\frac{1}{2},0$\big]
& \big[$0,0,\frac{c}{a}$\big] 
& \big[$\frac{1}{2},-\frac{1}{2},\frac{c}{a}$\big]
& $\sigma_{Y}$
& $I$
& $\sigma^{L}$ \\ [+4pt]
\hline \\ [-8pt]            
$(XY,XY)$                    & -509 &   -410 &    -70 &    -11 &      3 &    -25 &      6 & $+$ & $+$ &      $+$ \\
$(XY,YZ)$                    &    0 &    273 &    131 &     -9 &     -6 &      0 &     -9 & $+$ & $-$ & $-$(1,4) \\
$(XY,3Z^{2}-R^{2})$          &    0 &   -347 &      0 &     22 &     -8 &      0 &     11 & $-$ & $+$ &      $+$ \\ 
$(XY,ZX)$                    &    0 &    273 &      0 &     -9 &     18 &      0 &     -3 & $-$ & $-$ & $-$(1,2) \\
$(XY,X^{2}-Y^{2})$           &    0 &      0 &      0 &      0 &     -9 &      0 &     -4 & $-$ & $+$ &      $-$ \\ 
$(YZ,YZ)$                    &   46 &    197 &    128 &    -17 &     -8 &      8 &     27 & $+$ & $+$ &    (4,4) \\
$(YZ,3Z^{2}-R^{2})$          &    0 &   -119 &      0 &      7 &      2 &      0 &     11 & $-$ & $-$ & $-$(4,3) \\ 
$(YZ,ZX)$                    &    0 &    127 &      0 &    -23 &    -19 &      0 &     12 & $-$ & $+$ &    (4,2) \\ 
$(YZ,X^{2}-Y^{2})$           &    0 &    223 &      0 &      1 &     -3 &      0 &     20 & $-$ & $-$ &    (4,5) \\ 
$(3Z^{2}-R^{2},3Z^{2}-R^{2})$& -388 &     -4 &    -15 &    -14 &     -6 &    -23 &     -9 & $+$ & $+$ &      $+$ \\
$(3Z^{2}-R^{2},ZX)$          &    0 &    119 &    199 &     -7 &    -13 &      0 &    -10 & $+$ & $-$ & $-$(3,2) \\ 
$(3Z^{2}-R^{2},X^{2}-Y^{2})$ &    0 &      0 &   -115 &      0 &      1 &     -8 &     -6 & $+$ & $+$ &      $-$ \\ 
$(ZX,ZX)$                    &   46 &    197 &    335 &    -17 &     13 &      8 &      0 & $+$ & $+$ &    (2,2) \\
$(ZX,X^{2}-Y^{2})$           &    0 &   -223 &     82 &     -1 &    -15 &      0 &      7 & $+$ & $-$ &    (2,5) \\ 
$(X^{2}-Y^{2},X^{2}-Y^{2})$  &  -34 &    -56 &     93 &      0 &     17 &    -28 &      4 & $+$ & $+$ &      $+$ \\
\hline \hline 
  FeTe    \\ [+2pt] 
\hline \\ [-4pt]
$(m, n)$ $\backslash$ $\bm{R}$ 
& \big[$0,0,0$\big] 
& \big[$\frac{1}{2},-\frac{1}{2},0$\big] 
& \big[$1,0,0$\big] 
& \big[$1,-1,0$\big] 
& \big[$\frac{3}{2},-\frac{1}{2},0$\big]
& \big[$0,0,\frac{c}{a}$\big] 
& \big[$\frac{1}{2},-\frac{1}{2},\frac{c}{a}$\big]
& $\sigma_{Y}$
& $I$
& $\sigma^{L}$ \\ [+4pt] 
\hline \\ [-8pt]            
$(XY,XY)$                    &  -452 &   -378 &    -11 &    -41 &     -1 &    -31 &     12 & $+$ & $+$ &      $+$ \\
$(XY,YZ)$                    &       0 &    237 &    109 &      3 &     -6 &      0 &    -12 & $+$ & $-$ & $-$(1,4) \\
$(XY,3Z^{2}-R^{2})$          &      0 &   -336 &      0 &     33 &     -9 &      0 &     21 & $-$ & $+$ &      $+$ \\ 
$(XY,ZX)$                    &     0 &    237 &      0 &      3 &     35 &      0 &     -1 & $-$ & $-$ & $-$(1,2) \\
$(XY,X^{2}-Y^{2})$           &       0 &      0 &      0 &      0 &    -14 &      0 &      3 & $-$ & $+$ &      $-$ \\ 
$(YZ,YZ)$                    &   44 &    156 &    103 &    -15 &    -12 &     13 &     37  & $+$ & $+$ &    (4,4) \\
$(YZ,3Z^{2}-R^{2})$          &     0 &   -122 &      0 &     17 &      6 &      0 &     11 & $-$ & $-$ & $-$(4,3) \\ 
$(YZ,ZX)$                    &     0 &    101 &      0 &    -27 &    -25 &      0 &     14  & $-$ & $+$ &    (4,2) \\ 
$(YZ,X^{2}-Y^{2})$           &      0 &    178 &      0 &      0 &    -10 &      0 &     22 & $-$ & $-$ &    (4,5) \\ 
$(3Z^{2}-R^{2},3Z^{2}-R^{2})$& -480 &    -73 &    -53 &      3 &      8 &    -67 &    -23 & $+$ & $+$ &      $+$ \\
$(3Z^{2}-R^{2},ZX)$          &    0 &    122 &    198 &    -17 &    -17 &      0 &    -32 & $+$ & $-$ & $-$(3,2) \\ 
$(3Z^{2}-R^{2},X^{2}-Y^{2})$ &      0 &      0 &    -29 &      0 &     -6 &     30 &    -30  & $+$ & $+$ &      $-$ \\ 
$(ZX,ZX)$                    &    44 &    156 &    300 &    -15 &     42 &     13 &      9  & $+$ & $+$ &    (2,2) \\
$(ZX,X^{2}-Y^{2})$           &     0 &   -178 &    136 &      0 &    -23 &      0 &     26  & $+$ & $-$ &    (2,5) \\ 
$(X^{2}-Y^{2},X^{2}-Y^{2})$  & -211 &     66 &     52 &      9 &     12 &     16 &    -24 & $+$ & $+$ &      $+$ \\
\hline \hline 
\end{tabular}
}
\label{t_LDA} 
\end{table*}


\begin{table*}[ptb] 
\caption{Transfer integral for the $3d$ orbitals of the Fe sites in the FeSe and FeTe, $t^{\text{cGW-SIC}}_{mn}(R_x, R_y, R_z)$, 
where $t^{\text{cGW-SIC}}$ is the transfer integral without double-counting :
$t^{\text{cGW-SIC}}=\langle \phi | \mathcal{H}^{\text{cGW-SIC}} |\phi \rangle$,
$m$ and $n$ denote symmetries of the $3d$ orbitals, and the axis of $(R_x, R_y, R_z)$ is taken along the Fe-Se/Te directions.
Units are given in meV.
} 
\
\scalebox{0.75}[0.75]{ 
\begin{tabular}{c|rrrrrrr|rrr} 
\hline \hline \\ [-4pt]
  FeSe    \\ [+2pt] 
\hline \\ [-4pt]
$(m, n)$ $\backslash$ $\bm{R}$ 
& \big[$0,0,0$\big] 
& \big[$\frac{1}{2},-\frac{1}{2},0$\big] 
& \big[$1,0,0$\big] 
& \big[$1,-1,0$\big] 
& \big[$\frac{3}{2},-\frac{1}{2},0$\big]
& \big[$0,0,\frac{c}{a}$\big] 
& \big[$\frac{1}{2},-\frac{1}{2},\frac{c}{a}$\big]
& $\sigma_{Y}$
& $I$
& $\sigma^{L}$ \\ [+4pt]
\hline \\ [-8pt]            
$(XY,XY)$                    & -751 &   -466 &    -20 &     24 &      3 &    -38 &      6 & $+$ & $+$ &      $+$ \\
$(XY,YZ)$                    &    0 &    201 &    106 &    -26 &     -3 &      0 &     -9 & $+$ & $-$ & $-$(1,4) \\
$(XY,3Z^{2}-R^{2})$          &     0 &   -391 &      0 &     27 &     -3 &      0 &      8 & $-$ & $+$ &      $+$ \\ 
$(XY,ZX)$                    &   0 &    201 &      0 &    -26 &     -9 &      0 &     -4 & $-$ & $-$ & $-$(1,2) \\
$(XY,X^{2}-Y^{2})$           &    0 &      0 &      0 &      0 &      6 &      0 &    -10 & $-$ & $+$ &      $-$ \\ 
$(YZ,YZ)$                    &  -137 &     97 &    148 &    -35 &    -15 &      7 &     27 & $+$ & $+$ &    (4,4) \\
$(YZ,3Z^{2}-R^{2})$          &    0 &    -94 &      0 &     22 &     -1 &      0 &      8 & $-$ & $-$ & $-$(4,3) \\ 
$(YZ,ZX)$                    &    0 &    199 &      0 &    -37 &    -36 &      0 &     16 & $-$ & $+$ &    (4,2) \\ 
$(YZ,X^{2}-Y^{2})$           &     0 &    204 &      0 &     -1 &      7 &      0 &     17 & $-$ & $-$ &    (4,5) \\ 
$(3Z^{2}-R^{2},3Z^{2}-R^{2})$&  -1075 &    -67 &    -76 &      5 &     11 &     -4 &      3 & $+$ & $+$ &      $+$ \\
$(3Z^{2}-R^{2},ZX)$          &     0 &     94 &    195 &    -22 &    -16 &      0 &     -3  & $+$ & $-$ & $-$(3,2) \\ 
$(3Z^{2}-R^{2},X^{2}-Y^{2})$ &     0 &      0 &    -35 &      0 &    -13 &    -22 &      7 & $+$ & $+$ &      $-$ \\ 
$(ZX,ZX)$                    &    -137 &     97 &    263 &    -35 &     25 &      7 &     -2 & $+$ & $+$ &    (2,2) \\
$(ZX,X^{2}-Y^{2})$           &     0 &   -204 &    147 &      1 &    -39 &      0 &      0  & $+$ & $-$ &    (2,5) \\ 
$(X^{2}-Y^{2},X^{2}-Y^{2})$  &   -299 &    186 &     19 &    -20 &      8 &    -35 &     17 & $+$ & $+$ &      $+$ \\
\hline \hline 
  FeTe    \\ [+2pt] 
\hline \\ [-4pt]
$(m, n)$ $\backslash$ $\bm{R}$ 
& \big[$0,0,0$\big] 
& \big[$\frac{1}{2},-\frac{1}{2},0$\big] 
& \big[$1,0,0$\big] 
& \big[$1,-1,0$\big] 
& \big[$\frac{3}{2},-\frac{1}{2},0$\big]
& \big[$0,0,\frac{c}{a}$\big] 
& \big[$\frac{1}{2},-\frac{1}{2},\frac{c}{a}$\big]
& $\sigma_{Y}$
& $I$
& $\sigma^{L}$ \\ [+4pt] 
\hline \\ [-8pt]            
$(XY,XY)$                    &  -678 &   -410 &     52 &      5 &     -2 &     23 &    -13 & $+$ & $+$ &      $+$ \\
$(XY,YZ)$                    &     0 &    133 &     79 &    -11 &     -1 &      0 &     11 & $+$ & $-$ & $-$(1,4) \\
$(XY,3Z^{2}-R^{2})$          &     0 &   -340 &      0 &     26 &     -8 &      0 &     11 & $-$ & $+$ &      $+$ \\ 
$(XY,ZX)$                    &    0 &    133 &      0 &    -11 &     -2 &    -26 &     10  & $-$ & $-$ & $-$(1,2) \\
$(XY,X^{2}-Y^{2})$           &     0 &      0 &      0 &      0 &     -1 &      0 &     16 & $-$ & $+$ &      $-$ \\ 
$(YZ,YZ)$                    &   -146 &     42 &    120 &    -28 &    -17 &     11 &     15  & $+$ & $+$ &    (4,4) \\
$(YZ,3Z^{2}-R^{2})$          &     0 &    -88 &      0 &     24 &      6 &    -23 &     14 & $-$ & $-$ & $-$(4,3) \\ 
$(YZ,ZX)$                    &     0 &    186 &      0 &    -35 &    -46 &      0 &     16  & $-$ & $+$ &    (4,2) \\ 
$(YZ,X^{2}-Y^{2})$           &    0 &    139 &      0 &      1 &     -2 &     12 &     14 & $-$ & $-$ &    (4,5) \\ 
$(3Z^{2}-R^{2},3Z^{2}-R^{2})$& -974 &   -114 &   -114 &     21 &     26 &     -1 &     -1  & $+$ & $+$ &      $+$ \\
$(3Z^{2}-R^{2},ZX)$          &    0 &     88 &    181 &    -24 &    -12 &      0 &     -4 & $+$ & $-$ & $-$(3,2) \\ 
$(3Z^{2}-R^{2},X^{2}-Y^{2})$ &    0 &      0 &     49 &      0 &    -24 &     -1 &      2 & $+$ & $+$ &      $-$ \\ 
$(ZX,ZX)$                    &  -146 &     42 &    213 &    -28 &     51 &     22 &     13 & $+$ & $+$ &    (2,2) \\
$(ZX,X^{2}-Y^{2})$           &    0 &   -139 &    147 &     -1 &    -30 &      0 &      8 & $+$ & $-$ &    (2,5) \\ 
$(X^{2}-Y^{2},X^{2}-Y^{2})$  &   -495 &    292 &    -47 &     -9 &     20 &      5 &    -13 & $+$ & $+$ &      $+$ \\
\hline \hline 
\end{tabular}
}
\label{t_cGW-SIC} 
\end{table*}






