Let $A$ be a subset of $\{1,2,\ldots,2020\}$ such that the difference of any two distinct elements in $A$ is not prime. Determine the maximum number of elements in set $A$.

Let's think step by step and output the final answer within \boxed{}.
user