Charlie has a fair $n$-sided die (with each face showing a positive integer between $1$ and $n$ inclusive) and a list of $n$ consecutive positive integers. He first rolls the die, and if the number showing on top is $k$, he then uniformly and randomly selects a $k$-element subset from his list and calculates the sum of the numbers in his subset. Given that the expected value of this sum is $2020$, compute the sum of all possible values of $n$.

Let's think step by step and output the final answer within \boxed{}.
user