<s>[INST] Expected value of rolling 3 dice if we take square number 
Questions:
$(1)$ Suppose we play a game. I roll a die up to three times. Each time I roll, you can either take the square of the number showing as dollars, or roll again. How much should you pay to play this game?
$(2)$ What effect will it have if we are charged for $\$1$ for each additional roll?

Question $(1)$ above is very common if we replace 'square of the number' with just 'number. For example, this post in MSE.
For $(1),$ assuming that we have $1$ die. Then the expected value is
$$\sum_{k=1}^6 \frac{k^2}{6} = \frac{91}{6}.$$
Now, assume that we have $2$ dice. Then
$$\frac{91}{6} \times \frac{1}{6} + \frac{91}{6} \times \frac{1}{6} + \frac{91}{6} \times \frac{1}{6} + \frac{4^2}{6} + \frac{5^2}{6} + \frac{6^2}{6} = \frac{245}{12}.$$
Similarly, if we have $3$ dice, then
$$\frac{245}{12}\times\frac{1}{6} + \frac{245}{12}\times\frac{1}{6} + \frac{245}{12}\times\frac{1}{6} + \frac{245}{12}\times\frac{1}{6} + \frac{5^2}{6} + \frac{6^2}{6} = \frac{428}{18}.$$
Are my calculations above correct?
For $(2),$ I think we should pay less. But I do not how much lesser is it.
 [/INST] I agree with your calculations for the first part.
For the second part, your calculation for one roll is still correct.  But for the second roll, your expected value for rerolling would be $\frac{91}6-1=\frac{85}6$
$$\frac{85}{6} \times \frac{1}{6} + \frac{85}{6} \times \frac{1}{6} + \frac{85}{6} \times \frac{1}{6} + \frac{4^2}{6} + \frac{5^2}{6} + \frac{6^2}{6} = \frac{239}{12}$$ and similarly for the third part except with $\frac{227}{12}$ instead of $\frac{245}{12}$.  None of these subtractions change the decisions about when you would hold and when you would re-roll with just a \$1 charge, but you would want to stay aware of that in general.
</s>