<s>[INST] Mysterious sum equal to $\frac{7(p^2-1)}{24}$ where $p \equiv 1 \pmod{4}$ Consider a prime number $p \equiv 1 \pmod{4}$ and $n_p$ denotes the remainder of $n$ upon division by $p$. Let $A_p=\{ a \in [[0,p]] \mid {(a+1)^2}_p<{a^2}_p\}$.
I Conjecture
$$\sum_{a \in A_p } a=\dfrac{7(p^2-1)}{24}$$
**Addition, a simpler way to prove equality after have answer **.
We denote the sequence $s_p=\sum_{a=1}^{p-1} a({(a+1)^2}_p-{a^2}_p)$.
By expanding, we verify that $s_p=-\sum_{a=1}^{p-1} {a^2}_p$.
We can also verify that if  $a\in A_p$, then
${(a+1)^2}_p-{a^2}_p  =({2a+1})_p-p$
and if  $a\notin A_p$, then
${(a+1)^2}_p-{a^2}_p =({2a+1})_p.$
We deduce $p\sum_{p \in A_p } a =\sum_{a=1}^{p-1} a ({2a+1})_p-s_p=\sum_{a=1}^{p-1} a ({2a+1})_p + \sum_{a=1}^{p-1} {a^2}_p$.
to finish, it is easy to calculate $\sum_{a=1}^{p-1} a ({2a+1})_p\quad  \forall p\in\mathbb N$ and $\sum_{a=1}^{p-1} {a^2}_p\quad \forall p\equiv 1\mod 4$
 [/INST] I will leave several details for you to check, ask me if there are any issues with verifying them.
Let $$A_p=\{ a \in [[0,p-1]] \mid {(a+1)^2}_p<{a^2}_p\},$$
$$B_p=\{ a \in [[0,p-1]] \mid {(a+1)^2}_p>{a^2}_p\},$$
$$C_p=\{ a \in [[0,p-1]] \mid {(a+1)^2}_p={a^2}_p\}.$$
Then $A_p \cup B_p \cup C_p = [[0,p-1]]$. Check that $C_p$ is the singleton set $\{(p-1)/2 \}$.
Now, note that $a \in A_p \iff (p-1-a) \in B_p$ (Verify!).
Let $A_p' = A_p \cap [[0,(p-1)/2]]$, $B_p' = B_p \cap [[0,(p-1)/2]]$.
Then $A_p = A'_p \cup A''_p$ where $A_p'' = \{ (p-1)-b | b\in B_p' \}$.
Final thing to note is that $A_p' = \{ \lfloor \sqrt{jp} \rfloor | 1 \leq j \leq (p-5)/4 \}$, so $|A_p'| = (p-5)/4$ and $|B_p'| = (p+3)/4$.
Now, we just do a computation:
\begin{align}
\sum_{a \in A_p'} a + \sum_{a \in A_p''} a &= \sum_{a \in A_p'} a + \sum_{a \in B_p'} [(p-1) - a] \\ &= 2 \sum_{a \in A_p'} a + \sum_{a \in B_p'} (p-1) - \sum_{a=0}^{(p-3)/2} a \\ &= 2 \sum_{j=0}^{(p-5)/4} \lfloor \sqrt{jp} \rfloor + \frac{(p-1)(p+3)}4 - \frac{(p-3)(p-1)}8
\end{align}
Now, consider the subset of $\mathbb Z \times \mathbb Z$
$$\eqalign{
  &\left\{(t,j)\ \bigg|\ 1 \leq j \leq \frac{p-5}{4}\ \hbox{and}\ 
                    1 \leq t \leq \sqrt{jp} \right\}\cr
  &\qquad\qquad= \left\{(t,j)\ \bigg|\ 1 \leq t \leq \frac{p-3}{2}\ \hbox{and}\ 
                     \frac{t^2}{p} \leq j \leq \frac{p-5}{4} \right\}\cr}
$$
Comparing both cardinalities we get (q.r. = quadratic residues)
\begin{align}
\sum_{j=0}^{(p-5)/4} \lfloor \sqrt{jp} \rfloor &= \sum_{t=1}^{(p-3)/2} \left \{ \frac {(p-5)} 4 - \left \lfloor \frac{t^2} p  \right \rfloor \right \} \\ &= \frac{(p-3)(p-5)}{8} - \sum_{t=1}^{(p-3)/2} \frac{t^2} p + \sum_{t=1}^{(p-3)/2} \frac{t^2 \pmod p} p \\ &= \frac{(p-3)(p-5)}{8} - \frac{(p-1)(p-2)(p-3)}{24p} \\ &\ \ \ + \frac{\text{Sum of q.r. mod p}- ((p-1)/2)^2 \pmod p}p
\end{align}
Now, if $p \equiv 1 \pmod 4$, then $((p-1)/2)^2 \equiv (3p+1)/4 \pmod p$ and sum of quadratic residues is $p(p-1)/4$.
Thus,
\begin{align}
\sum_{j=0}^{(p-5)/4} \lfloor \sqrt{jp} \rfloor &= \frac{(p-3)(p-5)}{8} - \frac{(p-1)(p-2)(p-3)}{24p} + \frac{p(p-1)- (3p+1)}{4p} \\ &= \frac{(p-1)(p-5)}{12}.
\end{align}
Now, the final sum is
\begin{align}
\sum_{a \in A_p} a &= \frac{(p-1)(p-5)}{6} + \frac{(p-1)(p+3)}4 - \frac{(p-3)(p-1)}8 \\ &= \frac{7(p^2-1)}{24}.
\end{align}
Here are two problems employing similar techniques. (That is where I learnt this method from)
</s>