<s>[INST] Find all primes $m$ and $l$ such that $2m-1, 2l-1, 2ml-1$ are perfect squares Find all primes $m$ and $l$ such that the integers $2m-1$, $2l-1$ and $2ml-1$ are perfect squares.
I have been trying to solve this problem from the 2022 Moroccan Maths Olympiad training program, but I wasn’t able to find anything.
By some few calculations, I found that (5,5) is the only couple between 1-100 satisfying the conditions, so I tried proving that it is the only solution using congruence but nothing worked. Help me please.
 [/INST] I don't know of any way to solve this using congruences. However, it can be done using the uniqueness, plus the format & number of representations, of sums of squares. First, the stated conditions mean there is a positive integer $a$ such that
$$\begin{equation}\begin{aligned}
2m - 1 & = (2a + 1)^2 \\
2m - 1 & = 4a^2 + 4a + 1 \\
m & = 2a^2 + 2a + 1 \\
m & = a^2 + (a^2 + 2a + 1) \\
m & = a^2 + (a + 1)^2
\end{aligned}\end{equation}\tag{1}\label{eq1A}$$
Similarly, there are positive integers $b$ and $c$ with
$$l = b^2 + (b + 1)^2 \tag{2}\label{eq2A}$$
$$lm = (a^2 + (a + 1)^2)(b^2 + (b + 1)^2) = c^2 + (c + 1)^2 \tag{3}\label{eq3A}$$
With \eqref{eq1A} and \eqref{eq2A}, note Fermat's Two Squares Theorem states a "prime number $p$ can be represented as a sum of two nonzero squares if and only if $p = 2$ or $p \equiv 1 \pmod{4}$; and that this representation is unique". Since neither $m$ or $l$ are $2$, this means $m \equiv l \equiv 1 \pmod{4}$.
Next, Wikipedia's "Sum of squares function" article's $k = 2$ Formulae section shows, if $l \neq m$, there are $4(1 + 1)(1 + 1) = 16$ ways to write $lm$ as a sum of $2$ squares (if $l = m$, then there are $4(2 + 1) = 12$ ways, with this reduction due to $a - b = 0$ not having distinct positive & negative values). Note the Brahmagupta–Fibonacci identity shows how to write the product of two sums of $2$ squares as a sum of two squares in two different ways. The order of the terms in each way can be switched to give a factor of $2$. Also, the values being squared in each term in each way can be made negative, giving another factor of $2 \times 2 = 4$. Thus, altogether this gives all of the $2 \times 2 \times 4 = 16$ ways to write $lm$ as a sum of $2$ squares. Note that switching the orders of the factors of $l$ or $m$, or using negatives for $a$, $a + 1$, $b$ and/or $b + 1$, will not add any additional ways. This is also indicated in Sum of two squares theorem which states

The product of any two representable numbers is another representable number. Its representation can be derived from representations of its two factors, using the Brahmagupta–Fibonacci identity.

Using the ordering in \eqref{eq3A} with the first Brahmagupta–Fibonacci identity option gives
$$\begin{equation}\begin{aligned}
(a^2 + (a + 1)^2)(b^2 + (b + 1)^2) & = ((ab) - (a + 1)(b + 1))^2 + (a(b + 1) + (a + 1)b)^2 \\
& = (a + b + 1)^2 + (a + b + 2ab)^2
\end{aligned}\end{equation}\tag{4}\label{eq4A}$$
This can match the RHS of \eqref{eq3A} only if $c = a + b + 1$ and $a + b + 2ab = c + 1$, which means $2ab = 2 \; \to \; a = b = 1$. This gives $l = m = 5$, which is the solution you found.
Using the second identity option gives
$$\begin{equation}\begin{aligned}
(a^2 + (a + 1)^2)(b^2 + (b + 1)^2) & = ((ab) + (a + 1)(b + 1))^2 + (a(b + 1) - (a + 1)b)^2 \\
& = (a + b + 2ab + 1)^2 + (a - b)^2
\end{aligned}\end{equation}\tag{5}\label{eq5A}$$
There are no positive integer values of $a$ and $b$ for which the terms being squared can be consecutive integers. As stated earlier, this comprises all of the ways to write the sum of squares.
Nonetheless, to help confirm there are no different results, continuing by switching the $a^2$ and $(a + 1)^2$ terms around, then switching $b^2$ and $(b + 1)^2$, and finally switching both at the same time, gives these results:
$$\begin{equation}\begin{aligned}
& (b - a)^2 + (a + b + 2ab + 1)^2 \\
& (a + b + 2ab)^2 + (a + b + 1)^2 \\
& (a - b)^2 + (a + b + 2ab + 1)^2 \\
& (a + b + 2ab)^2 + (- a - b - 1)^2 \\
& (a + b + 1)^2 + (a + b + 2ab)^2 \\
& (a + b + 2ab + 1)^2 + (b - a)^2
\end{aligned}\end{equation}\tag{6}\label{eq6A}$$
As expected, these just repeat the possibilities expressed earlier in \eqref{eq4A} and \eqref{eq5A}. Thus, the only primes which work are $l = m = 5$.
</s>