<s>[INST] Distinct roots of $ax^2-bx+c=0$ in $(0,1)$, where $a,b,c\in \mathbb{Z}^+$ 
Question Statement:-
Let $a,b,c$ be positive integers and consider all the quadratic equations of the form $ax^2-bx+c=0$ which have two distinct real roots in $(0,1)$. Find the least positive integers $a$ and $b$ for which such a quadratic equation exist.


Attempt at a solution:-
Let $f(x)=ax^2-bx+c$. Now as the roots of $f(x)=0$ lie in $(0,1)$ hence $f(0)\cdot f(1)\gt0$.
$$D\gt0\implies b^2-4ac\gt0$$
$$f(0)=c \\ f(1)=a-b+c$$
As, $a,b,c\in\mathbb{Z}^+$, so
$$f(0)\ge1 \implies c\ge1\tag{1}$$
$$f(1)\ge1\implies a-b+c\ge1\tag{2}$$
$$f(0)\cdot f(1)\ge1\implies c(a-b+c)\ge1\implies c^2+(a-b)c-1\ge0\tag{3}$$
Now, for the quadratic inequality in $c$ that we obtained in $(3)$, for it to hold we get
$$D\le0\implies (a-b)^2+4\le0$$
The above inequality doesn't hold $\forall a,b \in \mathbb{Z}^+$.
After arriving at this conclusion I am kind of stuck on what to conclude from the above result or rather am I going in the right direction.
A little push in the right direction in the form of a hint or rather what should be the line of thought for attempting the question. Please don't write the solution, just the line of thought to arrive at the solution would be enough.
 [/INST] The brute force approach works out fairly quickly in this case.


*

*$f(0) \gt 0$ $\implies$ $c \gt 0$ and $f(1) \gt 0$ $\implies$ $b \lt a + c$

*the product of the roots must be in $(0,1)$ $\implies$ $c < a$

*the discriminant $\Delta \gt 0$ $\implies$ $b^2 \gt 4 ac$
It follows that $c \ge 1$, $a \ge c+1$ and $b \in (\sqrt{4ac}, a+c)$. Trying the first few values starting at the lowest possible $c=1$ and $a=2$ gives:
$$
\begin{align}
& c = 1 & a = 2 \quad \implies \quad & b \in (\sqrt{8},3) & = & \;\emptyset \\
& & a = 3 \quad \implies \quad & b \in (\sqrt{12},4) & = & \;\emptyset \\
& & a = 4 \quad \implies \quad & b \in (\sqrt{16},5) & = & \;\emptyset \\
& & a = 5 \quad \implies \quad & b \in (\sqrt{20},6) & = & \;\{\;5\;\}
\end {align}
$$
The latter gives the solution $5 x^2 - 5x +1$ which can be easily verified to have both roots in $(0,1)$.

[ EDIT ]   To complete the proof that $a=5$ is indeed the smallest possible value, the cases $c=2,3$ must also be checked (we can stop at $3$ since $c \ge 4$ implies $a \ge 5$).

$$
\begin{align}
& c = 2 & a = 3 \quad \implies \quad & b \in (\sqrt{24},5) & = & \;\emptyset \\
& & a = 4 \quad \implies \quad & b \in (\sqrt{32},6) & = & \;\emptyset
\end {align}
$$
$$
\begin{align}
& c = 3 & a = 4 \quad \implies \quad & b \in (\sqrt{48},7) & = & \;\emptyset
\end {align}
$$
</s>