<s>[INST] Use coordinate method to solve a pretty hard geometry problem Here is a hard geometry problem for my homework.

Let $D$ be a point inside $\Delta ABC$ such that $\angle BAD=\angle BCD$ and $\angle BDC=90^\circ$. If $AB=5,BC=6$ and $M$ is the midpoint of $AC$, find the length of $DM$. 

(Taken from HK IMO Prelim 2012)
The teacher wants us to use the coordinate geometry to solve this problem. It does not appear too hard, but there is a big problem.

I set the coordinates of $A,B,C,D,M$ as in the diagram. Then, we can get the following statement:
$$\begin{cases} b^2+c^2=36 \\ x^2+\left(b+y\right)^2=25 \\ \sin\angle BAD=\sin\angle BCD \end{cases}$$
We need to find the value of $\sin\angle BAD$ and $\sin\angle BCD$
$$\sin\angle BCD=\dfrac{BD}{BC}=\dfrac{b}{6}\\ \text{The area of }\Delta ABD=\dfrac{1}{2}\left(AB\right)\left(AD\right)\sin\angle BAD \\ \dfrac{bx}{2}=\dfrac{5\sqrt{x^2+y^2}\sin\angle BAD}{2} \\ \sin\angle BAD=\dfrac{bx}{5\sqrt{x^2+y^2}}\\ \dfrac{b}{6}=\dfrac{bx}{5\sqrt{x^2+y^2}} \\ \sqrt{x^2+y^2}=\dfrac{6}{5}x \\ x^2+y^2=\dfrac{36}{25}x^2 \\ y^2=\dfrac{11}{25}x^2 \\ y=\dfrac{\sqrt{11}}{5}x$$ 
Then, when I proceed further, the expression gets very complicated. I felt puzzled, so I stopped. I found a geometric solution, which gives the answer of $\dfrac{\sqrt{11}}{2}$. 
I hope you guys can help me solve this question using coordinate method. Thank you!
 [/INST] After your calculations we have the system:
$$\begin{cases} b^2+c^2=36, \\ x^2+\left(b+y\right)^2=25, \\ \sqrt{11}x = 5y, \end{cases}\tag{1}$$
where $x>0,\;y>0$.
Substituting $x=\dfrac{5}{\sqrt{11}}y\;$ into $2$nd equation of $(1)$, we get:
$$
\dfrac{25}{11}y^2+(b+y)^2=25;
$$
$$
\dfrac{36}{11}y^2+2by+(b^2-25)=0;\tag{2}
$$
and positive $y$-solution of quadratic equation $(2)$ is:
$$
y = -\dfrac{11}{36}b + \dfrac{5\sqrt{11}}{36} \sqrt{36-b^2};
$$
when apply $1$st equation of $(1)$, we have:
$$
y = \dfrac{-11b + 5\sqrt{11} c}{36};\tag{3}
$$
therefore 
$$
x = \dfrac{-5\sqrt{11}b + 25c}{36}.\tag{3'}
$$
And
$$
DM = \dfrac{1}{2}\sqrt{(c-x)^2+y^2}=\\
\dfrac{\sqrt{(11c+5\sqrt{11}b)^2 + (-11b+5\sqrt{11}c)^2}}{2\cdot 36} = \\
\dfrac{\sqrt{(11^2+25\cdot 11)(b^2+c^2)}}{2\cdot 36} = \dfrac{\sqrt{36\cdot 11\cdot 36}}{2\cdot 36} = \dfrac{\sqrt{11}}{2}.
$$
</s>