<s>[INST] Second partial derivative with $f(x,y)=x^3+5x^2y+y^3$ I have this problem:

I found $F_x=3x^2+10yx$ and $F_y=5x^2+3y^2$, then $D_uf=\frac{9}{5}x^2+6yx+4x^2$
$F_{x2}= \frac{58}{5}x + 6y, F_{y2}=6x+{24}{5}y$
$D_uf_2=\frac{174}{25}x+\frac{18}{5}y+\frac{32}{5}x+\frac{96}{25}y=>at(2,1)=>534/25$ 
The answer however is $\frac{774}{25}$
 [/INST] Your mistake was at the first derivative, you forgot to add $\frac{12}5y^2$. See this full calculation:

$$f(x,y) = x^3+5x^2y+y^3$$
$$\nabla f = \left(3x^2+10xy \atop 5x^2+3y^2 \right) $$
$$\nabla f \cdot u = \left(3x^2+10xy \atop 5x^2+3y^2 \right)\cdot \left(\frac 3 5 \atop \frac 4 5\right) =\frac {29} 5x^2+6xy+ \frac{12}5y^2 $$
$$\nabla \left(\frac {29} 5x^2+6xy+ \frac{12}5y^2\right)\cdot u = \left( \frac{58}5x+6y\atop6x+\frac{24}5 y\right) \cdot \left(\frac 3 5 \atop \frac 4 5\right) $$
$$ = \frac{174}{25}x+\frac{18}{5}y+\frac{24}5x + \frac{96}{25}y$$
$$ = \frac{294}{25}x + \frac{186}{25}y
$$
Evaluating at $(2,1)$ gives:
$$\left(\frac{294}{25}x + \frac{186}{25}y\right)\left(2,1\right) =\frac{588}{25}+\frac{186}{25}=\frac{775}{25}$$
</s>