<s>[INST] What is the maximum volume of an equilateral triangular prism inscribed in a sphere of radius 2? What is the maximum volume of an equilateral triangular prism inscribed in a sphere of radius 2?
Since the volume of an equilateral triangular prism is $\frac{\sqrt3}{4}a^2h$,where $a$ is the side length of the base triangle and $h$ is the height of the prism.How to express this volume in terms of radius of the sphere so that i can differentiate it and equate it to zero.Thanks. 
 [/INST] Let each side of the equilateral triangular face of the prism be $x$ inscribed in a sphere of radius $R=2$ the inscribed radius of the equilateral triangular base $$=\frac{x}{\sqrt 3}$$
Using geometry of right triangle, the normal distance of the center of equilateral triangular face of prism from the center of sphere $$=\sqrt{R^2-\left(\frac{x}{\sqrt 3}\right)^2}=\frac{\sqrt{3(2)^2-x^2}}{\sqrt 3}=\frac{\sqrt{12-x^2}}{\sqrt 3}$$
then the normal height $(H)$ of the prism is given as
$$H=2\frac{\sqrt{12-x^2}}{\sqrt 3}$$ 
The volume of the inscribed prism $$V=\text{(area of equilateral triangular face)}\times \text{(normal height of prism (H)}$$
$$V=\frac{\sqrt3}{4}x^2\times 2\frac{\sqrt{12-x^2}}{\sqrt 3}$$
$$V=\frac{1}{2}x^2\sqrt{12-x^2}\tag 1$$
Now, differentiating (1) w.r.t. $x$, we get 
$$\frac{dV}{dx}=\frac{1}{2}\left(x^2\frac{-2x}{2\sqrt{12-x^2}}+2x\sqrt{12-x^2}\right)=\frac{3}{2}\left(\frac{8x-x^3}{\sqrt{12-x^2}}\right)$$
Now, for maximum or minimum, we have $\frac{dV}{dx}=0$ 
$$\frac{3}{2}\left(\frac{8x-x^3}{\sqrt{12-x^2}}\right)=0\implies 8x-x^3=0$$
$$x(8-x^2)=0\implies x=0, \pm 2\sqrt 2$$
But, side, $x>0$ hence, $x=2\sqrt 2$ is acceptable 
Now, it can be checked that $\frac{d^2V}{dx^2}<0$ at $x=2\sqrt 2$ i.e. the volume of prism is maximum at $x=2\sqrt 2$  
hence setting $x=2\sqrt 2$ in (1), we get the maximum volume of the inscribed equilateral triangular prism in the sphere 
$$V_{max}=\frac{1}{2}(2\sqrt 2)^2\sqrt{12-(2\sqrt 2)^2}=\color{}{8}$$
$$\bbox[5pt, border:2.5pt solid #FF0000]{\color{blue}{\text{maximum volume of equilateral triangular prism}=8\ \text{unit}^3}}$$
</s>