<s>[INST] problem with joint probability function The joint probability density function of $X$ and $Y$
is given by
$$f(x,y) = \begin{cases}c(y^2-x^2) e^{-y}, & -y \le x < +y, \ 0 \le y <  \infty,\\
0,  & \text{otherwise}. \end{cases}$$
(a) Find $c$.
(b) Find the marginal densities of $X$ and $Y$.
(c) Find $\operatorname{E}[X]$.
I have some problems with the point (b)
(a) $$
  \int_0^\infty\ \int_{-y}^y c(y^2-x^2)e^{-y}\,dy\,dx = 1 \Leftrightarrow c= \frac 1 8.
$$
(b) I calculate the marginal density of $Y$ as
$$ 
\int_{x=-y}^{x=+y} \frac 1 8 (y^2-x^2)e^{-y}\,dx = \frac 1 6 y^3 e^{-y},
$$
and the density of $X$ as
$$
\int_{y=0}^\infty\  c(y^2-x^2)e^{-y}\,dy, 
$$
but there something wrong because the solution is different.
Can someone help me to understand my mistake?
 [/INST] The marginal density of $Y$ is
$$\int_{-y}^y c(y^2-x^2) e^{-y} \mathop{dx}=ce^{-y}(2y^3 - (2/3)y^3 ) = \frac{4c}{3} y^3 e^{-y}.$$
Integrating this over $y$ gives
$$1=\int_0^\infty \frac{4c}{3} y^3 e^{-y} \mathop{dy} = \frac{4c}{3} \cdot 3! = 8c \implies c=1/8.$$
So, the marginal density of $Y$ is $\frac{1}{6} y^3 e^{-y}$ as you obtained.

The marginal density of $X$ is
$$\int_{|x|}^\infty c(y^2-x^2) e^{-y} \mathop{dy}
.$$
The tricky part is the limits of integration, which comes from the condition $|x| \le y$.
First, integration by parts twice gives
\begin{align}
\int_{|x|}^\infty y^2 e^{-y} \mathop{dy}
&=[-y^2 e^{-y}]_{y=|x|}^\infty + 2\int_{|x|}^\infty ye^{-y} \mathop{dy}\\
&= x^2 e^{-|x|} + 2[-ye^{-y}]_{y=|x|}^\infty + 2\int_{|x|}^\infty e^{-y}\mathop{dy}\\
&= x^2 e^{-|x|} + 2|x|e^{-|x|} + 2e^{-|x|}\\
&= e^{-|x|}(|x|^2+2|x|+2).
\end{align}
So, the marginal density of $X$ is
$$\int_{|x|}^\infty c(y^2-x^2) e^{-y} \mathop{dy}
= c(e^{-|x|}(|x|^2+2|x|+2) - x^2 e^{-|x|}) = \frac{1}{4} e^{-|x|}(|x|+1).$$

Edit (explanation for why the lower limit of integration is $|x|$):
In general, the marginal density of $X$ is
$$f_X(x) = \int_{-\infty}^\infty f(x,y) \mathop{dy}.$$
For a specific $x$, note that by definition $f(x,y)$ is zero if $|x|>y$; otherwise $f(x,y) = c(y^2-x^2)e^{-y}$. So
$$f_X(x) = \int_{-\infty}^\infty f(x,y) \mathop{dy} = \int_{-\infty}^{|x|} f(x,y) \mathop{dy} + \int_{|x|}^\infty f(x,y) \mathop{dy}
= 0 + \int_{|x|}^\infty c(y^2-x^2)e^{-y} \mathop{dy}.$$
</s>