<s>[INST] Stewart's Calculus (6 edn 2007) : p. 1097. §16.8, Exercise #4. Source: Stewart. Calculus: Early Transcendentals (6 edn 2007). p. 1097. §16.8, Exercise #4.

Use Stokes' Theorem to evaluate $\int \int_S curl \ \vec F \cdot d \vec S $
$4. \; F(x,y,z) = x^2 \ y^3\ z \ \vec i + \sin(x\ y\ z)\  \vec j + x\ y\ z\  \vec k$,
$S$ is the part of the cone $y^2 =x^2 + z^2$ that lies between the planes $y = 0$ and $y = 3$, oriented in the direction of the positive $y$-axis.

I calculated $ curl \ \vec F $ :
$$ curl \ \vec F = ( xz-xy \cos(xyz) \ , \ -yz+x^2y^3 \ , \ yz \cos(xyz)-3x^2y^2z ) $$
Now I have to evaluate the domain $S$, and I have no idea how. I have a lot of difficulties to find a domain, and when I should use $\int \int_S curl \ \vec F \cdot d \vec S $ or $\int_c \vec F \cdot d \vec r $, because I always have difficulties on how to evaluate the domain of integration.
 [/INST] As Fantini points out, the boundary is a circle - we have a winner - of radius $3$.
Then, we fix $y=3$ and parameterise $(x,z) = (\sqrt{9-z^2},z)$ and then integrate from $z=3$ to $z=-3$ (going counterclockwise). One must also note that by the Implicit Function Theorem, there no unique function $x(z)$ in an open neighbourhood of $(x,y,z) = (0,3,3)$ or $(x,y,z)=(0,3,-3)$ and so we must also integrate from $z=-3$ to $z=3$ with the parameterisation $(x,z) = (-\sqrt{9-z^2},z)$. Now $dx = z/\sqrt{9-z^2}\; dz$ in the first case (opposite sign in the second).
Note that the $y$-component vanishes and so we are left with $\oint_{P} (27x^2z,3xz)\cdot (dx, dz) = - \int_{-3}^{3} (27z^2\sqrt{9-z^2}+3z\sqrt{9-z^2}) dz + \int_{-3}^{3} (-27z^2\sqrt{9-z^2}-3z\sqrt{9-z^2}) dz = -54 \int_{-3}^{3} z^2\sqrt{9-z^2} dz.$
So the problem reduces to (letting $u = \sqrt{9 - z^2}$):
$-54 \int_{-3}^{3} z^2\sqrt{9-z^2} dz = -108 \int_{0}^{3} u(9-u^2)\cdot \frac{-u}{\sqrt{9-u^2}} du = -108\cdot \frac{-81\pi}{16} = \frac{3^7 \pi}{4}.$
</s>