<s>[INST] Find the area of the region that lies inside the first curve and outside the second curve. $r = 10 \cos\theta,\ r = 5$ I am not sure of my answer. In the figure, $r=10 \cos\theta$ is a circle that doesn't look like a circle. 
The area of $r=5$ is $\pi r^2 = 25 \pi$.
You remove the area from $-\pi/3$ to $\pi/3$ of $10 \cos\theta$ from $25\pi$. 
That is remove $$\frac 12 \int (10 \cos\theta)^2\,d\theta = 74.0105$$
Required area = $25 \pi - 74.0105 = 4.5293$
 [/INST] Let $S$ be the area of the red part in the following figure :
$\qquad\qquad\qquad$
You want to find $2S$.
$S$ can be obtained by 
$$S=\frac 12\times 5^2\pi-[\text{sector $OAB$}]-[\text{the gray part}]\tag1$$
Since $\triangle{OAB}$ is an equilateral triangle, we have $\angle{AOB}=\frac{\pi}{3}$. So, $$[\text{sector $OAB$}]=\frac 12\times 5^2\times\frac{\pi}{3}=\frac{25}{6}\pi\tag2$$
Also, 
$$[\text{the gray part}]=\int_{\frac{\pi}{3}}^{\frac{\pi}{2}}\frac 12(10\cos\theta)^2 d\theta=25\left[\frac 12\sin(2\theta)+\theta\right]_{\frac{\pi}{3}}^{\frac{\pi}{2}}=\frac{25}{6}\pi-\frac{25}{4}\sqrt 3\tag3$$
From $(1)(2)(3)$, we have
$$2S=5^2\pi-2\cdot\frac{25}{6}\pi-2\left(\frac{25}{6}\pi-\frac{25}{4}\sqrt 3\right)=\color{red}{\frac{25}{3}\pi+\frac{25}{2}\sqrt 3}.$$
</s>