<s>[INST] Least squares fitting using cosine function? Hello I am trying to fit a harmonic of the form $$y = b + c\cos(x)$$ to four data points (0,6.1) (.5,5.4) (1,3.9) (1.5,1.6) using least squares for homework. I know that the error $= Y_i - f(x_i)$ but am pretty confused with the partial derivatives and linear algebra part of this question. Apparently the professor wants only $y=b+c\cos(x)$ and not $y=b+c\cos(x)+d\sin(x)$
I get two equations from taking the derivative of $E = \sum_{i=0}^n [y_i - (b+c\cos(x_i))]^2$, with respects to $b$ and $c$.
$$ \frac{dE}{db} = bn + \sum_{i=0}^n c\cos(x_i) = \sum_{i=0}^n y_i $$ and
$$ \frac{dE}{dc} = b\cos(x_i) + \sum_{i=0}^n c\cos^2(x_i) = \sum_{i=0}^n \cos(x_i) y_i $$
Putting this into matrix form gives me:
$$ \left[ \begin{array}{cc|c}
n & \sum_{i=0}^n (x_i) &  \sum_{i=0}^n (y_i) \\
\sum_{i=0}^n \cos(x_i) & \sum_{i=0}^n \cos^2(x_i) &  \sum_{i=0}^n \cos(x_i)y_i \\
\end{array}\right] $$
which lands me with $$ \left[\begin{array}{cc|c}
4 & 3 & 17 \\
2.48862 & 2.067077 & 13.05931
\end{array}\right] $$
Sorry for the terrible formatting | is meant to make this an augmented matrix (the four data points are included at the top). Anyways
I'm not really sure how to solve this matrix so I assume I did something wrong with the partial derivatives. Is there anyone out there that could be so helpful as to let me know what I did wrong or any hints on how to solve this system if it is indeed solvable?
 [/INST] The solution of @user147263 relies on calculus. Another option is to exploit linear algebra.
Form linear system
Start with a series of data points $(x_{k}, y_{k})_{k=1}^{m}$, and the trial function
$$
y(x) = c_{1} + c_{2} \cos x,
$$
We have the linear system
$$
\begin{align}
  A c & = y \\
    \left[ 
      \begin{array}{cc}
         1 & \cos x_{1} \\
         1 & \cos x_{2} \\
         \vdots & \vdots\\
         1 & \cos x_{m}
      \end{array}
    \right]
    \left[ 
      \begin{array}{c}
         c_{1} \\
         c_{2} 
      \end{array}
    \right]
    & = 
    \left[ 
      \begin{array}{c}
         y_{1} \\
         y_{2} \\
         \vdots \\
         y_{m}
      \end{array}
    \right]
\end{align}
$$
Find the solution vector $c$ which minimizes the sum of the squares of the residuals:
$$
r^{2}(c) = \lVert A c - y \rVert_{2}^{2} = \sum_{k=1}^{m} (y_{k} - c_{1} - c_{2} \cos x_{k})^2.
$$
Normal equations: Form the normal equations
$$
  A^{T}A c = A^{T}y.
$$
Solve linear system
The solution vector is 
$$
  c = (A^{T}A)^{-1} A^{T}y.
$$
Defining the $m-$vector 
${\mathbf{1}} = 
\left[ \begin{array}{c} 1 \\ 1 \\ 1 \\ 1 \end{array} \right]$, 
and the the $m-$vector 
$\eta = 
\left[ \begin{array}{c} \cos x_{1} \\  \cos x_{2} \\  \cos x_{3} \\  \cos x_{4} \end{array} \right]$, the system matrix has the column vector form 
$A = \left[ \begin{array}{c} \mathbf{1} & \eta \end{array} \right]$, and the   product matrix is 
$$
  A^{T} A =
      \left[ 
      \begin{array}{cc}
         \mathbf{1} \cdot \mathbf{1} & \mathbf{1} \cdot \eta \\
         \eta \cdot \mathbf{1} & \eta \cdot \eta
      \end{array}
    \right] =
%
      \left[ 
      \begin{array}{cl}
         m & \sum_{k=1}^{m} \cos x_{k} \\
         \sum_{k=1}^{m} \cos x_{k} & \sum_{k=1}^{m} \cos^{2} x_{k} 
      \end{array}
    \right]
$$
with inverse
$$
  ( A^{T} A )^{-1} =
      (\det (A^{T}A))^{-1}
      \left[ 
      \begin{array}{rc}
         \eta \cdot \eta & -\mathbf{1} \cdot \eta \\
         - \mathbf{1} \cdot \eta & m
      \end{array}
    \right]
$$
where the determinant is
$$
\det (A^{T}A) = m (\eta \cdot \eta) - (\mathbf{1}\cdot \eta)^{2}.
$$
For the data set above, $x =       
      \left[ 
      \begin{array}{l}
         0 \\
         0.5 \\
         1 \\
         1.5
      \end{array}
    \right]$, $\ $
$\eta =       
      \left[ 
      \begin{array}{l}
         1 \\
         0.877583 \\
         0.540302 \\
         0.0707372
      \end{array}
    \right]$, $\ $
$y =       
      \left[ 
      \begin{array}{c}
         6.1 \\
         5.4 \\
         3.9 \\
         1.6
      \end{array}
    \right]$, $\ $
and $\mathbf{1} =       
      \left[ 
      \begin{array}{c}
         1 \\
         1 \\
         1 \\
         1
      \end{array}
    \right].$ 
The inner products are
$$
\begin{align}
  \mathbf{1} \cdot \mathbf{1} &= 4, \\
  \mathbf{1} \cdot \eta = \eta \cdot \mathbf{1} &= 2.48862, \\
  \eta \cdot \eta & = 2.06708.
\end{align}
$$
The matrices are
$$
  A =
    \left[ 
      \begin{array}{rl}
         1 & 1 \\
         1 & 0.877583 \\
         1 & 0.540302 \\
         1 & 0.0707372
      \end{array}
    \right], \quad
%
  A^{T}A = 
    \left[ 
      \begin{array}{rl}
         4 & 3 \\
         3 & 3.5
      \end{array}
    \right], \quad
%
  (A^{T}A)^{-1} = 
    \det (A^{T}A)^{-1}
    \left[ 
      \begin{array}{rr}
         2.06708 & -2.48862 \\
         -2.48862 & 4
      \end{array}
    \right].
$$
The determinant is $\det (A^{T}A) = 2.07509$. The solution vector is
$$
c = 
    \left[ 
      \begin{array}{c}
         c_{1} \\
         c_{2} 
      \end{array}
    \right]
  =
    \left[ 
      \begin{array}{r}
         1.27258 \\
         4.78565 
      \end{array}
    \right]
$$
Check result
The solution curve is plotted against the data points.

A contour plot showing $r^{2}$ as a function of the solution vector $c$ with a white dot marking the solution.

</s>