 <p>In order to compute the rmse it is neccessary to have a maximum of 1 y value per x value, mathematically speaking a "function". You can plot later again the way your sketch shows it. But for the computation: rotate by 90 degree.</p> <p>For the optimization, you need an initiall guess (your <code>Alpha</code>), which is not defined. However, with some minor changes in the code below, one can get the optimal parameters. Also it might be better readable. The plot is shown below.</p> <pre><code>plot(r, XMean ,'ko', 'markerfacecolor', 'black') % // need a "function" (max 1y value for 1x-value) Eqn = @(par, x) par(1).*( 1 - ( (par(2) - x) / par(3) ).^2 ); par0 = [1, 1, 1]; % // an initial guess as start for optimization par_fit = nlinfit(r, XMean, Eqn, par0 ); % // optimize hold on r_fit = linspace(min(r), max(r)); % // more points for smoother curve plot(r_fit, Eqn(par_fit, r_fit), 'r') </code></pre> <p><img src="https://i.stack.imgur.com/w6PDy.png" alt="enter image description here"> The optimal parameters i get returned are</p> <pre><code>par_fit = 0.1940 -0.4826 9.0916 </code></pre> <p>You might want to rotate the plot again to get it into the right orientation, as in your sketch.</p> <h1>Additional Info: polyfit() instead of nlinfit</h1> <p>Since one can compute this flow profile analytically, one knows in advance that is is a parabola. You can also use <code>polyfit</code> to fit a parabola.</p> <pre><code>par_poly_fit = polyfit(r, XMean, 2) % // it yields the values: par_poly_fit = -0.0023 -0.0023 0.1934 </code></pre> <p>Plotting that paraobla gives a line pretty much identical to the line, which we got with the optimizer, but it is just more stable since it does not depend on an initial value.</p> <pre><code> plot(r_fit, polyval(par_poly_fit, r_fit),'g--' </code></pre>