user46893: I am trying to transmit MPU6050 accelerometer data + data from 3 potentiometers from Arduino to Raspberry Pi. I do post processing here (on Raspberry Pi) and then transfer the data back to Arduino to move actuators. However my problem is that when I try to write the processed data back to Arduino, I see that even the stream of incoming data from Arduino starts getting corrupted and everything becomes really slow . Even If I just read serial data from Arduino, things are slower if than if I do the processing on Arduino directly. Also, the accelerometer data seems to become unresponsive i.e. its value doesn't change if I move the accelerometer. Everything just works fine if all the processing is done on Arduino. I don't know what is going wrong. Below are my Arduino and Python functions / codes.
Arduino Code:
void send_rec_data(){

    Serial.print(Kp);
    Serial.print(",");
    Serial.print(Kd);
    Serial.print(",");
    Serial.print(Ki);
    Serial.print(",");
    Serial.println(Theta_x);

    delay(20);    

  if (Serial.available()>0){ // Read data sent by Raspberry Pi / Piserial

    str_data = Serial.readString();    

    int firstCommaIndex = str_data.indexOf(',');

    cmd = str_data.substring(0, firstCommaIndex); // This stores the Scaled output from the Raspberry Pi

    param1 = str_data.substring(firstCommaIndex+1); // This stores the error from the Raspberry Pi to decide motor roattion direction

    Output = cmd.toFloat(); my_error = param1.toFloat(); // Change the outputs from string to float

    blink_led(); // Blink LEDS if data is received from Raspberry Pi



  }

}

Python Code:
while 1:

    data = cont.get_serial_data(arduino)

    time.sleep(0.02)

    if data is not None:

        Kp, Kd, Ki, Theta_x = data

        print(" \n Arduino sent Kp = ", Kp , "Kd = ",Kd , "Ki = ", Ki, "Theta_x = ", Theta_x)

        cont.set_tunings(Kp, Kd, 0)

        Output_scaled = cont.Compute_PID_Output(Input = Theta_x) # Compute the output of PID Algorithm

        ff = str(Output_scaled)+','+str(cont.error)

        arduino.write(ff.encode())

        print("\n Computed data sent TO ARDUINO Output_scaled = ", Output_scaled, "Error = ", cont.error)

arduino.close()

I have spent quiet sometime to resolve the issue byputting delays, flushing buffer but nothing has worked so far. Can someone please help. 
Thanks.

user-1: Ok, without rewriting all your code, one thing I'm seeing here is a lot of sloooow string functions.
Pythons print function is extremely slow, for example. You should insert some timers in your code and see how long everything's taking. I think you will be surprised.
For a real-time application, which it sounds like this is, you need to be very careful and use timeouts and non-blocking code to make sure that everything doesn't grind to a halt unexpectedly.
You really need to re-write all this code, but I'll point out a few specific problems.
delay(20); 
if (Serial.available()>0){
   ...
}

This is sloppy. If your python code does not process and return data before the 20ms elapses everything is going to go out of sync.
str_data = Serial.readString();

A look at the documentation reveals that this will only return on timeout, so this is guaranteed to block for 1s (I think that's the default), so there's another show-stopper problem. See this question for more info.
cmd = str_data.substring(0, firstCommaIndex);

This is also not sufficient. There is no synchronization between the pi and the arduino, so you have no guarantee that you won't get half of a previous transmission tacked onto the front of your data string (or worse). You need to make your own buffer and parse for the datastring. Or do some careful buffer clearing and use generous timeouts.
So TLDR; here's what you want to do (assuming you want to block while this exchange happens)

Clear the arduino receive buffer.
Send data packet from arduino.
Start receiving. Add received bytes to a char buffer.
Continuously search that buffer for your complete data packet.
Timeout after x ms. Handle the lack of returned data somehow.
If you do find the complete data packet, then process it.

On the Pi:

Poll for serial data. Add it to a buffer as it comes in.
Start deleting the oldest characters from the buffer if it's longer than 2x the longest data packet
Continuously search the buffer for your data packet. If you find it then continue below.
delete the data packet from the receive buffer.
process the data.
Send out return data packet to arduino.


46893: <p>I am trying to transmit MPU6050 accelerometer data + data from 3 potentiometers from Arduino to Raspberry Pi. I do post processing here (on Raspberry Pi) and then transfer the data back to Arduino to move actuators. However my problem is that when I try to write the processed data back to Arduino, I see that even the stream of incoming data from Arduino starts getting corrupted and everything becomes really slow . Even If I just read serial data from Arduino, things are slower if than if I do the processing on Arduino directly. Also, the accelerometer data seems to become unresponsive i.e. its value doesn't change if I move the accelerometer. Everything just works fine if all the processing is done on Arduino. I don't know what is going wrong. Below are my Arduino and Python functions / codes.</p>

<p><strong>Arduino Code:</strong></p>

<pre><code>void send_rec_data(){

    Serial.print(Kp);
    Serial.print(",");
    Serial.print(Kd);
    Serial.print(",");
    Serial.print(Ki);
    Serial.print(",");
    Serial.println(Theta_x);

    delay(20);    

  if (Serial.available()&gt;0){ // Read data sent by Raspberry Pi / Piserial

    str_data = Serial.readString();    

    int firstCommaIndex = str_data.indexOf(',');

    cmd = str_data.substring(0, firstCommaIndex); // This stores the Scaled output from the Raspberry Pi

    param1 = str_data.substring(firstCommaIndex+1); // This stores the error from the Raspberry Pi to decide motor roattion direction

    Output = cmd.toFloat(); my_error = param1.toFloat(); // Change the outputs from string to float

    blink_led(); // Blink LEDS if data is received from Raspberry Pi



  }

}
</code></pre>

<p><strong>Python Code:</strong></p>

<pre><code>while 1:

    data = cont.get_serial_data(arduino)

    time.sleep(0.02)

    if data is not None:

        Kp, Kd, Ki, Theta_x = data

        print(" \n Arduino sent Kp = ", Kp , "Kd = ",Kd , "Ki = ", Ki, "Theta_x = ", Theta_x)

        cont.set_tunings(Kp, Kd, 0)

        Output_scaled = cont.Compute_PID_Output(Input = Theta_x) # Compute the output of PID Algorithm

        ff = str(Output_scaled)+','+str(cont.error)

        arduino.write(ff.encode())

        print("\n Computed data sent TO ARDUINO Output_scaled = ", Output_scaled, "Error = ", cont.error)

arduino.close()
</code></pre>

<p>I have spent quiet sometime to resolve the issue byputting delays, flushing buffer but nothing has worked so far. Can someone please help. </p>

<p>Thanks.</p>
 -1: <p>Ok, without rewriting all your code, one thing I'm seeing here is a lot of <em>sloooow</em> string functions.</p>

<p>Pythons print function is extremely slow, for example. You should insert some timers in your code and see how long everything's taking. I think you will be surprised.</p>

<p>For a real-time application, which it sounds like this is, you need to be very careful and use timeouts and non-blocking code to make sure that everything doesn't grind to a halt unexpectedly.</p>

<p>You really need to re-write all this code, but I'll point out a few specific problems.</p>

<pre><code>delay(20); 
if (Serial.available()&gt;0){
   ...
}
</code></pre>

<p>This is sloppy. If your python code does not process and return data before the 20ms elapses everything is going to go out of sync.</p>

<pre><code>str_data = Serial.readString();
</code></pre>

<p>A look at the documentation reveals that this will only return on timeout, so this is guaranteed to block for 1s (I think that's the default), so there's another show-stopper problem. See <a href="https://stackoverflow.com/questions/42863973/arduino-readstring-code-runs-slow">this question</a> for more info.</p>

<pre><code>cmd = str_data.substring(0, firstCommaIndex);
</code></pre>

<p>This is also not sufficient. There is no synchronization between the pi and the arduino, so you have no guarantee that you won't get half of a previous transmission tacked onto the front of your data string (or worse). You need to make your own buffer and parse for the datastring. Or do some careful buffer clearing and use generous timeouts.</p>

<p>So TLDR; here's what you want to do (assuming you want to block while this exchange happens)</p>

<ol>
<li>Clear the arduino receive buffer.</li>
<li>Send data packet from arduino.</li>
<li>Start receiving. Add received bytes to a char buffer.</li>
<li>Continuously search that buffer for your complete data packet.</li>
<li>Timeout after x ms. Handle the lack of returned data somehow.</li>
<li>If you do find the complete data packet, then process it.</li>
</ol>

<p>On the Pi:</p>

<ol>
<li>Poll for serial data. Add it to a buffer as it comes in.</li>
<li>Start deleting the oldest characters from the buffer if it's longer than 2x the longest data packet</li>
<li>Continuously search the buffer for your data packet. If you find it then continue below.</li>
<li>delete the data packet from the receive buffer.</li>
<li>process the data.</li>
<li>Send out return data packet to arduino.</li>
</ol>
