user88368: I have an Arduino UNO and an ESP32 that need to communicate to each other using SoftwareSerial. The problem I am coming across is that when I call readString and print it out using Serial, it will not give me everything that has been printed out from my ESP32.
I have my Arduino RX Pin hooked up to 4 and TX hooked up to 5
I have my ESP32 RX2 Pin hooked up to 16 and TX2 pin hooked up to 17, with the GND hooked up to directly to the Arduino GND
I am not using any resistors, straight up 5v of pure power
Arduino UNO Code:
// C-standard library
#include <stdbool.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <assert.h>

#include <SoftwareSerial.h>

#define RX_PIN 4
#define TX_PIN 5

SoftwareSerial ardSerial(RX_PIN, TX_PIN);

void setup(void) {
  // Get access to network
  pinMode(RX_PIN, INPUT);
  pinMode(TX_PIN, OUTPUT);
  Serial.begin(9600);
  while(!Serial) {}
  ardSerial.begin(9600);
  while(!ardSerial) {}
}

void loop(void) {
  while (ardSerial.available() > 0) {
    String payload = ardSerial.readString();
    Serial.println(payload);
  }
  delay(1000);

}

ESP32 Code
#include <stdlib.h>
#include <string.h>

#include <SoftwareSerial.h>

SoftwareSerial ardSerial;

void setup(void) {
  // Begin connections
  Serial.begin(115200);
  while (!Serial) {}

  ardSerial.begin(9600, SWSERIAL_8N1, RX2_PIN, TX2_PIN);
  while (!ardSerial) {}
  return;
}

void loop(void) {
  {
    String spotData = "Huge JSON File that needs transferred";
    ardSerial.print(spotData);
  }

  delay(5000);
}

Expected result:

Able to retrieve every little bit of the JSON file that was printed out

Actual result:

Only receive like half of the JSON file


user7508: There are two ways to implement a parser for a language like JSON or
XML:

Have the parser swallow the whole document at once, and produce an
in-memory representation of the same data.

Have the parser return control to the caller for each “event” it
detects, where an “event” is typically a primitive data item, or the
start or the end of a container.


Parsers of the first kind are much easier to use, but this can be
expensive in RAM usage if you want to parse a alrge document. The second
kind of parsers are harder to use, but they are able to parse huge
documents using a minimal amount of memory.
On an Arduino, you would typically use ArduinoJson, which is a parser of
the first kind, not ideal for parsing huge documents on little memory. A
quick search for “event driven C++ JSON parser” got me JsonReader. I
did not try it, and do not know whether that would work on Arduino, but
it shows that at least this kind of parsers do exist for JSON in C++.
You may want to search further along this way.
Before you dig into this, however, I suggest you give ArduinoJson a
second chance. ArduinoJson has a couple of tricks that can help dealing
with large documents:

You can ask it to filter the data as it reads it, in order to only
keep in memory the items you are actually interested in.

You can deserialize in chunks, and use the data extracted from one
chunk at a time.


Both techniques are explained in the tutorial How to deserialize a very
large document?.
Also, let's not forget the obvious: you should not attempt to store the
raw JSON document in memory. Instead, you should give ArduinoJson a
reference to your stream (namely ardSerial), and let it directly pull
the bytes as it parses.

88368: <p>I have an Arduino UNO and an ESP32 that need to communicate to each other using SoftwareSerial. The problem I am coming across is that when I call readString and print it out using Serial, it will not give me everything that has been printed out from my ESP32.</p>
<p>I have my Arduino RX Pin hooked up to 4 and TX hooked up to 5
I have my ESP32 RX2 Pin hooked up to 16 and TX2 pin hooked up to 17, with the GND hooked up to directly to the Arduino GND
I am not using any resistors, straight up 5v of pure power</p>
<p>Arduino UNO Code:</p>
<pre><code>// C-standard library
#include &lt;stdbool.h&gt;
#include &lt;stdio.h&gt;
#include &lt;stdlib.h&gt;
#include &lt;string.h&gt;
#include &lt;assert.h&gt;

#include &lt;SoftwareSerial.h&gt;

#define RX_PIN 4
#define TX_PIN 5

SoftwareSerial ardSerial(RX_PIN, TX_PIN);

void setup(void) {
  // Get access to network
  pinMode(RX_PIN, INPUT);
  pinMode(TX_PIN, OUTPUT);
  Serial.begin(9600);
  while(!Serial) {}
  ardSerial.begin(9600);
  while(!ardSerial) {}
}

void loop(void) {
  while (ardSerial.available() &gt; 0) {
    String payload = ardSerial.readString();
    Serial.println(payload);
  }
  delay(1000);

}
</code></pre>
<p>ESP32 Code</p>
<pre><code>#include &lt;stdlib.h&gt;
#include &lt;string.h&gt;

#include &lt;SoftwareSerial.h&gt;

SoftwareSerial ardSerial;

void setup(void) {
  // Begin connections
  Serial.begin(115200);
  while (!Serial) {}

  ardSerial.begin(9600, SWSERIAL_8N1, RX2_PIN, TX2_PIN);
  while (!ardSerial) {}
  return;
}

void loop(void) {
  {
    String spotData = &quot;Huge JSON File that needs transferred&quot;;
    ardSerial.print(spotData);
  }

  delay(5000);
}
</code></pre>
<p>Expected result:</p>
<ul>
<li>Able to retrieve every little bit of the JSON file that was printed out</li>
</ul>
<p>Actual result:</p>
<ul>
<li>Only receive like half of the JSON file</li>
</ul>
 7508: <p>There are two ways to implement a parser for a language like JSON or
XML:</p>
<ol>
<li><p>Have the parser swallow the whole document at once, and produce an
in-memory representation of the same data.</p>
</li>
<li><p>Have the parser return control to the caller for each “event” it
detects, where an “event” is typically a primitive data item, or the
start or the end of a container.</p>
</li>
</ol>
<p>Parsers of the first kind are much easier to use, but this can be
expensive in RAM usage if you want to parse a alrge document. The second
kind of parsers are harder to use, but they are able to parse huge
documents using a minimal amount of memory.</p>
<p>On an Arduino, you would typically use ArduinoJson, which is a parser of
the first kind, not ideal for parsing huge documents on little memory. A
quick search for “event driven C++ JSON parser” got me <a href="https://github.com/xquintana/JsonReader" rel="nofollow noreferrer">JsonReader</a>. I
did not try it, and do not know whether that would work on Arduino, but
it shows that at least this kind of parsers do exist for JSON in C++.
You may want to search further along this way.</p>
<p>Before you dig into this, however, I suggest you give ArduinoJson a
second chance. ArduinoJson has a couple of tricks that can help dealing
with large documents:</p>
<ul>
<li><p>You can ask it to filter the data as it reads it, in order to only
keep in memory the items you are actually interested in.</p>
</li>
<li><p>You can deserialize in chunks, and use the data extracted from one
chunk at a time.</p>
</li>
</ul>
<p>Both techniques are explained in the tutorial <a href="https://arduinojson.org/v6/how-to/deserialize-a-very-large-document/" rel="nofollow noreferrer">How to deserialize a very
large document?</a>.</p>
<p>Also, let's not forget the obvious: you should not attempt to store the
raw JSON document in memory. Instead, you should give ArduinoJson a
reference to your stream (namely <code>ardSerial</code>), and let it directly pull
the bytes as it parses.</p>
