How to deserialize a very large document?

By design, ArduinoJson stores the complete JSON document in RAM. Therefore, it cannot deserialize a document larger than your microcontroller’s capacity. The ArduinoJson Assistant will warn you if you try to deserialize a document that is too big.

Some ESP32s come with a large external PSRAM that you can use with ArduinoJson.

ArduinoJson supports two techniques to deserialize very large inputs: “filtering” and “deserialization in chunks.” These techniques allow you to read a document that would otherwise not fit in the RAM of your microcontroller.

Filtering

deserializeJson() supports a filtering feature that reduces memory consumption by ignoring irrelevant fields from the input document. It allows you to keep only the fields that matter in the JsonDocument.

To use this feature, you must create an ancillary JsonDocument that will serve as a filter. This document must contain the value true as a placeholder for each field you want to keep; every other field will be discarded. For arrays, only create one element in the filter document; it will serve as the filter for every element of the original document.

For example, suppose your input looks like this:

{
  "list": [
    {"temperature":21.2,"humidity":68.9,"pressure":1003},
    {"temperature":19.7,"humidity":62.1,"pressure":1007},
    {"temperature":18.6,"humidity":59.8,"pressure":1009}
   ] 
}

If you only want to keep the temperature field, you must create the following filter:

{
  "list": [
    {"temperature":true}
   ] 
}

Let’s put this example into code. Here is how we can create the filter document:

JsonDocument filter;
filter["list"][0]["temperature"] = true;

Once the filter document is ready, you must wrap it with DeserializationOption::Filter, and pass it to deserializeJson():

deserializeJson(doc, input, DeserializationOption::Filter(filter));

After executing this line, doc will contains the following document:

{
  "list": [
    {"temperature":21.2},
    {"temperature":19.7},
    {"temperature":18.6}
   ] 
}

In this case, filtering reduced the memory consumption in half.

The OpenWeatherMap case study in Mastering ArduinoJson shows how to use this technique in a real project.

Deserialization in chunks

Instead of parsing the whole JSON document in one shot, you can parse only a part of it and repeat the operation.

For example, suppose you are interested in the “characters” array from the following JSON document:

{
   "characters": [
        {"name":"Homer Simpson"},
        {"name":"Marge Simpson"},
        {"name":"Bart Simpson"},
        {"name":"Lisa Simpson"},
        // ...
    ]
}

If you try to deserialize the JSON document, it will probably not fit in the RAM of the microcontroller. However, you could easily deserialize each JSON object (i.e., each “character”), one by one.

One cool feature of ArduinoJson is that, when it parses an object from a Stream, it stops reading as soon as it sees the closing brace (}). The same is true with arrays: it stops reading as soon as it sees the closing bracket (]).

This feature allows you to parse streams in chunks: you just need to call deserializeJson() in a loop. Of course, you must skip the commas (,) between the objects.

Here is how this technique works:

Jump to the beginning of the array.
Call deserializeJson()
Read the next character; if it’s a comma, go to 2.

For step 1, you can use Stream::find(); for step 2, you can use Stream::findUntil()

Here is how you can implement this technique:

input.find("\"characters\":[");
do {
    deserializeJson(doc, input);
    // ...
} while (input.findUntil(",","]"));

The Reddit case study in Mastering ArduinoJson shows how to use this technique in a real project.

This technique only works for arrays; you cannot deserialize a large object in chunks.

Combining both techniques

You can also combine both techniques to deserialize in chunks and filter the content of each chunk.

The code is similar to the one above, except that you pass a filter document to deserializeJson():

JsonDocument filter;
filter["name"] = true;

input.find("\"characters\":[");
do {
    deserializeJson(doc, input, DeserializationOption::Filter(filter));
    // ...
} while (input.findUntil(",","]"));

Filtering

Deserialization in chunks

Combining both techniques

See also