Parsing an input that doesn’t fit in memory

“I have an Arduino UNO with 2KB of RAM, and I want to parse a JSON document of 300 KB… Can I do that with ArduinoJson?”

Yes, but not in one shot.

By design, ArduinoJson stores the complete representation of the JSON document in memory. If you don’t have enough memory on your microcontroller, it will not work.

You can use an external PSRAM with ArduinoJson, but you must define a custom allocator.

Instead of parsing the whole JSON document, you can parse only a part of it and repeat the operation.

For example, suppose you are interested in the “characters” array from the following JSON document:

   "characters": [
        {"name":"Homer Simpson"},
        {"name":"Marge Simpson"},
        {"name":"Bart Simpson"},
        {"name":"Lisa Simpson"},
        // ...

If you try to deserialize the JSON document, it will probably not fit in the RAM of the microcontroller. However, you could easily deserialize each JSON object (i.e., each “character”), one by one.

I call this technique “deserialize in chunks”; it works every time you need to parse a large array.

How to deserialize in chunks?

One cool feature of ArduinoJson is that, when it parses an object from a Stream, it stops reading as soon as it sees the closing brace (}). The same is true with arrays: it stops reading as soon as it sees the closing bracket (]).

This feature allows to parse streams in chunks: you just need to call deserializeJson() in a loop. Of course, you must skip the commas (,) between the objects.

Here is how this technique works:

  1. Jump to the beginning of the array.
  2. Call deserializeJson()
  3. Read the next character; if it’s a comma, go to 2.

For step 1, you can use Stream::find(); for step 2, you can use Stream::findUntil()

Here is how you can implement this technique:

do {
    deserializeJson(doc, input);
    // ...
} while (input.findUntil(",","]"));

A complete example

Mastering ArduinoJson

In the chapter “Case studies” of Mastering ArduinoJson, two projects use this technique, one for OpenWeatherMap and another for Reddit. The book gives a complete walk-through and explains how this technique works. The complete source code is available in the companion zip file.

Of course, the book contains several other chapters, including a quick C++ course, two complete tutorials and an inside view of the library.