Can I parse a JSON input that is too big to fit in memory?

Is it possible?

Yes, but not in one shot.

Why?

By design, ArduinoJson stores the complete representation of the JSON document in memory.

If you use the zero-copy mode (mutable input), the whole JSON input must stay in memory.

If the input is read-only (like a Stream), the JsonBuffer needs to make copies of the input, so the result is almost the same (except for spaces and punctuation).

How to solve?

One cool feature of ArduinoJson is that, when it parses an object from a Stream, it stops reading as soon as it sees the closing brace (}). The same is true with arrays: it stops reading as soon as it sees the closing bracket (]).

This feature allows to parse streams in chunks, you just need to call JsonBuffer::parseObject() in a loop. Of course, don’t forget to skip the commas (,) between the objects. As usual, don’t reuse the JsonBuffer, declare it inside the loop.

For example, if you want to parse the huge response of a 10-day forecast of Weather Underground, you can skip the beginning until you see "forecastday": [ in the stream (use Stream::find()), and then parse the objects for each day one after the other.

In short:

1. jump to the beginning of the object with Stream::find()
2. call JsonBuffer::parseObject()
3. read next byte from the stream
4. if it’s a comma, go to 2., otherwise stop the loop.

A complete example

In the chapter “Case studies” of Mastering ArduinoJson, two projects use this technique, one for OpenWeatherMap and another for Weather Underground. The book gives a complete walk-through and explains how this technique works. The complete source code is available in the companion zip file.

Of course, the book contains several other chapters, including a quick C++ course, two complete tutorials and an inside view of the library.