ArduinoJson 6.15: Filtering done right

I just published ArduinoJson 6.15. This revision includes something you’ve been waiting for a long time: filtering! This feature helps to reduce memory consumption by eliminating unnecessary data form large inputs. Version 6.15 also brings a new function to recover from memory leaks in JsonDocument, as well as a few changes in the copy-construction rules.

Filtering

Motivation

If you ever wrote a program that downloads data from a web service, you probably already had this problem: the server replies with a colossal JSON document, but most of it is irrelevant to your application.

When the code runs on a PC, that’s not an issue because you have a virtually unlimited amount of memory at your disposal. On an embedded platform, however, you only have a small number of kilobytes to store this document in RAM.

In previous versions of ArduinoJson, the solution was to deserialize the document in chunks by calling deserializeJson() several times. While this technique is still needed in some cases, most projects can now use a much simpler method: filtering.

The filter document

Version 6.15 introduces a new filtering feature that allows you to specify which fields ArduinoJson should store and which ones it can safely discard. To use this feature, you must create an ancillary JsonDocument that serves as a filter. This document must contain the value true for each member you want to keep in the final document. Regarding arrays, the first element serves as a filter for all items of the source array; the other elements are ignored.

For example, suppose your input looks like:

{
  "list": [
    {"temperature":21.2,"humidity":68.9,"pressure":1003},
    {"temperature":19.7,"humidity":62.1,"pressure":1007},
    {"temperature":18.6,"humidity":59.8,"pressure":1009}
   ] 
}

If you only want to keep the temperature field, you must create the following filter:

{
  "list": [
    {"temperature":true}
   ] 
}

Implementation

Let’s put this example into code. Here is how we can create the filter document:

StaticJsonDocument<64> filter;
filter["list"][0]["temperature"] = true;

As you can see, we can create the nested array and object implicity without calling createNestedArray() and createNestedObject(). This is another new feature from version 6.15.

The next step is to wrap the filter document with DeserializationOption::Filter and pass it to deserializeJson():

deserializeJson(doc, input, DeserializationOption::Filter(filter));

After executing this line, doc will contains the following document:

{
  "list": [
    {"temperature":21.2},
    {"temperature":19.7},
    {"temperature":18.6}
   ] 
}

In this case, filtering reduced the memory consumption in half.

This feature is currently only available for deserializeJson(). Let me know if you need it for deserializeMsgPack().

Garbage collect

Memory leaks

The secret behind the performance of ArduinoJson is the monotonic allocator inside JsonDocument. It’s small and fast, but by definition, it cannot release memory.

If you only use ArduinoJson for serialization, the monotonic allocator should not be an issue. Indeed, if you use a JsonDocument for a short period and destroy it quickly, there is no need to release blocks inside the JsonDocument.

However, if you push ArduinoJson beyond its original intent, for example, if you use a JsonDocument to store the state of your app in a global variable, then you’ll surely have a memory leak.

A leak occurs when:

• you remove a member from an object
• you remove an element from an array
• you replace a string stored in the JsonDocument (by copy, not by pointer)

If you do these operations repeatedly, your JsonDocument will saturate sooner or later.

Fixing the leaks

It’s possible to reclaim the leaked blocks by making a copy of the JsonDocument and discarding the original. This operation is slow because it duplicates the entire content of the document; it’s also memory consuming because you temporarily have two instances of the same document.

This technique was already available with the previous versions of ArduinoJson 6, but it was quite tricky; that’s why I decided to add the function JsonDocument::garbageCollect(), which performs this operation in a straightforward manner. Internally, this function still performs a deep-copy of the document, so it’s slow and memory consuming, but at least it’s easy to use.

Copy constructor of BasicJsonDocument

I don’t think anyone will notice, but I changed the way the copy-constructor of BasicJsonDocument (the real class behind DynamicJsonDocument) determines the capacity. In previous versions, the copied document had a capacity that matched the current memory usage of the original document. Now, the copied document has the same capacity as the original.

Here is an example that illustrate the difference:

DynamicJsonDocument doc1(64);
doc1.set(String("example"));

DynamicJsonDocument doc2 = doc1;
Serial.print(doc2.capacity());  // 8 with ArduinoJson 6.14
                                // 64 with ArduinoJson 6.15

I made this change because I added a move-constructor to BasicJsonDocument, and I wanted the move-constructor and the copy-constructor to be consistent.

Copy constructor of JsonDocument

Recently, someone opened a question on Stack Overflow because he didn’t understand why its program didn’t work. The program used a std::vector<JsonDocument> to store a list of documents. Unfortunately, this code didn’t do what the programmer expected: he expected a copy of the documents, but instead, the list contained truncated versions of the original DynamicJsonDocuments.

A few days before, another user had problems because he wrote a program that passes JsonDocument by value:

void myFunction(JsonDocument doc) {}

Here too, the JsonDocument looks like a copy, but it’s not. It’s a truncated version of the original document that only contains the internal pointers.

These two problems were due to a flaw in the library: it should not be possible to copy a JsonDocument in the first place. Only copying a StaticJsonDocument or a DynamicJsonDocument makes sense.

For this reason, I removed (more precisely, I hid) the copy-constructor of JsonDocument, so one cannot fall into this trap again. Starting from version 6.15, if you need to pass a JsonDocument to a function, either take a reference or a copy:

void myFunction1(JsonDocument& doc);
void myFunction2(DynamicJsonDocument doc);

Conclusion

Whether you use the filtering feature or the garbage collection function, this release should greatly simplify your code.

As usual, I took extreme care not to increase the size of the library. The graph below shows the evolution of the size of the two examples JsonGeneratorExample and JsonParserExample. As you can see, the size is relatively constant since 6.8 and is comparable to what we had in version 5.

With all these changes, Mastering ArduinoJson needed a refresh; that’s why I prepared a new edition of the book. I’ve been working on it for several weeks, and it should be ready in the next few days.

One more time, I’d like to thank every person who purchased the book because you truly help me keep the project rolling. Without your support, I would probably have moved on to other projects, but knowing that many people appreciate the work really motivates me in delivering the best possible library.

See also

• deserializeJson()
• JsonDocument::garbageCollect()
• I found a memory leak in the library!
• Mastering ArduinoJson