ArduinoJson 7

The microcontroller landscape has significantly changed in the past five years. Previously monopolized by 8-bit microcontrollers, the market is now dominated by 32-bit microcontrollers. This hardware revolution required a redesign of ArduinoJson.

Once a competitive advantage of ArduinoJson 6, fixed memory allocation progressively became irrelevant. ArduinoJson 7 redefines the memory management strategy to provide an elegant and straightforward API.

In this article, I’ll give you an overview of what’s new in this release.

Code size

Before we go into the good stuff, I must warn you that ArduinoJson 7 is significantly bigger than version 6. For example, on an Arduino UNO R3, the parser example is 41% bigger, and the generator example is 45% bigger.

Previously, all my design decisions were aimed at keeping the code small. Indeed, when I designed ArduinoJson 6, most users ran their programs on 8-bit microcontrollers, and that’s why I focused so much on code size.

That’s how ArduinoJson 6 came to implement a fixed memory allocation strategy. Not only does it reduce the code size, but it also eliminates heap fragmentation and allows stack-based allocations. Fixed memory allocation is perfect when resources are scarce but requires more discipline from the programmer.

During the past five years, we attended to the rise of 32-bit microcontrollers, first with the ESP8266, then with the ESP32, and all the ARM-based MCUs.

32-bit microcontrollers have much more memory but also a bigger runtime framework, so ArduinoJson now represents a small fraction of the executable. If we compare the parser example on the two versions of the Arduino UNO, we see that ArduinoJson 7 makes up about two-thirds of the executable on R3 but only 7% on R4.

Because the proportion is much smaller, the difference between ArduinoJson 6 and 7 is neglectable on 32-bit microcontrollers. For example, on Arduino UNO R4 Minima, the parser example only grew by 2.3%, and the generator example by 1.7%. As you can see, the size of the library is not so important anymore.

ArduinoJson 7 can run on 8-bit microcontrollers, but if the memory is tight, it’s probably better if you stick with version 6.

Major changes

ArduinoJson 7 includes stubs for every deprecated feature, so most existing programs should continue to work. The stubs produce informative warning messages that tell you how to upgrade the code. Of course, since your programs continue to work, you don’t have to upgrade right away, but you’ll see that the process is very straightforward. I already published an article explaining the changes in detail, so I’ll only give an overview here.

The biggest change concerns JsonDocument. ArduinoJson 7 exclusively relies on dynamic memory allocation, and you no longer need to specify the capacity upon creation.

Since we can no longer choose between stack and heap memory, I merged StaticJsonDocument and DynamicJsonDocument into a single JsonDocument class.

- StaticJsonDocument<256> doc;
+ JsonDocument doc;

- DynamicJsonDocument doc(256);
+ JsonDocument doc;

To maintain excellent performance, ArduinoJson 7 reduces heap allocations by allocating blocks of 1KB. String deduplication also reduces the number of allocations, and a short-string optimization will soon reduce it even more. The impact on heap fragmentation should be very limited. In fact, the only function that should increase the fragmentation is deserializeJson() because it needs to reallocate blocks for strings.

Because the capacity of the JsonDocument is now elastic, several functions related to memory management became irrelevant: JSON_ARRAY_SIZE(), JSON_OBJECT_SIZE(), JsonDocument::capacity(), JsonDocument::memoryUsage(), and JsonDocument::garbageCollect().

JsonDocument::shrinkToFit(), which releases the over-allocated memory, is still available in ArduinoJson 7, but it is automatically called by deserializeJson(), so you probably don’t need to call it anymore.

I changed how we customize the allocator: instead of passing a template parameter, you must pass a pointer to a polymorphic allocator. This change, inspired by std::pmr, allows all JsonDocuments to be compatible with each other, regardless of the allocator. It also allowed me to get rid of the template class BasicJsonDocument.

- BasicJsonDocument<SpiRamAllocator> doc(4096);
+ SpiRamAllocator allocator;
+ JsonDocument doc(&allocator);

I managed to keep all features of ArduinoJson 6, except one: shallowCopy(). Indeed, due to significant changes in the library, a slot can no longer point to a different document. If your program calls shallowCopy(), ArduinoJson 7 will do a deep copy instead, which could break your program, so watch out!

On a different topic, I took the opportunity to remove one of the library’s biggest warts. createNestedArray() and createNestedObject() are a legacy of ArduinoJson 4, and they don’t fit with the rest of the API. In ArduinoJson 7, I replaced them with add<T>() and to<T>().

  // [["hello","world"]]
- JsonArray arr2 = arr1.createNestedArray();
+ JsonArray arr2 = arr1.add<JsonArray>();
  arr2.add("hello");
  arr2.add("world");

  // [{"hello":"world"}]
- JsonObject obj = arr.createNestedObject();
+ JsonObject obj = arr.add<JsonObject>();
  obj["hello"] = "world";

  // {"data":["hello", "world"]}
- JsonArray arr = obj.createNestedArray("data");
+ JsonArray arr = obj["data"].to<JsonArray>();
  arr.add("hello");
  arr.add("world");

  // {"data":{"hello":"world"}}
- JsonObject obj2 = obj1.createNestedObject("data");
+ JsonObject obj2 = obj1["data"].to<JsonObject>();
  obj2["hello"] = "world";

Other changes

Now, let’s see less important changes that should only affect a minority of users and probably don’t require your attention.

The first is the removal of the zero-copy mode. In version 6, deserializeJson() behaved differently depending on the input type. If the input was a char*, it used the zero-copy mode: instead of copying strings into the JsonDocument, it kept them in the input buffer and stored pointers into the JsonDocument. I was initially a big fan of this feature because it could save a lot of memory. Unfortunately, the zero-copy mode was misunderstood and caused many bugs, so I removed it from ArduinoJson 7. If your program relied on the zero-copy mode, you will see a significant increase in memory consumption after the upgrade.

Next, I fixed the weird behavior of serializeJson() with string classes. Indeed, in ArduinoJson 6, serializeJson() treated string objects (such as String or std::string) as streams, so instead of replacing their content, it appended to the end. In ArduinoJson 7, serializeJson() overrides the content as one would expect. In theory, it is a breaking change, but I doubt it will break anything in practice.

Finally, I changed the string copy policy of the serialized() function. In ArduinoJson 6, raw strings used the same policy as regular strings: all types were stored by copy, except const char*, which were stored by pointer. In ArduinoJson 7, all raw strings are stored by copy.

ArduinoJson Assistant

I already published a new version of the ArduinoJson Assistant. As you’ll see, it is slightly different from what you are used to.

First of all, the raison d’être of the Assistant changed. Before, the main goal was to help you compute the right capacity for the JsonDocument. Now, it is to validate that your JsonDocument can fit in the RAM of your microcontroller.

Consequently, step 3, which showed how much RAM you should reserve for your JsonDocument, was removed. The new Assistant still shows the memory consumption, but this was moved to step 2 and doesn’t give you as many details.

In addition to showing the memory consumption, the new Assistant tells you what percentage of the RAM is used. Of course, this requires knowing the amount of memory available on your board, and that’s why the first step now asks you to enter the board instead of the processor architecture.

The new Assistant still allows you to design a filter for deserializeJson(), although this feature has changed a little. Instead of choosing “Deserialize and filter” in the first step, you must now check “Enable input filter” in the second step.

The last step, which generates a sample program, remained roughly the same, except you can now customize the deserialization program. Two settings are available. The first chooses the output for the error messages: Serial or std::cout. The second turns Flash strings on or off.

arduinojson.org

As you probably already saw, all the documentation for ArduinoJson 7 is already there. You can select the version on the navigation bar at the top of the screen.

I also updated the Troubleshooter to support ArduinoJson 7. You’ll see that it now starts by asking which version you use, and I even included ArduinoJson 5 for completeness.

If you encounter any issue with the library, please try to self-diagnose your problem with the ArduinoJson Troubleshooter. If it is unable to help, please open an issue on GitHub. Remember to include the Troubleshooter report in the issue description. The report gives me the context and tells me what you already tried. It also allows me to continue improving the Troubleshooter by adding more choices at the right locations.

Mastering ArduinoJson

I published a new edition of my book Mastering ArduinoJson for version 7.

The overall structure remained the same:

• Chapter 1 briefly introduces ArduinoJson and what you can do with it. When updating the manuscript, I was amazed by the number of Web APIs that had disappeared since the last edition. I included the list so that you realize how important it is to choose your service provider wisely.
• Chapter 2 teaches elementary C++ aspects that most Arduino users ignore. It’s called “The missing C++ course” because it covers what is usually omitted by other courses.
• Chapters 3 and 4 are the deserialization and serialization tutorials. They are available for free on arduinojson.org.
• Chapter 5 covers advanced topics, such as filtering, deserialization in chunks, JSON streaming, allocators, readers, writers, converters, and other valuable techniques.
• Chapter 6 explains how the library works and how to navigate the code. I rewrote this chapter from scratch, and this version is radically different from the previous one.
• Chapter 7 is a troubleshooting guide; it shows how things can go wrong and how to prevent bugs. Most of this chapter applies to any C++ program, whether it uses ArduinoJson or not.
• Chapter 8 contains five case studies showing how to use ArduinoJson in different scenarios.

I invite you to purchase the book directly at arduinojson.org/book. Not only will you learn essential things, but you’ll also support my work, allowing me to continue investing a crazy amount of time in improving the library and helping the community.

Future plans

As you can imagine, this new release has required tremendous work, so I might take a break before adding new features.

If you’ve been following me, you know that I rarely announce what’s coming in the next version. As usual, I won’t make any promises about what’s coming up, but I can tell you what’s on my mind at the moment.

1. Implement short-string optimization, which would continue to reduce the number of allocations.
2. Rewrite the parser in a non-recursive way, which would allow me to get rid of DeserializationOption::NestingLimit and DeserializationError::TooDeep.
3. Add a new parser for JSON5, which would eliminate the need for ARDUINOJSON_ENABLE_COMMENTS.
4. Refactor the parsers to support incremental data, which would be fantastic for scenarios where you receive the JSON document piece by piece, like with ESPAsyncWebServer.

Of course, I’m not making any commitments since my priorities might change. All I can promise is that I’ll continue working on ArduinoJson as much as my time allows.

See also

• How to upgrade your code from ArduinoJson 6 to 7
• How to upgrade your code from ArduinoJson 5 to 7