ArduinoJson 6.20: shallow copy and documentation

ArduinoJson 6.20 is finally out! This long-awaited release contains many changes on the inside but not so much on the outside. Indeed, in the year that passed since 6.19.0, I regularly committed changes to the code, but I didn’t publish a new release because I wanted to complete the internal cleanup before doing so.

In this article, I’ll review all the visible changes that 6.20 brings:

1. a new “shallow copy” feature for JsonVariant,
2. a slightly stricter JSON parser,
3. documentation for most public symbols,
4. the rename of internal symbols,
5. the removal of many undocumented functions,
6. a new overload of JsonArray::add(),
7. the removal of support for naked char.

Shallow copy

Sometimes, you compose a JsonDocument from multiple other JsonDocuments. For example, you could have a general configuration object you construct by including the configuration from several modules.

Up till now, we had to copy all the data into a huge JsonDocument, like so:

DynamicJsonDocument gpsConfig = getGpsConfig();
DynamicJsonDocument gpioConfig = getGpioConfig();
DynamicJsonDocument config(4096);
config["gps"] = gpsConfig;
config["gpio"] = gpioConfig;
serializeJson(config, file);

Now, we can use JsonVariant::shallowCopy() to include the data without making a copy:

DynamicJsonDocument gpsConfig = getGpsConfig();
DynamicJsonDocument gpioConfig = getGpioConfig();
StaticJsonDocument<128> config;
config["gps"].shallowCopy(gpsConfig);
config["gpio"].shallowCopy(gpioConfig);
serializeJson(config, file);

By avoiding the deep copy, this feature reduces the size of the final JsonDocument and improves performance. Of course, since the new JsonDocument refers to the nested ones, you must ensure they remain in memory for the whole operation.

Stricter JSON parser

ArduinoJson’s JSON parser has always favored code size over strict conformance. It never rejects a valid JSON document, but it may accept an invalid JSON document in some cases.

For example, up till now, the parser avoided string comparisons for the true, false, and null. Instead, it just looked at the first character and the string length. In other words, it accepted any four-letter word starting with n, like none or nope, for null.

Unfortunately, the optimization made the parser confuse error messages with valid JSON documents. For example, a user reported that deserializeJson(doc, "force esp exception") returned Ok. This is a small problem, but the diagnosis can take some time; that’s why I decided to remove this optimization.

Now, ArduinoJson 6.20 checks the entire word for null, false, and true. There remain some cases where the parser will overlook errors in the input; for example, it will not report incorrect UTF-16 surrogates pairs.

Documentation

I added a short comment on the top of almost every public symbol of the library. These comments should appear in your IDE when you hover a symbol.

As you can see, I didn’t use any markup in the comments because I couldn’t not find something that worked correctly in both Visual Studio and Visual Studio Code. As a result, each comment is a one- or two-line description followed by a link to the full documentation.

Unfortunately, the Arduino IDE is not showing symbol comments, but hopefully, it will do in the future.

Massive internal renames

The internal classes in ArduinoJson used to have different names. For example, the implementation of JsonArray was ArrayRef, and the one of JsonArrayConst was ArrayConstRef.

While I liked the flexibility and expressiveness these names gave me, I now think it was a bad idea. The first issue is that the internal names frequently appear in the error messages, which confuses users. The second issue is that I had to use typedef to rename the symbols, and Visual Studio Code refused to show the documentation for those.

That is why I decided to rename every internal symbol to match the public ones.

Hide internals

Another issue we used to have with IDEs is that they suggested functions reserved for internal use, and some users ended up using them as if they were part of the official API. For example, I saw some of you use functions like getMember() or getOrCreateMember() even though they were supposed to be internal to the library.

As part of this “developer experience” package, I decided to hide as much internal stuff as possible. Of course, this is a breaking change, but that’s what happens when one uses undocumented APIs. Fortunately, you can easily recreate the behavior of internal functions with the public API.

Here is how you can update you code:

// before
JsonVariant a = variant.getElement(idx);
JsonVariant b = variant.getOrAddElement(idx);
JsonVariant c = variant.getMember(key);
JsonVariant d = variant.getOrAddMember(key);

// after
JsonVariant a = variant[idx];
JsonVariant b = idx < variant.size() ? variant[idx] : variant[idx].to<JsonVariant>();
JsonVariant c = variant[key];
JsonVariant d = variant.containsKey(key) ? variant[key] : variant[key].to<JsonVariant>();

New overload of JsonArray::add()

In all these removed internal functions, addElement() couldn’t be easily recreated with the public ones, so I decided to rename it and make it part of the public API. It’s now available as the parameterless overload of JsonArray::add(), JsonDocument::add(), and JsonVariant::add().

This function appends an empty element to an array and returns a reference to the new element. I don’t expect this function to be very popular, but I call it in many places in the library, so I figured I might as well make it available to everyone.

Fun fact: this undocumented function used to be named JsonArray::add() three years ago when ArduinoJson 6 was still in beta. I literally went full circle on that one.

Removed support for char and char*

Support for naked chars was marked deprecated since ArduinoJson 6.18 because it caused issues with std::string. After a year and a half, I removed this deprecated code.

This is a breaking change: you must replace char with either signed char, unsigned char, or any other integer type. Similarly, you must replace char* with const char*.

For example:

// before
char age = doc["age"];
auto name = doc["name"].as<char*>();

// after
int8_t age = doc["age"];
auto name = doc["name"].as<const char*>();

Code size

As usual, I try to keep the library as small as possible, so I constantly monitor the size of the five most representative examples. You can see the evolution in the two charts below.

The library grew a little because of a simplification I introduced in the string adapters. These classes allow ArduinoJson to support several types of string (const char*, Flash strings, String, std::string, std::string_view), and I wanted to simplify the API so that you could easily add your custom string adapter in the future.

Conclusion

The next version of ArduinoJson should come very soon but will not bring any new features because I’ll dedicate the release to dropping support for C++03. Apparently, I was one of the last library developers still caring about old C++ compilers. Chances are that all ArduinoJson users switched to modern compilers a long time ago; that’s why I’ll make C++11 a requirement for version 6.21. As you should see in the banner at the top of arduinojson.org, I’m currently running a survey to know if any of you still need support for C++03. If that’s your case, now is your last chance to raise your voice!

See you next year!