ArduinoJson 6.16: String Deduplication

I just published version 6.16.0 of the library. As we’ll see, this revision reduces memory consumption significantly with minimal impact on speed. Let’s review the changes together.

String deduplication

As you should already know, when you add a string to a JsonDocument, ArduinoJson stores it either by pointer or by copy. It uses a pointer when the string has the type const char*, and it duplicates the string in all other cases (char[], String, Flash string…). Similarly, when you call deserializeJson(), it copies the strings into the JsonDocument (except, of course, if you use the zero-copy mode).

Previously, ArduinoJson blindly duplicated a string even if it was already present in the JsonDocument. Now, with ArduinoJson 6.16, it stores only one copy of each string.

For example, imagine we want to deserialize the following input :

{
  "list": [
    {"temperature":21.2,"humidity":68.9,"weather":"overcast clouds"},
    {"temperature":19.7,"humidity":62.1,"weather":"clear sky"},
    {"temperature":18.6,"humidity":59.8,"weather":"clear sky"}
   ] 
}

Let’s write a program to see the memory consumption:

StaticJsonDocument<512> doc;
deserialize(doc, input);
Serial.println(doc.memoryUsage());

If you run this program on ArduinoJson 6.15 and 6.16, you’d get these results:

To deduplicate strings, ArduinoJson has to scan the list of existing strings, which slightly slows down the process:

The scan duration increases with the number of unique strings in the JsonDocument, so your mileage may vary. If you want to see the result on your board, here is the program I used. You can disable this feature by setting ARDUINOJSON_ENABLE_STRING_DEDUPLICATION to 0.

Currently, the ArduinoJson Assistant doesn’t know about this feature. I plan to keep it that way until the majority of users had the opportunity to install the new version.

This awesome feature is largely due to the contribution of Ewald Comhaire. Thanks, Ewald!

Comparison operators

In previous versions, you could compare a JsonVariant with an integer, a double, or a string, but you could not compare a JsonVariant with another JsonVariant. This is possible with ArduinoJson 6.16, for example you can now write:

if (request["power"] <= config["maxPower"])

The result of the comparison depends on the type of both operands, so it requires a bit more work than just comparing two numbers. If you worry about speed or code size, prefer casting explicitly like so:

if ((double)request["power"] <= (double)config["maxPower"])

New default for ARDUINOJSON_DECODE_UNICODE

Introduced in ArduinoJson 6.9, ARDUINOJSON_DECODE_UNICODE allows you to enable the decoding of Unicode escape sequence in deserializeJson().

When set to 1, deserializeJson() converts the Unicode escape sequences to UTF-8 characters; for example, \u00EE becomes î.

When set to 0, deserializeJson() returns NotSupported if the input contains a Unicode escape sequence.

What changed in ArduinoJson 6.16 is the default value: it used to be 0, and now it’s 1. I managed to reduce the size of the parser (again), that’s why I thought it was acceptable to enable it by default. In the end, the size of the parser on AVR grew only by a hundred bytes and is still smaller than version 6.14.

Improved copyArray()

copyArray() is a utility function that allows you to copy values between a C array and a JsonArray, and it works in both ways. For example:

int[] values = [1,2,3];
copyArray(values, doc);
serializeJson(doc, Serial); // prints [1,2,3].

copyArray() already existed; what changed is that you can now use it with an ElementProxy (the class returned by doc[0]) and MemberProxy (the class returned by doc["key"]). This feature allows you to do stuffs like this:

int[] values = [1,2,3];
copyArray(values, doc["config"]["values"]);
serializeJson(doc, Serial);  // prints {"config":{"values":[1,2,3]}}

Reduced stack usage

Several users reported weird issues when building in debug mode on ESP8266 and ESP32. It took us some time to realize that this problem was due to an excessive stack usage in the recursive part of deserializeJson().

The problem only appeared with the optimization flag -Og. This flag produced a stack usage 5 times bigger than -Os.

After a long investigation, I discovered that the -Og behaves like this when you return a struct by value. Indeed, I tend to use this pattern a lot; for example, the implementation of deserializeJson() contains many functions that return a DeserializationError.

By refactoring the code, I managed to remove about a hundred return-struct-by-value, and the stack usage in debug mode is now back to normal. While working on that, I found several optimizations that allowed me to reduce the stack usage in release mode as well.

This fix wouldn’t have been possible without the help of Adam Iredale. Thanks, Adam!

Conclusion

As usual, I encourage you to download the new revision and to report any problem to the GitHub issue board.

Remember that you can support the development by purchasing my book Mastering ArduinoJson. Not only you’ll encourage the development of more releases like this one, but you’ll also learn how to get the best out of the library.

Have a beautiful summer!