ArduinoJson 6.14.0: a Service Pack

I just published ArduinoJson 6.14.0.
No big feature this time, but many nice improvements, and unfortunately, some breaking changes, so be sure to read on.

New function: `shrinkToFit()`

We all know that DynamicJsonDocument has a fixed-size memory pool and that it’s impossible to change the capacity after we created it. Well, there is now one exception. When you call DynamicJsonDocument::shrinkToFit(), the DynamicJsonDocument reduces the capacity of its memory pool to match the current usage.

This feature was added as an answer to a particular use case: when you want to deserialize with the maximum amount of memory and then return a “compact” JsonDocument. See the following function designed for ESP8266 and ESP32:

DynamicJsonDocument parseJsonInput(Stream& input) {
  const size_t capacity = ESP.getMaxFreeBlockSize();  // ESP8266
  const size_t capacity = ESP.getMaxAllocHeap();  // ESP32

  DynamicJsonDocument doc(capacity);
  deserializeJson(doc, input);
  doc.shrinkToFit();

  return doc;
}

As you see, this function calls ESP.getMaxFreeBlockSize() (on ESP8266) or ESP.getMaxAllocHeap() (ESP32) to get the size of the largest free block of memory. The result may be significantly lower than the total available memory (ESP.getFreeHeap()) if the heap is fragmented. It passes the result to the constructor of DynamicJsonDocument, thereby creating a gigantic memory pool (potentially 500KB on ESP32).

It deserializes by calling deserializeJson() as usual. Then, before returning the DynamicJsonDocument, it calls shrinkToFit() to release all unused bytes from the memory pool, freeing a lot of RAM. This pattern is very handy if you want a generic parsing function or if you want to store a DynamicJsonDocument as a member of an object.

Advantages

The DynamicJsonDocument returned by parseJsonInput() is compact.
You can save it as a member of an object, for example, and not worry about the wasted RAM.

Drawbacks

The DynamicJsonDocument returned by parseJsonInput() is full, so you cannot add any value in it.
ArduinoJson 6.14 doesn’t offer a reverse operation to grow the memory pool.
The program doesn’t use constant-size allocation anymore, so it’s subject to heap fragmentation.

Please note that if you use a custom allocator, you need to implement a new function: reallocate().

By the way, the name of the function was inspired by std::string::shrink_to_fit() and std::vector::shrink_to_fit(), which were added in C++11.

Support for comments is now optional

ArduinoJson has supported comments in JSON documents since version 5.0.0: deserializeJson() simply ignores all comments in the input. While this feature is still available in version 6.14.0, it’s now an opt-in feature.

The JSON specification doesn’t support comments, and only a minority of users need this feature. That’s why I decided to disable it by default, thereby saving a bunch of bytes in the Flash memory of most users.

The default behavior changed, so existing code may break. If so, switch comments back on.

To enable support for comments in ArduinoJson, you must define ARDUINOJSON_ENABLE_COMMENTS to 1 before including ArduinoJson.h:

#define ARDUINOJSON_ENABLE_COMMENTS 1
#include <ArduinoJson.h>

As with most compile-time settings, make sure you use the same value on all compilation units (all .cpp and .ino files); otherwise, your program will embed two versions of the library.

Better UTF-16 to UTF-8 conversion

Since version 6.9.0, deserializeJson() can translate Unicode escape sequences to UTF-8 characters. Here too, it’s an opt-in feature: you must set ARDUINOJSON_DECODE_UNICODE to 1 to enable it.

However, the conversion was not perfect and created a degenerated variant of UTF-8 known as CESU-8. When a character required two UTF-16 escape sequences (like \ud83d\udda4 for the character 🖤), it was converted to two UTF-8 characters instead of one.

Thanks to the contribution of Kay Sievers, ArduinoJson 6.14.0 produces standard UTF-8. The code is slightly bigger than before, but I did my best to reduce the overhead.

Detect `std::string` and `std::stream`

ArduinoJson supports Arduino’s String and the standard std::string. Similarly, it supports Stream and std::stream. However, to use the standard types on Arduino, you had to enable support with ARDUINOJSON_ENABLE_STD_STRING and ARDUINOJSON_ENABLE_STD_STREAM.

Previously, I could not detect if those types were available, so I disabled them by default to prevent unwanted compilation errors. Meanwhile, C++17 brought a new preprocessor function: __has_include(). This function allows detecting the presence of specific headers like <string> or <stream>, so ArduinoJson can now reliably tell if std::string and std::stream are available.

Therefore, if you have a reasonably recent compiler, you don’t have to worry about ARDUINOJSON_ENABLE_STD_STRING and ARDUINOJSON_ENABLE_STD_STREAM; you can safely remove them from your programs and Makefiles.

Faster serialization

When I benchmarked ArduinoJson against Arduino_JSON, I discovered that ArduinoJson was roughly 10% faster on all use cases except one: when you pass a String to serializeJson().

The problem came from the String class, which is horribly slow when adding characters one by one. Here is an excerpt from the AVR core:

unsigned char String::concat(char c)
{
    char buf[2];
    buf[0] = c;
    buf[1] = 0;
    return concat(buf, 1);
}

unsigned char String::concat(const char *cstr, unsigned int length)
{
    unsigned int newlen = len + length;
    if (!cstr) return 0;
    if (length == 0) return 1;
    if (!reserve(newlen)) return 0;
    strcpy(buffer + len, cstr);
    len = newlen;
    return 1;
}

So, instead of simply pushing one character at the end of the buffer (virtually one instruction), it creates string that contains one character and appends it to the current string (probably a hundred instructions).

Moreover, the reserve() function is not greedy enough. See the code below; it just allocates the given number of bytes, not less, not more:

unsigned char String::reserve(unsigned int size)
{
    if (buffer && capacity >= size) return 1;
    if (changeBuffer(size)) {
        if (len == 0) buffer[0] = 0;
        return 1;
    }
    return 0;
}

unsigned char String::changeBuffer(unsigned int maxStrLen)
{
    char *newbuffer = (char *)realloc(buffer, maxStrLen + 1);
    if (newbuffer) {
        buffer = newbuffer;
        capacity = maxStrLen;
        return 1;
    }
    return 0;
}

All std::string implementations use some kind of growth strategy to reduce the number of allocations (for example, multiply current capacity by 2), but String does not. This means that, unless you call reserve() yourself, String calls realloc() every time you append one character!!!

Fortunately, the solution was simple. To increase the speed of ArduinoJson, I simply added a temporary buffer. Now serializeJson() appends to String in chunks of 31 characters.

In the end, ArduinoJson 6.14.0 is faster than Arduino_JSON on all use cases.

Conclusion

That’s all for this time. I hope I didn’t upset you with yet another rant about String. Unfortunately, this class is the source of many troubles in Arduino projects. To me, it is the worse piece of code in the Arduino core. Some, like the ESP8266 core, have improved their String class, but most cores simply copied the code from the original AVR core, and with it, they duplicated all the flows.

OK, enough for now. No more ranting about this class… until next time.

Well, I hope you’ll enjoy this new version of ArduinoJson, and I’m sorry for the breaking changes. As usual, if you need any assistance, please open an issue on GitHub.

Stay informed!

...or subscribe to the RSS feed

New function: shrinkToFit()