ArduinoJson 7.2: smaller arrays

I just released a new revision of ArduinoJson which brings memory optimizations for arrays.

In most cases, arrays in version 7.2 consume twice less memory as in version 7.1.
In most cases, objects still consume as much memory as before, but in some cases, they might consume more.

Let’s see why in detail.

Object key-value pairs

The memory optimization in ArduinoJson 7.2 revolves around two significant changes, the first being the use of two slots for key-value pairs, instead of one.

As a reminder, ArduinoJson 7 stores every value (whether a number, an array, or an object) in a pool of fixed-size slots. This strategy reduces the number of heap allocations and significantly reduces fragmentation.

In previous versions of ArduinoJson, lone values and key-value pairs consumed one slot, and the memory reserved for the key remained unused for lone values. Storing key-value pairs in a single slot was an optimization for 8-bit microcontrollers

Using two slots enables memory optimizations for 32-bit microcontrollers at the detriment of 8-bit ones. In other words, ArduinoJson 7.2 consumes more memory than previous versions on 8-bit microcontrollers, as shown in the examples below.

64-bit numbers

The second change is about 64-bit integers and floating-point values.

In previous versions of ArduinoJson, a number would always consume the same amount of memory, regardless of its type. This meant a short integer used as much RAM as a long long. This changed in ArduinoJson 7.2: 64-bit integers now require two slots, and smaller integers, only one.

Floating-point numbers are treated the same way: single-precision floating-point numbers use one slot, whereas double-precision floating-point numbers use two. This required a slight change in deserializeJson() which now has to decide whether a floating-point number needs single or double precision. Currently, it uses the number of decimal places as a criterion, and only numbers with more than six digits use doubles.

Slot size

Getting the keys and 64-bit numbers out of the slot allowed cutting its size in half on 32-bit microcontrollers:

As you can see, the picture is less pretty on 8-bit and 64-bit processors. Luckily, very few projects still use ArduinoJson on 8-bit microcontrollers, and no microcontroller currently uses a 64-bit architecture.

Remember that objects now need twice as many slots as before, so they should consume the same amount of memory (on 32-bit MCUs), except if they contain 64-bit numbers.

Examples

In the following examples, I’ll compare the memory consumption in ArduinoJson 7.1 (and all previous versions) and 7.2. I’ll only count the memory consumed by the array or object itself, not the other overheads, so the results in the ArduinoJson Assistant might differ slightly.

As a reminder, support for 64-bit integers is disabled by default on 8-bit microcontrollers, so you would have to set ARDUINOJSON_USE_LONG_LONG to 1 to reproduce these examples. Also, note that 8-bit microcontrollers don’t support double-precision floating-point numbers.

Example 1: array of small integers

[1,2,3]

This array contains three elements and, therefore, consumes three slots.

In this ideal, yet realistic, case, the memory consumption is twice smaller on 32-bit and 25% smaller on 8-bit microcontrollers.

Example 2: array of large integers

[1000000000,2000000000,3000000000]

The array requires three slots, and each number consumes an extra slot, totaling six slots.

In this case, memory consumption is the same for 32-bit but increases by 30% for 8-bit microcontrollers.

Example 3: object with integers

{"a":1,"b":2,"c":3}

Each key-value pair requires two slots, so this object uses six slots.

Here, memory consumption is the same on 32-bit but increases by 50% on 8-bit microcontrollers.

Example 4: object with large integers

{"a":1000000000,"b":2000000000,"c":3000000000}

The object uses six slots plus one extra slot for each 64-bit integer, totaling nine slots.

In this worst-case scenario, memory consumption increases by 50% on 32-bit and doubles on 8-bit microcontrollers.

Removal of containsKey()

After being on the death row for years, the containsKey() method has finally been deprecated. You must now replace doc.containsKey("key") with doc["key"].is<T>(), which not only checks that the key exists but also that the value is of the expected type.

// Before
if (doc.containsKey("value")) {
  int value = doc["value"];
 // ...
}

// After
if (doc["value"].is<int>()) {
  int value = doc["value"];
 // ...
}

The motivation behind this change is to make an API that is easy to use correctly and hard to use incorrectly. Indeed, containsKey() was potentially harmful, as in the following example:

if (doc.containsKey["status"])
   strlcpy(currentStatus, doc["status"], 16);  // 💀

This program looks correct but is vulnerable: if an attacker sends a message like {"status":0}, the program crashes because doc["status"] would return nullptr. The vulnerability can easily be fixed with is<const char*>():

if (doc["status"].is<const char*>())
   strlcpy(currentStatus, doc["status"], 16);  // 🛡️

In addition, this syntax clearly shows the repeated lookup of the key, which should invite you to store the result in a variable:

JsonVariant status = doc["status"];  // only one lookup 👍
if (status.is<const char*>())
   strlcpy(currentStatus, status, 16);

If you want to check that a key exists regardless of its type, you can use is<JsonVariant>() or is<JsonVariantConst>() if the reference is read-only.

Code size

You can see a noticeable increase in the code size, but that’s the price we must pay to reduce memory consumption.

Conclusion

This new version brings significant memory savings in most cases but can use more RAM in some edge cases. 8-bit microcontrollers were sacrificed for the profit of 32-bit microcontrollers. As I said before, projects targetting 8-bit MCUs should stick to ArduinoJson 6, which was optimized for them.

Please let me know if you experience a significant increase in your memory consumption after upgrading to ArduinoJson 7.2.

In the next version, I plan to implement a short-string optimization, further reducing memory consumption.