ArduinoJson 7.3: safer copy policies

I just released ArduinoJson 7.3, which introduces a few changes in how strings are copied. While technically a breaking, it should be transparent for most people. This release also prevents copying the internal proxy objects, which might affect you if you use the auto keyword in your programs.

New string copy policy

The string copy policy refers to how ArduinoJson stores strings in the JsonDocument. Most strings are stored by copy, meaning the characters are copied into the JsonDocument; other strings are stored by pointer, meaning the JsonDocument only saves the address of the character array.

Storing a pointer is more efficient but unsafe if not used cautiously. Indeed, this pointer may become invalid if the string is destroyed, leading to an undefined behavior. The device would usually crash or spit random characters, but it could also accidentally work.

Since ArduinoJson 5.13, released 7 years ago, the policy has been the following: all strings are stored by copy, except const char*, which is stored by address. The motivation was to ensure that string literals would not be copied wastefully. Indeed, literals reside at fixed memory locations and, therefore, are safe to store by address.

Unfortunately, there are many situations where you get a const char* that doesn’t point to a string literal. For example, if a third-party function returns a const char*, it is usually unsafe to store it by address. This was a major pitfall of ArduinoJson, and many of you fell into it.

With ArduinoJson 7.3, I changed the string copy policy so that const char* is by copy, too. Thanks to template metaprogramming witchcraft, ArduinoJson still stores string literals by copy. Instead of using the type const char*, it now detects literals with the type const char(&)[N], which you are unlikely to use for temporary strings.

This new policy might require some changes in your code. For example, with previous versions of ArduinoJson, when you wanted to bypass the copy policy to force storing a pointer, you would usually cast the pointer to const char*. This trick doesn’t work anymore.

Now, if you want to store a pointer to a string that is not a literal, you must wrap it with a JsonString and pass true as the last argument to tell that the string is safe to store by pointer

  char buffer[256];
  sprintf(buffer, "value-%d", i);
- doc["key"] = const_cast<const char*>(buffer);
+ doc["key"] = JsonString(buffer, true);

I’m sure you’ll be happy to get rid of these yucky const_casts 😉

Non-copyable proxies

When you access a member (or an element) of a JsonDocument, ArduinoJson doesn’t return the value right away; instead, it returns a proxy object that allows reading or writing the value. Indeed, the C++ language doesn’t provide a way to tell if the member is accessed for reading or for writing, so we must return an object that will handle the read or write operation when it is known.

Most of the time, this mechanism is transparent but can cause problems if the client code stores a proxy object in a variable. For example, the following code is problematic because it stores a proxy object that points to a temporary string:

auto val = doc[String("value-") + i]; // "value-1", "value-2", ...
Serial.println(val.as<const char*>());

When you look at this code, it’s hard to tell that it is incorrect because the auto keyword hides the type of the proxy object. The problem is that this object stores a pointer to a temporary string that is destroyed before the second line executes. Even if it’s incorrect, this code will work most of the time because the characters are still in memory, but the slightest change might break this precarious code.

To prevent these errors, I made the proxy classes non-copyable so that the code above doesn’t compile with ArduinoJson 7.3. If you are affected by this breaking change, you must either call as<T>() or to<T>(), depending on the situation.

For example, if you are extracting values from a JSON document, you should update your code like so:

- auto config = doc["config"];
+ auto config = doc["config"].as<JsonObject>();
  const char* name = config["name"];

Conversely, if you are building a JSON document, you should update your code like so:

- auto config = doc["config"];
+ auto config = doc["config"].to<JsonObject>();
  config["name"] = "ArduinoJson";

This change should prevent many issues but is not a silver bullet. For example, it can be bypassed using const auto& instead of auto.

SFINAE in return types

SFINAE is a template metaprogramming technique that allows library authors to exclude certain function overloads for specific types. For example, ArduinoJson uses SFINAEs to provide a different behavior when you try to insert an integer or a string into a JsonDocument.

In previous versions of ArduinoJson, the SFINAEs were placed in the return type, which polluted the Intellisense tooltips in the Arduino IDE. In ArduinoJson 7.3, I moved all the public-facing SFINAEs to the template declarations so they don’t appear in the IDE’s tooltips.

Code size

Before wrapping up, let’s have a look at the code size. The charts below show the size of the compiled examples on Arduino UNO R3 and R4.

You can notice a slight increase due to the new string copy policy. To avoid having a template specialization for each string length, I grouped all RAM string adapters into a single class, losing the original optimization for zero-terminated strings.

The increase is visible in the programs that only use string literals and no other string type. As soon as the program uses multiple string types, the code size is smaller than the previous version, as you can see with StringExample.

Final words

I hope you’ll be happy with this new release.
Please let me know if something goes wrong after the upgrade.

Remember that you can support my work by sponsoring me on GitHub or purchasing my ebook.

For the next release, I plan on working on memory optimization.

Best wishes for 2025 🎉