How to use ArduinoJson with HTTPClient?

What is HTTPClient?

HTTPClient is a class that performs HTTP requests on ESP8266 and ESP32.

Assuming that the board is connected to the WiFi network, you can send an HTTP request like so:

WiFiClient client;  // or WiFiClientSecure for HTTPS
HTTPClient http;

// Send request
http.begin(client, "http://arduinojson.org/example.json");
http.GET();

// Print the response
Serial.print(http.getString());

// Disconnect
http.end();

While you can perform HTTP requests without HTTPClient (cf “See also” below), this class greatly simplifies your code.

On ESP8266, this class is provided by the ESP8266HTTPClient library, which is bundled with the ESP8266 core for Arduino.

On ESP32, it’s provided by the HTTPClient library, which is bundled with the Arduino core for the ESP32.

For HTTPS, you need to use WiFiClientSecure instead of WiFiClient.
You must also choose the certificate validation strategy. For example, you can call WiFiClientSecure::setInsecure() to skip the validation.

How to parse a JSON document from an HTTP response?

Unfortunately, HTTPClient doesn’t implement the Stream interface, so you cannot pass it directly to deserializeJson() like so:

deserializeJson(doc, http);  // error: 'class HTTPClient' has no member named 'read'; did you mean 'end'?

We could pass the result of http.getString() directly to deserializeJson(), but it would be quite inefficient because it would copy the complete response in RAM before parsing.

We can do much better by letting ArduinoJson pull the bytes from the HTTP response. To do that, we must get HTTPClient’s underlying Stream by calling http.getStream() instead of http.getString(). Unfortunately, when doing so, we bypass the code that handles chunked transfer encoding, making the stream unusable. Let’s see two solutions to this problem.

The simplest way to deal with chunked transfer encoding it to disabled by downgrading to HTTP version 1.0.

WiFiClient client;  // or WiFiClientSecure for HTTPS
HTTPClient http;

// Send request
http.useHTTP10(true);
http.begin(client, "http://arduinojson.org/example.json");
http.GET();

// Parse response
DynamicJsonDocument doc(2048);
deserializeJson(doc, http.getStream());

// Read values
Serial.println(doc["time"].as<long>());

// Disconnect
http.end();

Notice the call to http.useHTTP10(true) before sending the request.

Solution 2: use ChunkDecodingStream

If you cannot downgrade to HTTP 1.0, you can use the ChunkDecodingStream from the StreamUtils library. This class transform a chunked-encoded stream into a regular stream. Here is how you can use it with HTTPClient:

WiFiClient client;  // or WiFiClientSecure for HTTPS
HTTPClient http;

// Ask HTTPClient to collect the Transfer-Encoding header
// (by default, it discards all headers)
const char* keys[] = {"Transfer-Encoding"};
http.collectHeaders(keys, 1);

// Send request
http.begin(client, "http://arduinojson.org/example.json");
http.GET();

// Get the raw and the decoded stream
Stream& rawStream = http.getStream();
ChunkDecodingStream decodedStream(http.getStream());

// Choose the right stream depending on the Transfer-Encoding header
Stream& response =
    http.header("Transfer-Encoding") == "chunked" ? decodedStream : rawStream;

// Parse response
DynamicJsonDocument doc(2048);
deserializeJson(doc, response);

// Read values
Serial.println(doc["time"].as<long>());

// Disconnect
http.end();

Notice the call to http.collectHeaders() before sending the request.

How to send a JSON document in an HTTP request?

Unfortunately, HTTPClient doesn’t let us use the same trick as above, so we have to use a temporary buffer. Here is how we can do so with a String:

// Prepare JSON document
DynamicJsonDocument doc(2048);
doc["hello"] = "world";

// Serialize JSON document
String json;
serializeJson(doc, json);

WiFiClient client;  // or WiFiClientSecure for HTTPS
HTTPClient http;

// Send request
http.begin(client, "http://httpbin.org/post");
http.POST(json);

// Read response
Serial.print(http.getString());

// Disconnect
http.end();

If you run this program, you’ll see that it prints information on the HTTP requests. That’s the response returned by httpbin.org; it’s very handy to debug your programs.

You can significantly improve this code’s performance by calling String::reserve() before serializeJson().
Here are some other tips for using the String class efficiently.

If you want to send the payload as a stream, you need to use WiFiClient directly, as shown in the Serialization Tutorial (around page 127).

Why is the JsonDocument empty?

The most likely explanation is that the server returned a redirection code, like 301 Moved Permanently or 302 Found.

http.setFollowRedirects(HTTPC_FORCE_FOLLOW_REDIRECTS);

This feature was added quite recently on ESP32, so make sure you are up-to-date.

BONUS: How to see the document when I use a Stream?

As we saw, it’s more efficient to pass the Stream to deserializeJson() because it saves a large amount of memory. However, we cannot directly print the content of the Stream to see what was sent to ArduinoJson.

To see the content that is received from the HTTP response, we can use the ReadLoggingStream from the StreamUtils library.

Replace:

deserializeJson(doc, http.getStream());

with:

ReadLoggingStream loggingStream(http.getStream(), Serial);
deserializeJson(doc, loggingStream);

ReadLoggingStream will forward everything to deserializeJson() and will print the content to the serial port.

StreamUtils is a powerful library that deserves more attention. Please give it a star to spread the word.

See also

Global warming stripes by Professor Ed Hawkins (University of Reading)