`serializeJson()` continues appending to `String` when allocation fails

Created by: softplus

I'm using deserializeJson / serializeJson to receive, store, send data and noticed weird behavior for "longish" inputs. My guess is something is being incorrectly buffered and overwriting itself, leading to invalid serialized JSON output.

This is on Arduino 1.8.13, using ArduinoJson 6.18.0, compiling for ATmega32U4 on Mac OSX 11.3.1

Code:

#include <ArduinoJson.h>

void setup() {
  Serial.begin(115200); delay(2000);

  DynamicJsonDocument doc(1000);

  //String s = "{\"cmd\":\"set\",\"ver\":2,\"key\":[{\"p\":2,\"t\":1,\"k\":[\"textsingleoh boy\",\"textdouble\"]},{\"p\":3,\"t\":2,\"k\":[\"2extsingle\",\"2extdouble\"]}],\"led\":[{\"p\":4,\"t\":1,\"c\":{\"p\":1,\"v\":2,\"i\":255,\"d\":10},\"s\":[2]}]}"; // fails
  String s = "{\"cmd\":\"set\",\"ver\":2,\"key\":[{\"p\":2,\"t\":1,\"k\":[\"textsingleoh boy\",\"textdouble\"]},{\"p\":3,\"t\":2,\"k\":[\"2extsingle\",\"2extdouble\"]}],\"led\":[{\"p\":4,\"t\":1,\"c\":{\"p\":1,\"v\":2,\"i\":255,\"d\":10},\"s\":20}]}"; // fails
  //String s = "{\"cmd\":\"set\",\"ver\":2,\"key\":[{\"p\":2,\"t\":1,\"k\":[\"textsingleoh boy\",\"textdouble\"]},{\"p\":3,\"t\":2,\"k\":[\"2extsingle\",\"2extdouble\"]}],\"led\":[{\"p\":4,\"t\":1,\"c\":{\"p\":1,\"v\":2,\"i\":255,\"d\":10},\"s\":2}]}"; // ok
  //String s = "{\"cmd\":\"set\",\"ver\":2,\"key\":[{\"p\":2,\"t\":1,\"k\":[\"textsingleoh boy\",\"textdouble\"]},{\"p\":3,\"t\":2,\"k\":[\"2extsingle\",\"2extdouble\"]}],\"led\":[{\"p\":4,\"t\":1,\"c\":{\"p\":1,\"v\":2,\"i\":255,\"d\":10}]}"; //ok
  //String s = "{\"cmd\":\"set\",\"ver\":2,\"key\":[{\"p\":2,\"t\":1,\"k\":[\"textsingleoh boy\",\"textdouble\"]},{\"p\":3,\"t\":2,\"k\":[\"2extsingle\",\"2extdouble\"]}]}"; // ok

  Serial.println("Deserializing: ");
  Serial.print("input ="); Serial.println(s);
  Serial.print("len="); Serial.println(s.length()); /* Serial.println(strlen(s)); */ 
  DeserializationError error = deserializeJson(doc, s);
  Serial.print("error="); Serial.println(error.c_str());
  int input_length = s.length();

  // Serialize again
  String result;
  Serial.println("Serializing: ");
  int bytes = serializeJson(doc, result);
  Serial.print("bytes="); Serial.println(bytes);
  Serial.print("result="); Serial.println(result);
  Serial.print("len="); Serial.println(result.length());

  int output_length = result.length();
  if (input_length == output_length) { Serial.println("PASS"); } else { Serial.println("FAIL"); }
}

void loop() { }

Uncommenting / commenting a line with "fails" fails, one with "ok" works. Longer ones all fail. The difference seems to be in string length, since changing an int parameter from "2" to "20" triggers the same behavior (data size should be the same). serializeJson returns the correct number of bytes, but the output string is shorter, and sometimes invalid JSON.

Sample output:

10:05:48.591 -> Deserializing: 
10:05:48.591 -> input ={"cmd":"set","ver":2,"key":[{"p":2,"t":1,"k":["textsingleoh boy","textdouble"]},{"p":3,"t":2,"k":["2extsingle","2extdouble"]}],"led":[{"p":4,"t":1,"c":{"p":1,"v":2,"i":255,"d":10},"s":20}]}
10:05:48.591 -> len=189
10:05:48.591 -> error=Ok
10:05:48.591 -> Serializing: 
10:05:48.591 -> bytes=189
10:05:48.591 -> result={"cmd":"set","ver":2,"key":[{"p":2,"t":1,"k":["textsingleoh boy","textdouble"]},{"p":3,"t":2,"k":["2extsingle","2extdouble"]}],"led":[{"p":4,"t":1,"c":{"p"}]}
10:05:48.591 -> len=158
10:05:48.591 -> FAIL
...
10:06:23.389 -> Deserializing: 
10:06:23.389 -> input ={"cmd":"set","ver":2,"key":[{"p":2,"t":1,"k":["textsingleoh boy","textdouble"]},{"p":3,"t":2,"k":["2extsingle","2extdouble"]}],"led":[{"p":4,"t":1,"c":{"p":1,"v":2,"i":255,"d":10},"s":2}]}
10:06:23.389 -> len=188
10:06:23.389 -> error=Ok
10:06:23.389 -> Serializing: 
10:06:23.389 -> bytes=188
10:06:23.389 -> result={"cmd":"set","ver":2,"key":[{"p":2,"t":1,"k":["textsingleoh boy","textdouble"]},{"p":3,"t":2,"k":["2extsingle","2extdouble"]}],"led":[{"p":4,"t":1,"c":{"p":1,"v":2,"i":255,"d":10},"s":2}]}
10:06:23.389 -> len=188
10:06:23.389 -> PASS