Unescaped control character in JSON string in 2.3.9

[cmock]

Describe the bug
2.3.9 produces invalid JSON for the "power" MQTT topic:

00000000  7b 22 6c 76 22 3a 22 22  2c 22 69 64 22 3a 22 53  |{"lv":"","id":"S|
00000010  41 47 59 05 e8 94 b6 22  2c 22 74 79 70 65 22 3a  |AGY....","type":|

Specifically, the 0x05 at offset 0x13. In previous versions, the "id" key was empty.

To Reproduce
Steps to reproduce the behavior:

1. Parse the MQTT JSON
2. See parser throw an exception

Expected behavior
I'm not a JSON expert, but "jq" says "control characters from U+0000 through U+001F must be escaped".

Screenshots
If applicable, add screenshots to help explain your problem.

Hardware information:

• Country: Austria
• Meter: Sagemcom T210-D
• Encryption enabled yes
• AMS reader: ESP32
• M-bus adapter (if applicable): resistive divider

Relevant firmware information:

• Version: 2.3.9
• MQTT: yes
• MQTT payload type: HomeAssistant
• HAN GPIO: [e.g. GPIO16
• HAN baud and parity: 2400 8E1
• Temperature sensors: n/a
• ENTSO-E API enabled: no

Additional context
Here's the complete JSON string from the power topic attached so nothing gets lost in markdown:
ams.json

[ArnieO]

Indeed, your JSON string gives a parse error due to that character.
Using https://jsonbeautify.com/, I get this:

It could look like your meter is providing unexpected input to the fields "id" and "type".

On my Kamstrup meter, I receive:

• id: Meter serial number
• type: Meter model number (in my case "6841131BN243101040")

In order to see what your meter is sending, please run a Telnet debug and post the output here for decoding / verification.

[cmock]

Do I need to do anything special to enable configuration of debug mode? I can't see it in my config page... (using 2.3.9)

[ArnieO]

Do I need to do anything special to enable configuration of debug mode? I can't see it in my config page... (using 2.3.9)

You do not have this at the bottom of the Config screen?

[Noschvie]

Seems to be an ESP32-C3
#783 (comment)

[ArnieO]

Seems to be an ESP32-C3

In that case: Try to run debug with an older firmware, I think v2.3.4 might work.
We have seen issues with binaries on C3, and since we are not using them in our products, finding the issue does not have high priority

[Noschvie]

Version 2.2.28 is working for me.
#835 (comment)

[cmock]

No, it's an "original" ESP32, Info page says "Chip: esp32 (240MHz)". The last tile on the config screen is "Hardware"
So, back to 2.2.28, and we get

(V) MBUS frame:
(V) 68 FA FA 68  53 FF 00 01  67 DB 08 53  41 47 59 05 
(V) E8 94 B6 81  F8 20 00 EA  75 E7 5C A4  1D 8F 8B 75 
(V) 4F 47 29 91  EC 08 B7 43  DA C8 A8 72  1D 48 13 D3 
(V) BE E1 2C 7C  5A BF 0C 71  BC 54 8C 31  4E 0F 6D B4 
(V) CD 94 52 CB  7B AB 7F B4  34 6C 55 AF  60 1F 0E 9A 
(V) CB EA B4 95  8B FD B9 2F  2D 1D 4F 88  54 EF F1 91 
(V) C6 24 48 F2  49 90 F5 1F  CF F5 9E 20  BB B9 63 86 
(V) 64 D5 C2 4D  A5 5E 67 F4  B1 95 5B 79  E2 9C 14 48 
(V) DD E7 8F 3B  77 7F 6E 5A  C7 B0 12 F7  34 1D C9 49 
(V) 7A 5D BD F7  2D 39 C7 6A  FE 76 EF D5  0C 45 39 3E 
(V) 07 82 23 33  05 EB 3E FD  1E 49 B3 9B  E9 0B AC 72 
(V) 6A B9 29 0B  DC 4B CA ED  30 F8 68 89  39 52 B9 2A 
(V) 7B 38 54 F1  EA A1 BD E2  50 DE F1 3C  A8 78 3E 09 
(V) 41 4C EA 99  4E 35 70 8B  F7 A3 1A 98  78 53 AA C0 
(V) 7B 35 84 E8  C3 24 7F 62  53 8E 7A 82  89 01 33 6A 
(V) D1 83 11 8D  D4 7C AB C3  98 DE B3 AF  81 B1 F3 16 
(V) 
(V) MBUS frame:
(V) 68 14 14 68  53 FF 11 01  67 33 53 DB  55 B1 C5 6D 
(V) B8 F5 B9 61  99 18 7E 72  CC 16 
(V) GCM frame:
(V) DB 08 53 41  47 59 05 E8  94 B6 81 F8  20 00 EA 75 
(V) E7 0F 80 70  DD 27 0C 07  E8 0B 0F 05  0E 2D 2D 00 
(V) FF C4 02 02  23 09 0C 07  E8 0B 0F 05  0E 2D 2D 00 
(V) FF C4 02 09  06 01 00 01  08 00 FF 06  00 48 82 9A 
(V) 02 02 0F 00  16 1E 09 06  01 00 02 08  00 FF 06 00 
(V) 4E B1 99 02  02 0F 00 16  1E 09 06 01  00 01 07 00 
(V) FF 06 00 00  00 00 02 02  0F 00 16 1B  09 06 01 00 
(V) 02 07 00 FF  06 00 00 00  09 02 02 0F  00 16 1B 09 
(V) 06 01 00 20  07 00 FF 12  09 1E 02 02  0F FF 16 23 
(V) 09 06 01 00  34 07 00 FF  12 09 15 02  02 0F FF 16 
(V) 23 09 06 01  00 48 07 00  FF 12 09 11  02 02 0F FF 
(V) 16 23 09 06  01 00 1F 07  00 FF 12 00  02 02 02 0F 
(V) FE 16 21 09  06 01 00 33  07 00 FF 12  00 30 02 02 
(V) 0F FE 16 21  09 06 01 00  47 07 00 FF  12 00 53 02 
(V) 02 0F FE 16  21 09 06 01  00 0D 07 00  FF 12 00 5E 
(V) 02 02 0F FD  16 FF 09 0C  31 37 38 32  31 30 31 32 
(V) 38 35 30 32  
(V) DLMS frame:
(V) 0F 80 70 DD  27 0C 07 E8  0B 0F 05 0E  2D 2D 00 FF 
(V) C4 02 02 23  09 0C 07 E8  0B 0F 05 0E  2D 2D 00 FF 
(V) C4 02 09 06  01 00 01 08  00 FF 06 00  48 82 9A 02 
(V) 02 0F 00 16  1E 09 06 01  00 02 08 00  FF 06 00 4E 
(V) B1 99 02 02  0F 00 16 1E  09 06 01 00  01 07 00 FF 
(V) 06 00 00 00  00 02 02 0F  00 16 1B 09  06 01 00 02 
(V) 07 00 FF 06  00 00 00 09  02 02 0F 00  16 1B 09 06 
(V) 01 00 20 07  00 FF 12 09  1E 02 02 0F  FF 16 23 09 
(V) 06 01 00 34  07 00 FF 12  09 15 02 02  0F FF 16 23 
(V) 09 06 01 00  48 07 00 FF  12 09 11 02  02 0F FF 16 
(V) 23 09 06 01  00 1F 07 00  FF 12 00 02  02 02 0F FE 
(V) 16 21 09 06  01 00 33 07  00 FF 12 00  30 02 02 0F 
(V) FE 16 21 09  06 01 00 47  07 00 FF 12  00 53 02 02 
(V) 0F FE 16 21  09 06 01 00  0D 07 00 FF  12 00 5E 02 
(V) 02 0F FD 16  FF 09 0C 31  37 38 32 31  30 31 32 38 
(V) 35 30 32 
(D) Received valid DLMS at 35
(V) Using application data:
(V) 02 23 09 0C  07 E8 0B 0F  05 0E 2D 2D  00 FF C4 02 
(V) 09 06 01 00  01 08 00 FF  06 00 48 82  9A 02 02 0F 
(V) 00 16 1E 09  06 01 00 02  08 00 FF 06  00 4E B1 99 
(V) 02 02 0F 00  16 1E 09 06  01 00 01 07  00 FF 06 00 
(V) 00 00 00 02  02 0F 00 16  1B 09 06 01  00 02 07 00 
(V) FF 06 00 00  00 09 02 02  0F 00 16 1B  09 06 01 00 
(V) 20 07 00 FF  12 09 1E 02  02 0F FF 16  23 09 06 01 
(V) 00 34 07 00  FF 12 09 15  02 02 0F FF  16 23 09 06 
(V) 01 00 48 07  00 FF 12 09  11 02 02 0F  FF 16 23 09 
(V) 06 01 00 1F  07 00 FF 12  00 02 02 02  0F FE 16 21 
(V) 09 06 01 00  33 07 00 FF  12 00 30 02  02 0F FE 16 
(V) 21 09 06 01  00 47 07 00  FF 12 00 53  02 02 0F FE 
(V) 16 21 09 06  01 00 0D 07  00 FF 12 00  5E 02 02 0F 
(V) FD 16 FF 09  0C 31 37 38  32 31 30 31  32 38 35 30 
(V) 32 
(V) DLMS
(V) (AmsDataStorage) Day data timestamp age 1731678347 - 0 > 3600
(D) Its time to update data storage
(V) (EnergyAccounting) Adding 0.0000 kWh export
(V) (EnergyAccounting)  calculating threshold, currently at 2
(V) (EnergyAccounting)  new threshold 2
(D) Serving /data.json over http...
(D) Serving /data.json over http...

[ArnieO]

Using this tool to decode the DLMS frame gives:

<DataNotification>
  <LongInvokeIdAndPriority Value="8070DD27" />
  <!--2024-11-15 14:45:45-->
  <DateTime Value="07E80B0F050E2D2D00FFC402" />
  <NotificationBody>
    <DataValue>
      <Structure Qty="23" >
        <!--2024-11-15 14:45:45-->
        <OctetString Value="07E80B0F050E2D2D00FFC402" />
        <!--1.0.1.8.0.255-->
        <OctetString Value="0100010800FF" />
        <UInt32 Value="0048829A" />
        <Structure Qty="02" >
          <Int8 Value="00" />
          <Enum Value="1E" />
        </Structure>
        <!--1.0.2.8.0.255-->
        <OctetString Value="0100020800FF" />
        <UInt32 Value="004EB199" />
        <Structure Qty="02" >
          <Int8 Value="00" />
          <Enum Value="1E" />
        </Structure>
        <!--1.0.1.7.0.255-->
        <OctetString Value="0100010700FF" />
        <UInt32 Value="00000000" />
        <Structure Qty="02" >
          <Int8 Value="00" />
          <Enum Value="1B" />
        </Structure>
        <!--1.0.2.7.0.255-->
        <OctetString Value="0100020700FF" />
        <UInt32 Value="00000009" />
        <Structure Qty="02" >
          <Int8 Value="00" />
          <Enum Value="1B" />
        </Structure>
        <!--1.0.32.7.0.255-->
        <OctetString Value="0100200700FF" />
        <UInt16 Value="091E" />
        <Structure Qty="02" >
          <Int8 Value="FF" />
          <Enum Value="23" />
        </Structure>
        <!--1.0.52.7.0.255-->
        <OctetString Value="0100340700FF" />
        <UInt16 Value="0915" />
        <Structure Qty="02" >
          <Int8 Value="FF" />
          <Enum Value="23" />
        </Structure>
        <!--1.0.72.7.0.255-->
        <OctetString Value="0100480700FF" />
        <UInt16 Value="0911" />
        <Structure Qty="02" >
          <Int8 Value="FF" />
          <Enum Value="23" />
        </Structure>
        <!--1.0.31.7.0.255-->
        <OctetString Value="01001F0700FF" />
        <UInt16 Value="0002" />
        <Structure Qty="02" >
          <Int8 Value="FE" />
          <Enum Value="21" />
        </Structure>
        <!--1.0.51.7.0.255-->
        <OctetString Value="0100330700FF" />
        <UInt16 Value="0030" />
        <Structure Qty="02" >
          <Int8 Value="FE" />
          <Enum Value="21" />
        </Structure>
        <!--1.0.71.7.0.255-->
        <OctetString Value="0100470700FF" />
        <UInt16 Value="0053" />
        <Structure Qty="02" >
          <Int8 Value="FE" />
          <Enum Value="21" />
        </Structure>
        <!--1.0.13.7.0.255-->
        <OctetString Value="01000D0700FF" />
        <UInt16 Value="005E" />
        <Structure Qty="02" >
          <Int8 Value="FD" />
          <Enum Value="FF" />
        </Structure>
        <!--178210128502-->
        <OctetString Value="313738323130313238353032" />
      </Structure>
    </DataValue>
  </NotificationBody>
</DataNotification>

We find these fields:

OBIS code        Description
1.8.0	 	Positive active energy, total
2.8.0		Negative active energy, total
1.7.0		Positive active instantaneous power
2.7.0		Negative active instantaneous power
32.7.0		Instantaneous voltage in phase I
52.7.0		Instantaneous voltage in phase II
72.7.0		Instantaneous voltage in phase III
31.7.0		Instantaneous current in phase I
51.7.0		Instantaneous current in phase II
71.7.0		Instantaneous current in phase III
13.7.0		Instantaneous power factor

I see no data from the meter that could be used in the "id:" or "type:" fields in the MQTT payload.
In which case the fields should be empty.

So this seems to be a bug that @gskjold will take care of when he finds time.

[cmock]

I notice the "SAGY..." string at the start (offset 2) of the GCM frame, whatever that may be...

[ArnieO]

I notice the "SAGY..." string at the start (offset 2) of the GCM frame, whatever that may be...

My guess is that it is from a random memory location due to a bug - but I'm just guessing.
We'll have to wait until @gskjold finds time to look into this.