Element names containing spaces and functional characters can not be reached

[SMCinc]

Again questions from a newbie user:
when I try to reach an element in a json message containing an functional character e.g. '-' in 'E-Total' with bracket notation:

{
 "fromTopics": ["solar/inverter/2000144729","solar/inverter/2000144729 "],
 "toTopic": "openWB/set/evu/WhExported",
 "emitType": "combineLatest",
 "template": {"$eval": "(messages[0].values['E-Total']+messages[1].values['E-Total'])*-0.9906542-80*1000"}
   },

in following object solar/inverter/2000144729

 {"sn":2000144729,"time":1667563080,"values":{"Upv-Ist":381,"Upv-Soll":382,"Iac-Ist":392,"Uac":230,"Fac":49.989998882636428,"Pac":90,"Zac":0.44800002127885818,"Riso":3000,"Ipv":307,"Aktives Team":1,"E-Total":41159.525954972487,"h-Total":68438.1999388286,"h-On":70897.2655337967,"Netz-Ein":24466,"Seriennummer":2000144729,"Status":"Mpp","Teamfunktion":0,"Fehler":"-------"}}}

the result is null

When I try $eval: values['E-Total'] in jsone-e playground it works:

When I use this bracket notation for normal element names e.g. 'Seriennummer' which is one of the last elements:

works fine.

-When I use .values['E-Total'] in a map e.g.

  {
     "fromTopic": "solar/inverter/1000123",
      "toTopic": "openWB/set/evu/Status",
      "emitType": "map",
      "wrapper":"test",
       "template": {"$eval": "test.values['E-Total']"}
    },

the result is null

So is there a special notation to be used or is that an feature which can be solved?

And a second question I have: is it possible to have LF CR in result message e.g.
I have created:

obislog "/ 1-0:1.8.1*255(96435.65119118353) 1-0:2.8.1*255(111062.242362) 1-0:21.7.0*255(148*W)!"

in my resultfile with using mosquitto_sub -t "obislog" -h localhost > file.txt every 10 secs in a ramdisk via crontab:

"/ 1-0:1.8.1*255(96435.65119118353) 1-0:2.8.1*255(111062.242362) 1-0:21.7.0*255(148*W)!"

I would need is:

"/<crlf>
1-0:1.8.1*255(96435.65119118353)<crlf>
1-0:2.8.1*255(111062.242362)<crlf>
1-0:21.7.0*255(148*W)<crlf>
!<crlf>
"

Is that possible with mqtt-transformer ?

Thanks in advance
Christian

[tg44]

hmmm interesting problems...

As I started to dig into the possible problems, I found out that I move all the --s lower to _-s in every input object key name. I can't really tell why we doing this, but at some point there was a reason. Can you confirm that the map below works?

{
     "fromTopic": "solar/inverter/1000123",
      "toTopic": "openWB/set/evu/Status",
      "emitType": "map",
      "wrapper":"test",
       "template": {"$eval": "test.values['E_Total']"}
    },

As the second part I have honestly no idea. In theory json-e can handle \n and \r. In theory it should work. I would not be really surprised if end to end it would broke at some point, but;

const jsone = require('json-e');
const inStr2 = {"values":{"E-Total": "--\n\n--"}}
const template2 = {"$eval": "values['E-Total']"}
console.log(jsone(template2, inStr2))

This code prints 1 empty line as it supposed to. I think if you add \r\n to the json and do the concatenations, it will probably work...

[SMCinc]

Okay, part one is solved: underscore '_' for minus '-' works fine. Also space works in object key name.
So I have connected my devices yasdi2mqtt via mqtt-transforme to openWB with the available high resolution with all necessary corrections. That was the last peace of this puzzle.

Regarding \r\n I have to do some additional research....
Meanwhile for my problem, a file containing OBIS formated text tob bring D0 data to Solarview,
mosquitto_sub -t "obislog" -h localhost >> /var/www/html/openWB/ramdisk/obis/obis.txt and
sed -i 's/\s\+/\n/g' /var/www/html/openWB/ramdisk/obis/obis.txt hopefully help - seems to work
Is it possible to define constants and use them for an amount of topis in conf.json file ?

Thanks a lot for the support.

[tg44]

Okay, part one is solved

Yayy!

Is it possible to define constants and use them for an amount of topis in conf.json file ?

Right now definitely not. But if you can came up with some kind of format or sth I can implement it. (At the beginning there was not topic as a variable input/output for example, but that makes a LOT of flexibility so I added in when the first user asked for it.)

[SMCinc]

I have some ideas, what would be nice to have:

• global constants e.g. for correction in my case for the energy counters: CorrAdd and CorrFac so corrections might be done on one place in conf - file and considered in many topics

• global variables to use an input in several topics. (at the moment variable are defined for every topic - could be the same vriablename for all . right?)

• Initialisation of topics with combineLatest: In my case the solar converters are enot sending topics when the sun is down. combineLatest waits for at least one of each input topic before calculation. At the moment at systemstart I send a mosquitto_pub message with initialisation values (here zeros). So it would be nice to be able to define a imitialsation for global variables and topics whitch are used in combineLatest topics which are not casted all the time.

• Absotutely great would be a time based interpolation: e.g. a energycounter topic is provided (CtrEgy [kWh] resolution 1 kWh) and a power topic (Pac [W]) and needed is the energy in a higher resolution (EgyHres [kWh] resolution 0.000 000 1 kWh (~3 Ws)).

CtrEgy, Pac are read and the time between the messages is content of variable time [s]
the constant Factor can be defined in the topic or obove in init values

If CtrEgy == CtrEgy n-1                                          //CtrEgy is unchanged since last received topic
then 
EgyHres = EgyHres n-1 + Pac*time*Factor            // Factor =  (1/(3600*1000)) // [Ws ->kWh]
  if EgyHres > (CtrEgy+1)
  then
  EgyHres = CtrEgy
  end if
else
EgyHres ==  CtrEgy
endif

What do you think?

[tg44]

I'm not sure I understand some of these...

global constants

Something like this?

        {
            "constant": "secToHour",
            "value": 3600,
        },
        {
            "fromTopic": "tele/sp-desk/STATE",
            "toTopic": "transformed/sp-desk-state",
            "emitType": "map",
            "template": {"uptimeHrs": {"$eval": "UptimeSec / constants.secToHour"}}
        },

Probably this is easy to implement, but than the constants will be a reserved toplevel key. Another way would be to add the useConstants as part of the definitions like;

        {
            "constant": "secToHour",
            "value": 3600,
        },
        {
            "fromTopic": "tele/sp-desk/STATE",
            "toTopic": "transformed/sp-desk-state",
            "useConstants": {"renamed_secToHour": "secToHour"}
            "emitType": "map",
            "template": {"uptimeHrs": {"$eval": "UptimeSec / renamed_secToHour"}}
        },

I think the second is versatile enough to cover a lot of usecases, also not really hard to implement.

global variables

I think we can do this, with the current zip/combineLatest implementations, or I don't get the usecase.

default values for combineLatest

      {
            "fromTopics": ["tele/sp-desk/STATE", "tele/wemos-dev/STATE"],
            "toTopic": "transformed/combineLatest",
            "emitType": "combineLatest",
           "defaultValues": [null, {"UptimeSec": 0}]
            "template": {"uptimeDesk": "${messages[0].UptimeSec}", "uptimeWemos": "${messages[1].UptimeSec}", "sum": {"$eval": "messages[0].UptimeSec + messages[1].UptimeSec"}}
        },

and in this case, if the first topic gets a message, it will run (like the second would had the given message already), but if the second topic would get messages it would do nothing until the first emitting at least one (like now).

I think this is doable and easy.

time based interpolation

        {
            "fromTopic": "tele/sp-desk/STATE",
            "toTopic": "transformed/sp-desk-state",
            "useConstants": {"renamed_secToHour": "secToHour"}
            "useMetrics": {"ms": "sincePrevMsg", "avgTime": "avgTimeBetweenMessages", "sumTime": "sumObservedTime", "msgCount": "msgCount"}
            "emitType": "map",
            "template": {"estimatedMessagePerHour": {"$eval": "renamed_secToHour * 1000 / avgTime"}}
        },

If I would start sth like this, I would add multiple metrics, mostly the ones described above. This is what you thinked about?

I think this is also not really hard to add.

implement and reimplement

If I would add these features, I would surely rewrite the whole thing to TS. Which would add some more time to the implementation time, but I think the all things I described here could be done in an afternoon.

[SMCinc]

global constants

on the first sight I thougt, your first proposal would be sufficient. - But the longer I thinkl about it your second proposal is much better:

        {
            "constant": "secToHour",
            "value": 3600,
        },
        {
            "fromTopic": "tele/sp-desk/STATE",
            "toTopic": "transformed/sp-desk-state",
            "useConstants": {"renamed_secToHour": "secToHour"}
            "emitType": "map",
            "template": {"uptimeHrs": {"$eval": "UptimeSec / renamed_secToHour"}}
        },

global variables

e.g. at the following "wrapper":"energy_pv_sum" the name ist only used inside the topic, right?

    {
    "fromTopic": "hres",
      "toTopic": "calc/out/energy/pv_sum",
      "emitType": "map",
      "wrapper":"energy_pv_sum",
       "template": {"$eval": "energy_pv_sum/3600"}
    },

or is it possible to use "energy_pv_sum" in an other topic?

default values for combineLatest - looks fine !

time based interpolation

yes , might be good

Just as ispiration:
My workaroud at the moment ist to collect with mosqutto_sub topis in ram disk files every 5min @ crontab:

hist=$( tail -n 1 /var/www/html/openWB/ramdisk/obis/KT0.txt ) && echo $hist > /var/www/html/openWB/ramdisk/obis/KT0.txt && mosquitto_sub -t solarview/WR0/KT0_corr -W 299 >> /var/www/html/openWB/ramdisk/obis/KT0.txt
hist=$( tail -n 1 /var/www/html/openWB/ramdisk/obis/PAC.txt ) && echo $hist > /var/www/html/openWB/ramdisk/obis/PAC.txt && mosquitto_sub -t solarview/WR0/PAC -W 299 >> /var/www/html/openWB/ramdisk/obis/PAC.txt
#

and do the interpolation in a sript running in the background.
after the second increment at every increment of the low resolution counter a daption is done.

#!/bin/sh

#All calculation here has to be done with integer variables

#initialisation
fac_phys=3600000 #1 kWh = 1000 Wh * 3600Ws
add_round=$((-fac_phys/2)) #dependend to cut method of input counter: if rounded : (- 1*resolution / 2) (here -0.5 kWh = 180000W)s // if input counter is cut (floor): 0

KT0old=$(tail -n 1 /var/www/html/openWB/ramdisk/obis/KT0.txt) #input from file which collects mosquitto_sub (resolution 1  [kWh])
KT0old=$(((KT0old) *fac_phys + add_round)) # -1800000 Ws = 0.5 kWh due to input counter is rounded 0.5...1.5 = 1 kWh
hres=$((KT0old))
ctr=0       #counter delay for mqtt topc (
corr=10000   #correction factor [1/10000 = 0.01%] e.g. +5% -> 10500
sync=false #trigger for adaption @synchronisation
start=true #First sync shall not be considered in adaption / adaption shall be done from first  complete cycle on
diff=0 #difference Ws

#calculatiom in 1 s recurrance /therefore script runs in background nohup <filemame> &
while true ;do
PAC=$(tail -n 1 /var/www/html/openWB/ramdisk/obis/PAC.txt) #input from file which collects mosquitto_sub (integer resolution 1 [W])
KT0=$(tail -n 1 /var/www/html/openWB/ramdisk/obis/KT0.txt) #input from file which collects mosquitto_sub (integer resolution 1 [kWh])
if [ $KT0 -ne 0 ]; then KT0=$((KT0 * fac_phys + add_round)) # -1800000 Ws = 0.5 kWh due to input counter is rounded 0.5...1.5 = 1 kWh
fi
# snchonisation request
if [ $KT0 -ge $((KT0old + fac_phys)) ];then diff=$((hres - KT0)) &&  hres=$((KT0)) && startold=$start && start=false  && sync=true
#counter works
else hres=$((hres + PAC * corr / 10000)) #PAC [W]* 1 s -> [Ws)
fi

#output is limited  to (KT0old + max-1hex)until KT0 is incremented
if [ $hres -le $((KT0old + fac_phys*999/1000)) ];then hresout=$hres
else hresout=$((KT0old + fac_phys*999/1000))
fi

#calculation of adaption
if [ $sync = true ] && [ $start = false ] && [ $startold = false ];then corr=$(( fac_phys * corr / (diff + fac_phys))) && sync=false
else sync=false
fi

# mqtt output 5s
if [ $ctr -ge 5 ]; then mosquitto_pub -t hres -m $hresout && mosquitto_pub -t calc/hres/corr -m $corr && mosquitto_pub -t calc/hres/diff -m $diff && ctr=0
fi

if [ $KT0 -ne 0 ];then KT0old=$KT0
fi
ctr=$((ctr +1))
sleep 1
done
#

[tg44]

#19 Added "global constants" and "default values for combineLatest". Could broke other stuffs, bcs I moved all the codebase to TS. I added some tests, so the bare minimum is covered (about 60%, but I don't believe in coverage numbers).

"global variables" most of the times could be done if you output them to other/random topics, and parse them back with combineLatest

time based interpolation, or metrics or whatever it will be, coming in the next updates, I also want to add output types/web-hooks

EDIT; docs will be coming in the next updates...

[SMCinc]

I would like to test "global constants" and "default values for combineLatest" - when I use sha-404a9ef the old config does not work longer.
Can you provide the expected syntax for define constants , use constant and default values. - I guess the line "useConstants": {"renamed_secToHour": "secToHour"} has to be set always ibn combineLatest.

Regarding global variables I agree absolutely. - Can be done with combineLatest

Regarding the time based thing I think I could be done after abstraction in a few lines code.
only the sum of time*Input and a possible reset of the counter @ triggervalue n <> triggervalue n-1

• The calculation of adaption is not necessary. Factors for physical correction can be multiplied in the output line.
• Also the break of sum > trigger + 1 increment - resolution (e.g. 1.999 as long trigger = 1 if the sum is too fast)

My solution above works fine but is unnecessarily complicated regarding the adaption of the correction factor.
Due to using bash script the calculation is done in integer I used [Ws] what leads to big numbers to keep the accuracy.
Saving the mqtt topics may not be the best way. But the best I was able to do.

[tg44]

when I use sha-404a9ef the old config does not work longer

I have a test here which was intended to test backward compatibility, and also the parser should telling you which part of the config is not right. Can you provide more information? What happens when you try to start the latest on your prev working config?

Can you provide the expected syntax for define constants , use constant and default values.

We have a test for that too so the syntax is;

[
            {
                emitType: "constant",
                name: "a",
                value: 1,
            },
            {
                emitType: "constant",
                name: "b",
                value: 4,
            },
        {
            emitType: "map",
            fromTopics: ["t"],
            toTopicTemplate: "out",
            template: {"message": "I ${k.i}, ${f}, ${g}"},
            useConstants: {"f": "a", "g": "b"}
        }
]

[tg44]

I documented everything and added the metrics too. Cleaned up the config too.

I think the next steps would be to create some kind of UI where we can lego the inputs and outputs more easily, but I think this will be the future, and I added a lot of cool functionalities in the last week.

[tg44]

@SMCinc Did you have any time to test the modifications?

[SMCinc]

Hi, yes I did.
witth

{"transforms":[
   {
   "emitType": "constant",
   "name": "FAC_PV",
   "value": "0.99065657"
  },
   {
 "fromTopics": ["shellies/shellyem3/emeter/0/power","shellies/shellyem3/emeter/1/power","shellies/shellyem3/emeter/2/power","solarview/WR0/PAC"],
 "toTopic": "calc/out/power/sum",
 "emitType": "zipLast",
 "template": {"$eval": "floor(0.5+(messages[0]+messages[1]+messages[2])*1.02103678113188-messages[3]*${renamed})"},
 "useConstants": {"renamed": "FAC_PV"}
  },
.....

I get the following message:

pi@raspberrypi:/otp/mqtt-transformer $ sudo docker-compose up
Creating network "mqtt-transformer_default" with the default driver
Creating mqtt-transformer_mqtt-transformer_1 ... done
Attaching to mqtt-transformer_mqtt-transformer_1
mqtt-transformer_1  | App started
mqtt-transformer_1  | MQTT(0): Connected to mqtt://broker:1883
mqtt-transformer_1  | TypeError: Cannot create property 'renamed' on number '0'
mqtt-transformer_1  |     at /home/node/app/build/handleMessage.js:41:37
mqtt-transformer_1  |     at Array.forEach (<anonymous>)
mqtt-transformer_1  |     at /home/node/app/build/handleMessage.js:38:44
mqtt-transformer_1  |     at Array.forEach (<anonymous>)
mqtt-transformer_1  |     at handleMessage (/home/node/app/build/handleMessage.js:30:13)
mqtt-transformer_1  |     at globalHandler (/home/node/app/build/app.js:45:39)
mqtt-transformer_1  |     at MqttClient.<anonymous> (/home/node/app/build/io/mqtt.js:37:9)
mqtt-transformer_1  |     at MqttClient.emit (node:events:390:28)
mqtt-transformer_1  |     at MqttClient._handlePublish (/home/node/app/node_modules/mqtt/lib/client.js:1547:12)
mqtt-transformer_1  |     at MqttClient._handlePacket (/home/node/app/node_modules/mqtt/lib/client.js:535:12)
mqtt-transformer_mqtt-transformer_1 exited with code 0

What did I wrong here?
defining constats seems to be OK

[tg44]

I see, this is not so trivial, and this will be hacky as hell. Your messages are constants and not json objects. So the "correct" way is;

{
  "fromTopics": ["shellies/shellyem3/emeter/0/power","shellies/shellyem3/emeter/1/power","shellies/shellyem3/emeter/2/power","solarview/WR0/PAC"],
  "wrapper": "value"
   "toTopic": "calc/out/power/sum",
   "emitType": "zipLast",
   "template": {"$eval": "floor(0.5+(messages[0].value+messages[1].value+messages[2].value)*1.02103678113188-messages[3].value*messages[3].renamed)"},
   "useConstants": {"renamed": "FAC_PV"}
  }

I will try to come up with a more natural feel to this, but until then this should probably work.

[tg44]

With the latest version (that I just merged), your original rule should work too!