Automation between nodes

[FrengerH]

Is your feature request related to a problem/use-case? Please describe.

I have two EspHome nodes. One node is a wifi light bulb configured to control the light.
The other node is behind my wall switch and configured as input boolean/switch to control the light using an automation in homeassistant.
The problem is: when for some reason homeassistant is not running/responding, I can't switch the light anymore.
When the light and the switch were configured on one EspHome node I could have used an automation in EspHome to still be able to control the light.

Describe the solution you'd like:

I would love to have the possibility to create an automation in EspHome between different nodes.

Additional context:

Example:

Node1 (light):

esphomeyaml:
  name: example_light

light:
  - platform: cwww
    id: main_light
    name: "Main light"
    cold_white: output_cold
    warm_white: output_warm
    cold_white_color_temperature: 153 mireds
    warm_white_color_temperature: 500 mireds
    
output:
  - platform: esp8266_pwm
    pin: GPIO4
    frequency: 1000 Hz
    id: output_cold
    
  - platform: esp8266_pwm
    pin: GPIO5
    frequency: 1000 Hz
    id: output_warm

Node2 (switch):

esphomeyaml:
  name: example_switch

binary_sensor:
  - platform: gpio
    pin:
      number: GPIO5
    name: "Light button"
    on_state:
      - light.toggle: example_light.main_light

[OttoWinter]

This will be a part of #46

However, the ESPs will not communicate directly with each other. They will communicate through the central native API broker. Why? ESPs have very little resources, and I have the strong feeling people will create a network where each ESP is connected to each other ESP. Even with only a couple of ESPs that will quickly kill all resources.

That of course means there will be a central point of failure like with MQTT, but the plan is to make the central API server very lightweight - it will not even have to know what is inside packets, it will just blindly forward them to all clients that subscribed to that. Such a lightweight design will:

• reduce the amount of possible bugs
• make it possible to host on any machine, even a small RPi with minimal overhead.

[bruxy70]

Follow up from a discord conversation.....building a mesh network would be bad. But many people clearly have a need for simple "remote switch" functionality, where they have a sensor on one microcontroller, and want to toggle a switch on another microcontroller - for example to create 2-way or 3-way switch. Only today I have seen 3 different people asking for that.
Doing that via home assistant works - calling a homeassistant service directly is far better than sending mqtt message and triggering home assistant automation (thanks for that), but IMHO it is still overkill. It would be nice if the pair can operate autonomously (so that I do not have to explain to my wife why the switch suddenly does not turn on the light on the staircase whilst I am rebooting home assistant). @OttoWinter, please consider. Thanks.

[OttoWinter]

@bruxy70 Yes... that's why this feature request exists.

[squirrel289]

@bruxy70, afraid I'm asking for an obvious answer here but, as I was considering submitting a feature request to take advantage of native mesh networking capabilities of the ESPs (using, i.e., https://gitlab.com/painlessMesh/painlessmesh), I was hoping you'd be willing to elaborate on why mesh is bad. It seems like it would solve the resource constraints mentioned by @OttoWinter and provide an inherent level of resilience and security by removing dependency on a preexisting wifi network.

[bruxy70]

@squirrel289, I am not going to argue with that! I still believe that the "switches" have two classes of services:

• The "core function" is to turn on/off the light - that's what you'd (read my wife) expect from a switch to do, right?
• And then there is the "secondary" remote control, automation, voice control, etc. that are "nice to have".

If a switch relies on ESP, WiFi, RPi, and Hassio that will definitely have the worst service availability than a simple mechanical switch. If I make an electronic switch, I can't do that without the ESP, so I'll have to accept that. But I guess this is the least likely to fail (and if it does, it will probably need to be changed anyhow).
But in 3 instances, I created a two-way switch that does rely on ICT components, so I have sacrificed its "core function" for an effect, ultimately risking my marriage :). So this is my use case for the mesh feature. But I did not manage to convince others, so I had to accept this. In my work, I would not accept that, but this is an open source community, so that's it I guess.

[l3ok]

Why not make the mesh a feature that can be switched on when required. It can then be enabled on devices where it is required. You could also then use it as backup only when communication with the API broker is not possible

[JayElDubya]

Another idea on this is how about accomplishing this completely out of band of wifi? For example, use a 433mhz transceiver pair to do some sort of inter-device communications that has no requirement for wifi or home assistant. You could still use home assistant as a primary coordinator/relay and use a conditional check to see if it is up and then fall back to the out-of-band method of choice. (rf, bluetooth, wired i2c, etc.) To be robust, maybe two checks...one that HASS is up and another that your target device is connected.
Then you would just need to ensure HASS gets the current state whenever it reconnects.

[danps1]

I think another way to do achieve this functionality would be to take advantage of the existing rest API that is exposed with the web server...
https://esphome.io/web-api/index.html#rest-api
For example I can already control a light using curl...

curl -X POST http://192.168.X.Y/light/stairs/toggle

Is there a way to have a lightweight http/rest client able to make such a call from one node to another?

[brandond]

@danps1 https://next.esphome.io/components/http_request.html

[danps1]

@danps1 https://next.esphome.io/components/http_request.html

@brandond - that is awesome! Thanks!

[LouisF410]

How can I add the http_request component to my ESPHome instance?

[kvn1351]

In my opinion mesh solutions are vastly superior to classical server/client based solutions. IoT has already been moving in that direction for a while now. It would be nice if ESPHome followed that trend.

[10bn]

I think another way to do achieve this functionality would be to take advantage of the existing rest API that is exposed with the web server...
https://esphome.io/web-api/index.html#rest-api
For example I can already control a light using curl...

curl -X POST http://192.168.X.Y/light/stairs/toggle

Is there a way to have a lightweight http/rest client able to make such a call from one node to another?

Did figure out how to make it work?
Please leave a example if you managed to get it working.

[danps1]

I haven't yet given it a try, but if you want to what you'll need to do is use the dev branch of esphome as the http_request component has not yet been merged into the master branch.

[danps1]

@brandond https://next.esphome.io/components/http_request.html

One thing I can't work out is if the http_request component allows for authorization?

The http_server component supports digest auth, so it would be really good if we could use http_request from one ESP to control another ESP in a secure manner

[CarlosGS]

This would be very nice. My use case is a power sensor (node 1) that controls a water heater (node 2). Sending the data directly would allow for the lowest latency & significantly reduce load on Home Assistant.

Has anybody succeeded in using the http_request module for this purpose?

[iliyang]

FWIW, Tasmota offers several options to communicate between devices. I tried each and here's my experience:

1. Direct HTTP requests: no middle-man required (apart from WiFi router, obviously). Works but there's a noticeable lag. My explanation is that each request entails initiating a new connection between the nodes, and then HTTP is a rather heavy protocol for simple requests like "turn relay on".
2. KNX: again, no middle-man required, but it was quite unreliable for me.
3. MQTT: an MQTT server is required. But this option is lightning-fast. My explanation is that it's because each node maintains an active connection to the MQTT server at all times, and then individual messages have really small payloads compared to HTTP for example. So I took this option and has been working well for over a month now. In retrospect, this result is not surprising – it's what MQTT was designed for, i.e. quick communication between devices.

So the only disadvantage of the MQTT option is the reliance on a server, in addition to the router. I'm thus contemplating running the MQTT server on my router directly.

So it seems that if ESPhome is to implement a direct communication solution that does not rely on a middle-man, its performance should be measured against MQTT.

[ascillato]

FWIW, @iliyang

Besides all 3 options, you also have UDP groups in Tasmota. This last option could be an alternative for ESPHome for device to device communication.

About KNX in Tasmota, you should open a new issue there, because KNX is pretty reliable for me. Most of the time, the initial issues some other users had with KNX were not enabling Multicast on theirs wifi routers.

Anyway, UDP groups could be a light weight alternative for device to device communication on ESPHome.

[iliyang]

Thanks for the tips regarding KNX and UDP groups @ascillato!

[rradar]

I think esp now could also be a interesting approach for ad-hoc communication between nodes. See #275 for more (and a custom_component 🎉)

[BWilky]

I managed to successfully use one SONOFF as the "remote" which called HTTP on the other SONOFF.

Light

web_server:
  port: 80

  
output:
  - platform: gpio
    pin: GPIO12
    id: relay_1

light:
  - platform: binary
    name: "default"
    id: light_1
    output: relay_1

status_led:
  pin:
    number: GPIO13
    inverted: yes

"Remote"

web_server:
  port: 80
  
http_request:
  useragent: esphome/device
  timeout: 10s
  

    
binary_sensor:
  - platform: gpio
    pin:
      number: GPIO0
      mode: INPUT_PULLUP
      inverted: true
    name: "Sonoff Basic Button"
    on_press:
      - http_request.post: "http://192.168.0.50/light/default/toggle"
    

[glyndon]

The way Tasmota provided this feature was through a limited set of 'Device Groups' which can be handy.
It's not so much a mesh, but a short list of 'peers' that locate each other by nodename.
This approach prevents someone over-using it for more than just a few nodes.
It lets you, for example, have a set of lights which share commands with each other, so any automation just tells one light what to do and the others follow it.
https://tasmota.github.io/docs/Device-Groups/

[bkaufx]

It would be nice if switches and lights etc. could directly talk to each other. The specific use case for me is having a temp sensor directly communicate with some relays that run my air conditioner so that the air conditioner still works properly when Home Assistant is down.

Perhaps one ESPHome device could add an entity from another ESPHome device using the Home Assistant native API. There could be a platform for remote ESPHome entities.

YAML configuration something like this:

sensor:
  - platform: esphome_remote  
    address: bedroom_temp.local    # remote devices DDNS address or IP address
    remote_id: temp                # id or name set in remote device's config
    id: temp                       # id for entity on this ESPHome device; could default to same id as remote

[Cougar]

For me this is the main show stopper to move everything to ESPHome. I still use Tasmota in Sonoff RF Bridge and lamp switches that I'd like to work without Home Assistant (and during HA upgrades).

For just lighting control, what about using E1.31 protocol? Client (light) side is already implemented and works fine. It is meant for a LED strips but I don't see any problem why it can't be used (or modified) to address single light(s).

Only controlling code is missing. It should be quite lightweight to send UDP packets. Bigger challenge is to make this component easy to use and universal enough. Thoughts? - Sorry, I first posted it under wrong FR (#275)

[cabinlab]

Would it be practical to break out the HA or ESPHome API broker into a standalone package for OpenWRT or a tiny Docker container?

While I don't currently have the hardware to test this approach, Docker containers are supported in OpenWRT. With enough memory on the router, it's already possible to spin up a minimal HA or ESPHome container inside of OpenWRT. While perhaps not the most elegant solution, that would enable communication between nodes without a dedicated HA server.

If it's feasible to take the next step and make the ESPHome (or HA) API broker run directly on OpenWRT, that would eliminate all of the Docker stuff, and perhaps allow this to work on commodity router hardware.

That wouldn't be a lot different than running an MQTT broker on OpenWRT for the nodes to talk to one another, which is also already an option. But as discussed earlier, the API has advantages.

[RoganDawes]

Would it be practical to break out the HA or ESPHome API broker into a standalone package for OpenWRT or a tiny Docker container?

The api is already a package/repository of its own, so that part is done. It "just" requires python3 to run, and a small number of additional requirements. Of course, you would still need to write the inter-node comms part, allowing one node to query another nodes values, or invoke a service, and at that point, you are basically running a minimal HA installation anyway.

And that still doesn't address the original request: direct node-node comms, WITHOUT an intermediary.

[bkbartk]

For just lighting control, what about using E1.31 protocol? Client (light) side is already implemented and works fine. It is meant for a LED strips but I don't see any problem why it can't be used (or modified) to address single light(s).

good question. the manual isn't completely clear to me, but for as far as I can see this is for receiving broadcast messages, not for sending them. This is a part which might be missing.
also, the part where the number of leds for a ledstrip is important at all is completely lost to me.
It would be great if the feature for (on/off) would also work for switches.
I require one feature which for some reason does not work on the light component so I use switches for lights.

[FARKIr]

radio comunication 433mhz transciever and reciver connected to esp chip and state comes to HA also ?

[samturner3]

This feature could be out of the box once esphome expands to support Thread/Zigbee on the new esp-h2 hardware. #1430

[nagyrobi]

See https://esphome.io/cookbook/http_request_sensor.html showing how to retrieve sensor data directly from one node to another. Similarly could build up automating POST requests to toggle switches like above.

[bkbartk]

See https://esphome.io/cookbook/http_request_sensor.html showing how to retrieve sensor data directly from one node to another. Similarly could build up automating POST requests to toggle switches like above.

but how to pass the authorization?
if you have it enabled on the server auth method is digest,
but http_request only supports basic

[nagyrobi]

I've added notes about that, PR is being merged. (check here until then)

[bkbartk]

can you also link the PR for me please?
So I maybe could include it like this

external_components:
  - source: github://pr#3500
    components:
      # list all components modified by this Pull Request here
      - web_server
      - web_server_idf
      - web_server_base
      - captive_portal
    refresh: 100days

sending the plain auth header for digest gave me some issues in the past,

[nagyrobi]

It's just a PR to the docs... (2967)

[Tofandel]

Can't we just repurpose the api component and add an automation to make request to an external api server (which would be another esphome), right now it seems only home assistant can make this kind of requests

The same way the http_request.post automation work, but using the Native api protocol

Like this if your ESPs have fixed ips you can easily have them talk directly to each other, without the web server overhead and super hard authentication to setup

[nagyrobi]

See: #46

[Tofandel]

@nagyrobi That's quite different from what I'm proposing, this issue is about creating a server in home assistant, it's still esp -> ha -> esp communication and not esp -> esp communication

[ssieb]

I don't know why he removed the other links, but #2081 is what you're looking for.

[ferbulous]

There’s a new device group component you could try

https://github.com/Cossid/TasmotaDeviceGroupsForESPHome

[bkbartk]

There’s a new device group component you could try

https://github.com/Cossid/TasmotaDeviceGroupsForESPHome

unfortunately I get several build errors when I try this one

Compiling /data/keuken-aanrecht/.pioenvs/keuken-aanrecht/src/esphome/components/json/json_util.cpp.o
Compiling /data/keuken-aanrecht/.pioenvs/keuken-aanrecht/src/esphome/components/light/addressable_light.cpp.o
In file included from src/esphome/components/device_groups/device_groups.cpp:1:
src/esphome/components/device_groups/device_groups.h:271:3: error: 'bools' does not name a type; did you mean 'bool'?
  271 |   bools DeviceGroupsStart();
      |   ^~~~~
      |   bool
src/esphome/components/device_groups/device_groups.cpp: In member function 'virtual void esphome::device_groups::device_groups::loop()':
src/esphome/components/device_groups/device_groups.cpp:140:9: error: 'DeviceGroupsStart' was not declared in this scope; did you mean 'DeviceGroupsStop'?
  140 |     if (DeviceGroupsStart()) {
      |         ^~~~~~~~~~~~~~~~~
      |         DeviceGroupsStop
src/esphome/components/device_groups/device_groups.cpp: At global scope:
src/esphome/components/device_groups/device_groups.cpp:211:6: error: no declaration matches 'bool esphome::device_groups::device_groups::DeviceGroupsStart()'
  211 | bool device_groups::DeviceGroupsStart() {
      |      ^~~~~~~~~~~~~
src/esphome/components/device_groups/device_groups.cpp:211:6: note: no functions named 'bool esphome::device_groups::device_groups::DeviceGroupsStart()'
In file included from src/esphome/components/device_groups/device_groups.cpp:1:
src/esphome/components/device_groups/device_groups.h:243:7: note: 'class esphome::device_groups::device_groups' defined here
  243 | class device_groups : public Component {
      |       ^~~~~~~~~~~~~
src/esphome/components/device_groups/device_groups.cpp: In member function 'bool esphome::device_groups::device_groups::_SendDeviceGroupMessage(int32_t, esphome::device_groups::DevGroupMessageType, ...)':
src/esphome/components/device_groups/device_groups.cpp:856:19: warning: suggest parentheses around assignment used as truth value [-Wparentheses]
  856 |       while (item = *previous_message_ptr++) {
      |              ~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~
Compiling /data/keuken-aanrecht/.pioenvs/keuken-aanrecht/src/esphome/components/light/automation.cpp.o
Compiling /data/keuken-aanrecht/.pioenvs/keuken-aanrecht/src/esphome/components/light/esp_color_correction.cpp.o
*** [/data/keuken-aanrecht/.pioenvs/keuken-aanrecht/src/esphome/components/device_groups/device_groups.cpp.o] Error 1
========================= [FAILED] Took 15.80 seconds =========================

Next to that, it's really cool if it would work.

[ferbulous]

So far it’s working fine with my switches and lights.
You could raise an issue there to fix it

[RoganDawes]

How about extending the API to send/receive broadcast UDP packets, prefixed by the originating node name, and including the sensor entity update packet? Perhaps by adding a flag (along the lines of the internal:, add broadcast, perhaps) to the entity, or else by configuring a list of sensors to be broadcast under api:

api:
  broadcast:
    - entity1
    - entity2

Then you could define remote entities such as a switch using the api platform, give it a name, etc, which would allow it to be referenced by other entities in the esphome configuration/main.cpp. This could also help to address #46 - by broadcasting the packet to everyone on the subnet, deciding who needs to receive it becomes a moot point. Obviously, there are security considerations, if you also broadcast the API password to all and sundry.

I still think this would be a good approach, and the security issues would be addressed by nodes sharing an API NOISE encryption key. The only snag is the use of an entity identifier in the protobuf api, rather than the actual name of the entity. While the entity identifier is derived from the name, it's not exposed anywhere obvious, making it difficult to configure the remote nodes.

One way of working around that would be for nodes to do a periodic poll of "subscribed nodes", to request the entity identifier for any remote names that it has as references. This is not required with the current API because a TCP connection is torn down when the node receives a new configuration (and reboots!), while a node could easily reboot in between UDP updates without any relying nodes being any wiser. Of course, nodes could also publish relevant API messages such as DeviceInfoResponse, and ListEntitiesBinarySensorResponse on startup, telling any relying nodes to refresh any state that they are holding for entities related to that node.

If I could actually code, this is what I would implement:

1. API client and server over UDP, with NOISE encryption/decryption support.
   1.1. Since NOISE negotiates a unique CipherState pair per pair of participants, the API would need to maintain a map of keys per communicating node. I'd define each node expected/allowed to communicate with this node under a Peer: key of the api:. This could even be used to define the static shared key to be used, if the remote node is not using the same API key as the local node. The IP Address of the peer could also be configured here, if it should not be discovered automatically. This can possibly be used to enable communication with devices on different networks if required.
   1.2. Since the protobuf API does not include the node details in each message, the node name would need to be prefixed in clear to the encrypted protobuf message, so that receiving nodes would know which CipherState to use to decrypt the body. It would probably make sense to define a new protobuf message PeerMessage with a node name, uptime and encrypted body for this, so it can be extended if necessary.
2. Nodes (on boot and periodically) broadcast a simple "I'm here" PeerMessage containing its name and uptime, and an empty body to the network address, and directly to any hardcoded peers if they do not fall within the network. This allows any existing peer nodes to start the NOISE handshake (probably after a random delay, to not swamp the new node). The uptime allows peers to decide whether a new handshake is required, if the age of the negotiated CipherState pair is older than the uptime of the peer. Peers can also (optionally) update the IP Address that the peer message was received from, to enable directed UDP messages to be sent, rather than broadcast to the network address. Note: There is a potential denial of service here, if a PeerMessage is sent from an invalid IP address, as this is an unauthenticated message. If this is a concern, the peer IP address should be hard-coded in the config, I suspect.
   2.1. Peers can discard any state held for other peers if they have not received a PeerMessage from that peer for a certain period. This will also allow "remote entities" to be updated to "unknown" states if necessary/desired.
3. Once the handshake has been performed, nodes inform peers of entities that they are interested in, using a new message SubscribeEntityRequest, containing the device_class and the entity name. I think this would require a new platform: api or platform: peer to be defined for each class of entity (BinarySensor, Sensor, Switch, Cover, etc) that would include a peer name, and the peer's entity name. These entities would start off in an unknown state, and be updated once the peer has been contacted and details received. I think these entities would need to be internal: true by default to prevent proliferation. Without this new message, the node would need to broadcast all its entity updates to all peers individually, multiplying the number of packets, because there is no simple shared key in the NOISE encryption (to my knowledge).
   4.1. The peer responds with a suitable ListEntitiesBinarySensorResponse, according to the requested device_class. This will have all the relevant information about the entity, including its entity_id.
   4.2. The peer also updates a local list of subscribed entities for the peer. This list may be cleared if the requesting node fails to send an Announce message for a defined period.
4. Once the peer has the mapping from entity name to the entity id, that can be stored in the local entity state.
5. Any updates to an entity within a node will be communicated to the API subsystem, which will iterate through the list of peers to see if any peers are subscribed to that entity. If they are, an update will be prepared, encrypted using the peer's CipherState, packed into a PeerMessage with that peer's name. The PeerMessage will be sent to the configured or discovered IP address for the peer, in a UDP datagram.
6. The node listens on the defined UDP port for PeerMessage datagrams. It reads the sending node name, looks up the relevant CipherState to enable it to decrypt the message. From there it looks up its internal entities to figure out which to update.

So a config file might look like:

api:
  encryption:
    key: whatever
  peer:
    - name: myswitch
        address: 192.168.1.100
        key: someotherkey
    - name: whatever

binary_sensor:
  - platform: peer
     peer_name: myswitch
     peer_entity_name: the_button
     on_press:
       - switch.toggle: relay

[RoganDawes]

I suspect I have missed a detail, in that the PeerMessage would need to have a field for the NOISE protocol handshake message, since those don't seem to be defined in the protobuf API. But that should not be too tricky to figure out.

[nagyrobi]

Tadaaaaam: esphome/esphome#6865

"Works On Everything" (TM).

[nagyrobi]

See https://esphome.io/cookbook/http_request_sensor.html showing how to retrieve sensor data directly from one node to another. Similarly could build up automating POST requests to toggle switches like above.

When the component linked above will be merged, this cookbook entry will be removed.

[adipierro]

My use would be a DIY NFC card reader with multiple protocols (including some proprietary ones, see my project) that could 'talk' to a lock controller (built on ESPhome) securely, either over network with native API or UART/RS-486.