Add SSL/TLS support

[Tuckie]

Describe the problem you have/What new integration you would like

As I understand it, all communication happens in the clear. It would be great if there was a way to encrypt the traffic.

Please describe your use case for this integration and alternatives you've tried:

When looking at creating IoT devices such as door locks, having the traffic in the clear is less than desirable. I realize there are limitations for the ESP8266, but at least some form of encryption is feasible on the ESP32.

Having some way to generate a long-lasting cert during the build process and add it to home assistant via your integration library would be awesome. (or even a copy/paste from the web ui)

Additional context

[OttoWinter]

Yeah that's been something I wanted to work on for some time. However, it cannot make the initial setup process harder.

This will be part of #46 - the system will be like this: there will be a new mode for the native API: client mode.

In this mode you have a central native API server hosted on some machine (not an ESP) that will act kind of like an MQTT broker does: All clients (ESPs + HA) connect to this server and exchange their data over that.

That mode will have TLS support as a requirement: All connections in this mode must be encrypted. The setup will be like this: On initial native API server creation, a root CA is auto-generated. Then, each time an ESP is setup in native API client mode that CA will automatically be copied into the sketch file and used to validate the connection.

It will however not be the default mode, that will still be the ESPs each acting as their own server. And in that mode TLS will not be enabled, because I have the strong feeling that managing the certificates for that would be a major pain (cert would need to be copied to each client, for example HA).

[Tuckie]

Thank you for the detailed response. I'm excited to hear that this is already being brainstormed!

Please let me know if you want any help with testing.

[grote]

Since SSL/TLS is very difficult to set up, wouldn't it be possible to encrypt traffic with a symmetric cipher (like AES) using a pre-shared secret which could maybe even be derived from the API password which already needs to be entered in client and server?

[OttoWinter]

@grote I'd like to use existing solutions here, mostly because creating your own encryption algorithm is very hard without ending up with a fundamental issue like replay attacks etc. Plus TLS is actually not too hard for ESPs - WiFiClientSecure handles that just fine. And the key generation on server side can also be automated.

[grote]

Good point. The complexity of using TLS could probably be reduced, if homeassistant would generate a self-signed cert for the API automatically, so that someone making an esphome node would just need to copy the fingerprint.

[OttoWinter]

@grote The way I have it planned is that whenever the ESPHome native API server is started and a certificate does not exist yet, a new self-signed one will be generated.

Re fingerprint: The WiFiClientSecure interface allows for specifying full certificates (and not just fingerprints) - that should also improve security. I'm not sure how secure the fingerprinting really is/how many bits of entropy it really has.

One different problem will of course also be what happens if the user looses the certificate.

[ericdes]

Is SSL encryption in the roadmap any soon? I personally wouldn't mind if it adds more complexity -- supplying a self-signed certificate is a no brainer. I really love what you've done but I feel uneasy about having even only 1 device transmitting in clear -- and creating a point of failure...

[autox86]

Sound promissing but has a sort of broker in place like Mqtt has. From my perspective this will reduce the benefit of having a 1:1 connection between server and client.

I tried to understand the api code which is used to connect to hass. I think I need much more time to understand the coding so I could judge if the following idea is an option.

My idea:
ESPHome will create the certification authority. This will create the root ca for esphome and make it available in hass.
For each client(ESP) a new client certificate will be created, if not available yet.
The partition table of the ESP must be extend by a partition for only permanent data like the private or public certificate.
When flashing the ESP the bin will be flashed as usual and the certificate will be placed into the new partition.

Some benefits I see here:
The certificate will be put on the device with the first flash of the esp by ESPHome.

When reflashing the ESP the certificate can stay untouched.

ESPHome is acting as management console to replace / renew the certificate

This proposal is based on the assumption that the ESP acts as the client, so the private key cert will be stored in the ESP, whenever hass acts as client it could be the other way around.

Furthermore, I doubt that the certificate could be downloaded from the ESP whenever a serial connector can be attached to the esp.

@otto Winter:
What do you think about it?

[carver7]

Has the the security standard, TLS encryption and authentication using Pre-Shared Keys (TLS-PSK), been considered?

I ask because it seems like it would be an ideal choice if it could be used with Esphome (optionally ?), though this is not my field of expertise:

• Client does not send its unique key to the server
• Client authenticates the server
• No CA certificate, no private key, no generated certificates, etc.
• Requires far fewer resources than traditional certificated-based TLS encryption, and has been implemented for resource-constrained devices
• Is a fully vetted security standard, which therefore implies that it was developed, and is continuously refined, by a community of university and professional researchers (e.g. cryptographers), and it is not a ‘roll your own’ attempt that they adamantly warn against implementing.

Points 1 and 2 above: protect against man-in-the-middle attacks.

Point 2 above: A number of tutorials out there for cert-based https tutorials for esp32 skip using a cert that authenticates the server to the client because it slows down the handshake (and very much slows it down for the esp8266). Therefore, the client is exposed to the risk of MITM attacks, and the strength of traditional TLS standard is not realized in the first place.

TLS-PSK has been recently ported to Arduino for the esp32 (and of course it is in the Espressif SDK for the esp32), but I’m not sure if it’s available for the Esp8266.

From Espressif for esp32
https://github.com/espressif/arduino-esp32/tree/master/libraries/WiFiClientSecure#using-pre-shared-keys-psk

Using Pre-Shared Keys (PSK)

TLS supports authentication and encryption using a pre-shared key (i.e. a key that both client and server know) as an alternative to the public key cryptography commonly used on the web for HTTPS. PSK is starting to be used for MQTT, e.g. in mosquitto, to simplify the set-up and avoid having to go through the whole CA, cert, and private key process.

A pre-shared key is a binary string of up to 32 bytes and is commonly represented in hex form. In addition to the key, clients can also present an id and typically the server allows a different key to be associated with each client id. In effect this is very similar to username and password pairs, except that unlike a password the key is not directly transmitted to the server, thus a connection to a malicious server does not divulge the password. Plus the server is also authenticated to the client.

[krambriw]

Hello, first thanks for providing the nice ESPHome project
This issue about a secure MQTT connection is a very important and missing part if you plan to connect an ESP32 device directly to a MQTT cloud service. Maybe there is a possibility now since a version of esp-mqtt supporting TLS has just been released
I have no knowledge how to create or update the mqtt-client component in ESPHome so I wanted to ask if this would be possible or not using the updated version?

[krambriw]

I can confirm that the following esp32 project works with MQTT over SSL:
https://github.com/lucadentella/esp32-tutorial/tree/master/21_mqtt
I am running this currently on a esp32, simply just using a ds18b20 to read temperature but it is working fine but it would of course be much more useful if MQTT over SSL could be supported by ESPHome

ESP32 MQTT --> CloudMQTT over SSL --> MQTT Node-RED

[mddeff]

Jumping on the dog pile here. Would love to see support for TLS in MQTT natively; Currently piloting ESPHome for data center environmental monitoring , and want to be able to use MQTT with TLS natively (without having to run yet another service/learning a new API). Happy to test images using the library @krambriw linked to. Thoughts @OttoWinter ?

Edit: To be fully clear, I have no idea what it would take to integrate the new library (I'm Ops, not Dev). =)

[timkoers]

I don't have the time to implement it right now. But I do have some verified and tested ESP8266/ESP32 example code that you can use to create your own implementation.
The base library can be found here: https://gitlab.com/timkoers/homeenvironment
A sample sketch can be found here: https://gitlab.com/-/snippets/1991445

[skandalfo]

I'd definitely prefer to have an encrypted connection rather than the current setup. Anyone with access to your network segment and sniffing abilities (think a worm in a compromised laptop) can potentially listen to your esphome password in clear.

I don't think it's strictly necessary to replace the whole protocol to implement encryption. I actually prefer the current setup (where you could potentially have a backup HA instance take over ESPHome devices from a different IP) than the plans to convert nodes to clients of a hub. You would need to do DNS or VIP tricks to switch over to a secondary HA or MQTT broker instance.

I've looked at mbedTLS, which claims to support PSK modes of operation like TLS-PSK-WITH-AES-128-CCM. I'm not sure if the versions provided with ESP8266 and ESP32 SDKs do support that, but I might have a look and try.

A simple approach would be to have the Connect request and response use an "encryption requested" boolean rather than a password, and after that the socket connection would have to be switched to start a TLS negotiation on both sides using the shared password as the PSK.

I understand modifying AsyncServer to integrate flow handling with mbedTLS might be the biggest difficulty.

[NarinLab]

Any update on this? At least for MQTT channel, we need to encrypt it.

[TomKeddie]

Same here. This is stopping me from migrating from Tasmota (as much as I'd like to). They seem to use BearSSL (from https://tasmota.github.io/docs/TLS/). Seems like ram and code space are tight though (as would be expected).

[NarinLab]

Yes, we are wondering if we can use MQTT encrypted channel only (MQTT via WiFiClientSecure), without native API to connect to Home Assistant.

[pastukhov]

https://github.com/nodemcu/nodemcu-firmware are using mbedtls and i can connect with it to MQTT over TLS without any fingerprints on both esp8266 and esp32

[Lumpusz]

@jesserockz @glmnet Now that esphome is backed by Nabu Casa, do you think there would be resources to pick this up? Having encryption in esphome opens up the possibility for use cases that require secure connection, e.g. home security, motion sensors, door locks etc. And here I am talking about all communication channels, not just MQTT,. HA native API could also greatly benefit from this.
EDIT: even though encryption is CPU hungry stuff, ESP32 has crypto hw, and wolfSSL has already support for it. The node could actually generate a self signed cert, with few months expiration time, and regenerate it every few months for even more safety, like LetsEncrypt.

[neuman1812]

Just commenting to give this an thumbs up for integration with TLS/SSL. Would love to convert all my tasmoda devices over to Esphome secure.

[mrkeuz]

Unfortunately temporary switched to from Esphome to Tasmota, exactly cause this issue. I will watch to this issue and wait TLS MQTT will work with hope. Currently, Tasmota used BearSSL so think it possible "peep" and "get" solution from them.

[danielrheinbay]

OpenMQTTGateway has introduced TLS support by way of WifiClientSecure.h

Relevant commit: 1technophile/OpenMQTTGateway@4b42f89

Find WifiClientSecure.h here:
https://github.com/espressif/arduino-esp32/blob/master/libraries/WiFiClientSecure/src/WiFiClientSecure.h

[OttoWinter]

Good news: a version of this is now supported 🎉

In esphome/esphome#2254 I added transport encryption support to the native API layer, and support will be added to HA side soon too: home-assistant/core#56216

Note: This is only for the native API (not MQTT). Note also we use [https://noiseprotocol.org/] instead of TLS/SSL, but noise is a well-known protocol framework that's also used for parts of Signal/WhatsApp. This is because during some early testing we saw that TLS/SSL is very heavy both in code size (100k) and compute time (~1s just for computation during handshake), whereas noise was surprisingly low overhead in both metrics.

For MQTT TLS/SSL may come at some point too, but that is not a focus and would require extensive work because the MQTT library would have to be changed.

[mrkeuz]

@OttoWinter maybe it needs to create additional issue for TLS MQTT?

I'm really looking forward to being added this feature.

Use case is simple: I want to integrate remote places behind NAT securely via external MQTT.

Currently, I should set up secure tunnel to up to my MQTT server, so I need support additional minimal server (Rasp PI, Orange, etc.) for forwarding tunnel. This is terribly inconvenient.

I really hope for the implementation of this feature 🙏

[krambriw]

In the OpenMQTT project secure MQTT connections are now supported but I would of course prefer to stick with ESPHome
Best regards, Walter
https://docs.openmqttgateway.com/upload/advanced-configuration.html#secure-connection-to-the-broker

[mrkeuz]

@krambriw thank you very much for link. Will add to bookmarks. Currently, I switched to Tasmota project firmware. Seems work stable (after last fixes MQTT reconnect issue). But I also prefer ESPHome and want to use it in the future.

[davey]

Just as a side note: having WireGuard for VPN on the ESP32 (behind some NAT router) could be another option (or workaround?) to secure MQTT connections to publicly (self) hosted brokers - I just opened feature request 1444 for that today :-)

[lcavalli]

Hi @OttoWinter, do you think that using the same ecryption protocol used to secure the API channel (noise protocol framework - https://noiseprotocol.org/), could be used to encrypt the MQTT payload at application layer (like described here: https://www.hivemq.com/blog/mqtt-security-fundamentals-payload-encryption/)?
This only works if both the ends (ESPHome and Home Assistant) are able to encrypt/decrypt the payload, but will be transparent for the MQTT broker. This shall improve the MQTT channel security, if not at the same level of API one (message topic is still in clear text), at least enough to transfer sensitive data like alarm panels arm/disarm PINs.
I don't know the effort needed to implement it at the light of the work done on the API channel. If needed I can open a dedicated new feature issue.

[OttoWinter]

@lcavalli No, i don't think that would be a good idea. First, as you said that would require support from both sides (and so has to be implemented for any mqtt device wanting to speak with the ESP), and I don't think there's a good way to make that secure. For example,1. it would probably be very difficult to prevent replay attacks, 2. the devices would need to communicate a handshake which goes against the mqtt pubsub model, and in general having the transport encryption at that layer seems like a bad idea (need to check for every mqtt functionality if it can be impacted in any way)

[mrkeuz]

@OttoWinter I was thinking here at my leisure, what about make revert connection directions. I mean add ability that esphome can connect to home-assistant itself. I.e. in esphome add home-assistant address and token. Connection can use noise protocol here. On home-assistant side we just need to add special listener on some sub-path. In this case we will be able to interact from esphome under NAT. Just as an idea.

[Tuckie]

@mrkeuz Flipping the client/server model is something that has been discussed in the past.