Zigbee suqare button and cube support via fake_device #709

bskaplou · 2020-05-28T10:02:27Z

Add support for square button and cube actions via script installation on gate and events receiving with fake_device.

Also added zigbee_command to make functionality avialable through CLI

coveralls · 2020-05-28T10:04:13Z

Coverage decreased (-0.9%) to 72.434% when pulling 1aba320 on bskaplou:zigbee_suqarebutton_cube into dd0255c on rytilahti:master.

starkillerOG

@bskaplou nice, can't wait to test this.
Unfortunately I do not own a squarebutton or cube :(
Would it be possible to implement this for the magnet device?

I made some sugestions for improvements, hope you are open for it.

If you like I could do a more extensive revieuw after you proccesed these sugestions.

starkillerOG · 2020-05-28T17:46:18Z

miio/gateway.py

@@ -201,6 +218,38 @@ def discover_devices(self):

        return self._devices

+    def x_del(self, script_id):


I would rename "x_del" to "unsubscribe_event"
and "script_id" to "subscription_id"

x_del is name of method... python method name mimics MIIO method... Less conversions => more comprehension... x_del is not executed directly, it's low-level method which is called from uninstall ...

alright I understand, you are probably right if this function is only called by other functions

I agree that this should be named otherwise, especially if this can be useful for other use cases. Otherwise it is probably simpler to simply use self.send where necessary (or at least rename this to something more descriptive, and prefix it with _ to indicate that it's a private method not to be used by others.

miio/gateway.py

starkillerOG · 2020-05-28T17:58:08Z

miio/gateway.py

@@ -738,6 +787,21 @@ def get_firmware_version(self) -> Optional[int]:
            )
        return self._fw_ver

+    @command()
+    def uninstall_scripts(self):


I think this uninstalls all scripts right?
I would then rename this to "Unsubscribe_all_events"

this is actual installation of program into gateway memory.... installation and deletion...

Could you explain what you can do with loading such a "program" into gateway memory?
Can you only get callbacks when an event happens/property changes or can it do more?

It would be really nice if you could include docstrings """explanation"""" for all new functions that briefly explain what the function can do and how it works.

Could you explain what you can do with loading such a "program" into gateway memory?
Gateway is able to store limited amount of such "scripts". Scripts are permanent upon restarts. External device call parameters are also encoded with script (i.e. rotate)...
For each devices receiving messages from scripts gateway maintains state... It means gateway is sending ping requests and will not send actual packages in pings are faulty..

Can you only get callbacks when an event happens/property changes or can it do more?
Unfortunately we have no callbacks... Script allows to send package to other miio device as a result of action....

docs are added for all methods

Subscription mechanism can be implement by scripts if you use target with ip is an ip of python-miio running device... But in xiaomi scripts are not expected to be used as callbacks. It was designed to make gateway implement complete automation listen=>process=>send data to other devices... Such behavior is still possbile. You can create script which turns on wi-fi power plug in cube rotation without processing of callback in hass.

starkillerOG · 2020-05-28T18:03:20Z

miio/gateway.py

+
+    @command()
+    def install_move_script(self):
+        return self._gw.install_script(self.sid, "move", build_move)


I would rename all of these to "subscribe_to_move" etc.
And make an aditional function "subscribe_to_all_events" that calls all these functions in one go.

maybe consider removing all these individual functions and just make one list "events" like the "properties" that contains all events that can be subscribed to ["move", "rotate", "shakeair", "flip90", "taptap", "flip180"] and then just make one function "subscribe_to_all_events" that subscibes to all events in a for loop.

Removed most of code duplication... Further reducing requires deep work with click....

starkillerOG · 2020-05-28T18:10:44Z

miio/gateway_scripts.py

+    lumi, source_id = source_sid.split(".")
+    method_name = f"move_{source_id}"
+
+    move = [


move -> subscription_info

starkillerOG · 2020-05-28T18:11:01Z

miio/gateway_scripts.py

+
+    move = [
+        [
+            action_id["move"](source_sid),


move --> event

starkillerOG · 2020-05-28T18:12:53Z

miio/gateway_scripts.py

+                    "0",
+                    {
+                        "did": source_sid,
+                        "extra": "[1,18,2,85,[6,256],0,0]",


The 256 is specific for the move event, so make a mapping dict at the top of this file that maps events to these specific values:

extra_parameter_mapping = {"move": 256, "flip90": 64} etc.

starkillerOG · 2020-05-28T18:14:19Z

miio/gateway_scripts.py

+                    {
+                        "did": source_sid,
+                        "extra": "[1,18,2,85,[6,256],0,0]",
+                        "key": "event." + source_model + ".move",


"event." + source_model + ".move"-> f"event.{source_model}.{event}"

starkillerOG · 2020-05-28T18:15:08Z

miio/gateway_scripts.py

+    return dumps(move)
+
+
+def build_flip90(


The rest of these functions are then not needed anymore

bskaplou

Would it be possible to implement this for the magnet device?

@starkillerOG Yes sure! Can you sniff interaction between mobile app and magnet during creation of automation involving magnet?

bskaplou · 2020-05-28T22:32:34Z

miio/gateway.py

+
+    @command()
+    def install_move_script(self):
+        return self._gw.install_script(self.sid, "move", build_move)


Removed most of code duplication... Further reducing requires deep work with click....

bskaplou · 2020-05-28T22:36:47Z

miio/gateway_scripts.py

+    target_id=fake_device_id,
+    source_model="lumi.sensor_cube.v1",
+    message_id=0,
+):


Implemented

starkillerOG · 2020-05-29T07:22:40Z

@bskaplou I have not yet been able to sniff local direct traffic to or from my gateway.
I do see the UDP packets on my desktop using wireshark but that is a diffrent protecoll and since I use my phone for the MiHome app it does not capture direct packets I think.

On my phone I also have an capture app "Packet Capture", that worked greath for another application, but with the MiHome app it just captures hundreds of packets to outside servers (I pressume the Xioami Cloud servers, but I do not see local packets).

What app are you capturing, is it the MiHome app?
How are you capturing, which program do you use and do you do it on your phone or do u use an android/ios emulater on your PC and then capture with your PC?

starkillerOG · 2020-05-29T07:37:18Z

miio/gateway_scripts.py

+    target_ip,
+    target_model=fake_device_model,
+    target_id=fake_device_id,
+    source_model="lumi.sensor_switch.aq3",


maybe consider adding "lumi.sensor_switch.aq3" as a property of the subdevice class:

@property def model(self): """Return the model of the device.""" return "lumi.sensor_switch.aq3"

also include a default for the base class:

@property def model(self): """Return the model of the device.""" return "lumi.unknown"

I think this can be usefull for more functions in the future and this can be used to show in HomeAssistant as the model in the device info in the GUI.

That property can than be passed to these functions.

Probably someone will eventually figure out how to get the model directly from the gateway, I have tryed but not suceeded yet since I cannot yet capture packets.
I know the models are shown in the MiHome app if you go to the gateway device --> click ... --> about --> Hub info.
That shows a list containing dicts that have model, sid, and name, would be very usefull to get that list since we would then also have nice default names for inside HomeAsssitant while setting up the integration

maybe consider adding "lumi.sensor_switch.aq3" as a property of the subdevice
I actually hope at least some of these methods will be reusable between devices for example between single/double/triple button switch...

Probably someone will eventually figure out how to get the model directly from the gateway
:) Impossible without new firmware... As far as I get original idea is use of independent agents instead if client<->server ... All calls between devices are independent...

bskaplou · 2020-05-29T20:57:09Z

Significantly updated...

starkillerOG · 2020-05-29T21:11:25Z

How do you sniff miio trafic?

starkillerOG · 2020-05-29T22:14:00Z

I think this looks very good!

Could you possibly make an script for the gateway alarm?
I saw that there is an automation to trigger stuff if the Alarm goes off

bskaplou · 2020-05-30T07:36:23Z

@starkillerOG
I use rt-ac68u asus router to sniff traffic with asuswrt-merlin firmware.
Steps:

Install asuswrt-merlin into router
Install Entware in SD card of router
Install tcpdump in router
Sniff traffic on router via ssh: tcpdump -i eth1 -w dump1 '(src 192.168.2.109 or dst 192.168.2.109)' . 109 is my gateway
Copy file to laptop with scp
Convert dump into json: tshark -r dump1 -T json >dump1.json
Decrypt data with nodejs miio tool miio protocol json-dump dump1.json --token XXX
All done :)

Could you possibly make an script for the gateway alarm?
Yep NP, will do it... There are some other nice possibilities with gateway... It can play network radio as well...

@rytilahti waiting for PR approve and release :)

bskaplou · 2020-06-02T06:22:54Z

@rytilahti is everything fine with PR?

starkillerOG · 2020-06-02T08:01:26Z

@bskaplou great work with this code!

In the mean time I have been able to sniff traffic and test the code.
I was able to sniff trafic using BlueStack (an android emulator for windows), install MiHome App and then capture packets using wireshark --> export as json and use the miio tool https://github.com/aholstenson/miio to decode the packets.

Your code worked mostly nicely and I was able to capture the magnet sensor open event.

The two major things that I think still need to improve in this code are:

the tokens dict in the gateway_scripts.py file
- the 'real' schould use the token of the gateway class
- there realy needs to be a method to obtain/use encoded token and data_tkn, otherwise this code will be unusable for HomeAssistant (otherwise everyone would need to first sniff traffic to figure out these codes, which I think is too much to ask for HomeAssistant users)
the fake_device.py file schould be implemented such that it can be called by HomeAssistant or inside a simple python script with a non blocking method to call the start server and stop server.
- maybe consider a asyncio server (see code example below)
- you could also look at https://github.com/Danielhiversen/PyXiaomiGateway/blob/master/xiaomi_gateway/__init__.py for inspiration with the "listen" and "stop_listen" functions inside the XiaomiGatewayDiscovery class

import asyncio

async def handle_message(reader, writer):
    data = await reader.read(1000)
    message = data.decode()
    if message:
        addr = writer.get_extra_info('peername')
        print("Received %r from %r" % (message, addr))

    #writer.write(data)
    #await writer.drain()

    #print("Close the client socket")
    #writer.close()

loop = asyncio.get_event_loop()
coro = asyncio.start_server(handle_message, IP, port, loop=loop)
server = loop.run_until_complete(coro)

# Serve requests until Ctrl+C is pressed
print('Serving on {}'.format(server.sockets[0].getsockname()))
try:
    loop.run_forever()
except KeyboardInterrupt:
    pass

# Close the server
server.close()
loop.run_until_complete(server.wait_closed())
loop.close()

bskaplou · 2020-06-06T10:04:46Z

@starkillerOG removed token from code... passed as cli args now...
FakeDevice is not expected to be directly called from hass. Fake device is simple example which gives the idea. .... hass will have it's own async cycle for sure...

…u/python-miio into zigbee_suqarebutton_cube

starkillerOG · 2020-06-06T13:28:36Z

Homeassistant will almost certanly require the server/fake_device code to be implemented in miio library instead of in HASS itself, since they always want to push as much as code possible to the libs.

Besides it is convenient to have the server code in miio lib because then things like open/closed state can be tracked internally in the miio lib.
There will just need to be a way to register an external callback function such that HASS can fire events when for instance a remote button is pressed.

@rytilahti what do you think?

rytilahti

Thanks for the PR, and my apologies for the belated answer. I'm currently very busy with some other things, and as I realized that this will require some planning, I have been postponing the review and this is also a short not comprehensive one...

The main critique from my side are:

There is plenty of repetition, the event system interface should be very generic and applicable to new devices without touching any other code besides the code of that new device. This means, that we need a way to expose the available events that can be bound to "handlers/actions". In the perfect case we will have a method with a signature not unlike install_event_handler(source_device, event, target_device, target_method).

Let me know if any of my thoughts make sense to you :-)

The data structures need to be converted to attrs classes. This will allow generating the "payload" directly from a class instance without any manual dict/list handing.

rytilahti · 2020-06-06T21:16:46Z

miio/fake_device.py

+                ),
+            ),
+        )
+    )


This reads really, really complicated.. What for this is and is it necessary? I suppose for binding on multiple interfaces? If yes, that could also be potentially useful for discovery (iirc there is an issue about not being able to discover devices from other subnets). Is there maybe a library that could be used to achieve this?

rytilahti · 2020-06-06T21:17:09Z

miio/fake_device.py

+class FakeDevice:
+    _device_id = None
+    _address = None
+    _token = None


There is no need to define these as class attributes.

rytilahti · 2020-06-06T21:18:37Z

miio/gateway.py

+    build_shake,
+    build_shakeair,
+    build_singlepress,
+    build_taptap,


I think this is waaaaay too specific to be inside the gateway class. We need a way to expose subdevice based actions some other way.

rytilahti · 2020-06-06T21:20:27Z

miio/gateway.py

@@ -201,6 +218,38 @@ def discover_devices(self):

        return self._devices

+    def x_del(self, script_id):


I agree that this should be named otherwise, especially if this can be useful for other use cases. Otherwise it is probably simpler to simply use self.send where necessary (or at least rename this to something more descriptive, and prefix it with _ to indicate that it's a private method not to be used by others.

rytilahti · 2020-06-06T21:23:19Z

miio/gateway.py

+    )
+    def subdevice_command(self, sid, command, encoded_token):
+        """Send command to subdevice."""
+        self.discover_devices()


Is it necessary to do the discovery for this? If yes, I think it would be better to modify the discover_devices to return a dictionary of {sid: SubDevice}. This would allow dropping the next line and make it much more easier to comprehend.

rytilahti · 2020-06-06T22:19:15Z

miio/gateway.py

+    def uninstall_scripts(self, encoded_token):
+        return dict(
+            map(
+                lambda action: (
+                    action,
+                    (
+                        action_id[action](self.sid),
+                        self._gw.x_del(action_id[action](self.sid)),
+                    ),
+                ),
+                action_id.keys(),
+            )
+        )


This is way coo complicated and rather hard to interpret.

Is the goal to remove all scripts? Is there a way to get a list of current events, that could be iterated and then removed? How does removal/overwriting of a single script look like?

How is the encoded token used?

yes... goal is to remove all scripts at least potentially could be installed in device... As far as i get list of installed scripts is not exposed through miio protocol... Here I xdel all scripts for all possible scripts to avoid memory leak on gw

Encoded token is used during script installation and somehow linked with real token. Unfortunately algo is unknown...

Actually, I think there were some sort of list functionality related to scenes in the gateway implementation at some point, that could be related to this?

Anyway, I pointed out the encoded_token here as it is given as a parameter but never used in the method itself?

rytilahti · 2020-06-06T22:25:47Z

miio/gateway.py

+
+    def install_script(self, sid, builder, encoded_token, ip=None):
+        """Install script for by building script source and sending it with miio method. You need to run fake or real device to capture script execution results."""
+        if ip is None:


I'd like to see this being much simpler, in my opinion the API should be more something like:

gw = Gateway() cube = gw.get_device_by_id("cube id") some_other_device = gw.get_device_by_id("my light") cube.install_event_handler(cube, 'shake', some_other_device, 'some action')

Likewise, if I have a cube instance, I would like to query the list of available events and already installed scripts:

print("available events: %s" % cube.events) # I mention this way of exposing the available events in some other comment # if there is only one script possible, otherwise we need a way to get a list of already installed scripts to make it possible for the user to decide which one to remove cube.remove_script("shake") # the other case handlers = cube.installed_event_handlers for handler in handers: print("handler %s")

rytilahti · 2020-06-06T22:26:38Z

miio/gateway.py

+    properties = []
+
+    @command()
+    def install_singlepress_script(self, encoded_token):


See my other comment about how to expose events to users.

rytilahti · 2020-06-06T22:30:44Z

miio/gateway_scripts.py

+    target_encoded_token,
+    target_model=fake_device_model,
+    target_id=fake_device_id,
+    source_model="lumi.sensor_cube.v1",


Considering that these seem to be passed everywher, I think the best approach is to create a new class, EventHandler could be a possible name. This would contain all the necessary information for the handler, so just to give you an idea:

@attr.s class EventHandler: source_sid = attr.ib() target_ip = attr.ib() target_token = attr.ib() target_model = attr.ib() # is this necessary?! target_did = attr.ib() # should this be `target_sid`? target_model = attr.ib() # is this necessary?

Potentially a lot of these are already known by the target/source devices, so potentially passing the source "device" and target "device" should be enough for making the connections.

rytilahti · 2020-06-06T22:33:23Z

miio/gateway_scripts.py

+            [
+                "1.0",
+                randint(1590161094, 1590162094),
+                [
+                    "0",
+                    {
+                        "did": source_sid,
+                        "extra": extra,
+                        "key": "event." + source_model + "." + event,
+                        "model": source_model,
+                        "src": "device",
+                        "timespan": ["0 0 * * 0,1,2,3,4,5,6", "0 0 * * 0,1,2,3,4,5,6"],
+                        "token": "",
+                    },
+                ],
+                [


This needs to be done using a container class, otherwise it is way too complicated and hard to interpret. See my comment above, I think some of this data can already be stored in the devices what we have, the event handler (or eventscript? or script? I'm not sure) needs to simply have the potential extra fields (like the timespan).

rytilahti · 2022-07-17T17:40:08Z

As #1446 #1459 are merged now, I think this can be closed. Thanks @bskaplou and @starkillerOG!

bskaplou added 2 commits May 28, 2020 13:00

Add support for cube and suqarebutton via fake_device

662e35d

cleanup

485da59

bskaplou mentioned this pull request May 28, 2020

Fake device #703

Closed

starkillerOG reviewed May 28, 2020

View reviewed changes

bskaplou added 5 commits May 29, 2020 01:02

cleanup

0f47461

cleanup

a83ed86

cleanup

2849a2e

cleanup

d7d6367

cleanup

8c23c5b

bskaplou commented May 28, 2020

View reviewed changes

starkillerOG reviewed May 29, 2020

View reviewed changes

cleanup

d8cae75

starkillerOG mentioned this pull request Jun 2, 2020

How to access zigbee peripherals (lumi.gateway.v3) #650

Open

bskaplou and others added 2 commits June 6, 2020 12:56

pass tokens as cli args

360658a

Merge branch 'master' into zigbee_suqarebutton_cube

9df0fd0

bskaplou added 2 commits June 6, 2020 13:07

styling

1862910

Merge branch 'zigbee_suqarebutton_cube' of https://github.com/bskaplo…

1aba320

…u/python-miio into zigbee_suqarebutton_cube

rytilahti reviewed Jun 6, 2020

View reviewed changes

This was referenced Jun 7, 2020

add firmware property + convert device list to dict #719

Merged

gateway: add model property & implement SwitchOneChannel #722

Merged

starkillerOG mentioned this pull request Jun 13, 2020

Gateway: add name + model property to subdevice & add loads of subdevices #724

Merged

starkillerOG mentioned this pull request Jul 26, 2020

New gateway lumi.gateway.mieu01 non discoverable home-assistant/core#22078

Closed

starkillerOG mentioned this pull request Aug 6, 2020

Xiaomi Miio integration fails to discover sensors home-assistant/core#38581

Closed

rytilahti mentioned this pull request Sep 2, 2020

Add support for 'lumi.gateway.mieu01' subdevices #728

Open

starkillerOG mentioned this pull request Sep 30, 2020

Xiaomi gateway - alarm trigger #822

Open

rytilahti mentioned this pull request Oct 9, 2020

intergration xiaomi_miio missing fields or attributes? home-assistant/core#41567

Closed

starkillerOG mentioned this pull request Aug 2, 2021

Xiaomi Miio - lumi.gateway.aqhm01 home-assistant/core#53097

Closed

starkillerOG mentioned this pull request Jan 9, 2022

Implement push server for xioami miio gateways #1288

Closed

rytilahti closed this Jul 17, 2022

		@@ -201,6 +218,38 @@ def discover_devices(self):

		return self._devices

		def x_del(self, script_id):

Zigbee suqare button and cube support via fake_device #709

Zigbee suqare button and cube support via fake_device #709

Conversation

bskaplou commented May 28, 2020 • edited Loading

coveralls commented May 28, 2020 • edited Loading

starkillerOG left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bskaplou May 28, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bskaplou May 29, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bskaplou left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

starkillerOG commented May 29, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bskaplou commented May 29, 2020

starkillerOG commented May 29, 2020

starkillerOG commented May 29, 2020

bskaplou commented May 30, 2020 • edited Loading

bskaplou commented Jun 2, 2020

starkillerOG commented Jun 2, 2020

bskaplou commented Jun 6, 2020

starkillerOG commented Jun 6, 2020

rytilahti left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rytilahti commented Jul 17, 2022

bskaplou commented May 28, 2020 •

edited

Loading

coveralls commented May 28, 2020 •

edited

Loading

bskaplou May 28, 2020 •

edited

Loading

bskaplou May 29, 2020 •

edited

Loading

bskaplou left a comment •

edited

Loading

bskaplou commented May 30, 2020 •

edited

Loading