Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Virgin Media (GB) recording capacity #31

Closed
callumeveratt opened this issue Dec 21, 2022 · 53 comments
Closed

Virgin Media (GB) recording capacity #31

callumeveratt opened this issue Dec 21, 2022 · 53 comments
Assignees
Labels
bug Something isn't working

Comments

@callumeveratt
Copy link

I'm not sure this ever worked on the original Arris integration (certainly for me the value was always unavailable for recording_capacity) but the following appears in the log for getting recording capacity in GB:

Logger: lghorizon.lghorizon_api
Source: /usr/local/lib/python3.10/site-packages/backoff/_common.py:120
First occurred: 10:39:10 (1 occurrences)
Last logged: 10:39:10
Giving up _do_api_call(...) after 3 tries (lghorizon.exceptions.LGHorizonApiConnectionError: Unable to call https://prod.spark.virginmedia.com/eng/web/recording-service/customers/xxx_gb/quota. Error:501 Server Error: Not Implemented for url: https://prod.spark.virginmedia.com/eng/web/recording-service/customers/xxx_gb/quota)

@callumeveratt callumeveratt added the bug Something isn't working label Dec 21, 2022
@Sholofly
Copy link
Owner

Sholofly commented Dec 21, 2022

@callumeveratt I think this works as intended. For me there's no way to determine if cloud recording is available but to query that url. If it gives me a 501 i know it isn't avaiable. I presume you don't have a sensor at all anymore for recording capacity?

@caraar12345
Copy link
Contributor

caraar12345 commented Dec 21, 2022

It looks like VM sends this over MQTT as a percentage --

Topic: {household_id}/{box_id}/localRecordings/capacity
Payload: {"messageType":"CPE.capacity","version":"1.0.7","used":97}

@Sholofly
Copy link
Owner

Ah m8 thats brilliant information. Until now i was only supporting cloud recordings. In this way I can add a sensor for local recordings. How did you find out?

@caraar12345
Copy link
Contributor

All the investigation I've been doing has been just by logging into https://virgintvgo.virginmedia.com with the Chrome dev tools open and watching the calls it makes.

In particular, in the call to mqtt, which is the websocket, you can see the MQTT messages either way!

@Sholofly
Copy link
Owner

Cool, I'm going to try to add this to the api. Thanks!

@Sholofly
Copy link
Owner

BTW, can you find out of anything in your account data states that recording is enabled and if this is network or local?

@callumeveratt
Copy link
Author

Ignore that - I imagine it's not specified anywhere for Virgin GB as channel owners have cold feet over cloud recording - so it's never been enabled in the UK from what I gather

@callumeveratt
Copy link
Author

Would definitely be cool to use the local recording data where applicable though :-)

@caraar12345
Copy link
Contributor

Will also have a dig around -- but with Virgin, I can confidently say it'll always be local as we don't have Cloud DVR!

@callumeveratt I think remote recording in that context is the ability to remotely instruct your box to record something rather than recording to VM's servers

@callumeveratt
Copy link
Author

@caraar12345 - Yeah you're right, I got ahead of myself for a second. It is the ability to remote record, not cloud record.

@caraar12345
Copy link
Contributor

Ha! Impressive timing from both of us 😁

Does look like there's something about local DVR though -- localDvrAvailable. I believe networkDvrAvailable refers to being able to play back local recordings on a mobile device.

Unrelated tidbit, I do wonder where they get 'gender' from, as I may be somewhat effeminate, but I've definitely never identified myself as female to anyone 😆

{
    "oespToken": "",
    "accessToken": "",
    "householdId": "{customer_id}_gb",
    "username": "{customer_id}",
    "locationId": "18",
    "originalLocationId": "18",
    "timeToIdleSeconds": 7200,
    "customer": {
        "givenName": "Aaron",
        "countryCode": "GB",
        "gender": "female",
        "status": [],
        "smartCardId": "",
        "physicalDeviceId": "{box_id}",
        "stbType": "EOS",
        "networkDvrAvailable": true,
        "replayOTTOnlyAvailable": false,
        "localDvrAvailable": true,
        "householdId": "{customer_id}_gb",
        "sharedProfileId": "",
        "replayTvEntitled": true,
        "replayTvAvailable": true,
        "replayTvMinimumBroadcastTime": 0,
        "replayTvOptedIn": true,
        "offlineViewingEntitled": true,
        "profileIds": [
            ""
        ],
        "oboProvisioned": true,
        "customerId": "{customer_id}",
        "purchaseEntitlementsToken": "",
        "isSkLite": false
    },

@Sholofly
Copy link
Owner

From what call is this data coming from?

@caraar12345
Copy link
Contributor

@Sholofly
Copy link
Owner

Are you guys (or guy and lady) able to update and can you check if every box have an attribute with recording capacity?

@caraar12345
Copy link
Contributor

Will give it a shot; unfortunately I'm still contending with the config flow issue (hopefully this restart will sort it!?)

@ChrisThomas42
Copy link

@Sholofly On VirginMedia GB, I have 1 device and 1 entity. No sign of recording capacity anywhere.

@caraar12345
Copy link
Contributor

If you click the box entity, does it show in attributes?

@Sholofly
Copy link
Owner

No you should check the developer tools

@callumeveratt
Copy link
Author

Yep - there for me. Null though - is that expected at the mo?
image

@Sholofly
Copy link
Owner

Depends, do you have local recordings?

@callumeveratt
Copy link
Author

callumeveratt commented Dec 21, 2022

Yep - have recordings on the box and on TVGo I can see the MQTT request @caraar12345 gave

{"messageType":"CPE.capacity","version":"1.0.7","used":72}

@Sholofly
Copy link
Owner

ok, i that case it doesn't work yet. I have to add more logging to find out where the problem is. Do you have debugging enabled in your component?

@callumeveratt
Copy link
Author

Nope - but if you need to add some more logging I can get it turned on in the meantime

@Sholofly
Copy link
Owner

Sholofly commented Dec 21, 2022

OK stupid. I didnt't commit the part to subscribe to the actual mqtt topic :( Can you manually update your manifest.json to

"requirements": [
    "lghorizon>=0.5.13"
  ]

And then reboot of course ;)

@Sholofly
Copy link
Owner

@Sholofly On VirginMedia GB, I have 1 device and 1 entity. No sign of recording capacity anywhere.

The local recordings are an attribute of the box. It's not an account thing like the cloud recordings.

@ChrisThomas42
Copy link

After changing the entry 0.5.12 to 0.5.13 and restarting, I no longer have an entry for recording capacity listed in states. I can see all my local recodings in the integration.

@Sholofly
Copy link
Owner

Do you see recording capacity of each box in the box attributes?

@callumeveratt
Copy link
Author

I updated the manifest also - the recording_capacity did disappear originally but it came back but sadly still null

@Sholofly
Copy link
Owner

Sholofly commented Dec 21, 2022

Can you check the logs for mqtt messages if you set it to debug?

@callumeveratt
Copy link
Author

Just trawling though, so far found these

2022-12-21 15:05:41.002 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Sending SUBSCRIBE (d0, m7) [(b'201434117_gb/+/localRecordings', 0)]

2022-12-21 15:05:41.003 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Sending SUBSCRIBE (d0, m8) [(b'201434117_gb/+/localRecordings/capacity', 0)]

@callumeveratt
Copy link
Author

I see a SUBACK shortly after - but not much else

@Sholofly
Copy link
Owner

If you visit the web page now and force the webpage to retrieve your capacity. Is your HA instance getting an update too?

@ChrisThomas42
Copy link

I've just re-enabled debug and restarted, SOOO much stuff in the logs I need to go through more fully, but this is a start

2022-12-21 15:16:54.468 INFO (SyncWorker_2) [lghorizon.lghorizon_api] Executing API call to https://prod.spark.virginmedia.com/eng/web/recording-service/customers/105210861_gb/quota 2022-12-21 15:16:54.567 DEBUG (SyncWorker_2) [lghorizon.lghorizon_api] Step 1 - Get Authorization data 2022-12-21 15:16:54.739 DEBUG (SyncWorker_2) [lghorizon.lghorizon_api] Step 2 - Get Authorization cookie 2022-12-21 15:16:54.961 DEBUG (SyncWorker_2) [lghorizon.lghorizon_api] Step 3 - Login 2022-12-21 15:16:55.194 DEBUG (SyncWorker_2) [lghorizon.lghorizon_api] Step 4 - Follow redirect 2022-12-21 15:16:55.305 DEBUG (SyncWorker_2) [lghorizon.lghorizon_api] Step 5 - Extract auth code 2022-12-21 15:16:55.305 DEBUG (SyncWorker_2) [lghorizon.lghorizon_api] Auth code: UBoZFR, Auth state: f00a129d-1974-44e4-842c-3f1b478c5ab4 2022-12-21 15:16:55.305 DEBUG (SyncWorker_2) [lghorizon.lghorizon_api] Step 6 - Post auth data with valid code 2022-12-21 15:16:56.001 INFO (SyncWorker_2) [lghorizon.lghorizon_api] Backing off _do_api_call(...) for 1.1s (lghorizon.exceptions.LGHorizonApiConnectionError: Unable to call https://prod.spark.virginmedia.com/eng/web/recording-service/customers/105210861_gb/quota. Error:501 Server Error: Not Implemented for url: https://prod.spark.virginmedia.com/eng/web/recording-service/customers/105210861_gb/quota)

@callumeveratt
Copy link
Author

@ChrisThomas42 I believe that's the cloud recording rather than local - and that error is expected with GB as no cloud recording

@Sholofly - I forced a refresh of my TV Go (if that's what you meant) and I couldn't see anything in the logs around MQTT and the localCapacity topic

@Sholofly
Copy link
Owner

Ok just one last try for today: can you update your manifest to v0.5.14?

@callumeveratt
Copy link
Author

Tried - didn't work originally but thought I'd relink as a test also but actually seems the integration can't connect now:

Logger: custom_components.lghorizon.config_flow
Source: custom_components/lghorizon/config_flow.py:81
Integration: LG Horizon (documentation, issues)
First occurred: 15:38:17 (4 occurrences)
Last logged: 15:38:47
'LGHorizonAuth' object has no attribute 'accessToken'

@callumeveratt
Copy link
Author

2022-12-21 15:40:24.721 DEBUG (SyncWorker_11) [lghorizon.lghorizon_api] Obtain Virgin GB mqtt token...
2022-12-21 15:40:24.725 ERROR (MainThread) [custom_components.lghorizon.config_flow] 'LGHorizonAuth' object has no attribute 'accessToken'

@caraar12345
Copy link
Contributor

caraar12345 commented Dec 21, 2022 via email

@callumeveratt
Copy link
Author

Aha of course - I'll give it a little

@ChrisThomas42
Copy link

so this is a small extract from my log,

2022-12-21 15:38:22.873 INFO (SyncWorker_0) [lghorizon.lghorizon_api] Retrieving recordingcapacity... 2022-12-21 15:38:22.873 INFO (SyncWorker_0) [lghorizon.lghorizon_api] Executing API call to https://prod.spark.virginmedia.com/eng/web/recording-service/customers/*****_gb/quota 2022-12-21 15:38:22.945 DEBUG (SyncWorker_0) [lghorizon.lghorizon_api] Step 1 - Get Authorization data 2022-12-21 15:38:23.119 DEBUG (SyncWorker_0) [lghorizon.lghorizon_api] Step 2 - Get Authorization cookie 2022-12-21 15:38:23.142 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Received SUBACK 2022-12-21 15:38:23.146 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Received SUBACK 2022-12-21 15:38:23.399 DEBUG (SyncWorker_0) [lghorizon.lghorizon_api] Step 3 - Login 2022-12-21 15:38:23.579 DEBUG (SyncWorker_0) [lghorizon.lghorizon_api] Step 4 - Follow redirect 2022-12-21 15:38:23.699 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Received SUBACK 2022-12-21 15:38:23.707 DEBUG (SyncWorker_0) [lghorizon.lghorizon_api] Step 5 - Extract auth code 2022-12-21 15:38:23.707 DEBUG (SyncWorker_0) [lghorizon.lghorizon_api] Auth code: PJ363e, Auth state: 3fed465e-20e8-44a1-86ee-f190c6f72618 2022-12-21 15:38:23.707 DEBUG (SyncWorker_0) [lghorizon.lghorizon_api] Step 6 - Post auth data with valid code 2022-12-21 15:38:23.792 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Received SUBACK 2022-12-21 15:38:23.793 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Received SUBACK 2022-12-21 15:38:23.793 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Received SUBACK 2022-12-21 15:38:24.097 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Received SUBACK 2022-12-21 15:38:24.115 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Received SUBACK 2022-12-21 15:38:24.116 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Received SUBACK 2022-12-21 15:38:24.116 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Received SUBACK 2022-12-21 15:38:24.117 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Received SUBACK 2022-12-21 15:38:24.118 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Received SUBACK 2022-12-21 15:38:24.118 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Received SUBACK 2022-12-21 15:38:24.266 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Received PUBLISH (d0, q0, r0, m0), '******_gb/0QhZJ30DAj/status', ... (72 bytes) 2022-12-21 15:38:24.266 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Received MQTT message. Topic: ******_gb/0QhZJ30DAj/status 2022-12-21 15:38:24.266 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Message: {'source': '0QhZJ30DAj', 'state': 'ONLINE_RUNNING', 'deviceType': 'HGO'} 2022-12-21 15:38:24.285 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Received PUBREC (Mid: 14) 2022-12-21 15:38:24.286 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Sending PUBREL (Mid: 14) 2022-12-21 15:38:24.322 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Received PUBCOMP (Mid: 14) 2022-12-21 15:38:24.402 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Received PUBLISH (d0, q0, r1, m0), '******_gb/6d3aaf2f-721a-4163-a638-ffcbc46e3cbc/status', ... (110 bytes) 2022-12-21 15:38:24.403 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Received MQTT message. Topic: ******_gb/6d3aaf2f-721a-4163-a638-ffcbc46e3cbc/status 2022-12-21 15:38:24.403 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Message: {'mac': '', 'ipAddress': '', 'state': 'OFFLINE', 'source': '6d3aaf2f-721a-4163-a638-ffcbc46e3cbc', 'deviceType': 'HGO'} 2022-12-21 15:38:24.421 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Received PUBLISH (d0, q0, r1, m0), '******_gb/3C36E4-EOSSTB-003469169803/status', ... (152 bytes) 2022-12-21 15:38:24.421 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Received MQTT message. Topic: ******_gb/3C36E4-EOSSTB-003469169803/status 2022-12-21 15:38:24.422 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Message: {'deviceType': 'STB', 'source': '3C36E4-EOSSTB-003469169803', 'state': 'ONLINE_STANDBY', 'mac': '**:**:', 'ipAddress': '192.168.*.*/255.255.255.0'} 2022-12-21 15:38:24.428 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Sending PUBLISH (d0, q2, r0, m15), 'b'******_gb/3C36E4-EOSSTB-003469169803'', ... (66 bytes) 2022-12-21 15:38:24.487 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Received PUBREC (Mid: 15) 2022-12-21 15:38:24.488 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Sending PUBREL (Mid: 15) 2022-12-21 15:38:24.584 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Received PUBCOMP (Mid: 15) 2022-12-21 15:38:24.890 INFO (SyncWorker_0) [lghorizon.lghorizon_api] Backing off _do_api_call(...) for 0.8s (lghorizon.exceptions.LGHorizonApiConnectionError: Unable to call https://prod.spark.virginmedia.com/eng/web/recording-service/customers/******_gb/quota. Error:501 Server Error: Not Implemented for url: https://prod.spark.virginmedia.com/eng/web/recording-service/customers/******_gb/quota) 2022-12-21 15:38:25.686 INFO (SyncWorker_0) [lghorizon.lghorizon_api] Executing API call to https://prod.spark.virginmedia.com/eng/web/recording-service/customers/******_gb/quota

I think(hope) I've removed any potentially sensitive info???? There's more before and after.

@caraar12345
Copy link
Contributor

caraar12345 commented Dec 21, 2022

Woohoo - I finally managed to get the config flow to work so I can chime in!

source_list: 
  - <many_channels>
media_content_type: episode
media_title: BBC Launcher
source: BBC Launcher
entity_picture_local: >-
  /api/media_player_proxy/media_player.virgin_tv?token=
play_mode: app
channel: BBC Launcher
title: BBC Launcher
image: >-
  https://cdn.metrological.com/lightning/apps/com.bbc.app.launcher/1.0.0-a646d9e/static/icon.png
recording_capacity: null
device_class: tv
entity_picture: >-
  https://cdn.metrological.com/lightning/apps/com.bbc.app.launcher/1.0.0-a646d9e/static/icon.png
friendly_name: Virgin TV
supported_features: 154497

No MQTT message is coming through with the recording_capacity which is just... weird.

Interesting note: https://prod.spark.virginmedia.com/eng/web/recording-service/customers/{household_id}/recordings?sort=time&sortOrder=desc&language=en does work. And the data there contains diskSpace for every item. And for me, adding them together gets 96.05% which is reasonable based on the MQTT message reporting 97% (always round up?).

So worst case: iterate through the recordings and add the disk spaces!???

@callumeveratt
Copy link
Author

Apologies as been out this evening, just got back and managed to readd. Still no luck and either no value reported for recording or a null value

@Sholofly
Copy link
Owner

Sholofly commented Jan 3, 2023

https://prod.spark.virginmedia.com/eng/web/recording-service/customers/{household_id}/recordings?sort=time&sortOrder=desc&language=en

@caraar12345 We have this call too. I results in a list AND a summary of quota at the end. Isn't that the case here?

@callumeveratt
Copy link
Author

@Sholofly - just realized we all went quiet on this. I have a bit of time this evening so I can have a look into some of the calls ect and see if I can tweak it to work on my HA if you want :)

@Sholofly
Copy link
Owner

@callumeveratt Sorry, I'm very busy lately. Please let me know when you have time. Maybe we can figure this out soon :)

@ColinRobbins
Copy link
Contributor

I've started to look at this, and think my findings are the same as above.
In GB, the quota URL does not exists, but the recordings one works just fine.

Using the Virgin web interface, I've used both Chrome developer tools and burp to capture HTTP traffic and there is no call to quota, but the webpage does get the data somehow.

image

Looking at the JS pages (hard because of the obfuscation) there is code that looks for .quota and .occupied and so I assume it is getting in from somewhere.

I can see some MQTT calls in Chrome dev tools, but the messages are only 2 bytes of binary data as far as I can see.
Maybe I'm not capturing the MQTT correctly - not sure, I not that experienced with MQTT.

I can't see any indication the code is iterating the recordings and adding it up.

Should we (at least) put some code that says "If GB don't make the quota call"?

@ColinRobbins
Copy link
Contributor

Ahh, just found the following MQTT message...

00000000: aaaa aaaa  aaaa aaaa aaaa aaaa aaaa          XXXX-XXXXXX
00000001: aaaa aaaa  aaaa aaaa aaaa aaaa aaaa          XXXX-XXXXXX
00000002: aaaa aaaa  aaaa aaaa aaaa aaaa aaaa           XXXXX/loca
00000003: 6c52 6563 6f72 6469 6e67 732f 6361 7061  lRecordings/capa
00000004: 6369 7479 0066 7b22 6d65 7373 6167 6554  city.f{"messageT
00000005: 7970 6522 3a22 4350 452e 6361 7061 6369  ype":"CPE.capaci
00000006: 7479 222c 2276 6572 7369 6f6e 223a 2231  ty","version":"1
00000007: 2e30 2e37 222c 2275 7365 6422 3a36 7d    .0.7","used":6}

which contains the "used:6" - which is the correct - 6% disk used.

@Sholofly if you can give me a few pointers on what this might mean, and where to modify the code, I'll give it a go.

@Sholofly
Copy link
Owner

Sorry my time is limited nowadays so I try to respond as quickly as possible. Let me start to thank you for your dedication!

I'm from NL and the Ziggo boxes don't use local recordings but cloud recordings. That's why my capacity is exposed using the API.
If the box want's to cummincate to the outside world it uses the MQTT protocol. I'm not sure if you're familiar with this protocol but its a massaging protocol where you can subscribe on topics to receive messages (or send messages with a topic of course)

More info: https://mqtt.org/

What you see in your example is the fact that a message is received on the /localRecordings/capacity topic. With the used percentage of your box.

I think the subscription already is in place (https://github.com/Sholofly/lghorizon-python/blob/2fa57cc4ea5decff549db3fa7f952ef9403bd018/lghorizon/models.py#L363)

And the processing is already in place too:
https://github.com/Sholofly/lghorizon-python/blob/2fa57cc4ea5decff549db3fa7f952ef9403bd018/lghorizon/lghorizon_api.py#L349

So probably my cloud update is frustrating the localrecording, or there's something wrong in my processing.

Probably you can figure that one out. I'd really like to hep you, but since I don't have a UK account with local recordings, that's kind of difficult for me...

@ColinRobbins
Copy link
Contributor

@Sholofly thank you for the quick reply, I think I can see how to make that work.
I just have one more quick question if I can.

The MQTT subscription to local/capacity will presumably only send a message when the capacity changes.

Is there a way to force a "read" of the current status via MQTT?

I appreciate your help.

@Sholofly
Copy link
Owner

I'm not sure, probably you can inspect your browser for the answer. Since you saw the message appear, probably the previous message is the trigger? Or it's send automatically on subscription?

@ColinRobbins
Copy link
Contributor

ColinRobbins commented Oct 13, 2024

Pull request created to inhibit error messages.

The quota sensor seems focused on web recordings and is not available in GB.
Each media_player has a capacity attribute anyway.

@ColinRobbins
Copy link
Contributor

Pull request accepted in v0.6.3.
Close issue?

@Sholofly
Copy link
Owner

If solved.... yes!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

5 participants