Better device proxy cache

[hardillb]

part of #3414

Description

Rather than requesting the 4 largest files from the device agent every time the device editor is loaded it will load them from a local instance on @node-red/editor-client of the matching version if present.

Still need to work out how to install the right versions of @node-red/editor

Related Issue(s)

#3414

Checklist

• I have read the contribution guidelines
• Suitable unit/system level tests have been added and they pass
• Documentation has been updated
  • Upgrade instructions
  • Configuration details
  • Concepts
• Changes flowforge.yml?
  • Issue/PR raised on FlowFuse/helm to update ConfigMap Template
  • Issue/PR raised on FlowFuse/CloudProject to update values for Staging/Production

Labels

• Includes a DB migration? -> add the area:migration label

[codecov]

Codecov Report

Attention: Patch coverage is 54.83871% with 14 lines in your changes missing coverage. Please review.

Project coverage is 78.59%. Comparing base (33ac8f7) to head (3b83122).
Report is 11 commits behind head on main.

Files with missing lines	Patch %	Lines
forge/ee/lib/deviceEditor/DeviceTunnelManager.js	45.45%	12 Missing ⚠️
forge/db/controllers/Device.js	50.00%	1 Missing ⚠️
...db/migrations/20241218-01-add-nr-version-device.js	80.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4792      +/-   ##
==========================================
- Coverage   78.64%   78.59%   -0.05%     
==========================================
  Files         324      325       +1     
  Lines       15265    15296      +31     
  Branches     3507     3512       +5     
==========================================
+ Hits        12005    12022      +17     
- Misses       3260     3274      +14     

Flag	Coverage Δ
backend	78.59% <54.83%> (-0.05%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[hardillb]

@Steve-Mcl the default path for a localfs test is var/device/cache in the flowfuse directory (same place as var/stacks and var/projects for localfs)

And @node-red/editor-client needs to be installed under that in the matching Node-RED version number (like stacks)

e.g. in var/device/cache create 4.0.5 and run npm install @node-red/editor-client@4.0.5 --prefix 4.0.5

Also needs the head of device agent to work

[Steve-Mcl]

@Steve-Mcl the default path for a localfs test is var/device/cache in the flowfuse directory (same place as var/stacks and var/projects for localfs)

And @node-red/editor-client needs to be installed under that in the matching Node-RED version number (like stacks)

e.g. in var/device/cache create 4.0.5 and run npm install @node-red/editor-client@4.0.5 --prefix 4.0.5`

Also needs the head of device agent to work

Thanks Ben. First job for me this morning will be to pull and test this locally.

[Steve-Mcl]

Ben, this review is next on my list but before I pull and test, can you update the date on the migration please - there have been other migrations since Nov 15th.

Thanks.

[hardillb]

@Steve-Mcl done (this will probably break the pre-staging env)

[Steve-Mcl]

@Steve-Mcl done (this will probably break the pre-staging env)

@ppawlowski is it possible to delete / refresh the pre-staging for this?

@hardillb will this affect staging?

[hardillb]

@Steve-Mcl we have already deleted the pre-staging env, we did not recreate it. It can't be tested on pre-staging as there is no broker for the device comms needed to enable device editor mode.

At the moment we are not deploying the @node-red/editor-client packages in the containers so this will run as "normal" and not use the cached copies. The update to the containers will get done later.

[Steve-Mcl]

As discussed in slack, I added the try-catch and fall through to original tunneling logic.

This works fantastically well Ben I gotta say, its like night & day!

For the record, I added temporary logging to ensure things were working as expected, here are the results:

Serving cached file: C:\Users\xxx\repos\github\flowfuse\dev-env\packages\flowfuse\var\device\cache\4.0.7\node_modules\@node-red\editor-client\public\red\style.min.css
Serving cached file: C:\Users\xxx\repos\github\flowfuse\dev-env\packages\flowfuse\var\device\cache\4.0.7\node_modules\@node-red\editor-client\public\vendor\vendor.js
Serving cached file: C:\Users\xxx\repos\github\flowfuse\dev-env\packages\flowfuse\var\device\cache\4.0.7\node_modules\@node-red\editor-client\public\red\red.min.js
Serving cached file: C:\Users\xxx\repos\github\flowfuse\dev-env\packages\flowfuse\var\device\cache\4.0.7\node_modules\@node-red\editor-client\public\vendor\monaco\dist\editor.js
Serving cached file: C:\Users\xxx\repos\github\flowfuse\dev-env\packages\flowfuse\var\device\cache\4.0.7\node_modules\@node-red\editor-client\public\red\style.min.css
Serving cached file: C:\Users\xxx\repos\github\flowfuse\dev-env\packages\flowfuse\var\device\cache\4.0.7\node_modules\@node-red\editor-client\public\vendor\vendor.js
Serving cached file: C:\Users\xxx\repos\github\flowfuse\dev-env\packages\flowfuse\var\device\cache\4.0.7\node_modules\@node-red\editor-client\public\red\red.min.js
Serving cached file: C:\Users\xxx\repos\github\flowfuse\dev-env\packages\flowfuse\var\device\cache\4.0.7\node_modules\@node-red\editor-client\public\vendor\monaco\dist\editor.js
[2024-12-18T12:20:59.777Z] INFO: Device 36LO0L3Odb tunnel id:1 - new editor connection req:54 
Serving cached file: C:\Users\xxx\repos\github\flowfuse\dev-env\packages\flowfuse\var\device\cache\4.0.7\node_modules\@node-red\editor-client\public\vendor\mermaid\mermaid.min.js
Serving cached file: C:\Users\xxx\repos\github\flowfuse\dev-env\packages\flowfuse\var\device\cache\4.0.7\node_modules\@node-red\editor-client\public\vendor\monaco\dist\css.worker.js
Serving cached file: C:\Users\xxx\repos\github\flowfuse\dev-env\packages\flowfuse\var\device\cache\4.0.7\node_modules\@node-red\editor-client\public\vendor\monaco\dist\ts.worker.js