feat(app): Update robots from USB flash drive (#13923)

* feat(app-shell-odd): watch for USB drives

The Flex operating system automatically mounts the filesystems of
well-formatted USB drives (FAT and ext4 and maybe ntfs but that's a bit
iffy) to /media when those USB drives are inserted on the robot. In
theory it will in fact do this for _any_ kind of media that presents a
filesystem interface.

To that end, add a node task that will use a node filesystem watch to
keep an eye on /media, and
- when something that looks like a USB drive (/media/sd\w\d+) appears,
notify via redux actions
   - then enumerate all the files on it and notify those via redux
   actions
- when something we were keeping an eye on disappears, notify via
redux actions

The redux actions don't alter state and so don't need new reducers or
selectors; they exist because it's a handy mechanism to talk between our
components.

This code is very tightly coupled to the way the node fs interfaces work
and so I don't see a lot of point in unit tests for it; it's almost
entirely fs calls originating everything and providing all of the data,
and all the complexity is from working around weirdnesses in those calls
and in the underlying system. For instance,
- There's a little bit of time in between when the fs watch on /media
fires and when you can actually find the contents of the newly-present
directory; if you readdir before that you'll get an empty list, so we
wait a second
- The node fs.watch interface looks very fully features but is
absolutely chock-full of warnings about various features not being
reliable. A lot of that unreliability is _probably_ across systems and
everything works as we expect on linux, but just in case we have a lot
of fallbacks for if our callback doesn't get filepaths, etc

* fix(app-shell-odd): handle errors in readstreams in http.post

We have our custom http interface that wraps around node-fetch that
provides things like "doing your own read stream when posting a file",
and "mapping everything into the promise interface", which is nice,
but has an issue specifically for that read stream: we don't monitor
errors on it. Read streams surface errors by emitting an 'error' event;
we hook up a listener to that error event _while we're creating the
stream_, but then we disconnect it. So if you have an error in the
stream - for instance, you're reading from a file on a USB flash drive
and the user unplugs the flash drive - then the error will never get
surfaced.

Unfortunately the fix to this is a bit fiddly. We can hook up an error
listener fine, but it needs to do something; specifically, it needs to
turn the error from a callback into a promise rejection. That means it
needs to have a promise to reject that has the same lifetime as the
stream itself. http.post didn't provide that because it returns a whole
big promise chain, and each time you move a link in that chain the old
promise is gone and a new one happens, so we'd need to move the listener
around.

Since promises are monadic, a better fix is to have post return a single
promise and do all the promise chaining _inside_ that promise; then, the
read stream error handler can reject the outer promise directly, while
relying on promises bubbling up rejections to preserve error handling
capability for the promises in the internal chain.

* fix(app): Poll for updates on the ODD

Though we have everything set up to automatically fetch, prompt for, and
execute robot updates from the ODD, we weren't actually _checking_ for
those updates except once on boot (which then wouldn't work if the robot
wasn't internet-connected during boot). This means in particular that
the software updates during onboarding were guaranteed to fail.

We can use the same hook in the ODD app root that we do in the desktop
app route, but if we're going to do that then we better remove a log
message that suddenly becomes extremely spammy.

* feat(app-shell-odd): Supply "system updates" from flash drives

Adds the capability to provide system updates from flash drives to the
ODD app-shell.

These are "system updates" in that the app-shell determines their
availability and provides it to the app, rather than the user indicating
the presence of a file alongside their intent to update. The app-shell
will advertise the flash drive updates in the same way it advertises
internet-discovered updates, with a RobotUpdateInfo redux message; since
those now provide the path to the file they mean, it will be easy for
the app to specify the system update to load.

We can duplicate the logic that we use for system updates by adding a
second let cache for the "current update"; the system-updates code will
then prefer an update in the mass storage update cache to an update in
the old system updates cache, and send new robot update info messages in
all the state changes between neither cache being full; either cache
being full; and both caches being full.

The determination that a flash drive system update is present is
triggered by a mass storage enumerated message; when that flash drive
gets removed, we'll get a removal message.

To figure out whether updates are actually present, we can the list of
files that just got enumerated for things that end with .zip, and then
try to open them as zip files and read the VERSION.json information out
of them. This is a somewhat fraught process; the file could not be a zip
file, it could be a zip file but corrupted, it could be a zip file but
not an update, it could be an update but it's for an OT-2,  and we need
to handle all that, so there's a pretty excessive amount of error
handling in here. Once we're sure that there are one or more zip files
containing robot system updates, we can provide something to redux; we
provide the highest-version update present.

There is one way in which updates from flash drives differ from system
updates found on the internet, however: plugging in a flash drive
requires user intent, while checking for updates on the internet
doesn't. Therefore, if the user plugs in a flash drive with an update
file, we always want to make that update file available no matter the
relative versions of the robot and the update file. So we can add a bool
to the system update message (and then to the update state) that shows
that this is a "forced notification" update, and the app can know to
display it without caring about the upgrade/downgrade/reinstall state.

Since there's a lot of duplication, we can also factor out some common
logic to make it feel a little better.

That process of duplication also fixes a bug that would have prevented
the ODD from ever prompting for updates. The function that gets
information about updates used the same promise to read the release
notes and provide the update information; but we overrode the downloaded
release files to null out the release notes, meaning that promise would
always fail, and we'd never get the notification. We no longer override
the release notes to be null, and we also treat reading the release
notes separately from reading the rest of the update.

* feat(app): allow robot updates from USB files

Now that the odd app-shell provides us with notifications of updates
from USB flash drives, we can allow the user to install them. While the
redux mechanisms allow this pretty easily - a system update is a system
update, after all, and with the force mechanism the app wouldn't even
know if the update was a downgrade or anything - we ran into a problem
where the general robot update machinery in the ODD was very tightly
bound with the onboarding experience for the ODD, since that's the
context in which it was developed.

This commit extracts the robot update mechanisms from onboarding by
- Hoisting onboarding-related logic out of lower level components and
instead injecting that logic into the organisms code from the top level
page
- Moving the current update page to a new one that is focused on
onboarding at a new route, and copying just the update-related code to
a generic RobotUpdate page

This means that the two pages - RobotUpdate and
RobotUpdateDuringOnboarding - share most of the same code but are bound
to different routes and can have different top level behavior by
injecting different contexts to the finish and error handling behaviors
of the update. RobotUpdateDuringOnboarding sets the unfinished
onboarding page breadcrumbs appropriately, and uses display language
appropriate to the update being just a component of the larger workflow,
and moves on to estop handling when cancelled; RobotUpdate doesn't touch
any of that, and goes back to the settings page when cancelled, and uses
wording more appropriate to being its own topline flow.

Closes RAUT-829

app-shell-odd/src/http.ts

@@ -90,20 +90,28 @@ export function postFile(
   name: string,
   source: string
 ): Promise<Response> {
-  return createReadStream(source).then(readStream => {
-    const body = new FormData()
-    body.append(name, readStream)
-    return fetch(input, { body, method: 'POST' })
+  return new Promise<Response>((resolve, reject) => {
+    createReadStream(source, reject).then(readStream =>
+      new Promise<Response>(resolve => {
+        const body = new FormData()
+        body.append(name, readStream)
+        resolve(fetch(input, { body, method: 'POST' }))
+      }).then(resolve)
+    )
   })
 }
 
 // create a read stream, handling errors that `fetch` is unable to catch
-function createReadStream(source: string): Promise<Readable> {
+function createReadStream(
+  source: string,
+  onError: (error: unknown) => unknown
+): Promise<Readable> {
   return new Promise((resolve, reject) => {
     const readStream = fs.createReadStream(source)
     const scheduledResolve = setTimeout(handleSuccess, 0)
 
     readStream.once('error', handleError)
+    readStream.once('error', onError)
 
     function handleSuccess(): void {
       readStream.removeListener('error', handleError)

app-shell-odd/src/main.ts

@@ -21,6 +21,7 @@ import {
   ODD_DIR,
 } from './config'
 import systemd from './systemd'
+import { watchForMassStorage } from './usb'
 
 import type { BrowserWindow } from 'electron'
 import type { Dispatch, Logger } from './types'
@@ -106,6 +107,8 @@ function startUp(): void {
   ipcMain.once('dispatch', () => {
     systemd.sendStatus('started')
     systemd.ready()
+    const stopWatching = watchForMassStorage(dispatch)
+    ipcMain.once('quit', stopWatching)
   })
 }
 

app-shell-odd/src/system-update/index.ts

@@ -2,6 +2,8 @@
 import path from 'path'
 import { ensureDir } from 'fs-extra'
 import { readFile } from 'fs/promises'
+import StreamZip from 'node-stream-zip'
+import Semver from 'semver'
 import { UI_INITIALIZED } from '@opentrons/app/src/redux/shell/actions'
 import { createLogger } from '../log'
 import {
@@ -23,15 +25,18 @@ import type { Action, Dispatch } from '../types'
 import type { ReleaseSetFilepaths } from './types'
 
 const log = createLogger('systemUpdate/index')
+const REASONABLE_VERSION_FILE_SIZE_B = 4096
 
 let isGettingLatestSystemFiles = false
-let updateSet: ReleaseSetFilepaths | null = null
+const isGettingMassStorageUpdatesFrom: Set<string> = new Set()
+let massStorageUpdateSet: ReleaseSetFilepaths | null = null
+let systemUpdateSet: ReleaseSetFilepaths | null = null
 
 const readFileInfoAndDispatch = (
   dispatch: Dispatch,
   fileName: string,
   isManualFile: boolean = false
-): Promise<void> =>
+): Promise<unknown> =>
   readUserFileInfo(fileName)
     .then(fileInfo => ({
       type: 'robotUpdate:FILE_INFO' as const,
@@ -48,6 +53,7 @@ const readFileInfoAndDispatch = (
     .then(dispatch)
 
 export function registerRobotSystemUpdate(dispatch: Dispatch): Dispatch {
+  log.info(`Running robot system updates storing to ${getSystemUpdateDir()}`)
   return function handleAction(action: Action) {
     switch (action.type) {
       case UI_INITIALIZED:
@@ -105,7 +111,8 @@ export function registerRobotSystemUpdate(dispatch: Dispatch): Dispatch {
         break
       }
       case 'robotUpdate:READ_SYSTEM_FILE': {
-        const systemFile = updateSet?.system
+        const systemFile =
+          massStorageUpdateSet?.system ?? systemUpdateSet?.system
         if (systemFile == null) {
           return dispatch({
             type: 'robotUpdate:UNEXPECTED_ERROR',
@@ -114,11 +121,154 @@ export function registerRobotSystemUpdate(dispatch: Dispatch): Dispatch {
         }
         // eslint-disable-next-line @typescript-eslint/no-floating-promises
         readFileInfoAndDispatch(dispatch, systemFile)
+        break
       }
+      case 'shell:ROBOT_MASS_STORAGE_DEVICE_ENUMERATED':
+        if (isGettingMassStorageUpdatesFrom.has(action.payload.rootPath)) {
+          return
+        }
+        isGettingMassStorageUpdatesFrom.add(action.payload.rootPath)
+        getLatestMassStorageUpdateFiles(action.payload.filePaths, dispatch)
+          .then(() => {
+            isGettingMassStorageUpdatesFrom.delete(action.payload.rootPath)
+          })
+          .catch(() => {
+            isGettingMassStorageUpdatesFrom.delete(action.payload.rootPath)
+          })
+        break
+      case 'shell:ROBOT_MASS_STORAGE_DEVICE_REMOVED':
+        if (
+          massStorageUpdateSet !== null &&
+          massStorageUpdateSet.system.startsWith(action.payload.rootPath)
+        ) {
+          console.log(
+            `Mass storage device ${action.payload.rootPath} removed, reverting to non-usb updates`
+          )
+          massStorageUpdateSet = null
+          getCachedSystemUpdateFiles(dispatch)
+        } else {
+          console.log(
+            `Mass storage device ${action.payload.rootPath} removed but this was not an update source`
+          )
+        }
+        break
     }
   }
 }
 
+const getVersionFromOpenedZipIfValid = (zip: StreamZip): Promise<string> =>
+  new Promise((resolve, reject) =>
+    Object.values(zip.entries()).forEach(entry => {
+      if (
+        entry.isFile &&
+        entry.name === 'VERSION.json' &&
+        entry.size < REASONABLE_VERSION_FILE_SIZE_B
+      ) {
+        const contents = zip.entryDataSync(entry.name).toString('ascii')
+        try {
+          const parsedContents = JSON.parse(contents)
+          if (parsedContents?.robot_type !== 'OT-3 Standard') {
+            reject(new Error('not a Flex release file'))
+          }
+          const fileVersion = parsedContents?.opentrons_api_version
+          const version = Semver.valid(fileVersion)
+          if (version === null) {
+            reject(new Error(`${fileVersion} is not a valid version`))
+          } else {
+            resolve(version)
+          }
+        } catch (error) {
+          reject(error)
+        }
+      }
+    })
+  )
+
+interface FileDetails {
+  path: string
+  version: string
+}
+
+const getVersionFromZipIfValid = (path: string): Promise<FileDetails> =>
+  new Promise((resolve, reject) => {
+    const zip = new StreamZip({ file: path, storeEntries: true })
+    zip.on('ready', () => {
+      getVersionFromOpenedZipIfValid(zip)
+        .then(version => {
+          zip.close()
+          resolve({ version, path })
+        })
+        .catch(err => {
+          zip.close()
+          reject(err)
+        })
+    })
+    zip.on('error', err => {
+      zip.close()
+      reject(err)
+    })
+  })
+
+const fakeReleaseNotesForMassStorage = (version: string): string => `
+# Opentrons Robot Software Version ${version}
+
+This update is from a USB mass storage device connected to your flex, and release notes cannot be shown.
+`
+
+export const getLatestMassStorageUpdateFiles = (
+  filePaths: string[],
+  dispatch: Dispatch
+): Promise<unknown> =>
+  Promise.all(
+    filePaths.map(path =>
+      path.endsWith('.zip')
+        ? getVersionFromZipIfValid(path).catch(() => null)
+        : new Promise<null>(resolve => {
+            resolve(null)
+          })
+    )
+  ).then(values => {
+    const update = values.reduce(
+      (prev, current) =>
+        prev === null
+          ? current === null
+            ? prev
+            : current
+          : current === null
+          ? prev
+          : Semver.gt(current.version, prev.version)
+          ? current
+          : prev,
+      null
+    )
+    if (update === null) {
+      console.log('no updates found in mass storage device')
+    } else {
+      console.log(`found update to version ${update.version} on mass storage`)
+      const releaseNotes = fakeReleaseNotesForMassStorage(update.version)
+      massStorageUpdateSet = { system: update.path, releaseNotes }
+      dispatchUpdateInfo(
+        { version: update.version, releaseNotes, force: true },
+        dispatch
+      )
+    }
+  })
+
+const dispatchUpdateInfo = (
+  info: { version: string | null; releaseNotes: string | null; force: boolean },
+  dispatch: Dispatch
+): void => {
+  const { version, releaseNotes, force } = info
+  dispatch({
+    type: 'robotUpdate:UPDATE_INFO',
+    payload: { releaseNotes, version, force, target: 'flex' },
+  })
+  dispatch({
+    type: 'robotUpdate:UPDATE_VERSION',
+    payload: { version, force, target: 'flex' },
+  })
+}
+
 // Get latest system update version
 //   1. Ensure the system update directory exists
 //   2. Get the manifest file from the local cache
@@ -129,9 +279,12 @@ export function registerRobotSystemUpdate(dispatch: Dispatch): Dispatch {
 export function getLatestSystemUpdateFiles(
   dispatch: Dispatch
 ): Promise<unknown> {
-  const fileDownloadDir = path.join(getSystemUpdateDir(), getLatestVersion())
+  const fileDownloadDir = path.join(
+    getSystemUpdateDir(),
+    'robot-system-updates'
+  )
 
-  return ensureDir(fileDownloadDir)
+  return ensureDir(getSystemUpdateDir())
     .then(() => getLatestSystemUpdateUrls())
     .then(urls => {
       if (urls === null) {
@@ -151,11 +304,13 @@ export function getLatestSystemUpdateFiles(
         if (size !== null) {
           const percentDone = Math.round((downloaded / size) * 100)
           if (Math.abs(percentDone - prevPercentDone) > 0) {
-            dispatch({
-              // TODO: change this action type to 'systemUpdate:DOWNLOAD_PROGRESS'
-              type: 'robotUpdate:DOWNLOAD_PROGRESS',
-              payload: { progress: percentDone, target: 'flex' },
-            })
+            if (massStorageUpdateSet === null) {
+              dispatch({
+                // TODO: change this action type to 'systemUpdate:DOWNLOAD_PROGRESS'
+                type: 'robotUpdate:DOWNLOAD_PROGRESS',
+                payload: { progress: percentDone, target: 'flex' },
+              })
+            }
             prevPercentDone = percentDone
           }
         }
@@ -165,37 +320,62 @@ export function getLatestSystemUpdateFiles(
         .then(filepaths => {
           return cacheUpdateSet(filepaths)
         })
-        .then(({ version, releaseNotes }) => {
-          dispatch({
-            type: 'robotUpdate:UPDATE_INFO',
-            payload: { releaseNotes, version, target: 'flex' },
-          })
-          dispatch({
-            type: 'robotUpdate:UPDATE_VERSION',
-            payload: { version, target: 'flex' },
-          })
-        })
+        .then(
+          updateInfo =>
+            massStorageUpdateSet === null &&
+            dispatchUpdateInfo({ force: false, ...updateInfo }, dispatch)
+        )
         .catch((error: Error) => {
           return dispatch({
             type: 'robotUpdate:DOWNLOAD_ERROR',
             payload: { error: error.message, target: 'flex' },
           })
         })
         .then(() =>
-          cleanupReleaseFiles(getSystemUpdateDir(), getLatestVersion())
+          cleanupReleaseFiles(getSystemUpdateDir(), 'robot-system-updates')
         )
         .catch((error: Error) => {
           log.warn('Unable to cleanup old release files', { error })
         })
     })
 }
 
+export function getCachedSystemUpdateFiles(
+  dispatch: Dispatch
+): Promise<unknown> {
+  if (systemUpdateSet) {
+    return getInfoFromUpdateSet(systemUpdateSet)
+      .then(updateInfo =>
+        dispatchUpdateInfo({ force: false, ...updateInfo }, dispatch)
+      )
+      .catch(err => console.log(`Could not get info from update set: ${err}`))
+  } else {
+    dispatchUpdateInfo(
+      { version: null, releaseNotes: null, force: false },
+      dispatch
+    )
+    return new Promise(resolve => resolve('no files'))
+  }
+}
+
+function getInfoFromUpdateSet(
+  filepaths: ReleaseSetFilepaths
+): Promise<{ version: string; releaseNotes: string | null }> {
+  const version = getLatestVersion()
+  const releaseNotesContentPromise = filepaths.releaseNotes
+    ? readFile(filepaths.releaseNotes, 'utf8')
+    : new Promise<string | null>(resolve => resolve(null))
+  return releaseNotesContentPromise
+    .then(releaseNotes => ({
+      version: version,
+      releaseNotes,
+    }))
+    .catch(() => ({ version: version, releaseNotes: '' }))
+}
+
 function cacheUpdateSet(
   filepaths: ReleaseSetFilepaths
-): Promise<{ version: string; releaseNotes: string }> {
-  updateSet = filepaths
-  return readFile(updateSet.releaseNotes, 'utf8').then(releaseNotes => ({
-    version: getLatestVersion(),
-    releaseNotes,
-  }))
+): Promise<{ version: string; releaseNotes: string | null }> {
+  systemUpdateSet = filepaths
+  return getInfoFromUpdateSet(systemUpdateSet)
 }

app-shell-odd/src/system-update/release-files.ts

@@ -55,8 +55,6 @@ export function downloadReleaseFiles(
   onProgress: (progress: DownloadProgress) => unknown
 ): Promise<ReleaseSetFilepaths> {
   const tempDir: string = tempy.directory()
-  // @ts-expect-error delete this when the OT-3 manifest has release notes
-  urls.releaseNotes = null
   const tempSystemPath = outPath(tempDir, urls.system)
   const tempNotesPath = outPath(tempDir, urls.releaseNotes ?? '')
 
@@ -67,12 +65,14 @@ export function downloadReleaseFiles(
   const systemReq = fetchToFile(urls.system, tempSystemPath, { onProgress })
   const notesReq = urls.releaseNotes
     ? fetchToFile(urls.releaseNotes, tempNotesPath)
-    : Promise.resolve('')
+    : Promise.resolve(null)
 
   return Promise.all([systemReq, notesReq]).then(results => {
     const [systemTemp, releaseNotesTemp] = results
     const systemPath = outPath(directory, systemTemp)
-    const notesPath = outPath(directory, releaseNotesTemp)
+    const notesPath = releaseNotesTemp
+      ? outPath(directory, releaseNotesTemp)
+      : null
 
     log.debug('renaming directory', { from: tempDir, to: directory })