Use the "File System Access API" to be able to "remember" last ZIM file

[mossroy]

It is a popular demand, and was not technically possible (except on some specific devices that have specific APIs : Firefox OS and Windows Mobile).

This new API should allow to re-open the last ZIM file after closing/reopening kiwix-js.
It's very recent, and not supported by a lot of browsers (currently only Chrome >=86). Firefox is unfortunately not actively working on it for now. See https://caniuse.com/native-filesystem-api

It should be tested if it also works inside a browser extension

[Jaifroid]

It works if the site is part of the Origin Trial (I did sign up for that for kiwix.github.io and have provided some feedback to Google). See the implementation in Kiwix JS Windows here (also some screenshots):

kiwix/kiwix-js-pwa#94

I discuss some limitations in that issue, but generally it has a native feel (a few more prompts than native). It works best in a PWA when the user installs the PWA, otherwise the user would need to visit the website each time, while a PWA is an offline version of that.

[mossroy]

Excellent!
Sorry, I had forgotten that you already worked on it. That's really cool.
Do you know if it works in a browser extension?

[Jaifroid]

Hmm, I'm not sure (I never implemented an extension version of KJSW because I feel it's properly covered by Kiwix JS, and it would be confusing to have two different extensions built on essentially the same codebase). My understanding is that the API requires https:// and that the chrome-extension:// protocol used by Chrome/Edge (since Mozilla doesn't support the API at all yet) does not qualify. However, I never tested that understanding. I think this is an API designed to work particularly with PWAs. It is not difficult (given that we have a Service Worker) to turn Kiwix JS into a PWA -- it's just a matter of getting the app to cache its own code using the Cache API we already use in SW, and then deciding on the update strategy. But that's #388, which I think would need to be implemented here before this issue.

[Jaifroid]

The Origin Trial token for kiwix.github.io seems to have finished or the token has expired (at least on Edge Chromium, for which it was working fine until very recently). :-(
I'll see if I can renew it for continued testing.
EDIT: Actually the trial is over and the API has changed (quite radically!)...

[Jaifroid]

I've now updated KJSW to use the finalized version of the API. Quite a few things had changed from the trial. I can also confirm that for a completely fresh install of both Chrome and Edge in Windows sandbox, the Native Filesystem API is used out-of-the-box in the PWA without the user having set any flags.

To test, visit: https://kiwix.github.io/kiwix-js-windows/ . If you have ever visited that version before, you will probably need to ensure the PWA Cache is updated to see the latest functional code. Easiest way is to open Dev Tools, go to Application -> Service Workers. Click Unregister next to the KJSW Service Worker (pwabuilder-sw.js). Then force reload (with Dev Tools still open), followed by normal reload.

[I need to write a proper code updating routine, but the PWA is currently only used for testing experimental features.]

[kelson42]

Here a user feedback about this:

To click on Configure and specify the .zim file takes a long time.
(I have 6 of them.)

I would like to be able to bookmark a particular Kiwix configuration
setting -- in other words, a dhoice of .zim file -- so I could quickly
open a particular encyclopedia or dictionary.  Can you implement that?

Alternatively, the Kiwix icon could handle right-click by
popping up a menu showing all the .zim files that have 
ever been used on this machine and still exist.
That manu would make it quick to reselect any one of them.
Can you implement that?

[Jaifroid]

@kelson42: Yes, as @mossroy said at the head of this issue, it is a popular request. This can be achieved in two ways currently:

1. Add logic to make Kiwix JS into an installable PWA (since the API requires https://) with an offline-first caching strategy, and then the File System Access API can be used relatively easily (with or without installing it); or

2. Package the app in a framework (Electron or NWJS; there are some others such as React Native), each of which has built-in methods for accessing the File System.

These both offer different APIs for accessing the File System. In the case of File System Access, a file handle is requested and this is serialized in IndexedDB. It can be accessed later, but it will ask the user for permission, so two simple clicks are required to re-open a file, but it is not necessary to re-pick the file. In Electron and NWJS, a path is added to the File object in JS, which allows the file to be accessed across sessions (with no further clicks).

The File System Access API is only currently available in Chrome and Edge. As for NWJS and Electron, these are Chromium platforms too, but come with a complete copy of Chrome embedded, which means that the app is quite heavy compared to a PWA, so it makes sense more in the context of packaged apps (#549).

[Jaifroid]

However, all that is for future development in Kiwix JS. In the meantime, you can tell this user to try drag-and-drop in Kiwix JS. We enabled it quite a long time ago, and it could well achieve what the user wants right now. You can drag and drop into the app without Configure being open. It is a very fast way to switch between different ZIMs in a folder.

[Jaifroid]

@mossroy I noticed that at least in an Edge Extension (and therefore with 99% probability also in a Chrome Extension), the File System Access APIs appear to be available even under the (chrome-)extension: protocol. I only tested by opening Dev Tools and testing for the availability of window.showOpenFilePicker.

This raises the possibility that we might support this API in the Chromium Extension, and also in the PWA (when/if accessed by a Chromium browser). Unfortunately, Firefox does not support the API (yet?).

As I have experience using this API successfully with Kiwix JS Windows, it would be fairly easy to backport. However, I can think of one disadvantage: it creates a lack of parity between the Chromium Extension and the Mozilla Extension.

Would you be in favour of backporting the API to Kiwix JS nevertheless? I would only want to do the work if there were consensus that it's OK to introduce a feature that is only fully supported by one browser type, of course. Of course the code falls back to the native file picker, so it wouldn't affect Firefox users negatively, only by comparison.

API details: https://developer.mozilla.org/en-US/docs/Web/API/File_System_Access_API

(There is no guarantee the API actually works in the chrome-extension: protocol, because the API depends on a secure context. However, Service Workers also depend on secure contexts, and they run in Chromium Extensions. We will only know for sure if we try.)

[mossroy]

I did not manage to test this feature using https://kiwix.github.io/kiwix-js-windows/ on Chromium 104 on Ubuntu 20.04 (with the official snap provided by Ubuntu). Even after installing it as a PWA.

When I click on "select folder", then choose the directory where all my ZIM files are (and confirm that to Chromium), nothing happens (ZIM files are not listed).

It triggers this error message in the console:

Uncaught (in promise) DOMException: A requested file or directory could not be found at the time an operation was processed.
Promise.then (async) 
  | iterateAsyncDirEntryArray | @ | app.js:2987
  | processNativeDirHandle | @ | app.js:3018
  | (anonymous) | @ | app.js:2917
  | Promise.then (async) |   |  
  | pickFolderNativeFS | @ | app.js:2914
  | (anonymous) | @ | app.js:1168

However, if I select another folder that contains a split zim file, it opens it properly.

Also, if I try to reopen the PWA from the desktop icon (after setting it as executable), it opens a new empty Chromium window (just like if I started a new Chromium) instead of starting kiwix-js-windows.

But maybe this comes from my Ubuntu/Chromium/Snap combination?

I just tested on Ubuntu 22.04 (with the same Chromium version). On the web site, I manage to make it browse my ZIM directory, even if the UX can be confusing (the "select" button of the pop-up window is grayed if I go inside the directory: I need to select the directory while being in the parent directory). It remembers my previous zim files, and allows to reopen them with 2 clicks: one to choose the ZIM file, another one to confirm I allow Chromium to access it.
If I try to start the PWA from the desktop icon, I have the same issue as on Ubuntu 20.04: it opens an empty Chromium window.

This might have nothing to do with your implementation, BTW

[Jaifroid]

Thank you for testing on Linux! I'll try to reproduce the error you are seeing on WSL and get back to you. If you want to select only one File rather than a Directory, you should use the other button (Select File). For the Select Folder button, it will only let you select a Directory, not specific Files in the Directory. This is by design, because one button uses window.showOpenFilePicker, and the other button uses window.showDirectoryPicker.

[Jaifroid]

Running Ubuntu 20.04.4 LTS in WSL, I wasn't able to install the snap version of Chromium, because the snapd environment is (apparently) very difficult to install on WSL. I tried with a script that was supposed to enable it, but it prevented GUI apps from running in WSL, so I had to uninstall and reinstall Ubuntu 😟 (luckily this is just a few clicks in WSL!).

So in the end I had to test in Chrome on Ubuntu. I realize that's not quite the same thing... But for what it's worth, below is a video of what the experience should be like on Chromium browsers -- this is just using pwa.kiwix.org, without installing the PWA (it's not needed to use any of the APIs).

While picking the folder looks a bit convoluted, its' because I'm having to click through to the attached Windows drives where I keep my ZIM archives. Where it goes black in the middle is where I exited and relaunched Chrome to show how much quicker it then is to open an archive (two clicks). Once a directory is picked and remembered between sessions, it's a lot easier to open any archive in the directory.

[Jaifroid]

I forgot to say that the exception you see in your first screenshot above is normal and to be expected. Because the user hasn't given permission, the API throws (it's what it's supposed to do). The second exception from your second screenshot is not one I've been able to reproduce yet.

EDIT: If you think it's worth backporting this, I'd be happy (very pleased!) to have your input on how the file/folder selection UI could be made less confusing. I also understand that there are a lot of other things for us to develop on Kiwix JS, so this may not be highest priority.

[mossroy]

Thanks for your feedback.
I looked a bit into my error on Chromium. It seems to depend on the content of the directory.
If I move a few ZIM files in another directory, it works well
I'll try to investigate

[mossroy]

It seems to come from the fact that the directory where my ZIM files are is in fact a symlink to a directory on another hard drive.
If I directly point to the other hard drive (without going through the symlink), it works properly.
So sorry for the noise here, I'll create a separate ticket on kiwix-js-windows for that

[Jaifroid]

So sorry for the noise here

I don't consider it noise. Testing is always useful!

[mossroy]

OK so I can test the API correctly now.
I think the UX is indeed a bit confusing. I understood the difference between "select file" and "select folder", but did not know what to choose between "select storage", "select folder" and the "reload" icon.
However, I like the fact that it "remembers" the last ZIM file (it's a very popular demand). I suppose we could save the click on "open" by automatically choosing the last ZIM file? (we still let the user choose another ZIM file later if needed). It's the behavior I had implemented for Firefox OS

[Jaifroid]

"Select Storage" is only visible after a user has selected a file or folder. It's just a way of tidying away the other two buttons when they're no longer needed and decluttering the UX. In custom app versions, clicking it has the additional function of resetting storage to the default location (the folder where the packaged archive is stored) - because this can't be selected with a file picker.

The "Reload" icon allows the user easily to refresh a picked directory, for example after adding more ZIM files to it, or after having downloaded a new ZIM file.

But I'm keen to find ways of decluttering and simplifying UX.

[Jaifroid]

Regarding automatically choosing the last ZIM file, this is only possible in a framework like Firefox OS, Electron, NWJS or UWP, and it works that way in all those versions of Kiwix JS. Unfortunately, with the File System Access API, for security reasons the user permisison prompt can ONLY be triggered by a user gesture. It can't be triggered programmatically. So we have to have a minimum of two clicks to re-launch the last used archive.

However, I still find two clicks to be much better UX than asking the user to re-pick the archive every time. The other big advantage is that switching between archives in the folder is very fast (one click on the file, because it doesn't ask for permission again in any one session).

The Chrome developers have said they are considering allowing an INSTALLED PWA to have persistent permission, for a native-like experience (so user would not need to grant permission each time). But this is not even in the draft specification yet, so don't hold your breath, and it probably wouldn't apply to the extension version.

Note that drag-and-drop integrates automatically with the File System Access API, so dragging a file or folder in will mean it's available to reload on next launch with two clicks and no file picker needed.

Finally, on Chrome, the File Handling API (https://web.dev/file-handling/) is also enabled. This means that you can just double click a ZIM archive and it will load in Kiwix JS if it is the default app for handling ZIM archives. This works on Windows if you install the PWA (it has to be installed). I don't htink this is relevant for the Chrome/Edge extensions, however. There may be an extension API that allows it.

[mossroy]

I'm not against using an API that is not supported by all browser engines, if there's a significant advantage of doing so.

"Remembering" the last ZIM file is a very popular demand, so might be a good reason to use this new API.

But I'm disappointed by the restrictions put on this API in the context of a browser extension, that make it much less interesting than I initially thought (It has nothing to do with your code, of course).

It seems to me that the main demand/use-case is to automatically re-open the last ZIM file. With this new API, it can be done, but still needs 2 clicks. With the current version of kiwix-js (without this new API), it needs 3 clicks (or one click and a double-click), as the browser remembers the last directory used.

It's true that it should be much faster afterwards to switch to another ZIM file (in the same session), but does not seem to be the main use-case to me.

The upstream bug on symlinks mentioned in kiwix/kiwix-js-pwa#283 also makes me reluctant to rush on this.

[Jaifroid]

@mossroy I understand your perspective. Personally, I don't think it's just about the number of clicks, it's also the fact that when you open the app, all the ZIM archives in your archive folder are listed right there in the app. The clicks needed are both in the same context. It feels less clunky as a result. There is a tangible advantage with split archives, as you only need to click on the .zimaa (in fact the others aren't shown), to open the whole set in the folder.

But I agree that the gain is only marginal currently, and there are other priorities that are more urgent.

[Jaifroid]

It seems there is an issue using this API in Chrome extensions. See WICG/file-system-access#289.

[Jaifroid]

The corresponding Chromium bug report regarding showDirectoryPicker not working in extensions is here: https://crbug.com/1368818#c6. There is no point developing this until that bug is fixed, because being able to pick a directory of ZIM archives is one of the most useful aspects of this API.

[Jaifroid]

Now that we were forced to use the PWA workaround in order to support Manifest V3 in the Chromium extension, it is more feasible to add the File System Access API. It looks like Chrome is close to implementing persistent permissions for installed PWAs -- see here and here. Persistent means that there would be no further user prompt. Currently there is a small prompt required when accessing files or folders across sessions: it's quick and only requires one click. The idea of the upcoming changes is that if the PWA is installed as an app, rather than merely accessed from an offline Web address, then no further user prompts would be required until permission is revoked.

The main issue, it seems, would be conceptual confusion between an Extension and an installed PWA, even if they are the same thing under the hood...

[Jaifroid]

I have just tested (using Kiwix JS PWA) and in Chromium 117.0.2045.31 the showDirectoryPicker method is now working inside an extension. See https://bugs.chromium.org/p/chromium/issues/detail?id=1368818#c14. This means that the last obstacle to using this API (and, hopefully #1032) has been removed, and we should now add this functionality here.

[Jaifroid]

@Rishabhg71 As mentioned in #1032, it may be easiest to work on this API holistically with the Webkitdirectory API.

You can take a look at the implementation in Kiwix PWA (use the development version, because it is unminified code: https://kiwix.github.io/kiwix-js-windows/, and let it update to latest version, currently v2.7.4).

The left-hand button in screenshot below (uncircled) works like this: if window.showOpenFilePicker is available, it uses that method to get a File Handle. It then serializes (saves) that File Handle in indexedDB (the only Storage where it can be saved), gets the file, and uses it just like any picked file. If window.showOpenFilePicker is not available, then it launches the basic file picker instead (i.e. the current implementation).

The right-hand (circled) button works like this: if window.showDirectoryPicker function is available, then it will use that to pick the directory, which returns a Handle of type directory. This is then serialized to indexedDB, it gets the directory, and enumerates its contents, which it then stores in the Archive List. If showDirectoryPicker is not available, then it checks to see if webkitdirectory is available instead. This is used in a very similar manner, except that there is no Handle to store in indexedDB. If that API is not available either, then the circled button is hidden.

[Jaifroid]

Rather than reinventing the wheel, you could import cache.js into Kiwix JS Browser Extension, or at least the code in there that opens an indexedDB database, and allows serializing objects to it. IndexedDB is a complicated API, and it's really not worth trying to make your own custom methods for using it.

There is also a useful verifyPermission function in cache.js that it would save you time to re-use.

There is a lot of stuff you don't currently need in cache.js, but you can leave pruning stuff out till the end.

I hope this helps.

[Jaifroid]

PS don't try to implement Origin Private File System (yet). That's a whole other level of complexity, and you have to have the building blocks in place first.

[Jaifroid]

@Rishabhg71 Sorry, one other thing -- when picking a directory and processing its contents, attention has to be paid to any split ZIM files in the directory. Only the .zimaa part should be shown to the user. But if the user clicks on the .zimaa part, then a dedicated function will need to enumerate the contents of the directory and collect all the parts. In app.js you will find the function processDirectoryOfFiles(), which does this for you, so you can re-use that (or move it into cache.js, which is where it should probably be along with other FSA utilities).

[Rishabhg71]

Please correct me if I am wrong, But I need to go on and add a similar implementation done in kiwix-js-windows

• Add webkitdirectory support
• Use code from kiwix-js-windows and create an indexedDB
• save the dir to DB and then try to load that dir/zim again

I am still reading docs about FSA and OriginPFS here so drop any resource if you have any

[Jaifroid]

Ignore the information about OPFS in that document. Also read https://developer.mozilla.org/en-US/docs/Web/API/HTMLInputElement/webkitdirectory .

Then I'd do it in this order:

1. Start with detecting whether window.showOpenFilePicker is available, and if it is, use that method to get the File Handle, store it in IndexedDB, get the file and display it in the Dropdown List of Archives (and launch it) -- that should be relatively easy;
2. Then work on what happens when the user clicks on the (single) entry in the dropdown list of archives - it should go and see if it has the Handle stored in IndexedDB. If it's there, and it corresponds, then verify permission, get the file, and launch it for the user.
3. After that, you could add a second button for opening directories, and we can discuss further then.

[Jaifroid]

Oh, between 2 and 3, I'd suggest working on what happens when the user reloads the app: it should try to launch the last selected archive. If you follow the process above, then when you attempt to verify permission for the stored File Handle in IndexedDB, it should ask the user for a quick permission prompt, user says yes, and the file should be displayed. Also handle user saying no.