possible to make a sequence of fixed-time spectrum recordings?

[jbeale1]

I am wondering if it is already possible, or would be easy to add, the capability to record a sequence of fixed-time-interval spectra automatically. For example, accumulate a spectrum for 1 hour, save the JSON file, clear that data and immediately start accumulating another spectrum for the next hour, and so on. This would be useful for tracking any shifts in the spectrum (or the hardware) over time.

[jbeale1]

Incidentally I was impressed by how steady my background, PMT, and the OpenGamma board is. 1-hour background recordings 3 hours apart gave totals of 305967 counts and 305469 counts, for a difference of 0.2% so there is not very much drift happening.

[NuclearPhoenixx]

Sweet! 🙂

[NuclearPhoenixx]

Hmm, that's currently not possible, but it sounds doable. The only issue is that you cannot automatically save files to a users computer without asking for permission in some way. I could ask for permission to write to a single file before starting the recording and then store all the individual spectra into one file. That could definitely be done. You'd have to individually load every spectrum from this one file after everything's done if you want to look at the results. Would that be ok?

I could also do sort of a 3D bar chart like in impulse, which would be a lot nicer to look at than just all the spectra individually. I think that goes in the right direction of what you're thinking about, right?

[jbeale-takara]

Storing multiple spectra into one JSON file seems fine, I presume I could then write my own code to generate separate files from that later, so I could do whatever comparisons seem interesting, or generate a 3D waterfall plot, or a time-lapse animation showing the change over time. Does that mean you have to set up a fixed endpoint in advance instead of just letting it run indefinitely, and you can't extract intermediate results? Unless as you suggest the app overlays each plot for you as it goes, which I agree would be neat if that's doable.

[jbeale1]

sorry, was logged into my other account, but what he said :-)

Just out of curiosity, what does the small green message on the Serial connection tab like "Autosaved Feb 3, 08:23" mean, if filesystem access always needs separate user permission? Does it mean saved on the Pi-Pico board, not my host PC, or it's OK to re-write over the same filename without asking, or there's a private scratchpad file area you can freely write?

[NuclearPhoenixx]

Storing multiple spectra into one JSON file seems fine, I presume I could then write my own code to generate separate files from that later, so I could do whatever comparisons seem interesting, or generate a 3D waterfall plot, or a time-lapse animation showing the change over time. Does that mean you have to set up a fixed endpoint in advance instead of just letting it run indefinitely, and you can't extract intermediate results? Unless as you suggest the app overlays each plot for you as it goes, which I agree would be neat if that's doable.

I'm not sure how long file handle permissions last in your browser, if they're indefinite as long as the same session is open, then the recordings could in theory also be indefinite. You can always go back and open individual spectra in Gamma MCA one by one. If you import a JSON file with multiple spectra, it will ask you to select one.

sorry, was logged into my other account, but what he said :-)

Just out of curiosity, what does the small green message on the Serial connection tab like "Autosaved Feb 3, 08:23" mean, if filesystem access always needs separate user permission? Does it mean saved on the Pi-Pico board, not my host PC, or it's OK to re-write over the same filename without asking, or there's a private scratchpad file area you can freely write?

It's not stored as a file on your system, but rather as a variable in your browser. Each website has it's own little storage space to save a couple of MB inside your browser. I use that to autosave the data and restore it if the app crashes during a recording. It's 100% separate from your filesystem, but it's great to keep persistant data "behind the scenes".

[jbeale1]

I read that "Browsers can store up to 5 MiB of local storage, and 5 MiB of session storage per origin." How much would I want? I don't think anything recorded by any of my instruments has true 4095 points worth of resolution, even without the funky Pi-Pico ADC channel width issues. So presuming I could usefully represent one spectrum using 1000 channels, each being a 32-bit value, then one spectrum takes only 4k memory. So if there weren't other things to store, seems like there's space for over 1000 spectrums in this private browser memory. Or 250, even at the full 4k points which is arguably sort of without much meaning at least on the current Pi-Pico. Either way it would be good enough for anything I had in mind. https://developer.mozilla.org/en-US/docs/Web/API/Storage_API/Storage_quotas_and_eviction_criteria

[NuclearPhoenixx]

From my experience, the average spectrum file is about 20 - 25KB total with all the metadata, which gives around 200 files max in the browser's local storage, yeah. There is also another API that would be more targeted at that kind of application giving a lot more space.

Are you suggesting to store the data in the browser only and make it optionally downloadable after the fact so to say? That sounds like an interesting idea. I don't see why it wouldn't be possible to save all imported and recorded spectra in kind of a database inside the app and make them all accessible through that. Downloading/storing to the computer's filesystem can still be done of course if so desired.

I'd have to look up the persistency of that kind of storage, though. IIRC everything you store as a website can be nuked at any time if the user deletes the cache, cookies or other storage or if the quota for that origin has been reached. Would be a real pain if you store everything in the app itself and then accidentally delete your browser history or something lol

[jbeale1]

It seems like it could work to me. I don't know how much redesign it would need. I'm not a web developer and I had no idea even a MB of local storage was available to a web page. I guess my web knowledge is still stuck in the early days of the internet. I suppose you would have to trust that someone using this web app for a long-run experiment is not also going to manually delete browser cache etc. while it's collecting data, or if they do, the consequence is on them.
Isn't this somewhat the way it works now, though? For example right now I'm doing an 8-hour acquisition with the app. If I delete all my browser memory in the middle of that, without first manually saving my spectrum to a file, isn't it gone?

[NuclearPhoenixx]

Nope, it's pretty resilient in that regard. Most data is being hold in the memory for the page, i.e. RAM, not some browser storage:

• If you clear the browser storage without closing the app, your data will still be there and it should continue to record just fine.
• If you clear the browser storage and then immediately close the app without saving, you will lose everything and autosaving will not be able to help you.
• If you clear the browser storage, wait 15 minutes max and then close the app, the autosave function will recover everything for you.

You can test it if you want, it should pretty much work just like I mentioned.

My only concern with storing in the browser is that people don't know that the storage can be somewhat volatile, i.e. at their browser's mercy, and that they rely on storing a bunch of spectra in the app itself instead of exporting them to their filesystem. If somehow they themselves unknowingly clear the history, cache, cookies or something and all the data is gone

I might add a disclaimer or something, I'll have a look and add it on my todo list. It would be nice to store stuff in the app itself, even if people have to be careful with their browser settings. 👍🏻

[jbeale1]

I write some very simple Python to try what I wanted, and it works ok for me so far. I'm just decimating the original 4096 channels from the OpenGamma board down to 409 channels for storage and display, using OpenGammaRec.py and plot-stack.py in https://github.com/jbeale1/gamma