Provide a breakdown of all packages in SS, not just the AIPs included in reports

[peterVG]

Include a log of the Exception errors that were reported for each.

Provide this count on the reports themselves?

[joel-simpson]

This could also be framed as a report on what AIPScan includes in its database and what it doesn't. That's not just about errors (i.e. couldn't scan x AIPs) but also about helping users reconcile what is in AIPScan with what is in their storage system, including all packages, including DIPs, replicated AIPs, transfers, SIPs etc.
One idea we discussed was providing an explicit way for users to reconcile what they see in the Storage Service (whether through SS UI or the Archival Storage tab) with what is included in AIPScan. This should give users confidence in the accuracy of data stored in AIPScan.

[ross-spencer]

I have got as far as cleaning up some of the METS work to make it easier to see where we need to log these issues. What I haven't figured out yet, is where to record them:

1. An errors table with a foreign key to the AIP and files (original/preservation) causing a problem?
2. Errors in the AIP or 'files' tables?

And we need to add a summary to the storage service view as well.

I believe there are currently two places to log errors in processing (there might be a few others indicated by try/except patterns in the code):

1. Overall error processing the METS: here
2. Problem processing a file: here. The problem with this particular except right now is that the error assumes ISO file formats but it could really be any format, and so it might be better to log the entire file as something like "not able to process" and then move on.

[peterVG]

Since this issue ticket was created, we've created several new issues to breakdown the error reporting requirement.

To address Joel's feature suggestion (statistics for packages per SS), we've added the ability to record the number of SIPs, DIPs, replicated copies and deleted copies as columns in the Fetch Job table. See https://github.com/artefactual-labs/AIPscan/blob/main/AIPscan/Aggregator/tasks.py#L159-L179

However, we don't surface this information in the GUI for the user to see. I think the most natural place for this information is in the Fetch Job table that is shown at the bottom of a Storage Service page. This information could simply be read in from the table.

However, if this information was to be provided as a report, or even for display on the SS page, it may be worthwhile to deliver it via an API. This would be in sync with the approach described in Issue #95.

Therefore, the more elaborate/elegant way to deliver this feature is suggested in Issue #110

[peterVG]

GUI option 1: display stats in small, greyed out text below total package count.

Pro: easy to see all package info for a Fetch Job at a glance.

Con: takes up a lot of space. Makes Fetch Job row a lot longer.

[peterVG]

GUI option 2: display stats when hovering over an Info icon. next to total package count. Hide the stats when mouse moves away.

Pro: conserves spaces. easy to see all package info for a Fetch Job in one glance.

[ross-spencer]

Love them both @peterVG. I like 2 most. How configurable is 2? Could it be a toggle, so click for the info and it displays. Click again and it disappears?

[peterVG]

Thanks @ross-spencer, me too, so I implemented a "hover" option in 'dev/package-stats-GUI' and submitted as PR. After fiddling with jQuery for a while I got stats display working when you hover the mouse over the Info icon. The stats are hidden again as soon as mouse moves away from the icon.

[ross-spencer]

@peterVG yep, it's looking good. Reason I ask about a toggle using a similar technique to the one you've adopted is that it might be nice to keep the information up on screen. Users might then toggle the information on between fetch jobs to compare without having to hover over and commit some of the info to memory. (I just generally like the idea of persisting information that is useful on screen but with the compromise here too that it doesn't by default use real-estate). What do you think? Is it at all feasible?

[peterVG]

Yeah, that's a good usability point. I'll try to implement that change and post an update here.

[peterVG]

I made the change from hover to click in the updated PR: #114

The behaviour now works as @ross-spencer suggested. Display on click for all info icons that are clicked. Hide the stats when the same info icon is clicked again.