Request for Requirements: ISIS Release Process and Long Term Support #124
Comments
|
As an individual scientist, I want to use ISIS to perform data processing for a particular project that will lead to a paper. During the lifetime of this work (six months to a year), I want the output of ISIS to be dependable and reproducible, but if there are bug fixes, I want those. In the past, new versions of ISIS would sometimes introduce non-reproducible behavior that wasn't necessarily just a bug fix, and so I have become wary of upgrading ISIS in the middle of working on a project. However, if I could be guaranteed that there were version upgrades during the lifetime of my project that were truly just bugfixes, then I would be more likely to upgrade during that lifetime. Otherwise, I would only upgrade ISIS before starting a new project. |
|
As a developer of a software package that depends upon ISIS (e.g. Ames Stereo Pipeline), I need to develop against a version of ISIS whose API and underlying data structure is unlikely to change significantly over the average time span between releases of my software. My project is a much smaller enterprise than ISIS, and we are only able to release about once a year. If the ISIS API and data structure changes significantly on a shorter time scale (this probably means major version changes of ISIS and its data), my users complain about functionality not working, but my team can't do anything about it until we can perform another release. Ideally, we could get our project release schedule synchronized with major version changes to ISIS (if we knew what that schedule was and it wasn't faster than our release cadence). We'd develop against an ISIS release candidate, wait for that RC to be made an official release, do a final check, and then make our own release, so that the lifetime of our release matches (as much as possible) the lifetime of the ISIS revision we are baselined against. Having ISIS software and data versioned and available for a time scale approximately similar to the release cadence of my software project would alleviate this problem (even if the API and data structure resulted in major rev ticks on a timescale faster than my project's release cycle), as then we can direct our users to a specific version of ISIS and its data structure, and they should be able to get by until our next release. |
|
As a spacecraft mission IT manager, I need different things from the ISIS release pattern. (1) During the development stage of my mission, either ASC developers or my own developers will be creating new features and finding new bugs that need to be addressed and merged as quickly as possible so that ISIS is in a good state for "my mission." (2) After this phase, I now need ISIS to be rock-solid and provide reproducible results while my mission operates. I understand that bugs will be found and features will be added. Ideally, I'd like to get the bug fixes, but would probably prefer to live without the features (because they sometimes have unintended consequences and I want to keep the changes minimal because those changes might ripple out into my other data processing systems and require a lot of patching). I appreciate that my mission may be active for years or even decades, and it is unreasonable to expect only bugfixes for that time, and the mission is likely to freeze on a particular major version of ISIS. The longer that version can receive bugfixes, the better. Likewise, knowing the approximate time that a major version would be getting bugfixes will help my staff and I plan for changes. We'd much rather be able to see a major revision coming well in advance, so that we can plan for workforce to accommodate the change, and test against the RC before it becomes a release. Otherwise, we might stay frozen on a particular version for many years. |
|
As an average scientist user, I would want to have access to the latest ISIS version to use to do whatever I need for my particular project, be it processing one LROC-NAC image, or ten Cassini ISS images. I would hope that new versions are more stable, are more correct (in case inaccuracies in calculations were found that have been fixed), and have more/new features or capabilities I might want. My projects typically require ISIS only for the initial data processing, and afterwards it does not matter if something is forwards or backwards compatible because I don't tend to reprocess my data in the middle of a project. |
|
As a developer who writes code that interacts with ISIS and expects that ISIS behave in a certain way, including various data structures and output files, I need ISIS to remain stable and I need changes to be documented. For example, if a cube header format changes, and my code expects a certain format for a certain line, that could break my code. As another example, if the way that ISIS writes control networks moves from its current version (5) to a new one, my code would be completely broken, and an absence of documentation would make debugging it very difficult; there is no documentation about what a version 5 control net is, versus the past version (2). So moving forward, extremely detailed documentation about how things have changed from one ISIS version to the next would greatly assist me in understanding what I need to change in my own code to make sure it still works. |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
Scenario #1: A scientist who is focused on getting a task done and is looking for a fast and furious rate of small releases so their work does not stall. Story: I am a researcher processing multiple data sets for a section of planet X. I build a pipeline using seven isis applications mixed with some unix utilities and process 289 images from one of the instruments. I discover an obscure issue with isis (maybe a bad interaction between special pixels and a calibration routine). I report it, work with the developers over the next few days, get a fix, and keep processing the following week. Brilliant publication follows shortly thereafter with glowing acknowledgement to isis and its amazing support team. Note: I am aware of very real and practical problems with providing this kind of support that is driven by the user's schedule with no consideration of other work the developers are doing. But I think it is a real use case that is worth keeping in mind. |
This comment has been minimized.
This comment has been minimized.
|
Scenario #2: A user returns to ISIS after a long hiatus, expecting that there are changes to how ISIS works and that the documentation will make it easy to find the specific changes that affect their work. Story: I am a researcher who returns to a project after a 2.5 year hiatus and am planning to process new data using the isis-heavy pipeline I used back then. But before I start I do a quick search on the isis documentation to see if anything has changed with the applications I am using. I find that an obscure issue (maybe a bad interaction between special pixels and a calibration routine) has been corrected, but I need to set a parameter to a different value to activate this fix. I make the small change to my old pipeline, reprocess the old and new data. Brilliant publication follows shortly with glowing acknowledgement to isis. Note: What this story/scenario is meant to emphasize is the need not just for thorough documentation but also the ability to "discover" or search for the most relevant pieces, like new parameters available within a specific application. |
This comment has been minimized.
This comment has been minimized.
|
I am an internal user and work on creating products of the moon and mars. I almost exclusively use the internal conda environments and rely on a cluster to handle my processing. I use 4.0.1 because I have not needed to update as I work with very stable/mature functions (jigsaw, automos, etc.) and I subscribe to an "if it ain't broke (and there are no enhancements), don't fix it" mentality. As for release stability needs, taken from the ISO 25010 standars linked in the original post:
I am skipping the last 3 categories, I feel like those are not applicable to a products user story. But I also I wanted to keep them in the table in case anyone wanted to copy/reuse! |
|
I am a ground systems manager and data processing pipeline developer. My story is very similar to that of rbeyer's. I work on a mature spacecraft mission where the majority of processing needs are met, but development of new data products sill occur. I am also involved in another mission that is maturing, but is still under under heavy development. In a mature mission context, reliability and consistency are most important to us. Changes that necessitate updating parameters in our existing code can be tricky to find, even when such changes are well documented within a new Isis release. Testing new versions of Isis can be very time consuming, especially as our staffing has been reduced. Frequent releases make it difficult for us to keep current and our solution to this is to not try to keep up, but be strategic about when we upgrade. Our current policy is to upgrade at least once a year, but no more than twice. We have chosen this route instead of "freezing" our GDS as we still have on going development that needs Isis support and falling behind too far means much more work than keeping up. In the context of a developing mission, we need to be able to work closely with the Isis team to be sure they are aware of our developing needs and be able to have rapid updates to the Isis applications we are using to develop our processing system. Bugs are frequent and need to be addressed quickly to allow the development process to progress. |
|
As a developer for a mature, ongoing mission, I need to develop against a version of ISIS that is stable and would not introduce new issues but also allows us to remain at that version until we can upgrade to a later version as mission time allows. One hard requirement is that I need to remain at a stable software version but have up-to-the-minute data. Our mission is actively producing data but can't match the release cadence. We need to be able to work off a set software version but still have the latest version of data. Ideally, I'd like to keep up with major versions, but the release cadence is too fast. Upgrading consists of updating our environment to make the version available to our our science team to test. Since the adoption of Anaconda, which is geared more for creating an individual environments, our network environment requires some effort. Adopting a new release takes mission time. Once this release is available, it must be validated with our pipelines and production processes. This requires a good amount of effort and coordination. The current cadence does not allow us to do this for every version released, thus creating a situation where in order to adopt bug fixes, we have to upgrade to a more current version. This requires testing and validation from not only our developers, but also our science team. Having to reproduce products that do not contain errors attributed to bugs or breaks created by bug fixes, is very time consuming and requires a lot effort coordination. We have over 10 years of data. If we are unable to move to a release that contains a critical bug fix, I have to develop and implement a local bug fix or release enhancement as a work-around until we can upgrade to the release containing the fix. Realistically, if we had a release schedule, we could synchronize our development, testing, and validation schedule proportionally against major versions, with skipping a set number of major versions at a time. Ideally, we would like the option to adopt releases containing only bug fixes, or features, but not containing both |
|
I am an image data analyst that often evaluates interactions between ISIS, DTMs and navigation data on small, irregular bodies. I rely heavily on output data produced by applications such as instrument import programs (i.e., label content), spiceinit, isisminer, camstats, footprintint, cnetcheck and jigsaw. Stability, consistency and content with regard to units and order are critical to my work. At times, its difficult to determine presence of data and units from jigsaw due to different input parameters. One of the most important aspects of our analysis work is comparisons and validation of jigsaw bundle adjustments to digital shape kernel (DSK) models. We use high resolution DSKs as ground truth. Comparisons of jigsaw radius solutions at adjusted latitude/longitudes are compared directly with radii in the DSK at the same locations. This is only effective when jigsaw units are in body fixed coordinates. I am not too concerned with control network formats as I find it sufficient to associate comprehensive databases/CSVs produced from isisminer and output files from cnetcheck and jigsaw in production cycles. Changes in application parameters, output file formats/content or units with new ISIS releases can cause disruptions in workflows and make comparisons/verifications between releases difficult. Thorough documentation related to these features is also a must. I realize that it may be too much to assume changes won’t occur between major ISIS releases, so I would try to make my analysis immune to changes as much as possible (e.g, relying on consistent column header names and units) by anticipating reordering in or new additions of data (columns). isisminer is not a problem – jigsaw is as it doesn’t currently have a reliable single column header line naming scheme in most if its output files. |
|
I am closing this out as we are moving onto decision making. If anyone has further comments please add them here and re-open. |
Overview:
For the few months, we are actively soliciting input from the community of ISIS users to help determine the pace and requirements of ISIS releases. This request will be used, by members of the ISIS TC, to develop a plan for the ISIS release cadence and to determine if and how a long-term support (LTS) model may be implemented by the ISIS project.
Request:
We are soliciting user feedback on the ISIS release cycle in order to better define release requirements. Specifically we want to know how you interact with ISIS releases, the life-cycle or update cadence that you use with ISIS, and your expectations for release stability through one or more user stories. In the context of this request, user stories are at most five sentences summarizing generally who you are (a persona), what release requirements would fulfill your needs, and why this story fulfills your needs.
How can I participate?
On 'stability'
I added this section because we are seeing the word stability or proxy phrases for stability (e.g., should always work) being used. When writing about 'stability', please consider that that word has different meanings for different users. One resource to help guide contributors in writing more specific user stories might be the ISO:25010 specification on software stability. Lots of other options exists too! The primary goal here is to have text that helps the ISIS TC write specific requirements to support a large subsection of the user base.
User Story Templates:
At the most simplistic, you could use the following template as a mad-libs style starting point:
Some more fuller examples:
Please consider if you have more than one user stories within your work and feel free to break them up. Additionally, it is okay to have different requirements/desires for different tasks. We are hoping to satisfy as many needs as possible so don’t be afraid to share conflicting desires.
Next steps:
A subset of the ISIS TC will review and synthesize all presented user stories with the goal of identifying a release cadence and long-term support (LTS) model that supports as many users as possible.
The text was updated successfully, but these errors were encountered: