End of sprint release 2020-10-13

• (in separate branch) Initial documentation for how journals may use the API for embedded submissions.
• Cleaned up journal entries in our database, so the list of Royal Society journals includes only the current names and not the historical names that have caused confusion.
• Addded a search API that allows searching by keyword, affiliation, tenant, and date modified.
• Zenodo file downloads and Remote Access Tokens (RATs) -- (in separate branch zenodo-code)
  • In review page able to download files from both Merritt and Zenodo, and whether public or private
  • In private for peer review sharing page, shows Zenodo section if there are files there and with download links
• Cleanup of publishers in code to correctly set all as Dryad (such as citations, DataCite metadata and what gets saved to db)
• Remove versions from public display because they make no sense (citations)
• Minor UI cleanup of a few links and copyright year and other small items

End of sprint release 2020-09-29

• Related works and extra UI changes
  • UI changes (entry, review, landing with new sidebar layout)
  • Remove all PDF generation, templates, libraries
  • Autodetection of doi/accession numbers and type when entering
  • Error reporting and updating near entry including live validation and formatting
  • Autofill from DOI updated to work with new related works format
  • Curator acceptance of metadata from journals puts related works into new format
• Hiding related works from public display for things that are incorrect and can't be easily corrected manually
• 4 scripts bringing manual updates from Google sheets back into database
• Merge of Rails 5.2 back into long-standing Zenodo branches so development can start on those branches again
• Validation that Remote Access Tokens are working (access to private files) with pre-signed URLs at Zenodo and minor changes to work with 1) different file download URLs, 2) "deposition_ids" as strings instead of integers they appear to be
• Query/report of datasets published by year
• Fixing new secrets/config bugs with configs
• Reinstated the remaining rspec tests that had been disabled due to intermittent failures.
• Expanded fields that are returned for datasets in the API: license, visibility, lastModificationDate
• Reworked how configuration files are handled. All configuration files are now in our main repository, with “secret” values stored in the Rails encrypted file.

2020-09-15 end of sprint

Release notes:

• Updated all code to run on Rails 5.2.
• Upgraded Bundler to version 2.1.4
• Added documentation about how to handle API accounts that do not
  have ORCIDs.
• Added documentation about creating a “System” user when installing
  Dryad.
• Corrected issue that prevented subject keywords from being saved.
• Disabled the new checks on origins of HTTP requests, because they
  were not compatible with our Apache configuration.
• Added DOIs to the table on the submission_queue page.
• Changed config email addresses for bcc on some items.

(in separate branch not yet in main)

• Related works
  • Validation of DOI format
  • Transformation of URLs that appear to contain a DOI into DOI format
  • Live validation of DOIs against https://doi.org
  • Error messages near entry when things do not validate
  • Review page requires valid DOIs (to change later)
  • Review page displays related works in main column
  • Changed layout on landing page for Related Works: Right column, by category of work and linked
  • Corrected tests for new data entry changes and added additional tests for models

2020-09-01 End of Sprint

Release notes

Note, this code will not be deployed this time since we'll do the minor upgrade to Rails 5.2 before doing further upgrade testing and deployment.

• Updated all code to run on Rails 5.1.
• Updated most tests to create objects with the factory methods.
• Improved use of WebMock within the test suite, so it doesn’t cause tests to fail randomly.
• Consolidated documentation about the deployment process and interactions with Merritt, with the resultant documents going into the
  main Dryad documentation.
• Corrected issue with rendering files for NCBI LinkOut.
• Added a double-encoding workaround for retrieving files containing the number/hash sign from Merritt. We may remove as they move to different APIs in the future that may make this workaround unnecessary.
• Added a query and documentation for finding datasets by author affiliation with ROR institutions.
• Preliminary work on RATs presigned URL implementation with Zenodo for private file download (not merged into tag yet).
• Initial work for simplified "Related Works" entry UI (not merged into tag yet).

2020-08-18 End of Sprint

• Updated all code to run on Rails 5.0.
• Migrated all engine-level testing code into the top level, to run as a coherent test suite.
• Converted all sortable tables to use a SortableTableHelper instead of the outdated sortable-table gem.
• Added a selection box for "Research Domain" on the dataset description page. The box can be filled with one of the values from the OECD Fields of Science. These values will be sent to DataCite along with the other dataset metadata.
• Renamed the master branch to main.
• Standardized the formatting of DOIs that are stored in Related Identifiers.
• Expanded the metadata supplied on landing pages in schema.org format.
• Minor textual updates to the file upload page, Submission Process help page, and emails sent to users after submission.

2020-08-04 End of Sprint

• Live downloading and caching of view and downloads stats (in branch counter-stats-from-datacite)
• Reliability enhancements and tests for notifier script
  • catches and logs all errors
  • ensures PID is cleaned up after even aborted runs

In rails5 branch

• All base level functionality working
• Corrections to tests

(other tasks not in this repo)

• Resubmission of MDC stats to, updated to end of June 2020
• Texas Tech to "Texas Tech University" in login list
• Fixed most Zenodo additional copy errors (and added tickets for others)
• Documentation of services for meetings with dev-ops

2020-07-21 End of Sprint

• Enhanced logic for detecting status of the notifier service.
• Corrected error with the messaging when users requests a download of a large file via the API.
• (in other branch, not yet released work on Rails upgrade)
• Changing configuration for Merritt to include possible protocol and port (and code changes) so can be used without https and from various ports for Docker.
• Additional handling of timeouts from Merritt
  • Handle 408 status code (ignore for status of assembly, different actions for initiating a download)
  • Warn user of download currently unavailable with message
• Replaced Cirneco gem methods for registering DataCite metadata for DOI
• COUNTER CoP for Research Dataset changes for submission system for reports
  • Report IDs option
  • Get all identifiers from DataCite to avoid having to maintain state and separate from other independent scripts
  • Smarter option for deciding which months haven't been updated and submit them
  • Tell which months need to be uploaded and force upload of those months
• New Tenant added -- Texas Tech
• Fix for issue that arose when people would add a file remove the file and add the same file again within the same version and it would send incorrect submission information to Merritt.

2020-07-07 End of Sprint

• Added scripts to manage ROR affiliations.
• Minor improvements for reindexing the entire repository into Solr.
• Minor improvements for updating the metadata for the entire
  repository in DataCite.
• Improved script that detects duplicate authors, and expanded it to
  merge the duplicates.
• Updated mapping for schema.org fields to be more effective when
  indexed by Google.
• Improved robustness of the dependency checker for the stash-notifier service.
• Added the Claremont Colleges as a consortium (7 schools and one admin institution)
  • Login tests on stage
  • Logos
  • Configuration for administration to see datasets associated with consortium authors
• Upgraded from Ruby 2.4 to Ruby 2.6 which is maintained until 2022
  • Updated Rubocop lint/style checker to work with new rails version in app and all engines
  • Resolved bundler problems
  • Compiled and installed new version of Ruby on servers and fixed services and Capistrano to work with new Ruby
  • Instructions for Ruby version upgrade
  • Travis.ci changes to use new version of Ruby

Work in progress outside of main branch

• Enabled user-downloads for file versions sent and published to Zenodo

End of sprint 2020-06-23

• Added metadata requirement to submission system. When a journal name is present, either a manuscript number or article DOI must also be present.
• Guaranteed that an article DOI entered at the top of a submission will be saved (once and only once) in the related identifiers.
• Updated /download API endpoints to always use presigned URLs from Merritt.
• Improved schema.org metadata on the landing pages to be compatible with the current requirements from Google. Added ROR and ORCID identifiers to this metadata.
• Added a report of basic dataset metadata, dataset_info_report, to support periodic requests about the content in Dryad.
• Added some protections against internal nil errors.
• Updated staff descriptions on the Our Team page.
• Moved all rspec tests from the stash_datacite engine into the main rspec directory.
• Added University of Rhode Island as partner
• Added code for handling non-json if it is returned by Merritt in context of presigned version status
• Delayed job is automatically stopped and restarted around deploy to avoid zombie versions running old code
• Change configuration of HTTP.rb gem to bypass url path normalization which caused mismatches with some foreign characters and Merritt retrieval.

Changes worked on, but not in this branch (in zenodo-code-upload-files)

• Send different license and type (software) to Zenodo for software uploads
• Use Zenodo-provided DOIs for software
• Add isSupplementTo relationship for software (and in reverse dataset)
• Rework of workflow and using Zenodo states (rather than internal Dryad states) to make it more reliable and appropriate for differences in versioning between the two systems.
• Tracking zenodo DOIs, concept record id.
• New ActiveJob configuration and utility methods to work with software replication jobs.
• Refactor of replication classes for zenodo to call main class Copier rather than Resource which is already too overloaded in meaning.
• Had Copier class take copy_id in initialization rather than resource, since it simplifies code and makes many methods clearer and more specific, but require some code and test rewrites.
• Automated tests for about 1/2 of the added workflow and related methods (the other half coming soon).

End of sprint 2020-06-09

Release notes:

• Updated listing of Dryad staff on the Our Team page.
• Internal fix to ensure statistics displays will work regardless of the date settings of the underlying database.
• Improvements of documentation pages for the API.
• In API responses, changed embargoEndDate to publicationDate to be more accurate.
• Added support for “editURL”. These URLs appear in API responses, and can be used to directly access the editing interface for a dataset, assuming the user has appropriate permissions
• Using presigned S3 URLS from Merritt for object version downloads
  • Caching Merritt version keys
  • Re-assemble object if version is cache miss (per Merritt query)
  • Begin download if only a short wait or it is cached
  • Assembling progress dialog box if the wait is longer
  • Auto-download when ready from dialog
  • Allow "private for peer review" links to use pre-signed versions for download
  • Remove the email process for large files
  • Logging for COUNTER Code of Practice for downloads (slightly different places)
  • Allow standard GET request in addition to AJAX requests from URL