TODO.md

TODO

Top

• Delete VSCode stuff inside containers (I think its only when using the container itself as devcontainer)
• Study Date is changed when using Selective Transfer
• Fix that ADIT DICOMweb supports STOW of multiple image files at once -- Currently it only allow to upload only one file after another -- When multiple files at once are stored than those are send as a chunked transfer -- This seems to be not supported by DRF
• Bring back some kind of ID in batch modes to better correlate the failures in the Excel files
  • Users must provide "QUERY_ID" in Batch Query and "TRANSFER_ID" in Batch Transfer in uploaded Excel files
  • BatchQueryTask gets an additional "query_id" and BatchTransferTask gets an additional "transfer_id"
  • Show those to the user in the task views so that they can find the corresponding Excel row more easily
  • Create new export batch query results view to optionally allow the users to export with failures
  • Also provide some more info in that view what users should do to transfer
  • https://github.com/openradx/adit/commit/c6d840b1872b3c8f39b0f0008cdce131d5863f4d
• Add a permission to only allow to add a new job when all other jobs of the user are finished
• Show failures in exported batch results
• Add permissions to dicom_web views (see TODOs there)
• Update documentation
• Just warn when one only one series of the study could not be transferred (only error when all series could not be transferred)
• test job_utils
• Test canceled task/job in test_workers.py
• Make sure all views are atomic
  • Use ATOMIC_REQUESTS database setting
  • Unfortunately we can't just set ATOMIC_REQUESTS in the database settings globally as those don't seem to work with async views
  • Try this again when Django 5.0 is released
  • Otherwise tell those async views to be @transaction.non_atomic_requests
  • Alternative is to decorate all appropriate views with @transaction.atomic
• Use DicomLogEntry during C-STORE
• Fix dicom explorer search over Accession Number
• Make warning when only one image fails
• Exclude SR and PR when in pseudonymization mode
• Cancel processing tasks actively
  • When cancelling a job we currently wait for an already processing task to be completed before setting the job as canceled
  • Those tasks should get actively killed, which is already implemented when killing the task from UI (staff only)
  • But maybe we can do it more gracefully by passing a is_aborted() somehow down to the connectors
  • In DimseConnector we could check a provided is_cancelled function if the next series should be fetched (or even the association be aborted)
  • in DicomWebConnector we chould also check such an function an close the Session https://requests.readthedocs.io/en/latest/api/#requests.Session.close. We can handle the Session manually when using DicomWebClient https://dicomweb-client.readthedocs.io/en/latest/package.html#dicomweb_client.api.DICOMwebClient
• Make registration Email unique and required. Also maybe check if an Email is of specific domains (optional).
  • We must first delete those users with duplicate or non existing Emails
  • Do this also for RADIS
• Make sure temporary folder created in retrieve DICOM web API is cleaned up (see TODO in /home/adm-adit/workspace/adit/adit/dicom_web/views.py)
• Look into how we can stream the file from disc (from the temp folder) with WADO (see https://chat.openai.com/share/d5a2f27f-4854-4deb-85df-b7f574638ae3)
• Look into how we can improve STOW (do we have to upload one file at a time, can we stream it somehow)
• DICOMwebClient currently does not support to access warnings when transferring images
  • See ImagingDataCommons/dicomweb-client#94
  • That way we can't show warnings when some images of study to transfer could not be received (Partial Response with HTTP status code 206)
  • Maybe we can fix some stuff and get the PR accepted or use an own fork
    • See also ImagingDataCommons/dicomweb-client#88
• Directly stream images from the connectors through the operators
  • Currently we download them to a folder first
  • This is ok for selective and batch transfer, but not really ok for our API calls to retrieve images
  • Also the DicomWebConnector seems to support this through iter_ interface, see ImagingDataCommons/dicomweb-client#88
  • For DimseConnector this is already partly implemented by using generators (see yield there)
• Remove files in test folders from autoreload
• Selective transfer choose series
• Locked info for other apps like batch_transfer_locked.html
• Encrypt data between swarm containers
  • https://docs.docker.com/network/drivers/overlay/#encrypt-traffic-on-an-overlay-network
  • https://forums.docker.com/t/configuring-encryption-for-swarm-overlay-network-in-compose/29469/2
  • We can also make the network attachable to do the "exec" stuff in tasks.py using one off containers using "run"
• Redirect after restart/retry/delete job
• Option in batch query to query whole study or explicit series
• Make whole receiver crash if one asyncio task crashes
• Auto refresh job pages und success or failure
• Query with StudyDateStart, StudyDateEnd, StudyDate
• Common search query Websocket component
• Improve cancel during transfer
• Allow admin to kill a job (with task revoke(terminate=True))
• Fix the ineffective stuff in transfer_utils, see TODO there
• Write test_parsers.py
• DICOM data that does not need to be modified can be directly transferred between the source and destination server (C-MOVE). The only exception is when source and destination server are the same, then the data will still be downloaded and uploaded again. This may be helpful when the PACS server treats the data somehow differently when sent by ADIT.

Fix

• Shorter timeout for offline studies
• Tests: test_query_utils, test serializers, test all views (as integration tests using real Orthanc), improve tests of transferutil, BatchFileSizeError
• c-get download timeout
• Choice to transfer all modalities of a studies or just the modalities which were searched for
• Make logging analyze Docker image with: http://crunchtools.com/software/petit/, less, vim, https://crypt.gen.nz/logsurfer/, ripgrep
• Evaluate (0008, 0056) Instance Availability CS: 'OFFLINE' ( (ONLINE, NEARLINE, OFFLINE, UNAVAILABLE)), see https://www.gehealthcare.com/-/jssmedia/1b62d771fb604ff7a4c8012498aea68b.pdf?la=en-us

Maybe

• Do some prechecks before trying the task (is source and destination online?)
• Generate exposed IDs for URLs by hashing the primary (number) keys:
  • https://sqids.org/python
  • https://stackoverflow.com/questions/43253602/django-short-non-linear-non-predictable-id-in-the-url
• New batch transfer
  • Create new batch transfer job and allow to add tasks
  • Add tasks manually or using a Excel file
  • Query tasks directly using new dicom_query_queue
  • New status: QUERYING, READY
  • Button: Start Transfer (only when no task is querying)
  • Allow to add tasks to an already existing job (even if already transferred)
  • Delete batch query
• Upload portal with drag&drop
  • Store those files perhaps in ORTHANC
  • Preview uploaded images
  • Allow to transfer thow uploaded image to a PACS
• Improve scheduler (with day in week and times)
• Show failed tasks in results of batch query and batch transfer
• Allow to find multiple Patients for the same PatientName + PatientBirthDate in batch query
  • Currently we don't allow this, but this can happen when a patient has multiple PatientIDs in the same PACS (e.g. has external images)
• exclude test folders from autorelad in ServerCommand (maybe a custom filter is needed)
• Switch from Daphne to Uvicorn (maybe it has faster restart times during development)
• Upgrade postgres server to v15, but we have to migrate the data then as the database files are incompatible a newer version
  • https://hollo.me/devops/upgrade-postgresql-database-with-docker.html
  • https://thomasbandt.com/postgres-docker-major-version-upgrade
  • https://betterprogramming.pub/how-to-upgrade-your-postgresql-version-using-docker-d1e81dbbbdf9
  • look into https://github.com/tianon/docker-postgres-upgrade
• Allow to search multiple source servers with one query (maybe only in dicom explorer)
• Bring everything behind Nginx as reverse proxy
  • Orthanc could then be directly behind Nginx (without Django-revproxy)
  • Use authentication module of nginx
  • http://nginx.org/en/docs/http/ngx_http_auth_request_module.html
  • https://stackoverflow.com/a/70961666/166229
  • Evaluate Nginx vs Traefik
• Reuse authentication in integration tests
  • https://playwright.dev/python/docs/auth
  • Unfortunately, we can't use live_server fixture inside session fixtures
  • example https://github.com/automationneemo/PlaywrightDemoYt
• Get rid of dicom_connector.download_study/move_study. Do everything at the series level. That way filtering series (e.g. exlcude modalities) is much easier.
• Evaluate if services should be better restarted with pywatchman instead of watchdog and watchmedo
  • pywatchman is used by Django autoreload
  • See https://github.com/django/django/blob/main/django/utils/autoreload.py
  • Unfortunately, I could not get it to work with Django autoreload itself, but we can use something similiar by using watchman directly and integrate it in ServerCommand
• BatchQuery with custom DICOM keywords
• Watchdog server
• Allow provide a regex of StudyDescription in batch file
• move date parsing part in parsers.py and consumers.py to date_util.py
• https://stackoverflow.com/questions/14259852/how-to-identify-image-receive-in-c-store-as-result-of-a-c-move-query
• https://www.yergler.net/2009/09/27/nested-formsets-with-django/
• http://the-frey.github.io/2014/08/18/monitoring-docker-containers-with-monit
• move or get rid of hijack_logger and store_log_in_task in task_utils
• log debug -> info in connector also in production
• Link owner in templates to user profile
• Improve .zip file encryption
  • Unfortunately, file names and directory names are still visible
  • Can be work around by wrapping another zip file in an encrypted zip file https://unix.stackexchange.com/a/290088/469228
• Rewrite dicom_connector to use asyncio (wrap all pynetdicom calls in asyncio.to_thread)
  • I don't think that this will gain any performance improvements, so maybe not worth it
• Evaluate other task runners
  • https://www.pyinvoke.org/ # used currently
  • https://github.com/taskipy/taskipy
  • https://github.com/nat-n/poethepoet
  • https://just.systems/
  • https://taskfile.dev/
• Make a job urgent retrospectively (maybe only staff members can do this)
  • A current workaround is to cancel the job, change urgency with Django Admin and then resume the job
• Try to bring channels_liver_server in official pytest_django release
  • https://github.com/pytest-dev/pytest-django/blob/master/pytest_django/fixtures.py#L514
  • https://github.com/pytest-dev/pytest-django/blob/42b7db2f4f5dbe785e57652d1d4ea9eda39e56e3/pytest_django/live_server_helper.py#L4
  • https://github.com/django/channels/blob/main/channels/testing/live.py#L21
  • https://github.com/django/daphne/blob/main/daphne/testing.py#L123
  • https://github.com/django/django/blob/main/django/test/testcases.py#L1810
• Maybe move label from from form to models using "verbose_name" and also the help_text
• Save specific form fields for later use with HTMX, currently we only save them in the post handler when the form is valid.
• Move user profile fields from User to a related UserProfile model. Improves the query performance a bit, but not sure if it is worth it.
• Setup pgadmin
  • https://stackoverflow.com/questions/64620446/adding-postgress-connections-to-pgadmin-in-docker-file
  • Not sure if we really need this as we have Django admin and can view data in there
• Replace ass AssertionError with assert

RADIS