Error when reconstituting CPT4 of newest release

[mpatel-cai]

Describe the problem in content or desired feature
Received the following error when reconstituting CPT4:

[INFO ] 2024-03-11 16:23:13.899 [main] ConceptService - Not processed cpt4 concepts: 13348. See logs/not-processed-concepts-03-11-2024-15-24-12.out, file. You can find more information about errors in the logs/logfile.log file

How to find it
File attached not-processed-concepts-03-11-2024-15-24-12.txt

[don-torok]

I've had similar problems in the past. Try just running the CPT4 program again. For me the program was able to recognize which rows were not processed and completed on the second try.

[odikia]

Thanks @mpatel-cai and @don-torok for reporting this issue, as I'm having the same problem with similar log output. If this has happened in the past, perhaps the vocab team should either push an update to the readme, or fix the issue with the code? As it can be solved via a rerun, bugfixing the code may not be worthwhile at the current time, but perhaps the readme could state "If there is an error experienced when first running the cpt updater, try re-running the application"

Other users will hopefullly find their way to Meera's issue report here, but just in case not!

[Alexdavv]

@OHDSI/odysseus-product-team Can we actually make the recursion a part of the process?

[leslieskung]

We are also having this issue. Have tried twice over the last day with the same results of around 13k CPT4 not processing.

All failures have similar output with code value being empty:

UMLS name: null was not successfully resolved
Update status:FAIL. The code value for id xxxxxxxx is []

Will keep trying in the meantime.

[Alexdavv]

We are also having this issue. Have tried twice over the last day with the same results of around 13k CPT4 not processing.

Wait. Do you start over with the clean tables each time or do you run it on the tables you got on the previous run?

If the second is true, you got zero concepts processed on a new run or some are processed?

[leslieskung]

We are also having this issue. Have tried twice over the last day with the same results of around 13k CPT4 not processing.

Wait. Do you start over with the clean tables each time or do you run it on the tables you got on the previous run?

If the second is true, you got zero concepts processed on a new run or some are processed?

My colleague and I both tried separately. Each time got a slightly different amount of records not processed, but both in the 13k range.

[leslieskung]

Just tried the method of running twice using the same base table. Got the same exact count of failures both times, so no, rerunning does not (always) work and I got 0 processed the second time.

[Alexdavv]

Just tried the method of running twice using the same base table. Got the same count of failures both times, so no, rerunning does not (always) work and I got 0 processed the second time.

Hmm. This is weird. It must be another issue. Can you attach the logs, please?

[leslieskung]

Sorry, I do not have those any longer, they were stored in a temp directory. Our usual process is to run the update using a Databricks notebook. However today I tried via just basic command line and it seemed to work with all CPT processing the first time, but that may have just been coincidence.

[odikia]

@Alexdavv , I thought about just taking a look at the UMLS API in order to determine whether or not this is an issue with the Java code for some user’s systems, or if it has to do with problems in receiving data from UMLS consistently. Before I do any kind of testing of that hypothesis with Python, as I’m not a Java developer(!), has that already been tested at Odysseus that you’re aware of? If you think such a Python test would be useful with the different systems on my end, I can do that. Don’t want to waste time though, if that’s already been done!

[leslieskung]

@Alexdavv , I thought about just taking a look at the UMLS API in order to determine whether or not this is an issue with the Java code for some user’s systems, or if it has to do with problems in receiving data from UMLS consistently. Before I do any kind of testing of that hypothesis with Python, as I’m not a Java developer(!), has that already been tested at Odysseus that you’re aware of? If you think such a Python test would be useful with the different systems on my end, I can do that. Don’t want to waste time though, if that’s already been done!

Before I commented in this thread, I had reached out to UMLS first. They responded they would look, but that no one else had reported this issue to them and to contact OHDSI. :) Inconclusive at best.

[Alexdavv]

@Alexdavv , I thought about just taking a look at the UMLS API in order to determine whether or not this is an issue with the Java code for some user’s systems, or if it has to do with problems in receiving data from UMLS consistently. Before I do any kind of testing of that hypothesis with Python, as I’m not a Java developer(!), has that already been tested at Odysseus that you’re aware of? If you think such a Python test would be useful with the different systems on my end, I can do that. Don’t want to waste time though, if that’s already been done!

I don't think we tested in Python. Let me tag @OHDSI/odysseus-product-team @alex-odysseus @konstjar here.
It's pretty much the UMLS API issue that we've been struggling with a lot.