Processing loop

[andreaskoelsch]

Which version of recognize are you using?

6.1.1

Enabled Modes

Object recognition, Face recognition

TensorFlow mode

Normal mode

Downstream App

Memories App

Which Nextcloud version do you have installed?

28.0.5

Which Operating system do you have installed?

Debian 12

Which database are you running Nextcloud on?

MariaDB

Which Docker container are you using to run Nextcloud? (if applicable)

No response

How much RAM does your server have?

32GB

What processor Architecture does your CPU have?

x86_64

Describe the Bug

I have three files which are processed by the object detector in a loop and not removed from the queue.

Expected Behavior

Files should be removed from the queue once processed

To Reproduce

Well, hard to say. Since this only occured for 3 of my >100k files, it's not clear to me what caused the issue

Debug log

MariaDB [nextcloud]> Select * FROM oc_recognize_queue_imagenet;
+--------+---------+------------+---------+--------+
| id     | file_id | storage_id | root_id | update |
+--------+---------+------------+---------+--------+
| 312601 |  822267 |          2 |  666480 |      0 |
| 312603 |  822269 |          2 |  666480 |      0 |
| 312604 |  822270 |          2 |  666480 |      0 |
+--------+---------+------------+---------+--------+
3 rows in set (0.000 sec)

{"reqId":"6hT9CpdzAzHZvx1hRr53","level":0,"time":"2024-05-15T23:20:12+02:00","remoteAddr":"","user":"--","app":"recognize","method":"","url":"--","message":"Classifying files of storage 2 using imagenet","userAgent":"--","version":"28.0.5.1","data":{"app":"recognize"}}
{"reqId":"6hT9CpdzAzHZvx1hRr53","level":0,"time":"2024-05-15T23:20:12+02:00","remoteAddr":"","user":"--","app":"recognize","method":"","url":"--","message":"fetching 100 files from imagenet queue","userAgent":"--","version":"28.0.5.1","data":{"app":"recognize"}}
{"reqId":"6hT9CpdzAzHZvx1hRr53","level":0,"time":"2024-05-15T23:20:12+02:00","remoteAddr":"","user":"--","app":"recognize","method":"","url":"--","message":"Running imagenet classifier","userAgent":"--","version":"28.0.5.1","data":{"app":"recognize"}}
{"reqId":"6hT9CpdzAzHZvx1hRr53","level":0,"time":"2024-05-15T23:20:12+02:00","remoteAddr":"","user":"--","app":"recognize","method":"","url":"--","message":"generating preview of 822267 with dimension 1024 using nextcloud preview manager","userAgent":"--","version":"28.0.5.1","data":{"app":"recognize"}}
{"reqId":"6hT9CpdzAzHZvx1hRr53","level":0,"time":"2024-05-15T23:20:12+02:00","remoteAddr":"","user":"--","app":"recognize","method":"","url":"--","message":"generating preview of 822269 with dimension 1024 using nextcloud preview manager","userAgent":"--","version":"28.0.5.1","data":{"app":"recognize"}}
{"reqId":"6hT9CpdzAzHZvx1hRr53","level":0,"time":"2024-05-15T23:20:12+02:00","remoteAddr":"","user":"--","app":"recognize","method":"","url":"--","message":"generating preview of 822270 with dimension 1024 using nextcloud preview manager","userAgent":"--","version":"28.0.5.1","data":{"app":"recognize"}}
{"reqId":"6hT9CpdzAzHZvx1hRr53","level":0,"time":"2024-05-15T23:20:12+02:00","remoteAddr":"","user":"--","app":"recognize","method":"","url":"--","message":"Classifying array (\n  0 => '/tmp/oc_tmp_48XDyw-.jpg',\n  1 => '/tmp/oc_tmp_NocVac-.jpg',\n  2 => '/tmp/oc_tmp_tkiaC1-.jpg',\n)","userAgent":"--","version":"28.0.5.1","data":{"app":"recognize"}}
{"reqId":"6hT9CpdzAzHZvx1hRr53","level":0,"time":"2024-05-15T23:20:12+02:00","remoteAddr":"","user":"--","app":"recognize","method":"","url":"--","message":"Running array (\n  0 => '/usr/bin/nice',\n  1 => '-0',\n  2 => '/var/www/nextcloud/apps/recognize/bin/node',\n  3 => '/var/www/nextcloud/apps/recognize/src/classifier_imagenet.js',\n  4 => '-',\n)","userAgent":"--","version":"28.0.5.1","data":{"app":"recognize"}}
{"reqId":"6hT9CpdzAzHZvx1hRr53","level":0,"time":"2024-05-15T23:20:12+02:00","remoteAddr":"","user":"--","app":"recognize","method":"","url":"--","message":"Classifier process output: 2024-05-15 23:20:12.562541: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA\nTo enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n","userAgent":"--","version":"28.0.5.1","data":{"app":"recognize"}}
{"reqId":"6hT9CpdzAzHZvx1hRr53","level":0,"time":"2024-05-15T23:20:15+02:00","remoteAddr":"","user":"--","app":"recognize","method":"","url":"--","message":"Classifier process output: {\n  className: 'bell cote',\n  probability: 0.8209066390991211,\n  rule: { label: 'tower', threshold: 0.1, categories: [ 'architecture' ] }\n}\n","userAgent":"--","version":"28.0.5.1","data":{"app":"recognize"}}
{"reqId":"6hT9CpdzAzHZvx1hRr53","level":0,"time":"2024-05-15T23:20:15+02:00","remoteAddr":"","user":"--","app":"recognize","method":"","url":"--","message":"Classifier process output: {\n  className: 'church',\n  probability: 0.04315225034952164,\n  rule: {\n    label: 'church',\n    context: 'nature stones',\n    threshold: 0.1,\n    categories: [ 'building', 'architecture' ]\n  }\n}\n","userAgent":"--","version":"28.0.5.1","data":{"app":"recognize"}}
{"reqId":"6hT9CpdzAzHZvx1hRr53","level":0,"time":"2024-05-15T23:20:15+02:00","remoteAddr":"","user":"--","app":"recognize","method":"","url":"--","message":"Classifier process output: {\n  className: 'lakeside',\n  probability: 0.004793553147464991,\n  rule: {\n    label: 'lakeside',\n    context: 'nature',\n    threshold: 0.6,\n    categories: [ 'water' ]\n  }\n}\n","userAgent":"--","version":"28.0.5.1","data":{"app":"recognize"}}
{"reqId":"6hT9CpdzAzHZvx1hRr53","level":0,"time":"2024-05-15T23:20:15+02:00","remoteAddr":"","user":"--","app":"recognize","method":"","url":"--","message":"Classifier process output: {\n  className: 'beacon',\n  probability: 0.004151171538978815,\n  rule: { threshold: 0.1, categories: [ 'tower', 'architecture' ] }\n}\n","userAgent":"--","version":"28.0.5.1","data":{"app":"recognize"}}
{"reqId":"6hT9CpdzAzHZvx1hRr53","level":0,"time":"2024-05-15T23:20:15+02:00","remoteAddr":"","user":"--","app":"recognize","method":"","url":"--","message":"Classifier process output: {\n  className: 'vault',\n  probability: 0.0038930969312787056,\n  rule: { label: 'building', context: 'nature stones', threshold: 0.1 }\n}\n","userAgent":"--","version":"28.0.5.1","data":{"app":"recognize"}}
{"reqId":"6hT9CpdzAzHZvx1hRr53","level":0,"time":"2024-05-15T23:20:15+02:00","remoteAddr":"","user":"--","app":"recognize","method":"","url":"--","message":"Classifier process output: {\n  className: 'pole',\n  probability: 0.0022382489405572414,\n  rule: { threshold: 0.1, priority: -2 }\n}\n{\n  className: 'nipple',\n  probability: 0.002208178164437413,\n  rule: { label: 'bottle', threshold: 0.2 }\n}\n","userAgent":"--","version":"28.0.5.1","data":{"app":"recognize"}}
{"reqId":"6hT9CpdzAzHZvx1hRr53","level":0,"time":"2024-05-15T23:20:15+02:00","remoteAddr":"","user":"--","app":"recognize","method":"","url":"--","message":"Result for /andreas/files/Share/2024/2024-05-10 09.31.19.jpg(oc_tmp_48XDyw-.jpg) = [\"Tower\",\"Architecture\"]","userAgent":"--","version":"28.0.5.1","data":{"app":"recognize"}}
{"reqId":"6hT9CpdzAzHZvx1hRr53","level":0,"time":"2024-05-15T23:20:16+02:00","remoteAddr":"","user":"--","app":"recognize","method":"","url":"--","message":"Classifier process output: {\n  className: 'church',\n  probability: 0.4917282164096832,\n  rule: {\n    label: 'church',\n    context: 'nature stones',\n    threshold: 0.1,\n    categories: [ 'building', 'architecture' ]\n  }\n}\n","userAgent":"--","version":"28.0.5.1","data":{"app":"recognize"}}
{"reqId":"6hT9CpdzAzHZvx1hRr53","level":0,"time":"2024-05-15T23:20:16+02:00","remoteAddr":"","user":"--","app":"recognize","method":"","url":"--","message":"Classifier process output: {\n  className: 'bell cote',\n  probability: 0.33402442932128906,\n  rule: { label: 'tower', threshold: 0.1, categories: [ 'architecture' ] }\n}\n{\n  className: 'lakeside',\n  probability: 0.018104856833815575,\n  rule: {\n    label: 'lakeside',\n    context: 'nature',\n    threshold: 0.6,\n    categories: [ 'water' ]\n  }\n}\n{\n  className: 'mosque',\n  probability: 0.005985795520246029,\n  rule: {\n    label: 'tower',\n    threshold: 0.1,\n    categories: [ 'building', 'architecture' ]\n  }\n}\n","userAgent":"--","version":"28.0.5.1","data":{"app":"recognize"}}
{"reqId":"6hT9CpdzAzHZvx1hRr53","level":0,"time":"2024-05-15T23:20:16+02:00","remoteAddr":"","user":"--","app":"recognize","method":"","url":"--","message":"Classifier process output: {\n  className: 'earthstar',\n  probability: 0.005197624210268259,\n  rule: { label: 'vegetables', threshold: 0.6, categories: [ 'food' ] }\n}\n{\n  className: 'vault',\n  probability: 0.0040014563128352165,\n  rule: { label: 'building', context: 'nature stones', threshold: 0.1 }\n}\n","userAgent":"--","version":"28.0.5.1","data":{"app":"recognize"}}
{"reqId":"6hT9CpdzAzHZvx1hRr53","level":0,"time":"2024-05-15T23:20:16+02:00","remoteAddr":"","user":"--","app":"recognize","method":"","url":"--","message":"Classifier process output: {\n  className: 'buckeye',\n  probability: 0.003190287621691823,\n  rule: { label: 'plant', threshold: 0.1, catgories: [ 'nature' ] }\n}\n","userAgent":"--","version":"28.0.5.1","data":{"app":"recognize"}}
{"reqId":"6hT9CpdzAzHZvx1hRr53","level":0,"time":"2024-05-15T23:20:16+02:00","remoteAddr":"","user":"--","app":"recognize","method":"","url":"--","message":"Result for /andreas/files/Share/2024/2024-05-10 09.30.29.jpg(oc_tmp_NocVac-.jpg) = [\"Church\",\"Building\",\"Architecture\",\"Tower\"]","userAgent":"--","version":"28.0.5.1","data":{"app":"recognize"}}
{"reqId":"6hT9CpdzAzHZvx1hRr53","level":0,"time":"2024-05-15T23:20:17+02:00","remoteAddr":"","user":"--","app":"recognize","method":"","url":"--","message":"Classifier process output: {\n  className: 'church',\n  probability: 0.6289583444595337,\n  rule: {\n    label: 'church',\n    context: 'nature stones',\n    threshold: 0.1,\n    categories: [ 'building', 'architecture' ]\n  }\n}\n","userAgent":"--","version":"28.0.5.1","data":{"app":"recognize"}}
{"reqId":"6hT9CpdzAzHZvx1hRr53","level":0,"time":"2024-05-15T23:20:17+02:00","remoteAddr":"","user":"--","app":"recognize","method":"","url":"--","message":"Classifier process output: {\n  className: 'bell cote',\n  probability: 0.190898135304451,\n  rule: { label: 'tower', threshold: 0.1, categories: [ 'architecture' ] }\n}\n{\n  className: 'monastery',\n  probability: 0.007419401314109564,\n  rule: {\n    label: 'historic',\n    threshold: 0.25,\n    context: 'buildings architecture',\n    categories: [ 'architecture' ]\n  }\n}\n{\n  className: 'lakeside',\n  probability: 0.005275358911603689,\n  rule: {\n    label: 'lakeside',\n    context: 'nature',\n    threshold: 0.6,\n    categories: [ 'water' ]\n  }\n}\n","userAgent":"--","version":"28.0.5.1","data":{"app":"recognize"}}
{"reqId":"6hT9CpdzAzHZvx1hRr53","level":0,"time":"2024-05-15T23:20:17+02:00","remoteAddr":"","user":"--","app":"recognize","method":"","url":"--","message":"Result for /andreas/files/Share/2024/2024-05-10 09.30.35.jpg(oc_tmp_tkiaC1-.jpg) = [\"Church\",\"Building\",\"Architecture\",\"Tower\"]","userAgent":"--","version":"28.0.5.1","data":{"app":"recognize"}}
{"reqId":"6hT9CpdzAzHZvx1hRr53","level":0,"time":"2024-05-15T23:20:17+02:00","remoteAddr":"","user":"--","app":"recognize","method":"","url":"--","message":"Classifier process output: {\n  className: 'earthstar',\n  probability: 0.0036971624940633774,\n  rule: { label: 'vegetables', threshold: 0.6, categories: [ 'food' ] }\n}\n{\n  className: 'vault',\n  probability: 0.0036862995475530624,\n  rule: { label: 'building', context: 'nature stones', threshold: 0.1 }\n}\n{\n  className: 'mosque',\n  probability: 0.0026431980077177286,\n  rule: {\n    label: 'tower',\n    threshold: 0.1,\n    categories: [ 'building', 'architecture' ]\n  }\n}\n","userAgent":"--","version":"28.0.5.1","data":{"app":"recognize"}}

[github-actions]

Hello 👋

Thank you for taking the time to open this issue with recognize. I know it's frustrating when software
causes problems. You have made the right choice to come here and open an issue to make sure your problem gets looked at
and if possible solved.
I try to answer all issues and if possible fix all bugs here, but it sometimes takes a while until I get to it.
Until then, please be patient.
Note also that GitHub is a place where people meet to make software better together. Nobody here is under any obligation
to help you, solve your problems or deliver on any expectations or demands you may have, but if enough people come together we can
collaborate to make this software better. For everyone.
Thus, if you can, you could also look at other issues to see whether you can help other people with your knowledge
and experience. If you have coding experience it would also be awesome if you could step up to dive into the code and
try to fix the odd bug yourself. Everyone will be thankful for extra helping hands!
One last word: If you feel, at any point, like you need to vent, this is not the place for it; you can go to the forum,
to twitter or somewhere else. But this is a technical issue tracker, so please make sure to
focus on the tech and keep your opinions to yourself. (Also see our Code of Conduct. Really.)

I look forward to working with you on this issue
Cheers 💙

[andreaskoelsch]

Interestingly, of the recently added photos, these three are the only ones which are added to the landmark classification queue. Probably has something to do with it. The loop is as follows:

1. imagenet classification of the three files. They are not removed from the queue aftwerwards
2. Files added to landmark queue
3. landmark classification
4. removal from landmark queue
5. Step 1 again

[marcelklehr]

They are not removed from the queue aftwerwards

Huh, that's curious. And no errors?

[andreaskoelsch]

No, nothing. I also removed the causing files and re-uploaded them. Same issue

[andreaskoelsch]

Okay, to be fair, I did not re-upload the files, but move them on the filesystem and re-scanned the directories. That seems to have caused a problem. After actually uploading the files using the webinterface, it worked as supposed. I think, there might be a few factors playing a role here:

• Group Folders
• Landmark Detection

Just to provide information why I moved and re-scanned the file instead of just moving: This is still my workaround for #1118. Unfortunately, I cannot update, yet, as Group Folders is not yet compatible with NC 29. I'm hoping the fix for mentioned issue will make my workaround obsolete, so this should not happen anymore anyways. I'll test and post my findings here again when I have update NC and recognize to the latest version.

[andreaskoelsch]

Oh no, I was wrong. I moved another folder and again one file caused the loop issue, so I removed it and re-uploaded it via the webinterface. That did not help though, it's stuck in mentioned loop, even though it was uploaded "correctly". Again, it's a file which triggers landmark detection

[marcelklehr]

I did quite a bit of refactoring of the file hooks in v7 so, I'd rather hold off on digging into this deeper until you've updated to v7 / nc29 and can confirm that this is still happening. I hope that's ok.

[andreaskoelsch]

I think, it might be gone with v7. At least it did not happen for the last files I added. I guess we can close this.

[andreaskoelsch]

I was too quick in closing this issue. The files I tested did not trigger the landmark detection pipeline. It just happened again when I added some files which triggered the landmark detection. So the issue is unfortunately not fixed

[marcelklehr]

Is it possible that the files get re-inserted into the queue instead of them staying in there? Could you check the id column before and after imagenet classification? If the ids change then the files get reinserted

[marcelklehr]

Also, could it be that you or some other users have set up some flows in nextcloud flow that touch files based on tags? This could trigger the file listener and cause the files to get re-added to the queue.

[andreaskoelsch]

Sorry for the late response. With the recent images I added, I could no longer reproduce the issue. It seems a bit random, very odd. To your question before, I had checked that and the queue id's did not change.

[andreaskoelsch]

I'm just having the issue again today. Can confirm, the ids stay the same. Right now, I'm having a loop with 30 files. The following screenshot is taken during the imagenet classification. the 30 files stay in the imagenet queue and get added to the landmarks queue one by one. Well, procedure as described in the opening post

[andreaskoelsch]

After letting this loop run for ~1h, it suddenly managed to remove the files from the imagenet queue. Very weird. Could it have something to do with the use of redis?

[marcelklehr]

Seems like a race condition then 🤔