A lot of error logs "TaskId: <ID> could not be found while executing HTTP"

[astelmashenko]

Describe the bug
Just seeing a lot of errors. Conductor works.

Details Updated
Conductor version: 3.13.6
Persistence implementation: Postgres
Lock: Redis

Conductor version: 3.7.2
Persistence implementation: Postgres
Queue implementation: Nats Streaming
Lock: None

ERROR	2022-06-01T14:50:37,211	1216523394	com.netflix.conductor.core.execution.AsyncSystemTaskExecutor	[system-task-worker-1]	TaskId: c6829bc4-88ce-4174-b620-3eeb5982fa32 could not be found while executing HTTP
--
  | ERROR	2022-06-01T14:50:37,820	1216524003	com.netflix.conductor.core.execution.AsyncSystemTaskExecutor	[system-task-worker-0]	TaskId: 7b03e0ba-eff2-4aba-bf8e-1d2aa5c36ef8 could not be found while executing HTTP
  | ERROR	2022-06-01T14:50:37,841	1216591712	com.netflix.conductor.core.execution.AsyncSystemTaskExecutor	[system-task-worker-1]	TaskId: f79a5574-b2dd-407e-9353-5e95e2348c63 could not be found while executing HTTP
  | ERROR	2022-06-01T14:50:38,425	1216524608	com.netflix.conductor.core.execution.AsyncSystemTaskExecutor	[system-task-worker-1]	TaskId: a00f280a-0c4e-45fe-8049-09a603509774 could not be found while executing HTTP
  | ERROR	2022-06-01T14:50:38,442	1216592313	com.netflix.conductor.core.execution.AsyncSystemTaskExecutor	[system-task-worker-0]	TaskId: 2358d359-c098-4d7f-85ef-c55196ffa8c5 could not be found while executing HTTP
  | ERROR	2022-06-01T14:50:38,732	1216524915	com.netflix.conductor.core.execution.AsyncSystemTaskExecutor	[system-task-worker-0]	TaskId: a6220454-8aba-40c4-953d-79f0ccb13195 could not be found while executing HTTP
  | ERROR	2022-06-01T14:50:38,746	1216592617	com.netflix.conductor.core.execution.AsyncSystemTaskExecutor	[system-task-worker-1]	TaskId: 2714d0eb-88aa-47fe-ab71-8b035c956ca2 could not be found while executing HTTP
  | ERROR	2022-06-01T14:50:39,338	1216525521	com.netflix.conductor.core.execution.AsyncSystemTaskExecutor	[system-task-worker-1]	TaskId: a028e2f7-2dfa-49c0-bcaa-d070d9846704 could not be found while executing HTTP
  | ERROR	2022-06-01T14:50:39,349	1216593220	com.netflix.conductor.core.execution.AsyncSystemTaskExecutor	[system-task-worker-0]	TaskId: f9900694-516e-470b-9a5a-f680d088bd9b could not be found while executing HTTP
  | ERROR	2022-06-01T14:50:39,946	1216526129	com.netflix.conductor.core.execution.AsyncSystemTaskExecutor	[system-task-worker-0]	TaskId: 352276c0-d063-4a7d-b628-47a908b2eb3b could not be found while executing HTTP
  | ERROR	2022-06-01T14:50:39,954	1216593825	com.netflix.conductor.core.execution.AsyncSystemTaskExecutor	[system-task-worker-1]	TaskId: ba6f4723-29c6-467b-8349-8db4de1d506c could not be found while executing HTTP
  | ERROR	2022-06-01T14:50:40,252	1216526435	com.netflix.conductor.core.execution.AsyncSystemTaskExecutor	[system-task-worker-1]	TaskId: db52f994-1f06-4436-84e5-5d43e40b4eb4 could not be found while executing HTTP
  | ERROR	2022-06-01T14:50:40,258	1216594129	com.netflix.conductor.core.execution.AsyncSystemTaskExecutor	[system-task-worker-0]	TaskId: 888be841-096f-4059-8958-466b3be2f419 could not be found while executing HTTP
  | ERROR	2022-06-01T14:50:40,860	1216527043	com.netflix.conductor.core.execution.AsyncSystemTaskExecutor	[system-task-worker-0]	TaskId: 233304d2-0e7b-443d-a464-c220d850aaea could not be found while executing HTTP
  | ERROR	2022-06-01T14:50:40,861	1216594732	com.netflix.conductor.core.execution.AsyncSystemTaskExecutor	[system-task-worker-1]	TaskId: 6bb378a9-941d-4c2f-a62d-6bf25ce06826 could not be found while executing HTTP
  | ERROR	2022-06-01T14:50:41,465	1216595336	com.netflix.conductor.core.execution.AsyncSystemTaskExecutor	[system-task-worker-0]	TaskId: 41b00506-2028-4a16-8af0-3e4a0d6c906b could not be found while executing HTTP
  | ERROR	2022-06-01T14:50:41,466	1216527649	com.netflix.conductor.core.execution.AsyncSystemTaskExecutor	[system-task-worker-1]	TaskId: 324ac94f-c659-4c54-afa3-69b7d7885886 could not be found while executing HTTP
  | ERROR	2022-06-01T14:50:41,769	1216595640	com.netflix.conductor.core.execution.AsyncSystemTaskExecutor	[system-task-worker-1]	TaskId: a7cd6785-4869-4f9b-93ed-767723037c45 could not be found while executing HTTP
  | ERROR	2022-06-01T14:50:41,772	1216527955	com.netflix.conductor.core.execution.AsyncSystemTaskExecutor	[system-task-worker-0]	TaskId: 4a899089-0704-418c-9ca1-ebfb8df4330b could not be found while executing HTTP
  | ERROR	2022-06-01T14:50:42,378	1216528561	com.netflix.conductor.core.execution.AsyncSystemTaskExecutor	[system-task-worker-1]	TaskId: 3a89ea86-b5ca-4de4-9516-6853d8e1fbf7 could not be found while executing HTTP
  | ERROR	2022-06-01T14:50:42,382	1216596253	com.netflix.conductor.core.execution.AsyncSystemTaskExecutor	[system-task-worker-0]	TaskId: 51944fcb-66b2-4bdb-ba31-f906a0e74b08 could not be found while executing HTTP
  | ERROR	2022-06-01T14:50:42,691	1216596562	com.netflix.conductor.core.execution.AsyncSystemTaskExecutor	[system-task-worker-1]	TaskId: 691cf9bf-0ad8-4905-854d-cd6750bcb58c could not be found while executing HTTP
  | ERROR	2022-06-01T14:50:42,985	1216529168	com.netflix.conductor.core.execution.AsyncSystemTaskExecutor	[system-task-worker-0]	TaskId: dc3e211a-5fa5-46d7-91f6-6a541e035c0d could not be found while executing HTTP
  | ERROR	2022-06-01T14:50:43,297	1216597168	com.netflix.conductor.core.execution.AsyncSystemTaskExecutor	[system-task-worker-0]	TaskId: 4165406c-8e72-4010-92a8-88c7d5ef97e4 could not be found while executing HTTP
  | ERROR	2022-06-01T14:50:43,591	1216529774	com.netflix.conductor.core.execution.AsyncSystemTaskExecutor	[system-task-worker-1]	TaskId: a4e4e7b3-1df5-4dab-b5de-5b2a47c44b04 could not be found while executing HTTP
  | ERROR	2022-06-01T14:50:43,898	1216530081	com.netflix.conductor.core.execution.AsyncSystemTaskExecutor	[system-task-worker-0]	TaskId: e0d335e2-d7a8-4d1b-bc77-f7b00c539c37 could not be found while executing HTTP
  | ERROR	2022-06-01T14:50:43,917	1216597788	com.netflix.conductor.core.execution.AsyncSystemTaskExecutor	[system-task-worker-1]	TaskId: 3880fbf9-1b92-46a1-92be-3c0db4628598 could not be found while executing HTTP
  | ERROR	2022-06-01T14:50:44,214	1216598085	com.netflix.conductor.core.execution.AsyncSystemTaskExecutor	[system-task-worker-0]	TaskId: 54f5cba5-797e-4506-b9de-e1769fb4e0c3 could not be found while executing HTTP
  | ERROR	2022-06-01T14:50:44,503	1216530686	com.netflix.conductor.core.execution.AsyncSystemTaskExecutor	[system-task-worker-1]	TaskId: e8f2bfc3-a96f-4faa-acec-b6a633bcdec9 could not be found while executing HTTP
  | ERROR	2022-06-01T14:50:44,518	1216598389	com.netflix.conductor.core.execution.AsyncSystemTaskExecutor	[system-task-worker-1]	TaskId: 8acb6389-f30d-4121-8ed9-c97413fb0572 could not be found while executing HTTP
  | ERROR	2022-06-01T14:50:45,111	1216531294	com.netflix.conductor.core.execution.AsyncSystemTaskExecutor	[system-task-worker-0]	TaskId: 2104ad06-d763-49ff-b3b9-ba6330dd9d82 could not be found while executing HTTP
  | ERROR	2022-06-01T14:50:45,123	1216598994	com.netflix.conductor.core.execution.AsyncSystemTaskExecutor	[system-task-worker-0]	TaskId: aaca31d1-06e1-4e3d-a035-088a5eb6a23b could not be found while executing HTTP
  | ERROR	2022-06-01T14:50:45,427	1216599298	com.netflix.conductor.core.execution.AsyncSystemTaskExecutor	[system-task-worker-1]	TaskId: 0ff9abd3-47f5-4828-97ae-b2f4311efcca could not be found while executing HTTP
  | ERROR	2022-06-01T14:50:45,717	1216531900	com.netflix.conductor.core.execution.AsyncSystemTaskExecutor

[manan164]

hI @astelmashenko , Which type of executionDAO are you using? Looks like it is in an unstable state. Can you clean up and restart the conductor again?

[aravindanr]

@astelmashenko this error is printed because a HTTP task was posted to the queue with its task id. But when the executor of that HTTP task polled and started executing it, the task is not found in the persistence store. This could be a race condition. For production use-cases, it is necessary to have a lock implementation. Please consider using one.

[astelmashenko]

Thanks guys, I've looked through source code and found out it was due to absence of a lock service. We are in process of adding lock service. I'm closing this issue.

[astelmashenko]

This happened again, this time we have a lock enabled. Most likely after conductor upgrade, restart happened such errors appeared in logs.
Conductor version 3.13.6
Storage type: postrgres
Lock: redis

[astelmashenko]

@manan164 , is it possible that during restart lock service is stopped in incorrect order, is there something like graceful shutdown and critical services are started first and stopped last?

[github-actions]

This issue is stale, because it has been open for 45 days with no activity. Remove the stale label or comment, or this will be closed in 7 days.

[github-actions]

This issue was closed, because it has been stalled for 7 days with no activity.