[BUG] Existing jobs can not be deleted due to maximum string size in Jackson.

[bryanbolles]

JobRunr Version

6.2.4

JDK Version

17

Your SQL / NoSQL database

Postgres 15.2

What happened?

log message was stuck in succeeded causing the server to keep crashing

How to reproduce?

   BackgroundJob.enqueue(() -> service.helpMe());

Relevant log output

2023-06-27T23:20:19.427Z ERROR  1 --- [pool-9-thread-2] org.jobrunr.server.BackgroundJobServer   : FATAL - JobRunr encountered too many processing exceptions. Shutting down.
org.jobrunr.JobRunrException: JobRunr encountered a problematic exception. Please create a bug report (if possible, provide the code to reproduce this and the stacktrace)
	at org.jobrunr.JobRunrException.shouldNotHappenException(JobRunrException.java:43) ~[jobrunr-pro-6.2.4.jar:6.2.4]
	at org.jobrunr.utils.mapper.jackson.JacksonJsonMapper.deserialize(JacksonJsonMapper.java:88) ~[jobrunr-pro-6.2.4.jar:6.2.4]
	at org.jobrunr.jobs.mappers.JobMapper.deserializeJob(JobMapper.java:22) ~[jobrunr-pro-6.2.4.jar:6.2.4]
	at org.jobrunr.storage.sql.common.JobTable.toJob(JobTable.java:316) ~[jobrunr-pro-6.2.4.jar:6.2.4]
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(Unknown Source) ~[na:na]
	at org.jobrunr.storage.sql.common.db.SqlSpliterator.tryAdvance(SqlSpliterator.java:44) ~[jobrunr-pro-6.2.4.jar:6.2.4]
	at java.base/java.util.Spliterator.forEachRemaining(Unknown Source) ~[na:na]
	at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source) ~[na:na]
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) ~[na:na]
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source) ~[na:na]
	at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[na:na]
	at java.base/java.util.stream.ReferencePipeline.collect(Unknown Source) ~[na:na]
	at org.jobrunr.storage.sql.common.JobTable.selectJobsByStateAndDeleteAt(JobTable.java:206) ~[jobrunr-pro-6.2.4.jar:6.2.4]
	at org.jobrunr.storage.sql.common.DefaultSqlStorageProvider.getJobsByStateAndDeleteAt(DefaultSqlStorageProvider.java:309) ~[jobrunr-pro-6.2.4.jar:6.2.4]
	at org.jobrunr.storage.ThreadSafeStorageProvider.getJobsByStateAndDeleteAt(ThreadSafeStorageProvider.java:177) ~[jobrunr-pro-6.2.4.jar:6.2.4]
	at org.jobrunr.server.zookeeper.tasks.DeleteSucceededJobsTask.lambda$runTask$0(DeleteSucceededJobsTask.java:26) ~[jobrunr-pro-6.2.4.jar:6.2.4]
	at org.jobrunr.server.zookeeper.tasks.ZooKeeperTask.getJobsToProcess(ZooKeeperTask.java:91) ~[jobrunr-pro-6.2.4.jar:6.2.4]
	at org.jobrunr.server.zookeeper.tasks.ZooKeeperTask.processJobList(ZooKeeperTask.java:57) ~[jobrunr-pro-6.2.4.jar:6.2.4]
	at org.jobrunr.server.zookeeper.tasks.DeleteSucceededJobsTask.runTask(DeleteSucceededJobsTask.java:27) ~[jobrunr-pro-6.2.4.jar:6.2.4]
	at org.jobrunr.server.zookeeper.tasks.ZooKeeperTask.run(ZooKeeperTask.java:47) ~[jobrunr-pro-6.2.4.jar:6.2.4]
	at org.jobrunr.server.JobZooKeeper.lambda$runMasterTasksIfCurrentServerIsMaster$0(JobZooKeeper.java:79) ~[jobrunr-pro-6.2.4.jar:6.2.4]
	at java.base/java.util.Arrays$ArrayList.forEach(Unknown Source) ~[na:na]
	at org.jobrunr.server.JobZooKeeper.runMasterTasksIfCurrentServerIsMaster(JobZooKeeper.java:79) ~[jobrunr-pro-6.2.4.jar:6.2.4]
	at org.jobrunr.server.JobZooKeeper.run(JobZooKeeper.java:59) ~[jobrunr-pro-6.2.4.jar:6.2.4]
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:na]
	at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) ~[na:na]
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:na]
	at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]
Caused by: com.fasterxml.jackson.databind.JsonMappingException: String length (5046272) exceeds the maximum length (5000000) (through reference chain: org.jobrunr.jobs.Job["metadata"]->java.util.concurrent.ConcurrentHashMap["jobRunrDashboardLog-2"]->org.jobrunr.jobs.context.JobDashboardLogger$JobDashboardLogLines["logLines"]->java.util.concurrent.ConcurrentLinkedQueue[3]->org.jobrunr.jobs.context.JobDashboardLogger$JobDashboardLogLine["logMessage"])
	at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:402) ~[jackson-databind-2.15.0.jar:2.15.0]
	at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:361) ~[jackson-databind-2.15.0.jar:2.15.0]
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.wrapAndThrow(BeanDeserializerBase.java:1830) ~[jackson-databind-2.15.0.jar:2.15.0]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:316) ~[jackson-databind-2.15.0.jar:2.15.0]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:177) ~[jackson-databind-2.15.0.jar:2.15.0]
	at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer._deserializeFromArray(CollectionDeserializer.java:359) ~[jackson-databind-2.15.0.jar:2.15.0]
	at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:244) ~[jackson-databind-2.15.0.jar:2.15.0]
	at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:28) ~[jackson-databind-2.15.0.jar:2.15.0]
	at com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:138) ~[jackson-databind-2.15.0.jar:2.15.0]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:314) ~[jackson-databind-2.15.0.jar:2.15.0]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:215) ~[jackson-databind-2.15.0.jar:2.15.0]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:187) ~[jackson-databind-2.15.0.jar:2.15.0]
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:170) ~[jackson-databind-2.15.0.jar:2.15.0]
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:136) ~[jackson-databind-2.15.0.jar:2.15.0]
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromAny(AsPropertyTypeDeserializer.java:240) ~[jackson-databind-2.15.0.jar:2.15.0]
	at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializerNR.deserializeWithType(UntypedObjectDeserializerNR.java:115) ~[jackson-databind-2.15.0.jar:2.15.0]
	at com.fasterxml.jackson.databind.deser.std.MapDeserializer._readAndBindStringKeyMap(MapDeserializer.java:625) ~[jackson-databind-2.15.0.jar:2.15.0]
	at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:449) ~[jackson-databind-2.15.0.jar:2.15.0]
	at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:32) ~[jackson-databind-2.15.0.jar:2.15.0]
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:170) ~[jackson-databind-2.15.0.jar:2.15.0]
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:136) ~[jackson-databind-2.15.0.jar:2.15.0]
	at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserializeWithType(MapDeserializer.java:492) ~[jackson-databind-2.15.0.jar:2.15.0]
	at com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:147) ~[jackson-databind-2.15.0.jar:2.15.0]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:314) ~[jackson-databind-2.15.0.jar:2.15.0]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:215) ~[jackson-databind-2.15.0.jar:2.15.0]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:187) ~[jackson-databind-2.15.0.jar:2.15.0]
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:170) ~[jackson-databind-2.15.0.jar:2.15.0]
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:136) ~[jackson-databind-2.15.0.jar:2.15.0]
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeWithType(BeanDeserializerBase.java:1296) ~[jackson-databind-2.15.0.jar:2.15.0]
	at com.fasterxml.jackson.databind.deser.impl.TypeWrappedDeserializer.deserialize(TypeWrappedDeserializer.java:74) ~[jackson-databind-2.15.0.jar:2.15.0]
	at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323) ~[jackson-databind-2.15.0.jar:2.15.0]
	at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4825) ~[jackson-databind-2.15.0.jar:2.15.0]
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3772) ~[jackson-databind-2.15.0.jar:2.15.0]
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3740) ~[jackson-databind-2.15.0.jar:2.15.0]
	at org.jobrunr.utils.mapper.jackson.JacksonJsonMapper.deserialize(JacksonJsonMapper.java:84) ~[jobrunr-pro-6.2.4.jar:6.2.4]
	... 28 common frames omitted

[rdehuyss]

Caused by FasterXML/jackson-core#863.

[rdehuyss]

String limit has been increased in Jackson 2.15.1 from 5MB to 20MB. See FasterXML/jackson-core#1014

I think this makes a sensible value. Can you upgrade Jackson to 2.15.1? Or even better, update Spring Boot to v3.1.1.

[bryanbolles]

Sure we can change the spring boot version, the real issue is that the exception caused the server to crash over and over.

[rdehuyss]

I understand but don't think there is anything I can do.

If it can't deserialize a job, something is really wrong. You have another idea? Or what would be your preferred approach?

[pdavidson]

@rdehuyss the server couldn't get the job, it was sitting in the succeeded jobs and we ended up having to truncate the table to get it working again. I'm not sure what I would do in that situation.