Use of recursion rather than iteration in CompoundProcessor limits ingest pipeline length #84274
Labels
:Data Management/Ingest Node
Execution or management of Ingest Pipelines including GeoIP
Team:Data Management
Meta label for data/management team
A case where somebody was trying to import a CSV file with 3000 numeric fields revealed that it's possible to get a stack overflow exception when executing an ingest pipeline with many processors.
The ingest pipeline was ingest_pipeline.json, consisting of a CSV processor to parse the CSV followed by 3000 convert processors to convert the strings parsed from the CSV to numbers.
On executing this pipeline it fails with a stack overflow exception:
(The stack trace continues with the same 3 calls over and over again.)
Could
CompoundProcessor.innerExecute
be changed to use iteration rather than recursion to avoid this?The sample CSV file that goes with the ingest pipeline is test.csv.
The text was updated successfully, but these errors were encountered: