outputs.content exceeds maximum allowed memory size

[lehmanju]

We are trying to use git-cliff with about 200 repositories at the same time and generate one changelog for it. This results in the github action crashing because of the output being too large. Is there an option to remove the output/make it optional if its too long?

Error message: orhun/git-cliff-action/v4/action.yml (Line: 26, Col: 12): The maximum allowed memory size was exceeded while evaluating the following expression: steps.run-git-cliff.outputs.content

Also, maybe there is some consideration to improve git-cliffs performance with multithreading? Running with so many repos takes quite some time to finish.

[orhun]

Hey, you have a pretty interesting setup going on there!

Is there an option to remove the output/make it optional if its too long?

Yup, I think it is doable. Do you know the maximum length of output that we can use?

Also, maybe there is some consideration to improve git-cliffs performance with multithreading?

There were some performance optimizations in the last releases but I haven't got to work on parallelization yet. First step would be finding the bottlenecks and see if we can possibly add caching. Do you mind sharing more information about your setup and the git-cliff command you are running?

[lehmanju]

The maximum output seems to be defined by size: https://docs.github.com/en/actions/sharing-automations/creating-actions/metadata-syntax-for-github-actions#outputs-for-composite-actions

I helped myself by using head --lines 50 instead of cat.

[orhun]

I see, thanks for sharing the docs about maximum size. Do you still need to optionally disable the output?

[lehmanju]

Whether it's disabled or not, I don't care to be honest 😄 - just that it shouldn't crash on long changelogs.

[orhun]

I see, created #40 for limiting the size. Can you check?

[lehmanju]

nope, issue still exists, although the error message is different:

The template is not valid. System.InvalidOperationException: Maximum object size exceeded
   at GitHub.DistributedTask.ObjectTemplating.TemplateMemory.AddBytes(Int32 bytes)
   at GitHub.DistributedTask.ObjectTemplating.TemplateUnraveler.ReaderState.CreateState(ReaderState parent, TemplateToken value, TemplateContext context, Int32 removeBytes)
   at GitHub.DistributedTask.ObjectTemplating.TemplateUnraveler.MappingValueBasicExpression()
   at GitHub.DistributedTask.ObjectTemplating.TemplateUnraveler.Unravel(Boolean expand)
   at GitHub.DistributedTask.ObjectTemplating.TemplateUnraveler.AllowScalar(Boolean expand, ScalarToken& scalar)
   at GitHub.DistributedTask.ObjectTemplating.TemplateEvaluator.Evaluate(DefinitionInfo definition)
   at GitHub.DistributedTask.ObjectTemplating.TemplateEvaluator.HandleMappingWithWellKnownProperties(DefinitionInfo definition, List`1 mappingDefinitions, MappingToken mapping)
   at GitHub.DistributedTask.ObjectTemplating.TemplateEvaluator.Evaluate(DefinitionInfo definition)
   at GitHub.DistributedTask.ObjectTemplating.TemplateEvaluator.HandleMappingWithAllLooseProperties(DefinitionInfo mappingDefinition, DefinitionInfo keyDefinition, DefinitionInfo valueDefinition, MappingToken mapping)
   at GitHub.DistributedTask.ObjectTemplating.TemplateEvaluator.Evaluate(DefinitionInfo definition)
   at GitHub.DistributedTask.ObjectTemplating.TemplateEvaluator.Evaluate(TemplateContext context, String type, TemplateToken template, Int32 removeBytes, Nullable`1 fileId, Boolean omitHeader)

Maybe this time it's about stdout? the log has to be truncated there as well I would guess

[lehmanju]

Also, 50MB references the sum of all outputs, not just one output value.

[lehmanju]

Oh, and the corresponding changelog file ist only 13MB

[orhun]

Also, 50MB references the sum of all outputs, not just one output value.

Oh, that's a bit more work than I anticipated then :D

Maybe this time it's about stdout? the log has to be truncated there as well I would guess

We can probably disable this based on output size as well:

git-cliff-action/run.sh

Line 28 in 14bb673

cat "$OUTPUT"

But I'm not sure if that'll work 🤔

[lehmanju]

Following worked for me:

FILESIZE=$(stat -c%s "$OUTPUT")
MAXSIZE=$((40 * 1024 * 1024))
if [ "$FILESIZE" -le "$MAXSIZE" ]; then
  echo "content<<EOF" >>$GITHUB_OUTPUT
  cat "$OUTPUT" >>$GITHUB_OUTPUT
  echo "EOF" >>$GITHUB_OUTPUT
  cat "$OUTPUT"
fi

I reduced the maxsize a little to account for other output. For now, it just works.

[orhun]

Awesome, thanks for testing!