You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This ticket requests that a mechanism is provided to set a default value of the --parallelism flag for the lakectl local commands. Currently, it defaults to 25 and must be explicitly specified on the command line in order to change the default.
It would be fantastic to allow an environment variable and/or config file to set the default value here, similarly to how some of the configs here: https://docs.lakefs.io/reference/configuration.html influence the local CLI, and allow for use of env vars to set them.
Motivation: I have seen a big performance improvement in lakectl local when working with lots of smaller files with an increased --parallelism flag. Rather than needing to remember to set this flag each time, wrap the tool in helper scripts, etc. it would be great if there was a way to provide a higher custom default directly to the tool.
This ticket requests that a mechanism is provided to set a default value of the
--parallelism
flag for thelakectl local
commands. Currently, it defaults to 25 and must be explicitly specified on the command line in order to change the default.It would be fantastic to allow an environment variable and/or config file to set the default value here, similarly to how some of the configs here: https://docs.lakefs.io/reference/configuration.html influence the local CLI, and allow for use of env vars to set them.
Motivation: I have seen a big performance improvement in
lakectl local
when working with lots of smaller files with an increased--parallelism
flag. Rather than needing to remember to set this flag each time, wrap the tool in helper scripts, etc. it would be great if there was a way to provide a higher custom default directly to the tool.First asked about here: https://lakefs.slack.com/archives/C016726JLJW/p1725922712803829
The text was updated successfully, but these errors were encountered: