You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
I was running qualification tool and writing to DBFS and it takes a really long time to write the csv files. It took like 10 hours to run qualification tool on 7 different batches of 1000 event logs, where the parsing of event log itself was very fast but writing of the summary csv, exec and stages csv took a very long time. I'm not sure if this is any different than S3, we may want to measure. Perhaps it would be faster to write to local disk and then copy into DBFS or S3.
investigate more and make changes or document best practices
The text was updated successfully, but these errors were encountered:
Describe the bug
I was running qualification tool and writing to DBFS and it takes a really long time to write the csv files. It took like 10 hours to run qualification tool on 7 different batches of 1000 event logs, where the parsing of event log itself was very fast but writing of the summary csv, exec and stages csv took a very long time. I'm not sure if this is any different than S3, we may want to measure. Perhaps it would be faster to write to local disk and then copy into DBFS or S3.
investigate more and make changes or document best practices
The text was updated successfully, but these errors were encountered: