Reduce memory and CPU use when scanning

[bradlarsen]

Rework git metadata calculation to reduce peak memory and wall clock time when scanning. This includes many changes. The net effect of all this is a typical 30% speedup and 50% memory reduction when scanning Git repositories; in pathological cases, up to 5x speedup and 20x memory reduction.

• Git metadata graph:

  • Do not construct in-memory graph of all trees and blob names; instead, read tree objects from repo as late as possible
  • Use SmallVec to reduce heap fragmentation and small heap allocations
  • Use more suitable initial size for worklists and scratch buffers to reduce reallocations and heap fragmentation
  • Use the fastest / slimmest order for iterating object headers from a git repository when initially counting objects
  • Eliminate redundant intermediate data structures; remove unused fields from remaining intermediate data structures
  • Avoid temporary allocations when concatenating blob filenames
  • Fix a longstanding bug where a blob introduced multiple times within a single commit would have only a single arbitrary pathname reported

• BStringTable:

  • change default initialization to create an empty table
  • change get_or_intern to avoid heap-allocated temporaries when an entry already exists

• Scanning:

  • Use an Arc<CommitMetadata> instead of CommitMetadata and Arc<PathBuf> instead of PathBuf within git blob provenance entries (allows sharing; sometimes reduces memory use of these object types 10,000x)
  • Use a higher default level of parallelism: require 3GiB RAM instead of 4GiB per parallel job
  • Open Git repositories a single time instead of twice