Code

Optimizations for large builds#37

Code

Merged

aras-p merged 34 commits intomasteraras-p/ClangBuildAnalyzer:masterfrom optimize-large-buildsaras-p/ClangBuildAnalyzer:optimize-large-buildsCopy head branch name to clipboardMar 2, 2020

+21,672-3,012

Commits

Commits on Feb 29, 2020

• Print memory allocated.

  Show description for 01c2e29
  aras-p

  committed
  01c2e29

• Optimize: --stop produces binary data dump instead of JSON; JSON parsing and name deduplication happens during --stop, not during --analyze.

  Show description for 91671fe
  aras-p

  committed
  91671fe

• Encapsulate internal JSON parsing & data building state, saves a bit of memory too:

  Show description for 4d4a6e6
  aras-p

  committed
  4d4a6e6

• Use precomputed string hashes in nameToIndex map:

  Show description for b821b02
  aras-p

  committed
  b821b02

• Replace unordered_map for name deduplication with Malte Skarupke's bytell_hash_map (https://probablydance.com/2018/05/28/a-new-fast-hash-table-in-response-to-googles-new-fast-hash-table/)

  Show description for ff9f42a
  aras-p

  committed
  ff9f42a

• Reuse string buffer memory for reading JSON files into.

  Show description for 151569e
  aras-p

  committed
  151569e

• Optimize file writing by using a buffer instead of tiny fwrite calls.

  Show description for 862dd68
  aras-p

  committed
  862dd68

• Optimize file reading by not doing tiny fread calls.

  Show description for 3009485
  aras-p

  committed
  3009485

• Optimize EmitCollapsedTemplateOpt.

  Show description for 2dfbdd1
  aras-p

  committed
  2dfbdd1

• Fixing Linux & Windows builds

  aras-p

  committed
  e02e42c

Commits on Mar 1, 2020

• Remove "bump pointer, never free" allocator. It's fast, but huge memory usage on large codebases is a problem, and is not thread safe either.

  Show description for bad7fa1
  aras-p

  committed
  bad7fa1

• Add enkiTS task scheduler library from master branch 3686b99

  aras-p

  committed
  1416fb5

• Add xxHash hashing library from dev branch 5dd837e

  aras-p

  committed
  ae993bd

• Use arena allocator for nameToIndex string names, and xxHash (XXH64) hash function.

  Show description for d417f81
  aras-p

  committed
  d417f81

• Use arena allocator for all BuildNames and other names produced during analysis.

  Show description for 055e853
  aras-p

  committed
  055e853

• Replace sajson with simdjson (master branch 140e4dd)

  aras-p

  committed
  4452f1c

• Change code to use simdjson

  Show description for 9659ba6
  aras-p

  committed
  9659ba6

• Multi-threaded JSON parsing

  Show description for 1a5a87c
  aras-p

  committed
  1a5a87c

• Update Make & CMake projects

  aras-p

  committed
  70b7430

• Set timeout for Github Actions (5min), and test CMake path in Linux builder too

  aras-p

  committed
  00e9ed3

Commits on Mar 2, 2020

• Fixing cmake test action

  aras-p

  committed
  c2ed649

• Actions: try to use gcc 7 on Linux

  aras-p

  committed
  d3dc4a2

• Actions: try to use gcc 7 on Linux

  aras-p

  committed
  7ba5db4

• Actions: try to use gcc 7 on Linux

  aras-p

  committed
  6142f41

• Windows: switch to VS2019 and fix project files

  aras-p

  committed
  c009810

• Update readme

  aras-p

  committed
  bbfe83b

• Linux: link to -lpthread

  aras-p

  committed
  7d74e74

• Fix double fclose

  aras-p

  committed
  5ecdb30

• Make data file have header "magic" and a checksum hash, to avoid blindly loading wrong files

  aras-p

  committed
  6f4597d

• Move filename prettifying to --stop phase instead of doing it in --analyze

  Show description for a7418ce
  aras-p

  committed
  a7418ce

• Move name demangling to --stop phase instead of doing it in --analyze

  Show description for c705536
  aras-p

  committed
  c705536

• Use ska::bytell_hash_map in analysis phase as well.

  Show description for 561c5cb
  aras-p

  committed
  561c5cb

• In "Function sets that took longest to compile / optimize" (EmitCollapsedTemplateOpt), only actually process templated functions.

  Show description for 36aa6d5
  aras-p

  committed
  36aa6d5

• Fixing Linux build

  aras-p

  committed
  258c624