Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimizations for large builds #37

Merged
merged 34 commits into from
Mar 2, 2020
Merged

Optimizations for large builds #37

Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
01c2e29
Print memory allocated.
aras-p Feb 29, 2020
91671fe
Optimize: --stop produces binary data dump instead of JSON; JSON pars…
aras-p Feb 29, 2020
4d4a6e6
Encapsulate internal JSON parsing & data building state, saves a bit …
aras-p Feb 29, 2020
b821b02
Use precomputed string hashes in nameToIndex map:
aras-p Feb 29, 2020
ff9f42a
Replace unordered_map for name deduplication with Malte Skarupke's by…
aras-p Feb 29, 2020
151569e
Reuse string buffer memory for reading JSON files into.
aras-p Feb 29, 2020
862dd68
Optimize file writing by using a buffer instead of tiny fwrite calls.
aras-p Feb 29, 2020
3009485
Optimize file reading by not doing tiny fread calls.
aras-p Feb 29, 2020
2dfbdd1
Optimize EmitCollapsedTemplateOpt.
aras-p Feb 29, 2020
e02e42c
Fixing Linux & Windows builds
aras-p Feb 29, 2020
bad7fa1
Remove "bump pointer, never free" allocator. It's fast, but huge memo…
aras-p Mar 1, 2020
1416fb5
Add enkiTS task scheduler library from master branch 3686b99
aras-p Mar 1, 2020
ae993bd
Add xxHash hashing library from dev branch 5dd837e
aras-p Mar 1, 2020
d417f81
Use arena allocator for nameToIndex string names, and xxHash (XXH64) …
aras-p Mar 1, 2020
055e853
Use arena allocator for all BuildNames and other names produced durin…
aras-p Mar 1, 2020
4452f1c
Replace sajson with simdjson (master branch 140e4dd)
aras-p Mar 1, 2020
9659ba6
Change code to use simdjson
aras-p Mar 1, 2020
1a5a87c
Multi-threaded JSON parsing
aras-p Mar 1, 2020
70b7430
Update Make & CMake projects
aras-p Mar 1, 2020
00e9ed3
Set timeout for Github Actions (5min), and test CMake path in Linux b…
aras-p Mar 1, 2020
c2ed649
Fixing cmake test action
aras-p Mar 2, 2020
d3dc4a2
Actions: try to use gcc 7 on Linux
aras-p Mar 2, 2020
7ba5db4
Actions: try to use gcc 7 on Linux
aras-p Mar 2, 2020
6142f41
Actions: try to use gcc 7 on Linux
aras-p Mar 2, 2020
c009810
Windows: switch to VS2019 and fix project files
aras-p Mar 2, 2020
bbfe83b
Update readme
aras-p Mar 2, 2020
7d74e74
Linux: link to -lpthread
aras-p Mar 2, 2020
5ecdb30
Fix double fclose
aras-p Mar 2, 2020
6f4597d
Make data file have header "magic" and a checksum hash, to avoid blin…
aras-p Mar 2, 2020
a7418ce
Move filename prettifying to --stop phase instead of doing it in --an…
aras-p Mar 2, 2020
c705536
Move name demangling to --stop phase instead of doing it in --analyze
aras-p Mar 2, 2020
561c5cb
Use ska::bytell_hash_map in analysis phase as well.
aras-p Mar 2, 2020
36aa6d5
In "Function sets that took longest to compile / optimize" (EmitColla…
aras-p Mar 2, 2020
258c624
Fixing Linux build
aras-p Mar 2, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,3 +6,5 @@ xcuserdata
xcschemes
_AnalysisOutput.txt
_TraceOutput.json
_TraceOutput.bin

12 changes: 12 additions & 0 deletions projects/xcode/ClangBuildAnalyzer.xcodeproj/project.pbxproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,8 @@
2B0993262308259400344A93 /* cute_files.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = cute_files.h; sourceTree = "<group>"; };
2B09932A2309600400344A93 /* BuildEvents.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = BuildEvents.h; sourceTree = "<group>"; };
2B09932B2309600400344A93 /* BuildEvents.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = BuildEvents.cpp; sourceTree = "<group>"; };
2B3092D3240AEE8A002A24F4 /* bytell_hash_map.hpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.h; path = bytell_hash_map.hpp; sourceTree = "<group>"; };
2B3092D4240AEE8A002A24F4 /* flat_hash_map.hpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.h; path = flat_hash_map.hpp; sourceTree = "<group>"; };
2B6FBE04230B0D8100095E82 /* Analysis.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = Analysis.h; sourceTree = "<group>"; };
2B6FBE05230B0D8100095E82 /* Analysis.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = Analysis.cpp; sourceTree = "<group>"; };
2B6FBE07230B280400095E82 /* Utils.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = Utils.cpp; sourceTree = "<group>"; };
Expand Down Expand Up @@ -114,6 +116,7 @@
2B09931A23080EF500344A93 /* external */ = {
isa = PBXGroup;
children = (
2B3092D1240AEE6D002A24F4 /* flat_hash_map */,
2B6FBE1D230BD70100095E82 /* sajson.h */,
2B6FBE0B230BB8DF00095E82 /* llvm-Demangle */,
2B09931C23080F0E00344A93 /* inih */,
Expand All @@ -134,6 +137,15 @@
path = inih;
sourceTree = "<group>";
};
2B3092D1240AEE6D002A24F4 /* flat_hash_map */ = {
isa = PBXGroup;
children = (
2B3092D3240AEE8A002A24F4 /* bytell_hash_map.hpp */,
2B3092D4240AEE8A002A24F4 /* flat_hash_map.hpp */,
);
path = flat_hash_map;
sourceTree = "<group>";
};
2B6FBE0B230BB8DF00095E82 /* llvm-Demangle */ = {
isa = PBXGroup;
children = (
Expand Down
1 change: 1 addition & 0 deletions readme.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,7 @@ License for the Clang Build Analyzer itself is [Unlicense](https://unlicense.org
includes several external library source files (all under `src/external`), each with their own license:

* `cute_files.h` from [RandyGaul/cute_headers](https://github.com/RandyGaul/cute_headers): zlib or public domain,
* `flat_hash_map`, from [skarupke/flat_hash_map](https://github.com/skarupke/flat_hash_map): Boost 1.0,
* `inih`, from [benhoyt/inih](https://github.com/benhoyt/inih): BSD 3 clause,
* `llvm-Demangle`, part of [LLVM](https://llvm.org/): Apache-2.0 with LLVM-exception,
* `sajson.h` from [chadaustin/sajson](https://github.com/chadaustin/sajson): MIT,
Expand Down
7 changes: 7 additions & 0 deletions src/Allocator.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ const size_t kBlockSize = 32 * 1024 * 1024;
static void* s_CurBlock = nullptr;
static size_t s_CurBlockUsed = 0;
static size_t s_CurBlockSize = 0;
static size_t s_TotalAllocs = 0;


void* operator new(size_t count)
Expand Down Expand Up @@ -52,9 +53,15 @@ void* operator new(size_t count)

void* res = (char*)s_CurBlock + s_CurBlockUsed;
s_CurBlockUsed += count;
s_TotalAllocs += count;
return res;
}

void operator delete(void * p) throw()
{
}

size_t GetTotalAllocatedBytes()
{
return s_TotalAllocs;
}
13 changes: 11 additions & 2 deletions src/Analysis.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@

#ifdef _MSC_VER
struct IUnknown; // workaround for old Win SDK header failures when using /permissive-
#define NOMINMAX
#endif

#include "Analysis.h"
Expand Down Expand Up @@ -360,9 +361,17 @@ void Analysis::EmitCollapsedTemplates()
void Analysis::EmitCollapsedTemplateOpt()
{
std::unordered_map<std::string, InstantiateEntry> collapsed;
std::unordered_map<DetailIndex, std::string> collapsedNameCache;

for (const auto& fn : functions)
{
auto &stats = collapsed[collapseName(llvm::demangle(GetBuildName(fn.first.first)))];
aras-p marked this conversation as resolved.
Show resolved Hide resolved
auto fnName = fn.first.first;
auto fnCacheIt = collapsedNameCache.find(fnName);
if (fnCacheIt == collapsedNameCache.end())
{
fnCacheIt = collapsedNameCache.insert(std::make_pair(fnName, collapseName(llvm::demangle(GetBuildName(fnName))))).first;
}
auto &stats = collapsed[fnCacheIt->second];
++stats.count;
stats.us += fn.second;
}
Expand Down Expand Up @@ -449,7 +458,7 @@ void Analysis::EndAnalysis()
if (dname.size() > config.maxName)
dname = dname.substr(0, config.maxName-2) + "...";
int ms = int(e.second.us / 1000);
int avg = int(ms / e.second.count);
int avg = int(ms / std::max(e.second.count,1));
aras-p marked this conversation as resolved.
Show resolved Hide resolved
fprintf(out, "%s%6i%s ms: %s (%i times, avg %i ms)\n", col::kBold, ms, col::kReset, dname.c_str(), e.second.count, avg);
}
fprintf(out, "\n");
Expand Down
Loading