Use measureme in self profiler #59515

wesleywiser · 2019-03-29T03:06:22Z

~~Changes are still very rough.~~

~~I'm not sure what the right way to add the measureme dependency is. Currently it's just added with a relative path which Works On My Machine ™️.~~

I'm also not sure what to do with the category data.

src/librustc/util/profiling.rs

michaelwoerister · 2019-04-02T09:42:20Z

Looks like a good start! Eventually I'd like to have query names instead of dep-node names here but I don't know how to exactly do that yet.

Have you done performance tests on Windows? If the mmap-sink performance worse than the file-sink there, I think I'd like to get rid of the memmap dependency and implement the mmap-sink only on Unix via direct libc calls (we'll need some stuff from libc that memmap doesn't support anyway). On Windows we'd just use the file-sink.

For now, I think we should just ignore the category data. Later we could encode it in the event_id if we wanted to.

I think we can just publish measureme on crates.io and have lots of major version changes. The other option would be a git dependency. But that's something we should discuss with wg-meta.

michaelwoerister · 2019-04-03T08:31:27Z

For reference: https://rust-lang.zulipchat.com/#narrow/stream/185694-t-compiler.2Fwg-meta/topic/out-of-tree.20crates.20-.20crates.2Eio.20or.20git.20dependency

wesleywiser · 2019-04-04T13:56:16Z

Looks like a good start! Eventually I'd like to have query names instead of dep-node names here but I don't know how to exactly do that yet.

Originally I passed Query<'tcx>s into the profiler methods but I couldn't figure out how to preregister the query names so I had to use a FxHashSet to keep track of which query names had been interned into the string table and which hadn't. Maybe the overhead of that could be reduced more but removing it in favor of using dep-node names and pre-registering the names made a noticeable difference in performance.

Have you done performance tests on Windows?

I've been working on getting rustc to build on my Windows box. It's been a long time since I used it and my msvc toolchain was completely out of date. I think that's resolved now and I'm just waiting for LLVM to build. I should have the performance data tonight.

For now, I think we should just ignore the category data.

Ok that's fine. I'll remove the other unused _category parameters as well.

I think we can just publish measureme on crates.io and have lots of major version changes. The other option would be a git dependency. But that's something we should discuss with wg-meta.

I'd personally vote for publishing on crates.io

wesleywiser · 2019-04-05T00:03:26Z

Results of testing on Windows:

Test Case	Time Taken
regex rustc	5.02s
regex rustc + `-Z self-profile` `FileSerializationSink`	5.92s
regex rustc + `-Z self-profile` `MmapSerializationSink`	6.04s

michaelwoerister · 2019-04-05T08:42:41Z

Great, thanks for getting those numbers. Let's use the FileSerializationSink on Windows then.

michaelwoerister · 2019-04-05T12:05:22Z

Eventually I'd like to have query names instead of dep-node names here but I don't know how to exactly do that yet.

It looks like a recent PR fixed this for us by making DepKind variants and query names match up :)

wesleywiser · 2019-04-05T13:11:15Z

Oh, wonderful! I can make that change tonight. Do we have consensus on how to add the measureme dependency? I think that's the last thing that needs a fixup before this can be merged.

michaelwoerister · 2019-04-05T13:35:03Z

Let's go via crates.io, I'd say.

michaelwoerister · 2019-04-05T13:42:58Z

Note that Q::to_dep_node() is potentially expensive and in non-incremental builds it's not called at all, so we'll have to cache the query name StringId afterall. Ideally it would be created at the same time as the query caches and stored in a field somewhere.

wesleywiser · 2019-04-08T03:13:25Z

@michaelwoerister Do you like the approach in edff02e better?

michaelwoerister · 2019-04-09T10:13:26Z

I do -- but it still has some runtime overhead for cloning the query key. Could you try to add an associated constant to QueryConfig that holds the right QueryName variant? Then we could pass the QueryName directly to the record methods. You could maybe even replace the current NAME constant (not sure if it's used anywhere else).

wesleywiser · 2019-04-09T11:26:40Z

Oh yeah, that's a better idea

src/librustc/session/mod.rs

src/librustc/util/profiling.rs

michaelwoerister · 2019-04-10T09:45:28Z

I left a few comments. Looks like this is almost ready to merge.

wesleywiser · 2019-04-10T12:26:38Z

Thanks for the feedback @michaelwoerister! I've made those changes, pushed, and marked this PR as ready for review.

wesleywiser · 2019-04-13T01:56:10Z

@bors r=michaelwoerister

bors · 2019-04-13T01:56:12Z

📌 Commit 56e434d has been approved by michaelwoerister

@wesleywiser

[WIP] Implement event filtering for self-profiler. This is a first sketch for event filtering in the self-profiler, something that we'll want in order to keep profiling overhead low in the common case. The PR contains the commits from #59515 and, for the moment, is meant for performance testing. r? @wesleywiser

bors · 2019-04-13T11:54:33Z

⌛ Testing commit 56e434d with merge 896c3a5...

@michaelwoerister

Use measureme in self profiler r? @michaelwoerister ~Changes are still very rough.~ ~I'm not sure what the right way to add the `measureme` dependency is. Currently it's just added with a relative path which Works On My Machine ™️.~ I'm also not sure what to do with the category data.

bors · 2019-04-13T14:45:38Z

☀️ Test successful - checks-travis, status-appveyor
Approved by: michaelwoerister
Pushing 896c3a5 to master...

@wesleywiser

…, r=wesleywiser Implement event filtering for self-profiler. This is a first sketch for event filtering in the self-profiler, something that we'll want in order to keep profiling overhead low in the common case. The PR contains the commits from rust-lang#59515 and, for the moment, is meant for performance testing. r? @wesleywiser

@wesleywiser

…, r=wesleywiser Implement event filtering for self-profiler. This is a first sketch for event filtering in the self-profiler, something that we'll want in order to keep profiling overhead low in the common case. The PR contains the commits from rust-lang#59515 and, for the moment, is meant for performance testing. r? @wesleywiser

@wesleywiser

…, r=wesleywiser Implement event filtering for self-profiler. This is a first sketch for event filtering in the self-profiler, something that we'll want in order to keep profiling overhead low in the common case. The PR contains the commits from rust-lang#59515 and, for the moment, is meant for performance testing. r? @wesleywiser

@wesleywiser

…, r=wesleywiser Implement event filtering for self-profiler. This is a first sketch for event filtering in the self-profiler, something that we'll want in order to keep profiling overhead low in the common case. The PR contains the commits from rust-lang#59515 and, for the moment, is meant for performance testing. r? @wesleywiser

@wesleywiser

…, r=wesleywiser Implement event filtering for self-profiler. This is a first sketch for event filtering in the self-profiler, something that we'll want in order to keep profiling overhead low in the common case. The PR contains the commits from rust-lang#59515 and, for the moment, is meant for performance testing. r? @wesleywiser

@wesleywiser

…, r=wesleywiser Implement event filtering for self-profiler. This is a first sketch for event filtering in the self-profiler, something that we'll want in order to keep profiling overhead low in the common case. The PR contains the commits from rust-lang#59515 and, for the moment, is meant for performance testing. r? @wesleywiser

@wesleywiser

…, r=wesleywiser Implement event filtering for self-profiler. This is a first sketch for event filtering in the self-profiler, something that we'll want in order to keep profiling overhead low in the common case. The PR contains the commits from rust-lang#59515 and, for the moment, is meant for performance testing. r? @wesleywiser

@wesleywiser

…, r=wesleywiser Implement event filtering for self-profiler. This is a first sketch for event filtering in the self-profiler, something that we'll want in order to keep profiling overhead low in the common case. The PR contains the commits from rust-lang#59515 and, for the moment, is meant for performance testing. r? @wesleywiser

@wesleywiser

…, r=wesleywiser Implement event filtering for self-profiler. This is a first sketch for event filtering in the self-profiler, something that we'll want in order to keep profiling overhead low in the common case. The PR contains the commits from rust-lang#59515 and, for the moment, is meant for performance testing. r? @wesleywiser

rust-highfive assigned michaelwoerister Mar 29, 2019

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Mar 29, 2019

wesleywiser force-pushed the measureme branch from 1ae1ddc to 389ff90 Compare April 1, 2019 03:08

michaelwoerister reviewed Apr 2, 2019

View reviewed changes

src/librustc/util/profiling.rs Outdated Show resolved Hide resolved

wesleywiser force-pushed the measureme branch from 389ff90 to 461b91a Compare April 3, 2019 00:16

wesleywiser force-pushed the measureme branch from 461b91a to 62000d9 Compare April 5, 2019 00:10

wesleywiser force-pushed the measureme branch from 62000d9 to edff02e Compare April 8, 2019 03:12

wesleywiser force-pushed the measureme branch from edff02e to ea0754f Compare April 9, 2019 12:09

This was referenced Apr 10, 2019

Change dependencies to match existing rustc deps rust-lang/measureme#19

Merged

Release to crates.io rust-lang/measureme#20

Closed

michaelwoerister reviewed Apr 10, 2019

View reviewed changes

src/librustc/session/mod.rs Show resolved Hide resolved

michaelwoerister reviewed Apr 10, 2019

View reviewed changes

src/librustc/util/profiling.rs Outdated Show resolved Hide resolved

michaelwoerister reviewed Apr 10, 2019

View reviewed changes

src/librustc/util/profiling.rs Outdated Show resolved Hide resolved

wesleywiser force-pushed the measureme branch from ea0754f to cdc68c7 Compare April 10, 2019 12:24

wesleywiser marked this pull request as ready for review April 10, 2019 12:24

wesleywiser force-pushed the measureme branch from cdc68c7 to cab7e5f Compare April 10, 2019 12:25

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Apr 13, 2019

bors added the merged-by-bors This PR was explicitly merged by bors. label Apr 13, 2019

bors merged commit 56e434d into rust-lang:master Apr 13, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use measureme in self profiler #59515

Use measureme in self profiler #59515

wesleywiser commented Mar 29, 2019 •

edited

Loading

michaelwoerister commented Apr 2, 2019

michaelwoerister commented Apr 3, 2019

wesleywiser commented Apr 4, 2019

wesleywiser commented Apr 5, 2019

michaelwoerister commented Apr 5, 2019

michaelwoerister commented Apr 5, 2019

wesleywiser commented Apr 5, 2019

michaelwoerister commented Apr 5, 2019

michaelwoerister commented Apr 5, 2019

wesleywiser commented Apr 8, 2019

michaelwoerister commented Apr 9, 2019

wesleywiser commented Apr 9, 2019

michaelwoerister commented Apr 10, 2019

wesleywiser commented Apr 10, 2019

wesleywiser commented Apr 13, 2019

bors commented Apr 13, 2019

bors commented Apr 13, 2019

bors commented Apr 13, 2019

Use measureme in self profiler #59515

Use measureme in self profiler #59515

Conversation

wesleywiser commented Mar 29, 2019 • edited Loading

michaelwoerister commented Apr 2, 2019

michaelwoerister commented Apr 3, 2019

wesleywiser commented Apr 4, 2019

wesleywiser commented Apr 5, 2019

michaelwoerister commented Apr 5, 2019

michaelwoerister commented Apr 5, 2019

wesleywiser commented Apr 5, 2019

michaelwoerister commented Apr 5, 2019

michaelwoerister commented Apr 5, 2019

wesleywiser commented Apr 8, 2019

michaelwoerister commented Apr 9, 2019

wesleywiser commented Apr 9, 2019

michaelwoerister commented Apr 10, 2019

wesleywiser commented Apr 10, 2019

wesleywiser commented Apr 13, 2019

bors commented Apr 13, 2019

bors commented Apr 13, 2019

bors commented Apr 13, 2019

wesleywiser commented Mar 29, 2019 •

edited

Loading