src: name EmbededderGraph edges and use class names for nodes #23072

joyeecheung · 2018-09-25T03:23:47Z

This patch:

Refactors the MemoryRetainer API so that the impementer no longer
calls TrackThis() that sets the size of node on the top of the stack,
which may be hard to understand. Instead they implements SelfSize()
to provide their self sizes. Also documents the API in the header.
Refactors MemoryTracker so it calls MemoryInfoName() and SelfSize()
of MemoryRetainer to retrieve info about them, and separate
node_names and edge_names so the edges can be properly named with
reference names and the nodes can be named with class names.
(Previously the nodes are named with reference names while the edges
are all indexed and appear as array elements).
Adds SET_MEMORY_INFO_NAME(), SET_SELF_SIZE() and
SET_NO_MEMORY_INFO() convenience macros
Fixes a few MemoryInfo calls in some MemoryRetainers to track
their references properly.
Refactors the heapdump tests to check both node names and edge names,
distinguishing between wrapped JS nodes (without prefixes)
and embedder wrappers (prefixed with Node / ).

Comparison in Chrome DevTools

`Node / Http2Session`

Before:

After:

`Node / FileHandleReadWrap`

Before:

After:

`Node / JSBindingConnection`

Before:

After:

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
documentation is changed or added
commit message follows commit guidelines

nodejs-github-bot · 2018-09-25T03:23:49Z

@joyeecheung build started: https://ci.nodejs.org/blue/organizations/jenkins/node-test-pull-request-lite-pipeline/detail/node-test-pull-request-lite-pipeline/1001/pipeline

joyeecheung · 2018-09-25T03:24:21Z

~~CI: https://ci.nodejs.org/job/node-test-pull-request/17416/~~
CI: https://ci.nodejs.org/job/node-test-pull-request/17417/

joyeecheung · 2018-09-25T03:25:09Z

cc @addaleax

devsnek · 2018-09-25T03:27:29Z

src/async_wrap.cc

-    tracker->TrackThis(this);
-  }
+  SET_NO_MEMORY_INFO()
+  SET_MEMORY_INFO_NAME(AsyncWrapObject);


extra semicolon?

devsnek · 2018-09-25T03:31:13Z

src/async_wrap.cc

-  void MemoryInfo(MemoryTracker* tracker) const override {
-    tracker->TrackThis(this);
-  }
+  SET_NO_MEMORY_INFO()


seeing as these three macros are used together quite a lot, maybe there's a better way they could be composed? seems like this introduces some opportunities for forgetfulness and ambiguity.

@devsnek These are pure virtual functions, if the implementer does not implement them a compilation error would be thrown, so I am not sure how this can introduce forgetfulness and ambiguity (if anything, before this patch not all of them were pure virtual functions so that's where forgetfulness and ambiguity used to be possible). The macros are just helpers for the most common cases - some classes, like TCPWrap, would implement their own methods for better debuggability.

addaleax

Looks very good so far!

addaleax · 2018-09-25T09:50:12Z

src/heap_utils.cc

-  if (graph.CreateObject().ToLocal(&ret))
-    args.GetReturnValue().Set(ret);
+  // Crash if we cannot build a proper graph
+  args.GetReturnValue().Set(graph.CreateObject().ToLocalChecked());


What’s the reason for this change? If the return value of CreateObject() is empty, that means we already have a scheduled exception, so returning to JS would be okay, right?

I ran into this in tests and it somehow returned an undefined so I got an undefined embeddedGraph that can’t be called with filter...hmm now that I come to think of it maybe there was a bug in CreateObject that returned an empty Maybe even when there was no pending exception, the bug is likely gone now

addaleax · 2018-09-25T09:52:54Z

src/memory_tracker-inl.h

@@ -5,25 +5,33 @@

 #include "memory_tracker.h"

+#define DEFAULT_NODE_NAME                                                      \
+  (node_name == nullptr ? (edge_name == nullptr ? "" : edge_name) : node_name)


Can we avoid this being a macro? I think I’d prefer a function that takes edge_name and node_name arguments, even if that is a bit more to spell out…

src/memory_tracker-inl.h

addaleax · 2018-09-25T10:07:18Z

src/memory_tracker-inl.h

+    // Shift the self size of this container out to a separate node
+    CurrentNode()->size_ -= sizeof(T);
+  }
+  PushNode(DEFAULT_NODE_NAME, sizeof(T), edge_name);


These lines here seem to make the assumption that the outer Node always already accounts for the memory of the outer container… I don’t think that’s guaranteed, though?

Maybe a comment in the API would be enough? Or a boolean field in case the implementer doesn’t use sizeof as the SelfSize, then they will need to figure out the calculation (so far I haven’t seen any implementer with a container member that doesn’t do this). I believe before this patch all of the container member sizes got double-counted anyway so this just makes the common case a bit more accurate.

Or, maybe we can lift the size 0 constraint and make the container node a size 0 node then it won’t get double counted..(though I am not sure how it looks in the DevTools..I am 30% sure some internal Nodes in v8 already do this)

EDIT: hmm, nope, I looked into the generator code a bit and I don't think V8 does that

Oh, does that mean that V8 just doesn’t support nodes of size 0…? I guess in that case this is the best possible way to do this 👍

addaleax · 2018-09-25T10:09:05Z

src/memory_tracker-inl.h

+  auto it = seen_.find(retainer);
+  if (it != seen_.end()) {
+    n = it->second;
+    return n;


Maybe return it->second;, and move the declaration of n to where it’s initialized below?

addaleax · 2018-09-25T10:19:20Z

src/memory_tracker.h

+ *       // elements in the container appear as grandchildren nodes
+ *       tracker->TrackField("vector", this->vector_field);
+ *       // Node name and size come from the JS object
+ *       tracker->TrackField("target", this->target);


Is there a reason for going with this-> in the examples / no trailing underscores? It’s not exactly our existing code style … ;)

Ah I should’ve polished this a bit more...it’s more like a copy-paste from a draft, I will tweak the style

This patch: - Refactors the `MemoryRetainer` API so that the impementer no longer calls `TrackThis()` that sets the size of node on the top of the stack, which may be hard to understand. Instead they implements `SelfSize()` to provide their self sizes. Also documents the API in the header. - Refactors `MemoryTracker` so it calls `MemoryInfoName()` and `SelfSize()` of `MemoryRetainer` to retrieve info about them, and separate `node_names` and `edge_names` so the edges can be properly named with reference names and the nodes can be named with class names. (Previously the nodes are named with reference names while the edges are all indexed and appear as array elements). - Adds `SET_MEMORY_INFO_NAME()`, `SET_SELF_SIZE()` and `SET_NO_MEMORY_INFO()` convenience macros - Fixes a few `MemoryInfo` calls in some `MemoryRetainers` to track their references properly. - Refactors the heapdump tests to check both node names and edge names, distinguishing between wrapped JS nodes (without prefixes) and embedder wrappers (prefixed with `Node / `).

joyeecheung · 2018-10-02T09:54:14Z

Rebased and addressed comments. PTAL @addaleax @devsnek

https://ci.nodejs.org/job/node-test-pull-request/17583/

joyeecheung · 2018-10-02T12:33:45Z

CI is happy and so am I

addaleax · 2018-10-02T14:06:46Z

src/memory_tracker.h

+ *     std::vector<uv_async_t> vector;
+ *     node::Persistent<Object> target;
+ *
+ *     node::Persistent<Object> wrapped;


tiny nit: We use trailing _ for non-static class members :)

joyeecheung · 2018-10-03T10:16:40Z

Fixed nits. CI: https://ci.nodejs.org/job/node-test-pull-request/17600/

(Not sure who to cc, we don't have a team for V8 integrations, do we? Tentatively cc @nodejs/v8 ) I plan to land this in 24 hours unless there are more reviews to address.

hashseed · 2018-10-03T10:25:27Z

@ulan

ulan · 2018-10-04T09:51:35Z

src/memory_tracker-inl.h

+  }
+  return "";
+}
+
 class MemoryRetainerNode : public v8::EmbedderGraph::Node {
 public:
  explicit inline MemoryRetainerNode(MemoryTracker* tracker,


Nit: "explicit" is not needed here and below.

ulan · 2018-10-04T09:57:51Z

V8 related parts lgtm

joyeecheung · 2018-10-04T10:17:29Z

CI: https://ci.nodejs.org/job/node-test-pull-request/17624/

joyeecheung · 2018-10-04T13:37:32Z

Landed in 92fa0fc

This patch: - Refactors the `MemoryRetainer` API so that the impementer no longer calls `TrackThis()` that sets the size of node on the top of the stack, which may be hard to understand. Instead now they implements `SelfSize()` to provide their self sizes. Also documents the API in the header. - Refactors `MemoryTracker` so it calls `MemoryInfoName()` and `SelfSize()` of `MemoryRetainer` to retrieve info about them, and separate `node_names` and `edge_names` so the edges can be properly named with reference names and the nodes can be named with class names. (Previously the nodes are named with reference names while the edges are all indexed and appear as array elements). - Adds `SET_MEMORY_INFO_NAME()`, `SET_SELF_SIZE()` and `SET_NO_MEMORY_INFO()` convenience macros - Fixes a few `MemoryInfo` calls in some `MemoryRetainers` to track their references properly. - Refactors the heapdump tests to check both node names and edge names, distinguishing between wrapped JS nodes (without prefixes) and embedder wrappers (prefixed with `Node / `). PR-URL: #23072 Reviewed-By: Anna Henningsen <anna@addaleax.net>

joyeecheung · 2018-10-04T17:01:44Z

It should be backportable to v10.x

joyeecheung · 2018-10-04T17:37:32Z

Nope, it doesn't, because it relies on #22106 which breaks the ABI

addaleax · 2018-10-04T17:40:58Z

@joyeecheung Just wondering, because this touches so much code all over the place … would it be worth looking into doing an ABI-compatible backport of #22106?

joyeecheung · 2018-10-04T18:07:07Z

@addaleax Maybe I could...by modify the calls to actual AddEdge. I will try it later.

TimothyGu · 2018-10-04T20:20:36Z

@joyeecheung This PR added an empty src/txt file. Is that intended?

joyeecheung · 2018-10-04T20:30:05Z

@TimothyGu that was accidental, sorry. I am away from my laptop now, feel free open a PR to remove it or I will do it later

PR-URL: #23273 Refs: #23072 Reviewed-By: Daniel Bevenius <daniel.bevenius@gmail.com> Reviewed-By: Michaël Zasso <targos@protonmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Richard Lau <riclau@uk.ibm.com> Reviewed-By: Gus Caplan <me@gus.host>

This patch: - Refactors the `MemoryRetainer` API so that the impementer no longer calls `TrackThis()` that sets the size of node on the top of the stack, which may be hard to understand. Instead now they implements `SelfSize()` to provide their self sizes. Also documents the API in the header. - Refactors `MemoryTracker` so it calls `MemoryInfoName()` and `SelfSize()` of `MemoryRetainer` to retrieve info about them, and separate `node_names` and `edge_names` so the edges can be properly named with reference names and the nodes can be named with class names. (Previously the nodes are named with reference names while the edges are all indexed and appear as array elements). - Adds `SET_MEMORY_INFO_NAME()`, `SET_SELF_SIZE()` and `SET_NO_MEMORY_INFO()` convenience macros - Fixes a few `MemoryInfo` calls in some `MemoryRetainers` to track their references properly. - Refactors the heapdump tests to check both node names and edge names, distinguishing between wrapped JS nodes (without prefixes) and embedder wrappers (prefixed with `Node / `). PR-URL: nodejs#23072 Reviewed-By: Anna Henningsen <anna@addaleax.net>

This patch: - Refactors the `MemoryRetainer` API so that the impementer no longer calls `TrackThis()` that sets the size of node on the top of the stack, which may be hard to understand. Instead now they implements `SelfSize()` to provide their self sizes. Also documents the API in the header. - Refactors `MemoryTracker` so it calls `MemoryInfoName()` and `SelfSize()` of `MemoryRetainer` to retrieve info about them, and separate `node_names` and `edge_names` so the edges can be properly named with reference names and the nodes can be named with class names. (Previously the nodes are named with reference names while the edges are all indexed and appear as array elements). - Adds `SET_MEMORY_INFO_NAME()`, `SET_SELF_SIZE()` and `SET_NO_MEMORY_INFO()` convenience macros - Fixes a few `MemoryInfo` calls in some `MemoryRetainers` to track their references properly. - Refactors the heapdump tests to check both node names and edge names, distinguishing between wrapped JS nodes (without prefixes) and embedder wrappers (prefixed with `Node / `). Backport-PR-URL: #23295 PR-URL: #23072 Reviewed-By: Anna Henningsen <anna@addaleax.net>

This patch: - Refactors the `MemoryRetainer` API so that the impementer no longer calls `TrackThis()` that sets the size of node on the top of the stack, which may be hard to understand. Instead now they implements `SelfSize()` to provide their self sizes. Also documents the API in the header. - Refactors `MemoryTracker` so it calls `MemoryInfoName()` and `SelfSize()` of `MemoryRetainer` to retrieve info about them, and separate `node_names` and `edge_names` so the edges can be properly named with reference names and the nodes can be named with class names. (Previously the nodes are named with reference names while the edges are all indexed and appear as array elements). - Adds `SET_MEMORY_INFO_NAME()`, `SET_SELF_SIZE()` and `SET_NO_MEMORY_INFO()` convenience macros - Fixes a few `MemoryInfo` calls in some `MemoryRetainers` to track their references properly. - Refactors the heapdump tests to check both node names and edge names, distinguishing between wrapped JS nodes (without prefixes) and embedder wrappers (prefixed with `Node / `). PR-URL: #23072 Reviewed-By: Anna Henningsen <anna@addaleax.net>

PR-URL: #23273 Refs: #23072 Reviewed-By: Daniel Bevenius <daniel.bevenius@gmail.com> Reviewed-By: Michaël Zasso <targos@protonmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Richard Lau <riclau@uk.ibm.com> Reviewed-By: Gus Caplan <me@gus.host>

nodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. lib / src Issues and PRs related to general changes in the lib or src directory. labels Sep 25, 2018

joyeecheung added the dont-land-on-v8.x label Sep 25, 2018

devsnek reviewed Sep 25, 2018

View reviewed changes

joyeecheung force-pushed the reverse-track branch from cef975b to 80f852a Compare September 25, 2018 03:34

addaleax reviewed Sep 25, 2018

View reviewed changes

joyeecheung mentioned this pull request Sep 25, 2018

CI failures: 20180925 nodejs/reliability#17

Closed

joyeecheung added 2 commits October 2, 2018 11:21

fixup! src: name EmbededderGraph edges and use class names for nodes

cd07235

joyeecheung force-pushed the reverse-track branch from 80f852a to cd07235 Compare October 2, 2018 09:52

addaleax approved these changes Oct 2, 2018

View reviewed changes

fixup! src: name EmbededderGraph edges and use class names for nodes

774ae11

ulan reviewed Oct 4, 2018

View reviewed changes

fixup! src: name EmbededderGraph edges and use class names for nodes

e56d4c0

joyeecheung closed this Oct 4, 2018

joyeecheung added the backport-requested-v10.x label Oct 4, 2018

joyeecheung added the dont-land-on-v10.x label Oct 4, 2018

joyeecheung removed the backport-requested-v10.x label Oct 4, 2018

joyeecheung mentioned this pull request Oct 5, 2018

src: remove accidentally added src/txt #23273

Closed

2 tasks

joyeecheung mentioned this pull request Oct 5, 2018

Post-mortem out of memory analysis nodejs/diagnostics#239

Closed

joyeecheung mentioned this pull request Oct 6, 2018

[v10.x] backport EmbededderGraph related changes to v10.x #23295

Closed

targos added backported-to-v10.x and removed dont-land-on-v10.x labels Oct 7, 2018

targos mentioned this pull request Oct 7, 2018

Release proposal: v10.12.0 #23313

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

src: name EmbededderGraph edges and use class names for nodes #23072

src: name EmbededderGraph edges and use class names for nodes #23072

joyeecheung commented Sep 25, 2018

nodejs-github-bot commented Sep 25, 2018

joyeecheung commented Sep 25, 2018 •

edited

Loading

joyeecheung commented Sep 25, 2018

devsnek Sep 25, 2018

devsnek Sep 25, 2018

joyeecheung Sep 25, 2018 •

edited

Loading

addaleax left a comment

addaleax Sep 25, 2018

joyeecheung Sep 25, 2018 •

edited

Loading

addaleax Sep 25, 2018

addaleax Sep 25, 2018

joyeecheung Sep 25, 2018

joyeecheung Sep 25, 2018 •

edited

Loading

addaleax Sep 25, 2018

addaleax Sep 25, 2018

addaleax Sep 25, 2018

joyeecheung Sep 25, 2018 •

edited

Loading

joyeecheung commented Oct 2, 2018

joyeecheung commented Oct 2, 2018

addaleax Oct 2, 2018

joyeecheung commented Oct 3, 2018

hashseed commented Oct 3, 2018

ulan Oct 4, 2018

ulan commented Oct 4, 2018

joyeecheung commented Oct 4, 2018

joyeecheung commented Oct 4, 2018

joyeecheung commented Oct 4, 2018

joyeecheung commented Oct 4, 2018

addaleax commented Oct 4, 2018

joyeecheung commented Oct 4, 2018

TimothyGu commented Oct 4, 2018

joyeecheung commented Oct 4, 2018

src: name EmbededderGraph edges and use class names for nodes #23072

src: name EmbededderGraph edges and use class names for nodes #23072

Conversation

joyeecheung commented Sep 25, 2018

Comparison in Chrome DevTools

Node / Http2Session

Node / FileHandleReadWrap

Node / JSBindingConnection

Checklist

nodejs-github-bot commented Sep 25, 2018

joyeecheung commented Sep 25, 2018 • edited Loading

joyeecheung commented Sep 25, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joyeecheung Sep 25, 2018 • edited Loading

Choose a reason for hiding this comment

addaleax left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joyeecheung Sep 25, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joyeecheung Sep 25, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joyeecheung Sep 25, 2018 • edited Loading

Choose a reason for hiding this comment

joyeecheung commented Oct 2, 2018

joyeecheung commented Oct 2, 2018

Choose a reason for hiding this comment

joyeecheung commented Oct 3, 2018

hashseed commented Oct 3, 2018

Choose a reason for hiding this comment

ulan commented Oct 4, 2018

joyeecheung commented Oct 4, 2018

joyeecheung commented Oct 4, 2018

joyeecheung commented Oct 4, 2018

joyeecheung commented Oct 4, 2018

addaleax commented Oct 4, 2018

joyeecheung commented Oct 4, 2018

TimothyGu commented Oct 4, 2018

joyeecheung commented Oct 4, 2018

`Node / Http2Session`

`Node / FileHandleReadWrap`

`Node / JSBindingConnection`

joyeecheung commented Sep 25, 2018 •

edited

Loading

joyeecheung Sep 25, 2018 •

edited

Loading

joyeecheung Sep 25, 2018 •

edited

Loading

joyeecheung Sep 25, 2018 •

edited

Loading

joyeecheung Sep 25, 2018 •

edited

Loading