Added inspect port overriding in cluster #13761

mutantcornholio · 2017-06-18T12:51:12Z

This is third part that I've mentioned in #9659 (comment)

Current cluster inspect port logic doen't let user to manually set inspect port for workers.
After this commit, adding any --inspect* options to cluster.settings.execArgv will disable port autoincrement.

Fixes #8495 and #12941.

I've got a problem with these cases:

If user starts master with --inspect-port but adds --inspect to cluster.settings.execArgv (set the base port, but don't run master with inspector), autoincrementing logic will be disabled. I can't say if this behavior is good or bad. Probably, port overriding should take place only if --inspect=* or --(inspect|debug)-port= are passed.
This approach doesn't fix the case if master starts on zero port and user wants workers to work on zero port too, instead of listening on ports incremented from some port: cluster.settings.execArgv will already contain everything in process.execArgv and adding --inspect=0 to cluster.settings doesn't change anything. (link: test: fix test-inspector-port-zero-cluster #13373 (comment))

So maybe instead of this, I should just add inspectPort option right to cluster.settings?

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
documentation is changed or added
commit message follows commit guidelines

Affected core subsystem(s)

mutantcornholio · 2017-06-19T15:11:12Z

cc/ @refack @Trott @bnoordhuis @cjihrig

refack

👍 Not even sure It's semver-major IMHO it's a bug fix.

refack · 2017-06-19T15:16:28Z

lib/internal/cluster/master.js

+    .filter((arg) => arg.match(debugArgRegex))
+    .sort();
+
+  let debugSettingsOverriden = false;


Suggestion:
const debugSettingsOverriden = clusterDebugArgs.any((v, i) => v != masterDebugArgs[i]);

Another option, make cluster.settings.execArgv a getter/setter property and do check-on-assignment

refack · 2017-06-19T15:28:25Z

If user starts master with --inspect-port but adds --inspect to cluster.settings.execArgv (set the base port, but don't run master with inspector), autoincrementing logic will be disabled. I can't say if this behavior is good or bad. Probably, port overriding should take place only if --inspect=* or --(inspect|debug)-port= are passed.

There is also #13228 and initiating debugger with SIGUSR1. IMHO if there is no simple solution they could be dealt with in a future PR.

This approach doesn't fix the case if master starts on zero port and user wants workers to work on zero port too, instead of listening on ports incremented from some port: cluster.settings.execArgv will already contain everything in process.execArgv and adding --inspect=0 to cluster.settings doesn't change anything. (link: #13373 (comment))

~~Are you sure? It works from the CLI:~~[edit] should be overridable with cluster.setupMaster({execArgv: ['--inspect', '--inspect-port=0']});

d:\code\node-cur$ node --inspect=9229 --inspect=0
Debugger listening on ws://127.0.0.1:52153/590e5e9c-6118-475a-94e5-78103994953c
For help see https://nodejs.org/en/docs/inspector

refack · 2017-06-19T15:46:54Z

IMHO since no tests were changes, and this feels more like a bug fix then behaviour change this could be semver-patch

P.S. @mutantcornholio https://github.com/nodejs/node/blob/master/doc/api/cluster.md#clustersetupmastersettings should be updated to reflect it does accept execArgv (and always did 4cd522d). Unless anyone knows of a reason why it was never documented? @AndreasMadsen ? (f9a47de)

mutantcornholio · 2017-06-19T15:56:08Z

I promise to update the docs before it gets merged (see unchecked box in checklist?)
But first, I want to cover the cases I've mentioned — maybe use another approach.

I like the idea of making cluster.settings.execArgv a getter/setter.
Can anyone see a downside there?

AndreasMadsen · 2017-06-19T19:16:03Z

Unless anyone knows of a reason why it was never documented? @AndreasMadsen ?

Sorry, I think it is 6 years ago since I wrote that code :p Node.js had poor support for debugging at that time, I think it mostly happened through the terminal gdb interface at that time. So debugging cluster workers was not something we had considered. We just ensured that it would be possible to execute the worker as standalone and debug it that way. Obviously, things have changed since then. I don't use the cluster module these days, so I don't have any opinion, just stories to tell :)

refack · 2017-06-19T19:21:13Z

so I don't have any opinion, just stories to tell :)

👍
BTW: I was using WebStorm 1 in 2010. GUI rulez.

Trott · 2017-06-19T20:15:26Z

/cc @mcollina @nodejs/v8-inspector

mutantcornholio · 2017-06-20T09:39:52Z

Wait, cluster.settings.execArgv is an array. And we're already using .push() on it.
Unless we use something like this (which, I hope, we won't), I don't see, how getter/setter would help :(

refack · 2017-06-20T12:46:53Z

Wait, cluster.settings.execArgv is an array. And we're already using .push() on it.
Unless we use something like this (which, I hope, we won't), I don't see, how getter/setter would help :(

hmmm...
Well at least setupMaster uses assignment: https://github.com/nodejs/node/pull/13761/files#diff-a8eff7604cf3ad3580c1949c54176127R57
And about cluster.settingsFortunatly we have this line in the docs:

This object is not intended to be changed or set manually.

So we can be sure noone is using it directly 😜 🤣
You could use a getter on cluster.settings to defensively mark it dirty-on-read
And BTW, Proxies are not that bad... Cumbersome but useful.

Next PR turn cluster.settings into a deep-copy getter, and rename setupMaster to setup.

mutantcornholio · 2017-06-20T13:29:18Z

So we can be sure noone is using it directly 😜 🤣

I just looked at my codebase and it shows that I've been using it directly for almost two years :D

Next PR turn cluster.settings into a deep-copy getter, and rename setupMaster to setup.
This object is not intended to be changed or set manually

Are you saying that we should enforce setting cluster.settings through cluster.setup?
Make cluster.settings a getter (which will return a hidden object) + noop/throwing setter, for instance.

Looks better than a warning in docs, for me.

refack · 2017-06-20T16:19:15Z

Are you saying that we should enforce setting cluster.settings through cluster.setup?
Make cluster.settings a getter (which will return a hidden object) + noop/throwing setter, for instance.

Yes enforcing, since there is quite a bit of logic in setupMaster. Returning an immutable copy on cluser.settings, should be good.
Also setupMaster should do a deep-copy, since util._extend(settings, cluster.settings); is shallow and I can do:

const smartAlec = ['myarg'];
cluster.setupMaster({execArgv: smartAlec});
smartAlec.push('--invalid');
smartAlec.shift();

mcollina · 2017-06-21T09:30:04Z

I'm a bit lost on the status of this. It seems the two points @mutantcornholio raised are still not addressed, right?

mutantcornholio · 2017-06-21T10:01:32Z

Well, first one is pretty straightforward and I'll just use a bigger regex.

For the second one, preventing cluster.settings from direct change and moving port override logic to setupMaster should do the trick. I guess, everything's settled and I will update the PR in couple of days.

P.S. Is it okay to do setupMaster enforcing in this PR? That looks like a bigger scope issue than overriding inspect port increment logic.

mcollina · 2017-06-21T10:13:46Z

P.S. Is it okay to do setupMaster enforcing in this PR? That looks like a bigger scope issue than overriding inspect port increment logic.

Can you explain/rephrase this?

mutantcornholio · 2017-06-21T10:22:02Z

https://github.com/nodejs/node/blob/master/doc/api/cluster.md#clustersettings says "This object is not intended to be changed or set manually."

I need to detect setting debug args to cluster.settings.execArgv for this.
Considering it's an array, it's not gonna be pretty.

So @refack suggested to forbid users to make changes to cluster.settings directly and enforce setting it through cluster.setupMaster.

That will make my task much easier, but should I do both tasks in one PR?

refack · 2017-06-21T10:22:07Z

P.S. Is it okay to do setupMaster enforcing in this PR? That looks like a bigger scope issue than overriding inspect port increment logic.

IMHO that should be in a separate PR, so this one can stay semver-patch since it fixes a known-limitation.

mutantcornholio · 2017-06-21T13:11:43Z

That way, I'll probably won't fix --inspect=0 case until next PR gets landed

refack · 2017-06-21T13:15:22Z

That way, I'll probably won't fix --inspect=0 case until next PR gets landed

Incremental progression is good...

refack · 2017-06-21T13:32:39Z

That way, I'll probably won't fix --inspect=0 case until next PR gets landed

cluster.setupMaster({execArgv: ['--inspect', '--inspect-port=0']}); should work after this PR, no?

mutantcornholio · 2017-06-21T16:55:55Z

cluster.setupMaster({execArgv: ['--inspect', '--inspect-port=0']}); should work after this PR, no?

If master starts with same debug args as worker, code of this PR won't detect it. But you can start master with ['--inspect=0'] and call setupMaster with execArgv: ['--inspect', '--inspect-port=0'] and that'll do.

Current cluster inspect port logic doen't let user to manually set inspect port for workers. After this commit, adding any --inspect* options to execArgv in cluster.setupMaster's options will disable port autoincrement

mutantcornholio · 2017-06-28T20:38:15Z

I've got a little different idea.

Check these parts of cluster docs:
cluster.settings ... This object is not intended to be changed or set manually.
cluster.setupMaster ... the defaults above apply to the first call only, the defaults for later calls is the current value at the time of cluster.setupMaster() is called

So, once I call setupMaster({execArgv: ['--inspect=12312']}), port incrementing logic should stay overridden, no matter with what I'll call setupMaster() later.

Considering this, just checking execArgv passed to setupMaster() until catching port setting option looks sufficient.
--inspect=0 case is covered too.

I've updated the code and tests (run CI please).
If everyone agrees with solution, I'll start documenting this.

cc/ @refack @Trott @mcollina

refack · 2017-06-28T21:19:55Z

lib/internal/cluster/master.js

@@ -41,15 +42,15 @@ if (schedulingPolicy === undefined) {

 cluster.schedulingPolicy = schedulingPolicy;

-cluster.setupMaster = function(options) {
+cluster.setupMaster = function(options = {}) {


This is a silly one, but this might be a breaking change, but IMHO we should keep it unless someone objects.
This changes the .length of the method:

> require('cluster').setupMaster.length 1 > (function a(a1) {}).length 1 > (function b(b1={}) {}).length 0

This is definitely a breaking change, albeit a subtle one. Setting an argument default necessarily makes this semver-major

@jasnell Can you explain why? It’s not obvious to me how this would break code that isn’t designed specifically to be broken by this change.

Not sure why its backwards incompatible, but it is unnecessary, as is util._extend(settings, options || {}). _extend() ignores non-object second args. Default here should be removed, and the || {} that is removed below can stay removed.

refack · 2017-06-28T21:28:18Z

lib/internal/cluster/master.js

@@ -26,6 +26,7 @@ cluster.SCHED_RR = SCHED_RR;      // Master distributes connections.
 var ids = 0;
 var debugPortOffset = 1;
 var initialized = false;
+let debugSettingsOverriden = false;


a global, augh...
💡
🤓
create a Symbol and hang it on cluster.settings

const kDebugSettingsOverriden = Symbol('Debug Settings Overriden'); ... var settings = { args: process.argv.slice(2), exec: process.argv[1], execArgv: process.execArgv, silent: false, [kDebugSettingsOverriden]: false };

What the heck, hang them all (maybe in the different PR though)

refack · 2017-06-28T21:29:59Z

lib/internal/cluster/master.js

@@ -60,6 +61,14 @@ cluster.setupMaster = function(options) {
    settings.execArgv = settings.execArgv.concat(['--logfile=v8-%p.log']);
  }

+  // This allows user to override inspect port for workers.
+  const debugPortArgsRegex = /--inspect(=|-port=)|--debug-port=/;


const debugOverrideInSettings = options.execArgv && options.execArgv.some((arg) => arg.match(debugPortArgsRegex)) settings.execArgv[kDebugSettingsInArgv] = settings.execArgv.some((arg) => arg.match(/--inspect(?:-brk|-port)?|--debug-port/)))

Or some variation of... Then reuse

refack · 2017-06-28T21:36:47Z

I like it. It's simple!
If I get it right now the known limitation is:

cluster.setupMaster({execArgv: ['--inspect=12312']});
fork() // goes to 12312
cluster.setupMaster({execArgv: [--inspect]});
fork() // goes to 9229
fork() // boom

refack · 2017-06-28T21:42:37Z

💥
🤓
Same code but expose [kDebugSettingsOverriden]: false as a new public property of settings you can undo, or set to a number, or a lambda, or a web-service, or....

cluster.setupMaster({execArgv: ['--inspect=12312']});
fork() // goes to 12312
cluster.setupMaster({execArgv: [--inspect], overideDebugPort: false});
fork() // goes to 9229
fork() // goes to 9230
let port = 6666;
cluster.setupMaster({execArgv: [--inspect], overideDebugPort: () => port++});
fork() // goes to 6666
fork() // goes to 6667

mutantcornholio · 2017-06-29T06:24:50Z

Same code but expose [kDebugSettingsOverriden]: false as a new public property of settings you can undo, or set to a number, or a lambda, or a web-service, or....

I thought about that:

So maybe instead of this, I should just add inspectPort option right to cluster.settings?

Maybe we should do that instead. Boolean option to disable autoincrement logic?

jasnell · 2017-06-30T15:15:13Z

Quick glance through the code looks good but didn't go through enough to sign off. I will say that it would be helpful to include some details about this in the docs.

sam-github · 2017-06-30T15:34:36Z

lib/internal/cluster/master.js

  var settings = {
    args: process.argv.slice(2),
    exec: process.argv[1],
    execArgv: process.execArgv,
    silent: false
  };
  util._extend(settings, cluster.settings);
-  util._extend(settings, options || {});
+  util._extend(settings, options);


+1, the || {} was unnecessary, _extend() already does the right thing.

refack · 2017-06-30T22:09:05Z

Maybe we should do that instead. Boolean option to disable autoincrement logic?

This is ~~chinatown~~ JavaScript so I say an overloaded property inspectPort (maybe overrideInspectPort):

Boolean - enable / disable auto increment — negated so if (!settings.inspectPort) do the current logic
Number - explicit port to use
Function - function to run that return a number that will be the port
~~String - order a t-shirt from cafe-press with the string printed on it~~

mutantcornholio · 2017-07-01T20:06:57Z

An option called inspectPort which could accept boolean doesn't sound right to me. inspectPort: true shows me that "Some inspector port is set", not "Port autoincrement logic for workers is disabled"

I'd better split it into boolean inspectPortAutoincrement and number/function inspectPort.

And setting inspectPort would override inspectPortAutoincrement setting (which would default to true, IMHO).

What do you think?

Trott · 2017-07-01T20:14:55Z

I'd better split it into boolean inspectPortAutoincrement and number/function inspectPort.

And setting inspectPort would override inspectPortAutoincrement setting (which would default to true, IMHO).

What do you think?

If you do go with an option, maybe drop the Auto and call it inspectPortIncrement or even portIncrement? I think increment is clearer than autoincrement.

mutantcornholio · 2017-07-01T20:20:23Z

Dropping Auto sounds good, but I don't want to drop inspect — some people (like me) might think of port that workers accept connections on.

So, I'm making two explicit options and removing implicit override in execArgv?

refack · 2017-07-02T00:14:48Z

So, I'm making two explicit options and removing implicit override in execArgv?

I'm just summarizing current thoughts:

settings.execArgv.push('--inspect-port=XXX') turns off settings.inspectPortIncrement (previously known as `debugSettingsOverriden')
resetting settings.execArgv does not automatically reset settings.inspectPortIncrement
new option settings.inspectPort can be number or lambda () => n

Plus:

remove default ={} from cluster.setupMaster = function(options = {}) { since util._extend(settings, options) does the right thing anyway, and return to semver-minor again.

Things to think about:

inspectPortIncrement implies opt-in to previous behaviour, and IMHO we need opt-out (so that if !'inspectPortIncrement' in settings || Boolean(settings.inspectPortIncrement) == false we do what we always did)
what do we do if settings.inspectPortIncrement === false & settings.inspectPort == null?

My alternative — inspectPortOveride:

if undefined act like before PR
if true don't do anything
if number / () => n use value for adding --inspect-port=n
if cluster.setupMaster({ execArgv: ['--inspect-port=XXX')}) then implicitly we set settings.inspectPortOveride = true
(4) can be reverted with cluster.setupMaster({ execArgv: [], inspectPortOveride: false}) and we go back the pre PR behaviour

oooff a lot to think about...

mutantcornholio · 2017-07-02T00:39:04Z

settings.execArgv.push('--inspect-port=XXX') turns off settings.inspectPortIncrement (previously known as `debugSettingsOverriden')

I don't see, why we should do both settings.inspectPortIncrement and controlling it by looking in settings.execArgv. If we add an option to disable increment manually, why should we have implicit complex logic? I think using one of these methods is sufficient.
I mean, right now we have zero methods, why add two at once?

inspectPortIncrement implies opt-in to previous behaviour, and IMHO we need opt-out

Well, maybe. But we need a better name for the option, this way. Any ideas?

My alternative — inspectPortOveride:

Once again, I don't think that I would understand what {inspectPortOverride: true} could mean, without getting in the docs. I would thought "Override it with what?", not "Oh, that's disables incrementing behaviour, but the port stays the same"

refack · 2017-07-02T01:23:35Z

I don't see, why we should do both settings.inspectPortIncrement and controlling it by looking in settings.execArgv. If we add an option to disable increment manually, why should we have implicit complex logic? I think using one of these methods is sufficient.
I mean, right now we have zero methods, why add two at once?

I'm convinced.

inspectPortIncrement implies opt-in to previous behaviour, and IMHO we need opt-out

Well, maybe. But we need a better name for the option, this way. Any ideas?

disableInspectorPortIncrement?

So:

disableInspectorPortIncrement (or any other better name) — mean we do not do any argv manipulation, what comes in is what goes to the process.
inspectPort — mean we add --inspector-port=N

So now I'm not sure why we need (1) as I see it (2) covers all use-cases. If inspectPort is set, we use it, if it's not, we act like before the PR? seems good enough to me... No argv mutations, just adding last arg.

mutantcornholio · 2017-07-02T09:33:03Z

So now I'm not sure why we need (1) as I see it (2) covers all use-cases. If inspectPort is set, we use it, if it's not, we act like before the PR? seems good enough to me...

Now I am convinced.

I'll do that in a couple of days.

mutantcornholio · 2017-07-08T22:56:43Z

Closed in favor of #14140

* Capitalization and punctuation. * `setupMaster` contained info about `settings` which where incomplete. PR-URL: nodejs#14140 Fixes: nodejs#8495 Fixes: nodejs#12941 Refs: nodejs#9659 Refs: nodejs#13761 Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

10 ports for each test-case is too much. Not enough ports for new test cases, considering ~100 ports per file. PR-URL: nodejs#14140 Fixes: nodejs#8495 Fixes: nodejs#12941 Refs: nodejs#9659 Refs: nodejs#13761 Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

Added an option to override inspector port for workers using `settings.inspectPort` will override default port incrementing behavior. Also, using this option allows to set 0 port for the whole cluster. PR-URL: nodejs#14140 Fixes: nodejs#8495 Fixes: nodejs#12941 Refs: nodejs#9659 Refs: nodejs#13761 Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

* Capitalization and punctuation. * `setupMaster` contained info about `settings` which where incomplete. PR-URL: #14140 Fixes: #8495 Fixes: #12941 Refs: #9659 Refs: #13761 Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

10 ports for each test-case is too much. Not enough ports for new test cases, considering ~100 ports per file. PR-URL: #14140 Fixes: #8495 Fixes: #12941 Refs: #9659 Refs: #13761 Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

Added an option to override inspector port for workers using `settings.inspectPort` will override default port incrementing behavior. Also, using this option allows to set 0 port for the whole cluster. PR-URL: #14140 Fixes: #8495 Fixes: #12941 Refs: #9659 Refs: #13761 Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

* Capitalization and punctuation. * `setupMaster` contained info about `settings` which where incomplete. PR-URL: #14140 Fixes: #8495 Fixes: #12941 Refs: #9659 Refs: #13761 Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

10 ports for each test-case is too much. Not enough ports for new test cases, considering ~100 ports per file. PR-URL: #14140 Fixes: #8495 Fixes: #12941 Refs: #9659 Refs: #13761 Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

Added an option to override inspector port for workers using `settings.inspectPort` will override default port incrementing behavior. Also, using this option allows to set 0 port for the whole cluster. PR-URL: #14140 Fixes: #8495 Fixes: #12941 Refs: #9659 Refs: #13761 Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

mscdex added cluster Issues and PRs related to the cluster subsystem. inspector Issues and PRs related to the V8 inspector protocol labels Jun 18, 2017

refack approved these changes Jun 19, 2017

View reviewed changes

refack self-assigned this Jun 19, 2017

mutantcornholio force-pushed the cluster-inspect-override branch 2 times, most recently from 57ea09c to 7ce2925 Compare June 28, 2017 20:30

Added inspect port overriding in cluster

fb9d660

Current cluster inspect port logic doen't let user to manually set inspect port for workers. After this commit, adding any --inspect* options to execArgv in cluster.setupMaster's options will disable port autoincrement

mutantcornholio force-pushed the cluster-inspect-override branch from 7ce2925 to fb9d660 Compare June 28, 2017 20:34

refack approved these changes Jun 28, 2017

View reviewed changes

Trott requested a review from bnoordhuis June 29, 2017 02:42

jasnell added the semver-major PRs that contain breaking changes and should be released in the next major version. label Jun 30, 2017

sam-github reviewed Jun 30, 2017

View reviewed changes

mutantcornholio mentioned this pull request Jul 8, 2017

cluster: overriding inspector port #14140

Merged

4 tasks

mutantcornholio closed this Jul 8, 2017

refack removed their assignment Oct 20, 2018

Added inspect port overriding in cluster #13761

Added inspect port overriding in cluster #13761

Conversation

mutantcornholio commented Jun 18, 2017

Checklist

Affected core subsystem(s)

mutantcornholio commented Jun 19, 2017

refack left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

refack commented Jun 19, 2017 • edited Loading

refack commented Jun 19, 2017

mutantcornholio commented Jun 19, 2017

AndreasMadsen commented Jun 19, 2017

refack commented Jun 19, 2017

Trott commented Jun 19, 2017

mutantcornholio commented Jun 20, 2017

refack commented Jun 20, 2017

mutantcornholio commented Jun 20, 2017

refack commented Jun 20, 2017

mcollina commented Jun 21, 2017

mutantcornholio commented Jun 21, 2017

mcollina commented Jun 21, 2017

mutantcornholio commented Jun 21, 2017

refack commented Jun 21, 2017

mutantcornholio commented Jun 21, 2017

refack commented Jun 21, 2017

refack commented Jun 21, 2017

mutantcornholio commented Jun 21, 2017

mutantcornholio commented Jun 28, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

refack commented Jun 28, 2017

refack commented Jun 28, 2017

mutantcornholio commented Jun 29, 2017

jasnell commented Jun 30, 2017

Choose a reason for hiding this comment

refack commented Jun 30, 2017

mutantcornholio commented Jul 1, 2017

Trott commented Jul 1, 2017

mutantcornholio commented Jul 1, 2017

refack commented Jul 2, 2017 • edited Loading

mutantcornholio commented Jul 2, 2017

refack commented Jul 2, 2017

mutantcornholio commented Jul 2, 2017

mutantcornholio commented Jul 8, 2017

refack commented Jun 19, 2017 •

edited

Loading

refack commented Jul 2, 2017 •

edited

Loading