Enhance runtime stats tracking #4144

dipinhora · 2022-06-17T13:31:21Z

This PR enhances the runtime stats tracking that was previously
implemented under the USE_MEMTRACK and USED_MEMTRACK_MESSAGES
defines. The new defines are called USE_RUNTIMESTATS and
USE_RUNTIMESTATS_MESSAGES.

Runtime stats tracking tracks the following actor info:

heap memory allocated
heap memory used
heap num allocated
heap realloc counter
heap alloc counter
heap free counter
heap gc counter
system message processing cpu usage
app message processing cpu usage
garbage collection cpu usage
messages sent counter
system messages processed counter
app messages processed counter

Runtime tracking tracks the following scheduler info:

mutemap memory used
mutemap memory allocated
memory used for gc acquire/release actormaps and actors created
memory allocated for gc acquire/release actormaps and actors created
created actors counter
destroyed actors counter
actor system message processing cpu for all actor runs on the scheduler
actor app message processing cpu for all actor runs on the scheduler
actor garbage collection cpu for all actor runs on the scheduler
scheduler message processing cpu usage
scheduler misc cpu usage while waiting to do work
memory used by inflight messages
memory allocated by inflight messages
number of inflight messages

This runtime stats tracking info has been exposed to pony programs as
part of the runtime_info package and an example runtime_info program
has been added to the examples directory.

The runtime stats tracking in a pony program can be used for some
useful validations for those folks concerned about
heap allocations in the critical path (i.e. if they
rely on the compiler's HeapToStack optimization pass
to convert heap allocations to stack allocations and
want to validate it is working correctly).

Example of possible use to validate number of heap allocations:

use "collections"
use "runtime_info"

actor Main
  new create(env: Env) =>
    let num_allocs_before = ActorStats.heap_alloc_counter(ActorStatsAuth(env.root))

    let ret = critical()

    let num_allocs_after = ActorStats.heap_alloc_counter(ActorStatsAuth(env.root))

    env.out.print("Allocations before: " + num_allocs_before.string())
    env.out.print("Allocations after: " + num_allocs_after.string())

    env.out.print("Critical section allocated " + (num_allocs_after - num_allocs_before).string() + " heap objects")

    env.out.print("Allocations at end: " + ActorStats.heap_alloc_counter(ActorStatsAuth(env.root)).string())

  fun critical(): U32 =>
    var x: U32 = 1
    let y: U32 = 1000

    for i in Range[U32](1, y) do
      x = x * y
    end

    x

SeanTAllen · 2022-06-17T23:13:55Z

I'd like to see the new ponyint_ functions exposed via the runtime_info package.

And changed from being ponyint_ to being PONY_API functions.

In the runtime_info, there should be an auth for memory tracking that is required.

dipinhora · 2022-06-18T00:41:00Z

When you say new ponyint_ functions are you referring to only the new ones in this PR or to all of the MEMTRACK related functions that were added when the original PR was merged?

Also, do you like the memtrack name? i'm starting to think that maybe memstats might be better.. what do you think?

SeanTAllen · 2022-06-18T00:46:22Z

I was referring to the ones you added but really, any that we expose via the runtime info primitive.

I think memstats is better. For the primitive, perhaps AllocatorInfo ?

dipinhora · 2022-06-18T00:46:40Z

Also, if we're going to expose this stuff via PONY_API and runtime_info, should we change the compile settings so that the support for this functionality is compiled/enabled by default (probably not the message level stats tracking due to too much overhead) with the ability for folks to turn it off (via make/cmake) if they don't want it or if they're concerned about performance?

SeanTAllen · 2022-06-18T00:58:25Z

I think off by default is the right approach with documentation on the primitive pointing to build instructions to turn on.

dipinhora · 2022-06-18T01:36:23Z

okey dokey.. makes sense..

SeanTAllen · 2022-06-18T11:29:41Z

this needs a rebase

dipinhora · 2022-06-18T15:54:16Z

rebased onto latest main

SeanTAllen · 2022-06-21T18:01:04Z

needs another rebase

SeanTAllen · 2022-06-21T18:28:27Z

This has outstanding changes requested.

During sync, Joe gave this a thumbs up.

SeanTAllen · 2022-06-24T13:54:58Z

Also needs a rebase against main

SeanTAllen · 2022-06-26T00:13:09Z

@dipinhora drop me a note when we should check this out again.

dipinhora · 2022-06-26T00:30:16Z

@SeanTAllen yes, i'll drop a note when ready.. this took a bit of a back seat to the backpressure/systematic testing stuff (and in general i'd prefer to get those merged first anyways and then rebase/update this one accounting for all those changes)..

dipinhora · 2022-06-26T00:31:53Z

argh, looks like a new comment automagically adds in the discuss during sync label so it's back again even though you just removed it... 8*/

Runtime allocation tracking now also tracks the number of heap allocations, the number of freed heap allocations and the number of GC iterations via counters. Additionally, there is now a way to check if runtime memory allocation tracking is enabled or not via `ifdef` statements in Pony code. This allows for some useful validations for those folks concerned about heap allocations in the critical path (i.e. if they rely on the compiler's `HeapToStack` optimization pass to convert heap allocations to stack allocations and want to validate it is working correctly).

This commit enhances the runtime stats tracking that was previously implemented under the `USE_MEMTRACK` and `USED_MEMTRACK_MESSAGES` defines. The new defines are called `USE_RUNTIMESTATS` and `USE_RUNTIMESTATS_MESSAGES`. Runtime stats tracking tracks the following actor info: * heap memory allocated * heap memory used * heap num allocated * heap realloc counter * heap alloc counter * heap free counter * heap gc counter * system message processing cpu usage * app message processing cpu usage * garbage collection cpu usage * messages sent counter * system messages processed counter * app messages processed counter Runtime tracking tracks the following scheduler info: * mutemap memory used * mutemap memory allocated * memory used for gc acquire/release actormaps and actors created * memory allocated for gc acquire/release actormaps and actors created * created actors counter * destroyed actors counter * actor system message processing cpu for all actor runs on the scheduler * actor app message processing cpu for all actor runs on the scheduler * actor garbage collection cpu for all actor runs on the scheduler * scheduler message processing cpu usage * scheduler misc cpu usage while waiting to do work * memory used by inflight messages * memory allocated by inflight messages * number of inflight messages This runtime stats tracking info has been exposed to pony programs as part of the `runtime_info` package and an example `runtime_info` program has been added to the `examples` directory.

dipinhora · 2022-07-11T03:13:55Z

@SeanTAllen This is ready for review.

jemc · 2022-07-12T18:32:38Z

src/libponyc/codegen/genprim.c

+static void platform_runtimestats(compile_t* c, reach_type_t* t, token_id cap)
+{
+  FIND_METHOD("runtimestats", cap);
+  start_function(c, t, m, c->i1, &c_t->use_type, 1);
+
+#if defined(USE_RUNTIMESTATS) || defined(USE_RUNTIMESTATS_MESSAGES)
+  bool runtimestats_enabled = true;
+#else
+  bool runtimestats_enabled = false;
+#endif


I made a comment in the sync call today that I'd eventually like to see this happen differently.

That is, the way this is written, it assumes that the USE_RUNTIMESTATS define is synchronized on both ponyc and libponyrt. What this implies is that in order to compile a program with runtime stats you need to use a completely different compiler instead of just linking to a different runtime.

I'd eventually like us to get to a place where we bundle both versions of the runtime (with and without stats) such that when compiling a program you can use a flag to switch it on or off.

Doesn't need to happen in this PR - we just want to make sure we create a followup ticket to capture that intended direction.

Sure. Ideally all of the various use options (

ponyc/Makefile

Lines 117 to 143 in a3b3bdf

$$(info Enabling use option: $1)

ifeq ($1,valgrind)

PONY_USES += -DPONY_USE_VALGRIND=true

else ifeq ($1,thread_sanitizer)

PONY_USES += -DPONY_USE_THREAD_SANITIZER=true

else ifeq ($1,address_sanitizer)

PONY_USES += -DPONY_USE_ADDRESS_SANITIZER=true

else ifeq ($1,undefined_behavior_sanitizer)

PONY_USES += -DPONY_USE_UNDEFINED_BEHAVIOR_SANITIZER=true

else ifeq ($1,coverage)

PONY_USES += -DPONY_USE_COVERAGE=true

else ifeq ($1,pooltrack)

PONY_USES += -DPONY_USE_POOLTRACK=true

else ifeq ($1,dtrace)

DTRACE ?= $(shell which dtrace)

ifeq (, $$(DTRACE))

$$(error No dtrace compatible user application static probe generation tool found)

endif

PONY_USES += -DPONY_USE_DTRACE=true

else ifeq ($1,scheduler_scaling_pthreads)

PONY_USES += -DPONY_USE_SCHEDULER_SCALING_PTHREADS=true

else ifeq ($1,systematic_testing)

PONY_USES += -DPONY_USE_SYSTEMATIC_TESTING=true

else ifeq ($1,memtrack)

PONY_USES += -DPONY_USE_MEMTRACK=true

else ifeq ($1,memtrack_messages)

PONY_USES += -DPONY_USE_MEMTRACK_MESSAGES=true

) would be easy to enable via cli arguments.

It's not easy to imagine at this time how we'd get all of those to be individually tweakable at Pony-program-compile-time unless we change how things work significantly in order to compile the runtime alongside the Pony program.

What I was thinking of in my previous comment would be to ship two versions of the runtime - one with extra stats enabled, and one without. We couldn't use that kind of approach with individually tweakable options unless we shipped an exponential number of runtimes with the Pony compiler releases.

A lot of those USE_* defines are more geared toward troubleshooting of the runtime by runtime developers (valgrind, sanitizers, etc), whereas I think a few of them (runtime stats, memtrack, pooltrack, dtrace) could be more helpful to application developers to instrument their application.

So I'm thinking it may be worthwhile to just lump a group of them together as a "with stats" version of the runtime and ship that. Then if some people really end up needing more granularity than that, they can custom build.

Another approach I've thought about in the past for this is seeing if it's possible to refactor these things such that they can be tweaked in the runtime bitcode individually, by rewriting certain instructions in the LLVM IR for the runtime bitcode, based on ponyc invocation flags. I did some brief work on this, but it got tricky because it's not enough to simply effect execution paths - a lot of these stats-related features also have an impact on memory layout (i.e. C ifdefs around struct field declarations), so I haven't worked further on that idea at this time.

i understand and agree with everything you've said the re: making use options controllable via ponyc and that it is not an easy or trivial thing to accomplish. i was simply stating that in an ideal scenario, they would all be selectable via ponyc as they can be useful for "power users" (along with folks working on/troubleshooting the runtime). Your idea around shipping multiple versions of the runtime and selecting between them based on compiler flags sounds like a great next step.

ergl · 2022-07-19T18:18:27Z

@dipinhora Can you add a note about the new --ponyprintstatsinterval flag to the release notes?

dipinhora · 2022-07-19T23:14:31Z

@ergl added

jemc · 2022-08-02T18:19:40Z

@SeanTAllen - do you want to review again before we merge?

SeanTAllen · 2022-08-02T18:26:53Z

I'm ok with no more review from me.

ponylang-main added the discuss during sync Should be discussed during an upcoming sync label Jun 17, 2022

SeanTAllen added the changelog - added Automatically add "Added" CHANGELOG entry on merge label Jun 17, 2022

dipinhora force-pushed the enhancememtrack branch from 7a16066 to 6fef040 Compare June 18, 2022 15:53

SeanTAllen added do not merge This PR should not be merged at this time and removed discuss during sync Should be discussed during an upcoming sync labels Jun 21, 2022

jemc removed the do not merge This PR should not be merged at this time label Jun 21, 2022

ponylang-main added the discuss during sync Should be discussed during an upcoming sync label Jun 21, 2022

SeanTAllen added do not merge This PR should not be merged at this time and removed discuss during sync Should be discussed during an upcoming sync labels Jun 21, 2022

ponylang-main added the discuss during sync Should be discussed during an upcoming sync label Jun 24, 2022

SeanTAllen removed the discuss during sync Should be discussed during an upcoming sync label Jun 26, 2022

ponylang-main added the discuss during sync Should be discussed during an upcoming sync label Jun 26, 2022

SeanTAllen removed the discuss during sync Should be discussed during an upcoming sync label Jun 26, 2022

dipinhora added 2 commits July 9, 2022 10:10

dipinhora force-pushed the enhancememtrack branch from 6fef040 to 12dd8cd Compare July 10, 2022 02:43

dipinhora changed the title ~~Enhance runtime memory allocation tracking~~ Enhance runtime stats tracking Jul 10, 2022

ponylang-main added the discuss during sync Should be discussed during an upcoming sync label Jul 10, 2022

Update release notes

9e15b7c

dipinhora mentioned this pull request Jul 11, 2022

Update backpressure implementation with runtime configurable threshold #4151

Open

SeanTAllen self-requested a review July 12, 2022 18:11

jemc reviewed Jul 12, 2022

View reviewed changes

jemc approved these changes Jul 14, 2022

View reviewed changes

dipinhora added 4 commits July 16, 2022 12:17

Split GC cpu tracking by mark and sweep phases

d299635

Add cli argument for printing stats

85eac54

Remove accidental binary

a088ef1

Fix unused parameter error

ae34944

jemc approved these changes Jul 19, 2022

View reviewed changes

dipinhora added 2 commits July 19, 2022 19:13

Fix printf format issue

7194a97

Add '--ponyprintstatsinterval' to release notes

037784a

ergl approved these changes Aug 2, 2022

View reviewed changes

SeanTAllen merged commit 1c8f7b9 into ponylang:main Aug 3, 2022

ponylang-main removed the discuss during sync Should be discussed during an upcoming sync label Aug 3, 2022

github-actions bot pushed a commit that referenced this pull request Aug 3, 2022

Updates release notes for PR #4144

8bb0c71

github-actions bot pushed a commit that referenced this pull request Aug 3, 2022

Update CHANGELOG for PR #4144

935b913

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enhance runtime stats tracking #4144

Enhance runtime stats tracking #4144

dipinhora commented Jun 17, 2022 •

edited

Loading

SeanTAllen commented Jun 17, 2022 •

edited

Loading

dipinhora commented Jun 18, 2022

SeanTAllen commented Jun 18, 2022

dipinhora commented Jun 18, 2022 •

edited

Loading

SeanTAllen commented Jun 18, 2022

dipinhora commented Jun 18, 2022

SeanTAllen commented Jun 18, 2022

dipinhora commented Jun 18, 2022

SeanTAllen commented Jun 21, 2022

SeanTAllen commented Jun 21, 2022

SeanTAllen commented Jun 24, 2022

SeanTAllen commented Jun 26, 2022

dipinhora commented Jun 26, 2022

dipinhora commented Jun 26, 2022

dipinhora commented Jul 11, 2022

jemc Jul 12, 2022

dipinhora Jul 13, 2022

jemc Jul 13, 2022

jemc Jul 13, 2022

dipinhora Jul 14, 2022

ergl commented Jul 19, 2022

dipinhora commented Jul 19, 2022

jemc commented Aug 2, 2022

SeanTAllen commented Aug 2, 2022

	$$(info Enabling use option: $1)
	ifeq ($1,valgrind)
	PONY_USES += -DPONY_USE_VALGRIND=true
	else ifeq ($1,thread_sanitizer)
	PONY_USES += -DPONY_USE_THREAD_SANITIZER=true
	else ifeq ($1,address_sanitizer)
	PONY_USES += -DPONY_USE_ADDRESS_SANITIZER=true
	else ifeq ($1,undefined_behavior_sanitizer)
	PONY_USES += -DPONY_USE_UNDEFINED_BEHAVIOR_SANITIZER=true
	else ifeq ($1,coverage)
	PONY_USES += -DPONY_USE_COVERAGE=true
	else ifeq ($1,pooltrack)
	PONY_USES += -DPONY_USE_POOLTRACK=true
	else ifeq ($1,dtrace)
	DTRACE ?= $(shell which dtrace)
	ifeq (, $$(DTRACE))
	$$(error No dtrace compatible user application static probe generation tool found)
	endif
	PONY_USES += -DPONY_USE_DTRACE=true
	else ifeq ($1,scheduler_scaling_pthreads)
	PONY_USES += -DPONY_USE_SCHEDULER_SCALING_PTHREADS=true
	else ifeq ($1,systematic_testing)
	PONY_USES += -DPONY_USE_SYSTEMATIC_TESTING=true
	else ifeq ($1,memtrack)
	PONY_USES += -DPONY_USE_MEMTRACK=true
	else ifeq ($1,memtrack_messages)
	PONY_USES += -DPONY_USE_MEMTRACK_MESSAGES=true

Enhance runtime stats tracking #4144

Enhance runtime stats tracking #4144

Conversation

dipinhora commented Jun 17, 2022 • edited Loading

SeanTAllen commented Jun 17, 2022 • edited Loading

dipinhora commented Jun 18, 2022

SeanTAllen commented Jun 18, 2022

dipinhora commented Jun 18, 2022 • edited Loading

SeanTAllen commented Jun 18, 2022

dipinhora commented Jun 18, 2022

SeanTAllen commented Jun 18, 2022

dipinhora commented Jun 18, 2022

SeanTAllen commented Jun 21, 2022

SeanTAllen commented Jun 21, 2022

SeanTAllen commented Jun 24, 2022

SeanTAllen commented Jun 26, 2022

dipinhora commented Jun 26, 2022

dipinhora commented Jun 26, 2022

dipinhora commented Jul 11, 2022

jemc Jul 12, 2022

Choose a reason for hiding this comment

dipinhora Jul 13, 2022

Choose a reason for hiding this comment

jemc Jul 13, 2022

Choose a reason for hiding this comment

jemc Jul 13, 2022

Choose a reason for hiding this comment

dipinhora Jul 14, 2022

Choose a reason for hiding this comment

ergl commented Jul 19, 2022

dipinhora commented Jul 19, 2022

jemc commented Aug 2, 2022

SeanTAllen commented Aug 2, 2022

dipinhora commented Jun 17, 2022 •

edited

Loading

SeanTAllen commented Jun 17, 2022 •

edited

Loading

dipinhora commented Jun 18, 2022 •

edited

Loading