CustomProperty improvements #1261

eddynaka · 2020-09-11T22:04:37Z

Refs #1206

Changes

Instrumntation no longer store raw objects like HttpRequest in
Activity.CustomProperty. To enrich activity, use the Enrich action on the
instrumentation.

Please provide a brief description of the changes here.

For significant contributions please make sure you have completed the following items:

CHANGELOG.md updated for non-trivial changes
Design discussion issue #
Changes in public API reviewed

codecov · 2020-09-11T22:44:47Z

Codecov Report

Merging #1261 into master will decrease coverage by 0.39%.
The diff coverage is 44.00%.

@@            Coverage Diff             @@
##           master    #1261      +/-   ##
==========================================
- Coverage   77.44%   77.05%   -0.40%     
==========================================
  Files         223      223              
  Lines        6362     6428      +66     
==========================================
+ Hits         4927     4953      +26     
- Misses       1435     1475      +40

Impacted Files	Coverage Δ
...s/StackExchangeRedisCallsInstrumentationOptions.cs	`100.00% <ø> (ø)`
...Implementation/AspNetInstrumentationEventSource.cs	`80.00% <40.00%> (-20.00%)`	⬇️
...umentation.AspNet/Implementation/HttpInListener.cs	`86.31% <40.00%> (-7.94%)`	⬇️
...ementation/AspNetCoreInstrumentationEventSource.cs	`80.00% <40.00%> (-20.00%)`	⬇️
...tation.AspNetCore/Implementation/HttpInListener.cs	`87.61% <40.00%> (-4.78%)`	⬇️
...ent/Implementation/GrpcClientDiagnosticListener.cs	`83.78% <40.00%> (-7.13%)`	⬇️
...t/Implementation/GrpcInstrumentationEventSource.cs	`62.50% <40.00%> (-37.50%)`	⬇️
...tp/Implementation/HttpHandlerDiagnosticListener.cs	`78.82% <40.00%> (-8.85%)`	⬇️
...p/Implementation/HttpInstrumentationEventSource.cs	`60.86% <40.00%> (-5.80%)`	⬇️
...ient/Implementation/SqlClientDiagnosticListener.cs	`75.36% <40.00%> (-3.10%)`	⬇️
... and 9 more

CodeBlanch · 2020-09-11T23:54:44Z

Just thinking out loud here. How would this work if you were shipping something that wanted to access the raw objects downstream? For example, let's say we are building a custom ActivityProcessor that adds extra data for http spans. The hosting application owns the options, so our custom ActivityProcessor would have to ask the user through documentation to turn these on in the host before it would work? Ideally we would have a way for it to request that extra data automatically. Could we use the runtime context somehow?

reyang · 2020-09-12T00:04:25Z

Just thinking out loud here. How would this work if you were shipping something that wanted to access the raw objects downstream? For example, let's say we are building a custom ActivityProcessor that adds extra data for http spans. The hosting application owns the options, so our custom ActivityProcessor would have to ask the user through documentation to turn these on in the host before it would work? Ideally we would have a way for it to request that extra data automatically. Could we use the runtime context somehow?

I guess the goal is to reduce the expensive dictionary operation AND the potential pressure from GC?
Consider the following approach:

Expose these instrumentation raw object via runtime context, so processors (the code running in the context) could access these raw objects and cherry-pick stuff (e.g. take something from the raw object and call SetTag).
Once we reached the end of the processor chain, remove all the raw objects so we don't stress GC too much.

eddynaka · 2020-09-12T00:15:23Z

Just thinking out loud here. How would this work if you were shipping something that wanted to access the raw objects downstream? For example, let's say we are building a custom ActivityProcessor that adds extra data for http spans. The hosting application owns the options, so our custom ActivityProcessor would have to ask the user through documentation to turn these on in the host before it would work? Ideally we would have a way for it to request that extra data automatically. Could we use the runtime context somehow?

I guess the goal is to reduce the expensive dictionary operation AND the potential pressure from GC?
Consider the following approach:

Expose these instrumentation raw object via runtime context, so processors (the code running in the context) could access these raw objects and cherry-pick stuff (e.g. take something from the raw object and call SetTag).

Once we reached the end of the processor chain, remove all the raw objects so we don't stress GC too much.

got it, will change to runtimecontext and see how that works. What would work best for the key? type+traceid? Something like that? What do you think?
when you say end of the processor chain, would we do that on flush?

reyang · 2020-09-12T00:18:43Z

got it, will change to runtimecontext and see how that works. What would work best for the key? type+traceid? Something like that? What do you think?

Haven't spent much time thinking about this, I guess we probably would end up with 1 key, which contains a linked list?

when you say end of the processor chain, would we do that on flush?

It is when the last processor.OnEnd returned.

eddynaka · 2020-09-12T00:19:05Z

Just thinking out loud here. How would this work if you were shipping something that wanted to access the raw objects downstream? For example, let's say we are building a custom ActivityProcessor that adds extra data for http spans. The hosting application owns the options, so our custom ActivityProcessor would have to ask the user through documentation to turn these on in the host before it would work? Ideally we would have a way for it to request that extra data automatically. Could we use the runtime context somehow?

If we use runtimecontext, so we wouldn't need the flag at all, right?

CodeBlanch · 2020-09-12T00:20:04Z

@eddynaka I was thinking we would store the flag on runtime context to tell the instrumentation to use the custom property to store them, but open to @reyang's version where we store the actual objects on the runtime context and don't need a flag at all.

eddynaka · 2020-09-12T00:23:14Z

got it, will change to runtimecontext and see how that works. What would work best for the key? type+traceid? Something like that? What do you think?

Haven't spent much time thinking about this, I guess we probably would end up with 1 key, which contains a linked list?

when you say end of the processor chain, would we do that on flush?

It is when the last processor.OnEnd returned.

What if we change from LinkedList to Dictionary? At least this way we would have O(1) based on some key to remove from the RuntimeContext.

eddynaka · 2020-09-12T00:54:01Z

@reyang @CodeBlanch , pushed one change moving to RuntimeContextSlot. Still a working in progress. But, the general idea:

so, i would save as object, so the activityprocessor wouldn't need to import system.web for example.
added as a list, so for each trace we could have multiple raw objects
added as a dictionary, so we would have O(1) to fetch and discard in onend

My thoughts:

thinking about concurrency. I will probably change to ConcurrentDictionary
thinking about ordering. If ActivyProcessor occurs before the instrumentation, then we would have an exception.

CodeBlanch · 2020-09-13T17:03:56Z

@eddynaka RuntimeContext is essentially bound to a single trace. For example, check out how Baggage is implemented. It doesn't worry about other traces or clearing at the end of the trace. The not clearing at the end is trippy. I wrote a bunch of tests when I was doing Baggage to see if it ever leaked to other threads, couldn't ever get it to leak.

Esstentially what I'm saying is you can remove the cleanup part and switch to:

private static readonly RuntimeContextSlot<List<object>> RuntimeContextSlot = RuntimeContext.RegisterSlot<List<object>>("OTel.Runtime.Context");

(@reyang Can you confirm the cleanup part?)

BUT, I don't think we have really accomplished anything. We just switched allocation from Activity to RuntimeContext. What we really need is a way for some library component (ActivityProcessor, Exporter, Instrumentation, etc) or user to opt into the behavior globally and then the instrumentation adds it based on the behavior being triggered?

CodeBlanch · 2020-09-13T18:49:47Z

One more thing I just realized...

opentelemetry-dotnet/src/OpenTelemetry/Trace/TracerProviderSdk.cs

Line 78 in 768eaa3

activity.SetResource(this.resource);

opentelemetry-dotnet/src/OpenTelemetry/Trace/ActivitySourceAdapter.cs

Line 92 in 768eaa3

activity.SetResource(this.resource);

We set the Resource on every sampled Activity we see, which is actually calling SetCustomProperty. So if the goal of this work is to avoid the automatic allocation of the internal CustomProperty Dictionary on every Activity, we also need to solve that 😄

eddynaka · 2020-09-13T23:30:16Z

Applying the change to RuntimeContext and doing a benchmark, I could see that the Allocated memory increased:

Before (master branch):

Method	Mean	Error	StdDev	Gen 0	Gen 1	Gen 2	Allocated
InstrumentedHttpClient	186.1 us	3.72 us	5.79 us	2.1973	-	-	4.68 KB
InstrumentedHttpClientWithParentActivity	179.6 us	3.00 us	2.34 us	2.4414	-	-	5.99 KB

Actual (Runtime Context):

Method	Mean	Error	StdDev	Gen 0	Gen 1	Gen 2	Allocated
InstrumentedHttpClient	179.0 us	3.51 us	2.93 us	2.1973	-	-	4.81 KB
InstrumentedHttpClientWithParentActivity	177.5 us	2.70 us	2.52 us	2.4414	-	-	6.13 KB

eddynaka · 2020-09-13T23:40:04Z

I think, in general, the idea of the flag was to prevent to write somewhere (activity, rutimecontext, etc). With that, if we had that flag as false, we would have less allocations (theoretically).

If condition:

Method	Mean	Error	StdDev	Gen 0	Gen 1	Gen 2	Allocated
InstrumentedHttpClient	178.6 us	2.17 us	1.93 us	2.1973	-	-	4.63 KB
InstrumentedHttpClientWithParentActivity	180.8 us	3.57 us	6.80 us	2.4414	-	-	5.95 KB

so, we would have a little less allocation comparing to master branch.

cijothomas · 2020-09-14T16:13:17Z

Two primary goals:

Check if storing the raw object (in customprop or context or anywhere else) - is an expensive thing.
If yes, then make it opt-in. (this also means to benefit from this feature, its not enough to write a processor, but also change the instrumentation options.)
Cleanup part - we certainly don't want to store HttpWebRequest inside activity forever. If exporters are slow, then this HttpWebRequest won't be claimed by GC for a potentially long time. If runtimecontext is naturally solving this problem, without any explicit cleanup, then lets go for it.

I also think of another alternative:
Similar to Instrumentation's Filter API, we could introduce a Enrich api on instrumentation, which is a callback which gets passed the activity and the corresponding raw object. Users can register Enrich callback, if they want to enrich activity with anything from the raw objects. This also avoids storing any raw object anywhere, so no question of cleaning up part.

eddynaka · 2020-09-14T19:34:46Z

Two primary goals:

Check if storing the raw object (in customprop or context or anywhere else) - is an expensive thing.
If yes, then make it opt-in. (this also means to benefit from this feature, its not enough to write a processor, but also change the instrumentation options.)

Cleanup part - we certainly don't want to store HttpWebRequest inside activity forever. If exporters are slow, then this HttpWebRequest won't be claimed by GC for a potentially long time. If runtimecontext is naturally solving this problem, without any explicit cleanup, then lets go for it.

I also think of another alternative:
Similar to Instrumentation's Filter API, we could introduce a Enrich api on instrumentation, which is a callback which gets passed the activity and the corresponding raw object. Users can register Enrich callback, if they want to enrich activity with anything from the raw objects. This also avoids storing any raw object anywhere, so no question of cleaning up part.

If we go to this direction, we can do in Options file:

/// <summary>
/// Gets or sets a Subscribe function.
/// </summary>
public Action<object> Subscribe { get; set; }

With that, we can simply subscribe for events like this:

using (Sdk.CreateTracerProviderBuilder()
    .AddHttpClientInstrumentation(options =>
    {
        options.Subscribe = (req) => { Console.WriteLine(req); };
    })
    .AddProcessor(activityProcessor.Object)
    .Build())
{
    using var c = new HttpClient();
    using var r = await c.GetAsync("https://opentelemetry.io/").ConfigureAwait(false);
}

For the instrumentation itself:

if (activity.IsAllDataRequested)
{
    this.options?.Subscribe(request);

    activity.SetTag(SemanticConventions.AttributeHttpMethod, HttpTagHelper.GetNameForHttpMethod(request.Method));
    activity.SetTag(SemanticConventions.AttributeHttpHost, HttpTagHelper.GetHostTagValueFromRequestUri(request.RequestUri));
    activity.SetTag(SemanticConventions.AttributeHttpUrl, request.RequestUri.OriginalString);

    if (this.options.SetHttpFlavor)
    {
        activity.SetTag(SemanticConventions.AttributeHttpFlavor, HttpTagHelper.GetFlavorTagValueFromProtocolVersion(request.Version));
    }
}

With that, the customer would have access to those properties, and we wouldn't have any issue with performance / allocation.

Below, the benchmark:

Method	Mean	Error	StdDev	Gen 0	Gen 1	Gen 2	Allocated
InstrumentedHttpClient	230.4 us	4.47 us	6.55 us	2.1973	-	-	4.48 KB
InstrumentedHttpClientWithParentActivity	230.8 us	4.55 us	5.42 us	2.4414	-	-	5.79 KB

I can see that Mean/Error/StdDev increased. But, allocated decreased.

What do you think? @cijothomas @reyang @CodeBlanch

CodeBlanch · 2020-09-14T22:38:12Z

@eddynaka Regarding this API:

public Action<object> Subscribe { get; set; }

I've started to build this API before and ended up backing off because there's a surprising amount of things to deal with. You have 3 things you might want to pull data from: request, response, & (potentially) an exception. Should we have an enrichment function for all three things? That seems like a lot. But if you combine it all, it starts to get a little messy:

public Action<Activity, HttpContext, Exception> ActivityEnrichmentAction { get; set; }

(That is passing HttpContext to combine at least request & response.)

Just some things to consider 😄

eddynaka · 2020-09-14T22:41:37Z

@eddynaka Regarding this API:
public Action<object> Subscribe { get; set; }
I've started to build this API before and ended up backing off because there's a surprising amount of things to deal with. You have 3 things you might want to pull data from: request, response, & (potentially) an exception. Should we have an enrichment function for all three things? That seems like a lot. But if you combine it all, it starts to get a little messy:
public Action<Activity, HttpContext, Exception> ActivityEnrichmentAction { get; set; }
(That is passing HttpContext to combine at least request & response.)

Just some things to consider 😄

Hi @CodeBlanch , that's why I added as object. And I would pass the request/response/exception/other types if needed.

And the user would have to typeof/gettype to see what is it to do something.

What is a surprise for me is the increase of time.

CodeBlanch · 2020-09-14T22:50:34Z

I'll look at it a bit more closely later to try and understand the performance. Giving user an object works, it just isn't that nice of an API to code against. Forces them to do casting, type checking, maybe even reflection. Things that cost performance. Preferable to have something that exposes a strongly-typed API.

eddynaka · 2020-09-18T16:34:36Z

I'll look at it a bit more closely later to try and understand the performance. Giving user an object works, it just isn't that nice of an API to code against. Forces them to do casting, type checking, maybe even reflection. Things that cost performance. Preferable to have something that exposes a strongly-typed API.

Hi @CodeBlanch , did you have time to take a look? Should I go and maintain the if / Action / other? Let me know! :)

CodeBlanch · 2020-09-18T19:28:59Z

@eddynaka Hey I'm kind of confused where are on this. I don't see the Subscribe callback on the branch so I can't perf test it. Can you throw that up somewhere so I can take it for a spin?

Right now I see:

Options flag StoreRawObjectInActivity.
We're storing on RuntimeContext.

That doesn't make a whole lot of sense to me. I think if we have the flag, we should store using activity.SetCustomProperty. My idea to use RuntimeContext was basically a way to get around needing the flag. It was originally to move the flag to RuntimeContext and use the activity for the storage.

Maybe we should do a call on this to get everyone on the same page with the design?

eddynaka · 2020-09-18T19:41:38Z

@eddynaka Hey I'm kind of confused where are on this. I don't see the Subscribe callback on the branch so I can't perf test it. Can you throw that up somewhere so I can take it for a spin?

Right now I see:

Options flag StoreRawObjectInActivity.

We're storing on RuntimeContext.

That doesn't make a whole lot of sense to me. I think if we have the flag, we should store using activity.SetCustomProperty. My idea to use RuntimeContext was basically a way to get around needing the flag. It was originally to move the flag to RuntimeContext and use the activity for the storage.

Maybe we should do a call on this to get everyone on the same page with the design?

Hum.. i think I didn't push it.

What I will do in this branch:

nplicate httpclient instrumentation
add to benchmark and we can choose

eddynaka · 2020-09-18T20:10:00Z

Below a simple benchmark:

Method	Mean	Error	StdDev	Gen 0	Gen 1	Gen 2	Allocated
SimpleHttpClient	169.5 us	3.31 us	5.88 us	1.4648	-	-	1.84 KB
InstrumentedHttpClientFlag	193.3 us	3.74 us	4.86 us	1.9531	-	-	4.13 KB
InstrumentedHttpClientWithParentActivityFlag	192.9 us	3.76 us	4.76 us	2.1973	-	-	4.93 KB
InstrumentedHttpClientRuntime	189.4 us	3.78 us	4.36 us	2.1973	-	-	4.3 KB
InstrumentedHttpClientWithParentActivityRuntime	194.9 us	3.82 us	5.22 us	2.1973	-	-	5.11 KB
InstrumentedHttpClientSubscribe	189.1 us	3.67 us	3.92 us	1.9531	-	-	4.13 KB
InstrumentedHttpClientWithParentActivitySubscribe	191.8 us	3.48 us	3.87 us	2.1973	-	-	4.93 KB

If we go with flag: the activity gets big
If we go with subscribe: this can get slow, since it depends in what logic you add in the Action
If we go with Runtime: it's restricted to OTel, right?

CodeBlanch · 2020-10-02T21:42:32Z

@eddynaka Apologies for not looking at this sooner!

@cijothomas To answer the fundamental question, how expensive is SetCustomProperty? I made a few quick low-level benchmarks:

        [Benchmark]
        public void BasicActivity()
        {
            using Activity activity = new Activity("Activity");
        }

        [Benchmark]
        public void ActivityWithSingleCustomProperty()
        {
            using Activity activity = new Activity("Activity");
            activity.SetCustomProperty("activity", activity);
        }

        [Benchmark]
        public void ActivityWithManyCustomProperties()
        {
            using Activity activity = new Activity("Activity");
            activity.SetCustomProperty("activity1", activity);
            activity.SetCustomProperty("activity2", activity);
            activity.SetCustomProperty("activity3", activity);
        }

Results:

Method	Mean	Error	StdDev	Gen 0	Gen 1	Gen 2	Allocated
BasicActivity	5.156 us	0.0966 us	0.0904 us	0.0381	-	-	376 B
ActivityWithSingleCustomProperty	5.246 us	0.0530 us	0.0495 us	0.0687	-	-	584 B
ActivityWithManyCustomProperties	5.461 us	0.0544 us	0.0482 us	0.0687	-	-	584 B

The results make sense to me. The first one you add, big hit. It has to create the storage. Anything added after that, negligible.

Right now we add the Resource on all sampled non-suppressed Activity objects:
https://github.com/open-telemetry/opentelemetry-dotnet/blob/master/src/OpenTelemetry/Trace/TracerProviderSdk.cs#L78

That means we are already paying the big price, regardless of what the instrumentation does. Unless we change that, all of this work is moot. But I was chatting with @eddynaka on Gitter earlier and his goal is to fix that 😄

Right now we have 3 options on this PR:

Add a flag to control setting the raw objects as custom properties.
Add custom properties to RuntimeContext.
Add a callback user can use via options to enrich Activity using the raw objects.

None of them are absolutely perfect. 1 & 2 allow you to plug in an ActivityProcessor to enrich based on the raw sources. Given the spec says ActivityProcessor is our extension point for enrichment, it feels to me like we should support that. But Option 3 is probably more useful for most users to quickly enrich without having to build an ActivityProcessor.

I'm thinking we scope this PR to Option 3 and then maybe we can do something to enable the other case?

That being said, I really don't like the API...

Action<object> Enrich { get; set; }

User will have to do expensive checks to figure out if object is request, response, or exception. They also need to access Activity.Current.

I think at the minimum it should be...

Action<Activity, string, object> Enrich { get; set; }

Where the string is the event name. That would help with knowing what object is going to be.

I would prefer something strongly typed but that also comes with some challenges.

updating tests updating to RuntimeContextSlot updating to runtimeContext adding runtime/subscibe/flag benchmark updating aspnet and http updating ramaining instrumentation

src/OpenTelemetry.Instrumentation.AspNetCore/Implementation/HttpInListener.cs

cijothomas · 2020-10-05T23:57:56Z

This PR looks good to me, assuming its implement option 3 @CodeBlanch mentioned.
This avoids touching Activity.CustomProperty and should save perf (of course assuming we get rid of every other usage of customproperty as well)
No need of worrying about cleaning up the strong objects.

We can discuss alternate namings, comments, example usage in a subsequent PR.

src/OpenTelemetry.Instrumentation.AspNet/AspNetInstrumentationOptions.cs

cijothomas · 2020-10-06T00:00:55Z

src/OpenTelemetry.Instrumentation.AspNetCore/Implementation/HttpInListener.cs

@@ -112,7 +112,7 @@ public override void OnStartActivity(Activity activity, object payload)

            if (activity.IsAllDataRequested)
            {
-                activity.SetCustomProperty(RequestCustomPropertyName, request);
+                this.options.Enrich?.Invoke(activity, "OnStartActivity", request);


Also - can you ensure that unit test validate that Enrich is called only if Activity is sampled-in.

Just added a test. Let me know what do you think before I update everything

….com/eddynaka/opentelemetry-dotnet into feature/custom-property-improvements

src/OpenTelemetry/Trace/ActivityProcessor.cs

CodeBlanch

LGTM

eddynaka requested a review from a team September 11, 2020 22:04

eddynaka changed the title ~~CustomProperty improvements~~ [WIP] CustomProperty improvements Sep 11, 2020

eddynaka marked this pull request as draft September 11, 2020 23:32

CustomProperty improvements

8e91500

updating tests updating to RuntimeContextSlot updating to runtimeContext adding runtime/subscibe/flag benchmark updating aspnet and http updating ramaining instrumentation

eddynaka self-assigned this Oct 4, 2020

eddynaka marked this pull request as ready for review October 4, 2020 11:18

eddynaka changed the title ~~[WIP] CustomProperty improvements~~ CustomProperty improvements Oct 4, 2020

Merge branch 'master' into feature/custom-property-improvements

1059a94

cijothomas reviewed Oct 5, 2020

View reviewed changes

src/OpenTelemetry.Instrumentation.AspNetCore/Implementation/HttpInListener.cs Outdated Show resolved Hide resolved

cijothomas reviewed Oct 5, 2020

View reviewed changes

src/OpenTelemetry.Instrumentation.AspNet/AspNetInstrumentationOptions.cs Show resolved Hide resolved

cijothomas reviewed Oct 6, 2020

View reviewed changes

cijothomas approved these changes Oct 6, 2020

View reviewed changes

eddynaka added 4 commits October 6, 2020 05:48

adding try/catch&log

1065b94

Merge branch 'feature/custom-property-improvements' of https://github…

5727e99

….com/eddynaka/opentelemetry-dotnet into feature/custom-property-improvements

improve docs for enrich action

edf078e

Checking enrichment with sampler

5c63d64

CodeBlanch reviewed Oct 6, 2020

View reviewed changes

src/OpenTelemetry/Trace/ActivityProcessor.cs Outdated Show resolved Hide resolved

CodeBlanch approved these changes Oct 6, 2020

View reviewed changes

eddynaka added 4 commits October 6, 2020 14:27

removing unneeded using

a297554

Merge branch 'master' into feature/custom-property-improvements

f3f47dd

updating changelog

29fc4a4

updating changelog

8bc4e92

cijothomas merged commit 874f9d7 into open-telemetry:master Oct 6, 2020

eddynaka deleted the feature/custom-property-improvements branch October 17, 2020 00:08

This was referenced Oct 26, 2020

Avoid Activity.SetCustomPropery usage for Resource. #1397

Closed

Enrich for HttpClient .NET Framework instrumentation #1401

Closed

eddynaka mentioned this pull request Nov 5, 2020

Adding resource tags dinamically after initialization #1470

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CustomProperty improvements #1261

CustomProperty improvements #1261

eddynaka commented Sep 11, 2020 •

edited

Loading

codecov bot commented Sep 11, 2020 •

edited

Loading

CodeBlanch commented Sep 11, 2020

reyang commented Sep 12, 2020

eddynaka commented Sep 12, 2020

reyang commented Sep 12, 2020

eddynaka commented Sep 12, 2020

CodeBlanch commented Sep 12, 2020

eddynaka commented Sep 12, 2020

eddynaka commented Sep 12, 2020

CodeBlanch commented Sep 13, 2020

CodeBlanch commented Sep 13, 2020

eddynaka commented Sep 13, 2020

eddynaka commented Sep 13, 2020

cijothomas commented Sep 14, 2020 •

edited

Loading

eddynaka commented Sep 14, 2020 •

edited

Loading

CodeBlanch commented Sep 14, 2020

eddynaka commented Sep 14, 2020

CodeBlanch commented Sep 14, 2020

eddynaka commented Sep 18, 2020

CodeBlanch commented Sep 18, 2020

eddynaka commented Sep 18, 2020

eddynaka commented Sep 18, 2020

CodeBlanch commented Oct 2, 2020

cijothomas commented Oct 5, 2020

cijothomas Oct 6, 2020

eddynaka Oct 6, 2020

CodeBlanch left a comment

CustomProperty improvements #1261

CustomProperty improvements #1261

Conversation

eddynaka commented Sep 11, 2020 • edited Loading

Changes

codecov bot commented Sep 11, 2020 • edited Loading

Codecov Report

CodeBlanch commented Sep 11, 2020

reyang commented Sep 12, 2020

eddynaka commented Sep 12, 2020

reyang commented Sep 12, 2020

eddynaka commented Sep 12, 2020

CodeBlanch commented Sep 12, 2020

eddynaka commented Sep 12, 2020

eddynaka commented Sep 12, 2020

CodeBlanch commented Sep 13, 2020

CodeBlanch commented Sep 13, 2020

eddynaka commented Sep 13, 2020

eddynaka commented Sep 13, 2020

cijothomas commented Sep 14, 2020 • edited Loading

eddynaka commented Sep 14, 2020 • edited Loading

CodeBlanch commented Sep 14, 2020

eddynaka commented Sep 14, 2020

CodeBlanch commented Sep 14, 2020

eddynaka commented Sep 18, 2020

CodeBlanch commented Sep 18, 2020

eddynaka commented Sep 18, 2020

eddynaka commented Sep 18, 2020

CodeBlanch commented Oct 2, 2020

cijothomas commented Oct 5, 2020

cijothomas Oct 6, 2020

Choose a reason for hiding this comment

eddynaka Oct 6, 2020

Choose a reason for hiding this comment

CodeBlanch left a comment

Choose a reason for hiding this comment

eddynaka commented Sep 11, 2020 •

edited

Loading

codecov bot commented Sep 11, 2020 •

edited

Loading

cijothomas commented Sep 14, 2020 •

edited

Loading

eddynaka commented Sep 14, 2020 •

edited

Loading