Event Reclassification Using Logger Input/Application Insights Output

[timomta]

EventFlow appears to reclassify events. Here is how I logged an error in my code:

   this.logger.LogError($"No response from WebCacheService, which indicates an exception was thrown attempting to open a connection to server hosting the url. Please check the WebCacheService logs for errors related to this url: '{id.ToString()}'");

The logger was injected from the Asp.Net Core startup.cs.

Attached is a screenshot of Application Insights. I see what was originally an error logged instead with Trace status with severity level of verbose.

[rjregenold]

I'm experiencing the same behavior using the EventSource input and Application Insights output.

[rjregenold]

For what it's worth, I switched to Serilog and changed my EventFlow input to the Serilog Input (while still using the Application Insights output). The severity level and message are set correctly using this combination.

[karolz-ms]

@rjregenold I can't reproduce this using EventSource. https://gist.github.com/karolz-ms/b3d30a5fdf99fcca38245e3d9e6e96cc seems to work fine

Trying Microsoft.Extensions.Logging now...

[karolz-ms]

@timomta Can't reproduce using Microsoft.Extensions.Logging either. Gist updated to include this case.

However I noticed you have some property name conflicts (e.g. you ave EventId and EventId_0, Level and Level_0), indicating that custom event properties are conflicting with standard event properties. That is probably why the events are defaulting to Informational level. Can you investigate and explain why these conflicts occur?

[timomta]

I'm not doing anything to assign properties. If you take a look at my calling statement, it is just a call to ILogger LogError() with some text for the message. I'm using these packages in an ASP.Net Core project:
Microsoft.Extensions.Logging
Microsoft.Extensions.Logging.Abstractions
Microsoft.Extensions.Logging.EventSource

The Configure method of the Startup class looks like this:
public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory)
{
loggerFactory.AddConsole(Configuration.GetSection("Logging"));
loggerFactory.AddEventSourceLogger();

        app.UseMvc();

...
}

If you'd like me to create a repro project and send it off line, let me know, happy to. Thanks a lot for your help.

[karolz-ms]

Yes, please do, that will help--thanks!

[rjregenold]

@karolz-ms Thank you for looking into this. I'll see if I can create a small repo that reproduces the issue today.

[rjregenold]

@karolz-ms Here is a repo that reproduces the issue I am seeing:

https://github.com/rjregenold/eventflow-eventsource-ai-test

Please let me know if I can do anything else to help track this down. Thanks!

[karolz-ms]

@rjregenold I can reproduce the problem, thanks. Looks like this is related to EventSources that use self-describing format. Let me do some digging to see what we can do about this...

[karolz-ms]

@rjregenold Unfortunately this seems to be a bug in the desktop .NET Framework. I tried .NET 4.6, 4.6.1 and the newly released 4.7, all have the same issue with self-describing EventSource.

.NET Core (tried 1.1) does not seem to be affected.

So the workaround is to either use .NET Core or not use self-describing EventSource. Kind of sucks. Sorry!

@timomta Let us know if yours is the same issue or something else--thx!

[rjregenold]

@karolz-ms No problem. Thank you very much for checking it out! Serilog is working fine for now.

[timomta]

Yes, if it is a .Net bug, it is the same issue then. I'll take a look at Serilog. I have not used it before.

[karolz-ms]

@timomta so what API are you using as your logging API? Microsoft.Extensions.Logging?

If so, any particular reason why you are going through Ms.E.Logging --> EventSource --> EventFlow instead directly from Ms.E.Logging --> EventFlow?

[timomta]

I’ve written a Service Fabric application, so there isn’t any single place that logs can be written since services migrate around the cluster as necessary for load balancing. Service Fabric writes to an EventSource natively. I’m following that same pattern by writing my application logs on an EventSource provider via ILogger.

I then use EventFlow to capture all the ETW/EventSource traffic and send it to Application Insights, which is a single repository I can read it from.

I’m completely open to suggestions on better approaches if there are.

[karolz-ms]

@timomta In your case I would suggest using Microsoft.Extensions.Logging implementation of ILogger pattern to write to EventFlow directly, as described here https://github.com/Azure/diagnostics-eventflow#microsoftextensionslogging This is faster and also I expect it will circumvent the bug described in this thread. This will apply to your application logs. For SF logs continue to use EventSource input.

[timomta]

Thanks! I'll give that a shot.

[timomta]

I looked into this and am at a bit of a loss as to the proper way to implement. Since I'm using a Service Fabric service, my pipeline is created in the services Program.Main (per the same article under Service Fabric Support heading). Main creates a host process for Web Listener. Web Listener then calls into Startup as part of its build/config process. Asp.Net Core injects the LoggingFactory, I don't create it and don't set its pipeline.

In the example you pointed me to, the whole pipeline and LoggingFactory are set in one place. What is the correct way to follow this type of pattern when you have a Asp.Net core injected LoggingFactory via its Startup class?

[karolz-ms]

@timomta If you are using Microsoft.Extensions.Logging in a ASP.NET Core environment and intend to send data to AI, you do not even need EventFlow. Just use Microsoft.ApplicationInsights.AspNetCore package--it integrates with Microsoft.Extensions.Logging, so it is all you need.

But assuming you need EventFlow for some other purposes, here is one way to hook things up (the following is a modification of the standard Service Fabric ASP.NET Core-based stateless service project):

1. Create the EventFlow diagnostic pipeline in Program.Main
2. Pass the pipeline as a constructor parameter to your service class:

        private static void Main()
        {
            try
            {
                using (var pipeline = ServiceFabricDiagnosticPipelineFactory.CreatePipeline("CoreBasedFabricPlusEventFlow-Diagnostics"))
                {
                    ServiceRuntime.RegisterServiceAsync("Web1Type",
                        context => new Web1(context, pipeline)).GetAwaiter().GetResult();

                    ServiceEventSource.Current.ServiceTypeRegistered(Process.GetCurrentProcess().Id, typeof(Web1).Name);

                    // Prevents this host process from terminating so services keeps running. 
                    Thread.Sleep(Timeout.Infinite);
                }
            }
            catch (Exception e)
            {
                ServiceEventSource.Current.ServiceHostInitializationFailed(e.ToString());
                throw;
            }
        }

3. In the CreateServiceInstanceListeners() method add the pipeline as a singleton service to ASP.NET dependency injection container

        protected override IEnumerable<ServiceInstanceListener> CreateServiceInstanceListeners()
        {
            return new ServiceInstanceListener[]
            {
                new ServiceInstanceListener(serviceContext =>
                    new WebListenerCommunicationListener(serviceContext, "ServiceEndpoint", (url, listener) =>
                    {
                        ServiceEventSource.Current.ServiceMessage(serviceContext, $"Starting WebListener on {url}");

                        return new WebHostBuilder().UseWebListener()
                                    .ConfigureServices(
                                        services => services
                                            .AddSingleton<StatelessServiceContext>(serviceContext)
                                            .AddSingleton<DiagnosticPipeline>(this.diagnosticPipeline))
                                    .UseContentRoot(Directory.GetCurrentDirectory())
                                    .UseStartup<Startup>()
                                    .UseApplicationInsights()
                                    .UseServiceFabricIntegration(listener, ServiceFabricIntegrationOptions.None)
                                    .UseUrls(url)
                                    .Build();
                    }))
            };
        }

4. In the Startup class configure the loggerFactory by calling AddEventFlow on it:

        public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory)
        {
            loggerFactory.AddConsole(Configuration.GetSection("Logging"));
            loggerFactory.AddDebug();
            var diagnosticPipeline = app.ApplicationServices.GetRequiredService<DiagnosticPipeline>();
            loggerFactory.AddEventFlow(diagnosticPipeline);

            app.UseMvc();
        }

5. Now you can assume the logger factory will be constructor-injected into your controllers:

    [Route("api/[controller]")]
    public class ValuesController : Controller
    {
        private readonly ILogger<ValuesController> logger;

        public ValuesController(ILogger<ValuesController> logger)
        {
            this.logger = logger;
        }

        // GET api/values
        [HttpGet]
        public IEnumerable<string> Get()
        {
            this.logger.LogInformation("Hey, someone just called us!");
            return new string[] { "value1", "value2" };
        }

      // (rest of controller code is irrelevant)

Let me know if this is useful--hope it is--if yes, I am going to add this to the documentation.

[timomta]

Yes, I believe this is extremely useful. At this point, given all the great information you've given me, I need to sort out the way that best fits my needs. Thanks a lot for all your help!

[karolz-ms]

Just FYI I added the above example to the docs. Let me know if you have further questions.

[jfloodnet]

Hi @karolz-ms - is the advice for those using service fabric EventSources (or any for that matter), to use .Net Core in order for EventFlow to log exceptions as exceptions in AI?

[karolz-ms]

@jfloodnet Does issue 92 provide enough information about AI exception telemetry?

[karolz-ms]

This pull request #116 might help too. I am going to merge it soon

[ghost]

this.diagnosticPipeline is showing error to me

[karolz-ms]

@vi1996ash tell me more about the error and your program

[kurallasaitej]

Hi, @karolz-ms is it possible to give two different sources(Application Insights instrumentationKey)? Based on my requirement the telemetry should flow to the concerned account.

[karolz-ms]

@kurallasaitej I assume the question is "can I send data to two different Application Insights instances"? If that is the question, the answer is yes, just use two different (Application Insights) outputs with different instrumentation keys (or two different configuration files). Make sure you filter the data to each output appropriately.

If I misunderstood your question, please clarify.

[kurallasaitej]

Hi, @karolz-ms yes you are right and the issue got resolved thank u very much.

one quick question I added a new parameter in my eventFlowConfig.json I need to read that param in custom output SendEventsAsync method.
How can I do that?
(Here in the below attachment I used message template to archive it )
Example : Log.Information("{Information} LogSource {Path}", information, path);

Thanks in advance..

[karolz-ms]

@kurallasaitej I assume you want to add a configuration parameter for your custom output?

If so, the usual pattern is this:

1. The IPipelineItemFactory that creates your output gets an IConfiguration instance that holds the JSON fragment with configuration for your output. For example here is the relevant code for EventHub output
2. You pass this IConfiguration instance to the output constructor, which Binds it to the "configuration" object that stores all configuration parameters
3. The configuration object is a member variable of the output type, so it can be accessed from inside SendEventsAsync

Here is more information on how to create custom pipeline elements using configuration

Does this answer your question?

[cyberdecker]

@karolz-ms About your example provided to use Logger with EventFlow, in the part CreateServiceInstanceListeners, what to do when I have

protected override IEnumerable<ServiceReplicaListener> CreateServiceReplicaListeners()
        {
            return this.CreateServiceRemotingReplicaListeners();
        }

How to insert the logger here?

[karolz-ms]

First, to create the logger you can basically just create a LoggerFactory, call AddEventFlow() on it and then create a Logger using the factory. It can be all done as part of your service instance initialization

The first example here is probably most relevant. You just need to pass the EventFlow pipeline instance to the CoreService class somehow

[karolz-ms]

To make complex telemetry like Exception etc you need to attach some metadata to events https://github.com/Azure/diagnostics-eventflow#standard-metadata-types
The Logger input in EventFlow does this for you because it does not "know" which outputs you are using and what metadata they understand.

[karolz-ms]

Instead, you use the metadata filter for that https://github.com/Azure/diagnostics-eventflow#metadata

What you do is you "describe" which events should be treated as Exceptions, Events, Metric, Request etc (the "include" property) and the metadata filter will add this metadata to events that satisfy the include criteria.

It is a bit convoluted with the metadata filter etc. but this way you do not have to change how you LOG the data. You just log whatever information is useful, and then through metadata filter configuration, you specify what kind of telemetry should it produce in Application Inisghts
And it works for data coming from any source, not just your application