Tracking Response Exceptions via Custom Event Logging in Azure Insights

Welcome to the fifth and final part of our Microsoft Azure Insights Integration documentation series! This series has guided you through various aspects of integrating Azure Insights across different technologies and platforms. Belo is an overview of the topics covered in this series:

1. Getting Started with Microsoft Azure Insights Integration

2. A Step-by-Step Guide to Azure Insights Integration with .NET Core API

3. Streamlining Microsoft Azure Insights Integration with Angular

4. Log Request Payloads Using Custom Events in Microsoft Azure Insights

5. Tracking Response Exceptions via Custom Event Logging in Azure Insights (You’re here!)

In this fifth part, we will dive into tracking response exceptions using custom event logging in Microsoft Azure Insights to improve error tracking and debugging.

This guide explains how to track response exceptions via custom event logging in Microsoft Azure Application Insights. By doing so, we can monitor, capture, and analyze exception data, helping to diagnose errors and improve the reliability of backend services. This approach provides critical insights into application failures, enabling more comprehensive error tracking and troubleshooting, ultimately ensuring a smoother and more resilient application performance.

Prerequisites

• .NET Core Application: Backend API built using .NET Core (minimum version 3.1 or higher).

• Azure Subscription: Access to Microsoft Azure services.

• Azure Application Insights SDKs: For .NET.

• Basic knowledge of Application Insights: Familiarity with logging and telemetry.

Step 1: Set Up Application Insights in Azure

1. Create Application Insights Resource in Azure:

   • Log in to your Azure Portal.

   • Navigate to Application Insights and create a new resource.

   • Provide a name, select the appropriate subscription and resource group, and set your region (choose the region closest to your application's hosting).

   • Once the resource is created, go to the Overview page and copy the Instrumentation Key (this will be needed for both backend and frontend integration).

2. Configure Role-Based Access Control (RBAC):

   • Ensure that the appropriate permissions are granted for accessing the telemetry data. Add necessary users or service accounts with read or contributor access.

Step 2: Install Application Insights SDK for .NET Core

• Microsoft.ApplicationInsights.AspNetCore based on different versions of .NET:

• Use the following command in the terminal or in Visual Studio Package Manager Console:

  dotnet add package Microsoft.ApplicationInsights.AspNetCore

Step 3: Configure Application Insights in Startup.cs

Open the Startup.cs file and add the Application Insights service in the ConfigureServices method:

public void ConfigureServices(IServiceCollection services)
{
    var applicationInsightsEnabled = Configuration.GetValue<bool>("ApplicationInsights:TelemetryFeatureEnabled");
    if (applicationInsightsEnabled)
    {
        services.AddApplicationInsightsTelemetry(Configuration.GetSection("ApplicationInsights:ConnectionString"));
    }
}

Step 4: Adding New Configurations to appsettings.json

In your appsettings.json, add the Instrumentation Key and a feature flag to enable or disable the functionality:

{
  "ApplicationInsights": {
    "TelemetryFeatureEnabled": true,
    "ConnectionString": "YOUR_INSTRUMENTATION_KEY"
  }
}

This table provides a clear overview of the JSON configuration, explaining each setting and its purpose.

Step 5: Create azureinsightsconfiguration.json for Azure Insights

To better manage Azure Application Insights settings, it's useful to place the configurations in a dedicated file. In the root or configuration folder of the project, create a new file called azureinsightsconfiguration.json.

{
  "Exception": {
    "LoggingOption": 0,
    "LogRequests": [
      {
        "Path": "/api/v1/User/UserLogin",
        "Method": "POST"
      },
      {
        "Path": "/api/v1/Auth/AuthenticateWithJwtToken",
        "Method": "POST"
      }
    ],
    "DisabledDatabaseList": [
      "FR_Database"
    ]
  }
}

This table provides a clear overview of the JSON configuration, explaining each setting and its purpose.

Step 6: Configure azureinsightsconfiguration.json in Program.cs

• Open the Program.cs file and add the azureinsightsconfiguration.json in the CreateHostBuilder method:

public static IHostBuilder CreateHostBuilder(string[] args) =>
    Host.CreateDefaultBuilder(args)
        .UseSerilog()
        .ConfigureAppConfiguration(
            (context, config) => {
                config.AddJsonFile("azureinsightsconfiguration.json", optional: false, reloadOnChange: true);
            }
        )
        .ConfigureWebHostDefaults(webBuilder => {
            webBuilder.UseStartup<Startup>();
        });

• Set up logging, error handling, and dynamic configuration loading in Program.cs using configuration sources like appsettings.json and environment variables:

  public static void Main(string[] args)
  {
      try
      {
          Log.Information("Application is Starting.");
          CreateHostBuilder(args).Build().Run();
      }
      catch (Exception ex)
      {
          Log.Fatal(ex, "The Application is failed to start.");
      }
      finally
      {
          Log.CloseAndFlush();
      }
  }
  private static IConfiguration Configuration =>
      new ConfigurationBuilder()
          .SetBasePath(Directory.GetCurrentDirectory())
          .AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
          .AddEnvironmentVariables()
          .Build();

Step 07: Azure Insights Middleware Implementation - AzureInsightsMiddleware.cs

public class AzureInsightsMiddleware(
    RequestDelegate next,
    TelemetryClient telemetryClient,
    IConfiguration configuration
)
{
    public async Task Invoke(HttpContext context)
    {
        bool applicationInsightsEnabled = configuration.GetValue <bool> (
            "ApplicationInsights:TelemetryFeatureEnabled"
        );

        try
        {
            await next(context);
        }
        catch (Exception ex)
        {
            int exceptionLoggingOption = configuration.GetValue <int> ("Exception:LoggingOption");

            if (applicationInsightsEnabled && exceptionLoggingOption != -1)
            {
                PathString requestPath = context.Request.Path;
                string requestMethod = context.Request.Method;

                List < LogRequest > logRequests = configuration.GetSection("Exception:LogRequests").Get <List<LogRequest>> () ?? [];

                bool shouldLog = (logRequests.Any(
                    logRequest =>
                    requestMethod.Equals(
                        logRequest.Method,
                        StringComparison.OrdinalIgnoreCase
                    ) && requestPath.StartsWithSegments(logRequest.Path)
                ) && exceptionLoggingOption == 0) || exceptionLoggingOption == 1;

                if (shouldLog)
                {
                    var customMetaData = new Dictionary <string, string>()
                        {
                            {
                                "exception_message",
                                ex.Message
                            },
                            {
                                "exception_stack_trace",
                                ex.StackTrace
                            },
                            {
                                "exception_inner_message",
                                ex?.InnerException?.Message ?? string.Empty
                            },
                            {
                                "exception_inner_stack_trace",
                                ex?.InnerException?.StackTrace ?? string.Empty
                            },
                            {
                                "exception",
                                ex.ToString()
                            },
                            {
                                "inner_exception",
                                ex?.InnerException?.ToString() ?? string.Empty
                            },
                        };

                    if (context.User.Claims.Any())
                    {
                        string userDatabase = context.User.Claims.FirstOrDefault(c => c.Type == "Database")?.Value ?? string.Empty;
                        customMetaData.Add("organisation", userDatabase);
                    }
                    else
                    {
                        customMetaData.Add("organisation", "not-specified");
                    }

                    telemetryClient.TrackEvent($ "Response Exception", customMetaData);
                }
            }
        }
    }
}

Step 08: Register the Middleware in Startup.cs

Open the Startup.cs file and add the AzureInsightsMiddleware in the Configure method:

public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
    app.UseMiddleware<AzureInsightsMiddleware>();
}

public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
    app.UseAuthentication();
    app.UseAuthorization();
    app.UseMiddleware<AzureInsightsMiddleware>();
}

Step 09: Visualize the Custom Event in Azure Insights

Once the response exception are logged as custom events, follow these steps to visualize them in Azure Insights:

1. Open Azure Portal:

   • Go to Azure Portal and navigate to your Application Insights resource.

2. Navigate to Logs (Analytics):

   • On the Application Insights overview page, select Logs (formerly known as Analytics).

3. Query Custom Events:

   • Use Kusto Query Language (KQL) to query the custom events. For example, to view the logged response exceptions:

     customEvents
     | where name == "Response Exception"
     | project 
         timestamp,
         name,
         operation_Name,
         customDimensions.organisation,
         customDimensions.exception_message,
         customDimensions.exception_stack_trace,
         customDimensions.exception_inner_message,
         customDimensions.exception_inner_stack_trace
     | order by timestamp desc

4. Analyze the Data:
   The query results will display logged response exceptions, stack traces, and any additional metadata (such as organization). Use the insights to analyze incoming data, diagnose issues, or optimize performance.
   

   

5. Set Up Dashboards and Alerts:

   • You can save these queries as part of a dashboard or set up alerts based on specific conditions (e.g. exception, inner exception and stack trace) to proactively monitor important events.

customEvents
| where name == "Response Exception"
| where customDimensions.organisation contains "YOUR_DATABASE_NAME" 
| project 
    timestamp,
    name,
    operation_Name,
    customDimensions.organisation,
    customDimensions.exception_message,
    customDimensions.exception_stack_trace,
    customDimensions.exception_inner_message,
    customDimensions.exception_inner_stack_trace
| order by timestamp desc

This step helps you to visualize and analyze the custom events captured by Azure Application Insights, providing deeper insight into your application’s behavior and performance.

Conclusion

In this guide, we've explored how to track response exceptions using custom event logging in Microsoft Azure Application Insights. By implementing this approach, you can capture and analyze exception data, providing valuable insights into errors and failures in your backend services. This method allows for detailed tracking of exceptions, user-related context, and application behavior, helping to diagnose issues and improve reliability. With these insights, you can enhance monitoring, streamline error handling, and ensure your application operates smoothly, enabling proactive improvements where necessary.