Log Request Payloads Using Custom Events in Microsoft Azure Insights

Welcome to the fourth part of our Microsoft Azure Insights Integration documentation series! This series will guide you through various aspects of integrating Azure Insights across different technologies and platforms. Below is an overview of the topics covered in this series:

1. Getting Started with Microsoft Azure Insights Integration

2. A Step-by-Step Guide to Azure Insights Integration with .NET Core API

3. Streamlining Microsoft Azure Insights Integration with Angular

4. Log Request Payloads Using Custom Events in Microsoft Azure Insights (You’re here!)

5. Tracking Response Exceptions via Custom Event Logging in Azure Insights

In this fourth part, we will cover how to log request payloads using custom events in Microsoft Azure Insights to enhance monitoring and troubleshooting.

This guide explains how to log request payloads using custom events in Microsoft Azure Application Insights. By doing so, we can monitor, capture, and analyze incoming request data, helping to diagnose issues and optimize the performance of backend services. This approach provides critical insights into the data flowing through the application, ensuring more comprehensive monitoring and troubleshooting capabilities.

Prerequisites

• .NET Core Application: Backend API built using .NET Core (minimum version 3.1 or higher).

• Azure Subscription: Access to Microsoft Azure services.

• Azure Application Insights SDKs: For .NET.

• Basic knowledge of Application Insights: Familiarity with logging and telemetry.

Step 1: Set Up Application Insights in Azure

1. Create Application Insights Resource in Azure:

   • Log in to your Azure Portal.

   • Navigate to Application Insights and create a new resource.

   • Provide a name, select the appropriate subscription and resource group, and set your region (choose the region closest to your application's hosting).

   • Once the resource is created, go to the Overview page and copy the Instrumentation Key (this will be needed for both backend and frontend integration).

2. Configure Role-Based Access Control (RBAC):

   • Ensure that the appropriate permissions are granted for accessing the telemetry data. Add necessary users or service accounts with read or contributor access.

Step 2: Install Application Insights SDK for .NET Core

• Microsoft.ApplicationInsights.AspNetCore based on different versions of .NET:

• Use the following command in the terminal or in Visual Studio Package Manager Console:

  dotnet add package Microsoft.ApplicationInsights.AspNetCore

Step 3: Configure Application Insights in Startup.cs

Open the Startup.cs file and add the Application Insights service in the ConfigureServices method:

public void ConfigureServices(IServiceCollection services)
{
    var applicationInsightsEnabled = Configuration.GetValue<bool>("ApplicationInsights:TelemetryFeatureEnabled");
    if (applicationInsightsEnabled)
    {
        services.AddApplicationInsightsTelemetry(Configuration.GetSection("ApplicationInsights:ConnectionString"));
    }
}

Step 4: Adding New Configurations to appsettings.json

In your appsettings.json, add the Instrumentation Key and a feature flag to enable or disable the functionality:

{
  "ApplicationInsights": {
    "TelemetryFeatureEnabled": true,
    "ConnectionString": "YOUR_INSTRUMENTATION_KEY"
  }
}

This table provides a clear overview of the JSON configuration, explaining each setting and its purpose.

Step 5: Create azureinsightsconfiguration.json for Azure Insights

To better manage Azure Application Insights settings, it's useful to place the configurations in a dedicated file. In the root or configuration folder of the project, create a new file called azureinsightsconfiguration.json.

{
  "PayLoad": {
    "MinimumPayloadSize": 5, 
    "LoggingOption": 0,
    "LogRequests": [
      {
        "Path": "/api/v1/User/UserLogin",
        "Method": "POST"
      },
      {
        "Path": "/api/v1/Auth/AuthenticateWithJwtToken",
        "Method": "POST"
      }
    ],
    "DisabledDatabaseList": [
      "FR_Database"
    ]
  }
}

This table provides a clear overview of the JSON configuration, explaining each setting and its purpose.

Step 6: Configure azureinsightsconfiguration.json in Program.cs

• Open the Program.cs file and add the azureinsightsconfiguration.json in the CreateHostBuilder method:

public static IHostBuilder CreateHostBuilder(string[] args) =>
    Host.CreateDefaultBuilder(args)
        .UseSerilog()
        .ConfigureAppConfiguration(
            (context, config) => {
                config.AddJsonFile("azureinsightsconfiguration.json", optional: false, reloadOnChange: true);
            }
        )
        .ConfigureWebHostDefaults(webBuilder => {
            webBuilder.UseStartup<Startup>();
        });

• Set up logging, error handling, and dynamic configuration loading in Program.cs using configuration sources like appsettings.json and environment variables:

  public static void Main(string[] args)
  {
      try
      {
          Log.Information("Application is Starting.");
          CreateHostBuilder(args).Build().Run();
      }
      catch (Exception ex)
      {
          Log.Fatal(ex, "The Application is failed to start.");
      }
      finally
      {
          Log.CloseAndFlush();
      }
  }
  
  private static IConfiguration Configuration =>
      new ConfigurationBuilder()
          .SetBasePath(Directory.GetCurrentDirectory())
          .AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
          .AddEnvironmentVariables()
          .Build();

Step 07: Azure Insights Middleware Implementation - AzureInsightsMiddleware.cs

    public class AzureInsightsMiddleware(
        RequestDelegate next,
        TelemetryClient telemetryClient,
        IConfiguration configuration
    )
    {
        public async Task Invoke(HttpContext context)
        {
            bool applicationInsightsEnabled = false;

            try
            {
                applicationInsightsEnabled = configuration.GetValue<bool> (
                    "ApplicationInsights:TelemetryFeatureEnabled"
                );
                int payLoadLoggingOption = configuration.GetValue<int>("PayLoad:LoggingOption");

                if (applicationInsightsEnabled && payLoadLoggingOption != -1)
                {
                    PathString requestPath = context.Request.Path;
                    string requestMethod = context.Request.Method;

                    List <LogRequest> logRequests = configuration.GetSection("PayLoad:LogRequests").Get<List<LogRequest>>() ?? [];

                    bool shouldLog = (logRequests.Any(
                        logRequest =>
                        requestMethod.Equals(
                            logRequest.Method,
                            StringComparison.OrdinalIgnoreCase
                        ) && requestPath.StartsWithSegments(logRequest.Path)
                    ) && payLoadLoggingOption == 0) || payLoadLoggingOption == 1;

                    if (shouldLog)
                    {
                        string[] logDisabledDatabases = configuration.GetSection("PayLoad:DisabledDatabaseList").Get<string[]>() ?? [];
                        int minimumPayloadSize = configuration.GetSection("PayLoad:MinimumSize").Get<int?>() ?? 5;

                        context.Request.EnableBuffering();
                        string bodyAsText = string.Empty;

                        if (context.Request.ContentLength == null || context.Request.ContentLength > minimumPayloadSize * 1024 * 1024)
                        {
                            bodyAsText = "Request body is too large to process.";
                        }
                        else
                        {
                            using(StreamReader reader = new(context.Request.Body, Encoding.UTF8, detectEncodingFromByteOrderMarks: false, leaveOpen: true))
                            {
                                bodyAsText = await reader.ReadToEndAsync();
                                context.Request.Body.Position = 0;
                            }
                            context.Request.Body.Seek(0, SeekOrigin.Begin);
                        }

                        Dictionary<string, string> customMetaData = new()
                        {
                            {
                                "payload",
                                bodyAsText
                            },
                            {
                                "content_type",
                                context.Request.ContentType
                            },
                        };

                        if (context.User.Claims.Any())
                        {
                            string userDatabase = context.User.Claims.FirstOrDefault(c => c.Type == "Database")?.Value ?? string.Empty;

                            if (!logDisabledDatabases.Contains(userDatabase))
                            {
                                customMetaData.Add("organization", userDatabase);
                            }
                        }
                        else
                        {
                            customMetaData.Add("organization", "not-specified");
                        }

                        telemetryClient.TrackEvent($ "Request PayLoad", customMetaData);
                    }
                }
            }
            catch (Exception ex)
            {
                telemetryClient.TrackException(ex);
            }

            await next(context);
        }
    }

Step 08: Register the Middleware in Startup.cs

Open the Startup.cs file and add the AzureInsightsMiddleware in the Configure method:

public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
    app.UseMiddleware<AzureInsightsMiddleware>();
}

public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
    app.UseAuthentication();
    app.UseAuthorization();
    app.UseMiddleware<AzureInsightsMiddleware>();
}

Step 09: Visualize the Custom Event in Azure Insights

Once the request payloads are logged as custom events, follow these steps to visualize them in Azure Insights:

1. Open Azure Portal:

   • Go to Azure Portal and navigate to your Application Insights resource.

2. Navigate to Logs (Analytics):

   • On the Application Insights overview page, select Logs (formerly known as Analytics).

3. Query Custom Events:

   • Use Kusto Query Language (KQL) to query the custom events. For example, to view the logged request payloads:

     customEvents
     | where name == "Request PayLoad"
     | project 
         timestamp, 
         customDimensions.payload, 
         customDimensions.organization, 
         customDimensions.content_type
     | order by timestamp desc

4. Analyze the Data:
   The query results will display logged request payloads, content types, and any additional metadata (such as organization). Use the insights to analyze incoming data, diagnose issues, or optimize performance.
   

   

5. Set Up Dashboards and Alerts:

   • You can save these queries as part of a dashboard or set up alerts based on specific conditions (e.g., payload size, request content type) to proactively monitor important events.

customEvents
| where name == "Request PayLoad"
| where customDimensions.organisation contains "YOUR_DATABASE_NAME" 
| project 
    timestamp, 
    customDimensions.payload, 
    customDimensions.organization, 
    customDimensions.content_type
| order by timestamp desc

This step helps you to visualize and analyze the custom events captured by Azure Application Insights, providing deeper insight into your application’s behavior and performance.

Conclusion

In this guide, we've explored how to log request payloads using custom events in Microsoft Azure Application Insights. By implementing this approach, you can capture and analyze incoming request data, offering valuable insights into the operation and performance of your backend services. This method allows for detailed tracking of request content, user actions, and application behavior, helping to diagnose issues and optimize performance. With these insights, you can enhance monitoring, streamline troubleshooting, and ensure that your application runs efficiently, making proactive improvements as needed.