/
Log Request Payloads Using Custom Events in Microsoft Azure Insights

Log Request Payloads Using Custom Events in Microsoft Azure Insights

Welcome to the fourth part of our Microsoft Azure Insights Integration documentation series! This series will guide you through various aspects of integrating Azure Insights across different technologies and platforms. Below is an overview of the topics covered in this series:

  1. Getting Started with Microsoft Azure Insights Integration

  2. A Step-by-Step Guide to Azure Insights Integration with .NET Core API

  3. Streamlining Microsoft Azure Insights Integration with Angular

  4. Log Request Payloads Using Custom Events in Microsoft Azure Insights (You’re here!)

  5. Tracking Response Exceptions via Custom Event Logging in Azure Insights

In this fourth part, we will cover how to log request payloads using custom events in Microsoft Azure Insights to enhance monitoring and troubleshooting.


This guide explains how to log request payloads using custom events in Microsoft Azure Application Insights. By doing so, we can monitor, capture, and analyze incoming request data, helping to diagnose issues and optimize the performance of backend services. This approach provides critical insights into the data flowing through the application, ensuring more comprehensive monitoring and troubleshooting capabilities.

Prerequisites

  • .NET Core Application: Backend API built using .NET Core (minimum version 3.1 or higher).

  • Azure Subscription: Access to Microsoft Azure services.

  • Azure Application Insights SDKs: For .NET.

  • Basic knowledge of Application Insights: Familiarity with logging and telemetry.


Step 1: Set Up Application Insights in Azure

  1. Create Application Insights Resource in Azure:

    • Log in to your Azure Portal.

    • Navigate to Application Insights and create a new resource.

    • Provide a name, select the appropriate subscription and resource group, and set your region (choose the region closest to your application's hosting).

    • Once the resource is created, go to the Overview page and copy the Instrumentation Key (this will be needed for both backend and frontend integration).

  2. Configure Role-Based Access Control (RBAC):

    • Ensure that the appropriate permissions are granted for accessing the telemetry data. Add necessary users or service accounts with read or contributor access.

image-20241001-080213.png

Step 2: Install Application Insights SDK for .NET Core

  • Microsoft.ApplicationInsights.AspNetCore based on different versions of .NET:

.NET Version

Microsoft.ApplicationInsights.AspNetCore NuGet Version

.NET Version

Microsoft.ApplicationInsights.AspNetCore NuGet Version

.NET Core 2.1

2.8.1 - 2.14.0

.NET Core 3.1

2.14.0 and later

.NET 5.0

2.16.0 and later

.NET 6.0

2.18.0 and later

.NET 7.0

2.20.0 and later

.NET 8.0

2.21.0 and later

  • Use the following command in the terminal or in Visual Studio Package Manager Console:

    dotnet add package Microsoft.ApplicationInsights.AspNetCore

Step 3: Configure Application Insights in Startup.cs

Open the Startup.cs file and add the Application Insights service in the ConfigureServices method:

public void ConfigureServices(IServiceCollection services) { var applicationInsightsEnabled = Configuration.GetValue<bool>("ApplicationInsights:TelemetryFeatureEnabled"); if (applicationInsightsEnabled) { services.AddApplicationInsightsTelemetry(Configuration.GetSection("ApplicationInsights:ConnectionString")); } }

Step 4: Adding New Configurations to appsettings.json

In your appsettings.json, add the Instrumentation Key and a feature flag to enable or disable the functionality:

{ "ApplicationInsights": { "TelemetryFeatureEnabled": true, "ConnectionString": "YOUR_INSTRUMENTATION_KEY" } }

Section

Key/Setting

Default Value

Description

Section

Key/Setting

Default Value

Description

ApplicationInsights

 

 

Configuration for the Application Insights service integration.

 

TelemetryFeatureEnabled

false

Enables telemetry data collection such as performance metrics, exceptions, etc.

 

ConnectionString

"YOUR_INSTRUMENTATION_KEY"

Placeholder for the actual Application Insights Instrumentation Key or Connection String.

This table provides a clear overview of the JSON configuration, explaining each setting and its purpose.


Step 5: Create azureinsightsconfiguration.json for Azure Insights

To better manage Azure Application Insights settings, it's useful to place the configurations in a dedicated file. In the root or configuration folder of the project, create a new file called azureinsightsconfiguration.json.

Configuration Name

Default Value

Definition

Configuration Name

Default Value

Definition

PayLoad

-

Main configuration object.

MinimumPayloadSize

5 (in MB)

Minimum payload size for requests in megabytes.

LoggingOption

0

Specifies logging behavior:
-1: No databases are logged.
0: All except disabled databases & filtered log requests are logged.
1: All except disabled databases & all log requests are logged.

LogRequests

-

Specifies API requests that should be logged. (Inner Objects of Requests)

Path

-

The path of the API request to be logged.

Method

-

HTTP method for the API request.

DisabledDatabaseList

-

Lists the databases that are disabled.

Database

-

Comma separated names of the disabled database.

This table provides a clear overview of the JSON configuration, explaining each setting and its purpose.


Step 6: Configure azureinsightsconfiguration.json in Program.cs

  • Open the Program.cs file and add the azureinsightsconfiguration.json in the CreateHostBuilder method:

  • Set up logging, error handling, and dynamic configuration loading in Program.cs using configuration sources like appsettings.json and environment variables:


Step 07: Azure Insights Middleware Implementation - AzureInsightsMiddleware.cs

Explanation: This code defines a middleware class, AzureInsightsMiddleware, for logging request payloads to Azure Application Insights in a .NET Core application.

  1. Configuration Check:

    • It checks if telemetry logging is enabled by reading the TelemetryFeatureEnabled setting from the configuration. If disabled, no logging occurs.

  2. Payload Logging Conditions:

    • Based on configuration settings (PayLoad:LoggingOption and PayLoad:LogRequests), it determines whether the request payload should be logged. You can configure specific request paths and methods to track or log all requests.

  3. Buffering and Payload Logging:

    • If logging is enabled and the payload is not too large, the middleware reads the request body (using buffering) and logs it along with metadata like content type and user organization.

  4. Telemetry Logging:

    • The payload and additional data are logged to Application Insights as a custom event, allowing monitoring of specific requests and tracking user actions.

  5. Error Handling:

    • If any errors occur, they are caught and logged as exceptions to prevent the middleware from breaking the request flow.

This middleware helps monitor, log, and analyze request payloads in a structured way, providing valuable insights for debugging and performance monitoring.


Step 08: Register the Middleware in Startup.cs

Open the Startup.cs file and add the AzureInsightsMiddleware in the Configure method:

Ensure that the AzureInsightsMiddleware is registered after app.UseAuthentication() and app.UseAuthorization() in the Configure method to correctly log requests after authentication and authorization have been applied. This ensures that any telemetry data related to user identity or roles is captured, providing more accurate insights into authenticated requests.


Step 09: Visualize the Custom Event in Azure Insights

Once the request payloads are logged as custom events, follow these steps to visualize them in Azure Insights:

  1. Open Azure Portal:

    • Go to Azure Portal and navigate to your Application Insights resource.

  2. Navigate to Logs (Analytics):

    • On the Application Insights overview page, select Logs (formerly known as Analytics).

  3. Query Custom Events:

    • Use Kusto Query Language (KQL) to query the custom events. For example, to view the logged request payloads:

  4. Analyze the Data:
    The query results will display logged request payloads, content types, and any additional metadata (such as organization). Use the insights to analyze incoming data, diagnose issues, or optimize performance.

    image-20241009-125157.png
  5. Set Up Dashboards and Alerts:

    • You can save these queries as part of a dashboard or set up alerts based on specific conditions (e.g., payload size, request content type) to proactively monitor important events.

To query request payloads based on a specific database, use the following filter in your query: customDimensions.organization == 'YourDatabaseName'. This will help you isolate and analyze logs related to the specified database, providing more targeted insights.

This step helps you to visualize and analyze the custom events captured by Azure Application Insights, providing deeper insight into your application’s behavior and performance.



Conclusion

In this guide, we've explored how to log request payloads using custom events in Microsoft Azure Application Insights. By implementing this approach, you can capture and analyze incoming request data, offering valuable insights into the operation and performance of your backend services. This method allows for detailed tracking of request content, user actions, and application behavior, helping to diagnose issues and optimize performance. With these insights, you can enhance monitoring, streamline troubleshooting, and ensure that your application runs efficiently, making proactive improvements as needed.

Related content

A Step-by-Step Guide to Azure Insights Integration with .NET Core API
A Step-by-Step Guide to Azure Insights Integration with .NET Core API
More like this
Streamlining Microsoft Azure Insights Integration with Angular
Streamlining Microsoft Azure Insights Integration with Angular
More like this
Getting Started with Microsoft Azure Insights Integration
Getting Started with Microsoft Azure Insights Integration
Read with this
Tracking Response Exceptions via Custom Event Logging in Azure Insights
Tracking Response Exceptions via Custom Event Logging in Azure Insights
More like this
Feature Release Note | Camms.Risk Incident Management, December 2020
Feature Release Note | Camms.Risk Incident Management, December 2020
More like this
Feature Release Note | Camms.Insights, January 2021
Feature Release Note | Camms.Insights, January 2021
More like this