Azure @ Enterprise - How AppInsight SDK hook to WebClient and HttpClient classes?

This is continuation from previous post related to .Net SDK for Azure AppInsight and end to end cross process correlation. One of the open item in that post is about how the AppInsight SDK track the http dependency calls. Without we doing nothing, how the AppInsight can subscribe or hook to the http calls going out of the process?

Capture the outgoing http traffic from a .Net process in C#

Ideally capturing or intercepting the http traffic from a .Net process by a component inside that same process is independent of Azure AppInsight SDK. The SDK is just one use case of capturing the outgoing http traffic. There could be many other use cases such as security, modification etc...

Anyway lets start from the AppInsight source code and dig to the place where the hook happens. The source code of AppInsight SDK is available @ below Github repository

https://github.com/Microsoft/ApplicationInsights-dotnet-server

Navigating Source code of AppInsight SDK

In order to find out how the AppInsight SDK for .Net Server (ie legacy .Net as .Net Core is latest) hook to the http traffic, the entry point can be the ApplicationInsights.config file. If we search for the word 'Dependency' we can get the 'DependencyTrackingTelemetryModule'. Yes this resembles like the HttpModule which is powerful to do anything in the ASP.Net web world. First thing is to comment that section and see whether the Dependencies are getting logged or not to ensure that is the right thing we are looking at.

Anyway lets move ahead. Once we identify the source code of DependencyTrackingTelemetryModule class we can see below code snippet inside

---

#if NET45
// Net40 does not support framework event source

private HttpDesktopDiagnosticSourceListener httpDesktopDiagnosticSourceListener;
private FrameworkHttpEventListener httpEventListener;
private FrameworkSqlEventListener sqlEventListener;

#endif

---

https://github.com/Microsoft/ApplicationInsights-dotnet-server/blob/develop/Src/DependencyCollector/Shared/DependencyTrackingTelemetryModule.cs#L26

This tells us that the AppInsight SDK behaves differently in different .Net frameworks. Next thing it tells is that there are 2 http listening mechanisms. Lets start with the HttpDesktopDiagnosticsSourceListener class and come back if required for the other class. Once we locate that class we can see the constructor of that class as below

---

        internal HttpDesktopDiagnosticSourceListener(DesktopDiagnosticSourceHttpProcessing httpProcessing, ApplicationInsightsUrlFilter applicationInsightsUrlFilter)
        {
            this.httpDesktopProcessing = httpProcessing;
            this.subscribeHelper = new HttpDesktopDiagnosticSourceSubscriber(this, applicationInsightsUrlFilter);
            this.requestFetcherRequestEvent = new PropertyFetcher("Request");
            this.requestFetcherResponseEvent = new PropertyFetcher("Request");
            this.responseFetcher = new PropertyFetcher("Response");

            this.requestFetcherResponseExEvent = new PropertyFetcher("Request");
            this.responseExStatusFetcher = new PropertyFetcher("StatusCode");
            this.responseExHeadersFetcher = new PropertyFetcher("Headers");
        }

---

https://github.com/Microsoft/ApplicationInsights-dotnet-server/blob/develop/Src/DependencyCollector/Shared/Implementation/HttpDesktopDiagnosticSourceListener.cs#L29

Now focus to the subscriberHelper which is initialized with HttpDesktopDiagnosticSoruceSubscriber instance. What is in there?

---

internal HttpDesktopDiagnosticSourceSubscriber(
            HttpDesktopDiagnosticSourceListener parent,
            ApplicationInsightsUrlFilter applicationInsightsUrlFilter)
        {
            this.parent = parent;
            this.applicationInsightsUrlFilter = applicationInsightsUrlFilter;
            try
            {
                this.allListenersSubscription = DiagnosticListener.AllListeners.Subscribe(this);
            }

---

It is simple as subscribing to DiagnosticListener.AllListeners.

Redirect the analysis to DiagnosticsListener

Now we got a big clue. Lets understand what is DiagnosticListener and what is can listen to? Can it listen to outgoing http traffic?

The answer is yes from the .Net versions it support. Meaning in older .Net versions we cannot use this mechanism to capture http traffic. Below goes the sample code for hooking into http traffic using DiagnosticListener class.

https://github.com/joymon/dotnet-demos/tree/master/diagnostics/HttpCapture

More details

If anybody interested in following the topic below are some links
https://www.azurefromthetrenches.com/capturing-and-tracing-all-http-requests-in-c-and-net/

AppInsight SDK changed their model to capture
https://github.com/Microsoft/ApplicationInsights-dotnet-server/issues/548

Disclaimer

The above analysis is done by looking at the code and recreating sample using the code logic.  It might now work with other source repos where the name of the classes or functions may not resemble what it does internally. It is easy to debug the source to find out what is happening