Introduction
The last post in this blog was about Power BI to analyze IIS logs. It used Log Parser and PowerShell to prepare the W3C IIS .log files into CSV. That conversion is needed as Power BI don't understand the W3C log file format produced by IIS. Though it works for small to medium load, it takes more time to prepare larger files. Let see another way to prepare IIS logs to CSV for log analysis.
Limits of log parser
One of the limit is it cannot output a default value if the columns are not available in the source log file. At the other side Power BI needs all the columns in CSV to load data. This mismatch happens frequently as the default setting of IIS web server doesn't include the cs-bytes and sc-bytes columns and the PowerBI needs that to analyze the payload.
The solution was to load the CSV produced by Log Parser into PowerShell and add missing columns and output the same into CSV file. PowerShell loading the CSV and adding columns is time consuming out of the box. We can tune the PowerShell script to optimize CSV handling. But still, there will be 2 tools IISLogParser and PowerShell.
Enter C#
What if we use C# to convert the IIS Logs to the desired CSV with default values? It is not advisable to rewrite the IIS log parsing logic in C# and maintain the same. It is better to use a library from nuget. Fortunately, there is one library called IISLogParser. We can use the same to parse the logs and use CSV libraries such as ServiceStack to output into CSV.
It is easy to set up such utility in C#. Source code can be found at below GitHub location.
Please note that this uses .Net Core 2.1 target. Ideally, there should be no issues in adding full .Net framework version as target as those libraries support multi-targeting. Add an issue in GitHub repo in case of issue better with PR. Comments in this post also welcome.
No comments:
Post a Comment