Visual Studio Online / Azure DevOps load testing - On-premise agent service is not starting

This is a simple troubleshooting post related to VisualStudio Online or Azure DevOps load testing.

The load testing needs agent programs to be installed on multiple machines to generate load. We can use agents provided by Microsoft as well if our application is straight forward and accessible from Microsoft hosted agents.

If we are in Enterprise or in restricted environments, we may have application hosted internally and only accessible via domain joined machines or via other constraints. Then we need to run the agent program on premise.

Problem

The agent program is a windows services which we can see in services.msc console. Sometimes the agent service gets stopped and below error will show up when we try to restart.

---------------------------
Services
---------------------------
The VSTSLoadAgentService service on Local Computer started and then stopped. Some services stop automatically if they are not in use by other services or programs.
---------------------------
OK 
---------------------------

Solution

Since this Azure DevOps load testing tool is deprecated now and ending its life next year around this time, lets not go into the details why its not working. Lets see a hack.

As standard Microsoft solution which is as famous as machine restart, lets uninstall and install the agent program again and it will work.

Since the installation is by running below PowerShell scripts, it doesn't seems polluting registry and all.

Uninstalling agent

Uninstall command needs the agent name. If the name if unknown, first we need to get agents list. Example is as follows.

---

.ManageVSTSCloudLoadAgent.ps1 -TeamServicesAccountName https://<organization>.visualstudio.com -PATToken <PATToken> -AgentGroupName <group name> -GetAgents

---

to uninstall an agent follow the below command.

---

.ManageVSTSCloudLoadAgent.ps1 -TeamServicesAccountName https://<organization>.visualstudio.com -PATToken <PATToken> -DeleteAgent -AgentGroupName <group name> -AgentName <agent name>

---

Now install the agent again.

---

.ManageVSTSCloudLoadAgent.ps1 -TeamServicesAccountName https://<organization>.visualstudio.com -PATToken <PATToken> -ConfigureAgent -AgentGroupName <group name>

---

The ManageVSTSCloudLoadAgent.ps1 file can be downloaded from this location. The commands above has some parameters enclosed in '<' & '>' as placeholders. We need to use actual values there.

Azure @ Enterprise - Downloading all the VSOnline / Azure DevOps load test .har files together

Background

After performing a load test via Azure Test Plans or Visual Studio Online, it produce the http archive  files with .har extension. Those .har files are available in the Azure DevOps portal as part of test results and we can download to analyze the nature of load test. But the problem is that we need to download files one by one. Its tedious to do via portal.

Solution

As everyone knows, the Azure is built around ReSTful Web APIs. The Portal UI itself is a wrapper around those APIs. Lets see how can we download those files easily using the same API layer.

Figuring out Url

Below Url gives the details of WebAPI which we can invoke to interact with the load tests.

https://docs.microsoft.com/en-us/rest/api/azure/devops/clt/results/get?view=azure-devops-rest-5.0

In short, the url format is below

https://<organization name>.vsclt.visualstudio.com/_apis/clt/testRuns/<Run GUID>/results?api-version=5.0

The GUID of test run can be obtained from the portal URL, when we are in the individual test run page. If we try with the integer sequence number what we are seeing the test runs list, it will not work.

Security

The MSDN documentation says Personal Access Token and type Basic.  That means, when we put the PAT, we have to use base64 encoding. The best way to make sure its working, is via Postman. Below are the steps.

• In Postman, make sure the http method is GET and enter the intended URL by replacing the organization name and Run GUID.
• Go to Authorization tab and select TYPE as "Basic Auth".
• Enter any user name or leave it empty and inside the password text box, put the Personal Access Token.
• Press on 'Send' buttong to send the API request to Azure DevOps, and we should get data in JSON format.

What is the role of storage account and how to access files

The WebAPI call /testRuns/<GUID>/results gives a JSON as explained above not the .har files which we need. Instead of .har files or har data, it has a link to Azure Blob Storage. Notice that the link has some query string which have the SAS token details. 

Azure developers already knows how to download files from an Azure blob storage, if they get a URL with SAS token suffixed to it.

But for the benefit of testers and automation guys the below link has been given which explains, how to download files from Azure blob storage using a tool called Azure Storage Explorer. It is free as of now.

https://discourse.xcalar.com/t/white-check-mark-how-to-use-sas-uri-with-azure-file-explorer/336/2

Once the connection is established in Azure Storage Explorer to storage account using the SAS token, we can navigate to the correct GUID folder of test run. Then to /TestResult/Diagnostics folder to download .har files.

Another way is to use another command line tool called AzCopy.exehttps://docs.microsoft.com/en-us/azure/storage/common/storage-use-azcopy which download the files from Azure Storage Account in higher speeds. AzCopy.exe needs the SAS token to be passed into it via command line switches. Since that is not in the scope of this post, letting it to learners.

Recording VSTS webtest of windows application which uses https web services

Whenever we develop a web site or web application, it is mandatory that we should do load testing before production. It will give us an idea about how many users and requests it can handle per second in a given server environment. In theory its easy to say we should do the load testing. But in practice, very less people knows, how it is being worked out and what are the software tools used to do load testing.

One of the load testing tool is Visual Studio itself ! But before going into load testing in Visual studio, let us see what a load testing software in general should provide and how a load testing is carried out.

Features of load testing software

• Recording
• Ability to record the different use cases of the application. Mainly uses script for storing purpose.
• Ability to add parameters / tokens into the recorded use cases / script for handling virtual users and passing other runtime values.
• Play back
• Ability to run the recorded use cases / scripts by replacing the tokens with run time values
• Ability to run simultaneously from multiple machines or agents.
• Controller agent model to collect data from the nodes which are executing the scripts.
• Ability to create virtual users in the controller agent environment.

Steps to do load testing

1. Record the use cases in a format which can be playable
2. Adding parameters (optional)
1. Add tokens or place holders to the recorded script so that it can be replaced by actual values at run time
2. Write the token / parameter replacement logic.
3. Define web load test cases. eg: 25 virtual users
4. Setup the agents.
5. Run the recorded load test cases and collect the results.

Web performance load testing using Visual Studio

This feature is available in high end versions (Ultimate) of Visual Studio. If your Visual Studio supports load testing, you can see the webtest template and load test template. If your application is a web site its very easy to record the scripts. This is because the Visual Studio has an in built mechanism to record the http(s) traffic from internet explorer. If your internet explorer is in record mode, you can easily perform use cases and record the action. But if you are developing a windows application which uses http(s), it is little difficult to record because Visual Studio cannot intercept your windows desktop application to capture the http(s) traffic. So what to do? In fact that is the aim of this post.

Recording visual studio web test in desktop world

There is a really really useful tool called fiddler. It must be familiar to the web developers but not for windows developers. This tool can act as proxy which sits between any application and the network to capture all the http(s) requests and responses. Now a days most of its duties can be done by the F12 browser tools. But in the desktop environment, fiddler still has importance. Below is the sequence to create web test of windows applications using fiddler.

1. Run Fiddler
1. Setup support for https if required
2. Run the application
3. Execute a use case
4. Goto fiddler and select File -> Export Sessions -> All Sessions
5. Select Visual Studio WebTest
6. Add the exported web test to Visual Studio project.
7. Add tokens / parameters to the web test.
8. Clear the fiddler session
9. Repeat the steps 3-8 for all the use cases in your application which you want to load test.

http://fiddler2.com/documentation/Save-And-Load-Traffic/Tasks/VSWebTest

http://blogs.msdn.com/b/buckh/archive/2007/04/18/vsts-testing-using-fiddler-to-record-vsts-web-tests.aspx