As a consultant, many times I have to deal with custom requests which cannot be handled in TFS out-of-the-box. Many customizations end up to become valuable for other customers as well. Unfortunately I don’t always find the time to write about it and to work out a more generic solution which could help other people.
But recently I got an interesting question to intervene during the test run on the TFS Build Server because a complete test run took way too much time. The solution which was built on the server consisted of a big set of Unit Tests and a big set of Integration Tests. The Integration Tests required a deployment of a SQL Server database with some reference data. All tests were run at the same time and this caused builds to run for a long time, even if one of the Unit Tests failed at the beginning of the test run. The test run only completes (success/failure) after running ALL tests.
So, the goal was to quickly detect when a test fails (= fail early!) and to have the possibility to stop the build process immediately after the first test failure (=stop/fail build at the point one of the tests fails). The customer didn’t see any added value to run the remaining tests, knowing that already one test failed. Instead of waiting 30’ or longer for the full test results, the developers could already start fixing the first test failure and stopping the build would also decrease the load on the build server and test environment. We also agreed to only deploy the database when all Unit Tests succeeded.
How to separate the Integration Tests from the Unit Tests?
My sample solution above contains 2 separate projects/assemblies to host the Unit Tests and the Integration Tests. During the configuration of a Build Definition, you can easily define 2 separate test runs.
The first test run definition will only fetch the Unit Tests, while the second test run definition will look for the Integration Tests. Note that I specified a specific name for the test run definition. I will use this name later to filter the test run definitions. Creating multiple test run definitions is a flexible and easy way to split your big test run in multiple smaller test runs.
How to filter the test run definitions before the execution of a Test Run?
Time to customize the build process a bit so that first only the Unit Tests can be run before deciding to proceed with a database deployment and the run of the Integration Tests.
Instead of running the default VS Test Runner activity which would run all test run definitions (“AutomatedTests”), you need to filter for the Unit Tests. This can be done by modifying the TestSpec parameter for the RunAgileTestRunner activity. A where clause is added to the AutomatedTests value to search only for the “UnitTests” test run definition(s).
Result => only the Unit Tests will be executed by the VS Test Runner.
After this test run we can check the TestStatus of the build to stop the build or (in case of no test failures) to continue with a database deployment and the run of the Integration Tests.
In the ForEach activity after the database deployment operation I added a TestSpec filter for the AutomatedTests to only fetch the “IntegrationTests”.
The sequence in the ForEach activity will then call again the VS Test Runner and check for test failures to potentially stop the build in case of failure in the Integration Tests.
The more fine-grained you define your Integration Tests (= different test run definitions, making use of the test assembly file specification or the test case filter), the sooner you can inspect the test results and fail the build without running the other set(s) of Integration Tests.
Inspect Test Results during a Test Run (no filters)?
In the beginning, I started looking into options to inspect the test results during the ongoing one-and-only test run (no different test run definitions / no requirement for filters). I quickly stumbled on this MSDN page to learn more about the VSTest.Console.exe command-line options. By adding the switch /Logger:trx it’s possible to drop the test results into a Visual Studio Test Results File (.trx), but the problem is that the .trx file is only persisted to disk once the test run finishes. I didn’t find a way to get to the test results while the test run was still executing.
To stop the build in the customized build process template, I did throw an exception which is sufficient to stop the build process.
You can download the build process template which I used to write this blog entry. It’s far from complete and not fully tested, but it could help you to understand the filtering of the test run definitions.