Into Visual Studio Team System » Team Test

Running Coded UI Tests (from action recordings with MTLM) in Team Builds (TFS2010)

pietergheysens — Thu, 21 Jan 2010 22:38:46 +0000

With Visual Studio 2010 (Premium/Ultimate) we are able to create several types of automated tests. Automated tests will execute a sequence of test steps and determine whether the tests pass or fail according to expected results.

Coded UI Tests provide functional testing of the user interface and validation of user interface controls.

How to create Coded UI Tests? You could create them directly into Visual Studio, but for this blogpost I want to start from an action recording in Microsoft Test and Lab Manager (MTLM). An action recording is quite useful in manual tests that you need to run multiple times and for recycling common steps in different manual tests that contain shared steps.

I did create a simple test case with different test steps in MTLM to test some behavior on my website.

From MTLM I started a test run for this test case.

Before running the test, I do need to check the action recording to be sure to capture my actions for this test.

The Test Runner will give a detailed overview of the recorded actions. Afterwards you will be able to replay all these stored actions in the Test Runner.

After saving the results of this test run (all data is associated to my test case) it’s time to open Visual Studio 2010 and to create a Coded UI Test.

Instead of choosing the default option to record actions I did choose to use an existing action recording after which I need to retrieve the appropriate test case to link to the associated actions.

By clicking OK, Visual Studio will start generating code that will represent my actions that were recorded in Microsoft Test and Lab Manager. On top of that you are also able to add assertions on parts of the user interface in a separate Coded UI Test that you may reuse in other Coded UI Tests.

Now, let’s integrate this entire UI test (MyCodedUITest) into the automated build. I created a default new build defintion where I also enabled to run the automated tests.

To run unit tests that interact with the desktop during a Team Build, we need to modify the Build Service Host properties in the Team Foundation Administration Console to run the build service as an interactive process instead of running the build service as a Windows Service.

That’s about it. Make sure that the Build Service Host is running in the command line that will pop up after starting the BuildServiceHost. Queue the build and explore the results!

Done!

With this post I wanted to highlight the powerful integration of (automated) testing into the upcoming Visual Studio 2010 offering.

MSTest.exe exited with code -2146233082 during Team Build

pietergheysens — Thu, 17 Sep 2009 20:39:05 +0000

Recently I was faced with a failure in a Team Build caused by MSTest that crashed when firing up the Unit Tests.

MSBUILD : warning MSB6006: "MSTest.exe" exited with code -2146233082

Digging a bit deeper and running the Unit Tests manually with MSTest on the build server via the command line showed me a dialog box with the following notification:

MSTest.exe has encountered a problem and needs to close. We are sorry for the inconvenience.

Nothing more, nothing less! Hmm … Note that this behavior only surfaced after SP1 of Visual Studio 2008 and TFS 2008 was installed on the build server. Before, everything worked fine. On my local development pc (with SP1 of Visual Studio 2008) all Unit Tests run without any problems.

Let’s take a look at the eventviewer to get some extra information of what’s going wrong here …

Two interesting events were logged:

.NET Runtime version 2.0.50727.3053 – Fatal Execution Engine Error (7A035E00) (80131506)
Faulting application mstest.exe, version 9.0.30729.1, stamp 488f21a6, faulting module mscorwks.dll, version 2.0.50727.3053, stamp 4889dc18, debug? 0, fault address 0x001c5e00.

After searching for these errors on the web I came across some blog posts that mentioned the same Fatal Execution Engine Error but the scenario where this error occurred was totally different. It had something to do with Visual Studio disappearing unexpectedly but apparently there was a hotfix available on Microsoft Connect for this Fatal Execution Engine Error. So, with no other options at that moment I decided to go for the hotfix and see what happened.

The hotfix indeed did fix the problem, but I’m still puzzled what actually caused the problem. Strange was that certain Unit Test assemblies did run well on the build server while others even didn’t get started due to the above conditions …

[I did also log this error on the Testing forum of Visual Studio Team System]

Test Impact Analysis

pietergheysens — Thu, 05 Feb 2009 20:25:18 +0000

Test Impact Analysis is a new feature that will be part of Visual Studio Team System 2010 and it will enable developers to view which Tests are impacted by your latest code changes that are not checked in yet. I was struggling a bit to get it up-and-running but once you know the drill, it’s quite straightforward. The fact is that you need to have published test results by a Team Build on the previous stable situation.

How to set it up?

Create a new Team Build for your solution that also triggers your UnitTests and publishes the test results :
1. Select Solution to Build and select your tests to run
2. Enable Get Impacted Tests
3. Enable Publish the Test Impact Data
Enable Code Coverage for appropriate assemblies
Queue a new build of the newly created Team Build
Change some code that would impact a Unit Test
Open the Test Impact View Window
See the results …

This window reads as follows : 2 Unit Tests are impacted by my latest code changes that are not yet checked in. The AddTest Unit Test should be rerun because of changes made to the Add method.

I hope that this process will be simplified a bit more in the final release. If you don’t get it working immediately, check the Build log in search for Test Impact. Note that the Build log in TFS2010 has improved a lot! Maybe a topic for one of my next posts …