Running Coded UI Tests (from action recordings with MTLM) in Team Builds (TFS2010)

January 21, 2010

With Visual Studio 2010 (Premium/Ultimate) we are able to create several types of automated tests. Automated tests will execute a sequence of test steps and determine whether the tests pass or fail according to expected results.

Coded UI Tests provide functional testing of the user interface and validation of user interface controls.

How to create Coded UI Tests? You could create them directly into Visual Studio, but for this blogpost I want to start from an action recording in Microsoft Test and Lab Manager (MTLM). An action recording is quite useful in manual tests that you need to run multiple times and for recycling common steps in different manual tests that contain shared steps.

I did create a simple test case with different test steps in MTLM to test some behavior on my website.

TestCaseDefinition

From MTLM I started a test run for this test case.

TestSuiteOverview

Before running the test, I do need to check the action recording to be sure to capture my actions for this test.

CreateActionRecording

The Test Runner will give a detailed overview of the recorded actions. Afterwards you will be able to replay all these stored actions in the Test Runner.

ActionRecordings 

After saving the results of this test run (all data is associated to my test case) it’s time to open Visual Studio 2010 and to create a Coded UI Test.

TestCaseAttachments

CodedUITestStart

Instead of choosing the default option to record actions I did choose to use an existing action recording after which I need to retrieve the appropriate test case to link to the associated actions.

ActionRecordingPicker

By clicking OK, Visual Studio will start generating code that will represent my actions that were recorded in Microsoft Test and Lab Manager. On top of that you are also able to add assertions on parts of the user interface in a separate Coded UI Test that you may reuse in other Coded UI Tests.

CodedUITestAssertions

Now, let’s integrate this entire UI test (MyCodedUITest) into the automated build. I created a default new build defintion where I also enabled to run the automated tests.

BuildDefinition

To run unit tests that interact with the desktop during a Team Build, we need to modify the Build Service Host properties in the Team Foundation Administration Console to run the build service as an interactive process instead of running the build service as a Windows Service.

BuildServiceHost

That’s about it. Make sure that the Build Service Host is running in the command line that will pop up after starting the BuildServiceHost. Queue the build and explore the results!

TestResults

Done!

With this post I wanted to highlight the powerful integration of (automated) testing into the upcoming Visual Studio 2010 offering.


MSTest.exe exited with code -2146233082 during Team Build

September 17, 2009

Recently I was faced with a failure in a Team Build caused by MSTest that crashed when firing up the Unit Tests.

MSBUILD : warning MSB6006: "MSTest.exe" exited with code -2146233082

Digging a bit deeper and running the Unit Tests manually with MSTest on the build server via the command line showed me a dialog box with the following notification:

MSTest.exe has encountered a problem and needs to close. We are sorry for the inconvenience.

Nothing more, nothing less! Hmm … Note that this behavior only surfaced after SP1 of Visual Studio 2008 and TFS 2008 was installed on the build server. Before, everything worked fine. On my local development pc (with SP1 of Visual Studio 2008) all Unit Tests run without any problems.

Let’s take a look at the eventviewer to get some extra information of what’s going wrong here …

Two interesting events were logged:

  1. .NET Runtime version 2.0.50727.3053 – Fatal Execution Engine Error (7A035E00) (80131506)
  2. Faulting application mstest.exe, version 9.0.30729.1, stamp 488f21a6, faulting module mscorwks.dll, version 2.0.50727.3053, stamp 4889dc18, debug? 0, fault address 0x001c5e00.

After searching for these errors on the web I came across some blog posts that mentioned the same Fatal Execution Engine Error but the scenario where this error occurred was totally different. It had something to do with Visual Studio disappearing unexpectedly but apparently there was a hotfix available on Microsoft Connect for this Fatal Execution Engine Error. So, with no other options at that moment I decided to go for the hotfix and see what happened.

The hotfix indeed did fix the problem, but I’m still puzzled what actually caused the problem. Strange was that certain Unit Test assemblies did run well on the build server while others even didn’t get started due to the above conditions …

[I did also log this error on the Testing forum of Visual Studio Team System]


Test Impact Analysis

February 5, 2009

 
Test Impact Analysis is a new feature that will be part of Visual Studio Team System 2010 and it will enable developers to view which Tests are impacted by your latest code changes that are not checked in yet. I was struggling a bit to get it up-and-running but once you know the drill, it’s quite straightforward. The fact is that you need to have published test results by a Team Build on the previous stable situation.

How to set it up?

  • Create a new Team Build for your solution that also triggers your UnitTests and publishes the test results :
    1. Select Solution to Build and select your tests to run

    2. Enable Get Impacted Tests

    3. Enable Publish the Test Impact Data

  • Enable Code Coverage for appropriate assemblies

  • Queue a new build of the newly created Team Build
  • Change some code that would impact a Unit Test
  • Open the Test Impact View Window

  • See the results …

    This window reads as follows : 2 Unit Tests are impacted by my latest code changes that are not yet checked in. The AddTest Unit Test should be rerun because of changes made to the Add method.

I hope that this process will be simplified a bit more in the final release. If you don’t get it working immediately, check the Build log in search for Test Impact. Note that the Build log in TFS2010 has improved a lot! Maybe a topic for one of my next posts …


Follow

Get every new post delivered to your Inbox.