I couldn’t find any additional information (eventlog, TFS Log, …) about the root cause of this failure, but while limiting the test methods for the test run I bumped into a test method which name consisted of 461 characters!

Apparently there’s a hard limit of 256 characters for the test method names that are published to the TFS data warehouse.

Apart from the fact how this “build automation” should be called, there are many reasons why you should enforce this behavior on different branch types for your applications. The ultimate goal is to improve the quality of the software application and to reduce the time to release the application in production. By setting up early validation (compilation, automatic testing + other quality gates) through “build automation” you will at least be notified as soon as possible of all kinds of validation errors (= quality check) and you will have a chance to fix them before other team members will be impacted by pulling a get latest on the repository.

Automatically firing a validation build after a check-in will in the end not prevent broken builds and that’s where the Gated Check-in Build will come into play with Team Foundation Server 2010.

The Gated Check-in Build in TFS2010 will prevent broken builds by not automatically committing your pending changes to the repository, but the system will instead create a separate shelveset that will be picked up by the Gated Check-in Build. The build itself will finally decide if the pending changes need to be committed to the repository based on the applied quality gates.

The picture above describes the full process of a Gated Check-In build.

How to setup a Gated Check-in build?

The Trigger Tab in the Build Definition window has now an extra option for selecting Gated Check-in.

At the moment a check-in is attempted by a developer in the branch where the Gated Check-in build is active, the developer will be faced with a dialog box.

Cancelling this window will not kick off the build, but will also not commit your pending changes to the repository. If you really want to overrule this build with committing your changes directly to the repository, you may select the 2nd checkbox to bypass the validation build (not recommended). By default your pending changes will reside in your local workspace (first checkbox). In the situation where you immediately want to start with new changes – not relying on previous changes – it might be appropriate to uncheck the first option.

In the ideal situation, the build will complete without any validation errors and will eventually commit the changes to the repository. This will also lead to a Gated Check-in notification for the original committer via the Team Build Notification tool.

If you had previously chosen to preserve the changes locally (default), you may have noticed that the files you were working on were still checked out during the build … and in fact after a successful build these changes do not reflect the as-is situation anymore of the repository. With the above window you get the option to immediately reconcile your workspace with the up-to-date repository. So, clicking the “Reconcile …” button will give you the opportunity to select the desired files to force an undo in your local workspace and to pickup the changes that were committed by the Gated Check-in build for these files.

Another way to reconcile your workspace (if you for example ignored this window or when the build notification is way too slow) is by right-clicking the completed Gated Check-in Build in the Build Explorer and selecting the option to reconcile your workspace.

If you did not choose to preserve the changes locally, there won’t be any changes to reconcile after the Gated Check-in build, even if you forced the reconciliation.

The Gated Check-in build may also be kicked off manually where you may need to create a shelveset or where you may point to an existing shelveset.

A last thing to note is that the comment that was originally supplied to the changeset by the developer will be suffixed with the NoCICheckinComment variable (default = ***NO_CI***) to prevent that another continuous integration build will be fired after the final check-in done by the Gated Check-in build.

Summary

What meant to be a small post on the Gated Check-in feature in Team Foundation Server 2010 ended up in a more detailed explanation of how it works and how you can work with it in the Visual Studio IDE. Remember that you should setup the most appropriate build types according to your specific branches. Not all branches may need a Gated Check-in build. Only configure this for branches that should never have a broken build. A Gated Check-in build may for example validate a big merge operation from a development branch to a stable main branch.

Coded UI Tests provide functional testing of the user interface and validation of user interface controls.

How to create Coded UI Tests? You could create them directly into Visual Studio, but for this blogpost I want to start from an action recording in Microsoft Test and Lab Manager (MTLM). An action recording is quite useful in manual tests that you need to run multiple times and for recycling common steps in different manual tests that contain shared steps.

I did create a simple test case with different test steps in MTLM to test some behavior on my website.

From MTLM I started a test run for this test case.

Before running the test, I do need to check the action recording to be sure to capture my actions for this test.

The Test Runner will give a detailed overview of the recorded actions. Afterwards you will be able to replay all these stored actions in the Test Runner.

After saving the results of this test run (all data is associated to my test case) it’s time to open Visual Studio 2010 and to create a Coded UI Test.

Instead of choosing the default option to record actions I did choose to use an existing action recording after which I need to retrieve the appropriate test case to link to the associated actions.

By clicking OK, Visual Studio will start generating code that will represent my actions that were recorded in Microsoft Test and Lab Manager. On top of that you are also able to add assertions on parts of the user interface in a separate Coded UI Test that you may reuse in other Coded UI Tests.

Now, let’s integrate this entire UI test (MyCodedUITest) into the automated build. I created a default new build defintion where I also enabled to run the automated tests.

To run unit tests that interact with the desktop during a Team Build, we need to modify the Build Service Host properties in the Team Foundation Administration Console to run the build service as an interactive process instead of running the build service as a Windows Service.

That’s about it. Make sure that the Build Service Host is running in the command line that will pop up after starting the BuildServiceHost. Queue the build and explore the results!

Done!

With this post I wanted to highlight the powerful integration of (automated) testing into the upcoming Visual Studio 2010 offering.

Code coverage is a measure used in software testing. It describes the degree to which the source code of a program has been tested.

I won’t start the discussion what’s a good percentage for code coverage and why you should care or not, but let me just say that Code Coverage results may be very welcome while testing your applications. We all know that 100% coverage for a specific application doesn’t mean that your application is fail-proof and it won’t tell you how good your unit tests are. But it may give you a nice indication of the reach of your unit tests and those results should always be combined with other metrics of your code/tests.

Recently I noticed a strange thing when setting up unit tests (including code coverage) in Team Builds for TFS2008. To run unit tests after the compilation process in the build, you basically have two options.

1. specify test list(s) in the vsmdi file

    

2. specify test containers  

    

Now the tricky part about code coverage results, considering you have a solution with unit tests where you enabled code coverage in the testrunconfig file.

Setting up a Team Build with activating unit tests via a vsmdi file will give you the requested Code Coverage results.

On the other hand, setting up a Team Build (same solution) with enabling unit tests via a test container won’t give you the desired Code Coverage results.

Why is that?! Well, Code Coverage settings are contained in the testrunconfig file and the thing is that the vsmdi file contains a reference to this testrunconfig file (open up notepad to verify), while the test containers don’t have a link with any testrunconfig file. If you want to include a testrunconfig file to your build that will be used during the execution of the unit tests, you need to specify the RunConfigFile property in the TfsBuild.proj file so that the Code Coverage settings are picked up during the run of the unit tests.

Result:

Today I filed a suggestion for the next release of Team Foundation Server: Build Definition History. Unfortunately it’s still not possible to view history of changes made to the Build Defintion: “Drop Location”, “Build Agent”, “Trigger”, … On the Microsoft Connect site, you can easily look up other wanted featured and vote for them … but first vote for my suggestion!

Since a few weeks, there’s also a feedback survey running on Microsoft Visual Studio 2010 and the .NET Framework 4 Beta 2. If you care about the product and want your voice heard, please take some minutes to complete this online survey!

This is what I did with Beta 2 of Visual Studio 2010 Ultimate and TFS 2010 Basic on my Win7 laptop:

• created a new Team Project
• added a new solution with a C# Library Project to the Team Project
• added a default Team Build to build the C# Library Project
• added a new solution with a C# WPF Project to the Team Project
• referenced the library assembly (file reference to dll) into the C# WPF Project and called a method on a class in that assembly
• set a breakpoint on that line and hit F5 to start/debug the WPF application
• pressed F11 (Step Into) when breakpoint was hit

Guess what?! Yes, Visual Studio 2010 was immediately stepping into the source file of the C# Library Project! Sweet!

I remember that it took me some time to get this working for TFS2008!

When you create a new Build Definition with TFS2010, the Index Sources option is set to true by default and this will make sure that source indexing is part of the build.

I took a peek into the DefaultTemplate.xaml file in the BuildProcessTemplates folder and found out that the Index Sources and Publish activity is indeed completely baked in! I love it already!

• possible differences between retention policy for triggered and manual builds vs private builds

  

  You will be able to set another retention policy on private builds: builds that are run by the enabled Gated Check-in trigger.

• extra What to delete column

  

  You will be able to specify more in detail what information must be deleted when the retention policy kicks in.

  

MSBUILD : warning MSB6006: "MSTest.exe" exited with code -2146233082

Digging a bit deeper and running the Unit Tests manually with MSTest on the build server via the command line showed me a dialog box with the following notification:

MSTest.exe has encountered a problem and needs to close. We are sorry for the inconvenience.

Nothing more, nothing less! Hmm … Note that this behavior only surfaced after SP1 of Visual Studio 2008 and TFS 2008 was installed on the build server. Before, everything worked fine. On my local development pc (with SP1 of Visual Studio 2008) all Unit Tests run without any problems.

Let’s take a look at the eventviewer to get some extra information of what’s going wrong here …

Two interesting events were logged:

1. .NET Runtime version 2.0.50727.3053 – Fatal Execution Engine Error (7A035E00) (80131506)
2. Faulting application mstest.exe, version 9.0.30729.1, stamp 488f21a6, faulting module mscorwks.dll, version 2.0.50727.3053, stamp 4889dc18, debug? 0, fault address 0x001c5e00.

After searching for these errors on the web I came across some blog posts that mentioned the same Fatal Execution Engine Error but the scenario where this error occurred was totally different. It had something to do with Visual Studio disappearing unexpectedly but apparently there was a hotfix available on Microsoft Connect for this Fatal Execution Engine Error. So, with no other options at that moment I decided to go for the hotfix and see what happened.

The hotfix indeed did fix the problem, but I’m still puzzled what actually caused the problem. Strange was that certain Unit Test assemblies did run well on the build server while others even didn’t get started due to the above conditions …

[I did also log this error on the Testing forum of Visual Studio Team System]

Reference in the manifest does not match the identity of the downloaded assembly x

It was clear that something was wrong with the manifest, but I could not find what was keeping the application from launching. I checked and double checked the build script and the way the package was built but could not find an indication of the above error. Publishing the application from within Visual Studio 2008 seemed to work without any hitches and it was then that I noticed the following setting in the Application Tab of the Start Project in Visual Studio.

Visual Studio 2008 embeds by default a manifest inside an assembly. This means that the ClickOnce executable was using this default manifest instead of the manifest that was generated with Team Build. Switching to the other option to not generate a manifest with Visual Studio did solve the issue!

Hopefully I can save someone’s time with this blogpost!

The good news is that TFS2010 will offer a lot more fine-grained permission sets! You will now have the possibility to set permissions on the Team Project Collection, on the Team Project, on the Build Definition level and on the Version Control repository!

• Team Project Collection
  • Administer shelved changes
  • Administer test controllers
  • Administer warehouse
  • Administer workspaces
  • Alter trace settings
  • Create a workspace
  • Create new projects
  • Delete a team project
  • Delete team project collection
  • Edit collection-level information
  • Make requests on behalf of others
  • Manage build resources
  • Manage process template
  • Manage work-item link types
  • Trigger events
  • Use build resources
  • View build resources
  • View collection-level information
  • View system synchronization information
• Team Project
  • Administer test environments
  • Create test runs
  • Create team project
  • Delete test runs
  • Edit project-level information
  • View project-level information
  • View test runs
• Build Definition
  • View Builds
  • Edit build quality
  • Retain indefinitely
  • Delete builds
  • Manage build qualities
  • Destroy builds
  • Update build information
  • Queue builds
  • Manage build queue
  • Stop builds
  • View build definition
  • Edit build definition
  • Delete build definition
  • Override check-in validation by build
• Version Control
  • Read
  • Check Out
  • Check In
  • Label
  • Lock
  • Revise other users’ changes
  • Unlock other users’ changes
  • Undo other users’ changes
  • Administer labels
  • Manage permissions
  • Check-in other users’ changes
  • Merge
  • Manage branch

Great! There are a few permission that are new and that I certainly want to look into a bit deeper … but now let’s go back to my problem in TFS2008 and how to fix it in TFS2010. Right clicking the Team Project Collection brings me to the permissions on the Project Collection level.

The permission to Manage build resources allows people to create and modify build controllers and agents.

Right clicking Builds brings you to the permissions on the build definition level.

The permission to Edit build definition allows people to create and modify new build defnitions.