Code Coverage results with Team Builds (TFS2008)

January 20, 2010

From Wikipedia:

Code coverage is a measure used in software testing. It describes the degree to which the source code of a program has been tested.

I won’t start the discussion what’s a good percentage for code coverage and why you should care or not, but let me just say that Code Coverage results may be very welcome while testing your applications. We all know that 100% coverage for a specific application doesn’t mean that your application is fail-proof and it won’t tell you how good your unit tests are. But it may give you a nice indication of the reach of your unit tests and those results should always be combined with other metrics of your code/tests.

Recently I noticed a strange thing when setting up unit tests (including code coverage) in Team Builds for TFS2008. To run unit tests after the compilation process in the build, you basically have two options.

  1. specify test list(s) in the vsmdi file


  2. specify test containers  



Now the tricky part about code coverage results, considering you have a solution with unit tests where you enabled code coverage in the testrunconfig file.


Setting up a Team Build with activating unit tests via a vsmdi file will give you the requested Code Coverage results.


On the other hand, setting up a Team Build (same solution) with enabling unit tests via a test container won’t give you the desired Code Coverage results.


Why is that?! Well, Code Coverage settings are contained in the testrunconfig file and the thing is that the vsmdi file contains a reference to this testrunconfig file (open up notepad to verify), while the test containers don’t have a link with any testrunconfig file. If you want to include a testrunconfig file to your build that will be used during the execution of the unit tests, you need to specify the RunConfigFile property in the TfsBuild.proj file so that the Code Coverage settings are picked up during the run of the unit tests.




Feedback VS2010 Beta2?

November 6, 2009

So, you are playing around with VS2010/TFS2010 and you have some remarks, suggestions, bugs, … Please go the Microsoft Connect site for product feedback and bug reporting.

Today I filed a suggestion for the next release of Team Foundation Server: Build Definition History. Unfortunately it’s still not possible to view history of changes made to the Build Defintion: “Drop Location”, “Build Agent”, “Trigger”, … On the Microsoft Connect site, you can easily look up other wanted featured and vote for them … but first vote for my suggestion!

Since a few weeks, there’s also a feedback survey running on Microsoft Visual Studio 2010 and the .NET Framework 4 Beta 2. If you care about the product and want your voice heard, please take some minutes to complete this online survey!

Out-of-the-box source server indexing with TFS2010

October 27, 2009

Wow! This is great! Tonight I just wanted to find out if and how it would work …

This is what I did with Beta 2 of Visual Studio 2010 Ultimate and TFS 2010 Basic on my Win7 laptop:

  • created a new Team Project
  • added a new solution with a C# Library Project to the Team Project
  • added a default Team Build to build the C# Library Project
  • added a new solution with a C# WPF Project to the Team Project
  • referenced the library assembly (file reference to dll) into the C# WPF Project and called a method on a class in that assembly
  • set a breakpoint on that line and hit F5 to start/debug the WPF application
  • pressed F11 (Step Into) when breakpoint was hit

Guess what?! Yes, Visual Studio 2010 was immediately stepping into the source file of the C# Library Project! Sweet!

I remember that it took me some time to get this working for TFS2008!

When you create a new Build Definition with TFS2010, the Index Sources option is set to true by default and this will make sure that source indexing is part of the build.

I took a peek into the DefaultTemplate.xaml file in the BuildProcessTemplates folder and found out that the Index Sources and Publish activity is indeed completely baked in! I love it already!

Build Retention Policies in TFS2010

October 11, 2009

I noticed some interesting changes for the Build Retention Policies in TFS2010 (Beta 1):

  • possible differences between retention policy for triggered and manual builds vs private builds

    You will be able to set another retention policy on private builds: builds that are run by the enabled Gated Check-in trigger.

  • extra What to delete column

    You will be able to specify more in detail what information must be deleted when the retention policy kicks in.

MSTest.exe exited with code -2146233082 during Team Build

September 17, 2009

Recently I was faced with a failure in a Team Build caused by MSTest that crashed when firing up the Unit Tests.

MSBUILD : warning MSB6006: "MSTest.exe" exited with code -2146233082

Digging a bit deeper and running the Unit Tests manually with MSTest on the build server via the command line showed me a dialog box with the following notification:

MSTest.exe has encountered a problem and needs to close. We are sorry for the inconvenience.

Nothing more, nothing less! Hmm … Note that this behavior only surfaced after SP1 of Visual Studio 2008 and TFS 2008 was installed on the build server. Before, everything worked fine. On my local development pc (with SP1 of Visual Studio 2008) all Unit Tests run without any problems.

Let’s take a look at the eventviewer to get some extra information of what’s going wrong here …

Two interesting events were logged:

  1. .NET Runtime version 2.0.50727.3053 – Fatal Execution Engine Error (7A035E00) (80131506)
  2. Faulting application mstest.exe, version 9.0.30729.1, stamp 488f21a6, faulting module mscorwks.dll, version 2.0.50727.3053, stamp 4889dc18, debug? 0, fault address 0x001c5e00.

After searching for these errors on the web I came across some blog posts that mentioned the same Fatal Execution Engine Error but the scenario where this error occurred was totally different. It had something to do with Visual Studio disappearing unexpectedly but apparently there was a hotfix available on Microsoft Connect for this Fatal Execution Engine Error. So, with no other options at that moment I decided to go for the hotfix and see what happened.

The hotfix indeed did fix the problem, but I’m still puzzled what actually caused the problem. Strange was that certain Unit Test assemblies did run well on the build server while others even didn’t get started due to the above conditions …

[I did also log this error on the Testing forum of Visual Studio Team System]

Generating ClickOnce packages with Team Build 2008

September 7, 2009

Last week I spent way too much time on troubleshooting an issue with generating a ClickOnce package with a Team Build. Generating the package was actually not the problem, but running the application always resulted in …

Reference in the manifest does not match the identity of the downloaded assembly x

It was clear that something was wrong with the manifest, but I could not find what was keeping the application from launching. I checked and double checked the build script and the way the package was built but could not find an indication of the above error. Publishing the application from within Visual Studio 2008 seemed to work without any hitches and it was then that I noticed the following setting in the Application Tab of the Start Project in Visual Studio.

Visual Studio 2008 embeds by default a manifest inside an assembly. This means that the ClickOnce executable was using this default manifest instead of the manifest that was generated with Team Build. Switching to the other option to not generate a manifest with Visual Studio did solve the issue!

Hopefully I can save someone’s time with this blogpost!

More fine-grained permissions in TFS2010

September 3, 2009

Recently (in TFS2008) I was stuck with the fact that I could not split up the permission to create/modify builds and the permission to create/modify build agents. In certain enterprise environments it might be necessary to revoke the right from development teams to create/modify build agents. Build agents may be for instance controlled centrally by an operations team that manages all build servers. In TFS2008 both tasks belong to the “Administer a buid” permission.

The good news is that TFS2010 will offer a lot more fine-grained permission sets! You will now have the possibility to set permissions on the Team Project Collection, on the Team Project, on the Build Definition level and on the Version Control repository!

  • Team Project Collection
    • Administer shelved changes
    • Administer test controllers
    • Administer warehouse
    • Administer workspaces
    • Alter trace settings
    • Create a workspace
    • Create new projects
    • Delete a team project
    • Delete team project collection
    • Edit collection-level information
    • Make requests on behalf of others
    • Manage build resources
    • Manage process template
    • Manage work-item link types
    • Trigger events
    • Use build resources
    • View build resources
    • View collection-level information
    • View system synchronization information
  • Team Project
    • Administer test environments
    • Create test runs
    • Create team project
    • Delete test runs
    • Edit project-level information
    • View project-level information
    • View test runs
  • Build Definition
    • View Builds
    • Edit build quality
    • Retain indefinitely
    • Delete builds
    • Manage build qualities
    • Destroy builds
    • Update build information
    • Queue builds
    • Manage build queue
    • Stop builds
    • View build definition
    • Edit build definition
    • Delete build definition
    • Override check-in validation by build
  • Version Control
    • Read
    • Check Out
    • Check In
    • Label
    • Lock
    • Revise other users’ changes
    • Unlock other users’ changes
    • Undo other users’ changes
    • Administer labels
    • Manage permissions
    • Check-in other users’ changes
    • Merge
    • Manage branch

Great! There are a few permission that are new and that I certainly want to look into a bit deeper … but now let’s go back to my problem in TFS2008 and how to fix it in TFS2010. Right clicking the Team Project Collection brings me to the permissions on the Project Collection level.

The permission to Manage build resources allows people to create and modify build controllers and agents.

Right clicking Builds brings you to the permissions on the build definition level.

The permission to Edit build definition allows people to create and modify new build defnitions.

DYK #3 : retain build indefinitely

March 17, 2009

If you hold on to a specific Team Build retention policy (new since TFS2008), you might lose all valuable build information of a particular build that you actually wanted to keep.

There’s an option available via the Build Explorer to retain builds indefinitely!

DYK #1 : IIS not required for Team Build

February 22, 2009

DYK = Did You Know?

Internet Information Server (IIS) is not required to be installed for a TFS 2008 Team Build machine

The application server for TFS 2005 still used .NET Remoting to communicate with the Team Build machine, but as from TFS 2008 the communication is set up with Windows Communication Foundation. The WCF service runs in a managed Windows Service (TfsBuildService.exe).

Test Impact Analysis

February 5, 2009

Test Impact Analysis is a new feature that will be part of Visual Studio Team System 2010 and it will enable developers to view which Tests are impacted by your latest code changes that are not checked in yet. I was struggling a bit to get it up-and-running but once you know the drill, it’s quite straightforward. The fact is that you need to have published test results by a Team Build on the previous stable situation.

How to set it up?

  • Create a new Team Build for your solution that also triggers your UnitTests and publishes the test results :
    1. Select Solution to Build and select your tests to run

    2. Enable Get Impacted Tests

    3. Enable Publish the Test Impact Data

  • Enable Code Coverage for appropriate assemblies

  • Queue a new build of the newly created Team Build
  • Change some code that would impact a Unit Test
  • Open the Test Impact View Window

  • See the results …

    This window reads as follows : 2 Unit Tests are impacted by my latest code changes that are not yet checked in. The AddTest Unit Test should be rerun because of changes made to the Add method.

I hope that this process will be simplified a bit more in the final release. If you don’t get it working immediately, check the Build log in search for Test Impact. Note that the Build log in TFS2010 has improved a lot! Maybe a topic for one of my next posts …