(upcoming) ALM/DevOps Community Talks

February 8, 2016

Q1 of 2016 has already been quite busy with the organisation of VISUG 10 Years and the upcoming conference Techorama 2016. The user group and all community activities are simply not a simple side job anymore.

For the celebration of VISUG 10 years I delivered a session DevOps with Visual Studio Release Management.

In March (part of a Techorama 2016 promo tour), I will be travelling to Scotland for a User Group Talk in Glasgow (March 10): Advanced Techniques for Web Deploy (msdeploy) to simplify the deployment of web applications. The next day (March 11), I will again deliver my DevOps with Visual Studio Release Management talk in Edinburgh. Thanks to the Scottish Developers User Group for setting this up!


Destroying deleted branches in TFS

January 13, 2016

Everyone using a version control system will have ever used a delete command to delete files/folders which are not required anymore.

Team Foundation Verion Control (TFVC) treats a delete [tf delete] as a pending change in your workspace (type = delete). The final checkin command removes the item from the version control server, but does not delete the item permanently. All historical changes to the file remain available for lookup. In fact it’s also possible to undelete [tf undelete] an item and bring it back into play.

In only a few scenarios you might want to go one step further and really destroy the deleted files from TFS …

Recently a customer was complaining about the exponential growth of their TFS 2013 databases (exceeding 400GB for a specific TPC database). Running some SQL scripts on the tbl_content table of the Team Project Collection database revealed that a lot of version control and file container data was added monthly to the TFS database. Ranging from 2.5GB a few months ago up to 8GB in the last months.

After some investigation I noticed that the development team was creating a lot of feature branches (150+) for a specific application to isolate their feature development. This shouldn’t have a big impact on the exponential growth of the TFS databases because a branch is like a shallow copy of each of the files from the source branch. Both branches will originally refer to the same copy of each file in TFS. Extra changes will trigger a deltafication process to optimize storage. More info of this complex procedure is explained in this old TFS 2010 blog post which should still be valid for the latest version of TFS. But the main problem was that the feature branches also contained a lot of references (NuGet packages / binary files) which were added to version control as well and many dependency changes were initiated in the feature branches. As described in the old blog post, TFS will also try to compute deltas for binary files, but it’s way more difficult to predict database growth based on the sizes of these deltas. The results from deltafication vary greatly depending on what type of file it is, and the nature of the changes.

After further discussions, we decided to permanently destroy old deleted feature branches in version control and see the impact of the database size. I wrote a powershell script to iterate over a version control root folder to search for deleted branches and to destroy it only if the latest check-in did not occur in the last 100 days. In the script I used the startcleanup switch to immediately trigger the clean up process in TFS instead of waiting for the daily clean up background process.

I don’t remember the exact results on the database size, but it certainly decreased the total size of the TFS database for the involved Team Project Collection. With the upcoming planned upgrade to TFS 2015, we also used the Test Attachment Cleaner (part of the TFS Power Tools) to remove old diagnostic test data. So, in the end we were able to reduce the total size of the TPC database with a fair amount of GB in order to also speed up the duration of the future upgrade to TFS 2015.

Another initiative was started to not use TFS anymore to keep (physical) track of the dependencies. TFS is indeed a version control repository to store sources, but there are better tools out there to store software packages/artefacts.


TFS Build 2015 … and versioning!

August 24, 2015

Lately I got some time to play a bit more with the new build system which was released with TFS 2015 and which is also available for Visual Studio Online. The new build system was in the beginning announced as build vNext, but now with the release of TFS 2015, it’s safe to call it Team Foundation Build 2015 (TFBuild 2015) while the “old” build system can be referred to as the xaml (workflow) build system. Colin Dembovsky has a great post why you should switch to the new build sytem.

In the last years, I had to implement a lot of customizations into the xaml build system and I became very productive with the workflow activities. Along the way I developed a number of generic activities which I could reuse for other assignments and I really knew my way around in the build workflow. In many cases, the TFS Build Extensions were used to not reinvent the wheel. So, in the first phase I was a bit sceptic about the rise of yet another build system, but I clearly saw some interesting advantages which are explained in the post by Colin. One disadvantage of the xaml build system is the steep learning curve to master the customization process and also the deployment mechanism to refresh the TFS build controller(s). But like I experienced, once you got there, you were able to integrate very powerful customizations into the build process. Anyway, the “old” build system won’t disappear and you can still rely on this functionality for quite some time in the future, but I recommend to have a good look at the new build system and use it for your new/future build definitions.

In this post I want to share how I integrated a common activity in the build process: Versioning. With the available build steps it has become extremely simple to hook your own scripts into the build process. In your scripts you will have access to some predefined build variables.

In my previous blogpost I wrote about adopting a Global .NET Versioning Strategy and the existence of a third (optional) version attribute: AssemblyInformationalVersion. Let’s use this strategy to add versioning to a sample Fabrikam software application.

My build definition:

BuildDefinition-1

In the screenshot above you will see that I launch a powershell script (PreBuild.ps1) before building the solution and I pass one argument productVersion to the script. The powershell script will do the magic in the background to replace all versioning values for AssemblyVersion, AssemblyFileVersion and AssemblyInformationalVersion in the Assembly Info files, based on this product version. The product version will be passed as a whole to the AssemblyVersion and the AssemblyInformationalVersion attributes. The AssemblyFileVersion will be replaced with a full version number which will consist of the major and minor version number of the product version, a Julian based date and an incremental build number.

BuildResult-1

Assembly File Version = 1.0.15236.3

  • 1 => taken from “Major” product version
  • 0 => taken from “Minor” product version
  • 15236 => generated by build process: “15” = year 2015, “236” = day of year 2015
  • 3 => third build, run on day 236 in year 2015

Looking at the assembly details of a custom built Fabrikam assembly now reveals correct meta data:

AssemblyDetails

I also modified the build number format to have some more version information displayed in the build run.

BuildDefinition-2

BuildRuns-1

I added a gist at GitHub to share the powershell script. Note that the script has been used for experimentation and may not be ready to be used for production. it certainly lacks some proper validation and error-handling. Use at your own risk.

Also have a look at some similar inspiring blog posts about versioning TFS Builds which helped me to develop the powershell script that works for my scenario.


Global .NET Versioning Strategy – AssemblyInformationalVersion

August 24, 2015

Ever heard of a third (optional) versioning attribute in the AssemblyInfo files: AssemblyInformationalVersion. No? Please read!

Without a methodical (assembly) version numbering strategy, the ability to determine what changes were included in which version is lost. In my opinion, you always need to know exactly which source files went into which build and which version of the sofware is currently deployed in a specific environment. A random version numbering system creates confusion and will soon or later cause deployment risks. It will become a nightmare to fetch the exact source files to reproduce a bug from production.

All versioning of .NET assemblies that use the common language runtime is done at the assembly level. The specific version of an assembly and the versions of dependent assemblies are recorded in the assembly’s manifest. The default version policy for the runtime is that applications run only with the versions they were built and tested with, unless overridden by explicit version policy in configuration files.

Each .NET project has an AssemblyInfo file which contains an AssemblyVersion attribute and an AssemblyFileVersion attribute.

  • AssemblyVersion: this is the version number used by the .NET framework during build and at runtime to locate, link and load the assemblies. When you add a reference to any assembly in your project, it is this version number which gets embedded. At runtime, the CLR looks for assembly with this version number to load. But remember this version is used along with name, public key token and culture information only if the assemblies are strong-named signed. If assemblies are not strong-named signed, only file names are used for loading.
  • AssemblyFileVersion: This is the version number given to a file as in file system. It is displayed by Windows Explorer. It’s never used by .NET framework or runtime for referencing.

But what about this difference between AssemblyVersion and AssemblyFileVersion? Many times, I see that the same version is applied to both attributes … but why are these two (different) attributes provided by the .NET Framework? The AssemblyVersion should be the public version of an entire software application, while the AssemblyFileVersion is more the version of a specific component which may only be a small part of the entire application. The AssemblyFileVersion is the best place to put extra build version information which can be important for patching individual components of a software application.

Please follow the Semantic Versioning recommendations to dictate how the AssemblyVersion should be assigned and incremented. For the AssemblyFileVersion, I tend to include specific build information. Often, you will need to build (and test) a number of time a specific SemVer version of your software.

For example: release 1 of a software application could have the AssemblyVersion set to 1.0.0 (all components), while the AssemblyFileVersion of the individual components could be set to 1.0.15234.2 which refers to a unique build number of the build system and is linked to a particular date and a revision: “15” = year 2015; “234” = day number in 2015; “2” = second build processed that day. This also allows to later patch individual components in production with a similar AssemblyVersion (1.0.0), but a different AssemblyFileVersion (1.0.15235.1).

So, let’s try to apply this to a test project in Visual Studio and see the assembly details after building the project …

Versioning-1

Now you should be confused! Why does the Product Version display the AssemblyFileVersion and where’s the AssemblyVersion? The problem here is that a new Visual Studio project doesn’t include a third version attribute AssemblyInformationalVersion which is intended to represent the public version of your entire software application. Note that the CLR doesn’t care about this third (optional) version attribute. In short, the same Semantic Versioning rules of AssemblyVersion should be applied to AssemblyInformationalVersion.

Versioning-2

Aha! This looks much better right? Now it’s also easy to extract this metadata from your deployed assemblies and this information can be nicely listed in the about box of your software. The only issue with this approach is that the AssemblyFileVersion doesn’t include the “patch” number (Semantic Versioning) of the AssemblyVersion, but this can be ignored with the fact that the AssemblyFileVersion will be unique and can be linked to a unique build run in the build system. This way of working is my personal interpretation of how versioning can be properly applied in complex software applications and doesn’t reflect official guidelines from Microsoft. My goal here is to make software developers aware of the potential risks of not having a clear versioning strategy.

Now, forget about manually setting version information in the AssemblyInfo files and never ever release software from a local Visual Studio build. In a streamlined build process, generating unique build and version numbers are centrally coordinated. For effective troubleshooting and traceability, it’s imperative that the generated assemblies are stamped with a unique identifier that can be easily traced back to a system build number.

In a next post I will talk about how you can achieve this global .NET versioning strategy with the new build system in TFS 2015.


Deep copy of Test Cases to a TFS Team Project in another Team Project Collection

July 22, 2015

A few days ago I got stuck in a project to migrate a set of test cases from an “old” Team Project to another (new) Team Project in another Team Project Collection (TFS 2013 Update 4 environment). Via the TFS API I used the work item Copy method to perform a deep copy [including links/attachments] with the WorkItemCopyFlags set to CopyFiles.

ITestCase testCase = <MyTestCase>;
WorkItem wiCopy = testCase.WorkItem.Copy(<TargetProjectWorkItemType>, WorkItemCopyFlags.CopyFiles);
wiCopy.Save();

This always worked when copying test cases to a Team Project in the same Team Project Collection (TPC) as the original Team Project, but copying the test cases to a Team Project in another TPC always caused an error:

TF237136: File attachment was not found on attachment server.

I expected this to work, but after I verified that the MTMCopyTool also generated the same error when copying test cases across Team Projects, I decided to search for a workaround … downloading and uploading the attachments as after-processing.

DownloadUploadAttachments

Not perfect, but good enough to keep going and moving all attachments to the other Team Project in the other Team Project Collection.


Reporting Services issues after migration to TFS 2015 RC2 (from TFS 2010)

July 10, 2015

Yesterday, I blogged about the migration activities to get to TFS 2015 RC2, but I was still stuck at an error during new Team Project creation.

TeamProjectFailure

Looking into the logs it looked like there was some kind of issue during the upload of the first Reporting Services report “Bug Status” (part of the latest TFS 2015 Agile process template).

TeamProjectFailure-ReportUpload

First I thought there was something wrong with permissions on the Reporting Services site because I also applied a certificate to offer accessibility via https, but that wasn’t it. All seemed to be setup correctly. A sign to look further into the details of the Bug Status .rdl report definition and why it couldn’t be correctly uploaded to the Reporting Services site. After downloading the process template from the Team Project Collection and opening the report with notepad, I knew I was getting closer to the root cause.

DataSourceIssue

As shown in the picture above, the (default TFS 2015) Data Source mentioned in the report definition did not exist in the Report Server database after the migration from TFS 2010 SP1. In the TFS 2010 timeframe, all report defintions were linked to a “2010” Data Source.

Instead of renaming the existing Data Sources (that would break the existing reports, created from a TFS 2010 Team Project), I duplicated the entries to provide extra Data Sources with the expected names for usage with TFS 2015.

Issues resolved. Back on track!


Upgrading TFS 2010 SP1 to TFS 2015 RC2

July 9, 2015

Today I got the opportunity to upgrade a customer from TFS 2010 SP1 to TFS 2015. As blogged by Brian Harry a few days ago, the RTM release of TFS 2015 (not Visual Studio 2015!) is delayed (for a good reason!) and in order to further test the overall migration/upgrade process a new RC2 release has been made available (with go-live license). That’s the build I used today to upgrade from TFS 2010 SP1. The official installation guide is not yet available, so here are some tips to get you started in the right direction … At the customer I got 2 new servers at my disposal with a clean Windows Server 2012 R2 OS. The first server is used for the TFS Application and TFS Data Tier (Single Server Topology). The second one will be used as the TFS Build Server.

Prerequisites

Installing SQL Server 2014 Standard Edition

Before starting the installation of SQL Server 2014, I enabled the .NET 3.5 feature on Windows Server 2012 R2. OSFeature

Required SQL Server features:

SQL2014-FeatureSelection

I always recommend to use dedicated domain service accounts for running SQL Server. In this case I decided to reuse the <TFSSERVICE> account. Note that all services are set to start automatically. ServiceAccountsForSQL

SQL2014-InstallationResult

After successful installation of SQL Server 2014 I started the backup procedure on the old TFS 2010 environment and I copied all SQL .bak files to the new server for restoring all TFS related databases to the SQL Server 2014 instance. SQLDatabases

Configuration of SQL 2014 Reporting Services

Start-RS

Link to the existing restored “ReportServer” SQL Server database.

SetRSDatabase

RS-ExistingDB

RS-SelectReportServerDatabase

RS-Summary

Restore the SQL Server Reporting Services Encryption key.

RS-RestoreEncryptionKey

RS-RestoreEncryptionKey2

Run/Apply the web service and the report manager service.

RS-Webservice

RS-ReportManager

Remove the “old” server from the scale-out deployment configuration [http://intovsts.net/2014/07/14/tfs-migration-upgrade-scale-out-reporting-services-issues/] Test the Report Server website …

RS-Running

Installation TFS 2015 RC2

InstallTFS2015

trialLicense

The trial license is valid for 90 days and can be extended for 30 days. At this moment, there are no product keys available for the RC2 release.

Wizard-Upgrade

Wizard-Upgrade2

Wizard-Upgrade3

Wizard-Upgrade4

No Build Configuration yet.

Wizard-Upgrade5

Wizard-Upgrade6

Wizard-Upgrade7

Wizard-Upgrade8

Wizard-Upgrade9

Wizard-Upgrade10

No SharePoint link (yet).

Wizard-Upgrade11

Verification …

Wizard-Upgrade-Verification

Successful upgrade!

Upgrade-Success

Once the TFS 2015 RTM release becomes available, it’s only a minor upgrade to move to the official RTM bits. Interested in the release notes for TFS 2015? Check out the official news update, posted in April 2015.


Follow

Get every new post delivered to your Inbox.