Part IV: A fully integrated testing experience with TFS 2010
The third release of Microsoft’s ALM Solution (Visual Studio 2010) no longer used the Team System moniker and brought all the available SKUs back under the good old Visual Studio brand. The 2010 release was definitely the most significant release with many new functionality and improvements since the launch of the product with Team System 2005. Microsoft put a lot of effort into building a full testing experience into Team Foundation Server with a test management suite for managing test planning and for running manual/automated UI tests, including the ability to analyze test impact based on code changes. The Lab Management solution on top of TFS was quite revolutionary and can now be seen as the first indications of the growing importance of automated deployment and testing.
As a result, Gartner included Microsoft as Visionary for the first time in their Magic Quadrant for Integrated Software Quality Suites (published report of January 2011).
Microsoft Test Manager (MTM) was the new kid on the block and could be purchased separately with the Test Professional Suite (with MSDN Subscription). The Ultimate edition of Visual Studio also contained Microsoft Test Manager. Many existing testing tools on the market focused particularly on helping specialists testers (strong programming/scripting skills) with white box testing and API testing, but the majority of testing activities in the industry was still done manually (black box testing) while not a lot of testing tools provided extra support for this manual work. Consider MTM as an integrated test environment (ITE) for testers, compared to an integrated development environment (IDE) for developers.
The new SKU wanted to provide a bridge between the testing team and the development team. It’s the interface that a tester will use to create test cases, organize test plans, track test results, and file bugs when defects are found. Test Manager is integrated with Team Foundation Server, and is designed to improve the productivity of testers.
An interesting feature was for example the ability to play back action recordings to fast forward to a particular test step in a manual test to verify if a bug was fixed. It was also possible to create automated UI tests in Visual Studio (known as Coded UI Tests) using new automation libraries. You could generate C#/VB.NET code in Visual Studio from an existing action recording and add validation code to verify that the application under test was performing correctly.
The most compelling story was to file actionable bugs as a tester to avoid the typical ping-pong game between developers and testers. Familiar with the famous “it works on my machine” reply from the developer who marked the bug as non-reproducible? During a manual/automated test run with the built-in Test Runner, a lot of Diagnostic Data Adapters (DDA) could be enabled (test settings) to automatically collect and attach all data to a new Bug report: video recording, system information, IntelliTrace, Test Impact Analysis, screenshots, … It became plain easy for a tester to gather all important information and send it to the development team for further validation. Next to this, test managers could follow up which builds had bug fixes and/or new features added. They were also presented with recommendations about which tests needed to be rerun, based on the effect of code changes that were made to the codebase (test impact analysis on managed code). Instead of rerunning all test cases over and over again, testers could stay up-to-date with a new iteration by designing new test cases.
The icing on the cake for me was Visual Studio Lab Management 2010, an add-on for TFS 2010 which extended the Build Automation feature. It allowed builds to easily deploy their output to virtual environments which could exist of multiple virtual machines. Relying on the snapshotting features of the virtualization technology (Hyper-V), complete environments could be rolled back to a known clean snapshot/checkpoint to always enforce a clean deployment for each new manual/scheduled build. Automated Tests could be integrated in the Build-Deploy cycle of the Lab Build to validate each new deployment. The ultimate goal was to detect bugs as soon as possible in every incremental update of your application in Development and once a defect was found, all collected data through the collectors could be easily stored in an actionable bug report. No more no repro!
Team Foundation Server 2010 was designed to be easily customizable and extensible. TFS ensures that developers using any development platform can participate. This was also identified by an independent ALM study of Gartner, published in a report in June 2012. Microsoft has been identified as a leader in this area with the TFS 2010 offering. This is a clear indication that Microsoft has great tools for Application Lifecycle Management. Many larger companies also needed to think about compliance (Sarbanes-Oxley Act). This is often a very important and crucial requirement when choosing an enterprise ALM tool. Team Foundation Server 2010 provided three main pieces of functionality as it relates to SOx – Work Item Tracking, Version Control and Automated Build. These three features working together provided the capability to eliminate many of the risks associated with SOx 404 compliance. At the heart of SOx is the ability to know who made what change when, what requirement was the change made for and who authorized the change to be made. Was the requirement tested to make sure that it works correctly and who authorized the release of that requirement to production? Or, to state it another way, is the requirement traceable? Many technical solutions require multiple tools to provide this kind of traceability and most of the time they break the traceability chain at some point because they are not integrated at all required points. With Team Foundation Server 2010, full traceability was baked in. Team Foundation Server provided a comprehensive and flexible set of capabilities through traceability, test case management and automated builds. The rich extensibility of both the built-in features and a third-party ecosystem produce tools your company need in order to meet the requirements of SOx. A full article on Sox and TFS 2010 is available at http://msdn.microsoft.com/en-us/library/gg983694.aspx.
In the next part (Part V) I will continue with the release of TFS 2012 and the spring of the cloud story: running TFS on Windows Azure.