Code & Work Item Search for TFS 2017 – Troubleshooting

November 12, 2017

The Code Search extension for VSTS/TFS makes it easy to search for information across all your projects, from anywhere and any computer or mobile device, using just a simple web browser. You can narrow down your results and focus in on what you need by using filters.

CodeSearch

For TFS 2017 on-premises, Code Search includes Elasticsearch and will be configured on a server running TFS 2017. Work Item Search is now also relying on this functionality.

For large TFS enterprise environments with a lot of big code repositories the Search service might impact the performance of the TFS Application Tier when the Search service has been installed/configured on the same server.

These type of performance issues have been the case for a specific customer and the IT operations team have seen various huge CPU spikes for the Elasticsearch service on the TFS Application Tier.

Together with an in-place upgrade to TFS 2017 Update 3, I recommended to move the Search service to a dedicated server in order to avoid performance issues on the TFS Application Tier. Compared to the typical straightforward TFS upgrade wizard experience, the move of the Search service to another service contains a number of manual activities.

SearchConfiguration

After completing all actions on the dedicated Search server and making sure that the TFS Application Tier could access the Search service on the Search Server via the default port 9200, I was able to complete the upgrade.

The TFS upgrade was a success, except the Search service was not working anymore.

SNAGHTML42be1913

The TFS Administration Console was showing no errors/warnings about the installed Search component.

SNAGHTML42c07979

Looking into some troubleshooting actions from the documentation didn’t help me to get to the root issue because the Search service seemed to be up-and-running, but wasn’t processing any data.

SNAGHTML42c6ccdc

Time to get in contact with the VSTS/TFS Product Team to log this incident and sharing all possible log files (detailed instructions how to do this) and look for potential solutions.

In the end – after some analysis of the logs – I was requested to perform a cleanup of the Index Data and to restart the indexing process on all Team Project Collections.

After going through this entire process, I was happy to see the ElasticSearch process coming to life again and claiming lots of CPU.

image

Also the TFS Search Data/Index folder on the Search Server was quickly getting flooded with a lot of data.

image

Mission finally accomplished! Upgraded to the lastest version of TFS 2017 and moving the Search Service to another server to avoid performance issues on the TFS Application Tier.

If you don’t want to get bothered with all this infrastructure and configuration, there’s an easy way out … Migrate your TFS on-premises environment to VSTS! 🙂

Advertisements

Upgrade TFS Team Project features

April 20, 2017

When upgrading TFS, the existing Team Projects won’t automatically adopt the new features of the new TFS version. Some of the new features might require some updates to the Team Project. Note that this will only be required for TFS Team Projects … VSTS Team Projects are automatically updated with each service upgrade.

You can perform this update yourself via the Configure Features wizard. If the Configure Features link is visible for your Team Project, it means that the Team Project requires an update. Otherwise, the new features are already enabled.

alm_cfw_configfeatures

This might work if you don’t have a lot of Team Project Collections and Team Projects. During a recent upgrade to TFS 2017 Update 1 at a customer, I was confronted with 31 Team Project Collections and in total a bit more than 400 Team Projects. No way I was going to hit the configure features link 400 times …

I remember having done this already programmatically in the past (https://www.visualstudio.com/en-us/docs/work/customize/configure-features-after-upgrade#program-updates), but the issue now was that there wasn’t a ready-to-use solution for TFS 2017 Update 1. So, I used some tips & tricks from https://www.visualstudio.com/en-us/docs/work/customize/configure-features-after-upgrade#program-updates and also the Features4tfs CodePlex solution was a good starting point. I wanted to have a scenario where it’s possible to scan a complete TFS 2017 environment with all on-line Team Project Collections and all available Team Projects.

As a result, you can find my solution in Github: https://github.com/pietergheysens/TFSUpgradeTeamProjectFeatures. Because it worked for me with TFS 2017 (Update 1), it doesn’t mean it will work for you. Please test it first during a trial-upgrade and see if it helps you to upgrade your Team Projects in one go.


TFS Production upgrades: stay calm and stick to the plan!

March 4, 2017

When planning a big migration upgrade to TFS 2017 (from TFS 2013) on new hardware, the exact planning of all actions can be very important to make sure the downtime of the TFS environment can be as short as possible and there’s at least some buffer to fix unexpected issues. That’s why I always try to perform production migrations in a week-end and that’s why you should always run a trial migration to have an idea about the total duration.

For this specific migration I’m doing this week-end, TFS 2017 is only an intermediate step because the customer also wants to migrate to VSTS from the TFS 2017 environment. I managed to do this without any issues during a trial run.

So, the plan for the production migration was: bringing the TFS environment offline on Friday evening and already launching the TFS 2017 upgrade wizard on Friday evening to make sure the long upgrade process can continue to run during the night. During the trial upgrade, this process took about 4 hours.

Unfortunately when logging back in on Saturday morning, I noticed the upgrade process failed after more than 3 hours (step 1523 of 1621) due to error TF30042: The log file for the database is full.

UpgradeError

The dedicated log disk on the server was indeed full. Seeing this error might freak you out because first you will believe that the complete upgrade failed and you need to start all over again. This might jeopardize the full plan to have a working VSTS environment on Monday morning.

This is for sure a moment to stay calm and to properly assess the situation and read all text which is available for you in the log file and also have a good look at the warning message in the TFS Upgrade wizard:

One or more project collections failed to upgrade … Start the Administration Console and navigate to the Team Project Collections node to attempt retrying the upgrade for each failed collection.

No need to start all over again! Fix the error which can be found in the error log and try to resume the upgrade process. In my situation I had to clean up the dedicated log file disk before rerunning the job from the TFS Administration Console.

ResumeUpgradeProcess

And indeed, the upgrade process resumed from step 1523 …

I only lost a bit of processing time, but still ok to finish the complete upgrade process before Monday morning …

Having done about 50 TFS upgrades in the last couple of years, I never had to cancel a production upgrade. I always delivered the new environment on time. Of course, there were times were unexpected issues came up or where I needed to perform some aftercare when the new environment was already up-and-running.

Rule #1: always have a backup plan in case of a hard failure

Rule #2: stay calm and properly assess the situation

Rule #3: call help before doing crazy stuff in a production environment