Saturday, July 4, 2015

DevOps and PowerShell – Test automation with TFS API

Using the combination of TFS API’s and PowerShell, you can easily manage your TFS test cases, either as part of a build process or from a tool that supports invoking PowerShell cmdlets or scripts. In this post, I’ll show how to make use of the [Microsoft.TeamFoundation.TestManagement.Client] API’s to manage test runs in TFS or Microsoft Test Manager. The test management client API’s are internally used by MS visual studio Ultimate/ Test Professional editions to manage TFS test plans, and cases.
At the end we’ll create a PowerShell module with the functions that are needed for managing the test runs in TFS and use the functions to meet our requirements. The rest of the post contains sections defining the functions that are exposed as part of the Module.

Assemblies involved in the example:

The first step is to load the assemblies which are needed for our functions to interact with the TFS API’s

[void][System.Reflection.Assembly]::LoadWithPartialName("Microsoft.TeamFoundation.Client"
[void][System.Reflection.Assembly]::LoadWithPartialName("Microsoft.TeamFoundation.TestManagement.Client"
[void][System.Reflection.Assembly]::LoadWithPartialName("Microsoft.TeamFoundation.Build.Client"

The Microsoft.TeamFoundation.Build.Client assembly is used to get the details of the latest build that will be used to create a test run.

Get-TeamProjectCollection

In TFS a team project collection is used to group multiple team projects that needs a common set of resources or have a similar set of requirements or objectives.  A team project collection can be retrieved by using the GetTeamProjectCollection method on the TfsTeamProjectCollectionFactory class.

Function Get-TeamProjectCollection
{
      [CmdletBinding()]
      param
      (
            [Parameter()]
            [ValidateNotNull()]
            [Uri]$Uri
      )

      if(-not($Uri))
      {
            $Uri = "http://yourtfsserver:8080/tfs/defaultcollection"
      }

      [Microsoft.TeamFoundation.Client.TfsTeamProjectCollectionFactory]::GetTeamProjectCollection($Uri)     

}

PS C:\> Get-TeamProjectCollection


Name                                    : tfsprajeesh\DefaultCollection
DisplayName                             : tfsprajeesh
CatalogNode                             : CatalogNode instance 59865478
                                            ChangeTypeValue: 0
                                            IsDefault: False
                                            MatchedQuery: False
                                            NodeDependencies: [0]
                                            NodeDependenciesIncluded: False

Get-TeamProject

A team project is a collection of work items, code, tests, work products, metrics, and so forth that is used by a defined team to track a common set of related work. The logical concept of a team project helps you determine what to include or exclude from the development of a software application.
In our sample, we will make use of the TestManagement API (you can also use the TeamFoundation.Client API to get the team project) to get the TeamProject object.

Function Get-TeamProject
{
      [CmdletBinding()]
      param
      (
            [Parameter(ValueFromPipeline=$true, Position=1)]
            [ValidateNotNull()]
            [Microsoft.TeamFoundation.Client.TfsTeamProjectCollection]$TeamProjectCollection,

            [Parameter(Mandatory=$true, Position=0)]
            [ValidateNotNullOrEmpty()]
            [String]$Name
      )

    $service = $TeamProjectCollection.GetService([Microsoft.TeamFoundation.TestManagement.Client.TestManagementService])
    $service.GetTeamProject($Name)   

}

PS C:\> Get-TeamProjectCollection | Get-TeamProject -Name "TeamProject1"


TestEnvironments             : Microsoft.TeamFoundation.TestManagement.Client.TestEnvironmentHelper
IsValid                      : True
DefaultPageSize              : 200
DefaultPageSizeForTestPoints : 400

Get-TestPlan

A test plan lets you specify what you want to test and how to run those tests. You can create Test plans in Microsoft Test Manager or using the TFS API's. For more details on Test plans and how to create them follow the doumentation at https://msdn.microsoft.com/en-us/library/vstudio/dd286583(v=vs.110).aspx . To get a test plan instance based on the test plan name or test plan id, we'll make use of the TestPlans collection in the TeamProject object as given below.
Function Get-TestPlan
{
      [CmdletBinding()]
      param
(
            [Parameter(Mandatory=$true, ValueFromPipeline=$true, Position=0)]
            [ValidateNotNull()]
        $TeamProject,

            [Parameter(Position=1)]
            $Id,

        [Parameter(Position=2)]
        $Name
      )

    if($Id)
    {
        $TeamProject.TestPlans.Find($Id)
    }
    else
    {
        $TeamProject.TestPlans.Query("SELECT * FROM TestPlan WHERE PlanName='$Name'")
    }
}

PS C:\> Get-TeamProjectCollection | Get-TeamProject -Name "TeamProject1" | Get-TestPlan -Name "TestPlan1"


ObjectType                 : TestPlan
Name                       : TestPlan1
Description                :

Get-TestSuite

By using Microsoft Test Manager, you can organize test cases into a hierarchy of test suites inside test plans. For more information on test suites, follow the documentation at https://msdn.microsoft.com/en-us/library/vstudio/dd286738(v=vs.110).aspx. We can make use of the TestSuites collection of the TeamProject object to query a test suite based on the Id.
Function Get-TestSuite
{
      [CmdletBinding()]
      param
      (
            [Parameter(Mandatory=$true, Position=0, ParameterSetName="TeamProject")]
            [ValidateNotNull()]
        $TeamProject,

            [Parameter(Mandatory=$true,  ParameterSetName="TeamProject")]
            [Parameter(Mandatory=$false,  ParameterSetName="TestPlan")]
            [ValidateNotNull()]
            $Id,

            [Parameter(Mandatory=$true, ValueFromPipeline=$true, ParameterSetName="TestPlan")]
            [ValidateNotNull()]
            [Microsoft.TeamFoundation.TestManagement.Client.ITestPlan]$TestPlan
    )

    switch($PsCmdlet.ParameterSetName)
    {
        "TeamProject"
        {
            $TeamProject.TestSuites.Find($Id)
            break;
        }
        "TestPlan"
        {
            $TestPlan.Project.TestSuites.Find($Id)
            break;
        }
    }   
}

PS C:\> Get-TeamProjectCollection | Get-TeamProject -Name "TeamProject1" | Get-TestPlan -Name "TestPlan1" | Get-TestSuite -Id  2


TestSuiteType          : StaticTestSuite
ObjectType             : TestSuite
Plan                   : TestPlan instance 23

Get-TestEnvironment

A lab environment is a group of computers that you manage as a single entity. Usually you use them for system testing. If you’re testing a distributed application such as a web app, you can perform realistic tests by deploying each component on a separate machine.
Lab environments let you collect diagnostic data from all the machines in the lab while you’re performing your tests. The data, such as event logs or Intellitrace files, are attached to the test results and to any bug that you create. For more details on Lab environments refer to the documentation https://msdn.microsoft.com/en-us/library/ee943321.aspx. The TeamProject has a collection of TestEnvironments which we can Query the Environment by name or id
Function Get-TestEnvironment
{
      [CmdletBinding()]
      param (
            [Parameter(Mandatory=$true, ValueFromPipeline=$true, Position=0)]
            [ValidateNotNull()]
        $TeamProject,

            [Parameter(Position=1)]
            $Name
    )

      $TeamProject.TestEnvironments.Query() | ? { $_.Name.EndsWith($Name) }
}

PS C:\> Get-TeamProjectCollection | Get-TeamProject -Name "TeamProject1" | Get-TestEnvironment -Name "Env1"


Id                      : bgvf6fb7-96f3-483c-4356-078148cf45de
Name                    : TeamProject1.0.Env1
Description             :


Get-TestConfiguration

Test configurations will help users to test applications with different configurations like web browsers, different operating systems etc. A detailed description about test configurations is provided here https://msdn.microsoft.com/en-us/library/vstudio/dd286643(v=vs.110).aspx. The TestConfigurations collection property in the TeamProject can be used to find and use a specific configuration during a test run.

Function Get-TestConfiguration
{
      [CmdletBinding()]
      param
    (
            [Parameter(Mandatory=$true, ValueFromPipeline = $true)]
            [ValidateNotNull()]
        $TeamProject,
     
            [Parameter(Mandatory=$false,  ParameterSetName="Id", Position = 1)]
            [int]$Id,
     
            [Parameter(Mandatory=$false,  ParameterSetName="Name", Position = 1)]
            [string]$Name
    )

    switch($PsCmdlet.ParameterSetName)
    {
        "Id"
        {
            $TeamProject.TestConfigurations.Find($Id)
            break;
        }
        "Name"
        {
            $TeamProject.TestConfigurations.Query("SELECT * FROM TestConfiguration WHERE Name='$Name'")
            break;
        }
    }
}

PS C:\> Get-TeamProjectCollection | Get-TeamProject -Name "TeamProject1" | Get-TestConfiguration -Name "Internet Explorer Browser"


ObjectType        : TestConfiguration
Name              : Internet Explorer Browser
Description       : Use this configuration to run an automation test in internet explorer
AreaPath          : TeamProject1
IsDefault         : False

Get-TfsBuild

During a test run, it’s important to specify the TFS build to be used that contains the test methods that the tests are executed against. A team build can be queried using the Microsoft.TeamFoundation.Build.Client API, by providing the TeamProject and Build definition name.
Function Get-TfsBuild
{
      [CmdletBinding()]
      param
     (
            [ValidateNotNull()]
            [Microsoft.TeamFoundation.Client.TfsTeamProjectCollection]
            $TeamProjectCollection,

            [Parameter(Mandatory=$true)]
            [ValidateNotNullOrEmpty()]
            [string]
            $TeamProject,

            [Parameter(Mandatory=$true)]
            [ValidateNotNullOrEmpty()]
            [string]
            $BuildDefinition,

            [string]
            $BuildNumber
      )

      $server = $TeamProjectCollection.GetService([Microsoft.TeamFoundation.Build.Client.IBuildServer])
    if($BuildNumber)
    {
        $server.QueryBuilds($TeamProject, $BuildDefinition) |? {$_.BuildNumber -eq $BuildNumber}
    }
    else
    {
        $server.QueryBuilds($TeamProject, $BuildDefinition) |? {$_.Status -eq "Succeeded"} | Select -Last 1
    }   
}

PS C:\> Get-TfsBuild -TeamProjectCollection $collection -TeamProject $teamProject.TeamProjectName -BuildDefinition BlogsPrajeesh.Samples.Web_CI | select BuildNumber

BuildNumber
-----------
1.0.0.5  


Invoke-TestRun

A test run is the execution of test cases in a test suite or test case, to be executed against a specific configuration in an test environment. You also need to mention a TFS build to be used for the test run. For details on how to run tests in Microsoft test manager follow the article https://msdn.microsoft.com/en-us/library/dd286680(v=vs.110).aspx.

In our function, we will make use of the output from the previous functions created in this post to start a test run. Later we’ll create functions to get the result of a test run.
Function Invoke-TestRun
{
      [CmdletBinding()]
      param(
            [Parameter(Mandatory=$true)]
            [ValidateNotNull()]
            $TeamProject,

            [Parameter(Mandatory=$true)]
            [ValidateNotNull()]
            [string]$Title,

            [Parameter(Mandatory=$true)]
            [ValidateNotNull()]
            $TestPlan,

            [Parameter(Mandatory=$true)]
            [ValidateNotNull()]
            $TestSuite,

            [Parameter(Mandatory=$true)]
            [ValidateNotNull()]
            $TestConfiguration,

            [Parameter(Mandatory=$true)]
            [ValidateNotNull()]
            $TestEnvironment,

            [Parameter(Mandatory=$true)]
            [ValidateNotNull()]
            [Microsoft.TeamFoundation.Build.Client.IBuildDetail]$Build
      )

      $TestRun = $TestPlan.CreateTestRun($true)
    $TestRun.BuildUri = $Build.Uri
    $TestRun.BuildNumber = $Build.BuildNumber
    $TestRun.BuildDirectory = $Build.DropLocation
    $QueryText = "SELECT * FROM TestPoint WHERE SuiteId=$($TestSuite.Id) AND ConfigurationId=$($TestConfiguration.Id)"
    $TestPoints = $TestPlan.QueryTestPoints($QueryText)
    $TestPoints | ? { -not($_.State -eq "NotReady" -and $_.MostRecentResultOutcome -eq "Blocked") -and $_.MostRecentResultOutcome -ne "NotApplicable" } | % { $TestRun.AddTestPoint($_, $null) }
    $TestRun.Title = $Title
    $TestRun.Owner = $TestTeamProject.TestManagementService.AuthorizedIdentity
    $TestRun.TestEnvironmentId = $TestEnvironment.Id
    $TestRun.Controller = $TestEnvironment.ControllerName
    $TestRun.Save()
$TestRun
}
PS C:\> Invoke-TestRun -TeamProject $teamProject -Title "Demo Testrun2" -TestPlan $plan -TestSuite $suite -TestConfiguration $config -TestEnvironment $environment -Build $build


ObjectType            : TestRun
BuildDirectory        : \\TFSSER\TRUNK\1.0.0.5
State                 : Waiting
PostProcessState      : Complete
DateDue               : 1-1-0001 0:00:00
Statistics            : Microsoft.TeamFoundation.TestManagement.Client.TestRunStatistics
ErrorMessage          :
Iteration             : TeamProject1
LegacySharePath       :
Type                  : Unspecified
IsAutomated           : True
Version               : 1000
IsBvt                 : False
TotalTests            : 0
IncompleteTests       : 0
NotApplicableTests    : 0
PassedTests           : 0
UnanalyzedTests       : 0
TestMessageLogEntries : {}
Title                 : Demo Testrun2
BuildUri              : vstfs:///Build/Build/5
BuildNumber           : 1.0.05
BuildConfigurationId  : 0
BuildPlatform         :
BuildFlavor           :
DateCreated           : 1-1-0001 0:00:00
DateStarted           : 1-1-0001 0:00:00
DateCompleted         : 1-1-0001 0:00:00
LastUpdated           : 4-7-2015 9:33:00
LastUpdatedBy         : TeamFoundationIdentity instance 34352

Receive-TestRun

The receive test run method gets the results of a test run at a later point of time by providing a TestRun Id.

Function Receive-TestRun
{
      [CmdletBinding()]
      param(
            [Parameter(Mandatory=$true, Position=0, ParameterSetName="TeamProject")]
            [ValidateNotNull()]
            $TeamProject,

            [Parameter(Mandatory=$false, ParameterSetName="TestPlan")]
            [Parameter( ParameterSetName="TeamProject")]
            [int]$Id,

            [Parameter(Mandatory=$true, ValueFromPipeline=$true, ParameterSetName="TestPlan")]
            $TestPlan
      )

    switch($PsCmdlet.ParameterSetName)
    {
        "TeamProject"
        {
            $TeamProject.TestRuns.Find($Id)
            break;
        }
        "TestPlan"
        {
            $TestPlan.Project.TestRuns.Query("SELECT * FROM TestRun WHERE TestPlanId=$($TestPlan.Id) AND TestRunId=$Id")
            break;
        }
    }      
}

Wait-TestRun

The wait run is invoked in conjunction with the Invoke-TestRun, to wait till the test run is completed. This is useful when you are using the Invoke-TestRun method as part of a Tfs Build activity.

Function Wait-TestRun
{
      [CmdletBinding()]
      param
      (
            [Parameter(Mandatory=$true, ValueFromPipeline=$true)]
            [ValidateNotNull()]
            [Microsoft.TeamFoundation.TestManagement.Client.ITestRun]$TestRun
      )
   
    do
    {
            Start-Sleep -Seconds 5
            $TestRun = Receive-TestRun -Id $TestRun.Id -TeamProject $TestRun.Project
      }while($TestRun.State -ne "Completed" -and $TestRun.State -ne "NeedsInvestigation" -and $TestRun.State -ne "Aborted")

      $TestRun
}

$run = Invoke-TestRun -TeamProject $teamProject -Title "Demo Testrun2" -TestPlan $plan -TestSuite $suite -TestConfiguration $config -TestEnvironment $environment -Build $build | Wait-TestRun 



That will be the last method in our TeamFoundation.TestManager module. The functions in this module can be used in various stages of your Continuous Delivery pipeline specially during test execution phase. If you want the source code of the module, I’ve added this to my GitHub project at https://github.com/prajeeshprathap/Powershell-PowerPack/blob/master/TeamFoundationServer.TestManager.psm1