Jenkins CI with Xamarin.iOS, Xamarin Test Cloud and TestFlight – Part 2

In Part 1 we took care of all the prerequisites of preparing a clean Mac Mini as our Jenkins build server. Now we are ready to set up Jenkins and configure our jobs.

The Goal

It’s good to summarize what we want to accomplish here. I’m not going into the merits of a CI server and why you should have one. Just trust me, you need one. If you want to read on the basics, see this Xamarin guide.

We want to automate the BuildTestDeploy cycle for our Xamarin.iOS app, let’s call it MyProject from now on. It’s considered good practice to split up these tasks in separate jobs, both for maintainability and to limit job¬†execution time. This way you can manage and track progress for each job separately. Separate jobs have its downsides too, but we will come to that. We are going to create 3 jobs:

  1. MyProject (Build)
  2. MyProject-testcloud (Test)
  3. MyProject-testflight (Deploy)

The MyProject job will trigger the jobs MyProject-testcloud and MyProject-testflight only when the build succeeds. This is what we call Downstream projects. Notice that we are deploying to TestFlight right after the build succeeds, we are not waiting for the Test Cloud tests to finish. This is because I want to deploy to TestFlight regardless of the Test Cloud results as TestFlight is used by our developers and manual testers.

The starting point for setting up Jenkins is the Xamarin guide: Using Jenkins with Xamarin. I suggest you follow that guide first until the section: Setting up a Job. Then come back to this guide. I’ll wait…

Ok, good you’re back, let’s continue. ūüôā

A few things on the Xamarin guide. You’ll notice that the guide shows a pretty old version of Jenkins, version 1.531 from September 2013. At the time of this writing Jenkins is at 1.598 and lots of things have changed, including the user interface.

Second, when I followed the instructions on Configuring the MSBuild Plugin, I was presented with this warning:

jenkins-msbuild

Not sure why it gives me this warning, but don’t worry about it, it works nonetheless.

Plugins

Before I start creating the first job, here’s a list all of the plugins that we will need later on in our jobs. When you first install Jenkins, there’s just a few plugins available out of the box. But you know the standard answer to the question: Does Jenkins support X, Y, Z? There’s a plugin for that!

https://wiki.jenkins-ci.org/display/JENKINS/MSBuild+Plugin

https://wiki.jenkins-ci.org/display/JENKINS/EnvInject+Plugin

https://wiki.jenkins-ci.org/display/JENKINS/Copy+Artifact+Plugin

https://wiki.jenkins-ci.org/display/JENKINS/JUnit+Plugin

https://wiki.jenkins-ci.org/display/JENKINS/Workspace+Cleanup+Plugin

https://wiki.jenkins-ci.org/display/JENKINS/BitBucket+Plugin

https://wiki.jenkins-ci.org/display/JENKINS/Parameterized+Trigger+Plugin

https://wiki.jenkins-ci.org/display/JENKINS/Testflight+Plugin

https://wiki.jenkins-ci.org/display/JENKINS/Upstream+Downstream+View+Plugin

https://wiki.jenkins-ci.org/display/JENKINS/Email-ext+plugin

https://wiki.jenkins-ci.org/display/JENKINS/Dashboard+View

Build

The MyProject job will build the Xamarin.iOS solution and package the artefacts for further use. We’ll start by creating a Freestyle project MyProject. To do this, click on New Item on the Jenkins dashboard and you’ll see this page.

jenkins-myproject

Enter the name and click OK and you will be brought to the Configuration page of the job.

General

jenkins-parameter1

jenkins-parameter2

We are creating a Parameterized build and specify two Choice parameters: CONFIGURATION and DEVICE. We will use those parameters later in our build script.

Source Code Management

We are using BitBucket as our Git source code repository, so here we enter the repository url to our BitBucket repository. You can add the Credentials locally, but it’s easier to configure the credentials globally in Manage Jenkins -> Manage Credentials and then refer to them here.

Furthermore you can specify which branch to build and as an additional behavior I’ve set it to Clean before checkout, because I want to make sure that every time I run this job it starts fresh and there’s no leftovers from previous builds.

jenkins-scm

 

Build Triggers

You could use the BitBucket+Plugin to only trigger the build when a change is pushed to BitBucket. However, in our case Jenkins is not publicly accessible to BitBucket (which lives in the cloud) cannot reach it. Therefore we will do a simple Poll SCM every 15 minutes. If you want to know how H/15 * * * * translates to “every 15 minutes”, click on the question mark behind the field and read the explanation. If you understand it, please drop me an email and explain it to me ūüėČ

jenkins-buildtriggers

 

Build

Now we arrive at the meat of the matter, the actual build. For this I use a bash script that I adapted from https://nicolasgoles.com/blog/2011/08/continuous-integration-with-jenkins/.

# !/bin/sh
# By Anonymous user on Stack Overflow
# Freely translated from https://nicolasgoles.com/blog/2011/08/continuous-integration-with-jenkins/

 function fail {
    echo "$*" >&2
    exit 1
}

function section_print {
    echo "\n=== $* ==="
}

if [ -z $CONFIGURATION ]; then
    fail "No configuration specified";
    exit 1;
fi

if [ -z $DEVICE ]; then
    fail "No device specified";
    exit 1;
fi

# Put your project folder here
PROJECT_FOLDER=MyProject

#Put your solution file here
SOLUTION_FILE=MyProject.sln

BUILD_PATH=$PROJECT_FOLDER/bin/"$DEVICE"/"$CONFIGURATION"

section_print "Updating Build Number"
cd "$WORKSPACE"
VERSION_NUMBER=$(/usr/libexec/PlistBuddy -c "Print CFBundleShortVersionString" $PROJECT_FOLDER/Info.plist)
VERSION=$VERSION_NUMBER.$BUILD_NUMBER
/usr/libexec/PlistBuddy -c "Set :CFBundleVersion $VERSION" $PROJECT_FOLDER/Info.plist

section_print "Updating App Name"
/usr/libexec/PlistBuddy -c "Set :CFBundleDisplayName Mobile CRM $BUILD_NUMBER" $PROJECT_FOLDER/Info.plist

section_print "Restoring nuget packages"
nuget restore $SOLUTION_FILE

section_print "Building $CONFIGURATION"
/Applications/Xamarin\ Studio.app/Contents/MacOS/mdtool -v build "--configuration:$CONFIGURATION|$DEVICE" $SOLUTION_FILE || fail "Build failed"

# Get the .app and .ipa file names
cd "$BUILD_PATH"
for file in "*.app"
do
    APP_NAME=`echo $file`
done
APP_NAME=${APP_NAME%.*}

for file in "*.ipa"
do
    IPA_FILE=`echo $file`
done

section_print "Compressing dSYM"
DSYM_FILE="$APP_NAME-$VERSION.dSYM.zip"
zip -r $DSYM_FILE "$APP_NAME.app.dSYM"

section_print "Removing old artefacts from Workspace folder"
cd "$WORKSPACE"
rm -f *.ipa
rm -f *.dSYM.zip
rm -rf *.dSYM

section_print "Copying artefacts"
cd "$BUILD_PATH"
cp -v "$IPA_FILE" "$WORKSPACE/." || fail "Failed to copy ipa"
cp -v "$DSYM_FILE" "$WORKSPACE/." || fail "Failed to copy dSYM zip"
cp -r "$APP_NAME.app.dSYM" "$WORKSPACE/$APP_NAME.app.dSYM/" || fail "Failed to copy dSYM"

section_print "Build succeeded"

I’ll give some explanation on the script.

First we are updating the iOS Build number with the Jenkins build number. This way we can always relate a TestFlight build and a Test Cloud test run with the exact Jenkins job number. This is very important for traceability.

We also update the app Display name to also show the build number. Again, easy to see which build you are testing when you look at the app on your device. Another option of course is to put the build number somewhere in an about or settings screen.

Then we are restoring the NuGet packages. This caused me some headaches in the beginning because not all NuGet packages were restored. After some digging I found that there are a few ways to restore NuGet packages in a solution as described in this article NuGet Package Restore. It turned out that most of the projects were using MSBuild-Integrated Package Restore, but one project in the solution was missing this reference and didn’t restore NuGet packages causing the build to fail.

In the same article the NuGet team suggests to use the Command-Line Package Restore instead. For this you first have to remove the MSBuild-Integrated Package Restore which is described in this article. After that you can just do

nuget restore MyProject.sln

as described here.

After that we are building the solution passing the $CONFIGURATION and $DEVICE parameters to the script.

In the next steps, we are dynamically getting the app names for later use in the script, compressing the dSYM folder which we will need to submit to TestFlight, removing any old artifacts and copying the new artifacts to the $WORKSPACE folder.

Post-build Actions

Now that the build is done, we need to trigger some Post-build Actions. The first one is to Archive the artifacts. We will archive the ipa file, the dSYM folder (for Test Cloud) and the zipped dSYM folder (for TestFlight). There are 3 reasons to archive these files:

  1. So that they are easily available from the Jenkins web page if someone wants to download them.
  2. So that they can be passed on to the downstream jobs, in this case the MyProject-testflight job.
  3. This one is a bit tricky, but we will need it for the next Post-build action: Aggregate downstream test results. For this purpose we also need to Fingerprint all archived artifacts.

jenkins-postbuild-archiveartefacts

 

The next post-build action is Aggregate downstream test results. This will collect all the test results from downstream jobs, in this case MyProject-testcloud and present them in this job, which contains the build under test. Now you can easily see the test results in one glance.

Getting this to work however wasn’t obvious. At first, the upstream job always reported 0 test results from the downstream jobs. But as so many times, StackOverflow to the rescue. This post describes the answer for this problem: Aggregating results of downstream is no test in Jenkins. The thing here is that for this to work, both jobs need to have the same fingerprinted artifacts!

jenkins-postbuild-aggregateresults

 

The third post-build action is Editable Email Notification using the Email-ext+plugin. For this to work you first have to make sure that your E-mail Notification settings are set up correctly in Manage Jenkins -> Configure System.

I use this post-build action to send an email to the developers on the following triggers: Failure – Any and Fixed. The first email is a gentle reminder to fixed the build they broke and the second email is a Thank You for fixing the build :). There are lots of options in this plugin so be sure to read the documentation!

jenkins-postbuild-email

 

The final post-build action is maybe the most crucial one, namely triggering the downstream builds. We are not using the standard trigger for this, but the Trigger parameterized build on other projects from the Parameterized+Trigger+Plugin. This way we can pass some parameters to the downstream jobs.

We are triggering both the TestFlight and Test Cloud jobs when the build is stable and we’re passing the workspace folder and build number of the upstream job.

jenkins-postbuild-triggerbuilds

And this concludes the configuration of the MyProject Build job. Now on to the Test and Deploy jobs!

Test with Xamarin Test Cloud

The MyProject-testcloud project will also be a Freestyle project, so create it the same way you created the MyProject job. However, this will be a much simpler project. You only need 2 Build steps and 1 Post-build Action.

Build

The first Build steps is Copy artifacts from another project. The ONLY reason we need this is for the Aggregate downstream test results from the MyProject job to work.

jenkins-myproject-testcloud-copyartifacts

You need to Fingerprint Artifacts because the two jobs need to have the same fingerprinted artifacts for this to work.

The second Build step is an Execute shell script that uploads the ipa file, the Xamarin.UITest dlls and (optionally) the dSYM file to Xamarin Test cloud. What’s important here is the $UPSTREAM_WORKSPACE parameter that we passed from the MyProject job using the Parameterized+Trigger+Plugin. This will give us the path to the MyProject workspace folder. The reason we need this is that the uploader executable¬†Xamarin.UITest.0.6.8/tools/test-cloud.exe is located in the MyProject workspace and not in the MyProject-testcloud workspace.

function section_print {
    echo "\n=== $* ==="
}

# Parameter passed from the upstream job that built the iOS app
cd "$UPSTREAM_WORKSPACE"

for file in "*.ipa"
do
    IPA_FILE=`echo $file`
done

for file in "*.dSYM"
do
    DSYM_FILE=`echo $file`
done

# Path to the Xamarin.UITest NuGet package
UPLOADER_PATH=packages/Xamarin.UITest.0.6.8/tools/test-cloud.exe

# Location of the test project dlls
TEST_DIRECTORY=MyProject.UITests/bin/"$CONFIGURATION"

API_KEY='your-api-key'
APP_NAME='your-app-name'
TEST_SERIES='master'
LOCALE='en_US'
DEVICE_SET='4cd087d5' #iPhone 8.1 devices

section_print "Uploading to XTC on device set: $DEVICE_SET"
mono $UPLOADER_PATH submit $IPA_FILE $API_KEY --devices $DEVICE_SET --series $TEST_SERIES --locale $LOCALE --app-name $APP_NAME --assembly-dir $TEST_DIRECTORY --dsym $DSYM_FILE --nunit-xml "$WORKSPACE/report.xml"

You need to fill in a few parameters such as the App name, the API key and the Device set. The way to get these values is to login to the Xamarin Test cloud portal and create a New Test Run. This will guide you through a wizard and at the end of the wizard you will receive a sample script that you can use.

testcloud-newtestrun

 

Select New iOS app

testcloud-newiosapp

In this screen you can select the devices that you want to run your test on. You can use the left side to limit the devices that you want to see. In our case we want to run our test only on iOS 8.1.3 Phones. Then select them on the right.

testcloud-selectdevices

After that you can choose some settings, e.g. the Test series and the Locale.

testcloud-testseries

 

Finally you are presented with a screen that shows you a script based on the options you chose in the wizard. E.g. all iPhone devices with iOS 8.1.3 results in a device set of d8c3da75. It will also show your API key (which you need to keep secret obviously).

To be able to report back the test results, you need to pass the parameter –nunit-xml filename. In our case we choose¬†–nunit-xml $WORKSPACE/report.xml. We will need this in our last step.

testcloud-done

Post-build Actions

The final step in this job is a Post-build action Publish NUnit test result report. Here we need to refer to the report.xml file that we used in our shell script.

jenkins-myproject-testcloud-publishreport

 

That’s it. We’ve uploaded our app and the test scripts to Xamarin Test Cloud and it will report back the results and show them in a nice chart. Yeah, I know, my test results are not that great yet, but you get the picture right? ūüôā If you want to learn more about the subject read the article Submitting Tests to Xamarin Test Cloud.

jenkins-myproject-testcloud-results

 

We’ve come to our last step.

Deploy with TestFlight

The final step is to deploy our app to TestFlight so that other developers and QA can run manual tests on it. For this we create another Freestyle job MyProject-testflight. This project is triggered in parallel with the MyProject-testcloud job, so doesn’t wait for it to finish, unless the Jenkins job queue is full.

The main part of this job is handled by the Testflight+Plugin. This plugin takes an ipa and dSYM file and uploads them to TestFlight. The first step is to create Test Flight Tokens in Manage JenkinsConfigure System.

jenkins-testflight-tokens

 

You can get these tokens from your TestFlight portal at https://www.testflightapp.com/api/doc/. You need an API token and a team token.

Unfortunately, you’ll notice another, rather important, message at the top of the page. Apple recently bought TestFlight and is now shutting down the standalone TestFlight service. ūüôĀ Soon we need to start using the new TestFlight Beta Testing in iTunes Connect. That means there’s a Part 3 coming where I will replace the TestFlight service with the new iTunes Connect version.

testflight-api

 

Build Environment

First we are going¬†to prepare our build environment. I’m checking the¬†Delete workspace before build starts check box because I want to make sure there’s no leftover files in my workspace before I upload to TestFlight. When I didn’t do this the plugin would upload all the ipa and dSYM files it could find in the workspace folder and each run that became more and more.

Build

The next step is to¬†Copy artifacts from another project¬†just like we did in the MyProject-testcloud job. This time we need it to have the ipa and dSYM files in the workspace folder because that’s where Testflight+Plugin looks for them.

jenkins-myproject-testcloud-copyartifacts

 

Next we execute a shell script to get the changelog from the upstream project. We need this because we want to put the original changelog from MyProject in the build notes of the TestFlight upload. Unfortunately the changelog is not exposed as a build parameter so we need to get it in another way. Luckily Jenkins has an API that you can call and we can get to the changelog, thanks to this post on SO. To get access to the correct upstream build that triggered this job we use the $UPSTREAM_BUILD_NUMBER parameter that was passed.

# Based on http://stackoverflow.com/questions/11823826/get-access-to-build-changelog-in-jenkins
# Get changelog and format it properly
CHANGELOG=$(curl "http://localhost:8080/job/MyProject/$UPSTREAM_BUILD_NUMBER/api/xml?wrapper=changes&xpath=//changeSet//comment" | sed -e "s/<\/comment>//g; s/<comment>/* /g; s/<\/*changes>//g" | sed '/^$/d;G')

# Write result to properties file
echo CHANGELOG=$CHANGELOG > build.properties

Also, we are writing the CHANGELOG variable to a properties file. Why do we need this? Because the TestFlight upload happens in a different Post-build Action step and that step does not have access to the variables in this bash script. Therefore, we store it in a properties file and then use the EnvInject+Plugin to turn the contents of the properties file into Environment variables that can be used throughout the job. We will do this in the next step.

jenkins-testflight-envinject

Post-build Actions

The final step is to upload the files to TestFlight. You need to fill in the name of the Token Pair that you created in the Jenkins global configuration. We are not specifying a particular ipa or dSYM file. That means that the plugin will just look for all ipa files and matching dSYM files in the workspace folder. That’s the reason we used the¬†Delete workspace before build starts step before.

In the build notes we can now reference the $CHANGELOG environment variable to put the changelog in the Build Notes. Do not check the Append changelog to build notes check box because that will append the changelog of the current MyProject-testflight job instead of the job that actually did the build and has the real changelog, the MyProject job.

jenkins-testflight-upload

 

Dashboard

To top things off we use the Dashboard-View plugin to create a nice dashboard on our Jenkins homepage that shows our jobs and test results in one view.

jenkins-dashboard

Conclusion

I hope I was able to show you that Jenkins in combination with Xamarin Test Cloud and TestFlight are a great team and can bring great value to a development team to automate building, testing and deploying of mobile apps and to minimize the feedback loop between development and test.

Developers will get immediate feedback when their commit broke the build, but also when they fixed the build. Developers and QA get immediate notification when a new build is available on TestFlight, including the changelog and the build number of that build so that they can reference the Jenkins job. Automated UI test results are published to the Jenkins dashboard, but notifications are also sent to all members in Xamarin Test Cloud. These are just a few of the advantages of this system.

Watch out for Part 3 where I will replace TestFlight with the iTunes Connect version of TestFlight.

Happy Coding!