Unity3d: from commit to deployment onto tester devices in 20 min using Jenkins

In our goal to automate most of the development process, at We Want To Know, we've set up a Jenkins build server that continuously generates our builds and distributes them to our testers.

There are quite a few pages on the Web to explain how to make testflight deployment for xcode projects. Yet I haven't found a page that describes step by step instructions to set up a build pipeline for unity3d projects, which bear a few differences and caveats.

So here are some notes that describe the current setup of our CI server, the tradeoffs, and some of the future improvements. We use Jenkins, but you can reuse most of the notes here to set up your own solution.


As you will see, most of the complexity are related to the setup of iOS projects. If you are starting from scratch, don't know about jenkins and don't need iOS deployment, you should still be capable of having a CI server in a few hours. Configuring the iOS target alone will probably take you as long as the rest, because of the several manual operations involved. Because of this I recommend you to read this document once fully before you start your unity3d automation journey.

This setup assumes you have code stored in a SCM (subversion/git/mercurial/Perforce/... whatever is your kick). To store unity projects under SCM, follow the Unity doc. Note that 3.5 adds better support for text meta files and proper separation of ProjectSettings assets. Upgrade recommended!

The setup will drive you to automate the build gradually:

• first from within the editor
  
• second from a command line shell script
  
• third from a jenkins server
  
• and last, in the case of iOS projects:
  
  
  
  • automate the xcode building
    
  • deploy the builds onto your private store
    
  
  

Caveats:

• the current setup will require you to perform setup & maintenance using keyboard & mouse. Make sure to get a proper access to your build server.
  

Overview of the build server

Our server builds unity projects for Mac, Windows, iOS & Android completely automatically. Mac, Windows generate downloadable Zip files, iOS & Android generate IPAs and apk that we can archive and/or send to private stores (such as testflight and appaloosa) for distribution to our testers.

What you need:

• a mac. We are using a mac mini sitting on the desk close to our developers
  
• an apple developer profile. I am currently reusing mine, although you might want to create a separate one and give it developer right to your Company type apple developer profile
  
• a license for your Unity3d editor
  
• a distribution profile for Ad Hoc deployment where you have registered the UDID of your testers
  
• a recent jenkins (>= 1.449 if you want to use the better installer for iOS)
  

Tip: you probably want to create an email address for your build server, you will reuse it to create your apple developer profile (if you chose to use one) and your unity3d license.

0. Preparation

1. install xcode (we use 4.2.1)
   
2. install android sdk (download bare SDK, then run script to download the various targets you will want to use)
   
3. install unity
   
   
   
   • don't forget to put the license, otherwise a pesky dialog will show up.
     
   • if you are on the way to migrate from 3.5 and 3.4, e.g. your SCM contains 3.4 version, make sure to install the same version on your CI server. Otherwise, due to the difference in ProjectSettings and meta files, you might have a problem.
     
   
   
4. if you indend to do iOS builds, for the user that will perform your builds you will need to do a few configuration related to code signing. Note: if that user is jenkins you will need to install jenkins beforehand in order for the installer to appropriately create the user (more information below).
   
   
   
   • install apple developer certificate (create a new one for the build server or export your own developer certificate to pk12 from the keychain on your computer and import into login keychain or server computer)
     
   • install apple distribution profile into xcode (D&D file on xcode icon)
     
   
   

1. Unity3d command line

Let's first automate the running of the Unity Editor

• Get familiar with unity3d automation. You can start with the official command line documentation.
  
• create an Assets/Editor/CIEditor.cs file in which you will write one method per target. I tend to use the PerformXxxBuild() format. See this help.
  
  
  
  • I tend to follow the maven convention for repository layout, so the build will create a target/ directory and place the build files under it
    
  • if you want to provide a single file, you might want to zip your contents. We've added the DotNetZip library to the editor and after the build is performed, we create a zip file to contains all the files
    
  
  
• test the commands from the editor. If you've added the proper annotations, you will have a new menu item, e.g. under Custom / CI / etc..
  
• use the following editor.sh script to test the build from the command line. Note how in case of the failure, the content of the Editor.log is pasted on the standard output.
  

2. Jenkins setup

The following setup describe a centralized server running on a mac. On a larger scale project, you will probably have a master/slave setup with various. Note: Macs aren't yet that friendly to being used as virtualization hosts, even if the latest Lion release supposedly allows it. To run jenkins slaves on mac for unity3d (or plain iOS projects) you will definitively want a recent version of jenkins to pick that option. This allows proper SSH access.

On a mac, use the provided dmg to install jenkins. During the installation process, pick the option that allows the installer to run jenkins as a user not a daemon. This will create a user which will ease the access to the keychain for signing.

If this doesn't work for you, then there are various solutions:

• place the developer certificate system keychain (I haven't tried this)
  
• run jenkins manually from within the gui. sudo su -l jenkins... (you will have to enable the sudo rights for the jenkins user if you chose this solution)
  

Once you've installed jenkins, go the the configuration page and do whatever needs to be done

• secure the server
  
• install plugins. I usually add at least: xcode, unity3d, post build task, apaloosa, testflight, build timeout
  
• configure unity3d on the node. See the plugin's doc.
  

Note unity3d doesn't support opening/building multiple projects simultaneously. To ensure this won't happen, we've reduced the queue to a single simultaneously build. This will be good enough for now, even if it causes a few other drawbacks.

On mac, to have more memory you can do the following:

sudo defaults write /Library/Preferences/org.jenkins-ci heapSize 1024M

3. Unity3d on Jenkins

To automate unity3d on Jennkins you have here several alternatives. On your free style job, you can either

• add a shell script build step that reuses the aforementioned editor.sh script or
  
• use the jenkins unity3d-plugin to add an Unity3d Editor build step. The main benefit of this plugin are that you are able to monitor the console log while the editor is building your project instead of waiting for the full build to be done. With time the plugin will probably have more features, like log parsing, finer error reporting, etc.
  

In this document I will use the jenkins unity3d-plugin.

We will now create jenkins jobs for our server. I usually follow naming convention such as unity_$projectname_$target (which allows me to create views based on regular expressions). I also tend to create one job with good default settings and copy from that one. So even if unity allows you to switch environments on the CI server, and even if your underlying editor script supports this, each job will check out its own version of the code and be focused on one target.

Unity uses a special Temp directory for its build operations. There's no need to clean the target/ directory between each build, except if you do something special. In fact keeping the target directory will come in handy when it comes to iOS target.

Note: building on the various platforms assumes you already have properly configured your Editor for those platforms. These usually result in assets stored as EditorBuildSettings.asset and EditorUserBuildSettings.asset files.

Let's see how this looks like for different targets

4a Automate a Windows or Mac target

This will be very simple. Create a unity_test_win32 freestyle job

• configure a source entry (e.g. SVN), polling period, etc
  
• add a build timeout. For most jobs i guess 15 min is sufficient
  
• add a unity3d-builder step. Follow the plugin's doc.
  
• select archive the artifacts target/*.zip)
  

save your project. Now click Build Now. While building, look at the console output. It should start first checking out then unity will kick in. You will see the Editor.log contents being copied onto the jenkins console output and the unity application will start. When everything is finished, jenkins will have archived your first build artifact.

4b. Automate an Android build target

For android, repeat the same process. The only difference is that on first run, Unity will show up a dialog to let you select the android_sdk path. This should be persisted accross runs. If it doesn't you probably have a setup issue. Check if the user running the jenkins server has a real home directory. So run a couple of builds to make sure all is running as expected.

If this is working, you can add deploy to a private store, like appaloosa as a post build task. For this you will have to make sure your project uses a different build number to make sure the store accepts the new builds. I will come back to that in a later installment.

4c. Automate an iOS build target

Apple greatly restricts your ability to send your build to devices. This would bypass their app store. So you have to sign every single build, using signing certificates and distribution profiles (that restrict your developer profile to maximum 100 devices per year, and every device counted for a year cannot be removed).

So Unity for iOS projects has a special build workflow. You first build for xcode, then you use xcode to build your project.

As there's (yet) no way to create the xcode configuration programmatically, we first need to make sure the configuration is not going to be erased. If you were planning to erase your build target there, against my previous recommendation, be careful!

So after you've configured jenkins to build for iOS, make it perform a build. Unity will generate an xcode project. Stop jenkins (to make sure no Unity3d editor will be spawned) and open the xcode project that was generated under target/. Once opened, perform the following steps to allow for Ad Hoc distribution

• make sure you use the unity-iPhone target as default ! By default without connected iDevice on your server, it seems unity picks up the simulator target. You DO NOT WANT THAT. This can cause very subtle errors.
  
• copy the Release configuration into a new Ad Hoc one. Actually I don't think this is really necessary, but it seems like a good practice.
  
• edit the build settings so that the proper signing certificate is associated for the Ad Hoc build.
  
• edit the scheme for that configuration and make the Release step perform the Archive operation. Check once more that you are using the proper target not the simulator !
  

Save the xcode project and close it.

Open the unity editor, open the project for that particular jenkins job, and build the project for iOS. Unity will detect the existing directory (generated in the first build, the one you just´modified) and will ask you whether you want to replace or append to it. Pick "Append". Once the job is finished, close unity.

Now, whenever your build from Unity, Unity should be appending to the project, alleviating you from the need of modifying the xcode project for Ad Hoc distribution.

Now very important, in order to reproduce this appending operation from the command line, you will have to make sure that in the CIEditor.cs file, the BuildOptions instance used in the method matching your iOS executeMethod argument contain BuildOptions.AcceptExternalModificationsToPlayer. Otherwise Unity will replace the project and destroy your xcode project changes.

Restart jenkins, make a few test builds. You can open the resulting project in xcode in order to test the project configuration.

Now to automate the xcode building, we use the Jenkins xcode plugin. The configuration isn't hard but requires a bit more tuning than standard simple xcode projects, because the project isn't in standard location. In my project, the saved configuration looks like this:

You should be able to deduce easily the required values for your configuration based on the above.

Once you've added the xcode build step, make a build. At the end of the build process, the xcode dialog showing the newly archived IPA will be shown.

Now last step, pretty easy, configure the deployment to your private store. For testflight, enter the proper keys and information. You probably want to add a default distribution list (I use QA), so that some selected devices will always have the ability to receive the latest build.

Add profile automation

If you follow the configuration described earlier.

You will add one separate job to download the apple developer connection offsite and one build step to your iOS build to automatically update the profiles within xcode and within the unity project. Follow the documentation in the previous blog entry.

Summary for Jenkins iOS unity jobs

All in all your jenkins job configuration will look like this:


1. Automatic iOS profile provisionning build step


2. Unity3d Editor build step


3. Xcode build step


4. Archive the artifacts


5. Upload to testflight



When it comes to XML job configuration:

  <builders>
    <hudson.tasks.Shell>
      <command>echo &quot;Updating xcode project file and provisioning profiles&quot;
ADC_DIR=${WORKSPACE}/../../infra_apple_developper_site/workspace/adc/
SITE_JSON=${ADC_DIR}/site.json
xcode_project_file=${WORKSPACE}/target/dragonbox_ios/Unity-iPhone.xcodeproj/project.pbxproj
configuration=&quot;Ad Hoc&quot;
profile_type=&quot;distribution&quot;
profile_name=&quot;TestFlight WWTK All Projects&quot;

cd ${WORKSPACE}/../../infra_apple_developper_site/workspace/xcode/
./xcode_update_pp.sh &quot;${ADC_DIR}&quot; &quot;${SITE_JSON}&quot; &quot;${xcode_project_file}&quot; &quot;${configuration}&quot; &quot;${profile_type}&quot; &quot;${profile_name}&quot;</command>
    </hudson.tasks.Shell>
    <org.jenkinsci.plugins.unity3d.Unity3dBuilder>
      <unity3dName>Unity 3.5</unity3dName>
      <argLine>-quit -batchmode -executeMethod MyEditorScript.PerformIOSBuild</argLine>
    </org.jenkinsci.plugins.unity3d.Unity3dBuilder>
    <au.com.rayh.XCodeBuilder>
      <cleanBeforeBuild>false</cleanBeforeBuild>
      <cleanTestReports>false</cleanTestReports>
      <configuration>Ad Hoc</configuration>
      <target>Unity-iPhone</target>
      <sdk>/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS5.0.sdk/</sdk>
      <symRoot></symRoot>
      <configurationBuildDir>${WORKSPACE}/target/dragonbox_ios/build/Unity-iPhone.build</configurationBuildDir>
      <xcodeProjectPath>target/dragonbox_ios</xcodeProjectPath>
      <xcodeProjectFile></xcodeProjectFile>
      <xcodebuildArguments></xcodebuildArguments>
      <xcodeSchema></xcodeSchema>
      <xcodeWorkspaceFile></xcodeWorkspaceFile>
      <embeddedProfileFile></embeddedProfileFile>
      <cfBundleVersionValue>DragonBox-${BUILD_NUMBER}</cfBundleVersionValue>
      <cfBundleShortVersionStringValue></cfBundleShortVersionStringValue>
      <buildIpa>true</buildIpa>
      <unlockKeychain>false</unlockKeychain>
      <keychainPath>${HOME}/Library/Keychains/login.keychain</keychainPath>
      <keychainPwd>Lmcefjs</keychainPwd>
    </au.com.rayh.XCodeBuilder>
  </builders>

 <publishers>
    <hudson.tasks.ArtifactArchiver>
      <artifacts>target/dragonbox_ios/build/Unity-iPhone.build/*.ipa,target/dragonbox_ios/build/Unity-iPhone.build/*dSYM.zip</artifacts>
      <latestOnly>false</latestOnly>
    </hudson.tasks.ArtifactArchiver>
    <testflight.TestflightRecorder>
      <apiToken>XXXXXXXXXXX</apiToken>
      <teamToken>XXXXXXXXXXX</teamToken>
      <notifyTeam>false</notifyTeam>
      <buildNotes>DragonBox-${BUILD_NUMBER}
</buildNotes>
      <filePath></filePath>
      <dsymPath></dsymPath>
      <lists>qa</lists>
      <replace>false</replace>
      <proxyHost></proxyHost>
      <proxyUser></proxyUser>
      <proxyPass></proxyPass>
      <proxyPort>0</proxyPort>
    </testflight.TestflightRecorder>
    <hudson.tasks.Mailer>
      <recipients>devteam@XXXXXXXX</recipients>
      <dontNotifyEveryUnstableBuild>false</dontNotifyEveryUnstableBuild>
      <sendToIndividuals>false</sendToIndividuals>
    </hudson.tasks.Mailer>
  </publishers>
  <buildWrappers>
    <hudson.plugins.build__timeout.BuildTimeoutWrapper>
      <timeoutMinutes>20</timeoutMinutes>
      <failBuild>true</failBuild>
    </hudson.plugins.build__timeout.BuildTimeoutWrapper>
  </buildWrappers>

Note: If you haven't selected "development build" in the Unity project settings, then your xcode project will not generate a .DSYM file. testflight will pick it up automatically otherwise.

Current limitations

• as for the Unity xcode project Append operation, we might want to script the xcode project configuration instead. This will as well ease automatic setup of build slaves to allow setup with less manual interaction.
  
• unity is known to have a few issues which might require to remove all temporary build files. This would require a manual setup of the project again.
  
• the testflight deployment can take quite a bit of time. It is tied to the jenkins job, and as we only use one executor, this will cause the build queue to be unavailable for other pure unity3d jobs. Not a problem for us right now
  

Things not yet addressed in this document:

• unit testing with unity & jenkins
  

Feel free to send some feedback!