Automatically Releasing from TeamCity

August 16, 2015

3 comments

Part of my job as the Embedded Team Leader is to bring the team to the 21st century. I still have a plan to write about this whole experience in more detail, but in the meantime I want to share with you something I did a few days ago and I’m very proud of (mainly because I’m a total geek).

One of the problematic areas I noticed in our development and release process was that there is no automation at all. In order to make a release, people tag the code, compile it and put the result in a shared network location. There are a few failure-prone links in this process:

  • IAR, the IDE we work with, doesn’t have Mercurial integration and this results in people sometimes forgetting to add files to the source control. This means that the code compiles on the development machine but might not compile on a clean environment.
  • Sometimes the code does not compile, but people don’t notice it because they have a previous build result and they release the wrong binary.
  • Nothing assures us that what got compiled and released is the revision which was actually tagged.

All these can be fixed with some automation. In the BrightSource development department, we use TeamCity (TC) for continuous integration (CI). My team is still far away from having automatic tests ran in a CI environment, but at least we can compile! Our TC now detects tags, maps them to pre-defined products, builds them in a clean environment and copies the result to a shared release folder. Here’s how I achieved this:

Turns out that TC has this absolutely awesome feature of treating tags like branches. This has two effects:

  • First, when you manually choose to run a Custom Build, you can select specific tags and not just branches:

image

  • Second, just like TC can detect newly pushed branches, TC can also detect newly pushed tags and apply all the build steps on the exact required revision (yellow is for tag and red is for branch):

image

Now, if one of my build steps runs a release script then I’m almost done. Well, almost is an understatement. There are still two major issues which need to be addressed. I need to make sure that just the tagged revisions run the release script. I don’t want each push to the repository to create a new release folder in our shared release network location. Moreover, since we recently moved all our embedded code into a single repository, we now release several products from the same repository and each of these products might have different release scripts. So TC somehow needs to figure out which release script to run on which tag. And to top all that, we want to be able to put several tags on the same revision, where some of them represent releases that should be made and some are just for fun.

And now we’re getting to the awesome (nah, geeky) part. I wrote a ninja batch script, with recursion and all, which does all that. Here’s the main flow:

  • Figure out if we’re on a tagged revision
  • If not there’s nothing to do
  • For each of the tags T on the revision
    • Invoke release_dispatcher on T

release_dispatcher is a script which the user of the ninja script can provide and it basically knows how to map a name of a tag to a release script. For example, something like this:

if "%1 %2" == "Product A" (
 call release_a "%*" 
) else if "%1 %2" == "Product B" (
 call release_b "%*" 
) else if "%1 %2 %3" == "Product C D" (
 call release_c_d "%*" 
)

* I pass “%*” to the release scripts because the dispatcher only checks the tag’s prefix. But the full tag name also includes the release version number and if the release script has this information then it can put its results in a folder whose name includes the that number.

Now to the details.

Getting the revision tags into a variable we’re later going to parse: it turns out that in batch you can’t directly insert a command’s result into a variable. Instead, you can redirect it into a file and then read from the file (thanks StackOverflow). Make sense, doesn’t it?:

hg log -r "." --template "{join(tags,';')}" > tags.txt
set /p TAGS=<tags.txt
del tags.txt

Now that we have the tags separated by “;” we can parse the string and go over each tag and call the dispatcher on it. Batch supports parsing strings (or files) according to a delimiter. But the weird part is that you have to know in advance how many parts you’re going to have. So instead you have to use recursion: you treat the first part and then recursively call your function on the tail (thanks StackOverflow 2):

call :parse "%TAGS%"
goto :EOF

:parse
setlocal
set list=%1
set list=%list:"=%
FOR /f "tokens=1* delims=;" %%a IN ("%list%") DO (
  if not "%%a" == "" call %ReleaseDispatcher% %%a
  if not "%%b" == "" call :parse "%%b"
)
endlocal
exit /b


:EOF

Let’s dissect this line by line.

Line Explanation
1 We call :parse on “%TAGS%” in double quotes because we need it to be treated as a single parameter. Otherwise, if there are spaces there it would get treated as several parameters which isn’t good for our needs.
6 We put the tags in a parameter called list mainly for the sake of clarity but also for the next stage. Note that here the list of tags is enclosed in double quotes.
7 We remove the double quotes from around the list of tags. This is required in order for us to split it correctly by the delimiter “;”
8 We call the for command which splits %list% by “;”. Inside the loop body we can use %%a as the first part and %%b as the tail of the list.
9 We handle the first tag by calling the dispatcher on it.
10 We call :parse recursively on the tail of the list. Again – note we wrap the tail in double quotes so that it’s treated as a single parameter. Come to think of it, the quotes thing could be avoided if we called set list=%* instead of set list=%1 in line 6 [note to self: do it!].

And that’s it! Now all was left to do was add this script to the TC build steps and make sure the result was saved as artifacts. I tested it locally on my machine and on TC and it worked!

image

So I wrote batch recursion (well, I copied from the Internet, but still…) and I have automatic releases from TC and I totally feel like a genius. Now I should go rewrite this whole thing in Python so that the people who stay here after me can maintain this little piece of awesomeness.

 

Update 1:

After reading this blog post, a colleague of mine, who’s working on a unified release process for the entire development department, presented me with the following requirements:

  1. Each release type (as defined by a single internal build script that accept the same set of argument) has a unique TC project configuration
  2. A build script associated with a given project configuration shouldn’t be started because of a false trigger (a false trigger is one the developer didn’t intend to raise). This is required in order to avoid redundant TC processes that waste agent’s time and contaminate the project configurations build histories with meaningless failures/successes.
  3. Different TC project configurations could be associated with the same repository.
  4. Some kind of an automatic trigger should be used to start build processes. “Automatic trigger” is defined as an action that doesn’t require the developer to use the TC GUI.

The way my solution is implemented right now, (2) is not supported. The cost of TC resources is very low because if the tag doesn’t match then the build process won’t start. But indeed the history of the project is contaminated. This morning I figured out how (2) can be supported as well. If you want/willing to have different VCS roots defined for each product you want to release, then you can simply filter out in the VCS definition all branches/tags that don’t start with the correct prefix. For example, if I want to create a TC project which releases WHC Product and only WHC Product I can define a new VCS root which will only be used for this product and configure its branches like so:

image

Then, I can define a new TC project which uses this VCS root, and use the same script as before. The result is that I can only build branches/tags which start with the words “WHC Product”:

image

[A side note: I don’t know why the branch names that appear in the Custom Build windows don’t show the prefix “WHC Product” but just the version number. This might cause clashes if we have several products with the same version number. Probably need to check this with the TC people.]

I didn’t want to tag and push a dummy revision just for testing, but since these are the only branches that appear in the Custom Build window, I’m sure that when I finally do tag and push, only tags that start with “WHC Product” will trigger the build process. I will update here next time we do a release.

If I wanted to release several products from the same TC project I could just add a few more branch specifications to the VCS root definition. And if I actually wanted to have a separate TC project for each product I wouldn’t even need all the batch voodoo I used. TC’s VCS root would just filter the tags for me per project and each build step would use the corresponding release script.

Update 2:

It turns out that even though I supposedly filtered out all branches that don’t start with “WHC Product”, TC doesn’t support not monitoring the default branch. This means that my TC project was still polluted by “empty” default branch builds with no tags and no artifacts. I did what the StackOverflow question suggested and created a dummy branch which will never be pushed to. This solved the issue and when we push to default the WHC Product Release project build process is no longer triggered. We have a release planned for tomorrow. I will update when it all works Smile. In the meantime, like it says in the link, there’s an open issue for TC about it. Feel free to upvote.

Add comment
facebook linkedin twitter email

Leave a Reply

Your email address will not be published.

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

*

3 comments

  1. DmitryAugust 20, 2015 ב 4:49 PM

    If you want to get long branch names in the Custom Build dialog and other places in UI you can use brackets in your branch specification like this:

    +:(WHC Produce *)

    Reply
    1. dinazil
      dinazilAugust 20, 2015 ב 5:50 PM

      Cool. I’ll check this first thing when I get back to the office after the weekend!

      Reply
      1. dinazil
        dinazilAugust 21, 2015 ב 6:32 PM

        Worked! Woohoo! Couldn’t wait till after the weekend 🙂

        Reply