Cruise (from thoughtworks, the authors of CCNet)had an update that said they had better perforce integration. I decide it's time to revisit.
Last visit to cruise, we passed on it. It had nice dependency and build machine farm setups for cascading builds and queueing blocks of commands between machines, but it always did a full 'p4 sync' to a temp client spec, and couldn't use multiple clients from different p4 servers. This was a blocker, because at work, art data and code files live in different repositories. It would also probably eventually clutter p4 metadata beyond belief.
Let's see if things have improved.
- Installed client and server on win7 and kubuntu. Petty easy on both, even though I can't test the license across the virtualbox virtual machines. IIRC, the cruise trial license used to allow multiple machines with test license, but doesn't now. It's a licencing restriction of the 10 user evaluation version.
- Documentation mismatch. Claims that installer will prompt you to select artifact directory. Does not actually do this on win7 x64. Had to figure out where in the server config to change this. One thing that's great about cruise is that the coordinating server stores everything, the build logs, the build output you care about, test results. And each thing can be fetched back with specific html pages.
- Set service user to a local admin so it can do everything. This is pretty standard, since it will be running a compiler on a win7 machine with UAC enabled.
- Found a project to build, nullDC, a dreamcast emulator. Uses svn, a supported source control repo.
- Made a new pipeline, but svn can't be found. Needed to reinstall. I keep PATH really short, so I re-edited the batch wrapper and confirmed it worked. Using Sliksvn this time.
- Repo can't connect in the pipeline. Cruise is trying to invoke svn.exe over PATH, but I have svn.cmd. WTF. This is not how Windows does things. I see a post on the cruise support forums after 15 mins searching that says doing this is not supported. Really? Make my path filthy with additional directories.
- I test the connection, and see that cruise is trying to send username and pass arguments all the time. If they're empty, couldn't it skip that? I set user to 'anonymous' and password to my email address. Luckily, that works. Supposing it didn't though.
- I make a build robot email alias, and get stuck trying to get yahoo mail to forward from a new address. Swell. Eventually I get the new address approved by yahoo.
- I go to set the pipeline defaults, chose 'exec' maker, and set 'make' as the placeholder command. Poke through the cruise UI pages, and I see the repo history. I get a red box in a little while, and my firewall notifies me that cmd is being launched. Wait, I haven't asked it to build yet, have I? Visiting the pipeline page has tripped a build somehow. Not what I was expecting, since I'm not done configuring yet.
- Go to fetch the other components like winpcap and sdl. Where is the nice guided UI for adding pipeline parts? Fiddle around, and it turns out you can only invoke the text editor of the xml config file for the server, or create a pipeline from a web form. Nothing in the middle. VisualBuild is superior, at this point.
- Find that compile requirements include Microsoft Visual Studio C++ 2010. Boo. Download the Express iso image.
- A modern dxsdk is required. Download the Feb 2010, because instructions are non-specific. It later turns out that include paths are hardcoded to April in the projects, instead of using the DXSDK_DIR environment variable. Unlike all the other installers I get, this stupid installer must validate that 'windows is genuine' to download the dxsdk. I suck down another cd-sized installer from the 'net. This DSL speed is pretty lame. Idly dream about fiber, and remember seeing Oak Park's South Blvd. painted up for street maintenance with fiber in orange spraypaint next to triangular figure-eight shapes.
- Grab the win7 .net 4 sdk, another CD-sized installer, because the express edition probably does not come with it. No 'genuine' requirement there, do you hear me dxsdk team?
- Thinking about it, I probably don't want to constantly suck my code from google to do clean builds. I want a local mirror, or known versions of code and binaries that are stable or tagged. I look into svnsync.
- Instead, I follow the mercurial handbook and mirror the svn repo. Will need to hook this up to a mercurial repo that's tracking the mirrored changes, since we'd prefer to commit changes without being approved, and push them wherever we want.
- Install visual studio express, which actually only installs the prereqs and prompts for reboot without scheduling the install I really do want. Run the dxsdk installer, and realize visual studio is nowhere to be found. Re-run the installer to get visual studio, and re-install dxsdk.
- Realize only the web installer for the windows sdk was downloaded, track down the iso.
- Download and install mercurial, one of the 4 supported source control systems (svn, git, hg, p4). fix up the old python26 batch wrapper in c:\bin to point to the right place.
- Try to authorize visual studio 2010 express, but the email never arrives. Change email to my gmail, because yahoo trashes the inbound mail.
- Try to 'hg convert repo' but svn bindings aren't installed with the win32 build of hg. Seriously? just shove them in there. The workaround from stackoverflow - install and use tortoiseHg instead.
- Confirm the reinstall of dxsdk set the include/lib project paths, only to discover that they are not set anymore, the dialog claims the option is deprecated in favor of project settings. Great - the one thing I've had to configure in Visual Studio since 2001 has changed. Let's hope it's for the good.
- The tortoiseHg installer was win32, not the x64 I need. Wargh! Suppose my browser would have to leak my architecture in order to provide preferred arch for downloads. Wish for a package manager for windows, again. Steam people, are you listening?!
- Turns out I need TortoiseHg with the bindings, AND the mercurial dist with convert included, and to set tortoise first in PATH. Seriously, would it have been easier to get python and build? Probably I would need mingw and msys, and then where would I be?
- Ok, with convert working, I now have an 'svnsync sync', 'hg convert', and 'hg pull' pipeline for getting new changes from svn. The first two create a converted mirror for hg, and the last creates my local branch for edits. Things like installers or integration with 3rd party apis. Fixes. We're going to build my branch of course.
- Oh god. Now to configure the local repo url. 'hg serve' will suffice. Lots of options, but no way to make it run as a windows service? Web says bitbucket has a request for this, so I sign up and +1 for it. I try the 'sc' command. I get the windows server 2003 toolkit. But, all of this doesn't help, so I fall back to scheduled task to fire up the repo with 'hg serve'. Waste of time. But there are the files.
- Ok, so now I make a new pipleline, because that's the only way to get the web form to do data entry and modification. But, the free edition throws red up on the UI, because you can't have more than one pipeline group in the free version. And, embarrassingly, there's no way to delete the pipeline from the UI. So, how am I supposed to get this working? What is the web UI for, if not to work with configuring the builds?!
- While I'm in there, I find cruise has saved my email smtp password stored in plain text in the config file. That is not a real good idea. #insecure
- Open up the .sln in express, and discover release doesn't build. How have the authors of nullDC made their dxsdk directory paths show up? If they are now properties of projects, should those changes not be checked in? Or does windows write the settings out to some local temp file, which isn't part of a project any more? Not having default include paths is going to bite.
- It's not that bad to add dx to the projects. Some web searching reveals why the .res files don't compile; no MFC headers nor resource editor come with this free-as-in-beer compiler, and Platform SDK doesn't seem to be part of Windows7 SDK. I switch some defines. Now PuruPuru and nullExtDev don't compile, one is missing maybe libpcap, and one SDL? Good lord, there's no solution nor project for SDL bundled. Maybe I can make a makefile project.
- Fetch winpcap source from the web and add to hg source repo because it comes from a zip, instead of a repo proper. Added the target directory 'make headers' batch file, and figured out how to use cruise to stash build artifacts. But, headers don't work without libs, and it's an old vc2005 project, so it needs converting to vc2010, needs some tweaks to .rc files and the finished .vcxproj, and needs a global .sln that builds the required files. Got this done for release target, sort of. TODO: check in changes, get SDL fetched, built.
- Fixed winpcap vc2010 .sln and hunted for command line builder. MSBuild it is. You can dump out html and xml log files easily. That might be nice for a cruise tab representing the build log, instead of straight text. Build with options ok and add to pipeline. It would be great to have the msbuild as a builder in the dropdown, and reuse the CCNet xml plugin or a pre-existing xml report / logger, so a cool tab could be made with build report.
- Quoting properties in cruise with spaces is non-obvious. and it's impossible to view the actual command passed in the exec without echoing the entire thing yourself. For example, you need to use xml-entities for quotes and ampersands. If you put the quotes in the wrong place, they get doubled up or duped. You can't '/property:foo="bar with spaces"', it gets mangled into '"/property:foo=""bar with spaces". Boo. Instead, we quote the entire arg "/property:foo=bar with spaces" and msbuild figures out what we want. It would be really nice if this echoed the command line in the log instead of printing the xml of the command, and echoed it every time it ran something.
- Starting computer from sleep, cruise complained about server being over its remote license quota. Restarted the agent, and started working again. The server/client had different IP versions because a tornado reset of the network equipment.
- Forgot to mention, set the headers and libs up as artifacts. Cruise stages can be configured to fetch them from other collected artifacts on the build server, if you use the fetchartifact block or use wget or curl with the correct urls. It would be great to get a remote agent going to get simul linux builds up, to make sure I can keep the project building across multiple architectures. Some cruise thread on support page about using port forwarding to make it appear you have a remote agent also locally. interesting, but confused. What, would you disable one and then the other. Even having a secondary win32 xp virtual machine to build or run simple tasks on would prove whether or not cruise was interesting.
- With CCNet, the project was available for public use and contributing. Without cruise source and more features, fewer places to adopt the product. Work less likely to invest, since core source is not open.
- Cruise has the bamboo-like feature for guessing when product would be built based on metrics of build stats. The progress bar with estimated time to completion was nice. Wonder if it works per-host-per-task. Curious if this is hudson or cruisecontrol java under the hood, since each have the feature.
- Used the build properties url of latest cruise pipeline build, and artifacts url of latest build to view downloadable artifacts. Cute. Annoying, now that I've set up tasks that might work well across a farm, to not be able to make a farm to test development and features of cruise. Might still pull back the binaries into the build workspace to inspect. Wonder if there is a non-latest url that is the last successful binaries/headers from that stage.
- Fix some missing include file dir for winpcap. Make it relative to the Include dir, since that's sort of the standard. Add additional header and lib dirs, and make dxsdk_dir paths instead of April dxsdk paths. Release now builds using cruise, except for purupuru, which needs SDL. Which I don't even have time to make a builder for tonight. Except, I think, I made a mistake; the .vcxproj now expects the files in the right place, but nothing makes them there. I've made a (bad) dependency for the project, since no lib/header are available unless you run the magic steps first (make header, and make modified winpcap vc10 sln. sounds lame, but additional users won't be using cruise, right?
- Realize it's checking into mercurial that's tripping the build. It's on continual integration mode, right? you have to select manual staging if that's what you want (if you need human triggering or approval). Will have to figure that out another time.
- Figuring out how to get the artifacts from one stage to another wasn't too bad, but picking the right dirname or filename was difficult at the root level, since cruise wanted to include the artifact root directory in its destination, and didn't offer the same wildcard expansion for fetching them as it did for selecting them. Also, setting an attribute to an existing stage or pipeline name is impossible without tabs in the browser, because there's not a UI-driven pickbutton, and while the edit text box is active, other properties of the page are inactive.
- Now that everything but purupuru builds, time to investigate pulling SDL down. Yay! Its repo is already Hg. Boo - apparently we can only have one source control per cruise pipeline block, trying to figure out if it's per-pipeline, or per-task-chunk. Ok, no. It's like it will pull each one, as long as the destination is just one place. Not sure I like this, since I really want to pull all of SHARKBAIT, not some of this and some of that. Going to init a pull to SHARKBAIT of SDL 1.2, and if it's not awesome, try 1.3
- The pull was no problem, but now the msvc 2010 building requires unzipping, possibly converting or upgrading the sln/vcproj, and then stashing the .dll and headers. todo another day when more stamina is available. Interesting that commercial SDL development is in Sam's future. Nice going, Sam! Must be nice being an (ex?) WoW developer, hopefully making bank! Long time since wizport.
- Hey? Where are my emails from the build robot? My build is still busted, but there's no emails in my inbox. Do I have to subscribe to them or something? The 'test your smtp server' emails come in ok.
- Yep, reading the docs, you have to enable authenticated users. There's no default admin email.
- Add the extract zip with 7z to the stages. Still need to convert and build.
- Email mercurial list, because the documented hgwebdir.cgi doesn't exist, and I want to publish both local repos over http using it. Its wiki seems to tout this as the right way to do things.
- Reply from list comes back: hgweb.cgi had hgwebdir.cgi folded into it. It's not clear how to use this, and the comments in hgweb.cgi are broken: #Path to repo or hgweb config to serve (see 'hg help hgweb')'. This command doesn't exist, so I check help on 'hg help serve'. Perfect. Another arg to use, and I've got SDL and SHARKBAIT repos shown.
- Hey, VSUpdate.exe can update the SDL visual studio 2008 project on the command line, and msbuild can bake it! YAY! I don't need to modify the repo, I just need to run commands on it. Sweet.
- Except, why does the pulled version of SDL-1.2 auto expand the VisualC.zip for me into VisualC/ directory? I was trying to do that manually with 7z.exe. Delete that block, and setup include/ directory as artifact, Release/SDL.dll as artifact. Excellent. Maybe there's some kind of built in hook or functionality. I ignore this, and go with the flow.
- The SDL cruise pipeline is building stage SDL-1.2 at random interval. I need SHARKBAIT/nullDC to build after it completes. I set the nullDC pipeline to depend on materials from SDL, as suggested from the help files, and add fetchartifact blocks to pull the SDL include and lib dirs. Epic fail: licensing restrictions of cruise free edition prevent the user from using multiple repo materials, and stages from cruise count as a repo. I could check all of SDL into SHARKBAIT for no good reason, to work around this. I email thoughtworks and ask for a 1 year license instead. The nullDC stage hangs, pending license restriction fix.
Is it an improvement? Over the last version of cruise, sure. The documentation is there, and the product seems more intuitive than Bamboo.
Is it something I can convince work to switch to? Unknown. The license is not open source, the price is not opaque (they create customized quotes, usually synonymous with robbing you blind). We've certainly rejected software on these criterion.
But it has its charms, and it seems like it might scale well. With the restrictions, it's not really possible to evaluate past the good feel.
Give it a try, and see if you don't agree!
I'll follow up if I get a chance to test the other features.
[2010/07/02 update]
Fetched some new source with mercurial, and was confused about where my files went to after an 'hg pull'. Nothing but deleted files disappearwhen you 'p4 integrate', so this was a little spooky. And annoying to see cruise building away with invisible missing files. After 'hg merge', everything went back to normal. Would be good to block builds of incomplete integrations. I mean merges.
So everything builds. No storage support for the build software configurations. Doesn't one typically want to check these in?
Talked to Tim Mensch a little about git, which he says is goofy on windows when cygpath and path collide. I'm more of an msys/mingw guy, will have to investigate. He's using bzr for his own things, which jives with my anti-TLA rant. Not like I don't have half-a-dozen things on my local box under darcs, hg, bzr, and p4. This fragmentation of source control is both interesting and stupid. Not having access to source means I'm unlikely to extend cruise to support my distributed revision control flavor of the month.
If there's a reasonable way to export my perforce tree of binary video game source files to mercurial or git, let's hear it - git fast-export of //data...#head sounds promising. I tried hg convert, but at the 500 changelist, it was completely painful and I killed it. We have so many changes and revisions, we'd need something fast to try any kind of switch.
Leave a comment