We have decided to use CI aka Continuous Integration, a software development practice where members of a team integrate their work often. We will go through the pro's and con's of CI and its impact within the organization. This wiki holds information related to the decision on requirement, tools, installation and road ahead w.r.t CI.
The term "continuous integration" refers to a process that builds and tests code on a frequent basis. The continuous integration servers constantly monitor source code repositories and as soon as new changes/commits are detected, they initiate a new build cycle. The build cycle actually involves code compilation and, in addition, may involve various tests and code analysis. If the process encounters errors, it notify the build master.
- First, a developer commits code to the version control repository.
- CI server is polling this repository for changes (e.g., every few minutes/few commits).
- Execute build scripts.
- Feedback build results to members (e-mailing/feeds/irc).
- Wait for next commit.
We cannot assume that we are safe from integration problem, CI is a supporting tool in the development/release cycle to detect defects early. The participation of development / QA team plays a very important role. Educating the importance of commit early commit often and detect errors early is the key. CI is not just a technical implementation; it is also an organizational and cultural implementation.
Who is gonna benefit
Developers: making software integration a nonevent, you can focus on what you love the most, which i assume is software development ;-)
Release/Configuration/Build Management: Help create deployable software multiple times in a day without waiting till the end of development sprint. In OB we have a sprint of 2 weeks. So at the end of 2 weeks we have a deployable software.
Testers/QA: Enable testers / QA to do incremental testing.
Managers: you can talk to customers with confidence and promise a working software.
Continuous Compilation Vs Continuous Integration
Lets do it right
The following questions will answer if we are doing our CI correctly:
- All changes in code/db/conf pushed through SCM - cvs/subversion/mercurial/git
- Is you project/module build automated - make/rake/ant + ivy/maven
- Do you write/execute tests as part of build - junit/xunit/selenium
- Do you have coding and design standards, if yes how are we enforcing it - checkstyle/pmd/jdepends in case of Java
- Is the Integration machine separate from development machine - to ensure clean build(s)
Selection of build tools
We had the following consideration for selecting the build tool. The tools should be:
- Open source, with active community working on new features
- Everyone in the CI community knows that cruise control is probably one of the best tools available. But the open source version lacks UI. With active community working on the product as a whole we can assume it to evolve holistically.
- If there is active community then we have a great possibility that we can request for some feature that's missing or we can implement and submit a patch/plugin for the same.
- Plugin Architecture
- As we grow we will have new challenges and new systems that will get added to development/build/release environment. Plug in architecture ensures that this is taken care.
- Native Java support will be an added advantage
- 99% of Openbravo is Java and tools we rely are also Java centric (like ant, maven, etc). It makes a lot of sense to use a product developed by Java community which understands the challenges of a Java development environment.
- distributed build farm.
- At some point in time we will have to create a pool of servers and do distributed build, the tool should be capable of doing the same.
- support incremental builds
- this is a very important and critical decision point in tool selection. Without this we end up building the whole system for a minor patch update.
- Take care of 75% of challenges that we have, rest 10% should be in the roadmap of the product
- Even if all above points are not supported, then based on this point we can decide which is a better system for us to work with.
- Few more from the team (from the work priyam has done)
- support SCM Mercurial, Openbravo development has moved to mercurial from subversion. More about mercurial @ openbravo
- dashboard to browse the status of builds, tests
- support for e-mail, CIA
- mantis support, Openbravo runs mantis issue tracker
- Support for Apache ant, this might change as we are gonna discuss Ant Vs Maven in the next release.
- Support for multiple version builds.
What do we use and Why
Started with buildbot, the initial setup was done by stefan as a weekend activity, as he was familiar with it. We have not spend enough time on this tool to make it usable and at the moment we have decided to use hudson as primary choice for CI.
After a long internal / external debate we have zeroed in on Hudson as our new build tool probably because of following reasons:
- Easy installation: Just java -jar hudson.war, or deploy it in a servlet container. No additional install, no database.
- i know what you are thinking; "easy is relative", i hope you know that we are a java development company and people here are comfortable with java than any other language; hence easy - got it!
- configuration: Hudson can be configured entirely from its friendly web GUI with extensive on-the-fly error checks and in-line help.
- There's no need to tweak XML manually anymore, although if you'd like to do so, you can do that, too.
- since we have lot of interesting tasks to do (other than working with hudson), it doesn't make any sense to spent lot of time writing the configuration. Hudson promises a UI which is capable of addressing most of the day to day challenges.
- Change set support: Hudson can generate a list of changes made into the build from CVS/Subversion/Mercurial (via plugin). This is also done in a fairly efficient fashion, to reduce the load of the repository.
- most of the newest SCM's support this. The challenge would be to figure out how to do incremental builds, which we will discuss later.
- Permanent links: Hudson gives you clean readable URLs for most of its pages
- permalinks link "latest build"/"latest successful build", can be easily linked from elsewhere.
- this will help put information in wiki or other text publishing mediums
- RSS/E-mail/IM Integration: Monitor build results by RSS or e-mail to get real-time notifications on failures.
- from now you don't have to open a build status page, no one will stop from doing so though.
- after every build you will be intimated based on success / failure of the build.
- After-the-fact tagging: Builds can be tagged long after builds are completed
- not sure if this is a feature, fact or feature fact ;-)
- JUnit/TestNG test reporting: JUnit test reports can be tabulated, summarized, and displayed with history information, such as when it started breaking, etc. History trend is plotted into a graph.
- not sure if this is of any use, but lets keep it for the time being.
- File fingerprinting: Hudson can keep track of which build produced which jars, and which build is using which version of jars, and so on.
- This works even for jars that are produced outside Hudson, and is ideal for projects to track dependencies.
- Plugin Support: Hudson can be extended via 3rd party plugins. You can write plugins to make Hudson support tools/processes that your team uses.
- go on extending ...
- Distributed builds: Hudson can distribute build/test loads to multiple computers. This lets you get the most out of those idle workstations sitting beneath developers' desks.
- This is a real thread to Intel/AMD. We are going to make you work 24/7 ;-)
Comparison of Tools
In this article we have tested and evaluated tools like Cruise Control (CC), Buildbot and Hudson. All three tools have the ability to build and test code.
Cruise Control (CC)
- License: BSD-style license and is free for use. Commercial version has more feature than the open source edition.
- Configuration: XML Based, you also need to set up a copy of your project manually in the CruiseControl projects directory.
- UI: Can see the build/test progress (monitor) through a web dashboard but build itself is not configurable through UI.
- Installation: [Level - High]
Remark: however, once installed and properly configured, it proves to be very powerful. The tool is extremely powerful and can be adapted in the enterprise environment with medium to large teams. Its UI is definitely lacking.
- License: GPL-2
- Configuration: INI file format
- UI: Limited UI, where you can browser through the status of build/test in a waterfall/grid display
- Installation: [Level - Low] A few easy_install steps are now sufficient to create a buildbot waterfall.
Remark: configuration is easy and can be accomplished in less than half an hour, but tweaking the master.cfg takes time. UI is very primitive, not sure of close integration with Java subsystems.
- License: Creative Commons Attribution Share-Alike license / MIT license.
- Configuration: friendly web GUI with extensive on-the-fly error checks and in-line help. We can tweak XML manually as well (if we have lot of free time).
- UI: pretty decent UI support for most the challenges that we face. It has RSS/E-mail/IM Integration to get real time notification.
- Installation: [Level - Low] Installing Hudson is even simpler, and requires execution of a Java command after the jar file is placed on the machine that will be a build server. Hudson starts up, its web-based dashboard can be accessed via a link.
Remark: It's relatively young, but very well designed and lots of Java community members involved (soon we will also be part ;-) ). Learning curve is relatively less and more over we have tons of third party plugins.
continuous integration tools in the open source world