The CI/CD orchestrator is the backbone of a DevOps toolchain. It is the tool that ties all the other tools in your DevOps toolchain together. You can probably not imagine a DevOps implementation without a CI/CD orchestrator. However, this vital piece of the puzzle had very humble beginnings. The journey of the CI/CD orchestrator started around the same time I started my career. In this post, I will explore the past, present, and my views on the future of the CI/CD orchestrator.
The age of the artisans
Circa 2001. I had recently landed a job in IT. It was not my dream job - I did not even dream of being gainfully employed within a year of graduation. I expected a few more years of study and toil before I became independent. But fate had other plans.
During my initial years as a developer, I had to set up builds for projects using Microsoft Visual Source Safe as a code repo. I remember writing some batch scripts to fetch code and build and store artifacts to versioned folders in a network drive. Then, when the team had finished coding, I would manually trigger the script from the command prompt on my workstation and wait for it to finish before I sent out a release email. This was my first pipeline. Hardly CI/CD. Believe it or not, we did client releases like that.
During that time, most small and medium-sized companies had little to no build automation, and build and release were primarily manual. Large organisations had developed in-house tools and infrastructure to orchestrate builds. However, there were no industry standard tools for doing builds and releases.
The good old script and the scheduler were the most prevalent way of orchestrating builds then.
As a result of the manual nature of the work, build and release engineering jobs had a hint of indigo on their collars - they were not very sought after and were considered unglamorous. However, that was all going to change soon.
The advent of the orchestrator
Around 2001, there was a lot of talk going around in tech circles about Continuous Integration and Extreme Programming. Continuous Integration had been around for a few years since Grady Booch (one of the co-creators of UML) coined the term in 1991 in his Booch method of object-oriented software development. Extreme Programming adopted the principles from the Booch method and evolved Continuous Integration into the form that we know now. During this time, Martin Fowler from Thoughtworks was at the forefront of this new idea of building software more frequently than nightly builds. As an outcome of all these new developments:
Thoughtworks released the first version of CruiseControl in the year 2001. CruiseControl was the first ever widely used build orchestrator.
Continuous Delivery or Deployment was still not prevalent then, and it would be a few more years before they came to the scene. CruiseControl lacked a grandiose vision and focused mainly on providing fast feedback to developers - and it did that well. However, the industry needed a little more sophistication, and CruiseControl could not keep up.
Rise of the butler king
In the summer of 2004, Kohsuke Kawaguchi was an engineer at Sun Microsystems. He was facing the heat of his team for breaking builds too often. A tinkerer at heart, he wrote a utility to ensure his check-in always compiled - a 5 line shell script kicked by a cron every 15 minutes. All it did was get the code from CVS, start an Ant build, and mail the log file. As they say, all great things start small. This is where Hudson (now popularly known as Jenkins) started. He then wrote a dashboard to monitor his builds. On February 7th, 2005, Hudson 1.0 was released.
From 2006-2008 Hudson took the developer community by storm and won awards and accolades. Since Hudson was open source, it saw widespread adoption across big and small organisations.
Hudson became the most popular tool for build and release orchestration.
In 2010 Sun Microsystems was acquired by Oracle, which applied to trademark Hudson. This was when the Hudson community decided to part ways, and as a result of that separation, Jenkins was born. Jenkins continued to see widespread adoption across the technology industry. What made it such a powerhouse was its community, which churned out plugin after plugin - there were hardly any tools that did not integrate with Jenkins. Unfortunately, it also became a bane of sorts when all these plugins created a spaghetti mess, making upgrades and management of Jenkins a headache. A lot of these issues have been addressed over the years.
CloudBees has an Enterprise version of Jenkins which takes away many of the cluster management overheads. Jenkins has seen competition from several quarters in recent years, and there are several worthy competitors in the market now. However:
Jenkins is still the most widely used CI/CD orchestrator in the industry.
Worthy contenders
Anthill
Around the year 2001, Anthill was released by UrbanCode. Anthill focused not only on builds but also on managing other stages like testing and build promotion. UrbanCode’s enterprise version Anthill Pro saw widespread adoption. Their community edition was also widely used. IBM acquired UrbanCode in the year 2013. Anthill Pro was retired in the year 2018 and was replaced by IBM UrbanCode Build and IBM UrbanCode Deploy.
Bamboo
Atlassian released its pipeline orchestrator Bamboo around the year 2008. Being licensed software, it saw good adoption from medium to large enterprises. Bamboo is still sold as Atlassian’s on-prem enterprise CI/CD solution.
Electric Cloud
Co-founded in 2002 by the creator of TCL, John Ousterhout, Electric Cloud started with a solution to significantly reduce build times with their product Electric Accelerator. Their second product Electric Commander, a licensed build and release orchestrator, saw adoption from large enterprises. CloudBees acquired Electric Cloud in 2019.
What about Windows?
While Bill Gates was at the helm of affairs at Microsoft, Microsoft continued to have an image of the rich bully kid in class who did not get along with everyone else. Most of the new open-source tools were more Java and Linux focused. CruiseControl came up with a Windows version called CruiseControl.Net. In those days, while Windows support was present in most of the CI tools, it was patchy.
Microsoft started shipping MSBuild with .NET around 2005. MSBuild provided the capability to manage builds using code. From 2013 onwards, MSBuild started shipping with Visual Studio, and with the advent of Team Foundation Server, Microsoft had a robust solution for defining and managing build pipelines.
The current avatar of the Microsoft CI/CD orchestrator is Azure DevOps, which ties together several DevOps tools and is one of the leaders in this space.
New kids on the block
While Jenkins continues to rule the roost, the last decade has seen a few developments that have brought a sea of changes in the CI/CD domain and opened up the playing field.
There was a time when the Version Control tools space was going through trials and tribulations, with a plethora of solutions in the market. That race seems to have been settled now, with Git emerging as the straightforward tool of choice. Teams also want to keep their pipelines as close to the code as possible and maintain them just like code. Version control vendors realized soon that just providing a web interface for git workflows would not be sufficient to survive - so they started implementing more toolchains around version control workflows.
GitLab CI/CD
GitLab has taken the game to the next level in this area. All you need to do is quickly look at their features page and realize there are very few things in the CI/CD process flow that it cannot do. A community-run organisation in the true sense, GitLab is entirely remote and owns no real estate anywhere in the world. Instead, it has contributors worldwide who have access to all the details of company operations online in a document aptly named the handbook. GitLab introduced its CI/CD feature in the year 2018 and has done continual enhancements to it over the years.
GitHub Actions
The world’s most used version control tool introduced its pipeline feature in 2018, the same year Microsoft acquired it for 7.5 billion dollars. GitHub Actions has evolved into a mature pipeline tool since then. Microsoft has also steadily added several other tools (like security tools) with GitHub Actions to fortify its position as a CI/CD orchestrator of choice.
Bitbucket Pipelines
Bitbucket is Atlassian’s cloud-native CI/CD orchestrator. Atlassian is not a newcomer in this domain - it has been in the game since 2008 with Bamboo. Bamboo is now Atlassian’s on-prem only solution, while Bitbucket Pipelines is its cloud-native CI/CD orchestrator. With seamless integration with other popular Atlassian tools like Jira and Opsgenie, Bitbucket Pipelines has seen increased industry adoption.
CircleCI
A relatively new contender in the market, CircleCI has acquired a great deal of popularity in the DevOps community and has seen widespread adoption - and rightfully so. Its approach to modularising pipeline components and integrations into Orbs is much cleaner and more seamless than the plugin architecture of Jenkins and similar solutions. With well-maintained documentation and usable pipeline reference code, it is probably the fastest tool to onboard. Over the years, CircleCI has integrated many popular tools in the DevOps segment and is now easily one of the top CI/CD tools in the market.
Harness
Co-founded in 2017 by Jyothi Bansal - who sold his first company AppDynamics to Cisco for 3.7 billion dollars - Harness is relatively new in the game. However, it is already a unicorn. They started with a CD tool and added a CI capability to their portfolio by acquiring the open source CI tool Drone.io in 2020. Harness has been quietly building up a portfolio of offerings catering to various DevOps requirements with a vision to be a next-generation software delivery platform for everyone. Harness is still not a name synonymous with CI/CD. However, with a strategy of catering to enterprise and community users, Harness might be a name to watch for in times to come.
The meteoric rise of cloud-native orchestrators
More and more solutions are cloud-only these days, and with the burgeoning list of cloud-native services and tools, CI/CD has seen the advanced implementation of orchestrators from cloud vendors.
Azure DevOps
Known as Team Foundation Service/Server in its earlier versions, Azure DevOps is often confused as an Azure-only service. Even though Azure DevOps provides first-class support for Azure cloud services, it is entirely independent of Azure cloud services and can be used separately. Azure DevOps was one of the earliest to take a holistic approach to CI/CD - instead of building just orchestration, it tried to integrate other tools like collaboration, artifacts, deployment, security, etc. It was one of the earliest attempts at creating a DevSecOps platform for the market.
AWS Code tools
The AWS Code set of tools - namely CodeBuild, CodeDeploy, CodePipeline, and CodeCommit - have come a long way from “also-ran” to strong contenders in the CI/CD space. With strong integration with AWS native services, these tools provide a solid set of capabilities for AWS-native implementations. For AWS shops trying to curtail their vendor sprawl, Code tools offer a compelling option. Even though adoption has not been as widespread as other tools in the market, more and more projects are exploring this option to orchestrate their pipelines.
Google Cloud Build
Not much talk about this GCP capability is out there in the market. However, in the Forrester Wave: Cloud-Native Continuous Integration Tools, 2019 report, Google Cloud Build beat all the heavyweight contenders to come out on top. Under the hood, Google Cloud Build uses its open-source project Tekton, which it has donated to the Continuous Delivery Foundation. As the adoption of Google Cloud offerings picks up, I am sure we will hear more and more about this tool.
Several other tools in this space are worth mentioning, like Travis CI, Codefresh, and Buildkite.
Game on
With CI/CD and DevOps taking center stage in the past few years, the tools market has also seen a lot of action - and it will continue to be like this in the coming years. With such widespread adoption of cloud-native solutions, cloud vendors will have an edge in attracting customers to their CI/CD offerings. Specialised CI/CD vendors will continue pushing boundaries to provide teams with feature-rich, fast, intuitive, and easy-to-maintain pipeline tools.
It would be interesting to see how this segment evolves in times to come. How do teams select a CI/CD tool with so many options available in the market? A topic for another post, maybe.