What is Continuous Integration in DevOps?
In DevOps, Continuous Integration (CI) involves automating the process of building and deploying code every time a developer in a team commits code to version control. Developers share code by merging all changes to a shared repository, including the smallest of changes. Every time code is committed, it initiates an automated pipeline that retrieves that latest code and proceeds to build, test and validate the main or trunk branch.
CI was imagined and implemented as a best practice to tackle a specific problem. When coder worked in isolation and had to integrate changes with the team’s codebase at the end of days or weeks, they found it led to frequent merge conflicts, frustrating bugs, incompatible coding strategies and duplications.
With consistent code commits to a shared branch, these problems were significantly lessened. Code is tested to iron out bugs early on, and inconsistencies with the larger body of code are identified early on.
Benefits of Continuous Integration in DevOps
Smaller and Easier Code Changes
With every code change being pushed to version control immediately, the CI/CD pipeline has to deal with smaller code changes and integrations at a time. Obviously, such changes are easier to handle, test and debug than larger scripts that may contain multiple, hard-to-detect errors. The idea is to break down features and new development into byte-sized pieces that are fast to write.
Easier Debugging
A CI-based pipeline facilitates fault isolation AKA the practice of formulating systems in which errors lead to limited negative consequences. This protects a systems against major damage and makes for easier maintenance.
Fault isolation is facilitated because smaller code changes make it easier to find bugs and resolve them before they can adversely affect the entire ecosystem - something not easily achieved with CI in place.
Faster Product Releases
A CI-powered pipeline is a continuously moving system in which failures are detected and debugged faster. This invariably leads to faster and more frequent releases which translates to happier customers and a positive edge over competitors.
Lighter Backlog
As explained above, CI enables quicker bug identification and debugging, all within the early stages of code changes and integration. With small defects quickly fixed in pre-production, developers have a lighter backlog of avoidable, non-critical bugs to fix. They can devote that time to focus on larger problems, write better code and keep updating the systems for greater efficiency.
Needless to say, lesser errors result in lower costs, easier maintenance, better quality code and increased ROI.
Increased Transparency and Accountability
Frequent code commits lead to immediate and frequent feedback from the automated system as well as the team. This keeps problems visible and transparent to the team, keeps everyone on the same page.
Feedback from the CI stage has direct effects on build issues, merging conflicts, larger architectural snags, etc. Thus, it doesn’t just keep the developer's conscious of what they and their teammates are doing, but also of the health of the system they are operating in.
Of course, CI alone cannot implement the seamless operability required to create software quickly and with minimal flaws. Neither can it, by itself, establish DevOps principles within a software engineering team or organization. For that, the pipeline needs CD - Continuous Delivery/Continuous Deployment
Continuous Delivery
In DevOps, Continuous Delivery (CD) is a practice in which all code changes are automatically tested and readied for release. Once Continuous Integration takes care of code build and integration, delivery pushes changes to the testing environment and then keeps it prepared for production. Essentially, with CD in place, a team will always have a deployment-ready product that has been tested and cleared for public release.
Continuous Delivery takes the code beyond simple unit tests that are run in the CI stage. It allows the software to be run through multiple evaluation layers so as to verify functionality from all dimensions. This can include integration tests, UI tests, load tests, end-to-end tests, API tests, regression tests, security tests, etc. It allows developers to validate the product more comprehensively and weed out issues before it reaches customer hands.
Continuous Deployment
CD can sometimes also refer to Continuous Deployment - a practice that is the next stage from Continuous Delivery. Once code has been made release-ready, Continuous Deployment is the process of automatically pushing it to the production environments.
Automation, in this case, is conditional on a series of preconceived and pre-established tests in the pipeline. Code changes pass through these tests, and if all goes well, the pipeline triggers their release directly to production. No human intervention or approval required.
Continuous Delivery vs Continuous Deployment
Continuous Deployment essentially adds an extra step to Continuous Delivery. The former pushes every code to production automatically, without explicit approval from a human supervisor. The pipeline takes the code from the repository, pulls the appropriate configurations, builds VMs, containers, etc. on the fly, and deploys the code, all in one fell swoop.
In case of Continuous Delivery, the code is built, integrated and tested to be production ready. However, it requires a developer, product manager or Team Lead’s approval to actually be released to the production environment.
The question of - which continuous model to use? - depends on an organization’s goals, the skill level of their employees as well as the resources they can devote to acquiring the right CI/CD tools. Implementing CI/CD in DevOps is to have the proper tools and checks in place to manage configurations and rollback in the event of errors or failures.
What is a CI/CD pipeline in DevOps?
A CI/CD pipeline comprises a number of serial processes that are executed to deliver a new version of software (website/app). CI/CD aims to accelerate and improve software delivery by utilizing DevOps principles and leveraging automation at every step.
Consistent automation and monitoring is incorporated into the CI/CD pipeline so as to make development more efficient while consuming less resources (time, money, effort). Every step in the pipeline is automated - from the moment code is pushed to the repository it is built, integrated, tested, deployed and monitored via automated mechanisms.
Needless to say, the business benefits of CI/CD pipelines are many. They accelerate development and reduce the likelihood of errors. Essentially, it enables enterprises to release software multiple times a day with minimum human involvement.
Best Practices for CI/CD pipeline
Commit early and often
Every time a dev commits code, they initiate a series of automated tests that provide feedback and inform the team that a change has occurred. Regular code commits ensure that the whole team remains on the same page which enables better collaboration. It also lowers the likelihood of frustrating merge conflicts that usually show up when integrating larger code changes.
By sharing all changes with the entire team (through code pushes to the main branch), everyone stays updated and can modify their own work to match the best and latest version of software. It is wise to commit code at least once a day, though most major companies commit far more.
Stick to one build
Don’t create a new building for each stage of the pipeline. Doing so in different environments can introduce inconsistencies in the software, which means that you cannot depend on previous test results. Instead, one build artifact should be travelling through each stage and finally, released to production.
Keep environments pristine
To achieve accurate test results, clean the pre-prod environment between deployments. If environments run for a long time, teams have to deal with multiple configurational changes and updates, which are hard to track.
With returning them to a pristine state, tests that pass in one environment might fail in another. To prevent this, use containers to host environments and run tests. This makes it easier to tear down the environment after a deployment.
Monitor and Measure
Most teams will set up monitoring mechanisms for the production environment so as to detect any errors or anomalies quickly. Similarly, a CI/CD pipeline must be equipped with metrics for monitoring and evaluation. By analyzing these metrics, developers and team managers can narrow down potential bugs and handle them before they metastasize. They can also detect modules for improvement.
Streamline tests
Don’t expect to automate every test from the very beginning. Start with tests that run the fastest (unit tests), run them early to get an initial layer of feedback. Once these are complete and you have some idea of the build stability, move onto longer and more complex tests.
Manual tests take longer and are dependent on availability of the right personnel. Therefore, keep them minimal (you’ll never be able to eliminate them completely) and leave them for after the completion of automated tests. Instead, ensure that testers focus on creating mature, comprehensive automated test scripts that get the job done.
CI/CD pipeline stages: A breakdown
To define CI/CD pipelines, look at the basic steps: Develop → Build → Test → Deploy.
As we expand the pipeline, we also have Monitoring → Feedback → Operations. A continuous orchestration platform can help you design such a pipeline, this is where DevOps Engineers play their role.
Develop
Here we are writing the code, meaning that we need a repository for storing and checking out code.
Build
When code is checked-in to the repository, that code is integrated into the master branch. Here is where version control is necessary. In older waterfall style workflows, integration/build would only occur after the completion of a major feature set or large volume code changes. When working in a more agile, continuous model, code is integrated into the feature times per day. Code is broken down into smaller working functions, allowing for iterative improvements and fast integration results.
Test/Quality
Once the code is built, the resulting application must be tested for errors, functional failures, and quality. These tests can and should be automated using any number of purpose-built tools. By testing at every build, feedback is received quickly and corrections can be implemented swiftly.
Security Scan
As part of our transition away from bolt-on and after-the-fact auditing, security scanning is a critical part of the continuous loop. Security scans can automatically detect vulnerabilities and insecure implementations before being released into the wild and exploited by bad actors. It is essential to include this step in the testing portion of your pipeline - as they say, “an ounce of prevention is worth a pound of cure.”
Deploy
The code is built, testing has provided the green light, and it’s time to push our changes to an environment, whether pre-production or production. With continuous deployment, regular automated push to non-production environments gives clear feedback and metrics for how the code will perform when released to a customer-facing environment. After passing the appropriate tests in pre-production, code can be automatically released to production using whatever method or model meets your requirements.
There are a large number of tools for facilitating CI/CD processes in DevOps-based teams. Some of the most popular ones include:
Configuration Management - Ansible, Puppet, Chef
Code Management - GitHub, GitLab, BitBucket
Build - Jenkins, Bamboo, TeamCity
Testing - Selenium, JUnit, SonarQube
Deployment - Argo, Spinnaker, Octopus Deploy
I tried to break-down the building blocks involved in CI/CD pipeline. Have a nice day :)
