By Isaac Sacolick
Configuring basic continuous integration and continuous delivery (CI/CD) pipelines that automate packaging, compiling, and pushing code to application delivery environments is considered a fundamental devsecops practice. By automating a path to production, devsecops teams can reduce errors, increase deployment frequency, more quickly resolve production issues, and improve team culture.
Creating a basic CI/CD pipeline can be a catalyst for driving a culture of continuous improvement. For example, many teams will add test automation, error checking, and alerting to their pipelines to avoid publishing defects or breaking builds that impact developer productivity.
“Some organizations believe developer experience to be a mystical art involving arcade machines or table tennis, but developers get the most satisfaction from the smooth flow of their changes from commit to customer,” says Paul Stovell, founder and CEO of Octopus Deploy. “When your developers want to do great work, CI/CD is the difference between total frustration and developer joy.”
Developing CI/CD pipelines and their underlying platforms is a mature devsecops discipline, but there is room for improvement. Here are six ways to improve the current state of CI/CD pipelines and deliver meaningful business impacts.
6 ways to get more from your CI/CD pipelines
Increase continuous testing with genAI
Eighty percent of respondents to the 2023-24 World Quality Report said that 25% to 50% of their automated testing was integrated into delivery pipelines. So it’s no surprise that 39% of respondents identified CI/CD as the top-most critical skill for quality engineering associates, ranked second behind coding skills. The implication is that there is a will to improve continuous testing, but many organizations still have a “quality debt”—a backlog of tests that aren’t automated in their CI/CD pipelines.
David Brooks, SVP of evangelism at Copado, says, “You would think test automation is well adopted, but the truth is that many companies still rely on manual testing, and those that automate barely cover a third of their features. In reality, maintenance proves to be too much.”
Brooks refers to the maintenance work, which includes updating automation when the code changes, improving test performance, and increasing test data. Synthetic data is a potential solution to generating a more comprehensive test data set, and genAI may prove to be a game changer for QA in expanding the number of automated tests and simplifying their maintenance.
“AI will finally make automated testing a reliable part of CI/CD pipelines, rather than a flaky gatekeeper that slows teams down,” says Gevorg Hovsepyan, head of product at Mabl. “Most development teams are looking to genAI to generate test cases, but if those new tests constantly fail, CI/CD pipelines will grind to a halt. Using genAI to automatically update tests as the product changes is the more impactful way to advance CI/CD capabilities.”
Another path to improve continuous testing is to embed performance, stress, and scalability testing into CI/CD pipelines. Performance testing tools like Gatling, LoadNinja, LoadRunner, and Katalon have integrations with top CI/CD platforms.
Target continuous deployment
Continuous testing is one prerequisite to continuous deployment, a process where devsecops teams extend CI/CD to deploy to production environments. My checklist for continuous deployment readiness also includes having development teams use feature flagging, developing a canary release strategy, and using an AIops platform in IT operations.
The business impacts of continuous deployment can be important for organizations where deploying frequent changes and addressing application production issues quickly is essential. Many SaaS businesses, companies developing customer-facing applications, and others building mission-critical employee applications use DORA metrics to measure how continuous deployment and other devsecops practices drive business impacts.
According to the State of CI/CD Report 2024: The Evolution of Software Delivery Performance, continuous deployment significantly reduces the lead time for code changes, a DORA metric defined as the time from code committed to having the code successfully in production. Of those able to deploy multiple times per day, 53% saw a lead time for code changes of less than one day, compared to the 27% who deployed between once per hour and once per week, and 9% who deployed between once per week and once per month.
“AI-enabled devops tools promise to deliver 30% or more in developer productivity,” says Kumar Chivukula, founder and CEO of Opsera. “After the first wave of deployments, enterprises are now desperately looking for an automated mechanism to capture insights, KPIs, and DORA metrics to prove industry claims and realized ROI.”
Improving lead time for code changes can be significant for applications where defects and downtime result in lost revenue, poor customer experiences, or employee workflow disruptions.
Embrace hybrid CI/CD
One surprising data point in the State of CI/CD report was the number of CI/CD platforms respondents had in place and how it impacted DORA metrics. Companies using a hybrid approach of self-hosted and managed CI/CD platforms outperformed those who standardized on one approach or were not using CI/CD platforms.
Of the companies using a hybrid approach, 49% had a lead time of less than one week for changes, and 24% had a lead time of less than one day. Sixty-six percent could typically restore service performance from an unplanned outage in under a day, and 25% could do so in under an hour. These rates were significantly better than those using only one approach. The report also showed that organizations using three or fewer CI/CD platforms generally outperformed those with more than three tools.
There are many reasons why organizations may have multiple CI/CD platforms. For example, a company may use Copado or Opsera to deploy apps to Salesforce, use Jenkins for data center apps, GitHub Actions for cloud-native applications, and then inherit implementations using AWS CodeBuild and AWS CodePipeline after acquiring a business. The research suggests the benefits of having multiple solutions but recommends consolidating and standardizing solutions with similar capabilities.
Shift-left security with CI/CD plugins
One important area to research, proof-of-concept, and implement is using plugins to integrate third-party capabilities into CI/CD pipelines. Jenkins, the CI/CD platform with the highest market share, advertises 1,900 plugins, with its top plugins supporting connections to Git, Jira, and Kubernetes. Security and code quality plugins are critical to evaluate and can minimize vulnerabilities before the code passes builds and is deployed.
“Underutilized capabilities include predictive analytics for identifying potential deployment failures and AI with quality code review to identify bugs, security vulnerabilities, and data governance issues,” says Aislinn Wright, VP of product management at EDB. “These tools can greatly enhance the agility and efficiency of devops processes, yet they require a higher level of technological maturity and integration effort, which may contribute to their slower adoption rates.”
Security capabilities that plug into CI/CD pipelines include container security scanning, static application security testing (SAST), code quality scanning, and software supply chain vulnerability checking.
“Business leaders prioritize reliable, secure, high-value features for customers with zero SEV-1 or SEV-2 issues in production, delivered swiftly and at scale,” says Peter McKee, head of developer relations and community, Sonar. “Devops serves as the gatekeeper to these needs, but code quality testing is a crucial aspect often overlooked. While unit testing, integration testing, and end-to-end testing ensure functionality, they miss assessing code quality. Including static code analysis in the CI/CD process ensures clean code, fostering reliability, maintainability, and security, vital for meeting modern demands.”
Secure and improve pipeline observability
Beyond plugins that provide security capabilities, devsecops teams must also take steps to secure CI/CD pipelines. OWASP’s CI/CD security cheat sheet is a good resource for reviewing CI/CD risks, secure pipeline configurations, identity and access management (IAM) considerations, managing third-party code, and other best practices. Top CI/CD security risks include:
Devsecops teams must strike a balance between CI/CD enhancements that accelerate deployment frequencies and those that address security risks. In addition to securing their pipelines, teams should improve devops observability to help identify performance issues, track testing bottlenecks, and enable debugging of issues connecting to third-party services.
One approach to bringing security and operational considerations together is leveraging tools that support policy as code (PaC). These systems abstract policies and rules into code, providing devsecops teams with a scalable way to capture, implement, and scale security and operational business rules.
“Policy-as-code is a powerful capability for consistently managing critical policies on highly sensitive data, providing a self-documenting, automated system for security, governance, and devops teams,” says Mike Scott, CISO of Immuta. “Policy-as-code can be implemented using a CI/CD pipeline that performs testing and validation, automatically deploying validated policies to production environments.”
Devsecops organizations with many active pipelines, integrated services, and plugins may find ways to simplify and create reusable pipelines when the underlying business rules are developed with PaC platforms and services.
Understand the business impacts
Devsecops teams should explore these more advanced options and determine which DORA metrics to implement through continuous improvement cycles. But Srikumar Ramanathan, chief solutions officer at Mphasis, shares a key reminder that the final objective of any system is to serve the business.
He says, “Often, we as technologists get carried away in implementing the latest and the geekiest technology, just for its own sake. Shifting left really means taking the business view of things. This is very much required when considering QA, security, observability, and automation.”
A best practice is to define beneficiaries and value propositions from those who benefit from devsecops operational and security improvements. From there, devsecops teams can decide what capabilities to focus on and target performance metrics with meaningful business value.
