Starting with the CI/CD process, the series of steps that include all the phases and are responsible for creating an automated and seamless software delivery is called the CI/CD pipeline workflow. Using the CI/CD pipeline, software release artifacts can move and advance in the pipeline from the code commit phase to the test, build, deploy, and production phases. This concept is powerful because once a pipeline is specified, some or all of it can be automated, speeding up the process and reducing errors. In other words, the CI/CD pipeline makes it easier for businesses to automatically deliver software multiple times a day.
DevOps engineers are often confused with the CI/CD pipeline because of the automation of the phases in CI/CD. Although different tools can automate the various complex phases of CI/CD, the entire software supply chain of CI/CD can still be disrupted by human intervention. Start by understanding the various phases of the CI/CD process and why the CI/CD pipeline is critical for an organization to deliver code quickly and on a large scale.
CI/CD phase: Get to know the people, process, and technology
Enterprise application development teams are typically composed of developers, testers /QA engineers, operations engineers, and SRE (Site Reliability Engineer) or IT operations teams. They work closely together to deliver high-quality software to customers. CI/CD is a combination of two separate processes: continuous integration and continuous deployment. The main steps are listed below.
CI continuous integration
Continuous integration (CI) is the process of building software and completing initial testing. Continuous deployment (CD) is the process of integrating code with the infrastructure, ensuring that all tests are completed and policies are followed, and then deploying the code to the intended environment. Of course, many companies have their own processes, but the main steps are as follows.
CI: Code submission
Personnel: Developers and engineers, database administrators (DBAs), infrastructure teams Technologies: GitHub, GitLab, BitBucket Process: Code commit phase also known as version control. Commit is the action of sending the latest changes written by the developer to the repository. Each version of the code written by the developer is stored indefinitely. After discussing and reviewing changes with collaborators, developers write code and submit it when software requirements, feature enhancements, bug fixes, or change requests are complete. The repository that manages editing and committing changes is called source code management (SCM tools). After the developer submits the code (code push request), the code changes are merged into a branch of base code stored in a central repository, such as GitHub.
CI: Static code analysis
People: Developers and engineers, database administrators (DBAs), infrastructure teams, testers Technical: GitHub, GitLab, BitBucket Process: Once a developer writes code and pushes it to the repository, the system automatically triggers the next code analysis process. Imagine a step like this: Submitted code goes directly to build, but fails during build or deployment. This is a slow and expensive process in terms of resource utilization, whether machine or human. The static policy of the code must be checked. SAST (Static Application Security Test) : SAST is a white-box testing method that uses SAST tools such as SonarQube, Veracode, AppScan, etc., to examine code internally to find software defects, vulnerabilities, and weaknesses (such as SQL injection). This is a quick check to see if your code has syntax errors. While this phase lacks the ability to check for runtime errors, this will be done at a later stage.
Putting additional policy checks in the automation pipeline can significantly reduce the number of errors found later in the process.
CI: build
Personnel: developers and engineers
Technology: Jenkins, Bamboo CI, Circle CI, Travis CI, Maven, Azure DevOps
Process: The goal of the continuous integration process is to accept regular code submissions and continuously build binary artifacts. The continuous integration process helps find bugs faster by checking that new modules added work well with existing ones. This helps reduce the time it takes to validate new code changes. Build tools help you compile and create executables or packages (.exe,. DLL,.jar, etc.) depends on the programming language used to write the source code. SQL scripts are also generated during the build process and then tested with the infrastructure configuration file. In short, the Construction phase is the phase where the application is compiled. Other sub-activities of the build process include artifact storage, build validation, and unit testing.
CI: Test phase
Personnel: testers and QA engineers
Technologies: Selenium, Appium, JMeter, SOAP UI, Tarantula
Process: Release a build process with a set of automated tests to verify the accuracy of the code. This phase helps prevent errors from reaching the product. Depending on the size of the build, this check can last from seconds to hours. For large organizations with multiple teams committing and building code, these checks will be run in a parallel environment to save valuable time and notify developers of bugs early.
These automated tests are created by testers, or QA engineers, who have created test cases and scenarios based on user stories. They perform regression analyses, stress tests, to check for deviations from expected output. The activities involved in testing include fitness testing, integration testing, and stress testing. This is a very advanced level of testing. You’ll find problems here that the developers who developed the code might not have known about.
Integration testing:
Integration tests are performed using tools such as Cucumber and Selenium, in which individual application modules are assembled and tested as a group while being evaluated for compliance with specified functional requirements. After the integration test, someone needs to approve moving the update set in that group to the next stage, which is usually a performance test. This validation process can be cumbersome, but it is an important part of the process. A number of new solutions have emerged during the verification process.
Load and pressure testing:
Load balancing and stress testing are also performed using automated testing tools (such as Selenium, JMeter, etc.) to check whether the application is stable and performing well in a high-traffic environment. This test is usually not run on every update because full stress tests are long run. When major new features are released, multiple updates are grouped together and complete performance testing is done. In cases where a single update is moved to the next phase, the pipeline may include Canary testing as an alternative.
Continuous deployment: bake and deploy
Personnel: infrastructure engineer, field reliability engineer (SRE), operation and maintenance engineer
Technology: Spinnaker, Argo CD, Tekton CD
Process: Once the testing phase is complete, the code cleared of the standard can be deployed to the server, where it will be integrated with the main application. Before they are deployed to production, they will be deployed to test/staging or beta environments used internally by the production team. Before moving the build to these environments, the build must go through two subphases Bake and Deploy. Both stages are inherent to Spinnaker.
CD: Bake
Bake means to create an instance of an immutable image from source code that has the current configuration in a production environment. These configurations might be things like database changes and other infrastructure updates. Spinnaker can trigger Jenkins to perform this task, and some organizations prefer to use Packer.
CD: deployment
Spinnaker will automatically pass the baked image to the deployment phase. This is where the server group is set to deploy to the cluster. Similar to the testing process above, the same functional process is performed during the deployment phase. Deployment moves first to test, phase, and finally to production, and then to approval and inspection. The whole process is handled by tools like Spinnaker.
CD: validation
This is the key to optimizing the entire CI/CD process for the team. Since so much testing has been done now, failures should be rare. But any breakdowns at this point need to be resolved as quickly as possible to minimize the impact on the end customer. Teams should also consider automating this part of the process.
Deployment to production is performed using deployment policies such as blue-green deployment, Canary analysis, rolling updates, and so on. During the deployment phase, the running application is monitored to verify that the current deployment is correct or that a rollback is required.
CD: monitoring
Technology: Zabbix, Nagios, Prometheus, Elastic Search, Splunk, AppDynamics, Tivoli Process: To make a software distribution failsafe and robust, it is critical to track the health of the distribution in a production environment. Application monitoring tools track performance metrics such as CPU utilization and release latency. The log analyzer scans the log flow generated by the underlying middleware and operating system to identify behavior and trace the source of the problem. In case of any problem in the production process, the relevant personnel will be notified to ensure the safety and reliability of the production environment. In addition, the monitoring phase helps the enterprise gather information about how new software changes contribute to revenue and helps the infrastructure team track trends in system behavior and capacity planning.
Continuous deployment: Feedback and collaboration tools
Technical: Zen, ServiceNow, Slack, Email, HipChat The goal of the DevOps team is to deliver faster, more consistently, and then reduce errors and performance issues. This is achieved through Slack or email frequent feedback to developers and project managers on the quality and performance of new releases, as well as timely fare increases in ITSM tools. Often, feedback systems are part of the overall software delivery process; So any changes in the delivery process are frequently recorded in the system so that the delivery team can take action on them.
The enterprise must evaluate a holistic continuous delivery solution that can automate or facilitate the automation of the above phases.