First posted on Jenkins Chinese community

Implementing DevSecOps practices on an enterprise scale can be challenging. Because different applications within an organization are using multiple programming languages, automated testing frameworks, and security compliance tools, it becomes difficult for each team to build and maintain pipelining.

Regardless of which particular technology stack the application uses, most pipelines will follow the same common workflow. The template Engine plug-in (JTE for Jenkins Template Engine) allows you to gain efficiency by creating tool-independent template workflows that can be reused by each team.

As technical advisors to both public and private sector clients, we find at Booz Allen that every new project builds the DevSecOps pipeline from the ground up. By developing the Jenkins template engine, we have seen pipeline development reduced from months to days, and now we can reuse tool integration while bringing new levels of governance to the Jenkins pipeline.

Pipeline template

Organizations benefit from having application developers focus on what they do best: building applications. Supporting this means establishing a centralized DevOps team responsible for maintaining the platform infrastructure and creating a CI/CD pipeline for development teams to use.

With the rise of microservices-based architectures, a centralized DevOps team can support many different development teams at once; All of these teams may leverage different programming languages and automated testing tools.

While tools may differ between development teams, the workflow is usually the same: unit testing, static code analysis, building and publishing artifacts, deploying it, and then performing different types of testing against the deployed application.

The template engine plug-in allows you to remove Jenkinsfiles from each repository defined by the team for inheritable common workflows. Instead of defining the entire pipeline for each repository, the team provides a tool profile that uses the workflow.

JTE of actual combat

Let’s demonstrate template reusability with a simple example: The pipeline template example: Unit_test () build() Static_code_analysis () template uses the steps provided by the library to outline the steps that workflow teams must implement. While templates are executed like any other Jenkinsfile (which means support for standard scripting and declarative syntax), the goal of templates should be to read in plain English and avoid any technical implementation.

Using templates in this way, you can separate the business logic of the pipeline (when it should happen) from the technical implementation (what will actually happen). The result is a CI/CD pipeline that proves very manageable when supporting multiple teams simultaneously.

The steps outlined by this template (UNIT_test, Build, and Static_code_analysis) are specifically named. In this way, the different libraries the team can use share the same pipeline.

To implement the template

Implementing shareable pipelining using a template engine requires several key components:

  1. Pipeline template: Outlines the workflow to be executed
  2. Library: Provides technical implementations of workflow steps
  3. Configuration file: Specifies the library to use and its configuration

Step 1 Create the pipeline configuration repository

The pipeline configuration repository is used to store common configuration and pipeline templates inherited by the team.

This sample pipeline configuration repository will later be configured as part of the governance layer: JTE’s mechanisms allow you to build hierarchical configurations that represent your organization.

The governance layer has three aspects:

  1. Pipeline template
  2. Library resource list
  3. Layer configuration file (pipeline_config.groovy )

Pipeline templates and configuration files for the governance layer are stored in the pipeline configuration repository.

When configuring the governance layer in Jenkins, you provide a source control location for the repository containing the above components as well as the base directory where these artifacts can be found.

Step 2 Create a pipeline template

Next, we will create a Jenkinsfile for the governance layer. In JTE, Jenkinsfile is the default pipeline template that will be used for execution.

Jenkinsfile
unit_test()
build()
static_code_analysis()
Copy the code

Step 3 Create a library

The template engine plug-in implements a version of the Jenkins shared library to enhance the library’s reusability. The repository is the root directory in the source code repository that has been configured as the repository source at the governance layer.

In our example, the pipeline template needs to perform unit tests, package artifacts, and run static code analysis.

Suppose we have some teams using Gradle and some teams using Maven to build and test their applications, but they will all use SonarQube to perform static code analysis.

In this scenario, we should create gradle, Maven, and Sonarqube libraries.

|- gradle/
  \-- build.groovy
  \-- unit_test.groovy
|- maven/
  \-- build.groovy
  \-- unit_test.groovy
|- sonarqube/
  \-- static_code_analysis.groovy
Copy the code

Step 4. Implementation procedure

The library implementation steps are exactly the same as writing regular global variables as part of the default Jenkins shared library.

For the purposes of this demonstration, we will have each step print out the step name and contribution library.

gradle/build.groovy
void call(){
    println "gradle: build()"
}
Copy the code

Read more about the JTE development library.

Step 5 Create a configuration file

The JTE configuration file is called pipeline_config.groovy.

At the governance layer, we will create a configuration file that specifies common configuration between applications. In this case, both applications use sonarqube libraries.

pipeline_config.groovy
libraries{
  merge = true // allow individual apps to contribute additional libraries
  sonarqube
}
Copy the code

Next, we will create two more repositories that represent Maven and Gradle applications. In these repositories, we only need a specific pipeline_config.groovy file.

Each of these repositories contains the pipeline_config.groovy configuration file for the application.

maven app: pipeline_config.groovy
libraries{
    maven
}
Copy the code
gradle app: pipeline_config.groovy
libraries{
    gradle
}
Copy the code

Step 6 Configure the governance layer in Jenkins

Now that we have the pipeline configuration repository and repository source repository, we can configure the governance layer in Jenkins:

The configuration shown above can be found below Manage Jenkins >> Configure System.

With the template engine, you can represent this structure through folders in Jenkins to create a pipeline governance hierarchy that matches the organization’s classification.

Step 7. Create a multi-branch pipeline for both applications

When creating a multi-branch pipelined Project for each application, the template Engine plug-in provides a new Project Recognizer called the Jenkins Template Engine. The e project is set up to use the template engine framework for all branches in the repository.

You can also set up the Jenkins Template Engine Project identifier for GitHub organization projects, enabling you to easily share the same pipeline across the GitHub organization!

Step 8. Run the assembly line

In this way! Now, both applications will leverage exactly the same pipeline template, with the flexibility to choose which tools should be used at each stage of the workflow.

Here is sample output from console logs for two applications running pipelined:

Gradle:
[JTE] Obtained Template Configuration File pipeline_config.groovy from git https://github.com/steven-terrana/example-jte-configuration
[JTE] Obtained Template Configuration File pipeline_config.groovy from git https://github.com/steven-terrana/example-jte-app-gradle.git
[JTE] Loading Library sonarqube from git https://github.com/steven-terrana/example-jte-libraries.git
[JTE] Loading Library gradle from git https://github.com/steven-terrana/example-jte-libraries.git
...
[JTE] Obtained Template Jenkinsfile from git https://github.com/steven-terrana/example-jte-configuration
[JTE][Step - gradle/unit_test]
[Pipeline] echo
gradle: unit_test()
[JTE][Step - gradle/build]
[Pipeline] echo
gradle: build()
[JTE][Step - sonarqube/static_code_analysis]
[Pipeline] echo
sonarqube: static_code_analysis()
[Pipeline] End of Pipeline
Copy the code
Maven:
[JTE] Obtained Template Configuration File pipeline_config.groovy from git https://github.com/steven-terrana/example-jte-configuration
[JTE] Obtained Template Configuration File pipeline_config.groovy from git https://github.com/steven-terrana/example-jte-app-maven.git
[JTE] Loading Library sonarqube from git https://github.com/steven-terrana/example-jte-libraries.git
[JTE] Loading Library maven from git https://github.com/steven-terrana/example-jte-libraries.git
...
[JTE] Obtained Template Jenkinsfile from git https://github.com/steven-terrana/example-jte-configuration
[JTE][Step - maven/unit_test]
[Pipeline] echo
maven: unit_test()
[JTE][Step - maven/build]
[Pipeline] echo
maven: build()
[JTE][Step - sonarqube/static_code_analysis]
[Pipeline] echo
sonarqube: static_code_analysis()
[Pipeline] End of Pipeline
Copy the code

Benefits of template engines

Applied Organizational Governance

With the template engine plug-in, you can define enterprise-level, approved workflows that teams can use regardless of the tool. This top-down approach makes it easy to extend and enforce DevSecOps principles in an organization.

Optimize code reuse

Virtually every team in an organization doesn’t have to think twice about how to do the same thing. At Booz Allen, we’ve seen pipeline development times drop from months to days as we continue to reuse and extend the template engine library portfolio as part of our solution delivery platform.

Simplify line maintainability

Often, DevOps engineers find themselves building and supporting pipelines for multiple development teams at once. The template engine plug-in allows DevOps engineers to scale faster by separating workflow from technical implementation and consolidating pipeline definitions into a centralized location.

Get involved!

The template engine plug-in is open source and available at the Jenkins Update Center.

Feedback and contributions are always appreciated! If you have an interesting use case or want to ask questions, try the template Engine plugin in Gitter.

Advanced features

  • Configuration file conditional inheritance
  • Externalize the library configuration
  • Aspect-oriented lifecycle hooks
  • Multiple pipeline templates
  • Default step implementation
  • Configuration file DSL sandbox

More resources

For this demonstration

  • Pipeline configuration repository
  • The sample library
  • Maven repository example
  • Gradle repository example

Other resources

  • Template engine documentation
  • The source code
  • Booz Allen’s SDP pipeline library
  • Booz Allen Hamilton

Translator: linan607