What is an artifact?

Artifacts are binary files generated by source code compilation and packaging. Different development languages correspond to different formats of binary files. These binaries are usually used to run on the server or as compilation dependencies, and “artifact management” is an important part of configuration management.

Typically, these components are archives of various files, including Java bytecodes in class files, C object files, text files, and binaries. Component formats such as Java JAR, WAR, EAR; A common ZIP or.tar.gz file. Other package formats, such as NuGet packages, Ruby Gems, NPM packages; Executable file formats, such as. Exe or. Sh files, Android APK files, various installer formats.

According to the use scenario, products can be roughly divided into three categories

  1. External imported third-party components
  2. Product internal dependency package, common SDK
  3. Product delivery installation package

According to the development language, artifact types include the following types:

  • A Generic File is an artifact of a common File type.
  • Docker
  • Maven
  • npm
  • PyPI
  • Helm
  • Composer
  • NuGet
  • Conan

Why product management?

  1. External dependencies download slowly
  • It affects the speed of r&d and construction
  1. Incorrect Version management (SVN, FTP)
  • Delivery packages are managed using FTP or SVN, and the management granularity is relatively coarse. Under this extensive product management approach, storing and retrieving different types of packages is a headache, version tracking is extremely confusing, and teamwork is a barrier.
  • Due to regulatory constraints, one-click deployment is an impossible task, and intelligent package delivery across network segments relies on manual copying
  1. Risk of security breach
  • The more dependent components, the higher the risk of introducing vulnerabilities
  • The download management of third-party dependent packages is chaotic and there is no access control
  • The deeper the bug, the longer it will take to fix it
  1. Product storage risk
  • The in-house artifact repository is a single point without cluster deployment
  1. Waste of resources
  • Because there is no unified product library, there is the problem of repeated construction; Maintenance costs are high, or currently there is no maintenance at all

Products and CI/CD assembly line

For CI/CD assembly line, the product plays a roleThe essentialIt is the end of CI and the beginning of continuous delivery of CDS.

If there is no effective product management strategy and tools, it is impossible to build an efficient pipeline; Moving away from artifact management and building from code at a time is unacceptable for any enterprise organization and does not meet the “Build once, use many times” principle.

Throughout the development process, artifacts are critical to testers and operations staff, who are focused on getting the release they need to test and deploy. Without effective artifacts management, the entire DevOps value stream will have problems connecting. You may find yourself in a situation where the test student will ask in various ways which version is available to test, where the package is, etc.

Metadata for the package

What is metadata for a package? When someone gives you a package, how do you know what defect changes it contains, what code commits it contains, md5,hash, etc. This information is crucial for testers to track the introduction of problems, subsequent improvements, version regressions and, in plain English, the history of the product.

So where does this information come from? Of course, the CI pipeline is constantly being built, where requirements, code submissions can be collected. If your organization has purchased Jfrog products, this feature is particularly prominent on its platform.

Promotion of products

​In development practice, most teams will prepare DEV, TEST, UAT, RELEASE and other different environments, build different assembly lines accordingly, and TEST different products before deploying them to different environments. The promotion of products is derived here, that is, to set different access control for products.To sum up, products are closely related to CI/CD assembly line and inseparable, so the use scenario of good products should be considered when designing assembly line.

Product management tool

​

As mentioned above, due to the importance of product management, corresponding product solutions have been developed to unify the management of software products in different formats. In addition to the basic storage functions, it also provides important functions such as version control, access control, security scanning, dependency analysis, and finally establishes a “single trusted source”, which is a standardized way for enterprises to deal with all package types generated during software development.At present, the mainstream product management tools in the main market mainly include the following:

Nexus

Nexus is an “out of the box” system that doesn’t need a database, but uses a file system plus Lucene to organize data. Nexus uses ExtJS for interface development and Restlet to provide complete REST APIs that are integrated with Eclipse through M2Eclipse. Nexus supports WebDAV and LDAP secure authentication. ​

Nexus is one of the few products that supports almost all major product formats and offers a free version of the product management product, which is the choice of most small and medium companies for most business scenarios. However, the free version does not offer a high availability solution. Price reference:www.sonatype.com/products/pr… Because Nexus has no agents in China, its awareness is limited. In fact, Nexus is justsonatypeA product solution that provides product management solutions for the software development cycle.

Jfrog Artifactory

Jfrog is an Israeli company that specializes in product management environments and provides commercial solutions, so its products cost money.The following is a comparison of Jfrog Artifactory and Nexus for reference only. If you pay for it, you’ll get more support and services than the free Nexus, including high availability, vulnerability analysis of components, multi-location distribution, and so on. It’s not that the Nexus doesn’t work, but most of us use the free version of the Nexus, which has a similar plan for the paid version.

Harbor

Harbor is an open source enterprise Docker Registry project of VMware, whose goal is to help users quickly build an enterprise Docker Registry service.

Based on the official Registry V2 implementation, it provides management UI, Role Based AccessControl, AD/LDAP integration, and Auditlogging and other functions required by enterprise users. Open source Docker Distribution has been extended by adding enterprise essential features such as security, identity, and administration.

Harbor has become the main tool for private Docker/Helm management. Compared with Nexus, Harbor has more advantages in Docker image management, providing image synchronization service and supporting team project isolation. In practice, the author found that Nexus has some problems in team isolation of Docker images.

WePack

​

WePack is a separate product management service spun out of Tencent Coding’s previous DevOps to support private deployment. Perhaps a separate product management tool is a better way to tap into the user landscape than a large, all-encompassing DevOps platform.

How to manage products?

In order to unify the management of packages in different language formats, almost all of the above product management tools manage organizational artifacts as follows.

The hierarchy of artifacts is Repository > Package > version, with each level described as follows:

  • Warehouse: Used to manage different types of warehouses and package resources under the warehouse. You can set the access permission of the warehouse.
  • Package: The basic unit that a build product provides external access to and describes the purpose and usage of the current build product.
  • Version: Lists all build artifacts under a package, documenting in detail the iteration changes of each build artifact.

Specification of artifact library naming

If the team is relatively large and the requirements for product management are not high, the above methods can basically meet the needs. However, if the building company level needs to standardize some naming, as shown below

Product version number is standardized

The version number of an artifact is used to mark a particular artifact, and the normalization of naming helps to automate script writing and pipeline reuse.

Normalize artifact permissions

Whether based on open source tools or proprietary tools, it is also necessary to design permissions based on the artifact repository to achieve the isolation of the team’s products.

Security risks of open source products

For the management of products, most people stay in the idea of just storage, pull and use, I was this kind of thinking before this year. The Log4j2 security incident at the end of 2021 caused an uproar in the IT community. This open source component is involved in almost all Java applications, and every company had to examine their products to see if this risk was introduced.

Through this incident, we began to pay attention to the possible risks of open source components, which is currently a hot research and development process of “supply chain security”, is also an important part of DevSecOps.

As the product management in the research and development process, the audit mechanism in the introduction stage, the safety in use, has become more and more attention. As shown above, the organization needs to introduce the component review system to prevent developers from randomly pulling open source products from the Internet, and establish a real-time vulnerability scanning mechanism to form an organization-level whitelist warehouse.

SBOM- Software Bom

Modern software is largely assembled using third-party and open source components that fuse together in complex and unique ways and integrate with original code to achieve desired functions. In addition to adding security review mechanisms during the introduction of open source components, IT companies often need to pay attention to the composition of the software products they develop or use, such as the list of ingredients we see on the food package when we buy food in the supermarket, indicating all the ingredients used.

SBOM (Software Bill Of Materials) comes into being in order to accurately understand the components contained in Software. SBOM includes a variety Of key information, such as component name, version number, supplier, etc., which plays a key role in analyzing Software security. This information can be used to trace the original supply chain of software, greatly improving developers’ understanding of the security risks of their software, and helping enterprises improve their efficiency in network security risk analysis, vulnerability management, and emergency response. ​

For software development enterprises, SBOM can effectively control the risks of open source components, and help enterprises identify and eliminate security defects and licensing risks of open source components earlier. For software procurement enterprises, SBOM can help procurement decision makers easily understand whether the developer’s software has open source component risk. For software developers, SBOM helps them fully and accurately understand the open source components of their software.

conclusion

  • Product management is an important link in DevOps practice. It plays an important role in collecting process information.
  • At the same time, the introduction and use of products will have security risks, organizations need to pay attention to this, to avoid a series of risks like Log4j2 security incidents;
  • As a practitioner, the management of products needs to be combined with the needs of the organization and assembly line, specify the corresponding norms, to avoid confusion;
  • Good product management process, can reduce the development of self-testing and testing personnel to receive the test of the link process of inefficient communication;

Here is just to the product management to do a global comb, the follow-up will be on the specific knowledge points in detail.