With more and more components sinking into the Business, everyone is likely to face the stages of component creation, configuration and release. The basic ability of componentization is becoming more and more important. We hope to have a universal release plug-in to improve the development efficiency of our components and solve some problems we may face in componentization. Our release plugin provides the following capabilities to address the problems we are currently experiencing with fastlook componentization:

  • Easy and fast maven configuration upload
  • Pre-validation of component releases
  • API Module Export
  • One-click publishing for Intranet or extranet

Let’s take a look at each of these abilities in detail.

Optimize maven publish configuration efficiency

Currently, components are configured to publish using Maven, and it would be a lot of work for each component to configure Maven separately.

Maven Publishing configuration

Let’s start with the general process of defining a Maven release.

Plugins {id 'Java' id 'maven'} //2. Define Configuation Configurations {testArchives} //3. Define collection artifacts artifacts {testArchives file: file('libs/ XXXX.aar ')} //4. Define publish task uploadTestArchives {repositories {mavenDeployer {repository(URL: URI (uploadRepo)) {authentication(userName: localProps.getProperty("emailName"), password: localProps.getProperty("emailPass")) } pom.project { groupId GROUP_ID version rootProject.xxxRVersion artifactId "xxxxx" }}}}Copy the code

When we add the maven configuration above to the Gradle file, we define an upload task named uploadTestArchives.

Maven-publish publish configuration

However, after you upgrade the Android Build Plugin to 4.2.0, you will find that Maven is deprecated and can no longer be used. Check gradle’s official API, maven is replaced by Maven-publish. Next, a brief introduction to Maven-publish. The overall release process is basically the same as previous Maven releases.

Plugins {id 'Java' id 'maven-publish'} task sourceJar(type: Jar) { archiveClassifier = "sources" } publishing { //2. Publications {maven(MavenPublication) {from components. Java artifacts = ["my-custom-jar.jar", sourceJar] } } //3. Address to be published definition repositories {maven {def releasesRepoUrl = layout. BuildDirectory. Dir (' repos/releases) def snapshotsRepoUrl =  layout.buildDirectory.dir('repos/snapshots') url = version.endsWith('SNAPSHOT') ? snapshotsRepoUrl : releasesRepoUrl } } }Copy the code

Publish configuration optimization for publish plug-ins

Let’s take a look at how the various components of Maven define the release task so far

Apply plugin: 'plugin.kuaikan.complier' compileConfig {upload {version = "3.4.release.0" artifactId = "arch"}}Copy the code

This has the advantage of saving both maven’s cumbersome configuration and the familiar cost of Maven-publish and Maven. This cumbersome configuration can be optimized using the Gradle plug-in. If you are not familiar with Gradle plugin, you can first understand gradle plugin implementation process. The official API portal: doc.yonyoucloud.com/doc/wiki/pr… The overall implementation can be divided into the following steps:

  1. Customize a Gradle plugin
  2. Define publish parameter configuration Extension
  3. Register the actions in the Gradle Project lifecycle, resolve the configuration parameters, and configure them to the necessary parameters for Maven.

Non-critical process code is ignored below

@Override void proceedAfterEvaluate(Project project) { CompilerExtension compilerExtension = project.compileConfig if (compilerExtension == null) { return } UploadExtension extension = compilerExtension.upload if (! extension.enable) { return } ... project.publishing.publications({ publications -> publications.create("kk", MavenPublication.class, { MavenPublication publication -> publication.groupId = getGroupName(project) publication.artifactId = extension.artifactId publication.version = extension.version if (project.extensions.findByName("android") == null) { publication.from(project.components.java) } else { publication.from(project.components.release) if (extension.source) { artifact project.tasks.getByName("sourceJar") } } }) }) project.publishing.repositories { artifactRepositories -> ArtifactRepositories. Maven {mavenArtifactRepository - > / / url, username, password here will distinguish between inner and outer net mavenArtifactRepository. Url = getRemoteKKUrl(project) mavenArtifactRepository.credentials { credentials -> credentials.username = getRemoteKKUserName(project) credentials.password = getRemoteKKUserPass(project) } } } }Copy the code

Pre-validation of component releases

With the advancement of componentization, almost all basic components and some business components are published to remote warehouses in the way of Maven. This method improves the reusability of components, but also introduces new problems compared to the previous time when all modules are in one project. There are a few obvious problems:

  • After each module is modified, a new version must be released before validation can be packaged in dependent modules.

When some bugs occur and the module needs to be modified for verification, a new version needs to be released. The verification cost is very high, and the released version is actually the test version, which is very risky

  • A module is developed in parallel with multiple branches and released separately, which may result in the loss of functionality of the released version

If the module changes very frequently, the corresponding dependent module is required to update frequently, which will be relatively high cost for the dependent party

In order to solve the above problems, it is necessary to establish a strict specification of module branching and versioning strategy, and to avoid the above problems by using compilation tools to detect violations whenever possible.

Branching strategy

There are two publishing branches:

  • Release_master: All official releases can only be released on this branch
  • Development branch: Release dev versions at will

Release_master has strict permission control. Only the owner of each business line has the merge permission. Other students can initiate merge requests. The developer will pull the development branch based on the above two branches and publish the changes to the local for verification.

Version of the strategy

There are two different naming methods for the official and grayscale versions:

  • Official: major.minor.relex.fix_version
  • Development version: Major.minor.5_dev_branch_name. Fix_version

Including: major: the major version number, usually on upgrading has great function to upgrade major minor: the version number, the function of the normal upgrade, usually is to increase the new interface, or the function of the changes will significantly affect the relying party or changes in the need to rely on cooperation to modify the release: 5_dev_branch_name: indicates the development version.

The main reason for adding a number is that you want dev version dependencies to work with the same version. For example 3.0.dev.0 and 3.0.release.0, gradle by default selects 3.0.release.

Release strategy

  1. Release versions can only be released in the Release_master branch

  2. You must rebase to the latest code

  3. The working directory must be clean, meaning that code must be merged into the remote branch before it can be published

  4. Libraries of the Release version are not allowed to rely on dev libraries

  5. Releasing a release version of the library can only be done in the Release_master branch.

Publish plug-in detection

There is no way to avoid all problems when there are only specifications, so it is very necessary to automate the inspection of these specifications. We have automated checks for our component version number management in our plug-in release

if (isUpload2ServerTask) { def checkTask = project.tasks.findByName('preBuild') if (checkTask == null) { checkTask = project.tasks.findByName('jar') } if (checkTask ! = null) { checkTask.doFirst { checkDependencies(project, extension.version) checkWorkingDirectory() checkBranchAndVersionForPublish(project, extension.version) checkUploadUser(project) } } }Copy the code
  • At the very beginning of the release plug-in execution, check that the current release version type matches the local branch. If the release number contains a release string, the current local branch must be release_master, otherwise an error is reported directly
  • The git command is used to check whether the local code has been changed, whether there is a commit that has not been pushed, and whether there is a commit that has been updated on the remote side. If so, an error is reported
  • If dev dependencies are included, an error is reported. If dev dependencies are included, an error is reported

Component one-click publishing extranet

One day, my colleagues A and B came to me at the same time. Both of them needed to make an SDK for external distribution, indicating that they hoped to use the basic components of the current APP as A general ability. It is reasonable and appropriate that the platform can ensure that all the basic capabilities are shared by multiple parties, whether it is provided for the use of apps within the company or for the use of SDKS distributed externally. However, at present, all our components are published on the Intranet of the company, and there is no way to rely on the SDK distributed externally in the way of remote dependence. For the scenario of internal and external network communication of this App component, the following solutions were considered at that time: #### Based on the basic components required by the external SDK, simply develop a set again and publish it to the external network Maven. This scheme has been passed at the very beginning. The main reason is the cost of development and maintenance. At present, there is no manpower to independently develop and maintain the basic components for external SDK.

Fat AAR technology is used to do AN AAR merge, combining the base components in a package into an external SDK

Using the technology of incorporating an AAR, all of our base components were bundled into an external SDK. At first glance, there seems to be no problem, but after in-depth investigation and thinking, there are many possible problems in the follow-up. Because we used Okhttp, Fresco and other commonly used libraries in the industry, once we put these libraries into the distributed SDK as source code, once the host project also relies on the same network library and image library, etc., the compilation will fail and the class duplication will occur. Once this problem occurs, the host project needs to dynamically remove dependencies that duplicate our class. This point is passed by us because the subsequent SDK maintenance problems will cause a relatively large cost.Copy the code

All components that need to be used are re-published to the Internet

Look at the current number of components is relatively large, there are more than 100 large and small basic components, the overall release cost is acceptable, but the components need to be maintained and upgraded later. If you maintain components for Intranet and extranet at the same time, you need to add different Maven addresses in different code repositories, provide different dependencies for Intranet and extranet, and maintain version numbers for Intranet and extranet respectively, which is costly. If we only maintain the components of the external network in the future, this scheme is not realistic, because we hope to only obtain the components we want to provide, and we want to target the components that are confused with the external SDK and components without source code, and the components used on the internal network are not confused with source code.Copy the code

Of the three different options, we chose the third option. However, the third scheme also has obvious disadvantages, such as high cost of subsequent release and extranet dependency configuration and maintenance. Therefore, we developed a set of release plug-ins for the third scenario, embedding all the complex switching logic inside the release plug-in to increase overall usage and access costs.

Support one click extranet component publishing

For the same component to support extranet publishing, essentially the same component needs to be published to both Intranet and extranet repositories. If we publish the same artifactId as groupId for both Intranet and extranet components, there is no guarantee that Gradle will obtain the dependency of Intranet and extranet in the end ifa project is configured with both Intranet and extranet repositories. So you also need to make sure that artifactId and groupId have a different parameter. The artifactId we chose is different from the one for exets and exets. All artifactId components published on exets will have a suffix of “-sdk” than artifactId components published on Intranet. For example, the following publication definition:

Apply plugin: 'plugin.kuaikan.complier' compileConfig {upload {version = "3.4.release.0" artifactId = "arch"}}Copy the code

If you publish to the Intranet, the component name of the Intranet is “arch”. If you choose to publish to the extranet, the extranet component name is “arch-SDK”.

The custom Configuration

If you use implementation or API provided by Gradle to rely on it, you must provide a implementation or API configuration for both Intranet components and exo-components. Like this:

If (upload2Sdk) {implementation 'com. Kuaikan. Client. Library: kv - SDK: 3.0 the 0' implementation 'com. Kuaikan. Client. Library: net - SDK: 3.0 the 0' implementation 'com. Kuaikan. Client. Library: base - SDK: 3.0 the 0'} The else {implementation 'com. Kuaikan. Client. Library: kv: 3.0 the 0' implementation 'com. Kuaikan. Client. Library: net: 3.0 the 0' implementation 'com. Kuaikan. Client. Library: base: 3.0 the 0'}Copy the code

The configuration of the whole Intranet and extranet release is tedious, and it is not easy to read, and subsequent modification is prone to error. To solve this problem, we customize gradle’s Configuration.

Project.configurations {// Automatically selects SDK or Intranet dependencies when you publish: Implementation automaticImpl // Implementation automaticImpl uses this dependency :API sdkApiOnly // Only release look Intranet uses this dependency :API KkApiOnly // Only publish look the Intranet will select this dependency :Implementation kkImplOnly // Only publish Sdk will select this dependency :Implementation sdkImplOnly // Enforce the default dependency normalApi normalImpl }Copy the code
  • AutomaticImpl/automaticApi: according to the task, automatic selection network dependent/since networks outside in translating
  • SdkApiOnly /sdkApiOnly: Participate in compiling and publishing only if the publishing task is selected to publish the extranet
  • KkApiOnly /kkImplOnly: Only the publishing task chooses to publish the Intranet, will participate in compiling and publishing
  • NormalApi /normalImpl: Participates in compiling and publishing regardless of the publishing task

Let’s take a look at gradle writing using a custom configuration.

AutomaticImpl 'com. Kuaikan. Client. Library: kv: 3.0 the 0' automaticImpl 'com. Kuaikan. Client. Library: net: 3.0 the 0' AutomaticImpl 'com. Kuaikan. Client. Library: base: the 3.0 release. 0'Copy the code

If you are not familiar with the configuration defined, can first take a look at the official document: docs.gradle.org/current/use…

In our scenario, all the configurations are relatively similar, we can look at the automaticImpl configuration processing.

project.configurations .automaticImpl .getAllDependencies() .forEach {dependency -> def isSdkOutTask = KKCompilePluginUtil.isSdkUploadTask(project) if (isSdkOutTask) { implementation dependency.group + ':' + dependency.name  + "-sdk" + ':' + dependency.version } else { implementation dependency.group + ':' + dependency.name + ':' + dependency.version } }Copy the code

In brief, the automaticImpl process is divided into the following steps: 1. Gets all dependencies for the current project’s automaticImpl 2. If the current publication type is extranuclear, replace automaticImpl with implementation, splicing – SDK 3 after component name. If the current publication type is Intranet, only replace automaticImpl with implementation, and other dependency configurations remain unchanged.

Let’s look at another gradle configuration scenario, our image library component. We use the Fresco secondary development component on the Intranet, but in the external SDK scenario, this component will cause the entire external SDK package size to grow too much, so we hope that the image library has Glide implementation at the bottom.

sdkApiOnly 'com.kuaikan.client.library:image-glide:' + rootProject.libImageGlideVersion
automaticApi 'com.kuaikan.client.library:image-api:' + rootProject.libImageApiVersion
kkApiOnly 'com.kuaikan.client.library:image-fresco:' + rootProject.libImageFrescoVersion
Copy the code

This way, when the gallery publishes the Intranet, it uses Fresco. An implementation of Glide used when publishing extranets.

Extranet release task

Based on the maven definition above, after supporting SDK task publishing, we only need to add a parameter enableSdk = true in the upload parameter to support publishing to the external repository.

apply plugin: 'plugin.kuaikan.plier' compileConfig {upload {enableSdk = true version = "3.4.release.0" artifactId = "arch"}}Copy the code

We can see the gradleTask display with or without this configuration. Extranet tasks cannot be publishedPublish extranet tasks

Just add the enableSdk configuration in the maven-Publish configuration.

 void proceedAfterEvaluate(Project project) {
   .....
    defineKkUploadTask(project, extension, artifactName)
        if (extension.enableSdk) {
            defineSdkUploadTask(project, extension, artifactName)
        }
   ....
   }
   
   private void defineSdkUploadTask(Project project, UploadExtension extension, String artifactName) {
        project.publishing.publications({ publications ->
            publications.create("sdk", MavenPublication.class, { MavenPublication publication ->
                publication.groupId = getGroupName(project)
                publication.artifactId = artifactName + "-sdk"
                publication.version = extension.version
                if (project.extensions.findByName("android") == null) {
                    publication.from(project.components.java)
                } else {
                    publication.from(project.components.release)
                }
            
            })
        })
   }
Copy the code
User-defined release task name

Maven-publish generates multiple tasks by default.

  1. GeneratePomFileForPubNamePublication: Pom file is generated
  2. PublishPubNamePublicationToRepoNameRepository: to issue the component to the remote warehouse
  3. PublishPubNamePublicationToMavenLocal: to issue the component to the local maven, the default address is the local m2 folder
  4. Publish: Publish components, POM files, and all tasks to the remote repository
  5. PublishToMavenLocal: Publishes components, POM files, and all tasks to local Maven

Once we support both Intranet and extranet publishing, there will be 10 tasks, and some tasks we don’t need, so we can customize the publishing task.

project.tasks.create(name: TASK_KK_TO_LOCAL, dependsOn: ["publishKkPublicationToMavenLocal"], group: "upload")
project.tasks.create(name: TASK_KK_TO_SERVER, dependsOn: ["publishKkPublicationToMavenRepository"], group: "upload")
project.tasks.create(name: TASK_SDK_TO_LOCAL, dependsOn: ["publishSdkPublicationToMavenLocal"], group: "upload")
project.tasks.create(name: TASK_SDK_TO_SERVER, dependsOn: ["publishSdkPublicationToMavenRepository"], group: "upload")
Copy the code

In this way, we will see four published tasks in the Upload group.

  1. UploadSdk2Local: uploadSdk2Local: uploadSdk2Local
  2. UploadSdk2Server: Publishes extranet components to remote repository
  3. UploadKK2Local: uploadKK2Local uploadKK2Local: uploadKK2Local uploadKK2Local: uploadKK2Local uploadKK2Local
  4. UploadKK2Server: uploadKK2Server publishes Intranet components to remote repositories

Support API export

With the advancement of modularization, each module is released in the form of Maven. Each module is independently compiled, and complex dependencies between modules can easily lead to version compatibility problems of module dependencies. For example, there are two modules, LibA and LibB, and the Main project, with the following dependencies:

LibB and Main both depend on LibA:

LibB is compiled based on LibA1.0.Main is compiled based on LibA2.0Copy the code

After the gradle dependency conflict is resolved, LibA version 2.0 is packaged into Apk. Since LibB relies on version 1.0, this requires that version 2.0 of LibA be compatible with 1.0. However, in the process of version upgrade, it is very easy to break version compatibility, for example, the following examples are from the App access shenze 3.2.4, and 3.2.11 version:

3.2.4 Public Class SensorDataAPI {public void trackTimerStart()} 3.2.11 Public Class SensorDataAPI {public String  trackTimerStart() }Copy the code

The second version of this method just returns a String, which looks compatible with the first version, but is actually incompatible at the bytecode level of the caller, respectively:

3.2.4 invoked-Virtual LSensorDataAPI->trackTimerStart()V LSensorDataAPI->trackTimerStart()Ljava/lang/String;Copy the code

The example above is a very common version incompatibility problem. There are many similar version incompatibility problems in daily development. Therefore, there needs to be a mechanism to avoid version compatibility problems.

For each module published to Maven, it will Export documents according to an API and generate two Maven modules. For example, the module LibA will generate: LibA: contains all the resources and class files liba-export: The export module contains only classes, methods, and member variables declared in the API export document. Using a custom dependency kkApiOnly or kkImplOnly, if the inventory exports the document in the API, two dependency declarations will be generated, such as:

KkImplOnly 'com. Kuaikan. Client. Library: the router - API: 1.1 the +' transformation is: CompileOnly 'com. Kuaikan. Client. Library: the router - API - export: the 1.1 release. +' runtimeOnly 'com. Kuaikan. Client. Library: the router - API: 1.1 the +'Copy the code

In this way, the dependency can only access the interface exported by the module, which brings the following advantages:

  1. Can greatly reduce module interface changes caused by compatibility problems.
  2. The unexported interface can be modified arbitrarily, without considering compatibility issues, and can be flexibly modified
  3. It provides maximum decoupling on the basis of providing only one module, so that the user can only see the exported API library.

Note the following points:

  1. Since the Android Gradle plugin disables the APK project compileOnly/ Provided, using kkImplOnly in the main project does not actually apply the API export
  2. Since the main project is recompiled each time, there are no version compatibility issues
  3. Module projects should never rely on another module using the Gradle API, as this will cause the entire module of the dependent module to be passed

API Export document

The rules

The API export document records all the interfaces exported by the module, and only the interfaces in the API document are exported to export. Meanwhile, in principle, the API export document can only be added and cannot be modified unless it is 100% sure that all the dependent parties have updated to the latest version. Here is an example of an API export document:

com.alibaba.android.arouter.facade.template.IInterceptor
    public void process(com.alibaba.android.arouter.facade.Postcard, com.alibaba.android.arouter.facade.callback.InterceptorCallback)
com.alibaba.android.arouter.facade.template.IInterceptorGroup
    public void loadInto(java.util.Map)
com.alibaba.android.arouter.facade.template.IPolicy
    public int getFlag()
com.alibaba.android.arouter.facade.template.IProvider
    public void init(android.content.Context)
com.alibaba.android.arouter.facade.template.LocalInterceptor
com.alibaba.android.arouter.launcher.ARouter
    public static final java.lang.String RAW_URI
    public static final java.lang.String PATH
    public void init(com.alibaba.android.arouter.launcher.ARouterConfig)
    public static com.alibaba.android.arouter.launcher.ARouter getInstance()
    public void inject(java.lang.Object)
    public void inject(java.lang.Object, java.lang.Class)
    public com.alibaba.android.arouter.facade.Postcard build(java.lang.String)
    public java.lang.Object navigation(java.lang.Class)
    public boolean isSupportedActivityPath(java.lang.String)
    public java.lang.Class findBizClass(java.lang.String, java.lang.String)
    public java.util.Map findAllBizClassByType(java.lang.String)
    public java.lang.Object createBizClassObject(java.lang.String, java.lang.String)
Copy the code

The above configuration comes from part of KKRouter configuration, and the configuration rules are as follows:

  • Each class begins a line with a fully qualified name
  • Methods and member variables start with four Spaces, one line for each method and member variable
  • A parameter can contain only the type, not the parameter name
  • Cannot contain generic information, can only be declared using raw type
  • Access properties can only be public and protected, not private

A compilation failure occurs when:

  • When the configuration does not match the code, for example, access properties are not equal, such as public void inject and protected void inject are not matched
  • The configuration does not exist in the code, such as deleting a method

Implementation principle of API export

Api-export library implementation

The flow realized by export library can be roughly divided into the following flow:

  1. Read the API export document and compare it with the latest API export document to ensure that the existing rules are not modified or deleted
  2. In gradle’s Transform phase, the compiled product of the current component, the AAR or JAR package, is retrieved and converted into a ClassVisitor using the ASM API
  3. If the current ClassVisitor is defined in the API export document, then the ClassVisitor is iterated over and all members and methods matching the export document are written to the Export library
  4. When a component is published, the corresponding export library is also published.

Dependency conversion for apI-export

project.configurations
                .kkImplOnly
                .getAllDependencies()
                .forEach { dependency ->
                    boolean hasExport = QueryArtifactProcess.hasExportArtifact(project, false, dependency)
                    project.dependencies {
                        if (hasExport) {
                            compileOnly dependency.group + ':' + (dependency.name + "-export" + ':') + dependency.version
                            runtimeOnly dependency.group + ':' + dependency.name + ':' + dependency.version
                        } else {
                            implementation dependency.group + ':' + dependency.name + ':' + dependency.version
                        }
                    }
                }
Copy the code

Before replacing the dependency, check whether the -export library exists on the remote end. Nexus provides an API to check whether the corresponding library exists on the remote end. If the remote API export exists, then replace it with compileOnly, runtimeOnly the actual library.