Looking at the title, if you have done CI/CD and know anything about Monorepo, plus the front and back ends are not completely separated, you must know how difficult we are.

Pain points analysis

Monorepo pain

There are many subprojects, yes, many, dozens or hundreds of them.

Almost all CI/CD tools are not friendly to Monorepo support. How long would it take to build all the subprojects in one CI/CD run? Are development and QA acceptable?

Currently build a single subproject in about 20 seconds (webpack optimized) and build five in about 1 minute and 10 seconds.

Building 100 will take at least 20 minutes.

And with so many business subprojects, it is unlikely to manage their versions alone.

This is the number one problem to be solved by automation!

The front and rear ends are not completely separated

Because the leadership of the company is. NET back end origin, so many projects are back end dominated MVC architecture.

In fact, there is no back-end drawing code on the refactored page. We used the React project to build the project, and copied the package results to it. Used in.NET projects, page references are packaged as a result, but versioning is still the responsibility of the back end.

Colleagues repeatedly complained that the process was shockingly complex: develop in React projects, commit git, push, and push MR; Then build and copy the package results to. NET project, pull branch again to commit git, push, push MR. This copying process can easily lead to. NET project csPROj file synchronization failure. NET projects can’t find resources during build and will be blown up (for chunk files, old files must be deleted, then removed from VS, and then Include new ones to the project. If you miss a csPROj item, the synchronization fails, and the automatic build of.NET projects will be blown up, as anyone who has used VS will know).

For projects with front-end pages drawn by back-end controllers to do resource separation, the biggest challenge is versioning (i.e., cache reuse and penetration)!

The front-end takes care of the build and release
Versioning is left to the back end because the front end cannot change the back-end CSHTML view layer

Even if your Webpack build is handled badly, it will be. This hind leg drags me down.

PS: This is not my fault. It is caused by the reconstruction of the legacy project, which has been a headache for me since I joined the company. The old code view is drawn with knock + Query +.NET code, and after refactoring it is drawn with React. It is a separate project, but the result needs to be copied into the legacy project to run.

This is the second problem to be solved by automation!

The introduction of CDN

What? It’s 2020 and you’re telling me there are still projects not using CDN?

Yes, the leadership of the company is. NET backend origin, does not care much about front-end resource processing.

All scripts and styles are loaded into memory at project startup and accessed through back-end code.

@Script.Src("/page/a.js",
"/page/b.js");
@Style.Src("/css/a.css",
"/css/b.css");
Copy the code

To carry out automatic deployment, front-end resources must be separated, so the introduction of CDN is also natural.

The solution

Since want to do, above these problems always want to solve. After a few days of deliberation, I finally came to the conclusion that we needed a build platform of our own

One by one:

First, take care of Monorepo’s pain

Monorepo is a problem because there are too many sub-projects to build efficiently, and you don’t actually need a full build.

I have studied the API document of GitLab and learned the usage of trigger. I know that gitLab pipeline can be triggered by this API, and the most important thing is that it can carry the parameter variables.

So here’s the solution:

define.gitlab-ci.yml, define commands for the build phase, using predefined variablesvariables[NERKO_PROJECTS]To get fromtriggerTo build (oursMonorepoSupport free composition of subprojects at design time),NERKO_PROJECTSRepresents the subproject to build and is a string, such as"ProjectA projectB, projectE".

   - |
      if [ "$BUILD_ALL"= ="true" -o "$BUILD_PROJECTS" ]; then
        npm run build ${BUILD_PROJECTS};
      fi
Copy the code

Build a build platformNERKOTo list all subprojects in the current environment, optionally build, clickdetermineAfter sendingtriggerThe request triggersgitlabthepipeline.

Finally, upload the packing resultsCDN.

However, although the first problem is solved, the second problem still exists. The separation of the front and back ends has always been the biggest difficulty.

Then, address versioning that is not separated from the front and back ends

In the previous point, the build result is available, and the upload CDN is also available. NET project where to get which version which files?

Many Internet companies use the index.html file as an index, pointing to the current version of the file, and rendering the page with script references generated in the file.

We can use that idea here, but it’s a little bit more complicated, because our actual view is. NET renders CSHTML files instead of webpack output index.html!

Solutions are as follows:

When the build is complete, theprojectA.html,projectB.html… And so on.
Scan these files, read them one by one, and match them using regexscriptwithlinkTag, get a copyjsReference list andcssList of references.
Traverse thesejswithcss, read for relative pathswebpackBuilt files (absolute paths most arecdnThird party library reference, can be used directly), calculate these filesmd5Value, and perform the rename operation, such as willmenu.jsInstead ofmenu.{md5}.js.
To generate amanifest.jsonIndex file, as follows:

{
  "projects": {
    "allCalls": {
      "js": [
        "/dll/vendor.dll.90c504c0d36899ef24fd43024f0ffad2.js"."/public.e107ea21bcd121626fb24fcc6280961e.js"."/lib/common-plugins.88cad0d3eeaa5c32057a0d0216f980c4.js"."/pages/allCalls.7ba43cc799a7ff5dea331705e713bea0.js"]."css": [
        "/css/common.5b78df382b56d09b67371ca2bd785cf2.css"."/css/business/allCalls.37c37d31ea0626cedd909a38bdacf501.css"]},"autoSchedule": {
      "js": [
        "/dll/vendor.dll.90c504c0d36899ef24fd43024f0ffad2.js"."/dll/echarts.dll.73360fa76a5ceed9cd8861ea1f1e7acc.js"."/dll/fe-toolkit.3eb3dea4259316aa67b70b5e1e966edf.js"."/lib/common-plugins.88cad0d3eeaa5c32057a0d0216f980c4.js"."/public.e107ea21bcd121626fb24fcc6280961e.js"."/pages/autoSchedule.154a78a12b54976c351ebcc1493d8454.js"]."css": [
        "/css/common.5b78df382b56d09b67371ca2bd785cf2.css"."/css/business/autoSchedule.fb12e319a513cc983279c177156e5f41.css"]}}}Copy the code

Will takemd5Identify the file uploadedCDNGo up.
Of the generatedmanifest.jsonFiles are generatedmd5And changed his name tomanifest.{md5}.jsonUpload,CDNCorresponding directory ofversionUnder the folder, for archiving; And will bemanifest.jsonFile uploadCDNCorresponding directory, used as an index.
.NETWhen a project starts, proactively look for itmanifest.jsonIndex file, yeahcshtmlRender accordinglyjswithcssList.
.NETThe project provides an authentication-free interface to retrieve the index and refresh the version.NERKOThe build platform notifies each time a build is completed through this interface.NETUpdate index files.
Version restore by willCDNonversionThe corresponding version in the directory (beltmd5Id) index file copied tomanifest.jsonThe directory is overwritten and notified.NETUpdate the index file to complete a version rollback, about 7 seconds, a true second version rollback.

The back-end docking

When the service starts (after the front-end releases the version, the back-end provides a login-free refresh interface), it needs to get the manifest file from the source server and parse it into memory.

The page calls @cdn.js (projectName) and @cdn.css (projectName) for resource output.

The benefits are immediately apparent: js and CSS reference maintenance has been shifted to index.html in the front-end project, and the back-end view has been “decoupled”

That’s the ideal scenario, but it’s not all plain sailing

CDN distribution delay caused by`.NET`Failed to synchronize index files while refreshing the cache

Due to the large synchronization delay of CDN, it is called after the completion of release. NET project refresh interface, this time access to the manifest.json file is not up to date.

The solution is to get the index files directly from the source.

As we know, CDN distribution is to get files from the specified server, so. NET simply fetched the index file from the source server.

We are using AWS S3, so we can get it directly from S3 buckets.

The index file has many duplicate public references

The sharp-eyed or optimization-minded should have noticed that the sample index file above, if expanded to 100 sub-items, would have a lot of redundant public references (especially since the introduction of MD5 causes file names to be extremely long).

The solution is to pull out the common references and redesign the manifest structure.

{
  "projects": {
    "allCalls": {
      "js": [
        "lib:vendor"."lib:common-plugins"."lib:public"."/pages/allCalls.5e2596707a3aa3f3dbde5951ce64f7c6.js"]."css": [
        "lib:common-css"."/css/business/allCalls.d1af277947613dbc39b5cb24a4229288.css"]},"autoSchedule": {
      "js": [
        "lib:vendor"."lib:echarts"."lib:fe-toolkit"."lib:common-plugins"."lib:public"."/pages/autoSchedule.js"]."css": [
        "lib:common-css"."/css/business/autoSchedule.fdf913542f1ceb74e201a6e820fe42b1.css"]}},"library": {
    "vendor": "/dll/vendor.dll.90c504c0d36899ef24fd43024f0ffad2.js"."echarts": "/dll/echarts.dll.73360fa76a5ceed9cd8861ea1f1e7acc.js"."fe-toolkit": "/dll/fe-toolkit.3eb3dea4259316aa67b70b5e1e966edf.js"."common-plugins": "/lib/common-plugins.88cad0d3eeaa5c32057a0d0216f980c4.js"."common-css": "/css/common.1846ca18b2f3b23b38494df63df9eed6.css"."public": "/public.690672434e9d4201347f2956ae5773ae.js"
  },
  "host": "http://xx.xx.xx.xx:8080/client"."isCdn": false
}
Copy the code

PS: Add a host attribute for NET can directly spell out the full access address.

conclusion

We actually do a lot more than that.

In addition to the above, we also made a build details interface, similar to gitLab’s assembly line details, log polling, and version management interface to support version switching.

mo4tech.com (Moment For Technology) is a global community with thousands techies from across the global hang out!Passionate technologists, be it gadget freaks, tech enthusiasts, coders, technopreneurs, or CIOs, you would find them all here.

Gitlab CI/CD practice based on Monorepo + incomplete separation of front and back ends

Pain points analysis

Monorepo pain

The front and rear ends are not completely separated

The introduction of CDN

The solution

First, take care of Monorepo’s pain

Then, address versioning that is not separated from the front and back ends

The back-end docking

That’s the ideal scenario, but it’s not all plain sailing

CDN distribution delay caused by`.NET`Failed to synchronize index files while refreshing the cache

The index file has many duplicate public references

conclusion

Gitlab CI/CD practice based on Monorepo + incomplete separation of front and back ends

Pain points analysis

Monorepo pain

The front and rear ends are not completely separated

The introduction of CDN

The solution

First, take care of Monorepo’s pain

Then, address versioning that is not separated from the front and back ends

The back-end docking

That’s the ideal scenario, but it’s not all plain sailing

CDN distribution delay caused by.NETFailed to synchronize index files while refreshing the cache

The index file has many duplicate public references

conclusion

Related Posts

The difference between XHR, Ajax, Axios, and FETCH

JS- Tips on input

[HTML] it

CDN distribution delay caused by`.NET`Failed to synchronize index files while refreshing the cache