The original address: lessworsemorebetter.com/2018/04/17/…
Original author: github.com/natebosch
Published: April 17, 2018
Where we are: Pub Build (Barback)
When Barback was written, it was intended to meet a small need in Web development: like sASS compilation, or compressing Sprites into a single image. This is a very flexible system, and it has proven very useful for more intensive build steps, such as the Angular compiler. But flexibility comes at a price. If any file can be rewritten (or deleted) at any step of the build, care needs to be taken not to let other build steps read it until it is “stable.” While Barback adds protection, it also adds an important limitation — Transformer running in package Foo cannot read any files in package Bar until all transformers on the Bar are complete. This is problematic in the context of the Dart source file because of features such as type reasoning. To understand the “meaning” of a Dart source file, you also need to understand the details from the imported library. If two Transformers running in different packages want to understand the parsing meaning of a Dart library, they must read cross-package import libraries — which can cause a Barback deadlock when they loop across package boundaries. Neither Transformer is allowed to read the imported Dart library until the other is complete, and neither Transformer will complete until it can read the file.
This fundamental limitation means that Angular compilers prior to version 4 had to cheat the system. Barback doesn’t let it read the source Dart files — but the Angular converter also runs on those source files and will (hopefully) read the information before the point where it is needed. Angular starts storing what it needs in (valid) global variables that are shared across packages. This introduces a race condition — it depends on all the converters in the packet starting together and not reaching a point until the global data is populated.
We hit a roadblock when we rewrote the Angular compiler to use parser and executor deep type resolution. To parse the type, we need to provide escaped Dart imports to the parser, but we can’t do that if we have a package loop. We were forced to disallow packet loops using Angular converters and found that performance suffered significantly. We can only work until we need to read assets from dependencies — which in the case of profilers means immediately. We serialized angular Transformer work in different packages because we couldn’t start doing anything in a package until Barback told us to read in our Transitive imports.
Pub’s intended use case also does not include compilation steps that take a long time. There are limited caches and incremental compilations that cross run. The fact that there is no consistent view of what individual files look like makes these difficult or impossible to add.
How did we get here: withpackage:buildClose the gap
While teams within Google were building larger and larger projects, we also ran into other difficulties, integrating Pub’s “write whatever whenever” approach with Bazel’s more rigorous static analysis build diagrams. We can’t make the PUB model incremental or modular – it has to be holistic. Bazel does not allow you to rewrite files. The source file cannot be changed, and anything that produces output needs to happen in a build step. The package: Build we wrote has a stricter model that allows us to have a set of bazel-like constraints at run time, while also easily shimming to pub interfaces when we want to run in that build environment. Over time, we have added to bazel’s static analysis by adopting definitions in addition to bazel’s limitations. Instead of executing the Dart code to determine which files will be written, we switch to configuration metadata, which says what output extensions can be output from a given input extension. The concept of Builder was more restrictive for the author, but gave us more flexibility to integrate it with the build system. We can run a single Builder implementation in three ways — as a Transformer in pub, as a build rule in Bazel builds, or by writing directly to the source tree using build_Runner for a small, local, dedicated Builder.
Our long-term goal with the Build pack is to enable external users to use Bazel to build the full functionality of the system just as our internal users do. We built a prototype for Dazel — a tool that captures builder configuration metadata and generates Starlark BUILD rules and BUILD files needed for the Dart project. We ran into some obstacles with this approach.
- Internally, we only need to build Dart on Unix hosts, but some Dart users develop on Windows. It took a lot of effort to migrate our Bazel build rules and tools across platforms.
- Bazel builds the system, despite trying to hide its complexity, for simpler
pub buildWorking together on small projects still feels heavyweight.
We are in a difficult situation. Our development compiler was tricky to use without being tightly integrated into the build system. External Angular builds are too slow for most users, and the cost of generating Angular code is paid every time pub Serve is launched. We can use Bazel to give a good experience, but only for a small number of users.
Where we are now:build_runnerAs a complete build system
While it was originally written to satisfy a small set of use cases for the builder — generating code in a single package and intending to ship with the package — we do have a proof-of-concept build system that is pure Dart and follows a more rigorous (read “easier to optimize”) model. We have put a lot of effort into making Build_Runner a fully capable build system for the Dart project. We made improvements in two main areas.
- We’ve extended
build_runnerEnables it to run builds in all packages and in files that are not intended for distribution. - We improved
build_runnerAvailability to make it more likepub build, automatically discover Builders instead of expressing all needs in a manual build script
O.
In Dart 2, we transitioned the build of the Web project completely to Build_Runner (we called the CLI webdev), and we have dropped support for build and server commands in Pub.
Our goal: better, faster builds.
More simple
We pushed more configuration and complexity onto Builder authors so that end users didn’t have to do manual work. Most Builders can be automatically enabled based on dependencies, and the Builder configuration is expressive enough to automatically determine things like work order. Builder authors can also decide on the default options for development mode and publish mode, so enabling dart2js compilation only requires –release!
better
We have shown build_Runner to be effective in many projects using a small set of Builders. We will work to expand the scope of the happiness path and reduce the sharp edges that surround it. In completing the migration from PUB to Build_Runner, we will explore patterns that work within the constraints of the Package: Build paradigm.
We will also work to improve integration with other tools, such as analysis servers. Unlike pub builds, generated assets are saved on disk rather than in memory. These files are visible to other tools, making them easier to examine, debug, and understand.
faster
We wanted to move quickly to a working end-to-end system — and when we could reuse code, we did. Some of the apis we used were written with PUB compatibility in mind. Even if one Builder won’t overwrite a file, we can’t be sure that other Transformers won’t, so we need to be careful. Now that we have a lot of room for improvement, we can focus on building systems that have a model for static analysis.
stronger
Despite its limitations, we are focusing on ensuring that the new build system has the correct versatility of Dart. In fact, it was able to run our network compilers without any hard-coded knowledge — they appeared on the system just like any other generator — which was not possible in Barback. The advantage of this is that we can change the implementation, such as the compiler tuned for Node.js, without changing the build system itself.
Translation via www.DeepL.com/Translator (free version)