Ant Financial’s SOFAStack is a well-known example of successfully transforming an enterprise private project into an open source core model. We have previously done a special analysis on the thinking and driving force behind it. However, many readers have expressed their great interest in how this happened inside Ant Financial and how it was implemented in practice. And I think that’s what other technology companies are exploring and thinking about.

At the end of last month, When Wang attended KubeCon 2019 in Shanghai, he met Yu Huai, a member of ant Financial SOFA team. He is currently in charge of developing framework and SOFAStack open-source work in Ant Financial middleware team services and Frameworks group. Therefore, after attending the meeting, YU Huai and I have conducted in-depth communication on the practical operation of SOFA open source. Now we will sort out the results of the conversation for you.

SOFA with open source

In the first half of this year, they also opened the distributed transaction framework Seata and the service registry SOFARegistry, after opening SOFAStack last year. I once asked Yang Bing, the middleware leader of Ant Financial, about the thinking behind the open source SOFA. As well as the development and iteration of SOFA, you can see the previous article “Ant Financial Technical Director Yang Bing: Why fintech companies Embrace open source” for details. Get to know.

At present, SOFA architecture has been developed to SOFA 5 stage. In February this year, lu Zhi, the former head of SOFA open source, shared with me the key development directions of SOFA 5, including Service Mesh and Serverless. As well as the implementation of distributed transaction Seata, see the article “Ant Financial SOFA open-source Lu Zhi: Not only middleware, more open source in the future”.

As a successful open source core mode project, I am very concerned about the practical operation of SOFA open source and how to conduct open source governance. As an old friend of SOFA team, our topic will directly start from the open source governance of SOFA.

Take SOFA as an example: the open source process for in-house software

Yu said that Since 2015, Ant Financial has started the strategy of export of fintech, and SOFAStack has also gone beyond Ant Financial, even across national borders, and has been used by more financial institutions and partners, such as Tianhong Fund, Xinmei Trust, Nanjing Bank, PayTM and DANA Wallet.

In the process of communication and cooperation with partners and customers, they found that SOFAStack’s philosophy and capabilities are exactly what many companies in the financial industry need. In the process of ant fintech’s export, SOFAStack has undergone some internal code reconstruction, such as the stripping of historical compatibility logic, but it has not reached the standard of direct open source.

As for open source, there has been internal discussion of open source, and it was officially decided to open source after the end of The Double 11 in 2017. After a series of preparations, SOFAStack immediately announced some of the key components of the open source framework in April 2018, after completing the SOFA project to meet the standards of open source transformation.

The SOFA team adopted an Open source strategy called “Open Core”, which means to Open source both the interface layer and the Core implementation, and to build the capability of SOFAStack layer by layer in an extensible manner, ensuring that the internal version of SOFAStack and the Open source version use the same kernel. SOFAStack did a lot of remodeling and refactoring.

In the specific consideration of open source, Yu Huai said that the open source transformation of SOFAStack basically has three principles, which are respectively high scalability, internal compatibility with historical versions, external compatibility with industry standards.

SOFARPC refactoring, for example, goes something like this:

1. SOFARPC first need to carry out a core interface and model abstraction, and then increase the extension point mechanism and event bus mechanism, all internal and external implementation are based on these core interface and model to expand, and ensure that these expansion capabilities are equal, optional;

2. Then migrate the core processing logic implementation to this set of interfaces and models to ensure the complete availability of RPC capability;

3. Then it is necessary to make internal extensions of some RPC codes that are compatible with the history logic of the internal system, and conduct full test and verification to ensure compatibility with the existing online history scheme, and release online;

4. Finally, we will investigate some open source standard solutions and implementations in the industry and make them compatible. For example, SOFARPC not only supports its own implementation of SOFARegistry, but also supports excellent registry solutions in the industry such as Zookeeper, Etcd, and Nacos.

Although the above refactoring process doesn’t sound that complicated, it still tests the technical ability of the team in the actual process, especially when abstracting core interfaces and models. In order to achieve internal and external compatibility, it requires a lot of research work to do a good job of this relatively general abstraction. Secondly, when it came to internal logic compatibility, the team also did a lot of things to make the reconstituted code go online safely due to the heavy internal historical burden.

Taking SOFARPC as an example, the internal service routing process of ants is much more complex than that of open source, especially with the unique unitization deployment and the ability of remote multi-activity of ants, sometimes multi-layer routing is needed to find the target address. In order to verify the correctness of the refactored logic, the SOFA team built a very comprehensive integration framework internally to test the compatibility and correctness of the existing logic, with the exception of unit testing in the open source code.

After SOFAStack is built based on the idea of Open Core, in fact, the workload for students to develop will not be less, but may be increased. This is because when writing code, more internal and external use needs to be considered, and higher requirements on code quality are put forward, and the development process will become more complex.

Inside a new characteristic, for example, in the past may be directly modify the code tested is published online, but now he would have to think about what ability which is universal, the ability to abstract the and take them to the open source version and then released after open source version of the test, when the internal version based on the open source edition to extend, after test again released online.

Although the development of students work more, but this can let SOFAStack core code by more developers Review, run in more systems, in more scenarios for verification, SOFAStack quality assurance has a very great help.

In addition, on the progress of open source, Yu Huai said that SOFAStack does not pursue open source all internal components, but according to the characteristics of the product and open source prepared to have a choice of open source.

For example, the sub-database sub-table component under SOFAStack will not be open source for the time being because the product features are closely integrated with OB and other internal components. SOFAStack will integrate with other outstanding open source projects in the industry to ensure the integrity and diversity of the functions of the financial distributed architecture.

So for SOFAStack, it is not only about its own open source products, but more about building a suite of financial distributed architectures quickly together with all the excellent open source products in the community.

Open Source Project Management

To open source a project, the company behind it actually has to pay quite a lot of human and financial costs, and it inevitably involves the approval process. As More and more projects of Ant Financial are open source in more and more fields, including SOFAStack, AI, blockchain, etc., Ant Financial has introduced a corresponding strict internal review mechanism, including technology, compliance, legal affairs, security and other departments to review. At the same time, it also examines the significance of open source projects to the company and whether they are valuable to the community. After the approval of the project will be officially open source to everyone.

Ant Financial is very friendly to the open source culture, and most of its internal codes are open in the GitLab warehouse on the Intranet. Business teams often submit some merge requests (pull requests) to SOFAStack to help the development of the project.

At the same time, ant Financial’s engineers also generally embrace open source, which can help generate more and better ideas for projects. At the same time, it can also absorb contributions from the community to make projects better, which is welcomed by everyone.

Community governance of SOFA

Open source projects are not the end of the road. In fact, they are only the beginning, and continuous open source governance is the road to open source. How to transform an open source project from a community-oriented project led by the company behind the open source project at the very beginning is a topic worthy of consideration and exploration.

With the current open source model and community building efforts, the SOFA team is also trying to attract outside contributors as much as possible. However, SOFAStack project is widely applied in Ant Financial and many online environments of enterprises, so it requires relatively high technical ability and code quality of developers at present. Therefore, this task still needs more work.

The SOFA community already has a number of active developers and contributors addressing some of the most important needs the community has raised. Contributors have provided a significant code base for the completion of some of these functional components, and members of the SOFA community have been actively involved in the improvement and standardization of the functionality, and some pull requests have to go through a dozen rounds before being merged into the functional branch.

Yu also said that the next step will be to introduce more user-friendly processes and tools to make it easier for more developers to join the Community and contribute to the development of SOFAStack, including contributing code, documentation and promotion.

In terms of community teams, SOFAStack also designs Contributor certification mechanisms at multiple levels, such as contributors, committers, and PMCS, so that people with different levels of enthusiasm and energy can join in the project contribution.

SOFAStack open source community so far has more than 120 Contributor community, also more than 10 external committers through their contributions to the code recognized by the community, and further achieve influence on the community development. Talk about this, Yu Huai said that in order to ensure the quality of code, code contributions from the community often require SOFAStack community members and contributors to modify and improve to many times of reciprocating the trunk into the code, but through these interactions, can truly selection of the community elite, will also further projects into the community.

In terms of community construction, in addition to some exchanges on Github and Wechat group, SOFAStack community will also have a wealth of online and offline activities. SOFAWeekly, a weekly newspaper, helps you keep track of the latest community information and project progress; Community built SOFALab source code analysis laboratory, and community students to learn and analyze the source code, organized into a book, to help everyone better learning projects; SOFAChannel live broadcast several times a month, you can communicate with lecturers online. At the same time, SOFAStack will also regularly hold offline SOFAMeetup activities across the country, we face to face communication, has been held in five cities in the country 7, a total of more than 1,000 people to the scene; There will also be workshops and CodeLab and other practical activities. For example, KubeCon has a Workshop this time, leading people to practice SOFAStack hand in hand.

In terms of community cooperation, there are many excellent developers and open source projects in the community. SOFA team often learns, shares and exchanges technology with the community. SOFAStack has established a good relationship with many open source communities, including Domestic ServiceMesher, K8S China, ShardingSephere, SkyWalking, Ant Design, EggJS, etc., also including foreign light4j, etc.

In addition, Yu Huai also said that SOFAStack will do more integration and enhancement with the open source community this year. For example, he said that SOFAStack will do deeper integration and enhancement with Spring Cloud this year. For example, Spring Cloud is also a file level configuration update, and SOFAStack can be used for more granular configuration updates.

conclusion

If THERE’s one domestic open source project I’ve researched the most, it’s SOFAStack. Since SOFAStack open source, I have had in-depth communication with Yang Bing and Lu Zhi from the ideological and strategic aspects of SOFA open source. This time, I hope to share their experience in SOFA open source to the broad open source community and open source enterprises from the tactical and practical aspects.