We choose to gradually open source SOFA middleware framework, while contributing to the community, but also looking forward to community, partners and even customers, can participate in the construction, forming industry standards and best practices.

— Yang Bing, director of Ant Financial

The introduction

Recently, I heard that Ant Financial will open source the two core components of SOFA — the Distributed Transaction framework and the Service Registry.

Those of you who are familiar with middleware know that these two components are targeted at the most popular microservices architecture. Among them, distributed transaction framework is the key to solve the problem of data consistency. Service registries are the foundation of service governance. With these two open sources, SOFA will become a truly complete distributed solution.

As an open source person, I was so interested in this news that I contacted Yang Bing, director of Ant Financial’s middleware team, for confirmation. As luck would have it, Yang Bing spent her precious time telling me in depth about SOFA open source thinking and recent planning.

Through this conversation, I also saw how a successful commercial company embraced open source and took it as its fundamental strategy to leverage technology dividends and support its rapidly developing business needs.

The following is the text, I will share it out for readers.

Today, open source has become the mainstream, it can be said that the entire information industry has changed from the closed source model of the past to the open source model of the present. Various open source companies have been innovating different open source models, among which RedHat, Google, Facebook and other companies have made the most outstanding achievements.

In 2018, I participated in the writing of “2018 China Open Source Annual Report” hosted by “Open Source Society”, and established a mathematical analysis model to analyze the activity and health of open source projects of Chinese Internet companies. To my surprise and no surprise, Alibaba’s open source projects ranked first, second and fourth among the top five in terms of activity. Even in the top 50 projects, ali open source projects accounted for more than half of the share! I’m not surprised that the industry has been feeling about Alibaba’s actions and strength in open source; To my surprise, it was more powerful than I had imagined. These include alibaba Group and Ant Financial, which have contributed considerable open source projects.

Therefore, when I met Yang Bing this time, I had an in-depth conversation with him about open source, hoping to know how Ant Financial thinks and practices open source, and how it organically combines open source with the company’s commercial value.

Why open source

It’s not uncommon for a commercial company to declare itself open source, or even formally open source some of its code. However, there are not many that actually integrate open source with a company’s technological evolution and effectively boost its business. There’s more to it than just uploading it to GitHub.

According to the experience of the industry, in terms of the open source of the company’s technical products, to open source the code of the existing scene, at least 30% more technical input should be paid on the basis of the existing code that has run stably and has a clear structure, and the code should be combed, improved and generalized to achieve the initial open source code; The additional technical costs of maintaining open source code, or even keeping pace with the production code on a company’s line of business, go far beyond that. As an Internet tech veteran, I agree.

So how did Ant Financial convince its executives to agree to pay so much extra to support open source when they didn’t see the payoff coming? Where did the push for open source come from?

“First of all, open source is a win-win model. For Ant Financial, open source can expand the technical service scene, provide services to more customers such as payment and finance, and improve the efficiency of partners.” Yang Bing said, “Although Ant Financial has a lot of business scenarios and has achieved super-large-scale practical experience in many scenarios, there are still financial service scenarios that are not covered. By making the technology open source, more customers can apply it to their own scenarios — these scenarios effectively supplement ant Financial’s technical application and lay a foundation for a more perfect technical framework. Therefore, we chose to open source the SOFA middleware framework step by step to contribute to the community, but also look forward to the community, partners and even customers, to participate in the construction of industry standards and best practices.”

“Second, for financial services, there are more requirements for regulation and autonomy,” Mr Yang added. “Customers also want more control over the technology they use.” Open source is a feasible mode that enables customers and upstream and downstream industries to participate and develop together.

“So it wasn’t the technology department that convinced the decision makers to open source. It was a natural choice for the business, and it was a logical way to go.” “He concluded. The result, in fact, is with the current popular open source business model.

“Plus, as you said, it’s true that when we opened source, we made a big change. The ability to build the SOFA framework layer by layer in an extensible manner, ensuring that internal versions of SOFA and open source versions share the same kernel. In open source, the logic of a particular business is stripped away, while keeping the core of the company’s internal line of business code and open source code consistent. This way, as long as the company’s business continues to grow, open source code will be maintained and evolved. So the internal version of SOFA extends the compatibility logic of internal and historical versions on top of the open source version. The core logic of the open source version is consistent with the internal and external, and is widely used in Ant Financial’s production environment, and will evolve with the drive of Ant Financial’s own business needs.” “But it’s worth it,” Yang added. “When making improvements for the open source code, it’s also for the company’s own business. It’s win-win and sustainable.”

At the beginning of open source, many companies often have doubts about whether the opening of core technologies to the outside world will lead to the improvement of competitors’ technology, and whether it will cause greater pressure of technological competition.

“In fact, we talked about it when we first started to open source. Technology can only grow when it is used by more people and in more scenarios. The open technology brings the development of the team, because the technology is dynamic, as the open source side, is actually relatively ahead of the technology. “Open source and mastering are two different things, and mastering and using it well are two different things. Therefore, competition caused by open source actually boosts the development of the whole open source system. It is benign and beneficial.” “From the perspective of the community and industry, everyone is opening up, and the closed technology system will gradually lag behind. Only by opening up can we seek common ground while shelving differences and achieve common development.”

trivia

I asked my ant Financial friends, are there any interesting “jokes” in your open source? We can talk about it.

A few days later, my friend sent me this text:

“What is the norm for middleware teams involved in Singles’ Day? That night, the normal group is probably drinking tea and other midnight peak, after the peak, of course, is to participate in the shopping. Many of our initial ideas of some things come from the night chat on Singles’ Day. It seems that our brain cells are especially active after experiencing the tense peak of midnight. For the infrastructure team, double Eleven is the end of a big exam. After the exam results come out, we want to think about something challenging, so we will talk about what we need to do technically for the next year in an unimaginative way. On the day of Double Eleven in 2017, several students of SOFA gathered around and talked about whether SOFA could be open source. Why open source? The relationship between open source and commercialization? What should be done after open source? This is the first internal discussion of SOFA open source. After this internal discussion and about half a year of preparation, we officially announced open source in April 2018 and have been in the process of gradually opening source.”

He said, this is the “joke” they have been waiting a long time to come up with, ha ha ha, these lovely technical people ah.

Evolution of SOFA and the road to open Source

SOFA middleware framework is a finance-level distributed middleware independently researched and developed by Ant Financial, which contains various components required for the construction of the finance-level cloud native architecture. It is a complete solution for distributed architecture and also a best practice forged in the financial scene.



SOFA open source panorama covers all aspects of the micro-services field, and also actively combines with popular open source components in the industry, including Alibaba Group’s open source Nacos, Sentinel, etc., to provide users with a wider range of choices.

SOFA is a framework that has been evolving for several years, and to some extent represents the evolution of Ant Financial’s technology system, and now forms an open core, open (component) open source model. SOFA has been open source since 2018, but I wonder what the journey of SOFA was before it was open source.

Yang Bing said, “In the early days, when Ant Financial was not separated from Taobao, the company used a framework called Webx, but with the development of the business and technical speeches, this framework has not adapted to the needs of Taobao. At that time hu xi (now ant Financial’s deputy CTO) created a new framework — SOFA Middleware framework.”

“SOFA middleware within the ant has experienced 10 years of development and the evolution of the five dynasties architecture, is widely used in including payment, lending, credit, fund, insurance and other financial scene, supporting the ants spend all previous double tenth a smoothly, 12-12, big test such as New Year red envelopes, created 256000 transactions per second trading records, and continues to break the record.”

“The SOFA version iteration is closely related to the company’s architectural development. For example, when we first started with SOFA1, we did modular things. SOFA2 began to introduce the whole system of microservices. With SOFA3, we want to be able to aggregate services together into a single link under microservices, which is what we call integrated deployment.”

“As the volume of transactions continues to increase each year, we have to move from a monolithic architecture to a service-oriented architecture, and then to a monolithic architecture, and then to an elastic architecture. “Most of the functional upgrades in SOFA have been accompanied by greater challenges to the technical architecture goals of the entire company.” Yang Bing said, “I personally experienced SOFA1 to SOFA2 development process, from THE overall completion of SOFA1 landing, to the introduction of SOFA2 service architecture. In SOFA3, we replaced the Tomcat JBoss layer and became a self-developed application server, which solved the problems of class isolation, module isolation and combined deployment of our internal system.”

SOFA4: Step into open source

As mentioned earlier, open source existing code requires a lot of heavy lifting, so SOFA is not ready for open source until the fourth generation and makes open source the core goal of SOFA4.

“SOFA has been developing for so many years, so it has a very heavy historical burden, which is why we have been preparing for open source for a long time. In the whole process from SOFA3 to SOFA4, in the process of making the whole SOFA lightweight, is actually preparing for open source. There are some common capabilities (not strongly related to ant Financial’s entire architecture or business) that we want to open source. We’re not going to split our code into two things and then evolve them separately, which we don’t think is the right way to open source.” Yang Bing said. “in addition, open source for the consumption of our whole team is very big, we hope to use the same set of compatible code to implement internal and external source, we will have some good in the middle of the open source code extension point, which is itself code architecture scalability challenge for us, the degree that we currently do.”

SOFA has been open source since April 2018. Thanks to the recognition of the community and developers, all the relevant open source code of SOFA has gained more than 10,000 Star and more than 80 code contributors in more than half a year.



As for the online community, SOFA’s open source site (www.sofastack.tech/) is also constantly contributing articles and documents.

“We thought open source was the best, but during the evolution of SOFA, we found that our scenario was capable of building a system that is more complex and capable than open source software,” Yang said. I myself have learned a lot from participating in this process. This incident also reflects the process of technological evolution of Ant Financial from one side. We went from the earliest IOE architecture to a partial adoption of open source software, and then gradually to the full development of our own. In the process of self-research, embrace part of open source software, and do deep creation, do some of their own innovation. It was a very difficult road.”

Yang bing said, “I think the entire infrastructure should go open source, because from Linux to Kubernetes, it has been proven time and time again. The underlying code, including things like SOFA middleware, looks less complex than the logic of a business system, but the details, elegance of design, and robustness of architecture are actually very strong. Moreover, once it has to support so many business systems, the complexity of the whole project is also very high, and it is difficult to use a programmer in one company to handle the whole technology stack. SOFA is the same, and we are increasingly seeing some trends in this regard. It is difficult for us to do this on our own, plus we have to support both domestic and foreign countries. Now I understand such a trend, so I think it is better to give back to the community and build with the community. The crazy growth of Linux, and the commercialization ecosystem that has accompanied it, has proved to the world again and again that the combination of open source and commercialization can be perfect.”

SOFA5: Embrace the Service Mesh

From the SOA architecture of many years ago to the current technological evolution of containerization and microservices, we can see that although the technology is gradually changing, the main guiding principles are still the same. Ant Financial has been at the forefront of micro services exploration. What role will the current Service Mesh technology play in the development of SOFA?

“In the past two years, we have been actively embracing and laying out Mesh, and I personally focus on the Mesh direction in terms of evolution, because I personally think it is very critical to precipitate the infrastructure into our overall technology platform. This is our main goal for SOFA5.” “With the rise of container technology, traditional middleware components can be divided into finer and more flexible components, while resource utilization and operation and maintenance efficiency have been further improved,” Yang said.

In recent years, as Kubernetes established its dominant position in the container field, and the architecture of immutable infrastructure such as Serverless pattern was further mature, services can be further abstracted in the upper layer, laying a solid foundation for large-scale containerization and microservitization.

There is also some industry opinion that Service Mesh will replace middleware.

Yang Bing said, “Ant Financial has explored far and wide in middleware, and has made a lot of optimizations to get business dividends. These capabilities will still exist, but the SDK will be lighter by changing a way to integrate with applications. As the base technology platforms move to the cloud, we see ServiceMesh as the bridge between the current era and the cloud native era, and both middleware and many of the base technology components move to Serverless. Serverless will be a new programming model for developers, a subversive operation model for operations personnel, and even a kind of Opsless. I believe that Serverless mode in a broad sense will become the leading role in the cloud native era.”

The emerging area of Service Mesh is a big hotspot. Ant gold launched http://www.servicemesher.com/ suit and community together such a Service for Chinese Mesh developers in the field of community, there are a lot of volunteers, the ant gold suit is one of them.

In the field of Service Mesh of Ant Financial, two open source projects SOFAMesh and SOFAMosn are involved. The two projects are positioned as follows:

  • SOFAMesh project: it is the control plane of Ant Financial Service Mesh, which is branched from Istio project. In the Istio system and framework, the function is supplemented, expanded, enhanced and improved, based on the exploration and solution of Istio production ground, especially the practical problems encountered in large-scale ground, after full verification, contribute back to the upstream of Istio.

  • SOFAMosn project: Ant Financial’s new infrastructure and middleware underlying network universal solution, which can be in a variety of product forms. Assume the role of data plane in Ant Financial Service Mesh, cooperate with SOFAMesh project, compatible with Istio system. In addition, SOFAMosn is also used in Ingress/API Gateway/Serverless Function Gateway scenarios.

At present, ant Financial development team has established contact with Istio development team to cooperate and explore the large-scale implementation of Istio. Relevant practices and innovations will be contributed after production verification/code stabilization.

Service Mesh is a very new technology, and Istio is a very new project. There are many details to understand and problems to be encountered in production. And some requirements are common, such as RPC protocol support, old non-microservice applications support, existing applications upgrade Istio smooth migration and so on.

“We open source SOFAMesh and SOFAMosn. On the one hand, we hope to share our knowledge and experience in Service Mesh and Istio. On the other hand, we hope to make these two projects more perfect through open source construction.” Yang Bing said, “Although we are not many open source versions, but in the future we will make up the whole field of financial distributed cloud native architecture. On the one hand, we will open up the internal products in the same field more completely. On the other hand, we will achieve compatibility and mutual integration with the community in each field.”

NEXT: Open source distributed transaction and service registry

Ant began to use the self-developed distributed transaction framework in its core business in 2007 to solve the problem of sertization and data consistency after database split. After more than 10 years of evolution, SOFA distributed transaction framework has been widely used and practiced by ant internal business. It includes ant Financial’s most core transaction, payment and accounting systems, etc. During this period, SOFA distributed transaction framework also evolved to use mode for different business scenarios, giving consideration to r&d efficiency and performance. And Alibaba a few months ago also open source its distributed transaction framework Fescar, Ant Financial and Alibaba will be in the field of distributed transaction, SOFA distributed transaction framework will be merged with Fescar, for the industry to provide more excellent, more comprehensive open source distributed transaction solution.

Service registry is an important part of the micro-service field. SOFA service registry originated from the initial version of Taobao ConfigServer. Driven by the business development of Alipay/Ant Financial, it has evolved to the fifth generation in the past ten years, with unlimited level expansion, efficient push and high availability based on AP architecture. At the same time combined with SOFARPC with remote live unit architecture support ability, SOFA service registry open source will greatly enrich the whole SOFA open source ecosystem, but also for the micro service field to bring a new service discovery solution.

conclusion

Throughout the history of ant Financial’s SOFA middleware open source, we can see how a large commercial company closely integrates the open source core mode with its business model. Its path from adopting proprietary software, applying part of open source software, evolving its own self-developed software, to open source core software, and fully embracing the open source world is worth learning and thinking of many technical enterprises.

Meet with SOFA

On March 24th (next Sunday afternoon), SOFA Meetup#1- service registry, distributed transaction will be launched in Beijing. Click on the link to sign up: tech.antfin.com/activities/…

This Meetup is the first offline event of SOFAStack and also an offline celebration of the one-year anniversary of SOFA open source. In addition to the SOFA team meeting and exchange, we will also arrange the anniversary celebration. I am looking forward to meeting my friends.

Financial Class Distributed Architecture (Antfin_SOFA)