The authors introduce

Pan Juan, senior DBA of JD Data Department, Apache ShardingSphere PMC

Zhang Liang, Head of data Research and development of Jd Data Department, Apache ShardingSphere VP, Apache Dubbo PMC, author of popular open source project Elastice-Job

Before the order

From Sharding-JDBC to Apache ShardingSphere; From lightweight sub-database sub-table middleware to complete closed loop distributed database middleware platform; From the first line of code in January 2016 to the current 300K+ line of code; From few followers to GitHub 10K+ star; From a community that nobody cares about to 100+ contributors; From an internal application library to a list of 100+ adopted companies; From looking for mentors to successfully becoming Apache's top project. ...Copy the code

The core startup members of the Apache ShardingSphere team will tell you the ups and downs and the timeline behind its open source journey.

Project introduction

Apache ShardingSphere is an open source ecosystem of distributed database middleware solutions. It consists of three independent products that can be deployed and used together. They all provide standardized data sharding, distributed transactions, and database governance functions, and can be applied to diverse application scenarios such as Java isomorphism, heterogeneous languages, and cloud native. The core functions are shown in 1-1.

Apache ShardingSphere consists of three sub-projects that form a complete database solution, collectively known as J.P.S. Ecosystem.

Shardingsphere-jdbc: Positioned as a lightweight Java framework that provides additional services in Java’s JDBC layer. It uses the client directly connected to the database, in the form of JAR package to provide services, without additional deployment and dependence, can be understood as an enhanced VERSION of THE JDBC driver, fully compatible with JDBC and various ORM frameworks.

Shardingsphere-proxy: positioned as a transparent database Proxy, shardingSphere-Proxy provides server versions that encapsulate database binary protocols to support heterogeneous languages. MySQL/PostgreSQL is currently available, which can manipulate data using any MySQL/PostgreSQL compliant access client, making it more DBA-friendly.

Shardingsphere-sidecar (TODO) : positioned as the cloud native database agent of Kubernetes, it proxies all access to the database in the form of Sidecar. The Database Mesh, also known as the data grid, is provided through a central-free, zero-intrusion solution to interact with the Database.

Highlights of Apache ShardingSphere include:

  1. Complete distributed database solution: provides core capabilities such as data sharding, distributed transactions, data elastic migration, database and data governance.

  2. Independent SQL parsing engine: Fully independent SQL parsing engine that supports multiple SQL dialects and can be used independently from ShardingSphere.

  3. Pluggable microkernels: All SQL dialects, database protocols, and functionality can be loaded and unloaded via SPI’s pluggable mode, and the microkernels can even run in a blank environment with no functionality in the future.

Prepare for Apache

Finding a mentor is the first and most important step into the Apache Foundation. After understanding the operation of Apache Foundation, we embarked on the journey of looking for mentors. Get to know Apache members by attending various open source related sharing meetings or meetups. But things are not going well. Repeated attempts, repeated contacts only resulted in verbal recognition. We were really stressed and anxious during this period, and we were even going to put it on the back burner.

Later, we got to know Wu Sheng and Jiang Ning from Huawei. Wu Sheng is VP of Apache SkyWalking and has extensive experience in open source. He has a long history with ShardingSphere’s predecessor, Sharding-JDBC, and participated in the design of sharding-JDBC project prototype. Therefore, he eventually worked together as the PPMC of ShardingSphere to build the community. During the more than one year of participating in ShardingSphere community construction, he successively served as the Mentor of several Apache incubation projects and was elected as Apache Member this year. Jiang Ning is also a warm-hearted and experienced veteran, one of the most senior Apache members in China. In the process of communicating with him, we finally saw some hope, and he finally became our mentor. Later, Zhang Liang, VP of the team, went to Shanghai to attend the HDC Conference and got to know Craig L Russell, our other mentor. Craig was the secretary general of Apache at that time, and he was responsible for signing all SGA, ICLA and other legal documents. During the Incubation of the ShardingSphere, Craig was elected president of the Apache Software Foundation. He was friendly and kind, gave us practical advice on community norms and was willing to help us; The third mentor is Feng Jia, a core member of Apache RocketMQ. In the end, Roman Shaposhnik was chosen as the Champion of the project, which marked a perfect end to the project’s search for a mentor.

Still remember our joy and excitement at that time. Before the helplessness, wandering, lost in this moment have a bright future. Every project that enters the Apache Foundation must have its own story. Especially for projects in China, language and geographical barriers make things worse. Fortunately, more and more projects from China have entered the Apache Foundation, and more and more Chinese are active in the Apache mailing list. Moreover, the establishment of ALC Beijing has lowered the threshold of participation, which is really good news for Chinese friends who want to participate in the Apache Foundation.

Enter the Apache incubator

In order to officially enter the Apache incubator, project code, community, documentation, and so on all need to go through a series of specifications and tidies. This is indeed a trivial but important matter.

At the code level, compliance is the first principle. We comb through licensing agreements that third parties rely on to ensure compliance with the Apache Software License Agreement (ASL); On the community side, we started to change from Chinese to English. In terms of documents, we need to prepare English documents and relevant proposals. Since the initial goal of the project was to enter the Apache Foundation, at the beginning of the project, the dependency was as simple as possible, the community was relatively standardized, and the documents were constantly translated. This part of the work went smoothly without being unprepared, and the moment the project was granted the Apache domain name, everyone truly felt that all their efforts had been the most valuable return.

In addition to learning to write spec code, team members also began to learn the details of Apache spec, operation, English communication channels and so on. We are beginning to learn how to focus on the community, what consensus Decision is, and how to communicate via email asynchronously. In particular, it is important to study mailing lists, where you can find historical records of problems, compliant solutions, good examples, etc.

Exploration of Apache Way

Many people think that as long as the code is open, it’s called open source. But really, this is just the first step on the open source journey. How to build a vibrant community, and how to understand the Apache Way, is a much more important topic. ShardingSphere did not fully understand Apache way at the initial stage of entering Apache incubator, and due to excessive emphasis on code style, the participation threshold was high and the community activity was mediocre. At first, we didn’t know what the problem was and we were confused for a long time until we started talking to Apache members and realized the problem, so the community started a discussion about the Committer Bar, as shown in Figure 1-2. This was a turning point in the road to community building, as the concept of Community over Code began to permeate our minds and guide our actions.

Read the Apache Way concerns carefully: Earned Authority, Community of Peers, Open Communications, Consensus Decision Making, Responsible Oversight. You will find that it has been emphasizing compliance, openness, equality and collaboration, in order to build a compliant and active project community, as many people as possible to participate in equal communication, promote project development and promote personal growth.

With this concept, ShardingSphere began to adjust in multiple dimensions,

  • Code: standardize the code structure, divide module functions, provide project plugable ability, so as to allow users to partially participate in a module, while trying not to destroy the overall code structure.

  • Mentality: Be open to developing community tasks, encourage community friends to participate, and provide guidance and help from relevant PPMC or Committers.

  • Specification: Comb through documentation and code specifications, and provide detailed subscription and participation guidelines to facilitate voluntary community contributions on a large scale.

  • Communication: Encourage the community to use email and Issue to discuss as much as possible so as to open the discussion content, and put detailed discussion in wechat group. In addition, the official account will introduce the progress of the community, Release, publish technical articles, etc.

  • Collaboration: Build connections and increase communication with the rest of the Apache community, and learn and grow from these collaborations.

During the incubation period, Apache ShardingSphere cooperated and integrated with Apache SkyWalking and Apache ServiceComb successively, which not only improved the product functions of each other, but also increased the communication between community members. In addition, you have incubated co-meetups with Apache DolphinScheduler (Incubating) and Apache IoTDB (Incubating), as shown in Figure 1-3. We’ve also had a lot of conversations and discussions with core Apach Pulsar and Apache APISIX (Incubating) members.

Over time, the community has changed qualitatively. From the community’s email discussions and GitHub data display, you can see that the ShardingSphere community has really become active and diversified. Figure 1-4 shows the community data changes of ShardingSphere in Apache Incubator for more than one year.

The interdependence and reciprocity between the community and contributors is also evident throughout the process. For contributors, they communicate and collaborate with others in the open source community. And this ongoing process will lead to the following outcomes,

  • Expand your circle of friends

  • Keep learning and growing

  • Improve your technical influence

  • Broaden career channels

  • Combine interests and enjoy the process

And for the community, this process of helping and communicating with each other,

  • Expand the functionality of the project

  • Harvest an active and diversified ecosphere

  • Increase project visibility

  • Access to sustainable community development

From this point of view, the continuous exploration of Apache Way is also the hope of such a win-win and mutual assistance situation. Please remember community over code.

Graduate from an incubator

All incubator projects hope to eventually move towards TLP (Top Level Project). Under mentor’s guidance, PPMC’s exploration, committer and contributor’s support and contribution, ShardingSphere began to prepare for the graduation of Apache incubator. Use Apache’s Maturity assessment model, Figure 1-5, to assess the maturity of the community and project in the following areas. In fact, in the initial phase of the Apache project community, we recommend that you work on these aspects, because these are the official graduation standards and guidelines. From this direction, we can explore the unique community operation of our own projects, and it can be said that a hundred flowers bloom.

After a series of events such as Release, community building, Apache member’s guidance and Meetup, ShardingSphere finally initiated graduation discussion in the community and began to accept the guidance and evaluation of Apache members and all Apache members. Although graduation votes are finally passed by 10 +1 binding votes, 6 +1 non-binding votes and NO-1 or +/-0 votes, the process is also full of twists and turns.

Even after more than one year of community construction, the project is basically mature, but there are still a lot of work to meet the graduation standards. For example, confirm whether the trademark can be used, complete the statement of Apache brand and trademark on the official website of the project, and the website conforms to Apache way, etc. During this voting period, since the slogan fork Me on Github existed on the official website, and this problem always appeared frequently with no conclusion, other Apache members independently opened thread to discuss this problem. Check the Email List for details. Although the ShardingSphere was successfully promoted to the foreground and indirectly increased the exposure of the project, it can also be seen that Apache attaches great importance to and is strict about the independence of the third party and the prohibition of participating in commercial activities. On April 16, 2020, Apache ShardingSphere finally passed the resolution of the Board of Directors of the Foundation to join the TLP industry!

The way of the future

Graduating from Apache Incubator to TLP is not an end, but a beginning for ShardingSphere. In terms of product functions, ShardingSphere will continue to work deeply on the distributed database middleware platform, polishing the database middleware ecosystem with “distributed” as the core, so as to provide complete solutions, as shown in Figure 1-6. From the perspective of community, ShardingSphere will continue to be active in the community and encourage more friends to become committers and contributors in the community. Therefore, we welcome everyone to follow the ShardingSphere and join the community to move forward with more friends.

The future road is unpredictable, but based on the present, look to the future, the original heart has not changed, even if follow the same steps, also wish a reed to sail!

Apache ShardingSphere Committer list

Mentor

Craig L Russell feng Jia, Jiang Ning alibaba, Huawei

PMC

Zhang Liang, PAN Juan, ZHAO Jun, ZHANG Yonglun, Chen Qingyang, YI Pay, Cao Hao, Hainan Xinsoft, Ma Xiaoguang, Du Hongjun, Yang Yi, WU Sheng, Tetrate. IO, Gao Hongtao, Tetrate. IO

Committer

Li Ya, Nine Little Seals Yan Zhiyi, DaoCloud Dong Zonglei, JINGdong Retail Sun Haisheng, Melon Seeds Wang Qi, JINGdong Retail Ouyang Wen, Yikai Jiang Xiaofeng, Alibaba Wang Guangyuan Qin Jinwei, Jingdong Data Yueling Zhao Yanan

Liverpoolfc.tv: shardingsphere.apache.org/