After four years, from the first line of code to today, ShardingSphere is officially Apache’s top open source project. Martin Woodward, Director of Developer relations at GitHub, commented: “We’re excited to see how well the community has grown over the past two years and now has over 120 direct contributors.” In this article, its core founders share their open source approach with us, which will be of great benefit to all developers who love open source and want to make a difference.

The author

Pan Juan, senior DBA of JD Data Department, Apache ShardingSphere PMC.

Zhang Liang, Head of data Research and development at JD Data Department, Apache ShardingSphere VP, Apache Dubbo PMC, author of popular open source project Elastice-Job.

preface

From Sharding-JDBC to Apache ShardingSphere;

From lightweight sub-database sub-table middleware to complete closed loop distributed database middleware platform;

From the first line of code in January 2016 to the current 300K+ line of code;

From few followers to GitHub 10K+ star;

From a community that nobody cares about to 100+ contributors;

From an internal application library to a list of 100+ adopted companies;

From looking for mentors to successfully becoming Apache’s top project.

…

The core startup members of the Apache ShardingSphere team will tell you the ups and downs and the timeline behind its open source journey.

Project introduction

Apache ShardingSphere is an open source ecosystem of distributed database middleware solutions. It consists of three independent products that can be deployed and used together. They all provide standardized data sharding, distributed transactions, and database governance functions, and can be applied to diverse application scenarios such as Java isomorphism, heterogeneous languages, and cloud native. The core functions are shown in 1-1.

Apache ShardingSphere consists of three sub-projects that form a complete database solution, collectively known as J.P.S. Ecosystem.

  • Shardingsphere-jdbc: Positioned as a lightweight Java framework that provides additional services in Java’s JDBC layer. It uses the client directly connected to the database, in the form of JAR package to provide services, without additional deployment and dependence, can be understood as an enhanced VERSION of THE JDBC driver, fully compatible with JDBC and various ORM frameworks.

  • Shardingsphere-proxy: positioned as a transparent database Proxy, shardingSphere-Proxy provides server versions that encapsulate database binary protocols to support heterogeneous languages. MySQL/PostgreSQL is currently available, which can manipulate data using any MySQL/PostgreSQL compliant access client, making it more DBA-friendly.

  • Shardingsphere-sidecar (TODO) : positioned as the cloud native database agent of Kubernetes, it proxies all access to the database in the form of Sidecar. The Database Mesh, also known as the data grid, is provided through a central-free, zero-intrusion solution to interact with the Database.

Highlights of Apache ShardingSphere include:

  • Complete distributed database solution: provides core capabilities such as data sharding, distributed transactions, data elastic migration, database and data governance.

  • Independent SQL parsing engine: Fully independent SQL parsing engine that supports multiple SQL dialects and can be used independently from ShardingSphere.

  • Pluggable microkernels: All SQL dialects, database protocols, and functionality can be loaded and unloaded via SPI’s pluggable mode, and the microkernels can even run in a blank environment with no functionality in the future.

The Apache foundation

The most popular open source software is Apache.

This is the best description and affirmation of the Apache Open Source Software Foundation. It aims to provide high-quality open source software to the world, and welcomes friends from all over the world to contribute to the Apache community, and in the process of growing, gaining recognition, and building the open source community. Whether or not you are involved in software development, you are more or less aware of its existence and the quality of the open source projects it provides, which shows how big and far-reaching its influence is.

In addition, participating in the Apache open source community and becoming an Apache Committer in their own right is what most software development engineers are committed to.

ShardingSphere has a plan to enter the Apache Open Source Organization Foundation since the beginning of open source. The reasons are as follows:

  • The Apache Foundation’s unique Idea of Apache Way and community building will make an open source project more standardized and vigorous.

  • The Apache Foundation’s mentor system provides excellent guidance and assistance to an open source community;

  • The Apache Foundation will provide copyright and trademark protection for its open source projects;

  • The influence of the Apache Foundation around the world is self-evident. If it can enter the Apache Foundation, it will expand the influence of the project and make it enter the open source field around the world.

  • Projects within the Apache Foundation will have a better chance of stepping on to the world stage, establishing friendly communication with other open source projects, and attracting more contributors to the community.

However, the threshold to enter the Apache Open Source Foundation is not easy, especially for domestic projects, language and regional barriers are undoubtedly worse.

How do I prepare for Apache?

Finding a mentor is the first and most important step into the Apache Foundation.

After understanding the operation of Apache Foundation, we embarked on the journey of looking for mentors. Get to know Apache members by attending various open source related sharing meetings or meetups. But things are not going well. Repeated attempts, contact in exchange for verbal recognition. We were really stressed and anxious during this period, and we were even going to put it on the back burner.

Later, we got to know Wu Sheng and Jiang Ning from Huawei. Wu Sheng is VP of Apache SkyWalking and has extensive experience in open source. He has a long history with ShardingSphere’s predecessor, Sharding-JDBC, and participated in the design of sharding-JDBC project prototype. Therefore, he eventually worked together as the PPMC of ShardingSphere to build the community. During the more than one year of participating in ShardingSphere community construction, he successively served as the Mentor of several Apache incubation projects and was elected as Apache Member this year.

Jiang Ning is also a warm-hearted and experienced veteran, one of the most senior Apache members in China. In the process of communicating with him, we finally saw some hope, and he finally became our mentor.

Later, Zhang Liang, VP of the team, went to Shanghai to attend the HDC Conference and got to know Craig L Russell, our other mentor. Craig was the secretary general of Apache at that time, and he was responsible for signing all SGA, ICLA and other legal documents. During the Incubation of the ShardingSphere, Craig was elected president of the Apache Software Foundation. He was friendly and kind and gave us practical advice on community norms and was willing to help us.

The third mentor is Feng Jia, a core member of Apache RocketMQ. In the end, Roman Shaposhnik was chosen as the Champion of the project, which marked a perfect end to the project’s search for a mentor. Still remember our joy and excitement at that time. Before the helplessness, wandering, lost in this moment have a bright future.

Every project that enters the Apache Foundation must have its own story. Especially for Chinese projects, language and geographical barriers make it more difficult. Fortunately, more and more projects from China have entered the Apache Foundation, and more and more Chinese are active in the Apache mailing list. Moreover, the establishment of ALC Beijing has lowered the threshold of participation, which is really good news for Chinese friends who want to participate in the Apache Foundation.

Enter the Apache incubator

In order to officially enter the Apache incubator, project code, community, documentation, and so on all need to go through a series of specifications and tidies. This is indeed a trivial but important matter.

  • At the code level, compliance is the first principle. We comb through licensing agreements that third parties rely on to ensure compliance with the Apache Software License Agreement (ASL);

  • On the community side, we started to change from Chinese to English.

  • In terms of documents, we need to prepare English documents and relevant proposals.

Since the initial goal of the project was to enter the Apache Foundation, at the beginning of the project, the dependency was as simple as possible, the community was relatively standardized, and the documents were constantly translated. This part of the work went smoothly without being unprepared, and the moment the project was granted the Apache domain name, everyone truly felt that all their efforts had been the most valuable return.

In addition to learning to write spec code, team members also began to learn the details of Apache spec, operation, English communication channels and so on. We are beginning to learn how to focus on the community, what consensus Decision is, and how to communicate via email asynchronously. In particular, it is important to study mailing lists, where you can find historical records of problems, compliant solutions, good examples, etc.

Exploration of Apache Way

Many people think that as long as the code is open, it’s called open source. But really, this is just the first step on the open source journey.

How to build a vibrant community, and how to understand the Apache Way, is a much more important topic. ShardingSphere did not fully understand Apache way at the initial stage of entering Apache incubator, and due to excessive emphasis on code style, the participation threshold was high and the community activity was mediocre.

At first, we didn’t know what the problem was and we were confused for a long time until we started talking to Apache members and realized the problem, so the community started a discussion about the Committer Bar, as shown in Figure 1-2. This was a turning point in the road to community building, as the concept of Community over Code began to permeate our minds and guide our actions.

Read the Apache Way concerns carefully:

Earned Authority, Community of Peers, Open Communications, Consensus Decision Making, Responsible Oversight

You will find that it has been emphasizing compliance, openness, equality and collaboration, in order to build a compliant and active project community, as many people as possible to participate in equal communication, promote project development and promote personal growth. Adhering to this concept, ShardingSphere began to adjust in multiple dimensions:

  • Code: standardize the code structure, divide module functions, provide project plugable ability, so as to allow users to participate in a part of a module, while trying not to destroy the overall code structure;

  • Mentality: open mind, prepare community tasks, encourage community friends to participate, PPMC or Committer actively provide guidance and help;

  • Specification: Comb through the documentation and code specifications, and provide detailed subscription and participation guidelines to promote users to contribute independently to the community on a large scale;

  • Communication: Encourage the community to use email and Issue to discuss as much as possible so as to open the discussion content, and put detailed discussion in wechat group. In addition, the official account will also introduce the progress of the community, Release, publish technical articles, etc.

  • Collaboration: Build connections and increase communication with the rest of the Apache community, and learn and grow from these collaborations.

During the incubation period, Apache ShardingSphere cooperated and integrated with Apache SkyWalking and Apache ServiceComb successively, which not only improved the product functions of each other, but also increased the communication between community members. In addition, you have incubated co-meetups with Apache DolphinScheduler (Incubating) and Apache IoTDB (Incubating), as shown in Figure 1-3. We’ve also had a lot of conversations and discussions with core Apach Pulsar and Apache APISIX (Incubating) members.

Over time, the community has changed qualitatively. From the community’s email discussions and GitHub data display, you can see that the ShardingSphere community has really become active and diversified. Figure 1-4 shows the community data changes of ShardingSphere in Apache Incubator for more than one year.

The interdependence and reciprocity between the community and contributors is also evident throughout the process. For contributors, they communicate and collaborate with others in the open source community. This continuous process will bring the following results:

  • Expand your circle of friends

  • Keep learning and growing

  • Improve your technical influence

  • Broaden career channels

  • Combine interests and enjoy the process

For the community, this process of helping and communicating with each other leads to:

  • Expand the functionality of the project

  • Harvest an active and diversified ecosphere

  • Increase project visibility

  • Access to sustainable community development

From this point of view, the continuous exploration of Apache Way is also the hope of such a win-win and mutual assistance situation. Please remember community over code.

Graduate from an incubator

All incubator projects hope to eventually move towards TLP (Top Level Project). Under mentor’s guidance, PPMC’s exploration, committer and contributor’s support and contribution, ShardingSphere began to prepare for the graduation of Apache incubator.

Use Apache’s Maturity assessment model, Figure 1-5, to assess the maturity of the community and project in the following areas. In fact, in the initial phase of the Apache project community, we recommend that you work on these aspects, because these are the official graduation standards and guidelines. From this direction, we can explore the unique community operation of our own projects, and it can be said that a hundred flowers bloom.

After a series of events such as Release, community building, Apache member’s guidance and Meetup, ShardingSphere finally initiated graduation discussion in the community and began to accept the guidance and evaluation of Apache members and all Apache members. Although graduation votes are finally passed by 10 +1 binding votes, 6 +1 non-binding votes and NO-1 or +/-0 votes, the process is also full of twists and turns.

Even after more than one year of community construction, the project is basically mature, but there are still a lot of work to meet the graduation standards. For example, confirm whether the trademark can be used, complete the statement of Apache brand and trademark on the official website of the project, and the website conforms to Apache way, etc. During this voting period, since the slogan fork Me on Github existed on the official website, and this problem always appeared frequently with no conclusion, other Apache members independently opened thread to discuss this problem. Check the Email List for details. Although the ShardingSphere was successfully promoted to the foreground and indirectly increased the exposure of the project, it can also be seen that Apache attaches great importance to and is strict about the independence of the third party and the prohibition of participating in commercial activities.

On April 16, 2020, Apache ShardingSphere finally passed the resolution of the Board of Directors of the Foundation to join the TLP industry!

The way of the future

Looking back on this journey, both gain and pay. The purpose of this article is to use the story to bring us a more vivid, comprehensive exposition and evidence support, and at the end of the article, we also want to say to the developer friends:

  • I hope this article can open up new ideas for you. It is like a window, presenting another career landscape for those who are interested in it, bringing new thoughts.

  • Hopefully the section on “what it means to be part of the open source community” answers your questions about the value of open source.

  • Each Apache open source project has its own unique community direction and values. It’s valuable and fun to get involved and choose communities that smell like you.

  • There is relaxation, there is a retreat; Don’t belittle yourself, don’t be supercilious; In the field of open source, more specifically in the Apache open source community, it is more about compliance with norms, openness and inclusiveness, equal exchanges and mutual learning.

  • Bow to work, then go all out; Hearing out of the window, they receive new trends, update the brain.

Graduating from Apache Incubator to TLP is not an end, but a beginning for ShardingSphere. In terms of product functions, ShardingSphere will continue to work deeply on the distributed database middleware platform, polishing the database middleware ecosystem with “distributed” as the core, so as to provide complete solutions, as shown in Figure 1-6. From the perspective of community, ShardingSphere will continue to be active in the community and encourage more friends to become committers and contributors in the community. Therefore, we welcome everyone to follow the ShardingSphere and join the community to move forward with more friends. The future road is unpredictable, but based on the present, look to the future, the original heart has not changed, even if follow the same steps, also wish a reed to sail!

Apache ShardingSphere Committer list

Mentor

  • Craig L Russell

  • Feng Jia, Alibaba

  • Ning Jiang, Huawei

PMC

  • Zhang Liang, Jingdong Data Department

  • Pan Juan, Jingdong Data Department

  • Zhao Jun, Jingdong Data Department

  • Zhang Yonglun, Jingdong Data Department

  • Chen Qingyang, Wing pay

  • Cao Hao, Hainan Xinsoft

  • Xiao-guang ma

  • Du Hongjun, lead chuangzhi letter

  • Yang Yi, Jingdong Data Department

  • Wu Sheng, tetrate. IO

  • Gao Hongtao, Tetrate.io

Committer

  • Leah, nine seal pups

  • Yan Zhiyi, DaoCloud

  • Dong Zonglei, JD Retail

  • Sun Haisheng, melon seeds

  • Wang Qi, JINGdong Retail

  • Ouyang Wen, Yi Kayi

  • Jiang Xiaofeng, Alibaba

  • Guangyuan,

  • Qin Jinwei, Jingdong Data Department

  • Yue to

  • Zhao Yanan