Zhijiang Laboratory is a major scientific and technological innovation platform for zhejiang Provincial Party Committee and provincial government to deeply implement the innovation-driven development strategy and explore the path of zhejiang’s new nationwide system. The open source project to be introduced today is an artificial intelligence open source platform — Tianshu, led by Zhijiang Laboratory.
Zhijiang Laboratory started the development of artificial intelligence open source platform in October 2018, which was supported by the National Development and Reform Commission in January 2019, and went online in August 2020. The “Zhijiang Tianshu” team of nearly 100 researchers completed the development of the first version of the platform after more than 650 days of scientific research. And launch computing framework, development platform and tool components with unique performance advantages. After more than half a year of iteration, it has now evolved to V0.3.0 and is open source on Gitee.
Project name: Dubhe
Project author: Noko Tianshu
Open Source License: Apache-2.0
Project Address:Gitee.com/zhijiangtia…
Project introduction
Tianshu ARTIFICIAL Intelligence Open Source Platform (hereinafter referred to as: Tianshu Platform) is an artificial intelligence open source platform jointly developed by Zhijiang Laboratory, Beijing First-class Science and Technology, China Information and Communication Institute and Zhejiang University. The whole platform is composed of three subsystems: one-stop AI model development platform, high-performance deep learning framework (OneFlow) and model learning framework.
The One-stop AI model development platform (hereinafter referred to as “one-stop Development platform”) is oriented to the life cycle of AI model production, providing functions including data processing, model development, model training and model management to facilitate the one-stop construction of AI algorithms by users.
The project architecture
Project preview
Project characteristics
The development of friendly
The first version of Tianshu supports full-link functions from data management, model development, training management to model management, and data management can also realize intelligent annotation and data enhancement.
Training efficient
The distributed training framework of Zhijiang Tianshu platform supports three parallel modes: data parallelism, model parallelism and flow parallelism. It can automatically arrange the parallel modes according to different models, and has excellent linear acceleration ratio, which greatly improves the performance of data parallelism.
In addition, Tianshu also developed its own efficient communication protocol, native support RDMA; At the same time, it also supports a variety of deep learning compilers, and multiple chips can easily achieve software-defined data streams, which are friendly and compatible with AI chips.
Visual analysis
In the process of model training, developers can intuitively and real-time insight into the model structure, parameter trend, dimension reduction analysis and other important information. Through visual analysis, auxiliary parameter adjustment, so as to change the training path.
Model refined knowledge
The current mainstream deep learning algorithm construction process usually needs to go through data processing, model development and large-scale training calculation steps to produce models, but this time Tianshu launched a new model production mode “model refining”, through the leading model recombination refining technology, to achieve flexible customized models.
The depth model learning framework of Tianshu has the measurement function, which can automatically measure whether multiple visual models can be recombined, and recombine different model structures through layer by layer recombination, common feature extraction, multi-task adaptive branch decoding, etc., to directly produce a new model for application in a new scene.
As long as two pre-training models are input, the framework can reassemble a new multi-task model, which can handle two tasks at the same time, which not only improves the computational efficiency, reduces the energy consumption, but also strengthens the capability boundary of the model.
The development plan
Up to now, Tianshu platform has gathered alibaba Cloud, Ant Financial, Xinhua three (H3C), Hikvision and other dozens of ecological partners.
In the subsequent versions, Tianshu will further enrich the algorithm library and build a visual-oriented AutoML platform to achieve drag-and-drop model development. Furthermore, the distributed training performance will be further optimized, the training scale and parallel efficiency will be continuously improved, and finally the decentralized distributed training will be formed. At the same time, the efficient end-to-end reasoning platform in the RESEARCH and development plan, will improve the deployment link, to achieve the end-to-end reasoning function.
It is worth mentioning that, in order to support the development of enterprises and effectively expand the enterprise ecosystem, Tianshu platform supports small, medium and micro enterprises to use it for free, and further promotes the penetration and application of ARTIFICIAL intelligence technology to all walks of life.
Dubhe is now officially open source on Gitee. If you are a deep learning developer and fan, then this one-stop deep learning development platform is definitely a stop you should not miss. Now go to the project home page and give it a Star: gitee.com/zhijiangtia…