The application and development of big data is biased towards the bottom layer, which is difficult to learn and involves a wide range of technologies, which restricts the popularization of big data. Now a technology is needed to encapsulate some common and reused basic codes and algorithms in big data development into class libraries, so as to reduce the learning threshold of big data, reduce the difficulty of development and improve the development efficiency of big data projects.

There are three applications of big data in work: business related, such as user portrait and risk control; Related to decision making, data science, knowledge of statistics, algorithms, which is the domain of the data scientist; Related to engineering, how to implement, how to implement, what business problems to solve, that’s the job of data engineers.
The characteristics of data sources determine the technology selection of data collection and storage. According to the characteristics of data sources, I divide them into four categories:

The first category is divided into internal data and external data from the perspective of source.

The second category is divided into unstructured data and structured data from the perspective of structure.

The third category: from the perspective of variability, it can be divided into immutable data that can be added and data that can be modified and deleted;

The fourth category is divided into large amount of data and small amount of data from the perspective of scale

The first element of big data platform is data source. The data source we need to deal with is usually on the business system. In data analysis, the business data source may not be directly processed, but will be collected and stored before data analysis and processing.

As can be seen from the whole large ecosystem, to complete the data engineering needs a lot of resources; Large amount of data need to cluster; Controlling and coordinating these resources requires monitoring and coordinated dispatching; How to deploy more conveniently and easily in the face of large scale data; There’s also logging, security, and possibly integration with the cloud, all of which are on the fringes of big data and are equally important.

Big Fast Big Data Platform (

DKH), is a one-stop search engine-level, big data universal computing platform designed by Dakuai Company in order to break through the channel between the big data ecosystem and traditional non-big data companies. By using DKH, traditional companies can easily overcome the technological gap of big data and achieve the performance of big data platform at the level of search engine.

DKH effectively integrates all components of the entire HADOOP ecosystem, and is deeply optimized. It is newly compiled into a complete general computing platform for big data with higher performance, realizing the organic coordination of all components. Therefore, COMPARED with open source big data platform, DKH has up to 5 times (maximum) performance improvement in computing performance.

DKH simplifies the complex big data cluster configuration to three types of nodes (master node, management node and computing node) through the unique middleware technology of Dakuai, which greatly simplifies the management operation and maintenance of the cluster and enhances the high availability, high maintainability and high stability of the cluster.

DKH, although highly integrated, still retains all the advantages of the open source system and is 100% compatible with the open source system. Big data applications developed based on the open source platform can run efficiently on DKH without any changes, and the performance can be improved by up to 5 times.

DKH integrates the fast Big Data integration development framework (FreeRCH). FreeRCH development framework provides more than 20 classes commonly used in big data, search, natural language processing and artificial intelligence development. Through a total of more than 100 methods, the development efficiency has been improved by more than 10 times.

The SQL version of DKH also provides the integration of distributed MySQL, the traditional information system, which can be seamlessly implemented for big data and distributed across.

DKH standard platform technical architecture diagram