With the arrival of big data wave, enterprises are trying to build big data analysis platform in order to conduct comprehensive, rapid and effective intelligent analysis of massive data, dig and release its value. However, many enterprises still use the analysis platform architecture based on traditional database, which leads to the failure to respond to business requirements quickly, and even some analysis application systems are almost paralyzed.
Typically, data analysis architectures based on traditional databases have the following pain points and challenges:
- In the face of the expansion of new business, it is expected that the amount of data will grow at an alarming rate in the future, and the existing data platform will not be able to support it.
- The existing schemes have complex operations, many steps, time-consuming and laborious, and high maintenance costs.
- The analysis performance is greatly affected when there is a large amount of data. The report query takes minutes and even cannot be opened and used, causing paralysis.
- Poor scalability of the platform, the performance of the original data warehouse encountered serious bottlenecks, low scalability, limited by foreign manufacturers, high cost.
- A large amount of IT resources are wasted in repetitive work, which increases the pressure and difficulty of IT staff and also puts in high operation and maintenance costs.
- Unable to provide timely response to business needs, hindering business development and innovation.
As Hadoop becomes the de facto standard of big data platform, some enterprises begin to seek for transformation and build enterprise-level distributed big data platform based on Hadoop, which can well solve the problems of massive data storage, management and offline computing and processing. However, interactive analysis on Hadoop faces the following problems:
- The query performance is slow, and the response time is not ideal in massive data analysis scenarios.
- Weak concurrency support and slow query performance limit the overall throughput of the system, and each query consumes a lot of computing resources.
- The technical threshold is high, data access depends on programming implementation, data analysts can not directly analyze data, and there are many engines, high learning cost.
- The cost of migration is high. When traditional OLAP projects are migrated to Hadoop platform, it is necessary to find suitable components and tools for fast data modeling and seamless connection with existing analysis application tools.
The big data intelligent analysis solutions provided by Kyligence can perfectly respond to the needs of enterprises.
Replace the old scheme and upgrade the architecture smoothly
This section takes traditional database and Tableau front-end architecture as examples to introduce Kyligence upgrade and replacement solutions. In the old cases, the enterprise will first extract the massive data in the business system to the traditional data warehouse, and carry out conversion, logical operation and other processing operations in the background. The front-end analysis tool Tableau analyzes and presents the processed data by establishing the data connection with the data warehouse.
Kyligence Enterprise is an enterprise-level intelligent big data analysis platform with Apache Kylin as the core, which is a sharp tool to improve the ability of big data analysis and build a bridge between business personnel and data within the platform. The new architecture and data analysis mode make data analysis extremely efficient. Enable business autonomy analysis capability.
The new big data platform architecture is Hadoop + Kyligence Enterprise + Tableau. All system data stored in traditional RDBMS are stored in Hadoop platform after centralized processing. Create themed data marts and cubes for different businesses in Kyligence Enterprise, and then publish the industry models to business analysts.
Kyligence Enterprise intelligent analysis platform transforms rigid and technical tables and fields into business indicators and analysis dimensions that business personnel can understand through data mart, enabling business analysts to easily conduct self-service analysis on relevant data mart through front-end BI tools such as Tableau. Analysis ideas can be developed by visual drag-and-drop method, relying on the high-speed SQL query capability of Kyligence Enterprise on super-large data sets, which can quickly generate analysis results and further optimize analysis ideas and business decisions.
The smooth upgrade of big data analysis platform architecture conforms to the evolution trend of big data platform architecture and meets the needs of future development.
The data modeling is intelligent and the development efficiency is amazing
As a new generation of big data intelligent analysis platform, Kyligence Enterprise utilizes the latest artificial intelligence technology to accelerate the intelligent cognition of logical computation contained in business scenarios, complete rapid modeling operations, reduce the development cost of manual operation, and automatically build a bridge between big data analysis application and underlying data platform.
Through the Kyligence ODBC Driver, Tableau connects to the Kyligence data source. Based on the unified query entry, Kyligence automatically collects Tableau query history. The query history represents the pattern and frequency of the analysis scenario, and the selected key query pattern will be syntactic checked and validated in Kyligence, and will be used for subsequent automatic model generation:
A. Filter query modes that are critical to performance:
B. Rapid automatic modeling:
After submission, the optimal data model is automatically generated, reducing redundant manual operations.
C. Automatically recommend dimensions and measures and intelligently build data cubes:
The next generation of Kyligence Enterprise intelligence engine automatically analyzes business processing logic and recommends dimensions and metrics to greatly improve development efficiency compared to previous manual modeling methods.
Kyligence Enterprise’s intelligent modeling capabilities improve the development process and reduce redundant and repetitive operations so that workers can develop efficiently, greatly freeing up productivity. From the implementation level, a person only needs 2 days to complete the migration and development of nearly 100 statements, the work efficiency has been improved more than 10 times.
At the same time, compared with other solutions, Kyligence Enterprise intelligent modeling provides a convenient and easy-to-use wizard operation interface, without manual coding to complete modeling operations, simple and rich functions, reducing the learning threshold of relevant personnel, reducing development costs, and greatly shortening the system online cycle.
Analysis application seamless docking, smooth migration
Since huge resources have been invested in the development of front-end Tableau analysis reports in the early stage, the customer hopes to retain the existing Tableau assets without making any changes and seamlessly connect to the underlying data analysis driver platform to complete smooth migration. From the perspective of the overall project, in the implementation process, the application layer can be seamlessly migrated without changing the analysis application report, avoiding repeated development and greatly reducing the cost. In addition, the migration does not affect the existing usage habits of service personnel, and provides consistent user experience for service personnel who are familiar with the analysis operations.
- Just two steps, replace the data source, fast and smooth migration
Kyligence Enterprise quickly migrates data sources and switches data connections in just two steps, fully covering all business analytics applications, reducing duplication and protecting existing investments:
A. Establish Kyligence data connection in Tableau through Kyligence ODBC Driver (example)
B. Replace the data source used in the report with Kyligence Enterprise data source to quickly complete the seamless migration of all reports:
- Seamless integration, one-click model synchronization, native TDS files
As the technical partner of Tableau, Kyligence Enterprise, the big data intelligent analysis platform of Kyligence, has carried out a lot of certification work related to Tableau. Tableau contains its own functions and statements can be resolved and run in Kyligence Enterprise, eliminating customers’ problems of integration and compatibility.
In order to better realize the intercommunication of model layer, Kylgience supports one-click export of Cube model as Tableau Data Source definition file TDS (Tableau Data Source). Tableau directly open TDS file, Cube defined dimensions, measures, hierarchy, table association will be automatically synchronized to Tableau, convenient Tableau users seamless connection with Kyligence Enterprise big data model, reduce repetitive modeling work.
Kyligence has the advantage of ultra-high performance and stable performance
Kyligence Enterprise is a multi-dimensional analysis solution for massive data. With a number of advanced technologies such as predictive computation and intelligent optimization, it can achieve sub-second response to large-scale data summary statistics. At the same time, the query performance of cubes with different dimensions can remain stable, which fully improves the query performance and interactive query experience of the big data analysis platform.
After migrating the original platform to Kyligence scheme, we can obviously see the performance of Kyligence Enterprise and its strong performance advantages through Tableau’s own performance recording function in a customer’s multi-billion level self-service analysis scenario. Compared with the response performance of the old scheme at the level of minutes or even hours, the performance is improved by more than ten times, and the overall performance is stable:
The ultimate improvement in query performance can bring better user experience for customers and help business development and innovation.
About Kyligence
Founded by the core team of Apache Kylin, the first top-level open source project of Apache Software Foundation from China, Kyligence is a data technology company focusing on innovation in the field of big data analysis. Kyligence provides Enterprise intelligent big data analysis product Kyligence Enterprise and Cloud computing product Kyligence Cloud based on Apache Kylin. At present, Kyligence has won many enterprise customers in insurance, securities, telecommunications, manufacturing, retail, advertising and other industries at home and abroad, including Huawei, China Unicom, OPPO, SAIC, China Pacific Insurance Group, China Union Pay, Guotai Junan, Xinhe and other industry leaders. Headquartered in Shanghai, the company has offices in Beijing, Shenzhen and other places, as well as branch offices in Silicon Valley, USA. At present, the company has received multiple rounds of investment from investment institutions such as Red Dot China, Cisco, Broadband Capital, Shun Wei Capital and Stau Capital (Fidelity International’s own investment agency).
About Kyligence Enterprise
Kyligence Big Data Intelligent Analysis platform (hereinafter referred to as Kyligence Enterprise) is an Enterprise software product based on Apache Kylin, providing sub-second analysis capability on large data sets. At the same time, it provides simple and fast big data analysis services for business users, analysts and engineers, with outstanding advantages such as ultra-high performance, programming free development, multi-protocol support and non-invasive deployment. In addition, Kyligence Enterprise has made all-round improvement and innovation in the aspects of rich application scenarios, implementation efficiency, security and control, storage efficiency, performance optimization, self-service agile BI, system monitoring and other aspects that are concerned by Enterprise users. It is the leading big data intelligent analysis platform solution in the industry.
Contact us at [email protected]