Log Service (Log for short) is a one-stop Service for Log data, developed in alibaba Group through a large number of big data scenarios. You can quickly complete log data collection, consumption, delivery, query and analysis functions without development, improving operation and maintenance efficiency, and establishing massive log processing capability in the DT era.
The core functions are as follows:
Real-time Collection and Consumption (LogHub)
Function:
Access real-time log data (such as Metric, Event, BinLog, TextLog, Click, etc.) through ECS, containers, mobile terminals, open source software, AND JS to provide real-time consumption interface, and connect with real-time computing and services. Data cleaning (ETL), Stream Compute, monitoring and alarm, machine learning and iterative computing
Delivery number bin (LogShipper)
Stable and reliable log delivery. Deliver log central data to storage services for storage and big data analysis. Supports compression, custom Partition, and column storage.
Purpose: Data warehouse + data analysis, audit, recommendation system and user portrait
Query and Real-time Analytics (Search/Analytics)
Real-time index, query and analysis data.
Query: keywords, fuzzy, context, scope statistics: SQL aggregation and other rich query means visualization: Dashboard + report function connection: Grafana, JDBC/SQL92 Purpose: DevOps/ online operation and maintenance, log real-time data analysis, security diagnosis and analysis, operation and customer service system
Product advantage
Fully managed service
Easy to use, 5 minutes to access the service for use, Agent support any network data acquisition LogHub coverage Kafka 100% function, and provide complete monitoring, alarm and other functional data, elastic expansion (can support PB/Day scale), LogSearch/Analytics provides save query, dashboard and alarm functions, and the cost is less than 20%. 30+ access mode, With the cloud products (OSS/E – graphs/MaxCompute/Table Store/MNS/CDN/very different, etc.), open source software (Storm, Spark) seamless docking ecologically rich
LogHub supports 30+ acquisition terminals, including Logstash and Fluent, which can be easily accessed from embedded devices, web pages, servers and programs. On the consumer side, the LogShipper supports rich data formats (TextFile, SequenceFile, Parquet, etc.) and customized partitions, and supports Spark Streaming, Storm, cloud monitoring and ARMS. Data can be directly Presto, Hive, Spark, Hadoop, E-mapReduce, MaxCompute, HybridDB and other storage engines LogSearch/Analytics query analysis syntax complete, compatible with SQL92, will provide: The connection between JDBC protocol and Grafana is real-time
LogHub: write to consume; Logtail (acquisition Agent) Real-time acquisition and transmission, within 1 second to the server (99.9% of the cases) LogSearch/Analytics: write can query analysis, in multiple query conditions can query 1 billion data in 1 second, in multiple aggregation conditions can analyze 100 million data in 1 second complete API/SDK
Easy to support custom management and secondary development all functions can be realized through API/SDK, provide multiple language SDK, easy to manage services and millions of device query analysis syntax simple and convenient (compatible with SQL92), friendly interface suitable for docking with ecological software (Grafana docking scheme will be provided)
Details about Ali Cloud log service:
Ali cloud log service use tutorial
(Log Service (Log for short) is a one-stop Service for Log data, developed in alibaba Group through a large number of big data scenarios. Users can quickly complete data collection, consumption, delivery, query analysis and other functions without development, helping to improve operation and maintenance, operation efficiency, and establish massive log processing capability in DT era.)
Official website of Ali Yun University (Official website of Ali Yun University, Innovative Talent Workshop under cloud Ecology)