Abstract: Ali Cloud table storage has been upgraded recently. The upgraded table storage supports persistent storage of data collected by DataHub. The Python SDK supports Python 3.x, and the latest TimelineLib can help users easily build ten-million-level IM and Feed streaming systems.
Click here to view the original: click.aliyun.com/m/43196/
Ali Cloud table storage has been upgraded recently. The upgraded table storage supports persistent storage of data collected by DataHub. Its Python SDK has added support for Python 3.x, and the newly released TimelineLib can help users easily build ten-million-level IM and Feed streaming systems.
Function 1: Supports persistent storage of data collected by DataHub
Ali Cloud table storage supports persistent storage of data collected by DataHub. Users can continuously collect a large amount of streaming data generated by various mobile devices, application software, website services and sensors through DataHub service, and write it into table storage for persistent storage in real time, providing low-cost, flexible and high-performance online data services based on table storage.
The new feature is suitable for users who use DataHub for data collection and need persistent storage and online services.
The realization of persistent storage function relies on DataHub DataConnector function to synchronize streaming data from DataHub service to other cloud products. Data in Topic can be synchronized to MaxCompute(ODPS), OSS, Elasticsearch, RDS, Mysql, ADS, and TableStore in real time or quasi-real time. Users only need to write data to DataHub once and configure the synchronization function in DataHub service to use this data in various cloud products. Data synchronization supports at least once semantics, which may cause data duplication at the destination in scenarios with low probability, such as network service exceptions.
precondition
Creating a TableStore DataConnector requires the following prerequisites:
· TableStore related information, including TableStore service Endpoint, TableStore instance and its corresponding Table;
· Only the Owner/Creator account of DataHub Topic has the corresponding permission to operate DataConnector, including creation and deletion, etc.;
· Primary key columns of the TableStore table must have one-to-one field correspondence under DataHub Topic (definition order may be inconsistent);
· The authorization service role AliyunDataHubDefaultRole enables DataHub to access the user’s TableStore resources (authorization links).
Operation process
1. Enter the Topic details page:
2. Select Synchronous TableStore and fill in the relevant configuration:
3. Check the archiving status on the DataConnector page:
Configuration instructions
| The name of the | Whether must | describe |
| TableStore Endpoint | yes | TableStore service address |
| TableStore instance name | yes | TableStore instance name |
| TableStore table name | yes | Sync to TableStore table |
Matters needing attention
· The primary key column of the TableStore table must have a field with the same name under the corresponding DataHub Topic. Since DataHub is case-insensitive, if there are uppercase letters in the primary key name of TableStore, there will be corresponding lowercase fields in DataHub Topic;
· DataConnector task of synchronizing TableStore table only supports TUPLE Topic;
· The maximum number of rows that can be written in batches on the TableStore server is 200. Please refer to help.aliyun.com/document_de… .
The Python SDK supports Python 3.x
The TableStore Python SDK now supports Python 3, version 3.0 of Python, which is a major upgrade over earlier versions of Python. Python 3.0 was not designed with backward compatibility in mind, so Python2 and Python3 have major incompatibilities. There are already a number of projects and systems using Python 3, and official Python support for Python 2 will be dropped within the next three years.
In order to better meet the needs of Python3 users, the new version of the TableStore SDK ADAPTS Python3. The same version supports Python2.6, Python2.7, Python3.3, Python3.4, Python3.5 and Python3.6. When the user installs the TableStore Python SDK, whether through setup.py or PIP, the installation script automatically selects different installation contents according to the current Python version, ensuring that the installed TableStore Python SDK can perfectly adapt to the Python version.
This new feature is available to web-wide users who use tables to store the Python SDK.
Function 3: TimelineLib
TableStore launches TimelineLib, which now allows users to easily build ten-million-level IM and Feed streaming systems. TimelineLib implements the conceptual model of Timeline based on table storage, converts the model into a read/write interface for storage systems and synchronous systems, and encapsulates the interaction logic for underlying storage systems.
TimelineLib allows users to easily implement a tab-based IM or Feed streaming system and take full advantage of the high concurrency and low latency features of tab-based storage. This feature is especially suitable for games and social networks that require IM and Feed streaming systems.