On 10 May 2019, TiDB released 3.0.0-RC.1, corresponding to tiDB-Ansible version 3.0.0-RC.1. Compared to version 3.0.0-beta.1, there are many improvements to system stability, ease of use, functionality, optimizer, statistics, and execution engine.

TiDB

  • The SQL optimizer

    • The order correlation between columns is used to improve the accuracy of cost estimation and provide heuristic parameterstidb_opt_correlation_exp_factorUsed to control the degree of preference for index scanning in scenarios where correlation cannot be directly used to estimate.
    • When related columns are included in the filtering criteria, as many prefix columns of the index are matched as possible when extracting the access criteria of the composite index.
    • Use dynamic programming to determine the order of join execution when the number of tables participating in the join is not greater thantidb_opt_join_reorder_thresholdWhen enabled.
    • In the inner table that constructs an Index Join, match as many prefix columns of the Index as possible when the composite Index is used as the access condition.
    • Improves the accuracy of row count estimation with a value of NULL on a single-column index.
    • Special handling when eliminating aggregate functions during the logical optimization phaseGROUP_CONCATTo prevent incorrect execution results.
    • When the filter condition is constant, it is properly pushed down to the children of the join operator.
    • Special processing of some functions during column clipping during the logical optimization phase, for exampleRAND()To prevent execution results incompatible with MySQL.
    • supportFAST ANALYZEThrough thetidb_enable_fast_analyzeVariable control. This feature speeds up statistics collection by sampling a Region instead of scanning the entire Region.
    • supportSQL PLAN MANAGEMENT. This feature ensures execution stability through execution plan binding of SQL. This feature is currently in testing and only supports the use of bound execution plans for SELECT statements. It is not recommended to use it in production scenarios.

  • Execution engine

    • Support forTableReader,IndexReader 和 IndexLookupReaderOperator for memory tracking control.
    • More details about COPROCESSOR tasks are displayed in the slow log. For example, number of COPROCESSOR tasks, average/maximum /90% execution/waiting time, TiKV address with the longest execution/waiting time, etc.
    • DDL statements with PREPARE without placeholders are supported.

  • Server

    • When TiDB starts, only the DDL owner is allowed to perform bootstrap
    • newtidb_skip_isolation_level_checkVariable control checks no errors when the isolation level is set to SERIALIZABLE
    • In slow logging, the implicit commit time is merged with the SQL execution time
    • RBAC permission management
      • supportSHOW GRANT
      • supportSET DEFAULT ROLE
      • supportGRANT ROLE
    • Fixed an issue that caused TiDB to exit when plugins quit
    • Fixed an issue where read-only statements were incorrectly placed in the transaction history
    • The kill statement can end SQL execution faster and release resources faster
    • Add boot optionsconfig-checkTo check the validity of the configuration file
    • Fixed invalid check for writing NULL fields in non-strict mode

  • DDL

    • Pre_split_regions option is added to CREATE TABLE. This option can pre-allocate TABLE regions during TABLE construction to avoid write hotspots caused by massive data writes after TABLE construction
    • Optimized the performance of some DDL statements
    • FULLTEXT KEY added warning that does not support full-text index
    • Fixed UTF8 and UTF8MB4 encoding compatibility issues in older versions of TiDB
    • Fixed a potential BUG with shard_ROW_ID_bits for a table
    • Fixed BUG where Column Charset would not follow changes after ALTER TABLE Charset
    • Fixed a BUG where the SHOW Column could be wrong when using BINARY/BIT as the Column Default Value
    • Fixed compatibility issues with CHARSET/COLLATION display in SHOW FULL COLUMNS statements
    • Now the SHOW COLLATIONS statement only lists COLLATIONS that TiDB actually supports

PD

  • Upgrade the ETCD version
    • The unified ETCD log format is the same as that of pd Server
    • Fixed an issue where PreVote could not elect a Leader
    • Rapidly drop failed withdraw and read requests to reduce the request time after blocking
    • Fix Lease deadlock problem
  • Fixed incorrect store read hotspot keys statistics
  • Forcibly reconstructing a PD cluster from a single PD node
  • Fixed invalid Operator Step in Scatter Region
  • Fix the problem that the Region Merge Operator timeout duration is too short
  • Hotspot scheduling uses high priority
  • Add Metrics for how long PD Server takes to process TSO requests
  • Add the corresponding Store ID and Address to Store Metrics
  • The GetOperator service is supported
  • Fixed Heartbeat Stream error not finding store

TiKV

  • Engine
    • Fix inaccurate read traffic statistics
    • Fixed issue with Prefix Extractor Panic
    • Optimized memory management, reducedIterator Key Bound OptionMemory allocation and copy
    • Fix the panic problem caused by Learner log gap when Merge Region is not considered
    • Support differentcolumn familiesSharedblock cache
  • Server
    • To reducebatch commandsContext switch overhead
    • Check the validity of seek Iterator status
  • RaftStore
    • Can be configured to changeproperties index distance
  • Coprocessor
    • Added Batch Index Scan executor
    • Added vectorization evaluation framework
    • Added batch Executor statistics framework
    • Check Max Column when constructing RPN expression to prevent column offset from crossing the threshold in the Evaluation phase
    • implementationBatchLimitExecutor
    • ReadPool usetokio-threadpoolReplace the originalfutures-cpupoolReduce the context switch
    • Added batch aggregation framework
    • newBatchSelectionExecutor
    • Implement batch Aggression FunctionAVG
    • Realize the RPN functionLogicalAnd
  • Misc
    • Tcmalloc memory allocator is supported

Tools

  • TiDB-Binlog
    • Fixed bug Mc-117634 – Synchronization error interrupts when unsigned int primary key column binlog is negative
    • Delete the compression option for the downstream PB and change the downstream pb name to file
    • Pump Added the storage.sync-log configuration item to support Pump local storage asynchronous disk flushing
    • The communication between Pump and Drainer supports traffic compression
    • Added the Drainer configuration item syncer.sql-mode, which enables DDL query parsing using different SQL-modes
    • Added the Drainer configuration item syncer.ignore-table to filter out unnecessary lists
  • lightning
    • Fill the missing column data in the dump file with the row ID or column default value
    • The Importer fixes the bug that some SST import failure still returns the import success
    • The Importer supports the speed limit from Upload SST to TiKV
    • Lightning optimizes the order of importing tables by table data size order, reduces the impact of checksum and Analyze execution on the cluster during import, and improves the success rate of checksum and Analyze execution
    • Improve the performance of Lightning Encode SQL, improve the performance by 50%, directly parse the data source file content into types.Datum of TiDB, save the redundant parsing work of KV Encoder
    • Change the Log Format to Unified Log Format
    • Added some command-line options that can be used even if a configuration file is missing
  • Sync-diff-inspector Data Synchronization comparison tool
    • Supports checkpoint, which records the check status and continues the check from the last progress after the restart
    • Add the configuration item only-use-checksum to check whether the data is consistent by calculating the checksum

TiDB-Ansible

  • TiKV monitors changes and updates to Ansible, Grafana, and Prometheus versions
    • Summary monitoring allows users to view cluster status
    • The trouble_shooting monitoring is applicable to DBA troubleshooting
    • Details monitoring is suitable for developing analysis problems
  • Fixed failure to download Kafka version Binlog
  • Modify the OS version restriction to support only CentOS 7.0 or later and Red Hat 7.0 or later
  • Version detection during rolling upgrade changed to multi-concurrency
  • Update document links in the README
  • Removed duplicate TiKV monitoring item and added Trouble shooting monitoring item
  • To optimize thetable-regions.pyA script that displays the Leader distribution by table
  • Updates the drainer profile
  • Optimized TiDB monitoring, added SQL categories to display delayed monitoring items
  • Update the Lightning configuration file and add the tidb_lightning_ctl script