preface

This chapter moves into the first step of sharding-JDBC parsing from the execution of a SELECT statement.

private static void select(DataSource dataSource, long userId, long orderId) throws SQLException { Connection connection = dataSource.getConnection(); PreparedStatement preparedStatement = connection.prepareStatement("select * from t_order where user_id = ? and order_id = ?" ); preparedStatement.setLong(1, userId); preparedStatement.setLong(2, orderId); preparedStatement.execute(); ResultSet resultSet = preparedStatement.getResultSet(); while (resultSet.next()) { System.out.print("order_id = " + resultSet.getLong(1) + ","); System.out.print("user_id = " + resultSet.getLong(2) + ","); System.out.print("address_id = " + resultSet.getLong(3) + ","); System.out.println("status = " + resultSet.getString(4)); } resultSet.close(); preparedStatement.close(); connection.close(); }Copy the code

A, ShardingConnection

ShardingDataSource Creates a ShardingConnection using the constructor.

public class ShardingDataSource extends AbstractDataSourceAdapter { private final ShardingRuntimeContext runtimeContext;  public ShardingDataSource(final Map<String, DataSource> dataSourceMap, final ShardingRule shardingRule, final Properties props) throws SQLException { super(dataSourceMap); checkDataSourceType(dataSourceMap); runtimeContext = new ShardingRuntimeContext(dataSourceMap, shardingRule, props, getDatabaseType()); } @override public final ShardingConnection getConnection() {// TransactionTypeHolder holds ThreadLocal, which is used to set transaction type. The default LOCAL return new ShardingConnection (getDataSourceMap (), runtimeContext, TransactionTypeHolder. Get ()); }}Copy the code

ShardingConnection constructor.

@ Getter public final class ShardingConnection extends AbstractConnectionAdapter {/ / data source map private final map < String, DataSource> dataSourceMap; // sharding-jdbc runtimeContext private final ShardingRuntimeContext runtimeContext; // the default TransactionType is LOCAL private final TransactionType TransactionType; / / transaction management, the default is null private final ShardingTransactionManager ShardingTransactionManager; public ShardingConnection(final Map<String, DataSource> dataSourceMap, final ShardingRuntimeContext runtimeContext, final TransactionType transactionType) { this.dataSourceMap = dataSourceMap; this.runtimeContext = runtimeContext; this.transactionType = transactionType; shardingTransactionManager = runtimeContext.getShardingTransactionManagerEngine().getTransactionManager(transactionType); }}Copy the code

The inheritance relationship of ShardingConnection is similar to that of ShardingDataSource. Method to realize the Connection interface does not support AbstractUnsupportedOperationConnection (an exception is thrown), AbstractConnectionAdapter sharding – jdbcConnection implementation class of abstract parent class, provides some default implementation method.

Second, the ShardingPreparedStatement

Create ShardingPreparedStatement ShardingConnection, put himself and SQL to ShardingPreparedStatement construction method.

@Getter public final class ShardingConnection extends AbstractConnectionAdapter { @Override public PreparedStatement prepareStatement(final String sql) throws SQLException { return new ShardingPreparedStatement(this, sql); }}Copy the code

ShardingPreparedStatement constructor.

public final class ShardingPreparedStatement extends AbstractShardingPreparedStatementAdapter {
    @Getter
    private final ShardingConnection connection;
    private final String sql;
    @Getter
    private final ParameterMetaData parameterMetaData;
    private final BasePrepareEngine prepareEngine;
    private final PreparedStatementExecutor preparedStatementExecutor;
    private final BatchPreparedStatementExecutor batchPreparedStatementExecutor;
    private final Collection<Comparable<?>> generatedValues = new LinkedList<>();
    private ExecutionContext executionContext;
    private ResultSet currentResultSet;
    public ShardingPreparedStatement(final ShardingConnection connection, final String sql) throws SQLException {
        this(connection, sql, ResultSet.TYPE_FORWARD_ONLY, ResultSet.CONCUR_READ_ONLY, ResultSet.HOLD_CURSORS_OVER_COMMIT, false);
    }
    private ShardingPreparedStatement(final ShardingConnection connection, final String sql,
                                      final int resultSetType, final int resultSetConcurrency, final int resultSetHoldability, final boolean returnGeneratedKeys) throws SQLException {
        if (Strings.isNullOrEmpty(sql)) {
            throw new SQLException(SQLExceptionConstant.SQL_STRING_NULL_OR_EMPTY);
        }
        this.connection = connection;
        this.sql = sql;
        ShardingRuntimeContext runtimeContext = connection.getRuntimeContext();
        // ParameterMetaData
        parameterMetaData = new ShardingParameterMetaData(runtimeContext.getSqlParserEngine(), sql);
        // PreparedQueryPrepareEngine
        prepareEngine = new PreparedQueryPrepareEngine(runtimeContext.getRule().toRules(), runtimeContext.getProperties(), runtimeContext.getMetaData(), runtimeContext.getSqlParserEngine());
        // PreparedStatementExecutor
        preparedStatementExecutor = new PreparedStatementExecutor(resultSetType, resultSetConcurrency, resultSetHoldability, returnGeneratedKeys, connection);
        // BatchPreparedStatementExecutor
        batchPreparedStatementExecutor = new BatchPreparedStatementExecutor(resultSetType, resultSetConcurrency, resultSetHoldability, returnGeneratedKeys, connection);
    }
}
Copy the code

1, the ParameterMetaData

ParameterMetaData Metadata for placeholder parameters. SQLParserEngine held by ShardingRuntimeContext is passed in during construction.

Only supports a method for ShardingParameterMetaData getParameterCount, getParameterCount obtain the number of placeholders in the SQL.

@RequiredArgsConstructor public final class ShardingParameterMetaData extends AbstractUnsupportedOperationParameterMetaData { private final SQLParserEngine sqlParserEngine; private final String sql; @Override public int getParameterCount() { return sqlParserEngine.parse(sql, true).getParameterCount(); }}Copy the code

2, BasePrepareEngine

BasePrepareEngine is very important because it implements three important steps of parsing, routing, and rewriting in its only public method.

@RequiredArgsConstructor public abstract class BasePrepareEngine { private final Collection<BaseRule> rules; private final ConfigurationProperties properties; private final ShardingSphereMetaData metaData; private final DataNodeRouter router; private final SQLRewriteEntry rewriter; public BasePrepareEngine(final Collection<BaseRule> rules, final ConfigurationProperties properties, final ShardingSphereMetaData metaData, final SQLParserEngine parser) { this.rules = rules; this.properties = properties; this.metaData = metaData; router = new DataNodeRouter(metaData, properties, parser); rewriter = new SQLRewriteEntry(metaData.getSchema(), properties); }}Copy the code

The BasePrepareEngine constructor creates the DataNodeRouter, responsible for routing; Create SQLRewriteEntry, which is responsible for creating the SQLRewriteContext rewrite context.

There are two BasePrepareEngine implementation classes:

  • PreparedQueryPrepareEngine: handlingPrepareStatement.
  • SimpleQueryPrepareEngine: handlingStatement.

3, PreparedStatementExecutor

PreparedStatementExecutor inheritance AbstractStatementExecutor abstract class is responsible for executing SQL, first take a look at the constructor, just skip.

public final class PreparedStatementExecutor extends AbstractStatementExecutor { @Getter private final boolean returnGeneratedKeys; public PreparedStatementExecutor( final int resultSetType, final int resultSetConcurrency, final int resultSetHoldability, final boolean returnGeneratedKeys, final ShardingConnection shardingConnection) { super(resultSetType, resultSetConcurrency, resultSetHoldability, shardingConnection); this.returnGeneratedKeys = returnGeneratedKeys; }}Copy the code

Fill in placeholders

Abstract superclass AbstractShardingPreparedStatementAdapter ShardingPreparedStatement realized the function of filling a placeholder.

public abstract class AbstractShardingPreparedStatementAdapter extends AbstractUnsupportedOperationPreparedStatement { private final List<SetParameterMethodInvocation> setParameterMethodInvocations = new LinkedList<>(); @Getter private final List<Object> parameters = new ArrayList<>(); @Override public final void setLong(final int parameterIndex, final long x) { setParameter(parameterIndex, x); }}Copy the code

AbstractShardingPreparedStatementAdapter setXXX method is to save parameters to the parameters of the list.

private void setParameter(final int parameterIndex, final Object value) {
	if (parameters.size() == parameterIndex - 1) {
	    parameters.add(value);
	    return;
	}
	for (int i = parameters.size(); i <= parameterIndex - 1; i++) {
	    parameters.add(null);
	}
	parameters.set(parameterIndex - 1, value);
}
Copy the code

4. Parsing entry

Execute SQL (preparedStatement. Execute) is for the user, in factShardingPrepareStatementFour important actions are done at this stage:Parse, route, override, execute.

public final class ShardingPreparedStatement extends AbstractShardingPreparedStatementAdapter { private final PreparedStatementExecutor preparedStatementExecutor; @ Override public Boolean the execute () throws SQLException {try {/ clear/resource preparedStatementExecutor. The clear (); // Prepare (); / / initialize PreparedStatementExecutor initPreparedStatementExecutor (); / / execute SQL return preparedStatementExecutor. The execute (); } finally {// Resource cleanup clearBatch(); }}}Copy the code

1. Resource cleaning

PreparedStatementExecutor abstract superclass AbstractStatementExecutor realized clear method, mainly to empty all kinds of collections.

@ Getter public abstract class AbstractStatementExecutor {/ / database connections set private final Collection < Connection > connections = new LinkedList<>(); ParameterSets = new LinkedList<>(); private final List<List<Object>> parameterSets = new LinkedList<>(); Private final List<Statement> statements = new LinkedList<>(); Private Final List<ResultSet> resultSets = new CopyOnWriteArrayList<>(); // ResultSet private Final List<ResultSet> resultSets = new CopyOnWriteArrayList<>(); // StatementExecuteUnit private Final Collection<InputGroup<StatementExecuteUnit>> inputGroups = new LinkedList<>(); public void clear() throws SQLException { clearStatements(); // Close all Statement statements. Clear (); parameterSets.clear(); connections.clear(); resultSets.clear(); inputGroups.clear(); } private void clearStatements() throws SQLException { for (Statement each : getStatements()) { each.close(); }}}Copy the code

2, BasePrepareEngine# prepare

public final class ShardingPreparedStatement extends AbstractShardingPreparedStatementAdapter { private final String sql; private final BasePrepareEngine prepareEngine; private ExecutionContext executionContext; ExecutionContext = prepareEngine.prepare(SQL, getParameters()); GeneratedValues findGeneratedKey().ifPresent(generatedKey -> generatedValues.add(generatedKey.getGeneratedValues().getLast())); }}Copy the code

The prepare method of BasePrepareEngine contains three core logic: parsing, routing and rewriting.

public abstract class BasePrepareEngine { public ExecutionContext prepare(final String sql, List<Object> clonedParameters = cloneParameters(parameters); RouteContext = executeRoute(SQL, clonedParameters); RouteContext = executeRoute(SQL, clonedParameters); ExecutionContext result = new ExecutionContext(routeContext.getSqlStatementContext()); Result.getexecutionunits ().addAll(executeRewrite(SQL, clonedParameters, routeContext)); / / print SQL if (properties. Boolean > getValue (ConfigurationPropertyKey. SQL_SHOW)) {SQLLogger. LogSQL (SQL, properties.<Boolean>getValue(ConfigurationPropertyKey.SQL_SIMPLE), result.getSqlStatementContext(), result.getExecutionUnits()); } return result; }}Copy the code

ExecutionContext The result of parsing, routing, and rewritingExecutionContextRepresents the context in which SQL is executed.

@RequiredArgsConstructor
@Getter
public class ExecutionContext {
    private final SQLStatementContext sqlStatementContext;
    private final Collection<ExecutionUnit> executionUnits = new LinkedHashSet<>();
}
Copy the code

SQLStatementContext

public interface SQLStatementContext<T extends SQLStatement> {
    T getSqlStatement();
    TablesContext getTablesContext();
}
Copy the code

SQLStatementContext Obtains SQLStatement and TablesContext.

For example, SelectStatementContext includes query fields (ProjectionsContext), GroupByContext (GroupByContext), OrderByContext (PaginationContext), and tables (TablesC) Ontext) and so on.

@Getter
public final class SelectStatementContext extends CommonSQLStatementContext<SelectStatement> implements TableAvailable, WhereAvailable {
    private final TablesContext tablesContext;
    private final ProjectionsContext projectionsContext;
    private final GroupByContext groupByContext;
    private final OrderByContext orderByContext;
    private final PaginationContext paginationContext;
    private final boolean containsSubquery;
}
Copy the code

ExecutionUnit

The ExecutionContext ExecutionContext contains multiple ExecutionUnit execution units.

Each ExecutionUnit ExecutionUnit corresponds to a SQLUnitSQL unit for a dataSource such as demo_ds_1.

public final class ExecutionUnit {
    private final String dataSourceName;
    private final SQLUnit sqlUnit;
}
Copy the code

Each SQLUnitSQL unit corresponds to a completed SQL rewrite (including? And a list of parameters.

public final class SQLUnit {
    private final String sql;
    private final List<Object> parameters;
}
Copy the code

3, BasePrepareEngine# executeRoute

The executeRoute method of BasePrepareEngine registers the RouteDecorator with the DataNodeRouter instance and then calls the route method implemented by the subclass.

Public BasePrepareEngine {private Final Collection<BaseRule> rules; private final DataNodeRouter router; private RouteContext executeRoute(final String sql, Final List<Object> clonedParameters) {// Register BaseRule's RouteDecorator registerRouteDecorator(); Return Route (Router, SQL, clonedParameters); } private void registerRouteDecorator() {// Loop through all implementations of RouteDecorator registered through the SPI mechanism Class for (Class<? extends RouteDecorator> each : OrderedRegistry. GetRegisteredClasses (RouteDecorator. Class)) {/ / reflection to instantiate the RouteDecorator omit RouteDecorator RouteDecorator = createRouteDecorator(each); // Get the RouteDecorator supported BaseRule Class<? > ruleClass = (Class<? >) routeDecorator.getType(); // Filter out the BaseRule instance supported by the RouteDecorator in Collection<BaseRule> rules. // Register the mapping between the BaseRule and RouteDecorator to the DataNodeRouter rules.stream().filter(rule -> rule.getClass() == ruleClass || rule.getClass().getSuperclass() == ruleClass).collect(Collectors.toList()) .forEach(rule -> router.registerDecorator(rule, routeDecorator)); } } protected abstract RouteContext route(DataNodeRouter dataNodeRouter, String sql, List<Object> parameters); }Copy the code

PreparedQueryPrepareEngine implementation, to the method of the route is direct call DataNodeRouter route method, the third parameter passed true said enable parse SQL cache.

public final class PreparedQueryPrepareEngine extends BasePrepareEngine { @Override protected RouteContext route(final DataNodeRouter dataNodeRouter, final String sql, final List<Object> parameters) { return dataNodeRouter.route(sql, parameters, true); }}Copy the code

The DataNodeRouter parses the SQLStatement with SQLParserEngine and routes it with RouteDecorator.

Public Final Class DataNodeRouter {// Contains data source and table structure information private final ShardingSphereMetaData metaData; // Private final ConfigurationProperties properties; Private final SQLParserEngine parserEngine; // BaseRule-RouteDecorator mapping, <BaseRule, RouteDecorator> decorators = new LinkedHashMap<>(); Private SPIRoutingHook routingHook = new SPIRoutingHook(); public RouteContext route(final String sql, final List<Object> parameters, Final Boolean useCache) {// Execute all RoutingHook start methods routinghook. start(SQL); RouteContext result = executeRoute(SQL, parameters, useCache); RouteContext result = executeRoute(SQL, parameters, useCache); / / perform all RoutingHook finishSuccess method RoutingHook finishSuccess (result, metaData getSchema ()); return result; } the catch (final Exception ex) {/ / perform all RoutingHook finishFailure method RoutingHook. FinishFailure (ex); throw ex; } } private RouteContext executeRoute(final String sql, final List<Object> parameters, Final Boolean useCache) {RouteContext result = createRouteContext(SQL, parameters, useCache); // Route for (Entry<BaseRule, RouteDecorator> Entry: decorators.entrySet()) { result = entry.getValue().decorate(result, metaData, entry.getKey(), properties); } return result; }}Copy the code

Five, the parsing

Parsing is the first step in ShardingJDBC. It converts SQL statements in the form of strings into sqlstatements in the form of objects. DataNodeRouter# createRouteContext to parse SQL entrance, directly call the SQLParserEngine parse method, for PreparedQueryPrepareEngine into this method, useCache = true.

Public Final Class DataNodeRouter {// Contains data source and table structure information private final ShardingSphereMetaData metaData; Private final SQLParserEngine parserEngine; private RouteContext createRouteContext(final String sql, final List<Object> parameters, SQLStatement SQLStatement SQLStatement = parserEngine. Parse (SQL, useCache); Create SQL context try {/ / SQLStatementContext SQLStatementContext = SQLStatementContextFactory. NewInstance (metaData. The getSchema (),  sql, parameters, sqlStatement); Return new RouteContext(sqlStatementContext, parameters, new RouteResult()); } catch (final IndexOutOfBoundsException ex) { return new RouteContext(new CommonSQLStatementContext(sqlStatement), parameters, new RouteResult()); }}}Copy the code

SQLParserEngine does the actual parsing of the SQL, and this SQLParserEngine is the same SQLParserEngine that was constructed earlier when ShardingRuntimeContext was created.

public final class SQLParserEngine { public SQLStatement parse(final String sql, final boolean useCache) { ParsingHook parsingHook = new SPIParsingHook(); parsingHook.start(sql); SQLStatement result = parse0(SQL, useCache); SQLStatement result = parse0(SQL, useCache); parsingHook.finishSuccess(result); return result; } catch (final Exception ex) { parsingHook.finishFailure(ex); throw ex; }}}Copy the code

As with DataNodeRouter, THE VARIOUS SPI hook methods are executed in SQLParserEngine’s public method Parse, and finally the actual SQL parsing is performed through the private method parse0.

@requiredargsconstructor public final class SQLParserEngine {// databaseTypeName e.g. MySQL private final String databaseTypeName; SQL - SQLStatement private final SQLParseResultCache Cache = new SQLParseResultCache(); private SQLStatement parse0(final String sql, Final Boolean useCache) {// SQLStatement if (useCache) {Optional<SQLStatement> cachedSQLStatement = cache.getSQLStatement(sql); if (cachedSQLStatement.isPresent()) { return cachedSQLStatement.get(); ParseTree ParseTree = new SQLParserExecutor(databaseTypeName, SQL).execute().getrootNode (); / / build SQLStatement SQLStatement result = (SQLStatement) ParseTreeVisitorFactory. NewInstance (databaseTypeName, VisitorRule.valueOf(parseTree.getClass())).visit(parseTree); If (useCache) {cache. Put (SQL, result); } return result; }}Copy the code

In a PrepareStatement, SQLParserEngine attempts to retrieve the parse result from the SQLParseResultCache. The SQLParseResultCache use guava com.google.com mon. Cache. The cache.

public final class SQLParseResultCache {
    private final Cache<String, SQLStatement> cache = CacheBuilder.newBuilder().softValues()
    .initialCapacity(2000).maximumSize(65535).build();
}
Copy the code

The next two lines of code convert SQL to SQLStatement using antLR (Another Tool for Language Recognition) for lexical parsing and syntax parsing. This is what I did when I expanded the nested method call between two lines of code.

SQLParserExecutor SQLParserExecutor SQLParserExecutor = new SQLParserExecutor(databaseTypeName, SQL); / / 2. According to the different database types (for example MySQLStatementParser# execute) to create the parse tree ParseASTNode astNode = sqlParserExecutor. The execute (); ParseTree parseTree = astNode.getRootNode(); / / 3. Through the different database types to create org. Antlr. V4. Runtime. Tree. ParseTreeVisitor ParseTreeVisitor ParseTreeVisitor = ParseTreeVisitorFactory.newInstance(databaseTypeName, VisitorRule.valueOf(parseTree.getClass())); // 4. Run the visit method to convert the parseTree to SQLStatement SQLStatement result = (SQLStatement) parseTreeVisitor. Visit (parseTree);Copy the code

Selectstatements are structured like this.

Public final class extends DMLStatement {private ProjectionsSegment extends; Private final Collection<TableReferenceSegment> tableReferences = new LinkedList<>(); // where private WhereSegment where; // groupBy private GroupBySegment groupBy; // orderBy private OrderBySegment orderBy; // limit private LimitSegment limit; // parentStatement private SelectStatement parentStatement; // lock private LockSegment lock; }Copy the code

Back to DataNodeRouter# createRouteContext, create SQLStatementContext through SQLStatementContextFactory factory.

 SQLStatementContext sqlStatementContext = SQLStatementContextFactory.newInstance(metaData.getSchema(), sql, parameters, sqlStatement);
Copy the code

SQLStatementContextFactory first period of the if – else according to different types of SQL.

public final class SQLStatementContextFactory { public static SQLStatementContext newInstance(final SchemaMetaData schemaMetaData, final String sql, final List<Object> parameters, final SQLStatement sqlStatement) { if (sqlStatement instanceof DMLStatement) { return getDMLStatementContext(schemaMetaData, sql, parameters, (DMLStatement) sqlStatement); } if (sqlStatement instanceof DDLStatement) { return getDDLStatementContext((DDLStatement) sqlStatement); } if (sqlStatement instanceof DCLStatement) { return getDCLStatementContext((DCLStatement) sqlStatement); } if (sqlStatement instanceof DALStatement) { return getDALStatementContext((DALStatement) sqlStatement); } return new CommonSQLStatementContext(sqlStatement); }}Copy the code

For DMLStatement, there is an if-else paragraph that distinguishes which constructor to call based on the SQL type.

private static SQLStatementContext getDMLStatementContext(final SchemaMetaData schemaMetaData, final String sql, final List<Object> parameters, final DMLStatement sqlStatement) { if (sqlStatement instanceof SelectStatement) { return new SelectStatementContext(schemaMetaData, sql, parameters, (SelectStatement) sqlStatement); } if (sqlStatement instanceof UpdateStatement) { return new UpdateStatementContext((UpdateStatement) sqlStatement); } if (sqlStatement instanceof DeleteStatement) { return new DeleteStatementContext((DeleteStatement) sqlStatement); } if (sqlStatement instanceof InsertStatement) { return new InsertStatementContext(schemaMetaData, parameters, (InsertStatement) sqlStatement); } throw new UnsupportedOperationException(String.format("Unsupported SQL statement `%s`", sqlStatement.getClass().getSimpleName())); }}Copy the code

The constructor of the SelectStatementContext is to parse the properties in the SelectStatement and construct them as contexts without looking at them.

@Getter @ToString(callSuper = true) public final class SelectStatementContext extends CommonSQLStatementContext<SelectStatement> implements TableAvailable, WhereAvailable {// table private Final TablesContext TablesContext; Private Final ProjectionsContext ProjectionsContext; // groupBy private final GroupByContext groupByContext; // orderBy private final OrderByContext orderByContext; // Paging private final PaginationContext PaginationContext; Private final Boolean containsSubquery; public SelectStatementContext(final SchemaMetaData schemaMetaData, final String sql, final List<Object> parameters, final SelectStatement sqlStatement) { super(sqlStatement); tablesContext = new TablesContext(sqlStatement.getSimpleTableSegments()); groupByContext = new GroupByContextEngine().createGroupByContext(sqlStatement); orderByContext = new OrderByContextEngine().createOrderBy(sqlStatement, groupByContext); projectionsContext = new ProjectionsContextEngine(schemaMetaData).createProjectionsContext(sql, sqlStatement, groupByContext, orderByContext); paginationContext = new PaginationContextEngine().createPaginationContext(sqlStatement, projectionsContext, parameters);  Return false containsSubquery = containsSubquery(); }}Copy the code

conclusion

  • fordataSource.getConnectionThe Connection implementation class created by ShardingDataSource is ShardingConnection, which holds the data source Map and sharding runtime context.
  • forconnection.prepareStatement, the PrepareStatement implementation class created by ShardingConnection is ShardingPrepareStatement. The execute method performs four key operations, namelyParse, route, override, execute.
  • SQLParserEngineThe parse0 method is the core logic used for SQL parsingantlr(Another Tool for Language RecognitionSQLStatement.