This is the first day of my participation in the Gwen Challenge in November. Check out the details: the last Gwen Challenge in 2021

A few days ago, I encountered a problem when implementing MyBatis batch insert. When the volume of batch insert data is relatively large, the program execution will report an error, as shown in the following figure:The reason is that MySQL can only execute a certain length of SQL statements, but when a large amount of data is inserted, a very long SQL statement is generated, so that the program will report errors during execution.

There are two ways to solve this problem: first, set the maximum length of SQL that MySQL can execute; Second, divide a large List into N small lists. Since the maximum SQL length in a program cannot be accurately defined, the best solution is the second option, which leads to this article.

Introduction to the

The process of breaking up a List into smaller lists is called sharding, or List splitting, whatever you like to call it.

In Java, sharding is commonly implemented in the following ways:

  1. Using Google’s Guava framework to achieve sharding;
  2. Sharding using Apache Commons framework;
  3. The use of domestic god – class framework Hutool to achieve fragmentation;
  4. Use JDK 8 to provide Stream sharding;
  5. Custom sharding function.

So let’s look at them separately.

1.Google Guava

Start by adding framework support to the project’s POM.xml by adding the following configuration:

<! -- Google Guava Tool class -->
<! -- https://mvnrepository.com/artifact/com.google.guava/guava -->
<dependency>
  <groupId>com.google.guava</groupId>
  <artifactId>guava</artifactId>
  <version>31.0.1 - jre</version>
</dependency>
Copy the code

With the Guava framework, you just need to use Lists. Partition to achieve sharding, as shown in the following code:

import com.google.common.collect.Lists;

import java.util.Arrays;
import java.util.List;

/** * Guava fragments */
public class PartitionByGuavaExample {
    / / the original collection
    private static final List<String> OLD_LIST = Arrays.asList(
            "Tang Seng, Monkey Kong, Ba Jie, Sha Seng, Cao Cao, Liu Bei, Sun Quan.".split(","));

    public static void main(String[] args) {
        // Set fragments
        List<List<String>> newList = Lists.partition(OLD_LIST, 3);
        // Prints a collection of fragments
        newList.forEach(i -> {
            System.out.println("Set length:"+ i.size()); }); }}Copy the code

The execution result of the above code is shown below:

2.apache commons

Start by adding framework support to the project’s POM.xml by adding the following configuration:

<! -- Apache Collection Tools class -->
<! -- https://mvnrepository.com/artifact/org.apache.commons/commons-collections4 -->
<dependency>
  <groupId>org.apache.commons</groupId>
  <artifactId>commons-collections4</artifactId>
  <version>4.4</version>
</dependency>
Copy the code

With the Commons framework, sharding can be done using only the listutils.partition method, as shown in the following code:

import org.apache.commons.collections4.ListUtils;

import java.util.Arrays;
import java.util.List;

/** * Commons. Collections4 */
public class PartitionExample {
    / / the original collection
    private static final List<String> OLD_LIST = Arrays.asList(
            "Tang Seng, Monkey Kong, Ba Jie, Sha Seng, Cao Cao, Liu Bei, Sun Quan.".split(","));

    public static void main(String[] args) {
        // Set fragments
        List<List<String>> newList = ListUtils.partition(OLD_LIST, 3);
        newList.forEach(i -> {
            System.out.println("Set length:"+ i.size()); }); }}Copy the code

The execution result of the above code is shown below:

3.Hutool

Start by adding framework support to the project’s POM.xml by adding the following configuration:

<! -- Hutool -->
<! -- https://mvnrepository.com/artifact/cn.hutool/hutool-all -->
<dependency>
  <groupId>cn.hutool</groupId>
  <artifactId>hutool-all</artifactId>
  <version>5.7.14</version>
</dependency>
Copy the code

Once you have the Hutool framework, you just need to use the listutil. partition method to implement sharding, as shown below:

import cn.hutool.core.collection.ListUtil;

import java.util.Arrays;
import java.util.List;

public class PartitionByHutoolExample {
    / / the original collection
    private static final List<String> OLD_LIST = Arrays.asList(
            "Tang Seng, Monkey Kong, Ba Jie, Sha Seng, Cao Cao, Liu Bei, Sun Quan.".split(","));

    public static void main(String[] args) {
        // Fragment processing
        List<List<String>> newList = ListUtil.partition(OLD_LIST, 3);
        newList.forEach(i -> {
            System.out.println("Set length:"+ i.size()); }); }}Copy the code

The execution result of the above code is shown below:

4.JDK

Stream sharding is implemented using a Stream in JDK 8 without the need to add any framework.

import java.util.Arrays;
import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;

/** * JDK Stream Partition */
public class PartitionByStreamExample {
    / / the original collection
    private static final List<Integer> OLD_LIST = Arrays.asList(
            1.2.3.4.5.6);

    public static void main(String[] args) {
        // Set sharding: divide data greater than 3 and data less than or equal to 3 into two groups
        Map<Boolean, List<Integer>> newMap = OLD_LIST.stream().collect(
                Collectors.partitioningBy(i -> i > 3));// Print the resultSystem.out.println(newMap); }}Copy the code

The execution result of the above code is shown below:The advantage of this approach is that it does not need to add any framework, but the disadvantage is that it can only implement simple sharding (splitting a List into two) and has clear sharding conditions. For example, in this case, the sharding condition is whether the array is greater than 3. If it is greater than 3, it is grouped into one group, otherwise it is grouped into another group.

5. Customize sharding

If you don’t want to introduce a third-party framework, and you can’t use Stream to meet your needs, consider writing your own code to implement sharding. Because this method is not commonly used, we only present the key method here.

The key implementation method of custom sharding is the subList method in JDK, as shown in the figure below:The following is an example:

import java.util.Arrays;
import java.util.List;

public class App {
    private static final List<String> _OLD_LIST = Arrays.asList(
            "Tang Seng, Monkey Kong, Ba Jie, Sha Seng, Cao Cao, Liu Bei, Sun Quan.".split(","));

    public static void main(String[] args) {
        // Set delimiter
        List<String> list = _OLD_LIST.subList(0.3);
        // Prints the elements in the collectionlist.forEach(i -> { System.out.println(i); }); }}Copy the code

The execution result of the above code is shown below:

conclusion

This article introduces five ways to implement List sharding. The most convenient way to implement List sharding is to introduce a third-party framework, such as Google’s Guava, Apache Commons or Hutool. Of course, if your project already includes any of the above, Just use it. For simple sharding, consider using the JDK Stream or subList method built into the List.

Follow the public account “Java Chinese Community” for more articles in this series on Java.