Hadoop is 3.1.1

1. Start the Hadoop service

$ start-all.sh
Copy the code

2. Create the Maven project of IDEA

2.1 Select Maven, Project SDK 1.8, and click Next

2.2 After filling in GroupId and ArtifactId, click Next

2.3 click Finish

3. Change the Target Bytecode version

Open Setting, select Build, Execution, Deployment -> Compiler -> Java and change the Target Bytecode version to 1.8 or 8.

Make sure the JDK version is 1.8 for each configuration

4. Import the REQUIRED JAR package

4.1 After selecting Dependencies, click the “+” below and select “JARs or Directories”

4.2 Go to share/ Hadoop/in the Hadoop directory and import these packages

4.2 Add the following dependencies to pom. XML

    <dependencies>
        <! -- https://mvnrepository.com/artifact/junit/junit -->
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.12</version>
            <scope>test</scope>
        </dependency>

        <! --&lt; ! &ndash; https://mvnrepository.com/artifact/commons-logging/commons-logging &ndash; &gt; -->
        <dependency>
            <groupId>commons-logging</groupId>
            <artifactId>commons-logging</artifactId>
            <version>1.2</version>
        </dependency>

        <! --&lt; ! &ndash; https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common &ndash; &gt; -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>3.1.1</version>
        </dependency>

        <! -- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-core -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-core</artifactId>
            <version>1.2.1</version>
        </dependency>

        <! -- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-hdfs -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdfs</artifactId>
            <version>3.1.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>3.1.1</version>
        </dependency>
        
    </dependencies>
Copy the code

5. Write Java code for Hadoop project

5.1 Creating the Java Class Test.java

5.2 Writing Code

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;

import java.io.IOException;
import java.net.URI;
import java.net.URISyntaxException;

public class Test {

    // Create a test folder in HDFS
    public static void main(String[] args) {

        FileSystem fileSystem = null;
        try {
            fileSystem = FileSystem.get(new URI("hdfs://localhost:9000/"),new Configuration(),"binguner");
            fileSystem.mkdirs(new Path("/test"));
            fileSystem.close();
        } catch (IOException e) {
            e.printStackTrace();
        } catch (InterruptedException e) {
            e.printStackTrace();
        } catch(URISyntaxException e) { e.printStackTrace(); }}}Copy the code

5.3 Running Java Programs

6. Running result

6.1 The Test folder does not exist in the HDFS directory before running

6.2 The test folder is added to the HDFS directory after the HDFS is run

7. Common interfaces of FileSystem

  • 7.1 mkdirs
public boolean mkdirs(Path f) throws IOException {
    return this.mkdirs(f, FsPermission.getDirDefault());
}
Copy the code

The parameter is the path to a new folder that can be created by nesting folders within the folder.

  • 7.2 create
    public FSDataOutputStream create(Path f) throws IOException {
        return this.create(f, true);
    }

    public FSDataOutputStream create(Path f, boolean overwrite) throws IOException {
        return this.create(f, overwrite, this.getConf().getInt("io.file.buffer.size".4096), this.getDefaultReplication(f), this.getDefaultBlockSize(f));
    }

    public FSDataOutputStream create(Path f, Progressable progress) throws IOException {
        return this.create(f, true.this.getConf().getInt("io.file.buffer.size".4096), this.getDefaultReplication(f), this.getDefaultBlockSize(f), progress);
    }

    public FSDataOutputStream create(Path f, short replication) throws IOException {
        return this.create(f, true.this.getConf().getInt("io.file.buffer.size".4096), replication, this.getDefaultBlockSize(f));
    }

    public FSDataOutputStream create(Path f, short replication, Progressable progress) throws IOException {
        return this.create(f, true.this.getConf().getInt("io.file.buffer.size".4096), replication, this.getDefaultBlockSize(f), progress);
    }

    public FSDataOutputStream create(Path f, boolean overwrite, int bufferSize) throws IOException {
        return this.create(f, overwrite, bufferSize, this.getDefaultReplication(f), this.getDefaultBlockSize(f));
    }

    public FSDataOutputStream create(Path f, boolean overwrite, int bufferSize, Progressable progress) throws IOException {
        return this.create(f, overwrite, bufferSize, this.getDefaultReplication(f), this.getDefaultBlockSize(f), progress);
    }

    public FSDataOutputStream create(Path f, boolean overwrite, int bufferSize, short replication, long blockSize) throws IOException {
        return this.create(f, overwrite, bufferSize, replication, blockSize, (Progressable)null);
    }

    public FSDataOutputStream create(Path f, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress) throws IOException {
        return this.create(f, FsCreateModes.applyUMask(FsPermission.getFileDefault(), FsPermission.getUMask(this.getConf())), overwrite, bufferSize, replication, blockSize, progress);
    }

    public abstract FSDataOutputStream create(Path var1, FsPermission var2, boolean var3, int var4, short var5, long var6, Progressable var8) throws IOException;

    public FSDataOutputStream create(Path f, FsPermission permission, EnumSet<CreateFlag> flags, int bufferSize, short replication, long blockSize, Progressable progress) throws IOException {
        return this.create(f, permission, flags, bufferSize, replication, blockSize, progress, (ChecksumOpt)null);
    }

    public FSDataOutputStream create(Path f, FsPermission permission, EnumSet<CreateFlag> flags, int bufferSize, short replication, long blockSize, Progressable progress, ChecksumOpt checksumOpt) throws IOException {
        return this.create(f, permission, flags.contains(CreateFlag.OVERWRITE), bufferSize, replication, blockSize, progress);
    }
Copy the code

Create has multiple overloaded functions, and its arguments specify whether to overwrite existing files, the number of file backups, the buffer size for writing files, the block size of files, and file permissions. The return value is an FSDataOutputStream from which files can be written.

  • 7.3 copyFromLocal
    public void copyFromLocalFile(Path src, Path dst) throws IOException {
        this.copyFromLocalFile(false, src, dst);
    }

    public void copyFromLocalFile(boolean delSrc, Path src, Path dst) throws IOException {
        this.copyFromLocalFile(delSrc, true, src, dst);
    }

    public void copyFromLocalFile(boolean delSrc, boolean overwrite, Path[] srcs, Path dst) throws IOException {
        Configuration conf = this.getConf();
        FileUtil.copy(getLocal(conf), srcs, this, dst, delSrc, overwrite, conf);
    }
Copy the code

To copy a local file to a file system, you can specify the Path to upload the local file, the Path array consisting of multiple uploaded paths, the target pair Path, and whether to delete local files or overwrite files created in the HDFS.

  • 7.4 copyToLocalFile
    public void copyToLocalFile(Path src, Path dst) throws IOException {
        this.copyToLocalFile(false, src, dst);
    }

    public void copyToLocalFile(boolean delSrc, Path src, Path dst) throws IOException {
        this.copyToLocalFile(delSrc, src, dst, false);
    }
Copy the code

Copy the destination file to the specified local path. The delSrc parameter specifies whether to delete the source file after the file is moved.

  • 7.6 moveToLocalFile
    public void moveToLocalFile(Path src, Path dst) throws IOException {
        this.copyToLocalFile(true, src, dst);
    }
Copy the code

To move the target file to the specified path, the function calls copyToLocalFile internally.

  • 7.6 exists
    public boolean exists(Path f) throws IOException {
        try {
            return this.getFileStatus(f) ! =null;
        } catch (FileNotFoundException var3) {
            return false; }}Copy the code

Enter a path and check whether the path exists in HDFS. The value is true if the path exists and false if the path does not exist.

  • 7.7 delete
    public abstract boolean delete(Path var1, boolean var2) throws IOException;
Copy the code

The first argument is the path to delete, and the second argument, true, forces the deletion if there are files in the destination folder.

Welcome to the author of this article:

Scan code to pay attention to and reply to “dry goods”, and obtain thousands of GIGABytes of Android, iOS, JavaWeb, big data, artificial intelligence and other learning resources sorted by me.