Common HDFS shell commands are used

1. Display the current directory structure

Hadoop fs-ls -r <path> # display the contents of the root directory hadoop fs-ls /Copy the code

2. Create a directory

Hadoop fs -mkdir -p <path> hadoop fs -mkdir -p <path>Copy the code

3. Delete the vm

Hadoop fs -rm -r <path> hadoop fs -rm -r <path>Copy the code

4. Load the file from the local PC to the HDFS

Hadoop fs-put [localsrc] [DST] hadoop fs-copyfromLocal [localsrc] [DST] hadoop fs-put [localsrc] [DST]Copy the code

5. Export files from the HDFS to the local PC

Hadoop fs-get [DST] [localsrc] hadoop fs-copytolocal [DST] [localsrc] hadoop fs-get [DST] [localsrc]Copy the code

6. View the file content

Hadoop fs -text <path> hadoop fs -cat <path>Copy the code

7. Display the last 1000 bytes of the file

Hadoop fs -tail -f <path> hadoop fs -tail -f <path>Copy the code

8. Copy files

hadoop fs -cp [src] [dst]
Copy the code

9. Move files

hadoop fs -mv [src] [dst] 
Copy the code

10. Calculate the size of each file in the current directory

  • Default unit byte
  • -s: Displays the total size of all files.
  • -h: Will display the file size in a friendlier way (e.g. 64.0m instead of 67108864)
hadoop fs -du  <path>  
Copy the code

11. Merge and download multiple files

  • -nl adds a newline character (LF) to the end of each file
  • -skip-empty-file Skips empty files
Hadoop fs-getmerge # example Merge hbase-policy-xml and hbase-site. XML files on HDFS and download them to /usr/test. XML hadoop fs-getmerge -nl /test/hbase-policy.xml /test/hbase-site.xml /usr/test.xmlCopy the code

12. Collect statistics about the available space of the file system

hadoop fs -df -h /
Copy the code

13. Change the file replication factor

hadoop fs -setrep [-R] [-w] <numReplicas> <path>
Copy the code
  • Change the replication factor of a file. If path is a directory, change the replicators for all files under it
  • -w: Indicates whether the request command waits for replication to complete
Hadoop fs -setrep -w 3 /user/hadoop/dir1Copy the code

14. Permission control

Change the group to which a file or directory belongs. The user must be the owner or superuser of the file. hadoop fs -chgrp [-R] GROUP URI [URI ...] The user must be the owner or superuser of the file. hadoop fs -chmod [-R] <MODE[,MODE]... | OCTALMODE> URI [URI ...] The owner of the modified file must be a superuser. hadoop fs -chown [-R] [OWNER][:[GROUP]] URI [URI ]Copy the code

15. File detection

hadoop fs -test - [defsz]  URI
Copy the code

Optional options:

  • -d: 0 is returned if the path is a directory.
  • -e: 0 is returned if the path exists.
  • -f: Returns 0 if the path is a file.
  • -s: Returns 0 if the path is not empty.
  • -r: 0 is returned if the path exists and the read permission is granted.
  • -w: 0 is returned if the path exists and the write permission is granted.
  • -z: Returns 0 if the file length is zero.
Hadoop fs-test -e filenameCopy the code