Write the test

Write test bottom layer:

Test command line:

Doop -mapreduce-clientjobclient-3.1.3-tests.jar TestDFSIO -write -nrFiles 10 -fileSize 128MB

Ps :nrFiles n indicates the number of mapTasks to be generated. In the production environment, you can run hadoop103:8088 to check the number of CPU cores. Set this parameter to (Number of CPU cores -1).

Command Execution Result

2021-02-09 10:43:16,854 INFO fs.TestDFSIO: Number of files: 10

2021-02-09 10:43:16,854 INFO fs.TestDFSIO: Total MBytes processed: 1280

2021-02-09 10:43:16,854 INFO fs.TestDFSIO: Throughput mb/sec: 1.61

2021-02-09 10:43:16,854 INFO fs.TestDFSIO: Average I/O rate MB/SEC: 1.9

2021-02-09 10:43:16,854 INFO fs.TestDFSIO: IO rate std deviation: 0.76

2021-02-09 10:43:16,854 INFO fs.TestDFSIO: Test exec time sec: 133.05

  • Total MBytes processed:
  • File size processed by a single map
  • Throughput mb/sec:
  • Throughput of a single mapTak
  • Calculation method: total file size processed/time taken to write data for each mapTask
  • Total cluster throughput: Number of mapTasks generated * throughput of a single mapTak
  • Average IO rate mb/sec::
  • Average throughput of mapTak
  • Calculation method: Add the file size of each mapTask/the data writing time of each mapTask divided by the number of tasks
  • IO rate std deviation:
  • Variance, reflecting the difference of each mapTask processing, the smaller the more balanced

Read performance

Read 10 128 MB files in HDFS

Hadoop jar/opt/module/hadoop – 3.1.3 / share/hadoop/graphs/hadoop – graphs – the client – jobclient – 3.1.3 – tests. The jar TestDFSIO -read -nrFiles 10 -fileSize 128MB