Getting HDFS Storage UsageΒΆ

Let us get an overview of HDFS usage using du and df commands.

  • We can use hdfs dfs -df to get the current capacity and usage of HDFS.

  • We can use hdfs dfs -du to get the size occupied by a file or folder.

%%sh

hdfs dfs -help df

-df [-h] [<path> ...] :
  Shows the capacity, free and used space of the filesystem. If the filesystem has
  multiple partitions, and no path to a particular partition is specified, then
  the status of the root partitions will be shown.
                                                                                 
  -h  Formats the sizes of files in a human-readable fashion rather than a number
      of bytes.                                                                  

%%sh

hdfs dfs -df

Filesystem                                Size           Used      Available  Use%
hdfs://nn01.itversity.com:8020  18088946967552  7096522956949  9985130049852   39%

%%sh

hdfs dfs -df -h

Filesystem                        Size   Used  Available  Use%
hdfs://nn01.itversity.com:8020  16.5 T  6.5 T      9.1 T   39%

%%sh

hdfs dfs -help du

-du [-s] [-h] <path> ... :
  Show the amount of space, in bytes, used by the files that match the specified
  file pattern. The following flags are optional:
                                                                                 
  -s  Rather than showing the size of each individual file that matches the      
      pattern, shows the total (summary) size.                                   
  -h  Formats the sizes of files in a human-readable fashion rather than a number
      of bytes.                                                                  
  
  Note that, even without the -s option, this only shows size summaries one level
  deep into a directory.
  
  The output is in the form 
  	size	name(full path)

%%sh

hdfs dfs -du /user/${USER}/retail_db

1029     /user/itversity/retail_db/categories
953719   /user/itversity/retail_db/customers
60       /user/itversity/retail_db/departments
5408880  /user/itversity/retail_db/order_items
2999944  /user/itversity/retail_db/orders
174155   /user/itversity/retail_db/products

%%sh

hdfs dfs -du -s /user/${USER}/retail_db

9537787  /user/itversity/retail_db

%%sh

hdfs dfs -du -h /user/${USER}/retail_db

1.0 K    /user/itversity/retail_db/categories
931.4 K  /user/itversity/retail_db/customers
60       /user/itversity/retail_db/departments
5.2 M    /user/itversity/retail_db/order_items
2.9 M    /user/itversity/retail_db/orders
170.1 K  /user/itversity/retail_db/products

%%sh

hdfs dfs -du -s -h /user/${USER}/retail_db

9.1 M  /user/itversity/retail_db
HDFS Replication Factor Using HDFS Stat Commands