HDFS File Permissions

Let us go through file permissions in HDFS.

  • As we create the files, we can check the permissions on them using -ls command.

  • Typically the owner of the user space will have rwx, while members of the group specified as well as others have r-x.

  • rwx stands for read, write and execute while r-x stands for only read and execute permissions.

  • We can change the permissions using hadoop fs -chmod or hdfs dfs -chmod. However one can change the permissions of their own files.

  • We can specify permissions mode (e.g.: +x to grant execute access to owner, group as well as others) as well as octal mode (e.g.: 755 to grant rwx for owner, rx for group and others)

If you are not familiar with linux command chmod, we would highly recommend you to spend some time to get detailed understanding of it as it is very important with respect to file permissions.

%%sh

hdfs dfs -ls /user/${USER}/retail_db

  • Granting write permissions on the folder to others.

%%sh

hdfs dfs -chmod -R o+w /user/${USER}/retail_db/orders

%%sh

hdfs dfs -ls /user/${USER}/retail_db

  • Granting write permissions on the folder to group.

%%sh

hdfs dfs -chmod -R g+w /user/${USER}/retail_db/order_items

%%sh

hdfs dfs -ls /user/${USER}/retail_db

  • We can remove write permissions for every one.

%%sh

hdfs dfs -chmod -R -w /user/${USER}/retail_db/orders

%%sh

hdfs dfs -ls /user/${USER}/retail_db

  • No files can be copied to the folder or can be deleted from the folder. Below command will fail.

%%sh

hdfs dfs -rm /user/${USER}/retail_db/orders/part-00000

%%sh

hdfs dfs -chmod -R -w /user/${USER}/retail_db/order_items

%%sh

hdfs dfs -ls /user/${USER}/retail_db

  • Adding write permissions only to owner. Now the owner will be able to delete the file, but others cannot.

%%sh

hdfs dfs -chmod -R u+w /user/${USER}/retail_db/orders

%%sh

hdfs dfs -rm /user/${USER}/retail_db/orders/part-00000

%%sh

hdfs dfs -chmod -R u+w /user/${USER}/retail_db/order_items

%%sh

hdfs dfs -ls /user/${USER}/retail_db

  • Let us go through the details using octal format.

%%sh

hdfs dfs -ls /user/${USER}/retail_db

  • Granting write permissions on the folder to others. We can set these permissions for owner, group and other.

Binary Value

Permissions Mode

Decimal Value

000

—

0

001

–x

1

010

-w-

2

011

-wx

3

100

r–

4

101

r-x

5

110

rw-

6

111

rwx

7

  • Granting write permissions on the folder to others.

%%sh

hdfs dfs -chmod -R 757  /user/${USER}/retail_db/orders

%%sh

hdfs dfs -ls /user/${USER}/retail_db

  • Granting write permissions on the folder to group.

%%sh

hdfs dfs -chmod -R 775 /user/${USER}/retail_db/order_items

%%sh

hdfs dfs -ls /user/${USER}/retail_db

  • We can remove write permissions for every one.

%%sh

hdfs dfs -chmod -R 555 /user/${USER}/retail_db/orders

%%sh

hdfs dfs -chmod -R 555 /user/${USER}/retail_db/order_items

%%sh

hdfs dfs -ls /user/${USER}/retail_db

  • Adding write permissions only to owner.

%%sh

hdfs dfs -chmod -R 755 /user/${USER}/retail_db/orders

%%sh

hdfs dfs -chmod -R 755 /user/${USER}/retail_db/order_items

%%sh

hdfs dfs -ls /user/${USER}/retail_db
Using HDFS Stat Commands Overriding Properties