Copying files from HDFS to HDFS¶

Let us understand how to copy files with in HDFS (from one HDFS location to another HDFS location). I

We can use hdfs dfs -cp command to copy files with in HDFS.
One need to have at least read permission on source folders or files and write permission on target folder for cp command to work as expected.

%%sh

hdfs dfs -rm -R -skipTrash /user/${USER}/retail_db

%%sh

hdfs dfs -ls /public/retail_db

%%sh

hdfs dfs -ls /user/${USER}

%%sh

hdfs dfs -help cp

Let us create directory to store all the folders and files related to HDFS under user space. You can review the permissions on retail_db, user have write permissions on the target folder.

%%sh

hdfs dfs -mkdir /user/${USER}/retail_db

%%sh

hdfs dfs -ls /user/${USER}

%%sh

hdfs dfs -cp /public/retail_db/* /user/${USER}/retail_db

%%sh

hdfs dfs -ls /user/${USER}/retail_db

Note

This will fail as retail_db folder already exists.

%%sh

hdfs dfs -cp /public/retail_db /user/${USER}

Note

Alternative approach, where the folder and contents are copied directly.

%%sh

hdfs dfs -rm -R -skipTrash /user/${USER}/retail_db

%%sh

hdfs dfs -ls /user/${USER}

%%sh

hdfs dfs -cp /public/retail_db /user/${USER}

%%sh

hdfs dfs -ls -R /user/${USER}/retail_db

We can also use patterns while using cp command to copy files within HDFS. Also, we can pass multiple files or folders in HDFS to cp command.

%%sh

hdfs dfs -rm -R -skipTrash /user/${USER}/retail_db

Deleted /user/itversity/retail_db

%%sh

hdfs dfs -ls /user/${USER}

Found 3 items
drwx------   - itversity students          0 2021-01-17 19:11 /user/itversity/.Trash
drwxr-xr-x   - itversity students          0 2021-01-14 07:46 /user/itversity/.sparkStaging
drwxr-xr-x   - itversity students          0 2021-01-14 07:41 /user/itversity/warehouse

%%sh

hdfs dfs -mkdir /user/${USER}/retail_db

%%sh

hdfs dfs -cp /public/retail_db/order* /user/${USER}/retail_db

%%sh

hdfs dfs -ls /user/${USER}/retail_db

Found 2 items
drwxr-xr-x   - itversity students          0 2021-01-17 20:03 /user/itversity/retail_db/order_items
drwxr-xr-x   - itversity students          0 2021-01-17 20:03 /user/itversity/retail_db/orders

%%sh

hdfs dfs -cp /public/retail_db/departments /public/retail_db/products /user/${USER}/retail_db

%%sh

hdfs dfs -ls /user/${USER}/retail_db

Found 4 items
drwxr-xr-x   - itversity students          0 2021-01-17 20:04 /user/itversity/retail_db/departments
drwxr-xr-x   - itversity students          0 2021-01-17 20:03 /user/itversity/retail_db/order_items
drwxr-xr-x   - itversity students          0 2021-01-17 20:03 /user/itversity/retail_db/orders
drwxr-xr-x   - itversity students          0 2021-01-17 20:04 /user/itversity/retail_db/products

%%sh

hdfs dfs -cp /public/retail_db/categories /public/retail_db/customers /user/${USER}/retail_db

%%sh

hdfs dfs -ls /user/${USER}/retail_db

Found 6 items
drwxr-xr-x   - itversity students          0 2021-01-17 20:05 /user/itversity/retail_db/categories
drwxr-xr-x   - itversity students          0 2021-01-17 20:05 /user/itversity/retail_db/customers
drwxr-xr-x   - itversity students          0 2021-01-17 20:04 /user/itversity/retail_db/departments
drwxr-xr-x   - itversity students          0 2021-01-17 20:03 /user/itversity/retail_db/order_items
drwxr-xr-x   - itversity students          0 2021-01-17 20:03 /user/itversity/retail_db/orders
drwxr-xr-x   - itversity students          0 2021-01-17 20:04 /user/itversity/retail_db/products

Apache Spark using SQL

Copying files from HDFS to HDFS¶