Running OS CommandsΒΆ

Let us understand how to run OS commands using Spark SQL CLI.

  • We can run OS commands using ! at the beginning.

    • Listing local Files !ls -ltr;

    • Listing HDFS Files !hdfs dfs -ls /public/retail_db;

import sys.process._

"ls -ltr"!

import sys.process._

"hdfs dfs -ls /public/retail_db"!
Overview of Spark SQL Properties Understanding Warehouse Directory