Overview of File FormatsΒΆ
Let us go through the details about different file formats supported by STORED AS Clause.
Let us start spark context for this Notebook so that we can execute the code provided. You can sign up for our 10 node state of the art cluster/labs to learn Spark SQL using our unique integrated LMS.
val username = System.getProperty("user.name")
import org.apache.spark.sql.SparkSession
val username = System.getProperty("user.name")
val spark = SparkSession.
builder.
config("spark.ui.port", "0").
config("spark.sql.warehouse.dir", s"/user/${username}/warehouse").
enableHiveSupport.
appName(s"${username} | Spark SQL - Managing Tables - Basic DDL and DML").
master("yarn").
getOrCreate
If you are going to use CLIs, you can use Spark SQL using one of the 3 approaches.
Using Spark SQL
spark2-sql \
--master yarn \
--conf spark.ui.port=0 \
--conf spark.sql.warehouse.dir=/user/${USER}/warehouse
Using Scala
spark2-shell \
--master yarn \
--conf spark.ui.port=0 \
--conf spark.sql.warehouse.dir=/user/${USER}/warehouse
Using Pyspark
pyspark2 \
--master yarn \
--conf spark.ui.port=0 \
--conf spark.sql.warehouse.dir=/user/${USER}/warehouse
Go to this page and review supported file formats.
Supported File Formats
TEXTFILE
ORC
PARQUET
AVRO
SEQUENCEFILE - is not important
JSONFILE - only available in recent vesions of Hive.
and more
We can even specify custom file formats (out of scope)
%%sql
DROP DATABASE IF EXISTS itversity_sms CASCADE
%%sql
CREATE DATABASE IF NOT EXISTS itversity_sms
%%sql
USE itversity_sms
%%sql
CREATE TABLE students (
student_id INT,
student_first_name STRING,
student_last_name STRING,
student_phone_numbers ARRAY<STRING>,
student_address STRUCT<street:STRING, city:STRING, state:STRING, zip:STRING>
) STORED AS parquet
%%sql
INSERT INTO students VALUES (1, 'Scott', 'Tiger', NULL, NULL)
%%sql
INSERT INTO students VALUES (2, 'Donald', 'Duck', ARRAY('1234567890', '2345678901'), NULL)
%%sql
INSERT INTO students VALUES
(3, 'Mickey', 'Mouse', ARRAY('1234567890', '2345678901'), STRUCT('A Street', 'One City', 'Some State', '12345')),
(4, 'Bubble', 'Guppy', ARRAY('5678901234', '6789012345'), STRUCT('Bubbly Street', 'Guppy', 'La la land', '45678'))
%%sql
SELECT * FROM students
import sys.process._
val username = System.getProperty("user.name")
s"hdfs dfs -ls /user/${username}/warehouse/${username}_sms.db/students"!