Create Spark Metastore Tables

Let us understand how to create tables in Spark Metastore. We will be focusing on syntax and semantics.

Let us start spark context for this Notebook so that we can execute the code provided. You can sign up for our 10 node state of the art cluster/labs to learn Spark SQL using our unique integrated LMS.

val username = System.getProperty("user.name")

import org.apache.spark.sql.SparkSession

val username = System.getProperty("user.name")
val spark = SparkSession.
    builder.
    config("spark.ui.port", "0").
    config("spark.sql.warehouse.dir", s"/user/${username}/warehouse").
    enableHiveSupport.
    appName(s"${username} | Spark SQL - Managing Tables - Basic DDL and DML").
    master("yarn").
    getOrCreate

If you are going to use CLIs, you can use Spark SQL using one of the 3 approaches.

Using Spark SQL

spark2-sql \
    --master yarn \
    --conf spark.ui.port=0 \
    --conf spark.sql.warehouse.dir=/user/${USER}/warehouse

Using Scala

spark2-shell \
    --master yarn \
    --conf spark.ui.port=0 \
    --conf spark.sql.warehouse.dir=/user/${USER}/warehouse

Using Pyspark

pyspark2 \
    --master yarn \
    --conf spark.ui.port=0 \
    --conf spark.sql.warehouse.dir=/user/${USER}/warehouse

  • Let us drop and recreate the table. We need to determine table type, file format based up on the files that will be copied to the table. If the file format is delimited text file then we need to understand field delimiter as well.

    • Managed Table

    • Text File Format

    • Field Delimiter ‘,’

  • We will create the table based on the structure of data in /data/retail_db/orders

  • If you are using spark-sql make sure to end the statements with semi-colon. With spark-shell or pyspark make sure to use spark.sql to pass these commands.

%%sql

USE itversity_retail

%%sql

DROP TABLE orders

%%sql

CREATE TABLE orders (
  order_id INT,
  order_date STRING,
  order_customer_id INT,
  order_status STRING
) ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
STORED AS TEXTFILE

%%sql

SHOW tables

%%sql

USE itversity_retail

%%sql

DROP TABLE orders

%%sql

CREATE TABLE orders (
  order_id INT,
  order_date STRING,
  order_customer_id INT,
  order_status STRING
) ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
STORED AS TEXTFILE

%%sql

SHOW tables

  • Using Spark SQL with Python or Scala

spark.sql("USE itversity_retail")

spark.sql("DROP TABLE orders")

spark.sql("""
CREATE TABLE orders (
  order_id INT,
  order_date STRING,
  order_customer_id INT,
  order_status STRING
) ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
STORED AS TEXTFILE
""")

spark.sql("SHOW tables").show()

spark.sql("USE itversity_retail")

spark.sql("DROP TABLE orders")

spark.sql("""
CREATE TABLE orders (
  order_id INT,
  order_date STRING,
  order_customer_id INT,
  order_status STRING
) ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
STORED AS TEXTFILE
""")

spark.sql("SHOW tables").show()

  • Once the table is created either we can copy files using LOAD or insert query results using INSERT. We will see how to use insert to insert query results as part of the next section.

  • We can also use INSERT to insert individual records. But this approach is used less often. Let us create additional table and see how we can use INSERT to insert individual records.

%%sql
DROP DATABASE IF EXISTS itversity_sms CASCADE

%%sql

CREATE DATABASE IF NOT EXISTS itversity_sms

%%sql

USE itversity_sms

%%sql

CREATE TABLE students (
    student_id INT,
    student_first_name STRING,
    student_last_name STRING,
    student_phone_number STRING,
    student_address STRING
) STORED AS TEXTFILE

%%sql

INSERT INTO students VALUES (1, 'Scott', 'Tiger', NULL, NULL)

%%sql

INSERT INTO students VALUES (2, 'Donald', 'Duck', '1234567890', NULL)

%%sql

INSERT INTO students VALUES 
    (3, 'Mickey', 'Mouse', '2345678901', 'A Street, One City, Some State, 12345'),
    (4, 'Bubble', 'Guppy', '6789012345', 'Bubbly Street, Guppy, La la land, 45678')

%%sql

SELECT * FROM students

%%sql

INSERT INTO students VALUES ('Scott', 'Tiger', 1, NULL, NULL)

%%sql

SELECT * FROM students

  • Using Spark SQL with Python or Scala

spark.sql("DROP DATABASE IF EXISTS itversity_sms CASCADE")

spark.sql("CREATE DATABASE IF NOT EXISTS itversity_sms")

spark.sql("USE itversity_sms")

spark.sql("""
CREATE TABLE students (
    student_id INT,
    student_first_name STRING,
    student_last_name STRING,
    student_phone_number STRING,
    student_address STRING
) STORED AS TEXTFILE
""")

spark.sql("INSERT INTO students VALUES (1, 'Scott', 'Tiger', NULL, NULL)")

spark.sql("INSERT INTO students VALUES (2, 'Donald', 'Duck', '1234567890', NULL)")

spark.sql("""
INSERT INTO students VALUES 
    (3, 'Mickey', 'Mouse', '2345678901', 'A Street, One City, Some State, 12345'),
    (4, 'Bubble', 'Guppy', '6789012345', 'Bubbly Street, Guppy, La la land, 45678')
""")    

spark.sql("SELECT * FROM students").show()
Basic DDL and DML Overview of Data Types