Retrieve Metadata of TablesΒΆ

As the table is created, let us understand how to get the metadata of a table.

  • We can get metadata of Hive Tables using several commands.

    • DESCRIBE - e.g.: DESCRIBE orders;

    • DESCRIBE EXTENDED - e.g.: DESCRIBE EXTENDED orders;

    • DESCRIBE FORMATTED - e.g.: DESCRIBE FORMATTED orders;

  • DESCRIBE will give only field names and data types.

  • DESCRIBE EXTENDED will give all the metadata, but not in readable format in Hive. It is same as DESCRIBE FORMATTED in Spark SQL.

  • DESCRIBE FORMATTED will give metadata in readable format.

As the output is truncated using Jupyter, we will actually see the details using spark-sql

import org.apache.spark.sql.SparkSession

val username = System.getProperty("user.name")
val spark = SparkSession.
    builder.
    config("spark.ui.port", "0").
    config("spark.sql.warehouse.dir", s"/user/${username}/warehouse").
    enableHiveSupport.
    master("yarn").
    appName(s"${username} | Spark SQL - Getting Started").
    getOrCreate

%%sql

SELECT current_database()

%%sql

USE itversity_retail

%%sql

SHOW tables

%%sql

DESCRIBE orders

%%sql

DESCRIBE EXTENDED orders

%%sql

DESCRIBE FORMATTED orders
Managing Spark Metastore Tables Role of Spark or Hive Metastore