Windowing FunctionsΒΆ

As part of this section we will primarily talk about Windowing Functions. These are also known as Analytic Functions in Databases like Oracle.

  • Prepare HR Database

  • Overview of Windowing Functions

  • Aggregations using Windowing Functions

  • Getting LEAD and LAG values

  • Getting first and last values

  • Ranking using Windowing Functions

  • Understanding order of execution of SQL

  • Overview of Nested Sub Queries

  • Filtering - Window Function Results

import org.apache.spark.sql.SparkSession

val username = System.getProperty("user.name")
val spark = SparkSession.
    builder.
    config("spark.ui.port", "0").
    config("spark.sql.warehouse.dir", s"/user/${username}/warehouse").
    enableHiveSupport.
    appName(s"${username} | Spark SQL - Windowing Functions").
    master("yarn").
    getOrCreate

%%sql

SET spark.sql.shuffle.partitions=2
Exercises - Pre-Defined Functions Prepare HR Database