SparkSQL.jl

SparkSession Function

Submits application to Apache Spark cluster with config options.

Arguments

Examples

Basic spark session:

spark = SparkSession("spark://example.com:7077", "Julia SparkSQL Example App")

Spark Session with Hive config:

spark = SparkSession("spark://example.com:7077", "Julia SparkSQL Example App", Dict{String,String}("spark.sql.warehouse.dir"=>"LOCATION","spark.sql.catalogImplementation"=>"hive"))

Spark Session with Delta Lake config:

spark = SparkSession("spark://example.com:7077", "Julia SparkSQL Example App", Dict{String,String}("spark.sql.warehouse.dir"=>"LOCATION", "spark.sql.extensions"=>"io.delta.sql.DeltaSparkSessionExtension", "spark.sql.catalog.spark_catalog"=>"org.apache.spark.sql.delta.catalog.DeltaCatalog"))