Explanation of all PySpark RDD, DataFrame and SQL examples present on this project are available at Apache PySpark Tutorial, All these examples are coded in Python language and tested in our development environment.

SparkSession

SparkSession also includes all the APIs available in different contexts –

SparkContext,
SQLContext,
StreamingContext,
HiveContext.

Docs

• https://sparkbyexamples.com/pyspark-tutorial/
• https://sparkbyexamples.com/pyspark/pyspark-what-is-sparksession/

Install

JAVA_HOME = C:\Program Files\Java\jdk1.8.0_201
PATH = %PATH%;C:\Program Files\Java\jdk1.8.0_201\bin


SPARK_HOME  = C:\apps\spark-3.0.0-bin-hadoop2.7
HADOOP_HOME = C:\apps\spark-3.0.0-bin-hadoop2.7
PATH=%PATH%;C:\apps\spark-3.0.0-bin-hadoop2.7\bin

** Spark Shell + Web UI
- $SPARK_HOME/sbin/pyspark

Spark-shell also creates a Spark context web UI and by default, it can access from http://localhost:4041.

• Spark

scala -version
spark-submit --version
spark-shell --version
spark-sql --version

• Jupyter notebook

pip install jupyter
jupyter notebook

The spark-submit

The spark-submit command is a utility to run or submit a Spark or PySpark application program (or job) to the cluster by specifying options and configurations, the application you are submitting can be written in Scala, Java, or Python (PySpark). spark-submit command supports the following.

Submitting Spark application on different cluster managers like Yarn, Kubernetes, Mesos, and Stand-alone.
Submitting Spark application on client or cluster deployment modes.

 spark-3.0.1-bin-hadoop3.2/bin/spark-submit test.py

Create SparkSession from builder

import pyspark
from pyspark.sql import SparkSession
spark = SparkSession.builder.master("local[1]") \
                    .appName('SparkByExamples.com') \
                    .getOrCreate()

Table of Contents (Spark Examples in Python)

PySpark Basic Examples

• How to create SparkSession
• PySpark – Accumulator
• PySpark Repartition vs Coalesce
• PySpark Broadcast variables
• PySpark – repartition() vs coalesce()
• PySpark – Parallelize
• PySpark – RDD
• PySpark – Web/Application UI
• PySpark – SparkSession
• PySpark – Cluster Managers
• PySpark – Install on Windows
• PySpark – Modules & Packages
• PySpark – Advantages
• PySpark – Features
• PySpark – What is it? & Who uses it?

PySpark DataFrame Examples

• PySpark – Create a DataFrame
• PySpark – Create an empty DataFrame
• PySpark – Convert RDD to DataFrame
• PySpark – Convert DataFrame to Pandas
• PySpark – StructType & StructField
• PySpark Row using on DataFrame and RDD
• Select columns from PySpark DataFrame
• PySpark Collect() – Retrieve data from DataFrame
• PySpark withColumn to update or add a column
• PySpark using where filter function
• PySpark – Distinct to drop duplicate rows
• PySpark orderBy() and sort() explained
• PySpark Groupby Explained with Example
• PySpark Join Types Explained with Examples
• PySpark Union and UnionAll Explained
• PySpark UDF (User Defined Function)
• PySpark flatMap() Transformation
• PySpark map Transformation

PySpark SQL Functions

• PySpark Aggregate Functions with Examples
• PySpark Window Functions

PySpark Datasources

• PySpark Read CSV file into DataFrame
• PySpark read and write Parquet File