How to convert list of dictionaries into Pyspark DataFrame ?

Last Updated : 30 May, 2021

In this article, we are going to discuss the creation of the Pyspark dataframe from the list of dictionaries.

We are going to create a dataframe in PySpark using a list of dictionaries with the help createDataFrame() method. The data attribute takes the list of dictionaries and columns attribute takes the list of names.

dataframe = spark.createDataFrame(data, columns)

Example 1:

Python3

# importing module
import pyspark

# importing sparksession from
# pyspark.sql module
from pyspark.sql import SparkSession

# creating sparksession and giving
# an app name
spark = SparkSession.builder.appName('sparkdf').getOrCreate()

# list of dictionaries of students  data
data = [{"Student ID": 1, "Student name": "sravan"},
        {"Student ID": 2, "Student name": "Jyothika"},
        {"Student ID": 3, "Student name": "deepika"},
        {"Student ID": 4, "Student name": "harsha"}]

# creating a dataframe
dataframe = spark.createDataFrame(data)

# display dataframe
dataframe.show()

Output:

Example 2:

Python3

# importing module
import pyspark

# importing sparksession from 
# pyspark.sql module
from pyspark.sql import SparkSession

# creating sparksession and giving 
# an app name
spark = SparkSession.builder.appName('sparkdf').getOrCreate()

# list of dictionaries of crop  data
data = [{"Crop ID": 1, "name": "rose", "State": "AP"},
        {"Crop ID": 2, "name": "lilly", "State": "TS"},
        {"Crop ID": 3, "name": "lotus", "State": "Maharashtra"},
        {"Crop ID": 4, "name": "jasmine", "State": "AP"}]

# creating a dataframe
dataframe = spark.createDataFrame(data)

# display dataframe
dataframe.show()

Output:

Example 3:

Python3

# importing module
import pyspark

# importing sparksession from 
# pyspark.sql module
from pyspark.sql import SparkSession

# creating sparksession and giving 
# an app name
spark = SparkSession.builder.appName('sparkdf').getOrCreate()

# list of dictionaries of crop  data
data = [{"Crop ID": 1, "name": "rose", "State": "AP"},
        {"Crop ID": 2, "name": "lilly", "State": "TS"},
        {"Crop ID": 3, "name": "lotus", "State": "Maharashtra"},
        {"Crop ID": 4, "name": "jasmine", "State": "AP"}]

# creating a dataframe
dataframe = spark.createDataFrame(data)

# display dataframe count
dataframe.count()