Pyspark - Converting JSON to DataFrame

Pyspark - Converting JSON to DataFrame

Last Updated : 29 Jun, 2021

In this article, we are going to convert JSON String to DataFrame in Pyspark.

Method 1: Using read_json()

We can read JSON files using pandas.read_json. This method is basically used to read JSON files through pandas.

Syntax: pandas.read_json("file_name.json")

Here we are going to use this JSON file for demonstration:

Code:

Python3

# import pandas to read json file
import pandas as pd

# importing module
import pyspark

# importing sparksession from pyspark.sql module
from pyspark.sql import SparkSession

# creating sparksession and giving an app name
spark = SparkSession.builder.appName('sparkdf').getOrCreate()


# creating a dataframe from the json file named student
dataframe = spark.createDataFrame(pd.read_json('student.json'))

# display the dataframe (Pyspark dataframe)
dataframe.show()

Output:

Method 2: Using spark.read.json()

This is used to read a json data from a file and display the data in the form of a dataframe

Syntax: spark.read.json('file_name.json')

JSON file for demonstration:

Code:

Python3

# importing module
import pyspark

# importing sparksession from pyspark.sql module
from pyspark.sql import SparkSession

# creating sparksession and giving an app name
spark = SparkSession.builder.appName('sparkdf').getOrCreate()

# read json file
data = spark.read.json('college.json')

# display json data
data.show()

Output:

Pyspark - Converting JSON to DataFrame

S

sravankumar_171fa07058

Improve

Article Tags :

Practice Tags :

python

Similar Reads

Convert PySpark RDD to DataFrame

In this article, we will discuss how to convert the RDD to dataframe in PySpark. There are two approaches to convert RDD to dataframe. Using createDataframe(rdd, schema)Using toDF(schema) But before moving forward for converting RDD to Dataframe first let's create an RDD Example: Python # importing

Convert JSON to Pandas DataFrame

When working with data, it's common to encounter JSON (JavaScript Object Notation) files, which are widely used for storing and exchanging data. Pandas, a powerful data manipulation library in Python, provides a convenient way to convert JSON data into a Pandas data frame. In this article, we'll exp

How to Convert Pandas to PySpark DataFrame ?

In this article, we will learn How to Convert Pandas to PySpark DataFrame. Sometimes we will get csv, xlsx, etc. format data, and we have to store it in PySpark DataFrame and that can be done by loading data in Pandas then converted PySpark DataFrame. For conversion, we pass the Pandas dataframe int

Convert PySpark Row List to Pandas DataFrame

In this article, we will convert a PySpark Row List to Pandas Data Frame. A Row object is defined as a single Row in a PySpark DataFrame. Thus, a Data Frame can be easily represented as a Python List of Row objects. Method 1 : Use createDataFrame() method and use toPandas() method Here is the syntax

Convert PySpark dataframe to list of tuples

In this article, we are going to convert the Pyspark dataframe into a list of tuples. The rows in the dataframe are stored in the list separated by a comma operator. So we are going to create a dataframe by using a nested list Creating Dataframe for demonstration: Python3 # importing module import p