Scribd
Upload
17 views

Data Extraction: Parse A 3-Nested JSON Object and Convert It To A Pandas Dataframe

The document discusses querying APIs in Python and parsing the JSON output. It covers making a request to an API, the structure of nested JSON, and how to flatten a nested JSON response into a Pandas dataframe.

Uploaded by

fabiofurlans
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
17 views

Data Extraction: Parse A 3-Nested JSON Object and Convert It To A Pandas Dataframe

The document discusses querying APIs in Python and parsing the JSON output. It covers making a request to an API, the structure of nested JSON, and how to flatten a nested JSON response into a Pandas dataframe.

Uploaded by

fabiofurlans
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 1

Published in Python in Plain English

Get unlimited access

Tejeswini
Search
Jun 6, 2021 · 4 min read · Listen

Data Extraction: Parse a 3-Nested JSON Object


and Convert it to a pandas dataframe
Tejeswini
22 Followers

Data| Machine Learning| Technology| Design|


Leadership

Follow

You can now subscribe to


More from Medium
get stories delivered directly
to your inbox.
DIAN EXSI PALUPI
Got it
133 Static Data Types, Operators, and
Assignments-Python

Matt Eland

Installing Anaconda for Python


Development

Felipe Veiga in Python in Plain English

Integrate Python and Oracle


Databases

Mohammad Alim

Python Packages For Data


Analytics

Help Status Writers Blog Careers Privacy Terms About


Reference: Jason Bourne Poster Knowable

JSON, or JavaScript Object Notation, is a human-readable text-based format


for data exchange between a server and web application, as an alternative to XML.

Querying API in Python with JSON Output


As a follow-up to my previous blog which mentions rapidapi.com as one of the
data sources to extract OHLC (Open, High, Low and Close price) data for
cryptocurrency, let's look at querying an API in Python.

We can use the requests package for this purpose. To get the data as a JSON
output, we call the request.get method. The response variable in the below
code, stores the returned values, and the JSON output is parsed using the json( )
method.

API request to rapidapi.com

The request.get method has the following parameter values: requests.get(url,


params={key: value}, args).

Detailed documentation can be looked at here:

Python Requests get Method


❮ Requests Module Make a request to a web page, and return the
status code: import requests x =…
www.w3schools.com

What is a Nested-JSON ?
A JSON object is a collection of key and value pairs.

For example, { ‘data’ : [ {‘screen_ID’ : ‘63’, in this instance ‘data’ and ‘screen_ID’
are the keys, and ‘63’ is the value. Keys must be strings, and values must be a
valid JSON data type (string, number, object, array, boolean or null).

So, what is a Nested-JSON? A Nested-JSON is a JSON object which has other


JSON objects or Javascript arrays as its values.

Nested-JSON Types (Source:www.digitalocean.com)

The API request to rapidapi.com produces a nested JSON output with Javascript
arrays, structured as follows:

JSON output of API request to rapidapi.com

JSON Output to Pandas Dataframe


Each nested JSON object has a unique access path. To get first-level keys, we can
use the json.keys( ) method. In this case, it returns ‘data’ which is the first level
key and can be seen from the above image of the JSON output.

pd.json_normalize is a function of pandas that comes in handy in flattening the


JSON output into a datatable. So, using the first level key in the following code
format returns a datatable like below:

df stores flattened table of json output

Our problem statement here is to extract the keys and corresponding values of
‘color’, ‘date’, ‘high’, ‘low’, ‘open’, ‘perc_chg’, ‘price’, ‘vol’ to a data table format.
These values are contained in ‘screen_data.data’( To access nested fields,
concatenate the field names with a . (dot) as separator) column of the df,
dataframe, using below code we extract this column to a datatable:

dt stores df[‘screen_data.data’]

But here all the values are stored as a list. To transform each element of this list
to a row in the dataframe, we can use pd.dataframe.explode. This returns
exploded lists as rows of the subset columns; index will be duplicated for these
rows as below.

Here, dt stores exploded dataframe, dt

Now that we have a dictionary in each row of the datatable dt, we can use
pandas.Series to convert the dictionaries to a Pandas series format and apply
pandas.Dataframe to further convert it into type dataframe.

Final datafram df_final

“Data extraction through API requests and web scraping is especially useful for
analyzing or automating data service on web-based applications.”

The JSON format is already compatible in itself and pandas functions add
extra flexibility in making the parsing and readability of this data more user-
friendly.

Follow me on Medium for more articles on cryptocurrency and topics


related to Data Analysis, AI, Machine Learning, Design and Technology

Please feel free to share your thoughts, comments and feedback

Follow me here on Medium or connect with me on LinkedIn to collaborate


and share ideas.

More content at plainenglish.io

133

More from Python in Plain English Follow

New Python content every day. Follow to join 500k+ monthly readers.

Devindi Jayathilaka · Jun 6, 2021

How Does a Machine Understand Text or Speech?


Convert Text to Features using TF-IDF — How can a machine understand
what we talk or what we write? As we already know, it cannot understand
our words or sentences except numbers. So, we should make unstructure…

Data Science 3 min read

Share your ideas with millions of readers. Write on Medium

Sanjana Chakravarty · Jun 6, 2021

Deciphering Competitive Code — Part 1 of…?


How I stumbled upon a piece of code that made me rethink my whole
understanding of programming — I have never attempted competitive
coding seriously before — and neither have I started now. I started codin…

Python 6 min read

Aishah Ismail · Jun 6, 2021

How to convert google colab jupyter notebook


(.ipynb) to HTML
Often, we need to share our code explanations and gures written in a
Google colab Jupyter notebook (.py or .ipynb les) with our…

Python 2 min read

Tejas A · Jun 6, 2021

An Introduction to the List Data Structure in Python


for Beginners
A beginner’s tutorial on the list data structure in Python and its usage. —
The di erent types of majorly used data structures present in Python are:…

Python 5 min read

Haider Imtiaz · Jun 6, 2021

How to Earn Money through Coding


15 things you can do to earn money as a programmer — In this article, we
are going to see 15 ways to make money through coding. Coding is fun and
we can make money with it. But we don’t know how or where to start, wh…

Programming 9 min read

Read more from Python in Plain English

You might also like