The process of extracting required data/information from a web page by accessing the HTML of the web page is called Web Scraping or Web Harvesting or Web Data Extraction.
This article discusses the steps involved in Web Scraping by using Flutter's html and http packages.
Step 1: Set up a new Flutter App
Create a new flutter app by running the command :
flutter create YOUR_APP_NAME
- Open the app in VS Code or Android Studio. I am using VS Code.
- Open the lib/main.dart file and clear all the default code
- Add the code for your desired widgets. I will be having an AppBar, a Column containing three Text widgets, a CircularProgressIndicator, and a MaterialButton widget.
Dart
import 'package:flutter/material.dart';
void main() => runApp(MaterialApp(
theme: ThemeData(
accentColor: Colors.green,
scaffoldBackgroundColor: Colors.green[100],
primaryColor: Colors.green,
),
home: MyApp()));
class MyApp extends StatefulWidget {
const MyApp({Key key}) : super(key: key);
@override
_MyAppState createState() => _MyAppState();
}
class _MyAppState extends State<MyApp> {
// Strings to store the extracted Article titles
String result1 = 'Result 1';
String result2 = 'Result 2';
String result3 = 'Result 3';
// boolean to show CircularProgressIndication
// while Web Scraping awaits
bool isLoading = false;
@override
Widget build(BuildContext context) {
return Scaffold(
appBar: AppBar(title: Text('GeeksForGeeks')),
body: Padding(
padding: const EdgeInsets.all(16.0),
child: Center(
child: Column(
mainAxisAlignment: MainAxisAlignment.center,
children: [
// if isLoading is true show loader
// else show Column of Texts
isLoading
? CircularProgressIndicator()
: Column(
children: [
Text(result1,
style: TextStyle(
fontSize: 20, fontWeight: FontWeight.bold)),
SizedBox(
height: MediaQuery.of(context).size.height * 0.05,
),
Text(result2,
style: TextStyle(
fontSize: 20, fontWeight: FontWeight.bold)),
SizedBox(
height: MediaQuery.of(context).size.height * 0.05,
),
Text(result3,
style: TextStyle(
fontSize: 20, fontWeight: FontWeight.bold)),
],
),
SizedBox(height: MediaQuery.of(context).size.height * 0.08),
MaterialButton(
onPressed: () {},
child: Text(
'Scrap Data',
style: TextStyle(color: Colors.white),
),
color: Colors.green,
)
],
)),
),
);
}
}
Output :
Step 2: Add the HTML and HTTP packages.
- Open the pubspec.yaml file and under the dependencies:  add two lines http: ^0.12.0+4 and  html: ^0.14.0+3 with proper indentation and save the file.
- Then in your terminal run the command :
flutter pub get
- Open the main.dart file and import the packages by adding these lines on the top:
import 'package:html/parser.dart' as parser;
import 'package:http/http.dart' as http;
Step 3: Adding Web Scraping functionality
- Now to extract a particular data we first need to decide on a parent class with a unique class name from the rest of the document and the hierarchy of its children, for this we need to view the HTML document of the page. We can do that by opening the website on Chrome Browser and then right-click on the required text and click on Inspect.
- From the above image, you can see that I have selected a Parent class with the class name = "articles-list" , because it has a different name from all the other classes in the document. Now If we look at the Children class we want to extract, we can see for the first article's title we need this kind of hierarchy :
"articles-list" class  >> children[0]  >> children[0]  >> children[0]Â
- Similarly, for the second and third titles, it would be :
"articles-list" class  >> children[1]  >> children[0]  >> children[0]Â
"articles-list" class  >> children[2]  >> children[0]  >> children[0]  Â
- Now that we have the Class Name and the Hierarchy we can go ahead and write the function that does Web Scraping :
Future<List<String>> extractData() async {
//Getting the response from the targeted url
final response =
await http.Client().get(Uri.parse('https://www.geeksforgeeks.org/'));
//Status Code 200 means response has been received successfully
if (response.statusCode == 200) {
//Getting the html document from the response
var document = parser.parse(response.body);
try {
//Scraping the first article title
var responseString1 = document
.getElementsByClassName('articles-list')[0]
.children[0]
.children[0]
.children[0];
print(responseString1.text.trim());
//Scraping the second article title
var responseString2 = document
.getElementsByClassName('articles-list')[0]
.children[1]
.children[0]
.children[0];
print(responseString2.text.trim());
//Scraping the third article title
var responseString3 = document
.getElementsByClassName('articles-list')[0]
.children[2]
.children[0]
.children[0];
print(responseString3.text.trim());
//Converting the extracted titles into string and returning a list of Strings
return [
responseString1.text.trim(),
responseString2.text.trim(),
responseString3.text.trim()
];
} catch (e) {
return ['', '', 'ERROR!'];
}
} else {
return ['', '', 'ERROR: ${response.statusCode}.'];
}
}- Now we will call this function in the onPressed: parameter of MaterialButton and show CircularProgressIndicator till it gets the result.
onPressed: () async {
//Setting isLoading true to show the loader
setState(() {
isLoading = true;
});
//Awaiting for web scraping function to return list of strings
final response = await extractData();
//Setting the received strings to be displayed and making isLoading false to hide the loader
setState(() {
result1 = response[0];
result2 = response[1];
result3 = response[2];
isLoading = false;
});
}- Now after all this our main.dart looks something like this :
Dart
import 'package:flutter/material.dart';
import 'package:html/parser.dart' as parser;
import 'package:http/http.dart' as http;
void main() => runApp(MaterialApp(
theme: ThemeData(
accentColor: Colors.green,
scaffoldBackgroundColor: Colors.green[100],
primaryColor: Colors.green,
),
home: MyApp()));
class MyApp extends StatefulWidget {
const MyApp({Key key}) : super(key: key);
@override
_MyAppState createState() => _MyAppState();
}
class _MyAppState extends State<MyApp> {
// Strings to store the extracted Article titles
String result1 = 'Result 1';
String result2 = 'Result 2';
String result3 = 'Result 3';
// boolean to show CircularProgressIndication
// while Web Scraping awaits
bool isLoading = false;
Future<List<String>> extractData() async {
// Getting the response from the targeted url
final response =
await http.Client().get(Uri.parse('https://www.geeksforgeeks.org/'));
// Status Code 200 means response has been received successfully
if (response.statusCode == 200) {
// Getting the html document from the response
var document = parser.parse(response.body);
try {
// Scraping the first article title
var responseString1 = document
.getElementsByClassName('articles-list')[0]
.children[0]
.children[0]
.children[0];
print(responseString1.text.trim());
// Scraping the second article title
var responseString2 = document
.getElementsByClassName('articles-list')[0]
.children[1]
.children[0]
.children[0];
print(responseString2.text.trim());
// Scraping the third article title
var responseString3 = document
.getElementsByClassName('articles-list')[0]
.children[2]
.children[0]
.children[0];
print(responseString3.text.trim());
// Converting the extracted titles into
// string and returning a list of Strings
return [
responseString1.text.trim(),
responseString2.text.trim(),
responseString3.text.trim()
];
} catch (e) {
return ['', '', 'ERROR!'];
}
} else {
return ['', '', 'ERROR: ${response.statusCode}.'];
}
}
@override
Widget build(BuildContext context) {
return Scaffold(
appBar: AppBar(title: Text('GeeksForGeeks')),
body: Padding(
padding: const EdgeInsets.all(16.0),
child: Center(
child: Column(
mainAxisAlignment: MainAxisAlignment.center,
children: [
// if isLoading is true show loader
// else show Column of Texts
isLoading
? CircularProgressIndicator()
: Column(
children: [
Text(result1,
style: TextStyle(
fontSize: 20, fontWeight: FontWeight.bold)),
SizedBox(
height: MediaQuery.of(context).size.height * 0.05,
),
Text(result2,
style: TextStyle(
fontSize: 20, fontWeight: FontWeight.bold)),
SizedBox(
height: MediaQuery.of(context).size.height * 0.05,
),
Text(result3,
style: TextStyle(
fontSize: 20, fontWeight: FontWeight.bold)),
],
),
SizedBox(height: MediaQuery.of(context).size.height * 0.08),
MaterialButton(
onPressed: () async {
// Setting isLoading true to show the loader
setState(() {
isLoading = true;
});
// Awaiting for web scraping function
// to return list of strings
final response = await extractData();
// Setting the received strings to be
// displayed and making isLoading false
// to hide the loader
setState(() {
result1 = response[0];
result2 = response[1];
result3 = response[2];
isLoading = false;
});
},
child: Text(
'Scrap Data',
style: TextStyle(color: Colors.white),
),
color: Colors.green,
)
],
)),
),
);
}
}
Output:
Similar Reads
Video Streaming App in Flutter
Designing a video streaming app is a fun task to improve your development skills and design fun applications. Using Flutter, an open-source UI toolkit developed by Google, you can design for apps in iOS and Android, web and desktop all at once. Here, in this article, the reader will be introduced to
4 min read
SQLite in Flutter
SQLite is a very popular choice of local storage database. It is an SQL lightweight and serverless database engine that is highly efficient and user-friendly. Flutter, Google's UI toolkit for building natively compiled applications, doesn't come with built-in support for local data storage but provi
5 min read
Flutter - Splitting App into Widgets
Splitting an app into widgets refers to the breaking up of large widgets into smaller, more manageable widgets which can be inferred as smaller chunks of codes. The principal thing here is to understand when to split a widget into smaller ones. We are going to discuss when to split a widget and meth
5 min read
What is Widgets in Flutter?
Flutter is Google's UI toolkit for crafting beautiful, natively compiled iOS and Android apps from a single code base. To build any application we start with widgets - The building block of Flutter applications. Widgets describe what their view should look like given their current configuration and
5 min read
PageView Widget in Flutter
The PageView widget allows the user to transition between different screens in their flutter application. All you need to set it up are a PageViewController and a PageView. Constructor of PageView class:Syntax: PageView({Key key, Axis scrollDirection, bool reverse, PageController controller, ScrollP
3 min read
Making Calls in Flutter
In this online world, customer care plays a major role in the success of a company. Users are quite satisfied when they converse with the executives through calls. This has forced companies to add phone numbers to their apps for their customers to contact them easily. But dialing those numbers from
7 min read
Flutter - initState()
There are two types of widgets provided in Flutter. The Stateless WidgetThe Stateful Widget As the name suggests Stateful Widgets are made up of some 'States'. The initState() is a method that is called when an object for your stateful widget is created and inserted inside the widget tree. It is bas
4 min read
Flutter - Inherited Widget
If you are a flutter developer then you know how easy is to create Flutter UI. But when you have lots of widgets in your app and you want to pass data from one widget to another widget, this will be a pain for many developers,s especially for the newbie, this will be really hard for them to pass the
6 min read
Flutter - Using the Inspector
Flutter is based on Widgets and Widget Trees. If you are new to Flutter, imagine Widgets as structures of Data or Classes of Data. And to create a flutter application we nest these Widgets within one another. Flutter Inspector Tool Introduction to Flutter Inspector: Just like in Native android we ca
3 min read
URLs in Flutter
While surfing the net, every user will come across many buttons, text, etc., that would redirect the user to a different webpage in the same tab or in a different tab when clicked. In the same way, when a developer links a URL to a button or text in an app, the app will open the website when clicked
5 min read