The process of extracting required data/information from a web page by accessing the HTML of the web page is called Web Scraping or Web Harvesting or Web Data Extraction.
This article discusses the steps involved in Web Scraping by using Flutter's html and http packages.
Step 1: Set up a new Flutter App
Create a new flutter app by running the command :
flutter create YOUR_APP_NAME
- Open the app in VS Code or Android Studio. I am using VS Code.
- Open the lib/main.dart file and clear all the default code
- Add the code for your desired widgets. I will be having an AppBar, a Column containing three Text widgets, a CircularProgressIndicator, and a MaterialButton widget.
Dart
import 'package:flutter/material.dart';
void main() => runApp(MaterialApp(
theme: ThemeData(
accentColor: Colors.green,
scaffoldBackgroundColor: Colors.green[100],
primaryColor: Colors.green,
),
home: MyApp()));
class MyApp extends StatefulWidget {
const MyApp({Key key}) : super(key: key);
@override
_MyAppState createState() => _MyAppState();
}
class _MyAppState extends State<MyApp> {
// Strings to store the extracted Article titles
String result1 = 'Result 1';
String result2 = 'Result 2';
String result3 = 'Result 3';
// boolean to show CircularProgressIndication
// while Web Scraping awaits
bool isLoading = false;
@override
Widget build(BuildContext context) {
return Scaffold(
appBar: AppBar(title: Text('GeeksForGeeks')),
body: Padding(
padding: const EdgeInsets.all(16.0),
child: Center(
child: Column(
mainAxisAlignment: MainAxisAlignment.center,
children: [
// if isLoading is true show loader
// else show Column of Texts
isLoading
? CircularProgressIndicator()
: Column(
children: [
Text(result1,
style: TextStyle(
fontSize: 20, fontWeight: FontWeight.bold)),
SizedBox(
height: MediaQuery.of(context).size.height * 0.05,
),
Text(result2,
style: TextStyle(
fontSize: 20, fontWeight: FontWeight.bold)),
SizedBox(
height: MediaQuery.of(context).size.height * 0.05,
),
Text(result3,
style: TextStyle(
fontSize: 20, fontWeight: FontWeight.bold)),
],
),
SizedBox(height: MediaQuery.of(context).size.height * 0.08),
MaterialButton(
onPressed: () {},
child: Text(
'Scrap Data',
style: TextStyle(color: Colors.white),
),
color: Colors.green,
)
],
)),
),
);
}
}
Output :
Step 2: Add the HTML and HTTP packages.
- Open the pubspec.yaml file and under the dependencies: add two lines http: ^0.12.0+4 and html: ^0.14.0+3 with proper indentation and save the file.
- Then in your terminal run the command :
flutter pub get
- Open the main.dart file and import the packages by adding these lines on the top:
import 'package:html/parser.dart' as parser;
import 'package:http/http.dart' as http;
Step 3: Adding Web Scraping functionality
- Now to extract a particular data we first need to decide on a parent class with a unique class name from the rest of the document and the hierarchy of its children, for this we need to view the HTML document of the page. We can do that by opening the website on Chrome Browser and then right-click on the required text and click on Inspect.
- From the above image, you can see that I have selected a Parent class with the class name = "articles-list" , because it has a different name from all the other classes in the document. Now If we look at the Children class we want to extract, we can see for the first article's title we need this kind of hierarchy :
"articles-list" class >> children[0] >> children[0] >> children[0]
- Similarly, for the second and third titles, it would be :
"articles-list" class >> children[1] >> children[0] >> children[0]
"articles-list" class >> children[2] >> children[0] >> children[0]
- Now that we have the Class Name and the Hierarchy we can go ahead and write the function that does Web Scraping :
Future<List<String>> extractData() async {
//Getting the response from the targeted url
final response =
await http.Client().get(Uri.parse('https://www.geeksforgeeks.org/'));
//Status Code 200 means response has been received successfully
if (response.statusCode == 200) {
//Getting the html document from the response
var document = parser.parse(response.body);
try {
//Scraping the first article title
var responseString1 = document
.getElementsByClassName('articles-list')[0]
.children[0]
.children[0]
.children[0];
print(responseString1.text.trim());
//Scraping the second article title
var responseString2 = document
.getElementsByClassName('articles-list')[0]
.children[1]
.children[0]
.children[0];
print(responseString2.text.trim());
//Scraping the third article title
var responseString3 = document
.getElementsByClassName('articles-list')[0]
.children[2]
.children[0]
.children[0];
print(responseString3.text.trim());
//Converting the extracted titles into string and returning a list of Strings
return [
responseString1.text.trim(),
responseString2.text.trim(),
responseString3.text.trim()
];
} catch (e) {
return ['', '', 'ERROR!'];
}
} else {
return ['', '', 'ERROR: ${response.statusCode}.'];
}
}- Now we will call this function in the onPressed: parameter of MaterialButton and show CircularProgressIndicator till it gets the result.
onPressed: () async {
//Setting isLoading true to show the loader
setState(() {
isLoading = true;
});
//Awaiting for web scraping function to return list of strings
final response = await extractData();
//Setting the received strings to be displayed and making isLoading false to hide the loader
setState(() {
result1 = response[0];
result2 = response[1];
result3 = response[2];
isLoading = false;
});
}- Now after all this our main.dart looks something like this :
Dart
import 'package:flutter/material.dart';
import 'package:html/parser.dart' as parser;
import 'package:http/http.dart' as http;
void main() => runApp(MaterialApp(
theme: ThemeData(
accentColor: Colors.green,
scaffoldBackgroundColor: Colors.green[100],
primaryColor: Colors.green,
),
home: MyApp()));
class MyApp extends StatefulWidget {
const MyApp({Key key}) : super(key: key);
@override
_MyAppState createState() => _MyAppState();
}
class _MyAppState extends State<MyApp> {
// Strings to store the extracted Article titles
String result1 = 'Result 1';
String result2 = 'Result 2';
String result3 = 'Result 3';
// boolean to show CircularProgressIndication
// while Web Scraping awaits
bool isLoading = false;
Future<List<String>> extractData() async {
// Getting the response from the targeted url
final response =
await http.Client().get(Uri.parse('https://www.geeksforgeeks.org/'));
// Status Code 200 means response has been received successfully
if (response.statusCode == 200) {
// Getting the html document from the response
var document = parser.parse(response.body);
try {
// Scraping the first article title
var responseString1 = document
.getElementsByClassName('articles-list')[0]
.children[0]
.children[0]
.children[0];
print(responseString1.text.trim());
// Scraping the second article title
var responseString2 = document
.getElementsByClassName('articles-list')[0]
.children[1]
.children[0]
.children[0];
print(responseString2.text.trim());
// Scraping the third article title
var responseString3 = document
.getElementsByClassName('articles-list')[0]
.children[2]
.children[0]
.children[0];
print(responseString3.text.trim());
// Converting the extracted titles into
// string and returning a list of Strings
return [
responseString1.text.trim(),
responseString2.text.trim(),
responseString3.text.trim()
];
} catch (e) {
return ['', '', 'ERROR!'];
}
} else {
return ['', '', 'ERROR: ${response.statusCode}.'];
}
}
@override
Widget build(BuildContext context) {
return Scaffold(
appBar: AppBar(title: Text('GeeksForGeeks')),
body: Padding(
padding: const EdgeInsets.all(16.0),
child: Center(
child: Column(
mainAxisAlignment: MainAxisAlignment.center,
children: [
// if isLoading is true show loader
// else show Column of Texts
isLoading
? CircularProgressIndicator()
: Column(
children: [
Text(result1,
style: TextStyle(
fontSize: 20, fontWeight: FontWeight.bold)),
SizedBox(
height: MediaQuery.of(context).size.height * 0.05,
),
Text(result2,
style: TextStyle(
fontSize: 20, fontWeight: FontWeight.bold)),
SizedBox(
height: MediaQuery.of(context).size.height * 0.05,
),
Text(result3,
style: TextStyle(
fontSize: 20, fontWeight: FontWeight.bold)),
],
),
SizedBox(height: MediaQuery.of(context).size.height * 0.08),
MaterialButton(
onPressed: () async {
// Setting isLoading true to show the loader
setState(() {
isLoading = true;
});
// Awaiting for web scraping function
// to return list of strings
final response = await extractData();
// Setting the received strings to be
// displayed and making isLoading false
// to hide the loader
setState(() {
result1 = response[0];
result2 = response[1];
result3 = response[2];
isLoading = false;
});
},
child: Text(
'Scrap Data',
style: TextStyle(color: Colors.white),
),
color: Colors.green,
)
],
)),
),
);
}
}
Output:
Similar Reads
Video Streaming App in Flutter Designing a video streaming app is a fun task to improve your development skills and design fun applications. Using Flutter, an open-source UI toolkit developed by Google, you can design for apps in iOS and Android, web and desktop all at once. Here, in this article, the reader will be introduced to
4 min read
SQLite in Flutter SQLite is a very popular choice of local storage database. It is an SQL lightweight and serverless SQL database engine that is highly efficient and user-friendly. Flutter, Google's UI toolkit for building natively compiled applications, doesn't come with built-in support for local data storage but p
5 min read
What is Widgets in Flutter? Flutter is Google's UI toolkit for crafting beautiful, natively compiled iOS and Android apps from a single code base. To build any application we start with widgets - The building block of Flutter applications. Widgets describe what their view should look like given their current configuration and
5 min read
URLs in Flutter While surfing the net, every user will come across many buttons, text, etc., that would redirect the user to a different webpage in the same tab or in a different tab when clicked. In the same way, when a developer links a URL to a button or text in an app, the app will open the website when clicked
5 min read
Mail and SMS in Flutter The world works in text. From advertisements to conversations, text is everywhere. The most popular modes of official textual communication are Mail followed by SMS. Companies use these modes to communicate not only with their employees but also with their customers. This has led app developers to i
5 min read
Flutter - Create Fortune Wheel Spin Deciding what to eat can be a daily dilemma, especially when you have a plethora of options to choose from. To make this decision-making process more exciting and interactive, in this article, we will be creating – the Lunch Spinning Wheel app.Steps to Implement Fortune Wheel in FlutterStep 1: Creat
6 min read