Search Results for "data scraper website"
Sort By:
Showing 81 open source projects for "data scraper website"
View related business solutions-
$300 Free Credits to Build on Google CloudPut your $300 in credit toward real workloads, then keep building with free monthly usage for 20+ products. No commitment and no charge until you upgrade.
-
Ship Agents FasterGemini Enterprise Agent Platform lets you rapidly build, scale, govern and optimize production-ready agents grounded in your organization's data. The platform enables developers to build custom or pre-built agents for virtually any use case. New customers get $300 in free credits.
-
1
Linkedin Scraper
A library that scrapes Linkedin for user data
Linkedin Scraper is a library that scrapes Linkedin for user data. Version 2.0.0 and before is called linkedin_user_scraper and can be installed via pip3 install --user linkedin_user_scraper. The reason is that LinkedIn has recently blocked people from viewing certain profiles without having previously signed in. So by setting scrape=False, it doesn't automatically scrape the profile, but Chrome will open the linkedin page anyways. -
2
JobFunnel
Scrape job websites into a single spreadsheet with no duplicates.
...Run funnel with your settings YAML to populate your master CSV file with jobs from available providers. JobFunnel can be easily automated to run nightly with crontab. If you have a job website you'd like to write a scraper for, you are welcome to implement it, Review the Base Scraper for implementation details. JobFunnel supports scraping jobs from the same job website across locales & domains. If you are interested in adding support, you may only need to define session headers and domain strings, Review the Base Scraper for further implementation details. -
3
CyberScraper 2077
A Powerful web scraper powered by LLM | OpenAI, Gemini & Ollama
CyberScraper 2077 is not just another web scraping tool – it's a glimpse into the future of data extraction. Born from the neon-lit streets of a cyberpunk world, this AI-powered scraper uses OpenAI, Gemini and LocalLLM Models to slice through the web's defenses, extracting the data you need with unparalleled precision and style. -
4
ScrapeGraphAI
Python scraper based on AI
Extracting content from websites and local documents using LLM. ScrapeGraphAI is a web scraping python library that uses LLM and direct graph logic to create scraping pipelines for websites and local documents (XML, HTML, JSON, Markdown, etc.). Just say which information you want to extract and the library will do it for you. -
Veeam Data Platform v13.1 - Get Your Free TrialTry Veeam Data Platform today. Experience the unified platform that's secure by design, portable by default, and proven to recover clean, fast, and anywhere.
-
5
watercrawl
AI-ready web crawler that extracts and structures website content
WaterCrawl is an open source web crawling and data extraction platform designed to transform website content into structured data suitable for machine learning and AI workflows. It enables developers and researchers to crawl web pages, extract meaningful information, and convert it into formats that are easier to process and analyze. It provides a modern crawling system that can automatically navigate links, control crawl depth, and collect content from targeted sections of a website. ... -
6
Crawl4AI
Open-source LLM Friendly Web Crawler & Scraper
Crawl4AI is a high-performance, AI‑ready web crawler tailored for LLM data ingestion and RAG pipelines. It supports adaptive crawling heuristics (stopping when enough info is gathered), structured markdown output, and high-speed parallel execution. Designed to operate at scale with optional Docker deployment and framework integrations. -
7
FinalRecon
All-in-one Python web reconnaissance tool for fast target analysis
FinalRecon is an all-in-one web reconnaissance tool written in Python that helps security professionals gather information about a target website quickly and efficiently. It combines multiple reconnaissance techniques into a single command-line utility so users do not need to run several separate tools to collect similar data. FinalRecon focuses on providing a fast overview of a web target while maintaining accuracy in the collected results. It includes modules for gathering server information, analyzing SSL certificates, performing WHOIS lookups, and crawling website resources. ... -
8
DuckDuckGo Android App
Privacy browser for Android
DuckDuckGo is an app that gives you utmost privacy when browsing online. It stops you from getting tracked and protects your personal and private information, no matter where the internet may take you. Apart from providing standard browsing functionality, DuckDuckGo blocks all hidden third-party trackers, forces sites to use an encrypted connection where available, and provides a Privacy Grade rating for each website you visit. -
9
MDCx
Movie metadata scraper and organizer for media libraries and NFO
MDCx is an open source media metadata scraping and organization tool designed to automate the process of collecting detailed information for movie files. It retrieves metadata from multiple online sources and applies it to local media collections, helping users maintain structured and well-organized libraries. MDCx can download information such as titles, cast data, artwork, and other metadata, then generate standardized NFO files compatible with media management systems. It also supports... -
Paessler: Easy to Use With Enterprise Power. Free TrialYou shouldn't have to choose between a monitoring tool that's easy to use and one that's powerful enough for a complex environment. PRTG's low-code interface lets any admin build dashboards, set alerts and monitor devices without scripting, while custom sensors and full API access are there when your team needs deeper control. One platform, no compromise. Download a free 30-day trial now.
-
10
Datasette
An open source multi-tool for exploring and publishing data
Datasette is a tool for exploring and publishing data. It helps people take data of any shape or size, analyze and explore it, and publish it as an interactive website and accompanying API. Datasette is aimed at data journalists, museum curators, archivists, local governments, scientists, researchers and anyone else who has data that they wish to share with the world. It is part of a wider ecosystem of tools and plugins dedicated to making working with structured data as productive as possible. ... -
11
linkedin2username
Generate probable usernames from LinkedIn company employee lists
...This process helps security researchers, penetration testers, and investigators perform reconnaissance by building potential username lists for further security testing or OSINT analysis. Unlike tools that rely on official APIs, linkedin2username operates as a pure web scraper and therefore does not require API keys. The script uses Selenium to automate browser interactions and perform searches within LinkedIn to gather employee data. -
12
OSCAL
Open Security Controls Assessment Language (OSCAL)
...Public contributions to this project are welcome. With this effort, we are stressing the agile development of a set of minimal formats that are generic enough to capture the breadth of data in scope (controls specifications), while also capable of ad-hoc tuning and extension to support peculiarities of both (industry or sector) standards and new control types. The OSCAL website provides an overview of the OSCAL project, including an XML and JSON schema reference, examples, and other resources. -
13
changedetection.io
The best free open source website change detection and restock service
Loved by smart shoppers, data journalists, research engineers, data scientists, security researchers, and more. From simply monitoring website pages that have a change (such as watching prices, and restocking notifications), to deep inspection such as PDF text support, JSON and XML monitoring, and extensive text triggers. Monitor out-of-stock products and get alerts when those products are back in stock, get restock alerts via Discord, Slack, email, and many other platforms. ... -
14
Skyvern
Automate browser-based workflows with LLMs and Computer Vision
...Skyvern understands how to solve CAPTCHAs to complete complicated workflows. Support for authenticating into user accounts, including support for 2FA/TOTP. Extract data from workflows in any schema of your choice including CSV or JSON. Automate procurement pipelines, breeze through government forms, and complete workflows in any language. -
15
Parsera
Lightweight library for scraping web-sites with LLMs
Scrape data from any website with only a link and column descriptions. Parsera is a tool designed to scrape web content, specifically handling poorly structured or messy websites. -
16
finvizfinance
Finviz analysis python library
finvizfinance is a package that collects financial information from FinViz website. Stock charts, fundamental & technical information, insider information and stock news. Forex charts and performance. Crypto charts and performance. Screener and Group provide data frames for comparing stocks according to different filters and trading signals. Getting information (fundament, description, outer rating, stock news, inside trader) of an individual stock. -
17
Home Assistant
Open source home automation that puts local control and privacy first
...See more examples on the official website: https://www.home-assistant.io/ -
18
GPT Discord
A robust, all-in-one GPT interface for Discord
GPTDiscord is an all-in-one AI bot that brings ChatGPT-style workflows directly into Discord. It supports long-running conversations, model settings, multimodal image understanding, and generated images. The bot can answer questions using custom documents, websites, PDFs, and Discord channel history as context. It also includes internet-connected chat, AI moderation, translation, usage tracking, and automatic retries when upstream requests fail. More advanced setups can use code... -
19
CloakBrowser
Stealth Chromium for developers and browser automation.
...The custom Chromium binary is downloaded separately and distributed under a proprietary license. Proxies and CAPTCHA-solving services are not included. Official website and pricing: https://cloakbrowser.dev/ Official SourceForge product profile: https://sourceforge.net/software/product/CloakBrowser/ -
20
Pysheeet
Python Cheat Sheet
Pysheeet is a community-driven collection of Python code snippets covering common patterns and tasks like sockets, file I/O, data structures, and more. Each snippet is concise and battle-tested, designed to save coding time and reduce boilerplate. With documentation hosted on Read the Docs and an active GitHub repo, it’s a go-to resource for Python developers. -
21
OSV.dev
Open source vulnerability DB and triage service
osv.dev (Open Source Vulnerabilities) is Google’s open source platform and API for aggregating, managing, and analyzing vulnerability data across multiple ecosystems. It powers the osv.dev website, providing a unified, queryable database of vulnerabilities that map directly to open source packages and versions. The system hosts vulnerability data for ecosystems such as PyPI, npm, Go, Maven, and Debian, among others. The platform includes a web UI, API, and a Go-based dependency scanner that checks software dependencies, container images, SBOMs (SPDX, CycloneDX), and Git repositories for known vulnerabilities. ... -
22
Dendrite
Tools to build web AI agents that can authenticate
Dendrite Python SDK is a toolkit for building web AI agents that can authenticate, interact with, and extract data from any website, facilitating web automation tasks. -
23
AutoKeras
AutoML library for deep learning
AutoKeras: An AutoML system based on Keras. It is developed by DATA Lab at Texas A&M University. The goal of AutoKeras is to make machine learning accessible to everyone. AutoKeras only support Python 3. If you followed previous steps to use virtualenv to install tensorflow, you can just activate the virtualenv. Currently, AutoKeras is only compatible with Python >= 3.7 and TensorFlow >= 2.8.0. AutoKeras supports several tasks with extremely simple interface. AutoKeras would search for the... -
24
ClatScope
OSINT reconnaissance tool for IP, domain, email, and username lookups
ClatScope is a Python-based OSINT (open source intelligence) utility designed to gather and analyze publicly available information from multiple online sources. It is primarily aimed at investigators, cybersecurity professionals, penetration testers, and researchers who need a centralized platform for reconnaissance tasks. It integrates with numerous public APIs and internet services to retrieve detailed data about IP addresses, domains, email addresses, phone numbers, usernames, and other... -
25
Index
The SOTA Open-Source Browser Agent
...The project is built to integrate easily with applications through a simple programming interface, allowing developers to embed browser automation capabilities directly into their software systems. Index can perform tasks such as navigating pages, filling forms, collecting data, and analyzing web content without requiring manual scripting for each website.
