x-scrapper

Scrape a single public X/Twitter post into clean Markdown. Opens the URL in a real browser (Selenium), waits for the tweet to render, parses it with BeautifulSoup, and saves the result.

• Main script: /home/ashusevim/x-scrapper/main.py
• Requirements: /home/ashusevim/x-scrapper/requirements.txt

• A Markdown file at Data/Data.md containing:
  • Display name and handle
  • The tweet text (best-effort)
• The same content printed to your terminal

1. Launches Chrome via Selenium and webdriver-manager.
2. Opens your tweet URL.
3. Waits (up to 15s) for article[data-testid="tweet"] to appear.
4. Parses the page source with BeautifulSoup.
5. Extracts the display name, handle, and main text.
6. Writes Markdown to Data/Data.md.

• Python 3.8+
• Google Chrome or Chromium installed and on PATH
• Internet access (for webdriver-manager to fetch the driver)

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

If you’re missing packages, the script will tell you how to install them.

python main.py <post_url>

• Waits up to 15s for the tweet to load.
• Saves to Data/Data.md and prints the same content.

Example output:

Display Name - @handle

Tweet content line 1
Tweet content line 2

• Make sure the URL is a public tweet(x does not allow scraping on private links/posts for obvious reasons)

.
├── main.py
├── requirements.txt
└── experiment
    ├── exp.py
    ├── markdown.py
    └── sample.py

• Page structure on X changes frequently; you may need to adjust selectors/slicing.
• Only extracts the main tweet text (no media, replies, or quoted tweets).

responsibly and follow X/Twitter’s Terms of Service. Only scrape content you are allowed to access.