Skip to content
This repository has been archived by the owner on Mar 30, 2023. It is now read-only.

Add thumbnail attribute to tweet object #889

Merged
merged 3 commits into from
Sep 5, 2020
Merged

Conversation

RtiM0
Copy link
Contributor

@RtiM0 RtiM0 commented Aug 29, 2020

Fetch the thumbnail of a video in a tweet

Fetch the video thumbnail of a tweet
@RtiM0 RtiM0 closed this Aug 29, 2020
@RtiM0 RtiM0 reopened this Aug 29, 2020
@pielco11
Copy link
Member

pielco11 commented Aug 29, 2020

Missing support for:

  • Pandas
  • SQLite
  • CSV
  • JSON
  • Elasticsearch

@RtiM0 RtiM0 mentioned this pull request Aug 29, 2020
@RtiM0
Copy link
Contributor Author

RtiM0 commented Aug 29, 2020

Missing support for:

  • Pandas
  • SQLite
  • CSV
  • JSON
  • Elasticsearch

Last commit should add support for SQLite, CSV, JSON and Elasticsearch. I have not updated Pandas because similar data fields: URLs, Photos and Videos are already missing for some reason.

@@ -265,6 +266,7 @@ def tweets(conn, Tweet, config):
",".join(Tweet.cashtags),
",".join(Tweet.urls),
",".join(Tweet.photos),
",".join(Tweet.thumbnail)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is missing a , at the end of the row

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And below at cursor.execute is missing a ,? too, since there's a new field

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Embarrassing! it wouldn't require join() either since its not a list. Good catch.

@pielco11
Copy link
Member

Thanks for the updates; about Pandas, I guess it's not a must also people can load from JSON/CSV so there's still a way

If you want to add the few missing fields, it'd be great otherwise don't worry about that. (Anyway is just a dict so not that hard)

Add missing datafields: video, photos, urls and thumbnail.
@RtiM0
Copy link
Contributor Author

RtiM0 commented Aug 30, 2020

I made an amend to last commit with that db.py fix and also updated elasticsearch/index-tweets.json.
I've also made a Update panda.py commit to add the missing data fields. (which should solve #737)

@pielco11 pielco11 merged commit 87b5b66 into twintproject:master Sep 5, 2020
darvell pushed a commit to darvell/twint that referenced this pull request Nov 16, 2020
* Add thumbnail attribute to tweet object

Fetch the video thumbnail of a tweet

* Add Thumbnail support for DB, CSV, JSON and Elasticsearch

* Update panda.py

Add missing datafields: video, photos, urls and thumbnail.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Development

Successfully merging this pull request may close these issues.

2 participants