fix(profile): ported user profile to v2 API endpoint

[himanshudabas]

Fixed user profile feature which was broken since v1 endpoints were deprecated by twitter.

• very quick method to get the timeline of a user (~3200 tweets, which includes retweets, replies).
• unlike twint -u xyz results are sorted in this method.
• fixed retweet_id, user_rt_id, user_rt and retweet_date which were earlier commented out.
• code cleanup
• refactoring
• fixes No data is being retrieved if geo config is set !!! #947

[himanshudabas]

One more thing i'd like to add.

Travis build will fail for the library even when everything works fine. that's because Travis is probably hosted on AWS/GCP and twitter doesn't provide guest token to AWS/GCP IP addresses. I have tested this on AWS with multiple IPs and none of them works.
But If we can get the guest token with a proxy then twitter doesn't check the IP and we can keep on fetching data without proxy.
So, we only need proxy for token refreshing.

I have no idea how proxies are implemented in twint, If some one could take this up it'd fix the travis issue.

[essentialols]

Thanks for the commit! Will this scrape followers and followings?

[himanshudabas]

(Sounds like only needed for initial token get, so no need to run full CI on tor)

@lmeyerov
check this out. I implemented the new solution for the above mentioned AWS problem.

What it does is:

1. try to get the initial token using requests without proxy.
2. If 1 fails, it creates a TOR Session and tries to get the initial token through that.

So the catch here is, TOR is really SLOW, but, I'am only using the TOR Session for requesting the initial token, which gets refreshed after 200 requests. So it doesn't actually degrade the performance of Twint more than a few seconds

[himanshudabas]

-- use it: https://github.com/TheDataRideAlongs/ProjectDomino/blob/3e258e151495be9012e5d75650b53679384e977a/modules/TwintPool.py#L25

I see that you are running tor on docker? (forgive me if I'm wrong, I don't have much idea about Docker)
I initially tried running TOR on my machine and modified the initial Guest Token fetch for that but later realized that it'd be better to simply provide this functionality bundled into twint because user's might not want to (or may not know how to) setup TOR on their machines.
So I found this library torpy, which does exactly this. Since we don't actually require to make a lot of Token Fetch requests, it won't affect the speed of scraping at all.

[himanshudabas]

@essentialols followers & following was never broken if I am correct.
it even works on the current version. have your tried fetching the followers & following?

[ixsure]

This is my first comment in Github, thank you so much. It's useful for my issue, despite tor isn't stable sometimes, may request timeout, still thank you so much. God bless you.

[himanshudabas]

This is my first comment in Github, thank you so much. It's useful for my issue, despite tor isn't stable sometimes, may request timeout, still thank you so much. God bless you.

Yes you are right, Tor isn't stable a lot of times. But for that I'm catching the TimeoutException and create a new session if Timeout occurs.
Even better is Tor is only used to fetch the Token, so this would hardly affect the performance of twint.

I am glad this helped you. 👍

[himanshudabas]

@pielco11 if you think there are some problems with this PR, please let me know, so that I can make necessary adjustments.

[lmeyerov]

FWIW, looking forward to trying this! cc @webcoderz

[Greatdane]

On Python 3.8 I keep getting this error relating to dataclasses;

Traceback (most recent call last):
  File "/layers/google.python.webserver/gunicorn/gunicorn/arbiter.py", line 583, in spawn_worker
    worker.init_process()
  File "/layers/google.python.webserver/gunicorn/gunicorn/workers/gthread.py", line 92, in init_process
    super().init_process()
  File "/layers/google.python.webserver/gunicorn/gunicorn/workers/base.py", line 119, in init_process
    self.load_wsgi()
  File "/layers/google.python.webserver/gunicorn/gunicorn/workers/base.py", line 144, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/layers/google.python.webserver/gunicorn/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/layers/google.python.webserver/gunicorn/gunicorn/app/wsgiapp.py", line 49, in load
    return self.load_wsgiapp()
  File "/layers/google.python.webserver/gunicorn/gunicorn/app/wsgiapp.py", line 39, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/layers/google.python.webserver/gunicorn/gunicorn/util.py", line 358, in import_app
    mod = importlib.import_module(module)
  File "/opt/python3.8/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/srv/main.py", line 1, in <module>
    from ElonTweets.wsgi import application
  File "/srv/ElonTweets/wsgi.py", line 16, in <module>
    application = get_wsgi_application()
  File "/layers/google.python.pip/pip/django/core/wsgi.py", line 12, in get_wsgi_application
    django.setup(set_prefix=False)
  File "/layers/google.python.pip/pip/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/layers/google.python.pip/pip/django/apps/registry.py", line 122, in populate
    app_config.ready()
  File "/srv/website/apps.py", line 8, in ready
    from website import updater
  File "/srv/website/updater.py", line 5, in <module>
    import twint
  File "/layers/google.python.pip/pip/twint/__init__.py", line 12, in <module>
    from .config import Config
  File "/layers/google.python.pip/pip/twint/config.py", line 5, in <module>
    class Config:
  File "/layers/google.python.pip/pip/dataclasses.py", line 958, in dataclass
    return wrap(_cls)
  File "/layers/google.python.pip/pip/dataclasses.py", line 950, in wrap
    return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen)
  File "/layers/google.python.pip/pip/dataclasses.py", line 800, in _process_class
    cls_fields = [_get_field(cls, name, type)
  File "/layers/google.python.pip/pip/dataclasses.py", line 800, in <listcomp>
    cls_fields = [_get_field(cls, name, type)
  File "/layers/google.python.pip/pip/dataclasses.py", line 659, in _get_field
    if (_is_classvar(a_type, typing)
  File "/layers/google.python.pip/pip/dataclasses.py", line 550, in _is_classvar
    return type(a_type) is typing._ClassVar
AttributeError: module 'typing' has no attribute '_ClassVar'

I assume this is in relation to #1000 and the fact the requirements now specify dataclasses, which are not needed in 3.8. Any help would be appreciated.

[himanshudabas]

On Python 3.8 I keep getting this error relating to dataclasses;

I assume this is in relation to #1000 and the fact the requirements now specify dataclasses, which are not needed in 3.8. Any help would be appreciated.

@Greatdane
hi, I have put up a patch for it, your issue should be resolved now. let me know if it works for by installing directly from my branch.
I'll create a PR for it if it resolves your issue.

[Greatdane]

Thank you! I forked it and did the exact same code as you and it worked fine, so should be good to go!

…

On Thu, Nov 12, 2020 at 4:26 PM Himanshu Dabas ***@***.***> wrote: On Python 3.8 I keep getting this error relating to dataclasses; I assume this is in relation to #1000 <#1000> and the fact the requirements now specify dataclasses, which are not needed in 3.8. Any help would be appreciated. @Greatdane <https://github.com/Greatdane> hi, I have put up a patch <https://github.com/himanshudabas/twint/tree/patch-1> for it, your issue should be resolved now. let me know if it works for by installing directly from my branch <https://github.com/himanshudabas/twint/tree/patch-1>. I'll create a PR for it if it resolves your issue. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#955 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABGXHAFIYD5JNOFBC3GAMQDSPOEZHANCNFSM4SMEIGBQ> .

[hadisfr]

Hi!
It seems that --profile-full option is removed from app in this PR

because it is no longer required, default method will do this
https://github.com/twintproject/twint/blame/master/twint/cli.py#L207

It seems that in current version of code, twint won't handle shadow-banned or formerly-private accounts. Using --profile-full still fixes the issue.
Am I wrong or missing something? 🤔 If I am right, could you please bring that option back?

[himanshudabas]

It seems that in current version of code, twint won't handle shadow-banned or formerly-private accounts. Using --profile-full still fixes the issue.

Hi,
Thanks for bringing this up. Could you please share se examples where the default or --timeline flag fails?
So that I can do a little bit of testing.

One more thing I'd like to add is, the older --profile-full feature is gonna be disfunctional anyway after 15 December, 2020.
Why?
Because it uses the mobile version (no javascript) of twitter, and twitter is gonna kill the mobile version on 15th.
So even if we bring that feature back, it'll stop working soon.

PS : feature like followers, following are gonna stop working too, because they too use the mobile version for scraping.

[hadisfr]

@himanshudabas
I was not aware of --timeline option. I tested my case with --timeline and I got 3188 tweets, same as with --profile-full option. (Although I expected about 13k tweets, but I think it's impossible to get all those tweets due to limitations of Tweeter itself.)
So it may be a good idea to just update FAQ part of README and use --timeline there. 🤔

Is there any way to rewrite code and use desktop web version or insider graphql api instead of mobile version, to bring followers, followings, etc back?
And what was the difference between --profile-full and --timeline mechanisms?
I am too busy these months, but I may help writing some codes and create PRs after that. 🤔

[himanshudabas]

I tested my case with --timeline and I got 3188 tweets, same as with --profile-full option. (Although I expected about 13k tweets, but I think it's impossible to get all those tweets due to limitations of Tweeter itself.)

It's because --timeline & --profile-full are essentially the same, except their internal working. While --profile-full uses the older mobile website to scrape tweets using BeautifulSoup, --timeline uses the newer v2 API. Limitation for both is, they can fetch a maximum of 3200 Tweets, Retweets, Replies. Because if you try to manually scroll to the bottom of a user's timeline on twitter, you can not fetch more than 3200 tweets.

So it may be a good idea to just update FAQ part of README and use --timeline there. 🤔

Yes, you are correct. Twint Wiki is past due for a long time now. But I am currently busy and unable to update it.

Is there any way to rewrite code and use desktop web version or insider graphql api instead of mobile version, to bring followers, followings, etc back?

Not that Iam aware of. I tried going through all the graphql endpoints and wasn't able to find any endpoint which can be used for this. Newer APIs require AUTh for this.
Do let me know if you find a way around this.

And what was the difference between --profile-full and --timeline mechanisms?

Technical difference is what I explained above. Non technically they are the same, then why did I rename it?
Because on Twitter it's called a timeline of the user, and it's better to keep things consistent with it.
Although I do apologise I didn't add it to the Wiki yet, perhaps you can make the changes?
Note : Only the command-line flag is changed from --profile-full to --timeline, important twint as a package in python script still works the same.

[hadisfr]

There was only one occurrence of --profile-full. As you said, there isn't any config.Timeline yet. And I didn't change --followers and --followings.
Ref #1052

[himanshudabas]

Looks good.

One other thing regarding your previous issue.

Although I expected about 13k tweets

If the user you are searching for is not shadow banned and is reachable through Twitter Advanced Search, you can use something like this,

twint -s "from:realDonaldTrump"

This will certainly fetch *almost all the tweets + replies from realDonaldTrump.

• almost, because the total number of tweets that are shown on a user's profile contains all the tweets + retweets + replies + deleted tweets. Out of these Twitter Advanced Search can only fetch tweets + replies, there is a way to fetch retweets using Advanced Search but it's been broken for years now.

[hadisfr]

The user was formerly private (I think), so I could not fetch all tweets from search and had to use the flag to scrab from the timeline.