Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Twitter API throttling requests (429 Too Many Requests) #11

Open
ToniPortal opened this issue Jul 7, 2023 · 10 comments
Open

Twitter API throttling requests (429 Too Many Requests) #11

ToniPortal opened this issue Jul 7, 2023 · 10 comments
Labels
bug Something isn't working

Comments

@ToniPortal
Copy link

Hello, when I use certain scraper functions it gives me this error

      status: 429,
      statusText: 'Too Many Requests',

It was mostly to ask do you know when it's going to stop saying that ?
Or will it be fixed ?

@karashiiro
Copy link
Collaborator

I haven't seen that error from this before, but I'm currently working on porting over a month's worth of commits (#9, #6), so once that's done this might be fixed.

@karashiiro karashiiro added the bug Something isn't working label Jul 7, 2023
@karashiiro
Copy link
Collaborator

I just released v0.3.0 which ports over the most critical changes, let me know if that helps.

@karashiiro
Copy link
Collaborator

Actually, I just noticed the repo tests are failing with 429, looks like a regular rate limit. I'll need to set up a backoff for this.

@karashiiro
Copy link
Collaborator

Should be fixed with v0.3.1 (#12).

@karashiiro
Copy link
Collaborator

Made a minor adjustment in v0.3.2 btw, v0.3.1's backoff can get stuck in a retry loop for a while sometimes, which the latest version fixes.

@karashiiro
Copy link
Collaborator

Apparently that wasn't what was happening - the API actually just rate-limits the client for 14 minutes after a certain point. I updated the throttling mechanism to handle this, but I'm not sure if there's any real way to handle this beyond the delay.

@karashiiro karashiiro reopened this Jul 8, 2023
@karashiiro karashiiro changed the title Error when do getTweetsByUserId and other Twitter API throttling requests (429 Too Many Requests) Jul 8, 2023
@ToniPortal
Copy link
Author

ToniPortal commented Jul 9, 2023

Oh thanks for the clarification!
Really not cool to have put so many limitations on twitter, hope one day they change all these limitations they put in place...

@ImTheDeveloper
Copy link

ImTheDeveloper commented Aug 31, 2023

Is the restriction on IP address or just the tokens being used? If its the token can you regenerate the auth and carry on? Given we approximately know the limits of each endpoint (50 requests per 15 minutes on tweets/replies) we could count the usage of a token and generate before. I see in the code already it takes into account the token validity over x time so this would be now based on usage too.

Edit: Specifically in regards to guest token usage. I expect using a fully authed account would limit the consumer.

@karashiiro
Copy link
Collaborator

I just gave that a try, and it doesn't work - as soon as you try to get a new guest token you get rate-limited.

diff for reference: diff.txt

@ethos-vitalii
Copy link

I added a request timeout, but when it got relate-limited, it hung for a long time and ignored the timeout. This is how I added a timeout:

const scraper = new Scraper({
  transform: {
    request(input: RequestInfo | URL, init: RequestInit = {}) {
      init.signal = AbortSignal.timeout(REQUEST_TIMEOUT);

      return [input, init];
    },
  },
});

I also noticed that there are two types of 429. One of them hangs for ~13-14 minutes and another throws immediately.

  1. I think, the first 429 is related to the endpoint I'm hitting (getProfile), this one is being retried.
  2. Another one is for requesting a new guest token, this one fails immediately. I've noticed that if I create a new instance of Scraper, it fixes the rate-limit requesting a profile. So if we request a new guest token whenever we get 429, that should fix the rate limit.
    But if requesting the guest token is rate-limited, then it won't help for 14 minutes.

Would it be possible to refresh the guest token when the request for profile (or probably any other request) gets rate-limited instead of waiting for 13-14 minutes?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

4 participants