Update api call to v5 spec in TwitchPlaylistBaseIE

[geauxlo]

Before submitting a pull request make sure you have:

• At least skimmed through adding new extractor tutorial and youtube-dl coding conventions sections
• Searched the bugtracker for similar pull requests
• Checked the code with flake8

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

• I am the original author of this code and I am willing to release it under Unlicense
• I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

• Bug fix
• Improvement
• New extractor
• New feature

---

Description of your pull request and other information

This PR closes #25553. The v5 api uses a numerical string for the channel ID instead of the channel name, and to get it you have to request /kraken/users?login=NAME instead of /kraken/channels/NAME: v5 docs.

[geauxlo]

sorry about the garbage commits

[doersino]

Is there any reason why this is not being merged? It'd be nice to have this functionality back in youtube-dl – manually going through a list of videos and downloading new ones individually is a bit tedious.

[TexanMD]

Is there anything anyone in the community can do to assist the youtube-dl team with getting this PR merged or at least feedback as to why it isn't merged so that it can either be re-worked/fixed so it can be merged?

[vxbinaca]

Please merge this for the next release if this works. My project is literally missing internet history because of the bug this (hopefully) fixes.

[doersino]

@dstftw @phihag @remitamine Sorry about tagging you all – but please consider merging this, or at least giving any amount of feedback at all.

[wlritchi]

@vxbinaca It looks like your use case is very similar to mine. With this patch, my archival scripts are working with Twitch again. Fortunately this project is quite easy to build locally, so hopefully you can deploy a temporary build while waiting for this to merge.

TL;DR for this project: this patch works for me

[Pete5x5]

If anyone is desperate to get this working you can apply the patch yourself easily:

1. In your youtube-dl folder, in the extractor folder, rename twitch.py to twitch.py.old
2. Make a new twitch.py in the same location, and copy the contents of this patch's new twitch.py (found at https://raw.githubusercontent.com/geauxlo/youtube-dl/33afd662d96f82f7a5af4ac89513666f2a078a84/youtube_dl/extractor/twitch.py)
3. Instead of running "youtube-dl your_flags_here" run "python -m youtube_dl your_flags_here" (notice in the new command youtube-dl uses an underscore, not a dash)

Note: this patch works correctly with the "old" URL for all twitch vods (i.e. https://www.twitch.tv/CHANNELNAME/videos/all)

Hope this helps @vxbinaca @doersino and others temporarily.

[vxbinaca]

Or ya know, this could be merged up so I don't have to maintain a separate youtube-dl. I'm thinking of getting ahold of the maintainers and politely requesting this be merged this is a ridiculous bug to literally let sit. It's holding back testing we want to do to fix our project that relies on youtube-dl because we can't properly test a YTDL flag without Twitch channels working.

Edit 2: I don't feel like re-pointing library calls to the administrative nightmare that would be a separate youtube-dl package just to get Twitch channel support. I'm forced to wait for this to be merged in along with the 600+ other PR's this project has waiting. But in the meantime internet history is being lost because my scripts rely on Twitch channels as targets w/ crontabs to automate ingestion to Archive.org. Twitch channel extraction is literally broken right now, therefor unless I manually enuumerte a channels videos they don't get ingested.

So for the last 20 days internet history hasn't been getting saved, because the bug this PR fixes hasn't been fixed. The upgrade we want to test that this bug is preventing would also likely fix Periscope ingestion.

[doersino]

You could always just temporarily apply this patch as outlined above, then revert to mainline youtube-dl once this pull request has been merged. That's what I've done on my end.

It's a bit of a hassle, of course, but it'll get your internet history preservation project back up and running faster than demanding that the (unpaid) maintainers merge this patch and cut a release immediately.

[vxbinaca]

You could always just temporarily apply this patch as outlined above, then revert to mainline youtube-dl once this pull request has been merged. That's what I've done on my end.

It's a bit of a hassle, of course, but it'll get your internet history preservation project back up and running faster than demanding that the (unpaid) maintainers merge this patch and cut a release immediately.

It takes two mouse clicks to merge this fix. The users of my script are much more diverse than just me. Probably hundreds by now. I'll just find all of them, somehow, and tell them if they need to save Twitch just do this complicated series of steps to get it to work - oh and be sure to keep up to date with the rest of the extractors and code changes too until two mouse clicks happen and it's merged then you can go back to the vanilla install from pypi. Totally that will work just fine.

Or, ya know, two mouse clicks.

[doersino]

There was no indication in your previous comments that you weren't the only user of your script – that sure complicates things! Mind you, I hope for a swift merge as well.

[TexanMD]

I can confirm this PR does fix all of the issues I was running in to.

I'm building a custom youtube-dl binary for my projects now. I'd still prefer this be merged, but we're at 3 weeks with no feedback from the project about if this PR has any issues or needs any change, while other issues have been "managed" (updated/closed/triaged) by project contributors.

If you don't want to go the python module route that @Pete5x5 provided above, this is what I had to do to build a drop in replacement for /usr/local/bin/youtube-dl

git clone https://github.com/ytdl-org/youtube-dl.git
cd youtube-dl/
sudo apt-get install make python pandoc zip
cd youtube_dl/extractor
rm twitch.py
wget https://raw.githubusercontent.com/geauxlo/youtube-dl/33afd662d96f82f7a5af4ac89513666f2a078a84/youtube_dl/extractor/twitch.py
cd ../..
make
chmod 755 ./youtube-dl
sudo cp ./youtube-dl /usr/local/bin/youtube-dl

Any errors I got were due to a dependency being missing which is why I installed the 4 packages above.

It probably won't be much help for you @vxbinaca but it's at least a better solution for some use cases than using a patched version of the python module.

[TexanMD]

@dstftw Sorry for the ping, but would it be possible to get any feedback on this PR/issue please? I think there are multiple people willing to update it and/or address feedback if needed and multiple people have been testing this patch without reporting any issues.

[vxbinaca]

Can I point ytdl at whatever that ID number is and would it pick up the channel or is this problem a of the URL structure changing?

[ahmedtds]

Well, there's also this small bug regarding URL schemes that wasn't fixed for more than a year: #24263 #21811

I wonder how you guys were handling these new twitch URLs with "videos?filter=archives" when downloading all videos from a channel even before this "channel ID" bug? Because that's what someone gets from URL bar and youtube-dl doesn't recognize them for more than a year. No way to get URLs like "videos/past-broadcasts" anymore.

[Crusage]

Seriously, its been like a month since the last release. Can they soon merge this?

[vxbinaca]

Show of reacts: Who is interested in a 1:1 fork of youtube-dl, with update mirrors going into the future, I change none of the code myself (because I can't code), but I merge as many of the PRs in this repo as are viable and call it "video-dl" (if it's not taken in pypi) to avoid trademark infringement? I could spend a few hours banging out improvements (see: clicking the mouse twice), and just push a version of video-dl (or whatever I'll call it), and this would be the first code I merge. Looking into merging xvideos fixes among others after that.

I'm serious. I'm a get things done kind of guy. Dstfw wants us to pay attention to submitted issues and PRs, and it seems I'm among the small number of people who actually do this unlike the maintainers.

So react away boys, I'll log into Github in a few days and see whos interested.

Edit:

Oh wow, dstfw is tagging a probably bad PR as "do-not-merge" but his desired solution has been in discussions since 2016.. We've burned half-way through 2020 as of this comment. Wow.

[wlritchi]

I don't want to drag the maintainer here - this is a project that only lives because of people's unpaid contributions in their spare time - but I do get the sense that one of two things is happening:

1. The maintainer doesn't have the resources to effectively review the sheer number of PRs being submitted. When a module is completely broken, there's a simple PR to fix it, user testimonials are saying the PR works, and it's still not being looked at? That's a sign that things are getting pretty bad and the project is at risk of stalling out, and there needs to be more maintainers to help keep things moving. I don't have an answer for where those maintainers come from, but they are needed. In this circumstance, I'd much prefer to see people joining/stepping up in the original project, rather than forking. But, absent a response from the current maintainer, forking might be the only path forward in the short term.
2. The maintainer is uninterested in keeping support for some subset of modules. I'm still seeing PRs get merged for YouTube - not many, but some. That could be a symptom of point 1 combined with an attempt to prioritize the most popular use case. But I'm concerned that there's a chance it indicates a lack of interest in keeping download functionality working for sites like Twitch. I hope this isn't the problem. But if it is, I really don't see a choice but to fork.

[TexanMD]

@dstftw / @remitamine Any chance one of you could lend a hand with getting this PR merged? Do you need us to update/fix/rework anything?

[Crusage]

I have been using this to download a few videos, but just recently got bugs.
[download] 51.7% of ~8.86GiB at 4.00MiB/s ETA 17:03 ERROR: unable to download video data: <urlopen error [Errno 11001] getaddrinfo failed> [download] Downloading video 67 of 895 [twitch:vod] 350474046: Downloading vod info JSON ERROR: Unable to download JSON metadata: <urlopen error [Errno 11001] getaddrinfo failed> (caused by URLError(gaierror(11001, 'getaddrinfo failed'))) [download] Downloading video 68 of 895 [twitch:vod] 350474045: Downloading vod info JSON ERROR: Unable to download JSON metadata: <urlopen error [Errno 11001] getaddrinfo failed> (caused by URLError(gaierror(11001, 'getaddrinfo failed'))) [download] Downloading video 69 of 895 [twitch:vod] 350474044: Downloading vod info JSON ERROR: Unable to download JSON metadata: <urlopen error [Errno 11001] getaddrinfo failed> (caused by URLError(gaierror(11001, 'getaddrinfo failed'))) [download] Downloading video 70 of 895 [twitch:vod] 350474043: Downloading vod info JSON ERROR: Unable to download JSON metadata: <urlopen error [Errno 11001] getaddrinfo failed> (caused by URLError(gaierror(11001, 'getaddrinfo failed'))) [download] Downloading video 71 of 895 [twitch:vod] 350460928: Downloading vod info JSON ERROR: Unable to download JSON metadata: <urlopen error [Errno 11001] getaddrinfo failed> (caused by URLError(gaierror(11001, 'getaddrinfo failed'))) [download] Downloading video 72 of 895 [twitch:vod] 350460381: Downloading vod info JSON ERROR: Unable to download JSON metadata: <urlopen error [Errno 11001] getaddrinfo failed> (caused by URLError(gaierror(11001, 'getaddrinfo failed')))
And so on
Anyone have any idea what this mean?
The command I used: py -m youtube_dl -o "%(upload_date)s_%(id)s %(title)s.%(ext)s" -f best -ci --batch-file=links.txt
links.txt: https://www.twitch.tv/notcrusage/videos/all

[ergosteur]

@notcrusage looks like you are having a network/DNS issue unrelated to the issue here.

Confirm that this pull request still solves the issue on my end. I've resorted to writing a script that replaces the twitch.py extractor with the one from this PR every time youtube-dl updates on my server.

[vxbinaca]

I have been using this to download a few videos, but just recently got bugs.
[download] 51.7% of ~8.86GiB at 4.00MiB/s ETA 17:03 ERROR: unable to download video data: <urlopen error [Errno 11001] getaddrinfo failed> [download] Downloading video 67 of 895 [twitch:vod] 350474046: Downloading vod info JSON ERROR: Unable to download JSON metadata: <urlopen error [Errno 11001] getaddrinfo failed> (caused by URLError(gaierror(11001, 'getaddrinfo failed'))) [download] Downloading video 68 of 895 [twitch:vod] 350474045: Downloading vod info JSON ERROR: Unable to download JSON metadata: <urlopen error [Errno 11001] getaddrinfo failed> (caused by URLError(gaierror(11001, 'getaddrinfo failed'))) [download] Downloading video 69 of 895 [twitch:vod] 350474044: Downloading vod info JSON ERROR: Unable to download JSON metadata: <urlopen error [Errno 11001] getaddrinfo failed> (caused by URLError(gaierror(11001, 'getaddrinfo failed'))) [download] Downloading video 70 of 895 [twitch:vod] 350474043: Downloading vod info JSON ERROR: Unable to download JSON metadata: <urlopen error [Errno 11001] getaddrinfo failed> (caused by URLError(gaierror(11001, 'getaddrinfo failed'))) [download] Downloading video 71 of 895 [twitch:vod] 350460928: Downloading vod info JSON ERROR: Unable to download JSON metadata: <urlopen error [Errno 11001] getaddrinfo failed> (caused by URLError(gaierror(11001, 'getaddrinfo failed'))) [download] Downloading video 72 of 895 [twitch:vod] 350460381: Downloading vod info JSON ERROR: Unable to download JSON metadata: <urlopen error [Errno 11001] getaddrinfo failed> (caused by URLError(gaierror(11001, 'getaddrinfo failed')))
And so on
Anyone have any idea what this mean?
The command I used: py -m youtube_dl -o "%(upload_date)s_%(id)s %(title)s.%(ext)s" -f best -ci --batch-file=links.txt
links.txt: https://www.twitch.tv/notcrusage/videos/all

Install nscd to cache nameserver lookups and try that again.

[doersino]

@dstftw @phihag @remitamine Any progress on this?