[mixcloud] Add support for new frontend #14132

ishitatsuyuki · 2017-09-06T02:35:25Z

Before submitting a pull request make sure you have:

At least skimmed through adding new extractor tutorial and youtube-dl coding conventions sections
Searched the bugtracker for similar pull requests

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

I am the original author of this code and I am willing to release it under Unlicense

What is the purpose of your pull request?

Bug fix
Improvement

Description of your pull request and other information

The decryption part is refactored to use iterators, and therefore a compat function was added.

The code is mostly self explanatory.

ishitatsuyuki · 2017-09-11T07:04:34Z

@dstftw Friendly ping, is there anything that I should elaborate?

dstftw · 2017-09-12T16:41:05Z

youtube_dl/extractor/mixcloud.py

+                {
+                    'format_id': 'dash',
+                    'url': self._decrypt_xor_cipher(key, base64.b64decode(info_json['streamInfo']['dashUrl']))
+                }


151-162 code duplication.

Missing format keys should not breaks extraction.

HLS and DASH should be extracted with appropriate _extract_* methods.

HLS and DASH extraction should be non fatal.

dstftw · 2017-09-12T16:41:46Z

youtube_dl/extractor/mixcloud.py

+            uploader = info_json['owner']['displayName']
+            uploader_id = info_json['owner']['username']
+            description = info_json['description']
+            view_count = info_json['plays']


None of the optional fields should break extraction if missing.

dstftw · 2017-09-12T16:42:37Z

youtube_dl/extractor/mixcloud.py

+            for item in full_info_json:
+                item_data = item.get("cloudcast", {}) \
+                    .get("data", {}) \
+                    .get("cloudcastLookup", {})


try_get, single quotes.

kayb94 · 2017-09-12T22:49:01Z

Not sure if that is intented or not:
I just tried out your branch, ishitatsuyuki, with the media from #14088 and had to specify format as normal to get a listenable audio.


 > python -m youtube_dl https://www.mixcloud.com/aemidesign/mdma/ --list-formats[mixcloud] aemidesign-mdma: Downloading webpage
[mixcloud] aemidesign-mdma: Downloading webpage
[info] Available formats for aemidesign-mdma:
format code  extension  resolution note
normal       m4a        unknown    
hls          m3u8       unknown    
dash         mpd        unknown    (best)
 > python -m youtube_dl https://www.mixcloud.com/aemidesign/mdma/ -f normal
[mixcloud] aemidesign-mdma: Downloading webpage
[mixcloud] aemidesign-mdma: Downloading webpage
[download] Destination: MDMA-aemidesign-mdma.m4a
[download]  14.4% of 32.57MiB at  1.25MiB/s ETA 00:22^C
ERROR: Interrupted by user

The other file is just 4K.

 > du -h MDMA-aemidesign-mdma.mpd 
4.0K	MDMA-aemidesign-mdma.mpd

Feel free to ignore, if I got something wrong here.

ishitatsuyuki · 2017-09-12T22:57:01Z

I'm using DASH with mpv without issues. Their server seems to terminate the connection weirdly though.

ishitatsuyuki · 2017-09-13T13:25:20Z

@kayb94 thanks for pointing out. That was indeed my mistake.

@dstftw I hope I addressed all your comments.

kayb94 · 2017-09-13T19:57:22Z

Glad to be able to help!
Regards

There seems to be some "special" URLs that are not served from their CDN.

ishitatsuyuki · 2017-09-15T13:11:13Z

@dstftw friendly ping, I think this is ready for another review!

dstftw · 2017-09-17T15:55:53Z

youtube_dl/extractor/mixcloud.py

+
+            dash_encrypted = try_get(info_json, lambda x: x['streamInfo']['dashUrl'])
+            if dash_encrypted is not None:
+                dash_url = self._decrypt_xor_cipher(key, base64.b64decode(dash_encrypted))


157, 160-162, 165-167 code duplication.

What would you suggest then? The duplication is minimal.

Extracting duplicate code into a function obviously.

dstftw · 2017-09-17T15:56:16Z

youtube_dl/extractor/mixcloud.py

+                r'<script id="relay-data" type="text/x-mixcloud">([^<]+)</script>', webpage, 'play info'), 'play info')
+            for item in full_info_json:
+                item_data = try_get(item, lambda x: x['cloudcast']['data']['cloudcastLookup'])
+                if try_get(item_data, lambda x: x['streamInfo']['url']) not in ['', None]:


if try_get(item_data, lambda x: x['streamInfo']['url']) is enough.

ishitatsuyuki · 2017-09-20T07:47:24Z

@dstftw another review ping. If you have solutions to code duplication, please tell me how to do so.

ishitatsuyuki added 2 commits September 6, 2017 11:34

[compat] Add compat_zip

20dbcfc

[mixcloud] Add handling for new frontend

d5a11b4

dstftw requested changes Sep 12, 2017

View reviewed changes

dstftw added the pending-fixes label Sep 12, 2017

kayb94 mentioned this pull request Sep 12, 2017

[mixcloud] can't download periodically (very often) #14088

Closed

8 tasks

Address review issues

7ab078e

ishitatsuyuki added 2 commits September 15, 2017 22:05

Add fallback known-plaintext

6dd969a

There seems to be some "special" URLs that are not served from their CDN.

Ensure data variables are assigned

b507277

dstftw requested changes Sep 17, 2017

View reviewed changes

Address review issue (try 2)

a26f294

Reduce code duplication

79ec7fb

dstftw closed this in 095774e Sep 22, 2017

dstftw added a commit that referenced this pull request Sep 23, 2017

Credit @ishitatsuyuki for mixcloud fix (#14132)

2f3933a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[mixcloud] Add support for new frontend #14132

[mixcloud] Add support for new frontend #14132

ishitatsuyuki commented Sep 6, 2017

ishitatsuyuki commented Sep 11, 2017

dstftw Sep 12, 2017

dstftw Sep 12, 2017

dstftw Sep 12, 2017

kayb94 commented Sep 12, 2017 •

edited

Loading

ishitatsuyuki commented Sep 12, 2017

ishitatsuyuki commented Sep 13, 2017

kayb94 commented Sep 13, 2017

ishitatsuyuki commented Sep 15, 2017

dstftw Sep 17, 2017

ishitatsuyuki Sep 17, 2017

dstftw Sep 20, 2017

dstftw Sep 17, 2017

ishitatsuyuki commented Sep 20, 2017

[mixcloud] Add support for new frontend #14132

[mixcloud] Add support for new frontend #14132

Conversation

ishitatsuyuki commented Sep 6, 2017

Before submitting a pull request make sure you have:

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

What is the purpose of your pull request?

Description of your pull request and other information

ishitatsuyuki commented Sep 11, 2017

dstftw Sep 12, 2017

Choose a reason for hiding this comment

dstftw Sep 12, 2017

Choose a reason for hiding this comment

dstftw Sep 12, 2017

Choose a reason for hiding this comment

kayb94 commented Sep 12, 2017 • edited Loading

ishitatsuyuki commented Sep 12, 2017

ishitatsuyuki commented Sep 13, 2017

kayb94 commented Sep 13, 2017

ishitatsuyuki commented Sep 15, 2017

dstftw Sep 17, 2017

Choose a reason for hiding this comment

ishitatsuyuki Sep 17, 2017

Choose a reason for hiding this comment

dstftw Sep 20, 2017

Choose a reason for hiding this comment

dstftw Sep 17, 2017

Choose a reason for hiding this comment

ishitatsuyuki commented Sep 20, 2017

kayb94 commented Sep 12, 2017 •

edited

Loading