urllib3 version 2 incompatibility

[ceesem]

We are starting to use some code in caveclient that requires urllib3 version 2, but with urllib3, e.g. version 2.2.3, import cloudvolume raises the error:

cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_

This seems to be a breaking change in urllib3 between version 1.x.x and 2.x.x, but that version req is not enforced by requirements.txt from cloudvolume. It would be nice to be compatible with recent versions of urllib3.

[william-silversmith]

It looks like this warning comes from the use of botocore:

https://urllib3.readthedocs.io/en/stable/v2-migration-guide.html

ImportError: cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_'

This likely happens because you’re using botocore which [does not support urllib3 2.0 yet](https://github.com/boto/botocore/issues/2921). The good news is that botocore explicitly declares in its dependencies that it only supports urllib3<2. Make sure to use a recent pip. That way, pip will install urllib3 1.26.x until botocore starts supporting urllib3 2.0.

If you’re deploying to an AWS environment such as Lambda or a host using Amazon Linux 2, you’ll need to explicitly pin to urllib3<2 in your project to ensure urllib3 2.0 isn’t brought into your environment. Otherwise, this may result in unintended side effects with the default boto3 installation.

botocore is required for the use of Amazon S3 or S3 emulating endpoints like the Cloudian Hyperstore (matrix://) and Tigerdata (tigerdata://).

botocore has the following pin on <2 to support Amazon Linux 2 and AWS Lambda.

https://github.com/boto/botocore/blob/develop/setup.py#L25-L33

Does the latest CaveClient need urllib>=2? What kind of resolution would you like to see? There is significant use of S3 and S3 emulating endpoints by CloudVolume and CloudFiles users.

[ceesem]

Yup, that totally checks out.

The most recent version of caveclient added some increased retry configuration that used on an import that does not work in urllib3 1.x.x. I'm honestly not sure what the right answer for us is — it feels pretty negligent of the botocore package to not support a well-telegraphed version change after well more than a year, but it's also critical functionality as you point out.

[william-silversmith]

Hmm have you considered using https://github.com/jd/tenacity for the retry function? That might get you out of the jam.