py03 Panic when requesting to localstack instance

[ttonelli-via]

What's going on:

I am getting a panic error when trying to use this library to test against a localstack instance running inside a docker container.

Minimal steps to reproduce:

The use of the library here:

from object_store import ObjectStore

if __name__ == "__main__":
    s3 = ObjectStore("s3://mock-bucket")

    print(s3.list())

Here are the environment variables I am using, exported in the current shell that I run the above python code from:

AWS_BUCKET=mock-bucket
AWS_ACCESS_KEY_ID=mock_id
AWS_SECRET_ACCESS_KEY=mock_secret
AWS_DEFAULT_REGION=us-east-1
AWS_ENDPOINT=http://mock-bucket.s3.localhost:4566:
AWS_VIRTUAL_HOSTED_STYLE_REQUEST=true

Here is the docker compose setup I am using for the localstack instance

version: "3.9"
services:
  localstack:
    image: localstack/localstack:latest
    container_name: localstack
    ports:
      - 4566:4566
    environment:
      - SERVICES=s3
      - DEBUG=1
      - DATA_DIR=/localstack/data
      - AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
      - AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
      - AWS_DEFAULT_REGION=$AWS_DEFAULT_REGION

The error

thread '<unnamed>' panicked at 'not stream', /Users/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/object_store-0.5.6/src/aws/credential.rs:189:14
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Traceback (most recent call last):
  File "/Users/timtonelli/Development/blob-the-builder/test.py", line 6, in <module>
    print(s3.list())
          ^^^^^^^^^
  File "/Users/timtonelli/Development/blob-the-builder/.venv/lib/python3.11/site-packages/object_store/__init__.py", line 113, in list
    return super().list(prefix_)
           ^^^^^^^^^^^^^^^^^^^^^
pyo3_runtime.PanicException: not stream

The panic originates from this .expect() call from version 0.5.6 of the object store crate, which appears to be a workaround to an issue in reqwest that has since been resolved. The implementation of this same method in version 0.9.0 of the object store crate appears to have adjusted to the fix to the underlying issue in reqwest. My guess is that it's highly likely that the changes in #5 make this a non-issue.

[ttonelli-via]

@roeap I believe this issue should be resolved with some of the latest PRs. Any chance you could release a new version of this so the fix is available?

[kylebarron]

For my own purposes I have a minimal fork here, but I'm more focused on creating an ObjectStore instance for use by Rust code, rather than making IO calls from Python directly.

[ttonelli-via]

Thanks @kylebarron. In my case, I am maintaining an internal python library at my job that uses this package, so we need this version to be pip installable.

[rupurt]

I'd also love to be able to install this with pip. We've standardized on sticking with pip so can't use poetry