Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add storage_options to create_table? #1198

Open
raybellwaves opened this issue Nov 21, 2020 · 1 comment
Open

add storage_options to create_table? #1198

raybellwaves opened this issue Nov 21, 2020 · 1 comment

Comments

@raybellwaves
Copy link

I was testing this in dask-sql (dask-contrib/dask-sql#84 (comment)) and wanted to test this here.

It seems storage_options isn't an arg for create_table. The docs point to creating a dask.dataframe first: https://docs.blazingdb.com/docs/dask

$ conda create -n test_env python=3.7 -y
$ conda activate test_env
$ conda install -c blazingsql-nightly -c rapidsai-nightly -c nvidia -c conda-forge -c defaults blazingsql python=3.7 cudatoolkit=10.2 -y
$ python

from blazingsql import BlazingContext
bc = BlazingContext()

storage_options = {'account_name': 'azureopendatastorage'}
bc.create_table("taxi", "az://nyctlc/green/puYear=2019/puMonth=1/*.parquet", storage_options=storage_options)
>>> bc.create_table("taxi", "az://nyctlc/green/puYear=2019/puMonth=1/*.parquet", storage_options=storage_options)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ray/local/bin/anaconda3/envs/test_env/lib/python3.7/site-packages/pyblazing/apiv2/context.py", line 1833, in create_table
    kwargs_validation(kwargs, "create_table")
  File "/home/ray/local/bin/anaconda3/envs/test_env/lib/python3.7/site-packages/pyblazing/apiv2/context.py", line 881, in kwargs_validation
    + params_info
Exception: ERROR: The parameter 'storage_options' does not exists. Please make sure you are using the correct parameter:
To get the correct parameters, check:  https://docs.blazingdb.com/docs/create_table
@raybellwaves raybellwaves added the ? - Needs Triage needs team to review and classify label Nov 21, 2020
@wmalpica
Copy link
Contributor

Hello @raybellwaves . storage_options is not a parameter for create_table.

We currently do not support Azure Blob Store, which is why we recommend using dask in our documentation as a temporary workaround. We do plan on implementing a storage plugin for Azure in the next couple months.
We do support s3, HDFS and Google Compute Storage (https://docs.blazingdb.com/docs/connecting-data-sources)

@roaramburu roaramburu removed the ? - Needs Triage needs team to review and classify label Nov 25, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants