Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

s5cmd drops leading slashes in keys #352

Closed
gusostow opened this issue Sep 10, 2021 · 2 comments · Fixed by #459
Closed

s5cmd drops leading slashes in keys #352

gusostow opened this issue Sep 10, 2021 · 2 comments · Fixed by #459
Labels

Comments

@gusostow
Copy link

gusostow commented Sep 10, 2021

Version: v1.3.0-bb1775a

Steps to reproduce:

$ touch file.txt
$ s5cmd cat s3://bucket/file.txt
ERROR "cat s3://bucket/file.txt": NoSuchKey: The specified key does not exist. status code: 404, request id: Z...BMQB76, host id: Yy5Y9JW0D9Nvx....NMdpe9/dbE=
$ s5cmd cp file.txt s3://bucket//file.txt
cp file.txt s3://bucket/file.txt
$ s5cmd cat s3://bucket/file.txt
$

You can compare this behavior to the aws CLI:

$ aws s3 cp file.txt s3://bucket//file.txt
upload: ./file.txt to s3://bucket//filet.txt
$ aws s3 cp s3://bucket//file.txt .
download: s3://bucket//file.txt to ./file.txt

Even more problematic, if there's an existing key with a leading slash, s5cmd won't be able to address it.

@gusostow
Copy link
Author

gusostow commented Nov 4, 2021

Any thoughts on this?

@sonmezonur sonmezonur added the bug label Nov 4, 2021
@sonmezonur
Copy link
Member

As you have pointed out, s5cmd does not handle leading slashes properly. Looks like there is an issue with storage.URL package.

boraberke added a commit to boraberke/s5cmd that referenced this issue Jul 5, 2022
This commit allows url with adjacent slashes to be handled correctly.

- Disabled RestProtocolURICleaning (link) as it do not allow adjacent slashes.
- Changed path.Join method to strings.Join when joined remote urls. path.Join also cleans paths which removes additional slashes.
- Kept local urls to be joined with path.Join.
- Added test cases for copy;
    - Remote to local
    - Local to remote
    - Remote to remote
- Removed redundant trailing slashes in the keys of s3 objects in ls_test.go and sync_test.go files.

Fixes: peak#352 ,peak#449
igungor pushed a commit that referenced this issue Jul 22, 2022
This commit allows url with adjacent slashes to be handled correctly.

- Disabled RestProtocolURICleaning (link) as it do not allow adjacent slashes.
- Changed path.Join method to strings.Join when joined remote urls. path.Join also cleans paths which removes additional slashes.
- Kept local urls to be joined with path.Join.
- Added test cases for copy;
    - Remote to local
    - Local to remote
    - Remote to remote
- Removed redundant trailing slashes in the keys of s3 objects in ls_test.go and sync_test.go files.

Fixes #352
Fixes #449
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants