e2e/util_test: enable tests to work using external endpoint #506

boraberke · 2022-09-02T11:30:30Z

With this PR, e2e tests can now be tested using external endpoint.

To do so, following environment variables need to be set:

S5CMD_ACCESS_KEY_ID
S5CMD_SECRET_ACCESS_KEY
S5CMD_ENDPOINT_URL
S5CMD_IS_VIRTUAL_HOST
S5CMD_REGION
S5CMD_I_KNOW_WHAT_IM_DOING

s5cmd will use S5CMD_ENDPOINT_URL with credentials S5CMD_ACCESS_KEY_ID, S5CMD_SECRET_ACCESS_KEY and S5CMD_REGION to run all e2e tests instead of local gofakess3 server. Additionally, if the endpoint tested uses virtual host style S5CMD_IS_VIRTUAL_HOST needs to be a truthy value and control env variable S5CMD_I_KNOW_WHAT_IM_DOING needs to be set to 1. If any of those environment variables is empty, tests will fallback to use gofakess3 instead.

Additionally, tests are refactored to have the following characteristics:

Every test cleanup the bucket(s) after the test is done.
Each created test bucket has a unique bucket name.

use testing.cleanup instead of passing cleanup functions.

Tests are now can be run using an external URL instead of local s3-server. With this, it is technically possible to observe if test files will pass in any other s3-storage such as gcs, wasabi, amazon etc.

add random suffix to all bucket names.

to be continued...

in v0.7.0, SlowDown was a retryable error. https://github.com/peak/s5cmd/blob/v0.7.0/core/error.go#L49 Add that again as retryable because with peak#506, one might get slowdown errors while testing with other endpoint urls.

in [v0.7.0](https://github.com/peak/s5cmd/blob/v0.7.0/core/error.go#L49), SlowDown was a retryable error. Add that again as retryable because with #506, one might get slowdown errors while testing with other endpoint urls.

@igungor

Realized TestCopySingleFileToS3 was a faulty test. gofakess3 never reads/assigns content-type. By fixing that with following pr on @igungor's fork, it works as expected. p.s.: this commit will fail TestCopySingleFileToS3 as it depends on this change: igungor/gofakes3#7

igungor · 2022-09-06T15:59:53Z

@boraberke igungor/gofakes3#7 is merged. v0.0.13 release is ready.

igungor

This change will be very useful for us to be confident about the tool for certain s3 compatible services. Thank you.

e2e/cp_test.go

e2e/ls_test.go

e2e/run_test.go

e2e/util_test.go

there was no way of understanding if a specific endpoint uses virtual host style or path style when testing. Add another environment variable to get that information.

e2e/cat_test.go

e2e/rm_test.go

e2e/run_test.go

e2e/sync_test.go

storage/s3.go

seruman

🚀 Thank you!

There's a merge conflict though.

boraberke · 2022-09-15T07:27:11Z

🚀 Thank you!

There's a merge conflict though.

Just resolved the merge conflict 🏁

seruman · 2022-09-16T07:10:01Z

For the records, I did try to run tests with Backblaze B2 and there are a couple things to consider -and there might be more-;

B2 limits maximum buckets per account to 100.
Versioning is enabled by default. To delete a bucket, all versions of the files had to be deleted.
Looks like it does not keep spaces between media-type and parameters in Content-Type header;
- Tests sets and expects text/html; charset=utf-8 but B2 responds with text/html;charset=utf-8.

Do not think these are blockers for this PR. We might update the tests as we start add more compatible services.

boraberke · 2022-09-16T07:12:33Z

For the records, I did try to run tests with Backblaze B2 and there are a couple things to consider -and there might be more-;

B2 limits maximum buckets per account to 100.

Versioning is enabled by default. To delete a bucket, all versions of the files had to be deleted.

Looks like it does not keep spaces between media-type and parameters in Content-Type header;

Tests sets and expects text/html; charset=utf-8 but B2 responds with text/html;charset=utf-8.

Do not think these are blockers for this PR. We might update the tests as we start add more compatible services.

Thanks for testing Backblaze and pointing these out! As you said, after this PR, additional compatible services can be added and tests can be updated accordingly 👍

boraberke and others added 15 commits August 26, 2022 13:30

refactor cleanup behavior

dfdc8a3

use testing.cleanup instead of passing cleanup functions.

e2e: refactor cleanup behavior using t.Cleanup

91c9fea

Allow tests to use external endpoint url from env

e120ab1

Tests are now can be run using an external URL instead of local s3-server. With this, it is technically possible to observe if test files will pass in any other s3-storage such as gcs, wasabi, amazon etc.

add cleanup function to remove buckets after test

c403925

uniquify bucket names for all tests

5ab6264

add random suffix to all bucket names.

Merge branch 'master' into change-test-endpoint-read-from-env

8dfa8d9

add unit test for addRandomSuffixTo func

c5adc24

add unit test to addRandomSuffixTo func

8b129db

test github actions

32e867d

test github actions with env

ae372aa

fix endpoint for env tests

08f0f38

change test structures to use unique bucket names

726f5f4

to be continued...

refactor all tests to use unique bucket names

9837b50

Merge branch 'peak:master' into change-test-endpoint-read-from-env

4301747

add test to s3BucketFromTestNameWithPrefix

6708fcc

boraberke changed the title ~~e2e/util_test: enable tests to work in external endpoint~~ e2e/util_test: enable tests to work using external endpoint Sep 2, 2022

refactor TestCatS3ObjectFail

480aaa1

boraberke mentioned this pull request Sep 2, 2022

storage/s3: add SlowDown as retryable #507

Merged

add slowDownRetryer

208a81d

boraberke and others added 6 commits September 2, 2022 19:08

Merge branch 'peak:master' into change-test-endpoint-read-from-env

1775bee

fix some of the failed tests when used with gcs

ed1ea21

fix most of the failed tests on gcs

8e85e41

conclude path escape issues for with space char

238547f

fix TestAppProxy in gcs

6c6eb98

boraberke marked this pull request as ready for review September 6, 2022 15:15

boraberke requested a review from a team as a code owner September 6, 2022 15:15

boraberke removed the request for review from a team September 6, 2022 15:15

boraberke requested review from igungor and seruman September 6, 2022 15:15

boraberke mentioned this pull request Sep 6, 2022

.github/workflows: Add external endpoint test before release #508

Merged

boraberke added 2 commits September 7, 2022 09:27

vendor: update gofakes3 to v0.0.13

ed663fc

change chunk size to 1 when gcs is used

5005910

igungor reviewed Sep 9, 2022

View reviewed changes

boraberke added 2 commits September 9, 2022 15:38

add S5CMD_IS_VIRTUAL_HOST

7527fa6

there was no way of understanding if a specific endpoint uses virtual host style or path style when testing. Add another environment variable to get that information.

fix reviews

9e0fd43

seruman requested changes Sep 13, 2022

View reviewed changes

boraberke and others added 2 commits September 13, 2022 15:54

fix reviews

9dc6e80

Merge branch 'peak:master' into change-test-endpoint-read-from-env

5450e1e

boraberke requested a review from seruman September 13, 2022 13:38

seruman previously approved these changes Sep 15, 2022

View reviewed changes

Merge branch 'master' into change-test-endpoint-read-from-env

3bc7a91

boraberke dismissed seruman’s stale review via 3bc7a91 September 15, 2022 07:23

seruman approved these changes Sep 16, 2022

View reviewed changes

igungor approved these changes Sep 16, 2022

View reviewed changes

igungor merged commit d949fe0 into peak:master Sep 16, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

e2e/util_test: enable tests to work using external endpoint #506

e2e/util_test: enable tests to work using external endpoint #506

boraberke commented Sep 2, 2022 •

edited

Loading

igungor commented Sep 6, 2022

igungor left a comment

seruman left a comment

boraberke commented Sep 15, 2022

seruman commented Sep 16, 2022

boraberke commented Sep 16, 2022

e2e/util_test: enable tests to work using external endpoint #506

e2e/util_test: enable tests to work using external endpoint #506

Conversation

boraberke commented Sep 2, 2022 • edited Loading

igungor commented Sep 6, 2022

igungor left a comment

Choose a reason for hiding this comment

seruman left a comment

Choose a reason for hiding this comment

boraberke commented Sep 15, 2022

seruman commented Sep 16, 2022

boraberke commented Sep 16, 2022

boraberke commented Sep 2, 2022 •

edited

Loading