Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a new smithy-modeled rules based endpoint resolution. #2214

Merged
merged 58 commits into from
Jul 31, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
fe4c3b3
Add aws standard lib functions (#2057)
isaiahvita Mar 20, 2023
c40b4dd
remove error collector for now (#2073)
isaiahvita Mar 29, 2023
f4381d4
add partition generator, and run gofmt (#2076)
isaiahvita Apr 5, 2023
f79579a
Add aws library functions for endpoint rules engine and fix partition…
isaiahvita May 5, 2023
0eb37e5
Add tests codegen and regen s3 (#2126)
isaiahvita May 17, 2023
72ed6c0
Add client config adapters (#2146)
isaiahvita Jun 12, 2023
98f1519
Add S3 Model Preprocessing for EP 2.0 (#2153)
Jun 21, 2023
50ceb26
Add BuiltIn generation (#2152)
isaiahvita Jun 23, 2023
7cb6d19
Add endpoint-based auth scheme resolution (#2158)
isaiahvita Jun 29, 2023
5a56b8a
Remove httpLabel trait from Bucket members (#2173)
isaiahvita Jul 7, 2023
b44fbbf
convert s3 handle_200_error_test into an actual unit test with a fake…
lucix-aws Jul 7, 2023
98caa35
Add functionality for non-SSL endpoints for backwards compatibility (…
isaiahvita Jul 7, 2023
f0fe836
fix borked go mod files
lucix-aws Jul 7, 2023
0839634
Fix Properties Auth Scheme Path (#2177)
Jul 7, 2023
260ce31
fix empty string signing name and region for sigv4 on ctx (#2178)
lucix-aws Jul 7, 2023
1229116
Check S3 signer customization insensitively (#2179)
Jul 7, 2023
151def9
Fix EP20 url encoding issues with S3 (#2181)
isaiahvita Jul 10, 2023
9d7de33
serde buckets into immutable hostname endpts (#2182)
lucix-aws Jul 11, 2023
774213d
adjust some expected errs
lucix-aws Jul 11, 2023
bd74619
remove incorrectly-applied {RequestRoute} filter
lucix-aws Jul 11, 2023
6921c80
fix borked serializers from last commit
lucix-aws Jul 11, 2023
50e8ce8
use httpbinding to escape instead
lucix-aws Jul 11, 2023
6c91a64
Check hostname immutable in EndpointDisableHttps and make it service …
isaiahvita Jul 11, 2023
4a484db
add back unintentionally deleted test cases (#2185)
isaiahvita Jul 11, 2023
85637ba
fix s3 go.mods
lucix-aws Jul 12, 2023
50d9d41
update expected errs for (2)
lucix-aws Jul 12, 2023
679e5d9
fix mrap error message asserts
lucix-aws Jul 12, 2023
233a39e
only serialize nonarn buckets on immutable host
lucix-aws Jul 12, 2023
97d4fb7
Delegate custom v1 resolvers to legacy customizations, remove adapter…
isaiahvita Jul 13, 2023
e2252e0
fix custom resolvers with path style (#2188)
isaiahvita Jul 13, 2023
3215621
add docs for migrating endpoint resolver v1
lucix-aws Jul 13, 2023
3fe6736
Fix S3Control endpoint customization tests (#2195)
isaiahvita Jul 19, 2023
1bbaab0
regen s3 and eventbridge; add context checker to eb; eb tests passing…
isaiahvita Jul 20, 2023
ecd96cc
Fix host prefix bug (#2200)
isaiahvita Jul 21, 2023
906b316
Fix broken S3 tests (#2203)
isaiahvita Jul 24, 2023
13216bc
don't serde the bucket (#2204)
lucix-aws Jul 25, 2023
09b3a36
ADD noserde to bucket instead of swapping with httplabel to prevent s…
lucix-aws Jul 25, 2023
5c4a4d3
mark EndpointResolverWithOptions deprecated
lucix-aws Jul 25, 2023
1a66c87
Added fallback behavior to custom v1 resolvers (#2206)
isaiahvita Jul 25, 2023
6872543
Make builtin resolver private (#2208)
isaiahvita Jul 25, 2023
5ce3521
remove unnecessary line thats breaking full build (#2209)
isaiahvita Jul 26, 2023
1dd1f49
fix no ruleset bug (#2210)
isaiahvita Jul 26, 2023
e4bb6df
regen for ep2
lucix-aws Jul 26, 2023
4fe37f7
suppress internal rulesfn lint
lucix-aws Jul 26, 2023
b330936
fix auth scheme test
lucix-aws Jul 26, 2023
9bbecdd
remove middleware dependencies on resolveendpointv2 (#2211)
isaiahvita Jul 27, 2023
8ac86b6
regen
lucix-aws Jul 27, 2023
3d0374d
regen sdk (#2213)
isaiahvita Jul 27, 2023
0ffe660
add changelog file
isaiahvita Jul 27, 2023
3b19a0d
temp patch ci workflow to use upstream smithy-go
lucix-aws Jul 27, 2023
9920f71
fix windows pathexp, point to ep20 smithygo for codegen
lucix-aws Jul 27, 2023
bbb3c26
have lint use ep2 smithy-go, netip -> net for min go version
lucix-aws Jul 27, 2023
87a6564
no source change: comment on ipv6 host id parse
lucix-aws Jul 27, 2023
99b20d3
regen sdk
isaiahvita Jul 27, 2023
3afdb0f
upgrade golint GH action
isaiahvita Jul 28, 2023
b092577
remove ep20 hardcoding in GH action
isaiahvita Jul 31, 2023
13394ca
fix deprecated comment
isaiahvita Jul 31, 2023
e9c9311
fixed changelog file
isaiahvita Jul 31, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Added fallback behavior to custom v1 resolvers (#2206)
  • Loading branch information
isaiahvita committed Jul 31, 2023
commit 1a66c8739a608fef3980acf00b221d1da796a3f1
Original file line number Diff line number Diff line change
Expand Up @@ -185,6 +185,13 @@ public class AddAwsConfigFields implements GoIntegration {
.documentation("API stack mutators")
.generatedOnClient(false)
.build(),
AwsConfigField.builder()
.name(ENDPOINT_RESOLVER_CONFIG_NAME)
.type(getAwsCoreSymbol("EndpointResolver"))
.generatedOnClient(false)
.awsResolveFunction(SymbolUtils.createValueSymbolBuilder(RESOLVE_AWS_CONFIG_ENDPOINT_RESOLVER)
.build())
.build(),
AwsConfigField.builder()
.name(LOGGER_CONFIG_NAME)
.type(getAwsCoreSymbol("Logger"))
Expand Down Expand Up @@ -263,6 +270,7 @@ private void writeAwsDefaultResolvers(GoWriter writer) {
writeHttpClientResolver(writer);
writeRetryerResolvers(writer);
writeRetryMaxAttemptsFinalizeResolver(writer);
writeAwsConfigEndpointResolver(writer);
}

private void writerAwsDefaultResolversTests(GoWriter writer) {
Expand Down Expand Up @@ -581,6 +589,24 @@ private void writeHttpClientResolver(GoWriter writer) {
writer.popState();
}

private void writeAwsConfigEndpointResolver(GoWriter writer) {
writer.pushState();
writer.putContext("resolverName", RESOLVE_AWS_CONFIG_ENDPOINT_RESOLVER);
writer.putContext("clientOption", ENDPOINT_RESOLVER_CONFIG_NAME);
writer.putContext("wrapperHelper", EndpointGenerator.AWS_ENDPOINT_RESOLVER_HELPER);
writer.putContext("awsResolver", ENDPOINT_RESOLVER_CONFIG_NAME);
writer.putContext("awsResolverWithOptions", AWS_ENDPOINT_RESOLVER_WITH_OPTIONS);
writer.write("""
func $resolverName:L(cfg aws.Config, o *Options) {
if cfg.$awsResolver:L == nil && cfg.$awsResolverWithOptions:L == nil {
return
}
o.$clientOption:L = $wrapperHelper:L(cfg.$awsResolver:L, cfg.$awsResolverWithOptions:L)
}
""");
writer.popState();
}

@Override
public List<RuntimeClientPlugin> getClientPlugins() {
List<RuntimeClientPlugin> plugins = new ArrayList<>();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -394,33 +394,13 @@ private void generateAwsEndpointResolverWrapper(GoWriter writer) {
writer.write("""
type $T struct {
awsResolver $T
resolver $T
}
""", wrappedResolverSymbol, endpointResolverWithOptions, resolverInterface);
""", wrappedResolverSymbol, endpointResolverWithOptions);
var endpointNotFoundError = SymbolUtils.createValueSymbolBuilder("EndpointNotFoundError",
AwsGoDependency.AWS_CORE).build();

writeExternalResolveEndpointImplementation(writer, wrappedResolverSymbol, "w", () -> {
var endpointNotFoundError = SymbolUtils.createValueSymbolBuilder("EndpointNotFoundError",
AwsGoDependency.AWS_CORE).build();
var errorf = SymbolUtils.createValueSymbolBuilder("Errorf",
SmithyGoDependency.FMT).build();
writer.write("""
if w.awsResolver == nil {
goto fallback
}
endpoint, err = w.awsResolver.ResolveEndpoint(ServiceID, region, options)
if err == nil {
return endpoint, nil
}

if nf := (&$T{}); !errors.As(err, &nf) {
return endpoint, err
}

fallback:
if w.resolver == nil {
return endpoint, $T("default endpoint resolver provided was nil")
}
return w.resolver.ResolveEndpoint(region, options)""", endpointNotFoundError, errorf);
writer.write("return w.awsResolver.ResolveEndpoint(ServiceID, region, options)");

writer.addUseImports(SmithyGoDependency.ERRORS);
});
Expand All @@ -441,28 +421,31 @@ private void generateAwsEndpointResolverWrapper(GoWriter writer) {
// Generate exported helper for constructing a wrapper around the AWS EndpointResolver type that is compatible
// with the clients EndpointResolver interface.
writer.write("""
// $L returns an EndpointResolver that first delegates endpoint resolution to the awsResolver.
// If awsResolver returns aws.EndpointNotFoundError error, the resolver will use the the provided
// fallbackResolver for resolution.
// $1L returns an $3T that first delegates endpoint resolution to the awsResolver.
// If awsResolver returns $7T error, the v1 resolver middleware will swallow the error,
// and set an appropriate context flag such that fallback will occur when $8L is invoked
// via its middleware.
//
// fallbackResolver must not be nil
func $L(awsResolver $T, awsResolverWithOptions $T, fallbackResolver $T) $T {
var resolver $T
// If another error (besides $7T) is returned, then that error will be propagated.
func $1L(awsResolver $2T, awsResolverWithOptions $3T) $6L {
var resolver $3T

if awsResolverWithOptions != nil {
resolver = awsResolverWithOptions
} else if awsResolver != nil {
resolver = $L(awsResolver.ResolveEndpoint)
resolver = $4L(awsResolver.ResolveEndpoint)
}

return &$T{
return &$5T{
awsResolver: resolver,
resolver: fallbackResolver,
}
}
""", AWS_ENDPOINT_RESOLVER_HELPER, AWS_ENDPOINT_RESOLVER_HELPER,
endpointResolver, endpointResolverWithOptions, resolverInterface,
resolverInterface, endpointResolverWithOptions, AWS_ENDPOINT_RESOLVER_ADAPTOR, wrappedResolverSymbol);
""", AWS_ENDPOINT_RESOLVER_HELPER,
endpointResolver, endpointResolverWithOptions,
AWS_ENDPOINT_RESOLVER_ADAPTOR, wrappedResolverSymbol,
resolverInterface, endpointNotFoundError,
EndpointResolutionGenerator.RESOLVER_INTERFACE_NAME
);
}

private void generateMiddleware(GoWriter writer) {
Expand Down Expand Up @@ -532,9 +515,22 @@ private void generateMiddlewareResolverBody(GoStackStepMiddlewareGenerator g, Go
w.write("var endpoint $T", SymbolUtils.createValueSymbolBuilder("Endpoint", AwsGoDependency.AWS_CORE)
.build());
w.write("endpoint, err = m.Resolver.ResolveEndpoint(awsmiddleware.GetRegion(ctx), eo)");
w.openBlock("if err != nil {", "}", () -> {
w.write("return out, metadata, fmt.Errorf(\"failed to resolve service endpoint, %w\", err)");
});
w.write(
"""
if err != nil {
nf := (&$1T{})
if $2T(err, &nf) {
ctx = $3T(ctx, false)
return next.HandleSerialize(ctx, in)
}
return out, metadata, $4T(\"failed to resolve service endpoint, %w\", err)
}
""",
SymbolUtils.createValueSymbolBuilder("EndpointNotFoundError", AwsGoDependency.AWS_CORE).build(),
SymbolUtils.createValueSymbolBuilder("As", SmithyGoDependency.ERRORS).build(),
SymbolUtils.createValueSymbolBuilder("SetRequiresLegacyEndpoints", AwsGoDependency.AWS_MIDDLEWARE).build(),
SymbolUtils.createValueSymbolBuilder("Errorf", SmithyGoDependency.FMT).build()
);
w.write("");

w.write("req.URL, err = url.Parse(endpoint.URL)");
Expand Down Expand Up @@ -625,7 +621,6 @@ private void generatePublicResolverTypes(GoWriter writer) {
writer.write("return fn(region, options)");
});


// Generate EndpointResolverFromURL helper
writer.writeDocs(String.format("%s returns an EndpointResolver configured using the provided endpoint url. "
+ "By default, the resolved endpoint resolver uses the client region as signing region, and "
Expand Down
3 changes: 2 additions & 1 deletion service/s3/api_client.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

36 changes: 12 additions & 24 deletions service/s3/endpoints.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

12 changes: 12 additions & 0 deletions service/s3/internal/customizations/update_endpoint_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -957,6 +957,18 @@ func TestVPC_CustomEndpoint(t *testing.T) {
expectedSigningName: "s3",
expectedSigningRegion: "us-west-2",
},
"custom resolver to v2 fallback": {
bucket: "bucketname",
options: s3.Options{
EndpointResolver: EndpointResolverFunc(func(region string, options s3.EndpointResolverOptions) (aws.Endpoint, error) {
return aws.Endpoint{}, &aws.EndpointNotFoundError{}
}),
Region: "us-west-2",
},
expectedReqURL: "https://bucketname.s3.us-west-2.amazonaws.com/",
expectedSigningName: "s3",
expectedSigningRegion: "us-west-2",
},
"AccessPoint with custom endpoint url": {
bucket: "arn:aws:s3:us-west-2:123456789012:accesspoint:myendpoint",
options: s3.Options{
Expand Down
4 changes: 3 additions & 1 deletion service/s3control/api_client.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

36 changes: 12 additions & 24 deletions service/s3control/endpoints.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 3 additions & 1 deletion service/s3outposts/api_client.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading