Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a new smithy-modeled rules based endpoint resolution. #2214

Merged
merged 58 commits into from
Jul 31, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
fe4c3b3
Add aws standard lib functions (#2057)
isaiahvita Mar 20, 2023
c40b4dd
remove error collector for now (#2073)
isaiahvita Mar 29, 2023
f4381d4
add partition generator, and run gofmt (#2076)
isaiahvita Apr 5, 2023
f79579a
Add aws library functions for endpoint rules engine and fix partition…
isaiahvita May 5, 2023
0eb37e5
Add tests codegen and regen s3 (#2126)
isaiahvita May 17, 2023
72ed6c0
Add client config adapters (#2146)
isaiahvita Jun 12, 2023
98f1519
Add S3 Model Preprocessing for EP 2.0 (#2153)
Jun 21, 2023
50ceb26
Add BuiltIn generation (#2152)
isaiahvita Jun 23, 2023
7cb6d19
Add endpoint-based auth scheme resolution (#2158)
isaiahvita Jun 29, 2023
5a56b8a
Remove httpLabel trait from Bucket members (#2173)
isaiahvita Jul 7, 2023
b44fbbf
convert s3 handle_200_error_test into an actual unit test with a fake…
lucix-aws Jul 7, 2023
98caa35
Add functionality for non-SSL endpoints for backwards compatibility (…
isaiahvita Jul 7, 2023
f0fe836
fix borked go mod files
lucix-aws Jul 7, 2023
0839634
Fix Properties Auth Scheme Path (#2177)
Jul 7, 2023
260ce31
fix empty string signing name and region for sigv4 on ctx (#2178)
lucix-aws Jul 7, 2023
1229116
Check S3 signer customization insensitively (#2179)
Jul 7, 2023
151def9
Fix EP20 url encoding issues with S3 (#2181)
isaiahvita Jul 10, 2023
9d7de33
serde buckets into immutable hostname endpts (#2182)
lucix-aws Jul 11, 2023
774213d
adjust some expected errs
lucix-aws Jul 11, 2023
bd74619
remove incorrectly-applied {RequestRoute} filter
lucix-aws Jul 11, 2023
6921c80
fix borked serializers from last commit
lucix-aws Jul 11, 2023
50e8ce8
use httpbinding to escape instead
lucix-aws Jul 11, 2023
6c91a64
Check hostname immutable in EndpointDisableHttps and make it service …
isaiahvita Jul 11, 2023
4a484db
add back unintentionally deleted test cases (#2185)
isaiahvita Jul 11, 2023
85637ba
fix s3 go.mods
lucix-aws Jul 12, 2023
50d9d41
update expected errs for (2)
lucix-aws Jul 12, 2023
679e5d9
fix mrap error message asserts
lucix-aws Jul 12, 2023
233a39e
only serialize nonarn buckets on immutable host
lucix-aws Jul 12, 2023
97d4fb7
Delegate custom v1 resolvers to legacy customizations, remove adapter…
isaiahvita Jul 13, 2023
e2252e0
fix custom resolvers with path style (#2188)
isaiahvita Jul 13, 2023
3215621
add docs for migrating endpoint resolver v1
lucix-aws Jul 13, 2023
3fe6736
Fix S3Control endpoint customization tests (#2195)
isaiahvita Jul 19, 2023
1bbaab0
regen s3 and eventbridge; add context checker to eb; eb tests passing…
isaiahvita Jul 20, 2023
ecd96cc
Fix host prefix bug (#2200)
isaiahvita Jul 21, 2023
906b316
Fix broken S3 tests (#2203)
isaiahvita Jul 24, 2023
13216bc
don't serde the bucket (#2204)
lucix-aws Jul 25, 2023
09b3a36
ADD noserde to bucket instead of swapping with httplabel to prevent s…
lucix-aws Jul 25, 2023
5c4a4d3
mark EndpointResolverWithOptions deprecated
lucix-aws Jul 25, 2023
1a66c87
Added fallback behavior to custom v1 resolvers (#2206)
isaiahvita Jul 25, 2023
6872543
Make builtin resolver private (#2208)
isaiahvita Jul 25, 2023
5ce3521
remove unnecessary line thats breaking full build (#2209)
isaiahvita Jul 26, 2023
1dd1f49
fix no ruleset bug (#2210)
isaiahvita Jul 26, 2023
e4bb6df
regen for ep2
lucix-aws Jul 26, 2023
4fe37f7
suppress internal rulesfn lint
lucix-aws Jul 26, 2023
b330936
fix auth scheme test
lucix-aws Jul 26, 2023
9bbecdd
remove middleware dependencies on resolveendpointv2 (#2211)
isaiahvita Jul 27, 2023
8ac86b6
regen
lucix-aws Jul 27, 2023
3d0374d
regen sdk (#2213)
isaiahvita Jul 27, 2023
0ffe660
add changelog file
isaiahvita Jul 27, 2023
3b19a0d
temp patch ci workflow to use upstream smithy-go
lucix-aws Jul 27, 2023
9920f71
fix windows pathexp, point to ep20 smithygo for codegen
lucix-aws Jul 27, 2023
bbb3c26
have lint use ep2 smithy-go, netip -> net for min go version
lucix-aws Jul 27, 2023
87a6564
no source change: comment on ipv6 host id parse
lucix-aws Jul 27, 2023
99b20d3
regen sdk
isaiahvita Jul 27, 2023
3afdb0f
upgrade golint GH action
isaiahvita Jul 28, 2023
b092577
remove ep20 hardcoding in GH action
isaiahvita Jul 31, 2023
13394ca
fix deprecated comment
isaiahvita Jul 31, 2023
e9c9311
fixed changelog file
isaiahvita Jul 31, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add client config adapters (#2146)
  • Loading branch information
isaiahvita committed Jul 31, 2023
commit 72ed6c00302a16d78150b3d3a3b0eee95350258e
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@
import software.amazon.smithy.go.codegen.GoWriter;
import software.amazon.smithy.go.codegen.SymbolUtils;
import software.amazon.smithy.go.codegen.TriConsumer;
import software.amazon.smithy.go.codegen.endpoints.EndpointResolutionGenerator;
import software.amazon.smithy.go.codegen.integration.ConfigField;
import software.amazon.smithy.go.codegen.integration.ConfigFieldResolver;
import software.amazon.smithy.go.codegen.integration.GoIntegration;
Expand Down Expand Up @@ -68,7 +69,16 @@ public List<RuntimeClientPlugin> getClientPlugins() {
.name(ENDPOINT_RESOLVER_CONFIG_NAME)
.type(SymbolUtils.createValueSymbolBuilder(EndpointGenerator.RESOLVER_INTERFACE_NAME)
.build())
.documentation("The service endpoint resolver.")
.documentation(String.format("The service endpoint resolver."))
.deprecated(String.format(
"""
%s and With%s are deprecated. See %s and With%s
""",
EndpointGenerator.RESOLVER_INTERFACE_NAME,
EndpointGenerator.RESOLVER_INTERFACE_NAME,
EndpointResolutionGenerator.RESOLVER_INTERFACE_NAME,
EndpointResolutionGenerator.RESOLVER_INTERFACE_NAME
))
.withHelper(true)
.build(),
ConfigField.builder()
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,338 @@
package software.amazon.smithy.aws.go.codegen;


import static software.amazon.smithy.go.codegen.GoWriter.goTemplate;

import software.amazon.smithy.codegen.core.SymbolProvider;
import software.amazon.smithy.go.codegen.GoSettings;
import software.amazon.smithy.go.codegen.GoWriter;
import software.amazon.smithy.go.codegen.SmithyGoDependency;
import software.amazon.smithy.go.codegen.SymbolUtils;
import software.amazon.smithy.go.codegen.TriConsumer;
import software.amazon.smithy.go.codegen.endpoints.EndpointResolutionGenerator;
import software.amazon.smithy.go.codegen.integration.ConfigField;
import software.amazon.smithy.go.codegen.integration.ConfigFieldResolver;
import software.amazon.smithy.go.codegen.integration.GoIntegration;
import software.amazon.smithy.go.codegen.integration.RuntimeClientPlugin;
import software.amazon.smithy.model.Model;
import software.amazon.smithy.utils.ListUtils;
import software.amazon.smithy.utils.MapUtils;
import software.amazon.smithy.utils.SetUtils;

import java.util.List;
import java.util.Map;
import java.util.function.Consumer;

public class AwsEndpointResolverAdapterGenerator implements GoIntegration {

public static final String LEGACY_ADAPTER_TYPE = "legacyEndpointResolverAdapter";
public static final String COMPATIBLE_ADAPTER_TYPE = "compatibleEndpointResolver";
public static final String FINALIZE_ENDPOINT_RESOLVER_V2 = "finalize" + EndpointResolutionGenerator.RESOLVER_INTERFACE_NAME;


private Map<String, Object> commonCodegenArgs;

@Override
public void writeAdditionalFiles(
GoSettings settings,
Model model,
SymbolProvider symbolProvider,
TriConsumer<String, String, Consumer<GoWriter>> writerFactory
) {
this.commonCodegenArgs = MapUtils.of(
"goContext", SymbolUtils.createValueSymbolBuilder("Context", SmithyGoDependency.CONTEXT).build(),
"legacyAdapterType", SymbolUtils.createValueSymbolBuilder(LEGACY_ADAPTER_TYPE).build(),
"legacyResolverType", SymbolUtils.createValueSymbolBuilder(EndpointGenerator.RESOLVER_INTERFACE_NAME).build(),
"resolverType", SymbolUtils.createValueSymbolBuilder(EndpointResolutionGenerator.RESOLVER_INTERFACE_NAME).build(),
"paramsType", SymbolUtils.createValueSymbolBuilder(EndpointResolutionGenerator.PARAMETERS_TYPE_NAME).build(),
"smithyEndpointType", SymbolUtils.createValueSymbolBuilder("Endpoint", SmithyGoDependency.SMITHY_ENDPOINTS).build(),
"awsEndpointType", SymbolUtils.createValueSymbolBuilder("Endpoint", AwsGoDependency.AWS_CORE).build(),
"fmtErrorf", SymbolUtils.createValueSymbolBuilder("Errorf", SmithyGoDependency.FMT).build());

var content = new GoWriter.ChainWritable()
.add(generateLegacyAdapter())
.add(generateCompatibleAdapter())
.add(generatePseudoRegionUtility())
.add(generateFinalizeMethod())
.compose();

writerFactory.accept("endpoints.go", settings.getModuleName(), writer -> {
writer.write("$W", content);
});

}

private GoWriter.Writable generateLegacyAdapter() {
return goTemplate("""
type $legacyAdapterType:L struct {
legacyResolver $legacyResolverType:T
resolver $resolverType:T
}

func (l *$legacyAdapterType:T) ResolveEndpoint(ctx $goContext:T, params $paramsType:T) (endpoint $smithyEndpointType:T, err error) {
$legacyResolveMethodBody:W
}
""",
commonCodegenArgs,
MapUtils.of(
"legacyResolveMethodBody", generateLegacyResolveMethodBody()
));
}


private GoWriter.Writable generateLegacyResolveMethodBody() {
return goTemplate(
"""
$requiredOptions:W

$resolveInvocation:W

$hostnameImmutableCheck:W

$pushdownToResolver:W

""",
commonCodegenArgs,
MapUtils.of(
"requiredOptions", generateRequiredOptions(),
"resolveInvocation", generateResolveInvocation(),
"hostnameImmutableCheck", generateHostnameImmutableCheck(),
"pushdownToResolver", generatePushdownToResolver()));
}

private GoWriter.Writable generateRequiredOptions() {
return goTemplate(
"""
var fips $awsFipsEndpointStateType:T
var dualStack $awsDualStackEndpointStateType:T

if aws.ToBool(params.UseFIPS) {
fips = $awsFipsEndpointStateEnabledType:T
}
if aws.ToBool(params.UseDualStack) {
dualStack = $awsDualStackEndpointStateEnabledType:T
}
""",
commonCodegenArgs,
MapUtils.of(
"awsFipsEndpointStateType", SymbolUtils.createValueSymbolBuilder("FIPSEndpointState", AwsGoDependency.AWS_CORE).build(),
"awsFipsEndpointStateEnabledType", SymbolUtils.createValueSymbolBuilder("FIPSEndpointStateEnabled", AwsGoDependency.AWS_CORE).build(),
"awsDualStackEndpointStateEnabledType", SymbolUtils.createValueSymbolBuilder("DualStackEndpointStateEnabled", AwsGoDependency.AWS_CORE).build(),
"awsDualStackEndpointStateType", SymbolUtils.createValueSymbolBuilder("DualStackEndpointState", AwsGoDependency.AWS_CORE).build()
));

}

private GoWriter.Writable generateResolveInvocation() {
return goTemplate(
"""
resolveEndpoint, err := l.legacyResolver.ResolveEndpoint($toString:T(params.Region), EndpointResolverOptions{
ResolvedRegion: $toString:T(params.Region),
UseFIPSEndpoint: fips,
UseDualStackEndpoint: dualStack,
})
if err != nil {
return endpoint, err
}
""",
commonCodegenArgs,
MapUtils.of(
"toString", SymbolUtils.createValueSymbolBuilder("ToString", AwsGoDependency.AWS_CORE).build()
));
}

private GoWriter.Writable generateHostnameImmutableCheck() {
return goTemplate(
"""
if resolveEndpoint.HostnameImmutable {
uriString := resolveEndpoint.URL
uri, err := $parseUrl:T(uriString)
if err != nil {
return endpoint, $fmtErrorf:T(\"Failed to parse uri: %s\", uriString)
}

return $smithyEndpointType:T{
URI: *uri,
}, nil
}
""",
commonCodegenArgs,
MapUtils.of(
"parseUrl", SymbolUtils.createValueSymbolBuilder("Parse", SmithyGoDependency.NET_URL).build()
));
}

private GoWriter.Writable generatePushdownToResolver() {
return goTemplate(
"""
if resolveEndpoint.Source == $endpointSourceMetadata:T {
return l.resolver.ResolveEndpoint(ctx, params)
}

params = params.WithDefaults()
params.Endpoint = &resolveEndpoint.URL

return l.resolver.ResolveEndpoint(ctx, params)
""",
commonCodegenArgs,
MapUtils.of(
"endpointSourceMetadata", SymbolUtils.createValueSymbolBuilder("EndpointSourceServiceMetadata", AwsGoDependency.AWS_CORE).build()
));

}


private GoWriter.Writable generateCompatibleAdapter() {
return goTemplate(
"""
type isDefaultProvidedImplementation interface {
isDefaultProvidedImplementation()
}

type $compatibleResolverType:T struct {
EndpointResolverV2 $resolverType:T
}

func (n *$compatibleResolverType:T) isDefaultProvidedImplementation() {}

func (n *$compatibleResolverType:T) ResolveEndpoint(region string, options EndpointResolverOptions) (endpoint $awsEndpointType:T, err error) {
$compatibleResolveMethodBody:W
}
""",
commonCodegenArgs,
MapUtils.of(
"compatibleResolverType", SymbolUtils.createValueSymbolBuilder(COMPATIBLE_ADAPTER_TYPE).build(),
"compatibleResolveMethodBody", generateCompatibleResolveMethodBody()
));
}

private GoWriter.Writable generateCompatibleResolveMethodBody() {
return goTemplate(
"""
reg := region
fips := options.UseFIPSEndpoint
if len(options.ResolvedRegion) > 0 {
reg = options.ResolvedRegion
} else {
// $resolverInterfaceName:L needs to support pseudo-regions to maintain backwards-compatibility
// with the legacy $legacyResolverInterfaceName:L
reg, fips = mapPseudoRegion(region)
}
ctx := context.Background()
resolved, err := n.EndpointResolverV2.ResolveEndpoint(ctx, $paramsType:T{
Region: &reg,
UseFIPS: $awsBoolType:T(fips == $awsFipsEndpointStateEnabledType:T),
UseDualStack: $awsBoolType:T(options.UseDualStackEndpoint == $awsDualStackEndpointStateEnabledType:T),
})
if err != nil {
return endpoint, err
}

endpoint = $awsEndpointType:T{
URL: resolved.URI.String(),
HostnameImmutable: false,
Source: $endpointSourceMetadata:T,
}

return endpoint, nil
""",
commonCodegenArgs,
MapUtils.of(
"awsBoolType", SymbolUtils.createValueSymbolBuilder("Bool", AwsGoDependency.AWS_CORE).build(),
"awsFipsEndpointStateEnabledType", SymbolUtils.createValueSymbolBuilder("FIPSEndpointStateEnabled", AwsGoDependency.AWS_CORE).build(),
"awsDualStackEndpointStateEnabledType", SymbolUtils.createValueSymbolBuilder("DualStackEndpointStateEnabled", AwsGoDependency.AWS_CORE).build(),
"endpointSourceMetadata", SymbolUtils.createValueSymbolBuilder("EndpointSourceServiceMetadata", AwsGoDependency.AWS_CORE).build(),
"resolverInterfaceName", EndpointResolutionGenerator.RESOLVER_INTERFACE_NAME,
"legacyResolverInterfaceName", EndpointGenerator.RESOLVER_INTERFACE_NAME
));
}

private GoWriter.Writable generatePseudoRegionUtility() {
return goTemplate(
"""
// Utility function to aid with translating pseudo-regions to classical regions
// with the appropriate setting indicated by the pseudo-region
func mapPseudoRegion(pr string) (region string, fips $awsFipsEndpointStateType:T) {
const fipsInfix = \"-fips-\"
const fipsPrefix = \"fips-\"
const fipsSuffix = \"-fips\"

if strings.Contains(pr, fipsInfix) ||
strings.Contains(pr, fipsPrefix) ||
strings.Contains(pr, fipsSuffix) {
region = strings.ReplaceAll(strings.ReplaceAll(strings.ReplaceAll(
pr, fipsInfix, "-"), fipsPrefix, ""), fipsSuffix, "")
fips = $awsFipsEndpointStateEnabledType:T
} else {
region = pr
}

return region, fips
}
""",
commonCodegenArgs,
MapUtils.of(
"awsFipsEndpointStateEnabledType", SymbolUtils.createValueSymbolBuilder("FIPSEndpointStateEnabled", AwsGoDependency.AWS_CORE).build(),
"awsFipsEndpointStateType", SymbolUtils.createValueSymbolBuilder("FIPSEndpointState", AwsGoDependency.AWS_CORE).build()
));
}


private GoWriter.Writable generateFinalizeMethod() {
return goTemplate(
"""
func $finalizeMethodName:L(options *Options) {
// Check if the EndpointResolver was not user provided
// but is the SDK's default provided version.
_, ok := options.EndpointResolver.(isDefaultProvidedImplementation)
if options.EndpointResolverV2 == nil {
options.EndpointResolverV2 = $newResolverFuncName:L()
}
if ok {
// Nothing further to do
return
}

options.EndpointResolverV2 = &$legacyAdapterType:T{
legacyResolver: options.EndpointResolver,
resolver: $newResolverFuncName:L(),
}
}

""",
commonCodegenArgs,
MapUtils.of(
"finalizeMethodName", FINALIZE_ENDPOINT_RESOLVER_V2,
"newResolverFuncName", EndpointResolutionGenerator.NEW_RESOLVER_FUNC_NAME
));

}

@Override
public List<RuntimeClientPlugin> getClientPlugins() {
return ListUtils.of(
RuntimeClientPlugin.builder()
.configFields(SetUtils.of(
ConfigField.builder()
.name(EndpointResolutionGenerator.RESOLVER_INTERFACE_NAME)
.type(SymbolUtils.createValueSymbolBuilder(EndpointResolutionGenerator.RESOLVER_INTERFACE_NAME)
.build())
.documentation(String.format(
"""
Resolves the endpoint used for a particular service. This should be used over the
deprecated %s
""",
EndpointGenerator.RESOLVER_INTERFACE_NAME
))
.withHelper(true)
.build()
))
.addConfigFieldResolver(ConfigFieldResolver.builder()
.location(ConfigFieldResolver.Location.OPERATION)
.target(ConfigFieldResolver.Target.FINALIZATION)
.resolver(SymbolUtils.createValueSymbolBuilder(
FINALIZE_ENDPOINT_RESOLVER_V2).build())
.build())
.build());
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@
import software.amazon.smithy.go.codegen.SmithyGoDependency;
import software.amazon.smithy.go.codegen.SymbolUtils;
import software.amazon.smithy.go.codegen.TriConsumer;
import software.amazon.smithy.go.codegen.endpoints.EndpointResolutionGenerator;
import software.amazon.smithy.go.codegen.integration.ConfigField;
import software.amazon.smithy.go.codegen.integration.ProtocolUtils;
import software.amazon.smithy.model.Model;
Expand Down Expand Up @@ -617,7 +618,9 @@ private void generatePublicResolverTypes(GoWriter writer) {
writer.openBlock("func $L(o $P) {", "}", CLIENT_CONFIG_RESOLVER,
SymbolUtils.createPointableSymbolBuilder("Options").build(), () -> {
writer.openBlock("if o.EndpointResolver != nil {", "}", () -> writer.write("return"));
writer.write("o.EndpointResolver = $L()", RESOLVER_CONSTRUCTOR_NAME);
writer.openBlock("o.EndpointResolver = &$L{", "}", AwsEndpointResolverAdapterGenerator.COMPATIBLE_ADAPTER_TYPE, () -> writer.write(
"$L: $L(),", EndpointResolutionGenerator.RESOLVER_INTERFACE_NAME, EndpointResolutionGenerator.NEW_RESOLVER_FUNC_NAME
));
});

// Generate EndpointResolverFromURL helper
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -50,3 +50,4 @@ software.amazon.smithy.aws.go.codegen.customization.S3100Continue
software.amazon.smithy.aws.go.codegen.customization.ApiGatewayExportsNullabilityExceptionIntegration
software.amazon.smithy.aws.go.codegen.customization.LambdaRecursionDetection
software.amazon.smithy.aws.go.codegen.customization.SQSCustomizations
software.amazon.smithy.aws.go.codegen.AwsEndpointResolverAdapterGenerator
Loading