Cross-Region/Account References

[SachinShekhar]

An idea to fix cross-region/account resource sharing issue which goes beyond resource sharing capabilities of CloudFormation.

Use Case

CDK is a wonderful product which should get Best Cloud Innovation Award. It allows us to create super complex cloud infrastructure quickly without needing a team of hundreds or thousands.

Cross-region/account resource sharing is only natural for even small projects. It should have worked smoothly like same env resource sharing. Unfortunately, only those references which doesn't involve CloudFormation work (that's also if we provide PhysicalName or use core.PhysicalName.GENERATE_IF_NEEDED). To reference CloudFormation generated refs, we need to use complex strategies using NestedStack, CustomResource, ProviderFramework.

This feature will make everything smooth and simple. Users will be able to share resources across stacks without thinking about stack's env. Splitting existing CDK apps across multiple accounts will also become easy.

Proposed Solution

1. If there are circular dependencies across stacks, throw error.

2. Synthesize and deploy only those stacks which are dependencies.

3. Retrieve resolved refs from previous step and use them to synthesize and deploy dependent stacks.

4. Repeat the last step if there are more dependent stacks.

Other

CDK Synth won't be able to work without deployment, so it should throw error if there are cross-region/account references. It shouldn't be called BREAKING CHANGE because existing CDK apps won't be affected. This is a small price to pay if someone wants to use new cross-region/account reference feature. Maybe, to make existing behavior same, a new flag in App class can be introduced which will need to be turned ON to use this new feature.

• 👋 I may be able to implement this feature request
• ⚠️ This feature might incur a breaking change

---

This is a 🚀 Feature Request

[SachinShekhar]

Proof of Concept

This is how I make it work:

Let's say, there are two stacks viz., Stack1 & Stack2. Both exist in completely different regions and accounts. Stack2 needs to use resources from Stack1.

To solve the issue, I define two helper functions, one of which is this:

export function createLocalExports(
  scope: cdk.Construct,
  exports: { key: string; value: string }[]
) {
  exports.forEach(export => {
    new cdk.CfnOutput(scope, export.key, {
      value: export.value
    });
  });
}

Stack1 can use it like this:

createLocalExports(this, [{
  key: 'UserIdentityPoolId',
  value: userIdentityPool.ref
}]);

Then, running cdk deploy Stack1 --outputs-file=output.json --profile xyz will write outputs to output.json file (Command piping can also be used).

To consume outputs, here's another helper function:

export function importStackOutput(outputKey: string) {
  const outputFilePath = path.join(__dirname, 'relative/path/to/output.json' );

  let stackOutputs: {
    [key: string]: { [key: string]: string };
  };

  try {
    stackOutputs = JSON.parse(
      fs.readFileSync(outputFilePath).toString('UTF-8')
    );
  } catch (error) {
    stackOutputs = {};
  }

  return Object.keys(stackOutputs).length ? stackOutputs['Stack1'][outputKey] : '';
}

Stack2 can use it like this:

const userIdentityPoolId = importStackOutput('UserIdentityPoolId');
// use userIdentityPoolId wherever you want

So, the final command to deploy everything would be: cdk deploy Stack1 --outputs-file=output.json --profile xyz && cdk deploy Stack2 --profile abc (can be saved as npm scripts).

It works!

[SachinShekhar]

CDK Toolkit can share outputs internally without writing them to file. I don't want to touch it because I am not entirely familiar with core systems. To share entire resource objects, new objects with resolved refs need to be created so that their methods like grantX() continue to work. Maybe, token system also needs to be modified to take into account the completely resolved refs.

[eladb]

Hi @SachinShekhar!

Thanks for the thoughtful proposal. We know that cross-environment references have interesting use cases and would love to consider ideas for solutions. I am moving this issue to our RFCs repo, which is where we can discuss such topics.

Based on our current plans, I doubt that this major feature is something we can commit to in the coming months, but if you wish to continue the conversation, I'd recommend that you follow our RFC process and submit a detailed proposal.

From the outset, as you observed, the main challenge I see with this approach is that it breaks a fundamental design tenets which says that synthesis is isolated from deployment. This tenet is important for security reasons and in order to allow CDK apps to be deployed through CI/CD systems.

I would encourage you to consider designs that "stay within the bounds" of the current model. For example, use CFN custom resources to query values across environments during deployment.

[pgollucci]

"I would encourage you to consider designs that "stay within the bounds" of the current model. For example, use CFN custom resources to query values across environments during deployment."

If that's the best practice that needs to go in the docs.

[mrgrain]

Marking this RFCs as stale since there has been little recent activity and it is not currently close to getting accepted as-is. We appreciate the effort that has gone into this proposal. Marking an RFCs as stale is not a one-way door. If you have made substantial changes to the proposal, please open a new issue/RFC. You might also consider raising a PR to aws/aws-cdk directly or self-publishing to Construct Hub.