Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integration tests #31

Closed
1 of 13 tasks
eladb opened this issue Dec 9, 2019 · 6 comments
Closed
1 of 13 tasks

Integration tests #31

eladb opened this issue Dec 9, 2019 · 6 comments
Labels
ops Operations status/done Implementation complete testing Testing

Comments

@eladb
Copy link
Contributor

eladb commented Dec 9, 2019

PR Champion
#

Description

Progress

  • Tracking Issue Created
  • RFC PR Created
  • Core Team Member Assigned
  • Initial Approval / Final Comment Period
  • Ready For Implementation
    • implementation issue 1
  • Resolved
@eladb eladb added ops Operations testing Testing labels Dec 9, 2019
@eladb eladb mentioned this issue Dec 10, 2019
7 tasks
@MrArnoldPalmer MrArnoldPalmer added the status/proposed Newly proposed RFC label Jan 4, 2020
@mneil mneil mentioned this issue Jun 23, 2020
10 tasks
@douglasnaphas
Copy link

I definitely want the ability to automate the execution of these kinds of stack verification steps.

For example, the file linked above has:

/*
 * Stack verification steps:
 * * `curl -s -o /dev/null -w "%{http_code}" <url>` should return 401
 * * `curl -s -o /dev/null -w "%{http_code}" -H 'Authorization: deny' <url>` should return 403
 * * `curl -s -o /dev/null -w "%{http_code}" -H 'Authorization: allow' <url>` should return 200
 */

So if I run yarn integ authorizers/integ.token-authorizer.lit.js, I'd like for it to run the curl commands listed above prior to stack cleanup, and fail if it does not get the expected status codes.

I believe this included in the task Support Assertions, based on the first bullet point in #88.

@eladb
Copy link
Contributor Author

eladb commented Jul 25, 2021

Thanks, copying @nija-at

@robert-pitt-foodhub
Copy link

Hey, so I have just came across this PR and as I am currently implementation a solution for my team I wanted to just share my somewhat simplistic approach to performing post deploying integration testing on a stack.

  1. Environment
    In our company we use jest as our testing environment and framework for unit testing, this is common practice for typescript based applications we so wanted to ensure a consistent and unified experience for developers when writing unit tests or integration tests.

My approach was to simply create a secondary jest configuration that would allow us to distinguish between unit and integration test files (MyStack.unit.ts and MyStack.integ.ts), allowing for us to provide the developer with the ability to run yarn test or yarn test:integration to invoke each environment.

  1. Resource Identifiers
    In order for the jest integration test environment to know what resources identifiers it should use to perform the tests we have updated the cdk.json config file to add the following property ("outputsFile": ".deploy-output.json"), this outputs file will contains the fully resolved values exported via "CfnOutput" calls.

We have created some support utilities to make it simpler for developers to export those values within having a lot of duplicated CfnOutput within the classes, a simple function such as outputs({ "paramName": this.someResource.arn }).

When we execute the yarn test:integration command the configuration of jest will load a setup file that loads those values into memory and provides utilities that can be used within test environment to get the parameter value, e.g const someResourcesArn = getOutput("paramName"), this allows developers to not have to deal with dynamic names and can explicitly control the contract between the CDK Outputs and the accessing of those values within the test environment.

  1. Testing Resources.
    Now that we have jest environment that has the latest parameters for the previous deployment we simply write traditional jest tests using the aws-sdk library to interact with AWS services and perform integration testing in the similar way that we do unit testing.

This may have already been discussed and ruled out as an approach and if so it would be great to hear back from those who may have more experience with this approach to highlight some of the long term issues.

Regards :)

@nija-at
Copy link
Contributor

nija-at commented Oct 4, 2021

Thanks for your comment @robert-pitt-foodhub.

Could you also provide details into how you execute these integration tests? Ex: do they run as part of your CI pipeline?

Are you deploying a fresh stack each time for every integration test? How have you set up your AWS accounts? How do you clean up after a failure?

@nija-at nija-at removed their assignment Nov 25, 2021
@misterjoshua
Copy link

misterjoshua commented Feb 22, 2022

Our team has been experimenting with Step Functions to perform integration testing. We like this approach because the errors are easy to see in the execution history, they can take as long as they need, and it's easy to get at any x-ray traces to see where faults lie. It's also pretty easy to bundle up any custom lambdas and invoke them in one or several of the state machine steps - the code for that stuff can live right next to the definition of the test.

We've been starting the state machines from staging environment pipeline steps. We built a tool called cdk-exec to ease locating and running these tests, both in development and in the pipeline, from the cloud assembly.

We've also been starting and monitoring a few state machine integration tests via Custom Resources. We've liked these because they back out of deployments automatically if something is wrong, but we use them more sparingly.

We're also experimenting with low-code ways to make assertions in the state machines, but we're not sure yet what we'll do there.

@mrgrain
Copy link
Contributor

mrgrain commented Oct 27, 2023

integ-runner is available

@mrgrain mrgrain closed this as completed Oct 27, 2023
@mrgrain mrgrain added status/done Implementation complete and removed status/proposed Newly proposed RFC labels Oct 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ops Operations status/done Implementation complete testing Testing
Projects
None yet
Development

No branches or pull requests

7 participants