Sign k6 requests with HMAC to enable WAF bypass

[sarayourfriend]

Fixes

Fixes https://github.com/WordPress/openverse-infrastructure/issues/1031 by @obulat

Description

This PR adds signing requests with an HMAC to enable bypassing WAF rules, so that k6 requests do not get rate limited.

Testing Instructions

Follow the test instructions in this infrastructure PR: https://github.com/WordPress/openverse-infrastructure/pull/1048

These must be tested together.

You should also be able to run the k6 tests against your local frontend, without passing -e signing_secret. Run your frontend with ov j p frontend prod and then in another terminal, run the k6 tests:

ov j k6 frontend static-en -e FRONTEND_URL=https://staging.openverse.org/ -e scenario_vus=1 -e scenario_iterations=1

Checklist

• My pull request has a descriptive title (not a vague title likeUpdate index.md).
• My pull request targets the default branch of the repository (main) or a parent feature branch.
• My commit messages follow best practices.
• My code follows the established code style of the repository.
• I added or updated tests for the changes I made (if applicable).
• I added or updated documentation (if applicable).
• I tried running the project locally and verified that there are no visible errors.
• [N/A] I ran the DAG documentation generator (ov just catalog/generate-docs for catalog
  PRs) or the media properties generator (ov just catalog/generate-docs media-props
  for the catalog or ov just api/generate-docs for the API) where applicable.

Developer Certificate of Origin

Developer Certificate of Origin

Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.


Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

[sarayourfriend]

The infrastructure side of things is underway in this draft PR: https://github.com/WordPress/openverse-infrastructure/pull/1048

Nothing is working yet... 🙂 but it will soon!

[sarayourfriend]

New rollup plugins used to support bundling npm dependencies, because k6 doesn't support them unless they are bundled in. In our case, we need it for core-js to provide a sensible URL API.

I chose this as an alternative to the jslib URL implementation because (a) that still requires importing URL and (b) there are no means to support types for it. The suggested approach for supporting types from k6 jslib libraries is to vendor the library code.

I did not try declaring the types using declare module "...jslib..." so I'll try that for good measure (which would be nice for the other jslib function we use).

However, I've found that so far the k6 and jslib implementations of certain Web APIs have very subtle and annoying differences to the specification, which mostly just causes a bit of mental overload trying to keep track of. Which is to say, it's nice to be able to bundle in polyfills from core-js that will behave according to the specification!

[sarayourfriend]

k6-jslib-url just exports core-js: https://github.com/grafana/k6-jslib-url/blob/main/index.src.js

For the sake of making sure bundling in node dependencies is sorted for anyone else working in these tests, I think it's best to go ahead and use core-js directly in this PR, as a means of integrating the bundling process before it's needed otherwise.

[sarayourfriend]

I'm fairly certain the issue with this is on the Cloudflare side with my usage of their hmac validation function (made a slight error). Excited to test later today and maybe get this in a state where we can start running these after each staging deploy starting next week 😁

cc @obulat who created the issue and I suppose will be interested to see this become available.

Once this HMAC pattern is established, we can use it to run Playwright tests as well #4706

[sarayourfriend]

Putting this back as draft because I realised I still had a todo comment here, and can undo the async redactor now that I've switched this back to using the k6/crypto library instead of WebCryptoAPI. K6 really isn't meant for async yet, so its okay if we stick to sync code here for now.

Aside from that, if you're reading this and care to test the WAF bypass bit out, that's all working well.

I also decided to leave the GitHub Workflow bits out of this PR to avoid it getting too big/complex... There are a couple of things I need to think about for those. Main thing being... We should run all test scenarios in CI/CD anytime something relevant to a test namespace changes, which is also important to avoid pushing changes to the load tests to main that haven't at least run against the local frontend in the PR CI checks.

Anyway, that will be a separate PR. This one will stay scoped to only getting the HMAC signing working... Which it does! Just need to do a bit more cleanup/undo some of to unnecessary changes in this PR.

Update: this is ready for review now 🙂 Check out the PR description for testing instructions.

[obulat]

I'm going to review this PR tomorrow morning

[openverse-bot]

Based on the medium urgency of this PR, the following reviewers are being gently reminded to review this PR:

@obulat
@zackkrida
This reminder is being automatically generated due to the urgency configuration.

Excluding weekend¹ days, this PR was ready for review 4 day(s) ago. PRs labelled with medium urgency are expected to be reviewed within 4 weekday(s)².

@sarayourfriend, if this PR is not ready for a review, please draft it to prevent reviewers from getting further unnecessary pings.

Footnotes

1. Specifically, Saturday and Sunday. ↩

2. For the purpose of these reminders we treat Monday - Friday as weekdays. Please note that the operation that generates these reminders runs at midnight UTC on Monday - Friday. This means that depending on your timezone, you may be pinged outside of the expected range. ↩

[zackkrida]

This whole file is great 👨‍🍳 💋

[zackkrida]

Random thought but it'd be kinda cool to make folks co-authors of the PR when we reuse code like this! The comment is totally sufficient, of course.

[sarayourfriend]

Hmmm, I'd be worried about going to that extent, it might imply they were involved in the PR more broadly, because we squash commits onto main? I'd be worried about doing that without their consent.

[zackkrida]

LGTM. Verified against staging and the local frontend.