Runtime of CircleCI jobs exceed the limit for the free tier

[geedo0]

CircleCI's free tier limits a job's run time to 1 hour doc. The CI checks run by OpenSSH have been approaching that 1 hour limit and are now at the point where they just barely run past that limit and fail the CI checks example. Previous runs were already pretty close to the limit, the addition of new algorithms to test was enough to tip the run-time over the limit.

There's no solution that doesn't come with trade-offs, but here are some possibilities:

• Migrate to a new CI framework like Github Actions. This seems to be the right choice, but I don't have the context on why CircleCI was chosen in the first place.
• Upgrade to the paid Performance Plan pricing.
• Audit the unit tests and skip tests that have long running times and aren't particularly interesting for OQS.
• Look into refactoring the tests into parallel jobs that fit within the limits.

[geedo0]

@SWilson4 Here's that CircleCI issue I mentioned. I've got a first pass at porting it to Github Actions here. However, there's some difference in how the container/host handles process reaping which breaks a unit test, but I can patch around that.

name: CI Checks
on: [ push, pull_request ]
jobs:
  ubuntu_build:
    runs-on: ubuntu-latest
    container:
      image: openquantumsafe/ci-ubuntu-focal-x86_64:latest
    steps:
    - uses: actions/checkout@v4
    - name: Set up SSH environment
      run: |
        mkdir -p -m 0755 /var/empty
        groupadd sshd
        useradd -g sshd -c 'sshd privsep' -d /var/empty -s /bin/false sshd
    - name: Clone liboqs
      run: ./oqs-scripts/clone_liboqs.sh
    - name: Build liboqs
      run: ./oqs-scripts/build_liboqs.sh
    - name: Build OpenSSH
      run: env WITH_OPENSSL=true ./oqs-scripts/build_openssh.sh
    - name: Run tests documented to pass
      run: ./oqs-test/run_tests.sh
    - name: Ensure we have the ssh and sshd syntax right once for each algorithm
      run: python3 oqs-test/try_connection.py doone

[SWilson4]

That config looks reasonable to me. Would it help if I made a draft PR to move the liboqs downstream trigger from OQS-v8 (on CircleCI) to OQS-v9 (on GitHub Actions)?

[geedo0]

Yeah, that'll help. I can put together the PR in OpenSSH to get Github Actions working.

[baentsch]

Migrate to a new CI framework like Github Actions.

That's the way to go -- we already did in liboqs (thanks to @SWilson4) and plan doing elsewhere too, e.g., open-quantum-safe/oqs-provider#248.

why CircleCI was chosen in the first place

That was because at the time it had support for platforms other CI systems didn't. For oqs-openssh this is irrelevant as a) GH CI supports more platforms by now and b) there's (currently) no PLATFORMS.md (guaranteed supported systems) this sub project "warrants", so let's get off CCI here rather than spend time working around its quirks.

[SWilson4]

Merging open-quantum-safe/liboqs#1898 will complete the move to GitHub Actions.

[geedo0]

Thanks everyone for quickly driving this to completion!

[baentsch]

Thanks everyone for quickly driving this to completion!

The reverse is more due: Thanks for the quick and thorough update (really, upgrade) to oqs-openssh, @geedo0 !