Custom airgap #4

areed · 2019-08-19T23:12:24Z

All supported versions of Kubernetes, Docker and addons are bundled into
individual tar archives in CI and uploaded to S3. When a new installer
is created, the server will create a Job to generate an airgap bundle.
The server passes the packages required for the installer to the Job
along with a signed URL for S3 where it can upload the bundle. The Job
is a simple bash script that downloads all the packages and reassembles
into a single archive.

The online install script uses the same packages used in the airgap
bundle but downloads them at runtime. The online server redirects all
dist/ (pacakge) and bundle/ requests to S3.

Future work:

For new named installers the bundle may exist with a different ID. It
would be more efficient to re-use bundles when available.
After POSTing a new installer it takes a minute for the airgap bundle
to be ready.
There's no reconciler that detects failed airgap bundles and retries
or escalates.
No garbage collection for ConfigMaps and Jobs.

All supported versions of Kubernetes, Docker and addons are bundled into individual tar archives in CI and uploaded to S3. When a new installer is created, the server will create a Job to generate an airgap bundle. The server passes the packages required for the installer to the Job along with a signed URL for S3 where it can upload the bundle. The Job is a simple bash script that downloads all the packages and reassembles into a single archive. The online install script uses the same packages used in the airgap bundle but downloads them at runtime. The online server redirects all dist/ (pacakge) and bundle/ requests to S3. Future work: - For new named installers the bundle may exist with a different ID. It would be more efficient to re-use bundles when available. - After POSTing a new installer it takes a minute for the airgap bundle to be ready. - There's no reconciler that detects failed airgap bundles and retries or escalates.

areed · 2019-08-21T13:57:20Z

bin/create-bundle-alpine.sh

@@ -0,0 +1,24 @@
+#!/bin/sh


This is the script that creates custom airgap bundles at runtime. The server adds it to a ConfigMap that the Job mounts. The server then passes the signed PUT url as the first arg and a list of URLs to individual packages to include.

areed · 2019-08-21T13:57:54Z

bin/docker-save.sh

@@ -0,0 +1,15 @@
+#!/bin/bash


This runs in CI to save docker images as tar archives so they can be included in individual packages.

areed · 2019-08-21T13:58:45Z

bin/upload-dist.sh

@@ -0,0 +1,13 @@
+#!/bin/bash


This uploads all the individual packages to S3 during CI. Online installs download at runtime and airgap bundles download these when the Job runs.

areed · 2019-08-21T13:59:15Z

bin/upload-latest-bundle.sh

@@ -0,0 +1,39 @@
+#!/bin/bash


This runs in CI to create the special airgap bundle named latest.

areed · 2019-08-21T13:59:48Z

kustomize/base/namespace.yaml

@@ -0,0 +1,4 @@
+apiVersion: v1


All the airgap Jobs run in this namespace so it's easier to keep track of what's been created.

areed · 2019-08-21T14:00:26Z

packages/kubernetes/1.15.0/Manifest

@@ -0,0 +1,4 @@
+kubernetes gcr.io/google-containers/hyperkube:v1.15.0


List of images to include in the package. This is parsed in CI.

areed · 2019-08-21T14:00:58Z

scripts/Manifest

@@ -0,0 +1,8 @@
+KUBERNETES_VERSION=1.15.2


Latest manifest. This is turned into a template for the server to serve other install scripts.

areed · 2019-08-21T14:02:23Z

web/src/controllers/Scripts.ts

-      installer = i;
-    } catch (error) {
-      return { error };
+      installer = installer.resolve();


replace "latest" with actual versions

areed · 2019-08-21T14:03:40Z

web/src/kubernetes/index.ts

+    this.bucket = process.env["KURL_BUCKET"] || "kurl-dev";
+  }
+
+  public async runCreateAirgapBundleJob(i: Installer) {


First create a ConfigMap with three scripts:

the bash script to rebundle all the individual packages

the rendered install.sh script

the rendered join.sh script

divolgin · 2019-08-21T14:53:10Z

web/src/client/index_test.ts

+    });
+
+    it("returns yaml with version", async () => {
+      const yaml = await client.getInstallerYAML("latest", true);


doesn't look like that second parameter is used in getInstallerYAML

I forgot to implement that in the client. Added now.

divolgin · 2019-08-21T15:12:42Z

web/src/kubernetes/index.ts

+
+    const { signedUrl } = await this.s3Signer.signPutRequest({
+      Bucket: this.bucket,
+      Key: `bundle/${i.id}.tar.gz`,


is it worth checking here if the file already exists in s3 and then just not running the job if it does?

For named installers belonging to teams the bundle will need to be replaced.

divolgin · 2019-08-21T15:14:40Z

web/src/util/persistence/s3.ts

+  public parse(url: string): GetParams {
+    const parsed = new URL(url);
+    const Bucket = parsed.host.split(".")[0];
+    const Key = parsed.pathname.replace(/\//, "");


not sure if this function is used, and what the purpose of this is, but slashes are an important part of the key in S3.

not used, I'll delete

Andrew Reed added 4 commits August 19, 2019 16:02

tar archive for each component

7249ba9

online serve packages from S3 redirect

f72ba1b

shell job to build airgap bundles

00e142c

areed requested a review from divolgin August 19, 2019 23:15

areed commented Aug 21, 2019

View reviewed changes

bin/upload-latest-bundle.sh

@@ -0,0 +1,39 @@

#!/bin/bash

Copy link

Contributor Author

areed Aug 21, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This runs in CI to create the special airgap bundle named latest.

areed commented Aug 21, 2019

View reviewed changes

divolgin reviewed Aug 21, 2019

View reviewed changes

divolgin approved these changes Aug 21, 2019

View reviewed changes

areed merged commit 383a4e5 into replicatedhq:master Aug 21, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Custom airgap #4

Custom airgap #4

areed commented Aug 19, 2019 •

edited

Loading

areed Aug 21, 2019

areed Aug 21, 2019

areed Aug 21, 2019

areed Aug 21, 2019

areed Aug 21, 2019

areed Aug 21, 2019

areed Aug 21, 2019

areed Aug 21, 2019

areed Aug 21, 2019

divolgin Aug 21, 2019

areed Aug 21, 2019

divolgin Aug 21, 2019

areed Aug 21, 2019

divolgin Aug 21, 2019

areed Aug 21, 2019

		@@ -0,0 +1,4 @@
		kubernetes gcr.io/google-containers/hyperkube:v1.15.0

Custom airgap #4

Custom airgap #4

Conversation

areed commented Aug 19, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

areed commented Aug 19, 2019 •

edited

Loading