[alpine] Upgrade 3.11.6 -> 3.12.1, [docs] Customization (cloudposse#639)

.github/CODEOWNERS

@@ -19,3 +19,4 @@
 
 # Of course, admins must approve changes to CODEOWNERS itself
 .github/CODEOWNERS  @cloudposse/admins
+.github/workflows/validate-codeowners.yml   @cloudposse/admins

.github/workflows/docker.yml

@@ -39,7 +39,7 @@ jobs:
           LATEST_TAGS="$TAGS"
         fi
         echo ::set-output name=tags::${TAGS}
-        echo ::set-output name=latest_tags::${TAGS}
+        echo ::set-output name=latest_tags::${LATEST_TAGS}
     - name: Set up Docker Buildx
       uses: docker/setup-buildx-action@v1
     - name: Login to DockerHub
@@ -60,5 +60,5 @@ jobs:
       uses: docker/build-push-action@v2
       with:
         push: ${{ (github.event_name == 'release' && github.event.action == 'created') || github.event.pull_request.head.repo.full_name == github.repository }}
-        tags: ${{ format(steps.prepare.outputs.latest_tags, 'debian') }}
+        tags: ${{ format(steps.prepare.outputs.tags, 'debian') }}
         file: ./os/debian/Dockerfile.debian

Dockerfile.options

@@ -9,6 +9,18 @@ ENV MOTD_URL=http://geodesic.sh/motd
 # other control codes that can cause `less` to mess up the screen formatting
 ENV LESS=R
 
+# Our older Geodesic configurations relied on `direnv`, which we no longer recommend,
+# preferring YAML configuration files instead.
+ENV DIRENV_ENABLED=true
+# Our older Geodesic configuration uses multiple Makefiles, like Makefile.tasks
+# and depends on this setting, however this setting is set by default by `direnv`
+# due to rootfs/conf/.envrc, but `direnv` is now disabled by default, too.
+# If you are using (and therefore enable) `direnv`, consider the advantage
+# of using `direnv` to set MAKE_INCLUDES, which is that it will only set
+# it for trusted directories under `/conf` and therefore it will not affect
+# `make` outside of this directory tree.
+ENV MAKE_INCLUDES="Makefile Makefile.*"
+
 #
 # Configure aws-okta to easily assume roles
 #

README.md

@@ -106,6 +106,8 @@ Let's roll...
 
 
 
+### Customizing your Docker image
+
 In general we recommend creating a customized version of Geodesic by creating your own Dockerfile starting with
 ```
 ARG VERSION=0.138.0
@@ -165,6 +167,14 @@ RUN apt-get update && apt-get install -y google-cloud-sdk=300.0.0-0
 Note the `-y` flag to `apt-get install`. That is required for scripted installation, otherwise the command
 will ask for confirmation from the keyboard before installing a package.
 
+### Customizing your shell at launch time
+
+After you have build your Docker image, or if you are using a shared Docker image, you can
+add further customization at launch time. When Geodesic stars up, it looks for customization
+scripts and configuration so you can do things like add command aliases or override preconfigured options.
+Detailed information about launch-time configuration is in the [customization](./docs/customization.md)
+document, available from within the shell via `man customization`.
+
 
 
 

README.yaml

@@ -86,6 +86,8 @@ introduction: |-
   4. Quickly provision [kops clusters](docs/kops.md)
 
 usage: |-
+  ### Customizing your Docker image
+
   In general we recommend creating a customized version of Geodesic by creating your own Dockerfile starting with
   ```
   ARG VERSION=0.138.0
@@ -145,6 +147,14 @@ usage: |-
   Note the `-y` flag to `apt-get install`. That is required for scripted installation, otherwise the command
   will ask for confirmation from the keyboard before installing a package.
 
+  ### Customizing your shell at launch time
+
+  After you have build your Docker image, or if you are using a shared Docker image, you can
+  add further customization at launch time. When Geodesic stars up, it looks for customization
+  scripts and configuration so you can do things like add command aliases or override preconfigured options.
+  Detailed information about launch-time configuration is in the [customization](./docs/customization.md)
+  document, available from within the shell via `man customization`.
+
 # Other files to include in this README from the project folder
 include:
 

docs/customization.md

@@ -0,0 +1,109 @@
+---
+title: Customization(5) | Geodesic
+author:
+- Nuru
+date: March 2019
+---
+
+## NAME
+
+Customization - How to customize Geodesic at launch time
+
+## SYNOPSIS
+
+Several features of Geodesic can be customized at launch time (rather than
+during the build of the Docker image) so that people can share an image
+yet still have things set up the way they like. This document describes
+how to configure the customization.
+
+## DESCRIPTION
+
+Users can place bash shell scripts on their host computer, to be read either at the start of `bash` profile 
+script processing or at the end of it. These shell scripts can set up environment variables, command
+aliases, shell functions, etc. and through setting environment variables, can cause Geodesic to 
+enable or disable certain features.
+
+Users can also choose whether to have a single `bash` history file 
+for all containers or to have separate history files.
+
+### Root directory for configuration 
+
+All configuration files are stored under `$GEODESIC_CONFIG_HOME`, which defaults to `/localhost/.geodesic`. 
+At this time, `/localhost` is mapped to the host `$HOME` directory and this cannot be configured yet, 
+so all configuration files must be under `$HOME`, but within that limitation they can be placed anywhere. 
+So if you set `$GEODESIC_CONFIG_HOME` to `/localhost/work/config/geodesic`, 
+then files would go in `~/work/config/geodesic/` and below on your Docker host machine.
+
+### Resources
+
+There are currently 3 Resources used for configuration:
+- Preferences, which are shell scripts loaded very early in the launch of the Geodesic shell. 
+- Overrides, which are shell scripts loaded very late in the launch of the Geodesic shell.
+- `bash` history files, which store `bash` command line history.
+
+Additionally, when Geodesic exits normally, it will run the host command `geodesic_on_exit`
+if it is available. This is intended to be a script that you write and install
+anywhere on your PATH to do whatever cleanup you want. For example, change the window title.
+
+Both preferences and overrides can be either a single file, named `preferences` and `overrides` respectively, 
+or can be a collection of files in directories named `preferences.d` and `overrides.d`. 
+If they are directories, all the visible files in the directories will be sourced, 
+except for hidden files and files with names matching the `GEODESIC_AUTO_LOAD_EXCLUSIONS` regex, 
+which defaults to `(~|.bak|.log|.old|.orig|.disabled)$`. 
+
+`bash` history is always stored in a single file named `history`, never a directory of files
+nor files with any other name. If you want to use a separate history file for one
+Geodesic-based Docker image not shared by other Geodesic-based Docker images, you 
+must create an empty `history` file in the image-specific configuration directory (see below).
+
+### Configuration by file placement
+Resources can be in several places, and will be loaded from most general to most specific, according to the name of the docker container image. 
+
+- The most general resources are the ones directly in `$GEODESIC_CONFIG_HOME`. These are applied first. To keep the top-level directory less cluttered and to avoid name clashes, you can put them in a subdirectory named `defaults`. If that subdirectory exists, then `GEODESIC_CONFIG_HOME ` itself is not searched.
+- The `DOCKER_IMAGE` name is then parsed. Everything before the final `/` is considered the "company" name and everything after is, following the Cloudposse reference architecture, referred to as the "stage" name. So for the `DOCKER_IMAGE` name `cloudposse/geodesic`, the company name is `cloudposse` and the stage name is `geodesic`
+- The next place searched for resources is the directory with the same name as the "stage", which is generally the name of the project. In our example, that would be `~/.geodesic/geodesic`. Resources here would apply to all containers with the same base name, perhaps various forks of the same project.
+- The final place searched is the directory with the full name of the Docker image: `$GEODESIC_CONFIG_HOME/$DOCKER_IMAGE`, 
+i.e. `~/.geodesic/cloudposse/geodesic`. Files here are the most specific to this container. 
+
+By loading them in this order, you can put your defaults at one level and then override/customize them at another, minimizing the amount of duplication needed to customize a wide range of containers. 
+
+### Usage details
+Preferences and Overrides are loaded in the order specified above and all that are found are loaded. 
+For history files, only the last one found is used. To start keeping separate history, 
+just create an empty history file in the appropriate place. 
+
+While Preferences and Override files themselves must be `bash` scripts and will be directly loaded into 
+the top-level Geodesic shell, they can of course call other programs. 
+You can even use them to pull configuration out of other places.
+
+Symbolic links must be relative if you want them to work both inside Geodesic and outside of it. 
+Symbolic links that reference directories that are not below `$HOME` on the host will not work.
+
+When possible, Geodesic mounts the host `$HOME` directory as `/localhost` and creates a symbolic link
+from `$HOME` to `/localhost` so that files under `$HOME` on the host can be referenced by the 
+exact same absolute path both on the host computer and inside Geodesic. For example, if the
+host `$HOME` is `/Users/fred`, then `/Users/fred/src/example.sh` will refer to the same file both
+on the host and from inside the Geodesic shell. 
+
+In general, you should put most of your customization in the Preferences files. 
+Geodesic (usually) takes care to respect and adapt to preferences set before it starts adding on top of them. 
+The primary use for overrides is if you need the results of the initialization process as inputs to your configuration, 
+or if you need to undo something Geodesic does not yet provide a configuration option for not doing in the first place. 
+
+## WARNING
+One of the key benefits of Geodesic is that it provides a consistent environment for all users regardless of their 
+local machine. It eliminates the "it works on my machine" excuse. While these customization options can be great 
+productivity enhancements as well as provide the opportunity to install new features to try them out before committing 
+to installing them permanently, they can also create the kind of divergence in environments that brings back 
+the "it works on my machine" problem. 
+
+Therefore, we have included an option to disable the customization files: the preferences, the overrides, 
+and the docker environment files. Simply set and export the host environment variable `$GEODESIC_CUSTOMIZATION_DISABLED` 
+to any value other than "false" before launching Geodesic.
+
+## TROUBLESHOOTING
+If customizations are not being found or are not working as expected, 
+you can set the host enviornment variable `$GEODESIC_TRACE` to "custom" before
+launching Geodesic and a trace of the customization process will be output
+to the console.
+

os/alpine/Dockerfile.alpine

@@ -1,10 +1,12 @@
+ARG ALPINE_VERSION=3.12.1
+ARG GOOGLE_CLOUD_SDK_VERSION=316.0.0
 #
 # Python Dependencies
 #
-FROM alpine:3.11.6 as python
+FROM alpine:$ALPINE_VERSION as python
 
 RUN sed -i 's|http://dl-cdn.alpinelinux.org|https://alpine.global.ssl.fastly.net|g' /etc/apk/repositories
-RUN apk add python3 python3-dev libffi-dev gcc linux-headers musl-dev openssl-dev make
+RUN apk add python3 python3-dev py3-pip libffi-dev gcc linux-headers musl-dev openssl-dev make
 
 ## Note:
 # To install aws-gogle-auth:
@@ -24,12 +26,12 @@ RUN python3 -m pip install --upgrade pip setuptools wheel && \
 #
 # Google Cloud SDK
 #
-FROM google/cloud-sdk:316.0.0-alpine as google-cloud-sdk
+FROM google/cloud-sdk:$GOOGLE_CLOUD_SDK_VERSION-alpine as google-cloud-sdk
 
 #
 # Geodesic base image
 #
-FROM alpine:3.11.6
+FROM alpine:$ALPINE_VERSION
 
 # Set XDG environment variables per https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html
 # This is not a "multi-user" system, so we'll use special directories under
@@ -60,7 +62,7 @@ USER root
 
 # install the cloudposse alpine repository
 ADD https://apk.cloudposse.com/[email protected] /etc/apk/keys/
-RUN echo "@cloudposse https://apk.cloudposse.com/3.11/vendor" >> /etc/apk/repositories
+RUN echo "@cloudposse https://apk.cloudposse.com/3.12/vendor" >> /etc/apk/repositories
 
 # Use TLS for alpine default repos
 RUN sed -i 's|http://dl-cdn.alpinelinux.org|https://alpine.global.ssl.fastly.net|g' /etc/apk/repositories && \
@@ -77,7 +79,7 @@ RUN apk add --update $(grep -h -v '^#' /etc/apk/packages.txt /etc/apk/packages-a
     mkdir -p /etc/bash_completion.d/ /etc/profile.d/ /conf && \
     touch /conf/.gitconfig
 
-RUN [[ $(geodesic-apkindex-md5) == $(cat /tmp/geodesic_apkindex.md5) ]] || echo "WARNING: apk package repos mismatch: '$(geodesic-apkindex-md5)' != '$(cat /tmp/geodesic_apkindex.md5)'" 1>&2
+RUN ["/bin/bash", "-c", "[[ $(geodesic-apkindex-md5) == $(cat /tmp/geodesic_apkindex.md5) ]] || echo \"WARNING: apk package repos mismatch: '$(geodesic-apkindex-md5)' != '$(cat /tmp/geodesic_apkindex.md5)'\" 1>&2"]
 RUN rm -f /tmp/geodesic-apkindex.md5
 
 RUN echo "net.ipv6.conf.all.disable_ipv6=0" > /etc/sysctl.d/00-ipv6.conf
@@ -157,39 +159,10 @@ RUN chmod -R a+rwX ${HELM_HOME}
 # See https://helm.sh/docs/faq/#xdg-base-directory-support
 
 #
-# Install kops
+# We no longer install kops or defaults. See https://github.com/cloudposse/geodesic/blob/master/Dockerfile.options
+# for example settings you can add to your own Dockerfile, or see what we used to set
+# here: https://github.com/cloudposse/geodesic/blob/a7a47a0d3ed558e0f5d1116291b2f2f3aa1a1b97/Dockerfile#L123-L155
 #
-ENV KOPS_MANIFEST=/conf/kops/manifest.yaml
-ENV KOPS_TEMPLATE=/templates/kops/default.yaml
-## Set these to better values in child Dockerfile:
-#ENV KOPS_STATE_STORE s3://undefined
-#ENV KOPS_STATE_STORE_REGION us-east-1
-#ENV KOPS_FEATURE_FLAGS=+DrainAndValidateRollingUpdate
-
-ENV KOPS_BASTION_PUBLIC_NAME="bastion"
-
-# Set the KOPS_BASE_IMAGE to match your kops version. See:
-# https://github.com/kubernetes/kops/blob/master/channels/stable
-# https://github.com/kubernetes/kops/blob/master/docs/images.md
-#
-# Do not rely on KOPS_BASE_IMAGE being set in Geodesic. This will go away in future versions.
-# Set it in your /conf/kops/kops.envrc file, along with KUBERNETES_VERSION
-# ENV KOPS_BASE_IMAGE=kope.io/k8s-1.11-debian-stretch-amd64-hvm-ebs-2018-08-17
-
-ENV KUBECONFIG=/dev/shm/kubecfg
-ENV KUBECONFIG_TEMPLATE=/templates/kops/kubecfg.yaml
-
-RUN /usr/bin/kops completion bash > /etc/bash_completion.d/kops.sh
-
-# Instance sizes
-ENV BASTION_MACHINE_TYPE "t3.small"
-ENV MASTER_MACHINE_TYPE "t3.medium"
-ENV NODE_MACHINE_TYPE "t3.medium"
-
-# Min/Max number of nodes (aka workers)
-ENV NODE_MAX_SIZE 2
-ENV NODE_MIN_SIZE 2
-
 
 # 
 # Install fancy Kube PS1 Prompt
@@ -207,31 +180,52 @@ ENV AWS_SHARED_CREDENTIALS_FILE=${AWS_DATA_PATH}/credentials
 #
 # Configure aws-vault to easily assume roles (not related to HashiCorp Vault)
 #
-ENV AWS_VAULT_ENABLED=true
+ENV AWS_VAULT_ENABLED=false
 ENV AWS_VAULT_SERVER_ENABLED=false
 ENV AWS_VAULT_BACKEND=file
-ENV AWS_VAULT_ASSUME_ROLE_TTL=1h
-ENV AWS_VAULT_SESSION_TTL=12h
-#ENV AWS_VAULT_FILE_PASSPHRASE=
+# ENV AWS_VAULT_ASSUME_ROLE_TTL=1h
+# ENV AWS_VAULT_SESSION_TTL=12h
+# ENV AWS_VAULT_FILE_PASSPHRASE=
 
 #
 # Configure aws-okta to easily assume roles
 #
 ENV AWS_OKTA_ENABLED=false
 
 #
-# Shell
+# Shell customization
 #
-ENV SHELL=/bin/bash
+# options for `less`. `R` allows ANSI color codes to be displayed while stripping out
+# other control codes that can cause `less` to mess up the screen formatting
 ENV LESS=R
+# MANOPT=--no-hyphenation disables hyphenation for man pages, which is generally preferable
+# for the man pages in Geodesic to preserve the ability to copy and paste code.
+ENV MANOPT=--no-hyphenation
+# Support for using an SSH key file on the host. No longer favored now that
+# Docker supports forwarding SSH_AUTH_SOCK, but retained for backward compatibility.
+# See rootfs/etc/profile.d/ssh-agent.sh
 ENV SSH_AGENT_CONFIG=/var/tmp/.ssh-agent
 
-# Set a default terminal to "dumb" (headless) to make `tput` happy
+# Set a default terminal to "dumb" (headless) to make `tput` happy when running scripts.
+# When we launch Geodesic for interactive use, we forward the host value of `TERM`
 ENV TERM=dumb
 
+# Our older Geodesic configurations relied on `direnv`, which we no longer recommend,
+# preferring YAML configuration files instead.
+ENV DIRENV_ENABLED=false
+# Our older Geodesic configuration uses multiple Makefiles, like Makefile.tasks
+# and depends on setting
+# ENV MAKE_INCLUDES="Makefile Makefile.*"
+# but that setting causes problems in other situations, so we no longer set it by default.
+# We encourage you to use https://github.com/cloudposse/atmos instead of `make` for running commands.
+# If you are using (and therefore enable) `direnv`, consider the advantage
+# of using `direnv` to set MAKE_INCLUDES, which is that it will only set
+# it for trusted directories under `/conf` and therefore it will not affect
+# `make` outside of this directory tree.
+
 # Reduce `make` verbosity
 ENV MAKEFLAGS="--no-print-directory"
-ENV MAKE_INCLUDES="Makefile Makefile.*"
+
 # Install "root" filesystem
 COPY rootfs/ /
 COPY os/alpine/rootfs/ /
@@ -240,9 +234,7 @@ COPY os/alpine/rootfs/ /
 COPY docs/ /usr/share/docs/
 
 # Build man pages
-# man-db is not working as of 2020-10-20 with Alpine 3.11, so we are using man
-# TODO revert to RUN /usr/local/bin/docs update
-RUN /usr/local/bin/docs build
+RUN /usr/local/bin/docs update
 
 # Make sure that "user specific" directories we are sharing
 # are in fact available to all users

os/alpine/geodesic_apkindex.md5

@@ -1 +1 @@
-e24a742a565fd319b89fcaae4b428d4b
+9815f7dff09ac69f3a74de50bc6cc2ce

os/alpine/packages-alpine.txt

@@ -7,10 +7,6 @@ keybase-client@testing
 libc6-compat
 libltdl
 make
-# man-db 2.9.3-r2 require libgdbm.so.6 which is not available for Alpine 3.11
-# so we replace it with `man` for now
-# man-db@community
-man
 musl-dev
 mysql-client
 ncurses