-
Notifications
You must be signed in to change notification settings - Fork 11
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
New PR for GHA #289
Merged
Merged
New PR for GHA #289
Changes from all commits
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
3529ec9
add docker workflow
swelborn 3394df8
squashed from a lot of CI triggers
swelborn 624bd14
add scripts to build locally
swelborn 7bf2b95
remove old dockerfiles
swelborn 8281dff
build on master branch
swelborn 268e86c
install stempy on build
swelborn 240fa75
Merge remote-tracking branch 'upstream/master' into gha-final
swelborn ec0d976
add mpi4py to requirements
swelborn File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
/docs/ | ||
/.github/ | ||
/examples/ | ||
/LICENSE | ||
/.readthedocs.yml | ||
/.gitignore |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,32 @@ | ||
name: 'Setup Workflow' | ||
description: 'Set up environment and Docker settings' | ||
|
||
inputs: | ||
github_event_name: | ||
description: 'GitHub Event Name' | ||
required: true | ||
DOCKERHUB_USERNAME: | ||
description: 'DockerHub Username' | ||
required: true | ||
DOCKERHUB_TOKEN: | ||
description: 'DockerHub Token' | ||
required: true | ||
DOCKERHUB_ORG: | ||
description: 'DockerHub Organization' | ||
required: true | ||
|
||
runs: | ||
using: 'composite' | ||
steps: | ||
- name: Set DOCKERHUB_ORG | ||
run: | | ||
echo "DOCKERHUB_ORG=${{ inputs.DOCKERHUB_ORG == '' && 'openchemistry' || inputs.DOCKERHUB_ORG }}" >> $GITHUB_ENV | ||
shell: bash | ||
- name: Login to Docker Hub | ||
uses: docker/login-action@v2 | ||
if: ${{ github.event_name != 'pull_request' }} | ||
with: | ||
username: ${{ inputs.DOCKERHUB_USERNAME }} | ||
password: ${{ inputs.DOCKERHUB_TOKEN }} | ||
- name: Set up Docker Buildx | ||
uses: docker/setup-buildx-action@v2 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,137 @@ | ||
name: docker | ||
|
||
on: | ||
push: | ||
branches: | ||
- 'master' | ||
pull_request: | ||
branches: | ||
- 'master' | ||
|
||
jobs: | ||
# Builds base image for stempy of various verions if Dockerfile.base has changed | ||
build-stempy: | ||
runs-on: ubuntu-latest | ||
strategy: | ||
fail-fast: false | ||
matrix: | ||
python-version: ['3.9'] | ||
mpi: ['ON', 'OFF'] | ||
ipykernel: ['', 'ipykernel'] | ||
dev: ['', 'dev'] | ||
exclude: | ||
- MPI: 'ON' | ||
ipykernel: 'ipykernel' | ||
|
||
steps: | ||
- name: Checkout | ||
uses: actions/checkout@v3 | ||
with: | ||
fetch-depth: 0 | ||
submodules: 'recursive' | ||
|
||
- name: 'Setup Docker Environment' | ||
uses: ./.github/actions/docker_setup | ||
with: | ||
github_event_name: ${{ github.event_name }} | ||
DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }} | ||
DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }} | ||
DOCKERHUB_ORG: ${{ vars.DOCKERHUB_ORG }} | ||
|
||
- name: Determine if Dockerfile.base changed | ||
id: changed-dockerfile-base | ||
uses: tj-actions/changed-files@v35 | ||
with: | ||
files: | | ||
./docker/Dockerfile.base | ||
./docker/apt-packages-common.txt | ||
./docker/apt-packages-dev.txt | ||
|
||
- name: Set up environment variables for python version | ||
id: set-conda-env | ||
run: | | ||
echo "PYTHON_VERSION=${{ matrix.python-version }}" >> $GITHUB_ENV | ||
PYTHON_VERSION_NODOT=$(echo ${{ matrix.python-version }} | tr -d .) | ||
echo "PYTHON_VERSION_NODOT=${PYTHON_VERSION_NODOT}" >> $GITHUB_ENV | ||
- name: Set up other environment variables | ||
id: set-tag-vars | ||
run: | | ||
echo "BASE_TARGET=$(if [[ "${{ matrix.mpi }}" == 'ON' ]]; then | ||
echo 'mpi' | ||
else | ||
echo 'base' | ||
fi)" >> $GITHUB_ENV | ||
|
||
MPI_TAG=$(if [[ "${{ matrix.mpi }}" == 'ON' ]]; then | ||
echo '-mpi' | ||
else | ||
echo '' | ||
fi) | ||
echo "MPI_TAG=${MPI_TAG}" >> $GITHUB_ENV | ||
|
||
DEV_TAG=$(if [[ "${{ matrix.dev }}" == 'dev' ]]; then | ||
echo '-dev' | ||
else | ||
echo '' | ||
fi) | ||
echo "DEV_TAG=${DEV_TAG}" >> $GITHUB_ENV | ||
|
||
echo "RELEASE_OR_DEBUG=$(if [[ "${{ matrix.dev }}" == 'dev' ]]; then | ||
echo 'Debug' | ||
else | ||
echo 'Release' | ||
fi)" >> $GITHUB_ENV | ||
|
||
COMMIT=$(git rev-parse --short HEAD) | ||
echo "COMMIT=${COMMIT}" >> $GITHUB_ENV | ||
|
||
IPYKERNEL_TAG=$(if [[ "${{ matrix.ipykernel }}" == 'ipykernel' ]]; then | ||
echo '-ipykernel' | ||
else | ||
echo '' | ||
fi) | ||
|
||
echo "IPYKERNEL_TAG=${IPYKERNEL_TAG}" >> $GITHUB_ENV | ||
|
||
BASE_TAG=${{ env.DOCKERHUB_ORG }}/stempy${MPI_TAG}:py${{ env.PYTHON_VERSION_NODOT }}-base${DEV_TAG} | ||
echo "BASE_TAG=${BASE_TAG}" >> $GITHUB_ENV | ||
|
||
TAG=${{ env.DOCKERHUB_ORG }}/stempy${MPI_TAG}${IPYKERNEL_TAG}:py${{ env.PYTHON_VERSION_NODOT }}-${COMMIT}${DEV_TAG} | ||
echo "TAG=${TAG}" >> $GITHUB_ENV | ||
|
||
LATEST_TAG=${{ env.DOCKERHUB_ORG }}/stempy${MPI_TAG}${IPYKERNEL_TAG}:latest${DEV_TAG} | ||
echo "LATEST_TAG=${LATEST_TAG}" >> $GITHUB_ENV | ||
Comment on lines
+58
to
+103
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If you have more than a few lines in a |
||
|
||
- name: Build/push Dockerfile.base | ||
uses: docker/build-push-action@v3 | ||
if: ${{ contains(github.event.head_commit.message, 'trigger-ci') || steps.changed-dockerfile-base.outputs.any_changed == 'true'}} | ||
with: | ||
context: . | ||
file: ./docker/Dockerfile.base | ||
push: ${{ github.event_name != 'pull_request' }} | ||
build-args: | | ||
PYTHON_VERSION=${{ env.PYTHON_VERSION }} | ||
DEV=${{ matrix.dev }} | ||
tags: ${{ env.BASE_TAG }} | ||
target: ${{ env.BASE_TARGET }} | ||
cache-to: type=gha, mode=max | ||
cache-from: type=gha, mode=max | ||
|
||
- name: Build/push Dockerfile.stempy | ||
uses: docker/build-push-action@v3 | ||
with: | ||
context: . | ||
file: ./docker/Dockerfile.stempy | ||
push: ${{ github.event_name != 'pull_request' }} | ||
build-args: | | ||
PYTHON_VERSION=${{ env.PYTHON_VERSION }} | ||
BASE_IMAGE=${{ env.BASE_TAG }} | ||
RELEASE_OR_DEBUG=${{ env.RELEASE_OR_DEBUG }} | ||
MPI=${{ matrix.mpi }} | ||
IPYKERNEL=${{ matrix.ipykernel }} | ||
tags: | | ||
${{ env.TAG }} | ||
${{ env.LATEST_TAG }} | ||
target: build | ||
cache-to: type=gha, mode=max | ||
cache-from: type=gha, mode=max |
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,47 @@ | ||
ARG PYTHON_VERSION | ||
FROM python:${PYTHON_VERSION} as base | ||
|
||
ENV DEBIAN_FRONTEND noninteractive | ||
WORKDIR /build | ||
|
||
# Install common packages | ||
COPY ./docker/apt-packages-common.txt /tmp/apt-packages-common.txt | ||
|
||
# Install dev packages if DEV is set | ||
COPY ./docker/apt-packages-dev.txt /tmp/apt-packages-dev.txt | ||
|
||
ARG DEV | ||
RUN if [ "$DEV" = "dev" ]; then \ | ||
cat /tmp/apt-packages-common.txt /tmp/apt-packages-dev.txt > /tmp/apt-packages.txt; \ | ||
else \ | ||
cp /tmp/apt-packages-common.txt /tmp/apt-packages.txt; \ | ||
fi && \ | ||
apt-get update && \ | ||
apt-get upgrade --yes && \ | ||
apt-get install --yes \ | ||
$(cat /tmp/apt-packages-common.txt) && \ | ||
apt-get clean all && \ | ||
rm -rf /var/lib/apt/lists/* | ||
|
||
FROM base as mpi | ||
|
||
# Build mpich | ||
ARG mpich=4.0.2 | ||
ARG mpich_prefix=mpich-$mpich | ||
RUN wget https://www.mpich.org/static/downloads/$mpich/$mpich_prefix.tar.gz && \ | ||
tar xvzf $mpich_prefix.tar.gz -C /build && \ | ||
rm -rf $mpich_prefix.tar.gz && \ | ||
cd /build/$mpich_prefix && \ | ||
./configure --disable-f77 --disable-fc --disable-fortran && \ | ||
make -j 16 && \ | ||
make install && \ | ||
make clean && \ | ||
cd .. && \ | ||
rm -rf $mpich_prefix | ||
|
||
# Build BigMPI | ||
RUN cd /build && wget https://github.com/jeffhammond/BigMPI/archive/refs/heads/master.tar.gz && \ | ||
tar zxvf master.tar.gz && cd /build/BigMPI-master && \ | ||
./autogen.sh && ./configure --with-pic && make -j4 && make install && make clean && rm /build/master.tar.gz | ||
|
||
RUN /sbin/ldconfig |
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,37 @@ | ||
ARG BASE_IMAGE | ||
FROM ${BASE_IMAGE} as build | ||
|
||
ENV DEBIAN_FRONTEND noninteractive | ||
|
||
ARG PYTHON_VERSION | ||
ARG IPYKERNEL | ||
ARG MPI | ||
ARG RELEASE_OR_DEBUG | ||
|
||
COPY . /source/stempy/ | ||
|
||
RUN pip install -r /source/stempy/requirements.txt | ||
|
||
RUN mkdir -p /build/stempy && \ | ||
cd /build/stempy && \ | ||
cmake -DCMAKE_BUILD_TYPE:STRING=${RELEASE_OR_DEBUG} \ | ||
-Dstempy_ENABLE_VTKm:BOOL=OFF \ | ||
-Dstempy_ENABLE_MPI:BOOL=${MPI} \ | ||
-DBIGMPI_INCLUDE_DIR:PATH=/usr/local/include \ | ||
-DBIGMPI_LIBRARY:PATH=/usr/local/lib/libbigmpi.a \ | ||
/source/stempy . && \ | ||
cmake --build . && \ | ||
cmake --install . && \ | ||
cp -r -L /build/stempy/lib/stempy \ | ||
/usr/local/lib/python${PYTHON_VERSION}/site-packages | ||
|
||
RUN if [ "${IPYKERNEL}" = "ipykernel" ]; then \ | ||
pip install -r /source/stempy/docker/requirements-ipykernel.txt; \ | ||
elif [ "${MPI}" = "ON" ]; then \ | ||
pip install -r /source/stempy/docker/requirements-mpi.txt; \ | ||
else \ | ||
pip install -r /source/stempy/docker/requirements-normal.txt; \ | ||
fi && \ | ||
rm -rf /source/stempy | ||
|
||
RUN /sbin/ldconfig |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,20 @@ | ||
apt-transport-https | ||
autoconf | ||
automake | ||
ca-certificates | ||
gcc | ||
g++ | ||
git | ||
gnupg | ||
libarchive-dev | ||
libeigen3-dev | ||
libffi-dev | ||
libhdf5-dev | ||
liblapack-dev | ||
libopenblas-dev | ||
libsqlite3-dev | ||
libtool | ||
make | ||
software-properties-common | ||
wget | ||
zlib1g-dev |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does the
Dockerfile.base
take a long time to build? If not, it could have been simpler to just build it every time, rather than caching it. We may actually want to rebuild it occasionally just because upstream dependencies change over time which include things like security updates.But we can leave this!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Dockerfile.base does take a long time to build (~40 minutes). I understand that it is CI, but say you are developing on a fork and want to make a quick change and test at NERSC.
All you have to do is edit this:
And commit your change, and CI will take care of building/pushing to your own Dockerhub repo (so long as you have set up these vars and secrets):
I can set up a cron job to build/push base, I don't think that would be too much work.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the info! It's probably okay for now unless @cjh1 thinks we should add a cron job.