xreds

XArray Environmental Data Services

Running Locally

Installing dependencies

System Dependencies

macOS

brew install netcdf4 h5 geos proj eccodes

Python Dependencies

Then install with uv in a virtualenv:

uv venv
source venv/bin/activate
uv pip install -r requirements.txt

Or install with pip in a virtualenv:

virtualenv -p python3 env/
source env/bin/activate
pip install -r requirements.txt

Running the server

Build the react app

cd viewer/
yarn install
yarn build

Run the following in the activated virtualenv:

datasets_mapping_file=./test.json python app.py

Where datasets_mapping_file is the path to the dataset key value store specified in the previous section. You can now navigate to http://localhost:8090/docs to see the supported operations

Running With Docker

Building and Running manually

The docker container for the app can be built with:

docker build -t xreds:latest .

Once built, it requires a few things to be run: The 8090 port to be exposed, and a volume for the datasets to live in, and the environment variable pointing to the dateset json file.

docker run -p 8090:8090 -e "datasets_mapping_file=/path/to/datasets.json" -v "/path/to/datasets:/opt/xreds/datasets" xreds:latest

Running with docker compose

docker compose --platform=linux/amd64 up -d

Specifying Datasets

Datasets are specified in a key value manner, where the keys are the dataset ids and the values are objects with the path and access control info for the datasets:

{
    "gfswave_global": {
        "path": "s3://nextgen-dmac/kerchunk/gfswave_global_kerchunk.json",
        "type": "kerchunk",
        "chunks": {},
        "drop_variables": ["orderedSequenceData"]
    },
    "dbofs": {
        "path": "s3://nextgen-dmac/nos/nos.dbofs.fields.best.nc.zarr",
        "type": "kerchunk",
        "chunks": {
            "ocean_time": 1
        },
        "drop_variables": ["dstart"]
    }
}

Currently zarr, netcdf, and kerchunk dataset types are supported. This information should be saved a file and specified when running.

Building and Deploying Docker Image

First follow instructions above to build the docker image tagged xreds:latest. Then thexreds:latest image needs to be tagged and deployed to the relevant docker registry.

# Auth with ECR
aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws/m2c5k9c1

# Tag the image
docker tag xreds:latest public.ecr.aws/m2c5k9c1/nextgen-dmac/xreds:latest

# Push the image
docker push public.ecr.aws/m2c5k9c1/nextgen-dmac/xreds:latest