Skip to content

Latest commit

 

History

History
601 lines (422 loc) · 24 KB

content.md

File metadata and controls

601 lines (422 loc) · 24 KB
Raw

InfluxDB

InfluxDB is a time series database built from the ground up to handle high write and query loads. InfluxDB is meant to be used as a backing store for any use case involving large amounts of timestamped data, including DevOps monitoring, application metrics, IoT sensor data, and real-time analytics.

InfluxDB Documentation

%%LOGO%%

latest updated to InfluxDB 2.x

The latest tag for this image now points to the latest released implementation of InfluxDB 2.x. If you are using the latest tag and would like to stay on the InfluxDB 1.x line, please update your environment to reference the 1.8 tag.

Using this Image - InfluxDB 2.x

Upgrading from InfluxDB 1.x

InfluxDB 2.x provides a 1.x-compatible API, but expects a different storage layout on disk. To bridge this mismatch, the InfluxDB image contains extra functionality to migrate 1.x data and config into 2.x layouts automatically before booting the influxd server.

The automated upgrade process bootstraps an initial admin user, organization, and bucket in the system. Additional environment variables are used to configure the setup logic:

  • DOCKER_INFLUXDB_INIT_USERNAME: The username to set for the system's initial super-user (Required).
  • DOCKER_INFLUXDB_INIT_PASSWORD: The password to set for the system's inital super-user (Required).
  • DOCKER_INFLUXDB_INIT_ORG: The name to set for the system's initial organization (Required).
  • DOCKER_INFLUXDB_INIT_BUCKET: The name to set for the system's initial bucket (Required).
  • DOCKER_INFLUXDB_INIT_RETENTION: The duration the system's initial bucket should retain data. If not set, the initial bucket will retain data forever.
  • DOCKER_INFLUXDB_INIT_ADMIN_TOKEN: The authentication token to associate with the system's initial super-user. If not set, a token will be auto-generated by the system.

It also requires extra volumes to be mounted into the 2.x container:

  • Data from the 1.x instance
  • Custom config from the 1.x instance (if any)

The upgrade process searches for mounted 1.x data / config in this priority order:

  1. A config file referred to by the DOCKER_INFLUXDB_INIT_UPGRADE_V1_CONFIG environment variable
  2. A data directory referred to by the DOCKER_INFLUXDB_INIT_UPGRADE_V1_DIR environment variable
  3. A config file mounted at /etc/influxdb/influxdb.conf
  4. A data directory mounted at /var/lib/influxdb

Finally, the DOCKER_INFLUXDB_INIT_MODE environment variable must be set to upgrade.

Automated upgrade will generate both data and config files, by default under /var/lib/influxdb2 and /etc/influxdb2. It's recommended to mount volumes at both paths to avoid losing data.

NOTE: Automated upgrade will not run if an existing boltdb file is found at the configured path. This behavior allows for the InfluxDB container to reboot post-upgrade without overwriting migrated data.

Find more about the InfluxDB upgrade process here. See below for examples of common upgrade scenarios.

Upgrade Example - Minimal

Assume you've been running a minimal InfluxDB 1.x deployment:

$ docker run -p 8086:8086 \
      -v influxdb:/var/lib/influxdb \
      %%IMAGE%%:1.8

To upgrade this deployment to InfluxDB 2.x, stop the running InfluxDB 1.x container, then run:

$ docker run -p 8086:8086 \
      -v influxdb:/var/lib/influxdb \
      -v influxdb2:/var/lib/influxdb2 \
      -e DOCKER_INFLUXDB_INIT_MODE=upgrade \
      -e DOCKER_INFLUXDB_INIT_USERNAME=my-user \
      -e DOCKER_INFLUXDB_INIT_PASSWORD=my-password \
      -e DOCKER_INFLUXDB_INIT_ORG=my-org \
      -e DOCKER_INFLUXDB_INIT_BUCKET=my-bucket \
      %%IMAGE%%:2.0

Upgrade Example - Custom InfluxDB 1.x Config

Assume you've been running an InfluxDB 1.x deployment with customized config:

$ docker run -p 8086:8086 \
      -v influxdb:/var/lib/influxdb \
      -v $PWD/influxdb.conf:/etc/influxdb/influxdb.conf \
      %%IMAGE%%:1.8

To upgrade this deployment to InfluxDB 2.x, stop the running container, then run:

$ docker run -p 8086:8086 \
      -v influxdb:/var/lib/influxdb \
      -v influxdb2:/var/lib/influxdb2 \
      -v influxdb2-config:/etc/influxdb2 \
      -v $PWD/influxdb.conf:/etc/influxdb/influxdb.conf \
      -e DOCKER_INFLUXDB_INIT_MODE=upgrade \
      -e DOCKER_INFLUXDB_INIT_USERNAME=my-user \
      -e DOCKER_INFLUXDB_INIT_PASSWORD=my-password \
      -e DOCKER_INFLUXDB_INIT_ORG=my-org \
      -e DOCKER_INFLUXDB_INIT_BUCKET=my-bucket \
      %%IMAGE%%:2.0

Upgrade Example - Custom Paths

Assume you've been running an InfluxDB 1.x deployment with data and config mounted at custom paths:

$ docker run -p 8086:8086 \
      -v influxdb:/root/influxdb/data \
      -v $PWD/influxdb.conf:/root/influxdb/influxdb.conf \
      %%IMAGE%%:1.8 -config /root/influxdb/influxdb.conf

To upgrade this deployment to InfluxDB 2.x, first decide if you'd like to keep using custom paths, or use the InfluxDB 2.x defaults. If you decide to use the defaults, you'd stop the running InfluxDB 1.x container, then run:

$ docker run -p 8086:8086 \
      -v influxdb:/root/influxdb/data \
      -v influxdb2:/var/lib/influxdb2 \
      -v influxdb2-config:/etc/influxdb2 \
      -v $PWD/influxdb.conf:/root/influxdb/influxdb.conf \
      -e DOCKER_INFLUXDB_INIT_MODE=upgrade \
      -e DOCKER_INFLUXDB_INIT_USERNAME=my-user \
      -e DOCKER_INFLUXDB_INIT_PASSWORD=my-password \
      -e DOCKER_INFLUXDB_INIT_ORG=my-org \
      -e DOCKER_INFLUXDB_INIT_BUCKET=my-bucket \
      -e DOCKER_INFLUXDB_INIT_UPGRADE_V1_CONFIG=/root/influxdb/influxdb.conf \
      %%IMAGE%%:2.0

To retain your custom paths, you'd run:

$ docker run -p 8086:8086 \
      -v influxdb:/root/influxdb/data \
      -v influxdb2:/root/influxdb2/data \
      -v influxdb2-config:/etc/influxdb2 \
      -v $PWD/influxdb.conf:/root/influxdb/influxdb.conf \
      -e DOCKER_INFLUXDB_INIT_MODE=upgrade \
      -e DOCKER_INFLUXDB_INIT_USERNAME=my-user \
      -e DOCKER_INFLUXDB_INIT_PASSWORD=my-password \
      -e DOCKER_INFLUXDB_INIT_ORG=my-org \
      -e DOCKER_INFLUXDB_INIT_BUCKET=my-bucket \
      -e DOCKER_INFLUXDB_INIT_UPGRADE_V1_CONFIG=/root/influxdb/influxdb.conf \
      -e DOCKER_INFLUXDB_CONFIG_PATH=/root/influxdb2/config.toml \
      -e DOCKER_INFLUXDB_BOLT_PATH=/root/influxdb2/influxdb.bolt \
      -e DOCKER_INFLUXDB_ENGINE_PATH=/root/influxdb2/engine \
      %%IMAGE%%:2.0

Upgrading from quay.io-hosted InfluxDB 2.x image

Early Docker builds of InfluxDB 2.x were hosted at quay.io/influxdb/influxdb. The builds were very bare-bones, containing the influx and influxd binaries without any default configuration or helper scripts. By default, the influxd process stored data under /root/.influxdbv2.

Starting with v2.0.4, we've restored our DockerHub build. This build defaults to storing data in /var/lib/influxdb2. Upgrading directly from quay.io/influxdb/influxdb to influxdb:2.0.4 without modifying any settings will appear to cause data loss, as the new process won't be able to find your existing data files.

To avoid this problem when migrating from quay.io/influxdb/influxdb to influxdb:2.0, you can use one of the following approaches.

Change volume mount point

If you don't mind using the new default path, you can switch the mount-point for the volume containing your data:

# Migrate from this:
$ docker run -p 8086:8086 \
      -v $PWD:/root/.influxdbv2 \
      quay.io/influxdb/influxdb:v2.0.3

# To this:
docker run -p 8086:8086 \
      -v $PWD:/var/lib/influxdb2 \
      %%IMAGE%%:2.0

Override default configs

If you'd rather keep your data files in the home directory, you can override the container's default config:

# Migrate from this:
$ docker run -p 8086:8086 \
      -v $PWD:/root/.influxdbv2 \
      quay.io/influxdb/influxdb:v2.0.3

# To this:
docker run -p 8086:8086 \
      -e INFLUXD_BOLT_PATH=/root/.influxdbv2/influxd.bolt \
      -e INFLUXD_ENGINE_PATH=/root/.influxdbv2/engine \
      -v $PWD:/root/.influxdbv2 \
      %%IMAGE%%:2.0

See the section about configuration below for more ways to override the data paths.

Running the container

The InfluxDB image exposes a shared volume under /var/lib/influxdb2. You can mount a host directory to that point to access persisted container data. A typical invocation of the container might be:

$ docker run -p 8086:8086 \
      -v $PWD:/var/lib/influxdb2 \
      %%IMAGE%%:2.0

Modify $PWD to the directory where you want to store data associated with the InfluxDB container.

You can also have Docker control the volume mountpoint by using a named volume.

$ docker run -p 8086:8086 \
      -v influxdb2:/var/lib/influxdb2 \
      %%IMAGE%%:2.0

Exposed Ports

The following ports are important and are used by InfluxDB.

  • 8086 HTTP UI and API port

The HTTP port will be automatically exposed when using docker run -P.

Find more about API Endpoints & Ports here.

Configuration

InfluxDB can be configured using a mix of a config file, environment variables, and CLI options. To mount a configuration file and use it with the server, you can use this command to generate the default configuration file:

$ docker run --rm %%IMAGE%%:2.0 influxd print-config > config.yml

Modify the default configuration, which will now be available under $PWD. Then start the InfluxDB container:

$ docker run -p 8086:8086 \
      -v $PWD/config.yml:/etc/influxdb2/config.yml \
      %%IMAGE%%:2.0

Modify $PWD to be the directory where you want to store the configuration file.

Individual config settings can be overridden by environment variables. The variables must be named using the format INFLUXD_${SNAKE_CASE_NAME}. The SNAKE_CASE_NAME for an option will be the option's name with all dashes (-) replaced by underscores (_), in all caps.

Examples:

# Config setting: bolt-path
INFLUXD_BOLT_PATH=/root/influxdb.bolt
# Config setting: no-tasks
INFLUXD_NO_TASKS=true
# Config setting: storage-wal-fsync-delay
INFLUXD_STORAGE_WAL_FSYNC_DELAY=15m

Finally, all config options can be passed as CLI options:

$ docker run -p 8086:8086 \
      %%IMAGE%%:2.0 --storage-wal-fsync-delay=15m

CLI options take precedence over environment variables.

Find more about configuring InfluxDB here.

Database Setup

InfluxDB 2.x requires authentication. A special API exists to bootstrap the first super-user in the database, along with an initial organization and bucket. It's possible to access this API manually, or to run it automatically via environment variables.

Manual Setup

If your InfluxDB container is running locally (or on a host exposed to the network), you can perform initial setup from outside the container using either the UI or the influx CLI. Find more about setting up InfluxDB using these methods here.

It's also possible to perform manual setup from within the container using docker exec. For example, if you start the container:

$ docker run -d -p 8086:8086 \
      --name influxdb2 \
      -v $PWD:/var/lib/influxdb2 \
      %%IMAGE%%:2.0

You can then run the influx client in the container:

$ docker exec influxdb2 influx setup \
      --username $USERNAME \
      --password $PASSWORD \
      --org $ORGANIZATION \
      --bucket $BUCKET

Running setup from within the container will cause CLI configs to be written to /etc/influxdb2/influx-configs. You can then use the influx CLI from within the container to extract the generated admin token:

# Using table output + cut
$ docker exec influxdb2 influx auth list \
      --user $USERNAME \
      --hide-headers | cut -f 3

# Using JSON output + jq
$ docker exec influxdb2 influx auth list \
      --user $USERNAME \
      --json | jq -r '.[].token'

Alternatively, you could configure your initial InfluxDB run to mount /etc/influxdb2 as a volume:

$ docker run -d -p 8086:8086 \
      --name influxdb2 \
      -v $PWD/data:/var/lib/influxdb2 \
      -v $PWD/config:/etc/influxdb2 \
      %%IMAGE%%:2.0

This will make the generated CLI configs available to the host.

Automated Setup

The InfluxDB image contains some extra functionality to automatically bootstrap the system. This functionality is enabled by setting the DOCKER_INFLUXDB_INIT_MODE environment variable to the value setup when running the container. Additional environment variables are used to configure the setup logic:

  • DOCKER_INFLUXDB_INIT_USERNAME: The username to set for the system's initial super-user (Required).
  • DOCKER_INFLUXDB_INIT_PASSWORD: The password to set for the system's inital super-user (Required).
  • DOCKER_INFLUXDB_INIT_ORG: The name to set for the system's initial organization (Required).
  • DOCKER_INFLUXDB_INIT_BUCKET: The name to set for the system's initial bucket (Required).
  • DOCKER_INFLUXDB_INIT_RETENTION: The duration the system's initial bucket should retain data. If not set, the initial bucket will retain data forever.
  • DOCKER_INFLUXDB_INIT_ADMIN_TOKEN: The authentication token to associate with the system's initial super-user. If not set, a token will be auto-generated by the system.

Automated setup will generate metadata files and CLI configurations. It's recommended to mount volumes at both paths to avoid losing data.

For example, a minimal invocation of automated setup is:

$ docker run -d -p 8086:8086 \
      -v $PWD/data:/var/lib/influxdb2 \
      -v $PWD/config:/etc/influxdb2 \
      -e DOCKER_INFLUXDB_INIT_MODE=setup \
      -e DOCKER_INFLUXDB_INIT_USERNAME=my-user \
      -e DOCKER_INFLUXDB_INIT_PASSWORD=my-password \
      -e DOCKER_INFLUXDB_INIT_ORG=my-org \
      -e DOCKER_INFLUXDB_INIT_BUCKET=my-bucket \
      %%IMAGE%%:2.0

And an example using all available options is:

$ docker run -d -p 8086:8086 \
      -v $PWD/data:/var/lib/influxdb2 \
      -v $PWD/config:/etc/influxdb2 \
      -e DOCKER_INFLUXDB_INIT_MODE=setup \
      -e DOCKER_INFLUXDB_INIT_USERNAME=my-user \
      -e DOCKER_INFLUXDB_INIT_PASSWORD=my-password \
      -e DOCKER_INFLUXDB_INIT_ORG=my-org \
      -e DOCKER_INFLUXDB_INIT_BUCKET=my-bucket \
      -e DOCKER_INFLUXDB_INIT_RETENTION=1w \
      -e DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=my-super-secret-auth-token \
      %%IMAGE%%:2.0

NOTE: Automated setup will not run if an existing boltdb file is found at the configured path. This behavior allows for the InfluxDB container to reboot post-setup without encountering "DB is already set up" errors.

Interacting with InfluxDB

Once an InfluxDB instance has completed initial setup, its APIs will unlock. See the main documentation site for reference information and examples on:

Custom Initialization Scripts

The InfluxDB image supports running arbitrary initialization scripts after initial system setup, on both the setup and upgrade paths. Scripts must have extension .sh and be mounted inside of the /docker-entrypoint-initdb.d directory. When multiple scripts are present, they will be executed in lexical sort order by name.

As a convenience for script-writers, the image will export a number of variables into the environment before executing any scripts:

  • INFLUX_CONFIGS_PATH: Path to the CLI configs file written by setup/upgrade
  • INFLUX_HOST: URL to the influxd instance running setup logic
  • DOCKER_INFLUXDB_INIT_USER_ID: ID of the initial admin user created by setup/upgrade
  • DOCKER_INFLUXDB_INIT_ORG_ID: ID of the initial organization created by setup/upgrade
  • DOCKER_INFLUXDB_INIT_BUCKET_ID: ID of the initial bucket created by setup/upgrade

For example, if you wanted to grant write-access to an InfluxDB 1.x client on your initial bucket, you'd first create the file $PWD/scripts/setup-v1.sh with contents:

#!/bin/bash
set -e

influx v1 dbrp create \
  --bucket-id ${DOCKER_INFLUXDB_INIT_BUCKET_ID} \
  --db ${V1_DB_NAME} \
  --rp ${V1_RP_NAME} \
  --default \
  --org ${DOCKER_INFLUXDB_INIT_ORG}

influx v1 auth create \
  --username ${V1_AUTH_USERNAME} \
  --password ${V1_AUTH_PASSWORD} \
  --write-bucket ${DOCKER_INFLUXDB_INIT_BUCKET_ID} \
  --org ${DOCKER_INFLUXDB_INIT_ORG}

Then you'd run:

$ docker run -p 8086:8086 \
      -v $PWD/data:/var/lib/influxdb2 \
      -v $PWD/config:/etc/influxdb2 \
      -v $PWD/scripts:/docker-entrypoint-initdb.d \
      -e DOCKER_INFLUXDB_INIT_MODE=setup \
      -e DOCKER_INFLUXDB_INIT_USERNAME=my-user \
      -e DOCKER_INFLUXDB_INIT_PASSWORD=my-password \
      -e DOCKER_INFLUXDB_INIT_ORG=my-org \
      -e DOCKER_INFLUXDB_INIT_BUCKET=my-bucket \
      -e V1_DB_NAME=v1-db \
      -e V1_RP_NAME=v1-rp \
      -e V1_AUTH_USERNAME=v1-user \
      -e V1_AUTH_PASSWORD=v1-password \
      %%IMAGE%%:2.0

NOTE: Custom scripts will not run if an existing boltdb file is found at the configured path (causing setup or upgrade to be skipped). This behavior allows for the InfluxDB container to reboot post-initialization without encountering errors from non-idempotent script commands.

Using this Image - InfluxDB 1.x

Running the container

The InfluxDB image exposes a shared volume under /var/lib/influxdb, so you can mount a host directory to that point to access persisted container data. A typical invocation of the container might be:

$ docker run -p 8086:8086 \
      -v $PWD:/var/lib/influxdb \
      %%IMAGE%%:1.8

Modify $PWD to the directory where you want to store data associated with the InfluxDB container.

You can also have Docker control the volume mountpoint by using a named volume.

$ docker run -p 8086:8086 \
      -v influxdb:/var/lib/influxdb \
      %%IMAGE%%:1.8

Exposed Ports

The following ports are important and are used by InfluxDB.

  • 8086 HTTP API port
  • 2003 Graphite support, if it is enabled

The HTTP API port will be automatically exposed when using docker run -P.

Configuration

InfluxDB can be either configured from a config file or using environment variables. To mount a configuration file and use it with the server, you can use this command:

Generate the default configuration file:

$ docker run --rm %%IMAGE%%:1.8 influxd config > influxdb.conf

Modify the default configuration, which will now be available under $PWD. Then start the InfluxDB container.

$ docker run -p 8086:8086 \
      -v $PWD/influxdb.conf:/etc/influxdb/influxdb.conf:ro \
      %%IMAGE%%:1.8 -config /etc/influxdb/influxdb.conf

Modify $PWD to the directory where you want to store the configuration file.

For environment variables, the format is INFLUXDB_$SECTION_$NAME. All dashes (-) are replaced with underscores (_). If the variable isn't in a section, then omit that part.

Examples:

INFLUXDB_REPORTING_DISABLED=true
INFLUXDB_META_DIR=/path/to/metadir
INFLUXDB_DATA_QUERY_LOG_ENABLED=false

Find more about configuring InfluxDB here.

Graphite

InfluxDB supports the Graphite line protocol, but the service and ports are not exposed by default. To run InfluxDB with Graphite support enabled, you can either use a configuration file or set the appropriate environment variables. Run InfluxDB with the default Graphite configuration:

docker run -p 8086:8086 -p 2003:2003 \
    -e INFLUXDB_GRAPHITE_ENABLED=true \
    %%IMAGE%%:1.8

See the README on GitHub for more detailed documentation to set up the Graphite service. In order to take advantage of graphite templates, you should use a configuration file by outputting a default configuration file using the steps above and modifying the [[graphite]] section.

HTTP API

Creating a DB named mydb:

$ curl -G http://localhost:8086/query --data-urlencode "q=CREATE DATABASE mydb"

Inserting into the DB:

$ curl -i -XPOST 'http://localhost:8086/write?db=mydb' --data-binary 'cpu_load_short,host=server01,region=us-west value=0.64 1434055562000000000'

Read more about this in the official documentation

CLI / SHELL

Start the container:

$ docker run --name=influxdb -d -p 8086:8086 %%IMAGE%%:1.8

Run the influx client in this container:

$ docker exec -it influxdb influx

Or run the influx client in a separate container:

$ docker run --rm --link=influxdb -it %%IMAGE%%:1.8 influx -host influxdb

Database Initialization

The InfluxDB image contains some extra functionality for initializing a database. These options are not suggested for production, but are quite useful when running standalone instances for testing.

The database initialization script will only be called when running influxd. It will not be executed when running any other program.

Environment Variables

The InfluxDB image uses several environment variables to automatically configure certain parts of the server. They may significantly aid you in using this image.

INFLUXDB_DB

Automatically initializes a database with the name of this environment variable.

INFLUXDB_HTTP_AUTH_ENABLED

Enables authentication. Either this must be set or auth-enabled = true must be set within the configuration file for any authentication related options below to work.

INFLUXDB_ADMIN_USER

The name of the admin user to be created. If this is unset, no admin user is created.

INFLUXDB_ADMIN_PASSWORD

The password for the admin user configured with INFLUXDB_ADMIN_USER. If this is unset, a random password is generated and printed to standard out.

INFLUXDB_USER

The name of a user to be created with no privileges. If INFLUXDB_DB is set, this user will be granted read and write permissions for that database.

INFLUXDB_USER_PASSWORD

The password for the user configured with INFLUXDB_USER. If this is unset, a random password is generated and printed to standard out.

INFLUXDB_READ_USER

The name of a user to be created with read privileges on INFLUXDB_DB. If INFLUXDB_DB is not set, this user will have no granted permissions.

INFLUXDB_READ_USER_PASSWORD

The password for the user configured with INFLUXDB_READ_USER. If this is unset, a random password is generated and printed to standard out.

INFLUXDB_WRITE_USER

The name of a user to be created with write privileges on INFLUXDB_DB. If INFLUXDB_DB is not set, this user will have no granted permissions.

INFLUXDB_WRITE_USER_PASSWORD

The password for the user configured with INFLUXDB_WRITE_USER. If this is unset, a random password is generated and printed to standard out.

Initialization Files

If the Docker image finds any files with the extensions .sh or .iql inside of the /docker-entrypoint-initdb.d folder, it will execute them. The order they are executed in is determined by the shell. This is usually alphabetical order.

Manually Initializing the Database

To manually initialize the database and exit, the /init-influxdb.sh script can be used directly. It takes the same parameters as the influxd run command. As an example:

$ docker run --rm \
      -e INFLUXDB_DB=db0 \
      -e INFLUXDB_ADMIN_USER=admin -e INFLUXDB_ADMIN_PASSWORD=supersecretpassword \
      -e INFLUXDB_USER=telegraf -e INFLUXDB_USER_PASSWORD=secretpassword \
      -v $PWD:/var/lib/influxdb \
      %%IMAGE%%:1.8 /init-influxdb.sh

The above would create the database db0, create an admin user with the password supersecretpassword, then create the telegraf user with your telegraf's secret password. It would then exit and leave behind any files it created in the volume that you mounted.