Head to the Deploy section of our docs site to get started.
See below for technical considerations and instructions.
Lightning enforces encryption at rest for credentials, TOTP backup codes, and webhook trigger authentication methods, for which an encryption key must be provided when running in production.
The key is expected to be a randomized set of bytes, 32 long; and Base64 encoded when setting the environment variable.
There is a mix task that can generate keys in the correct shape for use as an environment variable:
mix lightning.gen_encryption_key
0bJ9w+hn4ebQrsCaWXuA9JY49fP9kbHmywGd5K7k+/s=
Copy your key (NOT THIS ONE) and set it as PRIMARY_ENCRYPTION_KEY
in your
environment.
Lightning uses external worker processes for executing Runs. There are three settings required to configure worker authentication.
WORKER_RUNS_PRIVATE_KEY
WORKER_LIGHTNING_PUBLIC_KEY
WORKER_SECRET
You can use the mix lightning.gen_worker_keys
task to generate these for
convenience.
For more information see the Workers documentation.
Note that for secure deployments, it's recommended to use a combination of
secrets
and configMaps
to generate secure environment variables.
WORKER_MAX_RUN_MEMORY_MB
- how much memory (in MB) can a single run use?RUN_GRACE_PERIOD_SECONDS
- how long after theMAX_RUN_DURATION_SECONDS
should the server wait for the worker to send back data on a run.WORKER_MAX_RUN_DURATION_SECONDS
- the maximum duration (in seconds) that workflows are allowed to run (keep this plusRUN_GRACE_PERIOD_SECONDS
below your termination_grace_period if using kubernetes)WORKER_CAPACITY
- the number of runs a ws-worker instance will take on concurrently.MAX_DATACLIP_SIZE_MB
- the maximum size (in MB) of a dataclip created via the webhook trigger URL for a job. This limits the max request size via the JSON plug and may (in future) limit the size of dataclips that can be stored as run_results via the websocket connection from a worker.
Lightning enables connection to github via Github Apps. The following github permissions are needed for the github app:
Resource | Access |
---|---|
Actions | Read and Write |
Contents | Read and Write |
Metadata | Read only |
Secrets | Read and Write |
Workflows | Read and Write |
These envrionment variables will need to be set in order to configure the github app:
GITHUB_APP_ID
- the github app ID.GITHUB_APP_NAME
- the github app nameGITHUB_APP_CLIENT_ID
- the github app Client IDGITHUB_APP_CLIENT_SECRET
- the github app Client SecretGITHUB_CERT
- the github app private key
You can access these from your github app settings menu. Also needed for the configurtaion is:
REPO_CONNECTION_SIGNING_SECRET
- secret used to sign access tokens. This access token is used to authenticate requests made from the github actions. You can generate this usingmix lightning.gen_encryption_key
🧪 Experimental
Lightning can be configured to use an AI chatbot for user interactions.
See openfn/apollo for more information on the Apollo AI service.
The following environment variables are required:
OPENAI_API_KEY
- your OpenAI API key.APOLLO_ENDPOINT
- the endpoint for the OpenFn Apollo AI service.
🧪 Experimental
Lightning workflows can be configured with a trigger that will consume messages from a Kafka Cluster. By default this is disabled and you will not see the option to create a Kafka trigger in the UI, nor will the Kafka consumer groups be running.
To enable this feature set the KAFKA_TRIGGERS_ENABLED
environment variable
to yes
and restart Lightning. Please note that, if you enable this feature
and then create some Kafka triggers and then disable the feature, you will not
be able to edit any triggers created before the feature was disabled.
Kafka triggers currently rely on the existence of Message Candidate Sets. These are a temporary measure to ensure that message sequence for messages with identical keys are preserved. As part of this mechanism, Lightning uses a MessageCandidateSetWorker to convert messages into WorkOrders.
By default, there is only one worker process, but the number of workers can be
increased by setting the KAFKA_NUMBER_OF_MESSAGE_CANDIDATE_SET_WORKERS
environment variable. Increasing this may increase the rate at which messages
received by Kafka are converted to WorkOrders. If you find that you have a
large number of records in the trigger_kafka_messages
table, then increasing
the number of workers may help to reduce this backlog.
The MessageCandidateSetWorker has two configurable processing delays:
- The delay before the worker requests the next Message Candidate Set for
consideration after it has finished processing the current Message Candidate
Set. This can be set by configuring the
KAFKA_NEXT_MESSAGE_CANDIDATE_SET_DELAY_MILLISECONDS
environment variable. The default value is currently 250ms. - The delay before the worker requests the next Message Candidate Set for
consideration when there are no Message Candidate Sets available. This can be
set by configuring the
KAFKA_NO_MESSAGE_CANDIDATE_SET_DELAY_MILLISECONDS
environment variable. The default value is currently 10000ms.
The number of Kafka consumers in the consumer group can be modified by setting
the KAFKA_NUMBER_OF_CONSUMERS
environment variable. The default value is
currently 1. The optimal setting is one consumer per topic partition. NOTE:
This setting will move to KafkaConfiguration as it will be trigger-specific.
The number of messages that the Kafka consumer will forward is rate-limited by
the KAFKA_NUMBER_OF_MESSAGES_PER_SECOND
environment variable. This can be set
to a value of less than 1 (minimum 0.1) and will converted (and rounded-down) to
an integer value of messages over a 10-second interval (e.g. 0.15 becomes 1
message every 10 seconds). The default value is 1.
Processing concurrency within the Kafka Broadway pipeline is controlled by the
KAFKA_NUMBER_OF_PROCESSORS
environment variable. Modifying this, modifies the
number of processors that are downstream of the Kafka consumer, so an increase
in this value should inrcease throughput (when factoring in the rate limit set
by KAFKA_NUMBER_OF_MESSAGES_PER_SECOND
). The default value is 1.
ADAPTORS_PATH
- where you store your locally installed adaptorsDISABLE_DB_SSL
- in production the use of an SSL conntection to Postgres is required by default, setting this to"true"
allows unencrypted connections to the database. This is strongly discouraged in real production environment.K8S_HEADLESS_SERVICE
- this environment variable is automatically set if you're running on GKE and it is used to establish an Erlang node cluster. Note that if you're not using Kubernetes, the "gossip" strategy is used for establish clusters.LISTEN_ADDRESS
" - the address the web server should bind to, defaults to127.0.0.1
to block access from other machines.LOG_LEVEL
- how noisy you want the logs to be (e.g.debug
,info
)MIX_ENV
- your mix env, likelyprod
for deploymentNODE_ENV
- node env, likelyproduction
for deploymentORIGINS
- the allowed origins for web traffic to the backendPORT
- the port your Phoenix app runs onPRIMARY_ENCRYPTION_KEY
- a base64 encoded 32 character long string. See Encryption.SCHEMAS_PATH
- path to the credential schemas that provide forms for different adaptorsSECRET_KEY_BASE
- a secret key used as a base to generate secrets for encrypting and signing data.SENTRY_DSN
- if using Sentry for error monitoring, your DSNCORS_ORIGIN
- a list of acceptable hosts for browser/cors requests (',' separated)URL_HOST
- the host, used for writing urls (e.g.,demo.openfn.org
)URL_PORT
- the port, usually443
for productionURL_SCHEME
- the scheme for writing urls, (e.g.,https
)USAGE_TRACKER_HOST
- the host that receives usage tracking submissions (defaults to https://impact.openfn.org).USAGE_TRACKING_DAILY_BATCH_SIZE
- the number of days that will be reported on with each run ofUsageTracking.DayWorker
. This will only have a noticeable effect in cases where there is a backlog, or where reports are being generated retroactively (defaults to 10).USAGE_TRACKING_ENABLED
- enables the submission of anonymised usage data to OpenFn (defaults totrue
).USAGE_TRACKING_RESUBMISSION_BATCH_SIZE
- the number of failed reports that will be submitted on each resubmission run (defaults to 10).USAGE_TRACKING_UUIDS
- indicates whether submissions should include cleartext uuids or not. Options arecleartext
orhashed_only
, with the default beinghashed_only
.QUEUE_RESULT_RETENTION_PERIOD_SECONDS
- the number of seconds to keep completed (successful)ObanJobs
in the queue (not to be confused with runs and/or history)IS_RESETTABLE_DEMO
- If set toyes
it allows this instance to be reset to the initial "Lightning Demo" state. Note that this will destroy most of what you have in your database!ALLOW_SIGNUP
: Set totrue
to enable user access to the registration page. Set tofalse
to disable new user registrations and block access to the registration page. Default istrue
.EMAIL_ADMIN
- This is used as the sender email address for system emails. It is also displayed in the menu as the support email.EMAIL_SENDER_NAME
- This is displayed in the email client as the sender name for emails sent by the application.
Using your Google Cloud account, provision a new OAuth 2.0 Client with the 'Web application' type.
Set the callback url to: https://<ENDPOINT DOMAIN>/authenticate/callback
.
Replacing ENDPOINT DOMAIN
with the host name of your instance.
Once the client has been created, get/download the OAuth client JSON and set the following environment variables:
GOOGLE_CLIENT_ID
- Which isclient_id
from the client details.GOOGLE_CLIENT_SECRET
-client_secret
from the client details.
Using your Salesforce developer account, create a new Oauth 2.0 connected application.
Set the callback url to: https://<ENDPOINT DOMAIN>/authenticate/callback
.
Replacing ENDPOINT DOMAIN
with the host name of your instance.
Grant permissions as desired.
Once the client has been created set the following environment variables:
SALESFORCE_CLIENT_ID
- Which isConsumer Key
from the "Manage Consumer Details" screen.SALESFORCE_CLIENT_SECRET
- Which isConsumer Secret
from the "Manage Consumer Details" screen.