jogrogan / Hoptimator Public

forked from linkedin/Hoptimator

Notifications You must be signed in to change notification settings
Fork 0
Star 0

Multi-hop declarative data pipelines

Notifications

Name		Name	Last commit message	Last commit date
Latest commit History 18 Commits
bin		bin
deploy		deploy
gradle		gradle
hoptimator-catalog		hoptimator-catalog
hoptimator-cli		hoptimator-cli
hoptimator-flink-iterator		hoptimator-flink-iterator
hoptimator-flink-runner		hoptimator-flink-runner
hoptimator-operator		hoptimator-operator
hoptimator-planner		hoptimator-planner
models		models
.gitignore		.gitignore
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
NOTICE		NOTICE
README.md		README.md
build.gradle		build.gradle
gradlew		gradlew
gradlew.bat		gradlew.bat
settings.gradle		settings.gradle

Repository files navigation

Hoptimator

Multi-hop declarative data pipelines

Quick Start

  $ ./gradlew build
  $ ./bin/hoptimator
  > !intro

What is Hoptimator?

Hoptimator is an SQL-based control plane for complex data pipelines.

Hoptimator turns high-level SQL subscriptions into low-level SQL pipelines. Pipelines may involve an auto-generated Flink job (or similar) and any arbitrary resources required for the job to run.

How does it work?

Hoptimator has a pluggable adapter framework, which lets you wire-up arbtitary data sources. Adapters loosely correspond to "connectors" in the underlying compute engine (e.g. Flink Connectors), but they may bring along additional baggage. For example, an adapter may bring along a cache or a CDC stream as part of the resulting pipeline.