Multi-hop declarative data pipelines
Hoptimator is an SQL-based control plane for complex data pipelines.
Hoptimator turns high-level SQL subscriptions into low-level SQL pipelines. Pipelines may involve an auto-generated Flink job (or similar) and any arbitrary resources required for the job to run.
Hoptimator has a pluggable adapter framework, which lets you wire-up arbtitary data sources. Adapters loosely correspond to "connectors" in the underlying compute engine (e.g. Flink Connectors), but they may bring along additional baggage. For example, an adapter may bring along a cache or a CDC stream as part of the resulting pipeline.
$ make quickstart
... wait a while ...
$ ./bin/hoptimator
> !intro
> !q
Subscriptions are SQL views that are automatically materialized by a pipeline.
$ kubectl apply -f deploy/samples/subscription.yaml
In response, the operator will deploy a Flink job and other resources:
$ kubectl get subscriptions
$ kubectl get sqljobs
$ kubectl get flinkdeployments
$ kubectl get kafkatopics
You can verify the job is running by inspecting the output:
$ ./bin/hoptimator
> !tables
> SELECT * FROM RAWKAFKA."sample-subscription-1" LIMIT 5;
> !q