SQL to HTTP Stream

A microservice-based end-to-end stream flow.

Stack

Service	Libraries
data-crud	http4s (with Ember + Cats) + Doobie + Flyway + Circe
influxdb-sink-connector	Akka Stream
influx-stream	Akka Stream + Akka HTTP

Architecture overview

Dependencies

• Java 11
• Docker Compose
• sbt

Running locally

1. Place the cursor at the project's root folder
2. Install Debezium Connector
   1. Create a folder named kafka-connectors with read-write permissions for Docker container's usage
   2. Download Debezium Postgres Connector 2.1.3 and unpack it to the above folder
3. Run docker compose up -d to start Docker Compose
4. Make the POST request at kafka-connector.http to configure a Debezium Postgres Connector instance
5. Use sbt to start each of the services data-crud, influxdb-sink-connector and influx-stream

Now you can make the requests exemplified at data-crud.http to data-crud and observe the streamed average at http://localhost:8081/moving-average/{id}?period={seconds}&every={seconds} to change

Port mapping

Service	Port
Postgres	5432
InfluxDB	8086
Zookeeper	2187
Kafka	9092
Schema Registry	9093
Kafka Connect	8083
Kafka UI	8090
data-crud	8080
influxdb-sink-connector	N/A
influx-stream	8081