Sink Connector 2024 Roadmap

[hodgesrm]

This issue tracks deliverables on the 2024 roadmap for Sink Connector. Feel free to comment on current deliverables and suggest new ones in comments.

Upcoming Releases

2.2.1 - https://github.com/Altinity/clickhouse-sink-connector/issues?q=is%3Aopen+is%3Aissue+milestone%3A2.2.1

Source Database Support

• Production PostgreSQL support (currently experimental)
• Production MongoDB support

ClickHouse Target Table Types

* [Replicated]MergeTree - Map change log records to ClickHouse, not final state

Replication Capabilities

* Replicate multiple databases in one connector(#531)

• Add existing tables to replication
• Schema overrides to control mapping of upstream schema to ClickHouse (clickhouse_loader.py : support schema overrides in yaml #323)
• Support MergeTree with version timestamp (Request: Support MergeTree with version timestamp. #445)

Security Improvements

• Remove passwords from command line invocations (clickhouse-loader.py and *checksum.py security #398)

Deployment Configurations and Packaging

• Production Kafka Sink Connector
• Production Helm Chart for Kubernetes
• Tarball releases

Reliability and Performance

• Expanded DDL handling tests
• Ghost testing (MySQL)

Miscellaneous

• Move SQL test data to separate project
• Move dump/load utilities to separate project