Export issue metadata & agile metrics, transform and load to OLAP data storage. Metrics based on issue changelog.
Versions 1.x.x incompatible with 0.1.x. New versions works only on Python >= 3.10
You can fork this repository and refine the tool the way you want. Or use it as it is - this will allow you to build basic analytics on the tasks from Yandex.Tracker.
So, you can install Clickhouse with Proxy via Ansible role inside project (previous versions).
Edit the inventory file ansible/inventory/hosts.yml
and just run ansible-playbook.
Attention: For the role to work correctly, docker must be installed on the target server.
Example:
pip3 install -r requirements-dev.txt
cd ansible
ansible-playbook -i inventory/hosts.yml playbooks/clickhouse.yml --limit agile
Also, you can use this extended Clickhouse role
# prepare virtual environment
python3 -m venv venv
source venv/bin/activate
python3 setup.py install
# configure environment variables
export EXPORTER_TRACKER__TOKEN=your_token
export EXPORTER_TRACKER__CLOUD_ORG_ID=your_org_id
export EXPORTER_CLICKHOUSE__HOST=localhost
export EXPORTER_CLICKHOUSE__PORT=8123
export EXPORTER_STATEFUL="true"
export EXPORTER_STATE__STORAGE=jsonfile
export EXPORTER_STATE__JSONFILE_STRATEGY=local
export EXPORTER_STATE__JSONFILE_PATH=./state.json
# run
tracker-exporter
pip3 install tracker-exporter
tracker-exporter
Read about the settings here
tracker-exporter --env-file /home/akimrx/tracker/.settings
cd yandex-tracker-exporter/docker
docker-compose up -d
docker logs tracker-exporter -f
How to: https://cloud.yandex.com/en/docs/managed-clickhouse/operations/cluster-create
- Set user for exporter, example:
agile
- Set a database name, example:
agile
- Enable
Serverless access
flag - For testing enable host public access
- Enable
Access from the management console
flag - Run migration or manual create tables (see migration block here, see sql)
How to: https://cloud.yandex.com/en/docs/functions/quickstart/create-function/python-function-quickstart
- Use Python >= 3.10
- Copy/paste example content from
examples/serverless
(code) - Set entrypoint:
main.handler
(for code from examples) - Set function timeout to
600
, because the launch can be long if there are a lot of updated issues during the collection period - Set memory to
512MB
or more - Add environment variables (see variables block here)
EXPORTER_TRACKER__TOKEN=XXXXXXXXXXXXXXXX
EXPORTER_TRACKER__CLOUD_ORG_ID=123456
EXPORTER_TRACKER__SEARCH__RANGE=2h
EXPORTER_ENABLE__UPLOAD="true"
EXPORTER_CLICKHOUSE__PROTO=https
EXPORTER_CLICKHOUSE__CACERT_PATH=/etc/ssl/certs/ca-certificates.crt
EXPORTER_CLICKHOUSE__PORT=8443
EXPORTER_CLICKHOUSE__HOST=rc1b-xxxxxx.mdb.yandexcloud.net
EXPORTER_CLICKHOUSE__USER=agile
EXPORTER_CLICKHOUSE__PASSWORD=xxxx
- Release function
- Run test
- See logs
If you don't want to enable clickhouse public access, use service account with such permissions - serverless.mdbProxies.user
and set environment variables below:
EXPORTER_CLICKHOUSE__HOST=akfd3bhqk3xxxxxxxxxxx.clickhouse-proxy.serverless.yandexcloud.net
EXPORTER_CLICKHOUSE__SERVERLESS_PROXY_ID=akfd3bhqk3xxxxxxxxxxxxx
How to create database connection: https://cloud.yandex.com/en/docs/functions/operations/database-connection
Also, the EXPORTER_CLICKHOUSE__PASSWORD
variable with service account must be replaced by IAM-token. Keep this in mind.
Probably, you should get it in the function code, because the IAM-token works for a limited period of time.
How to: https://cloud.yandex.com/en/docs/functions/quickstart/create-trigger/timer-quickstart
- Create new trigger
- Choose type
Timer
- Set interval every hour:
0 * ? * * *
- Select your function
- Create serverless service account or use an existing one
- Save trigger
TODO
You can use any BI/observability tool for visualization, for example:
- Yandex DataLens (btw, this is opensource)
- Apache Superset
- PowerBI
- Grafana
Based on go-migrate tool.
wget https://github.com/golang-migrate/migrate/releases/download/v4.15.2/migrate.darwin-amd64.tar.gz -O migrate.tar.gz
tar xvf migrate.tar.gz
mv migrate ~/bin
wget https://github.com/golang-migrate/migrate/releases/download/v4.15.2/migrate.linux-amd64.tar.gz -O migrate.tar.gz
tar -xvf migrate.tar.gz
mv migrate /usr/local/bin
Example bash script below.
See full example script here
#!/usr/bin/env bash
set -Eeuo pipefail
CLICKHOUSE_HOST="localhost"
CLICKHOUSE_TCP_PORT=9000
CLICKHOUSE_HTTP_PORT=8123
CLICKHOUSE_USER="default"
CLICKHOUSE_PASSWORD="strongpassword"
MIGRATION_SOURCE_PATH="file://${PWD}/../migrations/clickhouse"
MIGRATION_HISTORY_TABLE="ci_gomigrate_migrations"
MIGRATION_DATABASE="agile"
MIGRATION_CLICKHOUSE_DSN="clickhouse://${CLICKHOUSE_HOST}:${CLICKHOUSE_TCP_PORT}?username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}&database=${MIGRATION_DATABASE}&x-multi-statement=true&x-migrations-table=${MIGRATION_HISTORY_TABLE}"
prepare_migration() {
echo "CREATE DATABASE IF NOT EXISTS ${MIGRATION_DATABASE}" | \
curl "http://${CLICKHOUSE_HOST}:${CLICKHOUSE_HTTP_PORT}/?user=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}" --data-binary @-
}
run_migration() {
migrate -verbose \
-source $MIGRATION_SOURCE_PATH \
-database $MIGRATION_CLICKHOUSE_DSN \
up
}
prepare_migration
run_migration
See config declaration here
variable | description |
---|---|
EXPORTER_STATEFUL |
Enable stateful mode. Required EXPORTER_STATE__* params. Default is False |
EXPORTER_STATEFUL_INITIAL_RANGE |
Initial search range when unknown last state. Default: 6mo |
EXPORTER_LOGLEVEL |
ETL log level. Default: info |
EXPORTER_WORKDAYS |
Workdays for calculate business time. 0 - mon, 6 - sun. Default: [0,1,2,3,4] |
EXPORTER_BUSINESS_HOURS_START |
Business hours start for calculate business time. Default: 09:00:00 |
EXPORTER_BUSINESS_HOURS_END |
Business hours end for calculate business time. Default: 22:00:00 |
EXPORTER_DATETIME_RESPONSE_FORMAT |
Yandex.Tracker datetime format in responses. Default: %Y-%m-%dT%H:%M:%S.%f%z |
EXPORTER_DATETIME_QUERY_FORMAT |
Datetime format for search queries. Default: %Y-%m-%d %H:%M:%S |
EXPORTER_DATETIME_CLICKHOUSE_FORMAT |
Datetime format for Clickhouse. Default: %Y-%m-%dT%H:%M:%S.%f |
EXPORTER_ETL_INTERVAL_MINUTES |
Interval between run ETL. Default: 30 (minutes) |
EXPORTER_CLOSED_ISSUE_STATUSES |
Statuses for mark issue as closed. Default: closed,rejected,resolved,cancelled,released |
EXPORTER_NOT_NULLABLE_FIELDS |
Fields that should never be null (e.g. dates). Default: all datetime fields |
variable | description |
---|---|
EXPORTER_TRACKER__LOGLEVEL |
Log level for Yandex.Tracker SDK. Default: warning |
EXPORTER_TRACKER__TOKEN |
OAuth2 token. Required if EXPORTER_TRACKER__IAM_TOKEN is not passed |
EXPORTER_TRACKER__ORG_ID |
Yandex360 organization ID. Required if EXPORTER_TRACKER__CLOUD_ORG_ID is not passed |
EXPORTER_TRACKER__IAM_TOKEN |
Yandex.Cloud IAM token. Required if EXPORTER_TRACKER__TOKEN is not passed |
EXPORTER_TRACKER__CLOUD_ORG_ID |
Yandex.Cloud organization ID. Required if EXPORTER_TRACKER__ORG_ID is not passed |
EXPORTER_TRACKER__TIMEOUT |
Yandex.Tracker HTTP requests timeout. Default: 10 (sec) |
EXPORTER_TRACKER__MAX_RETRIES |
Yandex.Tracker HTTP requests max retries. Default: 10 |
EXPORTER_TRACKER__LANGUAGE |
Yandex.Tracker language. Default: en |
EXPORTER_TRACKER__TIMEZONE |
Yandex.Tracker timezone. Default: Europe/Moscow |
EXPORTER_TRACKER__SEARCH__QUERY |
Custom query for search issues. This variable has the highest priority and overrides other search parameters. Default is empty |
EXPORTER_TRACKER__SEARCH__RANGE |
Search issues window. Has no effect in stateful mode. Default: 2h |
EXPORTER_TRACKER__SEARCH__QUEUES |
Include or exclude queues in search. Example: DEV,SRE,!TEST,!TRASH Default is empty |
EXPORTER_TRACKER__SEARCH__PER_PAGE_LIMIT |
Search results per page. Default: 100 |
variable | description |
---|---|
EXPORTER_CLICKHOUSE__ENABLE_UPLOAD |
Enable upload data to Clickhouse. Default is True |
EXPORTER_CLICKHOUSE__HOST |
Clickhouse host. Default: localhost |
EXPORTER_CLICKHOUSE__PROTO |
Clickhouse protocol: http or https. Default: http |
EXPORTER_CLICKHOUSE__PORT |
Clickhouse HTTP(S) port. Default: 8123 |
EXPORTER_CLICKHOUSE__CACERT_PATH |
Path to CA cert. Only for HTTPS proto. Default is empty |
EXPORTER_CLICKHOUSE__SERVERLESS_PROXY_ID |
Yandex Cloud Functions proxy ID. Default is empty |
EXPORTER_CLICKHOUSE__USERNAME |
Clickhouse username. Default: default |
EXPORTER_CLICKHOUSE__PASSWORD |
Clickhouse password. Can be empty. Default is empty |
EXPORTER_CLICKHOUSE__DATABASE |
Clickhouse database. Default: agile |
EXPORTER_CLICKHOUSE__ISSUES_TABLE |
Clickhouse table for issues metadata. Default: issues |
EXPORTER_CLICKHOUSE__ISSUE_METRICS_TABLE |
Clickhouse table for issue metrics. Default: issue_metrics |
EXPORTER_CLICKHOUSE__AUTO_DEDUPLICATE |
Execute OPTIMIZE after each INSERT . Default is True |
variable | description |
---|---|
EXPORTER_STATE__STORAGE |
Storage type for StateKeeper. Can be: jsonfile , redis , custom . Default: jsonfile |
EXPORTER_STATE__REDIS_DSN |
Connection string for Redis state storage when storage type is redis . Default is empty. |
EXPORTER_STATE__JSONFILE_STRATEGY |
File store strategy for jsonfile storage type. Can be s3 or local . Default: local |
EXPORTER_STATE__JSONFILE_PATH |
Path to JSON state file. Default: ./state.json |
EXPORTER_STATE__JSONFILE_S3_BUCKET |
Bucket for s3 strategy. Default is empty |
EXPORTER_STATE__JSONFILE_S3_REGION |
Region for s3 strategy. Default is eu-east-1 |
EXPORTER_STATE__JSONFILE_S3_ENDPOINT |
Endpoint URL for s3 strategy. Default is empty |
EXPORTER_STATE__JSONFILE_S3_ACCESS_KEY |
AWS access key id for s3 strategy. Default is empty |
EXPORTER_STATE__JSONFILE_S3_SECRET_KEY |
AWS secret key for s3 strategy. Default is empty |
EXPORTER_STATE__CUSTOM_STORAGE_PARAMS |
Settings for custom storage params as dict . Default: {} |
variable | description |
---|---|
EXPORTER_MONITORING__METRICS_ENABLED |
Enable send statsd tagged metrics. Default is False |
EXPORTER_MONITORING__METRICS_HOST |
DogStatsD / statsd host. Default: localhost |
EXPORTER_MONITORING__METRICS_PORT |
DogStatsD / statsd port. Default: 8125 |
EXPORTER_MONITORING__METRICS_BASE_PREFIX |
Prefix for metrics name. Default: tracker_exporter |
EXPORTER_MONITORING__METRICS_BASE_LABELS |
Tags for metrics. Default: ["project:internal",] |
EXPORTER_MONITORING__SENTRY_ENABLED |
Enable send exception stacktrace to Sentry. Default is empty |
EXPORTER_MONITORING__SENTRY_DSN |
Sentry DSN. Default is empty |
Based on DogStatsD tagged format. VictoriaMetrics compatible.
Metric name | Metric type | Labels | Description |
---|---|---|---|
tracker_exporter_issue_transform_time_seconds |
time | - | Duration of transform per task (data packing to the model) |
tracker_exporter_issues_total_processed_count |
count | - | Total issues processed |
tracker_exporter_issues_search_time_seconds |
time | - | Yandex.Tracker search duration time in seconds |
tracker_exporter_issues_without_metrics |
count | - | Issues with empty metrics (no changelog) |
tracker_exporter_issue_prefetch_seconds |
time | - | Pre-transform data duration in seconds |
tracker_exporter_comments_fetch_seconds |
time | - | Comments fetch duration in seconds |
tracker_exporter_etl_duration_seconds |
time | - | ETL full pipeline duration in seconds |
tracker_exporter_etl_upload_status |
gauge | - | Last upload status, 1 - success, 2 - fail |
tracker_exporter_export_and_transform_time_seconds |
time | - | Overall export and transform duration in seconds |
tracker_exporter_upload_to_storage_time_seconds |
time | - | Overall insert duration time in seconds |
tracker_exporter_last_update_timestamp |
gauge | - | Last data update timestamp |
tracker_exporter_clickhouse_insert_time_seconds |
time | database, table | Insert per table duration time in seconds |
tracker_exporter_clickhouse_inserted_rows |
count | database, table | Inserted rows per table |
tracker_exporter_clickhouse_deduplicate_time_seconds |
time | database, table | Optimize execute time duration in seconds |