Dependabot/go modules/GitHub.com/hashicorp/go sockaddr 1.0.2

[simonpasquier]

@mxinden @stuartnelson3 I'm working on automating dependency updates with dependabot and this PR is a first try. See also prometheus/prometheus#5265 for more context.

I have basically written some glue code that updates the dependabot PRs in case they don't pass the CI (eg vendor/ directory not synced) and push automatically to the upstream once the CI is green. It will also check that the PR is mergeable and ask dependabot to recreate a new PR if not.

[stuartnelson3]

I'm in favor of this, but my main concerns (which should always be concerns) is the flakiness of some of our tests, and the e2e prometheus operator test that we rely on when cutting releases being external (I believe that's what @mxinden has referenced, not sure of the correct name for it).

I've never seen the e2e test, but would it be possible to include this (or something like it)? Some of the work done in the api acceptance test could potentially be expanded to include a small webhook receiver to make sure the notifications are correctly sent, instead of spinning up the whole e2e environment.

What does "success" look like in the operator e2e test? Is it using a webhook and verifying a notification is received?

Also, a thought: Is prometheus using the swagger-generated alertmanager client to send alerts?

[mxinden]

@simonpasquier do I understand it correctly, that

1. dependabot creates a PR against github.com/simonpasquier/alertmanager.
2. Your script updates the PR to also vendor all changes.
3. Open a PR with the same change set against github.com/prometheus/alertmanager.

---

@stuartnelson3 I am not quite sure how your comment relates to these changes. Are you saying that our already high flakiness would increase more with continuously introducing changes in regards to our dependencies?

The Prometheus Operator test suite contains the following Alertmanager tests:

• Create and Delete a Cluster
• Scaling the cluster
• Version Migration between last minor releases
• StorageUpdate
• ExposingWithKubernetesAPI
• Mesh (/cluster) Initialization
• Mesh AfterRolling Update
• Cluster Gossip Silences
• Reload Config
• Zero Downtime Rolling Deployment (this includes a webhook, receiving firing alerts)

You can find the tests here: https://github.com/coreos/prometheus-operator/blob/master/test/e2e/alertmanager_test.go

We could add a Kubernetes end-to-end test suite via minikube running on the travis nodes as a single node cluster, but I would rather like to invest into our current bare-metal test setup, as it is not bound to particular orchestrator.

Also, a thought: Is prometheus using the swagger-generated alertmanager client to send alerts?

Not yet :(

[stuartnelson3]

@stuartnelson3 I am not quite sure how your comment relates to these changes. Are you saying that our already high flakiness would increase more with continuously introducing changes in regards to our dependencies?

I am happy moving forward with this, but I think we'll still have to manually run tests to make sure everything is "ok". I'm imagining a bright and shiny future where this can be fully automated.

I would rather like to invest into our current bare-metal test setup, as it is not bound to particular orchestrator.

This sounds good to me. Some form of "client sends an alert, alertmanager processes it, verify that a webhook outputs the alert to stdout". We could probably reuse https://github.com/prometheus/alertmanager/blob/master/examples/webhook/echo.go.

But, this would be for another ticket.

[simonpasquier]

I agree with your concerns and I'm also be in favor of extending the test suite of AlertManager in this repository. That being said and as @mxinden already said, they seem a bit orthogonal to this work. Also regarding e2e testing from alert to webhook, isn't it already covered by the acceptance tests or did you mean something different?

do I understand it correctly, that

dependabot creates a PR against github.com/simonpasquier/alertmanager.
Your script updates the PR to also vendor all changes.
Open a PR with the same change set against github.com/prometheus/alertmanager.

Yes.

Is prometheus using the swagger-generated alertmanager client to send alerts?

No, this is somehow related to prometheus/prometheus#5167. I'm quite sure that POST /api/v1/alerts and POST /api/v2/alerts are identical so we could already use the v2 API client for both cases.

[stuartnelson3]

No worries, I'm just making comments that are unnecessary for this PR. I'm a 👍

[mxinden]

Sorry for dropping the ball here @simonpasquier.