Panic on rolling update in Kubernetes

[hypnoglow]

Describe the bug

In Kubernetes on rolling update oathkeeper panics:

oathkeeper-58747c8474-nqvvp oathkeeper time="2019-07-22T14:41:44Z" level=debug msg="Detected access rule repository file change." event=fsnotify file="file:///etc/config/access-rules.json" op=CHMOD
oathkeeper-58747c8474-nqvvp oathkeeper time="2019-07-22T14:41:44Z" level=debug msg="Detected that a access rule repository file has been removed, reloading config."
oathkeeper-58747c8474-nqvvp oathkeeper time="2019-07-22T14:41:44Z" level=debug msg="Access rule watcher received an update." event=config_change source=fsnotify_remove
oathkeeper-58747c8474-nqvvp oathkeeper panic: send on closed channel
oathkeeper-58747c8474-nqvvp oathkeeper 
oathkeeper-58747c8474-nqvvp oathkeeper goroutine 118 [running]:
oathkeeper-58747c8474-nqvvp oathkeeper github.com/ory/oathkeeper/rule.(*FetcherDefault).watch.func4(0xc00007c540, 0xc00011dd00)
oathkeeper-58747c8474-nqvvp oathkeeper  /go/src/github.com/ory/oathkeeper/rule/fetcher_default.go:220 +0xb6
oathkeeper-58747c8474-nqvvp oathkeeper created by github.com/ory/oathkeeper/rule.(*FetcherDefault).watch
oathkeeper-58747c8474-nqvvp oathkeeper  /go/src/github.com/ory/oathkeeper/rule/fetcher_default.go:219 +0xedc

This is because race condition:

1. The new configmap is deployed (config gets updated in the mounted volume on the old container)
2. Old container is getting stopped (file watcher is closed)
3. There are still goroutines that are blocked sending to the channel, e.g. here

Expected behavior

Goroutines should not write to the closed channel.

How to fix

Make goroutines controllable using waitgroup, so no orphaned goroutines will try to write to the closed channel. Close the channel only when all goroutines exit.