KDD BGP: Issue getting an IP from host-local IPAM

[frnkdny]

Exercising calico-node v1.1.2-rc1 on K8s version 1.5.3 with the following docs:

• http://docs.projectcalico.org/master/getting-started/kubernetes/installation/hosted/kubernetes-datastore/
• kubectl apply -f http://docs.projectcalico.org/master/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico.yaml

Seeing the following error after calico-node is up but DNS would not start due to this issue:
Warning FailedSync Error syncing pod, skipping: failed to "SetupNetwork" for "kube-dns-v19-73g2s_kube-system" with SetupNetworkError: "Failed to setup network for pod \"kube-dns-v19-73g2s_kube-system(f4c0f4df-254c-11e7-88d2-42010af0000f)\" using network plugins \"cni\": no IP addresses available in network: k8s-pod-network; Skipping pod"

• The CNI config calico-ipam it’s using "type": "host-local" & ”subnet": "usePodCidr”.
• cluster-cidr is large enough: --cluster-cidr=10.244.0.0/16

——————Detail Output Below——————

NAMESPACE     NAME                 READY     STATUS              RESTARTS   AGE       IP            NODE
kube-system   kube-dns-v19-vng36   0/3       ContainerCreating   0          2m        <none>        tigera-scale-fd-highperf-004
kube-system   calico-node-1s5vl    2/2       Running             0          5m        10.240.0.45   tigera-scale-fd-highperf-004

CNI config:
$ kubectl logs calico-node-1s5vl -n kube-system install-cni
Wrote Calico CNI binaries to /host/opt/cni/bin/
CNI plugin version: v1.6.2-10-g78a894f
Wrote CNI config: {
    "name": "k8s-pod-network",
    "cniVersion": "0.3.1",
    "type": "calico",
    "log_level": "debug",
    "datastore_type": "kubernetes",
    "hostname": "tigera-scale-fd-highperf-004",
    "ipam": {
        "type": "host-local",
        "subnet": "usePodCidr"
    },
    "policy": {
        "type": "k8s",
        "k8s_auth_token": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkZWZhdWx0LXRva2VuLXI0MGJqIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImRlZmF1bHQiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIxODdhYzEzYy0yNTU2LTExZTctOTRlNy00MjAxMGFmMDAwMGYiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06ZGVmYXVsdCJ9.st9G0a4hGzBSojEfTe-J1la5_6Aqr2rITUdGdhitsF31qY6KktpHtH_35td52qBI7TOkY5HiK0fyVCuEO3_l4FYXGqwaLOCAC-QcNciL_wJ6HTRlk0mBQ7WZkbu3C4tcI5IHXmNe8qO36LnHhTDG2dCHmAuac2an8YUUKJ6Gi60o9t8LFwKMBcEsXqmNP_Y7zkG9XVqqtOkmEdlbFGSpiJsVAUVC1GlZ2NtFi5-uXZTrk62Xj7gidKDssnr4GKZnE0pF1psSz26CZSsRk3T_nONisnSSJw6b6J5kmcOlrc0qiSGBOLNRu3Wp7Ofs3gpEMwFCO6sl62TxChkZ9IL9Sg"
    },
    "kubernetes": {
        "k8s_api_root": "https://10.100.0.1:443",
        "kubeconfig": "/etc/cni/net.d/calico-kubeconfig"
    }
}
Done configuring CNI.  Sleep=true

$ cat /etc/systemd/system/kube-proxy.service
[Unit]
Description=Kubernetes Proxy
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
Requires=kubelet.service
After=kubelet.service
[Service]
ExecStartPre=/usr/bin/wget -N -P /opt/bin  https://storage.googleapis.com/kubernetes-release/release/v1.5.3/bin/linux/amd64/kube-proxy
ExecStartPre=/usr/bin/chmod +x /opt/bin/kube-proxy
ExecStart=/opt/bin/kube-proxy \
--master=http://10.240.0.15:8080 \
--proxy-mode=iptables \
--logtostderr=true
Restart=always
RestartSec=10

$ cat /etc/systemd/system/kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
Requires=kube-apiserver.service
After=kube-apiserver.service
[Service]
ExecStartPre=/usr/bin/wget -N -P /opt/bin https://storage.googleapis.com/kubernetes-release/release/v1.5.3/bin/linux/amd64/kube-controller-manager
ExecStartPre=/usr/bin/chmod +x /opt/bin/kube-controller-manager
ExecStart=/opt/bin/kube-controller-manager \
--master=10.240.0.15:8080 \
--service-account-private-key-file=/var/run/kubernetes/apiserver.key \
--root-ca-file=/var/run/kubernetes/apiserver.crt \
--allocate-node-cidrs=true \
--configure-cloud-routes=true \
--cloud-provider=gce \
--cloud-config=/etc/gce.conf \
--cluster-name=tigera-scale-fd \
--cluster-cidr=10.244.0.0/16 \
--logtostderr=true
Restart=always
RestartSec=10

[tmjd]

Here is what is passed to the ipam plugin
The strange characters in the token are from the conversion I believe.

{"cniVersion":"0.3.1","datastore_type":"kubernetes","hostname":"tigera-scale-fd-highperf-004","ipam":{"subnet":"10.244.11.0/24","type":"host-local"},"kubernetes":{"k8s_api_root":"https://10.100.0.1:443","kubeconfig":"/etc/cni/net.d/calico-kubeconfig"},"log_level":"debug","name":"k8s-pod-network","policy":{"k8s_auth_token":"eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkZWZhdWx0LXRva2VuLXI0��GJqIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImRlZmF1bHQiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIxODdhYzEzYy0yNTU2LTExZTctOTRlNy00MjAxMGFmMDAwMGYiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06ZGVmYXVsdCJ9.st9G0a4hGzBSojEfTe-J1la5_6Aqr2rITUdGdhitsF31qY6KktpHtH_35td52qBI7TOkY5HiK0fyVCuEO3_l4FYXGqwaLOCAC-QcNciL_wJ6HTRlk0mBQ7WZkbu3C4tcI5IHXmNe8qO36LnHhTDG2dCHmAuac2an8YUUKJ6Gi60o9t8LFwKMBcEsXqmNP_Y7zkG9XVqqtOkmEdlbFGSpiJsVAUVC1GlZ2NtFi5-uXZTrk62Xj7gidKDssnr4GKZnE0pF1psSz26CZSsRk3T_nONisnSSJw6b6J5kmcOlrc0qiSGBOLNRu3Wp7Ofs3gpEMwFCO6sl62TxChkZ9IL��Sg","type":"k8s"},"type":"calico"}

I think it looks correct from my look over the CNI spec docs but that was the first time I've looked at them.

[tmjd]

Here is a piece of the log that file that I think shows the problem but I do not understand why.

Apr 19 23:39:40 tigera-scale-fd-highperf-004.c.unique-caldron-775.internal kubelet[1525]: time="2017-04-19T23:39:40Z" level=debug msg="Fetched K8s labels" Workload=kube-system.kube-dns-v19-vng36 labels=map[k8s-app:kube-dns kubernetes.io/cluster-service:true version:v19 calico/k8s_ns:kube-system]
Apr 19 23:39:40 tigera-scale-fd-highperf-004.c.unique-caldron-775.internal kubelet[1525]: time="2017-04-19T23:39:40Z" level=debug msg="Fetched K8s annotations" Workload=kube-system.kube-dns-v19-vng36 annotations=map[scheduler.alpha.kubernetes.io/critical-pod: scheduler.alpha.kubernetes.io/tolerations:[{"key":"CriticalAddonsOnly", "operator":"Exists"}] kubernetes.io/created-by:{"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicationController","namespace":"kube-system","name":"kube-dns-v19","uid":"744ac676-2559-11e7-94e7-42010af0000f","apiVersion":"v1","resourceVersion":"3939"}}
Apr 19 23:39:40 tigera-scale-fd-highperf-004.c.unique-caldron-775.internal kubelet[1525]: ]
Apr 19 23:39:40 tigera-scale-fd-highperf-004.c.unique-caldron-775.internal kubelet[1525]: time="2017-04-19T23:39:40Z" level=debug msg="Calling IPAM plugin host-local" Workload=kube-system.kube-dns-v19-vng36
Apr 19 23:39:40 tigera-scale-fd-highperf-004.c.unique-caldron-775.internal kubelet[1525]: time="2017-04-19T23:39:40Z" level=debug msg="IPAM plugin returned: DNS:{Nameservers:[] Domain: Search:[] Options:[]}" Workload=kube-system.kube-dns-v19-vng36
Apr 19 23:39:40 tigera-scale-fd-highperf-004.c.unique-caldron-775.internal kubelet[1525]: E0419 23:39:40.912648    1525 cni.go:255] Error adding network: IPAM plugin returned missing IP config
Apr 19 23:39:40 tigera-scale-fd-highperf-004.c.unique-caldron-775.internal kubelet[1525]: E0419 23:39:40.912689    1525 cni.go:209] Error while adding to cni network: IPAM plugin returned missing IP config
Apr 19 23:39:40 tigera-scale-fd-highperf-004.c.unique-caldron-775.internal kubelet[1525]: E0419 23:39:40.912720    1525 docker_manager.go:2196] Failed to setup network for pod "kube-dns-v19-vng36_kube-system(744bb470-2559-11e7-94e7-42010af0000f)" using network plugins "cni": IPAM plugin returned missing IP config; Skipping pod

I think it would be helpful to see logs from the host-local plugin, but I don't know how to enable those or where they would be.

[tomdee]

@lwr20 with you to suggest a next action

[lwr20]

Another datapoint - I did not see this using k8s v1.6.1 installed using kubeadm.

I think I should try to repro this using the info above. If that fails, then we should close.

[lwr20]

I'm not seeing this when running a calico-k8s-cluster installed rig with:

K8S_VER=v1.5.3
CNI_VER=v0.5.2
CALICO_POLICY_VER=v0.6.0
CALICO_CNI_VER=v1.7.0
CALICO_CONT_VER=v1.1.2-rc2
CALICOCTL_VER=v1.1.2-rc2

Similar diags to before:

$ kubectl logs calico-node-3lssr -n kube-system install-cni
Installing any TLS assets from /calico-secrets
cp: can't stat '/calico-secrets/*': No such file or directory
Wrote Calico CNI binaries to /host/opt/cni/bin/
CNI plugin version: v1.7.0
Wrote CNI config: {
    "name": "k8s-pod-network",
    "cniVersion": "0.3.1",
    "type": "calico",
    "etcd_endpoints": "http://tigera-kubetest-lwr-etcd-01:2379",
    "etcd_key_file": "",
    "etcd_cert_file": "",
    "etcd_ca_cert_file": "",
    "log_level": "info",
    "ipam": {
        "type": "host-local"
        "subnet": "usePodCidr"
    },
    "policy": {
        "type": "k8s",
        "k8s_api_root": "https://10.100.0.1:443",
        "k8s_auth_token": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjYWxpY28tbm9kZS10b2tlbi1oczkydiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJjYWxpY28tbm9kZSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjUwYTlhYWU2LTJhYTctMTFlNy1iZjViLTQyMDEwYWYwMDAxYiIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlLXN5c3RlbTpjYWxpY28tbm9kZSJ9.tspiW9vSy_vhah8o1pc-o0qZYgESm-Oa-Jn61VejI_d9LEtHICGGSG7K4JRz6R_VyEuyDAJjU-j2X0Mr-oMwoDEA2ELSxutcuUpsI8VU_G_SIhgk-1f_GXnSvSt_YAh_bTdS6GsvuTnZi8u4kaZb-03c1WzkNEZwAAN8mX_dCsjvnz-PoNs7FG07ls5UhXLkLjTUEN9PqrYU8E4rYciA7gmPPPkz5r-uaovD5zS8YaHrh67TNRkaiJAAi-6WIyLaesmvrr38YsPc-5AU3jNtA39IQMa-s1sdekKhuDzHoXzOPWForkbvHOiNXBY5-7cljKubbJsyqJnyevrWBXNIHw"
    },
    "kubernetes": {
        "kubeconfig": "/etc/cni/net.d/calico-kubeconfig"
    }
}
Done configuring CNI.  Sleep=true

$ cat /etc/systemd/system/kube-proxy.service
[Unit]
Description=Kubernetes Proxy
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
Requires=kubelet.service
After=kubelet.service
[Service]
ExecStartPre=/usr/bin/wget -N -P /opt/bin  https://storage.googleapis.com/kubernetes-release/release/v1.5.3/bin/linux/amd64/kube-proxy
ExecStartPre=/usr/bin/chmod +x /opt/bin/kube-proxy
ExecStart=/opt/bin/kube-proxy \
--master=http://10.240.0.27:8080 \
--proxy-mode=iptables \
--logtostderr=true
Restart=always
RestartSec=10

[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
Requires=kube-apiserver.service
After=kube-apiserver.service
[Service]
ExecStartPre=/usr/bin/wget -N -P /opt/bin https://storage.googleapis.com/kubernetes-release/release/v1.5.3/bin/linux/amd64/kube-controller-manager
ExecStartPre=/usr/bin/chmod +x /opt/bin/kube-controller-manager
ExecStart=/opt/bin/kube-controller-manager \
--master=10.240.0.27:8080 \
--service-account-private-key-file=/var/run/kubernetes/apiserver.key \
--root-ca-file=/var/run/kubernetes/apiserver.crt \
--allocate-node-cidrs=true \
--configure-cloud-routes=false \
--cluster-name=tigera-kubetest-lwr \
--cluster-cidr=10.244.0.0/16 \
--logtostderr=true
Restart=always
RestartSec=10

[ozdanborne]

@lwr20 How was this resolved? Couldn't reproduce?

[lwr20]

Exactly - failed to repro.