weave-net not scheduling on nodes

I have gone through the course all the way to the weave net install.  I ran the apply command but my weavenet pods are not coming up.   It just keep show desired: 0 current: 0 ready: 0 available: 0.  I followed the course all the way through, the only difference I believe is I used 18.04 rather than 16.04 but I don't see any glaring differences between the two that would cause an issue.  Any assistance would be appreciated.  The nodes are showing NotReady and the only error in the logs is the missing cni config.

Thanks,
Chris
  • post-author-pic
    Michael M
    12-26-2018

    Hello, 


    It is possible that there may be some indication in the logs on the scheduler. If you run get pods --all-namespaces and locate the scheduler pod, and then get a bash shell on that pod you may be able to locate some more information in /var/log.
    There have been some reported issues with coredns on Ubuntu 18. 
    I would also review the weave command that you are using as it might need to be updated. it is possible that there may have been a patched version for the DNS issue. 

  • post-author-pic
    Christopher R
    12-26-2018

    Unfortunately I haven't even gotten to the DNS part.  Just bringing up weave the pods never get started.  I just have :

    NAME        DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
    weave-net 0 0 0 0 0 <none> 1d


    If I do describe nodes:


    Name:               creiche3c.mylabserver.com
    Roles: <none>
    Labels: beta.kubernetes.io/arch=amd64
    beta.kubernetes.io/os=linux
    kubernetes.io/hostname=creiche3c.mylabserver.com
    Annotations: volumes.kubernetes.io/controller-managed-attach-detach=true
    CreationTimestamp: Mon, 24 Dec 2018 14:25:20 -0500
    Taints: <none>
    Unschedulable: false
    Conditions:
    Type Status LastHeartbeatTime LastTransitionTime Reason Message
    ---- ------ ----------------- ------------------ ------ -------
    OutOfDisk False Wed, 26 Dec 2018 10:11:36 -0500 Mon, 24 Dec 2018 14:25:20 -0500 KubeletHasSufficientDisk kubelet has sufficient disk space available
    MemoryPressure False Wed, 26 Dec 2018 10:11:36 -0500 Mon, 24 Dec 2018 14:25:20 -0500 KubeletHasSufficientMemory kubelet has sufficient memory available
    DiskPressure False Wed, 26 Dec 2018 10:11:36 -0500 Mon, 24 Dec 2018 14:25:20 -0500 KubeletHasNoDiskPressure kubelet has no disk pressure
    PIDPressure False Wed, 26 Dec 2018 10:11:36 -0500 Mon, 24 Dec 2018 14:25:20 -0500 KubeletHasSufficientPID kubelet has sufficient PID available
    Ready False Wed, 26 Dec 2018 10:11:36 -0500 Mon, 24 Dec 2018 14:25:20 -0500 KubeletNotReady runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni config load failed: no network config found in /etc/cni/net.d: cni plugin not initialized: failed to load cni config
    Addresses:
    InternalIP: 172.31.29.230
    Hostname: creiche3c.mylabserver.com
    Capacity:
    cpu: 1
    ephemeral-storage: 20263484Ki
    hugepages-2Mi: 0
    memory: 2039724Ki
    pods: 110
    Allocatable:
    cpu: 1
    ephemeral-storage: 18674826824
    hugepages-2Mi: 0
    memory: 1937324Ki
    pods: 110
    System Info:
    Machine ID: 9ef2866073d1434aa3bdbde3f2f26eb1
    System UUID: EC2E0DD8-AA4E-C5CF-7B31-2F08827C9373
    Boot ID: d67417ad-7380-493d-a74e-46cddd52d071
    Kernel Version: 4.15.0-1031-aws
    OS Image: Ubuntu 18.04.1 LTS
    Operating System: linux
    Architecture: amd64
    Container Runtime Version: containerd://1.1.0
    Kubelet Version: v1.10.2
    Kube-Proxy Version: v1.10.2
    ExternalID: creiche3c.mylabserver.com
    Non-terminated Pods: (0 in total)
    Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits
    --------- ---- ------------ ---------- --------------- -------------
    Allocated resources:
    (Total limits may be over 100 percent, i.e., overcommitted.)
    CPU Requests CPU Limits Memory Requests Memory Limits
    ------------ ---------- --------------- -------------
    0 (0%) 0 (0%) 0 (0%) 0 (0%)
    Events:
    Type Reason Age From Message
    ---- ------ ---- ---- -------
    Normal Starting 25m kube-proxy, creiche3c.mylabserver.com Starting kube-proxy.
    Normal Starting 16m kubelet, creiche3c.mylabserver.com Starting kubelet.
    Warning InvalidDiskCapacity 16m kubelet, creiche3c.mylabserver.com invalid capacity 0 on image filesystem
    Normal NodeHasSufficientDisk 16m kubelet, creiche3c.mylabserver.com Node creiche3c.mylabserver.com status is now: NodeHasSufficientDisk
    Normal NodeHasSufficientMemory 16m kubelet, creiche3c.mylabserver.com Node creiche3c.mylabserver.com status is now: NodeHasSufficientMemory
    Normal NodeHasNoDiskPressure 16m kubelet, creiche3c.mylabserver.com Node creiche3c.mylabserver.com status is now: NodeHasNoDiskPressure
    Normal NodeHasSufficientPID 16m kubelet, creiche3c.mylabserver.com Node creiche3c.mylabserver.com status is now: NodeHasSufficientPID
    Normal NodeAllocatableEnforced 16m kubelet, creiche3c.mylabserver.com Updated Node Allocatable limit across pods
    Warning Rebooted 16m kubelet, creiche3c.mylabserver.com Node creiche3c.mylabserver.com has been rebooted, boot id: d67417ad-7380-493d-a74e-46cddd52d071
    Normal Starting 16m kubelet, creiche3c.mylabserver.com Starting kubelet.
    Warning InvalidDiskCapacity 16m kubelet, creiche3c.mylabserver.com invalid capacity 0 on image filesystem
    Normal NodeAllocatableEnforced 16m kubelet, creiche3c.mylabserver.com Updated Node Allocatable limit across pods
    Normal NodeHasSufficientDisk 16m kubelet, creiche3c.mylabserver.com Node creiche3c.mylabserver.com status is now: NodeHasSufficientDisk
    Normal NodeHasSufficientMemory 16m kubelet, creiche3c.mylabserver.com Node creiche3c.mylabserver.com status is now: NodeHasSufficientMemory
    Normal NodeHasNoDiskPressure 16m kubelet, creiche3c.mylabserver.com Node creiche3c.mylabserver.com status is now: NodeHasNoDiskPressure
    Normal NodeHasSufficientPID 16m kubelet, creiche3c.mylabserver.com Node creiche3c.mylabserver.com status is now: NodeHasSufficientPID


    Name: creiche4c.mylabserver.com
    Roles: <none>
    Labels: beta.kubernetes.io/arch=amd64
    beta.kubernetes.io/os=linux
    kubernetes.io/hostname=creiche4c.mylabserver.com
    Annotations: volumes.kubernetes.io/controller-managed-attach-detach=true
    CreationTimestamp: Mon, 24 Dec 2018 14:25:18 -0500
    Taints: <none>
    Unschedulable: false
    Conditions:
    Type Status LastHeartbeatTime LastTransitionTime Reason Message
    ---- ------ ----------------- ------------------ ------ -------
    OutOfDisk False Wed, 26 Dec 2018 10:11:30 -0500 Mon, 24 Dec 2018 14:25:18 -0500 KubeletHasSufficientDisk kubelet has sufficient disk space available
    MemoryPressure False Wed, 26 Dec 2018 10:11:30 -0500 Mon, 24 Dec 2018 14:25:18 -0500 KubeletHasSufficientMemory kubelet has sufficient memory available
    DiskPressure False Wed, 26 Dec 2018 10:11:30 -0500 Mon, 24 Dec 2018 14:25:18 -0500 KubeletHasNoDiskPressure kubelet has no disk pressure
    PIDPressure False Wed, 26 Dec 2018 10:11:30 -0500 Mon, 24 Dec 2018 14:25:18 -0500 KubeletHasSufficientPID kubelet has sufficient PID available
    Ready False Wed, 26 Dec 2018 10:11:30 -0500 Mon, 24 Dec 2018 14:25:18 -0500 KubeletNotReady runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni config load failed: no network config found in /etc/cni/net.d: cni plugin not initialized: failed to load cni config
    Addresses:
    InternalIP: 172.31.21.74
    Hostname: creiche4c.mylabserver.com
    Capacity:
    cpu: 1
    ephemeral-storage: 20263484Ki
    hugepages-2Mi: 0
    memory: 2039724Ki
    pods: 110
    Allocatable:
    cpu: 1
    ephemeral-storage: 18674826824
    hugepages-2Mi: 0
    memory: 1937324Ki
    pods: 110
    System Info:
    Machine ID: 9ef2866073d1434aa3bdbde3f2f26eb1
    System UUID: EC24050F-65F2-10BE-A99B-8C0708769F5E
    Boot ID: 9dac739b-affe-40d2-a2e1-5a683f19f28b
    Kernel Version: 4.15.0-1031-aws
    OS Image: Ubuntu 18.04.1 LTS
    Operating System: linux
    Architecture: amd64
    Container Runtime Version: containerd://1.1.0
    Kubelet Version: v1.10.2
    Kube-Proxy Version: v1.10.2
    ExternalID: creiche4c.mylabserver.com
    Non-terminated Pods: (0 in total)
    Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits
    --------- ---- ------------ ---------- --------------- -------------
    Allocated resources:
    (Total limits may be over 100 percent, i.e., overcommitted.)
    CPU Requests CPU Limits Memory Requests Memory Limits
    ------------ ---------- --------------- -------------
    0 (0%) 0 (0%) 0 (0%) 0 (0%)
    Events:
    Type Reason Age From Message
    ---- ------ ---- ---- -------
    Normal Starting 27m kube-proxy, creiche4c.mylabserver.com Starting kube-proxy.
    Normal NodeHasSufficientPID 27m kubelet, creiche4c.mylabserver.com Node creiche4c.mylabserver.com status is now: NodeHasSufficientPID
    Normal Starting 27m kubelet, creiche4c.mylabserver.com Starting kubelet.
    Warning InvalidDiskCapacity 27m kubelet, creiche4c.mylabserver.com invalid capacity 0 on image filesystem
    Normal NodeHasSufficientMemory 27m kubelet, creiche4c.mylabserver.com Node creiche4c.mylabserver.com status is now: NodeHasSufficientMemory
    Normal NodeHasNoDiskPressure 27m kubelet, creiche4c.mylabserver.com Node creiche4c.mylabserver.com status is now: NodeHasNoDiskPressure
    Normal NodeAllocatableEnforced 27m kubelet, creiche4c.mylabserver.com Updated Node Allocatable limit across pods
    Warning Rebooted 27m kubelet, creiche4c.mylabserver.com Node creiche4c.mylabserver.com has been rebooted, boot id: dee4880a-3d8d-468c-a629-aba479fe49d9
    Normal NodeHasSufficientDisk 27m kubelet, creiche4c.mylabserver.com Node creiche4c.mylabserver.com status is now: NodeHasSufficientDisk
    Normal Starting 3m kube-proxy, creiche4c.mylabserver.com Starting kube-proxy.
    Warning InvalidDiskCapacity 3m kubelet, creiche4c.mylabserver.com invalid capacity 0 on image filesystem
    Normal Starting 3m kubelet, creiche4c.mylabserver.com Starting kubelet.
    Normal NodeHasSufficientDisk 2m kubelet, creiche4c.mylabserver.com Node creiche4c.mylabserver.com status is now: NodeHasSufficientDisk
    Normal NodeHasSufficientMemory 2m kubelet, creiche4c.mylabserver.com Node creiche4c.mylabserver.com status is now: NodeHasSufficientMemory
    Normal NodeHasNoDiskPressure 2m kubelet, creiche4c.mylabserver.com Node creiche4c.mylabserver.com status is now: NodeHasNoDiskPressure
    Normal NodeHasSufficientPID 2m kubelet, creiche4c.mylabserver.com Node creiche4c.mylabserver.com status is now: NodeHasSufficientPID
    Warning Rebooted 2m kubelet, creiche4c.mylabserver.com Node creiche4c.mylabserver.com has been rebooted, boot id: 9dac739b-affe-40d2-a2e1-5a683f19f28b
    Normal NodeAllocatableEnforced 2m kubelet, creiche4c.mylabserver.com Updated Node Allocatable limit across pods



  • post-author-pic
    Christopher R
    12-26-2018

    Ah I finally got it.  I found occasionally I was getting:

    kubectl get componentstatuses
    NAME STATUS MESSAGE ERROR
    controller-manager Unhealthy Get http://127.0.0.1:10252/healthz: dial tcp 127.0.0.1:10252: getsockopt: connection refused
    Then saw in the kube-controller-manager logs:
    error reading CA key file "/var/lib/kubernetes/ca-key.pem": open /var/lib/kubernetes/ca-key.pem: no such file or directory

    somehow I copied everything but ca-key.pem to /var/lib/kubernetes, copied that over and right everything started up right up.

    NAME              READY     STATUS    RESTARTS   AGE
    weave-net-9smx9 2/2 Running 0 33s
    weave-net-bjwm5 2/2 Running 0 33s


    Looks like I'm set now but hopefully this helps others know where to look if they run into the same issue.

    Thanks,
    Chris

Looking For Team Training?

Learn More