K8's CNI error registering worker nodes with cluster

I am following the Kubernetes the Hard Way lesson and am unable to join the worker nodes to the cluster. My containerd, kube-proxy, and kubelet service active and running but I see the following error in journalctl. 

kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni config load failed: no network config found in /etc/cni/net.d: cni plugin not initialized: failed to load cni config


  • post-author-pic
    Will B
    11-12-2018

    Have you gotten to the Networking section yet? That is a normal message until you are ready to install the networking plugin. What does your kubectl get nodes say? It should list your worker nodes, but they will be in the NotReady state, which is OK until networking is installed.


    If the nodes aren't showing up in kubectl get nodes, can you try journalctl -u kubelet and provide the log? There's likely something else besides that CNI message.

  • post-author-pic
    goodyman05
    11-12-2018

    kubectl get nodes returns "no resources found". I completed the Worker node bootstrapping section.  

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:39.974542 1183 feature_gate.go:226] feature gates: &{{} map[]}

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: Flag --allow-privileged has been deprecated, will be removed in a future version

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:39.982167 1183 feature_gate.go:226] feature gates: &{{} map[]}

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.007799 1183 mount_linux.go:211] Detected OS with systemd

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: W1112 22:52:40.012771 1183 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.022979 1183 server.go:376] Version: v1.10.2

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.023016 1183 feature_gate.go:226] feature gates: &{{} map[]}

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.023094 1183 plugins.go:89] No cloud provider specified.

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.023105 1183 server.go:492] No cloud provider specified: "" from the config file: ""

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.033011 1183 manager.go:154] cAdvisor running in container: "/sys/fs/cgroup/cpu,cpuacct/system.slice/kubelet.service"

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.049401 1183 fs.go:142] Filesystem UUIDs: map[4573eb39-57f3-439b-9a73-8aef508afd3f:/dev/xvda1]

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.049420 1183 fs.go:143] Filesystem partitions: map[tmpfs:{mountpoint:/run major:0 minor:18 fsType:tmpfs blockSize:0} /dev/xvda1:{mountpoint:/ major:202 minor:1 fsType:ext4 blockSize:0

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.080415 1183 manager.go:227] Machine: {NumCores:1 CpuFrequency:2400040 MemoryCapacity:1038839808 HugePages:[{PageSize:2048 NumPages:0}] MachineID:e156aebfbcac49b4bed31684a6b812cb Syst

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.080660 1183 manager.go:233] Version: {KernelVersion:4.4.0-1070-aws ContainerOsVersion:Ubuntu 16.04.5 LTS DockerVersion:Unknown DockerAPIVersion:Unknown CadvisorVersion: CadvisorRevis

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.082023 1183 server.go:613] --cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to /

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.082657 1183 container_manager_linux.go:242] container manager verified user specified cgroup-root exists: /

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.082669 1183 container_manager_linux.go:247] Creating Container Manager object based on Node Config: {RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:remot

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.082746 1183 container_manager_linux.go:266] Creating device plugin manager: true

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.082753 1183 manager.go:102] Creating Device Plugin manager at /var/lib/kubelet/device-plugins/kubelet.sock

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.082803 1183 state_mem.go:36] [cpumanager] initializing new in-memory state store

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.083378 1183 state_mem.go:84] [cpumanager] updated default cpuset: ""

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.083392 1183 state_mem.go:92] [cpumanager] updated cpuset assignments: "map[]"

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.083421 1183 state_file.go:119] [cpumanager] state file: restored state from state file "/var/lib/kubelet/cpu_manager_state"

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.083426 1183 state_file.go:120] [cpumanager] state file: defaultCPUSet:

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.083470 1183 server.go:887] Using root directory: /var/lib/kubelet

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.085808 1183 kubelet.go:297] Watching apiserver

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: W1112 22:52:40.092955 1183 kubelet_network.go:132] Hairpin mode set to "promiscuous-bridge" but container runtime is "remote", ignoring

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.092974 1183 kubelet.go:556] Hairpin mode set to "none"

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: W1112 22:52:40.093054 1183 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.093063 1183 plugins.go:190] Loaded network plugin "cni"

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.093078 1183 remote_runtime.go:43] Connecting to runtime service unix:///var/run/containerd/containerd.sock

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.107164 1183 kuberuntime_manager.go:186] Container runtime containerd initialized, version: v1.1.0, apiVersion: v1alpha2

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.109714 1183 plugins.go:454] Loaded volume plugin "kubernetes.io/aws-ebs"

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.109731 1183 plugins.go:454] Loaded volume plugin "kubernetes.io/empty-dir"

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.109739 1183 plugins.go:454] Loaded volume plugin "kubernetes.io/gce-pd"

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.109746 1183 plugins.go:454] Loaded volume plugin "kubernetes.io/git-repo"

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.109753 1183 plugins.go:454] Loaded volume plugin "kubernetes.io/host-path"

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.109761 1183 plugins.go:454] Loaded volume plugin "kubernetes.io/nfs"

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.109768 1183 plugins.go:454] Loaded volume plugin "kubernetes.io/secret"

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.109779 1183 plugins.go:454] Loaded volume plugin "kubernetes.io/iscsi"

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.109790 1183 plugins.go:454] Loaded volume plugin "kubernetes.io/glusterfs"

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.109798 1183 plugins.go:454] Loaded volume plugin "kubernetes.io/rbd"

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.109806 1183 plugins.go:454] Loaded volume plugin "kubernetes.io/cinder"

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.109813 1183 plugins.go:454] Loaded volume plugin "kubernetes.io/quobyte"

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.109823 1183 plugins.go:454] Loaded volume plugin "kubernetes.io/cephfs"

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.109833 1183 plugins.go:454] Loaded volume plugin "kubernetes.io/downward-api"

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.109841 1183 plugins.go:454] Loaded volume plugin "kubernetes.io/fc"

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.109848 1183 plugins.go:454] Loaded volume plugin "kubernetes.io/flocker"

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.109855 1183 plugins.go:454] Loaded volume plugin "kubernetes.io/azure-file"

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.109863 1183 plugins.go:454] Loaded volume plugin "kubernetes.io/configmap"

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.109871 1183 plugins.go:454] Loaded volume plugin "kubernetes.io/vsphere-volume"

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.109879 1183 plugins.go:454] Loaded volume plugin "kubernetes.io/azure-disk"

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.109886 1183 plugins.go:454] Loaded volume plugin "kubernetes.io/photon-pd"

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.109894 1183 plugins.go:454] Loaded volume plugin "kubernetes.io/projected"

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.109902 1183 plugins.go:454] Loaded volume plugin "kubernetes.io/portworx-volume"

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.109910 1183 plugins.go:454] Loaded volume plugin "kubernetes.io/scaleio"

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.109942 1183 plugins.go:454] Loaded volume plugin "kubernetes.io/local-volume"

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.109952 1183 plugins.go:454] Loaded volume plugin "kubernetes.io/storageos"

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.111310 1183 csi_plugin.go:61] kubernetes.io/csi: plugin initializing...

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.111327 1183 plugins.go:454] Loaded volume plugin "kubernetes.io/csi"

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.111590 1183 server.go:129] Starting to listen on 0.0.0.0:10250

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.113146 1183 server.go:299] Adding debug handlers to kubelet server.

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.129672 1183 fs_resource_analyzer.go:66] Starting FS ResourceAnalyzer

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.131018 1183 status_manager.go:140] Starting to sync pod status with apiserver

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.131037 1183 kubelet.go:1777] Starting kubelet main sync loop.

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.131066 1183 kubelet.go:1794] skipping pod synchronization - [container runtime is down PLEG is not healthy: pleg was last seen active 2562047h47m16.854775807s ago; threshold is 3m0s]

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.131237 1183 server.go:944] Started kubelet

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.131375 1183 volume_manager.go:245] The desired_state_of_world populator starts

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.131381 1183 volume_manager.go:247] Starting Kubelet Volume Manager

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.149262 1183 desired_state_of_world_populator.go:129] Desired state populator starts to run

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: E1112 22:52:40.168180 1183 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni config load failed: no net

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: E1112 22:52:40.170419 1183 cri_stats_provider.go:236] Failed to get the info of the filesystem with mountpoint "/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs": unable to find data for

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: E1112 22:52:40.170701 1183 kubelet.go:1277] Image garbage collection failed once. Stats initialization may not have completed yet: invalid capacity 0 on image filesystem

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.171887 1183 factory.go:136] Registering containerd factory

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.172153 1183 factory.go:54] Registering systemd factory

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.172481 1183 factory.go:86] Registering Raw factory

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.172796 1183 manager.go:1205] Started watching for new ooms in manager

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.174192 1183 manager.go:356] Starting recovery of all containers

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.291786 1183 manager.go:361] Recovery completed

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.309721 1183 kubelet.go:1794] skipping pod synchronization - [container runtime is down]

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.310034 1183 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.351621 1183 kubelet_node_status.go:422] Recording NodeHasSufficientDisk event message for node goodyman053.mylabserver.com

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.351955 1183 kubelet_node_status.go:422] Recording NodeHasSufficientMemory event message for node goodyman053.mylabserver.com

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.352154 1183 kubelet_node_status.go:422] Recording NodeHasNoDiskPressure event message for node goodyman053.mylabserver.com

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.352350 1183 kubelet_node_status.go:422] Recording NodeHasSufficientPID event message for node goodyman053.mylabserver.com

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.352539 1183 kubelet_node_status.go:82] Attempting to register node goodyman053.mylabserver.com

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.459764 1183 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.462504 1183 kubelet_node_status.go:422] Recording NodeHasSufficientDisk event message for node goodyman053.mylabserver.com

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.462728 1183 kubelet_node_status.go:422] Recording NodeHasSufficientMemory event message for node goodyman053.mylabserver.com

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.462925 1183 kubelet_node_status.go:422] Recording NodeHasNoDiskPressure event message for node goodyman053.mylabserver.com

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.464669 1183 kubelet_node_status.go:422] Recording NodeHasSufficientPID event message for node goodyman053.mylabserver.com

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.464878 1183 cpu_manager.go:155] [cpumanager] starting with none policy

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.465052 1183 cpu_manager.go:156] [cpumanager] reconciling every 10s

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.465223 1183 policy_none.go:42] [cpumanager] none policy: Start

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.465599 1183 container_manager_linux.go:369] Updating kernel flag: vm/overcommit_memory, expected value: 1, actual value: 0

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.465842 1183 container_manager_linux.go:369] Updating kernel flag: kernel/panic, expected value: 10, actual value: 0

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.466061 1183 container_manager_linux.go:369] Updating kernel flag: kernel/panic_on_oops, expected value: 1, actual value: 0

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.485376 1183 manager.go:205] Starting Device Plugin manager

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: Starting Device Plugin manager

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:52:40.489693 1183 manager.go:237] Serving device plugin registration server on "/var/lib/kubelet/device-plugins/kubelet.sock"

    Nov 12 22:52:40 goodyman053.mylabserver.com kubelet[1183]: E1112 22:52:40.492034 1183 eviction_manager.go:246] eviction manager: failed to get get summary stats: failed to get node info: node "goodyman053.mylabserver.com" not found

    Nov 12 22:52:45 goodyman053.mylabserver.com kubelet[1183]: E1112 22:52:45.490670 1183 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni config load failed: no net

    Nov 12 22:52:50 goodyman053.mylabserver.com kubelet[1183]: E1112 22:52:50.491336 1183 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni config load failed: no net

    Nov 12 22:52:50 goodyman053.mylabserver.com kubelet[1183]: E1112 22:52:50.494336 1183 eviction_manager.go:246] eviction manager: failed to get get summary stats: failed to get node info: node "goodyman053.mylabserver.com" not found

    Nov 12 22:52:55 goodyman053.mylabserver.com kubelet[1183]: E1112 22:52:55.491998 1183 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni config load failed: no net

    Nov 12 22:53:00 goodyman053.mylabserver.com kubelet[1183]: E1112 22:53:00.492729 1183 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni config load failed: no net

    Nov 12 22:53:00 goodyman053.mylabserver.com kubelet[1183]: E1112 22:53:00.494617 1183 eviction_manager.go:246] eviction manager: failed to get get summary stats: failed to get node info: node "goodyman053.mylabserver.com" not found

    Nov 12 22:53:05 goodyman053.mylabserver.com kubelet[1183]: E1112 22:53:05.493457 1183 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni config load failed: no net

    Nov 12 22:53:10 goodyman053.mylabserver.com kubelet[1183]: E1112 22:53:10.088825 1183 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://52.40.249.106:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dg

    Nov 12 22:53:10 goodyman053.mylabserver.com kubelet[1183]: E1112 22:53:10.089301 1183 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://52.40.249.106:6443/api/v1/services?limit=500&resourceVersion=0: di

    Nov 12 22:53:10 goodyman053.mylabserver.com kubelet[1183]: E1112 22:53:10.089614 1183 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://52.40.249.106:6443/api/v1/nodes?fieldSelector=metadata.name%3Dgoodyma

    Nov 12 22:53:10 goodyman053.mylabserver.com kubelet[1183]: E1112 22:53:10.149738 1183 event.go:209] Unable to write event: 'Post https://52.40.249.106:6443/api/v1/namespaces/default/events: dial tcp 52.40.249.106:6443: i/o timeout' (may retry after sleepin

    Nov 12 22:53:10 goodyman053.mylabserver.com kubelet[1183]: E1112 22:53:10.353102 1183 kubelet_node_status.go:106] Unable to register node "goodyman053.mylabserver.com" with API server: Post https://52.40.249.106:6443/api/v1/nodes: dial tcp 52.40.249.106:64

    Nov 12 22:53:10 goodyman053.mylabserver.com kubelet[1183]: E1112 22:53:10.494588 1183 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni config load failed: no net

    Nov 12 22:53:10 goodyman053.mylabserver.com kubelet[1183]: E1112 22:53:10.495029 1183 eviction_manager.go:246] eviction manager: failed to get get summary stats: failed to get node info: node "goodyman053.mylabserver.com" not found

    Nov 12 22:53:10 goodyman053.mylabserver.com kubelet[1183]: I1112 22:53:10.553733 1183 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach

    Nov 12 22:53:10 goodyman053.mylabserver.com kubelet[1183]: I1112 22:53:10.555014 1183 kubelet_node_status.go:422] Recording NodeHasSufficientDisk event message for node goodyman053.mylabserver.com

    Nov 12 22:53:10 goodyman053.mylabserver.com kubelet[1183]: I1112 22:53:10.555261 1183 kubelet_node_status.go:422] Recording NodeHasSufficientMemory event message for node goodyman053.mylabserver.com

    Nov 12 22:53:10 goodyman053.mylabserver.com kubelet[1183]: I1112 22:53:10.555479 1183 kubelet_node_status.go:422] Recording NodeHasNoDiskPressure event message for node goodyman053.mylabserver.com

    Nov 12 22:53:10 goodyman053.mylabserver.com kubelet[1183]: I1112 22:53:10.555696 1183 kubelet_node_status.go:422] Recording NodeHasSufficientPID event message for node goodyman053.mylabserver.com

    Nov 12 22:53:10 goodyman053.mylabserver.com kubelet[1183]: I1112 22:53:10.555912 1183 kubelet_node_status.go:82] Attempting to register node goodyman053.mylabserver.com

    Nov 12 22:53:15 goodyman053.mylabserver.com kubelet[1183]: E1112 22:53:15.495657 1183 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni config load failed: no net

    Nov 12 22:53:20 goodyman053.mylabserver.com kubelet[1183]: E1112 22:53:20.495517 1183 eviction_manager.go:246] eviction manager: failed to get get summary stats: failed to get node info: node "goodyman053.mylabserver.com" not found

    Nov 12 22:53:20 goodyman053.mylabserver.com kubelet[1183]: E1112 22:53:20.496542 1183 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni config load failed: no net

    Nov 12 22:53:25 goodyman053.mylabserver.com kubelet[1183]: E1112 22:53:25.497549 1183 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni config load failed: no net

    Nov 12 22:53:30 goodyman053.mylabserver.com kubelet[1183]: E1112 22:53:30.496392 1183 eviction_manager.go:246] eviction manager: failed to get get summary stats: failed to get node info: node "goodyman053.mylabserver.com" not found

    Nov 12 22:53:30 goodyman053.mylabserver.com kubelet[1183]: E1112 22:53:30.498902 1183 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni config load failed: no net

    Nov 12 22:53:35 goodyman053.mylabserver.com kubelet[1183]: E1112 22:53:35.499836 1183 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni config load failed: no net

    Nov 12 22:53:40 goodyman053.mylabserver.com kubelet[1183]: E1112 22:53:40.514598 1183 eviction_manager.go:246] eviction manager: failed to get get summary stats: failed to get node info: node "goodyman053.mylabserver.com" not found

    Nov 12 22:53:40 goodyman053.mylabserver.com kubelet[1183]: E1112 22:53:40.514943 1183 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni config load failed: no net

    Nov 12 22:53:40 goodyman053.mylabserver.com kubelet[1183]: E1112 22:53:40.556470 1183 kubelet_node_status.go:106] Unable to register node "goodyman053.mylabserver.com" with API server: Post https://52.40.249.106:6443/api/v1/nodes: dial tcp 52.40.249.106:64

    Nov 12 22:53:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:53:40.956913 1183 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach

    Nov 12 22:53:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:53:40.958051 1183 kubelet_node_status.go:422] Recording NodeHasSufficientDisk event message for node goodyman053.mylabserver.com

    Nov 12 22:53:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:53:40.958302 1183 kubelet_node_status.go:422] Recording NodeHasSufficientMemory event message for node goodyman053.mylabserver.com

    Nov 12 22:53:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:53:40.958549 1183 kubelet_node_status.go:422] Recording NodeHasNoDiskPressure event message for node goodyman053.mylabserver.com

    Nov 12 22:53:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:53:40.958755 1183 kubelet_node_status.go:422] Recording NodeHasSufficientPID event message for node goodyman053.mylabserver.com

    Nov 12 22:53:40 goodyman053.mylabserver.com kubelet[1183]: I1112 22:53:40.958977 1183 kubelet_node_status.go:82] Attempting to register node goodyman053.mylabserver.com

    Nov 12 22:53:41 goodyman053.mylabserver.com kubelet[1183]: E1112 22:53:41.089863 1183 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://52.40.249.106:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dg

    Nov 12 22:53:41 goodyman053.mylabserver.com kubelet[1183]: E1112 22:53:41.091252 1183 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://52.40.249.106:6443/api/v1/services?limit=500&resourceVersion=0: di

    Nov 12 22:53:41 goodyman053.mylabserver.com kubelet[1183]: E1112 22:53:41.092338 1183 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://52.40.249.106:6443/api/v1/nodes?fieldSelector=metadata.name%3Dgoodyma

    Nov 12 22:53:41 goodyman053.mylabserver.com kubelet[1183]: E1112 22:53:41.429874 1183 event.go:209] Unable to write event: 'Post https://52.40.249.106:6443/api/v1/namespaces/default/events: dial tcp 52.40.249.106:6443: i/o timeout' (may retry after sleepin

    Nov 12 22:53:45 goodyman053.mylabserver.com kubelet[1183]: E1112 22:53:45.515864 1183 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni config load failed: no net

    Nov 12 22:53:50 goodyman053.mylabserver.com kubelet[1183]: E1112 22:53:50.515070 1183 eviction_manager.go:246] eviction manager: failed to get get summary stats: failed to get node info: node "goodyman053.mylabserver.com" not found

    Nov 12 22:53:50 goodyman053.mylabserver.com kubelet[1183]: E1112 22:53:50.516836 1183 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni config load failed: no net

    Nov 12 22:53:55 goodyman053.mylabserver.com kubelet[1183]: E1112 22:53:55.517773 1183 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni config load failed: no net

    Nov 12 22:54:00 goodyman053.mylabserver.com kubelet[1183]: E1112 22:54:00.515651 1183 eviction_manager.go:246] eviction manager: failed to get get summary stats: failed to get node info: node "goodyman053.mylabserver.com" not found

    Nov 12 22:54:00 goodyman053.mylabserver.com kubelet[1183]: E1112 22:54:00.519120 1183 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni config load failed: no net

    Nov 12 22:54:05 goodyman053.mylabserver.com kubelet[1183]: E1112 22:54:05.520224 1183 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni config load failed: no net

    Nov 12 22:54:10 goodyman053.mylabserver.com kubelet[1183]: E1112 22:54:10.516242 1183 eviction_manager.go:246] eviction manager: failed to get get summary stats: failed to get node info: node "goodyman053.mylabserver.com" not found

    Nov 12 22:54:10 goodyman053.mylabserver.com kubelet[1183]: E1112 22:54:10.521700 1183 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni config load failed: no net

    Nov 12 22:54:10 goodyman053.mylabserver.com kubelet[1183]: E1112 22:54:10.959558 1183 kubelet_node_status.go:106] Unable to register node "goodyman053.mylabserver.com" with API server: Post https://52.40.249.106:6443/api/v1/nodes: dial tcp 52.40.249.106:64

    Nov 12 22:54:11 goodyman053.mylabserver.com kubelet[1183]: I1112 22:54:11.760158 1183 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach

    Nov 12 22:54:11 goodyman053.mylabserver.com kubelet[1183]: I1112 22:54:11.761401 1183 kubelet_node_status.go:422] Recording NodeHasSufficientDisk event message for node goodyman053.mylabserver.com

    Nov 12 22:54:11 goodyman053.mylabserver.com kubelet[1183]: I1112 22:54:11.761654 1183 kubelet_node_status.go:422] Recording NodeHasSufficientMemory event message for node goodyman053.mylabserver.com

    Nov 12 22:54:11 goodyman053.mylabserver.com kubelet[1183]: I1112 22:54:11.761908 1183 kubelet_node_status.go:422] Recording NodeHasNoDiskPressure event message for node goodyman053.mylabserver.com

    Nov 12 22:54:11 goodyman053.mylabserver.com kubelet[1183]: I1112 22:54:11.762140 1183 kubelet_node_status.go:422] Recording NodeHasSufficientPID event message for node goodyman053.mylabserver.com

    Nov 12 22:54:11 goodyman053.mylabserver.com kubelet[1183]: I1112 22:54:11.762368 1183 kubelet_node_status.go:82] Attempting to register node goodyman053.mylabserver.com

    Nov 12 22:54:12 goodyman053.mylabserver.com kubelet[1183]: E1112 22:54:12.090849 1183 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://52.40.249.106:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dg

    Nov 12 22:54:12 goodyman053.mylabserver.com kubelet[1183]: E1112 22:54:12.092753 1183 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://52.40.249.106:6443/api/v1/services?limit=500&resourceVersion=0: di

    Nov 12 22:54:12 goodyman053.mylabserver.com kubelet[1183]: E1112 22:54:12.093354 1183 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://52.40.249.106:6443/api/v1/nodes?fieldSelector=metadata.name%3Dgoodyma

    Nov 12 22:54:15 goodyman053.mylabserver.com kubelet[1183]: E1112 22:54:15.522642 1183 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni config load failed: no net

    lines 286-342


  • post-author-pic
    Will B
    11-13-2018

    It looks like the relevant log message is this:

    22:53:10.353102 1183 kubelet_node_status.go:106] Unable to register node "goodyman053.mylabserver.com" with API server: Post https://52.40.249.106:6443/api/v1/nodes: dial tcp 52.40.249.106:64
    Part of it is cut off though, so I can't see the full reason. Can you try and grep for that and get the whole thing?
    journalctl -u kubelet | grep "Unable to register node"

    Make sure that 52.40.249.106 is the correct private IP of your load balancer, and that the load balancer and kube-apiservers are up and running.

  • post-author-pic
    goodyman05
    11-14-2018

    Thank you!


  • post-author-pic
    goodyman05
    11-14-2018

    the rest of the log is "i/o timeout" I entered the wrong ip address when setting up the api server.

  • post-author-pic
    Will B
    11-14-2018

    Glad to hear you found the issue!

  • post-author-pic
    goodyman05
    11-14-2018

    False alarm. :/  I double checked and I did enter the correct api server address. I am not sure when the 52.40... address is coming from. 

  • post-author-pic
    Will B
    11-14-2018

    It should be the private address of the load balancer server, if you are following Kubernetes the Hard Way.


    Come to think of it, that address looks like a public IP (private IPs usually start with 172). I think the problem may be that you used the public IP of the load balancer, not the private IP. That causes issues, because the public IPs change when the server stops and starts again. You should use the private IP since they stay the same.

    That address is set in a Kubeconfig. Go back to the lesson "Generating Kubeconfigs for the cluster". Make sure you use the load balancer private IP (not public) when you set the 

    KUBERNETES_ADDRESS environment variable. You will need to re-generate the kubeconfigs for each worker node as well as kube-proxy with the correct address, then move those new kubeconfigs to the appropriate locations on each worker, overwriting the old ones. That should fix the issue. Let me know how it goes!

  • post-author-pic
    goodyman05
    11-17-2018

    Thank you  @wboyd ! I will give that a try. 

  • post-author-pic
    goodyman05
    11-20-2018

     @wboyd  That fixed it!  This helped me really understand what kube-proxy and the the kubeconfigs are doing.

  • post-author-pic
    Will B
    11-20-2018

    Awesome! Glad to hear it's working!

Looking For Team Training?

Learn More