Failed

Kubernetes e2e suite.[sig-node] Pods Extended Pod Container Status should never report success for a pending container

Failing for the past 2 builds (Since #9501 )
Took 5 min 42 sec.

Stacktrace

/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/node/pods.go:206
May 16 20:29:02.959: timed out waiting for watch events for pod-submit-status-1-2
/usr/local/go/src/runtime/asm_amd64.s:1371

Standard Output

[BeforeEach] [sig-node] Pods Extended
  /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185
STEP: Creating a kubernetes client
May 16 20:23:48.043: INFO: >>> kubeConfig: /var/run/kubernetes/admin.kubeconfig
STEP: Building a namespace api object, basename pods
STEP: Waiting for a default service account to be provisioned in namespace
[BeforeEach] Pod Container Status
  /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/node/pods.go:202
[It] should never report success for a pending container
  /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/node/pods.go:206
STEP: creating pods that should always exit 1 and terminating the pod after a random delay
May 16 20:23:54.342: INFO: watch delete seen for pod-submit-status-1-0
May 16 20:23:54.342: INFO: Pod pod-submit-status-1-0 on node 172.18.3.220 timings total=6.274341607s t=421ms run=0s execute=0s
May 16 20:23:56.733: INFO: watch delete seen for pod-submit-status-0-0
May 16 20:23:56.733: INFO: Pod pod-submit-status-0-0 on node 172.18.3.220 timings total=8.665247627s t=1.17s run=0s execute=0s
May 16 20:24:01.135: INFO: watch delete seen for pod-submit-status-0-1
May 16 20:24:01.135: INFO: Pod pod-submit-status-0-1 on node 172.18.3.220 timings total=4.401729838s t=1.815s run=0s execute=0s
May 16 20:24:02.334: INFO: watch delete seen for pod-submit-status-1-1
May 16 20:24:02.335: INFO: Pod pod-submit-status-1-1 on node 172.18.3.220 timings total=7.992282855s t=1.532s run=0s execute=0s
May 16 20:24:05.132: INFO: watch delete seen for pod-submit-status-2-0
May 16 20:24:05.132: INFO: Pod pod-submit-status-2-0 on node 172.18.3.220 timings total=17.064043623s t=1.633s run=0s execute=0s
May 16 20:24:07.736: INFO: watch delete seen for pod-submit-status-0-2
May 16 20:24:07.736: INFO: Pod pod-submit-status-0-2 on node 172.18.3.220 timings total=6.601307651s t=794ms run=0s execute=0s
May 16 20:24:22.849: INFO: watch delete seen for pod-submit-status-2-1
May 16 20:24:22.849: INFO: Pod pod-submit-status-2-1 on node 172.18.3.220 timings total=17.716877748s t=1.441s run=0s execute=0s
May 16 20:24:29.733: INFO: watch delete seen for pod-submit-status-2-2
May 16 20:24:29.733: INFO: Pod pod-submit-status-2-2 on node 172.18.3.220 timings total=6.884622457s t=1.13s run=0s execute=0s
May 16 20:29:02.959: FAIL: timed out waiting for watch events for pod-submit-status-1-2

Full Stack Trace
created by k8s.io/kubernetes/test/e2e/node.glob..func10.3.2
	/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/node/pods.go:224 +0x1c5
May 16 20:29:09.187: FAIL: timed out waiting for watch events for pod-submit-status-0-3

Full Stack Trace
created by k8s.io/kubernetes/test/e2e/node.glob..func10.3.2
	/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/node/pods.go:224 +0x1c5
May 16 20:29:30.245: FAIL: timed out waiting for watch events for pod-submit-status-2-3

Full Stack Trace
created by k8s.io/kubernetes/test/e2e/node.glob..func10.3.2
	/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/node/pods.go:224 +0x1c5
[AfterEach] [sig-node] Pods Extended
  /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186
STEP: Collecting events from namespace "pods-3697".
STEP: Found 34 events.
May 16 20:29:30.249: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for pod-submit-status-0-0: { } Scheduled: Successfully assigned pods-3697/pod-submit-status-0-0 to 172.18.3.220
May 16 20:29:30.249: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for pod-submit-status-0-1: { } Scheduled: Successfully assigned pods-3697/pod-submit-status-0-1 to 172.18.3.220
May 16 20:29:30.249: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for pod-submit-status-0-2: { } Scheduled: Successfully assigned pods-3697/pod-submit-status-0-2 to 172.18.3.220
May 16 20:29:30.249: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for pod-submit-status-0-3: { } Scheduled: Successfully assigned pods-3697/pod-submit-status-0-3 to 172.18.3.220
May 16 20:29:30.249: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for pod-submit-status-1-0: { } Scheduled: Successfully assigned pods-3697/pod-submit-status-1-0 to 172.18.3.220
May 16 20:29:30.249: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for pod-submit-status-1-1: { } Scheduled: Successfully assigned pods-3697/pod-submit-status-1-1 to 172.18.3.220
May 16 20:29:30.249: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for pod-submit-status-1-2: { } Scheduled: Successfully assigned pods-3697/pod-submit-status-1-2 to 172.18.3.220
May 16 20:29:30.249: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for pod-submit-status-2-0: { } Scheduled: Successfully assigned pods-3697/pod-submit-status-2-0 to 172.18.3.220
May 16 20:29:30.249: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for pod-submit-status-2-1: { } Scheduled: Successfully assigned pods-3697/pod-submit-status-2-1 to 172.18.3.220
May 16 20:29:30.249: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for pod-submit-status-2-2: { } Scheduled: Successfully assigned pods-3697/pod-submit-status-2-2 to 172.18.3.220
May 16 20:29:30.249: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for pod-submit-status-2-3: { } Scheduled: Successfully assigned pods-3697/pod-submit-status-2-3 to 172.18.3.220
May 16 20:29:30.249: INFO: At 2022-05-16 20:23:51 +0000 UTC - event for pod-submit-status-0-0: {kubelet 172.18.3.220} Failed: Error: cannot find volume "kube-api-access-8zsvr" to mount into container "busybox"
May 16 20:29:30.249: INFO: At 2022-05-16 20:23:51 +0000 UTC - event for pod-submit-status-0-0: {kubelet 172.18.3.220} Pulled: Container image "k8s.gcr.io/e2e-test-images/busybox:1.29-1" already present on machine
May 16 20:29:30.249: INFO: At 2022-05-16 20:23:51 +0000 UTC - event for pod-submit-status-2-0: {kubelet 172.18.3.220} Failed: Error: cannot find volume "kube-api-access-m2j7m" to mount into container "busybox"
May 16 20:29:30.249: INFO: At 2022-05-16 20:23:51 +0000 UTC - event for pod-submit-status-2-0: {kubelet 172.18.3.220} Pulled: Container image "k8s.gcr.io/e2e-test-images/busybox:1.29-1" already present on machine
May 16 20:29:30.249: INFO: At 2022-05-16 20:23:52 +0000 UTC - event for pod-submit-status-1-0: {kubelet 172.18.3.220} Pulled: Container image "k8s.gcr.io/e2e-test-images/busybox:1.29-1" already present on machine
May 16 20:29:30.249: INFO: At 2022-05-16 20:23:52 +0000 UTC - event for pod-submit-status-1-0: {kubelet 172.18.3.220} Failed: Error: cannot find volume "kube-api-access-hqnqh" to mount into container "busybox"
May 16 20:29:30.249: INFO: At 2022-05-16 20:23:56 +0000 UTC - event for pod-submit-status-1-1: {kubelet 172.18.3.220} Pulled: Container image "k8s.gcr.io/e2e-test-images/busybox:1.29-1" already present on machine
May 16 20:29:30.249: INFO: At 2022-05-16 20:23:56 +0000 UTC - event for pod-submit-status-1-1: {kubelet 172.18.3.220} Started: Started container busybox
May 16 20:29:30.249: INFO: At 2022-05-16 20:23:56 +0000 UTC - event for pod-submit-status-1-1: {kubelet 172.18.3.220} Created: Created container busybox
May 16 20:29:30.249: INFO: At 2022-05-16 20:23:59 +0000 UTC - event for pod-submit-status-0-1: {kubelet 172.18.3.220} Pulled: Container image "k8s.gcr.io/e2e-test-images/busybox:1.29-1" already present on machine
May 16 20:29:30.249: INFO: At 2022-05-16 20:23:59 +0000 UTC - event for pod-submit-status-0-1: {kubelet 172.18.3.220} Failed: Error: cannot find volume "kube-api-access-92g7v" to mount into container "busybox"
May 16 20:29:30.249: INFO: At 2022-05-16 20:24:03 +0000 UTC - event for pod-submit-status-0-2: {kubelet 172.18.3.220} Failed: Error: container create failed: error stat'ing file `/var/lib/kubelet/pods/50597c00-b209-492c-8817-d9458e6de86c/volumes/kubernetes.io~projected/kube-api-access-dsjzb`: No such file or directory

May 16 20:29:30.249: INFO: At 2022-05-16 20:24:03 +0000 UTC - event for pod-submit-status-0-2: {kubelet 172.18.3.220} Pulled: Container image "k8s.gcr.io/e2e-test-images/busybox:1.29-1" already present on machine
May 16 20:29:30.249: INFO: At 2022-05-16 20:24:06 +0000 UTC - event for pod-submit-status-1-2: {kubelet 172.18.3.220} Pulled: Container image "k8s.gcr.io/e2e-test-images/busybox:1.29-1" already present on machine
May 16 20:29:30.249: INFO: At 2022-05-16 20:24:06 +0000 UTC - event for pod-submit-status-1-2: {kubelet 172.18.3.220} Failed: Error: cannot find volume "kube-api-access-w2mf7" to mount into container "busybox"
May 16 20:29:30.249: INFO: At 2022-05-16 20:24:08 +0000 UTC - event for pod-submit-status-2-1: {kubelet 172.18.3.220} Pulled: Container image "k8s.gcr.io/e2e-test-images/busybox:1.29-1" already present on machine
May 16 20:29:30.249: INFO: At 2022-05-16 20:24:08 +0000 UTC - event for pod-submit-status-2-1: {kubelet 172.18.3.220} Failed: Error: cannot find volume "kube-api-access-l9zsl" to mount into container "busybox"
May 16 20:29:30.249: INFO: At 2022-05-16 20:24:10 +0000 UTC - event for pod-submit-status-0-3: {kubelet 172.18.3.220} Pulled: Container image "k8s.gcr.io/e2e-test-images/busybox:1.29-1" already present on machine
May 16 20:29:30.249: INFO: At 2022-05-16 20:24:10 +0000 UTC - event for pod-submit-status-0-3: {kubelet 172.18.3.220} Failed: Error: cannot find volume "kube-api-access-8ncmd" to mount into container "busybox"
May 16 20:29:30.249: INFO: At 2022-05-16 20:24:25 +0000 UTC - event for pod-submit-status-2-2: {kubelet 172.18.3.220} Pulled: Container image "k8s.gcr.io/e2e-test-images/busybox:1.29-1" already present on machine
May 16 20:29:30.249: INFO: At 2022-05-16 20:24:25 +0000 UTC - event for pod-submit-status-2-2: {kubelet 172.18.3.220} Failed: Error: cannot find volume "kube-api-access-8fcvw" to mount into container "busybox"
May 16 20:29:30.249: INFO: At 2022-05-16 20:24:32 +0000 UTC - event for pod-submit-status-2-3: {kubelet 172.18.3.220} Pulled: Container image "k8s.gcr.io/e2e-test-images/busybox:1.29-1" already present on machine
May 16 20:29:30.249: INFO: At 2022-05-16 20:24:32 +0000 UTC - event for pod-submit-status-2-3: {kubelet 172.18.3.220} Failed: Error: cannot find volume "kube-api-access-mq5pl" to mount into container "busybox"
May 16 20:29:30.251: INFO: POD                    NODE          PHASE    GRACE  CONDITIONS
May 16 20:29:30.251: INFO: pod-submit-status-0-3  172.18.3.220  Pending  1s     [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-05-16 20:24:07 +0000 UTC  } {Ready False 0001-01-01 00:00:00 +0000 UTC 2022-05-16 20:24:07 +0000 UTC ContainersNotReady containers with unready status: [busybox]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2022-05-16 20:24:07 +0000 UTC ContainersNotReady containers with unready status: [busybox]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-05-16 20:24:07 +0000 UTC  }]
May 16 20:29:30.251: INFO: pod-submit-status-1-2  172.18.3.220  Pending  1s     [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-05-16 20:24:02 +0000 UTC  } {Ready False 0001-01-01 00:00:00 +0000 UTC 2022-05-16 20:24:02 +0000 UTC ContainersNotReady containers with unready status: [busybox]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2022-05-16 20:24:02 +0000 UTC ContainersNotReady containers with unready status: [busybox]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-05-16 20:24:02 +0000 UTC  }]
May 16 20:29:30.251: INFO: pod-submit-status-2-3  172.18.3.220  Pending  1s     [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-05-16 20:24:29 +0000 UTC  } {Ready False 0001-01-01 00:00:00 +0000 UTC 2022-05-16 20:24:29 +0000 UTC ContainersNotReady containers with unready status: [busybox]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2022-05-16 20:24:29 +0000 UTC ContainersNotReady containers with unready status: [busybox]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-05-16 20:24:29 +0000 UTC  }]
May 16 20:29:30.251: INFO: 
May 16 20:29:30.254: INFO: 
Logging node info for node 172.18.3.220
May 16 20:29:30.256: INFO: Node Info: &Node{ObjectMeta:{172.18.3.220    a63a89d9-0a05-4469-a62a-202a308f9dcd 23171 0 2022-05-16 20:05:58 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/os:linux kubernetes.io/arch:amd64 kubernetes.io/hostname:172.18.3.220 kubernetes.io/os:linux topology.hostpath.csi/node:172.18.3.220] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kubelet Update v1 2022-05-16 20:27:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}}} {kube-controller-manager Update v1 2022-05-16 20:28:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}}}}]},Spec:NodeSpec{PodCIDR:,DoNotUseExternalID:,ProviderID:,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[],},Status:NodeStatus{Capacity:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{79198666752 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16783200256 0} {<nil>} 16389844Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{4 0} {<nil>} 4 DecimalSI},ephemeral-storage: {{71278799959 0} {<nil>} 71278799959 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{16678342656 0} {<nil>} 16287444Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-05-16 20:29:12 +0000 UTC,LastTransitionTime:2022-05-16 20:05:58 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-05-16 20:29:12 +0000 UTC,LastTransitionTime:2022-05-16 20:05:58 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-05-16 20:29:12 +0000 UTC,LastTransitionTime:2022-05-16 20:05:58 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-05-16 20:29:12 +0000 UTC,LastTransitionTime:2022-05-16 20:06:08 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.18.3.220,},NodeAddress{Type:Hostname,Address:172.18.3.220,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec229a97cf9b93f04c6086ff8dba3997,SystemUUID:ec26875a-53e7-880c-0672-0eaed645c7c2,BootID:e7feaec5-85a2-4876-836a-d8424376ef48,KernelVersion:5.4.18-200.fc31.x86_64,OSImage:Fedora 31 (Cloud Edition),ContainerRuntimeVersion:cri-o://1.21.7,KubeletVersion:v1.21.13-rc.0.16+f3ad8e60207d1f-dirty,KubeProxyVersion:v1.21.13-rc.0.16+f3ad8e60207d1f-dirty,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8s.gcr.io/sig-storage/nfs-provisioner@sha256:c1bedac8758029948afe060bf8f6ee63ea489b5e08d29745f44fab68ee0d46ca k8s.gcr.io/sig-storage/nfs-provisioner:v2.2.2],SizeBytes:403155297,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/volume/nfs@sha256:124a375b4f930627c65b2f84c0d0f09229a96bc527eec18ad0eeac150b96d1c2 k8s.gcr.io/e2e-test-images/volume/nfs@sha256:90af3b1795d2669a4a07d3a0fecbaa2ac920ef69b3c588e93423e74501793cdc k8s.gcr.io/e2e-test-images/volume/nfs:1.2],SizeBytes:272582535,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:abf326ca6107a7aa8f8019ebf1acb077ed818f3761e50c289182046cef4bc973 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:260976965,},ContainerImage{Names:[docker.io/kubernetesui/dashboard@sha256:19207cca570f61bce2294595a767b51b4d7e1f223a003c7972c350cf8689f49e docker.io/kubernetesui/dashboard@sha256:a705c04e83badb4fdb2b95eb6b126f3c2759677b2f953742f3b08a1fada07d9d docker.io/kubernetesui/dashboard:v2.0.1],SizeBytes:225542210,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd@sha256:d8c12384c89b59053cd9c9d6bca380ccc56a143adc6c798762d0fe050c7aa7bf k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:132295921,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd@sha256:cba7b71304b6369c0d5e1ea5e70631354b5824c7f75dbce9d63149af216efbeb k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:128894977,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:758db666ac7028534dba72e7e9bb1e57bb81b8196f976f7a5cc351ef8b3529e1 k8s.gcr.io/e2e-test-images/agnhost@sha256:ef11a0f696f3489a1684af5525419ac332df8682a148c6843b4da63c1503ee5b k8s.gcr.io/e2e-test-images/agnhost:2.32],SizeBytes:126732584,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:1f36a24cfb5e0c3f725d7565a867c2384282fcbeccc77b07b423c9da95763a9a k8s.gcr.io/e2e-test-images/nautilus@sha256:d2444f4f1049cf55278bc54390218f2c54e46671879767b599726b3099fba5ea k8s.gcr.io/e2e-test-images/nautilus:1.4],SizeBytes:122559135,},ContainerImage{Names:[k8s.gcr.io/k8s-dns-kube-dns@sha256:7f1c5078c7efac7b18e90725136fca67c0eeb3e1b5fe5c9af200022758c8d443 k8s.gcr.io/k8s-dns-kube-dns@sha256:df4efd1cdfd9eed00e7c6de271c7df3b02b6bd44728cb4efac30fee0f62bc008 k8s.gcr.io/k8s-dns-kube-dns:1.15.10],SizeBytes:89091848,},ContainerImage{Names:[k8s.gcr.io/k8s-dns-sidecar@sha256:38c1821f486c101536aa658820754244d95cf66b500939aece383696f931743f k8s.gcr.io/k8s-dns-sidecar@sha256:bb667bc31a4ef3e66165f42aa23f0f9b3610cb22de150cce5bedf4e85c5f822f k8s.gcr.io/k8s-dns-sidecar:1.15.10],SizeBytes:81374662,},ContainerImage{Names:[k8s.gcr.io/k8s-dns-dnsmasq-nanny@sha256:2808637121af22933d9e0f47984d4571fa91bea5fecbbab992ab4423b3f70841 k8s.gcr.io/k8s-dns-dnsmasq-nanny@sha256:d34ae632c8ce2d51aee6669d53bf4c418f5ff191a369f48374edc0323fd63dab k8s.gcr.io/k8s-dns-dnsmasq-nanny:1.15.10],SizeBytes:79266955,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:695505fcfcc69f1cf35665dce487aad447adbb9af69b796d6437f869015d1157 k8s.gcr.io/sig-storage/csi-provisioner@sha256:96b0c0cab9369f7a3d17f85abb2464be87e28cb8fa1a69d9da472db3e2fa6995 k8s.gcr.io/sig-storage/csi-provisioner:v2.1.1],SizeBytes:52904131,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:20c828075d1e36f679d6a91e905b0927141eef5e15be0c9a1ca4a6a0ed9313d2 k8s.gcr.io/sig-storage/csi-provisioner@sha256:dc17308d5e6405307a4c6dc3d019013beecde77818473e428bf2d3b89a7af751 k8s.gcr.io/sig-storage/csi-provisioner:v2.1.0],SizeBytes:52891331,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:51f2dfde5bccac7854b3704689506aeecfb793328427b91115ba253a93e60782 k8s.gcr.io/sig-storage/csi-snapshotter@sha256:7f66155edcb6cb9a38ed54ad041e9b554b37b26633b1a82c4a9f185f1f9d5d12 k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0],SizeBytes:50804947,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:50c3cfd458fc8e0bf3c8c521eac39172009382fc66dc5044a330d137c6ed0b09 k8s.gcr.io/sig-storage/csi-attacher@sha256:b831dcea4b7e07f980d890a87c0a816ed7b576a89728239e00aa4c6fe575f155 k8s.gcr.io/sig-storage/csi-attacher:v3.1.0],SizeBytes:50440866,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer@sha256:ef7659a2f399429c1fd4bf64d154c04ef6a6c0fbeed00f178b44490e3eb2e420 k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:50421927,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:2ffa647e8107cfd39e5f464e738dce014c9f5e51b108da36c3ab621048d0bbab k8s.gcr.io/sig-storage/csi-attacher:v2.2.0],SizeBytes:47375145,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:6c6a0332693a7c456378f6abd2bb40611826c1e1a733cadbdae2daab3125b71c k8s.gcr.io/sig-storage/csi-resizer:v0.5.0],SizeBytes:47285534,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:4051e85640c22f8e00c34dbd273576fc9e1e2829992656588062be9c0f69b04b k8s.gcr.io/e2e-test-images/nonroot@sha256:93f8fe220940db5f92e1572e72b1457fc683ea3aebd24ac9474c6bca65660834 k8s.gcr.io/e2e-test-images/nonroot:1.1],SizeBytes:43878048,},ContainerImage{Names:[docker.io/kubernetesui/metrics-scraper@sha256:555981a24f184420f3be0c79d4efb6c948a85cfce84034f85a563f4151a81cbf docker.io/kubernetesui/metrics-scraper@sha256:d78f995c07124874c2a2e9b404cffa6bc6233668d63d6c6210574971f3d5914b docker.io/kubernetesui/metrics-scraper:v1.0.4],SizeBytes:36950651,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:d2b357bb02430fee9eaa43b16083981463d260419fe3acb2f560ede5c129f6f5 k8s.gcr.io/sig-storage/hostpathplugin@sha256:f411c1712711f94a556346eb80d747228694d7ac3afd76ce91381ad32553c8bb k8s.gcr.io/sig-storage/hostpathplugin:v1.4.0],SizeBytes:28181939,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:f8600ea8748fe0b38e84708bdc32e1d179ae368c628fb5712cf4b8cf379902c1 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:20909099,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:3938dbe6f05482045deb3a68ebffbf9be89b9498bbefd19f6475fb01ab65d1bc k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:e07f914c32f0505e4c470a62a40ee43f84cbf8dc46ff861f31b14457ccbad108 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.0.1],SizeBytes:19246095,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:48da0e4ed7238ad461ea05f68c25921783c37b315f21a5c5a2780157a6460994 k8s.gcr.io/sig-storage/livenessprobe@sha256:d82c69638630d34d658b66b5e93a779ecd20f2fd78aa6042204fc4edfaa5e054 k8s.gcr.io/sig-storage/livenessprobe:v2.2.0],SizeBytes:19022320,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:967db33d47bc530905b9ad2ae36f33cfc881e1312fc07c51e2b42ed65ea80e6f k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:18926726,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx@sha256:ebf4de42b3d660133f6f7d0feddabe31a44d07ed55f59471fd2072b0d8e8afae k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:17245687,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs@sha256:f6b1c4aef11b116c2a065ea60ed071a8f205444f1897bed9aa2e98a5d78cbdae k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:7373984,},ContainerImage{Names:[k8s.gcr.io/busybox@sha256:d8d3bc2c183ed2f9f10e7258f84971202325ee6011ba137112e01e30f206de67 k8s.gcr.io/busybox:latest],SizeBytes:2659266,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:244bdbdf4b8d368b5836e9d2c7808a280a73ad72ae321d644e9f220da503218f k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:1374910,},ContainerImage{Names:[docker.io/library/busybox@sha256:91ef6c1c52b166be02645b8efee30d1ee65362024f7da41c404681561734c465 docker.io/library/busybox@sha256:bbc3a03235220b170ba48a157dd097dd1379299370e1ed99ce976df0355d24f0 docker.io/library/busybox:1.27],SizeBytes:1343106,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause@sha256:369201a612f7b2b585a8e6ca99f77a36bcdbd032463d815388a96800b63ef2c8 k8s.gcr.io/pause:3.5],SizeBytes:689969,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:6c3835cab3980f11b83277305d0d736051c32b17606f5ec59f1dda67c9ba3810 k8s.gcr.io/pause@sha256:914e745e524aa94315a25b49a7fafc0aa395e332126930593225d7a513f5a6b2 k8s.gcr.io/pause:3.4.1],SizeBytes:689817,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-provisioning-3063^7040faa1-d555-11ec-910e-728155b2b318],VolumesAttached:[]AttachedVolume{},Config:nil,},}
May 16 20:29:30.256: INFO: 
Logging kubelet events for node 172.18.3.220
May 16 20:29:30.259: INFO: 
Logging pods the kubelet thinks is on node 172.18.3.220
May 16 20:29:30.271: INFO: kubernetes-dashboard-7fff8584c9-r2qrr started at 2022-05-16 20:06:16 +0000 UTC (0+1 container statuses recorded)
May 16 20:29:30.271: INFO: 	Container kubernetes-dashboard ready: true, restart count 0
May 16 20:29:30.271: INFO: pod-handle-http-request started at 2022-05-16 20:29:23 +0000 UTC (0+1 container statuses recorded)
May 16 20:29:30.271: INFO: 	Container agnhost-container ready: true, restart count 0
May 16 20:29:30.271: INFO: kube-dns-5bc69b84c5-72dzk started at 2022-05-16 20:06:16 +0000 UTC (0+3 container statuses recorded)
May 16 20:29:30.271: INFO: 	Container dnsmasq ready: true, restart count 0
May 16 20:29:30.271: INFO: 	Container kubedns ready: true, restart count 0
May 16 20:29:30.271: INFO: 	Container sidecar ready: true, restart count 0
May 16 20:29:30.271: INFO: ss-0 started at 2022-05-16 20:29:24 +0000 UTC (0+1 container statuses recorded)
May 16 20:29:30.271: INFO: 	Container webserver ready: false, restart count 0
May 16 20:29:30.272: INFO: nfs-server started at 2022-05-16 20:28:55 +0000 UTC (0+1 container statuses recorded)
May 16 20:29:30.272: INFO: 	Container nfs-server ready: false, restart count 0
May 16 20:29:30.272: INFO: nfs-injector started at 2022-05-16 20:29:10 +0000 UTC (0+1 container statuses recorded)
May 16 20:29:30.272: INFO: 	Container nfs-injector ready: false, restart count 0
May 16 20:29:30.272: INFO: csi-hostpath-snapshotter-0 started at 2022-05-16 20:28:11 +0000 UTC (0+1 container statuses recorded)
May 16 20:29:30.272: INFO: 	Container csi-snapshotter ready: false, restart count 0
May 16 20:29:30.272: INFO: dashboard-metrics-scraper-7b4c85dd89-cblnj started at 2022-05-16 20:06:16 +0000 UTC (0+1 container statuses recorded)
May 16 20:29:30.272: INFO: 	Container dashboard-metrics-scraper ready: true, restart count 0
May 16 20:29:30.272: INFO: csi-hostpathplugin-0 started at 2022-05-16 20:28:10 +0000 UTC (0+3 container statuses recorded)
May 16 20:29:30.272: INFO: 	Container hostpath ready: false, restart count 0
May 16 20:29:30.272: INFO: 	Container liveness-probe ready: false, restart count 0
May 16 20:29:30.272: INFO: 	Container node-driver-registrar ready: false, restart count 0
May 16 20:29:30.272: INFO: nfs-server started at 2022-05-16 20:28:58 +0000 UTC (0+1 container statuses recorded)
May 16 20:29:30.272: INFO: 	Container nfs-server ready: true, restart count 0
May 16 20:29:30.272: INFO: pod-with-poststart-http-hook started at <nil> (0+0 container statuses recorded)
May 16 20:29:30.272: INFO: httpd-deployment-8584777d8-fgdmr started at 2022-05-16 20:14:09 +0000 UTC (0+1 container statuses recorded)
May 16 20:29:30.272: INFO: 	Container httpd ready: false, restart count 0
May 16 20:29:30.272: INFO: pod-submit-status-2-3 started at 2022-05-16 20:24:29 +0000 UTC (0+1 container statuses recorded)
May 16 20:29:30.272: INFO: 	Container busybox ready: false, restart count 0
May 16 20:29:30.272: INFO: external-provisioner-988bk started at 2022-05-16 20:28:54 +0000 UTC (0+1 container statuses recorded)
May 16 20:29:30.272: INFO: 	Container nfs-provisioner ready: true, restart count 0
May 16 20:29:30.272: INFO: pod-submit-status-0-3 started at 2022-05-16 20:24:07 +0000 UTC (0+1 container statuses recorded)
May 16 20:29:30.272: INFO: 	Container busybox ready: false, restart count 0
May 16 20:29:30.272: INFO: pod-submit-status-1-2 started at 2022-05-16 20:24:02 +0000 UTC (0+1 container statuses recorded)
May 16 20:29:30.272: INFO: 	Container busybox ready: false, restart count 0
May 16 20:29:30.701: INFO: 
Latency metrics for node 172.18.3.220
May 16 20:29:30.701: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
STEP: Destroying namespace "pods-3697" for this suite.