欢迎光临，若觉得本博客的内容对您有帮助，请多点击边栏的Google广告，并提供意见及建议，谢谢！
Mar
30
生产环境发现不定时 Java 应用出现 coredump 故障，测试环境不定时出现写入 /cgroup/memory 报 no space left on device 的故障，导致整个 kubernetes node 节点无法使用。设置会随着堆积的 cgroup 越来越多，docker ps 执行异常，直到把内存吃光，机器挂死。
典型报错：
或者
该问题发生前后，进行过 kubernetes 1.6 到 1.9 的升级工作。怀疑问题与 kubernetes 、内核有关。
典型报错：
引用
kubelet.ns-k8s-node001.root.log.ERROR.20180214-113740.15702:1593018:E0320 04:59:09.572336 15702 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to start sa
ndbox container for pod "osp-xxx-com-ljqm19-54bf7678b8-bvz9s": Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "process_linux.go:258: applying cgroup configuration
for process caused \"mkdir /sys/fs/cgroup/memory/kubepods/burstable/podf1bd9e87-1ef2-11e8-afd3-fa163ecf2dce/8710c146b3c8b52f5da62e222273703b1e3d54a6a6270a0ea7ce1b194f1b5053: no space left on device\""
ndbox container for pod "osp-xxx-com-ljqm19-54bf7678b8-bvz9s": Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "process_linux.go:258: applying cgroup configuration
for process caused \"mkdir /sys/fs/cgroup/memory/kubepods/burstable/podf1bd9e87-1ef2-11e8-afd3-fa163ecf2dce/8710c146b3c8b52f5da62e222273703b1e3d54a6a6270a0ea7ce1b194f1b5053: no space left on device\""
或者
引用
Mar 26 18:36:59 ns-k8s-node-s0054 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x8020)
Mar 26 18:36:59 ns-k8s-noah-node001 kernel: cache: ip6_dst_cache(1995:6b6bc0c9f30123084a409d89a300b017d26ee5e2c3ac8a02c295c378f3dbfa5f), object size: 448, buffer size: 448, default order: 2, min order: 0
Mar 26 18:36:59 ns-k8s-noah-node001 kernel: cache: ip6_dst_cache(1995:6b6bc0c9f30123084a409d89a300b017d26ee5e2c3ac8a02c295c378f3dbfa5f), object size: 448, buffer size: 448, default order: 2, min order: 0
该问题发生前后，进行过 kubernetes 1.6 到 1.9 的升级工作。怀疑问题与 kubernetes 、内核有关。
