imagefs与nodefs
kubelet可以对磁盘进行管控，但是只能对nodefs与imagefs这两个分区进行管控。其中
imagefs: docker安装目录所在的分区
nodefs: kubelet的启动参数
--root-dir所指定的目录（默认/var/lib/kubelet）所在的分区
接下来，我们来验证一下我们对imagefs与nodefs的理解。
前置条件
k8s集群使用1.8.6版本
$ kubectl get node
NAME STATUS ROLES AGE VERSION
10.142.232.161 Ready <none> 263d v1.8.6
10.142.232.162 NotReady <none> 263d v1.8.6
10.142.232.163 Ready,SchedulingDisabled <none> 227d v1.8.610.142.232.161上docker安装在/app/docker目录下，kubelet的--root-dir没有设置，使用默认的/var/lib/kubelet。/app是一块盘，使用率为70%；/是一块盘，使用率为57%；而imagesfs与nodefs此时设置的阈值都为80%，如下：
$ df -hT
文件系统 类型 容量 已用 可用 已用% 挂载点
devtmpfs devtmpfs 16G 0 16G 0% /dev
tmpfs tmpfs 16G 0 16G 0% /dev/shm
tmpfs tmpfs 16G 1.7G 15G 11% /run
tmpfs tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/mapper/centos-root xfs 45G 26G 20G 57% /
/dev/xvda1 xfs 497M 254M 243M 52% /boot
/dev/xvde xfs 150G 105G 46G 70% /app$ ps -ef | grep kubelet
root 125179 1 37 17:50 ? 00:00:01 /usr/bin/kubelet --address=0.0.0.0 --allow-privileged=true --cluster-dns=10.254.0.10 --cluster-domain=kube.local --fail-swap-on=false --hostname-override=10.142.232.161 --kubeconfig=/etc/kubernetes/kubeconfig --pod-infra-container-image=10.142.233.76:8021/library/pause:latest --port=10250 --enforce-node-allocatable=pods --eviction-hard=memory.available<20%,nodefs.inodesFree<20%,imagefs.inodesFree<20%,nodefs.available<20%,imagefs.available<20% --network-plugin=cni此时，10.142.232.161该node没有报磁盘的错
$ kubectl describe node 10.142.232.161
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Starting 18s kubelet, 10.142.232.161 Starting kubelet.
Normal NodeAllocatableEnforced 18s kubelet, 10.142.232.161 Updated Node Allocatable limit across pods
Normal NodeHasSufficientDisk 18s kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasSufficientDisk
Normal NodeHasSufficientMemory 18s kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 18s kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasNoDiskPressure
Normal NodeNotReady 18s kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeNotReady
Normal NodeReady 8s kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeReady验证方案
验证imagefs是
/app/docker目录所在分区（/app分区使用率为70%）修改imagefs的阈值为60%，node应该报imagefs超标
修改imagefs的阈值为80%，node应该正常
验证nodefs是
/var/lib/kubelet目录所在的分区（/分区使用率为57%）修改nodefs的阈值为50%，node应该报nodefs超标
修改nodefs的阈值为60%，node应该正常
修改kubelet启动参数
--root-dir，将值设成/app/kubelet修改让imagefs的阈值为80%，nodefs的阈值为60%；此时应该报nodefs超标
修改让imagefs的阈值为60%，nodefs的阈值为80%；此时应该报imagefs超标
修改让imagefs的阈值为60%，nodefs的阈值为60%；此时应该报两个都超标
修改让imagefs的阈值为80%，nodefs的阈值为80%；此时node应该正常
验证步骤
一、验证imagefs是/app/docker目录所在分区
1.1 修改imagefs的阈值为60%，node应该imagefs超标
如下，我们把imagefs的阈值设为60%
$ ps -ef | grep kubelet
root 41234 1 72 18:17 ? 00:00:02 /usr/bin/kubelet --address=0.0.0.0 --allow-privileged=true --cluster-dns=10.254.0.10 --cluster-domain=kube.local --fail-swap-on=false --hostname-override=10.142.232.161 --kubeconfig=/etc/kubernetes/kubeconfig --pod-infra-container-image=10.142.233.76:8021/library/pause:latest --port=10250 --enforce-node-allocatable=pods --eviction-hard=memory.available<20%,nodefs.inodesFree<20%,imagefs.inodesFree<20%,nodefs.available<20%,imagefs.available<40% --network-plugin=cni然后我们查看节点的状态，Attempting to reclaim imagefs，意思为尝试回收imagefs
$ kubectl describe node 10.142.232.161
...
Normal NodeAllocatableEnforced 1m kubelet, 10.142.232.161 Updated Node Allocatable limit across pods
Normal Starting 1m kubelet, 10.142.232.161 Starting kubelet.
Normal NodeHasSufficientDisk 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasSufficientDisk
Normal NodeHasSufficientMemory 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasNoDiskPressure
Normal NodeNotReady 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeNotReady
Normal NodeHasDiskPressure 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasDiskPressure
Normal NodeReady 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeReady
Warning EvictionThresholdMet 18s (x4 over 1m) kubelet, 10.142.232.161 Attempting to reclaim imagefs1.2 修改imagefs的阈值为80%，node应该正常
我们把imagefs的阈值为80%
$ ps -ef | grep kubelet
root 51402 1 19 18:24 ? 00:00:06 /usr/bin/kubelet --address=0.0.0.0 --allow-privileged=true --cluster-dns=10.254.0.10 --cluster-domain=kube.local --fail-swap-on=false --hostname-override=10.142.232.161 --kubeconfig=/etc/kubernetes/kubeconfig --pod-infra-container-image=10.142.233.76:8021/library/pause:latest --port=10250 --enforce-node-allocatable=pods --eviction-hard=memory.available<20%,nodefs.inodesFree<20%,imagefs.inodesFree<20%,nodefs.available<20%,imagefs.available<20% --network-plugin=cni然后再来查看node的状态，NodeHasNoDiskPressure，说明imagefs使用率没有超过阈值了
$ kubectl describe node 10.142.232.161
...
Warning EvictionThresholdMet 6m (x22 over 11m) kubelet, 10.142.232.161 Attempting to reclaim imagefs
Normal Starting 5m kubelet, 10.142.232.161 Starting kubelet.
Normal NodeAllocatableEnforced 5m kubelet, 10.142.232.161 Updated Node Allocatable limit across pods
Normal NodeHasSufficientDisk 5m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasSufficientDisk
Normal NodeHasSufficientMemory 5m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 5m (x2 over 5m) kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasNoDiskPressure
Normal NodeNotReady 5m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeNotReady
Normal NodeReady 4m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeReady二、验证nodefs是/var/lib/kubelet目录所在的分区（/分区使用率为57%）
2.1 修改nodefs的阈值为50%，node应该报nodefs超标
修改nodefs的阈值为50%
$ ps -ef | grep kubelet
root 72575 1 59 18:35 ? 00:00:04 /usr/bin/kubelet --address=0.0.0.0 --allow-privileged=true --cluster-dns=10.254.0.10 --cluster-domain=kube.local --fail-swap-on=false --hostname-override=10.142.232.161 --kubeconfig=/etc/kubernetes/kubeconfig --pod-infra-container-image=10.142.233.76:8021/library/pause:latest --port=10250 --enforce-node-allocatable=pods --eviction-hard=memory.available<20%,nodefs.inodesFree<20%,imagefs.inodesFree<20%,nodefs.available<50%,imagefs.available<20% --network-plugin=cni查看node的状态，报Attempting to reclaim nodefs，意思是尝试回收nodefs，也就是nodefs超标了
$ kubectl describe node 10.142.232.161
...
Normal Starting 1m kubelet, 10.142.232.161 Starting kubelet.
Normal NodeAllocatableEnforced 1m kubelet, 10.142.232.161 Updated Node Allocatable limit across pods
Normal NodeHasSufficientDisk 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasSufficientDisk
Normal NodeHasSufficientMemory 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasNoDiskPressure
Normal NodeNotReady 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeNotReady
Normal NodeHasDiskPressure 53s kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasDiskPressure
Normal NodeReady 53s kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeReady
Warning EvictionThresholdMet 2s (x5 over 1m) kubelet, 10.142.232.161 Attempting to reclaim nodefs2.2 修改nodefs的阈值为60%，node应该正常
修改nodefs的阈值为60%
$ ps -ef | grep kubelet
root 78664 1 31 18:38 ? 00:00:02 /usr/bin/kubelet --address=0.0.0.0 --allow-privileged=true --cluster-dns=10.254.0.10 --cluster-domain=kube.local --fail-swap-on=false --hostname-override=10.142.232.161 --kubeconfig=/etc/kubernetes/kubeconfig --pod-infra-container-image=10.142.233.76:8021/library/pause:latest --port=10250 --enforce-node-allocatable=pods --eviction-hard=memory.available<20%,nodefs.inodesFree<20%,imagefs.inodesFree<20%,nodefs.available<40%,imagefs.available<20% --network-plugin=cni此时查看node的状态，已正常
$ kubectl describe node 10.142.232.161
...
Normal Starting 2m kubelet, 10.142.232.161 Starting kubelet.
Normal NodeReady 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeReady三、修改kubelet启动参数--root-dir，将值设成/app/kubelet
以下几个参数的默认值都与/var/lib/kubelet有关
--root-dir # 默认值为 /var/lib/kubelet
--seccomp-profile-root # 默认值为 /var/lib/kubelet/seccomp
--cert-dir # 默认值为 /var/lib/kubelet/pki
--kubeconfig # 默认值为 /var/lib/kubelet/kubeconfig为了能够不再使用/var/lib/kubelet这个目录，我们需要对这四个参数显示设置。设置如下：
--root-dir=/app/kubelet
--seccomp-profile-root=/app/kubelet/seccomp
--cert-dir=/app/kubelet/pki
--kubeconfig=/etc/kubernetes/kubeconfig3.1 修改让imagefs的阈值为80%，nodefs的阈值为60%；此时应该报nodefs超标
$ ps -ef | grep kubelet
root 14423 1 10 19:28 ? 00:00:34 /usr/bin/kubelet --address=0.0.0.0 --allow-privileged=true --cluster-dns=10.254.0.10 --cluster-domain=kube.local --fail-swap-on=false --hostname-override=10.142.232.161 --kubeconfig=/etc/kubernetes/kubeconfig --pod-infra-container-image=10.142.233.76:8021/library/pause:latest --port=10250 --enforce-node-allocatable=pods --eviction-hard=memory.available<20%,nodefs.inodesFree<20%,imagefs.inodesFree<20%,nodefs.available<40%,imagefs.available<20% --root-dir=/app/kubelet --seccomp-profile-root=/app/kubelet/seccomp --cert-dir=/app/kubelet/pki --network-plugin=cni查看节点的状态，只报Attempting to reclaim nodefs，也就是说nodefs超标
$ kubectl describe node 10.142.232.161
...
Normal NodeHasDiskPressure 3m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasDiskPressure
Normal NodeReady 3m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeReady
Normal Starting 3m kube-proxy, 10.142.232.161 Starting kube-proxy.
Warning EvictionThresholdMet 27s (x15 over 3m) kubelet, 10.142.232.161 Attempting to reclaim nodefs3.2 修改让imagefs的阈值为60%，nodefs的阈值为80%；此时应该报imagefs超标
$ ps -ef |grep kubelet
root 21381 1 30 19:36 ? 00:00:02 /usr/bin/kubelet --address=0.0.0.0 --allow-privileged=true --cluster-dns=10.254.0.10 --cluster-domain=kube.local --fail-swap-on=false --hostname-override=10.142.232.161 --kubeconfig=/etc/kubernetes/kubeconfig --pod-infra-container-image=10.142.233.76:8021/library/pause:latest --port=10250 --enforce-node-allocatable=pods --eviction-hard=memory.available<20%,nodefs.inodesFree<20%,imagefs.inodesFree<20%,nodefs.available<20%,imagefs.available<40% --root-dir=/app/kubelet --seccomp-profile-root=/app/kubelet/seccomp --cert-dir=/app/kubelet/pki --network-plugin=cni我们查看node的状态，只报imagefs超标
$ kubectl describe node 10.142.232.161
...
Normal Starting 1m kubelet, 10.142.232.161 Starting kubelet.
Normal NodeAllocatableEnforced 1m kubelet, 10.142.232.161 Updated Node Allocatable limit across pods
Normal NodeHasSufficientDisk 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasSufficientDisk
Normal NodeNotReady 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeNotReady
Normal NodeHasNoDiskPressure 1m (x2 over 1m) kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientMemory 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasSufficientMemory
Normal NodeReady 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeReady
Normal NodeHasDiskPressure 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasDiskPressure
Warning EvictionThresholdMet 11s (x5 over 1m) kubelet, 10.142.232.161 Attempting to reclaim imagefs3.3 修改让imagefs的阈值为60%，nodefs的阈值为60%；此时应该报两个都超标
$ ps -ef | grep kubelet
root 24524 1 33 19:39 ? 00:00:01 /usr/bin/kubelet --address=0.0.0.0 --allow-privileged=true --cluster-dns=10.254.0.10 --cluster-domain=kube.local --fail-swap-on=false --hostname-override=10.142.232.161 --kubeconfig=/etc/kubernetes/kubeconfig --pod-infra-container-image=10.142.233.76:8021/library/pause:latest --port=10250 --enforce-node-allocatable=pods --eviction-hard=memory.available<20%,nodefs.inodesFree<20%,imagefs.inodesFree<20%,nodefs.available<40%,imagefs.available<40% --root-dir=/app/kubelet --seccomp-profile-root=/app/kubelet/seccomp --cert-dir=/app/kubelet/pki --network-plugin=cni我们查看node的状态，果然imagefs与nodefs都超标了
$ kubectl describe node 10.142.232.161
...
Normal Starting 1m kubelet, 10.142.232.161 Starting kubelet.
Normal NodeAllocatableEnforced 1m kubelet, 10.142.232.161 Updated Node Allocatable limit across pods
Normal NodeHasSufficientDisk 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasSufficientDisk
Normal NodeHasSufficientMemory 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 1m (x2 over 1m) kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasNoDiskPressure
Normal NodeNotReady 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeNotReady
Normal NodeHasDiskPressure 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasDiskPressure
Normal NodeReady 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeReady
Warning EvictionThresholdMet 14s kubelet, 10.142.232.161 Attempting to reclaim imagefs
Warning EvictionThresholdMet 4s (x8 over 1m) kubelet, 10.142.232.161 Attempting to reclaim nodefs3.4 修改让imagefs的阈值为80%，nodefs的阈值为80%；此时node应该正常
$ ps -ef | grep kubelet
root 27869 1 30 19:43 ? 00:00:01 /usr/bin/kubelet --address=0.0.0.0 --allow-privileged=true --cluster-dns=10.254.0.10 --cluster-domain=kube.local --fail-swap-on=false --hostname-override=10.142.232.161 --kubeconfig=/etc/kubernetes/kubeconfig --pod-infra-container-image=10.142.233.76:8021/library/pause:latest --port=10250 --enforce-node-allocatable=pods --eviction-hard=memory.available<20%,nodefs.inodesFree<20%,imagefs.inodesFree<20%,nodefs.available<20%,imagefs.available<20% --root-dir=/app/kubelet --seccomp-profile-root=/app/kubelet/seccomp --cert-dir=/app/kubelet/pki --network-plugin=cni我们查看node的状态，果然没有报imagefs与nodefs的错了
$ kubectl decribe node 10.142.232.161
...
Normal Starting 1m kubelet, 10.142.232.161 Starting kubelet.
Normal NodeHasSufficientDisk 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasSufficientDisk
Normal NodeHasSufficientMemory 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeHasSufficientMemory
Normal NodeNotReady 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeNotReady
Normal NodeAllocatableEnforced 1m kubelet, 10.142.232.161 Updated Node Allocatable limit across pods
Normal NodeReady 1m kubelet, 10.142.232.161 Node 10.142.232.161 status is now: NodeReady总结
1、nodefs是--root-dir目录所在分区，imagefs是docker安装目录所在的分区
2、建议nodefs与imagefs共用一个分区，但是这个分区要设置的大一些。
3、当nodefs与imagefs共用一个分区时，kubelet中的其他几个参数--root-dir、--cert-dir、--seccomp-profile-root、--kubeconfig也要显示地设置
Last updated
Was this helpful?