[TOC]
資源配額,通過 ResourceQuota 對象來定義,對每個命名空間的資源消耗總量提供限制。 它可以限制命名空間中某種類型的對象的總數目上限,也可以限制命令空間中的 Pod 可以使用的計算資源的總上限。
## 啟用資源配額
資源配額的支持在很多 Kubernetes 版本中是默認啟用的。 當 API 服務器 的命令行標志 --enable-admission-plugins= 中包含 ResourceQuota 時, 資源配額會被啟用。
```shell
grep enable-admission-plugins /data/k8s/conf/kube-apiserver.conf
```
## 限制資源配置
```yaml
apiVersion: v1
kind: ResourceQuota
metadata:
name: quota
namespace: test
spec:
hard:
requests.cpu: "4"
requests.memory: "1Gi"
requests.storage: "100Gi"
pods: "4"
```
> **說明:**
> 如果所使用的是 CRI 容器運行時,容器日志會被計入臨時存儲配額。 這可能會導致存儲配額耗盡的 Pods 被意外地驅逐出節點。 參考日志架構 了解詳細信息。
## 生效資源限制
```shell
$ kubectl apply -f test-namespaces.yml
resourcequota/quota create
```
## 驗證
### 驗證內存
創建一個 `1Gi` 的deployment清單文件
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: client
name: client
spec:
replicas: 1
selector:
matchLabels:
app: client
strategy: {}
template:
metadata:
labels:
app: client
spec:
containers:
- image: busybox:1.24.1
name: busybox
args:
- sh
- -c
- "sleep 3600"
resources:
requests:
memory: 2Gi
```
創建容器
```shell
$ kubectl apply -f client.yml -n test
deployment.apps/client created
$ kubectl -n test get deployments.apps
NAME READY UP-TO-DATE AVAILABLE AGE
client 0/1 0 0 38s
$ kubectl -n test describe deployments.apps client
Name: client
Namespace: test
CreationTimestamp: Wed, 09 Feb 2022 11:21:25 +0800
Labels: app=client
Annotations: deployment.kubernetes.io/revision: 1
Selector: app=client
Replicas: 1 desired | 0 updated | 0 total | 0 available | 1 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
Labels: app=client
Containers:
busybox:
Image: busybox:1.24.1
Port: <none>
Host Port: <none>
Args:
sh
-c
sleep 3600
Requests:
cpu: 50m
memory: 2Gi
Environment: <none>
Mounts: <none>
Volumes: <none>
Conditions:
Type Status Reason
---- ------ ------
Progressing True NewReplicaSetCreated
Available False MinimumReplicasUnavailable
ReplicaFailure True FailedCreate
OldReplicaSets: <none>
NewReplicaSet: client-9d57dfdf6 (0/1 replicas created)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 20s deployment-controller Scaled up replica set client-9d57dfdf6 to 1
$ kubectl -n test describe rs client-9d57dfdf6
Name: client-9d57dfdf6
Namespace: test
Selector: app=client,pod-template-hash=9d57dfdf6
Labels: app=client
pod-template-hash=9d57dfdf6
Annotations: deployment.kubernetes.io/desired-replicas: 1
deployment.kubernetes.io/max-replicas: 2
deployment.kubernetes.io/revision: 1
Controlled By: Deployment/client
Replicas: 0 current / 1 desired
Pods Status: 0 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
Labels: app=client
pod-template-hash=9d57dfdf6
Containers:
busybox:
Image: busybox:1.24.1
Port: <none>
Host Port: <none>
Args:
sh
-c
sleep 3600
Requests:
cpu: 50m
memory: 2Gi
Environment: <none>
Mounts: <none>
Volumes: <none>
Conditions:
Type Status Reason
---- ------ ------
ReplicaFailure True FailedCreate
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedCreate 29s replicaset-controller Error creating: pods "client-9d57dfdf6-9x74f" is forbidden: exceeded quota: quota, requested: requests.memory=2Gi, used: requests.memory=1Gi, limited: requests.memory=1Gi
Warning FailedCreate 29s replicaset-controller Error creating: pods "client-9d57dfdf6-62kxp" is forbidden: exceeded quota: quota, requested: requests.memory=2Gi, used: requests.memory=1Gi, limited: requests.memory=1Gi
Warning FailedCreate 29s replicaset-controller Error creating: pods "client-9d57dfdf6-nx9bs" is forbidden: exceeded quota: quota, requested: requests.memory=2Gi, used: requests.memory=1Gi, limited: requests.memory=1Gi
Warning FailedCreate 29s replicaset-controller Error creating: pods "client-9d57dfdf6-zs9jl" is forbidden: exceeded quota: quota, requested: requests.memory=2Gi, used: requests.memory=1Gi, limited: requests.memory=1Gi
Warning FailedCreate 28s replicaset-controller Error creating: pods "client-9d57dfdf6-llrdj" is forbidden: exceeded quota: quota, requested: requests.memory=2Gi, used: requests.memory=1Gi, limited: requests.memory=1Gi
Warning FailedCreate 28s replicaset-controller Error creating: pods "client-9d57dfdf6-25qrk" is forbidden: exceeded quota: quota, requested: requests.memory=2Gi, used: requests.memory=1Gi, limited: requests.memory=1Gi
Warning FailedCreate 28s replicaset-controller Error creating: pods "client-9d57dfdf6-2tlxl" is forbidden: exceeded quota: quota, requested: requests.memory=2Gi, used: requests.memory=1Gi, limited: requests.memory=1Gi
Warning FailedCreate 28s replicaset-controller Error creating: pods "client-9d57dfdf6-fdl4j" is forbidden: exceeded quota: quota, requested: requests.memory=2Gi, used: requests.memory=1Gi, limited: requests.memory=1Gi
Warning FailedCreate 27s replicaset-controller Error creating: pods "client-9d57dfdf6-hjfnf" is forbidden: exceeded quota: quota, requested: requests.memory=2Gi, used: requests.memory=1Gi, limited: requests.memory=1Gi
Warning FailedCreate 8s (x4 over 26s) replicaset-controller (combined from similar events): Error creating: pods "client-9d57dfdf6-5xkj7" is forbidden: exceeded quota: quota, requested: requests.memory=2Gi, used: requests.memory=0, limited: requests.memory=1Gi
```
### 驗證pod數量
創建一個5副本的deployment清單文件
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: web
name: web
spec:
replicas: 5
selector:
matchLabels:
app: web
strategy: {}
template:
metadata:
labels:
app: web
spec:
containers:
- image: nginx
name: nginx
resources:
requests:
memory: 100m
cpu: 50m
```
創建容器
```shell
$ kubectl apply -f web.yml -n test
deployment.apps/web created
$ kubectl -n test get pod
NAME READY STATUS RESTARTS AGE
web-584b96b57-24pk5 1/1 Running 0 43s
web-584b96b57-czr6q 1/1 Running 0 43s
web-584b96b57-m9hkv 1/1 Running 0 43s
web-584b96b57-szg9x 1/1 Running 0 43s
$ kubectl -n test get deployments.apps
NAME READY UP-TO-DATE AVAILABLE AGE
web 4/5 4 4 30s
$ kubectl -n test describe rs web-584b96b57
Name: web-584b96b57
Namespace: test
Selector: app=web,pod-template-hash=584b96b57
Labels: app=web
pod-template-hash=584b96b57
Annotations: deployment.kubernetes.io/desired-replicas: 5
deployment.kubernetes.io/max-replicas: 7
deployment.kubernetes.io/revision: 1
Controlled By: Deployment/web
Replicas: 4 current / 5 desired
Pods Status: 4 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
Labels: app=web
pod-template-hash=584b96b57
Containers:
nginx:
Image: nginx
Port: <none>
Host Port: <none>
Requests:
cpu: 50m
memory: 100m
Environment: <none>
Mounts: <none>
Volumes: <none>
Conditions:
Type Status Reason
---- ------ ------
ReplicaFailure True FailedCreate
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 70s replicaset-controller Created pod: web-584b96b57-m9hkv
Normal SuccessfulCreate 70s replicaset-controller Created pod: web-584b96b57-24pk5
Normal SuccessfulCreate 70s replicaset-controller Created pod: web-584b96b57-szg9x
Warning FailedCreate 70s replicaset-controller Error creating: pods "web-584b96b57-4ttxz" is forbidden: exceeded quota: quota, requested: pods=1, used: pods=4, limited: pods=4
Normal SuccessfulCreate 70s replicaset-controller Created pod: web-584b96b57-czr6q
Warning FailedCreate 70s replicaset-controller Error creating: pods "web-584b96b57-jv9mp" is forbidden: exceeded quota: quota, requested: pods=1, used: pods=4, limited: pods=4
Warning FailedCreate 70s replicaset-controller Error creating: pods "web-584b96b57-7vsjh" is forbidden: exceeded quota: quota, requested: pods=1, used: pods=4, limited: pods=4
Warning FailedCreate 70s replicaset-controller Error creating: pods "web-584b96b57-7pbxc" is forbidden: exceeded quota: quota, requested: pods=1, used: pods=4, limited: pods=4
Warning FailedCreate 70s replicaset-controller Error creating: pods "web-584b96b57-sdlgw" is forbidden: exceeded quota: quota, requested: pods=1, used: pods=4, limited: pods=4
Warning FailedCreate 70s replicaset-controller Error creating: pods "web-584b96b57-ksjzx" is forbidden: exceeded quota: quota, requested: pods=1, used: pods=4, limited: pods=4
Warning FailedCreate 70s replicaset-controller Error creating: pods "web-584b96b57-gqk28" is forbidden: exceeded quota: quota, requested: pods=1, used: pods=4, limited: pods=4
Warning FailedCreate 70s replicaset-controller Error creating: pods "web-584b96b57-spczj" is forbidden: exceeded quota: quota, requested: pods=1, used: pods=4, limited: pods=4
Warning FailedCreate 70s replicaset-controller Error creating: pods "web-584b96b57-8kzvt" is forbidden: exceeded quota: quota, requested: pods=1, used: pods=4, limited: pods=4
Warning FailedCreate 20s (x12 over 69s) replicaset-controller (combined from similar events): Error creating: pods "web-584b96b57-rjkxh" is forbidden: exceeded quota: quota, requested: pods=1, used: pods=4, limited: pods=4
```
### 其他驗證
請自行測試,這里就不進行一一驗證。
> 說明:
> 1、創建deployment必須設置 `requests.memory` 和 `requests.cpu`,否則創建pod失敗
> 2、查看deployment創建成功,但是pod沒有創建出來。那得看 `rs` 創建的情況
## 參考文章
https://kubernetes.io/zh/docs/concepts/policy/resource-quotas/
- 前言
- 架構
- 部署
- kubeadm部署
- kubeadm擴容節點
- 二進制安裝基礎組件
- 添加master節點
- 添加工作節點
- 選裝插件安裝
- Kubernetes使用
- k8s與dockerfile啟動參數
- hostPort與hostNetwork異同
- 應用上下線最佳實踐
- 進入容器命名空間
- 主機與pod之間拷貝
- events排序問題
- k8s會話保持
- 容器root特權
- CNI插件
- calico
- calicoctl安裝
- calico網絡通信
- calico更改pod地址范圍
- 新增節點網卡名不一致
- 修改calico模式
- calico數據存儲遷移
- 啟用 kubectl 來管理 Calico
- calico卸載
- cilium
- cilium架構
- cilium/hubble安裝
- cilium網絡路由
- IP地址管理(IPAM)
- Cilium替換KubeProxy
- NodePort運行DSR模式
- IP地址偽裝
- ingress使用
- nginx-ingress
- ingress安裝
- ingress高可用
- helm方式安裝
- 基本使用
- Rewrite配置
- tls安全路由
- ingress發布管理
- 代理k8s集群外的web應用
- ingress自定義日志
- ingress記錄真實IP地址
- 自定義參數
- traefik-ingress
- traefik名詞概念
- traefik安裝
- traefik初次使用
- traefik路由(IngressRoute)
- traefik中間件(middlewares)
- traefik記錄真實IP地址
- cert-manager
- 安裝教程
- 頒布者CA
- 創建證書
- 外部存儲
- 對接NFS
- 對接ceph-rbd
- 對接cephfs
- 監控平臺
- Prometheus
- Prometheus安裝
- grafana安裝
- Prometheus配置文件
- node_exporter安裝
- kube-state-metrics安裝
- Prometheus黑盒監控
- Prometheus告警
- grafana儀表盤設置
- 常用監控配置文件
- thanos
- Prometheus
- Sidecar組件
- Store Gateway組件
- Querier組件
- Compactor組件
- Prometheus監控項
- grafana
- Querier對接grafana
- alertmanager
- Prometheus對接alertmanager
- 日志中心
- filebeat安裝
- kafka安裝
- logstash安裝
- elasticsearch安裝
- elasticsearch索引生命周期管理
- kibana安裝
- event事件收集
- 資源預留
- 節點資源預留
- imagefs與nodefs驗證
- 資源預留 vs 驅逐 vs OOM
- scheduler調度原理
- Helm
- Helm安裝
- Helm基本使用
- 安全
- apiserver審計日志
- RBAC鑒權
- namespace資源限制
- 加密Secret數據
- 服務網格
- 備份恢復
- Velero安裝
- 備份與恢復
- 常用維護操作
- container runtime
- 拉取私有倉庫鏡像配置
- 拉取公網鏡像加速配置
- runtime網絡代理
- overlay2目錄占用過大
- 更改Docker的數據目錄
- Harbor
- 重置Harbor密碼
- 問題處理
- 關閉或開啟Harbor的認證
- 固定harbor的IP地址范圍
- ETCD
- ETCD擴縮容
- ETCD常用命令
- ETCD數據空間壓縮清理
- ingress
- ingress-nginx header配置
- kubernetes
- 驗證yaml合法性
- 切換KubeProxy模式
- 容器解析域名
- 刪除節點
- 修改鏡像倉庫
- 修改node名稱
- 升級k8s集群
- 切換容器運行時
- apiserver接口
- 其他
- 升級內核
- k8s組件性能分析
- ETCD
- calico
- calico健康檢查失敗
- Harbor
- harbor同步失敗
- Kubernetes
- 資源Terminating狀態
- 啟動容器報錯