# 使用etcdctl訪問kubernetes數據 Kubenretes1.6中使用etcd V3版本的API，使用`etcdctl`直接`ls`的話只能看到`/kube-centos`一個路徑。需要在命令前加上`ETCDCTL_API=3`這個環境變量才能看到kuberentes在etcd中保存的數據。 ```bash ETCDCTL_API=3 etcdctl get /registry/namespaces/default -w=json|python -m json.tool ``` 如果是使用 kubeadm 創建的集群，在 Kubenretes 1.11 中，etcd 默認使用 tls ，這時你可以在 master 節點上使用以下命令來訪問 etcd ： ```bash ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt \ --cert=/etc/kubernetes/pki/etcd/peer.crt \ --key=/etc/kubernetes/pki/etcd/peer.key \ get /registry/namespaces/default -w=json | jq . ``` - `-w`指定輸出格式 將得到這樣的json的結果： ```json { "count": 1, "header": { "cluster_id": 12091028579527406772, "member_id": 16557816780141026208, "raft_term": 36, "revision": 29253467 }, "kvs": [ { "create_revision": 5, "key": "L3JlZ2lzdHJ5L25hbWVzcGFjZXMvZGVmYXVsdA==", "mod_revision": 5, "value": "azhzAAoPCgJ2MRIJTmFtZXNwYWNlEmIKSAoHZGVmYXVsdBIAGgAiACokZTU2YzMzMDgtMWVhOC0xMWU3LThjZDctZjRlOWQ0OWY4ZWQwMgA4AEILCIn4sscFEKOg9xd6ABIMCgprdWJlcm5ldGVzGggKBkFjdGl2ZRoAIgA=", "version": 1 } ] } ``` 使用`--prefix`可以看到所有的子目錄，如查看集群中的namespace： ```bash ETCDCTL_API=3 etcdctl get /registry/namespaces --prefix -w=json|python -m json.tool ``` 輸出結果中可以看到所有的namespace。 ```bash { "count": 8, "header": { "cluster_id": 12091028579527406772, "member_id": 16557816780141026208, "raft_term": 36, "revision": 29253722 }, "kvs": [ { "create_revision": 24310883, "key": "L3JlZ2lzdHJ5L25hbWVzcGFjZXMvYXV0b21vZGVs", "mod_revision": 24310883, "value": "azhzAAoPCgJ2MRIJTmFtZXNwYWNlEmQKSgoJYXV0b21vZGVsEgAaACIAKiQ1MjczOTU1ZC1iMzEyLTExZTctOTcwYy1mNGU5ZDQ5ZjhlZDAyADgAQgsI7fSWzwUQ6Jv1Z3oAEgwKCmt1YmVybmV0ZXMaCAoGQWN0aXZlGgAiAA==", "version": 1 }, { "create_revision": 21387676, "key": "L3JlZ2lzdHJ5L25hbWVzcGFjZXMvYnJhbmQ=", "mod_revision": 21387676, "value": "azhzAAoPCgJ2MRIJTmFtZXNwYWNlEmEKRwoFYnJhbmQSABoAIgAqJGNkZmQ1Y2NmLWExYzktMTFlNy05NzBjLWY0ZTlkNDlmOGVkMDIAOABCDAjR9qLOBRDYn83XAXoAEgwKCmt1YmVybmV0ZXMaCAoGQWN0aXZlGgAiAA==", "version": 1 }, { "create_revision": 5, "key": "L3JlZ2lzdHJ5L25hbWVzcGFjZXMvZGVmYXVsdA==", "mod_revision": 5, "value": "azhzAAoPCgJ2MRIJTmFtZXNwYWNlEmIKSAoHZGVmYXVsdBIAGgAiACokZTU2YzMzMDgtMWVhOC0xMWU3LThjZDctZjRlOWQ0OWY4ZWQwMgA4AEILCIn4sscFEKOg9xd6ABIMCgprdWJlcm5ldGVzGggKBkFjdGl2ZRoAIgA=", "version": 1 }, { "create_revision": 18504694, "key": "L3JlZ2lzdHJ5L25hbWVzcGFjZXMvZGV2", "mod_revision": 24310213, "value": "azhzAAoPCgJ2MRIJTmFtZXNwYWNlEmwKUgoDZGV2EgAaACIAKiQyOGRlMGVjNS04ZTEzLTExZTctOTcwYy1mNGU5ZDQ5ZjhlZDAyADgAQgwI89CezQUQ0v2fuQNaCwoEbmFtZRIDZGV2egASDAoKa3ViZXJuZXRlcxoICgZBY3RpdmUaACIA", "version": 4 }, { "create_revision": 10, "key": "L3JlZ2lzdHJ5L25hbWVzcGFjZXMva3ViZS1wdWJsaWM=", "mod_revision": 10, "value": "azhzAAoPCgJ2MRIJTmFtZXNwYWNlEmcKTQoLa3ViZS1wdWJsaWMSABoAIgAqJGU1ZjhkY2I1LTFlYTgtMTFlNy04Y2Q3LWY0ZTlkNDlmOGVkMDIAOABCDAiJ+LLHBRDdrsDPA3oAEgwKCmt1YmVybmV0ZXMaCAoGQWN0aXZlGgAiAA==", "version": 1 }, { "create_revision": 2, "key": "L3JlZ2lzdHJ5L25hbWVzcGFjZXMva3ViZS1zeXN0ZW0=", "mod_revision": 2, "value": "azhzAAoPCgJ2MRIJTmFtZXNwYWNlEmYKTAoLa3ViZS1zeXN0ZW0SABoAIgAqJGU1NmFhMDVkLTFlYTgtMTFlNy04Y2Q3LWY0ZTlkNDlmOGVkMDIAOABCCwiJ+LLHBRDoq9ASegASDAoKa3ViZXJuZXRlcxoICgZBY3RpdmUaACIA", "version": 1 }, { "create_revision": 3774247, "key": "L3JlZ2lzdHJ5L25hbWVzcGFjZXMvc3BhcmstY2x1c3Rlcg==", "mod_revision": 3774247, "value": "azhzAAoPCgJ2MRIJTmFtZXNwYWNlEoABCmYKDXNwYXJrLWNsdXN0ZXISABoAIgAqJDMyNjY3ZDVjLTM0YWMtMTFlNy1iZmJkLThhZjFlM2E3YzViZDIAOABCDAiA1cbIBRDU3YuAAVoVCgRuYW1lEg1zcGFyay1jbHVzdGVyegASDAoKa3ViZXJuZXRlcxoICgZBY3RpdmUaACIA", "version": 1 }, { "create_revision": 15212191, "key": "L3JlZ2lzdHJ5L25hbWVzcGFjZXMveWFybi1jbHVzdGVy", "mod_revision": 15212191, "value": "azhzAAoPCgJ2MRIJTmFtZXNwYWNlEn0KYwoMeWFybi1jbHVzdGVyEgAaACIAKiQ2YWNhNjk1Yi03N2Y5LTExZTctYmZiZC04YWYxZTNhN2M1YmQyADgAQgsI1qiKzAUQkoqxDloUCgRuYW1lEgx5YXJuLWNsdXN0ZXJ6ABIMCgprdWJlcm5ldGVzGggKBkFjdGl2ZRoAIgA=", "version": 1 } ] } ``` key的值是經過base64編碼，需要解碼后才能看到實際值，如： ```bash $ echo L3JlZ2lzdHJ5L25hbWVzcGFjZXMvYXV0b21vZGVs|base64 -d /registry/namespaces/automodel ``` ## etcd中kubernetes的元數據 我們使用kubectl命令獲取的kubernetes的對象狀態實際上是保存在etcd中的，使用下面的腳本可以獲取etcd中的所有kubernetes對象的key： > 注意，我們使用了ETCD v3版本的客戶端命令來訪問etcd。 ```bash #!/bin/bash # Get kubernetes keys from etcd export ETCDCTL_API=3 keys=`etcdctl get /registry --prefix -w json|python -m json.tool|grep key|cut -d ":" -f2|tr -d '"'|tr -d ","` for x in $keys;do echo $x|base64 -d|sort done ``` 通過輸出的結果我們可以看到kubernetes的原數據是按何種結構包括在kuberentes中的，輸出結果如下所示： ```ini /registry/ThirdPartyResourceData/istio.io/istioconfigs/default/route-rule-details-default /registry/ThirdPartyResourceData/istio.io/istioconfigs/default/route-rule-productpage-default /registry/ThirdPartyResourceData/istio.io/istioconfigs/default/route-rule-ratings-default ... /registry/configmaps/default/namerctl-script /registry/configmaps/default/namerd-config /registry/configmaps/default/nginx-config ... /registry/deployments/default/sdmk-page-sdmk /registry/deployments/default/sdmk-payment-web /registry/deployments/default/sdmk-report ... ``` 我們可以看到所有的Kuberentes的所有元數據都保存在`/registry`目錄下，下一層就是API對象類型（復數形式），再下一層是`namespace`，最后一層是對象的名字。 以下是etcd中存儲的kubernetes所有的元數據類型： ```ini ThirdPartyResourceData apiextensions.k8s.io apiregistration.k8s.io certificatesigningrequests clusterrolebindings clusterroles configmaps controllerrevisions controllers daemonsets deployments events horizontalpodautoscalers ingress limitranges minions monitoring.coreos.com namespaces persistentvolumeclaims persistentvolumes poddisruptionbudgets pods ranges replicasets resourcequotas rolebindings roles secrets serviceaccounts services statefulsets storageclasses thirdpartyresources ``` ## 參考 - [etcd中文文檔](https://github.com/doczhcn/etcd) - [etcd官方文檔](https://coreos.com/etcd/docs/latest/)