资源指标只包含CPU、内存，一般来说也够了。但如果想根据自定义指标:如请求qps/5xx错误数来实现HPA，就需要使用自定义指标了，目前比较成熟的实现是 Prometheus Custom Metrics。自定义指标由Prometheus来提供，再利用k8s-prometheus-adpater聚合到apiserver，实现和核心指标（metric-server)同样的效果。

1、部署Prometheus

自行部署Prometheus，需要采集Pod指标

2、部署 Custom Metrics Adapter

prometheus采集到的metrics并不能直接给k8s用，因为两者数据格式不兼容，还需要另外一个组件(k8s-prometheus-adpater)，将prometheus的metrics 数据格式转换成k8s API接口能识别的格式，转换以后，因为是自定义API，所以还需要用Kubernetes aggregator在主APIServer中注册，以便直接通过/apis/来访问。

GitHub - kubernetes-sigs/prometheus-adapter: An implementation of the custom.metrics.k8s.io API using PrometheusGitHub - kubernetes-sigs/prometheus-adapter: An implementation of the custom.metrics.k8s.io API using PrometheusGitHub - kubernetes-sigs/prometheus-adapter: An implementation of the custom.metrics.k8s.io API using Prometheus

该 PrometheusAdapter 有一个稳定的Helm Charts，我们直接使用。镜像可以替换成国内镜像。

验证适配器是否注册到APIServer：

[root@master chart]# kubectl get apiservices |grep custom 
v1beta1.custom.metrics.k8s.io          kube-system/prometheus-adapter   True        3m56s
[root@master chart]#

3、部署测试应用

apiVersion: apps/v1
kind: Deployment
metadata:labels:app: metrics-appname: metrics-app
spec:replicas: 2selector:matchLabels:app: metrics-apptemplate:metadata:labels:app: metrics-appannotations:prometheus.io/scrape: "true"prometheus.io/port: "80"prometheus.io/path: "/metrics"spec:containers:- image: art.local:8081/docker-local/lizhenliang/metrics-appname: metrics-appports:- name: webcontainerPort: 80resources:requests:cpu: 200mmemory: 256MireadinessProbe:httpGet:path: /port: 80initialDelaySeconds: 3periodSeconds: 5livenessProbe:httpGet:path: /port: 80initialDelaySeconds: 3periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:name: metrics-applabels:app: metrics-app
spec:type: NodePortports:- name: webport: 80targetPort: 80nodePort: 30099selector:app: metrics-app

 该metrics-app暴露了一个Prometheus指标接口，提供2个指标：

http_requests_total
http_requests_per_second

4、创建自定义HPA策略

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:name: metrics-app-hpa-custom
spec:scaleTargetRef:apiVersion: apps/v1kind: Deploymentname: metrics-appminReplicas: 1maxReplicas: 10metrics:- type: Podspods:metric:name: http_requests_per_secondtarget:type: AverageValueaverageValue: 800m

[root@master k8s]# kubectl get hpa
NAME                     REFERENCE                TARGETS          MINPODS   MAXPODS   REPLICAS   AGE
metrics-app-hpa-custom   Deployment/metrics-app   /800m   1         10        2          27s
[root@master k8s]#

此时适配器还不知道要什么指标（http_requests_per_second），HPA也就获取不到Pod提供指标。

5、配置适配器收集特定的指标

编辑PrometheusAdapter创建的prometheus-adapter Config Map：

新增查询规则

- seriesQuery: 'http_requests_total{kubernetes_namespace!="",kubernetes_pod_name!=""}'resources:overrides:kubernetes_namespace: {resource: "namespace"}kubernetes_pod_name: {resource: "pod"}name:matches: "^(.*)_total"as: "${1}_per_second"metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)'

重新部署PrometheusAdapter后，再次查看hpa，此时能够获取到自定义指标

[root@master log]# kubectl get hpa
NAME                     REFERENCE                TARGETS     MINPODS   MAXPODS   REPLICAS   AGE
metrics-app-hpa-custom   Deployment/metrics-app   499m/800m   1         10        2          76m
[root@master log]#

6、压测，观察自定义指标扩缩容

[root@localhost ~]# ab -n 100000 -c 100 http://179.220.56.232:30099/metrics

[root@master ~]# kubectl get hpa
NAME                     REFERENCE                TARGETS        MINPODS   MAXPODS   REPLICAS   AGE
metrics-app-hpa-custom   Deployment/metrics-app   163846m/800m   1         10        10         5h46m
[root@master ~]# kubectl describe hpa metrics-app-hpa-custom 
Warning: autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscaler
Name:                                  metrics-app-hpa-custom
Namespace:                             default
Labels:                                
Annotations:                           
CreationTimestamp:                     Mon, 20 Mar 2023 10:01:59 +0800
Reference:                             Deployment/metrics-app
Metrics:                               ( current / target )"http_requests_per_second" on pods:  499m / 800m
Min replicas:                          1
Max replicas:                          10
Deployment pods:                       3 current / 3 desired
Conditions:Type            Status  Reason               Message----            ------  ------               -------AbleToScale     True    ScaleDownStabilized  recent recommendations were higher than current one, applying the highest recent recommendationScalingActive   True    ValidMetricFound     the HPA was able to successfully calculate a replica count from pods metric http_requests_per_secondScalingLimited  False   DesiredWithinRange   the desired count is within the acceptable range
Events:Type    Reason             Age   From                       Message----    ------             ----  ----                       -------Normal  SuccessfulRescale  24m   horizontal-pod-autoscaler  New size: 4; reason: pods metric http_requests_per_second above targetNormal  SuccessfulRescale  24m   horizontal-pod-autoscaler  New size: 8; reason: pods metric http_requests_per_second above targetNormal  SuccessfulRescale  24m   horizontal-pod-autoscaler  New size: 10; reason: pods metric http_requests_per_second above targetNormal  SuccessfulRescale  17m   horizontal-pod-autoscaler  New size: 7; reason: All metrics below targetNormal  SuccessfulRescale  12m   horizontal-pod-autoscaler  New size: 5; reason: All metrics below targetNormal  SuccessfulRescale  7m1s  horizontal-pod-autoscaler  New size: 4; reason: All metrics below targetNormal  SuccessfulRescale  2m1s  horizontal-pod-autoscaler  New size: 3; reason: All metrics below targe