Prometheus Operator监控SpringCloud

前言

  • 在上一篇文章优雅的使用Prometheus Operator中我们搭建起来了监控的基础堆栈。下来我们来监控我们的业务服务。
  • 监控这个SpringCloud的java微服务的方式有很多,比如说基于eureka的服务型和基于kubernetes的endpoints的注解服务发现。

    基础指标依赖

  • 为了能让promethus采集到相关的指标数据我们需要加入一下依赖。
    1
    2
    3
    4
    5
    6
    7
    8
    9
    <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>
    <dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
    <version>1.0.6</version>
    </dependency>

基于eureka服务发现

  • 由于promethus默认提供的服务发现是不支持eureka的,but他支持consul,所以有老铁写了个桥接eureka-consul-adapter,让eurek伪装成consul,这样就能被promethus服务发现支持了。
  • 所以这里我们需要把这个依赖加入到eureka中。

    1
    2
    3
    4
    5
    <dependency>
    <groupId>at.twinformatics</groupId>
    <artifactId>eureka-consul-adapter</artifactId>
    <version>1.3.0</version>
    </dependency>
  • 这里我们已经部署好了eureka。我们看下eureka的地址。

    1
    2
    3
    4
    $ kubectl get po -owide | grep eureka
    eureka-0 1/1 Running 0 33m 172.224.3.222 node1 <none> <none>
    eureka-1 1/1 Running 0 33m 172.224.7.94 node6 <none> <none>
    eureka-2 1/1 Running 0 34m 172.224.4.127 node2 <none> <none>
  • 任选一个节点,查看我们的适配器是否已经成功生效
    $ curl -i http://172.224.3.222:8761/v1/catalog/services
    HTTP/1.1 200 OK
    Connection: keep-alive

    1
    2
    3
    4
    5
    6
    {
    "CONFIG-SERVICE": [],
    "DEMO1-SERVICE": [],
    "DEMO2-SERVICE": [],
    "EUREKA": []
    }
  • 查看服务的详细
    $ curl -i http://172.224.3.222:8761/v1/catalog/service/DEMO2-SERVICE
    HTTP/1.1 200 OK
    Connection: keep-alive

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    [
    {
    "Address": "172.224.3.223",
    "Node": "DEMO2-SERVICE",
    "ServiceAddress": "172.224.3.223",
    "ServiceName": "DEMO2-SERVICE",
    "ServiceID": "demo2-service:8101@5826031",
    "ServicePort": 8101,
    "NodeMeta": {},
    "ServiceMeta": {
    "management.port": "8101"
    },
    "ServiceTags": []
    }
    ]
  • 查看指标是否能成功获取
    $ curl -i http://172.224.3.223:8101/actuator/prometheus
    HTTP/1.1 200 OK
    Connection: keep-alive

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    # HELP jvm_classes_unloaded_classes_total The total number of classes unloaded since the Java virtual machine has started execution
    # TYPE jvm_classes_unloaded_classes_total counter
    jvm_classes_unloaded_classes_total 1.0
    # HELP jvm_buffer_memory_used_bytes An estimate of the memory that the Java virtual machine is using for this buffer pool
    # TYPE jvm_buffer_memory_used_bytes gauge
    jvm_buffer_memory_used_bytes{id="direct",} 1685563.0
    jvm_buffer_memory_used_bytes{id="mapped",} 0.0
    # HELP jvm_threads_live_threads The current number of live threads including both daemon and non-daemon threads
    # TYPE jvm_threads_live_threads gauge
    jvm_threads_live_threads 67.0
    ...
    # HELP jvm_buffer_count_buffers An estimate of the number of buffers in the pool
    # TYPE jvm_buffer_count_buffers gauge
    jvm_buffer_count_buffers{id="direct",} 154.0
    jvm_buffer_count_buffers{id="mapped",} 0.0
    # HELP jvm_classes_loaded_classes The number of classes that are currently loaded in the Java virtual machine
    # TYPE jvm_classes_loaded_classes gauge
    jvm_classes_loaded_classes 10297.0

这样我们就把spring boot配置完成了,接下来需要配置promethus。

追加自定义配置到promethus-oprator

我们需要继续修改文件prometheus-prometheus.yaml,我们从官方提供的文档可知,他为我们追加自定义的配置预留了一个属性additionalScrapeConfigs,我们只需将自定义的配置,通过这个配置就能追到上去。

additionalScrapeConfigs的格式为如下,为了方便我们使用,这里追加配置的方式,它采用了kubernetes的secret机制,我们自己编写的配置需要提交成一个secret,而后对应的配置,name就是secret的名称,key就是文件名。

1
2
3
4
spec:
additionalScrapeConfigs:
name:
key:

有了基本概念之后我们就来追加我们的服务发现配置。

  • prometheus-additional.yaml

    1
    2
    3
    4
    - job_name: "eureka-ds"
    consul_sd_configs:
    - server: "eureka-0.eureka.default.svc.cluster.local:8761"
    metrics_path: /actuator/prometheus
  • 创建

    1
    kubectl create secret generic additional-configs --from-file=prometheus-additional.yaml -n monitoring
  • 查看

    1
    2
    3
    4
    5
    6
    7
    8
    9
    $ kubectl get secret additional-configs -n monitoring -o yaml
    apiVersion: v1
    data:
    prometheus-additional.yaml: I2t1YmVjdGwgY3JlYXRlIHNlY3JldCBnZW5lcmljIGFkZGl0aW9uYWwtY29uZmlncyAtLWZyb20tZmlsZT1wcm9tZXRoZXVzLWFkZGl0aW9uYWwueWFtbCAtbiBtb25pdG9yaW5nCiNrdWJlY3RsIGdldCBzZWNyZXQgYWRkaXRpb25hbC1jb25maWdzIC1uIG1vbml0b3JpbmcgLW8geWFtbAoja3ViZWN0bCBkZWxldGUgc2VjcmV0IGFkZGl0aW9uYWwtY29uZmlncyAtbm1vbml0b3JpbmcKLSBqb2JfbmFtZTogImV1cmVrYS1kcyIKICBjb25zdWxfc2RfY29uZmlnczoKICAgIC0gc2VydmVyOiAiZXVyZWthLTAuZXVyZWthLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWw6ODc2MSIKICBtZXRyaWNzX3BhdGg6IC9hY3R1YXRvci9wcm9tZXRoZXVzCiMtIGpvYl9uYW1lOiAna3ViZXJuZXRlcy1zZXJ2aWNlLWVuZHBvaW50cycKIyAga3ViZXJuZXRlc19zZF9jb25maWdzOgojICAgIC0gcm9sZTogZW5kcG9pbnRzCiMgIHJlbGFiZWxfY29uZmlnczoKIyAgICAtIHNvdXJjZV9sYWJlbHM6IFtfX21ldGFfa3ViZXJuZXRlc19zZXJ2aWNlX2Fubm90YXRpb25fcHJvbWV0aGV1c19pb19zcHJpbmdzY3JhcGVdCiMgICAgICBhY3Rpb246IGtlZXAKIyAgICAgIHJlZ2V4OiB0cnVlCiMgICAgLSBzb3VyY2VfbGFiZWxzOiBbX19tZXRhX2t1YmVybmV0ZXNfc2VydmljZV9hbm5vdGF0aW9uX3Byb21ldGhldXNfaW9fc2NoZW1lXQojICAgICAgYWN0aW9uOiByZXBsYWNlCiMgICAgICB0YXJnZXRfbGFiZWw6IF9fc2NoZW1lX18KIyAgICAgIHJlZ2V4OiAoaHR0cHM/KQojICAgIC0gc291cmNlX2xhYmVsczogW19fbWV0YV9rdWJlcm5ldGVzX3NlcnZpY2VfYW5ub3RhdGlvbl9wcm9tZXRoZXVzX2lvX3BhdGhdCiMgICAgICBhY3Rpb246IHJlcGxhY2UKIyAgICAgIHRhcmdldF9sYWJlbDogX19tZXRyaWNzX3BhdGhfXwojICAgICAgcmVnZXg6ICguKykKIyAgICAtIHNvdXJjZV9sYWJlbHM6IFtfX2FkZHJlc3NfXywgX19tZXRhX2t1YmVybmV0ZXNfc2VydmljZV9hbm5vdGF0aW9uX3Byb21ldGhldXNfaW9fcG9ydF0KIyAgICAgIGFjdGlvbjogcmVwbGFjZQojICAgICAgdGFyZ2V0X2xhYmVsOiBfX2FkZHJlc3NfXwojICAgICAgcmVnZXg6IChbXjpdKykoPzo6XGQrKT87KFxkKykKIyAgICAgIHJlcGxhY2VtZW50OiAkMTokMgojICAgIC0gYWN0aW9uOiBsYWJlbG1hcAojICAgICAgcmVnZXg6IF9fbWV0YV9rdWJlcm5ldGVzX3NlcnZpY2VfbGFiZWxfKC4rKQojICAgIC0gc291cmNlX2xhYmVsczogW19fbWV0YV9rdWJlcm5ldGVzX25hbWVzcGFjZV0KIyAgICAgIGFjdGlvbjogcmVwbGFjZQojICAgICAgdGFyZ2V0X2xhYmVsOiBrdWJlcm5ldGVzX25hbWVzcGFjZQojICAgIC0gc291cmNlX2xhYmVsczogW19fbWV0YV9rdWJlcm5ldGVzX3NlcnZpY2VfbmFtZV0KIyAgICAgIGFjdGlvbjogcmVwbGFjZQojICAgICAgdGFyZ2V0X2xhYmVsOiBrdWJlcm5ldGVzX25hbWUK
    kind: Secret
    metadata:
    name: additional-configs
    namespace: monitoring
    type: Opaque
  • 删除

    1
    kubectl delete secret additional-configs -nmonitoring
  • 有了secret之后呢我们就可以配置promethus了

    1
    2
    3
    4
    spec:
    additionalScrapeConfigs:
    name: additional-configs #对应secret名
    key: prometheus-additional.yaml # 对应文件名
  • 完整配置如下

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    apiVersion: monitoring.coreos.com/v1
    kind: Prometheus
    metadata:
    labels:
    prometheus: k8s
    name: k8s
    namespace: monitoring
    spec:
    alerting:
    alertmanagers:
    - name: alertmanager-main
    namespace: monitoring
    port: web
    baseImage: freemanliu/prometheus
    nodeSelector:
    kubernetes.io/os: linux
    podMonitorSelector: {}
    replicas: 2
    resources:
    requests:
    memory: 400Mi
    ruleSelector:
    matchLabels:
    prometheus: k8s
    role: alert-rules
    securityContext:
    fsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
    serviceAccountName: prometheus-k8s
    serviceMonitorNamespaceSelector: {}
    serviceMonitorSelector: {}
    version: v2.11.0
    additionalScrapeConfigs:
    name: additional-configs
    key: prometheus-additional.yaml
    storage:
    volumeClaimTemplate:
    spec:
    storageClassName: local-storage-promethues
    accessModes: [ "ReadWriteOnce" ]
    resources:
    requests:
    storage: 10Gi
  • 重新apply,使其生效

    1
    kubectl apply -f manifests/
  • 再次通过SVCIP查看我们promethus的targets,可以看到已经成功采集到数据了,有一个错误是因为我们的config-service没有加入指标依赖。加入即可解决。

spring boot 微服务仪表盘

我们这里需要导入dashboard,需要的导入的id为4701。导入完成之后你将看到如下展示。

推荐文章