Question

What Grafana version and what operating system are you using?

Promtail:latest & Loki:2.2.0, Kubernetes (GitVersion:"v1.18.8") and Helm (Version:"v3.6.2")

What are you trying to achieve?

To can scrape my active targets and push them to Loki.

What happened?

All targets are marked as "not ready". If I am going to the /targets page, all my active_targets are marked as "false". In Loki I have no logs. As well the /var/logs/ folder is empty in Promtail. The logs I am receiving from the promtail pod are like this:

level=info ts=2021-08-06T05:30:09.076046169Z caller=filetargetmanager.go:254 msg="Adding target" key="{app="<app_name>", container="<container_name>", job="<job_name>", namespace="<namesapce_name>", node_name="<nodeName_name>", pod="<pod_name>"}"

level=info ts=2021-08-06T05:30:09.076046169Z caller=filetargetmanager.go:254 msg="Removing target" key="{app="<app_name>", container="<container_name>", job="<job_name>", namespace="<namesapce_name>", node_name="<nodeName_name>", pod="<pod_name>"}"

level=info ts=2021-08-06T05:30:14.095615824Z caller=filetarget.go:150 msg="filetarget: watcher closed, tailer stopped, positions saved" path=/var/log/pods/*<some_path>/<container_name>/*.log

Promtail/metrics:

 HELP promtail_targets_failed_total Number of failed targets.
 TYPE promtail_targets_failed_total counter
promtail_targets_failed_total{reason="empty_labels"} 2280
promtail_targets_failed_total{reason="exists"} 470
 HELP request_duration_seconds Time (in seconds) spent serving HTTP requests.

What did you expect to happen?

That my targets getting scraped and pushed to Loki.

Can you copy/paste the configuration(s) that you are having problems with?


  file: |
    server:
      log_level: {{ .Values.config.logLevel }}
      http_listen_port: {{ .Values.config.serverPort }}
      health_check_target: false
    client:
      url: {{ tpl .Values.config.lokiAddress . }}
      {{- tpl .Values.config.snippets.extraClientConfigs . | nindent 2 }}
    positions:
      filename: /run/promtail/positions.yaml
    scrape_configs:
      {{- tpl .Values.config.snippets.scrapeConfigs . | nindent 2 }}
      {{- tpl .Values.config.snippets.extraScrapeConfigs . | nindent 2 }}

scrapeConfigs: |
      # See also https://github.com/grafana/loki/blob/master/production/ksonnet/promtail/scrape_config.libsonnet for reference
      # Pods with a label  app.kubernetes.io/name 
      - job_name: kubernetes-pods-app-kubernetes-io-name
        pipeline_stages:
          {{- toYaml .Values.config.snippets.pipelineStages | nindent 4 }}
        kubernetes_sd_configs:
          - role: pod
        relabel_configs:
          - action: replace
            source_labels:
              - __meta_kubernetes_pod_label_app_kubernetes_io_name
            target_label: app
          - action: drop
            regex:   
            source_labels:
              - app
          - action: replace
            source_labels:
              - __meta_kubernetes_pod_label_app_kubernetes_io_component
            target_label: component
          {{- if .Values.config.snippets.addScrapeJobLabel }}
          - action: replace
            replacement: kubernetes-pods-app-kubernetes-io-name
            target_label: scrape_job
          {{- end }}
          {{- toYaml .Values.config.snippets.common | nindent 4 }}
      # Pods with a label  app 
      - job_name: kubernetes-pods-app
        pipeline_stages:
          {{- toYaml .Values.config.snippets.pipelineStages | nindent 4 }}
        kubernetes_sd_configs:
          - role: pod
        relabel_configs:
          # Drop pods with label  app.kubernetes.io/name . They are already considered above
          - action: drop
            regex: .+
            source_labels:
              - __meta_kubernetes_pod_label_app_kubernetes_io_name
          - action: replace
            source_labels:
              - __meta_kubernetes_pod_label_app
            target_label: app
          - action: drop
            regex:   
            source_labels:
              - app
          - action: replace
            source_labels:
              - __meta_kubernetes_pod_label_component
            target_label: component
          {{- if .Values.config.snippets.addScrapeJobLabel }}
          - action: replace
            replacement: kubernetes-pods-app
            target_label: scrape_job
          {{- end }}
          {{- toYaml .Values.config.snippets.common | nindent 4 }}
      # Pods with direct controllers, such as StatefulSet
      - job_name: kubernetes-pods-direct-controllers
        pipeline_stages:
          {{- toYaml .Values.config.snippets.pipelineStages | nindent 4 }}
        kubernetes_sd_configs:
          - role: pod
        relabel_configs:
          # Drop pods with label  app.kubernetes.io/name  or  app . They are already considered above
          - action: drop
            regex: .+
            separator:   
            source_labels:
              - __meta_kubernetes_pod_label_app_kubernetes_io_name
              - __meta_kubernetes_pod_label_app
          - action: drop
            regex:  [0-9a-z-.]+-[0-9a-f]{8,10} 
            source_labels:
              - __meta_kubernetes_pod_controller_name
          - action: replace
            source_labels:
              - __meta_kubernetes_pod_controller_name
            target_label: app
          {{- if .Values.config.snippets.addScrapeJobLabel }}
          - action: replace
            replacement: kubernetes-pods-direct-controllers
            target_label: scrape_job
          {{- end }}
          {{- toYaml .Values.config.snippets.common | nindent 4 }}
      # Pods with indirect controllers, such as Deployment
      - job_name: kubernetes-pods-indirect-controller
        pipeline_stages:
          {{- toYaml .Values.config.snippets.pipelineStages | nindent 4 }}
        kubernetes_sd_configs:
          - role: pod
        relabel_configs:
          # Drop pods with label  app.kubernetes.io/name  or  app . They are already considered above
          - action: drop
            regex: .+
            separator:   
            source_labels:
              - __meta_kubernetes_pod_label_app_kubernetes_io_name
              - __meta_kubernetes_pod_label_app
          - action: keep
            regex:  [0-9a-z-.]+-[0-9a-f]{8,10} 
            source_labels:
              - __meta_kubernetes_pod_controller_name
          - action: replace
            regex:  ([0-9a-z-.]+)-[0-9a-f]{8,10} 
            source_labels:
              - __meta_kubernetes_pod_controller_name
            target_label: app
          {{- if .Values.config.snippets.addScrapeJobLabel }}
          - action: replace
            replacement: kubernetes-pods-indirect-controller
            target_label: scrape_job
          {{- end }}
          {{- toYaml .Values.config.snippets.common | nindent 4 }}
      # All remaining pods not yet covered
      - job_name: kubernetes-other
        pipeline_stages:
          {{- toYaml .Values.config.snippets.pipelineStages | nindent 4 }}
        kubernetes_sd_configs:
          - role: pod
        relabel_configs:
          # Drop what has already been covered
          - action: drop
            regex: .+
            separator:   
            source_labels:
              - __meta_kubernetes_pod_label_app_kubernetes_io_name
              - __meta_kubernetes_pod_label_app
          - action: drop
            regex: .+
            source_labels:
              - __meta_kubernetes_pod_controller_name
          - action: replace
            source_labels:
              - __meta_kubernetes_pod_name
            target_label: app
          - action: replace
            source_labels:
              - __meta_kubernetes_pod_label_component
            target_label: component
          {{- if .Values.config.snippets.addScrapeJobLabel }}
          - action: replace
            replacement: kubernetes-other
            target_label: scrape_job
          {{- end }}
          {{- toYaml .Values.config.snippets.common | nindent 4 }}

Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
Did you follow any online instructions? If so, what is the URL?

I followed mostly the instructions of the offical repo.

https://github.com/grafana/helm-charts/tree/main/charts

I have created the following recourses:

For Loki: I have a Secret (with the configs), Service and Statefulset.

Promtail: I have a DaemonSet, Secret, powerful ClusterRole and CluserRoleBinding.

Answer 1

Well I m not sure this will help you, but I had the same promtail logs of adding target and immediately removing them.

My config was a bit different, I was running promtail locally and scraping some files. The problem was that promtail didn t have access rights to read those files.

So I d suggest double checking that your promtail pod has read access to the files you re trying to scrape and then restarting the service.

I also didn t see any errors in grafana or Loki, as the logs were never pushed to Loki.

Answer 2

my promtail_err.log

level=info ts=2023-08-09T02:26:00.468678877Z caller=filetargetmanager.go:255 msg="Adding target" key="{env="test", hostname="10.0.12.9", log_type="var-log-messages", service_name="var-log-messages"}"
level=info ts=2023-08-09T02:26:00.468989593Z caller=filetargetmanager.go:255 msg="Adding target" key="{env="test", hostname="10.0.12.9", log_type="var-log-secure", service_name="var-log-secure", test="aaa"}"
level=info ts=2023-08-09T02:26:00.469215283Z caller=filetargetmanager.go:271 msg="Removing target" key="{env="test", hostname="10.0.12.9", log_type="var-log-secure", service_name="var-log-secure", test="aaa"}"
level=info ts=2023-08-09T02:26:00.469320684Z caller=filetarget.go:151 msg="filetarget: watcher closed, tailer stopped, positions saved" path=/var/log/secure
level=info ts=2023-08-09T02:26:00.46937081Z caller=filetargetmanager.go:271 msg="Removing target" key="{env="test", hostname="10.0.12.9", log_type="var-log-messages", service_name="var-log-messages"}"
level=info ts=2023-08-09T02:26:00.469469895Z caller=filetarget.go:151 msg="filetarget: watcher closed, tailer stopped, positions saved" path=/var/log/messages

The reason for my problem: I don t have/var/log/messages and/var/log/secure in my system

My solution：

- targets:
    - localhost
  labels:
    __path__: /var/log/messages
    env: test
    hostname: 10.0.12.9
    service_name: var-log-messages
    log_type: var-log-messages
- targets:
    - localhost
  labels:
    __path__: /var/log/secure
    env: test
    hostname: 10.0.12.9
    service_name: var-log-secure
    log_type: var-log-secure
    test: aaa
- targets:
    - localhost
  labels:
    __path__: /var/log/syslog
    env: test
    hostname: 10.0.12.9
    service_name: var-log-syslog
    log_type: var-log-syslog
    test: aaa

add /var/log/syslog

promtail successful!

友情链接