English 中文(简体)
Kubernetes metadata not loading into OpenSearch after EKS Upgrade
原标题：

We have OpenSearch 2.5 and filebeat 7.12.1 that collects logs from EKS cluster (v1.27). We no longer get logs from cluster with kubernetes metadata. We get logs for cloud metadata, but we cannot get our kubernetes metadata objects to load into OpenSearch.

This was working previous to our EKS upgrade from v1.23-->1.27. Have been working on this for some time now with another engineer. Any help is appreciated.

Here is our filebeat config

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    logging.level: debug
    logging.selectors: [ "kubernetes" ]
    filebeat.config:
      inputs:
        # Mounted `filebeat-inputs` configmap:
        path: ${path.config}/inputs.d/*.yml
        # Reload inputs configs as they change:
        reload.enabled: false
      modules:
        path: ${path.config}/modules.d/*.yml
        # Reload module configs as they change:
        reload.enabled: false
    
    #filebeat.autodiscover:
    #providers:
    #  - type: kubernetes
    #    node: ${NODE_NAME}
    #    hints.enabled: true
    #    hints.default_config:
    #      type: container
    #      paths:
    #        - /var/log/pods/*/*/*.log
    #    templates:
    #      - condition:
    #          contains:
    #            kubernetes.container.name: "no-json-logging"
    #        config:
    #          - type: container
    #            paths:
    #              - "/var/log/pods/*-${data.kubernetes.container.id}.log"
    #      - condition:
    #          contains:
    #            kubernetes.container.name: "json-logging"
    #        config:
    #          - type: container
    #            paths:
    #              - "/var/log/pods/*-${data.kubernetes.container.id}.log"
    #            json.keys_under_root: true
    #            json.add_error_key: true
    #            json.message_key: message

    processors:
      - add_cloud_metadata:

    cloud.id: ${ELASTIC_CLOUD_ID}
    cloud.auth: ${ELASTIC_CLOUD_AUTH}

    output.elasticsearch:
      hosts: [ ${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT:443} ]
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}
      protocol:  https 
      ilm.enabled: false
    setup.ilm.enabled: false
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-inputs
  namespace: kube-system
  labels:
    k8s-app: filebeat
data:
  kubernetes.yml: |-
    - type: container
      multiline.pattern:  ^[[:space:]] 
      multiline.negate: false
      multiline.match: after
      symlinks: true
      paths:
        - /var/log/pods/*/*/*.log  
      processors:        
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            in_cluster: true
            add_resource_metadata:
            default_indexers.enabled: false
            default_matchers.enabled: false         
            indexers:
              - container:
            matchers:
              - fields:
                  lookup_fields: ["container.id"]
              - logs_path:
                  logs_path:  /var/log/pods/ 
                  resource_type:  container 
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: kube-system
  labels:
    k8s-app: filebeat
spec:
  selector:
    matchLabels:
      k8s-app: filebeat
  template:
    metadata:
      labels:
        k8s-app: filebeat
    spec:
      serviceAccountName: filebeat
      automountServiceAccountToken: true
      terminationGracePeriodSeconds: 30
      containers:
      - name: filebeat
        image: docker.elastic.co/beats/filebeat-oss:7.12.1       
        args: [
          "-c", "/etc/filebeat.yml",
          "-e",
        ]
        env:        
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        - name: ELASTICSEARCH_HOST
          value: "XXX"
        - name: ELASTICSEARCH_PORT
          value: "443" # 443
        - name: ELASTICSEARCH_USERNAME
          value: "XXX"
        - name: ELASTICSEARCH_PASSWORD
          value: "XXX"
        - name: ELASTIC_CLOUD_ID
          value:
        - name: ELASTIC_CLOUD_AUTH
          value:
        securityContext:
          runAsUser: 0
          # If using Red Hat OpenShift uncomment this:
          #privileged: true
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: config
          mountPath: /etc/filebeat.yml
          readOnly: true
          subPath: filebeat.yml
        - name: inputs
          mountPath: /usr/share/filebeat/inputs.d
          readOnly: true
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: varlibdockercontainers
          mountPath: /var/log/pods
          readOnly: true
      volumes:
      - name: config
        configMap:
          defaultMode: 0600
          name: filebeat-config
      - name: varlibdockercontainers
        hostPath:
          path: /var/log/pods
      - name: inputs
        configMap:
          defaultMode: 0600
          name: filebeat-inputs
      # data folder stores a registry of read status for all files, so we don t send everything again on a Filebeat pod restart
      - name: data
        hostPath:
          path: /var/lib/filebeat-data
          type: DirectoryOrCreate
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: filebeat
subjects:
- kind: ServiceAccount
  name: filebeat
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: filebeat
  labels:
    k8s-app: filebeat
rules:
- apiGroups: [""] # "" indicates the core API group
  resources:
  - namespaces
  - pods
  - nodes
  verbs:
  - get
  - watch
  - list
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat
  namespace: kube-system
  labels:
    k8s-app: filebeat
---

Tried several variations of the config, but cannot get kubernetes metadata to load. Even tried autodiscover mode, but when I try it no logs get into OpenSearch at all.

Expecting tags to show in OpenSearch for kubernetes - for example:

kubernetes.container.name kubernetes.labels.app kubernetes.labels.env kubernetes.pod.name

We have custom labels in pods to scope logs by env, and app name.

Again, all of this was working before the EKS upgrade.

There are no errors in filebeat logs to indicate a problem. Logs appear to be loading via harvesters and indexes are created. Looks like it should be working but its not. The only thing I see in logs is "...did not match any of the cached resources", but not sure what this means.

Attaching log output:

2023-10-05T02:01:06.506Z        INFO    instance/beat.go:660    Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]

2023-10-05T02:01:06.506Z        INFO    instance/beat.go:668    Beat ID: 72dee488-d083-4145-a5a2-ec77566c0519

2023-10-05T02:01:06.509Z        INFO    [add_cloud_metadata]    add_cloud_metadata/add_cloud_metadata.go:105    add_cloud_metadata: hosting provider type detected as aws, metadata={"account":{"id":"xxx"},"availability_zone":"us-east-1b","image":{"id":"ami-013895

b64fa9cbcba"},"instance":{"id":"i-0c4320f7a209ffa8d"},"machine":{"type":"t3.medium"},"provider":"aws","region":"us-east-1"}

2023-10-05T02:01:06.510Z        INFO    [seccomp]       seccomp/seccomp.go:124  Syscall filter successfully installed

2023-10-05T02:01:06.510Z        INFO    [beat]  instance/beat.go:996    Beat info       {"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "fi

lebeat", "uuid": "72dee488-d083-4145-a5a2-ec77566c0519"}}}

2023-10-05T02:01:06.511Z        INFO    [beat]  instance/beat.go:1005   Build info      {"system_info": {"build": {"commit": "651a2ad1225f3d4420a22eba847de385b71f711d", "libbeat": "7.12.1", "time": "2021-04-20T19:58:27.000Z", "version": "7.12.1"}}}

2023-10-05T02:01:06.511Z        INFO    [beat]  instance/beat.go:1008   Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.15.9"}}}

2023-10-05T02:01:06.515Z        INFO    [beat]  instance/beat.go:1012   Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2023-09-29T17:25:38Z","containerized":true,"name":"filebeat-74xql","ip":["127.0.0.1/8","::1/128","172.19.54.66/32","fe80

::6029:bdff:fea6:c030/64"],"kernel_version":"5.10.186-179.751.amzn2.x86_64","mac":["62:29:bd:a6:c0:30"],"os":{"type":"linux","family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":9,"patch":2009,"codename":"Core"},"timezone":"U

TC","timezone_offset_sec":0,"id":"5ea6a666bf6bfd4b2167796371a02dc5"}}}

2023-10-05T02:01:06.517Z        INFO    [beat]  instance/beat.go:1041   Process info    {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw",

"sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","

setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 7, "ppid": 1, "seccomp": {"mode":"filter","no_new_privs"

:true}, "start_time": "2023-10-05T02:01:06.430Z"}}}

2023-10-05T02:01:06.517Z        INFO    instance/beat.go:304    Setup Beat: filebeat; Version: 7.12.1

2023-10-05T02:01:06.517Z        INFO    eslegclient/connection.go:99    elasticsearch url: https://xxx.us-east-1.es.amazonaws.com:443

2023-10-05T02:01:06.518Z        INFO    [publisher]     pipeline/module.go:113  Beat name: filebeat-74xql

2023-10-05T02:01:06.518Z        INFO    [monitoring]    log/log.go:117  Starting metrics logging every 30s

2023-10-05T02:01:06.518Z        INFO    instance/beat.go:468    filebeat start running.

2023-10-05T02:01:06.519Z        INFO    memlog/store.go:119     Loading data file of  /usr/share/filebeat/data/registry/filebeat  succeeded. Active transaction id=222664

2023-10-05T02:01:06.523Z        INFO    memlog/store.go:124     Finished loading transaction log file for  /usr/share/filebeat/data/registry/filebeat . Active transaction id=222868

2023-10-05T02:01:06.523Z        INFO    [registrar]     registrar/registrar.go:109      States Loaded from registrar: 10

2023-10-05T02:01:06.523Z        INFO    [crawler]       beater/crawler.go:71    Loading Inputs: 0

2023-10-05T02:01:06.524Z        INFO    log/input.go:157        Configured paths: [/var/log/pods/*/*/*.log]

2023-10-05T02:01:06.525Z        INFO    [crawler]       beater/crawler.go:108   Loading and starting Inputs completed. Enabled inputs: 0

2023-10-05T02:01:06.525Z        INFO    cfgfile/reload.go:164   Config reloader started

2023-10-05T02:01:06.525Z        INFO    cfgfile/reload.go:224   Loading of config files completed.

2023-10-05T02:01:06.525Z        INFO    cfgfile/reload.go:164   Config reloader started

2023-10-05T02:01:06.529Z        INFO    log/input.go:157        Configured paths: [/var/log/pods/*/*/*.log]

2023-10-05T02:01:06.529Z        INFO    cfgfile/reload.go:224   Loading of config files completed.

2023-10-05T02:01:06.531Z        INFO    log/harvester.go:302    Harvester started for file: /var/log/pods/kube-system_metricbeat-ncnsr_90ac5901-9638-4b09-8fdd-3b1f6ff4f648/metricbeat/0.log

2023-10-05T02:01:06.531Z        INFO    log/harvester.go:302    Harvester started for file: /var/log/pods/kube-system_kube-proxy-2s26d_1b51389c-42b3-4f98-b67a-ce303b3a2c98/kube-proxy/0.log

2023-10-05T02:01:06.531Z        INFO    log/harvester.go:302    Harvester started for file: /var/log/pods/kube-system_filebeat-74xql_33ecdbe6-8d15-4b79-baa8-617e022bd877/filebeat/0.log

2023-10-05T02:01:06.550Z        INFO    add_kubernetes_metadata/kubernetes.go:71        add_kubernetes_metadata: kubernetes env detected, with version: v1.27.4-eks-2d98532

2023-10-05T02:01:06.550Z        DEBUG   [kubernetes]    add_kubernetes_metadata/matchers.go:72  logs_path matcher log path: /var/log/pods/

2023-10-05T02:01:06.550Z        DEBUG   [kubernetes]    add_kubernetes_metadata/matchers.go:73  logs_path matcher resource type: container

2023-10-05T02:01:06.550Z        INFO    [kubernetes]    kubernetes/util.go:99   kubernetes: Using node ip-172-19-55-203.ec2.internal provided in the config     {"libbeat.processor": "add_kubernetes_metadata"}

2023-10-05T02:01:06.550Z        DEBUG   [kubernetes]    add_kubernetes_metadata/kubernetes.go:162       Initializing a new Kubernetes watcher using host: ip-172-19-55-203.ec2.internal {"libbeat.processor": "add_kubernetes_metadata"}

2023-10-05T02:01:06.650Z        DEBUG   [kubernetes]    kubernetes/watcher.go:184       cache sync done

2023-10-05T02:01:06.751Z        DEBUG   [kubernetes]    kubernetes/watcher.go:184       cache sync done

2023-10-05T02:01:06.851Z        DEBUG   [kubernetes]    kubernetes/watcher.go:184       cache sync done

2023-10-05T02:01:06.851Z        DEBUG   [kubernetes]    add_kubernetes_metadata/kubernetes.go:206       Adding kubernetes pod: default/ui-blue-0        {"libbeat.processor": "add_kubernetes_metadata"}

2023-10-05T02:01:06.851Z        DEBUG   [kubernetes]    add_kubernetes_metadata/kubernetes.go:296       Created index ff93836067b818d36598827bb5f6355e7e7f0306d19de9fda6bca32f3977ef95 for pod default/ui-blue-0        {"libbeat.processor": "add_kubernetes_metadata"}

2023-10-05T02:01:06.851Z        DEBUG   [kubernetes]    add_kubernetes_metadata/kubernetes.go:206       Adding kubernetes pod: kube-system/aws-node-n4zb9       {"libbeat.processor": "add_kubernetes_metadata"}

2023-10-05T02:01:06.851Z        DEBUG   [kubernetes]    add_kubernetes_metadata/kubernetes.go:296       Created index 2e0feef7d881d27dbf6542a0d588751e661869cfdbb1cf95335618509bc29b6e for pod kube-system/aws-node-n4zb9       {"libbeat.processor": "add_kubernetes_metadata"

}

2023-10-05T02:01:06.851Z        DEBUG   [kubernetes]    add_kubernetes_metadata/kubernetes.go:296       Created index 2da92f12efb668e79a589484fa2aedbfdf0620c921b00f6c8c4b37d61cba7c25 for pod kube-system/aws-node-n4zb9       {"libbeat.processor": "add_kubernetes_metadata"

}

2023-10-05T02:01:06.851Z        DEBUG   [kubernetes]    add_kubernetes_metadata/kubernetes.go:206       Adding kubernetes pod: kube-system/ebs-csi-node-wk6nn   {"libbeat.processor": "add_kubernetes_metadata"}

2023-10-05T02:01:06.851Z        DEBUG   [kubernetes]    add_kubernetes_metadata/kubernetes.go:296       Created index e1391f837355370f7733689cb6193c7e034f6be1e8901f08e08e9cdd4ff4aa49 for pod kube-system/ebs-csi-node-wk6nn   {"libbeat.processor": "add_kubernetes_metadata"

}

2023-10-05T02:01:06.851Z        DEBUG   [kubernetes]    add_kubernetes_metadata/kubernetes.go:296       Created index d790a2114216777abd6f0902265d390a533f1245bcc9956cf560166a56d7a898 for pod kube-system/ebs-csi-node-wk6nn   {"libbeat.processor": "add_kubernetes_metadata"

}

2023-10-05T02:01:06.852Z        DEBUG   [kubernetes]    add_kubernetes_metadata/kubernetes.go:296       Created index 8e9494a4b785f98acc40ed3cba7a8db78bcc99b7455ef54bc3b601578372c3e2 for pod kube-system/ebs-csi-node-wk6nn   {"libbeat.processor": "add_kubernetes_metadata"

}

2023-10-05T02:01:06.852Z        DEBUG   [kubernetes]    add_kubernetes_metadata/kubernetes.go:206       Adding kubernetes pod: kube-system/filebeat-74xql       {"libbeat.processor": "add_kubernetes_metadata"}

2023-10-05T02:01:06.852Z        DEBUG   [kubernetes]    add_kubernetes_metadata/kubernetes.go:206       Adding kubernetes pod: kube-system/kube-proxy-2s26d     {"libbeat.processor": "add_kubernetes_metadata"}

2023-10-05T02:01:06.852Z        DEBUG   [kubernetes]    add_kubernetes_metadata/kubernetes.go:296       Created index 2b7110b7e3cc61d5fad18595ff0e962a1771ded45133f279fb108637c63305fa for pod kube-system/kube-proxy-2s26d     {"libbeat.processor": "add_kubernetes_metadata"

}

2023-10-05T02:01:06.852Z        DEBUG   [kubernetes]    add_kubernetes_metadata/kubernetes.go:206       Adding kubernetes pod: kube-system/metricbeat-ncnsr     {"libbeat.processor": "add_kubernetes_metadata"}

2023-10-05T02:01:06.852Z        DEBUG   [kubernetes]    add_kubernetes_metadata/kubernetes.go:296       Created index 9a0347946c5e0e28598a8ced3ece865308abe39215783125c9c8581fa8c3a9b1 for pod kube-system/metricbeat-ncnsr     {"libbeat.processor": "add_kubernetes_metadata"

}

2023-10-05T02:01:07.179Z        DEBUG   [kubernetes]    add_kubernetes_metadata/kubernetes.go:211       Updating kubernetes pod: kube-system/filebeat-74xql     {"libbeat.processor": "add_kubernetes_metadata"}

2023-10-05T02:01:07.179Z        DEBUG   [kubernetes]    add_kubernetes_metadata/kubernetes.go:296       Created index 2103af45628eb99b74286c17859bbeee3a7749d2566aa9794d5400474f0ec71b for pod kube-system/filebeat-74xql       {"libbeat.processor": "add_kubernetes_metadata"

}

2023-10-05T02:01:07.532Z        INFO    [publisher_pipeline_output]     pipeline/output.go:143  Connecting to backoff(elasticsearch(https://search-preprod-logging-alwadubacsyc4yts5wjecxdfdq.us-east-1.es.amazonaws.com:443))

2023-10-05T02:01:07.532Z        INFO    [publisher]     pipeline/retry.go:219   retryer: send unwait signal to consumer

2023-10-05T02:01:07.532Z        INFO    [publisher]     pipeline/retry.go:223     done

2023-10-05T02:01:07.533Z        DEBUG   [kubernetes]    add_kubernetes_metadata/matchers.go:88  Incoming log.file.path value: /var/log/pods/kube-system_filebeat-74xql_33ecdbe6-8d15-4b79-baa8-617e022bd877/filebeat/0.log

2023-10-05T02:01:07.533Z        DEBUG   [kubernetes]    add_kubernetes_metadata/matchers.go:126 Using container id: kube-system_filebeat-74xql_33ecdbe6-8d15-4b79-baa8-617e022bd877/

2023-10-05T02:01:07.533Z        DEBUG   [kubernetes]    add_kubernetes_metadata/kubernetes.go:252       Using the following index key kube-system_filebeat-74xql_33ecdbe6-8d15-4b79-baa8-617e022bd877/  {"libbeat.processor": "add_kubernetes_metadata"}

2023-10-05T02:01:07.533Z        DEBUG   [kubernetes]    add_kubernetes_metadata/kubernetes.go:255       Index key kube-system_filebeat-74xql_33ecdbe6-8d15-4b79-baa8-617e022bd877/ did not match any of the cached resources    {"libbeat.processor": "add_kubernetes_metadata"

Any help is appreciated. Thank you.

问题回答

暂无回答


上一篇：
下一篇：


相关问题
Autocomplete in opensearch

I have done autocomplete functionality for opensearch using WCF. But we are using 3.5, so I now need to achieve the same without using WCF. Any idea or please send me any useful link. My scenario is ...

时间：2011-02-08 16:00:06 回答：1 展示：1
Google Custom Search and Opensearch

I have a Google Custom Search Engine on our site and I have recently implemented the opensearch XML file so that I can add our site s search in the browsers search box. Any idea how I can add the ...

时间：2010-07-20 14:21:53 回答：1 展示：1
Confused about namespaces in Atom feed

Is there any difference between <opensearch:totalResults>1000</opensearch:totalResults> and <totalResults xmlns="opensearch">1000</totalResults> I m using the ...

时间：2010-07-01 17:55:53 回答：3 展示：1
Track custom firefox search engine with google analytics

I made a custom Firefox search engine. It used the code below. Is it possible for me to track the use of this on the site. My concern is that searches form the firefox search engine will look like ...

时间：2010-02-02 05:28:29 回答：2 展示：1
OpenSearch .net (C#) library: which one?

i would like to add OpenSearch to my C# webapplication. does someone recommend a certain .net library? or does a custom webapplication require a tailormade opensearch implementation?

时间：2010-01-18 08:59:47 回答：3 展示：1
热门标签

友情链接

Allggapp Alljchome-教程家园 mvfinale.com-影视剧情大结局大全