English 中文(简体)
logstash的csv数据解析问题
原标题：csv data parsing issue with logstash

我正在尝试将csv报告上传到logstash中，但它没有按预期工作。

我的csv文件有200多行，其中一行用于下面给出的引用。

$ cat app.csv
Sl.No,Emp_ID,Date,Emp_Name,Product_Name,Item_Details
1,1234556,30-12-2022,frank.van,SAMPLE_PRODUCT,"[Name] Frank Van Puffelen JAVA.
[Area/Pin] San Francisco, CA 
[Region/Status/Identify] Android Plaltfrom
[Case#] Jira-01234
[Problem] Messaging app not booting.
[Staring Point] Google service for the notifications
[Evaluate] Cloud Messaging.
[Verification Mode] Local Device.
[Empname] Frank Van.

Domain:Cloud_S,Android:S_OS
***** Ticket Status : https://jenkins.company.com/job/889900112 *****
"

我的＜code＞logstash conf＜/code＞文件如下。

input {
   file {
      path => "/home/user/logs/app.csv"
      start_position => "beginning"
      sincedb_path => "/dev/null"
      codec => multiline { 
      pattern =>  ^" 
      negate => "true"
      what => "next"
}
   }
}
filter {
    csv {
        separator => ","
        columns => ["Sl.No", "Emp_ID", "Date", "Emp_Name", "Product_Name", "Item_Details"]
    }

}
output {
  elasticsearch {
    hosts => "localhost:9200"
    index => "java-app"
    document_type => "Emp_ID"
  }
  stdout{}
}

在logstash日志中，它显示CSV列标题值，而不是加载实际值。

logstash         |                  "Emp_Name" => "Emp_Name",
logstash         |        "Product_Name" => "Product_Name",
logstash         |               "message" => "Sl.No,Emp_ID,Date,Emp_Name,Product_Name,Item_Details
1,1234556,30-12-2022,frank.van,SAMPLE_PRODUCT,"[Name] Frank Van Puffelen JAVA.
[Area/Pin] San Francisco, CA 
[Region/Status/Identify] Android Plaltfrom
[Case#] Jira-01234
[Problem] Messaging app not booting.
[Staring Point] Google service for the notifications
[Evaluate] Cloud Messaging.
[Verification Mode] Local Device.
[Empname] Frank Van.

Domain:Cloud_S,Android:S_OS
***** Ticket Status : https://jenkins.company.com/job/889900112 *****
",
logstash         |              "@version" => "1",
logstash         |                  "path" => "/home/user/logs/app.csv",
logstash         |          "Date" => "Date",
logstash         |           "Item_Details" => "Item_Details",
logstash         |     "Emp_ID" => "Emp_ID",
logstash         |            "@timestamp" => 2023-06-22T05:48:22.714Z,
logstash         |                  "tags" => [
logstash         |         [0] "multiline"
logstash         |     ],
logstash         |         "host" => "828967718f28",
logstash         |         "Sl.No" => "Sl.No"
logstash         | }

你能建议如何将我的csv文件数据上传到logstash中吗？我的Item_Details列包含双引号。

@保罗，这是我更新的logstashconf文件。

input {
  file {
    path => "/home/user/logs/app.csv"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    codec => multiline { 
    pattern =>  ^d 
    negate => "false"
    what => "next"
  }
 } 
}

filter {
    csv {
      separator => ","
      skip_header => true
      columns => ["Sl.No", "Emp_ID", "Date", "Emp_Name", "Product_Name", "Item_Details"]
  }
}

Logstash输出

logstash         | [2023-06-24T02:46:02,959][WARN ][logstash.filters.csv     ][main][aacc2f62062158dfe25eef66ddf5744e6c49abde51f3b44a542bcdacf04017fc] Error parsing csv {:field=>"message", :source=>""
", :exception=>#<CSV::MalformedCSVError: Unclosed quoted field on line 1.>}
logstash         | {
logstash         |           "tags" => [
logstash         |         [0] "multiline",
logstash         |         [1] "_csvparsefailure"
logstash         |     ],
logstash         |        "message" => "1,1234556,30-12-2022,frank.van,SAMPLE_PRODUCT,"[Name] Frank Van Puffelen JAVA.
[Area/Pin] San Francisco, CA 
",
logstash         |     "@timestamp" => 2023-06-24T02:46:02.757Z,
logstash         |           "path" => "/home/user/logs/app.csv",
logstash         |       "@version" => "1",
logstash         |           "host" => "11f690730fff"
logstash         | }
logstash         | {
logstash         |           "message" => "[Evaluate] Cloud Messaging.
",
logstash         |     "serial_number" => "[Evaluate] Cloud Messaging.",
logstash         |        "@timestamp" => 2023-06-24T02:46:02.759Z,
logstash         |              "path" => "/home/user/logs/app.csv",
logstash         |          "@version" => "1",
logstash         |              "host" => "11f690730fff"
logstash         | }
logstash         | {
logstash         |           "path" => "/home/user/logs/app.csv",
logstash         |       "@version" => "1",
logstash         |        "message" => "
",
logstash         |           "host" => "11f690730fff",
logstash         |     "@timestamp" => 2023-06-24T02:46:02.760Z
logstash         | }
logstash         | {
logstash         |           "message" => "***** Ticket Status : https://jenkins.company.com/job/889900112 *****
",
logstash         |     "serial_number" => "***** Ticket Status : https://jenkins.company.com/job/889900112 *****",
logstash         |        "@timestamp" => 2023-06-24T02:46:02.760Z,
logstash         |              "path" => "/home/user/logs/app.csv",
logstash         |          "@version" => "1",
logstash         |              "host" => "11f690730fff"
logstash         | }
logstash         | {
logstash         |           "message" => "[Staring Point] Google service for the notifications
",
logstash         |     "serial_number" => "[Staring Point] Google service for the notifications",
logstash         |        "@timestamp" => 2023-06-24T02:46:02.759Z,
logstash         |              "path" => "/home/user/logs/app.csv",
logstash         |          "@version" => "1",
logstash         |              "host" => "11f690730fff"
logstash         | }
logstash         | {
logstash         |           "message" => "[Problem] Messaging app not booting.
",
logstash         |     "serial_number" => "[Problem] Messaging app not booting.",
logstash         |        "@timestamp" => 2023-06-24T02:46:02.758Z,
logstash         |              "path" => "/home/user/logs/app.csv",
logstash         |          "@version" => "1",
logstash         |              "host" => "11f690730fff"
logstash         | }
logstash         | {
logstash         |           "message" => "[Verification Mode] Local Device.
",
logstash         |     "serial_number" => "[Verification Mode] Local Device.",
logstash         |        "@timestamp" => 2023-06-24T02:46:02.759Z,
logstash         |              "path" => "/home/user/logs/app.csv",
logstash         |          "@version" => "1",
logstash         |              "host" => "11f690730fff"
logstash         | }
logstash         | {
logstash         |           "message" => "[Region/Status/Identify] Android Plaltfrom
",
logstash         |     "serial_number" => "[Region/Status/Identify] Android Plaltfrom",
logstash         |        "@timestamp" => 2023-06-24T02:46:02.758Z,
logstash         |              "path" => "/home/user/logs/app.csv",
logstash         |          "@version" => "1",
logstash         |              "host" => "11f690730fff"
logstash         | }
logstash         | {
logstash         |           "message" => "[Case#] Jira-01234
",
logstash         |     "serial_number" => "[Case#] Jira-01234",
logstash         |        "@timestamp" => 2023-06-24T02:46:02.758Z,
logstash         |              "path" => "/home/user/logs/app.csv",
logstash         |          "@version" => "1",
logstash         |              "host" => "11f690730fff"
logstash         | }
logstash         | {
logstash         |           "message" => "[Empname] Frank Van.
",
logstash         |     "serial_number" => "[Empname] Frank Van.",
logstash         |        "@timestamp" => 2023-06-24T02:46:02.760Z,
logstash         |              "path" => "/home/user/logs/app.csv",
logstash         |          "@version" => "1",
logstash         |              "host" => "11f690730fff"
logstash         | }
logstash         | {
logstash         |               "message" => "Domain:Cloud_S,Android:S_OS
",
logstash         |         "serial_number" => "Domain:Cloud_S",
logstash         |            "@timestamp" => 2023-06-24T02:46:02.760Z,
logstash         |                  "path" => "/home/user/logs/app.csv",
logstash         |              "@version" => "1",
logstash         |                  "host" => "11f690730fff",
logstash         |     "changelist_number" => "Android:S_OS"
logstash         | }
logstash         | {
logstash         |           "tags" => [
logstash         |         [0] "_csvparsefailure"
logstash         |     ],
logstash         |        "message" => ""
",
logstash         |     "@timestamp" => 2023-06-24T02:46:02.761Z,
logstash         |           "path" => "/home/user/logs/app.csv",
logstash         |       "@version" => "1",
logstash         |           "host" => "11f690730fff"
logstash         | }

这不是预期的结果，字段值未正确对齐。你能帮我用conf文件把csv文件插入logstash吗？这会很有帮助。

问题回答

Tldr;

似乎在第一次迭代中，多行匹配将第一行的行与页眉和实际信息进行匹配。

Solution;

要么删除包含头的第一行，然后再将其发送到logstash。

或者，您也可以使用与每个条目正确匹配的另一种模式。

可以匹配的模式的想法如下：

codec => multiline { 
  pattern =>  ^d 
  negate => "false"
  what => "next"
}
时间：2023-06-23 13:31:45


上一篇：
下一篇：


相关问题
Roll over index with elastic search and serilog

We are using es 6.7 and serilog 7.1 in our dotnet core application. In our logger implementation vi are using the following index "app-{0:yyyy.MM}-1" for our ElasticsearchSinkOptions. This ...

时间：2019-04-29 19:39:46 回答：1 展示：1
Change the date format in kibana

I am working for my internship on the implementation of the ElasticSearch family (ElasticSearch+ Kibana+ logstash). Here is my question: I have a field "@ timestamp" with the following format: 2014-05-...

时间：2014-05-23 17:10:46 回答：1 展示：1
热门标签

友情链接

Allggapp Alljchome-教程家园 mvfinale.com-影视剧情大结局大全