星空网 > 软件开发 > 数据库

Logstashmultiline 插件

本文内容

  • 测试数据
  • 字段属性
  • 示例
  • 参考资料

在处理日志时，除了访问日志外，还要处理运行时日志，该日志大都用程序写的，比如 log4j。运行时日志跟访问日志最大的不同是，运行时日志是多行，也就是说，连续的多行才能表达一个意思。

本文主要说明，如何用 multiline 出来运行日志。

如果能按多行处理，那么把他们拆分到字段就很容易了。

测试数据

[16-04-12 03:40:01 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.
[16-04-12 03:40:02 DEBUG] impl.JdbcEntityInserter:- from product_category product_category
where product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null
order by product_category.ORDERS asc
[16-04-12 03:40:03 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.
[16-04-12 03:40:04 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.
[16-04-12 03:40:05 DEBUG] impl.JdbcEntityInserter:- from product_category product_category
where product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null
order by product_category.ORDERS desc
[16-04-12 03:40:06 DEBUG] impl.JdbcEntityInserter:- from product_category product_category
where product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null
order by product_category.ORDERS asc
[16-04-12 03:40:07 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.


测试是在7秒内发生的（当然是假数据）。可以看到，第二、五、六秒的日志是多行的，有条SQL语句。其他是单行的。

字段属性

对 multiline 插件来说，有三个设置比较重要：negate、pattern 和 what。

negate

    • 类型是 boolean

    • 默认为 false

      Negate the regexp pattern (if not matched).否定正则表达式（如果没有匹配的话）。

      pattern

        • 必须设置

        • 类型为 string

        • 没有默认值

          要匹配的正则表达式。

          what

            • 必须设置

            • 可以为 previous 或 next

            • 没有默认值

              如果正则表达式匹配了，那么该事件是属于下一个或是前一个事件？

              示例

              示例1：若配置文件如下所示，

              input {
                  file{
                      path=>"/usr/local/elk/logstash/logs/c.out"
                      type=>"runtimelog"
                      codec=> multiline {
                          pattern => "^\["
                          negate => true
                          what => "previous"
                      }
                      start_position=>"beginning"
                      sincedb_path=>"/usr/local/elk/logstash/sincedb-access"
                      ignore_older=>0
                  }
              }
              output{
                  stdout{
                      codec=>rubydebug
                  }
              }


              解析结果如下所示，能解析出6个JSON：

              {
                "@timestamp" => "2016-06-01T04:37:43.147Z",
                  "message" => "[16-04-12 03:40:01 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.",
                 "@version" => "1",
                   "path" => "/usr/local/elk/logstash/logs/c.out",
                   "host" => "vcyber",
                   "type" => "runtimelog"
              }
              {
                "@timestamp" => "2016-06-01T04:37:43.152Z",
                  "message" => "[16-04-12 03:40:02 DEBUG] impl.JdbcEntityInserter:- from product_category product_category\nwhere product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS asc",
                 "@version" => "1",
                   "tags" => [
                  [0] "multiline"
                ],
                   "path" => "/usr/local/elk/logstash/logs/c.out",
                   "host" => "vcyber",
                   "type" => "runtimelog"
              }
              {
                "@timestamp" => "2016-06-01T04:37:43.152Z",
                  "message" => "[16-04-12 03:40:03 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.",
                 "@version" => "1",
                   "path" => "/usr/local/elk/logstash/logs/c.out",
                   "host" => "vcyber",
                   "type" => "runtimelog"
              }
              {
                "@timestamp" => "2016-06-01T04:37:43.155Z",
                  "message" => "[16-04-12 03:40:04 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.",
                 "@version" => "1",
                   "path" => "/usr/local/elk/logstash/logs/c.out",
                   "host" => "vcyber",
                   "type" => "runtimelog"
              }
              {
                "@timestamp" => "2016-06-01T04:37:43.157Z",
                  "message" => "[16-04-12 03:40:05 DEBUG] impl.JdbcEntityInserter:- from product_category product_category\nwhere product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS desc",
                 "@version" => "1",
                   "tags" => [
                  [0] "multiline"
                ],
                   "path" => "/usr/local/elk/logstash/logs/c.out",
                   "host" => "vcyber",
                   "type" => "runtimelog"
              }
              {
                "@timestamp" => "2016-06-01T04:37:43.159Z",
                  "message" => "[16-04-12 03:40:06 DEBUG] impl.JdbcEntityInserter:- from product_category product_category\nwhere product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS asc",
                 "@version" => "1",
                   "tags" => [
                  [0] "multiline"
                ],
                   "path" => "/usr/local/elk/logstash/logs/c.out",
                   "host" => "vcyber",
                   "type" => "runtimelog"
              }


              解析时，最后一行日志，不会解析。只有当再追加一条日志时，才会解析最后一条日志。

              示例2：若将配置文件修改为，

              input {
                  file{
                      path=>"/usr/local/elk/logstash/logs/c.out"
                      type=>"runtimelog"
                      codec=>multiline {
                          pattern => "^\["
                          negate => true
                          what => "next"
                      }
                      start_position=>"beginning"
                      sincedb_path=>"/usr/local/elk/logstash/sincedb-access"
                      ignore_older=>0
                  }
              }
              output{
                  stdout{
                      codec=>rubydebug
                  }
              }


              解析结果为，能解析出7个JSON：

              {
                "@timestamp" => "2016-06-01T04:40:43.232Z",
                  "message" => "[16-04-12 03:40:01 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.",
                 "@version" => "1",
                   "path" => "/usr/local/elk/logstash/logs/c.out",
                   "host" => "vcyber",
                   "type" => "runtimelog"
              }
              {
                "@timestamp" => "2016-06-01T04:40:43.237Z",
                  "message" => "[16-04-12 03:40:02 DEBUG] impl.JdbcEntityInserter:- from product_category product_category",
                 "@version" => "1",
                   "path" => "/usr/local/elk/logstash/logs/c.out",
                   "host" => "vcyber",
                   "type" => "runtimelog"
              }
              {
                "@timestamp" => "2016-06-01T04:40:43.238Z",
                  "message" => "where product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS asc\n[16-04-12 03:40:03 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.",
                 "@version" => "1",
                   "tags" => [
                  [0] "multiline"
                ],
                   "path" => "/usr/local/elk/logstash/logs/c.out",
                   "host" => "vcyber",
                   "type" => "runtimelog"
              }
              {
                "@timestamp" => "2016-06-01T04:40:43.239Z",
                  "message" => "[16-04-12 03:40:04 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.",
                 "@version" => "1",
                   "path" => "/usr/local/elk/logstash/logs/c.out",
                   "host" => "vcyber",
                   "type" => "runtimelog"
              }
              {
                "@timestamp" => "2016-06-01T04:40:43.244Z",
                  "message" => "[16-04-12 03:40:05 DEBUG] impl.JdbcEntityInserter:- from product_category product_category",
                 "@version" => "1",
                   "path" => "/usr/local/elk/logstash/logs/c.out",
                   "host" => "vcyber",
                   "type" => "runtimelog"
              }
              {
                "@timestamp" => "2016-06-01T04:40:43.245Z",
                  "message" => "where product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS desc\n[16-04-12 03:40:06 DEBUG] impl.JdbcEntityInserter:- from product_category product_category",
                 "@version" => "1",
                   "tags" => [
                  [0] "multiline"
                ],
                   "path" => "/usr/local/elk/logstash/logs/c.out",
                   "host" => "vcyber",
                   "type" => "runtimelog"
              }
              {
                "@timestamp" => "2016-06-01T04:40:43.249Z",
                  "message" => "where product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null\norder by product_category.ORDERS asc\n[16-04-12 03:40:07 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.",
                 "@version" => "1",
                   "tags" => [
                  [0] "multiline"
                ],
                   "path" => "/usr/local/elk/logstash/logs/c.out",
                   "host" => "vcyber",
                   "type" => "runtimelog"
              }


              示例3：若将配置文件修改为，

              codec=>multiline {
                  pattern => "^\["
                  negate => false
                  what => "previous"
              }


              则解析结果为：

              {
                "@timestamp" => "2016-06-01T05:38:50.853Z",
                  "message" => "[16-04-12 03:40:01 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.\n[16-04-12 03:40:02 DEBUG] impl.JdbcEntityInserter:- from product_category product_category",
                 "@version" => "1",
                   "tags" => [
                  [0] "multiline"
                ],
                   "path" => "/usr/local/elk/logstash/logs/c.out",
                   "host" => "vcyber",
                   "type" => "runtimelog"
              }
              {
                "@timestamp" => "2016-06-01T05:38:50.856Z",
                  "message" => "where product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null",
                 "@version" => "1",
                   "path" => "/usr/local/elk/logstash/logs/c.out",
                   "host" => "vcyber",
                   "type" => "runtimelog"
              }
              {
                "@timestamp" => "2016-06-01T05:38:50.858Z",
                  "message" => "order by product_category.ORDERS asc\n[16-04-12 03:40:03 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.\n[16-04-12 03:40:04 DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.\n[16-04-12 03:40:05 DEBUG] impl.JdbcEntityInserter:- from product_category product_category",
                 "@version" => "1",
                   "tags" => [
                  [0] "multiline"
                ],
                   "path" => "/usr/local/elk/logstash/logs/c.out",
                   "host" => "vcyber",
                   "type" => "runtimelog"
              }
              {
                "@timestamp" => "2016-06-01T05:38:50.860Z",
                  "message" => "where product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null",
                 "@version" => "1",
                   "path" => "/usr/local/elk/logstash/logs/c.out",
                   "host" => "vcyber",
                   "type" => "runtimelog"
              }
              {
                "@timestamp" => "2016-06-01T05:38:50.861Z",
                  "message" => "order by product_category.ORDERS desc\n[16-04-12 03:40:06 DEBUG] impl.JdbcEntityInserter:- from product_category product_category",
                 "@version" => "1",
                   "tags" => [
                  [0] "multiline"
                ],
                   "path" => "/usr/local/elk/logstash/logs/c.out",
                   "host" => "vcyber",
                   "type" => "runtimelog"
              }
              {
                "@timestamp" => "2016-06-01T05:38:50.863Z",
                  "message" => "where product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null",
                 "@version" => "1",
                   "path" => "/usr/local/elk/logstash/logs/c.out",
                   "host" => "vcyber",
                   "type" => "runtimelog"
              }


              参考资料

              • Logstash multiline



              原标题：Logstashmultiline 插件

              关键词：

              *特别声明：以上内容来自于网络收集，著作权属原作者所有，如有侵权，请联系我们： admin#shaoqun.com （#换成@）。

              金华跨境电商：https://www.goluckyvip.com/tag/949.html
              泰国港爆炸：https://www.goluckyvip.com/tag/9490.html
              跨境海外仓专线：https://www.goluckyvip.com/tag/94900.html
              海外仓是谁提出的：https://www.goluckyvip.com/tag/94902.html
              正规的海外仓一件代发：https://www.goluckyvip.com/tag/94903.html
              自贸仓和海外仓：https://www.goluckyvip.com/tag/94904.html
              夹江千佛岩景区门票（夹江千佛岩景区门票价格）：https://www.vstour.cn/a/411232.html
              武陵山大裂谷周围景点 武陵山大裂谷周围景点图片：https://www.vstour.cn/a/411233.html
              相关文章
              我的浏览记录
              最新相关资讯
              跨境电商 软件开发 营销推广 产品运营 短视频运营

              Copyright © 星空网

              海外公司注册 | 跨境电商服务平台 | 深圳旅行社 | 东南亚物流