lundi 24 juillet 2017

Need to create a field for multiple type exceptions-logstash

I have configured logstash configuration to create an index in ES for all the logs present in three node cluster. Please find the below conf.

input {
  file {
    path => "D:\Development_Avecto\logstash-2.4.1\bin\dev_exceptions.txt"
    start_position => "beginning"
    sincedb_path => "NUL"
    codec => multiline {
            pattern => "^\[%{TIMESTAMP_ISO8601:TIMESTAMP}\]"
            negate => true
            what => "previous"
        }
  }
}
filter {
    grok {
      match => [ "message", "(?m)^\[%{TIMESTAMP_ISO8601:TIMESTAMP}\]\[%{LOGLEVEL:LEVEL}%{SPACE}\]\[%{DATA:ERRORTYPE}%{SPACE}\]%{SPACE}(?<ERRORMESSAGE>(.|\r|\n)*)"]
    }
# DEBUG Logs
if "grokked" not in [tags] and "DEBUG" == [LEVEL] {
grok { match => [ "ERRORMESSAGE", "(?m)^\[%{DATA:SERVERNAME}\]" ]
add_tag => [ "Debug Logs", "grokked" ]
tag_on_failure => [ ]
}
}
# INFO Logs
if "grokked" not in [tags] and "INFO" == [LEVEL] {
grok { match => [ "ERRORMESSAGE", "(?m)^\[%{DATA:SERVERNAME}\]" ]
add_tag => [ "Info Logs", "grokked" ]
tag_on_failure => [ ]
}
}
# WARN Logs
if "grokked" not in [tags] and "WARN" == [LEVEL] {
grok { match => [ "ERRORMESSAGE", "(?m)^\[%{DATA:SERVERNAME}\]" ]
add_tag => [ "Warn Logs", "grokked" ]
tag_on_failure => [ ]
}
}
if "Exception" in [ERRORMESSAGE] {
    mutate { add_field => { "Exception" => "Exception" } 
                }
}
date {
    match => [ "TIMESTAMP" , "yyyy-MM-dd HH:mm:ss,SSS" ]
    target => "TIMESTAMP"
  }
mutate {
        remove_field => ["message","@version","path","host","tags","type" ]
      }
}
output {
elasticsearch { hosts => ["localhost:9200"]
     index => "logstash-devlogs"
}
stdout { codec => rubydebug }
}

As per the conf file, a new field called "Exception"(specified in If condition) has created for the mentioned index.

And i need one more raw field should be created for the "Exception" field with respect to errors(for Eg: if it is a mapper parse exception or search parse exception or Invalid exception or Timeout exception) Kindly post your comments ASAP. very urgent.

Thanks, Narmatha

Aucun commentaire:

Enregistrer un commentaire