02/07/2017 - ELASTICSEARCH, LINUX
In this example we are going to use Filebeat to forward logs from two different logs files to Logstash where they will be inserted into their own Elasticsearch indexes.
filebeat:
prospectors:
-
paths:
- /var/log/apache2/access.log
input_type: log
document_type: apache-access
-
paths:
- /var/log/symfony/dev.log
input_type: log
document_type: symfony-dev
As you can see, we use mutate
block to define new variable for Elasticsearch index to use in "output" block. You can move mutate
in "if" and "else" blocks if you wish.
input {
beats {
port => 5044
}
}
filter {
if [type] == "apache-access" {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
} else if [type] == "symfony-dev" {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
}
mutate {
add_field => { "index_name" => "web-%{type}" }
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
sniffing => true
manage_template => false
index => "%{index_name}"
}
}
This is how you use type
without "if" condition in "output" block.
input {
beats {
port => 5044
}
}
filter {
if [type] == "apache-access" {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
} else if [type] == "symfony-dev" {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
sniffing => true
manage_template => false
index => "web-%{type}"
}
}
This is how you use type
with "if" condition in "output" block. I am duplicating lines just to show you how things are done.
input {
beats {
port => 5044
}
}
filter {
if [type] == "apache-access" {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
} else if [type] == "symfony-dev" {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
}
}
output {
if [type] == "apache-access" {
elasticsearch {
hosts => ["localhost:9200"]
sniffing => true
manage_template => false
index => "web-%{type}"
}
} else if [type] == "symfony-dev" {
elasticsearch {
hosts => ["localhost:9200"]
sniffing => true
manage_template => false
index => "web-%{type}"
}
}
}