How To Map User Location with GeoIP and ELK (Elasticsearch, Logstash, and Kibana)

Thanks for your nice tutorial.

Please also provide Step-by-Step Setup of ELK for NetFlow traffic Analytics. (Kibana 4 Preferred) so that we can monitor traffic from different sources like Router, Firewall and server.

Thanks Ashik

Hi,

Thanks for the nice tutorial.

Just to mention, you may need to change the mappings for geoip.location field and set it to ‘geo_point’ or else you won’t be able to get the Field ‘geiop.location’ displayed under Geo Co-ordinates > Field when you try to create a MAP.

I was using an index called ‘apache’ so initially the filed was set to double

$ curl http://localhost:9200/apache*/_mapping/apache-access/field/geoippretty
{
  "apache" : {
    "mappings" : {
      "apache-access" : {
        "geoip.location" : {
          "full_name" : "geoip.location",
          "mapping":{"location":{"type":"double"}}
        }
      }
    }
  }
}

I had to change it to ‘geo-point’ to get the filed on the map.

$curl http://localhost:9200/apache*/_mapping/apache-access/field/geoip.location?pretty
{
  "apache" : {
    "mappings" : {
      "apache-access" : {
        "geoip.location" : {
          "full_name" : "geoip.location",
          "mapping":{"location":{"type":"geo_point"}}
        }
      }
    }
  }
}

To change the mappings:

1. copy the default lib/logstash/outputs/elasticsearch/elasticsearch-template.json to something like lib/logstash/outputs/elasticsearch/elasticsearch-apache-template.json
2. Edit the elasticsearch-apache-template.json and give it a name “template” : “apache”,
3. You will require to delete your existing index (THIS WILL LOOSE DATA). curl -XDELETE http://localhost:9200/apache*
4. Edit your logstash.conf file to include the template in the output

output {
  elasticsearch {
     host => localhost
     cluster => es_24
     index   => "apache"
     template => "path_to_elasticsearch-apache-template.json"
     template_name => "apache"
  }
  stdout { codec => rubydebug }
}

5. Restart ES and LS and it should work.

Hello, Mitchell Anicas I encountered a problem about grok pattern in my elk environment. Could you help me? Follows:

1. My nginx log format:

log_format access '$remote_addr - $remote_user [$time_local] "$request"' '$status $body_bytes_sent $request_body "$http_referer"' '
"$http_user_agent"  $http_x_forwarded_for $host $request_time $upstream_response_time' " $upstream_cache_status";

2. log contents :

124.134.229.30 - - [18/Aug/2015:03:30:01 +0800] "POST /?nav=24xsc&flaglot=hnquick5&tit=%E6%B2%B3%E5%86%855%E5%88%86%E5%BD%A9 HTTP/1.0"200 23 lotteryid=14&issue=20150818-043&flag=gettime "http://x8shi.com/?nav=24xsc&flaglot=hnquick5&tit=%E6%B2%B3%E5%86%855%E5%88%86%E5%BD%A9""Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36"  124.134.229.30 x8shi.com 0.140 0.027 -

now how to write a grok pattern ? I have tried many times，but all failed.

tags _grokparsefailure

Please help me ! Thank you very much !

Hi Manica, first of all thanks for your tutorials.

I’ve problems with mi geoip ip configuration, i’ve kibana 4.1.1 with logstash 1.5.3 and apache 2.4.

This is my personalized geoip file configuration on logstash, i called 12-geoip.conf :

filter { if [type] == “apache_access” { grok { match => { “message” => “%{COMMONAPACHELOG}” }

} geoip { source => “clientip” target => “geoip.location” database => “/etc/logstash/GeoLiteCity.dat” add_field => [ “[geoip.location][coordinates]”, “%{[geoip.location][longitude]}” ] add_field => [ “[geoip.location][coordinates]”, “%{[geoip.location][latitude]}” ] } mutate { convert => [ “[geoip.location][coordinates]”, “float”] } } }

Mi apache configuration on other file filter { if [type] == “apache” { grok { match => { “message” => “%{COMBINEDAPACHELOG}” } add_field => [ “received_from”, “%{host}” ] } } }

My problem is that Kibana don’t appears the “geoip” field, but clientip, hostname, etc… appears.

Part of my logstash-forwarder configuration

{
  "paths": [
    "/var/log/apache2/*error.log",
    "/var/log/apache2/*access-ssl.log"
   ],
  "fields": { "type": "apache" }
},
{
  "paths": [
    "/var/log/apache2/*access.log"
   ],
  "fields": { "type": "apache_access" }
 }

] }

And my logstash-forwarder registered events:

2015/09/04 09:07:57.807119 Registrar: processing 5 events 2015/09/04 09:08:45.244783 Registrar: processing 2 events 2015/09/04 09:08:50.238178 Registrar: processing 2 events 2015/09/04 09:09:02.744967 Registrar: processing 1 events

Thax so much Manica.

Hi Manica i have problems

i finally created my own “index pattern”, i have my template, and when logstash starting catch my template, i see that with the kopf plugin but my problem continues

My geoip and client mappings:

"geoip": {
  "dynamic": "true",
  "properties": {
    "location": {
      "doc_values": true,
      "type": "geo_point"
    },
    "longitude": {
      "doc_values": true,
      "type": "float"
    },
    "latitude": {
      "doc_values": true,
      "type": "float"
    },
    "ip": {
      "doc_values": true,
      "type": "ip"
    }
  }
},

for client ip:

"clientip": {
    "doc_values": true,
    "type": "ip"
  },

I regenerated thel index and appears the new fields:

this is my apache configuration for logstash

filter {
  if [type] == "apache_error" {
    grok {
      pattern => "%{COMBINEDAPACHELOG}"
      add_field => [ "received_from", "%{host}" ]
    }
  }
}
filter {
  if [type] == "apache_access" {
    grok {
      match => {"message" => "%{COMBINEDAPACHELOG}" }
    }
    geoip {
      source => "clientip"
      target => "geoip"
      database => "/etc/logstash/GeoLiteCity.dat"
      add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
      add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
    }
    mutate {
      convert => [ "[geoip][coordinates]", "float" ]
    }
  }
}
filter {
  if [type] == "apache_access_ssl" {
    grok {
      match => {"message" => "%{COMBINEDAPACHELOG}" }
    }
    geoip {
      source => "clientip"
      target => "geoip"
      database => "/etc/logstash/GeoLiteCity.dat"
      add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
      add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
    }
    mutate {
      convert => [ "[geoip][coordinates]", "float" ]
    }
  }
}

and the output for logstash

output {
  elasticsearch {
    host => "marioneto01"
    cluster => "oknels"
    protocol => "http"
    index => "okn-%{+YYYY.MM.dd}"
    template => "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-1.0.5-java/lib/logstash/outputs/elasticsearch/okn-template.json"
    template_name => "okn-*"
    manage_template => "true"
    #template_overwrite => "true"
  }
  stdout { codec => rubydebug }
}

ideas?? thanks

Hi, Mitchell Anicas

I want to Thank you from the bottom of my heart, for your wonderful tutorials on ELK stack. They are so great and step by step with you, I have completed the all three tutorials. The only thing left is to try the final one, which is about Map User Location. the problem I don’t have Apache or Nginx server hosting a real website where people are accessing it with their IP.

But again Thank you mate for your wonderful tutorials.

What a neat tutorial … hats off to author and team … keep up good work !! SF From SW England ! P.S. Is there any place i can see a example of application log which use log4j logs … btway that’s what i m going to do next …

Any pointers on how to do this for fluentd (using logstash format) instead of logstash?

Followed every part of the tutorial ,

Getting the Error ,

No Compatible Fields: The “[filebeat-]YYYY.MM.DD” index pattern does not contain any of the following field types: geo_point

have searched Google and unable to find the root cause of this error , can you help