Download - Visualize NGiNX log with GoogleMap
Transcript
- 1. Visualize NGiNX log with GoogleMap
- 2.
- 3. purpose ! NGiNX ! !
- 4.
- 5. ! Internet ! ! GoogleMap
- 6. ! ngx_http_geoip_module ! Google BigQuery ! () GeoFUSE
- 7. GeoFUSE? ! GPL ! mbasa/GeoFuse GitHub : https://github.com/mbasa/geofuse ! GeoFUSE : http://geofuse.georepublic.net/geofuse/ ! ! (GUI)
- 8.
- 9.
- 10. overwiew GeoFUSE Big QueryNGiNX 1.Logging 2.Transfer 3.Query 4.CSV
- 11. 1.Logging ! NginxGeoIP ! GeoIP C APIGeoIP Datale ! nginx--with-http_geoip_module ! Nginx.confGeoIP Datale ! GeoIPIP http { geoip_country /usr/local/share/GeoIP/GeoIP.dat; geoip_city /usr/local/share/GeoIP/GeoIPCity.dat; real_ip_header X-Forwarded-For; set_real_ip_from 10.0.0.0/8; log_format ltsv "geoip_country_name:" "$geoip_city_country_name" "t" # "geoip_country_code3:" "$geoip_city_country_code3" "t" # JPN/USA "geoip_city:" "$geoip_city" "t" # "geoip_latitude:" "$geoip_latitude" "t" # "geoip_longitude:" "$geoip_longitude" # ; }
- 12. 2.Transfer ! NginxBigQuery ! uentd ! in_tail ! uent-plugin-google-cloud ! NGiNXltsv ! BigQueryjson ()
- 13. 3.Query ! textPayload json ! JSON_EXTRACT JSON_EXTRACT_SCALAR ! (SUM,AVG,MIN,MAX) SELECT ROW_NUMBER() OVER (ORDER BY count DESC) AS ID, COUNT (*) AS count, JSON_EXTRACT_SCALAR(textPayload,'$.geoip_longitude') AS lon, JSON_EXTRACT_SCALAR(textPayload,'$.geoip_latitude') AS lat, AVG(JSON_EXTRACT_SCALAR (textPayload,'$.response_time')) AS avelage_time FROM [HOGE.nginx_access_20150402] GROUP BY lat,lon ORDER BY count DESC
- 14. 4.csv ! BigQuerycsv ! Geo-FUSE ! Geo-FUSE ! ! ID ! lon.lat() ID,count,avelage_time,lon,lat 1,3135686,0.2498652794178637,139.6900,35.6900 2,1596097,0.2536400474791719,139.7514,35.6850 3,182082,0.23747259653015085,135.5200,34.6864 4,117914,0.22794176617882336,121.5250,25.0392
- 15.
- 16. ! EC ( IP) ! (Wi-)
- 17.