一、基于hadoop的nginx访问日志分析---解析日志篇

时间:2021-01-18 16:08:06

前一阵子,搭建了ELK日志分析平台,用着挺爽的,再也不用给开发拉各种日志,节省了很多时间。

这篇博文是介绍用python代码实现日志分析的,用MRJob实现hadoop上的mapreduce,可以直接放到hadoop集群上运行。

mrjob可以让我们使用Python编写MapReduce运算,并在多个不同平台运行,你可以:

  • 使用纯python编写multi-step MapReduce
  • 本机测试
  • 在hadoop集群上运行

安装mrjob

pip install mrjob

nginx访问日志格式

gamebbs.51.com 10.80.2.176 219.239.255.42 54220 [26/Dec/2016:04:34:39 +0800] "GET /forum.php?mod=ajax&action=forumchecknew&fid=752&time=1482697523&inajax=yes HTTP/1.0" 200 66 "http://gamebbs.51.com/forum.php?mod=forumdisplay&fid=752&page=1" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2626.106 Safari/537.36 Yunhai Browser" 0.016 0.011

日志格式分为下面几个部分:

server_name(域名): game.51.com
local_ip(本机内网IP):10.80.2.176
client_ip(客户端IP):219.239.255.42
remote_port(客户端建立连接端口):54220
time_local(请求时间):[26/Dec/2016:04:34:39 +0800]
method(请求方式):GET
request(请求url):/forum.php?mod=ajax&action=forumchecknew&fid=752&time=1482697523&inajax=yes HTTP/1.0
verb(http版本号):HTTP/1.0
status(状态码):200
body_bytes_sent:66
http_referer:http://gamebbs.51.com/forum.php?mod=forumdisplay&fid=752&page=1
http_user_agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2626.106 Safari/537.36 Yunhai Browser
request_time:0.016
upstream_response_time:0.011

处理nginx日志的类:

#!/usr/bin/env python
# coding=utf-8 import datetime
from urllib.parse import urlparse
from user_agents import parse as ua_parse class NginxLineParser(object): def parse(self, line):
""" 将 nginx 日志解析多个字段
"""
try:
line_item = line.strip().split('"')
self._server_name, self._local_ip, self._client_ip, self._remote_port = line_item[0].strip().split('[')[0].split()
self._time_local = line_item[0].strip().split('[')[-1].strip(']')
self._method, self._request, self._verb = line_item[1].strip().split()
self._status, self._body_bytes_sent = line_item[2].strip().split()
self._http_referer = line_item[3].strip()
self._http_user_agent = line_item[-2].strip()
self._request_time, self._upstream_response_time = line_item[-1].strip().split()
except:
with open('/tmp/parser_log_error.txt', 'a+') as f:
f.write(line + '\n') def logline_to_dict(self):
""" 将日志段转为字典
"""
line_field = {}
line_field['server_name'] = self.server_name
line_field['local_ip'] = self.local_ip
line_field['client_ip'] = self.client_ip
line_field['remote_port'] = self.remote_port
line_field['time_local'] = self.time_local
line_field['method'] = self.method
line_field['request'] = self.request
line_field['verb'] = self.verb
line_field['status'] = self.status
line_field['body_bytes_sent'] = self.body_bytes_sent
line_field['http_referer'] = self.http_referer
line_field['http_user_agent'] = self.http_user_agent
line_field['request_time'] = self.request_time
line_field['upstream_response_time'] = self.upstream_response_time return line_field @property
def server_name(self):
return self._server_name @property
def local_ip(self):
return self._local_ip @property
def client_ip(self):
return self._client_ip @property
def remote_port(self):
return self._remote_port @property
def time_local(self):
return datetime.datetime.strptime(self._time_local, '%d/%b/%Y:%H:%M:%S +0800') @property
def method(self):
return self._method @property
def request(self):
return urlparse(self._request).path @property
def verb(self):
return self._verb @property
def body_bytes_sent(self):
return self._body_bytes_sent @property
def http_referer(self):
return self._http_referer @property
def http_user_agent(self):
ua_agent = ua_parse(self._http_user_agent)
if not ua_agent.is_bot:
return ua_agent.browser.family @property
def user_agent_type(self):
us_agent = ua_parse(self._http_user_agent)
if us_agent.is_bot:
return us_agent.browser.family @property
def status(self):
return self._status @property
def request_time(self):
return self._request_time @property
def upstream_response_time(self):
return self._upstream_response_time def main():
"""程序执行入口
"""
ng_line_parser = NginxLineParser()
with open('test.log', 'r') as f:
for line in f:
ng_line_parser.parse(line) if __name__ == '__main__':
main()

该类主要有两个方法:

  1. parse:将日志行解析为几个字段
  2. logline_to_dict:将解析好的日志段转为字典类型