你的位置:首页 > Java教程

[Java教程]应用系统日志采集解决方案

概述



基于Flume + MongoDB,对现有的多个应用系统进行日志采集。

特点

  1. 采集范围
    每一次用户请求的请求信息。
  2. 数据量大
  3. 尽量减少现有系统的改动

数据流图





说明:
首先考虑的结构体系,是直接在应用系统中,将日志数据写到Flume;但是现有的应用系统都是非Maven的,需要在每一个应用系统中添加20+个jar包。为避免这种情况,抽出了一层日志服务,开放webservice服务给应用系统调用,最终形成上述的体系。

日志存储



1.需要解决的问题

1.1 借助Flume,写日志到MongoDB

参考:Flume学习应用:Java写日志数据到MongoDB
- 外网参考:Flume学习应用:Java写日志数据到MongoDB

1.2 发布webservice服务

参考:在web项目中发布jaxws
- 外网参考:在web项目中发布jaxws

2.日志服务实现

一个简单的web项目,对外发布一个webservice服务,实现写日志到Flume。

2.1 文件结构

src/main/java  |---- cn.sinobest.asj.log       |---- ISALog.java # 日志服务接口       |---- SALogImpl.java # 日志服务实现类  |---- cn.sinobest.asj.log.exception       |---- InvalidGradeException.java # 表示无效的日志等级       |---- InvalidFormatExceptioin.java # 表示无效的消息格式(要求是JSON格式字符串)  |---- cn.sinobest.asj.log.util       |---- ValidGrade.java # 枚举,所有有效的日志等级(DEBUG, INFO, WARN, ERROR)       |---- MessageTemplate.java # 消息模板src/main/resources  |---- log4j.propertiessrc/main/webapp  |---- WEB-INF       |---- sun-jaxws...jsp # 这个可以忽略pom.

 

2.2  文件内容


你可以直接从log-service拿到源代码,并跳过这一节的内容。
  1. pom.<project ="http://maven.apache.org/POM/4.0.0" ="http://www.w3.org/2001/ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>cn.sinobest.asj</groupId> <artifactId>log-service</artifactId> <packaging>war</packaging> <version>0.0.1-SNAPSHOT</version> <name>log-service Maven Webapp</name> <url>http://maven.apache.org</url> <dependencies> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>3.8.1</version> <scope>test</scope> </dependency> <dependency> <groupId>log4j</groupId> <artifactId>log4j</artifactId> <version>1.2.16</version> </dependency> <dependency> <groupId>commons-logging</groupId> <artifactId>commons-logging</artifactId> <version>1.1.1</version> </dependency> <!-- for log to Flume --> <dependency> <groupId>org.apache.flume.flume-ng-clients</groupId> <artifactId>flume-ng-log4jappender</artifactId> <version>1.6.0</version> </dependency> <!-- for jax-ws --> <dependency> <groupId>com.sun.</groupId> <artifactId>jaxws-rt</artifactId> <version>2.2.10</version> </dependency> <!-- for test the log content is a json-format or not --> <dependency> <groupId>org.mongodb</groupId> <artifactId>mongo-java-driver</artifactId> <version>2.13.0</version> </dependency> </dependencies> <build> <finalName>log-service</finalName> </build></project>

  • web.<??><web-app ="http://www.w3.org/2001/ ="http://java.sun.com/ ="http://java.sun.com/ xsi:schemaLocation="http://java.sun.com/ id="WebApp_ID" version="3.0" metadata-complete="false"> <display-name>Archetype Created Web Application</display-name></web-app>

    注意:如果是servlet3.0以下的版本,需要额外的配置。

  • log4j.properties
    # 配置Log4jAppender,能写日志到Flumelog4j.appender.flumeAvro=org.apache.flume.clients.log4jappender.Log4jAppenderlog4j.appender.flumeAvro.Hostname=localhostlog4j.appender.flumeAvro.Port=44444log4j.appender.flumeAvro.UnsafeMode=truelog4j.appender.flumeAvro.layout=org.apache.log4j.PatternLayoutlog4j.appender.flumeAvro.layout.ConversionPattern=%m# set root loggerlog4j.rootLogger=INFO, flumeAvro

  • ISALog.java
    package cn.sinobest.asj.log;import javax.jws.WebParam;import javax.jws.WebService;import cn.sinobest.asj.log.exception.InvalidFormatExceptioin;import cn.sinobest.asj.log.exception.InvalidGradeException;/** * SINOBEST ASJ Log - 为实现日志的统一采集和管理. * * @author lijinlong * */@WebServicepublic interface ISALog { /** * 日志记录. * * @param grade * 日志等级描述 - 忽略大小写. * @param content * 日志内容 - 需要为JSON格式的字符串. */ public void log(@WebParam(name = "grade") String grade, @WebParam(name = "content") String content) throws InvalidGradeException, InvalidFormatExceptioin;}

  • SALogImpl.java
    package cn.sinobest.asj.log;import javax.jws.WebService;import org.apache.commons.logging.Log;import org.apache.commons.logging.LogFactory;import cn.sinobest.asj.log.exception.InvalidFormatExceptioin;import cn.sinobest.asj.log.exception.InvalidGradeException;import cn.sinobest.asj.log.util.MessageTemplate;import cn.sinobest.asj.log.util.ValidGrade;import com.mongodb.util.JSON;@WebService(endpointInterface = "cn.sinobest.asj.log.ISALog")public class SALogImpl implements ISALog { static final Log log = LogFactory.getLog(SALogImpl.class); public void log(String grade, String content) throws InvalidGradeException, InvalidFormatExceptioin { checkGrade(grade); checkContent(content); ValidGrade vg = ValidGrade.valueOf(grade.toUpperCase()); log(vg, content); } /** * 根据日志等级,调用{@link log}的不同方法记录日志. * * @param vg * 日志等级 * @param content * 日志内容 */ private void log(ValidGrade vg, String content) { switch (vg) { case DEBUG: log.debug(content); break; case INFO: log.info(content); break; case WARN: log.warn(content); break; case ERROR: log.error(content); break; default: break; } } /** * 检查日志等级的有效性. * * @param grade * 日志等级描述. * @throws InvalidGradeException * 当日志等级无效时,抛出此异常. */ private void checkGrade(String grade) throws InvalidGradeException { boolean valid = ValidGrade.isValid(grade); if (!valid) { String message = String.format(MessageTemplate.INVALID_GRADE, grade, ValidGrade.getEnumContent()); throw new InvalidGradeException(message); } } /** * 检查日志内容格式的有效性.<br> * 要求为JSON格式的字符串. * * @param content * 日志内容. * @throws InvalidFormatExceptioin * 当日志内容格式无效时,抛出此异常. */ private void checkContent(String content) throws InvalidFormatExceptioin { boolean valid = true; if (content == null || content.isEmpty()) { valid = false; } else { try { JSON.parse(content); valid = true; } catch (com.mongodb.util.JSONParseException e) { valid = false; } } if (!valid) { String message = String.format(MessageTemplate.INVALID_FORMAT, content); throw new InvalidFormatExceptioin(message); } } /** * just for test. * * @param args */ public static void main(String[] args) { String[][] data = { { "info", "{'name':'ljl','age':26}" }, { "INFO", "trouble is a friend." }, { "JOKE", "{'message':'I am feeling down.'}" } }; ISALog ilog = new SALogImpl(); for (String[] dat : data) { String grade = dat[0]; String content = dat[1]; try { ilog.log(grade, content); } catch (Exception e) { e.printStackTrace(); } } }}

  • InvalidGradeException.java
    package cn.sinobest.asj.log.exception;/** * 表示无效的日志等级. * @author lijinlong * */public class InvalidGradeException extends Exception { private static final long serialVersionUID = 1341726127995938030L; public InvalidGradeException(String message) { super(message); }}

  • InvalidFormatExceptioin.java
    package cn.sinobest.asj.log.exception;/** * 表示无效的日志等级. * @author lijinlong * */public class InvalidGradeException extends Exception { private static final long serialVersionUID = 1341726127995938030L; public InvalidGradeException(String message) { super(message); }}

  • ValidGrade.java
    package cn.sinobest.asj.log.util;/** * 有效的日志等级. * * @author lijinlong * */public enum ValidGrade { DEBUG, INFO, WARN, ERROR; /** 有效日志等级的枚举内容 */ private static String enumContent; /** * 获取所有有效的日志等级. * * @return */ public static String getEnumContent() { if (enumContent != null && !enumContent.isEmpty()) return enumContent; ValidGrade[] vgs = ValidGrade.values(); StringBuilder builder = new StringBuilder(30); for (ValidGrade vg : vgs) { builder.append(vg).append(","); } builder.delete(builder.length() - 1, builder.length()); enumContent = builder.toString(); return enumContent; } /** * 判断日志等级是否有效. * @param grade 日志等级 - 忽略大小写. * @return */ public static boolean isValid(String grade) { if (grade == null || grade.isEmpty()) return false; boolean result = false; final String GRADE = grade.toUpperCase(); ValidGrade[] vgs = ValidGrade.values(); for (ValidGrade vg : vgs) { if (vg.toString().equals(GRADE)) { result = true; break; } } return result; } /** * just for test. * @param args */ public static void main(String[] args) { String content = getEnumContent(); System.out.println(content); String[] testGrade = {"DEBUG", "INFO", "WARN", "ERROR", "TEST"}; for (String tg : testGrade) { if (!ValidGrade.isValid(tg)) { String message = String.format("%s is invalid.", tg); System.out.println(message); } } }}

  • MessageTemplate.java
    package cn.sinobest.asj.log.util;/** * 消息模板. * @author lijinlong * */public class MessageTemplate { /** 无效的消息等级 */ public static final String INVALID_GRADE = "无效的日志等级[%s]。服务支持的日志等级有:%s。"; /** 无效的消息内容格式 */ public static final String INVALID_FORMAT = "无效的日志内容格式:\n%s\n,请检查是否为JSON格式的字符串。";}

  • sun-jaxws.<??><endpoints ="http://java.sun.com/ version="2.0"> <endpoint name="defaultLog" implementation="cn.sinobest.asj.log.SALogImpl" url-pattern="/log.action" /></endpoints>
  • 应用系统群




    1.需要考虑的问题

    1.1 拦截

    使用Filter可以实现拦截。

    1.2 组织日志内容

    视需求而定,当前仅对request中的部分信息进行了采集。

    1.3 格式化

    日志信息需要格式化为JSON字符串,才能正确的写到MongoDB。

    1.4 请求webservice服务

    参考:基于wsimport生成代码的客户端
    - 外网参考:基于wsimport生成代码的客户端

    2. demo

    2.1 文件结构图

    src |---- cn.sinobest.asj.log      |---- LogFilter.java |---- cn.sinobest.asj.log.wsimport # 存放wsimport生成的代码      # 省略basic |---- WEB-INF      |---- web.

    2.2 文件内容

    1. LogFilter.java
      package cn.sinobest.asj.log;import java.io.IOException;import java.net.MalformedURLException;import java.net.URL;import java.util.Date;import java.util.HashMap;import java.util.Map;import javax.servlet.Filter;import javax.servlet.FilterChain;import javax.servlet.FilterConfig;import javax.servlet.ServletException;import javax.servlet.ServletRequest;import javax.servlet.ServletResponse;import javax.servlet.http.HttpServletRequest;import org.apache.commons.logging.Log;import org.apache.commons.logging.LogFactory;import org.json.JSONObject;import cn.sinobest.asj.log.wsimport.ISALog;import cn.sinobest.asj.log.wsimport.InvalidFormatExceptioin_Exception;import cn.sinobest.asj.log.wsimport.InvalidGradeException_Exception;import cn.sinobest.asj.log.wsimport.SALogImplService;public class LogFilter implements Filter { static final Log log = LogFactory.getLog(LogFilter.class); static final String WSDL_LOCATION = "http://localhost:8080/logserv/log.action?wsdl"; @Override public void destroy() { } @Override public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException { try { log(request); } catch (InvalidFormatExceptioin_Exception e) { e.printStackTrace(); } catch (InvalidGradeException_Exception e) { e.printStackTrace(); } finally { chain.doFilter(request, response); } } private void log(ServletRequest request) throws MalformedURLException, InvalidFormatExceptioin_Exception, InvalidGradeException_Exception { Map<String, Object> data = new HashMap<String, Object>(); data.put("appid", "zfba"); data.put("time", new Date()); data.put("localAddr", request.getLocalAddr()); data.put("localName", request.getLocalName()); data.put("localPort", request.getLocalPort()); data.put("remoteAddr", request.getRemoteAddr()); data.put("remoteHost", request.getRemoteHost()); data.put("remotePort", request.getRemotePort()); // data.put("serverName", request.getServerName()); // data.put("serverPort", request.getServerPort()); HttpServletRequest hrequest = (HttpServletRequest) request; data.put("pathInfo", hrequest.getPathInfo()); data.put("pathTranslated", hrequest.getPathTranslated()); data.put("remoteUser", hrequest.getRemoteUser()); data.put("requestURI", hrequest.getRequestURI()); data.put("requestURL", hrequest.getRequestURL()); data.put("servletPath", hrequest.getServletPath()); JSONObject cont = new JSONObject(data); URL url = new URL(WSDL_LOCATION); SALogImplService ss = new SALogImplService(url); ISALog service = ss.getSALogImplPort(); service.log("info", cont.toString()); } @Override public void init(FilterConfig arg0) throws ServletException { }}

    2. web.这里仅贴出新增的内容:
       <!-- 测试日志 --> <filter> <filter-name>log-filter</filter-name> <filter-class>cn.sinobest.asj.log.LogFilter</filter-class> </filter> <!-- 测试日志 --> <filter-mapping> <filter-name>log-filter</filter-name> <url-pattern>/*</url-pattern> </filter-mapping>

    测试




    1. 启动MongoDB
      参考《Flume学习应用:Java写日志数据到MongoDB》
    2. 配置并启动Flume
      参考《Flume学习应用:Java写日志数据到MongoDB》
    3. 启动日志服务
      参考《在web项目中发布jaxws》
    4. 启动应用系统,并进行访问
    5. 查看MongoDB数据库
      参考《Flume学习应用:Java写日志数据到MongoDB》

    附录




    相关文章

    1. Flume学习应用:Java写日志数据到MongoDB
      博客园:Flume学习应用:Java写日志数据到MongoDB
    2. 在web项目中发布jaxws
      博客园:在web项目中发布jaxws
    3. 基于wsimport生成代码的客户端
      博客园:基于wsimport生成代码的客户端