你的位置:首页 > 操作系统

[操作系统]nginx中使用srcache_nginx模块构建缓存


openresty项目中将lua嵌入到nginx,让nginx执行lua脚本,高并发,非阻塞的处理各种请求,我们可以使用nignx可以直接构建 srcache_nginx + redis 缓存,而不用通过动态语言来处理(也就是说QPS可以轻松的提高了)

srcache-nginx-module的一张图

好了废话不多说,性能自然不用说了

一、安装

pcre

cd /usr/local/srcwget -c ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-8.38.tar.gztar zxf pcre-8.38.tar.gz

drizzle7

cd /usr/local/src/ wget http://openresty.org/download/drizzle7-2011.07.21.tar.gz tar xzvf drizzle-2011.07.21.tar.gz cd drizzle-2011.07.21/ ./configuremakemake installexport LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH

JIT(Just-In-Time Compiler)

wget -c http://luajit.org/download/LuaJIT-2.0.2.tar.gztar xzvf LuaJIT-2.0.2.tar.gzcd LuaJIT-2.0.2make install PREFIX=/usr/local/luajitecho "/usr/local/luajit/lib" > /etc/ld.so.conf.d/usr_local_luajit_lib.confldconfigexport LUAJIT_LIB=/usr/local/luajit/libexport LUAJIT_INC=/usr/local/luajit/include/luajit-2.0

nginx

cd /usr/local/srcwget -c http://nginx.org/download/nginx-1.9.9.tar.gzgit clone https://github.com/simpl/ngx_devel_kit.gitgit clone https://github.com/openresty/set-misc-nginx-module.gitgit clone https://github.com/openresty/memc-nginx-module.gitgit clone https://github.com/openresty/echo-nginx-module.gitgit clone https://github.com/openresty/lua-nginx-module.gitgit clone https://github.com/openresty/srcache-nginx-module.gitgit clone https://github.com/openresty/drizzle-nginx-module.gitgit clone https://github.com/openresty/rds-json-nginx-module.gitwget http://people.freebsd.org/~osa/ngx_http_redis-0.3.7.tar.gztar zxf nginx-1.9.9.tar.gzcd nginx-1.9.9./configure \--prefix=/usr/local/nginx-1.9.9 \--add-module=../memc-nginx-module \--add-module=../srcache-nginx-module \--add-module=../ngx_devel_kit \--add-module=../ngx_image_thumb \--add-module=../redis2-nginx-module \--add-module=../echo-nginx-module \--add-module=../lua-nginx-module \--add-module=../set-misc-nginx-module \--add-module=../ngx_http_redis-0.3.7 \--with-pcre=../pcre-8.38 \--with-pcre-jitmake && make install

redis

cd /usr/localwget http://download.redis.io/releases/redis-3.0.6.tar.gztar zxf redis-3.0.6.tar.gz cd redis-3.0.6make./src/redis-server &

配置文件

daemonize yespidfile /var/run/redis-6379.pidport 6379bind 127.0.0.1timeout 0tcp-keepalive 0loglevel noticelogfile stdoutdatabases 16stop-writes-on-bgsave-error yesrdbcompression yesrdbchecksum yesdbfilename dump.rdbslave-serve-stale-data yesslave-read-only yesrepl-disable-tcp-nodelay noslave-priority 100maxmemory 8096mb  maxmemory-policy volatile-ttlappendonly noappendfsync everysecno-appendfsync-on-rewrite noauto-aof-rewrite-percentage 100auto-aof-rewrite-min-size 64mblua-time-limit 5000slowlog-log-slower-than 10000slowlog-max-len 128hash-max-ziplist-entries 512hash-max-ziplist-value 64list-max-ziplist-entries 512list-max-ziplist-value 64set-max-intset-entries 512zset-max-ziplist-entries 128zset-max-ziplist-value 64activerehashing yesclient-output-buffer-limit normal 0 0 0client-output-buffer-limit slave 256mb 64mb 60client-output-buffer-limit pubsub 32mb 8mb 60hz 10aof-rewrite-incremental-fsync yes

nginx简单配置

user www www;worker_processes auto;error_log logs/error.log info;pid    logs/nginx.pid;events {  use epoll;  worker_connections 65536;}http {  include    mime.types;  default_type application/octet-stream;  charset utf-8;  log_format main '$remote_addr - $remote_user [$time_local] "$request" '    '$status $body_bytes_sent "$http_referer" '    '"$http_user_agent" "$http_x_forwarded_for"';  log_format srcache_log '$remote_addr - $remote_user [$time_local] "$request" '    '"$status" $body_bytes_sent $request_time $bytes_sent $request_length '    '[$upstream_response_time] [$srcache_fetch_status] [$srcache_store_status] [$srcache_expire]';    server_tokens off;  keepalive_timeout      60 20;  client_header_timeout    3m;  client_body_timeout     3m;  send_timeout        3m;  client_header_buffer_size      16k;  large_client_header_buffers     4 32k;  server_names_hash_max_size     512;  server_names_hash_bucket_size    64;  sendfile    on;  tcp_nopush   on;  tcp_nodelay   on;  gzip on;  gzip_min_length 1k;  gzip_buffers   4 16k;  gzip_http_version 1.0;  gzip_comp_level 2;  gzip_types    text/plain application/x-javascript text/css application/

二、srcache+memcache

server {  listen    8099;  server_name 192.168.1.30;  root /data/www;  index index.php index.html index.htm;  default_type text/plain;  access_log logs/host.access.log main;  location /hello{    echo "This is a test";  }  location = /lua-version {    content_by_lua '      if jit then        ngx.say(jit.version)      else        ngx.say(_VERSION)          end      ';  }  location /memc {    internal;    memc_connect_timeout 100ms;    memc_send_timeout 100ms;    memc_read_timeout 100ms;    set $memc_key $query_string;    set $memc_exptime 300;    memc_pass memcache;  }  location ~ \.php$ {    charset    utf-8;    default_type  text/html;    set $key $uri$args;    srcache_fetch GET /memc $key;    srcache_store PUT /memc $key;    add_header X-Cached-From $srcache_fetch_status;     add_header X-Cached-Store $srcache_store_status;    fastcgi_pass  127.0.0.1:9000;    fastcgi_index index.php;    include    fastcgi_params;    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;  }  error_page  500 502 503 504 /50x.html;  location = /50x.html {    root  html;  }}

第一次访问的时候

Connection:keep-aliveContent-Encoding:gzipContent-Type:text/html; charset=UTF-8Date:Wed, 20 Jan 2016 16:32:32 GMTKeep-Alive:timeout=20Server:nginxTransfer-Encoding:chunkedVary:Accept-EncodingX-Cached-From:MISSX-Cached-Store:STORE

第二次访问

Connection:keep-aliveContent-Encoding:gzipContent-Type:text/html; charset=UTF-8Date:Wed, 20 Jan 2016 16:33:17 GMTKeep-Alive:timeout=20Server:nginxTransfer-Encoding:chunkedVary:Accept-EncodingX-Cached-From:HITX-Cached-Store:BYPASS

可以自定义哪些需要访问

三、srcache+redis

redis配置测试

server {  listen    9001;  server_name 192.168.1.30;  root /data/www;  index index.php index.html index.htm;  default_type text/plain;  access_log logs/host.access.log main;  location /testx{    echo '1';  }  location ~ .*\.php {    srcache_store_private on;    srcache_methods GET;    srcache_response_cache_control off;    set $key $uri;    set_escape_uri $escaped_key $key;    srcache_default_expire 172800;    srcache_fetch GET /redis_get $key;    srcache_store PUT /redis_set key=$escaped_key&exptime=$srcache_expire;    add_header X-Cached-From $srcache_fetch_status;    set_md5 $md5key $key;    add_header X-md5-key $md5key;    add_header X-Cached-Store $srcache_store_status;    add_header X-Key $key;    add_header X-Query_String $query_string;    add_header X-expire $srcache_expire;    add_header X-uri $uri;    access_log logs/9001-access.log srcache_log;    include fastcgi_params;    fastcgi_pass 127.0.0.1:9000;    fastcgi_index index.php;    fastcgi_connect_timeout 60;    fastcgi_send_timeout 180;    fastcgi_read_timeout 180;    fastcgi_buffer_size 128k;    fastcgi_buffers 4 256k;    fastcgi_busy_buffers_size 256k;    fastcgi_temp_file_write_size 256k;    fastcgi_intercept_errors on;    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;    fastcgi_split_path_info ^(.+\.php)(.*)$;    fastcgi_param PATH_INFO $fastcgi_path_info;  }  location = /redis_get {    internal;    set_md5 $redis_key $args;    redis_pass redis;  }  location =/show{    echo $request_uri;     echo $args;  }  location = /redis_set {    internal;    set_unescape_uri $exptime $arg_exptime;    set_unescape_uri $key $arg_key;    set_md5 $key;    redis2_query set $key $echo_request_body;    redis2_query expire $key $exptime;    redis2_pass redis;  }  location = /one {    set $value 'first';    redis2_query set one $value;    redis2_pass redis;  }  location = /get {    set_unescape_uri $key $arg_key; # this requires ngx_set_misc    redis2_query get $key;    redis2_pass redis;  }  error_page  500 502 503 504 /50x.html;  location = /50x.html {    root  html;  }}

可以查看日志缓存是否命中

四、lua

Nginx下Lua处理阶段与使用范围

init_by_lua      httpset_by_lua       server, server if, location, location ifrewrite_by_lua     http, server, location, location ifaccess_by_lua     http, server, location, location ifcontent_by_lua     location, location ifheader_filter_by_lua  http, server, location, location ifbody_filter_by_lua   http, server, location, location iflog_by_lua       http, server, location, location iftimer

lua代码

ngx.req.read_body() -- explicitly read the req bodylocal data = ngx.req.get_body_data()if data then  ngx.say("body data:")  ngx.print(data)  returnend-- body may get buffered in a temp file:local file = ngx.req.get_body_file()if file then  ngx.say("body is in file ", file)else  ngx.say("no body found")endlocal res = ngx.location.capture("/foo/index.php")if res then  ngx.say("status: ", res.status)  ngx.say("body:")  ngx.print(res.body)end

nginx中配置

location /lua_test {    content_by_lua_file conf/lua_test.lua;                                           } 

 

 

参考文章

https://github.com/openresty/lua-nginx-module

http://chenxiaoyu.org/2011/10/30/nginx-modules.html

http://www.ttlsa.com/nginx/nginx-lua-redis/