Varnish 3安装部署及使用

时间：2021-12-15 00:05:19

Varnish安装

  • 源码安装Varnish

    安装pcre:

    # wget -c http://downloads.sourceforge.net/project/pcre/pcre/8.32/pcre-8.32.tar.bz2
    # tar jxvf pcre-8.32.tar.bz2
    # cd pcre-8.32
    # ./configure && make && make install

    源码安装Varnish:

    # wget -c http://repo.varnish-cache.org/source/varnish-3.0.3.tar.gz
    # tar zxvf varnish-3.0.3.tar.gz
    # cd varnish-3.0.3
    # export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
    # ./configure --prefix=/usr/local/varnish
    # make && make install

Varnish配置与管理

  • varnish启动配置
    # cp /data/install/varnish-3.0.3/redhat/varnish.sysconfig /etc/sysconfig/varnish
    # cp /data/install/varnish-3.0.3/redhat/varnish.initrc /etc/init.d/varnish
    # cp /data/install/varnish-3.0.3/redhat/varnish_reload_vcl /usr/local/varnish/bin/
    # groupadd varnish
    # useradd -g varnish -s /sbin/nologin varnish

    编辑varnish启动配置:

    # vi /etc/sysconfig/varnish
    # cat /etc/sysconfig/varnish
    ## Alternative 3, Advanced configuration
    #
    # See varnishd(1) for more information.
    #
    # # Main configuration file. You probably want to change it
    #VARNISH_VCL_CONF=/etc/varnish/default.vcl
    VARNISH_VCL_CONF=/usr/local/varnish/etc/varnish/fdfs.vcl

    # # Default address and port to bind to
    # # Blank address means all IPv4 and IPv6 interfaces, otherwise specify
    # # a host name, an IPv4 dotted quad, or an IPv6 address in brackets.
    # VARNISH_LISTEN_ADDRESS=
    #VARNISH_LISTEN_PORT=6081
    VARNISH_LISTEN_PORT=80
    #
    # # Telnet admin interface listen address and port
    VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1
    VARNISH_ADMIN_LISTEN_PORT=6082
    #
    # # Shared secret file for admin interface
    #VARNISH_SECRET_FILE=/etc/varnish/secret
    VARNISH_SECRET_FILE=/usr/local/varnish/etc/varnish/secret
    #
    # # The minimum number of worker threads to start
    VARNISH_MIN_THREADS=50
    #
    # # The Maximum number of worker threads to start
    VARNISH_MAX_THREADS=1000
    #
    # # Idle timeout for worker threads
    VARNISH_THREAD_TIMEOUT=120
    #
    # # Cache file location
    VARNISH_STORAGE_FILE=/var/lib/varnish/varnish_storage.bin
    #
    # # Cache file size: in bytes, optionally using k / M / G / T suffix,
    # # or in percentage of available disk space using the % suffix.
    VARNISH_STORAGE_SIZE=150M
    #
    # # Backend storage specification
    #VARNISH_STORAGE="file,${VARNISH_STORAGE_FILE},${VARNISH_STORAGE_SIZE}"
    VARNISH_STORAGE="malloc,${VARNISH_STORAGE_SIZE}"
    #
    # # Default TTL used when the backend does not specify one
    VARNISH_TTL=120
    #
    # # DAEMON_OPTS is used by the init script.  If you add or remove options, make
    # # sure you update this section, too.
    DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
                 -f ${VARNISH_VCL_CONF} \
                 -T ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \
                 -t ${VARNISH_TTL} \
                 -w ${VARNISH_MIN_THREADS},${VARNISH_MAX_THREADS},${VARNISH_THREAD_TIMEOUT} \
                 -u varnish -g varnish \
                 -S ${VARNISH_SECRET_FILE} \
                 -s ${VARNISH_STORAGE}"
    #

    ## Alternative 4, Do It Yourself. See varnishd(1) for more information.
    #
    # DAEMON_OPTS=""

    编辑varnish启动管理脚本:

    # vi /etc/init.d/varnish
    # cat /etc/init.d/varnish
    ...
    pidfile=/var/run/varnish.pid
    exec="/usr/local/varnish/sbin/varnishd"
    reload_exec="/usr/local/varnish/bin/varnish_reload_vcl"
    ...

    编辑动态加载VCL配置的脚本:

    # vi /usr/local/varnish/bin/varnish_reload_vcl
    # cat /usr/local/varnish/bin/varnish_reload_vcl
    ...
    # Done parsing, set up command
    VARNISHADM="/usr/local/varnish/bin/varnishadm $secret -T $VARNISH_ADMIN_LISTEN_ADDRESS:$VARNISH_ADMIN_LISTEN_PORT"
    ...

    注:也可以将/usr/local/varnish/bin添加到系统的PATH中,这样就不需要编辑varnish_reload_vcl

    生成varnish管理秘钥:

    # /usr/bin/uuidgen > /usr/local/varnish/etc/varnish/secret
    # chmod 644 /usr/local/varnish/etc/varnish/secret
  • VCL配置

    FastDFS部署的前端Varnish缓存的vcl配置列子:

    # cat /usr/local/varnish/etc/varnish/fdfs.vcl
    # backend health checking
    probe health {
        .url = "/favicon.ico";
        .timeout = 60ms;
        .interval = 2s;
        .window = 5;
        .threshold = 3;
    }

    backend server00 {
         .host = "192.168.1.190";
         .port = "8080";
         .probe = health;
    }

    backend server01 {
         .host = "192.168.1.191";
         .port = "80";
         .probe = health;
    }

    backend server02 {
        .host = "192.168.1.192";
        .port = "80";
        .probe = health;
    }

    # round-robin loadblancing
    director dr1 round-robin {
         { .backend = server01; }
         { .backend = server02; }
    }

    # Access Control
    acl internal {
       "localhost";
       "192.168.1.0"/24;
       ! "192.168.1.1";
    }

    sub vcl_recv {
        # Backend selection
        if (req.url ~ "^/M00") {
            set req.backend = dr1;
        } else {
            set req.backend = server00;
        }
        # purge request
        if (req.request == "PURGE") {
            if (!client.ip ~ internal) {
                error 405 "Not allowed.";
            }
            return (lookup);
        }

        # Use anonymous, cached pages if all backends are down
        if (!req.backend.healthy) {
            unset req.http.Cookie;
        }
        /* Bypass cache for large files.  The x-pipe header is
             set in vcl_fetch when a too large file is detected.*/
        if (req.http.x-pipe && req.restarts > 0) {
            unset req.http.x-pipe;
            return (pipe);
        }
        # Do not cache these paths
        if (req.url ~ "^/admin$" || req.url ~ "^/admin/.*$") {
            return (pass);
        }
        # Allow the backend to serve up stale content if it is responding slowly
        if (req.backend.healthy) {
            set req.grace = 30s;
        } else {
            set req.grace = 6h;
        }

        # Always cache the following file types
        if (req.url ~ "(?i)\.(pdf|asc|dat|txt|doc|xls|ppt|tgz|csv|png|gif|jpeg|jpg|ico|swf|css|js)(\?.*)?$") {
            unset req.http.Cookie;
        }
    }

    sub vcl_hit {
        if (req.request == "PURGE") {
            purge;
            error 200 "Purged.";
        }
    }

    sub vcl_miss {
        if (req.request == "PURGE") {
            purge;
            error 404 "Not in cache.";
        }
    }

    sub vcl_pass {
        if (req.request == "PURGE") {
            error 502 "PURGE on a passed object";
        }
    }

    # Set a header to track a cache HIT/MISS.
    sub vcl_deliver {
        if (obj.hits > 0) {
            set resp.http.X-Varnish-Cache = "HIT";
        }
        else {
            set resp.http.X-Varnish-Cache = "MISS";
        }
    }

    sub vcl_fetch {
        # Don't cache files larger than 10MB
        if (beresp.http.Content-Length ~ "[0-9]{8,}") {
            set req.http.x-pipe = "1";
            return (restart);
        }
        #  override the default time to live of a cached object
        if ( req.url ~ "(?i)\.(png|jpg|jpeg|gif)$") {
            set beresp.ttl = 30d;
            unset beresp.http.Set-Cookie;
        } else {
            set beresp.ttl = 30s;
        }
        # Allow items to be stale if needed
        set beresp.grace = 6h;
    }
  • 启动/停止varnish

    检查VCL配置是否正确:

    # /etc/init.d/varnish configtest

    或

    # /usr/local/varnish/sbin/varnishd -C -f /usr/local/varnish/etc/varnish/fdfs.vcl

    启动varnish:

    # service varnish start

    查看varnish状态:

    # /etc/init.d/varnish status

    动态加载VCL配置:

    # /etc/init.d/varnish reload

    停止varnish:

    # /etc/init.d/varnish stop

    查看当前varnish监听的80端口:

    # netstat -plan | grep :80
    tcp        0      0 0.0.0.0:80                  0.0.0.0:*                   LISTEN      3068/varnishd       
    tcp        1      0 192.168.1.190:44391         192.168.1.191:80            CLOSE_WAIT  3068/varnishd       
    tcp        0      0 :::80                       :::*                        LISTEN      3068/varnishd

    查看varnish进程:

    # ps -ef | grep varnishd | grep -v grep
    root      3067     1  0 15:06 ?        00:00:00 /usr/local/varnish/sbin/varnishd -P /var/run/varnish.pid -a :80 -f /usr/local/varnish/etc/varnish/fdfs.vcl -T 127.0.0.1:6082 -t 120 -w 50,1000,120 -u varnish -g varnish -S /usr/local/varnish/etc/varnish/secret -s malloc,150M
    varnish   3068  3067  0 15:06 ?        00:00:03 /usr/local/varnish/sbin/varnishd -P /var/run/varnish.pid -a :80 -f /usr/local/varnish/etc/varnish/fdfs.vcl -T 127.0.0.1:6082 -t 120 -w 50,1000,120 -u varnish -g varnish -S /usr/local/varnish/etc/varnish/secret -s malloc,150M
  • Varnish访问日志

    varnishncsa可以使用NCSA通用日志格式(NCSA Common Log Format)将HTTP请求记录到日志文件.

    # cp /data/install/varnish-3.0.3/redhat/varnishncsa.initrc /etc/init.d/varnishncsa
    # chmod +x /etc/init.d/varnishncsa
    # mkdir -p /usr/local/varnish/logs

    编辑varnishncsa启动配置:

    # vi /etc/init.d/varnishncsa
    # cat /etc/init.d/varnishncsa
    ...
    logfile="/usr/local/varnish/logs/varnishncsa.log"
    exec="/usr/local/varnish/bin/varnishncsa"
    ...

    启动varnishncsa：

    # /etc/init.d/varnishncsa start

    使用logrotate轮询日志文件(每天轮询,保留最近30天):

    # vi /etc/logrotate.d/varnish
    # cat /etc/logrotate.d/varnish
    /usr/local/varnish/logs/varnishncsa.log {
        missingok
        notifempty
        sharedscripts
        daily
        rotate 30
        dateext
        dateformat .%Y-%d-%m
        delaycompress
        postrotate
        /bin/kill -HUP `cat /var/run/varnishncsa.pid 2>/dev/null` 2> /dev/null || true
        endscript
    }

    日志轮询debug测试:

    # logrotate -df /etc/logrotate.d/varnish
  • 配置开机自动启动
    # chkconfig --add varnish
    # chkconfig varnish on
    # chkconfig --add varnishncsa
    # chkconfig varnishncsa on
  • Varnish辅助工具
    • varnishtop 显示客户端最频繁请求的URL的持续更新列表:
      # varnishtop -i RxURL

      显示最频繁命中后端的URL的持续更新列表:

      # varnishtop -b -i TxURL
    • varnishstat
      varnishstat实时查看缓存统计信息(如:连接数和命中率):

      # varnishstat

      仅查看特定项统计信息:

      # varnishstat -f client_conn,client_drop,client_req,cache_hit,cache_hitpass,cache_miss,backend_conn,backend_fail,backend_reuse,n_lru_nuked,n_wrk_lqueue,n_wrk_queued,n_wrk_drop
    • varnishlog
      Debug缓存操作(ReqStart:后面为某客户端请求的IP):

      # varnishlog -c -m "ReqStart:192.168.1.120"

      仅显示发送到后端的URLs(如:没命中缓存和内容还没被缓存):

      # varnishlog -O -i TxURL

      将varnishlog写入到文件,分析访问特定页面的客户端请求:

      # varnishlog -w /usr/local/varnish/logs/varnish.log -d
      # varnishlog -r /usr/local/varnish/logs/varnish.log -c -m 'RxURL:^/test/c.php$'
    • varnishadm
      查看varnishadm提供的命令:

      # varnishadm -T 127.0.0.1:6082 -S /usr/local/varnish/etc/varnish/secret help
      help [command]
      ping [timestamp]
      auth response
      quit
      banner
      status
      start
      stop
      vcl.load
      vcl.inline <quoted_VCLstring>
      vcl.use
      vcl.discard
      vcl.list
      vcl.show
      param.show [-l] []
      param.set 
      panic.show
      panic.clear
      storage.list
      backend.list
      backend.set_health matcher state
      ban.url 
      ban    [&&   ]...
      ban.list

      查看后端服务器健康状况:

      # varnishadm debug.health
      Backend server01 is Healthy
      Current states  good:  4 threshold:  3 window:  5
      Average responsetime of good probes: 0.001339
      Oldest                                                    Newest
      ================================================================
      4444444444444444444444444444444444444444444444444444444444444444 Good IPv4
      XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Good Xmit
      RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR-RRRR Good Recv
      HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH-HHHH Happy
      Backend server02 is Healthy
      Current states  good:  5 threshold:  3 window:  5
      Average responsetime of good probes: 0.001387
      Oldest                                                    Newest
      ================================================================
      4444444444444444444444444444444444444444444444444444444444444444 Good IPv4
      XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Good Xmit
      RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR-RRRRRRRRRRRR Good Recv
      HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH-HHHHHHHHHHHH Happy
      Backend server00 is Healthy
      Current states  good:  5 threshold:  3 window:  5
      Average responsetime of good probes: 0.000331
      Oldest                                                    Newest
      ================================================================
      4444444444444444444444444444444444444444444444444444444444444444 Good IPv4
      XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Good Xmit
      RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR Good Recv
      HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH Happy

      或

      # varnishadm backend.list
      Backend name                   Refs   Admin      Probe
      server01(192.168.1.191,,80)    3      probe      Healthy 5/5
      server02(192.168.1.192,,80)    3      probe      Healthy 5/5
      server00(192.168.1.190,,8080)  2      probe      Healthy 5/5

Varnish进阶使用

  • VCL流程简介

    VCL是用于定义Varnish缓存策略的配置语言(VCL参考手册).VCL配置大部分是写到子程序,并遵循一个预先定义的请求和响应阶段的流程.

    Varnish 3安装部署及使用

    处理HTTP请求的子程序(subroutine):

    • vcl_recv:当有HTTP请求进来的时候,vcl_recv是第一个被执行的子程序,它决定是否处理/如何处理请求.在这里可正常化URL,修改HTTP头,调整/删除cookies,定义使用哪个后端backend或director,控制访问...
    • vcl_hash:它是vcl_recv子程序执行之后才执行的一个子程序.它的作用是生成hash作为存储对象在内存中映射的key,对获得高命中率发挥重要作用.
    • vcl_pipe:当一个pipe指令在vcl_recv执行的之后(接下来的VCL code将不被进行处理),它就会将客户端请求直接传输给后端服务器.客户端和后端服务器之间直接来回传输数据,直至有一方关闭连接.
    • vcl_pass:它会将客户端请求传输给后端服务器,后端服务器的响应数直接传回给客户端,但不进行缓存.

    处理HTTP响应的子程序:

    • vcl_fetch:它是第一个处理响应阶段的子程序,对缓存策略和ESI发挥重要作用.它是在从后端服务器成功获得内容之后被调用.
    • vcl_deliver:它是在缓存对象发送给客户端之前调用.可以用于隐藏服务器信息,添加debug头部信息,最后清理http响应头部信息等.由于vcl_deliver是在对象进入缓存之后才调用的,所有在vcl_deliver内的操作都不会被持久化.
    • vcl_error:当遇到错误的时候调用,显式地或隐式地由于后端或是内部的错误.也可以用于拒绝请求访问或将请求导向新的地址,或是用于系统维护提示页面.

    建议在写自己的VCL代码之前,先阅读Varnish默认内嵌的VCL配置文件default.vcl:

    # cat /usr/local/varnish/etc/varnish/default.vcl
  • Varnish负载均衡

    Varnish负载均衡的director有random,client,hash,round-robin,dns,fallback等.
    下面是两种最常用的director的负载均衡.

    director使用round-robin的VCL:

    ...
    director dr1 round-robin {
         { .backend = server01; }
         { .backend = server02; }
    }
    sub vcl_recv {
       if (req.http.host ~ "(?i)^images.zrwm.com$"){
            set req.backend = dr1;
        }
    }
    ...

    director使用client的VCL:

    ...
    director dr2 client {
        .retries = 2;
        {
            .backend = server01;
            .weight = 1;
        }                       
        {
            .backend = server02;
            .weight = 1;
        }
    }

    if (req.http.host ~ "(?i)^images.zrwm.com$"){
            set req.backend = dr2;
            /* Load balance by client IP, this is the default */
            set client.identity = client.ip;

            /*
            ## Load balance by clients' real IP if other services in front of Varnish Cache
            set client.identity = req.http.X-Forwarded-For;
            ## Load balance by URL
            set client.identity = req.url;
            ## Load balance by user agent
            set client.identity = req.http.user-agent;
            */
        }
    ...
  • HTTP缓存头部

    HTTP1.1引入了cache-control响应头部来替换HTTP 1.0的expires头部.两者的主要区别在于expires是使用一个日期时间值,而cache-control可以接收一个age值.

    Expires: Fri, 1 Oct 2012 14:19:41 GMT
    Cache-Control: max-age=3600

    一个使用PHP处理HTTP缓存(缓存10秒)的例子:

    <?php 
    header('Cache-Control: public, must-revalidate, max-age=10, s-maxage=10');
    header('Last-Modified: ' . gmdate("D, d M Y H:i:s") . ' GMT');
    header('Expires: ' . gmdate ('D, d M Y H:i:s', time() + 10). ' GMT');

    echo "content from php";
    ?>

    Varnish会遵守这些缓存头部(cache headers)的值,除非Varnish被告知不需要遵守.因此,当内容应该被缓存而实际没缓存的时候,可检查响应头部(response headers)是否有no-store/no-cache或是一个已经过期的时间值.

  • 清除Varnish缓存内容

    清除Varnish缓存内容有三种方式:

    • Purge:每次从缓存中清除一个对象(清除所有Vary:-对象版本),需要在vcl_hit和vcl_miss两个子程序中实现.
    • Ban:可以通过正则表达式一次清除大量的内容.因此清除大量的缓存内容的时候,Ban方法会比Purge快很多.
    • 使缓存不命中:在vcl_recv子程序中set req.hash_always_miss
      = true;      这样会使Varnish每次都去缓存中寻找对象,但会忽略任何发现的对象副本.

    注意:Purge方式会马上将内容从缓存中删除,并释放内存.Ban方式不会马上删除内容,也不释放内存,而是创建一个ban列表,每次请求进来的时候都会检查这个列表.

    Purge只能在VCL代码中实现,参考上面的fdfs.vcl.
    下面是使用Purge方式清理缓存的一个PHP脚本:

    <?php
    function purge($ip, $url) {
        $fp = fsockopen($ip, 80, $errno, $errstr, 2);
        if (!$fp) {
            echo "$errstr ($errno)", PHP_EOL;
        } else {
            $out  = "PURGE $url HTTP/1.0\r\n";
            $out .= "HOST:www.zrwm.com\r\n";
            $out .= "Connection: Close\r\n\r\n";
            fwrite($fp, $out);
            while (!feof($fp)) {
               echo fgets($fp, 128);
            }
            fclose($fp);
        }
    }

    purge('127.0.0.1', '/M00/00/00/wKgBv1FSXxGADRiEAAO1XLFK2Tw732.jpg');

    Ban方式既可以在VCL代码vcl_recv子程序中实现,也可以通过CLI方式的varnishadm管理客户端来操作.
    下面是通过varnishadm使用ban方式清除缓存的几个例子:
    清除单个文件c.php的缓存:

    # varnishadm ban.url c.php

    # varnishadm ban "req.http.host ~ a.qingluobo.com && req.url ~ c.php"
    清除/M00目录下所有文件的缓存:

    # varnishadm ban.url "^/M00"

    清除所有缓存:

    # varnishadm ban.url "^/.*$"

    查看所有的ban列表:

    # varnishadm ban.list
  • HTTP压缩

    文本类型的响应内容可以进行压缩,比如(html,css,js,xml).
    在对象进入缓存之前进行压缩再存储:

    sub vcl_fetch {
    ...
        if (beresp.http.content-type ~ "text") {
            set beresp.do_gzip = true;
        }
    ...
    }

参考资料

  1. Varnish 3官方文档
  2. Varnish Book
标签：

相关文章