Varnish recipe for buildout

plone.recipe.varnish is a zc.buildout recipe to install Varnish. Even though the name contains the name Plone, there is nothing Plone-specific about this recipe: it works for non-Zope sites just as well.

PLEASE NOTE: Version 6.x of this recipe now 'supports' Varnish 6.0 LTS only. Template generation support for older unsupported Varnish versions or the intermediate development versions have been removed. Varnish Software (the company) has switched to a half yearly time boxed release cycles without rigorous quality assurance, from which an lts version is picked when deemed stable enough.

If you have custom vcl and/or want to use your current recipe, you can keep plone.recipe.varnish pinned to versions 2.x, or you can use this 6.x version of the recipe, but provide parameters for a custom download url and custom vcl file.

Configuring it is very simple. For example:

[varnish-build]
recipe = plone.recipe.varnish:build

[varnish-configuration]
recipe = plone.recipe.varnish:configuration
backends = 127.0.0.1:8081

[varnish-script]
recipe = plone.recipe.varnish:script
bind = 127.0.0.1:8000
cache-size = 512M

This configures three buildout parts:

varnish-build

which will download, compile and install varnish,

varnish-configuration

which generates the VCL configuration file, sending requests to a backend at 127.0.0.1:8081, and

varnish-script

which runs Varnish, configured to listen on 127.0.0.1:8000 for requests, using a 512 megabyte cache.

A wrapper script for the varnish startup command is created in the bin directory of your buildout.

Please note that the configuration generated by this recipe is specifically meant for the latest LTS version of Varnish, version 6.0.x .

Area's of interest to look at when you use this recipe

Even though this recipe tries to provide sane defaults for most parameters for a drop in placement, there are a few area's of expertise which you can learn about to understand and test for a performant but stable Varnish set up.

• If you use this recipe, you will use buildout and a common setup for 'single' projects is that your Varnish will only cache for one Plone site. An inherent problem with caching is cache invalidation: when an editor changes a page you want to make sure the next delivery for this content item fetched freshly from the backend (Plone) site. Plone's caching setup can be configured to send a purge requests for changed content to Varnish. The calculated has plays an essential role in this type of purging: if the calculated hash from the client request is different from the calculated hash on the purge request, purging will fail.
• There are clever alternative purge request setups, which can improve freshness but you really have to know what you are doing and experience so far is that more advanced schemes have broken between Varnish upgrades.
• Especially if you have multiple backends and you let Varnish do the load balancing, don't forget to enable the grace-sick and grace-healthy options. When the recipe notices there parameters, it automatically enables health probe settings in the generated vcl. grace helps serving pages temporarily when your backends are temporarily down.
• As soon as a cookie appears on an incoming request, Varnish will not cache the request and even store the url in a 'hit for pass' buffer so that subsequent similar requests don't get delayed by waiting in the backend request queue. The generated vcl has a function which strips off most irrelevant cookies from incomiing requests before they get passed to the backend to increase cache hit rate. the __ac cookie is the most notable exception, this indicates for Plone that a user is logged in and caching should be disabled.
• You can monitor Varnish caching operations in great detail by learning how to use varnishlog and the query language, but it will take at least a few hours if you have never used this tool before. It is easy to draw the wrong conclusions from just poking around a bit in varnishlog and seeing hits or misses.

Virtual hosting

Varnish supports virtual hosting by selecting a different backend server based on headers on the incoming request. You can configure the backends through the backends option:

[varnish-configuration]
backends =
   plone.org:127.0.0.1:8000
   plone.net:127.0.0.1:9000

This will generate a configuration which sends all traffic for the plone.org host to a backend server running on port 8000 while all traffic for the plone.net host is send to port 9000.

Zope 2 hosting (with Virtual Host Monster)

If you are using Zope 2 as backend server you will need to rewrite the URL so the Zope Virtual Host Monster (VHM) can generate correct links for links in your pages. This can be done either by a web server such as Apache or nginx (placed either in front or behind Varnish) but can also be done by Varnish itself.

The three options are described below.

Option 1 (rewrites after Varnish)

If generating these VHM-style URLs in a proxy behind Varnish (or if using VHM's 'mapping' feature), no extra Varnish configuration is needed. Just make sure the backends option directs the traffic to the proxy.

Option 2 (rewrites before Varnish)

If generating these VHM-style URLs in a proxy in front of Varnish, no extra Varnish configuration is needed as long as the original hostname is still retained in the URL. If the hostname is not retained, you can tell Varnish to direct requests based on the "path" instead of the hostname. For example:

[varnish-configuration]
backends =
  /VirtualHostBase/http/plone.org:80/Plone:127.0.0.1:8000
  /VirtualHostBase/http/plone.net:80/Plone:127.0.0.1:9000

This will generate a configuration which sends all traffic for any request whose path starts with /VirtualHostBase/http/plone.org:80/Plone to a backend server running at 127.0.0.1 on port 8000, while request paths starting with /VirtualHostBase/http/plone.net:80/Plone are sent to port 9000.

Option 3 (rewrites within Varnish)

To have Varnish generate these VHM-style URLs, you can use the zope2_vhm_map option. Here is an example:

[varnish-configuration]
zope2_vhm_map =
    plone.org:/plone
    plone.net:/plone

This tells us that the domain plone.org should be mapped to the location /plone in the backend. By combining this with the information from the backends option a varnish configuration will be generated that maps URLs correctly.

Load Balancing

Varnish supports load balancing by configuring a director for a pool of backends. This director sends the incoming requests that cannot be fulfilled by varnish to backends in the pool in either random or round robin fashion. You can configure the director via the balancer option:

[varnish-configuration]
balancer = random

This will generate a configuration which sends all traffic to the director, which will choose a 'random' backend server to fulfill the request if the content requested is not cached by varnish itself.

plone.recipe.varnish reference

The plone.recipe.varnish recipe does one or more of the following:

plone.recipe.varnish:build

compiles varnish from sources

plone.recipe.varnish:configuration

generates a VCL-configuration file

plone.recipe.varnish:script

generates a wrapper script inside your buildout that will start Varnish with the correct configuration.

Build varnish from sources

build is based on zc.recipe.cmmi - so all parameters from that recipe are available here too (but rarely used). These options are available for the recipe part plone.recipe.varnish:build.

Three parameters are different/ extra:

url

Location used for download of varnish sources. Defaults to the latest LTS version.

jobs

Passes the number of parallel jobs to make, defaults to 4. Adjust as needed to your CPU resources.

compile-vmods

Boolean flag defaults to False, used for building Varnish modules. By defaults, the Varnish modules are from module collection by Varnish Software

VCL Configuration Generator

These options are available for the recipe part plone.recipe.varnish:configuration.

backends

Specifies the backend or backends which will process the (uncached) requests. The syntax for backends:

[<hostname>][/<path>]:<ip address>:<port>

The optional hostname and path allows you to do virtual hosting. If multiple backends are specified then each backend must include either a hostname or path (or both) so that Varnish can direct the matching request to the appropriate backend. Defaults to 127.0.0.1:8080.

balancer

If included and set to either random or round_robin, this option configures varnish to load balance the servers specified by the backends directive. Possible values: none (default), round_robin or random.

between-bytes-timeout

If specified, this option configures the timeout (in seconds) for Varnish waiting between bytes when receiving data from a backend. Varnish will only wait this many seconds between bytes before giving up. A value of 0s means this will never time out. Defaults to 60s, as per Varnish's default settings.

bind

Hostname and port on which Varnish will listen for requests. Defaults to 127.0.0.1:8000.

connect-timeout

If specified, this option configures the connection timeout (in seconds) for Varnish connecting to a backend server. Varnish will only try to connect to a given backend for this many seconds before giving up. Defaults to 0.4s, as per Varnish's default settings.

cookie-pass

This list consists of lines with a cookie-match and urlexclude in the form: "cookiematch":"urlexcludes". If cookiematch applies for the cookiename, then the request is passed directly to the configured backend bypassing any caching. Additionally, if the current url matches urlexcludes, then the cookies are removed, and the request piped to the backend. Defaults are optimized for Plone, one line: "auth_token|__ac(|_(name|password|persistent))=":"\.(js|css|kss)$" So when you are authenticated, the request is always handled by Plone. When an authenticated user requests a js/css/kss file, Plone will see you as anonymous because no cookies reach Plone.

cookie-pass-not-exclude

If url matches this regexp, cookie-pass exclude rules are skipped. This is useful for url like ++resource++zmi that requires authentication also for resources like js, css, ...

cookie-whitelist

When the cookie-pass is processed and does not match, this means you are anonymous, at least with the default cookie-pass settings. In that case, this whitelist is used to to sanitize cookie data on the request. Cookie data to be sent to the backend includes only cookies with the given names. Defaults are optimized for Zope2/Plone: statusmessages __ac _ZopeId __cp auth_token The __ac and auth_token cookies should not be needed, as they are already in the cookie-pass list, but they are here for safety in case you have customized the cookie-pass setting to not include them. If you have custom code that sets cookies and needs to read them in the backend, then you must add the cookie names to this list.

first-byte-timeout

If specified, this option configures the timeout (in seconds) for Varnish receiving the first byte from a backend. Varnish will only wait for this many seconds before giving up. A value of 0s means Varnish will never time out. Defaults to 300s.

purge-hosts

Specifies hostnames or IP addresses for purge ACL. By default localhost and the backends are allowed to purge. Additional allowed hosts are listed here.

vcl_recv, vcl_hit, vcl_miss, vcl_backend_fetch, vcl_backend_response, vcl_deliver, vcl_pipe, vlc_purge, vcl_hash, vcl_import, vcl_init, vcl_pass, vcl_synth

Insert arbitrary VCL code into the generated config.

verbose-headers

Enable sending extra headers in responses that expose what varnish did with the request and the cache status. Useful for debugging cache settings and optimizations. Possible values: on or off (default).

zope2_vhm_map

Defines a virtual host mapping for Zope servers. This is a list of hostname:ZODB location entries which specify the location inside Zope where the website for a virtual host lives.

zope2_vhm_port

Defines a virtual host mapping port to use in the VHM URL to send back to clients. Useful if there is another port mapping in front of varnish, such as haproxy. Defaults to bind port.

zope2_vhm_ssl

If specified, this maps VHM URLs to https for all requests. Possible values: on or off (default).

zope2_vhm_ssl_port

Defines a virtual host mapping port to use in the VHM URL to send back to clients. Useful if there is another port mapping in front of varnish, such as haproxy. Defaults to 443.

vcl-version

Varnish VCL format version. If not given it defaults to 4.0.

health-probe-*

Settings for backend health probes. Probes are activated if grace-healthy is set.

See https://varnish-cache.org/docs/6.0/reference/vcl.html#probes for a detailed explanation of each setting.

• health-probe-url: defaults to /ok
• health-probe-timeout: defaults to 5s
• health-probe-interval: defaults to 15s
• health-probe-window: defaults to 10
• health-probe-threshold: defaults to 8
• health-probe-initial: If not given varnish will default to threshold -1

grace-healthy

Grace in the context of Varnish means delivering otherwise expired objects when circumstances call for it. This can happen because: (1) the backend-director selected is down, or (2) a different thread has already made a request to the backend that's not yet finished.

If the backend is healthy, accept objects that are this number of seconds old. Clients will be delivered content that is no more than number of seconds past its TTL.

Format: number followed by a time unit: ms, s, m, h.

Defaults to None. If this is set to None the grace feature is disabled.

grace-sick

If the backend is sick, accept objects that are this old. See also grace-healthy.

Format: number followed by a time unit: ms, s, m, h.

Defaults to 600s. Should be greater than grace-healthy.

To test the generated configuration for syntactic correctness, run varnishd -C -f ./parts/varnish-configuration/varnish.vcl.

Create script to start varnish

Start varnish as a daemon or in foreground with the given settings. These options are available for the recipe part plone.recipe.varnish:script.

bind

Hostname and port on which Varnish will listen for requests. Defaults to 127.0.0.1:8000.

build-part

References the buildout part in order to get settings from there. Defaults to varnish-build. Set it to false in order to switch it off.

cache-location

Customise the location for the Varnish file storage. Option only applicable when used with file or persistent cache-type options. Defaults to using a file named storage inside the relevant parts directory (eg parts/varnish/storage). Changing the default location can be useful in putting the storage somewhere with quicker read speeds (e.g. RAM disk).

cache-size

The size of the cache (limited to 2G on 32bit systems). Defaults to 256M.

cache-type

Specify the type of cache storage to use with Varnish. Possible values: file (storage for each object is allocated from an arena backed by a file), malloc (storage for each object is allocated with malloc; in memory), or persistent (experimental as at Varnish 2.1.4). Defaults to file.

configuration-file

Path to a Varnish VCL configuration file to use. Defaults to the generated file from the configuration-part setting. If no configuration was generated, this setting is mandatory.

configuration-part

Names the buildout part to get settings from. Defaults to varnish-configuration.

daemon

The file and path of the varnish daemon varnishd to use. If not given, it looks for the build part (see the build-part setting) and uses its location setting plus the string /sbin/varnishd. If there is no build part, it defaults to /usr/sbin/varnishd - the most common place where it's found on many Unix systems. Adjust it if needed.

group

The name of the group that varnish should switch to before accepting any request. This defaults to the main group for the specified user.

mode

Specify whether the varnish daemon should run in daemon or foreground mode. The latter is useful when varnish is run by service supervision tools like daemontools or runit. Defaults to daemon.

name

If specified this sets the name of the varnish instance (defaults to the host name).

From varnishd's manpage:

Amongst other things, this name is used to construct the name of the directory in which varnishd keeps temporary files and persistent state. If the specified name begins with a forward slash, it is interpreted as the absolute path to the directory which should be used for this purpose.

runtime-parameters

Runtime parameter configuration options. The full list of available options can be found in the manpage varnishd(1) for your version of varnish. Examples include thread_pool_max, thread_pool_min, sess_timeout.

telnet

If specified sets the hostname and port on which Varnish will listen for commands using its telnet interface.

script-filename

Name of the start script file in buildout:bin-directory. Defaults to the name of this buildout part.

secret-file

In Varnish 4.X the telnet interface is no longer usable without authentication by default. A pre shared key mechanism has been put in place which requires both the varnish daemon and a client connection over telnet (like the varnishadm tool) to have a shared key to authenticate. By default if no secret-file is specified, it's no longer possible to authenticate to the telnet interface.

To disable this security feature (and go back to the dark Varnish 2 & 3 days) use secret-file = disabled. This is discouraged.

To enable the secret-file, give the path to a file on the filesystem that preferably has random content and is both accessible to the varnish daemon and a command line utility like varnishadm.

An example buildout part to generate such a file could be:

[varnish-secret]
recipe = plone.recipe.command
command = dd if=/dev/random of=${buildout:directory}/var/varnish_secret count=1
          chmod 600 ${buildout:directory}/var/varnish_secret

Giving secret-file the location of this file will pass on the secret to the varnish daemon when it starts up. Afterwards you can use varnishadm with the parameters -T host:port -S /path/to/varnish_secret to connect to the admin telnet interface.

user

The name of the user varnish should switch to before accepting any requests. Defaults to nobody.

Examples:

Use system varnish at /usr/sbin/varnishd, generate start script in ./bin/varnishd using a VCL-file in ./parts/varnish-configuration/varnish.vcl:

[buildout]
parts =
    varnish-script
    varnish-configuration

[varnish-script]
recipe = plone.recipe.varnish:script

[varnish-configuration]
recipe = plone.recipe.varnish:configuration