sácale el jugo a varnish
DESCRIPTION
Ideas, plugins y trucos para sacarle un mayor partido a Varnish con o sin drupal.TRANSCRIPT
![Page 1: Sácale el jugo a Varnish](https://reader033.vdocuments.site/reader033/viewer/2022042508/547ba36cb37959822b8b4d52/html5/thumbnails/1.jpg)
Sácale jugo a Varnish
![Page 2: Sácale el jugo a Varnish](https://reader033.vdocuments.site/reader033/viewer/2022042508/547ba36cb37959822b8b4d52/html5/thumbnails/2.jpg)
¿Quién soy?
Responsable técnico de idealista/news @rodricels Curioso sin remedio
![Page 3: Sácale el jugo a Varnish](https://reader033.vdocuments.site/reader033/viewer/2022042508/547ba36cb37959822b8b4d52/html5/thumbnails/3.jpg)
VARNISH
![Page 4: Sácale el jugo a Varnish](https://reader033.vdocuments.site/reader033/viewer/2022042508/547ba36cb37959822b8b4d52/html5/thumbnails/4.jpg)
cURL
https://github.com/varnish/libvmod-curl
cURL +
vmod
![Page 5: Sácale el jugo a Varnish](https://reader033.vdocuments.site/reader033/viewer/2022042508/547ba36cb37959822b8b4d52/html5/thumbnails/5.jpg)
• Petición curl.get(url) curl.post(url, data) curl.header_add() curl.header_remove() curl.set_connect_timeout() curl.set_timeout()
• Devolución curl.header("foo") curl.status() curl.body() curl.error()
cURL
![Page 6: Sácale el jugo a Varnish](https://reader033.vdocuments.site/reader033/viewer/2022042508/547ba36cb37959822b8b4d52/html5/thumbnails/6.jpg)
![Page 7: Sácale el jugo a Varnish](https://reader033.vdocuments.site/reader033/viewer/2022042508/547ba36cb37959822b8b4d52/html5/thumbnails/7.jpg)
if (req.request == "BAN" && client.ip ~ purge_ban) { ban("obj.http.x-host == " + req.http.host + "&& obj.http.x-url == " + req.url); // ¡¡¡ AQUÍ VA LA MAGIA !!! curl.get(req.http.host + req.url); error 200 "Ban added"; }
cURL: Calentar caché tras banear
Spoiler: se puede hacer sin cURL
![Page 8: Sácale el jugo a Varnish](https://reader033.vdocuments.site/reader033/viewer/2022042508/547ba36cb37959822b8b4d52/html5/thumbnails/8.jpg)
cURL: autentificación
curl.fetch("http://authserver/validate?key=" + regsub(req.url, ".*key=([a-z0-9]+), "\1")); if (curl.status() != 200) { error 403 "Go away"; }
http://err.no/personal/blog/2011/Aug/03
![Page 9: Sácale el jugo a Varnish](https://reader033.vdocuments.site/reader033/viewer/2022042508/547ba36cb37959822b8b4d52/html5/thumbnails/9.jpg)
cURL: actualizar contenido relacionado
if (req.url ~ foo && obj.hits > 10000) { curl.get(req.http.host + bar); }
![Page 10: Sácale el jugo a Varnish](https://reader033.vdocuments.site/reader033/viewer/2022042508/547ba36cb37959822b8b4d52/html5/thumbnails/10.jpg)
![Page 11: Sácale el jugo a Varnish](https://reader033.vdocuments.site/reader033/viewer/2022042508/547ba36cb37959822b8b4d52/html5/thumbnails/11.jpg)
Calentar caché
Otras formas • Varnishreplay –r log.txt • Curl/wget sitemap.xml
#!/bin/bash URL='www.example.com' wget --quiet http://$URL/sitemap.xml --no-cache --no-cookies --output-document - | egrep -o "http://$URL/foo/bar" | while read line; do curl --user-agent 'Cache Warmer' --silent --location --max-time 10 $line > /dev/null 2>&1 sleep 5 done
![Page 12: Sácale el jugo a Varnish](https://reader033.vdocuments.site/reader033/viewer/2022042508/547ba36cb37959822b8b4d52/html5/thumbnails/12.jpg)
Calentar caché
• Purge + Restart sub vcl_hit { # y vcl_miss() if (req.request == "PURGE") { purge; set req.request = "GET"; set req.http.X-purger = "Purged"; error 800 "restart"; # set req-backend = SuperServer; # return(restart); } }
https://www.varnish-software.com/static/book/Saving_a_request.html#solution-combine-purge-and-restart
![Page 13: Sácale el jugo a Varnish](https://reader033.vdocuments.site/reader033/viewer/2022042508/547ba36cb37959822b8b4d52/html5/thumbnails/13.jpg)
Detección de dispositivos
https://www.varnish-cache.org/vmod/dclass-apache-devicemap https://github.com/OpenDDRdotORG/OpenDDR-Resources
Open Device Description Repository
OpenDDR
+
dClass Dtree Pattern Classification
Engine
![Page 14: Sácale el jugo a Varnish](https://reader033.vdocuments.site/reader033/viewer/2022042508/547ba36cb37959822b8b4d52/html5/thumbnails/14.jpg)
Detección de dispositivos
set req.http.dclass_openddr = dclass.classify(req.http.user-agent); if (dclass.get_field("is_tablet") == "true"){ set req.http.dclass_type = "tablet"; } else if ( dclass.get_field("is_wireless_device") == "true" && dclass.get_field("inputDevices") == "touchscreen"){ set req.http.dclass_type = "smartphone"; }
![Page 15: Sácale el jugo a Varnish](https://reader033.vdocuments.site/reader033/viewer/2022042508/547ba36cb37959822b8b4d52/html5/thumbnails/15.jpg)
Detección de dispositivos
sub vcl_recv() { if (req.http.Cookie ~ "^X-device=") { # do the magic } } sub vcl_fetch() { set obj.http.Set-Cookie = "X-device=" + req.http.dclass_type; domain=.example.com; path=/"; }
![Page 16: Sácale el jugo a Varnish](https://reader033.vdocuments.site/reader033/viewer/2022042508/547ba36cb37959822b8b4d52/html5/thumbnails/16.jpg)
Detección de dispositivos
Fabricante Modelo Ancho y alto de pantalla Inputs (táctil, teclado, etc) Soporte de Javascript Si es wireless / tablet / crawler / desktop Navegador, nombre y versión Sistema operativo
![Page 17: Sácale el jugo a Varnish](https://reader033.vdocuments.site/reader033/viewer/2022042508/547ba36cb37959822b8b4d52/html5/thumbnails/17.jpg)
Geo IP
Fijar conexión a servidores locales Estúpida ley de cookies europea Restricciones copyright por países Tres implementaciones La de Cosimo (Opera) permite nivel de ciudad
https://github.com/leed25d/geoip-vmod https://github.com/lampeh/libvmod-geoip https://github.com/cosimo/varnish-geoip
![Page 18: Sácale el jugo a Varnish](https://reader033.vdocuments.site/reader033/viewer/2022042508/547ba36cb37959822b8b4d52/html5/thumbnails/18.jpg)
Firewall varnish
https://github.com/comotion/VSF
Usa otros vmods ParseReq Shield Throttle UrlEncode
Basado en mod_security
Backend-less: mejor no mezclar con la lógica de caché
![Page 19: Sácale el jugo a Varnish](https://reader033.vdocuments.site/reader033/viewer/2022042508/547ba36cb37959822b8b4d52/html5/thumbnails/19.jpg)
Firewall varnish
• Mitigación de DDoS • Ataques SQL • XSS • URL malformadas • Robots y arañas • Vulnerabilidades
https://github.com/comotion/VSF
![Page 20: Sácale el jugo a Varnish](https://reader033.vdocuments.site/reader033/viewer/2022042508/547ba36cb37959822b8b4d52/html5/thumbnails/20.jpg)
Firewall varnish
• Toma de decisiones • Solo log • Bloquear • Devolver html • Honey-trap • Redirect / image
![Page 21: Sácale el jugo a Varnish](https://reader033.vdocuments.site/reader033/viewer/2022042508/547ba36cb37959822b8b4d52/html5/thumbnails/21.jpg)
Te va a parar unos cuantos balones…
pero no es Lev Yashin
Firewall varnish
![Page 22: Sácale el jugo a Varnish](https://reader033.vdocuments.site/reader033/viewer/2022042508/547ba36cb37959822b8b4d52/html5/thumbnails/22.jpg)
Autentificación
ldap
htpassw
vcl
*(y con cURL)
https://www.varnish-cache.org/vmod/ldap-authentication https://github.com/pariahsoft/libvmod-authentication https://www.varnish-cache.org/vmod/basicauth
![Page 23: Sácale el jugo a Varnish](https://reader033.vdocuments.site/reader033/viewer/2022042508/547ba36cb37959822b8b4d52/html5/thumbnails/23.jpg)
Autentificación con ldap
import ldap; if(req.url ~ "^/member/"){ if(!(req.http.Authorization && ldap.simple_auth( true, "cn=Manager,dc=ldap,dc=example,dc=com", "password", "ldap://192.168.1.1/ou=people,dc=ldap, dc=example,dc=com?uid?sub?(objectClass=*)", ldap.get_basicuser(), ldap.get_basicpass() ))){ error 401; }
![Page 24: Sácale el jugo a Varnish](https://reader033.vdocuments.site/reader033/viewer/2022042508/547ba36cb37959822b8b4d52/html5/thumbnails/24.jpg)
Autentificación con htpassw
Fichero con estructura htpassw usando md5 o sha1 (no es necesario apache) import basicauth; sub vcl_recv { if (!basicauth.match("/var/www/.htpasswd", req.http.Authorization)) { error 401 "Authentication required"; } }
![Page 25: Sácale el jugo a Varnish](https://reader033.vdocuments.site/reader033/viewer/2022042508/547ba36cb37959822b8b4d52/html5/thumbnails/25.jpg)
Autentificación harcoded en vcl
if(req.url ~ "^/protected/") { if(!authentication.match("admin", "test")) { error 401 "Authentication Required"; } }
![Page 26: Sácale el jugo a Varnish](https://reader033.vdocuments.site/reader033/viewer/2022042508/547ba36cb37959822b8b4d52/html5/thumbnails/26.jpg)
Ordenar parámetros
/video/480?title=0&byline=0&portrait=0&color=51a516 /video/480?byline=0&color=51a516&portrait=0&title=0 import boltsort; sub vcl_hash { set req.url = boltsort.sort(req.url); }
Aumenta el ratio de hit Disminuye el uso de memoria
https://www.varnish-cache.org/vmod/boltsort-querystring-params-sort https://github.com/Dridi/libvmod-querystring
![Page 27: Sácale el jugo a Varnish](https://reader033.vdocuments.site/reader033/viewer/2022042508/547ba36cb37959822b8b4d52/html5/thumbnails/27.jpg)
Reducir I/O de disco
“Si usas la RAM como almacén, todo Varnish funciona en memoria”
¡NO!
SHM
![Page 28: Sácale el jugo a Varnish](https://reader033.vdocuments.site/reader033/viewer/2022042508/547ba36cb37959822b8b4d52/html5/thumbnails/28.jpg)
Reducir I/O de disco
Shared Memory Log en /var/lib/varnish Junto con los .so de los vcls que hayas cargado
~ 80 MB /etc/fstab tmpfs /var/lib/varnish tmpfs rw,size=128M 0 0
Si tienes SSDs no hace falta
https://www.varnish-software.com/static/book/Tuning.html#the-shared-memory-log
![Page 29: Sácale el jugo a Varnish](https://reader033.vdocuments.site/reader033/viewer/2022042508/547ba36cb37959822b8b4d52/html5/thumbnails/29.jpg)
TTL personalizada
set beresp.ttl = 10m; if (beresp.http.X-TTL) { C{ char *ttl; ttl = VRT_GetHdr(sp, HDR_BERESP, "06X-TTL:"); VRT_l_beresp_ttl(sp, atoi(ttl)); }C }
http://www.slideshare.net/MaximeTopolov/varnish-14329696
![Page 30: Sácale el jugo a Varnish](https://reader033.vdocuments.site/reader033/viewer/2022042508/547ba36cb37959822b8b4d52/html5/thumbnails/30.jpg)
Proxys y Akamai
Varnish 3 no transforma string en IP ipcast.clientip(req.http.X-Forwarded-For); ipcast.clientip(req.http.True-Client-IP); ipcast.clientip("192.168.0.10"); ipcast.clientip("2001:db8::1");
https://github.com/lkarsten/libvmod-ipcast
![Page 31: Sácale el jugo a Varnish](https://reader033.vdocuments.site/reader033/viewer/2022042508/547ba36cb37959822b8b4d52/html5/thumbnails/31.jpg)
Imágenes
No las cachees.
![Page 32: Sácale el jugo a Varnish](https://reader033.vdocuments.site/reader033/viewer/2022042508/547ba36cb37959822b8b4d52/html5/thumbnails/32.jpg)
Imágenes
![Page 33: Sácale el jugo a Varnish](https://reader033.vdocuments.site/reader033/viewer/2022042508/547ba36cb37959822b8b4d52/html5/thumbnails/33.jpg)
Imágenes
Si tienes CDN ¿para qué cacheas los estáticos? Si no tienes CDN ¿para qué cacheas los estáticos? Apache 2.4 puede ser suficiente
OMFG!!!11one!!
Ban.nuke crece sin cesar
![Page 34: Sácale el jugo a Varnish](https://reader033.vdocuments.site/reader033/viewer/2022042508/547ba36cb37959822b8b4d52/html5/thumbnails/34.jpg)
Longtail
• Separar storages • Diferentes TTLs • TTLs por horas • TTLs por tipos de ficheros • TTLs por edad del contenido
![Page 35: Sácale el jugo a Varnish](https://reader033.vdocuments.site/reader033/viewer/2022042508/547ba36cb37959822b8b4d52/html5/thumbnails/35.jpg)
Longtail
sub vcl_fetch() { if (req.url ~ "^/archivo/20(0[1-9]|1[0-2])" && beresp.ttl > 0s ) { unset beresp.http.expires; set beresp.http.cache-control = "max-age=604800"; set beresp.ttl = 2w; // varnish ttl set beresp.storage = "disco"; } else { set beresp.storage = "memoria"; } }
https://www.varnish-cache.org/trac/wiki/VCLExampleLongerCaching
![Page 36: Sácale el jugo a Varnish](https://reader033.vdocuments.site/reader033/viewer/2022042508/547ba36cb37959822b8b4d52/html5/thumbnails/36.jpg)
Sitio en mantenimiento
Poner el sitio en mantenimiento puede ser una locura, avisa a Varnish de ello. Dos acercamientos: • Devolver todo lo que aún en caché y dar
mensaje en lo que no esté en caché. • Dar un mensaje de mantenimiento
![Page 37: Sácale el jugo a Varnish](https://reader033.vdocuments.site/reader033/viewer/2022042508/547ba36cb37959822b8b4d52/html5/thumbnails/37.jpg)
Varnish Bans Manager
Varnish Administration Console Gestor de bans libre Hecho en A Coruña por dot2code
https://github.com/dot2code/varnish-bans-manager
![Page 38: Sácale el jugo a Varnish](https://reader033.vdocuments.site/reader033/viewer/2022042508/547ba36cb37959822b8b4d52/html5/thumbnails/38.jpg)
Varnish Bans Manager
Web-manager en django/*SQL con ACLs
https://github.com/dot2code/varnish-bans-manager
![Page 39: Sácale el jugo a Varnish](https://reader033.vdocuments.site/reader033/viewer/2022042508/547ba36cb37959822b8b4d52/html5/thumbnails/39.jpg)
Varnish Bans Manager
Monitorización de bans actuales y pasados
![Page 40: Sácale el jugo a Varnish](https://reader033.vdocuments.site/reader033/viewer/2022042508/547ba36cb37959822b8b4d52/html5/thumbnails/40.jpg)
Varnish Bans Manager
Ban por nodo/grupo con expresiones regulares
![Page 41: Sácale el jugo a Varnish](https://reader033.vdocuments.site/reader033/viewer/2022042508/547ba36cb37959822b8b4d52/html5/thumbnails/41.jpg)
Varnish Bans Manager
Monitorización de bans actuales y pasados
![Page 42: Sácale el jugo a Varnish](https://reader033.vdocuments.site/reader033/viewer/2022042508/547ba36cb37959822b8b4d52/html5/thumbnails/42.jpg)
Memcached y Redis
https://www.varnish-cache.org/vmod/memcached https://github.com/zephirworks/libvmod-redis
Lectura y escritura en Memcached Lectura y escritura en Redis (soporte completo)
![Page 43: Sácale el jugo a Varnish](https://reader033.vdocuments.site/reader033/viewer/2022042508/547ba36cb37959822b8b4d52/html5/thumbnails/43.jpg)
Memcached y redis
Estadísticas if (memcached.incr("node-1234", 1)) { // set(STRING key, STRING value, INT expiration, INT flags) memcached.set("node-1234", "1", 0, 0); }
Recoger bloques set resp.http.block-123 = memcached.get("block-123");
![Page 44: Sácale el jugo a Varnish](https://reader033.vdocuments.site/reader033/viewer/2022042508/547ba36cb37959822b8b4d52/html5/thumbnails/44.jpg)
![Page 45: Sácale el jugo a Varnish](https://reader033.vdocuments.site/reader033/viewer/2022042508/547ba36cb37959822b8b4d52/html5/thumbnails/45.jpg)
![Page 46: Sácale el jugo a Varnish](https://reader033.vdocuments.site/reader033/viewer/2022042508/547ba36cb37959822b8b4d52/html5/thumbnails/46.jpg)
¿Preguntas?