Adding your Squid to Shoal

This post describes how to install, configure, and run shoal-agent to add your squid cache to shoal. If you are interested in shoal itself we published a description of the system here:

Dynamic web cache publishing for IaaS clouds using Shoal
I Gable, M Chester, P Armstrong, F Berghaus, A Charbonneau, C Leavett-Brown, M Paterson, R Prior, R Sobie, R Taylor.
Proceedings of the 2013 CHEP Conference, Journal of Physics: Conference Series 513 (2013) 062035 Amsterdam 2013

Squid Installation

If you already have a squid installed, move on to the next part. Otherwise, follow the instructions for installing the frontier squid variant. The frontier squid is nicely packaged and optimized for Frontier and CVMFS.

Shoal Agent Installation

If you want to set up Shoal Agent very quickly, do:

 curl https://raw.github.com/hep-gc/shoal/master/scripts/production-agent-install-for-hep.sh | bash
service shoal-agent start

More extensive instructions are available as Github documentation. In particular, note the shoal_agent.conf recommended for ATLAS. Once the shoal agent is running you should see your squid cache appear on shoal.

Squid Agent Configuration


The recommended modus operandi when using Shoal is to allow client connections to the squid from anywhere, but restrict destinations to the known CVMFS and Frontier servers. This way, CVMFS and Frontier access can be provided via the squid to any location, without the need for configuring specific ACLs for each known group of users. If the risk of DOS is a concern, you can employ rate limiting.

Here are good references for the syntax to use in customize.sh:
You should have this in customize.sh

uncomment("acl MAJOR_CVMFS")
uncomment("acl ATLAS_FRONTIER")
insertline("^# http_access deny !RESTRICT_DEST", "http_access allow MAJOR_CVMFS")
insertline("^# http_access deny !RESTRICT_DEST", "http_access allow ATLAS_FRONTIER")
setoption("acl NET_LOCAL src", "192.168.0.0/16")

This will allow connections that are either from the specified NET_LOCAL subnet(s) to anywhere or from anywhere to a CVMFS or Frontier server

These instructions were originally posted by Ryan on our twiki.

Comments

Post a Comment

Popular posts from this blog

Grid-mapfile based authentication for DynaFed

The Dynamic Federation project (DynaFed)  is a good way to federate existing Grid storage as well as making S3 based object stores available for Grid storage. To access DyanFed, the webdav/https protocol is used. The default authentication for accessing the storage backend is based on X.509 certificates and voms proxies. In that case, the user has to use grid tools to contact a voms server to authenticate and create a X.509 proxy locally on the client machine. This proxy can then be used by tools that support it to list and access files behind DynaFed. However, that kind of authentication must be supported by the tool used. It will for example not work using a web browser. In addition to the build-in authentication, DynaFed also supports Python based scripts to handle the authentication. One of the standard Grid authentication method is to authenticate against a so called grid-mapfile which is usually created by tools like  edg-mkgridmap . We implemented this grid-mapfile lookup i
Read more

Monitoring Dynafed with ELK

Image
The Dynamic Federation project (DynaFed)  being developed at CERN is intended to federate any number of different types of storage endpoints allowing users to read or write data transparently and efficiently. How it works in a nutshell: A client sends a COPY, GET, PUT, DELETE HTTP request to the DynaFed instance's URL. DynaFed decides which amongst the storage endpoints it federates is the "best" one to handle the request, sorted according to geographic location and available free space. The client receives a 302 redirect link pointing to this storage endpoint with the protocol and authentication tokens necessary for the transaction. The client re-sends the HTTP request directly to this storage endpoint. As we can see, after the redirect link is provided, DynaFed is out of the loop and as such is unaware of whether the client-storage endpoint transaction is successful or not. So while we cannot get this valuable information from Dynafed, we instead are interested i
Read more

Glint Version 2 Enters Production

After several months of prototyping and development Glint version 2.0 (Glint v2) has entered production. Glint v2 is a standalone web service inspired by Colin Leavett-Brown and Ron Desmarais' original glint service (Glint v1). The idea of glint was to allow for image replication across multiple openstack clouds using a simple interface instead of manually downloading and uploading images to new locations. Version 2 differs from the original in that it is a dedicated web service instead of an extension of the Openstack Horizon dashboard. Unfortunately the Openstack developers had a different philosophy regarding image and repository management and decided not to accept glint v1 as a proprietary module. The Openstack 6 month development cycle made it unreasonable for a small group like UVic's HEPRC group to maintain Glint v1 as an openstack plugin. Instead a new version of the service was conce
Read more