Shoal - Squid Proxy Discovery and Management

The Shoal system has been running stable in a production environment for several years now without much change. The goal of Shoal is to help provide contextualization to new virtual machines in a cloud production habitat. More simply- Shoal provides virtual machines with some squid proxies where they can retrieve the software and data they need to run their payloads without going all the way to the source.

The Shoal system is broken down into three components:
  • shoal-agent
  • shoal-server
  • shoal-client
The shoal-agent is a daemon process that runs on a squid proxy cache. The daemon collects various health metrics and configuration information about the squid and sends the shoal-server a message via AMQP (Advanced Message Queuing Protocol). Each installation of shoal-agent will have a shoal-agent configuration file typically found at /etc/shoal/shoal_agent.conf. This file allows you to configure several things about the squid cache such as which shoal-server to register to, who the shoal-server is allowed to serve this squid to (for issues with firewalls or users who don't wish to share their proxy with other shoal-users), and logging options. More details about all the configuration options can be found in the default config file.

The shoal-server is a RESTful web-service that processes the incoming AMQP messages and maintains a list of all active squid proxies (squids running shoal-agents). The server exposes the proxies via a REST interface that inspects the requester's IP and returns an ordered list of the most appropriate proxies based on the proxy's current load and geographic location in relation to the requester. For example anyone can visit http://shoal.heprc.uvic.ca/nearest/10 and receive a JSON package of the 10 most suitable proxies.

The shoal-client is a small python script that simply leverages the REST interface provided by the shoal-server. Running the shoal-client without any options will contact the shoal-server specified in the shoal-client configuration file for the most appropriate proxies then use cvmfs-talk to update the active proxy configuration.

In the recent 0.6.4 release of the shoal-client a new option has been added to support frontier clients. Running shoal-client with the --frontier (or -f) option produces an output string appropriate for setting the frontier proxy environment variable. More information about the frontier option can be found at the shoal github page. This release also changes the client to use the sys-log instead of it's own dedicated log which removes the need for user permissions and any pre-configuration of a log file.

If you'd like to add your squid proxies to the shoal-server list to begin using shoal with your system, Frank Berghaus has made some detailed instructions in a previous blog post.

Comments

Post a Comment

Popular posts from this blog

Grid-mapfile based authentication for DynaFed

The Dynamic Federation project (DynaFed)  is a good way to federate existing Grid storage as well as making S3 based object stores available for Grid storage. To access DyanFed, the webdav/https protocol is used. The default authentication for accessing the storage backend is based on X.509 certificates and voms proxies. In that case, the user has to use grid tools to contact a voms server to authenticate and create a X.509 proxy locally on the client machine. This proxy can then be used by tools that support it to list and access files behind DynaFed. However, that kind of authentication must be supported by the tool used. It will for example not work using a web browser. In addition to the build-in authentication, DynaFed also supports Python based scripts to handle the authentication. One of the standard Grid authentication method is to authenticate against a so called grid-mapfile which is usually created by tools like  edg-mkgridmap . We implemented this grid-mapfile...
Read more

Monitoring Dynafed with ELK

Image
The Dynamic Federation project (DynaFed)  being developed at CERN is intended to federate any number of different types of storage endpoints allowing users to read or write data transparently and efficiently. How it works in a nutshell: A client sends a COPY, GET, PUT, DELETE HTTP request to the DynaFed instance's URL. DynaFed decides which amongst the storage endpoints it federates is the "best" one to handle the request, sorted according to geographic location and available free space. The client receives a 302 redirect link pointing to this storage endpoint with the protocol and authentication tokens necessary for the transaction. The client re-sends the HTTP request directly to this storage endpoint. As we can see, after the redirect link is provided, DynaFed is out of the loop and as such is unaware of whether the client-storage endpoint transaction is successful or not. So while we cannot get this valuable information from Dynafed, we instead are interested i...
Read more

Authorization in DynaFed, Part 2

As we showed previously , there is an easy way to use the information derived from VOMS-server based on grid-mapfiles to authorize a specific user to access a specific part of the dynamic federation. This solution was based on 3 parts: a grid-mapfile listing the DNs of all users from all supported VOs with all possible roles a text file (accessfile) that specifies the different privileges for the different parts of the storage federation a python script that is doing the authentication and authorization based on the 2 previously mentioned files While in this solution the grid-mapfile and accessfile can be changed anytime without the need to reload/restart the httpd and memcache process, there is also a simpler solution based on the internal authentication methods possible which however needs to restart httpd and memcache after each change. This one will be explained in the following. Using the built in authentication in Dynafed, one can grant access to a specific part o...
Read more