News from the High Energy Physics Research Computing Group at the University of Victoria

The Shoal system has been running stable in a production environment for several years now without much change. The goal of Shoal is to help provide contextualization to new virtual machines in a cloud production habitat. More simply- Shoal provides virtual machines with some squid proxies where they can retrieve the software and data they need to run their payloads without going all the way to the source. The Shoal system is broken down into three components: shoal-agent shoal-server shoal-client The shoal-agent is a daemon process that runs on a squid proxy cache. The daemon collects various health metrics and configuration information about the squid and sends the shoal-server a message via AMQP (Advanced Message Queuing Protocol). Each installation of shoal-agent will have a shoal-agent configuration file typically found at /etc/shoal/ shoal_agent.conf. This file allows you to configure several things about the squid cache such as which shoal-server to register to, who the ...

The 18th International Workshop on Advanced Computing and Analysis Techniques in Physics Research ( ACAT ) took place in Seattle this week. We presented our work on integrating the dynamic web federation into HEP computing as a poster . The conference focused on the use of machine learning algorithm in physics research with contributions from industry offering effective computing technology to execute workflows employing deep neural nets. These technologies offer solutions to the computing issues the field is facing in light of a great increase in data with a constant computing budget. When the LHC experiments were planned it was assumed that Dennard Scaling would solve this problem for us, it has become clear that this is not the case. It was shown that generative adversarial neural nets may be used to do do simulation, and that supervised learning may provide options for triggering and reconstruction. In some places these technologies are already used. nVidia, Microsoft, and DW...

After several months of prototyping and development Glint version 2.0 (Glint v2) has entered production. Glint v2 is a standalone web service inspired by Colin Leavett-Brown and Ron Desmarais' original glint service (Glint v1). The idea of glint was to allow for image replication across multiple openstack clouds using a simple interface instead of manually downloading and uploading images to new locations. Version 2 differs from the original in that it is a dedicated web service instead of an extension of the Openstack Horizon dashboard. Unfortunately the Openstack developers had a different philosophy regarding image and repository management and decided not to accept glint v1 as a proprietary module. The Openstack 6 month development cycle made it unreasonable for a small group like UVic's HEPRC group to maintain Glint v1 as an openstack plugin. Instead a new version of the service was conce...

As we showed previously , there is an easy way to use the information derived from VOMS-server based on grid-mapfiles to authorize a specific user to access a specific part of the dynamic federation. This solution was based on 3 parts: a grid-mapfile listing the DNs of all users from all supported VOs with all possible roles a text file (accessfile) that specifies the different privileges for the different parts of the storage federation a python script that is doing the authentication and authorization based on the 2 previously mentioned files While in this solution the grid-mapfile and accessfile can be changed anytime without the need to reload/restart the httpd and memcache process, there is also a simpler solution based on the internal authentication methods possible which however needs to restart httpd and memcache after each change. This one will be explained in the following. Using the built in authentication in Dynafed, one can grant access to a specific part o...

This post describes how to install, configure, and run shoal-agent to add your squid cache to shoal. If you are interested in shoal itself we published a description of the system here: Dynamic web cache publishing for IaaS clouds using Shoal I Gable, M Chester, P Armstrong, F Berghaus, A Charbonneau, C Leavett-Brown, M Paterson, R Prior, R Sobie, R Taylor. Proceedings of the 2013 CHEP Conference, Journal of Physics: Conference Series 513 (2013) 062035 Amsterdam 2013 Squid Installation If you already have a squid installed, move on to the next part. Otherwise, follow the instructions for installing the frontier squid variant . The frontier squid is nicely packaged and optimized for Frontier and CVMFS. Shoal Agent Installation If you want to set up Shoal Agent very quickly, do: curl https://raw.github.com/hep-gc/shoal/master/scripts/production-agent-install-for-hep.sh | bash service shoal-agent start More extensive instructions are available as Github documentation . ...

The UVIC group has been using private and commercial clouds for High Energy Physics (HEP) applications from different experiments successfully over the last 5 years using CloudScheduler , an own development by the group.                                                                                 Recently, Microsoft Azure cloud resources were integrated into the distributed cloud computing infrastructure that was designed and is operated by the group. For this integration test, up to 400 cores were used in parallel. The Azure resources were found to be a very reliable and high-performance resource for HEP applications. Microsoft put a video about our work online   and also mentions our cooperation in their own blog .

The Dynamic Federation project (DynaFed)  is a good way to federate existing Grid storage as well as making S3 based object stores available for Grid storage. To access DyanFed, the webdav/https protocol is used. The default authentication for accessing the storage backend is based on X.509 certificates and voms proxies. In that case, the user has to use grid tools to contact a voms server to authenticate and create a X.509 proxy locally on the client machine. This proxy can then be used by tools that support it to list and access files behind DynaFed. However, that kind of authentication must be supported by the tool used. It will for example not work using a web browser. In addition to the build-in authentication, DynaFed also supports Python based scripts to handle the authentication. One of the standard Grid authentication method is to authenticate against a so called grid-mapfile which is usually created by tools like  edg-mkgridmap . We implemented this grid-mapfile...