Access

Introduction

Researchers are awarded time on iVEC’s supercomputers through a number of different allocation schemes. A typical compute-time allocation includes:

• a provision of compute time — measured in Service Units (see below), as appropriate;
• access to high-performance, temporary storage (called scratch) for in-progress computations;
• a quota’ed allocation of home and group file-system space, for application source code, binaries, and such like.

On all iVEC supercomputers, compute time is budgeted on quarterly periods, irrespective of the duration of the project. Typically, this is done by uniformly distributing the Service Unit (SU) allocation across relevant periods. For example, a project that is awarded 1,000,000 SUs during January–December 2013 would be credited with 250,000 SUs in each of the four quarters.

This breakdown of time is implemented in order to promote a more uniform demand on our systems – past experience tells us that our supercomputers are subject to excess levels of demand, and consequently significant delays in running jobs, at the end of the calendar year, when a number of merit-allocation periods come to a close.

With this in mind, unspent SUs from one quarter are not carried over into the subsequent quarter. However, a project can continue to run computations on our resources even after its SU budget is exhausted. We implement a fair-share system that means such jobs run, though are queued with a lower priority than is normally the case.

In exceptional circumstances, we will credit SUs to a project using a non-uniform quarterly distribution — most commonly to accommodate scientific needs or technical restrictions.

Our intent is to achieve fair access to iVEC supercomputers for all of our users, and to promote access to resources in a manner that makes effective use of the available computing capacity.

Service Units Explained

All projects that use the iVEC compute resources are allocated Service Units (SU) through one of the several merit allocation based schemes. How service units are defined and consumed will depend on which system you are using. SUs are consumed at the same rate regardless of job size or number. For example, on Epic a single job that runs for 4 hours on 288 cores will have used 1152 SU, and the reverse is the same, 288 single core jobs that run for 4 hours each, will also use 1152 SU. On Epic a single SU is defined as:

1 Epic SU = one core per hour of walltime

On Fornax a SU is defined differently, as the current version of the PBSpro resource management software is only able to record the usage on a walltime per core basis and not the GPU usage. Therefore on Fornax a SU is node based where:

1 Fornax SU = 1 node hour = 12 core hours = 1 GPU hour

SU on Magnus are also based on the ” one core per hour of walltime”. However you should note that the small job that can run on Magnus requires using the entire node and each node on Magnus has 16 cores. Any job that runs on Magnus will consume SU based on the usage of the entire node regardless, as it is not possible to share a node on the Cray with different users. Ideally all jobs on the Magnus should use some multiple of 16 cores per job. A single Magnus SU is defined as :

1 Magnus SU = one core per hour of walltime

SU are charged based on the fraction of actually walltime used not on the wall time requested in your job scripts. If you specify a walltime of 5 hours on 100 cores but the job finishes in 2 hours 30 minutes and 10 seconds then your allocation is charged 250.28 SU and not 500 SU.

Monthly Usage Statements

To help Project Leaders to monitor the compute-time usage of their researchers, iVEC sends out — at the beginning of each month — a short email detailing the accumulated compute-time usage for each project, along with the compute time remaining in the current quarter and in the project as a whole. A sample email (for a made-up project) is appended below.

At the time of writing, monthly statements are only created for projects on Epic and Fornax. iVEC is working to add this functionality for Magnus-hosted projects in due course.

Please contact the iVEC Helpdesk, if you have any questions, comments, or suggestions.

Dear Principal Investigator,
Please find, appended below, the monthly update on your project's usage of iVEC's "epic" Supercomputer.
Project Leader: Neil Armstrong
Project Group/ID: moonlander017
Report Date: 01/02/2014
Rolling Quarter Usage (CPU Hrs)
  Allocated: 500000
  Used: 13669.9 (2.7%)
  Remaining: 486330.1 (97.3%)
Total project usage (CPU Hrs)
  Allocated: 2000000
  Used: 13669.9 (0.7%)
  Remaining: 1986330.1 (99.3%)
Current Project Members:
  Edwin 'Buzz' Aldrin
  Alan Shepard
Should you have any questions or comments, please contact the iVEC Helpdesk <[email protected]>
rather than reply to this email.