iVEC

Data Stores @ iVEC

Home → Services → Data Stores @ iVEC

Pawsey Supercomputing Centre Data Stores

The Pawsey Supercomputing Centre is an ultra large scale data storage facility, that has multiple data storage options available to the research community with large scale data needs. Access to the Pawsey Supercomputing Centre Data Stores is governed by a Data Storage and Management Policy (DSMP), which was approved by the Pawsey Supercomputing Centre Board. The DSMP is available to all researchers from here.

The Pawsey Supercomputing Centre also has Comments / feedback on this statement can be provided

Access to the Pawsey Supercomputing Centre Data Stores

As Pawsey Data Stores are multiple petascale facilities, allocations are only provided to researchers that have more than 5TB of data to store at the point of application.

All data storage allocations, require researchers to complete the application form provided and need to meet the following criteria:

  • All researchers will be asked to provide a project title which will be associated with each directory provided; only one specific project will be permitted per directory on any Pawsey Supercomputing Centre store – irrespective of the status of the researcher. Researchers with more than one project data set should apply separately to have these housed within Pawsey.
  • Every directory can only have one PI designated PI, who owns all data in that directory; only the PI of that directory can grant access to others.
  • All new researchers can be added to a specific directory, only on conclusion of a signed declaration by the PI who owns the project space.
  • All researchers using iVEC’s Data Store must comply with the Pawsey Supercomputing Centre Data Storage and Management Policy.
  • Any application for an individual project must be > 5TB at the point of application
  • All applications for storage of > 100 TB will be referred to the Pawsey Supercomputing Centre Data Allocation Committee (established by the Board to represent all partners and stakeholders) for review. Details of representation of the data allocation committee and Terms of Reference are available online.
  • The Pawsey Supercomputing Centre will only approve applications where a second copy of the data is declared to be held elsewhere (except in exceptional circumstances). See Data Management and Storage Policy for more details.
  • The only access mechanism to access data to and from the Pawsey Supercomputing Centre Data Stores is via the assdet management tool, LiveARC (also known as Mediaflux).
  • At this time, due to funding the Pawsey Supercomputing Centre Data Stores will only provide storage applications until December 2016.
  • All Pawsey Data Stores long term storage is provided with four mandatory quarterly data ingest milestones. These milestones will be agreed in advance with all researchers.
  • All data allocations must be ingested to the Pawsey Supercomputing Centre within 12 months of an allocation being approved. Similar to Supercomputing processing allocations, Data ingest allocations cannot be rolled over from year to year.

The details of the Pawsey Supercomputing Centre storage requirements are also provided in a Guidance for applicants document.

What storage is available?

There are various large scale disk and tape based storage arrays available in the Pawsey Supercomputing Centre for research data only. The Pawsey Supercomputing Centre offers both CXFS and GPFS for completeness.

Currently the Pawsey Supercomputing Centre does not create any offsite or onsite back up copy of any data for any research project; in the event of a catastrophic technical failure, in most cases data can be recovered. However in the event of researcher error (e.g. deletion of data) the Pawsey Supercomputing Centre will not restore data. Therefore, all researchers are advised to have another copy of their data held elsewhere at all times.

How long do storage applications take to process?

Data storage applications are usually processed within 28 days of receipt. However in circumstances where the information received by the Pawsey Supercomputing Centre is incomplete or ambiguous, the process can take longer. All researchers are requested to complete all applications as fully and as accurately as possible to prevent delays.

Once I have an application, how do I ingest data to the Pawsey Supercomputing Centre?

The Pawsey Supercomputing Centre’s only data ingest tool is LiveARC (Mediaflux). Documentation on how to use LiveARC is available on the data portal (data.ivec.org). However LiveARC movie is available here to support researchers with the ingest process.

All Pawsey Supercomputing Centre long term storage is provided with four mandatory quarterly data ingest milestones. These milestones will be agreed in advance with all researchers. Researchers can ingest data faster than agreed, but are not permitted to ingest data slower than agreed. Researchers will be advised when they are in breach of milestone ingests and will be asked to take remedial action. All data ingest milestones must have an even distribution (therefore the complete allocation cannot be ingested in the final quarter). All data allocations must be ingested to the Pawsey Supercomputing Centre within 12 months of an allocation being approved.

Need more help?

Why don’t you come along to a Data Clinic? There is help available at a ‘drop-in’ Data Clinics near you. At a Clinic you can find out:

  • what data storage is available and how to access this
  • how to complete the application process
  • about data ingest procedures, tools and practices (e.g. what tools to use and why)
  • how Data staff can help you identify bottlenecks in present workflow
  • how best to share data (especially for researchers with collaborators elsewhere)
  • how best to make data discoverable by others.