SC14 blog – Day four, George Beckett

Today was my last day at the Supercomputing 2014 conference, as I start my long journey back to Perth this evening.

The first appointment of the day was a meeting with a team from Sandia National Labs, for Chris B and me, to learn about a monitoring tool that they have been developing called LDMS. The tool is deployed onto a supercomputer or cluster and monitors various aspects of the load on the system — such as communications performance, CPU load, and memory usage. LDMS is similar to Ganglia, which iVEC uses to monitor its older systems Epic and Fornax. However, LDMS is better suited to large-scale systems than Ganglia, say the developers, as it places a lower load on both the compute nodes and the fabric. Further, it supports the Cray Aries interconnect that provides the fabric for both iVEC’s Magnus and Galaxy supercomputers (whereas Ganglia does not). Definitely something to consider as an option for production monitoring on the Pawsey resources.

The second half of the morning was spent in an SKA workshop, discussing the impact of next-generation sensor networks — such as SKA — on HPC. As well as providing a chance to catch-up with some of our collaborators in the Science Data Processor element, it was also a chance to share some of the progress from the first year or so of SKA with the wider HPC community.

Today was also the day on which the Student Cluster Challenge results were announced. The winner was, as last year, University of Texas, though the iVEC team looks to have performed strongly and provided competitive attempts at all of the challenges that were set.

The exhibition closed mid-afternoon, wrapping up Australian HPCs third unified presence at the conference: with three days of welcoming visitors, presenting Tim Tam talks, and running demonstrations — including both Drishti and Maali (the new name for iBS). We have put in our selection for a booth place at SC’15, which will be held in Austin Texas next November, and for which we have great plans.

The conference does not actually finish until tomorrow (Friday),when iVEC is represented on three separate workshops during the morning — co-chairing sessions on HPC User Support Tools and Training Best Practice in Supercomputing, plus contributing to a third workshop on Women in HPC. I suspect you’ll be able to read all about these workshops over the next couple of days. However, for me, it’s time to board my flight to LAX,

MGB, 20/NOV/14

SC14 blog – Day three, George Beckett

HPC Matters

The theme of this year’s Supercomputing conference is ‘HPC Matters’ and, in keeping with this, the drive of the program has been to emphasise how supercomputing enables scientists to tackle problems that would otherwise be impossible, in fields as diverse as particle physics, manufacturing, and health care — in line with the aims of the current iVEC Petascale Pioneers’ Program. I was fortunate enough to have an appointment-free day, which meant I could immerse myself in the technical program.

During the morning, I attended a session on petascale bioinformatics, which felt particularly pertinent given iVEC’s recent, hugely popular symposium on bioinformatics. The session was very inspiring, showcasing the significant work that has been done to migrate some of the key elements of genomics workflow — such as gene assembly and gene-sequence alignment — to large, parallel systems. The first talk looked at a new tool called Meraculous, which optimises de novo gene assembly. Early results have demonstrated fantastic scalability and speedup, turning a 140-hour wheat genome assembly process into a 15-minute one, by properly exploiting a supercomputer of the scale of Magnus. This was followed by a talk on sequence alignment, which explored how algorithmic advances were delivering a step-change over traditional MPI-BLAST workflows in terms of database size and alignment accuracy, thanks to a tool called Orion, which is expected to be released to the community in the coming months.

At lunchtime, I stopped by at the exhibition hall to check on the iVEC Student Cluster Competition team. All was looking good and, with just a few hours to go, the team had completed the majority of the tasks they had been set. It was good to see everyone in good spirits, in spite of the near-two days of non-stop work to tune their hardware to run the challenging simulations set by the judges. I also stopped by the Australian HPC booth to listen to the now famous “Tim Tam” talks, today from VLSCI and Swinburne University.

This afternoon, I participated in the HPC Connections (HPCC) workshop, in which I had been invited to talk about the Pawsey Supercomputing Centre and our role in the SKA telescope. The HPCC workshop, which focuses mainly on HPC developments in Asia, attracted a large and interactive audience of 60 or so delegates, who were treated to a program that included multi-Petaflop programming on the Tianhe-2 system (the Chinese National University of Defense Technology (NUDT) system that currently holds the number one position on the Top 500), work at Argonne National Laboratories to create next-generation batteries for electric cars and, of course, a reprise of the Pawsey project and SKA pre-construction. It was good to meet some of iVEC’s colleagues from the HPC in Asia consortium and to build on our links with those supercomputing centres that are among our nearest neighbours.

I’m sad to say that tomorrow is the last day of SC’14 — for me, at least. However, as my flight is in the evening, I’ll get to partake of the full program, plus will be involved in an SKA industry-engagement workshop at which we will provide an update on telescope pre-construction activities plus progress prospects to engage in technology prototyping.

SC14 blog – Day one, George Beckett

Monday, 17 November
by George Beckett

To anyone significantly involved in supercomputing, November is known for the annual Supercomputing conference, which is the premier meeting for all aspects of scientific and high-performance computing and typically attracts an international attendance of more than 10,000 people through its doors.

The 2014 meeting, which kicks off tomorrow (Monday 17th), is being hosted in New Orleans, Louisiana, in one of the largest convention centres I have ever seen. The Ernest N. Morial convention centre, which sits on the shore of the Mississippi River, is so large that the Supercomputing Conference occupies less than half of its floor space. In fact, SC14 is the smaller of two conferences being held there this week, alongside the American Public Health Association. Despite playing second fiddle to the APHA, SC14 is as lively and packed as ever, with a range of satellite events clustering around a core program of technical papers, workshops, tutorials, and exhibitors.

I flew into New Orleans yesterday evening, winding up a long haul from Perth to Sydney, across the Pacific, and finishing with a relatively modest 3 hour 20 minute ‘hop’ from Los Angeles to Louis Armstrong International Airport. Though, at least I made it according to plan. A technical hitch in Perth left the Pawsey student cluster team, and their leader Rebecca, delayed in Sydney for more than 24 hours. Thankfully the team is now here and working hard to set up for the competition start tomorrow.

For me, the conference kicked off early this morning, as I attended the Intel Developer Workshop — a one-day briefing on the various technologies that Intel will be showcasing on the exhibition floor this year. The morning started off with a session on Intel’s latest-generation Xeon processor — based on the Haswell architecture — which most Pawsey researchers will know as the heart of iVEC’s petascale supercomputer Magnus. The session covered the whole gamut of what makes the Haswell (or E5-2600V3, as it is officially known) better than its predecessor, the Ivy Bridge (E5-2600V2, I mean). There was lots of interesting material in the session, which will help the Supercomputing Team to help researchers squeeze the most out of their Magnus jobs.

For the second part of the morning, I attended a session on vectorisation, which is the not-so-well-kept secret for getting the best performance out of the Xeon CPU family, including Haswell. This session was very much a guide for developers on how to most easily make sure that the compiler is able to organise computations into data-parallel groups called vectors.

Following on from lunch, Intel provided a sneak preview of their plans for a new fabric in 2015, building on the IP they have acquired in recent times from — most notably — Cray, and which has driven the design of the Aries interconnect that connects Magnus’s 35,000 compute cores together.

The last session of the day looked at the emerging — for Intel, at least — area of ‘Big Data’ (data-intensive research, as we more commonly refer to it). Intel is working hard to enable Hadoop to run effectively on the parallel, network file systems that form the backbone of most HPC centres — starting to bridge the gap between traditional supercomputing stalwarts and emerging data-science domains. It will be interesting to see how their approach plays out in the coming year.

I wrapped the day up by visiting the Pawsey Student Cluster Competition team, who have settled in to their booth and are busy testing and configuring the SGI-provided cluster ready for the competition opening on Monday night. The travel hiccup meant a late night, last night, for the students, though spirits are high and the cluster is running smoothly, so keep your fingers crossed for the next three days!

ISC’14 Blog: HPC in Asia, final day

Thursday, 27 June
by George Beckett

According to the saying “time flies when you are having fun”. Well I think this is doubly true if you are also busy. This sums up ISC this year: it has been a fun and full-on experience, which has flown by in a blur. Today is the final day of the conference but for iVEC and Australia, it is one of the high points as for the first time we have participated in HPC in Asia program as a full member.

HPC in Asia was set up in 2011, as a satellite meeting to ISC, with five countries — China, Japan, Korea, Singapore, and Taiwan. The intent is to showcase the technology, innovation, and computational research that comes out of Asia, as well as to seed discussions and collaborations with Asian researchers. In 2012, the session was promoted to the main program and India joined the committee. Then, in 2013, Australia appeared in the program as a guest contributor, with Australian membership being ratified during a Committee Meeting at the SC’13 conference in Denver.

I think we have made a good show for our first session as full members. Australia-based researchers had three posters accepted (about Australian SKA work, the NeCTAR Research Cloud, and NCI’s computational support activities) and delivered one of only three scientific talks (from a member of iVEC’s supercomputing team, no less).

As the representative for Australia, I was part of the judging panel for the Best Poster award. The HPC in Asia poster session was excellent. I’m pretty confident that HPC in Asia had more posters than the main ISC’14 poster session. And it was not just quantity: the quality was also very high. Being on the judging panel meant I had to carefully study all 24 posters. This has given me a different and better view of poster sessions. I’m sad to say that I don’t often spend much time looking through the posters at these conferences, though I will be more diligent in future as I now realise what interesting subjects I may be missing.

The poster session was not without its dramas, however. We discovered early in the morning that four of the posters were missing. What happened remains a mystery: the posters were all checked on the previous evening, though – by the morning – four had disappeared. Thankfully the ISC organising committee came to the rescue, reprinting the missing posters very quickly, allowing proceedings to continue uninterrupted.

The poster quality was so high that we decided to award two Best Poster prizes to Toshihiro Hanawa (U. Tokyo) and colleagues for their work on “Proprietary Interconnect with Low Latency for HA-PACS/TCA” and to Mikiko Sato (Tokyo U. of Agriculture and Technology) and colleagues for their work on “Multiple PVAS: Parallel Task Model for the Hybrid Architecture Consisting of Many-Core & Multi-Core”.

HPC in Asia best poster winners

The poster session was sandwiched between two sessions of talks. The first session focused on site reports from each of the seven members. It was pleasing to see the level of investment and that is planned for the coming years and the commitment of the national governments across Asia, from Japan’s ambitious plans for building a 1 Exaflop supercomputer (with an estimated cost of $1.4Bn USD) by 2020, through to India’s plans to deliver a new program of world-class research outputs — in areas such as environmental science, drug discovery, and engineering — based on the opportunities of Petascale computing and beyond over the remainder of the decade. On a slightly less serious note, we believe we have identified the world’s most elevated supercomputing centre, with A*Star’s machine room being hosted on the 17th floor of the A*CRC building in downtown Singapore. I wonder how easy we would find it to install Magnus on the 17th storey of a building.

The second session focused on Asian computational-science outputs and included a well-received talk from iVEC’s Nicola Varini, who gave a presentation on his work with Ryan Mead-Hunter and Ben Mullins to generate the most realistic simulation of a monkey’s respiratory system using the first phase of the Magnus supercomputer.

Nicola Varini presenting about the canonical simulation of respiratory airflow

Next year, ISC moves to Frankfurt, and HPC in Asia has secured a prominent and longer session during the main body of the conference. This trend towards an increasing profile for the session is a testament to the energy, diversity, and enthusiasm of those who work in Asia. It is a good thing for Australia to be a part of this.

Auf Wiedersehen!

George Beckett

ISC’14 Blog: Storage Corner #2

Wednesday 25 June
By Bob Cregan

Halfway through day three and the overall story from the storage part of supercomputing is that everyone is rather worried about where to put, and access, their bits and bytes with Exa-scale systems. I have heard several people say that they are already hitting an I/O wall on problems that should be CPU bound; the problem being that disk technology is not progressing as fast as compute. There are no real answers on display here, but most are suggesting that the data must somehow get closer to the CPU and are talking about PCI attached storage as at least being part of the solution. How they would use this to create a single name space that can be utilised with existing parallel code has not yet been explained.

Another approach is being investigated by the EIOW – Exascale I/O workgroup and it sounds like it could be an interesting development for the future. They are proposing a middleware layer to optimise I/O patterns for file systems, automatic I/O profiling and modules for automating such things as compression. This made my ears prick up until one of the speakers suggested that code would have to be re-written to take advantage some of the features, at which point my interest waned somewhat. This is not the first time this conference that I’ve heard that exa-scale I/O might require throwing away POSIX, which is something of a daunting task that I think most will not consider.

I’ve also been to lots of Genomics talks and have learned a couple things. Firstly that, whilst the human genome is a few GB in size, the data created when it is processed is much larger; the machines that sequence DNA produce a bunch of fragments that come to about 300GB, and when they are pieced together using a reference the size balloons to about 3TB. Quite why this happens was not explained. If anyone knows then please explain. Regardless of the reasons, Genomics data is already large and growing fast; next generation sequencers will increase the volumes even more.

Secondly I learned that Genomics processing is very often not very efficient. Code was written by postgraduate biologists several years ago in high level languages and is I/O and memory intensive. Vendors like it; personally I think a rewrite might be in order.

Talking of interpreted languages. I went to a talk on “R” in the HPC space and it looks like enthusiasm for it is unabated; this will no doubt make many sysadmin’s hearts sink, but I fear that as a language it is too accessible and easy to use for its use to do anything but grow. Power to the people!

Bob

ISC’14 Blog: Pawsey and SKA Presentations day

Tuesday, 24 June
by George Beckett

SKA Science Data Processor Industry Update

ISC’14 Blog: The exhibition is officially open

Tuesday, 24 June
by Karina Nunez

And the graphics finally arrived.

It was Monday morning, the HPC in Asia poster had to be on display by 11am, our booth needed to be ready to receive people from 3pm, and there was still no sign of my lost tube with the graphics. I was getting really stressed by that point when, at 7.45 am, reception called to let me know that a delivery from the airport was waiting for me at reception. Hence, we have poster and banner for the booth. A good way to start the day!

HPC in Asia poster session will run for the next four day. Starting today at hall 5

The iVEC poster is the first one of 23 posters (and I may be biased but it also looks especially good amongst the others). The iVEC poster was put on display, the banner and promotional material from iVEC, NCI and VLSCI were installed on the booth, and all was ready for our visitors by 11am.

It was a long day, especially as I was suffering from jet lag for the first time in my life, which had been driving me crazy since day one in Leipzig. The feeling that all your senses are out-of-sync was strange. But yesterday afternoon the excitement of the exhibition opening diluted that feeling and got the best from both Tegan (from NCI) and me.

The exhibition hall opened its door from 3pm, but it was not until 6pm, when all the presentations finished, that the majority of participant started visiting all the booths. For them it was also an opportunity to network and chill out with a glass of wine and some canapés.

Visitors were especially interested in the NCI koala. He was the star of the booth, that and the fact that NCI is recruiting.

NCI’s Koala

We received lots of positive feedback regarding the iVEC and NCI videos, which may translate into interesting contacts for the iVEC Head of Visualisation, Paul Bourke, and more Drishti users.

Today, I am planning to visit all the other booths to find out more about their experiences at ISC14 and to learn what other centres have planned for the Supercomputing conference later this year.

I am looking forward to the Thursday session HPC in Asia, where Dr. Beckett and Dr. Varini will deliver their presentations as Australia’s representative. If you are around join us at Hall 4 at 10am. More details here.

Karina

ISC’14 Blog: Storage Corner

Monday 23 June
By Bob Cregan

On the data/storage front ISC’14 has been quiet so far with one exception. The Texas Advanced Computing Centre (TACC) has presented their plans for a data-centred cluster called Wrangler at the BoF 02 session. It is a cluster designed to run applications that have I/O profiles that would upset regular parallel file systems- random I/O on millions of small files and databases are normally suggested as examples.

Their approach is a variation of the normal scratch/archive arrangement. Wrangler will have a 10PB, geographically replicated “conventional” file system which will hold the permanent copy of the data. No one asked what “conventional” was, but it was assumed by me at least that this meant a parallel file system of some description. This file system would be Infiniband connected to 98 compute nodes.

Then there is to be 500TB of storage available on the compute nodes for the processing element of the cluster. This storage is made from a very interesting type of SSD that I’ve not heard of before, produced by DSSD, a company recently acquired by EMC. Essentially a type of PCI attached NAND flash, this storage is accessed via an API that bypasses the Linux kernel block queues and provides extremely efficient access to the solid state devices. If the application cannot use these APIs for some reason then an optimised block device is available that does use them, and thus makes conventional file system access possible. They have demonstrated Lustre and HDFS access, and there should be no reason that others could not work also. The scheduler would presumably arrange for the appropriate type of access to be made available at run time.

The cluster is designed for Hadoop and other “fringe” HPC applications such as R, GIS apps and databases, both relational and NO-SQL and they expect impressive performance, considering the amount of storage, from the processing file system – 1TB/s and 250 Million IOP/s. However the system is not yet set up so we will have to wait to see if these are realisable in practice.

Bob Cregan
Data Infrastructure Specialist