As readers of bbgm know, one of the subjects that interests me the most is computing, even though I am hardly a guru, but I’ve been around long enough and close enough to the world of computing to notice and observe various trends in this space. Perhaps the most recent trend, and it appears a new phrase in the computing lexicon is data intensive computing, something that life scientists should be very cognizant of today, as I talk about in Science Big, Science Connected. This interest in data intensive computing and recent exposure to data centers and large scale operations has led me to pay even more attention to some blogs I’ve been following for a while. Specifically, two blogs that I read religiously are those of James Hamilton and Dan Reed.
James writes (and speaks) with much knowledge and passion about data center operations and economies, especially in services, e.g. as evidenced by his post on Cloud Computing Economies of Scale. I would like to point you to the slide deck linked to in that post, specifically slides 12-13. A couple of things jump out
Scaling and availability is only achieved through partitioning and redundancy
Best hardware is never good enough, while highly reliable software evolves very slowly
Lower quality hardware in large numbers is more reliable in aggregate than high-quality hardware
Assume that software and hardware will fail frequently and unpredictably
My personal opinion is that we need to rethink some of our dearly held beliefs and take a different approach. The degree of parallelism required at exascale, even with future manycore designs, will challenge even our most heroic application developers, and the number of components will raise new reliability and resilience challenges. Then there are interesting questions about manycore memory bandwidth, achievable system bisection bandwidth and I/O capability and capacity. There are just a few programmability issues as well!
and
I believe it is time for us to move from our deus ex machina model of explicitly managed resources to a fully distributed, asynchronous model that embraces component failure as a standard occurrence.
Notice some common themes there. Programming and systems design is not exactly the core competency of life scientists, which is why seeing people leverage frameworks like Hadoop, designed for the kinds of systems described above is encouraging, but that is hardly mainstream. We still have a fondness for exotic architectures, and fancy machines and tend to write code that chews up memory and databases that aren’t necessarily scalable. There are exceptions, and more people are paying attention to designing for the kinds of systems described above. I suspect that over the next decade or so, out of sheer necessity, the community will have to adopt scalable designs, and leverage Hadoop and other frameworks, as well as GPGPU and other systems as required.
The world of scientific software needs a new breed, many of whom I happen to know from the online world that we share. People who understand the importance of APIs that interact with each other, horizontal scaling, MapReduce and similar frameworks. What we don’t have as much of is people who live and breathe concurrency and scale. Maybe it is time.
The future of big compute for big science
As readers of bbgm know, one of the subjects that interests me the most is computing, even though I am hardly a guru, but I’ve been around long enough and close enough to the world of computing to notice and observe various trends in this space. Perhaps the most recent trend, and it appears a new phrase in the computing lexicon is data intensive computing, something that life scientists should be very cognizant of today, as I talk about in Science Big, Science Connected. This interest in data intensive computing and recent exposure to data centers and large scale operations has led me to pay even more attention to some blogs I’ve been following for a while. Specifically, two blogs that I read religiously are those of James Hamilton and Dan Reed.
James writes (and speaks) with much knowledge and passion about data center operations and economies, especially in services, e.g. as evidenced by his post on Cloud Computing Economies of Scale. I would like to point you to the slide deck linked to in that post, specifically slides 12-13. A couple of things jump out
In When petascale is just too slow, Dan Reed writes
and
Notice some common themes there. Programming and systems design is not exactly the core competency of life scientists, which is why seeing people leverage frameworks like Hadoop, designed for the kinds of systems described above is encouraging, but that is hardly mainstream. We still have a fondness for exotic architectures, and fancy machines and tend to write code that chews up memory and databases that aren’t necessarily scalable. There are exceptions, and more people are paying attention to designing for the kinds of systems described above. I suspect that over the next decade or so, out of sheer necessity, the community will have to adopt scalable designs, and leverage Hadoop and other frameworks, as well as GPGPU and other systems as required.
The world of scientific software needs a new breed, many of whom I happen to know from the online world that we share. People who understand the importance of APIs that interact with each other, horizontal scaling, MapReduce and similar frameworks. What we don’t have as much of is people who live and breathe concurrency and scale. Maybe it is time.
Related articles by Zemanta
Please read this disclaimer