Programming HPC for the domain

May 11, 2008

At Accelrys, a lot of the software I managed was in-licensed from academia. That approach allowed the company to tap into the intellectual resources of some of the smartest academic researchers in the world, but it also created a problem. One was the difference in software development practices. Some of the academic code barely had version control. But that’s the obvious one. In a new post at Computing at Scale, Bill McColl writes about Domain-specific parallel programming. Translating code parallelized for an academic setting, often under the assumption that huge clusters might be available, to an industrial setting where scaling and fault tolerance become critical, where resource availability varies widely, and speed is critical, is always a challenge. This is especially true when you’re trying to shrink wrap software and building interactive interfaces.

So in an era with more scale available, clouds to tap into, accelerators, and new data and distribution models, are we going to see a shift? I still feel that the underlying scientific research has to come from academia. They have the resources, time and incentive to do so, but I think industrial think tanks and expertise can contribute back by working with academia on advanced problems of relevance, e.g. in the area of computing. Will we tap into some of the new domain specific development being done today as a scientific community? It can’t be done by one side or the other. But rather we need to identify approaches as a community and understand what works best, without trying to duplicate efforts. Of course, we need people who understand these new methods and paradigms to implement them.

There will always be a tension between academic research efforts and commercial need. In the life sciences it is especially tough for industry specific apps to be developed from an economic point of view, which is why I believe it will have to be a joint effort.

Your thoughts?

Image via Wikipedia

Technorati Tags: High Performance Computing, Parallel Programming