I recently heard someone talk about software as being operational. The context was large scale systems and how it is important not just to run hardware systems that operate at scale but also software that operates at scale. And having spent a lot of time lately in a world where operational excellence is paramount as well as the Hadoop world, the importance of operational software is becoming only more evident. As we work with very large data sets that need to be processed across 100′s, even 1000′s of commodity compute nodes, you need to run software that accepts and embraces failure. What makes the Hadoop framework work isn’t just the part that you can run map-reduce jobs, but the part where data is replicated in multiple locations, where the filesystem is distributed across a number of nodes, where hardware failure is expected. As we go towards multiple terabytes per workload, maybe someday even petabytes of data in the life sciences, understanding and appreciating operational software is going to become critical. Most of us aren’t used to doing so and it’s going to require a bit of a cultural shift to get there, but it will be fun when people figure it out.
In the meantime follow the workRajarshi is doing with Hadoop and of course work coming from people like Mike Schatz.
When software embraces failure
I recently heard someone talk about software as being operational. The context was large scale systems and how it is important not just to run hardware systems that operate at scale but also software that operates at scale. And having spent a lot of time lately in a world where operational excellence is paramount as well as the Hadoop world, the importance of operational software is becoming only more evident. As we work with very large data sets that need to be processed across 100′s, even 1000′s of commodity compute nodes, you need to run software that accepts and embraces failure. What makes the Hadoop framework work isn’t just the part that you can run map-reduce jobs, but the part where data is replicated in multiple locations, where the filesystem is distributed across a number of nodes, where hardware failure is expected. As we go towards multiple terabytes per workload, maybe someday even petabytes of data in the life sciences, understanding and appreciating operational software is going to become critical. Most of us aren’t used to doing so and it’s going to require a bit of a cultural shift to get there, but it will be fun when people figure it out.
In the meantime follow the work Rajarshi is doing with Hadoop and of course work coming from people like Mike Schatz.