Over at the Money:Tech conference (O’Reilly does organize some of the coolest conferences), a couple of talks caught my eye, notably one by Michael Stonebraker on Streambase and another by Steve Skiena on using computer simulations and mathematical modeling to make bets (You can find the slides on the web site). After going through the slides, I wondered, is there anything there which would be useful for the life sciences. My first reaction was not really. Michael’s talk on processing streams of data does not really fit into the timescales that scientific data usually falls into (his other company, Vertica, on the other hand fits right in). Steve’s models also don’t really fit anywhere in the drug hunting world (at least in my brain).
But how about post marketing. We track epidemics, we track adverse events, but we don’t really do a good job. We also try and predict the success of a particular drug, both from a scientific/medical perspective and a financial one. I am sure pharma companies use advance financial modeling to try and predict drug success, so I won’t talk about that too much here, but I would like to ruminate over stream processing.
The above figure (taken from Michael’s talk), shows that in stream processing, you essentially have a stream of input data (event data) coming in from one side and coming out of the other side as alerts and actions based on certain queries. I feel that in a hospital, emergency, pandemic situation such a system might be very useful. In a safety situation, where the flow of data is not necessarily that fast, would such a system be overkill? We are constantly collecting data globally and real time results might be critical to bring up an adverse event signal.
Just some thoughts. We have been using the same old methods for a very long time and they, at least from my vantage point, need a refresh with innovative, nimble methods. It’s fascinating to see what people are thinking about in other fields. I am sure they can learn a lot from the life sciences as well.
Technorati Tags: Stream Processing, Data Analysis, Biomedical informatics
4 Comments
New to the StreamBase offering is support for 64-bit processors; including AMD Opteron. According to the vendor, 64-bit support pushes the potential performance of the SPE to over 1 million messages per second.
I don’t think stream data processing will _remain_ a niche for much longer. I think it will grow rapidly to become far more mainstream, partly because a lot of data will come from sensor networks, partly because machines will create much more data, partly because storage space grows so rapidly that all that sensor data can be kept.I think one has to consider how much of the perceived growth of stream processing is due to ~100 million USD in VC being blown on marketing by new comers to this scene. I agree that the market growing but am not convinced it really is growing fast enough to generate an open standard, yet. Niches do not support competition well, at least commercially.
I don’t think stream data processing will _remain_ a niche for much longer. I think it will grow rapidly to become far more mainstream, partly because a lot of data will come from sensor networks, partly because machines will create much more data, partly because storage space grows so rapidly that all that sensor data can be kept.I think one has to consider how much of the perceived growth of stream processing is due to ~100 million USD in VC being blown on marketing by new comers to this scene. I agree that the market growing but am not convinced it really is growing fast enough to generate an open standard, yet. Niches do not support competition well, at least commercially.
Very very interesting post..I like this one. gotta bookmark this one.
Cheers,
Buat Duit Dengan Blog
2 Trackbacks
[...] reading Streambase: Query your streaming data The CDC embraces the Semantic [...]
[...] Further Reading Streambase [...]