Streambase: Query your streaming data

February 15, 2008

Over at the Money:Tech conference (O’Reilly does organize some of the coolest conferences), a couple of talks caught my eye, notably one by Michael Stonebraker on Streambase and another by Steve Skiena on using computer simulations and mathematical modeling to make bets (You can find the slides on the web site). After going through the slides, I wondered, is there anything there which would be useful for the life sciences. My first reaction was not really. Michael’s talk on processing streams of data does not really fit into the timescales that scientific data usually falls into (his other company, Vertica, on the other hand fits right in). Steve’s models also don’t really fit anywhere in the drug hunting world (at least in my brain).

But how about post marketing. We track epidemics, we track adverse events, but we don’t really do a good job. We also try and predict the success of a particular drug, both from a scientific/medical perspective and a financial one. I am sure pharma companies use advance financial modeling to try and predict drug success, so I won’t talk about that too much here, but I would like to ruminate over stream processing.

The above figure (taken from Michael’s talk), shows that in stream processing, you essentially have a stream of input data (event data) coming in from one side and coming out of the other side as alerts and actions based on certain queries. I feel that in a hospital, emergency, pandemic situation such a system might be very useful. In a safety situation, where the flow of data is not necessarily that fast, would such a system be overkill? We are constantly collecting data globally and real time results might be critical to bring up an adverse event signal.

Just some thoughts. We have been using the same old methods for a very long time and they, at least from my vantage point, need a refresh with innovative, nimble methods. It’s fascinating to see what people are thinking about in other fields. I am sure they can learn a lot from the life sciences as well.

Technorati Tags: Stream Processing, Data Analysis, Biomedical informatics