Collective Intelligence in the life sciences

Here is the Table of Contents of Programming Collective Intelligence (a great book by the way)

• Introduction to Collective Intelligence
• Making Recommendations
• Discovering Groups
• Searching and Ranking
• Optimization
• Document Filtering
• Modeling with Decision Trees
• Building Price Models
• Advanced Classification: Kernel Methods and SVMs
• Finding Independent Features
• EVOLVING INTELLIGENCE
• Algorithm Summary
• Third-Party Libraries
• Mathematical Formulas

I was going through this the other day, and it occured to me that bioinformaticians/computational biologists program collective intelligence every day. The collective intelligence comes from evolution and the intricate evolutionary relationships that drive our knowledge about the world around us; the knowledge built into biology. We use many of the same methods described above, and many of the same paradigms. In a way, you can say that bioinformaticians are way ahead of the web 2.0 crowd (although we still don’t know how to write intuitive interfaces). The description of the book on Amazon says

This fascinating book demonstrates how you can build Web 2.0 applications to mine the enormous amount of data created by people on the Internet. With the sophisticated algorithms in this book, you can write smart programs to access interesting datasets from other web sites, collect data from users of your own applications, and analyze and understand the data once you’ve found it.

Over the years, life scientists have generated tons of data, and deposited it on the internet. While we do a great job os trying to leverage evolutionary relationships with cutting edge algorithms, we do a relatively poor job of harnessing the collective intelligence from the biological data distributed around the web.

The modern web is programmable. There are those who would say it’s always been that way, but I beg to differ. The biggest development in recent years, has been our ability to use the entire web as our platform through all the mechanisms available to us, REST APIs, RSS delivery, microformats, etc. I tend to look at the life science web as an independent subsystem living within the WWW. I just wish it were more programmable. The tide is beginning to change (the NCBI Universal Resource Locator would be one example), but as Duncan mentioned last year and I lamented earlier, the general trend, where available, is towards big web services. Given the amount of changes seen in life science data, WS-* interfaces are likely to break and usually just to “heavy” to be readily programmable. In other words, while there are a small group of bioinformaticians who are harnessing the web the way it should be harnessed, by and large, we are still limited due to the lack of simple REST APIs. With all due respect to IBM and the lovers of service oriented architectures (which are primarly WS-* based), we could do things much faster, cheaper and more effectively if we could harness data and focus on deriving knowledge from it.

So let’s move towards a new form of collective intelligence, where we can harness the power of evolution and of the network of public and private resource that are available via simple API’s, microformats and what have you. Let us empower the creative developers out there to make the life science web even more powerful.

Further reading
Will data be our undoing
Biology, computing and web services
Biological content, access and monetization

Technorati Tags: Collective Intelligence, Bioinformatics, Programmable Web