Cloud Computing & Recommendation Engines

Marketing, Collaborative Filtering, Technology, Cloud Computing, Computer Clusters

From time-to-time Summer Research addresses internal projects to expand our experience beyond our traditional practice areas.  We tackled the Netflix Prize, not to win, but to prove that using simple algorithms and cloud computing, a company could affordably match the results Netflix had achieved prior to the competition.  Our goal, prove we could match Netflix’s staff of professionals, on our own, without a bench of Silicon Valley whiz-kids and top-tier PhD’s. 

In our experience, particularly in finance and risk management, many companies don’t want to and don’t need to have cutting edge analysis.  They just need to be slightly ahead of their competitors or at least in the middle-of-the-pack.  More importantly, they need to do it cost-effectively and leveraging existing company personnel – and often none of them have PhD’s in advanced mathematics.

Summer Research built a 796-CPU computer cluster using Amazon EC2.  We developed the tools to manage the cluster using a completely open source platform of Linux, R Statistical Package, MySQL, and MPI.  And we did all this at an approximate cost savings of 90% from what could have been possible just a year or two earlier.

We ran one of the simplest algorithms in collaborative filtering – something someone with even the most casual understanding of statistics would comprehend.  And we essentially matched the results Netflix had started with.  No PhD’s. No fancy algorithm’s to go awry.  And something business people could manage, comprehend, and put their trust in.