Big Data – Are we tech-ready yet?

June 16, 2013

“Speed is the key”

I have been hearing this statement since my school days. All these years, the processor and Internet speeds have picked up phenomenally. However, these improvements don’t seem to dampen the spirits of this statement, for it gets younger with each passing day, just like Mr. Benjamin Button. I still hear people uttering it with varied degrees of emotion, accentuated by the situation in which they are in. Now, as we push ourselves into the next phase of information age with exabytes of data being collected and prepared to be analysed, I often ask myself about our readiness in terms of computing technology and its speed.

One of the applications, that I am currently working on, deals with the analysis of data fetched from multiple sources. It helps big companies with revenue predictions based on budget allocations. These predictions are churned out by an array of a complex set of algorithms. Breathtaking concept! However, it suffers a serious problem with speed. Some of the conceptual features that I have proposed could not see the daylight for the need of calculation speeds. Even the state-of-the-art servers and computing devices don’t seem to help. With a simple ‘what if’ flow taking about 2 minutes, the application had to cut down on a range of interactions that were in demand from the users. Result – a potentially powerful tool suffering a serious case of UX relevance.

I am often told that “speed cannot be increased beyond this”. The issue, as I understand, is not totally with the ability of the business layer architecture or the developers. The issue is more with the machines that are being used for the calculations.

That’s why the question, are we tech-ready yet to analyse big data? I personally feel we are not there yet, fully. Unless we get over the architectural problems of our computing devices and kiss goodbye to Von Neumann bottlenecks and processor-centric computing, we might not reach there. But, I am more than hopeful that the ‘change’ would push itself in as the demand is here to stay. The ‘change’ that harbingers in-memory database technologies like SAP HANA. Bring them on, we need more!

Leave a Reply