While Gartner describes a “trough of disillusionment” to describe the hangover that follows a period of commercial hype, on the IT side, I see a corresponding “mountain of ego”. Don’t get me wrong. This is not about a sales guy trying to go after the development community - one I proudly belonged to for many years and where I started my journey in this industry. But ask any developer how long it takes to code something and prepare to be amazed by how fast and easy it all is. Three budget cycles later and a couple of delay notifications and we all know better. Agile development tries to cope with this but it’s no silver bullet.
But here’s the problem. It’s one thing to develop beautiful algorithms and dazzling prototypes but what happens when the inevitable errors, exceptions and irregularities, or – worse - the continuous stream of user changes surface? These unglamorous, seemingly trivial inconveniences that seemed hardly worth factoring into the initial delivery estimates invariably wind up causing major headaches and delays. And fixing them is boring and unworthy of the self proclaimed and highly paid Data Scientist!
The reason I mention all this is that no group suffers the grave consequences of putting ego before pragmatism than IT itself. Business Intelligence 1.0 fell into disrepute for taking ages to implement, costing too much money, being inflexible and plagued with backlogs of IT requests. Our industry must avoid the situation in big data where searching for the Holy Grail of scripting nirvana gets in the way of delivering solutions on time to the business so we all can avoid a repeat failure.
When it comes to data analytics, sidestepping IT is reckless. If you thought ‘Excel Hell’ and ‘rogue spreadsheets’ led to inefficiency and poor decisions, just wait for the mayhem that ensues when ‘rogue analytics’ comes to town! Whether it knows it or not, the business needs IT to handle the data cleansing, warehousing, integration and assimilation that is vital to underpinning fast, meaningful, insightful analytics - especially when big data sets come into the picture.
All this means that IT and the business need to work together and find a rhythm instead of trying to get one over on each other. Since the business rarely gets blamed for bad IT decisions (even when it makes them) this rhythm will only happen when IT gets pragmatic and finds ways to work at the (rapid) pace of the business, especially when using big data sources. In addition to changes in culture and mentality, taking advantage of the promise of big data analytics will certainly involve IT using powerful data integration tools instead of coding scripts and hacking algorithms together. However, through these changes, IT stands to earn respect and even hero status, when the business is able to measure revenue gains and efficiency savings.
On the other hand if IT insists on putting ego before pragmatism, the business will find shortcuts, and believe you me they won’t be pretty! So don’t F* around…visit www.pentahobigdata.com to learn more.
*Cartoon drawing by Pentaho's own Steve Macfarlane