Max Cottica, is Head of Data Science and Solutions with AIB, and has joined the DatSci 2017 Judging Panel. This year Max will be judging the SME of the Year Award and the Student of the Year Award. We caught up with him to hear is thoughts on Modern Data:
When I was young summers were long and predictable. It would get so hot at times that me and my rascal mates visited often a fountain in the middle of the village. I was trying to collect as much water with my hands as possible to get the most out of the refreshing liquid. But my hands were so small, water would spill out all over the place. When I think of modern data, that is the image that comes to mind. Every drop of water we can’t collect equals a missing opportunity to place a sale, to support a customer, to update our reference data. You can understand how concepts like data warehouses, lakes, reservoirs, marts and ponds are getting in the way. Data needs to be collected, analysed, transformed and used for predictions in real time. Saving it on disk for one, two, three or more times cannot give us a millisecond response to our questions. Modern data is a river and we need to learn how to fish in it.
This is the reason why technologies like Kafka, the Kappa architecture, Flink, Apex, Storm, Beam, Spark streaming, Solr, Elastic and Samza are getting more and more curious adopters. What we did to data for the good part of the last quarter of a century does not scale with the river anymore. Data needs to sit in memory, needs to be summarised to statistics, needs to be written to disk only in certain contexts or we will not be able to find the competitive edge that differentiates the successful companies from everybody else.