Streaming Analytics going mainstream?
Updated: Jan 14
A quick look at how PaaS and IaaS work in Streaming Analytics.
So lately we are working in new architectures and discovering new ways to get data ingested. We find that the faster we go the more checking we need to really do and closer to where the ingestion is rather than at the backend (old days). As we start this Kafka really took off with great credentials in handling large amounts of data with thought built in to manage fault tolerance etc. However data persistence was an issue, not that easy to really roll back the data feed (now we have storage techniques to manage this) but as we transition to the cloud new ways to do this make the comparisons a little more difficult to really manage. The way Microsoft Eventhubs manages streams is comparable but its really a service and takes advantage of cloud based technology in all the right ways. Honestly I was surprised at how well it performs (not necessarily faster) and is fast to setup. This was something I realized that drove cloud adoption earlier, in some of my data modeling sessions I would hear architects talk about not wanting to wait for servers, loading things etc and how great it is to spool up the required database in minutes with little administration. Now it looks as if this is happening in the streaming world as well. What could this mean? Well faster time to information and better architectures that are flexible. Faster? Sort of but not in the way you imagined a direct comparison would work. Still this is an area of expansion where customers are beginning to move so be prepared.