Techaisle Blog

Various organizations define Big Data differently. Some use “petabytes of data” as a benchmark to isolate big data from other normalized and structured data sets that exist within an organization. However, this measure quickly boxes big data analytics into the large enterprise market segment. Small and mid-market businesses certainly do not have this extent of data but Big Data still relevant for them. In fact Big Data solutions are more relevant for Small and Mid-Market businesses. However, it will take some creativity on the part of solution providers to make Big Data accessible, easy to use and comprehend for segment that constitutes 97 percent of global businesses.

Cloud computing started as an enterprise play, however, it was quickly discovered that SMBs will be the more relevant target segment with a faster path to adoption. Similarly, as Virtualization market started getting fully penetrated within the enterprises, vendors shifted their focus to the SMBs with some very creative solutions. As far as big data is concerned SMBs are starting to show interest and even adoption. However, there is a stark difference in approaches between mid-market businesses and small businesses. While mid-market businesses are experimenting with bespoke solutions, small businesses are gravitating towards a multi-tenant, aggregated and federated big data solution that has a mix of publicly available data and their own internal data.

It is expected that in year 2016, global SMBs would spend US$1.6 Billion on big data solutions exhibiting a growth rate that is faster than what was exhibited by cloud computing solutions. Cumulatively between now and end of 2016, SMBs itself would have shelled out US$3.9 billion on big data solutions. This spending includes hardware, software and services.

So why are many big data solution providers ignoring SMBs? Simply put, because of perceived complexity and inability to create bite-sized messaging that directly address SMBs pain-points. But they should not forget that business intelligence has now become one of the fastest solutions to be adopted by SMBs. If done right, Big data address three key pain points of SMBs: Increase sales, Efficient operations, Improve Customer service.

Promise of Superior Decision Making

Let us take Techaisle’s recent global mid-market businesses’ Big Data Adoption & Trends study which clearly shows that the promise of superior data-driven decision making is motivating 43 percent of global mid-market businesses to either invest in or investigate Big Data technology. Out of these, 18 percent of mid-market businesses are actively investing in big data related projects. The possibilities of analyzing a variety of data sources, producing action-driven business insights is too big to ignore for these businesses.

Similar to cloud, the attitude towards Big Data is transitioning from “Over-Hype” to “Must-Have” technology with the size of business. Even within the businesses that consider big data to be over-hyped, 29 percent think that it will be an important part of their business decision making process in the future.

Extracting Business Perspectives

Business intelligence by itself has provided enough business insights, however, mid-market businesses are now looking for extracting business perspectives to drive superior decisions and ultimately achieve superior results.  Extracting business perspectives has become important as they rethink their marketing strategies because mobility, social media, and other transactional services have increased the number avenues for connections with their customers and partners.

CRM solutions had first established the analytics for analyzing customer data. However, the data was mostly two-way transactional data. This changed when customers began visiting business websites to explore, browse and perhaps make purchases thus leaving behind a trail of information. IT vendors and mid-market businesses figured out the need to analyze the data and combine it with transactional information.

However, everything changed with the onset of social media, blogs, forums, wikis and opinion platforms where the identification of false positives and negatives became difficult and knowledge about the customer and resulting segmentation became an inaccurate undertaking.

Big data analytics presents the possibilities of connecting together a variety of data sets from disconnected sources to produce business insights whether be for generating sales, improving products or detecting fraud.

It is therefore not surprising that global mid-market businesses are turning towards big data analytics to analyze social media data, web data, customer and sales data along with click-stream machine generated data and even communications data in the form of emails, chat, voicemails.

Leap of Faith or Solution Readiness

Analyzing data from diverse sources leads a mid-market business to naturally consider linking structured and unstructured data. This also drives them to evaluate and select the technology that can be used for simplified implementation. Simplified implementation is important because mid-market businesses do not yet have in-house capabilities to analyze unstructured data and those that have them consider the capabilities at best rudimentary.

Big data therefore is a major leap of faith for mid-market businesses resulting in treating big data analytics projects usually as separate to the existing analytics within the business. More aggressive adopters are planning to use big data analytics along with other analytics in a coordinated manner so that one does not become an inhibitor for the other.

In recent years technology and technology options have evolved extremely rapidly for an average business to understand, evaluate, purchase and implement. The complexity gets further exacerbated with lack of experience, lack of skilled manpower and innate difficulty in identifying external consultants that would be the most right fit for their business objectives and budget availability.

In spite of challenges, the study shows that there have been some successes when business units, IT & data analysts exhibit extraordinary alignment. Our study shows that mid-market businesses typically started their big data journey in one of four ways. Highest success rates for project implementation and generating new insights have been achieved when IT and data analysts work with external consultants from project inceptions.

SMBs as the Path to Big Data Simplicity

The global SMB spend on big-data related deployments will cross US$1.0 billion in 2013 which is a 32 percent increase from 2012. SMBs are still experimenting to see if big data analytics can provide newer insights into their operations and better knowledge about their customers. It is still very early days for small and mid-market businesses to fully embrace big data but they are planting the seeds in terms of re-architecting their IT infrastructure to plan for the future. But we believe that SMBs may very well race ahead of enterprises with their deployments as technology becomes simpler and consultants become experienced.

 

Techaisle’s global mid-market businesses’ Big Data Adoption & Trends study shows that the promise of superior data-driven decision making is motivating 43 percent of global mid-market businesses to either invest in or investigate Big Data technology. Out of these, 18 percent of mid-market businesses are actively investing in big data related projects. The possibilities of analyzing a variety of data sources, producing action-driven business insights is too big to ignore for mid-market businesses.

Big Data requires a certain level of IT sophistication and a history in the linear investment in Information Technology enablers to be successfully. While these factors predispose larger accounts to Big Data, the competitive imperative to understand customers, innovate products and improve operational efficiencies has already started to reach down to the mid-Market, forcing a search for how to leverage primary and secondary data that is generated by the business.

The current and planned investment represents a sizable opportunity considering that the segment is relatively new and requires a certain level of IT sophistication and a history in linear investment in Information Technology enablers to be successful. North America has both the largest market and the highest level of investment in Big Data overall in SMB and mid-market segments. Mid-Market attitude towards Big Data transitions from “Over-Hype” to “Must-Have” technology with the increase in employee size. However, nearly one-fourth of lower mid-market businesses consider big data to be over-hyped and yet 29 percent think that it will be an important part of their business decision making process in the future.

Business intelligence by itself has provided enough business insights, however, mid-market businesses are now looking for extracting business perspectives to drive superior decisions and ultimately achieve superior results.  Extracting business perspectives has become important as they rethink their marketing strategies because mobility, social media, and other transactional services have increased the number avenues for connections with their customers and partners.

In addition to understanding customers, mid-market businesses are also considering big data analytics as an important initiative to help them improve operational efficiencies.

Techaisle’s study shows that there are many different tactical objectives for deploying big data projects but the top among them are sentiment monitoring, generating new revenue streams & improving predictive analytics. It must also be said that businesses have figured out that there is a lot of publicly available data which could also be analyzed to their advantage.

The mid-market businesses actively investing in big data technologies are expecting some clear cut benefits from big data analytics such as increased sales, more efficient operations and improved customer service. These objectives differ slightly by different geographic regions. As the growth rates continue to lag in mature economies, the pressure to increase revenue grows resulting in developing robust analysis and extracting insights from all sales and customer data including transactions.

When specifically asked about preferred deployment choice in terms of on-premise vs. cloud, mid-market businesses are unsure as they are still navigating through their technology options. However, Hadoop dominates as the preferred platform but confusion exists.

In terms of analytics skill-set and long-term vision, the potential of linking structured and unstructured data sources to create new business insights is being considered very useful but at the same time mid-market businesses are not really prepared for it. In fact one-third of mid-market businesses agree that linking structured and unstructured data would be very useful for big data analytics but over 70 percent mention that they have either none or very limited capabilities of analyzing unstructured data. This is where they are turning to external help for guidance.

Needless to say, survey reveals that big data deployment is posing tremendous challenges. Technology confusion, lack of skilled resources and potential unclean data are being considered as the biggest roadblocks for big data project implementations. Big data technology and its far-reaching capabilities are being viewed by mid-market businesses as very complex resulting in very steep learning curves.

In spite of challenges, the study shows that there have been some successes when business units, IT & data analysts exhibit extraordinary alignment. Highest success rates for project implementation and generating new insights have been achieved when IT and data analysts work with external consultants from project inceptions.

Detailed Global Mid-Market Big Data Adoption and Trends report is available for purchase. Details are given here.

Underutilization and the complexity of managing growing data sprawl have spawned several trends during the last several years. Data-as-a-Service (DaaS) is one such trend which represents an opportunity to improve IT efficiency and performance through centralization of resources. DaaS strategies have increased dramatically in the last few years with the maturation of technologies such as data virtualization, data integration, MDM, SOA, BPM and Platform-as-a-service.

Within the corner offices of business heads, data scientists and analysts several questions are being asked:

• How to deliver the right data to the right place at the right time?

 

• How to “virtualize” the data often trapped inside applications?

 

• How to support changing business requirements (analytics, reporting, and performance management) in spite of ever changing data volumes and complexity?

In the early years most of DaaS initiatives were limited to financial services, telecom, and government sectors. However, in the past 24 months, we have seen a significant increase in adoption in the healthcare, insurance, retail, manufacturing, eCommerce, and media/entertainment sectors. This is because of massive amalgamation of extracting continuous insights from structured and unstructured data, liberation of data restricted and protected within silos to the enterprise level and the express desire to conduct real-time analytics.

Businesses are looking to solve tough data and process integration challenges as they once again begin to invest in new business capabilities. Data as a Service (DaaS) is based on the concept that the fragmented transaction, product, customer data can be provided on demand to the user regardless of geographic or organizational separation of provider and consumer. Additionally, the emergence of PaaS and service-oriented architecture (SOA) has rendered the actual platform on which the data resides also irrelevant.

Data as a Service (DaaS) has many use cases:

1. Providing a single version of the truth;

 

1. Integration of data from multiple systems of record

 

1. Enabling real-time business intelligence (BI),

 

1. Federating views across multiple domains;

 

1. Improving security and access;

 

1. Integrating with cloud and partner data and social media;

 

1. Delivering real-time information to mobile apps

Data as a Service (DaaS) brings the notion that data related services can happen in a centralized place – aggregation, quality, cleansing, enriching and offering it to different systems, applications or mobile users, irrespective of where they were. DaaS is a major enabler of the Master Data Management (MDM) concept.

Master Data Management is the Holy Grail in data management.  The focus for most businesses is on the single version of the truth or Golden Source “Product”, “Customer”, “Transaction” and “Supplier” data.  This is because:

• Fragmented inconsistent product data slows time-to-market, creates supply chain inefficiencies, results in weaker than expected market penetration, and drives up the cost of compliance.

 

• Fragmented inconsistent Customer data hides revenue recognition, introduces risk, creates sales inefficiencies, and results in misguided marketing campaigns and lost customer loyalty.

 

• Fragmented and inconsistent Supplier data reduces efficiency; negatively impacts spend control initiatives, and increases the risk of supplier exceptions.

MDM provides the plumbing that enables DaaS solutions. This plumbing allows for:

• Agility & Time to Market – Customers can move quickly due to the consolidation of data access and the fact that they don’t need extensive knowledge of the underlying data. If customers require a slightly different data structure or has location specific requirements, the implementation is easy because the changes are minimal.

 

• Cost-effectiveness – Providers can build a base with data experts and outsource the presentation layer, which makes for very cost-effective report and dashboard user interfaces and makes change requests at the presentation layer much more feasible.

 

• Data quality – Access to the data is controlled via data services, which tends to improve data quality, as there is a single point for updates. Once those services are tested thoroughly, they only need to be regression tested, if they remain unchanged for the next deployment.

 

• Cloud like Efficiency, High availability and Elastic capacity. These benefits derive from the virtualization foundation —one gets efficiency from high utilization of sharing physical servers, availability from clustering across multiple physical servers, and elastic capacity from the ability to dynamically resize clusters and/or migrate live cluster nodes to different physical servers.

We find that there is a common process that is appearing within the mid-market and customer customers focused on enabling and MDM strategy. It is the data logistics chain consisting of data acquisition, data stewardship, data aggregation and data servicing.

There is a sudden and dramatic shift in how data is handled in businesses as they are shifting away from a hierarchical, one-dimensional enterprise data warehouse initiative with fixed data sources to a fragmented network. This phenomenon has caused ripple effects throughout the old data logistics network.  Data-as-a-Service (DaaS) at its core is addressing this problem of fragmentation soundly enabled by MDM.

 

Many organizations are starting to think about “analytics-as-a-service” (no acronym allowed) as they struggle to cope with the problem of analyzing massive amounts of data to find patterns, extract signals from background noise and make predictions. In our discussions with CIOs and others, we are increasingly talking about leveraging the private or public cloud computing to build an analytics-as-a-service model.

The strategic goal is to harness data to drive insights and better decisions faster than competition as a core competency.  Executing this goal requires developing state-of-the-art capabilities around three facets:  algorithms, platform building blocks, and infrastructure.

Analytics is moving out of the IT function and into business — marketing, research and development, into strategy.  As a result of this shift, the focus is greater on speed-to-insight than on common or low-cost platforms.   In most IT organizations it takes anywhere from 6 weeks to 6 months to procure and configure servers.  Then another several months to load configure and test software. Not very fast for a business user who needs to churn data and test hypothesis. Hence cloud-as-a-analytics alternative is gaining traction with business users.

The “analytics-as-a-service” operating model that businesses are thinking about is already being facilitated by Amazon, Opera Solutions, eBay and others like LiquidHub.  They are anticipating the value migrating from traditional outmoded BI to an Analytics-as-a-service model.  We believe that Amazon’s analytics-as-a-service model provides a directional and aspirational target for IT organizations who want to build an on-premise equivalent.

 

Situation/Problem Summary: The Challenges of Departmental or Functional Analytics

The dominant design of analytics today is static or dependent on specific questions or dimensions. With the need for predictive analytics-driven business insights growing at ever increasing speeds, it’s clear that current departmental stove-pipe implementations are unable to meet the demands of increasingly complex KPIs, metrics and dashboards that will define the coming generation of Enterprise Performance Management. The fact that this capability will also be available to SMBs follows the trend of embedded BI and dashboards that is already sweeping the market as an integral part of SaaS applications. As we have written in the past, the move to true mobile BI can be provided as an application "bolt-ons" that work in conjunction with an existing Enterprise Applications or as pure play developed from scratch BI applications that take advantage of new technologies like HTML5. Generally, the large companies do the former through acquisition with existing technology and integration and with start-ups for the latter. Whether at the Departmental or Enterprise level, the requirements to hold down costs, minimize complexity and increase access and usability are pretty much universal, especially for SMBs, who are quickly moving away from on-premise equipment, software and services.

After years of cost cutting, organizations are looking for top-line growth again and finding that with the proliferation of front-end analytics tools and back-end BI tools, platforms and data marts, the burden/overhead of managing, maintaining and developing the “raw data to insights” value chain is growing in cost and complexity - a balance that brings SaaS and on-premise benefits together is needed.

The perennial challenge of a good BI deployment remains: it is becoming increasingly necessary to bring the disparate platforms/tools/information into a more centralized but flexible analytical architecture. Add to this the growth in volume of Big Data across all company types and the challenges accelerate.

Centralization of analytics infrastructure conflicts with the business requirement of time-to-impact, high quality and rate of user adoption - time can be more important than money if the application is strategic.  Line of Business teams need usable, adaptable, and flexible and constantly changing insights to keep up with customers.  The front-line teams care about revenue, alignment with customers and sales opportunities. So how do you bridge the two worlds and deliver the ultimate flexibility with the lowest possible cost of ownership?

The solution is Analytics-as-a-Service.

 

Emerging Operating Model:  Analytics-as-a-Service

It’s clear that sophisticated firms are moving along a trajectory of consolidating their departmental platforms into general purpose analytical platforms (either inside or outside the firewall) and then packaging them into a shared services utility.

This model is about providing a cloud computing model for analytics to anyone within or even outside an organization.  Fundamental building blocks (or enablers) like – Information Security, Data Integrity, Data and Storage Management, iPad and Mobile capabilities and other aspects – which are critical, don’t have to be designed, developed, tested again and again. More complex enablers like Operations Research, Data Mining, Machine Learning, Statistical models are also thought of as services.

Enterprise architects are migrating to “analytics-as-a-service” because they want to address three core challenges – size, speed, type – in every organization:

• The vast amount of data that needs to be processed to produce accurate and actionable results

 

• The speed at which one needs to analyze data to produce results

 

• The type of data that one analyzes - structured versus unstructured

The real value of this service bureau model lies in achieving the economies of scale and scope…the more virtual analytical apps one deploys, the better the overall scalability and higher the cost savings. With growing data volumes and dozens of virtual analytical apps, chances are that more and more of them leverage processing at different times, usage patterns and frequencies, one of the main selling points of service pooling in the first place.

 

Amazon Analytics-as-a-Service in the Cloud

Amazon.com is becoming a market leader in supporting the analytics-as-a-service concept. They are attacking this as a cloud-enabled business model innovation opportunity than an incremental BI extension.  This is a great example of value migration from outmoded methods to new architectural patterns that are better able to satisfy business’ priorities.

Amazon is aiming at firms that deal with lots and lots of data and need elastic/flexible infrastructure.  This can be domain areas like Gene Sequencing, Clickstream analysis, Sensors, Instrumentation, Logs, Cyber-Security, Fraud, Geolocation, Oil Exploration modeling, HR/workforce analytics and others. The challenge is to harness data and derive insights without spending years building complex infrastructure.

Amazon is betting that traditional enterprise “hard-coded” BI infrastructure will be unable to handle the data volume growth, data structure flexibility and data dimensionality issues.  Also even if the IT organization wants to evolve from the status quo they are hamstrung with resource constraints, talent shortage and tight budgets. Predicting infrastructure needs for emerging (and yet-to-be-defined) analytics scenarios is not trivial.

Analytics-as-a-service that supports dynamic requirements requires some serious heavy lifting and complex infrastructure. Enter the AWS cloud.  The cloud offers some interesting value 1) on demand; 2) pay-as-you-go; 3) elastic; 4) programmable; 5) abstraction; and in many cases 6) better security.

The core differentiator for Amazon is parallel efficiency - the effectiveness of distributing large amounts of workload over pools and grids of servers coupled with techniques like MapReduce and Hadoop.

Amazon has analyzed the core requirements for general analytics-as-a-service infrastructure and is providing core building blocks that include 1) scalable persistent storage like Amazon Elastic Block Store; 2) scalable storage like Amazon S3; 3) elastic on-demand resources like Amazon Elastic Compute Cloud (Amazon EC2); and 4) tools like Amazon Elastic MapReduce.  It offers choice in the database images (Amazon RDS, Oracle, MySQL, etc.)

 

How does Amazon Analytics-in-the-Cloud work?

BestBuy had a clickstream analysis problem — 3.5 billion records, 71 million unique cookies, 1.7 million targeted ads required per day. How to make sense of this data? They used a partner to implement an analytic solution on Amazon Web Services and Elastic MapReduce. Solution was a 100 node cluster on demand; processing time was reduced from 2+ days to 8 hours.

Predictive exploration of data, separating “signals from noise” is the base use case. This manifests in different problem spaces like targeted advertising / clickstream analysis; data warehousing applications; bioinformatics; financial modeling; file processing; web indexing; data mining and BI.  Amazon analytics-as-a-service is perfect for compute intensive scenarios in financial services like Credit Ratings, Fraud Models, Portfolio analysis, and VaR calculations.

The ultimate goal for Amazon in Analytics-as-a-Service is to provide unconstrained tools for unconstrained growth. What is interesting is that an architecture of mixing commercial off-the-shelf packages with core Amazon services is also possible.

 

The Power of Amazon’s Analytics-as-a-Service

So what does the future hold?  The market in predictive analytics is shifting.  It is moving from “Data-at-Rest” to “Data-in-motion” Analytics.

The service infrastructure to do “data-in-motion” analytics is pretty complicated to setup and execute.  The complexity ranges from the core (e.g., analytics and query optimization), to the practical (e.g., horizontal scaling), to the mundane (e.g., backup and recovery).  Doing all these well while insulating the end-user is where Amazon.com will be most dominant.

 

Data in motion analytics

Data “in motion” analytics is the analysis of data before it has come to rest on a hard drive or other storage medium. Due to the vast amount of data being collected today, it is often not feasible to store the data first before analyzing it. In addition, even if you have the space to store the data first, additional time is required to store and then analyze. This time delay is often not acceptable in some use cases.

 

Data at rest analytics

Due to the vast amounts of data stored, technology is needed to sift through it, make sense of it, and draw conclusions from it. Much data is stored in relational or OLAP stores. But, more data today is not stored in a structured manner. With the explosive growth of unstructured data, technology is required to provide analytics on relational, non-relational, structured, and unstructured data sources.

Now Amazon AWS is not the only show in town attempting to provide analytics-as-a-service.  Competitors like Google BigQuery, a managed data analytics service in the cloud is aimed at analyzing big sets of data… one can run query analysis on big data sets — 5 to ten terabytes — and get a response back pretty quickly, in a matter of seconds, ten to twenty seconds. That’s pretty useful when you just want a standardized self-service machine learning service. How is BigQuery used? Claritic has built an application for game developers to gather real-time insights into gaming behavior. Another firm, Crystalloids, built an application to help a resort network “analyze customer reservations, optimize marketing and maximize revenue.” (THINKstrategies’ Cloud Analytics Summit in April, Ju-kay Kwek, product manager for Google’s cloud platform).

 

Bottom-line and Takeaways

Analytics is moving from the domain of departments to the enterprise level.   As the demand for analytics grows rapidly the CIOs and IT organizations are going to be under increasing pressure to deliver.  It will be especially interesting to watch how companies that have outsourced and offshored extensively (50+%) to Infosys, TCS, IBM,  Wipro, Cognizant, Accenture, HP, CapGemini and others will adapt and leverage their partners to deliver analytics innovation.

At the enterprise level a shared utility model is the right operating model.  But given the multiple BI projects already in progress and vendor stacks in place (sunk cost and effort); it is going to be extraordinarily difficult in most large corporations to rip-and-replace.  They will instead take a conservative and incremental integrate-and-enhance-what-we-have approach which will put them at a disadvantage. Users will increasingly complain that IT is not able to deliver what innovators like Amazon Web Services are providing.

Amazon’s analytics-as-a-service platform strategy shows exactly where the enterprise analytics marketplace is moving to or needs to go. But most IT groups are going to struggle to implement this trajectory without some strong leadership support, experimentation and program management. We expect this enterprise analytics transformation trend will take a decade to play out (innovation to maturity cycle).

Shirish Netke