Email a colleague    

April 2014

Pumping Crude Intelligence: How to Manage Telco Big Data before you Monetize It

Pumping Crude Intelligence: How to Manage Telco Big Data before you Monetize It

The geologists who work at big energy firms like Exxon and BP know with great certainty where huge reservoirs of natural gas and oil lie 5 miles below ground.

But it wasn‘t until advanced technologies came along -- like deep water drilling and fracking -- that these giant firms could economically bring that crude energy to the surface and put it to good use powering the planet.

Mobile telecoms face a similar challenge.  Their time- and location-sensitive big data can profitably be sold to marketers in other industries.  Solution firms like Zettics and comScore even offer a turnkey way to analyze and enrich that data so telcos don‘t even have to touch it.

But a key bottleneck still exists: finding a way to access and manage these huge data volumes effectively.  How do you economically pump the data in, store it, and filter it so it’s easier to work with?

Well, one company who has specialized in massaging telco data since 1999 is Sweden-based DigitalRoute.  The firm’s claim to fame is in BSS where it OEMs its mediation solution to billing heavy-weights like Netcracker and Comverse.

But now, the great excitement around mobile data monetizing has brought DigitalRoute full-square into the OSS side too.  So here to explain this trend and the many new data management challenges he’s seeing in the market is Thomas Vasen, the firm’s Product Management and Marketing VP.

Dan Baker: Thomas, it would be great if you could explain a little bit about DigitalRoute’s history in the mediation business.

Thomas Vasen: Sure, Dan.  We were actually quite fortunate.  In the late 90s when DigitalRoute was formed, we could ride the wave of commodity hardware and blade servers emerging back then.  And we’ve always designed our software to do end-to-end in-memory processing so that nothing touches disk while we process it.  We take whole files in at once to give us a performance advantage, and we feel that’s still an important edge for us.  We built one of the first truly convergent systems and can today handle batch and real-time data in a hybrid way.

Altogether we have over 300 active customers, most of whom have come to us through our OEM partners, though we serve some 60 operators direct.

What is it about your mediation approach that these OEMs like?  Are you low cost?  What’s the secret sauce?

It is more complicated than people guess.  We have focused on mediation and it requires expertise to do it correctly and efficiently.  Over the years we’ve added more power and speed as you’d expect, but one point we try to emphasize is being partner-friendly.

To start with, the software is fully re-brand-able.  Also, our partners build their own agents to take our technology and make it compatible with their proprietary solutions.  We have also built one-step-upgrade types of functionality where you can correct, upgrade, and even easily roll back features.  And our APIs are useful for embedding the software in a larger system.  The customer doesn‘t even recognize that mediation from us is a separate module.

I understand an OEM you’ve recently attracted is SAP.  What’s that all about?

Yes, we are the data access point and collection engine for SAP who takes telco data and sells it as intelligence to advertising firms and marketing agents for example.  The idea is to take raw geo-location and radio access network data and feed it enriched with demographics information from CRM, but anonymized of personal identifiers to SAP’s HANA databases.  There they analyze it for marketing purposes and detect interesting behavioral patterns.  Their service is called SAP Consumer Insight 365; it really unlocks and monetizes the network data.

Now the typical marketing agency who consumes this data might serve the leading department store in Sweden, as an example.  And that retailer wants to know what kind of people walk by their stores but never come inside.  That intelligence is of great value to them because if they cross-correlate demographics with location data, they can market to them much more effectively.

Now SAP is an IT giant with huge capabilities.  So why can‘t they do the data gathering and filtering themselves?

Basically they can‘t speak telco data at a deep level.  Telco has very specific data formats, different aggregation rules, especially when you get into the radio access network world.  It tricky to extract that.  SAP and other IT vendors lack the capacity to process it accurate and efficiently.

What we do is intelligently merge telco data streams and put everything together into something manageable.  This is why SAP is working with us.

What kinds of data do you capture?

Well, our claim to fame, of course, is CDR processing for BSS applications, but DPI is an important new data stream we work with today.  We don‘t capture DPI directly: we take in the records reported by the DPI engines or probes of Ericsson, Sandvine, Allot, Tek, Procera, and others.  We then cross reference to identify what kind of service the customer has or the URLs being served.

This DPI data is invaluable in measuring customer experience.  It’s not just about knowing what customers are doing, but also looking at dropped calls and sessions and figuring out what the customer did before and after the drop.

This data is all available but most mobile operators are not using it right now because the volumes are so massive, and they haven‘t learned how to manage it.

Why is that a problem?  I mean, commodity hardware is here and standards like Hadoop are making access to large data sets more affordable.

Hadoop’s vision is great, but the idea of plucking out data streams and saving them forever is not really working in the world of truly massive telco data.

We think the answer is to be real clever about the questions you ask of the data.

Most operators know exactly what they need to do with their network data.  It’s pretty structured.  The challenge is to filter it down to a reasonable size so it can be easily digested and worked with.  Accessing the raw data alone doesn‘t work well because you are dealing with insane data volumes.

You need to prep and strategically look at your data.  And it’s hard because in telecom the data off the switch is usually in binary form, not ASCII.  So you can‘t just dump it into Hadoop and expect magic to happen.  Data has to be converted to be able to correlate and unlock its value.

To give you an idea: a typical tier 2 operator in Europe takes in about 200 million records a day on the BSS side.  And a tier 1 operator in the U.S. might be dealing with 6 or 7 billion mediation records a day for BSS.  But that’s peanuts compared to the OSS side: an LTE network for a U.S. tier one is 2 million records a second!

And yet, DigitalRoute is able to manage that volume on a small set of standard hardware blades.

OK, we’ve talked about big data monetization, but what about the BSS side of the house?  Any action happening there?

As you know, Dan, the billing paradigm has shifted in a big way from minutes to bytes.  And all the BSS logic around charging based on distance, area codes, time of day, etc. is not needed anymore.

But where there’s a lot of action in understanding the data usage -- counting billions of small records.

We call this capability “service control”, which starts with identifying the data records and routing them to a bucket that either counts subscriber usage or content provider usage.  And our system allows you to create these buckets dynamically with an unlimited hierarchy.

Speaking of content, that’s a huge topic of interest.  In particular, there’s a big tussle over exactly who is going to pay for streaming digital video.

Yes, and a lot of early action there is happening in the U.S. market where some recent court rulings have relaxed the net neutrality laws somewhat.

And to me, these new regulations introduce a little common sense and give the service providers a little relief.  The problem is that video subscription services like Hulu and Netflix are consuming an enormous amount of bandwidth across high speed Internet modems.  Sandvine says that Netflix alone is eating up 30% or more of the North American data traffic.

So one remedy that’s emerging -- and AT&T was the first to announce it publicity — is to enable a reverse charging model similar to 1-800 and toll-free numbers.  The idea is that the company who owns the content pays for the delivery charges.

And this is really the way it should be.  After all, when you buy a book from Amazon, you also pay for the postage to have that book delivered to your home.

This trend is also playing into the cable industry too.  Very recently Netflix and Comcast came to an agreement where Netflix agrees to pay an undisclosed amount for delivering its content to Comcast subscribers.

So anyway you slice it: either tracking usage or the number of times a user passes a retail store, sounds like there’s plenty of opportunity for DigitalRoute and its OEMs to stay busy.

We certainly think so, Dan.

Copyright 2014 Black Swan Telecom Journal

Thomas Vasen

Thomas Vasen

Thomas Vasen is VP Product Management & Marketing at DigitalRoute.  He has over 17 years operational experience with product and service development in telecoms.  Most recently head of solution development at service assurance pioneer Polystar OSIX, before that he was an entrepreneur in a series of Voice over IP start-ups at operators in Europe.

At B2 Bredband AB, the largest ETTH broadband operator in Sweden, he was responsible for the setup and operation of the first primary line local-loop replacement service launched on SIP technology in the world.  Thomas has studied at the Erasmus University in Rotterdam and at the London School of Economics.   Contact Thomas via

Black Swan Solution Guides & Papers

cSwans of a Feather

  • Pumping Crude Intelligence: How to Manage Telco Big Data before you Monetize It interview with Thomas Vasen — Mobile telecoms are eager to sell their time- and location-sensitive big data to marketers in other industries.  But a key bottleneck exists: finding a way to efficiently access and manage the huge data voluimes involved.  In this article, a supplier of mediation software explains his firm’s approach to tackling the problem.
  • When Big Data is Too Big: The Value of Real-Time Filtering and Formatting interview with Rick Aguirre — The volume of telecom network traffic is often so huge it outstrips the ability of even “big data” engines to analyze it fast enough.  In this article, you’ll learn about a business that filters and formats very large data sets and delivers the relevant data for applications like: data monetization, network optimization, network peering monitoring, and unstructured data storage.
  • Crusaders Clash: The Battle for Control of Telco 2.0 Service Delivery, Billing & Policy interview with Stephen Rickaby — Mobile is still reeling from the shock of being taken out of the driver’s seat in terms of services offered on the handset.  But  will telecoms make a services come-back?  This interview with an expert in the thick of Telco 2.0 transformation action discusses the strategic issues involved and also analyzes Oracle’s recent moves to acquire Acme Packet and Tekelec.
  • Telecom Mediation: Time to Move Back into the Limelight? by Dan Baker — While mediation technology remains crucial to assurance applications, solution vendors have been relatively quiet in recent years.  This article points to reasons why the mediation market may soon get more active.  Among the factors discussed are: consolidation, big data, group merger activity and the offload of mobile transactions to cheaper platforms.
  • Putting a Database at the ‘Nexus’ of Service and Revenue Assurance interview with Michael Olbrich — Closing the B/OSS gap — getting network-facing OSS systems to communicate with customer-facing business systems is one of telecom’s greatest challenges.  This article shows the virtue of unifying B/OSS data and  processes under a single database.  Also discussed is the issue of vendor management and choosing trusted supplier to grow with.

Related Articles

  • Subex’s New HyperSense AI-Platform Aims to Turn Telcos into Agile, Analytics-Driven Ubers and Rakutens interview with Rohit Maheshwari — An expert explains why telcos must swiftly adopt the AI/analytics methods of hyperscaler firms.  Giving telco examples where cross-company data sharing can make a difference, he explains the 12 key features of a new platform designed to deliver more agile and collaborative intelligence across a telco.
  • Subex’s COO: Machine Learning, Disruption & Adaptive Biz Models to Impact Telecoms in 2020 interview with Shankar Roddam — In our dynamic and digital-driven world, telecoms and enterprises face many business risks.  So what can you do to plot a successful future?  Hear 5 prophecies on 2020 telecom trends from the Chief Operating Officer of Subex.
  • TELUS Analytics Users Get Productivity Boost from Internal Team of Data Access & Showcasing Experts-for-Hire interview with Mange Kumarasamy — How a large organization with hundreds of analytics users gets help from an  internal Big Data team who sources multiple back-end databases; builds tailored reports; and drives campaigns that answer strategic questions for users.
  • Non-Verbal Speech Analytics: Monitoring Voice Calls in Real-Time for Customer Care, Sales, Retention & Onboarding interview with Yoav Degani — Non-verbal speech analysis studies the emotional context of voice qualities like intonation, tone, emphasis and rhythm.  A pioneer in voice analytics explains how its technology benefits customer care, sales, retention and onboarding.
  • Telecom CVM: From Scattered Campaigns to Unified & Consistent Communication with Customers interview with Cretièn Brandsma — Despite the many failures Customer Value Management has faced in telecom, CVM’s future is very hopeful.  A carrier expert explains why telecoms have faltered, how customer experience programs can be revitalized, and where telecoms should invest in better tactics and technology.
  • The Key to Driving 4G Profit: Sell Value, Not Bandwidth by Miri Duenias — Are you struggling to earn a profit on your 4G investments?  Many operators are failing today on the marketing side.  But aligning 4G products with a customer’s personal preferences and desires provides the necessary sizzle to boost sales and earn a handsome ROI.
  • Will Real-Time Decisioning Save Big Data Analytics from Overblown Hype? interview with Tom Erskine — Telecom analytics is more than just collecting and analyzing data.  It’s also about taking action — correct action — often in real-time and across a complex provisioning environment.  In this interview you’ll hear how next best actions are creating value in retention and upselling through a more flexible, business-process driven approach.
  • A Big Data Starter Kit in the Cloud: How Smaller Operators Can Get Rolling with Advanced Analytics interview with Ryan Guthrie — Medium to small operators know “big data” is out there alright, but technical staffing and cost issues have held them back from implementing it.  This interview discusses the advantages of moving advanced analytics to the cloud where operators can get up and running faster and at lower cost.
  • The Customer Engagement Era: How Personalization & Backend Integration Leads to a Richer Mobile Biz interview with Rita Tochner — How does a mobile operator move its subscribers to higher levels of spending and profit?  Fierce competition, social media scrutiny, and the high cost of new networks all conspire against these goals.  In this interview, however, you’ll learn how engaging better with customers, getting more personal, and being more sensitive to their individual needs is the path forward.
  • Telecoms Failing Badly in CAPEX: The Desperate Need for Asset Management & Financial Visibility interview with Ashwin Chalapathy — A 2012 PwC report put the telecom industry on the operating table, opened the patient up, and discovered a malignant cancer: poor network CAPEX management, a problem that puts telecoms in grave financial risk.  In this interview, a supplier of network analytics solutions provides greater detail on the problem and lays out its prescription for deeper asset management, capacity planning and data integrity checks.
  • Batting for More Churn Reduction and Revenue Assurance Home Runs interview with Peter Mueller — What’s it like to transform an IT shop to big data and cloud?  In this interview, the CTO of a boutique revenue assurance explains how his firm made the leap.  He shows how project-oriented programs and working with carrier customers to explore RA and churn reduction “hunches” is where much of the action is.
  • History Repeats: The Big Data Revolution in Telecom Service Assurance interview with Olav Tjelflaat — The lessons of telecom software history teach that new networks and unforeseen industry developments have an uncanny knack for disrupting business plans.  A service assurance incumbent reveals its strategy for becoming a leader in the emerging network analytics and assurance market.
  • From Alarms to Analytics: The Paradigm Shift in Service Assurance interview with Kelvin Hall — In a telecom world with millions of smart devices, the service assurance solutions of yesteryear are not getting the job done.  So alarm-heavy assurance is now shifting to big data solutions that deliver visual, multi-layered, and fine-grained views of network issues.  A data architect who works at large carriers provides an inside view of the key service provider problems driving this analytics shift.
  • The Shrink-Wrapped Search Engine: Why It’s Becoming a Vital Tool in Telecom Analytics interview with Tapan Bhatt — Google invented low cost, big data computing with its distributed search engine that lives in mammoth data centers populated with thousands of commodity CPUs and disks.  Now search engine technology is available as “shrink wrapped” enterprise software.  This article explains how this new technology is solving telecom analytics problems large and small.
  • Sharing Intelligence, Services, and Infrastructure across the Telecom Galaxy interview with Gary Zimmerman — The telecom industry is an industry of sharing.  In fact, the rise of mobile broadband is driving a greater reliance on real-time intelligence, services trading, and infrastructure exchange.  In this article, a leading info exchange provider explains the value of its services portfolio and points to other interoperability and sharing ideas under development.
  • Data Monetization: Why Selling Intelligence is a Hot New Revenue Stream for Mobile Carriers interview with Joe Levy — Data monetization is a revenue dream come true for mobile carriers: a highly profitable sideshow where the carrier analyzes and sells data it already collects for other purposes.  In this article you’ll learn how operators monetize their data through use cases in corporate advertising and media branding.
  • Harvesting Big Data Riches in Retailer Partnering, Actionable CEM & Network Optimization interview with Oded Ringer — In the analytics market there’s plenty of room for small solution firms to add value through a turnkey service or cloud/licensed solution.  But what about large services firms: where do they play?  In this article you’ll learn how a global services giant leverages data of different types to help telcos: monetize retail partnerships, optimize networks, and make CEM actionable.
  • Raising a Telco’s Value in the Digital Ecosystem: One Use Case at a Time interview with Jonathon Gordon — The speed of telecom innovation is forcing software vendors to radically adapt and transform their business models.  This article shows how a deep packet inspection company has  expanded into revenue generation, particularly  for mobile operators.  It offers a broad palette of value-adding use cases from video caching and parental controls to application-charging and DDoS security protection.
  • Radio Access Network Data: Why It’s Become An Immensely Useful Analytics Source interview with Neil Coleman — It’s hard to overstate the importance of Radio Access Network (RAN) analytics to a mobile operator’s business these days.  This article explains why the RAN data, which lives in the air interface between the base station and the handset --  can be used for a business benefit in network optimization and customer experience.
  • Back Office Streamlining to Enterprise Support: The Many Flavors of Wireline Analytics interview with Tom Nolting — Mobile analytics gets plenty of press coverage, but analytics is just as crucial for wireline operators.  In this article, a billing VP at a leading wireline operator discusses several diverse uses of analytics in billing, enterprise sales/retention, and network partner margin assurance.
  • Analytics Biology: The Power of Evolving to New Data Sources and Intelligence Gathering Methods interview with Paul Morrissey — Data warehouses create great value, yet it’s now time to let loose non-traditional big data platforms that create value in countless pockets of operational efficiency that have yet to be fully explored.  This article explains why telecoms must expand their analytics horizons and bring on all sorts of new data sources and novel intelligence gathering techniques.
  • B/OSS Mathematics: The Quest to Analyze Business Problems & Drive Operating Decisions interview with Matti Aksela — Analytics is the glory of mathematics brought to practical use.  And in telecom, analytics has merely stratched the surface of its full potential.  In this article, you’ll learn how machine learning is being combined with the power of CDR number crunching to optimize mobile top-ups, control churn — and in the future, help telecoms make critical network and operating decisions.
  • Leveraging the RA/FM Platform to Deliver Business Insights to Finance & Marketing by Amit Daniel — Carrier professionals using RA and fraud management tools are getting requests from internal customers who want the role of RA/FM platforms expanded to deliver up-to-date analytics data for finance and marketing purposes.  This article advocates a cross-product layer to serve such broader use cases.  The effect would be to transform the existing RA/FM platform into a combined business protection and business growth analytics engine.
  • A Mobile Marketer Service: Bridging Personalization & B/OSS Flowthrough interview with Efrat Nakibly — Marketing analytics is a prescriptive program for driving  actions such as sending a timely promotion to a mobile subscriber.  But completeness demands that you also be able to provision that treatment, qualify the promotion, and keep billing fully in the loop.  This article shows how a managed services program can deliver such an end-to-end process and manage customer life cycles on a one-to-one basis.
  • Science of Analytics: Bringing Prepaid Top Ups & Revenue Maximization under the Microscope interview with Derek Edwards — Prepaid subscribers are the customers that carriers know the least about.  The operator is not interacting with prepaid customers on a monthly basis.  You’re not sending a bill, nor do you have detailed profiles on these customers, especially in the developing world where customers are buying SIMs at a grocery store.  This interview explains how contextual marketing meets the unique analytics challenge of prepaid customers.
  • Connecting B/OSS Silos and Linking Revenue Analytics with the Customer Experience by Anssi Tauriainen — Customer experience analytics is a complex task that flexes B/OSS data to link the customer’s network experience and actions to improve it and drive greater revenue.  In this article, you’ll gain an understanding of how anayltics data needs to be managed across various customer life cycle stages and why it’s tailored for six specific user groups at the operator.
  • Profitable 3G: China’s Mobile Operators Monetize Networks with Retailers & Partners interview with Kevin Xu — Mobile operators are at the center of explosive growth in wireless services.  But to exploit this opportunity requires IT ingenuity and a broader view on how the mobile user can be served.  In this article you’ll learn the innovative techniques Chinese operators use to monetize 3G networks via analytics and partnerships with retailers, social networks, and advertisers.
  • Customer Analytics: Making the Strategic Leap From Hindsight to Foresight interview with Frank Bernhard — Are your company’s analytics programs scattered?  Is there a strategy in place for customer analytics?  This interview with a leading telecom analytics consultant explains why strategy and planning around the analytics function is crucial to getting your money’s worth.  Topics discussed include: hindsight vs. foresight; an advanced analytics program; and the interface sophistication required to support high end vs. low end analytics users.
  • Meeting the OTT Video Challenge: Real-Time, Fine-Grain Bandwidth Monitoring for Cable Operators interview with Mark Trudeau — Cable operators in North America are being overwhelmed by the surge in video and audio traffic.  In this article you’ll learn how Multi Service Operators (MSOs) are now monitoring their traffic to make critical decisions to protect QoS service and monetize bandwidth.  Also featured is expert perspective on trends in: network policy; bandwidth caps; and  customer care issues.
  • Analytics Meditations: The Power of Low-Cost Hardware and the Social Network Within interview with Ken King — Analytics didn‘t arrive yesterday.  Data warehousing and BI have been in the telecom vocabulary for twenty-five or more years.  In this interview, you’ll gain a perspective on why “big data” changes the game and why social network (or social circles) analysis promises the next level of insights.  Other interesting topics include: segmenting the analytics market, engaging with carrier clients, and upgrading from older- to newer-style methodologies.
  • LTE Analytics:  Learning New Rules in Real-Time Network Intelligence, Roaming and Customer Assurance interview with Martin Guilfoyle — LTE is telecom’s latest technology darling, and this article goes beyond the network jargon, to explain the momentous changes LTE brings.  The interview delves into the marriage of IMS, high QoS service delivery via IPX, real-time intelligence and roaming services, plus the new customer assurance hooks that LTE enables.
  • Shared Data Plans: The Challenge of Managing a Family of Pricing, Revenue Assurance, Fraud, and Network Policy Issues by Amit Daniel — Verizon Wireless‘ recent announcement of its move to shared data plans for families shook the mobile industry.  In this column, cVidya’s Amit Daniel shines a spotlight on the knowhow and analytics tools that operators now deperately need to offer the right  shared data price plans, ensure bandwidth throttling is handled correctly, and address new fraud concerns.
  • Analytics Guru: Are Telecoms Ready for the Biz Intelligence Explosion? interview with John Meyers — Business intelligence is evolving from the creation of dashboards and reports to taking action based on a deep knowledge of the environmental context.  The article explores the implications of “big data” in terms of IPTV, storage requirements, hardware, event collection, and deep packet inspection.
  • Social Networking for Telecoms: How To Enlist Friends and Family as Mobile Marketers interview with Simon Rees — Social Network Analysis (SNA) is about exploiting data on “friends and family” connections to combat churn and win new CSP business.  The article explores how an analysis of the ebb and flow of CDRs, phone calls, and messages, can identify key influencers and drive powerful marketing campaigns.
  • Making the Strategic Leap From Billing to Merchandising interview with Humera Malik — Today billing/charging technology has progressed to the point where the usage intelligence, the charges, the user behaviors, and the analytics can all come together in near real-time.  This article discusses the organizational and marketing strategies that enable a operator to create a true “merchandising” system that can revolutionize a CSP’s business.