Big Data Financial Analytics: Creating Business Value for Tier 1 Telecoms

“I do not know how I may appear to the world, but to myself I seem to have been only like a boy playing on the sea-shore, and diverting myself -- now and then finding a smoother pebble or a prettier shell than ordinary, whilst the great ocean of truth lay all undiscovered before me.”

Sir Isaac Newton (1643 — 1727)
perhaps the world’s most influential scientist

Kids finding cool pebbles and shiny shells at the seashore.  Sounds like the mission of an analytics company.  I’ll bet that’s more fun than finding needles in haystacks :- )

What exactly is an analytics company?  They’re actually pretty rare, but with the era of affordable big data platforms upon us, I suspect more and more analytics firms will begin to sprout up.

SAS Institute, one of the most successful high tech companies ever, is probably your classic analytics firm.  Over the years, whenever a Tier 1 carrier in the Americas or Europe needed help with churn reduction or subscriber analytics, they would often turn to SAS.

Everybody classifies SAS as a software firm, but I don‘t agree.  Sure, it employs software to do its exploratory work, but SAS is not really known for any particular software product it leaves behind.  SAS isn’t a consulting firm either, though throwing industry-savvy consultants and systems experts at problems is key to its work.

A few weeks ago I drove down from Pennsylvania to see another analytics firm, TEOCO, a telecom analytics firm, at its Fairfax, Virginia offices outside Washington DC.

I knew TEOCO’s marketing people pretty well, but I was particularly looking forward to this meeting because I would get a chance to converse with their top executives including Atul Jain, the CEO and founder of TEOCO, who I had never actually met before.

They sat me down in the middle of a 25-foot long conference table, then surrounded me with a dozen people from their top executive and marketing team.  What?  Am I going to be grilled with tough questions?  Thankfully, no.  Most of the people were there to listen in and contribute when they could; the conversation was led by Atul and General Manager, John Devolites.

It was great.  We dispensed with PowerPoints and other formalities and just talked.  I turned on my audio recorder and listened in on some of the best big carrier “war stories” I’d ever heard.  TEOCO’s perspective on the telecom analytics business is equally enlightening I think.

Given the upward trajectory of its business, its expansion into Europe, its vast experience working with Tier 1 telcos, and the “kids at the seashore” excitement I felt from their executive team, TEOCO has more than a fighting shot of becoming the SAS Institute of telco financial analytics.  Read the interview and see if you don‘t agree.

Atul, the last three acquisitions TEOCO has made have been all over the map.  First it was Vero Systems in 2006, a supplier of Least Cost Routing solutions.  That deal made sense to me because optimal routing fits nicely with your historic focus on cost management.  But then you bought TTI Telecom, a service assurance software vendor out of Israel, and then Schema, a specialist in wireless spectrum.  It’s kind of hard to figure out what the last two acquisitions have to do with financial and cost analytics, which is your specialty.

Atul Jain: Dan, our acquisitions only seem scattered if you assume our objective is to fill a particular hole in a portfolio of applications.  We actually don‘t think that way.  We are an analytics company who looks to deliver fast ROI projects to customers.  Only secondarily do we deliver on-going applications.

TEOCO’s Acquisitions of Vero Systems and TTI Telecom

Acquiring Vero Systems taught us an important lesson: the closer we get to the network, the closer we are to the place where telcos put a lot of time, money and effort.  With Vero, we could monitor that network from the standpoint of completing calls and routing them cost effectively.

It’s true, TTI Telecom and TEOCO are coming from different worlds.  TTI’s value proposition to the carrier is on the operational or OSS side.  By contrast, TEOCO delivers value on the BSS side.

But acquiring TTI was very exciting for us because they gave us a perspective we were missing — network element data from a fault and performance management point of view.  We now have access to network elements we have never spoken to before.  And so our challenge now is to take the OSS information and create a business value out of it.

And with TTI we are playing to our strength: analyzing huge volumes of data.  We are looking at TTI’s operational data and figuring out what may be going wrong so we can an ROI or something that will have a financial impact for a customer.

John Devolites: Dan, what do you think is the hardest problem for a company like us to deal with?

Dan Baker: Finding the right people who understand the systems you’re looking at?

John Devolites: Yes, that’s on the right track.  Actually our toughest challenge is simply getting hands on the data we need.

Hardest Job: Gaining Access to Data

So what we have done is to either buy companies or build products that get us access to data that is very difficult to get your hands on.

Give you an example.  On the cost management side, many of the North American carriers run their invoices through us.  That gives us a good outlook on costs.  And costs are nice: you can do a lot of good things with cost -- even figure out how to reduce it.  But costs alone don‘t really give you the complete picture of what’s going on.

Vero’s knowledge was pivotal because if it gave us access to CDRs.  Now we had CDRs before but we never really had CDRs at the deeper level we got when we acquired Vero.  Why?  Because with Vero every call that gets routed we can see a cost component and see a price.  That’s the way the wholesale model works.

Now you can certainly ask a telephone company to give you access to that data, but they almost never do that.  Best case: they spend 6 months trying to find the one guy who knows how to get what you need.  We know of a larger carrier whose top network guy doesn‘t quite know who to get his CDRs from.

Our strategy, then, is pretty simple.  We want to have applications that get in the way of data and that give us access to that data.  And it’s much easier to get access to the data by acquiring an application that already exists in the market than for us to go to a customer and ask for it.

If we go to any of the major carriers and ask them for access to their subscriber database, what’s the chance they will give it to us?  But if we’re already in the business of getting quality by subscriber data , that becomes a lot easier.

So our mission is to get access to data then enrich it to a level where it’s become more meaningful and shows people information in a way they haven‘t seen it before.

In the end it’s all about data access.  Getting applications into a position where we can collect that information so we can do something with it.  That’s the secret sauce.

	Thanks John.  I get It now.  When you buy a company, their current application is of secondary value.  It’s the ROI business application you develop on top of the data sources where the real value lies.

Atul Jain: Every time we figured the secret was to invest in probes to go after specific data, the answer turned out to be “no”.  Our goal is to get data streams, mostly data streams that have not been used for business value generating activities and have very valuable information associated with them.

That’s how we create the ROIs.  It’s pretty cool.  I mean for us guys who love data, it’s the best thing in the world.

	I understand why TTI Telecom was attractive now, but what about the Schema acquisition?

Atul Jain: Well, Schema gives us access to another kind of data — data in the Radio Access Network (RAN).  And this gives us access to dropped calls.  Those c alls didn‘t go through and are not known because they never made into CDRs.

In the Schema application, that’s the primary focus, knowing how well the radio access network is working.  Once again, TEOCO had access to logs we never had access to before.  And the logs give you access to all the dropped calls and bad quality calls.

That’s the data that’s used to optimize a RAN network.  Earlier we were focused on the calls that were actually completed, but now we see all the calls even the ones never completed.

Now does that play into customer retention?  Absolutely, because you can determine which customers you are apt to lose based on your bad quality and dropped call statistics.

As we move to LTE, they will see a slow session and sporadic session in the RAN and this will become a more important factor from a subscriber quality point of view.  And knowing about the quality of service in detail will be the key to churn reduction.

	A year ago I heard TEOCO present at the Telestrategies conference about what you did at a tier 1 U.S. provider.  It’s actually pretty amazing to be able to implement a company-wide profit assurance solution like that.  The idea of getting a large operator to cooperate across many internal organizations is maybe the most impressive part.

John Devolites: The biggest mistake you see integrators make is to try to do everything as a big bang project.  You can‘t just say, “Here’s the end goal, so let’s go after it.” You have to use an incremental approach.  It’s got to be phased, very clear, and measure results after Phase 1, Phase 2, etc.

Phased Projects, Measured Results

And we generally do that by data sources.  First phase is switch records.  Second phase is billing records, for example.  And we make sure we have clear cases so we can measure the return and compare it to the beginning state.

A nirvana solution delivered all at once is doomed to fail.

Every phone company we deal with in North America operates the same way.  There has to be a business case.  That’s where it starts.  We can come in and tell you what great things we can do, it all boils down to a business case that is going up to senior management for approval.

And in virtually every deal we have done recently, there is some promise where they have a way out of the program if we don‘t make our numbers.  And that is incremental — each phase.

So when Atul talks about ROI, the foundation of TEOCO is the ROI equation.

And we tend to love business cases because they show pretty well what we do.  And it’s interesting how clients tend to get greedier during the process.

If every time I give you a dollar, you get three back, you’d think that was a good deal.  Well, no.  It’s doesn‘t work that way.  Because now they want $5 back for every dollar of investment!  Generally we try to get a 90 day ROI on the short projects.  But on the analytics projects, they tend to be longer because the data sources are harder to get at and set up.

	John , I know you guys are champions of big data analytics, but when your data set runs into billions of records, isn‘t a statistical sampling just as good?  Why is looking at all the data important?

John Devolites: As an aside, when we acquired Vibrant Solutions several years ago, we found out they actually sold the first (Serial number 0001) Neteeza box (now IBM), so you could say our roots in big data go back a ways.

Switch-to-Bill Analytics

Let me give you a case study.  One of our large customers had an issue with switch-to-billing information.  The switch said one thing and the billing information said another.

Now if you wanted to do a deep every-CDR investigation back in 2006, your only choice was Teradata.  It was the only data warehouse that could handle that kind of scale.  But by the time you paid licenses to Teradata and went through all the effort to insert yourself into the carrier’s data warehouse ecosystem, you could never make any money as an analytics firm.

When we eventually did get the project to do the work using our Netezza box, we took all the switch records and billing records, and pulled them together.  The analysis work paid for itself very quickly.  In many cases, it turned out the switches didn‘t have CDRs turned on.  They ended up having to clean up various network elements to get it right.

But one of the problems was their sampling methods didn‘t identify underlying issues.  You could get into an argument with a statistician (there’s one sitting next to you [Atul] (laughing in the room) , but the correct answer is that nothing substitutes for analyzing all the data.

And part of the reason is there’s no argument about what statistical method you used — it is what it is.  If you start off with a billion minutes on the switches and you end up with 900 million in billing, you definitely want to know why.  And the way you do that is look at both sets of records and figure out why that’s actually occurring.  Some of it was legitimate.  Turns out that in a few of their states, you got an exemption if you were a Native American.  Same goes if you were blind, disabled, or over 65, you didn‘t have to pay for a phone.  And the list went on.

So we ended up finding about 150 of these exemptions that people had forgotten about.  And nobody had ever reporting on how much those exemptions were costing the business.  So when you go back to the Public Utility Commission (PUC) [regulator] and say, this is what people asked us to do, and here’s what it’s costing us, that has a big impact.

Traffic Pumping

Traffic pumping is a big issue that if a carrier doesn‘t do things right, it can really turn into some serious revenue issues.  This is an area where we saved carriers hundreds of millions of dollars.  But you can’t actually get there without the technology to see the record detail.

For instance, if you did sampling, you’d end up sampling the wrong place.  And the traffic pumpers would set up operations so they could traffic pumping overnight.  They set up a partnership with a local LEC in North Dakota who gave them some phone numbers to use.  And that’s where they set up their porn and chat lines to use those phone lines.

Well, many of the largest operators are paying for it.  The number got to be in the $300 to $400 million a year range for just about all the Tier 1 providers.

Atul Jain: Actually, it’s the legal advantages of using all the data that really seals the deal.

The Legal Advantages of Big Data vs.  Sampling

Let’s say someone is doing traffic pumping and you’ve decided to sue them.  Now when you’re in court, a statistician stands up and says: we sampled this data and found out that X% of the traffic was traffic pumping.

If you extrapolate the numbers and say the money owed is $20 million, I don‘t think the court is going to buy that.  The court is going to say: show me the evidence.  Don’t give me sampling that adds to $20 million.  Show me the actual records that add up to $20 million.  Don‘t base you number on a 2% sample that was only $40,000.

Now believe it or not, the statistical analysis is quite valid, but getting a court to buy into that is another story.  Frankly, the statistical approach isn‘t admissible in court anyway.  We’ve been a major source for this type of information, so our guys are the ones doing the expert testimony.  The statistical guys doing sampling are just chewed up by the defense lawyers.

Another problem with the statistical method is a lot of calls are dropped, and there’s a difference here between intra-state and inter-state calls that doesn‘t show in the sampling.  So the problem would not have been caught if you relied on sampling alone.

I’ve always considered TEOCO to be delivering most of your value in the cost management or inter-carrier reconciliation space.  But the stories I’m hearing now in switch-to-bill and traffic pumping are revenue assurance and fraud issues.

John Deloites: Often the problems large carriers have would be simple to solve if the huge scale of their operations didn‘t get in the way.

Analysing the Outliers of Wireless Usage

For one large carrier we went in and examined two months of records to look at what their top 100 subscribers were doing.  You’d think they’d have the ability to do that, but they didn‘t at the time.

So we ran off the top 100 wireless subscribers and found the top 20 had 20,000 minutes of usage a month.  Now that’s a lot of minutes a month.  I’m not even sure there are 20,000 minutes in a month.  Anyway, TEOCO called the numbers to investigate and we found out it was a taxi dispatch service in Seattle, Washington.  Each taxi driver basically just turned their mobile phone on all day long and used it as a walky-talky.

So this shows another reason for not sampling.  Looking at all the data, you have a comprehensive view of the outliers.  With sampling you might miss a few of those.

Sampling is basically like using training wheels.  That isn‘t the way fraudsters play the game in North America.  They play at the individual phone number level.  And if you can’t get down to that level, you’re not going to detect those bad guys.

---

Editor's Note: TEOCO promises to contribute a byline story detailing a few of its customer analytics stories.  I‘m going to hold them to that promise :-)