Email a colleague    

August 2015

Test Call Generators: An Essential Test & Debugging Tool in Mobile Billing Assurance

Test Call Generators: An Essential Test & Debugging Tool in Mobile Billing Assurance

Is it even possible to develop complex enterprise software without a powerful debugging environment?  Sure, you could do it, but getting to relatively error-free code would take you forever.

So what about the world of mobile operator billing?  Actually, that world is far more complex because it combines all the familiar IT headaches — databases, servers, clouds, and code — with telecom billing’s own weird issues: proprietary switches, event-driven tariff rules, regional network variations, international roaming, mediation processes, changing tariffs, new technologies and many other complexities, not to mention regulatory requirements.

Now mobile network revenue assurance (RA) systems that use the CDR as their starting point are vital, but they miss all sorts of problems that can happen south of the mediation system.

In short, an “active” test call generator (TCG) can see problems that a “passive” RA system is simply blind to.  Yet the TCG’s role in preventing, diagnosing, and debugging mobile billing problems has not been fully appreciated up to now.

Here, then, to brief us on the virtues of TCGs in billing and revenue assurance is Steffen Öftring, Revenue Assurance Manager at SIGOS.  Steffen discusses the  strengths of active RA systems, the differences between over-the air calls and core call injection, and the importance of global test call networks in roaming RA.

Dan Baker: Steffen, let me start with a question that’s probably on a lot of people’s minds.  For operators who have already spent a million dollars or more on a revenue assurance system, why should they spend more money to acquire a test call generator solution from SIGOS?

Steffen Öftring: Actually, Dan, I get asked that question all the time.  And the first thing I say is that this is not an either/or choice.  We often work in tandem with the major revenue assurance vendors to complement and enhance the capabilities of their system for customers.  And frankly, the revenue assurance interface to provisioning is an area we do not address.

But on the billing verification side of revenue assurance, we bring great value because we generate test CDRs on the network specifically to find and diagnose billing problems.  And because we control every aspect of those call events, we know exactly how many CDRs should be generated.  And we can check the metering, call duration, or download volume.

In fact, of the three most common errors in billing, test calls are needed to identify two of them: a passive RA system doesn’t even pick those errors up.  So here’s a diagram and text that explains what I mean:

  1. Incorrect Rating — A typical revenue assurance system excels at checking for incorrectly rated CDRs.  But because its starting point is CDRs coming off the switch, it has no visibility over errors caused by the switch or events prior to the call getting to the switch.
  2. Metering Issues — If the switch is not correctly recording the duration of the call or the size of a download, that’s a big issue and a passive revenue assurance system can’t identify that.  But test calls, because of their active and end-to-end nature can find where these metering issues exist.
  3. Lost CDRs — Finally, there are many cases where an MSC is not recording CDRs at all.  Here again, active test calls come to the rescue.  By comparing test call data with a daily or twice-daily CDR feed, you can readily detect switches that are malfunctioning.  We know exactly that from location A, we made 150 calls yesterday.  So if there are only 140 calls in the CDR database, we know there are 10 CDRs missing.
Passive vs. Active RA

A passive system can manage all the live traffic.  But our approach is to focus on test call generator traffic.  And it’s much less traffic, so the deployment is much cheaper than a passive solution.

Some of these revenue assurance systems cost $5 million or more and they might take 6 months to deploy.  But an average TCG from SIGOS costs around $0.5 million and can be up and running in eight weeks.  Because we have our own TCG system covering over 200 countries we can also provide a fully managed service which requires no equipment to be installed by an operator.

Now I understand one of the crucial areas where TCGs are useful is in testing calls for roaming.

This is very true.  If, for example, a US subscriber roams to Europe and has a 24-hour unlimited pass for voice and data service, we can test that with our system.

So the strength of a roaming solution like this is the breadth of global coverage in your TCG network.  Now SIGOS stacks up very well because we have 430+ global customers who use our platform for quality of service, fraud detection, roaming and revenue assurance.

Our biggest customers have over 200 test locations covering all technologies in one country so they can cover all the switch sites they have in a domestic market.

And any one of these local test call generator systems can be connected to our GlobalRoamer network footprint.  So we can combine domestic revenue assurance with global RA with everything managed from the customer’s own system.

How does TCG revenue assurance operate at a typical client?

Well, let’s use a US based operator as an example.  They need to supply us with one SIM card for every individual tariff.  And from that tariff we are going to use our expertise to create a test plan that tests all the risk points of this tariff.

Let’s suppose the tariff is a flat rate bundle with 50 SMS and 500 voice minutes and 5 Gigabytes of data.  Once we know the tariff rules, our test plan makes sure we consume all the free bundle elements.  And TCG operates with that tariff plan over the course of a daily or weekly basis, and we keep consuming.  So if all is working well, the 501st voice minute should cost something.  And we verify that.

Now each call we make creates a live CDR in the US-Operator’s network.  So we bring together the live CDRs that our customer supplies to us.

The trick is to then verify accuracy by comparing different data sources.  Here, we generally take three feeds: an un-rated switch feed with all test SIM CDRs; a feed from the mediation system; and a feed from the billing system.

Now aside from having a global network for testing roaming, what else should buyers be looking for as they evaluate TCG systems?

I would definitely say the rating capability is first.  Remember, after we do the CDR comparison to find the missing CDRs, metering errors, and correctness of the content, our second step is to re-rate the calls.

Actually, the rating engine is one of the hardest components to get right.  At SIGOS it took us three years to develop a rating engine to input live CDRs and re-rate them.

And the reason it took so long is the tariffs these day are immensely complex.  Flat rated tariffs are simple, but you’d be surprised how creative people in marketing can get inventing new tariffs.

For example, in South Africa they had an hourly-based discount model based on traffic volume.  If traffic in a BSC area was low, the discount was different for the next voice call than it was for a high traffic area.

Here’s another one, once again from Africa: for each incoming call that terminates on your mobile, you get a credit on your pre-paid account.  It takes a long time to groom a rating system to manage such complex events.

Speaking of Africa, that’s a market where most of the usage is prepaid.  But what about postpaid?  How much demand is there for TCG revenue assurance in the postpaid market with their flat rate bucket minutes?

Dan, you’re right about Africa — pre-paid is probably 98% there, but I think you’d be surprised about postpaid which, of course, is dominant in Europe and US/Canada.

If you look at T-Mobile USA, a healthy 40% of their revenue comes from non-recurring charges — even though the vast majority of their business is flat rate bucket plans.

So the term “flat rate” is a bit misleading.  A significant amount of revenue is still coming from outside the bundle.  Now families who don’t travel abroad can stay inside their flat rate limits, but roaming services and making long distance calls gets you beyond the bucket

So it’s critical to monitor and record this usage accurately.

Now in addition to supplying the operators with TCG, I understand you also serve government regulators.

Regulators are aggressively checking operators to make sure they don’t over-bill.  And I’m sure you have seen the press releases where an operator is being fined tens of millions of dollars simply for inaccurate billing.

The US regulators are very aggressive here.  Regulators in Africa are also keen on monitoring the local operators for overcharging errors.

So yes, we work with many regulators these days, and it’s simple for us to serve them because our GlobalRoamer TCG is practically everywhere.

I understand there’s versatility in the way you diagnose mobile network problems in a TCG environment.  You can either do end-to-end testing across the core of the radio access networks, or simulate the over-the-air calls.

Yes, if your objective is to focus on errors in the switch, mediation, or billing systems, you can simply inject the call directly through SS7 towards the switch.

So that’s what many of our customers do in revenue assuring domestic calls.  They inject calls using our core interfaces via Gb, S1, or SS7.

For end-to-end testing you always need air interfaces, for 2G, 3G, and 4G.  And it is used to test the quality of service.  That’s our main business that SIGOS started 25 years ago.

But there are people who need to manage the switches, SGSN and so on.  It’s useful for them to inject calls — which emulate a complete radio site — via the core interfaces to check the operation of a switch.

Say you suspect some quality issues on the radio side, you might think it’s the MSC software that’s the problem.  Well by injecting calls into the core and comparing what you get with the over-the-air results, you discover that the MSC is not the problem.

So this same principle applies to all the other technologies — 2G, 3G, LTE.  You can inject the calls directly to the core without using the radio site.

It really sounds like a complete debugging environment for mobile calls.

Exactly.  It’s much faster isolating problems when you can do tests across the core and radio network at the same time.

Steffen, thanks for the fine briefing.  It’s highly interesting and I learned a lot.  Are there any final comments you would like to make about the SIGOS solution?

Thanks for the opportunity, Dan.  Certainly a key strength of ours is the stability of the platform.  The first customer we acquired was in the 1990s and it was Vodafone Mannesman, the first GSM operator in Germany.  And through our long association with Vodafone, we developed a very robust system that we advertise as being available 99.8% of the time.

And it’s interesting.  Some of the big German carriers actually calculate the bonuses for their managers based on the KPIs our system is measuring.  These are service managers — one for voice, one for SMS, and another for MMS, and another for data.  And they get a certain bonus based on the quality of their service that’s being measured by our system.

One final comment: our platform is designed to serve many applications at the same time.  So you can do revenue assurance and QoS testing on the same hardware platform.  For instance, if the revenue assurance department buys it, it can be used by the QoS testing people too — and vice versa.  No need to invest in the hardware twice.

Copyright 2015 Black Swan Telecom Journal

Steffen Öftring

Steffen Öftring

Steffen Öftring serves as Revenue Assurance Manager since October 2014 and heads up the complete RA solution (product management, marketing, sales and service delivery).  Previously he headed up the Professional Services Group for Telecommunications as senior consultant where he was responsible for all SITE and GlobalRoamer consultancy projects and managed services worldwide.  Amongst them were various services for interconnect fraud (SIM Box Detection, international carrier quality) and active QoS/QoE monitoring (national & roaming) services.

Before Steffen joined SIGOS he worked as an external business consultant for the Robert-Bosch GmbH in the area of business modelling, corporate-wide time registration software and launching SAP R/3 since the year 2000.  He holds a University degree in computer science and economics from the University of Bamberg.   Contact Steffen via

Black Swan Solution Guides & Papers

cSwans of a Feather

Related Articles

  • Black Swan Guide: Araxxe’s Revenue Assurance Consulting, Testing, and High Definition Billing Analysis Service by Dan Baker — How Araxxe’s end-to-end revenue assurance complements switch-to-bill RA  through telescope RA (external and partner data) and microscope RA (high-definition analysis of complex services like bundling and digital services).
  • Subex’s IDcentral Monetizes Telco & Enterprise Data to Deliver Digital ID & Risk Metric Services for Financing, KYC & More interview with Shankar Roddam — A new digital intelligence service that monetizes the idle data of telecoms and enterprises while also earning a good return for the owner of the data.
  • Opportunities & Obstacles: Consultant Luke Taylor Muses on the State of the Telecom Risk Assurance Business interview with Luke Taylor — A rambling discussion on the state of the risk assurance business with Luke Taylor, independent consultant in telecom revenue/fraud assurance and solution requirements and marketing.
  • LATRO’s Tips for Launching a Successful Revenue & Fraud Assurance Program for Mobile Money Operations in Developing Countries interview with Don Reinhart — A company building mobile money RA/FM tools and  managed services gives a concise, but detailed tutorial on how the Mobile Money Ecosystem works.  Revenue assurance pros will get tips on  what to look for in analytics/assurance tools, controls, and professional services.
  • A WeDo Conference Talk: Consulting & Analytics: Improving your Business Today, Enhancing it Tomorrow interview with Carla Cardoso & Bernado Lucas & Thomas Steagall — Leading risk management consultants explain their mission and walk-through RA, subscription fraud, and collections cases.  They also explain how analytics and machine learning can supplement process optimization.
  • PrologMobile’s Simple and Brilliant Plan to Save US MNOs Billions a Year in Recovered Phones & Retained Customers interview with Seth Heine — An expert in the mobile phone reverse supply chain explains how MNOs — via a neutral third party information exchange — can recover their original phones on the used market and save huge sums in multi-year customer retention.
  • WeDo Explores the IoT Ecosystem in Search of Tomorrow’s Pivotal Fraud & Business Assurance Solutions interview with Carlos Marques — A veteran product manager scans the IoT terrain, discusses key fraud and assurance challenges, and explains the preparatory steps WeDo is taking to become a key player in this emerging market.
  • New Report: Telecom Fraud & Business Assurance Solutions, Services & Strategies by Dan Baker & Luke Taylor & Colin Yates — TRI publishes a new market research report, Telecom Fraud & Business Assurance Solutions, Services & Strategies.  Free executive summary available.
  • Subex Juggles a Wide Variety of Business Assurance and Big Data Analytics Use Cases interview with Rohit Maheshwari — A expert in business assurance solutions explains top use cases such as: IoT security, big data analytics/AI, network asset optimization, multi-player gaming assurance, onboarding mobile subs, and AI customer analytics.
  • MTN Agility: Mastering Exponential Technologies in Revenue/Fraud Assurance and Beyond interview with Danie Maritz & Tony Sani & Luke Taylor — An in-depth look at RAFM operations and innovation at the MTN Group.  Topics discussed include RA/fraud control challenges, strategies, and MTN’s journey to exploit exponential tech (AI, robotics, and ML) in its RAFM program and support of internal non-telco businesses.
  • From Byzantine Software Contracts to Simple & Flexible RA Managed Services interview with Philippe Orsini — Is the way B2B/enterprise software is sold and delivered today progressive — or is it Byzantine in the age of cloud?  An expert lays out the case for managed services in RA and billing verification.
  • Premiere Experts Set to Speak at Summer RAG Conference in London, July 7th and 8th by Dan Baker — The Risk and Assurance Group (RAG) has announced that its 2016 summer conference will expand into a two-day event and feature many premiere experts. 
  • WeDo Hosts Revenue Assurance & Fraud Management Conference in Washington DC by Dan Baker — Black Swan is pleased to announce what looks to be a first class revenue assurance and fraud management conference being put on by WeDo Technologies, on October 1st and 2nd in beautiful Washington DC.
  • Test Call Generators: An Essential Test & Debugging Tool in Mobile Billing Assurance interview with Steffen Öftring — An “active” test call generator (TCG) can see problems that a “passive” revenue assurance system is blind to.  Here’s a discussion on the test call RA  process, over-the-air calls versus core call injection, and test call networks in global roaming RA.
  • The Revenue Assurance Game: How the Rules Change in the Era of IoT & Mobile Broadband interview with Rene Felber & Gadi Solotorevsky — Revenue assurance is perhaps the hardest of telecom functions to define because the term is used in so many different senses.  This discussion on the evolving role of revenue assurance was catalyzed by a survey of experts in the profession.
  • Day in the Life of a Revenue Assurance Analyst interview with Michael Lazarou — Revenue assurance is much more than a software category.  It’s individual analysts struggling to help their larger organizations get a handle on system errors and coordination problems.  In this interview, an analyst reveals the many challenges of getting the revenue assurance job done at a small GSM operator in Europe.
  • Revenue Assurance: History and New Beginnings in RA Maturity interview with Daniela Giacomantonio & Gadi Solotorevsky — The Roman Forum was the center of commercial life in ancient Rome.  Now, two millennia later, the Forum lives on in the exchange of ideas across countless professions and  media.  In this interview, two Revenue Assurance experts discuss both the new RA Maturity initiative of the TM Forum and the value of telco/solution vendor collaboration.
  • Migrating systems or launching LTE next year?  Don‘t forget transformation assurance & optimisation by Efrat Nissimov — System transformations and network migrations are major  revenue impacting events and they should raise a big red flag.  Why?  Because data integrity issues are bound to crop up as CSPs move vital data from a legacy system to something new.  It’s time for transformation assurance.
  • How can Cable/DSL Internet Providers Meet the Usage-Based Billing Mandate? interview with Ryan Guthrie — The popularity of YouTube, Netflix, and Hulu other video outlets has turned the tables on service profitability for cable/DSL service providers.  Many are moving to usage-based billing, but that largely unprepared for the revenue assurance aspects of this move.  This interview explains the technical challenge and points to solutions in billing, speed caps, and traffic revenue monitoring.
  • CABS Revenue Assurance: How Rural LECs can Recover $284 Million in Revenue Shortfalls interview with Kelly Cannon & Darrell Merschak — Independent rural LECs in the U.S. still rely on the AMA/EMI billing formats for CABS billing, even as that format has proven to be highly inaccurate as a source of inter-carrier records.  This interview includes an analysis and discussion of revenue recovery techniques ILECs can use by leveraging SS7 probes.  Also discussed are billing strategies, traffic dumping threats, and the possible fallout from the FCC’s bill-and-keep mandate.
  • Make Business Assurance Progress Every Day: How to Set Goals, Automate, and Energize Your Team interview with Kathleen Romano — Business assurance (BA) skills have wide applicability outside the revenue assurance and fraud mangement domains.  In this article, a telecom executive explains how she’s applying her BA skills in the Payments area.  In addition to discussing the key operational challenges in Payments, the interview also provides keen insights on setting goals in business assurance, leading a team, and making critical decisions.
  • LTE Rollout: Make it a Smashing Success with Risk Assessment, Controls, and Marketing Offer Analytics by Gadi Solotorevsky — LTE brings splendid new capabilities to mobile users.  But like 2G and 3G deployments before, operators can only make money if they successfuly plan, coordinate, deploy fast, and pay attention to pricing plans and the customer experience.  This article lays out a 3-phase tactical guide on  how revenue analytics professionals can add value in LTE service risk assessment, controls, and marketing offer analytics.
  • RA Prevention: How to Manage Revenue Risks and Communicate RA’s Value to Senior Execs by Shaul Moav — The era of revenue assurance prevention and risk assessment is here.  Several of the mature operators of the world have developed their own methodologies and tools.  Using firefighting and fire prevention as a metaphor, the article details a new commercial software approach explaining the goals, method of risk evaluation, and senior executive dashboards developed for the process.
  • Precision Clockworks: How Revenue Assurance Synchronizes with the Business at Swisscom interview with Marco Pollinger — An expert revenue assurance department is one whose work dovetails well with the lines of businesses it supports.  In this interview you’ll learn how Swisscom manages its revenue assurance function for maximum effect.  The article discusses: the operator’s innovative RA organization, the screening and RA approval of new services, its pre-production bill audits, and its coordination with corporate risk management.
  • Versatile, Portable & Corrections-Savvy: Quest for the Swiss Army Knife of Revenue Assurance Software by Mark Yelland — Revenue assurance maturity models are not cast in stone.  Since  best practices will change over time, it’s healthy to explore moving maturity models forward.  For example, great gains have been made in leakage detection, but RA corrections has been harder to master.  The author dreams about seven functions that should ideally come together in a single all-purpose revenue assurance software tool.
  • Bringing Strategic Planning & Value Engineering to Revenue Assurance interview with Maged Fawzy — Engineering and architectural techniques have a role in revenue assurance.  This interview with a top Egyptian RA consultant explains how continuous risk assessment and long range — yet flexible — RA planning can sharpen a carrier’s RA program and lead to better use of revenue assurance software and integration services.
  • Forensic Fossils: Is Your Revenue Assurance Shop Fit for Display at a Natural History Museum? interview with Jim Marsh — Without the continuous guiding light of seasoned revenue assurance leaders, even the best teams of RA professionals, technology, and business processes can fossilize and lose their vitality.
  • Revenue Assurance: The Magical Market Cap Multiplier by Van Howard & Curtis Mills — Many operators today consider revenue assurance yesterday’s opportunity.  But this article shows why significant revenue and cost leakage can still go undetected, even in companies with dedicated RA departments.  Also discussed are the benefits of a broader or more “forensic” approach to revenue assurance, an approach that boosts the bottom line regardless of the automated tools already in place.
  • From Risk to Robust: Turning the Big Picture Into a Real Agenda for Change in Telecoms by Eric Priezkalns — Inspired by a Financial Times article written by Nassim Taleb, author of “The Black Swan”, here is an insightful and entertaining primer on telecom risk management.  The article takes ten risk management lessons from Taleb and applies them specifically to the communications industry.  You’ll learn about the value of small scale trials, organization accountability, cures for a blame culture, incentives that work, the power of simplicity, and more.
  • Synthesizing the Telecom Business Assurance Practice With the Analytics World by Dan Baker — Business assurance is a wrapper term that allows you to draw a circle around various telecom assurance, control, and optimization activities.  This article maps business assurance as a subset of telecom analytics, constrasting it with marketing analytics while a diagram shows where biz assurance fits in the larger B/OSS world.
  • CABS Revenue Assurance Disputes: May the Carrier With the Best Data Win by Cheryl Smith Rardin & David West — Revenue assurance innovation is far easier when partners cooperate to make it happen.  This articles shows how a U.S. operator, software vendor, and consultant teamed to develop a breakthrough in Carrier Access Billing (CABS) assurance.  Learn about: the dispute resolution data gap that needed to be filled, the partnering strategy, the implementation challenges, and payback results.
  • Revenue Assurance vs.  Business Assurance: Who’s the Rightful King of Controls Software? interview with Sergio Luis Silvestre — Business controls software, originally developed for RA, is finding application in other areas of the business such as internal audit, collections, security and risk management.  This article argues that “business assurance” is the best term to describe this broader set of  controls software that can find a home in numerous departments or functions of a CSP’s business.
  • PwC on the Business of Revenue Assurance Consulting & Mentoring interview with Tim Banks & Dan Stevens — Revenue assurance consulting firms offer a broad range of services to clients these days.  The article explains the practice of mentoring RA mangers and providing a CFO with visibility on the status of an operator’s business controls.  Perspective is also offered on the value of RA software and the opportunity to broaden the RA practice scope.
  • Robots for Hire: Verifying Accuracy In the Age of Complex Mobile Billing/Charging interview with Xavier Lesage — As real-time charging and complex lifestyle calling plans gain credence across the globe in wireless, billing quality issues will rise in importance.  This article discusses a unique managed services approach to invoice testing and roaming fraud protection that checks results against advertised or published source data for the utmost accuracy.
  • Ericsson: Revenue Assurance Consulting With an NGN Flavor interview with Thomas Steagall — Helping operators detect billing and provisioning problem is merely table stakes in the RA services business these days.  The article discuss why operators need to ramp up their RA function with service experience and group-wide financial health monitoring.  Advise is also offered on: key RA maturity questions, risk-and-reward contracts, and how to extract greater value from software investments.
  • Do-It-Yourself RA for Small Operators and MVNOs interview with Mark Yelland — Budget-minded small operators and MVNOs are no longer hamstrung in RA capability anymore.  This article offers high-leverage strategies for operators who cannot afford expensive RA software tools.  With  data access, brains, and a DIY philosophy, any small operator can map a  path to greater RA savings, maturity, and program growth.
  • Revenue Assurance Maturity: Report From the Arena interview with Eric Nelson — Revenue assurance maturity can‘t be easily computed.  How do you  compare the KPIs of Comcast billing with that of mobile money RA in Western Africa?  Even still, this article offers some universal RA wisdom from a straight-shooting veteran of carriers large and small.  Topics discussed include: dashboard or process, COTS vs. inhouse solutions, and tips on gaining internal support for the RA practice.