Home
About Black Swan Editorial Mission Reports & Papers
Fraud Systems SIM Box Roaming Fraud PBX Fraud Subscription Databases
RA Practices Business Assurance RA Maturity RA Consulting Bill Testing
Wholesale Systems Net Inventory Margin Assure Partner Mgmt
Customer Behavior Machine Learning Network Analytics Marketing Analytics
Network Behavior Mobile & IoT Security Trusted Communication Signaling
Email a colleague    

August 2015

Test Call Generators: An Essential Test & Debugging Tool in Mobile Billing Assurance

interview with Steffen Öftring

Test Call Generators: An Essential Test & Debugging Tool in Mobile Billing Assurance

Is it even possible to develop complex enterprise software without a powerful debugging environment?  Sure, you could do it, but getting to relatively error-free code would take you forever.

So what about the world of mobile operator billing?  Actually, that world is far more complex because it combines all the familiar IT headaches — databases, servers, clouds, and code — with telecom billing’s own weird issues: proprietary switches, event-driven tariff rules, regional network variations, international roaming, mediation processes, changing tariffs, new technologies and many other complexities, not to mention regulatory requirements.

Now mobile network revenue assurance (RA) systems that use the CDR as their starting point are vital, but they miss all sorts of problems that can happen south of the mediation system.

In short, an “active” test call generator (TCG) can see problems that a “passive” RA system is simply blind to.  Yet the TCG’s role in preventing, diagnosing, and debugging mobile billing problems has not been fully appreciated up to now.

Here, then, to brief us on the virtues of TCGs in billing and revenue assurance is Steffen Öftring, Revenue Assurance Manager at SIGOS.  Steffen discusses the  strengths of active RA systems, the differences between over-the air calls and core call injection, and the importance of global test call networks in roaming RA.

Dan Baker: Steffen, let me start with a question that’s probably on a lot of people’s minds.  For operators who have already spent a million dollars or more on a revenue assurance system, why should they spend more money to acquire a test call generator solution from SIGOS?

Steffen Öftring: Actually, Dan, I get asked that question all the time.  And the first thing I say is that this is not an either/or choice.  We often work in tandem with the major revenue assurance vendors to complement and enhance the capabilities of their system for customers.  And frankly, the revenue assurance interface to provisioning is an area we do not address.

But on the billing verification side of revenue assurance, we bring great value because we generate test CDRs on the network specifically to find and diagnose billing problems.  And because we control every aspect of those call events, we know exactly how many CDRs should be generated.  And we can check the metering, call duration, or download volume.

In fact, of the three most common errors in billing, test calls are needed to identify two of them: a passive RA system doesn’t even pick those errors up.  So here’s a diagram and text that explains what I mean:

  1. Incorrect Rating — A typical revenue assurance system excels at checking for incorrectly rated CDRs.  But because its starting point is CDRs coming off the switch, it has no visibility over errors caused by the switch or events prior to the call getting to the switch.
  2. Metering Issues — If the switch is not correctly recording the duration of the call or the size of a download, that’s a big issue and a passive revenue assurance system can’t identify that.  But test calls, because of their active and end-to-end nature can find where these metering issues exist.
  3. Lost CDRs — Finally, there are many cases where an MSC is not recording CDRs at all.  Here again, active test calls come to the rescue.  By comparing test call data with a daily or twice-daily CDR feed, you can readily detect switches that are malfunctioning.  We know exactly that from location A, we made 150 calls yesterday.  So if there are only 140 calls in the CDR database, we know there are 10 CDRs missing.
Passive vs. Active RA

A passive system can manage all the live traffic.  But our approach is to focus on test call generator traffic.  And it’s much less traffic, so the deployment is much cheaper than a passive solution.

Some of these revenue assurance systems cost $5 million or more and they might take 6 months to deploy.  But an average TCG from SIGOS costs around $0.5 million and can be up and running in eight weeks.  Because we have our own TCG system covering over 200 countries we can also provide a fully managed service which requires no equipment to be installed by an operator.

Now I understand one of the crucial areas where TCGs are useful is in testing calls for roaming.

This is very true.  If, for example, a US subscriber roams to Europe and has a 24-hour unlimited pass for voice and data service, we can test that with our system.

So the strength of a roaming solution like this is the breadth of global coverage in your TCG network.  Now SIGOS stacks up very well because we have 430+ global customers who use our platform for quality of service, fraud detection, roaming and revenue assurance.

Our biggest customers have over 200 test locations covering all technologies in one country so they can cover all the switch sites they have in a domestic market.

And any one of these local test call generator systems can be connected to our GlobalRoamer network footprint.  So we can combine domestic revenue assurance with global RA with everything managed from the customer’s own system.

How does TCG revenue assurance operate at a typical client?

Well, let’s use a US based operator as an example.  They need to supply us with one SIM card for every individual tariff.  And from that tariff we are going to use our expertise to create a test plan that tests all the risk points of this tariff.

Let’s suppose the tariff is a flat rate bundle with 50 SMS and 500 voice minutes and 5 Gigabytes of data.  Once we know the tariff rules, our test plan makes sure we consume all the free bundle elements.  And TCG operates with that tariff plan over the course of a daily or weekly basis, and we keep consuming.  So if all is working well, the 501st voice minute should cost something.  And we verify that.

Now each call we make creates a live CDR in the US-Operator’s network.  So we bring together the live CDRs that our customer supplies to us.

The trick is to then verify accuracy by comparing different data sources.  Here, we generally take three feeds: an un-rated switch feed with all test SIM CDRs; a feed from the mediation system; and a feed from the billing system.

Now aside from having a global network for testing roaming, what else should buyers be looking for as they evaluate TCG systems?

I would definitely say the rating capability is first.  Remember, after we do the CDR comparison to find the missing CDRs, metering errors, and correctness of the content, our second step is to re-rate the calls.

Actually, the rating engine is one of the hardest components to get right.  At SIGOS it took us three years to develop a rating engine to input live CDRs and re-rate them.

And the reason it took so long is the tariffs these day are immensely complex.  Flat rated tariffs are simple, but you’d be surprised how creative people in marketing can get inventing new tariffs.

For example, in South Africa they had an hourly-based discount model based on traffic volume.  If traffic in a BSC area was low, the discount was different for the next voice call than it was for a high traffic area.

Here’s another one, once again from Africa: for each incoming call that terminates on your mobile, you get a credit on your pre-paid account.  It takes a long time to groom a rating system to manage such complex events.

Speaking of Africa, that’s a market where most of the usage is prepaid.  But what about postpaid?  How much demand is there for TCG revenue assurance in the postpaid market with their flat rate bucket minutes?

Dan, you’re right about Africa — pre-paid is probably 98% there, but I think you’d be surprised about postpaid which, of course, is dominant in Europe and US/Canada.

If you look at T-Mobile USA, a healthy 40% of their revenue comes from non-recurring charges — even though the vast majority of their business is flat rate bucket plans.

So the term “flat rate” is a bit misleading.  A significant amount of revenue is still coming from outside the bundle.  Now families who don’t travel abroad can stay inside their flat rate limits, but roaming services and making long distance calls gets you beyond the bucket

So it’s critical to monitor and record this usage accurately.

Now in addition to supplying the operators with TCG, I understand you also serve government regulators.

Regulators are aggressively checking operators to make sure they don’t over-bill.  And I’m sure you have seen the press releases where an operator is being fined tens of millions of dollars simply for inaccurate billing.

The US regulators are very aggressive here.  Regulators in Africa are also keen on monitoring the local operators for overcharging errors.

So yes, we work with many regulators these days, and it’s simple for us to serve them because our GlobalRoamer TCG is practically everywhere.

I understand there’s versatility in the way you diagnose mobile network problems in a TCG environment.  You can either do end-to-end testing across the core of the radio access networks, or simulate the over-the-air calls.

Yes, if your objective is to focus on errors in the switch, mediation, or billing systems, you can simply inject the call directly through SS7 towards the switch.

So that’s what many of our customers do in revenue assuring domestic calls.  They inject calls using our core interfaces via Gb, S1, or SS7.

For end-to-end testing you always need air interfaces, for 2G, 3G, and 4G.  And it is used to test the quality of service.  That’s our main business that SIGOS started 25 years ago.

But there are people who need to manage the switches, SGSN and so on.  It’s useful for them to inject calls — which emulate a complete radio site — via the core interfaces to check the operation of a switch.

Say you suspect some quality issues on the radio side, you might think it’s the MSC software that’s the problem.  Well by injecting calls into the core and comparing what you get with the over-the-air results, you discover that the MSC is not the problem.

So this same principle applies to all the other technologies — 2G, 3G, LTE.  You can inject the calls directly to the core without using the radio site.

It really sounds like a complete debugging environment for mobile calls.

Exactly.  It’s much faster isolating problems when you can do tests across the core and radio network at the same time.

Steffen, thanks for the fine briefing.  It’s highly interesting and I learned a lot.  Are there any final comments you would like to make about the SIGOS solution?

Thanks for the opportunity, Dan.  Certainly a key strength of ours is the stability of the platform.  The first customer we acquired was in the 1990s and it was Vodafone Mannesman, the first GSM operator in Germany.  And through our long association with Vodafone, we developed a very robust system that we advertise as being available 99.8% of the time.

And it’s interesting.  Some of the big German carriers actually calculate the bonuses for their managers based on the KPIs our system is measuring.  These are service managers — one for voice, one for SMS, and another for MMS, and another for data.  And they get a certain bonus based on the quality of their service that’s being measured by our system.

One final comment: our platform is designed to serve many applications at the same time.  So you can do revenue assurance and QoS testing on the same hardware platform.  For instance, if the revenue assurance department buys it, it can be used by the QoS testing people too — and vice versa.  No need to invest in the hardware twice.

Copyright 2015 Black Swan Telecom Journal

 
Steffen Öftring

Steffen Öftring

Steffen Öftring serves as Revenue Assurance Manager since October 2014 and heads up the complete RA solution (product management, marketing, sales and service delivery).  Previously he headed up the Professional Services Group for Telecommunications as senior consultant where he was responsible for all SITE and GlobalRoamer consultancy projects and managed services worldwide.  Amongst them were various services for interconnect fraud (SIM Box Detection, international carrier quality) and active QoS/QoE monitoring (national & roaming) services.

Before Steffen joined SIGOS he worked as an external business consultant for the Robert-Bosch GmbH in the area of business modelling, corporate-wide time registration software and launching SAP R/3 since the year 2000.  He holds a University degree in computer science and economics from the University of Bamberg.   Contact Steffen via

Black Swan Solution Guides & Papers

cSwans of a Feather

Related Articles

  • Tokopedia, Indonesia’s E-Commerce King, Partners with 11 Million Merchants; Adopts Multi-Cloud to Drive Innovation — interview with Warren Aw & Ryan de Melo — Indonesia’s Tokopedia, founded in 2009, has grown to become one of world’s leading e-commerce players.  Read about its success, technology direction, and multi-cloud connectivity adoption.
  • Bridge Alliance: Knocking Down Regional & Mobile Connectivity Barriers so Connected Car Markets Get Rolling in Asia — interview with Kwee Kchwee — The CEO of an Asian consortium of mobile operators explains how they  help simplify and harmonize their members‘ operations in support of multi-national corporations.  This integration is enabling two huge industries to come together in Asia: auto manufacturing and telco.
  • Epsilon’s Infiny NaaS Platform Brings Global Connection, Agility & Fast Provision for IoT, Clouds & Enterprises in Southeast Asia, China & Beyond — interview with Warren Aw — Network as a Service, powered by Software Defined Networks, are a faster, more agile, and more partner-friendly way of making data global connections.  A leading NaaS provider explains the benefits for cloud apps, enterprise IT, and IoT.
  • PCCW Global: On Leveraging Global IoT Connectivity to Create Mission Critical Use Cases for Enterprises — interview with Craig Price — A leading wholesale executive explains the business challenges of the current global IoT scene as it spans many spheres: technical, political, marketing, and enterprise customer value creation.
  • Senet’s Cloud & Shared Gateways Drive LoRaWAN IoT Adoption for Enterprise Businesses, Smart Cities & Telecoms — interview with Bruce Chatterley — An IoT netowork pioneer explains how LoRaWAN tech fits in the larger IoT ecosystem.  He gives use case examples, describes deployment restraints/costs, and shows how partnering, gateway sharing, and flexible deployment options are stimulating growth.
  • ARM Data Center Software’s Cloud-Based Network Inventory Links Network, Operations, Billing, Sales & CRM to One Database — interview with Joe McDermott & Frank McDermott — A firm offering a cloud-based network inventory system explains the virtues of: a single underlying database, flexible conversions, task-checking workflow, new software business models, views that identify stranded assets, and connecting to Microsoft’s cloud platform.
  • Pure Play NFV: Lessons Learned from Masergy’s Virtual Deployment for a Global Enterprise — interview with Prayson Pate — NFV is just getting off the ground, but one cloud provider to enterprises making a stir in virtual technology waters is Masergy.  Here are lessons learned from Masergy’s recent global deployment using a NFV pure play software approach.
  • The Digital Enabler: A Charging, Self-Care & Marketing Platform at the Core of the Mobile Business — interview with Jennifer Kyriakakis — The digital enabler is a central platform that ties together charging, self-care, and marketing.  The article explains why leading operators consider digital enablers pivotal to their digital strategies.
  • Delivering Service Assurance Excellence at a Reduced Operating Cost — interview with Gregg Hara — The great diversity and complexity of today’s networks make service assurance a big challenge.  But advances in off-the-shelf software now permit the configuring and visualizing of services across multiple technologies on a modest operating budget.
  • Are Cloud-Based Call Centers the Next Hot Product for the SMB Market? — interview with Doron Dovrat — Quality customer service can improve a company’s corporate identity and drive business growth.  But many SMBs are priced out of acquiring modern call center technology.  This article explains the benefits of affordable and flexible cloud-based call centers.
  • Flexing the OSS & Network to Support the Digital Ecosystem — interview with Ken Dilbeck — The need for telecoms to support a broader digital ecosystem requires an enormous change to OSS infrastructures and the way networks are being managed.  This interview sheds light on these challenges.
  • Crossing the Rubicon: Is it Time for Tier Ones to Move to a Real-Time Analytics BSS? — interview with Andy Tiller — Will tier one operators continue to maintain their quilt works of legacy and adjunct platforms — or will they radically transform their BSS architecture into a new  system designed to address the new telecom era?  An advocate for radical transformation discusses: real-time analytics, billing for enterprises, partnering mashups, and on-going transformation work at Telenor.
  • Paradigm Shift in OSS Software: Network Topology Views via Enterprise-Search — interview with Benedict Enweani — Enterprise-search is a wildly successful technology on the web, yet its influence has not yet rippled to the IT main stream.  But now a large Middle Eastern operator has deployed a major service assurance application using enterprise-search.  The interview discusses this multi-dimensional topology solution and compares it to traditional network inventory.
  • The Multi-Vendor MPLS: Enabling Tier 2 and 3 Telecoms to Offer World-Class Networks to SMBs — interview with Prabhu Ramachandran — MPLS is a networking technology that has caught fire in the last decade.  Yet the complexity of MPLS has relegated to being mostly a large carrier solution.  Now a developer of a multi-vendor MPLS solutions explains why the next wave of MPLS adoption will come from tier 2/3 carriers supporting SMB customers.
  • Enabling Telecoms & Utilities to Adapt to the Winds of Business Change — interview with Kirill Rechter — Billing is in the midst of momentous change.  Its value is no longer just around delivering multi-play services or sophisticated rating.  In this article you’ll learn how a billing/CRM supplier has adapted to the times by offering deeper value around the larger business issues of its telecom and utility clients.
  • Driving Customer Care Results & Cost Savings from Big Data Facts — interview with Brian Jurutka — Mobile broadband and today’s dizzying array of app and network technology present a big challenge to customer care.  In fact, care agents have a hard time staying one step ahead of customers who call to report problems.  But network analytics comes to the rescue with advanced mobile handset troubleshooting and an ability to put greater intelligence at the fingertips of highly trained reps.
  • Hadoop and M2M Meet Device and Network Management Systems — interview with Eric Wegner — Telecom big-data in networks is more than customer experience managment: it’s also about M2M plus network and element management systems.  This interview discusses the explosion in machine-to-machine devices, the virtues and drawbacks of Hadoop, and the network impact of shrink-wrapped search.
  • The Data Center & Cloud Infrastructure Boom: Is Your Sales/Engineering Team Equipped to Win? — by Dan Baker — The build-out of enterprise clouds and data centers is a golden opportunity for systems integrators, carriers, and cloud providers.  But the firms who win this business will have sales and engineering teams who can drive an effective and streamlined requirements-to-design-to-order process.  This white paper points to a solution — a collaborative solution designs system — and explains 8 key capabilities of an ideal platform.
  • Big Data: Is it Ready for Prime Time in Customer Experience Management? — interview with Thomas Sutter — Customer experience management is one of the most challenging of OSS domains and some suppliers are touting “big data” solutions as the silver bullet for CEM upgrades and consolidation.  This interview challenges the readiness of big data soluions to tackle OSS issues and deliver the cost savings.  The article also provides advice on managing technology risks, software vendor partnering, and the strategies of different OSS suppliers.
  • Calculated Risk: The Race to Deliver the Next Generation of LTE Service Management — interview with Edoardo Rizzi — LTE and the emerging heterogeneous networks are likely to shake up the service management and customer experience management worlds.  Learn about the many new network management challenges LTE presents, and how a small OSS software firm aims to beat the big established players to market with a bold new technology and strategy.
  • Decom Dilemma: Why Tearing Down Networks is Often Harder than Deploying Them — interview with Dan Hays — For every new 4G LTE and IP-based infrastructure deployed, there typically a legacy network that’s been rendered obsolete and needs to be decommissioned.  This article takes you through the many complexities of network decom, such as facilities planning, site lease terminations, green-safe equipment disposal, and tax relief programs.
  • Migration Success or Migraine Headache: Why Upfront Planning is Key to Network Decom — interview with Ron Angner — Shutting down old networks and migrating customers to new ones is among the most challenging activities a network operators does today.  This article provides advice on the many network issues surrounding migration and decommissioning.  Topics discussed include inventory reconciliation, LEC/CLEC coordination, and protection of customers in the midst of projects that require great program management skills.
  • Navigating the Telecom Solutions Wilderness: Advice from Some Veteran Mountaineers — interview with Al Brisard — Telecom solutions vendors struggle mightily to position their solutions and figure out what to offer next in a market where there’s considerable product and service crossover.  In this article, a veteran order management specialist firm lays out its strategy for mixing deep-bench functional expertise with process consulting, analytics, and custom API development.
  • Will Telecoms Sink Under the Weight of their Bloated and Out-of-Control Product Stacks? — interview with Simon Muderack — Telecoms pay daily for their lack of product integration as they constantly reinvent product wheels, lose customer intelligence, and waste time/money.  This article makes the case of an enterprise product catalog.  Drawing on central catalog cases at a few Tier 1 operators, the article explains the benefits: reducing billing and provisioning costs, promoting product reuse, and smoothing operations.
  • Virtual Operator Life: Enabling Multi-Level Resellers Through an Active Product Catalog — interview with Rob Hill — The value of product distribution via virtual operators is immense.  They enable a carrier to sell to markets it cannot profitably serve directly.  Yet the need for greater reseller flexibility in the bundling and pricing of increasingly complex IP and cloud services is now a major channel barrier.  This article explains what’s behind an innovative product catalog solution that doubles as a service creation environment for resellers in multiple tiers.
  • Telecom Blocking & Tackling: Executing the Fundamentals of the Order-to-Bill Process — interview with Ron Angner — Just as football teams need to be good at the basics of blocking and tackling, telecoms need to excel at their own fundamental skillset: the order-to-cash process.  In this article, a leading consulting firm explains its methodology for taking operators on the path towards order-to-cash excellence.  Issues discussed include: provisioning intervals; standardization and simplicity; the transition from legacy to improved process; and the major role that industry metrics play.
  • Wireline Act IV, Scene II: Packaging Network & SaaS Services Together to Serve SMBs — by John Frame — As revenue from telephony services has steadily declined, fixed network operators have scrambled to support VoIP, enhanced IP services, and now cloud applications.  This shift has also brought challenges to the provisioning software vendors who support the operators.  In this interview, a leading supplier explains how it’s transforming from plain ol‘ OSS software provider to packager of on-net and SaaS solutions from an array of third party cloud providers.
  • Telecom Merger Juggling Act: How to Convert the Back Office and Keep Customers and Investors Happy at the Same Time — interview with Curtis Mills — Billing and OSS conversions as the result of a merger are a risky activity as evidenced by famous cases at Fairpoint and Hawaiian Telcom.  This article offers advice on how to head off problems by monitoring key operations checkpoints, asking the right questions, and leading with a proven conversion methodology.
  • Is Order Management a Provisioning System or Your Best Salesperson? — by John Konczal — Order management as a differentiator is a very new concept to many CSP people, but it’s become a very real sales booster in many industries.  Using electronics retailer BestBuy as an example, the article points to several innovations that can — and are — being applied by CSPs today.  The article concludes with 8 key questions an operator should ask to measure advanced order management progress.
  • NEC Takes the Telecom Cloud from PowerPoint to Live Customers — interview with Shinya Kukita — In the cloud computing world, it’s a long road from technology success to telecom busness opportunity.  But this story about how NEC and Telefonica are partnering to offer cloud services to small and medium enterprises shows the experience of early cloud adoption.  Issues discussed in the article include: customer types, cloud application varieties, geographic region acceptance, and selling challenges.
  • Billing As Enabler for the Next Killer Business Model — interview with Scott Swartz — Facebook, cloud services, and Google Ads are examples of innovative business models that demand unique or non-standard billing techniques.  The article shows how flexible, change-on-the-fly, and metadata-driven billing architectures are enabling CSPs to offer truly ground breaking services.
  • Real-Time Provisioning of SIM Cards: A Boon to GSM Operators — interview with Simo Isomaki — Software-controlled SIM card configuration is revolutionizing the activation of GSM phones.  The article explains how dynamic SIM management decouples the selection of numbers/services and delivers new opportunities to market during the customer acquisition and intial provisoining phase.
  • A Cynic Converted: IN/Prepaid Platforms Are Now Pretty Cool — interview with Grant Lenahan — Service delivery platforms born in the IN era are often painted as inflexible and expensive to maintain.  Learn how modern SDPs with protocol mediation, high availability, and flexible Service Creation Environments are delivering value for operators such as Brazil’s Oi.
  • Achieving Revenue Maximization in the Telecom Contact Center — interview with Robert Lamb — Optimizing the contact center offers one of the greatest returns on investment for a CSP.  The director of AT&T’s contact center services business explains how telecoms can strike an “artful balance” between contact center investment and cost savings.  The discussion draws from AT&T’s consulting with world class customers like Ford, Dell, Discover Financial, DISH Network, and General Motors.
  • Mobile Broadband: The Customer Service Assurance Challenge — interview with Michele Campriani — iPhone and Android traffic is surging but operators struggle with network congestion and dropping ARPUs.  The answer?  Direct  resources and service quality measures to ensure VIPs are indeed getting the quality they expect.  Using real-life examples that cut to the chase of technical complexities, this article explains the chief causes of service quality degradation and describes efficient ways to deal with the problem.
  • Telco-in-a-Box: Are Telecoms Back in the B/OSS Business? — interview with Jim Dunlap — Most telecoms have long since folded their merchant B/OSS software/services businesses.  But now Cycle30, a subsidiary of Alaskan operator GCI, is offering a order-to-cash managed service for other operators and utilities.  The article discusses the company’s unique business model and contrasts it with billing service bureau and licensed software approaches.
  • Bricks, Mortar & Well-Trained Reps Make a Comeback in Customer Management — interview with Scott Kohlman — Greater industry competition, service complexity, and employee turnover have raised the bar in the customer support.  Indeed, complex services are putting an emphasis on quality care interactions in the store, on the web, and through the call center.  In this article you’ll learn about innovations in CRM, multi-tabbed agent portals,  call center agent training, customer treatment philosophies, and the impact of  self-service.
  • 21st Century Order Management: The Cross-Channel Sales Conversation — by John Konczal — Selling a mobile service is generally not a one-and-done transaction.  It often involves several interactions — across the web, call center, store, and even kiosks.  This article explains the power of a “cross-channel hub” which sits above all sales channels, interacts with them all, and allows a CSP to keep the sales conversation moving forward seamlessly.
  • Building a B/OSS Business Through Common Sense Customer Service — by David West — Delivering customer service excellence doesn‘t require mastering some secret technique.  The premise of this article is that plain dealing with customers and employees is all that’s needed for a winning formula.  The argument is spelling out in a simple 4 step methodology along with some practical examples.