Email a colleague    

August 2012

Partners in Carrier Management: The Success Story Behind T-Mobile’s Fiber Rollout in Wireless Backhaul

Partners in Carrier Management: The Success Story Behind T-Mobile’s Fiber Rollout in Wireless Backhaul

iPhones and Androids didn‘t create the mobile broadband revolution on their own.  The smartphone market would have never taken off had operators not built out a reliable and cost effective fixed broadband infrastructure to offload radio data traffic.

Back in 2006, the massive fixed broadband infrastructure that exists today to support smartphones didn‘t really exist.  And that successful build out was a culmination of many gut-wrenching decisions, risk-laden investments, and full-scale cooperation from a variety of telecom industry players.

Truth be told, wireless backhaul is the unsung hero of the smartphone’s success.  And here to give us some perspective on how it all evolved is Bryan Fleming, Vice President, Technical Systems and Business Operations at T-Mobile USA, who I met at Razosight’s recent user conference in Washington D.C.

What I love about Bryan’s story is the way he tells it.  Bryan weaves an interesting -- even entertaining — tale that almost anyone in telecom can relate to.  Of course, for people in the interconnect and carrier management areas, Bryan’s first-hand perspective is invaluable.

Among the insights you’ll take away: the reasons for adopting a full-scale fiber strategy; the challenge of finding carriers to support T-Mobile’s vision; the clever techniques T-Mobile used to simplify and cut costs; his expert advice on how to build and maintain great relationships with suppliers; and the key role analytics, assurance and visualization software play in T-Mobile’s current backhaul program.

Dan Baker: Bryan, to begin, it would be great if you could give us some context, perhaps a quick backgrounder on T-Mobile’s situation.

Bryan Fleming:  Sure, Dan.  I guess like every other wireless carrier, T-Mobile USA has seen major smart phone penetration in the past two years.  Our latest figures say there are 11.5 million smart phones in our network.  In fact, 90% or more of the devices we sell today are smart phone devices.  And as you can well imagine, it’s things like social networking, video applications, and instant access to information that’s driving demand.

Another thing about T-Mobile — and I’ve been with the company almost 13 years — is it likes to be on the leading-edge.  We are frequently the first to adopt new innovations, whether it be Android rollouts or apps like wired-enabled autos.

So from the year 2007 forward, we’ve put a tremendous emphasis on trunking and data connectivity to support our heavy wireless usage.

At some point, T-Mobile chose to go “all in” with a full-blown fiber back haul strategy.  What were some of the reasons you went in that direction?

Well, it was back in 2006 and 2007 when we first faced the mobile data tsunami is coming.  What scared mobile operators like us was that if data grew as fast as we expected and users didn‘t pay a premium for it, our business model was going to be turned upside down.

T1 Economics Didn‘t Work

So the only way we could keep up with wireless network growth was through T1 lines, which would have been an unsustainable model.  A T1 gives you 1.5 Mbits and its typical cost -- including associated connectivity such DS3s and OCNs -- was between $350 and $500.  It wasn‘t going to work.

What’s more, the equipment manufacturers were also predicting Doomsday.  It freaked us out.  And when the iPhone launched in mid-2007 we could see all sorts of cost and customer experience problems in our future.

A Fiber Revelation

When we were in midst of this crisis, my boss called me up and excitedly explained that he’d had one of those famous “epiphanies in the shower” that morning.  He was absolutely convinced that the way to solve our problem was through fiber.

Now initially I was very skeptical of that decision, even though that indeed turned out to be the winning strategy.  At the time, the fiber choice wasn‘t so obvious.  In fact a lot of companies invested in the microwave route, but that ultimately didn’t work because you have to spend money over and over again with microwave.

What sort of virtues were you looking for in your carrier suppliers?

Well, we really focused on three key things.

Availability

First, we made sure that fiber was available in the majority of sites that meant the most to us.  We wanted to make sure that if we changed vendors or made long-term commitments to the ones we already had that they were committed to improving the reliability of the network.  And the former Ma Bells were reluctant to invest for the future in their old wireline networks — and I don‘t blame them.

Simplify Billing

Our second principle of partnering was around cost.  Not only did we have to stop the upward slope of the cost curve, we had to simplify billing.  Billing is an absolute nightmare.  The best analogy for a TDM network is it’s a big bowl of spaghetti where no one knows which end is which.  It creates a great deal of confusion not only in your inventory systems, but also in terms of auditing supplier bills and managing reliability.

Timely Delivery

We also had to know that our vendors were committed to deliver what we needed on time.  We work on very short timeframes and our mindset was to enable those vendors to deploy as quickly as possible.  So, with these points in mind, we started to engage the vendor community.

What kind of carriers did you approach as suppliers of your next generation back haul?

Well, as you can imagine, our first conversations were with the traditional and incumbent local exchange carriers because that’s who we had our current infrastructure with.

The Incumbent LECs Laughed at Our Plan

In my discussions with the LECs, I don‘t know how many times in that next six months I got laughed at and told, “Good luck, Fleming, but we’re not playing.”  So, I took note of that, jotted it down, and continued on my way.

The Microwave Carriers Step Up

So what we were able to find -- and what you always need when your ecosystem is new and needs to be developed — is someone who believes in your story that data is coming and that there’s going to be revenue growth.

The companies who first stepped up to support us were the microwave-based companies like FiberTower and TTM.  As you probably know, FiberTower currently is in bankruptcy, but back in 2005-2006, they were the first alternative access vendor.  And they made good on their promise to replace the LEC with a better cost and better reliability solution.  Their problem was they did it using microwave and didn‘t fully anticipate the big rollout in wireless data.

A Secondary Business Opportunity for the Cable Operators

Our next question became: who do we look at if the LECs won‘t play and the microwave players can’t keep up?  Well, that’s when we turned to the cable community.  And our pitch to the cable operators was that they had a valuable asset in the ground that they were not fully leveraging: delivering video service to the home was a great business model but there are other growth opportunities there too.  So, our first mover on that concept was Time Warner Cable.  And interestingly enough, as soon as that deal was done, everybody in the cable industry followed at a very rapid pace.

The Former Lunatic Becomes the Most Hunted

Fast forward another 8 to 12 months and we still had no LEC deals.  Every meeting I had with them ended with, “You’re a lunatic -- good luck, you’re not going to be able to get your plan done.” So to demonstrate we were serious, as we began to install fiber with Time Warner, we stopped talking to the LECs and started disconnecting their service.  And sure enough, all of a sudden the “lunatic” in the room became the most hunted because their bottom line was being affected and they realized their business was at risk.  So the LECs came along after that relatively quickly.

And as far as the LEC space, the first mover, if you can believe it was AT&T, and in some cases I would say they are probably one of the most cost effective vendors out there today.

Now while we were concerned with backhaul, a lot of the companies who supply backhaul deliver things other than backhaul and in most cases, backhaul is not their first business.  So we found other opportunities to leverage growth to drive out our costs and find synergies in which to grow together.  That’s really, really important.

Looking back to 2006-2007, roughly 95% of our backhaul network at T-Mobile was supplied by the LEC incumbents.  Today, if you look at our broadband footprint that supports our HSPDA plus network, the LEC incumbents have about 47% share and that continues to decline because there are new providers out there who are aggressive and have a strong capital backing.

Carrier/vendor negotiations can be a fairly brutal experience for the vendor.  Traditionally a lot of pressure is put on the carriers supplying software, hardware, and services.

That’s true.  But on the backhaul business, if you’re out to beat your supplier down to the lowest possible price, it’s not going to work because this is an extremely capital intensive business.  And you want the capital money to be available to grow that ecosystem because those are the things that drive your costs, innovation, and availability.

Knowing the Supplier’s Business Model & Constraints

To successfully negotiate and find a good partner in this environment, you have to understand what your supplier’s business model is.  That’s very critical.  Third thing is open communication: you really have to understand what is important to both parties to find a good deal that works for everybody without risking the relationship or coming away disappointed in what you got.

Each of the vendors will be in a different financial situation.  What kind of capital do they have?  You need to understand that and be flexible.  And what might work from a contract or deal perspective with a large company like AT&T or a Comcast or Time Warner, might not work for a middle or a small tier carrier.

The deal needs to be structured in a way that makes their financial backers comfortable that you are going to be around for a long time for them.  When they are confident of that, then they can offer you the right price.

Open Communications and Commitment

You want to foster creativity.  Instead of working on a traditional contract, there might be some other alternatives out there, but it has to start with open communication.

The fourth thing is being committed to that relationship.  If you hit a bump in the road, just don‘t dump the relationship.  Especially in a growing area like wireless backhaul that you know can only become more complex, you need strong allies in your partners.  As wireless carriers, to be able to deliver that content, the system will have to evolve, and that will require creative solutions that your provider will supply.

Good communications is absolutely key.  Sometimes, they may even want to pull you aside and honestly say, “Hey you’re being an idiot.  You are doing this the wrong way.  You need to listen to me.” So your relationship needs to be open enough for you to get that important feedback.

Another point: your vendor has to succeed for you to be successful.  Now, that doesn‘t mean you will turn the other away if they have issues.  It means you hold them to their commitment and see it through.

Partnering in Other Business Areas

The last thing is partnership.  You may get to the point where the volume of backhaul business you are sending a vendor is leveling out because you’re waiting for that next wave.  At that point you need to think: are there other opportunities to help them continue to grow revenue?  Is it in voice services, is it in internet connectivity, is it doing partnerships with other parts of your organization?  In this way, you continue to get them embedded and make sure that they are as committed to your company success as you are.

What was the end result of your fiber deployments?  What did you achieve?

One of the biggest things we achieved at T-Mobile was to simplify our network tremendously and our basic approach was the “one-throat-to-choke” philosophy.

Providers Responsible for End-to-End Circuits

The way our network is constructed today, I no longer have multiple providers delivering connectivity for one circuit: it is only one provider.  A provider has full responsibility for that.  The great thing about that strategy is that when the bill comes in, I don‘t need to have analysts piecing the puzzle together because the vendor responsible for a circuit is tied back to the cell site location building.  So, it helps from an inventory perspective as well.

Higher Reliability in the Thunderstorm Belt

Another plus is we saw a dramatic reduction in outages because we had fiber based technology.  I remember, during our first storm season in Texas, my ops manager gave me a call and said, “Bryan, you’re not gonna believe this, but the thunderstorm came through and nothing went down!” Our past history was such that as storms rolled through various regions of the country, the antiquated LEC infrastructure or bad grounding caused the network to go down and we had to fight the battle to get technicians on the case.

Costs Under Control

We also found a way to break the cost curve.  Today, as revenue and traffic grows, our costs over the past two years have been flat and in some cases have declined.  Our high-speed data network covers well over 200 to 250 million POPs.  And our HSPDA plus network covers about 184 million POPs, but backhaul is not a problem for T-Mobile.

The other nice thing is that 95% of our data traffic lies on this network, so the customers have a great experience and third party companies have put us right in the top two as far as network quality is concerned.

How does software figure into your strategy of managing carrier relationships?

For me, it’s all about managing the facts.  Many times in the last four years people would come into my office and say, “It’s time to rip those guys out of our network.” But then when you pulled up the facts you’d discovered that those people over-reacted because the vendor only had one glitch.

Managing Facts Efficiently and Accurately

But you also need to manage the facts efficiently and accurately.  When I first started working with the carrier management group, I asked: can you give me my carrier spend?  Three and a half weeks later I still didn‘t have those numbers.  Turns out a bunch of people were trying to work these numbers out on a spreadsheet.

So this is where advanced software comes into play.  My philosophy is: you can‘t have a large group of people whose job is to monitor costs and analytics.  It’s the robots’ job to do that.  The humans‘ job is to make the decisions, get the data, analyze it, and secure it.

So there are some great tools out there.  We like Razorsight because they have the flexibility and willingness to listen and understand the changing environment that the carrier has and are willing to adjust on the fly.

Visualization Drives Productivity

We also like the idea of visualizing data.  If your data is confined to a spreadsheet, you are not going to pick up trends.  I don‘t care if you have the highest IQ on the planet, if you are sitting in front of the computer using an Access database or an Excel spreadsheet with 32,000 lines, you’re not going to get the big picture.  I’m convinced that if you can visualize and get that data in a visual format, it drives productivity.  You can spot trends and take actions sooner.

Service Level Agreement Monitoring

Another issue is I’m getting one time charges.  And we’ll often get a charge for a technician who had to repair my circuits and it was the partner’s fault.  Now that’s fine, and we’ll work that out, but on the flip side, if you as a carrier have made a commitment to deliver to me a certain level of service and quality and you don‘t , I’m going to charge you for it because you are impacting my bottom line as far as my customers are concerned.

We are able to monitor our network in real-time.  That information is fed into our vendor management group, which allows us to check how our vendors are performing.  We also monitor Service Level Agreements (SLAs), which ensures our vendors are taking the necessary corrective actions whether it’s processes or not having enough equipment to monitor things.  And it’s amazing how when you start putting in tight SLAs, outages never seem to happen anymore.

Managing data is really, really critical, so the tools are very important.  We use Razorsight to manage our contracts and look at the addressability of our vendors‘ offerings.  And we use other vendors like Spirient to monitor our SLAs, too.  We also use a visual analytics engine called Tableau.

What is your team worried about for the future?  What are some of emerging challenges in the carrier management area?

The first challenge for us is transitioning the legacy transporter TDM network to all IP.  T-Mobile is one of one of the first movers of this trend and it’s key to reducing our costs.  We have basically ripped out 150,000 TDM circuits out of the network over the last 12 months.  We’re now disconnecting a significant number of DS3s and rings and we’ll continue to do this as fiber gets rolled out.  That one single circuit handles whole voice and data at our cell towers.

The Problem with Provisioning for Peak Speeds

The second thing is that while the backhaul ecosystem is strong, the trends in data and enabling technology, particularly around LTE and future versions of that standard, are going to introduce new problems.  What carriers are doing today is enabling backhaul and provisioning to peak speeds.  But that’s not the best strategy for either you or your underlying carrier in terms of costs and required investment.  That’s because peak speeds are only being experienced maybe 1% to 5% of the time.  So, you basically have a very under-utilized pipe the majority of the time.

The Ever-Shrinking Cell Site

One of the ways to handle that problem is to shrink those cells and get them down to low levels to be able to provide very localized capacity.  Now, there’s a challenge with that strategy because you can‘t afford to put a T1 at every one of these.  You also can’t build fiber to every one either.

The likely scenario is that the macro network will still stay in place but each macro cell, from a capacity standpoint becomes almost a min-MSC.  And it’s now too far fetched to think that those cells will have one to 10 Gig connections in the next three to five years.

So all these problems will require creative and cooperative relationships between T-Mobile and both its carrier and software partners.

Copyright 2012 Black Swan Telecom Journal

 

About the Expert

Bryan Fleming

Bryan Fleming is Vice President, Technical Systems and Business Operations at T-Mobile.  Over the past several years, he steered operational excellence, robust infrastructure, and state-of-art business processes at T-Mobile USA.  In his role as head of Carrier Management, he oversaw the connection of all 4G cell sites in 1.5 years.  He partnered with internal teams to secure contracts and to source wireless backhaul facilities from other carriers.  He also introduced an industry-first vendor management program with simplfied billing and dashboards to monitor trends and SLAs.  The scale of his interconnect/cost assurance and partner management programs is one of the most ambitious ever conducted in the mobile industry.

Related Stories

  • Webinar: From Wholesale Settlement  to Global Partner Management by Dan Baker — A 40 minute webinar providing a sweeping view of the challenges and opportunities service providers face as they try to manage a far more complex wholesale and partnering scene.
  • Partner Settlement: The Adaptable Chameleon that Lives Between Wholesale and Retail Billing interview with Arun Kalavath — Partner settlement has been around a long time, but its profile is on the rise because wholesalers are offering new services and content services are bringing settlements into the retail sector.  In this article, you’ll learn about the market forces changing the wholesale business, the expanded portfolio of services that require settlement, and advice on how to pick a settlements solution vendor.
  • Partners in Carrier Management: The Success Story Behind T-Mobile’s Fiber Rollout in Wireless Backhaul interview with Bryan Fleming — Wireless backhaul is the unsung hero of the smartphone’s success.  This interview with T-Mobile’s carrier management architect for backhaul reveals the behind the scenes game plan for one of the most ambitious wireless interconnect programs ever.  You’ll learn about: the reasons for adopting a full-scale fiber strategy; the challenge of finding carrier partners; the clever techniques T-Mobile used to simplify and cut costs; advice on building great relationships with suppliers; and the key role that analytics, assurance, and visualization software played.

Related Articles

  • Webinar: From Wholesale Settlement  to Global Partner Management by Dan Baker — A 40 minute webinar providing a sweeping view of the challenges and opportunities service providers face as they try to manage a far more complex wholesale and partnering scene.
  • Special Report: Wholesale Systems for Advanced Services by Dan Baker — Plenty of technology and market changes are disrupting the telecom wholesale market: advanced services, new kinds of partners,,and the need to monitor relationships and services more closely than was ever required in the circuit voice world.  This article introduces a TRI special report discussing the kind of Wholesale Systems needed in this challenging marketplace.
  • Data Integrity Issues Go Global in the Telecom Wholesale Exchange interview with Vic Bozzo — Wholesale systems enable operators to expand their global reach and work efficiently with partners in an increasingly interdependent telco business.  In this article, you’ll learn about: how many subsystems work together in support of the global trading exchange, why data integrity has become critical, and how wholesale business trends are affecting wholesale systems at Tier 1 and mobile operators.
  • Network Asset Choreography: Subex Teaches Analytics to Dance with Discovery & Life Cycle Management interview with John Brooks — Network provisioning and asset life cycle management is a foreign world to most business assurance professionals because it’s a domain that lives outside the order-to-cash stream.  This article explains how new analytics and network discovery techniques are enabling operators to better track assets, plan capacity, and pave the way for strategic network deployments and decommissioning.
  • Atlanta-lytics: New Telecom Conference Features Revenue and Customer Analytics by Dan Baker — A new telecom conference on analytics in Atlanta (January 29 — 31) is shaping up to be a key event for people who want to stay abreast of this fast moving field.  Black Swan is hosting a half-day, pre-conference workshop entitled, “Financial and Revenue Analytics” featuring seven speakers, many of whom are contributors to this magazine.  Detailed program guide is enclosed.
  • Big Data Financial Analytics: Creating Business Value for Tier 1 Telecoms interview with John Devolites & Atul Jain — What exactly is an analytics company?  This interview with executives at TEOCO helps defines this new breed of telecom solution vendor — a cross between software, consulting, and assurance supplier.  Along with a few Tier 1 case studies, the interview discusses: big-data vs. sampling, analytics project management, and the importance of data access.
  • Margin Analysis: Bolting Profit Assurance onto a Revenue Assurance Platform by Efrat Nissimov — Margin analysis is one of the hottest trends in business assurance.  But what’s all the excitement about?  This article explains the benefits, the market forces driving its adoption, and its advantages over Excel and heritage data warehouse approaches in terms of flexibility, data granularity and quicker time to access.  The author also touts major cost savings if margin analysis is deployed as a module within an existing revenue assurance suite.
  • Partner Settlement: The Adaptable Chameleon that Lives Between Wholesale and Retail Billing interview with Arun Kalavath — Partner settlement has been around a long time, but its profile is on the rise because wholesalers are offering new services and content services are bringing settlements into the retail sector.  In this article, you’ll learn about the market forces changing the wholesale business, the expanded portfolio of services that require settlement, and advice on how to pick a settlements solution vendor.
  • Carrier Usage Audits: Why Regular Checkups Are the Key to a CSP’s Bottom-Line Health by Brian Silvestri — Many telecoms are wholly unaware of the risks of not having a solid usage auditing program for interconnect bills.  Rather than a once-a-year or once-a-quarter audit, this article makes the case for regular monthly audits.  Also discussed are: the different types of audits and organizations involved; the dangers of not auditing often; and factors to consider when selecting a cost-management vendor.
  • Partners in Carrier Management: The Success Story Behind T-Mobile’s Fiber Rollout in Wireless Backhaul interview with Bryan Fleming — Wireless backhaul is the unsung hero of the smartphone’s success.  This interview with T-Mobile’s carrier management architect for backhaul reveals the behind the scenes game plan for one of the most ambitious wireless interconnect programs ever.  You’ll learn about: the reasons for adopting a full-scale fiber strategy; the challenge of finding carrier partners; the clever techniques T-Mobile used to simplify and cut costs; advice on building great relationships with suppliers; and the key role that analytics, assurance, and visualization software played.
  • Will Apple Take a Bite Out of Carrier SMS Revenues?  Likely. by Brian Silvestri — Text messaging generates $20 billion of revenue each year for U.S. carriers alone.  But Apple’s lastest iPhones come with iMessage, a free embedded messaging service.  But while the long term threat to SMS revenue is clear, what strategies should operators be following today?  This article discusses the many factors operators need to consider, offering advice on how mobile operators can keep SMS profits rolling in as long as possible.
  • Invoice Audit Productivity: From 19th Century Textile Mill to SaaS by Peter Yelle — When it comes to invoice reconciliation, are your processes and tools modern and efficient, or is your company stuck in a 19th century time warp?  This article show how operators can vastly reduce their manual audit steps by automating via a SaaS solution.  The  detailed case study and diagram show which tasks were automated to achieve an 80%+ productivity gain.
  • Least Cost Routing and Rate Audits Not Delivering the Savings You Expected?  Here’s Maybe Why by Brian Silvestri — Least cost routing (LCR) is supposed to be one of telecom’s greatest cost-saving innovations.  Yet Connectiv Solution’s research suggests that the LCR programs at many U.S. carriers are losing millions of dollars.  The story explains the traffic analytics steps service providers must take to ensure profitable LCR and other cost assurance programs.
  • Wireless Backhaul: Preparing for the Cleanup When the Dust Settles by Charlie Thomas — Wireless backhaul represents perhaps the largest telecom build-out in the last decade.  This article lays out a strategy for operators as they sooner or later must groom, optimize and re-engineer their backhaul networks based on actual capacity needs, new pricing and improved supplier agreements.  The article lays out a 6-point plan for ensuring success.
  • Is Offering Free International LD on Your Horizon?  Follow These Checkpoints by Hulya Altinsoy — Free is a magic word for consumers.  So how about adding free international long distance (LD) to your list of services?  While free international LD serivce is fraught with risk, the article presents a  5-point checklist to help get your bearings before you make the leap.  The article also presents an analysis of billions of minutes in international LD calls by U.S. wireless carriers.
  • Real-Time Network Intelligence: The New Way to Read Telecom Tea Leaves by Suren Nathan — Real-time network intelligence is the key to deciding which products to launch, whose facilities to lease, and where to route traffic.  The article explains why telecoms — and especially enhanced service providers --  should ideally be equipped with both a fine-grained margin analysis solution and a SaaS platform, offering an upgrade path that requires no internat IT support.
  • International Call Routing: A Challenging Area to Master, But Payoffs Can Be Big interview with John Fitzpatrick — International call routing offers a completely different set of challenges than the common practices found in North America.  This article explains the many issues faced in international LCR from imprecise routing and number portability headaches to dispute problems and the flood of wholesaler rate sheets that must be rapidly made sense of and implemented in the routing plan.
  • Dynamic Call Routing: The Market Enabler that Allows VoIP to Improve with Age interview with Neal Axelrad — Least cost routing (LCR) has matured greatly in recent years.  Today’s routing is far more dynamic because of new real-time capabilities.  This article explains technology advances that instaneously re-routie calls based on real-time intellgence gathering from the network.  Also discussed are template-driven rate addendums, SaaS delivery, and the economics of LCR.
  • Network Inventory Integrity: Taking Cost Management to a Higher Level interview with Suren Nathan — Recovering stranded assets and capacity is essential for minimizing CAPEX and leasing costs.  This article makes the case for reconciling network inventory through the many ordering, billing, and other systems that interact with that inventory.
  • An Automated Self-Audit Approach to Telecom Cost Assurance interview with Jim Buttafuoco — What’s the value of an automated approach to invoice validation?  This article explains the power of the SaaS model where the vendor supplies the data-processing expertise, relieves the operator of tedious manual work and boosts auditor productivity so more money is saved.
  • Meet Your New Cost Management Solution: A Tandem Hub Carrier interview with Surendra Saboo — In the telco interconnect game, frustrations are many: high tolls, jurisdiction scams, and the expense of connecting to a partner.  This article explains how tandem or aggregation carriers are making life easier and saving operators a boatload of money.  Learn the difference between peering and a “tandem hub” and why this trend is becoming increasingly global.
  • Cost Assurance: Dealing with Rating Variety in Wholesale Bills by David West — Validation of intercarrier wholesale invoices seems like a fairly straight forward process.  But regulatory rules and a variety of bill rating methods have made it very complex.  This article details the challenge by walking through a wireless carrier case, then laying out the argument why operators should opt for a custom solution based on a core reporting engine.