Partners in Carrier Management: The Success Story Behind T-Mobile’s Fiber Rollout in Wireless Backhaul

iPhones and Androids didn‘t create the mobile broadband revolution on their own.  The smartphone market would have never taken off had operators not built out a reliable and cost effective fixed broadband infrastructure to offload radio data traffic.

Back in 2006, the massive fixed broadband infrastructure that exists today to support smartphones didn‘t really exist.  And that successful build out was a culmination of many gut-wrenching decisions, risk-laden investments, and full-scale cooperation from a variety of telecom industry players.

Truth be told, wireless backhaul is the unsung hero of the smartphone’s success.  And here to give us some perspective on how it all evolved is Bryan Fleming, Vice President, Technical Systems and Business Operations at T-Mobile USA, who I met at Razosight’s recent user conference in Washington D.C.

What I love about Bryan’s story is the way he tells it.  Bryan weaves an interesting -- even entertaining — tale that almost anyone in telecom can relate to.  Of course, for people in the interconnect and carrier management areas, Bryan’s first-hand perspective is invaluable.

Among the insights you’ll take away: the reasons for adopting a full-scale fiber strategy; the challenge of finding carriers to support T-Mobile’s vision; the clever techniques T-Mobile used to simplify and cut costs; his expert advice on how to build and maintain great relationships with suppliers; and the key role analytics, assurance and visualization software play in T-Mobile’s current backhaul program.

	Dan Baker: Bryan, to begin, it would be great if you could give us some context, perhaps a quick backgrounder on T-Mobile’s situation.

Bryan Fleming:  Sure, Dan.  I guess like every other wireless carrier, T-Mobile USA has seen major smart phone penetration in the past two years.  Our latest figures say there are 11.5 million smart phones in our network.  In fact, 90% or more of the devices we sell today are smart phone devices.  And as you can well imagine, it’s things like social networking, video applications, and instant access to information that’s driving demand.

Another thing about T-Mobile — and I’ve been with the company almost 13 years — is it likes to be on the leading-edge.  We are frequently the first to adopt new innovations, whether it be Android rollouts or apps like wired-enabled autos.

So from the year 2007 forward, we’ve put a tremendous emphasis on trunking and data connectivity to support our heavy wireless usage.

	At some point, T-Mobile chose to go “all in” with a full-blown fiber back haul strategy.  What were some of the reasons you went in that direction?

Well, it was back in 2006 and 2007 when we first faced the mobile data tsunami is coming.  What scared mobile operators like us was that if data grew as fast as we expected and users didn‘t pay a premium for it, our business model was going to be turned upside down.

T1 Economics Didn‘t Work

So the only way we could keep up with wireless network growth was through T1 lines, which would have been an unsustainable model.  A T1 gives you 1.5 Mbits and its typical cost -- including associated connectivity such DS3s and OCNs -- was between $350 and $500.  It wasn‘t going to work.

What’s more, the equipment manufacturers were also predicting Doomsday.  It freaked us out.  And when the iPhone launched in mid-2007 we could see all sorts of cost and customer experience problems in our future.

A Fiber Revelation

When we were in midst of this crisis, my boss called me up and excitedly explained that he’d had one of those famous “epiphanies in the shower” that morning.  He was absolutely convinced that the way to solve our problem was through fiber.

Now initially I was very skeptical of that decision, even though that indeed turned out to be the winning strategy.  At the time, the fiber choice wasn‘t so obvious.  In fact a lot of companies invested in the microwave route, but that ultimately didn’t work because you have to spend money over and over again with microwave.

	What sort of virtues were you looking for in your carrier suppliers?

Well, we really focused on three key things.

Availability

First, we made sure that fiber was available in the majority of sites that meant the most to us.  We wanted to make sure that if we changed vendors or made long-term commitments to the ones we already had that they were committed to improving the reliability of the network.  And the former Ma Bells were reluctant to invest for the future in their old wireline networks — and I don‘t blame them.

Simplify Billing

Our second principle of partnering was around cost.  Not only did we have to stop the upward slope of the cost curve, we had to simplify billing.  Billing is an absolute nightmare.  The best analogy for a TDM network is it’s a big bowl of spaghetti where no one knows which end is which.  It creates a great deal of confusion not only in your inventory systems, but also in terms of auditing supplier bills and managing reliability.

Timely Delivery

We also had to know that our vendors were committed to deliver what we needed on time.  We work on very short timeframes and our mindset was to enable those vendors to deploy as quickly as possible.  So, with these points in mind, we started to engage the vendor community.

	What kind of carriers did you approach as suppliers of your next generation back haul?

Well, as you can imagine, our first conversations were with the traditional and incumbent local exchange carriers because that’s who we had our current infrastructure with.

The Incumbent LECs Laughed at Our Plan

In my discussions with the LECs, I don‘t know how many times in that next six months I got laughed at and told, “Good luck, Fleming, but we’re not playing.”  So, I took note of that, jotted it down, and continued on my way.

The Microwave Carriers Step Up

So what we were able to find -- and what you always need when your ecosystem is new and needs to be developed — is someone who believes in your story that data is coming and that there’s going to be revenue growth.

The companies who first stepped up to support us were the microwave-based companies like FiberTower and TTM.  As you probably know, FiberTower currently is in bankruptcy, but back in 2005-2006, they were the first alternative access vendor.  And they made good on their promise to replace the LEC with a better cost and better reliability solution.  Their problem was they did it using microwave and didn‘t fully anticipate the big rollout in wireless data.

A Secondary Business Opportunity for the Cable Operators

Our next question became: who do we look at if the LECs won‘t play and the microwave players can’t keep up?  Well, that’s when we turned to the cable community.  And our pitch to the cable operators was that they had a valuable asset in the ground that they were not fully leveraging: delivering video service to the home was a great business model but there are other growth opportunities there too.  So, our first mover on that concept was Time Warner Cable.  And interestingly enough, as soon as that deal was done, everybody in the cable industry followed at a very rapid pace.

The Former Lunatic Becomes the Most Hunted

Fast forward another 8 to 12 months and we still had no LEC deals.  Every meeting I had with them ended with, “You’re a lunatic -- good luck, you’re not going to be able to get your plan done.” So to demonstrate we were serious, as we began to install fiber with Time Warner, we stopped talking to the LECs and started disconnecting their service.  And sure enough, all of a sudden the “lunatic” in the room became the most hunted because their bottom line was being affected and they realized their business was at risk.  So the LECs came along after that relatively quickly.

And as far as the LEC space, the first mover, if you can believe it was AT&T, and in some cases I would say they are probably one of the most cost effective vendors out there today.

Now while we were concerned with backhaul, a lot of the companies who supply backhaul deliver things other than backhaul and in most cases, backhaul is not their first business.  So we found other opportunities to leverage growth to drive out our costs and find synergies in which to grow together.  That’s really, really important.

Looking back to 2006-2007, roughly 95% of our backhaul network at T-Mobile was supplied by the LEC incumbents.  Today, if you look at our broadband footprint that supports our HSPDA plus network, the LEC incumbents have about 47% share and that continues to decline because there are new providers out there who are aggressive and have a strong capital backing.

	Carrier/vendor negotiations can be a fairly brutal experience for the vendor.  Traditionally a lot of pressure is put on the carriers supplying software, hardware, and services.

That’s true.  But on the backhaul business, if you’re out to beat your supplier down to the lowest possible price, it’s not going to work because this is an extremely capital intensive business.  And you want the capital money to be available to grow that ecosystem because those are the things that drive your costs, innovation, and availability.

Knowing the Supplier’s Business Model & Constraints

To successfully negotiate and find a good partner in this environment, you have to understand what your supplier’s business model is.  That’s very critical.  Third thing is open communication: you really have to understand what is important to both parties to find a good deal that works for everybody without risking the relationship or coming away disappointed in what you got.

Each of the vendors will be in a different financial situation.  What kind of capital do they have?  You need to understand that and be flexible.  And what might work from a contract or deal perspective with a large company like AT&T or a Comcast or Time Warner, might not work for a middle or a small tier carrier.

The deal needs to be structured in a way that makes their financial backers comfortable that you are going to be around for a long time for them.  When they are confident of that, then they can offer you the right price.

Open Communications and Commitment

You want to foster creativity.  Instead of working on a traditional contract, there might be some other alternatives out there, but it has to start with open communication.

The fourth thing is being committed to that relationship.  If you hit a bump in the road, just don‘t dump the relationship.  Especially in a growing area like wireless backhaul that you know can only become more complex, you need strong allies in your partners.  As wireless carriers, to be able to deliver that content, the system will have to evolve, and that will require creative solutions that your provider will supply.

Good communications is absolutely key.  Sometimes, they may even want to pull you aside and honestly say, “Hey you’re being an idiot.  You are doing this the wrong way.  You need to listen to me.” So your relationship needs to be open enough for you to get that important feedback.

Another point: your vendor has to succeed for you to be successful.  Now, that doesn‘t mean you will turn the other away if they have issues.  It means you hold them to their commitment and see it through.

Partnering in Other Business Areas

The last thing is partnership.  You may get to the point where the volume of backhaul business you are sending a vendor is leveling out because you’re waiting for that next wave.  At that point you need to think: are there other opportunities to help them continue to grow revenue?  Is it in voice services, is it in internet connectivity, is it doing partnerships with other parts of your organization?  In this way, you continue to get them embedded and make sure that they are as committed to your company success as you are.

	What was the end result of your fiber deployments?  What did you achieve?

One of the biggest things we achieved at T-Mobile was to simplify our network tremendously and our basic approach was the “one-throat-to-choke” philosophy.

Providers Responsible for End-to-End Circuits

The way our network is constructed today, I no longer have multiple providers delivering connectivity for one circuit: it is only one provider.  A provider has full responsibility for that.  The great thing about that strategy is that when the bill comes in, I don‘t need to have analysts piecing the puzzle together because the vendor responsible for a circuit is tied back to the cell site location building.  So, it helps from an inventory perspective as well.

Higher Reliability in the Thunderstorm Belt

Another plus is we saw a dramatic reduction in outages because we had fiber based technology.  I remember, during our first storm season in Texas, my ops manager gave me a call and said, “Bryan, you’re not gonna believe this, but the thunderstorm came through and nothing went down!” Our past history was such that as storms rolled through various regions of the country, the antiquated LEC infrastructure or bad grounding caused the network to go down and we had to fight the battle to get technicians on the case.

Costs Under Control

We also found a way to break the cost curve.  Today, as revenue and traffic grows, our costs over the past two years have been flat and in some cases have declined.  Our high-speed data network covers well over 200 to 250 million POPs.  And our HSPDA plus network covers about 184 million POPs, but backhaul is not a problem for T-Mobile.

The other nice thing is that 95% of our data traffic lies on this network, so the customers have a great experience and third party companies have put us right in the top two as far as network quality is concerned.

	How does software figure into your strategy of managing carrier relationships?

For me, it’s all about managing the facts.  Many times in the last four years people would come into my office and say, “It’s time to rip those guys out of our network.” But then when you pulled up the facts you’d discovered that those people over-reacted because the vendor only had one glitch.

Managing Facts Efficiently and Accurately

But you also need to manage the facts efficiently and accurately.  When I first started working with the carrier management group, I asked: can you give me my carrier spend?  Three and a half weeks later I still didn‘t have those numbers.  Turns out a bunch of people were trying to work these numbers out on a spreadsheet.

So this is where advanced software comes into play.  My philosophy is: you can‘t have a large group of people whose job is to monitor costs and analytics.  It’s the robots’ job to do that.  The humans‘ job is to make the decisions, get the data, analyze it, and secure it.

So there are some great tools out there.  We like Razorsight because they have the flexibility and willingness to listen and understand the changing environment that the carrier has and are willing to adjust on the fly.

Visualization Drives Productivity

We also like the idea of visualizing data.  If your data is confined to a spreadsheet, you are not going to pick up trends.  I don‘t care if you have the highest IQ on the planet, if you are sitting in front of the computer using an Access database or an Excel spreadsheet with 32,000 lines, you’re not going to get the big picture.  I’m convinced that if you can visualize and get that data in a visual format, it drives productivity.  You can spot trends and take actions sooner.

Service Level Agreement Monitoring

Another issue is I’m getting one time charges.  And we’ll often get a charge for a technician who had to repair my circuits and it was the partner’s fault.  Now that’s fine, and we’ll work that out, but on the flip side, if you as a carrier have made a commitment to deliver to me a certain level of service and quality and you don‘t , I’m going to charge you for it because you are impacting my bottom line as far as my customers are concerned.

We are able to monitor our network in real-time.  That information is fed into our vendor management group, which allows us to check how our vendors are performing.  We also monitor Service Level Agreements (SLAs), which ensures our vendors are taking the necessary corrective actions whether it’s processes or not having enough equipment to monitor things.  And it’s amazing how when you start putting in tight SLAs, outages never seem to happen anymore.

Managing data is really, really critical, so the tools are very important.  We use Razorsight to manage our contracts and look at the addressability of our vendors‘ offerings.  And we use other vendors like Spirient to monitor our SLAs, too.  We also use a visual analytics engine called Tableau.

	What is your team worried about for the future?  What are some of emerging challenges in the carrier management area?

The first challenge for us is transitioning the legacy transporter TDM network to all IP.  T-Mobile is one of one of the first movers of this trend and it’s key to reducing our costs.  We have basically ripped out 150,000 TDM circuits out of the network over the last 12 months.  We’re now disconnecting a significant number of DS3s and rings and we’ll continue to do this as fiber gets rolled out.  That one single circuit handles whole voice and data at our cell towers.

The Problem with Provisioning for Peak Speeds

The second thing is that while the backhaul ecosystem is strong, the trends in data and enabling technology, particularly around LTE and future versions of that standard, are going to introduce new problems.  What carriers are doing today is enabling backhaul and provisioning to peak speeds.  But that’s not the best strategy for either you or your underlying carrier in terms of costs and required investment.  That’s because peak speeds are only being experienced maybe 1% to 5% of the time.  So, you basically have a very under-utilized pipe the majority of the time.

The Ever-Shrinking Cell Site

One of the ways to handle that problem is to shrink those cells and get them down to low levels to be able to provide very localized capacity.  Now, there’s a challenge with that strategy because you can‘t afford to put a T1 at every one of these.  You also can’t build fiber to every one either.

The likely scenario is that the macro network will still stay in place but each macro cell, from a capacity standpoint becomes almost a min-MSC.  And it’s now too far fetched to think that those cells will have one to 10 Gig connections in the next three to five years.

So all these problems will require creative and cooperative relationships between T-Mobile and both its carrier and software partners.