… but maybe some parts of the network should be. What the heck am I talking about? Actually, when you remove the fluff, I’m talking about optimizing the network – striking a long term, cost-advantaged balance between CapEx and OpEx. Which is another way of saying that, sometimes, spending a little more money now saves you lots more money over time. Here’s an example I’ve been working with for an SP lately. They have multiple data centers, where their subscriber traffic is supported. Subscriber numbers are growing, and the use per subscriber is also growing. They need to add capacity, but not necessarily evenly. Some subscribers still use older network access devices, and need more supporting services from the operators’ network. Most newer network access devices, however, use very little from the SP and simply want to get to the Internet. The result of the subscriber growth plus subscriber usage growth plus device evolution means the SP needs to both add capacity – lots of capacity – and rethink the network design at the same time. A problem with adding capacity in the network is that, unless proportionate capacity is incremented everywhere, the design has now changed – maybe a little, maybe a lot, most likely somewhere between those two extremes. As a result, every capacity add must be engineered from scratch, because things keep changing. Of course, if keeping the current proportionality of resources is important (so your design doesn’t fundamentally change), you’d have to create unnecessary capacity in those network elements which users are evolving away from, but hey – your design would stay the same. Unfortunately, that’s like hitting every golf shot with a driver, when maybe sometimes you need a wedge – or a putter. The problem, to continue the metaphor, is when you’ve put yourself in the game with only one club in your bag. Suppose you could have a bag with – what an idea!! – different clubs for different sizes of golf problems? How about different “modules” of network capacity for different network needs? Isn’t that sort of like having a driver for long shots, a wedge for lifted shots, and a putter for those close-controlled shots on the green? It’s true you’d winding up spending more money for three clubs than you spent for only one … but your golf game would be a lot better. If your network needs to grow faster than you can keep redesigning it, maybe it’s time to accept a little higher CapEx to create modules/zones/sectors which are designed as drop-in units. It’s not quite plug-and-play, but pre-engineered, standardized modules will be faster to deploy. The savings in design, review, redesign, catching up with what’s changed since the last design, and now fitting in a custom addition … just may add up to an OpEx reduction that exceeds the CapEx increment. Not to mention that increasing capacity as fast as you need just might reduce churn. Wouldn’t that be nice? ... View more

I recently had the less-than-entertaining experience of a series of emails exchanged with a customer (manager level) on how to measure the “perceived user experience of throughput” through one of our nodes in their network. Since this node is one of many – very many – in the end customer’s data path, I responded with a link to the DSL Reports web site and its series of speed tests (http://www.dslreports.com/speedtest?flash=1 – I find them useful for comparison purposes). That, after all, is the throughput the user will perceive. He replied, most unhappily. It seemed, after much email talk back-and-forth, that what he really wanted was a means to measure the per-user throughput via a MIB on this particular node. Since the node has up to 40 distributed processors, each of which is capable of supporting 65,000 (old hardware)/136,000 (new hardware) user sessions simultaneously, and creating/deleting them at a rate of 100/200 per second, we don’t track the per-session throughput for each session. What I was able to provide him was a set of MIBs – which they were already collecting – which measured the instantaneous throughput in each direction and which counted the number of active sessions at any given time. From this, they can derive averages, but only averages. Those averages may or may not be representative of their customers’ actual throughput experience … their customers, for instance, could be quite bursty in their traffic, but on average not use very much at all. In addition, the “user experience” involves latency at a large number of other nodes in the path of the data flows. What I’m afraid this manager is going to get is some specific data which conveys to him a picture that doesn’t necessarily match reality. Data is not the same thing as information. Information has contextual meaning. As an example, consider this blog post, which I found quite interesting: http://pajamasmedia.com/richardfernandez/2009/07/20/hockey-stick/ I, too, thought the graph displayed was something else, though the particular thing I expected it to be reflected my biases, and wasn’t the thing the author suggested most people would expect. That’s an important thought – I had biases in what I thought the data was telling me. And – like anyone else – I have biases in how I ask a question. What I look for in KPIs from my nodes reflects my biases as to what’s important. What I look for reflects my education, my experience, and my exposure to other peoples’   experiences, usually in the form of horror stories. As the saying goes, Your Mileage May Vary. ... View more

In the past five plus years that I've been working with mobile SPs, traffic has grown, then grown some more, and now is exploding. While that's a better business problem to have than the alternative (!), it does have its own little headaches. The one that's on my mind the most these days is the problem of scale. In an ideal SP's data center, all the nodes supporting customer traffic would be big enough, fast enough, and have enough headroom to absorb the slings and arrows of outrageous outages elsewhere. Likewise the nodes supporting the nodes carrying (paying) customer traffic would also be big/fast/headroom enough, too. The problem I keep seeing, though, is the inevitable mismatch of capacity among the nodes leads to network designs which attempt to minimize the stranded capacity. That makes sense, in a static universe, because it minimizes the excess expense to build and operate the data center. Unfortunately, the mobile SP's universe is anything but static these days. Mobile devices are becoming more capable, subscribers are more willing to use them for new services, and there's a growing use of the mobile network for machine-to-machine communications. That last item is invisible to most folks - it's not exciting, it's not dramatic, and it sure isn't sexy. But let me give you a couple of examples, just from the universe I've been working in lately: A field (obviously with high value crops!) has sensors embedded in the soil which report moisture and pH levels to the irrigation controller. No farmer intervention is required to know when to turn on or off the irrigation system. A vending machine company operates smart machines, which report their consumption/inventory levels before the truck rolls out on its resupply run - saving the operator time, money, and also (incidentally) enabling very powerful tracking of customer consumption trends. Alarm systems which don't depend on the "phone line" are less likely to be disrupted by the clever thief who first cuts the phone line before breaking his way in. Gas meters for larger operations should be read daily (think plastic manufacturing) - and they are, on a cron job, over the cellular network. That's happening today (and has been, for several years) - imagine that expanding to Smart Grid. When the mobile wireless is more about “wireless” than it is about “mobile” – the possibilities grow enormously. But that means the SP serving those customers needs to be able to grow the data center capacity, too. And fast. That’s the problem I see. Datacenters that are “just right” today are woefully inadequate six months from now. The needs will morph from today’s capacity mix to something different. Nodes/services which were the choke point last year will not be the critical point next year. Adding capacity becomes a fundamental network redesign effort – and it happens every time you need to increase capacity. What seems to work, though it is admittedly not as efficient as we’d like in a perfect world, is to change the fundamental principle behind how we design the SP’s data center. We need to go from a “big enough” mindset to a “fast enough” one. The fastest way to increase network capacity is to make the underlying design modular. Instead of increasing capacity in the current pain point, only to create a new one a few months from now, establish capacity sets. If I need more WAP capacity, I add these nodes, in this standard arrangement, with these standardized connection points into the existing network. If I need laptop connect (aircard) capacity, the nodes/arrangement/connection point set will be different, but it can still be standardized and known in advance. The modular design will result in somewhat more stranded capacity in each module (or zone … or set … or whatever you choose to call it). However, with the standardization of the hardware/software/connections/etc. in the design, you quite possibly gain enough in speed to be able to deploy capacity increases in a more timely fashion. Instead of having to build the data center today to carry the capacity you need in 18 months, if you can add capacity with 3 month lead times, you can stage in the increases, spreading capex over time instead of in massive surges. Obviously, the fundamental data center design must be made to accommodate these “drop in” modules. But that’s a design which can scale – and do so in the timelines SPs are dealing with now. Imagine what it will be like when we come out on the other side of the current economic slowdown, and everyone tries to catch up? Scale matters – and especially the ability to scale. ... View more