Re: Best Telco Stories - Page 5

Kimberly Adams · ‎01-05-2011

Most of us in the networking world have had issues working with telcos. Here is your chances to share your funny telco stories!

Here is mine....

One site had a frame relay circuit that was bouncing and spent more time down then up. The client was running on their backup ISDN circuit and getting huge bills. I kept working with the telco and they kept saying it was a CPE issue. We replaced the WIC 3 times and same condition. I told them that the issue was with the line and when they tested it they said they saw no issue. Finally after about two months of this, I asked my field engineer to go on site and hang out for at least 2 hours. Then I called the telco and told them that all the equipment was replaced, the chassis, card (again), everything. The telco tech broke into the circuit and told me that he now sees the issue. That the faulty Cisco equipment was masking a secondary issue with the circuit. They repaired the circuit and it came back up.

The funnies part of this, we didn't replace anything the last time. The chassis was never replaced and the WIC was, but it was never the issue.

I just love dealing with some of the telco's out there!

Happy New Year to all.

Kimberly

Thanks and Cheers! Kimberly Please remember to rate helpful posts.

smitesh kharecha · ‎10-21-2011

Hi All,

I have my own share of stories as well...

but one I enjoyed and laughed my A** out till today is as below:

Me and telco engineer were working on some issue related to BGP.

After much troubleshooting and all, i was surprise to hear from telco engineer was

"Sir why don't yuo told me that BGP is in Active state. However what is still confusing me is that even though it is in Active state, then why it is not passing traffic."

I was speechless for moment at that guy's bold statement that why it is not passing traffic even when BGP is in Active state. It took me couple of minutes to recover and then to explain him that Active state is not normal and in Active state; never any traffic will be forwared.

Regards,

Smitesh

darren.g · ‎11-08-2011

Ahhh, I should get out more and check out other forums once in a while. I didn't even realise this one was here!

For your enjoyment. Do ISP's qualify as Telco's? Close enough, I guess.

Current $POE has its own BGP AS and /23, so has two uplink ISP's, one of which is quite a bit slower than the other and only used for backup advertisement purposes - it also has a rather complex arrangement of private MPLS clouds linkint it to various teleworkers and clients.

Earlier this week, I had a complaint from some of our remote users - not connected to the teleworker cloud, but coming in via the internet - that our FTP server was running slow for outbound (download) traffic.

Since this server is used to deliver video files inbound and outbound of the half to one gigabyte range, this is somewhat of a concern - the regular 20 meg link speed is often complained about - if this was running slower than that, the brown sticky stuff would quickly hit the rotating air distribution blades.

I logged on to the FTP server and ran a couple of tests. Sure enough, inbound was fine - 18 meg a second (close enough, allowing for the shared office flotsam and jetsam it gets used for as well), but outbound was running at around 384 *KILOBITS* per second - major bad karma! I should add that I graph the interface utilisation using Cacti, so I know that there wasn't any other traffic clogging the pipe at the time.

Logged a job with the (outsourced) support organisation, who in turn forward this to the ISP - who get back to me about 3 hours later and say "We can;t find anything wrong - can you test to *this* server".

The server they wanted me to test to was located exactly two router hops away from my server - and one of those was my edge router! They wanted me to test to a server located on their premesis.

I came back with "Erm, no, we don't PAY you for a 20 meg link to YOUR router, we pay you for 20 meg to the INTERNET - you know, all those other IP addresses out there which people might like to use?". They kept insisting that since I could FTP at full speed to their server nothing was wrong with my link - at which point I got rather mad and said many, many rude words.

14 hours and *much* escalation later (I ended up on the phone to the CTO of our outsourced support organisation threatening to come down the road and camp on his doorstep until he got this fixed), the link came good and I was getting outbound traffic at the rates I expected.

The explaination from the ISP - "We couldn't find anything wrong, and didn't change anything". Another of those 'magic" fixes.

Far out. I'd be better off *with* Telstra than this bunch of muppets. As soon as I can convince the boss that we have ample grounds to break the contract with this ISP on the grounds of not meeting the requirements, I'm gonna talk to PIPE or Nextgen and see what they can offer me!

Cheers

Leo Laohoo · ‎11-09-2011

Hi Darren,

Australian telco's will never admit that it's their fault (particularly if only a few clients are affected) because they don't want to pay for the penalties. However, things change when you start posting your investigations into AusNOG and people start saying, "Hey, I'm seeing the same thing too!".

My story:

We have site in Jarvis Bay, NSW. It's an ADSL site that we got from TransACT (we call it with a different term but I'm afraid the filter may not like it) [OK, OK, OK. It's starts with "Tr" but rhymes with the word "Granny".], who got it from AAPT (aka Ask Another Person Tomorrow) who them got it from Telstra. Anyway, at around 2 pm on a Thursday, the link went dead. The site took the initiative and called us within 15 minutes. Normally, I would be expecting the lines of "... our network is down. How long can you get this fixed?". Not this time. This call gave everyone a chuckle. The caller went "... our network is down. We notice that a Telstra linesman was down the manhole before the outage. The manhole is located INSIDE our premises. When we asked the Telstra linesman what he was doing, he said he's running a new ADSL line to a property up the street."

Sensing I have some information, I called TransACT and relayed exactly what the client told us. Up the foodchain this information got. Lo and behold! The link went back up 3 hours later. What did Telstra say? "We didn't do anything. It's a problem with YOUR equipment."

The fact that a linesman was down the Telstra manhole and inside the mini-sub-exchange before the outage was "purely coincidental".

Pull the other leg Telstra, it's got bells.

darren.g · ‎11-09-2011

Leo.

Oh man, you're not kidding. I'll make the embarrassing admission in a whisper - I, once upon a time when I was young and foolish and naive and believed in a work ethic and that putting in hard yakka mattered, worked for Telstra. Well, they were called Telecom when I started working for them, then the name changed. I've seen it from the inside - in the Telecom days, there was none of this "public owned, make money for shareholders" cr*p - people actually cared, and did their best to make things work.

Fast forward a few years, and it became "Close the fault as quickly as possible regardless of the fix" - I worked on mainly internal systems, but I had cr*p like "We didn't do anything" thrown at me - when I had been standing watching the muppet *break* the damn link in the first place.

Thankfully, those days are long gone, and I now won't have anything to do with Telstra unless I absolutely have to - my mobile phone and mobile data SIM for my laptop are with them, but only because the company pays for it, and it's realisticaslly the only way to get 3G working reliably in some of the places I have to go. Optus stinks, and let's not get started on Vodafail!

The sooner the NBN gets rolled out the better, as far as I'm concerned. I've already got my ISP and plan selected for the day when they finally roll the cable down my street. :-)

Cheers.

Leo Laohoo · ‎11-10-2011

Yo Darren mate,

Unfortunately we are Aussies and we are used to or expected to pay a premium price for crappy service and support. I still don't understand why big European or American service companies are reluctant to set up shop here (and more competition).

I like to call this form of service as a-race-to-the-bottom.

darren.g · ‎11-10-2011

leolaohoo wrote:

Yo Darren mate,

Unfortunately we are Aussies. We are used to paying a premium price for crappy service. I still don't understand why big European or American service companies are reluctant to set up shop here (and more competition).

I like to call the service Aussies are getting the race-to-the-bottom.

Leo.

I do understand the lack of internationals, actually.

If you look at the geographics of Australia - we have 22 million people in a land area roughly equivalent to the continental United States (not including Alaska and Hawaii) - where they have better than 300 million people.

Simple economies of scale. We're a smaller pond, client wise, but would be just as expensive (more, really, because we're more spread out) to deliver services to anywhere but the major cities/hubs. Look at Optus - they tried, failed, and had to make do by merging with a mob from Singapore because they couldn't keep up with the capital costs in Australia.

Not that America has it so great if you're not in a big-ish city either - I have some friends in the US who are still on dial-up - they can't even get wireless (3G)!

Unfortunately for us, no government has had the stones to actually stand up to Telstra - Sol and his three amigos were so antagonstic towards regulation that Telstra gets handled with tongs even today - and break it into a wholesale and retail arm - it's just one big bundle, and can stifle competition by charging "itself" less for retail access than it charges other companies for wholesale!

That's why I'm hanging for the NBN - it removes Telstra's ability to screw over competitors by claiming they don't have "capacity" in exchanges for competiting equipment (a fallacy in 99.9% of the cases it's claimed) or by restricting access to others on the basis of "technical issues" with interconnects.

Ahhh, that's my rant for the morning. :-)

Cheers.

Leo Laohoo · ‎11-10-2011

I do understand the lack of internationals, actually.

I do recall a few months back when Aussies were complaining about prices in Australia being more expensive (than the rest of the world) and the response from the retailers was "It's the price of doing business in Australia.".

Next the Aussie dollar went parity and beyond, Aussies found out the wonder of doing e-commerce from overseas. Now the retailers are complaining and the public reply is "It's the price of doing business in Australia".

LOL