Best Telco Stories

[Kimberly Adams]

Most of us in the networking world have had issues working with telcos.  Here is your chances to share your funny telco stories!

Here is mine....

One site had a frame relay circuit that was bouncing and spent more time down then up.  The client was running on their backup ISDN circuit and getting huge bills.  I kept working with the telco and they kept saying it was a CPE issue.  We replaced the WIC 3 times and same condition.  I told them that the issue was with the line and when they tested it they said they saw no issue.  Finally after about two months of this, I asked my field engineer to go on site and hang out for at least 2 hours.  Then I called the telco and told them that all the equipment was replaced, the chassis, card (again), everything.  The telco tech broke into the circuit and told me that he now sees the issue.  That the faulty Cisco equipment was masking a secondary issue with the circuit.  They repaired the circuit and it came back up.

The funnies part of this, we didn't replace anything the last time.  The chassis was never replaced and the WIC was, but it was never the issue.

I just love dealing with some of the telco's out there!

Happy New Year to all.

Kimberly

[JohnHeller_2]

That really sounds like an interview from hell.

The sort of questions that they were asking are the sort of things that you memorise parrot fashion for the routing exam for your CCNP exam and then promptly forget.

If you ever get stuck on a real life WAN routing issue, you can always drag out your textbook again. But in all seriousness so much of what we learn in Cisco courses we are unlikely to use again. If we do come across something unfamiliar, there are so many other resources to refer to. Command references, course material, forum sites aka here.

I think that your hubby aquitted himself well if he could describe how to implement and troubleshoot WAN systems, and that company wasn't worth working for.

I know that America is having a hard time economically, but it is a great shame that people with your skills are getting laid off or can't find work.

Is it a problem in your area, or over the whole country?

Just move to Australia....we need you. Lots of work here . Or try for an overseas posting with someone like raytheon?

[Kimberly Adams]

Leo,

I would never get offended and I am glad he walked out the interview.  This is most definitely some place that I don't think anyone with any sense would want to work.  We are not that desperate that he would have to work at this place.  By the way, they really didn't pay THAT much and not enough to endure that kind of environment on a daily basis.

Honestly, my husband doesn't do well in places like this.  He can get seriously "lippy" and run his mouth.  Which means that he wouldn't have lasted very long there anyway.  I told him that this is beyond a good thing he didn't get this one, there is another out there waiting for him.

John,

The economy here in the States has gotten better in the last couple of years.  He gave up on looking for work when I beat him out for a job and decided to go back to school....until I got laid off.  But now he has an offer coming today and he is pretty excited about it.  We will see how things go here, but good to know that other areas are doing good.  There are jobs here, but you have to be willing to move or work in areas you really don't like.  But oh well, what can you do?

Cheers all!

Kimberly

[Leo Laohoo]

Back in 2007 I was involved in a project to upgrade about 20 (or so) sites from Frame Relay to Telstra's GWIP (Government-based Wideband IP).  It comes in various "sizes":  4, 8, 12, 20, 24 mbps, etc.

So anyway we decided to upgrade the small sites first.  We gave the list of sites to Telstra along with the bandwidth the client would like them to be.  Three weeks later Telstra sends us an email saying that everything is configured and the links are terminated at the site.  Good to go.

Upgrade of the routers went well and we plugged the sites in and all were well.  Ok.  So far so good.  The client was happy with the result so we went full steam ahead.

The last site was the head office and it's suppose to be upgraded to 20 mbps.  Ok.  Everything goes as plan and the site was upgraded to the new GWIP on Friday 6pm.  Perfect.  Verifications went well.  Pat-on-the-back and have-a-nice-weekend.  Lights out.

Come Monday everything came to a grinding halt.  We all looked at each other, scratched our heads and wished we called in sick today.  Everyone was hands on deck.  We checked server logs, router logs, link test.  The works.  We don't know why accessing the database found in the head office was slow and to the point of timing out.

Then someone decided to do something silly.  He brought in his personal laptop (a no-no because of security issues) into the network fired up CyberGauge and pointed to the router in the head office.  No one cared what this proggie can do.  Everyone was doing CLI on the routers, ASA, FWSM, switch.

Then came a question everyone dread:  Is the head office suppose to run 20 mbps or 8 mbps?

PIN DROPS.

Sorry, can you repeat the question again?

CyberGauge is seeing the link to be flat-lining on 8mbps.

It was already 2pm on Monday and the client has fuming at "our" s_crew-up.  Everyone starts crowding around the laptop running CyberGauge.  we pulled a report out and furnished to our manager.  Someone called up Telstra.  Then another one called Telstra.  Then comes the kicker.  The first call to Telstra was for the Telstra account team.  Yes.  You requested for a 20 mbps link and we "gave you" 20 mbps.  It's not our fault.  It's yours.

The second call went to the provisioning team.  20 mbps????  What 20 mbps?  We were told, in writing, to give this address 8 mbps.

Let me tell you this:  Everyone were G0d-fearing men/women on that floor but I've never, ever heard them swear like a Navy sailor before.

Anyone reading this and is laughing, STOP.  That's NOT the funny part.  Here's the funny part.

So after everyone's calmed down, we all sat around the speaker phone with Telstra's account team.  Ok, ok, ok.  So someone's s_crewed up.  So it makes sense that a few buttons can rectify the issue, right?  WRONG.  So Telstra how many hours will it take to bring the link up from 8 mbps to 20 mbps.  Their response:  72 hours.

More swearing.  More hairs being pulled.  Some soiled their pants too!

In the age of modern technology why does it take 72 hours to upgrade a link from 8 mbps to 20 mbps?  Telstra's response was that there's a s_crew-up somewhere.  One system reports the link to be 20 mbps and another disparate system is reporting 8mbps.  This is Telstra at it's best.  Built around a significant number of disparate systems (or databases) that couldn't/wouldn't talk to each other, the only way to "rectify" the issue is to CANCEL the existing service and create a new one.  And this takes 72 hours.

[JohnHeller_2]

Here is another Telstra story:-

About ten years ago two businesses decided to start online gambling in my town, due to the unique legislation that exists in the Northern Territory. One of the businesses was the local casino and the other was an independant bookmaker who was specialising on sports betting.

Two friends of mine owned a data cabling business and were given the task of cabling up the sports betting site, which was one of the first Category 6 sites in Australia.

No expense was spare on any part of the operation. The server room was a modern raised floor design with air circulating through the racks from below. The had really good UPS systems, top of the line HP servers. All very nice.

They approched Telstra and got a 2 meg syncronous internet connection, which was the best that was available. The local casino also rented the same connection for their operation.

After going live and getting a large number of American and Europen gamblers signed up, they complained about a lack of network speed. Clients were complaining that it was taking too long to place online bets.

The company wasn't happy, so we tested every component and there was nothing wrong with the cabling, switches etc.

I ran a number of speed tests on the link and found that it was maxing out at around 300k.

We complained to Telstra and they basically said the link was working as intended. They blamed overall traffic to the link being the problem. I did tests to the telstra routers in adelaide and the results were the same.The casino apparently had the same problem.

This link was costing about $5000 per month and they could have gotten the same bandwidth out of a couple of ISDN links for 5% of the price!!

I found out off the record that the total bandwidth to the town was only 8 megs at the time, so Telstra sold two corporate clients half the towns bandwidth without any equipment upgrades. Im not sure which is worse, the fact that they sold half the bandwidth effectively meaning that all other private and business clients were losing half their overall bandwidth, or that they were ripping of the two big companies by not delivering what they promised.

Eventually Telstra fixed the problem, but it took months for the links to deliver what they were supposed to. I suspect that they gave traffic prioritisation to the sports betting business and casino, rather than increase overall bandwidth to the town.

Now that ADSL-2 is being offered in Alice Springs, I am finding that the speed is barely better than what ADSL-1 uncapped can achieve. Probably the same issue. When will ISP's be forced to advertise the effective bandwidth of their services rather than the connection speed?

[Leo Laohoo]

When will ISP's be forced to advertise the effective bandwidth of their services rather than the connection speed?

Nice one John.  I'm afraid not in Australia.  As you and I know pretty well Australia is the only place in the world where the ISP term "unlimited" as a number of asteriks at the end.  You'll also need a high-priced lawyer to get the meaning of this. 

[burleyman]

Well hello everyone it has been awhile......mainly because I have been dealing with Telcos..... 

The latest good one....

We had a T-1 start flapping and finally went down to one of our offices 2 weeks ago. I called our provider and they checked the circuit and did see some errors. They then checked things out and it looked like it was the last mile provider that was having the errors, so our provider opened a ticket with them. The last mile guys called me and said it was our equipment and not their issue which this last mile carrier always does on the first call and I mean always.

For example, about two months prior we had the same issue with another office and they flat out refused to do anything until we replaced our card in the router. So what I did was tell them I had called for the replacement and it would be there with our contracted 2 hours, we have 4 hour but I told him two. After about two and a half hours I called them back and told them I replaced the card and it was still down. They told me they still had the circuit down for testing which was why I was still having the problem and they would turn it back up. Two hours later it came back up and they called me back and they said "well it looks like it was a failed card, sometimes they fail without showing in the router as failed", I chuckled and told them I did absolutely nothing on the router because I knew that was not the problem and I knew the issue was on your end, and that looks like the case. They were not too happy.

So when they said basically the same thing on this call I outright laughed at the person on the phone when they said I need to replace the card in my Cisco router after there was no errors at all on the router indicating a problem and told them of the other call. So they went back a started checking things. Two days later after allot of back and forth they sent someone on site, against their will. So I get this call from this very rude lady from the telco asking very snotty, "Where are you, I have a technician on site and I am NOT!! having him wait forever!!!" I calmly told her that I was not the on site contact and they were to call the on-site person to gain access. She again very rudely says WELL THAT NUMBER IS NOT IN THE TICKET!! Again calmly I said it was because I just talked to the technician who did call the on-site person from the number listed in the ticket. She asked if I could make sure he is still there. so I set my cell phone down and not on mute, on purpose. I called the office and ask my person there to find out what they found. I could hear the tech in the background answer, "I did not find anything and it must be our equipment" and my person relayed that to me. Now loud enough so the rude lady on my cell phone could here I told my person to relay to the tech,"It is NOT! our equipment and they need to do more testing and find this issue. This company always says that to buy more time to fix their issue and I am not going for it!" I then hung up from talking to my guy and picked up my cell phone. I said hello you still there? and the nice rude lady is not happy and the first thing out of her mouth was " you know you did not put the phone on mute and I could hear everything you said and the technician could as well!!, I said that I know I left mute off so you could hear it, if you missed anything I could repeat it. Needless to say that was fun for me but not her. That ended our phone call. I then called our provider who actually has been great through all this and told them what happened and that we may need to escalate this higher up the chain. It was escalated and the next day they found some faulty equipment on the last mile provider and repaired it and we have been good ever since.

So now if I have the same problem I am going to jump right to the chase and tell them on the first call back here is what I did before I called you..... I replaced the wiring, and when that did not work I replaced the card in the router, and when that did not work I replaced the router and when that did not work I rebooted the router 3 more times and now I am calling you. I figured then I would just get the circuit fixed and then call them and let them know it is backup and running so they can say the other line they love to use, "we did not do anything yet?"

Telco's gotta love'em

[Leo Laohoo]

Nice one Mike.

[Rob Huffman]

Hey Mike,

Hope all is well buddy!

Did you try and connect a "Cross-over" cable for testing +5

It sounds like you have been down this road many times with this Telco

so it's great that you had your checklist prepared. I always like to tell them that

I've moved their cct. to a known working card/module and the trouble followed. That

usually mitigates all the "back and forth". Nice thinking with not muting the phone

with the Telco "crazy lady!!" I hate their pompous attitude, so it's always good to

put them in their place once in a while. Sweeeeeeeeeeeeeeeeeeeeeeeet!!!

Cheers!

Huff

[burleyman]

I will have to remmeber to tell them next time... "My CSC engineer suggested that I try a crossover cable and even that did not work, so I think I covered everything." But I have to do it with out laughing because I think that might be iratating them...

I did enjoy saying to her....If you did not hear everything I can repeat it for you.....she did seem to enjoy that as much as I did....not sure why, I was just being nice... 

Mike

[Kimberly Adams]

Mike,

+5 for your telco story, they are so much fun to work with.

Huff,

I do love the cross-over cable idea and I may have to use that one too!  +5 for the ideas!

I will have another one shortly, I am currently doing battle with one now!  So much for a good and happy friday.

Cheers,

Kimberly

[gerald.suiza]

here's mine

i was talking to this tech from a telco in thailand since i had this issue with one of our clients over there. in short the client was complaining of an intermittent connection. (who doesn't..duh..). so i go on explaining what i need the telco to do and i ramble things off for a round 15-30 mins. after i finish i ask him "Did you understand what I wnat you to do?" and telco says"Yes" so I ask his name and he says "Yes"

Toink!!...duh..and i fell off my seat...

[Leo Laohoo]

after i finish i ask him "Did you understand what I wnat you to do?" and telco says"Yes" so I ask his name and he says "Yes"
Toink!!...duh..and i fell off my seat...

ROFL.

It's like watching a Lauren and Hardy show! 

[Jon Marshall]

So, not a telco story but to do with contractors so near the mark.

I worked on a contract where we had to relocate 3 offices into a new head office. We had 3 months to get all computer equipment moved and to install any new equipment (6500s) into the new computer room.

So we were given the all clear to start in the new computer room and we installed some new 6500 switches. 2 days later the building contractors started drilling new holes in the computer room. Dust everywhere. 6500s sent away to be professionally cleaned. This proved to be just the start. Everytime we were given the all clear to start the building contractors decided they needed another hole for some reason in the computer room.

3 months turns into 2 weeks. By this time the head of IT who sits on the board and has staked his reputation on this move is getting a bit twitchy. An emergency meeting is called at the new building and we find ourselves standing about 30 ft from the entrance to the computer room. My boss is patiently trying to explain why there is absolutely nothing powered up in the room. The head of IT turns to the building contractors and explains there are only 2 weeks to get this up and running and can they guarantee that the room is ready for us.  It's important to realise both of them have their backs to the computer room and me and my boss were facing them, so looking directly at the computer room door.

Just as the lead contractor was assuring the IT head that there was no more work to be done me and my boss spotted a contractor with a drill in his hand ambling along, heading for the computer room. He was completely oblivious to the meeting going on. He wanders into the computer room and sure enough, just as the lead contractor had finished promising the room was ready we heard the sound of a drill starting up.

Things got a bit heated after that....

[gerald.suiza]

..you been punkd!!

[Leo Laohoo]

He wanders into the computer room and sure enough, just as the lead contractor had finished promising the room was ready we heard the sound of a drill starting up.

LOL!