Troubleshooting lessons.

OK. For a couple of weeks, we've had major phone problems. We run a couple of Mitel VoIP boxes and a few hundred phones off them. Each Mitel box has 30 ISDN lines ( I don't really get ISDN, otherwise I'd describe it better ).

We've been getting one way calls to our enquiries people, so our operators can't hear the caller.

This is the story of the diagnosis.

First off, we questioned users and first line support folks. It appeared to be happening on calls coming in to one of the Mitel boxes, destined for a group of handsets on a distant campus.

I took some packet captures from the switch port one of the phones is plugged into. I saw normal background stuff, some mitel control traffic (port 6800 and 6900), and two RTP streams, one each direction. So I pulled the streams to audio files (wireshark is lovely..), and listened to them. Sure enough, the outgoing side was fine. "Hello, ???, can I help you?......Hello?.......Hello?....click". The incoming side was silent. Not quiet, silent.

So, it's not the LAN, I reasoned. The audio stream is getting to the phone, there's just nothing in it.

So we talked to our Mitel reseller, who remotely looked over stuff and said nothing was wrong. And we talked to our ISDN provider, ditto. Our Mitel reseller sent someone out.

And while he was looking, I got a notification that one of our internet routers was down. And I got a colleague to look it over, and restart it. And it's fine, but unreachable. So I go look at the layer 3 switch it's connected to. And the interface is up, but no arp entry and no pings. Oh bum.

So I try pinging the Mitel box from the core switch. Uh-uh. ARP entry? Nope. OK, add a static ARP entry. All of a sudden, all is well.

Turns out it was the network all along. Switch hardware, we're working it out with the vendor right now.

The point being, I was fully satisfied it wasn't the network. I could talk through my analysis with capable colleagues, who agreed. And we were wrong.

The moral of the story? When there's several components to a problem, and all of them check out fine, then someone doesn't understand the problem.

And it's probably you.

• Posted on: Sun, Feb 3 2008 10:06 AM