MPLS is back up, Cisco WIC at fault

Verizon wasn’t calling us back or being helpful for the first 24 hours while our network was down, so we starting yelling at them. After three hours of this, continuously on the phone with their support group, and working through four “escalations”, they eventually gave us some useful attention. With their help we determined that (unusually enough) the problem was not in Verizon’s equipment, although it’s not unlikely that the whole sequence of events started in their network. Embarrassing situation all around, really. Verizon did a great job once they got started, and from their point of view our system was the one failing and not theirs, but you still shouldn’t have to harangue people to get them to fulfill the most basic requirements of customer service.

Anyway, the MPLS circuit terminates at a Cisco 2811 router with 4 FastEthernet interfaces. The 2 port fast ethernet WAN Interface Card on that system was reporting completely false packet counts and diagnostics – pretending to work perfectly while actually generating no outgoing data packets at all, and ignoring all incoming packets. There was literally no way to diagnose this without using physical loopbacks and other caveman tricks since rebooting the router during business hours would wreak even more havoc.

All of such order generic cialis appalachianmagazine.com issues are serious, but completely treatable and should be addressed to a doctor before using any anti-impotent medicine. The recommended dosage viagra 25 mg appalachianmagazine.com of the medication will take effect by protecting the cGMP or cyclic guanosine monophospahte from becoming degraded by the PDE-5 or phosphodiesterase type five in the corpus cavernosum. It is one of the trusted Canadian online levitra on line drugs that effectively treat type 2 diabetes. The super viagra for sale more active the tissue (brain, muscle, heart etc.) the more mitochondria there are. Shutting down the router, pulling the power cord, reseating the WIC and restarting everything fixed it. But of course we had to fly Jay from Philly to Boston in order to do that, and the network was crippled for 41 hours before the situation resolved at right about 5AM. Three engineers working 19 hour shifts is not fun for people over 30!

Since clearly this WIC is unreliable we moved the line to a dedicated port and set up a hot spare router next to the problem child.

Leave a Reply