Discussion: Routing Protocols (Written Transcript)
Because it happened at 4 am, no one noticed or checked.
Gemma was manning the tier 2 helpdesk when the first call came in. The system logged the call at 8:13 am from John Hopkins, a branch manager at a restaurant franchise location in Wyoming, 6:13 am Mountain Standard Time.
Gemma: "IT services, how can I assist this morning?"
John: "Got a problem. I can't access the food ordering system for my next day's delivery."
Gemma: "When did you first notice the problem?"
John: "About 10 minutes ago, a few minutes after I arrived."
Gemma: "Have you noticed any other problems, like can you access the company website or print a document?"
John: "I can print, but now that you mention it...I can't access the website."
Gemma: "Can you access any websites?"
John: "Hang on...no, not even Google."
Just then Gemma noticed an uptick in the phone line activity of her colleagues. Everyone was suddenly very busy, chatter everywhere....unusual for this time of the morning.
Gemma: "John, let me get back to you. I need to check something out."
Knowing he could print basically meant that the local connection was good and the branch office LAN was functioning. It had to be something on the WAN link. Gemma, suspicious, trained her ears on the conversation of her nearest colleague, Mara. She heard the words, "maybe the WAN link is down."
"Hmm," she thought to herself, "sounds like more than one branch is having issues." She looked at the incoming phone number on Mara's IP phone. The area code was in Montana. Gemma got up and wandered around looking at the phone numbers on the desktops of other colleagues. They were all from Wyoming, Montana, and Idaho. She went back to her desk and looked up the phone number of her WAN provider.
The provider's engineer answered cheerily, "Grayson here, what can I do for Shake and Wake Breakfast Foods today?"
Gemma: "I think our WAN link to the northwest is down. We are deluged with calls from Wyoming and border states."
Grayson: "Hmm, I don't see any linkage issues. We would have called you if we had seen them. Let me test something; hold on...yep, they are up and carrying data for other customers just fine. But the bandwidth usage is down, suggesting that your folks aren't using their bandwidth slots. Have you checked your routers?"
Gemma thought for a second. "No, but I am going to VPN into one from this side." Gemma went for the main WAN connection router first. It was up and running. She then went to the second one. It was up and running. She tried to ping to the remote branch...nothing. She tried to ping the other side of the WAN router, the provider's IP. It worked.
Gemma scratched her head and thinking out loud said, "Grayson, I can ping internally, but I can't ping externally, are you sure that link is up?"
"Data link is up, and I pinged your WAN router's outside address from here," said Grayson.
Gemma was perplexed. She VPN-ed back into the WAN router. Not only could she ping the external address, but she could also reach the branch address in Wyoming. That was went something hit her. She issued the 'show ip route' command. Seeing the result, she then issued the 'show ip ospf neighbor' command and observed nothing. Show IP Protocols told the rest of the story.
After you view the scenario above, respond to the questions below. One important detail to know is that the WAN link is the local branch's Internet link, as opposed to a separate local Internet Service Provider.
- Using "problem isolation" troubleshooting, where do you think the "root cause" is physically located?
- What ideas do you have for how Gemma should proceed at this point to fix this?
- What do you think could have happened before everyone arrived at work that might have caused this? Was it avoidable? Explain in detail.