Last couple of week have been pretty interesting. Been sick and not working for a day a week. Did a hardware upgrade and just like anything IT, there’s always that chance that something will happen and things will go south. I also went for an emergency dental operation yesterday, which was not fun at all. Delay after delay after delay stopping me or keeps me off track of my CCIE goal.
Yeah I haven’t been feeling really well for the last couple of days. Maybe it was stress related, maybe I am pushing myself too much, maybe I need to relax a bit, maybe I have not been living healthy. Whatever it is, my body has just been complaining and whinging. Historically its always been like this, whenever I get to be at the peek of something, body gives in and tell me to slow down. May it be sports, school or any activity or project that I give my all, my body will eventually shut down and will want it recuperate. Hopefully this week, after the dental operation I had yesterday, everything will be all good.
I did a hardware upgrade of our stacked Brocade in Sydney this week. It was actually a number of things I had to do, upgrade the firmware, increase the frame size, turn off optical monitoring and increase the default vlan. For some weird reason, the Brocade ICX 6610s has a default vlan of 64 which would not be suitable for our environment as an ISP. It is capable of maxing to 4096 but warns that it may have a toll on other resources. Had to increase mtu size as we need to get a packet more than 1500 which is allowed by default. We maxed that out as well. Firmware was the tricky part as I was doing it remotely from QLD to NSW, so anything happening would cut off a number of customer on that link.
Everything was running perfectly on the night of the upgrade but when I ran the last command, which was to copy the bootrom file to the stack, the brocades power-cycled and by the time it came back online and they unstacked. Before doing the upgrade I was prepared for the worst, I got my email ticket ready with photos and instructions to Equinix smarthands in the event one of them gets bricked. All good and ready to go but that didn’t happen. The Brocades just unstacked and as I consoled in via OOB, I noticed that both devices had a the same config but they were running as 2 separate devices.
Further investigation I noticed that the bootrom file was only copied on unit 1 but didn’t on unit 2. A mismatch was keeping them from forming a stack. What I did was I erase the config on the 2nd device and started to plan to get this configured to reach the tftp server and copy the botroom and that should stack them back together.
What I didn’t know, while I was troubleshooting and resolving this issue, they were creating a broadcast storm on our core switching causing an outage. I only knew about it when I got a call saying Sydney was offline and was there something wrong. I said the Brocades unstacked themselves. The weird part about it was the Cisco 4500s had storm-control configured on them. I have no idea why despite storm-control implemented, it took a while to filter the traffic coming from the Brocades.
Any how, I managed to get the 2nd Brocade to upgrade the botroom and eventually they stacked with all the upgrades and configs that I applied. I am telling you, it was the longest 2 hours of my life and it wasn’t fun at all. Well you live and learn.
———————————-
On a lighter note, I got to configure a 3-way fail-over solution for a client running ethernet, fiber and wireless using BGP. Pretty cool, first time to configure BGP over our infrustructure as a 3-way failover solution. I was under the notion that you can only use BGP if you have multiple providers. Thinking about it, yeah it was multiple providers to reach their private network which we host.
I will be posting the configs soon.
Running-Config 