What a week and a nasty Brocade ICX upgrade experience

Last couple of week have been pretty interesting. Been sick and not working for a day a week. Did a hardware upgrade and just like anything IT, there’s always that chance that something will happen and things will go south. I also went for an emergency dental operation yesterday, which was not fun at all. Delay after delay after delay stopping me or keeps me off track of my CCIE goal.

Yeah I haven’t been feeling really well for the last couple of days. Maybe it was stress related, maybe I am pushing myself too much, maybe I need to relax a bit, maybe I have not been living healthy. Whatever it is, my body has just been complaining and whinging. Historically its always been like this, whenever I get to be at the peek of something, body gives in and tell me to slow down. May it be sports, school or any activity or project that I give my all, my body will eventually shut down and will want it recuperate. Hopefully this week, after the dental operation I had yesterday, everything will be all good.

I did a hardware upgrade of our stacked Brocade in Sydney this week. It was actually a number of things I had to do, upgrade the firmware, increase the frame size, turn off optical monitoring and increase the default vlan. For some weird reason, the Brocade ICX 6610s has a default vlan of 64 which would not be suitable for our environment as an ISP. It is capable of maxing to 4096 but warns that it may have a toll on other resources. Had to increase mtu size as we need to get a packet more than 1500 which is allowed by default. We maxed that out as well. Firmware was the tricky part as I was doing it remotely from QLD to NSW, so anything happening would cut off a number of customer on that link.

Everything was running perfectly on the night of the upgrade but when I ran the last command, which was to copy the bootrom file to the stack, the brocades power-cycled and by the time it came back online and they unstacked. Before doing the upgrade I was prepared for the worst, I got my email ticket ready with photos and instructions to Equinix smarthands in the event one of them gets bricked. All good and ready to go but that didn’t happen. The Brocades just unstacked and as I consoled in via OOB, I noticed that both devices had a the same config but they were running as 2 separate devices.

Further investigation I noticed that the bootrom file was only copied on unit 1 but didn’t on unit 2. A mismatch was keeping them from forming a stack. What I did was I erase the config on the 2nd device and started to plan to get this configured to reach the tftp server and copy the botroom and that should stack them back together.

What I didn’t know, while I was troubleshooting and resolving this issue, they were creating a broadcast storm on our core switching causing an outage. I only knew about it when I got a call saying Sydney was offline and was there something wrong. I said the Brocades unstacked themselves. The weird part about it was the Cisco 4500s had storm-control configured on them. I have no idea why despite storm-control implemented, it took a while to filter the traffic coming from the Brocades.

Any how, I managed to get the 2nd Brocade to upgrade the botroom and eventually they stacked with all the upgrades and configs that I applied. I am telling you, it was the longest 2 hours of my life and it wasn’t fun at all. Well you live and learn.


On a lighter note, I got to configure a 3-way fail-over solution for a client running ethernet, fiber and wireless using BGP. Pretty cool, first time to configure BGP over our infrustructure as a 3-way failover solution. I was under the notion that you can only use BGP if you have multiple providers. Thinking about it, yeah it was multiple providers to reach their private network which we host.

I will be posting the configs soon.

two weeks at work

Two weeks into working for a service provider and I fee like I’ve been with the company for months. I am going through so much technology in a span of days and just praying I could get familiar with them all the soonest possible way. At this point, I am against time and my capacity just to take it all in which is something I don’t complain about.

I feel like going back to basics and re-think about my plans to take the CCIE, instead go for something more relevant to what I am doing. In a few days I am thinking of booking my CCNA SPNG1 exam. It would have been nice if it was like all the other tracks wherein if you have your CCNA RS, all you have to take is a single exam and you have your 2nd Associate level cert. Unfortunately, its a different track meaning I have no choice but to take the two exams. Its a good review anyway.

Let me just share the technology I work with:

  • Cisco ASR 1000 series core routers
  • Cisco ME 3400 series switches
  • Cisco’s C170, used to be called Ironport before Cisco bought the company
  • Sonicwall firewall
  • Fortinet firewall
  • Cisco ADSL routers
  • Telstra’s LOLO, LOLS (funny names)
  • AAPT Frontier
  • Brocade ICX 6610 switches
  • Web services such as DNS, Domains
  • Solarwinds Netview
  • Netflow

There are more technologies but at this point I have no access to them yet.

It is really a humbling experience, that’s all I can say. All the more I am realizing that I don’t know a lot.

Living the dream

Two things that really pushed me or inspired me to go corporate when I was teaching: Firstly, when one of my students asked “With the cert that you have why are you teaching here at TAFE…” and when one of my teachers said for young guys like you tutoring/teaching in TAFE should just serve as a jumping board to bigger things. Gain experience and then just go back to teaching. He was most definitely right. Got myself out there and finally landed into something amazing.

I’ve always dreamed of getting my CCIE number but find no reason to get one. There’s really no point of getting it if I am not working with routers/switches in the enterprise level. This week I’ve started working for a service provider down the Gold Coast as part of their NOC team. I guess I’m still in the hibernate stage for the next few weeks since I will be introduced to all their technologies and products. This week I’m shadowing the senior level 1/2 tech engineer, next week I will be shadowing the service design manager and will be shading the level 2 engineer who I will be replacing in a couple of weeks/months.

It’s amazing being surrounded by people who are so passionate about what they do. These guys who are way younger than I am and knows so much more of the network are amazing. They don’t have their CCNA, well there are about to get theirs but I realized my CCNP is nothing compared to what they know. They showed me the data center, the equipment. I feel like a kid in Toys R Us. It was amazing.