Had a real busy day at work yesterday. When I came in to work in the morning I got alerted by the NOC that we had a major fault in one of our networks. The network is an open network for broadband customers and companies who can choose what ISP they want to use. The problem related to a switch that kept rebooting itself for no apparent reason and when it rebooted it replied to ping but logging in to it was not possible. The switch was one if not the most important since it’s the place where the service providers connect.
We decided to replace it by another switch. It took a while to get it up to snuff with the correct software and configuration. While I was working the phone was ringing non stop and the NOC was overflowed with customers who couldn’t surf the Internet. It’s kind of a weird feeling when you know what you are doing righ now is affecting hundreds of people but it feels great when you come up with a solution. To be a good network engineer I think you have to kind of like these moments or at least not dislike them entirely. If you have a simular story or have an opinion on how you react in these situations please post it in comments.