Architecting Supportable Designs…..and a rant or two.

Tennessee Aquarium in Chattanooga,TN. A fine piece of architecture.

I just spent 9 weeks on site with a client building out a network reference architecture. The goal was to provide them with a framework they could use to configure their network in the future without having to reinvent the wheel every single time. I ended up with almost 90 pages of documentation which included somewhere in the neighborhood of 30 Visio diagrams. From a network infrastructure perspective, this reference architecture included routing, switching, wireless, network management, QoS, WAN optimization, and a few other things.

It was a very interesting experience, as I have never really sat down and thought about how I would design the network as a whole. I’ve done my fair share of implementations, but they generally focus on one particular area of the network. Companies don’t generally forklift their entire network or change every device configuration across the board all at once. During this process, one thing kept nagging at me in the back of my mind. It followed me through every major section of the document. I needed to ensure that this particular reference architecture could be supported by the networking staff.

Due to that, I had to make some tradeoffs. That doesn’t mean I slacked off and didn’t consider all possibilities. It just meant that I had to ensure that any recommendation I made would make sense to the average networking person out there. Did I explain the reasons why I chose to use NSSA’s in OSPF multi-area environments? Did I include caveats to running things like VTP on Cisco switches? Did I provide enough information to help explain why I would limit the SSID’s on wireless AP’s to 4 or less?

By writing the document with the thought of people being able to support the proposed architecture, it wasn’t an attempt to say that I am much smarter than the design choices I recommended. I was being pragmatic. I DO have an ego(Don’t we all?), but I try not to let it come out and play too often. I’ve been wrong WAY too many times to go shooting off at the mouth, errr…..keystroke.

The Problem

What I am about to tell you isn’t something you don’t already know. There is a shortage of mid to senior level qualified networking people out there. There are a LOT of tactile engineers floating about. By tactile, I mean people who have done process XYZ a number of times and their level of understanding never goes beyond that. This is easily discovered when things break and their troubleshooting methodology is either non-existent, or doesn’t go beyond a few “show” commands on a device. If this is a junior level person, that’s not a huge problem, unless the thing that broke is well within their skill set. For senior level people, and I realize that “senior” is a very subjective term, there is less wiggle room for lack of understanding when things go wrong. Especially if they were the person who designed and implemented that particular aspect of the network.

I also fully admit that my opinion is based on the encounters I have had, and in no way could I claim to speak as an authority for the entire IT community at large. I will say that when I mention this subject to my peers or network managers out there in corporate America, they tend to validate as opposed to dispute my assumptions. As always, I reserve the right to be completely wrong and have someone tell me that I am wrong. I am also approaching this from the position of networking. I don’t pretend to understand the other silos in the way that I think I understand networking. This could probably apply to those areas as well.

Why This Problem Exists

There are a number of reasons I think this problem exists. The easy thing to do would be to blame everyone for not having the drive to develop themselves professionally. Realistically though, it’s a bit more complicated than that. Allow me to put forth some theories here.

1. Companies don’t care about professional development. – This isn’t true for all companies, but it tends to be the norm in my experience. Most IT managers I have dealt with really do have an interest in ensuring their people can do the job required. However, their directives are to get people in the door that can do the job with little ramp up time. They need performers from day one. Training may be promised during an interview, but all too often, it is never followed up with actual training time or resources. It is lip service. Employees that have a need to develop themselves are going to do it on their own, but don’t expect them to stick around for too long. They’ll gain additional experience at a particular company and move on to somewhere else. They know a dead-end when they see one.

If the networking team at your company consists of more than a few bodies, managers should be doing all they can to ensure their people have freedom and support for professional development. Lots of people stay at companies for years in spite of the fact that they could make more money somewhere else. There is something to be said for being in an environment where there is a solid technical team lead by competent managers. A good team can work wonders with even dated infrastructure. The better prepared your people are, the better your network will be run. That may not fit into a nice little spreadsheet in the accounting department, so network managers need to be prepared to defend the reasons for employee X taking a half day every week to train/learn. I can definitively state that there are a handful of people I have worked for/with over the years, that I would gladly work for/with again. When you find a good team, you don’t want it to ever end, even though it usually does at some point.

Tip: If you are interviewing for a job and they mention that they support training, ask them which of their employees have been on the receiving end of that training in the past year. If they can’t answer it, that should set off alarm signals in your head.

2. People don’t know how to develop themselves. – Remember your first job in IT? Were you overwhelmed? Did only a small amount of things make sense? You aren’t alone. I suspect it was like that for most of us. The difference, I think, is having an environment where learning and development is encouraged. The senior level people need to take time to ensure the junior level people understand what it is that they are doing. Looking back over the years, I have had the good fortune of working for people who took an interest in my development. They gave me projects and refused to hold my hand. They would help me out if I got stuck and needed further explanation, but overall, they instilled in me the need to research and learn things on my own. It has served me well over the years, and above all, I always want to know WHY something works the way it does. I credit my mentors for instilling that value within me, and hope I can do the same for others.

For people that didn’t have the fortune of good mentors, they may spend an entire career just going through the motions and repeating the same tasks over and over. While some out there are just lazy, I think there are a fair amount that just don’t know where to begin. They also may not be able to visualize how far they can actually go if they take the time to step out of mediocrity and go for it.

Tip: If you work in an environment where you are laughed at or belittled because you don’t know how everything works, find another job. People that withhold information from you are probably insecure and don’t know half as much as they let you believe they do. I’ve learned plenty of things from people that had far less experience than I did over the years.

3. We don’t screen candidates properly. – I could go on and on about recruiters sending in lambs to the slaughter, but I won’t. They aren’t going to change their tactics. A fair amount of them are just trying to meet that quota and don’t really care, or even know, if a candidate is viable. One of the things I do once a year or so is have lunch with a recruiter that I trust, along with some of his newer recruiters. We tend to eat at places that use paper as the table cloth, and in between bites of food, I draw out different network technologies for them. Very basic stuff. Here’s a switch. Here’s how it works. Here’s a load balancer. Here’s what it does. Etc. They will ask questions about things they have heard. I try my best to answer it in a way that makes sense to them. This is all done to ensure that when they start to vet potential candidates to put in front of a client, they can ask some basic things and perhaps understand the candidate’s resume a little more. I get a free lunch at a nice restaurant, and they get some basic introduction to networking. An even trade in my opinion. It helps my belly grow a little more, and hopefully, it makes them better at their job.

It isn’t just recruiters though. Sometimes network managers don’t have a technical background. I’ve seen good network managers with heavy business backgrounds, and I have seen good managers who have technical backgrounds. Everyone is different. Lots of different opinions on which approach is better. For network managers who only understand their infrastructure from a very high level, it means that they might end up hiring someone based on a resume and their ability to make themselves seem technically stronger than they are. They don’t know what to ask them unless they are hiring a very junior person. If you, as a technical person, see enough resumes, and provided a recruiter hasn’t “fixed” the resume, you get a general idea of whether they are the real deal. The personal interview will usually reinforce your initial instincts. If you aren’t doing technical interviews for your higher level positions, you WILL get burned at some point. Don’t settle on candidates just because you need a body. If all companies performed technical interviews at the level the position required, people would take interviews more seriously. That would make the candidates prepare a lot more than some of them do.

Tip: If you fail an interview due to technical reasons, remember those things you missed. Write them down during the interview, or shortly after leaving while they are still fresh in your mind. Go home, accept the fact that you failed, and research the things you missed. One of the saddest things I hear from people that failed technical interviews, is that they never bothered to figure out what the proper answer would have been. I once got administrative distance and EIGRP’s advertised distance confused in an interview. Although I ended up getting the job, I was so mad at myself for missing that(Hello type 1 personality!), that I went home and memorized the administrative distance table. Years later, I can still write it down if someone were to ask me to. That’s how bad it upset me. Don’t let a failed interview go to waste. Learn from it.

Additional Tip: If you fail an interview because someone was trying to make you feel stupid, consider it a blessing if you don’t get that job. You probably don’t want to work with people who like to flex their little bit of power and use interviews as a forum to show you how much they know and how little you know. Give it a few years in a healthy environment, and you will run circles around them. In the future, when you are conducting technical interviews, remember how it felt when someone was being openly hostile towards you. Don’t do that to others. It serves no purpose. Also, remember not to confuse standard technical interview pressure with open hostility. I’ve been a part of technical interviews where multiple people would hit a candidate with questions all at once in rapid fire. It was designed to see how they operated under pressure. The questions were not asked in a threatening manner. They were just probing to see if the candidate could take each question and provide an answer without becoming flustered. Yes, you’re nervous in an interview. Perfectly natural. However, when things go wrong at 2AM and you are the one who needs to fix it, you can’t shut down if the problem count goes up.

The Fix

Fixing this problem is two-fold. It’s “tough love”, but it needs to be said nevertheless.

1. Companies – As a general rule, you guys are lousy. Not ALL of you, but a fair amount of you. I get it though. You are focused on the stock price. The quarterly numbers. You don’t have time to sink more cash into that money pit we call IT. Unfortunately for you, you need to take better care of your people. Imagine a place where your employees were willing to do what it takes to get the job done. Imagine a place where you didn’t have to manage by intimidation. A place where you had droves of people waiting to interview for a position at your company, instead of just settling on you because a better offer didn’t come up. Imagine a place where you don’t have to micromanage everyone because you didn’t trust them. They’ll make it happen because they are committed to the success of the business and they are solid professionals.

What’s that? You have an ample supply of corporate drones that don’t want to rock the boat because they are happy to have a job? Here’s some news for you. When you have a group of performers in IT, they have options. If their personal networks are developed and they have in-demand skill sets, they don’t need you. You need them.

Is your IT department a revolving door? Maybe you should stop and ask yourself why. Of course, if the profits remain high, who cares, right? You think outsourcing is the answer? Go for it. Let me know if that works out for you and it comes in under budget.

Take an interest in your people and they will do wonders for you. Here’s a secret that your accounting department and CFO will love. You don’t necessarily have to pay them all a ton of money. You could spend a little on investing in their professional development and they’ll appreciate you for it. That doesn’t mean they will all stay. Sometimes you outgrow a job from a technical perspective. It happens. It’s part of the natural evolution of the IT professional. I can bet you though, that a lot more people will stay if they think you actually care about them. For those that do leave, do exit interviews. It takes a little extra time, but wouldn’t you REALLY like to know what makes people leave your company?

2. Individuals – Abhor mediocrity. I don’t get to use the word “abhor” much, so I am really glad it fit here. 🙂 That’s about it. Don’t be content with treading water. Go out and drive your career. Don’t be a passenger! I can assure you that you will have far greater rewards in the long run. Spend some time figuring out what you want to do. Identify your weak areas. Be honest with yourself. Take some extra time and bump up your knowledge in those areas you are weak in. This will probably require time outside of normal work hours. There’s a fair amount that can be done without spending tons of money on educational material. Google will show you the way. Social media will help as well. Get on Twitter, even if it means being passive and just watching. Read blog posts and networking forums. Be a sponge.

I’ve talked with numerous networking folks over the past several years in interviews and casual conversations. I’ve recommended people for jobs they didn’t have the exact experience the employer wanted because they had potential and were strong in other areas. You knew after talking to them that they were destined for bigger things. They were hungry, and I will take “hungry and inexperienced” any day over “experienced and treading water”.

Closing Thoughts

I love what I do for a living. Life has a way of sending you in directions you never intended to go in. As a child, I was in love with the idea of being a pilot in the US Air Force, being a police officer, or being a cartographer. I ended up in IT. Looking back though, it was the greatest thing that could have happened to me. IT is one of those unique fields where YOU can dictate where you end up. If you are willing to put in the time studying and never settling for the experience you have today, you really can be anything you want to be. This is a knowledge based industry. The playing field is fairly level, in my opinion. Those that are willing to do more than the next person will find success. I firmly believe that. Yes, there is some luck involved, but in a way, you make your own luck.

I’m never content with what I know today. This industry moves way too fast to sit still. Things change. Technologies change. Vendors come and go. Jobs come and go. As you move around and move up, your value increases provided you never stop learning. Companies WANT people who are high performers. It may take you several years and many sleepless nights, but opportunities will come your way provided you put forth the effort to be ready when that opportunity presents itself.

I started out taking about network architecture, so let me end this long-winded post talking about that same thing. I’ve seen my share of network designs that were lacking, not necessarily from a hardware perspective, but from a configuration perspective. They were lacking because the people implementing the design weren’t fully aware of what their options were or they didn’t understand the technology in general. This is not always true. There are some very deliberate things done on networks, for very specific scenarios, that at first glance seem erroneous. However, it seems more common that things are done due to a lack of understanding. The same might be said for even the recommendations I made in my designs. By constantly improving ourselves, every design will get better and we’ll have fewer 2AM calls and alerts due to problems we inadvertently caused. Look at it this way. The more you know, the more time you will get to sleep at night. Unless of course, you are up late studying. 😉