People Who Ask Why

There are a lot of things I want to teach my kids, but at the same time, I find myself not wanting to for certain things. Let me explain.

For the first years of my children’s lives, they made very little decisions for themselves. They ate when we fed them. They slept when we told them. They bathed when we told them. They played when we let them. And on and on it goes. As they have grown, they make more of their own day to day decisions. At some point, they will leave my house. I will not be around to tell them what to do 24 hours a day. It will be up to them.

Most kids are naturally curious. My son, for example, is always asking me why he has to do things a certain way, or at a certain time. While some of those things are not up for discussion(ie bed time, brushing teeth, bathing), a lot of them are. I always tell my kids that there is generally a reason for everything I do. If I drive somewhere on a certain route or  require certain attire at church, there is a reason. I don’t mind explaining that reason. Sometimes the answer is acceptable, but other times, I am the most unjust father in the world in their eyes. Such is the burden of parenting!

As my kids age, the questions become more complex. Why is the sky blue? Why do airplanes fly if they are made out of metal? For many of those questions, I find myself telling them to figure it out. Of course, discretion is warranted. I don’t want my kids hitting up Google to find out about human reproduction as I fear the answers might be muddied with improper websites. :)

What I want is to nurture that curiosity trait. I want them to question why. Always! Even if the answer they get isn’t what they want to hear.

A few years ago, I wrote a post entitled Chasing the “Ah-ha!” Moments. It was about those times in our IT careers where we got a little bit of clarity on some technical matter. It becomes an addiction. You look for those things that you don’t understand as well as you should, and you run it down until you have it figured out. Someone can explain it to you, but I think there is more value when you discover it on your own. Variable length subnet masking was one of those things for me. When it finally clicked in my head, I was excited. Some people might think that is a relatively trivial thing, but for me it was huge.

A big part of what we do in IT is education. Whether it is another IT professional or an end user, we are constantly explaining why things work the way they do, or more commonly, why it broke. For the end user, just explaining why it broke is generally good enough. Sometimes they don’t even care as long as it is fixed.

For the fellow IT professional though, I recommend a different approach if possible. Don’t explain why. Have them research it and see if they can explain it to you. Don’t just give away answers. At some point, the hope is that they will stop asking for answers and find out for themselves. In some cases, that isn’t an option. If it is a simple question from a co-worker(ie Which TCP option is it that Riverbed Steelheads use to discover each other?), just answer it. Don’t be an idiot. Use discretion and never condescend. However, when the question is more complex and you have the luxury of time, let them figure it out, with you providing hints or clues as needed. It will serve them well in their careers.

The end goal is not to seem like a jerk that won’t help out co-workers. The goal is to teach them how to solve problems on their own. I hate calling vendors for support. I’ll do it after a certain amount of time has passed, and the amount of time I wait changes based on the situation, but I still hate it. I hate the not knowing. Hate it with a passion. I want the same for everyone else. If you don’t care, I probably won’t work well with you over the long run. If you do care about the “why”, we’ll probably get along just fine. :)

Posted in career, learning, training, troubleshooting | 2 Comments

My Lego Datacenter

I was invited to take part in a datacenter building contest from Juniper Networks recently. I don’t need to give you the full story behind myself and Legos. Let’s just say I have loved them for a long time. Both of my kids have multiple sets. How could I refuse such a contest? The 5 year old boy that occupies a significant portion of my brain still wishes he could sit at home all day and play video games and build complex structures with Legos. Oh, and the best part is that whoever wins, gets a donation to their favorite charity courtesy of Juniper!

And so it was that I was eagerly anticipating my Lego package from my buddy Ashton at Juniper. As luck would have it, I was out of town last week toiling away for my employer the day the Lego set arrived. My wife texted me that the package had arrived. In the course of the week, I became rather ill. Maybe it was the cold north Texas air, or maybe it was breathing 10 year old dust from the ancient wireless access points I was replacing. In any event, I was laid up over the weekend trying to recover. As this week wore on, I began to see designs from my competition. They were all good designs and I was still trying to recover from my illness. I knew I would have to pull something together at the last minute due to pressing matters at work keeping me from abusing my work at home privilege.

This evening, with a design concept in hand, I summoned all of my Lego datacenter powers and merged them with the leftover palm residue enshrined on various business cards from Juniper employees. I’m only about 80% recovered from my illness, so I may have been a bit grumpy when the final plan came together. Not Ivan Pepelnjak grumpy, but still grumpy. :)


It is hard to make out any old palm residue, but I assure you it is there. I felt design powers flow through me like The Force!

Behold, the Leguniper datacenter!

DC OverviewThe first thing you notice is the dark void around the datacenter. This is not by accident. What appears to be void is simply the light being overshadowed by the Cloud. This IS a cloud datacenter, and in the interest of fully displaying the power of the Cloud, I thought it would be good to show you, dear reader, what the Cloud looks like. Nobody else has shown this before, so you are in for a treat!

The Cloud























The Cloud in all of its glory!

Not what you expected? Well, reality is often different than what we picture in our minds. You’ll just have to take my word for it.

Touring the Leguniper Datacenter

First up, is the Security section. In this datacenter, security is a big deal. They ensure that nobody can do their job except for the external hackers. All of their equipment is kept behind locked doors, even within the datacenter, as they do not trust even the other employees. Notice the fierce determination in their eyes. They take their job seriously!

Security Waiting Line























The long line of employees who have committed various infractions never ends. They must come and present themselves before the security department to explain their Internet abuse or why they cannot choose a 328 character password properly. All must come and all must apologize profusely. In fact, I was so terrified when I took their photos, that hairs from my head fell onto the datacenter floor out of sheer terror!

Security WIPSYou aren’t going to get anything by this datacenter security team. If you think you can bring in an unauthorized wireless device, think again! They have a state of the art omnidirectional antenna on top of their security suite that is always scanning for rogue clients and access points. You WILL be caught, and when you are caught, you will stand in line with everyone else and explain your infraction. You have been warned.

What’s that? You want to see inside the security suite? Are you nuts? You are NOT authorized to do that! Well, since you are my guest, and I am the designer, perhaps I can make an exception. Hold on a second and let me see about getting that door opened.

Security Retinal Scan

I forgot about the retinal scan device at the door! It keeps out all unauthorized people. Let me bribe one of the security guards with a chance at dating the blond haired lady working at the NOC desk and see if he will let me in.

Security DoorSuccess! Now that the door is opened, let’s see what super secret stuff lies inside.

Security InteriorLooks like we have us a pair of Juniper SRX security devices. There are several other things in this room, but we aren’t too interested in them since they are mainly a bunch of appliances from other vendors that compile logs that nobody reads. Let’s get out of here before the security guard realizes I was lying about the date with the NOC lady. She’s really only interested in dating network engineers.

Servers and StorageHere we can see all the racks containing the servers and storage. It is forbidden territory for us network folks. An unknown virtual void of gigantic databases, multi-tiered applications, and various trickery designed to make everyone think it is always a network problem. The demarcation line between the network gear and the servers and storage has been set by the warnings of dragons. Proceed further at your own risk. Occasionally, one must enter into that territory. You see, there are some QFX 3500 and 3600 top of rack QFabric nodes in every rack indicated by the blue bricks. We don’t want to get caught up in the server and storage trickery, so we use a different color to know what is safe to touch.

Ladder Trays

We zoom in a little closer and we can see all the fiber and copper coming from the servers and storage extending into the super cool network racks. It is suspended in the air above all of the racks via nice and tidy ladder trays. No more diving under the raised floor and plunging your arms into a cabling mess to find that one new drop that was run. No, no, no. Everything is fully populated and extended into the network hardware so that it is only touched during install and IF there is a failure, there are redundant connections so that nothing goes down. Well, the network doesn’t go down. There’s no telling what is going on over there in virtual land.

QFabric Plus DroidNow we come to the part of the tour you have all been waiting for. Get your cameras ready. This is going to be good! Here we have our QFabric Interconnects and Directors. Why did I design it this way? I didn’t want to have to configure a bunch of top of rack switches independently. I figured I might as well make the datacenter one big logical switch. It gives me more free time to work on the stuff that really matters, like my video gaming skills. In addition to the QFabric piece, I also included some really nifty MX routers. From these, I am doing cool stuff like BGP peering. You know? Real protocols. Manly protocols. Oh, BGP is an application you say? Shut your mouth! I designed this datacenter. It is what I say it is! (I told you I was grumpy.)

If I actually need to physically touch any of this stuff, my droid takes care of it. I don’t want this pristine equipment polluted by human hands. I need typing skills that rival Mavis Beacon. Your CCNA does not impress me and will not grant you access to this hardware. Might I suggest a Juniper certification to win my favor and a shot at replacing the droid once maintenance expires on it?

Let’s go visit the network operations center(NOC) now.

NOC Front View

Here are all of our NOC technicians slaving away at maintaining all of this hardware. They are watching environmental systems, ensuring backups run, and constantly wrestling with application issues caused by imperfect code, written by imperfect humans, and executed by imperfect end users. It’s cool though. They are all very nice and friendly. Sometimes their manager gets riled up and fires one of them for making a mistake, but……wait a sec. Let’s make sure they are working since the manager appears to be berating an employee at the moment.

NOC Back View

Looks good! Wait a sec. Does that guy with the hat on have his password sitting on top of his monitor? Uh oh. Don’t tell security or his manager. I’ll wait until the tour is over to go talk to him. That way he’ll owe me a favor down the road. That manager is yelling pretty loud. Let’s go see what has him all frustrated!

Manager Front

Oh. I see what happened. The manager was busy counting out his quarterly bonus and somebody interrupted him. They want him to pick up his red phone and call the network guy to do some troubleshooting. If there is one rule in the datacenter, it is that you do not interrupt the manager when he is counting his bonus. Who in the world interrupted him?

Manager Back

That guy? He’s a DBA who kicked off an indexing job 5 minutes ago and wonders why his servers are running slow. This won’t end well for him.

Well, that’s about it for the tour. You are welcome to come hang out in the network engineering office. There isn’t too much going on. We mainly just sit around and wait for the phone to ring. Since our datacenter is one big gigantic switch, not much happens in the way of changes. Still want to check out the office? Okay. Here you go. I’ll just relax for a bit while you check things out.

Network TV

I’m just watching a little TV. Not much going on. Just drinking my coffee and lying down in my datacenter bed.

Network Bed

It is pretty comfortable here in the fun zone. I wait around for that red phone to ring. When it doesn’t ring, I like to catch up on all the latest TV shows and take naps.

Hope you enjoyed the tour. You can show yourselves out.

That’s A Wrap

This contest was pretty fun. Obviously this isn’t a super serious datacenter design. That isn’t to say that I wouldn’t use any of this equipment. I would. I also may have been a bit harsh on the non-network sections of the datacenter, but it was all in good fun. If I was in the business of designing datacenters, I certainly wouldn’t have that many people running around the building. However, you can bet that I would have racks and racks of servers, storage, security, and network gear. Thanks to Juniper Networks for the chance to participate in something out of the box like this! Good luck to all my competition. There were some pretty cool entries that I have seen so far. I’ll link to the ones I am aware of below, in the order I came across them:

Stephen FoskettDatacenter History: Through the Ages in Lego

Amy ArnoldYou built a data center, out of a DeLorean?!

Ivan PepelnjakLego Data Center

Tom HollingsworthBuilding A Lego Data Center Juniper Style

Robert NovakBuilding the Best Lego Datacenter for Juniper

Posted in contest, data center, humor, juniper | Comments Off

Appreciating Complexity

SoundBoardFor four years, I played the French Horn in school. I learned a fair amount about music. Mainly, I learned how difficult it was to write music. Not just writing it, but also performing it at a high level. Once I understood that, I gained a new appreciation for musicians. It doesn’t even matter whether I like their particular music or not. I can appreciate the skill required to create and perform music. I have a fair amount of albums by musicians that some people think are garbage. One of those is John Tesh. When I would ask why they didn’t like his music, it was usually because it was too soft or boring for them. If they were a sports fan, I might actually play them a song from John Tesh’s Victory album. It contains an assortment of songs that have been used in various professional sports broadcasts. They would hear it and say something like: “I know that song. They play it during NFL or NBA games.” Sometimes, their opinion might change a little because they can now associate him with something they enjoy, or something they understand like sports.

Music is something we are all familiar with. We can play a song and just enjoy it for what it is. When we understand what is involved in creating it at a much deeper level, we can enjoy it even more. We are no longer just listening to simple song. We’re picturing the musicians that have practiced over and over until they produce a perfect product.


The Problem

Apply this same concept to what we do in the IT world. There is an art form to taking hardware and software and making it perform at a high level. We poke and prod and tweak our infrastructure to squeeze every ounce of performance out of it. We work nights and weekends to ensure everything is running as it should when the users come back in to the office each morning. We’ll spend hours and days plowing through logs to find out why an application is running a few milliseconds slower than it should. We do all this, and yet plenty of times, the IT department is seen as a cost center. A black hole for money. The IT people are expensive, and the equipment they work on can sometimes cost more than most people’s homes.

In many organizations, every dollar has to be justified and each additional IT body required is debated and debated. Do you REALLY need that extra person? Can’t you just handle it with the current staff? Can we outsource that function? Is maintenance REALLY required on all of that hardware and software? Certainly the vendor can come down on the price. Certainly that reseller can charge less. $150 an hour seems rather expensive for a consultant to come in and perform a routine upgrade. Why can’t our own people do that? The list goes on and on.

I’m not telling you anything you don’t already know. The problem has always been a lack of understanding on the business side when it comes to technical things. Of course, there is something to be said for engineers who just want to buy the latest toy because it is new and shiny. They don’t do IT departments any favors. Throw in the vendors and resellers that oversell a solution under the guise of “future proofing” the network and you have an even bigger problem. These days, there seems to be a push for the technical folks to understand the business requirements. I can’t disagree with that push. While I still contend that I don’t need an MBA to understand how to make the business successful, I DO need to understand the direction the business wants to go in to make sure I propose and implement the right solutions at the right cost. Sometimes those solutions don’t involve buying more hardware or software. Sometimes those solutions require more bodies or simply refining processes or advocating standards.

In a perfect world, the business would understand the technology and the IT department would understand the business. We don’t live in a perfect world, so we have to work around imperfect humans and imperfect execution. Maybe there’s a different approach to take for one side of that equation. I propose that if the business can appreciate the complexity of IT, they will tolerate it a little more. I suppose that already happens in plenty of organizations, but I still run into quite a few where that isn’t the case. I also understand that we all get paid to perform a certain role. I shouldn’t fault the accounting department for not having a thorough understanding of all things IT, just like the accounting department shouldn’t expect the IT department to have a thorough understanding for all things finance.


If I Ruled The World

This may all be a pipe dream on my part, but if I were trying to build a better appreciation of all things IT within a business, I might do the following:

1. Be nice. - There are plenty of stereotypes regarding IT people, and one of them is that they talk down to their end users they support. If you plan on making a career out of IT, you have to accept the fact that when people call you with a problem, it is usually because something is broken. They aren’t calling to tell you how good a job you are doing. They are reaching out because something is preventing them from doing their job properly. The best thing you can do is hear them out and assure them that you will take care of it as quickly as possible. After all, you are there to support the business and not the other way around. Yes, there are some people that are never happy, but I have found that most people are reasonable as long as you take the time to explain what is involved in fixing the problem. Sometimes, the end users can help you fix their problem even faster by sidestepping corporate bureaucracy via their management chain. If they know you are genuinely trying to help them, they’ll do what they can to remove any barriers within their control. However, if you treat them like idiots, they might take pleasure in seeing you struggle through the various obstacles you come in contact with when working on their problem. Be nice. It can only help you in the long run. The more allies you build in the various departments, the better your chances of getting what you need to be successful in your job.

2. Explain as much as they will let you, but do it in small increments. – People are naturally curious. I always like to explain as much as they will let me. Using analogies and examples they can relate to will help them understand technology that much more. However, I also don’t try to bombard them with more information than they are able to absorb in that particular moment. Over time, they’ll understand more and more and will be even more helpful when they encounter another problem down the road. At least, that has been my experience. I’ve never had an interaction with an end user where I told them that they gave me too much information. Successful problem solving can often come down to one little piece of information that would have gone unsaid had you not pried it out of them. If they know more about technology, they can give you better information to solve their problem. I know next to nothing about cars. Anytime I need work done on a car, I always want to know what the problem was and how it was fixed. I’ll ask the mechanic numerous questions about it. If something similar happens again, I am able to give better information to the mechanic in the hopes that it will help them come up with a solution quicker, and thus cost me less to fix it.

3. Know your infrastructure. – You MUST know as much as you can about how the software and hardware you support works. You cannot be content with simply being a tactile engineer and just pointing and clicking your way through your job. Of course, this has to be tempered with the level of depth required for your position. In larger departments, there are usually senior people that you can ask for greater understanding. In smaller departments, there might not be anyone else. Leveraging consultants, social media, or everyone’s best friend Google, can help out tremendously in that regard. It isn’t acceptable to be content with not knowing. Find an answer.

4. Spend what you must, but no more. – Focus on needs and not wants. If you have to fight for every dollar you spend, make sure you are only buying what you need. If you have a room full of unboxed equipment that exceeds normal spare levels, there’s a problem. If you bought some fancy software package that you just had to have and ended up not using it, there’s a problem. If the business knows you are cognizant of how much money you spend on gear, you’ll have an easier time when you need extra funds for something that wasn’t budgeted for. If you are able to explain in great detail why you need that upgrade or net new purchase and tie it to the success of the business, you’ll have a lot more wins than losses. If you are allowed to spend money at will and with little oversight, at some point, someone will shut off the money and you’ll have a harder time in the future getting projects funded.

5. Consider “A day in the life of IT” programs. – This one is probably the most unrealistic point, but it still bears mentioning. One of my friends in high school had a father who was a career police officer. One night, I rode out with him on the night shift. He was a shift supervisor, so a fair amount of his time was spent directing various officers here and there. I learned a lot in one night about the various things a police officer does in the course of their shift. It was an eye opening experience. By simply observing what he did and asking questions for clarification, I learned to appreciate how difficult that job can be. It helped me to understand why some officers might seem annoyed when interacting with the various citizens of whatever jurisdiction they work in. I’m not saying cops can do no wrong, but I gained a little more insight into their thought process. I can understand their lack of trust with people they come into contact with. In short, it helped me see things from their point of view. While I am sleeping soundly in bed at night, they are dealing with things I very rarely experience, and they do that every shift. If you could get people outside of IT to just come hang out with a few engineers for a half day or so, they might gain a better appreciation for how much work is involved in keeping a network up and running. This, coupled with my first two points(Be nice and explain as much as they’ll let you), can go a long way in building better relationships with departments outside of IT. While I don’t expect the CEO of a large corporation to come hang out with IT for several hours in a day, it would be great if they did. Just make sure than when people do something like this, that you aren’t sitting around surfing the Internet and reinforcing stereotypes that all we do in IT is play on the Internet instead of fixing problems.

Closing Thoughts

There are no easy answers to narrowing the gap between IT and the business. I could put it all on management within IT(ie CIO, CTO) and say that it is their problem. Unfortunately, that doesn’t do anything to help the problem. Everyone in the IT department has to help the business understand that we are there to enable them, but that it comes with a price tag. Getting businesses to view IT as something other than a cost center can make things a lot better for that IT department in terms of funding and head count.

What do you think? Is this just a bunch of unrealistic babble? Or, do you think that giving people a greater understanding of the complexities of IT will help?

Posted in career, efficiency | 9 Comments