Could Cisco ACI Kill APM?

APM TargetNote – This is ALL 100% speculation on my part. I may be WAY off base with what you are about to read, and if you know something I don’t, feel free to correct me in the comments below.

I attended the Cisco Live Local Edition event here in Nashville,TN last month. It was an all day event that gave a variety of presentations in different focus areas. While I spent the bulk of my time in the routing/switching/wireless/security presentations, I made a point to sit in on one in the data center track. It was entitled Data Center Fabric Futures. This session spent a lot of time talking about Cisco’s Application Centric Infrastructure(ACI) technology, so I was curious to learn a bit more about it since the company I work for sells a fair amount of Cisco Nexus switching.

If you want a little more information about Cisco’s ACI technology, here’s some really good writing on that subject:

Insieme and Cisco ACI [Part 1] – by Matt Oswalt

Insieme and Cisco ACI [Part 2] – by Matt Oswalt

Cisco’s ACI (Insieme) Launch – by John Herbert

While the presentation was moving along, one particular aspect of ACI caught my attention. It was a specific function within the Application Policy Infrastructure Controller(APIC). Before I dive into that, let me give you a brief overview of the purpose of the APIC.

APIC is the brains of ACI. Think of it as the controller for the network. Control plane operations can be orchestrated from here in the same manner that a wireless LAN controller would do for wireless access points. Anyone who has followed the industry buzz around SDN is familiar with this concept of a network controller. Instead of doing a lot of manual configuration on each and every switch and router, the controller(APIC) would handle optimized routing, QoS, and other configuration tasks automatically. Policies can be deployed and removed on the fly without a human having to intervene at every step along the way of a given data flow.

Getting My Attention

When the discussion moved to the monitoring aspects of APIC, I was suddenly even more interested. The ability to monitor traffic flowing across the network is something that every decent sized network requires. Well, it may be required, but the price tag associated with platforms that can provide this sort of monitoring tend to scare off all but the most committed organizations. They have a reputation for costing an arm and a leg. I have seen customers get excited about what a particular network monitoring product can do, and then lose interest once they see the price tag. I’m not arguing that companies with decent technology should sell it for next to nothing. It takes a lot of skill and hard work to develop any decent product and companies should be able to charge what they think is a fair price for their hardware or software. The price will be dictated by what customers are willing to pay, and if enough of them agree to that price, the vendor doesn’t necessarily have to come down on pricing to satisfy what a customer “thinks” they should pay for said product.

Here’s the slide that caught my eye:


Forget the basic flow based monitoring tools or up/down state monitoring tools using SNMP. Those are old news. Granted, they are still quite useful, but in today’s larger networks, they don’t go deep enough. Consider all of the multi-tier applications that companies are employing today. What may be looked at as a simple web based application to an end user could in reality be a multitude of servers, load balancers, and other devices on the back end to allow that web page to be displayed. The complexity is hidden from the end user, and rightfully so. They just want it to work. You and I get paid to figure out how to make that work.

Enter Application Performance Monitoring(APM)

Over the past several years, I have had the pleasure of working with a few different APM vendors. In addition to logging how much traffic has traversed the network, they can drill down even deeper and show you precisely what that traffic was made up of. Instead of just telling me that a flow was comprised of Microsoft SQL traffic, APM can tell me which specific operations to a given database were made. Let’s say that you have a web application that is running slow. With APM, I have the ability to look at each transaction within that SQL flow and see if a particular “select” statement was taking too long to process. Instead of just telling the DBA’s that their database was running slow, I can point them to the precise operation that is causing that slowness.

Of equal importance is the fact that I can map out all of the applications and see exactly which systems are talking to each other. I can’t tell you how many times I have had to help a client troubleshoot a slowness issue and there was little to no information about how all of their systems interacted with each other. There is always some box in the corner of a datacenter that has been running for years and nobody knows what it does, except that when it goes offline, everything breaks. APM can tell you precisely what the box does, as it knows all the other systems it talks to, and what type of traffic is being sent and received by that unknown box.. The value of mapping out all of an organization’s applications and who talks to who should not be overlooked. When it breaks and nobody knows how it works because the original system architects have moved on to other jobs, much pain will ensue. Been there. Done that. Got plenty of scars to show for it. THAT reason alone is enough to justify the cost of an APM solution, unless you don’t really know how much your downtime costs you. If that is the case, remind me not to buy any of your stock. ;)

Allow Me To Speculate……

APIC is not available for purchase yet. I just checked for SKU’s on the Cisco pricing list and couldn’t find any for APIC. As best I can tell based on what I have heard, it should be coming within the next year or so. This shouldn’t be too much of a surprise though. Juniper did the same thing with QFabric. The whole solution came in phases.

Based on the fact that APIC is not available today, I have to speculate on what it MIGHT be from an APM perspective due to lack of the ever popular Cisco design guides and extensive documentation. There’s also the understanding that over time, more and more capabilities will be added. This is NOT unusual within the industry, or even Cisco for that matter. The Nexus 7000 series product line didn’t come to market with everything it has today. It took time, and so will ACI.

I am also aware that Cisco’s ACI solution was probably not even intended for use as an APM-like platform. I’m just trying to think outside of the box here.

Still with me? Good.

Imagine the possibility of being able to see the health of an application from the same vendor that sold you all of your data center switches. No more span sessions or expensive network taps. No more high dollar appliances which are commodity Dell, HP, IBM, or Cisco servers with a different vendor name slapped on them. None of that. Granted, APIC won’t be free, and I wouldn’t be surprised if licensing wields its ugly head and the APIC functions are carved up under various licensing SKU’s. That’s just the way the game is played by the majority of vendors out there.

How deep will APIC be able to see into each application? I am unsure of this, but I am pretty sure it will be a bit more than just flow data or standard ICMP/SNMP health checks. I find myself wondering whether or not an “application” from an ACI perspective is the same as an “application” from a traditional APM vendor. An ACI “application” might just be a grouping of hardware and virtual resources as opposed to the lower level functions within an “application” such as database operations, etc.

The two terms that I am VERY interested in from that slide are “health scores” and “resource consumption”. They might have different meanings than what I would expect to see from an APM vendor. If they are, then you just wasted time reading this, and I wasted even more time writing it.

Closing Thoughts

I don’t know what Cisco ACI will do in terms of application monitoring. My guess is it will not be anywhere near as comprehensive as the mainline APM tools out there. However, it may be good enough, and quite a few networks out there are monitored with systems that are good enough, because the cost of APM solutions was too much for them to stomach. If it costs a little more to get even more insight into the applications running over the network, I would say that will be an easier sell than trying to bolt on a very expensive APM solution from a third party vendor. I say that as someone who LIKES the APM solutions from other vendors. I just happen to dwell in reality where lower cost often overrides the best technology choice.

Posted in cisco, data center, monitoring, network management | 1 Comment

Architecting Supportable Designs…..and a rant or two.

Tennessee Aquarium in Chattanooga,TN. A fine piece of architecture.

Tennessee Aquarium in Chattanooga,TN. A fine piece of architecture.

I just spent 9 weeks on site with a client building out a network reference architecture. The goal was to provide them with a framework they could use to configure their network in the future without having to reinvent the wheel every single time. I ended up with almost 90 pages of documentation which included somewhere in the neighborhood of 30 Visio diagrams. From a network infrastructure perspective, this reference architecture included routing, switching, wireless, network management, QoS, WAN optimization, and a few other things.

It was a very interesting experience, as I have never really sat down and thought about how I would design the network as a whole. I’ve done my fair share of implementations, but they generally focus on one particular area of the network. Companies don’t generally forklift their entire network or change every device configuration across the board all at once. During this process, one thing kept nagging at me in the back of my mind. It followed me through every major section of the document. I needed to ensure that this particular reference architecture could be supported by the networking staff.

Due to that, I had to make some tradeoffs. That doesn’t mean I slacked off and didn’t consider all possibilities. It just meant that I had to ensure that any recommendation I made would make sense to the average networking person out there. Did I explain the reasons why I chose to use NSSA’s in OSPF multi-area environments? Did I include caveats to running things like VTP on Cisco switches? Did I provide enough information to help explain why I would limit the SSID’s on wireless AP’s to 4 or less?

By writing the document with the thought of people being able to support the proposed architecture, it wasn’t an attempt to say that I am much smarter than the design choices I recommended. I was being pragmatic. I DO have an ego(Don’t we all?), but I try not to let it come out and play too often. I’ve been wrong WAY too many times to go shooting off at the mouth, errr…..keystroke.

The Problem

What I am about to tell you isn’t something you don’t already know. There is a shortage of mid to senior level qualified networking people out there. There are a LOT of tactile engineers floating about. By tactile, I mean people who have done process XYZ a number of times and their level of understanding never goes beyond that. This is easily discovered when things break and their troubleshooting methodology is either non-existent, or doesn’t go beyond a few “show” commands on a device. If this is a junior level person, that’s not a huge problem, unless the thing that broke is well within their skill set. For senior level people, and I realize that “senior” is a very subjective term, there is less wiggle room for lack of understanding when things go wrong. Especially if they were the person who designed and implemented that particular aspect of the network.

I also fully admit that my opinion is based on the encounters I have had, and in no way could I claim to speak as an authority for the entire IT community at large. I will say that when I mention this subject to my peers or network managers out there in corporate America, they tend to validate as opposed to dispute my assumptions. As always, I reserve the right to be completely wrong and have someone tell me that I am wrong. I am also approaching this from the position of networking. I don’t pretend to understand the other silos in the way that I think I understand networking. This could probably apply to those areas as well.

Why This Problem Exists

There are a number of reasons I think this problem exists. The easy thing to do would be to blame everyone for not having the drive to develop themselves professionally. Realistically though, it’s a bit more complicated than that. Allow me to put forth some theories here.

1. Companies don’t care about professional development. – This isn’t true for all companies, but it tends to be the norm in my experience. Most IT managers I have dealt with really do have an interest in ensuring their people can do the job required. However, their directives are to get people in the door that can do the job with little ramp up time. They need performers from day one. Training may be promised during an interview, but all too often, it is never followed up with actual training time or resources. It is lip service. Employees that have a need to develop themselves are going to do it on their own, but don’t expect them to stick around for too long. They’ll gain additional experience at a particular company and move on to somewhere else. They know a dead-end when they see one.

If the networking team at your company consists of more than a few bodies, managers should be doing all they can to ensure their people have freedom and support for professional development. Lots of people stay at companies for years in spite of the fact that they could make more money somewhere else. There is something to be said for being in an environment where there is a solid technical team lead by competent managers. A good team can work wonders with even dated infrastructure. The better prepared your people are, the better your network will be run. That may not fit into a nice little spreadsheet in the accounting department, so network managers need to be prepared to defend the reasons for employee X taking a half day every week to train/learn. I can definitively state that there are a handful of people I have worked for/with over the years, that I would gladly work for/with again. When you find a good team, you don’t want it to ever end, even though it usually does at some point.

Tip: If you are interviewing for a job and they mention that they support training, ask them which of their employees have been on the receiving end of that training in the past year. If they can’t answer it, that should set off alarm signals in your head.

2. People don’t know how to develop themselves. – Remember your first job in IT? Were you overwhelmed? Did only a small amount of things make sense? You aren’t alone. I suspect it was like that for most of us. The difference, I think, is having an environment where learning and development is encouraged. The senior level people need to take time to ensure the junior level people understand what it is that they are doing. Looking back over the years, I have had the good fortune of working for people who took an interest in my development. They gave me projects and refused to hold my hand. They would help me out if I got stuck and needed further explanation, but overall, they instilled in me the need to research and learn things on my own. It has served me well over the years, and above all, I always want to know WHY something works the way it does. I credit my mentors for instilling that value within me, and hope I can do the same for others.

For people that didn’t have the fortune of good mentors, they may spend an entire career just going through the motions and repeating the same tasks over and over. While some out there are just lazy, I think there are a fair amount that just don’t know where to begin. They also may not be able to visualize how far they can actually go if they take the time to step out of mediocrity and go for it.

Tip: If you work in an environment where you are laughed at or belittled because you don’t know how everything works, find another job. People that withhold information from you are probably insecure and don’t know half as much as they let you believe they do. I’ve learned plenty of things from people that had far less experience than I did over the years.

3. We don’t screen candidates properly. – I could go on and on about recruiters sending in lambs to the slaughter, but I won’t. They aren’t going to change their tactics. A fair amount of them are just trying to meet that quota and don’t really care, or even know, if a candidate is viable. One of the things I do once a year or so is have lunch with a recruiter that I trust, along with some of his newer recruiters. We tend to eat at places that use paper as the table cloth, and in between bites of food, I draw out different network technologies for them. Very basic stuff. Here’s a switch. Here’s how it works. Here’s a load balancer. Here’s what it does. Etc. They will ask questions about things they have heard. I try my best to answer it in a way that makes sense to them. This is all done to ensure that when they start to vet potential candidates to put in front of a client, they can ask some basic things and perhaps understand the candidate’s resume a little more. I get a free lunch at a nice restaurant, and they get some basic introduction to networking. An even trade in my opinion. It helps my belly grow a little more, and hopefully, it makes them better at their job.

It isn’t just recruiters though. Sometimes network managers don’t have a technical background. I’ve seen good network managers with heavy business backgrounds, and I have seen good managers who have technical backgrounds. Everyone is different. Lots of different opinions on which approach is better. For network managers who only understand their infrastructure from a very high level, it means that they might end up hiring someone based on a resume and their ability to make themselves seem technically stronger than they are. They don’t know what to ask them unless they are hiring a very junior person. If you, as a technical person, see enough resumes, and provided a recruiter hasn’t “fixed” the resume, you get a general idea of whether they are the real deal. The personal interview will usually reinforce your initial instincts. If you aren’t doing technical interviews for your higher level positions, you WILL get burned at some point. Don’t settle on candidates just because you need a body. If all companies performed technical interviews at the level the position required, people would take interviews more seriously. That would make the candidates prepare a lot more than some of them do.

Tip: If you fail an interview due to technical reasons, remember those things you missed. Write them down during the interview, or shortly after leaving while they are still fresh in your mind. Go home, accept the fact that you failed, and research the things you missed. One of the saddest things I hear from people that failed technical interviews, is that they never bothered to figure out what the proper answer would have been. I once got administrative distance and EIGRP’s advertised distance confused in an interview. Although I ended up getting the job, I was so mad at myself for missing that(Hello type 1 personality!), that I went home and memorized the administrative distance table. Years later, I can still write it down if someone were to ask me to. That’s how bad it upset me. Don’t let a failed interview go to waste. Learn from it.

Additional Tip: If you fail an interview because someone was trying to make you feel stupid, consider it a blessing if you don’t get that job. You probably don’t want to work with people who like to flex their little bit of power and use interviews as a forum to show you how much they know and how little you know. Give it a few years in a healthy environment, and you will run circles around them. In the future, when you are conducting technical interviews, remember how it felt when someone was being openly hostile towards you. Don’t do that to others. It serves no purpose. Also, remember not to confuse standard technical interview pressure with open hostility. I’ve been a part of technical interviews where multiple people would hit a candidate with questions all at once in rapid fire. It was designed to see how they operated under pressure. The questions were not asked in a threatening manner. They were just probing to see if the candidate could take each question and provide an answer without becoming flustered. Yes, you’re nervous in an interview. Perfectly natural. However, when things go wrong at 2AM and you are the one who needs to fix it, you can’t shut down if the problem count goes up.

The Fix

Fixing this problem is two-fold. It’s “tough love”, but it needs to be said nevertheless.

1. Companies – As a general rule, you guys are lousy. Not ALL of you, but a fair amount of you. I get it though. You are focused on the stock price. The quarterly numbers. You don’t have time to sink more cash into that money pit we call IT. Unfortunately for you, you need to take better care of your people. Imagine a place where your employees were willing to do what it takes to get the job done. Imagine a place where you didn’t have to manage by intimidation. A place where you had droves of people waiting to interview for a position at your company, instead of just settling on you because a better offer didn’t come up. Imagine a place where you don’t have to micromanage everyone because you didn’t trust them. They’ll make it happen because they are committed to the success of the business and they are solid professionals.

What’s that? You have an ample supply of corporate drones that don’t want to rock the boat because they are happy to have a job? Here’s some news for you. When you have a group of performers in IT, they have options. If their personal networks are developed and they have in-demand skill sets, they don’t need you. You need them.

Is your IT department a revolving door? Maybe you should stop and ask yourself why. Of course, if the profits remain high, who cares, right? You think outsourcing is the answer? Go for it. Let me know if that works out for you and it comes in under budget.

Take an interest in your people and they will do wonders for you. Here’s a secret that your accounting department and CFO will love. You don’t necessarily have to pay them all a ton of money. You could spend a little on investing in their professional development and they’ll appreciate you for it. That doesn’t mean they will all stay. Sometimes you outgrow a job from a technical perspective. It happens. It’s part of the natural evolution of the IT professional. I can bet you though, that a lot more people will stay if they think you actually care about them. For those that do leave, do exit interviews. It takes a little extra time, but wouldn’t you REALLY like to know what makes people leave your company?

2. Individuals – Abhor mediocrity. I don’t get to use the word “abhor” much, so I am really glad it fit here. :) That’s about it. Don’t be content with treading water. Go out and drive your career. Don’t be a passenger! I can assure you that you will have far greater rewards in the long run. Spend some time figuring out what you want to do. Identify your weak areas. Be honest with yourself. Take some extra time and bump up your knowledge in those areas you are weak in. This will probably require time outside of normal work hours. There’s a fair amount that can be done without spending tons of money on educational material. Google will show you the way. Social media will help as well. Get on Twitter, even if it means being passive and just watching. Read blog posts and networking forums. Be a sponge.

I’ve talked with numerous networking folks over the past several years in interviews and casual conversations. I’ve recommended people for jobs they didn’t have the exact experience the employer wanted because they had potential and were strong in other areas. You knew after talking to them that they were destined for bigger things. They were hungry, and I will take “hungry and inexperienced” any day over “experienced and treading water”.

Closing Thoughts

I love what I do for a living. Life has a way of sending you in directions you never intended to go in. As a child, I was in love with the idea of being a pilot in the US Air Force, being a police officer, or being a cartographer. I ended up in IT. Looking back though, it was the greatest thing that could have happened to me. IT is one of those unique fields where YOU can dictate where you end up. If you are willing to put in the time studying and never settling for the experience you have today, you really can be anything you want to be. This is a knowledge based industry. The playing field is fairly level, in my opinion. Those that are willing to do more than the next person will find success. I firmly believe that. Yes, there is some luck involved, but in a way, you make your own luck.

I’m never content with what I know today. This industry moves way too fast to sit still. Things change. Technologies change. Vendors come and go. Jobs come and go. As you move around and move up, your value increases provided you never stop learning. Companies WANT people who are high performers. It may take you several years and many sleepless nights, but opportunities will come your way provided you put forth the effort to be ready when that opportunity presents itself.

I started out taking about network architecture, so let me end this long-winded post talking about that same thing. I’ve seen my share of network designs that were lacking, not necessarily from a hardware perspective, but from a configuration perspective. They were lacking because the people implementing the design weren’t fully aware of what their options were or they didn’t understand the technology in general. This is not always true. There are some very deliberate things done on networks, for very specific scenarios, that at first glance seem erroneous. However, it seems more common that things are done due to a lack of understanding. The same might be said for even the recommendations I made in my designs. By constantly improving ourselves, every design will get better and we’ll have fewer 2AM calls and alerts due to problems we inadvertently caused. Look at it this way. The more you know, the more time you will get to sleep at night. Unless of course, you are up late studying. ;)

Posted in career, documentation | 8 Comments

Easier Cabling Migrations

SergeantClipLogoSwitching upgrades are a necessary part of any network over a given period of time. Port speed requirements change. Platforms change. What doesn’t seem to change as often are the cables themselves. This is especially true in wiring closets. While I wish that the majority of wiring closets I have worked in had pristine cabling, this is sadly not the case in most of them.

The biggest challenge with updating switching platforms in wiring closets is always the cabling. This is especially a challenge when your maintenance window does not include adequate time to replace or reorganize the cabling itself.

In the interest of doing the job correctly, given the overall limitations, it is generally advisable to label all of the patch cables so that you know which port it came out of and which port it needs to go into on the new switch. Labeling hundreds of drops takes a tremendous amount of time.

The last batch of wiring closets I upgraded would have been the same old process of labeling each cable ahead of time, except I was armed with a new tool that saved me from having to do that.

Sergeant Clip

In September of last year, I received the following message on Twitter from @sergeantclip:

I get these kinds of messages from time to time, but this particular product seemed intriguing, and so I went to the website and poked around a bit. I didn’t have a use for it at the time, but filed it away in my head as something I might be interested in down the road.

In December, I had some wiring closets to update for a client that involved removing older Cisco 6500, 4500, and 4003’s. They were to be replaced with Cisco 3850 switches. It is always a pain to move from chassis’ to single switches or vice versa. The cabling is going to go from a spread out configuration to a denser one or the other way around. When you have the ability to move the patch panels around to compensate for the physical changes, it works out pretty well. Unfortunately, that isn’t always the case.

In light of the work my co-workers and I would be doing in these wiring closets, I thought it would be a good idea to test out these Sergeant Clips and see if they could save us some time. They did. In fact, they saved us a LOT of time.

The product is pretty straightforward. Take a bundle of up to 12 cables and insert the clip around them. Clamp it down, and now a 12 cable bundle is able to be disconnected and reconnected in a pretty quick fashion. You just have to keep an eye on which bundle goes into which port group. Instead of labeling 48 cables per switch, you just had to worry about 4 clips.

Here are some photos of the Sergeant Clips without all the cabling:



Basic Clip

































Here is one of the wiring closets we had to perform work in. The cables are all over the place, and although they appear to be labeled, the labels did not always match the port number on each module.



Here is that same closet with the Sergeant Clips in use:

PostClip1 PostClip2

Not all closets were messy. This photo came from a closet that had relatively pristine Cat6 cabling. Even with the bulk of that cable when compared to Cat5, the clips still closed over a group of 12.

NeatCabling Clip2


Closing Thoughts

This product is all about time savings. The sooner I can get a wiring closet swapped out with the new switches, the sooner I can move on to the next one. Over the course of a week and a half, we migrated to over 50 new Cisco 3850 switches from numerous Cisco platforms, with the most common being the 6504/6/9.

Any future migrations I do in wiring closets will include the Sergeant Clip. It was reasonably priced. I bought 16 of the 12 port clips and it was well under $100 USD. The product ships from the UK, so the prices are all in UK pounds. I’ll let you do the math and figure out how much it will cost you. Click here for the pricing section on their website.

I know what an hour’s worth of time costs my company per engineer, and I can tell you that the Sergeant Clip paid for itself within the first hour of use. If you are doing any sort of work in wiring closets, this NEEDS to be in your tool bag.

Disclaimer: I paid for this product out of my own pocket. I was not asked to write this by the folks at Sergeant Clip. They had no knowledge of this post before it was written. Yes, it is that good, and worth breaking a multi-month blogging hiatus. :)


Posted in efficiency, hardware, switching | Comments Off