Module 2: Interrogation & Monitoring
Module 3a: Basic JSON XML Parsing
Module 3b: Advanced JSON XML Parsing
Module 4a: Setting Up IKL Environment
Full Video Transcript:
Let’s start off the call today to kind of review really quickly on Module 1. I think everyone’s on board. It’s a very… What we discussed in the past module is fairly simplified, so I just want to review and make sure everyone’s on the same baseline. As we talked about in the past, there’s going to be really four components to the Indeni architecture. It’s going to be the collector and the server components. The collector component has the interrogation and the monitoring scripts. These are going to probably be the four biggest emphasis, because obviously when you’re adding new devices to Indeni, we’ll need to build out these interrogation scripts so that we know how to read them properly. Likewise, with monitoring, that’s how we’re going to be able to parse the data from your devices.
Another thing to keep in mind is the server component and how we have rule templates in place. So, try to copy as much as you can from what we have on our rule templates. If it requires some sort of customization, we’ll definitely discuss that in later modules, I believe, a little bit. We’ll talk about the rules and how Scala works a little bit.
A little bit. Yeah.
Yeah, just a tiny bit. But, our main focus for the most part is going to be on the collector part. One thing I wanted to note is Liz and I were kind of hashing this out and we were trying to understand how do tags get identified for the monitoring component of the Indeni system. And so, when you first initiate the Indeni and you have a connected device, it goes ahead and goes through the interrogation script. You guys are very well aware of that. It goes ahead and writes tags on certain values that we want to identify, but those tags then get placed into, for lack of a better word, the device entity table. And, that device entity table is stored actually on the collector component.
So, you know how we talked about how the collector and most of the scripts are stateless? Well, it actually maintains the device tags on the collector component, primarily. And then, it syncs and provides another backup on the server component. So, every time you connect to a new device or every time the Indeni instance fails to connect to the device for a while and it tries to reinitiate a connection, it will go through the interrogation phase. So, that’s for your reference how those tags are getting written, when they get updated, and when they get appended into the current device tag table, so to speak.
Given that information, we actually have a logical process where we basically define certain tags that are prerequisites for a lot of these scripts. So, in here, I’m showing you an AWK script for our demo, which is the ACME hardware humidifier. Over here, there’s the requirements that we’re going to go to in depth later, but these requirements are actually the tags that we wrote in the interrogation scripts. And, when you outline them here, the Indeni instance will build a logic to understand that given the vendor for ACME and for the product humidifier, we will run a series of certain scripts that require those tags under the monitoring script. So, that’s baked into the intelligence of the collector component. All you have to do is basically dump those parsing scripts into a file path that we’ll talk about later actually.
So, yeah. I just wanted to give you guys a quick, high-level view on that and give you guys some more context on that component. Let’s go ahead and start the Module 2.
What we’re going to be going over today is primarily on the collector component, and that’s all we’re going to really talk about. We did a quick review session, which is great. We’re going to go ahead and talk about the interrogation scripts and the monitoring scripts – how they’re different. There are small nuances that you should be aware of. We’ll start talking about scripting framework, and by that, I mean the mental framework on how to approach scripting using Indeni. That way, you don’t get stuck on how to approach the right methodology for building out scripts. We’ll talk a little more in-depth into that. We obviously want to focus on running commands that will have the least impact to the devices. So, there’s some strategy involved in designing the right script.
We’re going to talk about dos and don’ts of metrics and tags. The reason for that being that there are some metrics and tags that are reserved for the R&D team. Some things that you should be aware of, and also just other things that you guys have available to you when you’re building out your scripts. Then, we’re going to talk about how to be a copycat. I think that will help you a lot when you’re building out these alerts. And then, finally, the best part is how to simulate some of these Indeni scripts using CommandRunner. There’s going to be an extra credit part. Again, not necessary, but as you’ll see after you understand how to use CommandRunner, this will really pivot you guys to get on board fast. So, let’s move on.
So, as we talked about in the past, the Indeni collector has two components. One of them is the interrogation script, and the way I like to call it is… Or, actually, Yoni called it this. It’s, “The 20 Question Game.” It basically is going to identify everything from product, vendor, serial number, model number, operating system, the name, the version, if it’s in a cluster. And, all of these things are going to be necessary to determine what kind of monitoring scripts run. So, we’ve beaten the horse on this quite a bit, so let’s go and dive into an actual Indeni interrogation script we have right now.
And, I come from the Cisco world. I have a CCNA. So, this is the most relevant for me to understand how the interrogation scripts work. So, let’s say in this scenario we connect to a Cisco device. What we don’t know – and, a lot of you guys will probably be very familiar with this – is that you don’t know whether or not the device is running as a Layer 2 or a Layer 3 device. And so, that actually becomes a problem for us when it comes to interrogating the device, because we care more than about just knowing the model number. We need to know how it’s used contextually in the architecture. Is it doing Layer 2 or Layer 3?
And so, when you’re interrogating the device, here is a component right here. It’s called the meta component. And, right here you outline the name of the scripts. You give it a description. You define the type, which in this case is interrogation. And then, you define the tags that are necessary to run the script. Now, what we didn’t talk about before is that there are some interrogation scripts that are run after tags are identified. So, if we run a command to know if it’s a Cisco device with IOS, we will then subsequently interrogate the device using this script, as well.
Then, on this component, we define what’s the mechanism that we’re going to be communicating over, whether it’s SSH or HTTP. Obviously, this is a Cisco IOS, so we’re going to do SSH and then run CLI commands. And, we also would advise to provide documentation on things that we’re looking out for. So, right here, we documented command is usually not found on switches, especially when we’re running the command, we want to see if those outputs come out.
So, let’s move along. As you can see, there is going to be a definition on this part on how we parse it. So, the reason I’m going through this and I’m kind of narrating over it is because each component is going to be important to build the Indeni script. You need a meta component. You need the remote component. And, by the way, this is case-sensitive, so make sure that you are word by word and structure by structure, following how this is templated. But, in the parsing component, we’re defining what parsing mechanism we’re going to use. Even though we sometimes run commands over SSH, that doesn’t necessarily mean we’re going to parse the data over AWK. It might be a more structured output like XML. So, we also need to define that here.
So, as you can see, those are the four main components to focus on. The actual scripting takes place immediately after you invoke the parser command here. Right here, you start putting in your scripts. As you can see with the IOS, we are looking for specific things on the output. We’re going to use a regex pattern on all outputs that include Cisco IOS software, and then we’re going to look at several things and attach them to certain values. I’m going to over-generalize a little bit, just in case people aren’t really familiar with the IOS. But, what we’re doing here is basically looking at the output, and if it has certain configurations that are unique to Layer 3, then we’ll define it as being a router; but otherwise, if there are more than two commands that are missing that are unrecognized, we will then go ahead and assign it a tag of being a switch.
On top of that, we go ahead and write other tags, such as being the product, the serial number, the IOS, and so on. So, I just wanted to show you the interrogation script – how it looks like. Very similar to the monitoring script, but there’s going to be small nuances to pay attention to.
So, going back, let’s move along. Let’s talk about the monitoring script a little bit. As we mentioned before, these monitoring scripts are going to be executed based on the tags that were provided by the interrogation script. So, after the initial phase of the interrogation, the Indeni instance will then go ahead and start running these scripts. Make sure that the tags are outlined in the interrogation scripts. Otherwise, the monitoring will just not work in a real, live environment. Another thing to mention here is that the Indeni collector is stateless. We’ve talked about this before. So, when you’re considering how to build these scripts, make sure to account the fact that when you’re building the script, you do not maintain any information in the Indeni script where the monitoring script has to remember over a series of multiple intervals. So, in that instance when you are running the script, you should be able to run, by the end of it, run a command that will push a value immediately that the server or the rules will be able to recognize. So, just keep that in mind.
We’re going to look at the sample monitoring script, but while we’re doing this, I actually want to navigate through Bitbucket and show it to you. So, you’re going to be getting very familiar with this very quickly – probably more than I will. But, when we go to Source, this is where we have our current build of scripts that have been merged into the master file. You’ll see that there’s a way to navigate through this, so branches are going to be mainly the location where you will be working your scripts in, and then later on, you would then commit it, or make a request to merge your scripts into the master, which is the source.
So, let’s look here. Navigate through the parser domain. And let’s go to, again Cisco. That’s where I’m more familiar with. I do apologize if you guys want to look at more XML type of parsing. We’ll go over that with Yoni, actually. So, let’s do this one.
So, after we’ve identified what kind of a device it is, the vendor and all the interrogation scripts are run; you’ll see here that you’ll define the type of the script as being monitoring. There’s nothing really different from a template standpoint of how to invoke each of these components. So, obviously, it still needs to be #! META. Make sure it’s uppercase, as always. You always are going to require a name, a description, a type. The monitoring interval will define how frequently this script runs on the collector component, and then again, the tags that are required for this script to run.
This part is going to be a little bit different in the monitoring scripts. And, the reason we add this is because we’re actually trying to create a documentation of why we are running these scripts. And, if we’re running them, how are we running them, and is it possible without Indeni? So, as you can see here, we outline the “why”, the “how”, the “without-indeni” component. And then, we also want to know if we can run it through SNMP, or if you can find this information through syslog. Over time, this database will be able to help us identify what are the key best-sellers, and what are the things that Indeni are looking out for that can’t be found through SNMP monitoring solutions right now.
So, documentation’s going to be really important on the monitoring scripts. The person that did this one… Actually, I don’t know who it was. I don’t know if I can pull it up here. Oh, it looks like it was Dimitris, who was the last one who updated it. But, he did a really good job with documentation and defined each different output and what we’re looking for. So, we’re hopefully looking for something like this, and I think Liz will definitely go over this in much better detail.
Again, invoke the communication method and invoke what kind of parsing mechanism that’s being utilized. Then, go ahead and run your scripts here in the AWK component. It’s advised that for each line of code, that you can kind of outline what this part of the AWK script is running. So, we want to document that as much as possible. So, based on this input of an AWK script, what would be the designated output of that? And so, yeah. Dimitris actually did a really good job in providing that here. Let’s see. Anything else? No.
I think when it comes to actually AWK scripting – I’m going to move along to the rest – it’s going to be important to try doing some exercises on your own, outside of Indeni. It’ll help you kind of understand how it works if you haven’t used AWK before in the past. I do have some references that you could try. There are a lot of daily exercises, challenges that you can try to achieve using AWK. I’ll show you in a bit. But, let’s start looking at the IKE framework and how you should think about it when you’re building scripts for the collector component. And, we’ve tried to break it down and simplify it as much as possible.
What is the issue you’re trying to catch, and how do I check for the issue? What are the commands that you run for it, and how do I make sure that those commands are the least impactful to the device, considering that these are production environments? Most importantly, what are the outputs I care about, and how do I measure the output? Is it going to be a binary? Is there a counter reply to it? Do you look at it and compare it against something else? Is it threshold-based? How can I track this output regularly in intervals?
So, we talked about it in the past, but some miscellaneous things about the AWK-based Indeni scripts is to make sure you have documentation on the reasoning. It’s really important for us to understand why we’re doing this in plain text, in plain English. And, we also want to make sure that you adhere to the required structure, because it’s very case-sensitive, and… Actually, something I forgot to add is that you see the blank space here? It is actually not a tab. It’s actually four spaces. So, every time you’re indenting or you’re adding to these values, such as name, description, type, and monitoring interval, you’ll have to press space four times and do the same for vendor and OS name. So, this is actually space sensitive. Yeah, just in the beginning. Actually, something to note if you’re not familiar with AWK – it’s not space or indentation sensitive. So, you can play around with it as much as you want. It actually helps us if you space out your scripts as you saw before to provide us better context and make it easier for us to read.
One very last thing; I’m going to hammer this every single time; is to properly tag the devices during interrogation. I know with a lot of our new IKE buddies, we are looking at adding these new devices that we currently don’t interrogate. So, this is going to be probably a fundamental process that we are going to be hounding on regularly.
So, using AWK for parsing. If you are familiar with it, we are using the most basic AWK language. There aren’t any add-ons. We aren’t using sed. We aren’t using anything else beyond that. You’ll notice that some AWK scripts are not defined functions that are fundamental to AWK. We actually define them in the helper AWK file that we have in both Indeni and CommandRunner. So, the reason I bring this up is when you are starting to simulate these things through CommandRunner – and, I’ll explain why you would want to do it through CommandRunner – those functions are going to be available to you.
So, if you want to see an example of some of these functions available, let me quickly look at the Confluence page and navigate to it. I actually have this all available in the syllabus if you guys need it in a single document, you can definitely take a look at that later. Let’s go ahead to metrics. And, here are some of these pre-defined metrics that we have provided. So, the most common one is being double metrics. These are basically numbers. Anything that’s text literal would be complex metrics. Tags are very straight-forward. We just need to assign certain things to it.
One thing I want to add. Okay, we’ll go over that in a bit. But, let’s go to the contribution… There it is. So, if you go to this page, you’ll see all our helper functions that we defined for AWK already. So, these functions will be able to… You’ll be able to leverage using CommandRunner. And, a lot of them are pre-keyed, especially when it comes to identifying day or day time. There is a lot of different ways how this will be useful for you instead of having to create your own functions on the Indeni script. Less is more. I think that’s the key ideology here.
Below are some resources for you guys to leverage if you want to get up to speed with AWK. Grymoire – I think that’s the proper way to pronounce it. I’m sorry if I’m butchering the language. But, use that, especially as a quick, high-level overview of the AWK. You know, what are some of the functions you can use right off the bat with the AWK language? What things may be required using the AWK helper that we have. Then, if you want to take a look, Lynda has a great training, quick, basically ten to thirty-minute overview of how AWK works. And then, HackerRank is really useful for running a couple exercises for you to see if you’re comfortable with using AWK.
So, those are the dos about AWK parsing. Here are the don’ts. Do not use any of the reserved tags and metrics. I’ll show you right here real quick. But, that’s actually right here under this part of the navigation of the Confluence page. These are reserved metrics that we are using right now for some of our unique alerts. And, because of that, if you were to run into this, you would generate another alert that may not be coinciding with what you’re looking into. So, just keep that in mind. You won’t break the device, but I think you’ll start generating some of these random alerts that aren’t part of building these new scripts for you. And, there are some tags also identified here. Just keep a look out for them. There are quite a lot, but get familiar with which ones you can use and which ones you can’t.
Okay. Some are actually considered known metrics, and those are actually leveraged for us as a regular basis. So, if you use these metrics and these tags, they may be already associated to a rule template, and that may be easier for you to immediately enable a new alert for what you’re looking for.
Finally, let’s go ahead and start thinking about how we can test the Indeni scripts. If you haven’t yet, you may want to start downloading CommandRunner. And, as we’ve talked about in the past, you will need Java Runtime Environment. Otherwise, it just will not work. We have some examples in Confluence of how to run CommandRunner, but for training purposes, we really want to focus on doing single inputs and outputs. And, what I mean by that is you can run CommandRunner and simulate it on a static input text file with the Indeni script that you build and leverage that to provide an output to see if it actually works. The reason I bring this up is that sometimes you might need to test it on maybe some devices that you currently don’t have access to from the KDLab, or it might be remote. If you know what the input looks like, or if you know someone that can provide you that input documentation, you can document it, provide it, and then save it as a text file with a .input ending. And then, run CommandRunner to go ahead and parse against that input file. So, at the bare minimum, that will let you know if the script is actually running or not.
So, the extra credit. We wanted the extra credit here, and the reason why it’s so important and why you will really feel just as happy as this girl here is that you’ll feel like you’re really close to building your own Indeni scripts. And, the reason why is… and, the extra credit will have you actually start building your own Indeni scripts, either using AWK, XML, or whatever kind of parsing mechanism that’s necessary, and then use it against CommandRunner to see if it actually provides an output. If it does and you’re successful, go ahead and post it on our IKE discussion forum and have everyone review it. We really like the idea of having everyone take a look at the script, not to see if you’re doing it well or not, but really to see how everyone’s approaching this in their own way with scripting. And also, just to see if there’s some best practices that we could have the whole community enable.
So, that concludes my component of the Module 2. I wanted to leave the questions towards the very end. So, yeah, if you have any questions, right now is the right opportunity to do it.
