Module 2: Interrogation & Monitoring
Module 3a: Basic JSON XML Parsing
Module 3b: Advanced JSON XML Parsing
Module 4a: Setting Up IKL Environment
Full Video Transcript:
MALE 1:
Thanks everybody for joining. The session today is going to be around the JSON and XML parsers. So, these parsers are very different to the AWK parser. The AWK parser is meant to parse largely unstructured text that is just flowing in and you’re going through it line by line, kind of like a human goes through a document. The JSON and XML parsers are meant to parse, essentially, hierarchical data. Both JSON and XML are hierarchical data structures, and the parser is built for that. One of the differences between the JSON and XML parsers and the AWK one is that this parser was completely invented by us. So, on the one hand, it’s very focused towards what we try to do at Indeni and what the technology is trying to do, but on the other hand, it’s a completely invented language, so sometimes the learning curve is a bit more steep and a bit more difficult.
In our experience, we’ve seen that it takes a bit more time to pick up the JSON and XML parsers than it is for the AWK parsers, and it takes a bit more time to think about them the way they actually work. But, once you do an d once you’re on top of it, you can have very good control of what you want to achieve, and it becomes very powerful.
So now, let’s think about the need for a moment. I’m going to share my screen over here. So, let’s think about the need for a moment. So, let’s say I’m working on supporting Palo Alto Networks. And, Palo Alto Networks, you can grab data through an API that they invented, which is not exactly a REST API and is not exactly a SOAP API. It’s just kind of a different API structure. And, the output from the API is XML. So, for example, here is an API call. And, the API call says I want to run show admins. Okay. And then, providing an API key. And, the output of this… Can you guys see what I’m seeing? Okay. So, the output of this…it looks like this. Okay? It’s in XML. Now, in this specific case, the XML is not well formatted. It’s a bit difficult to track, but if you look at an online XML viewer… I try to just use one of these. Just pick one that you like and is easy for you. But, what you can do is you can format it, and then it’s a bit easier to see. So, you can see there’s a hierarchy here. Okay?
So, there’s the response. Within it there’s the result, admins, entry, etc. etc. It’s very, very important to take note of, because that’s essentially the entire way our parsing works. Remember that the XML can only have one root element, so only one element at the bottom of the hierarchy. That element can contain whatever number of elements, and those elements can contain whatever number of elements they want to contain. Okay?
Now a specific example here… What we want to do is iterate over each of these entries under the admins and pull data pertaining to these fields. Okay? That’s what we’re trying to achieve here. An important tool with XML is called XPath. So, XPath is a language used to traverse XML data. So, if we take that XML that we just had, an example would be I want to start from the root. Okay? Response. Then, I want to go into results, then admins. And then, I want every single entry. Okay, this is the XPath language. The training today won’t talk about the XPath language as a training for that, because there are a lot of online resources for training on the XPath expression, and, you know, you can kind of dig into those, but it’s important to realize that we use XPath heavily, because that is the standard for navigating XML content. If I run tests, you’ll see that it grabs and runs test, and you can see that my XPath found two elements. This is one, and this is the other. Okay?
A resource I found to be very useful for XPath is XPath MSDN. They have basically these different pages of the MSDN site of the Microsoft site. It’s pretty good with explaining how XPath works, a bunch of example syntax, functions you can use – things like that. This is very, very useful – their documentation. So, I highly recommend looking into that and getting to know it. And, you’ll notice things like the asterisk here, or double slash, or things like that. More advanced things, essentially, and you’re going to see the use of some of these within our existing INE scripts, so if the device you’re set to support has XML output, it would be a very good idea to get familiar with this thing, okay?
JSON looks like this, okay? You’ll notice it’s hierarchical, just like XML, but it tries to be more shorthand in the amount of text you need to write to actually be able to describe a JSON structure. And also, it just tries to make things a lot simpler and simpler to read, as well for us humans. So, you can see that there is a root here, right? It gets opened here and it gets closed somewhere down here. Here. Okay. Ignore the rest part. I’m not going to get into that at the moment, but essentially, this is the end of the root, and then within it, you have sets of keys and values, they call it. This is a key, this is a value. The value can be a number. It can be a string. This is a string if it has quotes. Or, it can be an array. These square brackets are arrays. Okay? So, for example here, this is an array of slots. Now, it happens to have just one slot, but I’m sure there examples here of arrays that have multiple like here. Nope, that doesn’t. Let’s see. Well, tis is an array, actually, if you think about it, because it has this key and then under it there’s an array with three elements, okay? So, if the device you’re working with have JSON output, it’s important to learn JSON, to essentially spend the time to look at JSON.
Now, remember we have XPath for XML. Okay? For JSON, you have JSONPath, okay? And, JSONPath is very different to XPath. So, other than the name, there’s no other similarity between them. Okay, if you’re writing a JSONPath, don’t assume that they’re similar in any way, shape, or form. If you look at this as an example, this is the JSON test content and the JSONPath is I want to start from the root. That’s the ‘$’ sign. Then, I want to go into the phone numbers, and I want to get only the first one. Okay? This means I want only the first one, and I want to get the type of the first one. Okay? I could write something like this, and then I’m going to get. Shoot, what did I get wrong? Then I get the last one. Sorry, because they both start from zero. If I do 0 to 999, I get all of them. See the evaluation? This website is super useful. The guy who wrote the f5 content I think spent most of his days working with this thing.
So, find the online evaluator you like for these two technologies. I mean, the one here for XPath is not amazing. There are some better ones out there, but find one that you find is very comfortable for you and stick to it. It can be very helpful for you, because anything you want to build from a path perspective, it’s easiest to first figure out the XPath or the JSONPath.
Now, to make matters even slightly more complicated… Okay, so it’s not here. I need to find it later. But, to make things slightly more complicated, the implementation of JSONPath in Java is slightly different to the global implementation of JSONPath. So, there are certain functions that exist in the global one and not in this one, and vice versa. There was a website that listed that that I can try to look for. I think it’s Jayway, if I’m not mistaken. Here, this. Okay, this is a better JSONPath evaluator for the Indeni platform, because we specifically use the Jayway JSONPath library in Java. And, like I mentioned, it’s slightly different to the main one, so if you’re using a JSONPath evaluator, it’s better to use this one - jsonpath.herokuapp.com and not jsonpath.com. This one’s slightly misleading at times. It looks better, and 99% of the time, the results are the same, but we just noticed that sometimes they weren’t exactly the same. And, you can see that they actually let you compare different implementations of JSONPath, but stick to Jayway. Okay?
So, summarize: XML, JSON – they’re both hierarchical data structures. You know which one is which being used by the product you’re supporting. Sometimes they would actually use both of them. Before you get into them, make sure that you do the tutorials of XML and JSON, or whichever language, and then get very comfortable with the path terminology. The XPath here or JSONPath for JSON. Okay?
So, now, let’s talk a little bit about the parser in our world. Once you nail down the path, you’re going to use the parser in order to generate metrics. So, as a reminder, metrics are essentially the units of data that we use to essentially store data about what’s happening in the devices and then run rules on top of them. So, the JSON parser and the XML parser are very, very similar. Actually, we can stick to this example. The structure is as follows: It has operators that have an underscore, so whenever you’re seeing an underscore in the beginning, this is a reserved operator – essentially an operator we invented in our language. And, these operators are used to take certain actions or to describe certain structures within the parser.
The ‘_metrics’ is generally the most known operator, and generally, you have only one of them in a script. This operator says that everything under this are essentially metrics that I want to generate from this script. Some of the metrics are just individual metrics, and some of these metrics would be multiple metrics with the variation in tags, which we’ll get to in a second.
Under the metrics operator, you’re going to see this funky symbol. Now, the structure of this language is called YAML. So, this. YAML. Okay? YAML is like XML, but more human readable. We use YAML in all kinds of different elements of our code, and in the case of the JSON XML parsers, YAML is the syntax we use for describing the parsers. Now, YAML’s very straight-forward. It basically has a hierarchical structure and it just follows that.
A couple of things to note with YAML is it’s very, very sensitive to the number of spaces you have here. Our standard is just having four spaces, so you just hit the spacebar four times. Do not use the tab button. YAML does not like tabs. So, you need to stick to spaces. Four spaces is the structure. So, you’ll notice this is four, and then this is eight, and then this is twelve. If you make the mistake of not having the right number of spaces, YAML will get confused with the hierarchy, because if I have this, for example, and this, and this has eight spaces, and this has seven or nine spaces, the YAML parser is going to assume that these are at two different levels of hierarchy, whereas we actually need for them to be at the same level. So, spaces are important. Okay?
Now, each of the metrics is separated by this little hyphen. So, this is one metric, and this is another metric. Okay? So, every metric starts a section with this little hyphen, okay? Metrics have values, and they have tags, okay? The most basic tags all metrics have is ‘im.name’. ‘im.name’ is the name of the metric, okay? In the AWK parser, it’s the equivalent of this. So, when you write a metric in the AWK parser… Do you see this? The first parameter we pass, which is the name of the metric – that’s the same thing, okay? What’s actually happening in the backend here is it’s adding a tag called ‘im.name = water-level’. That’s what the function’s doing. Okay?
So, the tag ‘im.name’, and the value itself is our total entries. Notice the operator here, okay? ‘_constant’ means just use the string as is. Okay? There are other operators that we’re going to talk about in a second, but that’s the simple operator that says just use it as is. Okay? Notice that YAML has a structure where everything is key value, except for hierarchy. Okay? You’re never going to have a situation where there’s just an operator here and nothing after it, okay? If you have something like it, then it’s going to be like this, where it’s essentially opening a hierarchy level. Okay? But, you’re never going to have just a simple key and no value and no hierarchy that’s being opened. YAML will not let you do that. Okay?
A common thing is with this ‘#’ sign, and you can put the comment after. This is the same as AWK. It makes it a lot easier to read the code afterwards. The value part talks about the value that the metric has. We have two types of metrics: Double and Complex. Double are just numbers. Complex are JSON structures, which we’ll talk about. The value operator has to exist in every single metric we create. You can never create a metric without a value operator, okay? We have double, We have complex.
Under the value operator, depending on the type of value that it is, you’re going to have the value itself, okay? This value says the value of, so it looks a bit funky if you look at it, but the value of the metric is a double. The value of the metric is going to be the value of evaluating just this JSON path.
So, this underscore value and this underscore value are completely different operators. Okay? This one says I'm now going to define how I set the value of the metric, whereas this one says go and use the JSON tab to find this and take the value you see at that JSON path. Okay? So if we take this example and say this is our input, and this is the JSON path. And the value is going to be 17. Okay? See, the 17 here matches the 17 here. So, what this is saying is I want to create a metric called ‘arp-total-entries’. The metric is a double metric, and the value of it is the value of the result of parsing this path. Okay?
Similarly, here, I want to create an ‘arp-limit’, which is a result of parsing the max. Okay, so if I change this to max I get 512. Okay? This is the simplest example.
Now the JSON parser and the XML parser are the same from the perspective of the structure. The only two differences they have is one, JSONPath versus XPath. And, two, some of the operators are different here, because they depend on the terminology of the relevant language. In JSONPath, there is a terminology called value. In XML, it's a bit different. Okay, so that's why some of the operators are different. If you look at the wiki you'll notice that for each of the parser, we have a list of operators that you could look at, okay? Any questions so far?
MALE 2:
Nothing yet on my side.
MALE 1:
Great. So, now let's start making things slightly more complicated. Here is a group operator or groups. Look at this. Notice that the input here has multiple drives, so what we want to do is generate a metric for each of these drives, okay? So, the metric name is going to be ‘hardware-element-status’, okay? And, to differentiate between the drives we add a tag called ‘name’. And, the first time the name is going to be ‘1’, and the second time the name is going to be ‘2’, okay? The way to achieve this is first by adding an operator called ‘groups’, which tells the parser, “Hey, I want you to iterate over multiple entries that are all in this certain path and generate metrics for each of them.”
And, the groups itself iterates over this path. Okay, so this essentially means “Start from the root and give me all the drives.” So, it's going to give me these. Okay? And then, in the tags I say I want to create a name tag, and the value of the name tag is ‘disk_id’. Okay? The value essentially means, “When you give me these elements, I want to go into the ‘disk_id’ of each of them. Okay?
So let's run through this for a second. It will be easier to see. So, this is the data, and this is the initial JSONPath. Okay? Okay, this gives me essentially the two drives. So, it gives me an array of the two drives. Notice that the drive element and the root element are not here, because I just asked for the drives itself. Okay? Now what I'm doing is… So, that's what I'm iterating on. So, essentially, it's going to pick this one first and then this one second. Then, what I do is I say, I want the value of this ‘disk_id’ to be the name, and I want the value of the metric itself to be equal to the status – ‘0’ or ‘1’. Okay? So, what it's going to do is it's going to go into one of the elements. So, I'll show you how that looks like.
It's going to go into the first element, and in the first element it's going to pick the ‘disk_id’. So, essentially what it does is it concatenates the JSONPath, and it picks the first ‘disk_id’. Okay? Which is ‘1’. And, it's going to use that as the name tag for the metric. And then, it's going to go and pick the status for the actual value of the metric. So, instead of ‘disk_id’, it's going to put value. And, that’s going to be input status. And, that's going to be the value. Okay?
Then, it's going to go to the next one and say, “Okay, I'm going to pick the ‘disk_id’, which is ‘2’, and then I pick the status.” The total result is going to be two metrics both have the same ‘im.name’, so the same metric name. You're going to have different named tags. One is going to say, ‘1’, and the other is going to say ‘2’. Their values are also going to be different. The first one's going to say, ‘1’. The second one’s going to say ‘0’. Okay?
So, let's look at an example with the CommandRunner. Oh, did we do the CommandRunner yet? Not yet.
FEMALE 1:
Briefly.
MALE 1:
Briefly? So, let's try to use this. I'll show you the example. So, this is the JSON, okay? Now, what we want to do is we want to extract the different nodes in f5. So, we want the status… For each of these guys, we want to know if it's up or down. Okay? Now, notice that there are three states here: ‘down’, ‘up’, or ‘user-down’. Okay?
Now, let's flip over to the parsing, which is here and you'll notice that these are pretty well commented, so that's pretty good. But, look there are two metrics here. This is one and this is the other. Now, they have the same name, and that's okay. It's okay to generate multiple metrics with the same name as long as the values of their tags are going to be different. Okay? Because, if you generate two metrics that have the values of the tags, then they're going to overlap.
But, basically what he did here is he said, “I want to iterate over all the items” – so, all the nodes – “where the state is either ‘up’ or ‘unchecked’ and the user didn't disable it on their own. “If it's ‘up’ or ‘unchecked’ and the user didn't disable it on their own, I want to create a metric called ‘lb-server-state’ with the name being equal to the full path”, which you can see that here, basically – the name of the node. Okay? And, I want it to have a value of ‘1’. Notice that he set the value as a constant, instead of trying to calculate it from the field. Okay? So, basically what he said here is, “For each of the nodes where the state is ‘up’ or ‘unchecked’ and the user did not disable it on their own, I want to write a metric and say, ‘The server is up.’” Okay? Then, he also built a reverse and said, “I want to go over all of the items, and if it’s not ‘up’ and it’s not ‘unchecked’, or the user disabled it.” Notice this is “or.” “Or, the user disabled it, then I want to write ‘0’, which essentially means that it’s down.” Okay?
If I run it through the CommandRunner, notice that this guy is up. This guy is considered down, because the user disabled it. This guy is considered down because the state is down, and this guy is considered down, as well. So, look at the results, and if we put one right next to the other, you’ll see that APP01 is considered up. See, it’s ‘1’ here. Okay? And then, the others are all considered down. Okay? So, the metric name is… Remember, this is the ‘im.name’, and the name of the actual node is within the name tag. Okay?
So, if we kind of go back to the script, this is a complicated script. It’s not the easiest thing to start with, because there’s this whole complicated JSON path and the logic that he put here. But, it’s a good example of how the JSON path can be used, and the JSON parser in general. The fact is it’s very, very powerful, because you’re going to see all kinds of weird outputs coming from devices, and you need to somehow parse them into something that our system can understand and is very standardized. And, that’s why we have to rely on powerful things like JSONPath and XPath, and of course the operators.
So, the question is can we set a default so that if it doesn’t match any of these, can we have some kind of default value. Unfortunately, we can’t. So, you need to account for every single option. There is one way around that. There’s something called transform, which we’ll get into. But, generally, yes. You’ll need to account for every one.
Notice that the vast majority of JSON parsing doesn’t have this kind of structure of multiple ways of generating the same metric name. They tend to be just more fluid. Let’s see if we have a good example here. This has a transform. This has a transform, as well. Yeah, some of these are really advanced. I don’t know. I don’t know if we have a JSON for Cisco. I’ll get to XML in just a second. Yup.
So, I’m trying to look if there’s a simpler example here. Yeah, this is probably a simpler example. What it’s doing here is we have a metric called ‘ssl-secure-handshakes’. We are going to generate this metric multiple times by iterating over all the entries at this level. And, for each and every one of them, we save its name from its description, and then the actual value is by parsing a specific path within that. So, if we look at this, we’ll see that what it’s doing is it’s going into ‘$.entries.*.nestedStats.entries’, which means ‘entries’, ‘*’, ‘nestedStats’, ‘entries’. And, inside here, it’s looking for this guy. Okay? And, for the name tag, it’s got this. And, it’s going to do that again. So, it does that again here. Entries, asterisk, nested stats, entries, and then it goes again and pulls it. Okay? And, again, I realize this is a lot more complicated than the AWK parser, but it’s also a lot more powerful. And, it’s a lot better from a performance perspective, we’ve discovered as well. So, these parsers are really good once you’ve kind of mastered them.
Okay, let’s switch over to XML. So, I was trying to find a simple one. This is AWK. AWK. XML. This has transforms. Let’s see if we have one without transforms. Transform. Transform. Transform. Let me try to find one that’s simple. Okay, this doesn’t have a transform. Okay, so the input is ‘show_ip_pim_neighbor’. Okay, this is how the XML looks like. Now, unfortunately, with Cisco, they’re not exactly issuing a valid XML. Okay? If I copy this…XML formatter, let’s see if it catches it. Yeah, it didn’t catch it. This is illegal, from an XML structure perspective. The rest here is actually kind of okay, except for the namespace part, which also is not exactly legal XML format. But, this is not legal. So, any typical XML parser that looks at this says, “This is bad”, and throws an error. Okay?
So, to deal with this, we have to add a special function for our friends at Cisco that is this. It basically says, “Drop the last line.” Okay? Now, if you ask any good programmer, they’ll think you’re bonkers for doing this. So, it’s not something we like doing, but unfortunately, that is the structure of the XML output, so we had to do it. Okay? So, that’s the ‘_omit’ operator you have here. The ‘_omit’ operator, by the way, also has other modes like regex, which we won’t go into right now, but if there’s a chance where you need to drop lines, but you only sometimes need to drop lines and sometimes you don’t, we have a way around that.
The question is, “What are they doing with the closing tag that is illegal?” This little thing here? We don’t know, because we don’t work there, and we didn’t ask anybody, but it looks like the end of a CDATA element. So, in XML, there’s something called CDATA, and you see that the CDATA element ends with this. Okay? ‘]]>’. ‘]]>’. So, it looks like just a bug. Whoever wrote the XML output, just somehow that happened. And, it’s very possible that the XML tool they test with just ignores that, but the XML library that Java uses doesn’t like it very much. Okay? But, essentially, it’s called the CDATA. That’s what we’re assuming. Okay?
So, if we go back to this, this is the XML. Now, in the case of Nexus, XML is very verbose. There’s a lot of text here. Usually, XML is not this complicated, but usually simpler. And, there are ways to circumvent that, but basically, this is the XML structure, and what we want from here is to get every single neighbor. So, we want to iterate over this – each row in this table, right? So, what we did is we define a variable, which in this case is ‘//__readonly__’, which means, “Find all the read-only elements.” Basically, this is how they’re written: ‘__readonly__’, like here. Where is it? Here it is.
So, it finds this. [inaudible, 00:35:03]. This is just for [inaudible, 00:35:05]. You don’t have to do this, but it allows you to just write ‘${root}. It saves you some writing and makes it a bit more legible. You don’t have to do this. You could just the same write this. Okay? But, it just makes it a bit easier, and the guy who wrote these scripts just found it very useful for him. So, the metric here that we’re creating is called ‘pim-state’. We want to find a read-only, and then on the read-onlys, find the vrfs, and on the vrfs, find all the neighbors. Okay? So, Read-only, vrf table, vrfs, find the neighbor [inaudible, 00:35:51]. This is what we’re iterating over. Okay? So, for each line that is call ‘ROW_neighbor’, we’re going to create one metric. The name is ‘pim-state’.
The name tag of the name specific neighbor is going to be the text of nbr address. So, nbr address is this guy. This is the text. So, it’s just like the ‘_value’ that JSON has. In XML, it’s called ‘_text’. The reason we differentiate is because JSON uses the concept key, value, whereas XML calls it element and text. That’s why we’ve got to call it text, okay? So, while we created two different parsers with two different terminologies, we have to match the terminology of the destination language.
We can ignore these tags for now. They’re irrelevant for this conversation, but you can see that I can even drop these, and this is enough to look at. So, basically, for every row that we find, we’re going to create a ‘pim-state’ using the neighbor address, and it’s going to always be ‘1’. The reason is that the way this protocol is built, if you can find the pim neighbor, then it’s up, by definition. If you can’t find it, then it’s down. Okay? And, that’s a bit tricky, because you’re basically saying, “Whatever’s down, I don’t know of.” It’s tricky, and it’s handled completely separately – not the scope of this conversation or this training. But, this is enough. So, if I were to run this. Okay, so the result is going to show me one pim neighbor. Its name is going to be this, and its value is going to be ‘1’. So, there it is. Name is going to be this. ‘10.255.20.1’, and its value is going to be ‘1’, which means it is up. Okay?
Just to play around with this, let’s say I do this. By the way, one of the cool things about XML is it doesn’t mind whitespace. So, this, it doesn’t care about it. So, that’s nice. So, we’ll do this. I’ll save it. I’ll run it. And then, you’re going to see two of them. One is going to be ‘1’, and one is going to be ‘8’. Okay? XML, by the way, also has comments, which look like this. JSON does not have a commenting structure. You guys don’t need a comment because what you’re getting is input from the devices, so you shouldn’t mind, but if you ever need it, JSON doesn’t have it, XML does. Commenting in the parser, in both cases, is the same. All you need to do is just use the ‘#’. Okay?
Sublime is really nice for editing these kinds of parsers. Sublime’s the tool I’m using here. Just because it lines these things up, so it makes it a bit easier to see how things are in line together. Okay? That’s the simple training. Now, we have more advanced operators in XML and JSON. It may be a good idea to have another session around that. One second, I’ll just explain what I did. But, as you noticed, we’ve been using ‘_constant’ for the value for text operators to just grab the content out of the output, but sometimes you need to modify the content before you actually generate the metric, and sometimes you need more complicated metrics, which sometimes we call complex metrics. And, those things require a bit more effort and a bit more operators, and that’s essentially a bit more advanced parsing. So, we’re going to need to have that essentially in another module and run through those. Questions?
FEMALE 1:
Yeah, so one of the questions that came through was you know how we’re ignoring the ‘]]>’, and we’re ignoring the last two lines? How come we’re not just ignoring that pattern?
MALE 1:
Okay, so good question. That’s a good question. We actually have a version of ‘_omit’ that does that. Let’s see if it was used anywhere. Hmm. It looks like it wasn’t used yet for Nexus. Do we have this in here? Here. So, you could do an omit regex, so essentially, the way it would look like is this. Instead of doing ‘_bottom’, you would do ‘_omit’, ‘_regex’. Let’s do this and see if that works. Yeah. So, essentially, what I said here is omit all the lines that start with these brackets and things and ends with it as well. Let’s try to botch this just to make sure that’s doing what it’s supposed to do. See? It failed now. Okay? Because, it didn’t remove this line. Okay, so this basically says I want to omit that line, and then it doesn’t have the bottom one.
FEMALE 1:
Are we using that?
MALE 1:
We were supposed to, but now that little find thing that I ran showed me that we’re not, so we could spend the time to go through it and fix all of those scripts. I think there’s even a ticket for that somewhere. Any other questions from anyone?
MALE 2:
Yeah, so the question is the parsing on the JSON and XML – which one is more easier? If the source provides both, like the JSON and XML, which one is better to do parsing? I think JSON is easier to parsing.
MALE 1:
Yeah. I essentially agree with that that JSON is generally better to parse than XML. Unfortunately, it boils down to whatever the vendor wanted to do. Like, some vendors will say, “Okay, we used to have an XML API. Now, we’ve built a JSON API” because it’s more modern, more trendy, but the JSON API doesn’t cover all the stuff that the XML API used to cover, or it misses certain things, or the responses are different. So, unfortunately, at the end of the day, it’s not necessarily the technology that you want to parse, but rather what the vendor decided to expose. Like, a lot of times we’re going to use SSH commands and parse with AWK, just because the vendor made no way of actually getting this data through other formats. So, that’s usually what dictates. But, if you can prefer JSON over XML, that’s generally preferred.
MALE 2:
Okay, thanks.
MALE 1:
No problem.
FEMALE 1:
So, it just depends on the output of the device, essentially?
MALE 1:
Yeah. It depends on what the device is willing to give you. Any other questions?
FEMALE 2:
Yeah, I think it for Nexus, what [inaudible, 00:44:32] told us was that he started off with XML, and then finding that all the Nexus platforms they’re really bad in terms of [inaudible, 00:44:40] conversions, so he’s actually going back to actually undo some of what he’s done and using AWK instead. So, they might actually find issues like that, particularly in the Nexus platform, so that’s something we might have to work with [inaudible, 00:44:55].
MALE 1:
Yeah. We have the same problem with Palo Alto. So, the issue is that from version to version, the vendors just choose to change things sometimes. And, it’s unfortunate, right, because a developer says, “Oh, I’m going to change it.” They change it. It goes through the entire process. Nobody actually checks that part, and then you discover this two years later when one of the scripts break. Unfortunately, that’s common.
Actually, with Palo Alto, there’s a specific API call that changed between 6.0.1 and 6.0.2 or something like that, and low and behold our luck, our customer is running both of them, and half the scripts weren’t working. And, it was the same script, and we couldn’t understand why not. And, when we reached out the them, they explained that to us. So, unfortunately, that happens.
Multiple ways of dealing with that. One way is to just have both versions in the same script, because remember it’s going to iterate over the path, and if it doesn’t find anything, then it doesn’t find anything, and that’s okay. And, what you could do is you could just do this and just create two metrics with the same name. The other one just has a different path. You can say, “Okay”, so in one of the versions it doesn’t have the vrf, and then you just iterate over both of them. Yeah, it’s a pain in the butt, though, because then you’re maintaining two versions of the same script within the same script, and it starts to get confusing. And then, you need to add the comments here and explain what you did. So, that’s one problem.
Another way around this is sometimes they only change the level in the hierarchy that the element is at, so you could just use something called ‘//’ and then you can say, “Wherever it is in the hierarchy, I just want the table neighbor.” So, there’s that way as well. But, if they radically changed it, then either having two ways of pulling the same metric, or maybe changing to AWK or something like that. But, even AWK [inaudible, 00:47:03], because they’ll change the commands all the time. A bug we just opened to the Check Point guy was that in one of the versions of Check Point, they added another column in one of the commands that he’s using, and that column broke his script. So, you’re never fully insulated from these changes unfortunately. Any other questions? Thought?
So, the last thing is just words of encouragement. I gave them in the beginning. I’ll give them now. These parsers are more difficult than AWK, but far more powerful. Once you get the hang of them, you can build really, really cool things. So, don’t dismay as you’re hitting your head against the parsers sometimes in the beginning. After a few cycles of training, you can get some really cool stuff. Yeah, great. Thanks.
