Episode 180 – JSON All The Things!

- [Ben] Welcome to Episode 180 of the Microsoft Cloud IT Pro Podcast recorded live on May 29 2020. This is a show about Microsoft 365 and Azure from the perspective of IT pros and end users where we discuss the topic or recent news and how it relates to you. In this episode, Scott and Ben talked through a recent experience Scott had when working with service principals in Azure AD for use with Azure Kubernetes Service and then wrap up talking a little bit about Parsing JSON in the command line. Hey, Scott, we're recording Episode 180 today.

- [Scott] 180, amazing.

- [Ben] It really is. And you know what other mark we crossed recently, 'cause we don't talk about this much. 400,000 downloads over the lifetime of the podcast.

- [Scott] Nice, I gotta go update the About page.

- [Ben] You should. We should actually make some updates around that. So we have, it has been going on three years now. Because we record extra at events. I can't remember when we started this, but it was about this time three years ago. Which means I have talked to you every week for three years.

- [Scott] And you're better for it. I keep telling you that, but you don't believe me.

- [Ben] I would not be the same man I am today had it not been for you, Scott.

- [Scott] Boy, let's see if I can teach you some stuff today. Maybe you too could teach me some stuff.

- [Ben] If we talk about AKS, like you have suggested you would like to because you have been living in the world of AKS. Not to be confused with ACKS when you're talking about networking. Bad joke.

- [Scott] No, there's a TCP joke in there somewhere.

- [Ben] Yes, you will probably be teaching me more because AKS is not something that I have done much with or have much knowledge in. So would you like to talk AKS today, Scott?

- [Scott] Yeah, we can talk about AKS. I have a bit of a personal question first though. So my wife and I have been going back and forth on this one. Do you ever have dreams about work? Like, do you get so involved in your work that you go to bed at night and you dream about it?

- [Ben] Not a lot. I think I've done it once or twice, but it is not a regular occurrence. However, based on your Facebook status, I thought that was a joke. So is your Facebook status actually happened to you the other night? It was at twitter.

- [Scott] It's been happening to me on the regular lately. So I am either far too close to my work, or my brain is just warping in a very strange way. So last night, I was watching TV and fell asleep on the couch. And I'd been spending a little bit of time in the morning running through and playing within AKS deployment. And one of the things you end up doing in AKS quite a bit after it's deployed or heck even, while it's deploying, especially when you're working with Azure CLI or things like that is you're gonna be playing a lot with JSON objects. When you're in bash, you want to find better ways to play with JSON. Because you don't wanna be potentially just manipulating raw strings or not having things like access to logical operators for doing selects of given nodes in a JSON document or anything like that, right? You just need a way to like prettify it and make it usable to you, so that you can query within it and kind of figure things out. So one of the tools that I found that I like to do that is jq. So that's a lightweight and flexible command line JSON processor. Think like I said for JSON data. And, you're on the right path for what it does. Certainly, there's a lot of tooling built into other things like Azure CLI supports queries with JMESPath. If you're looking like I was doing a bunch of stuff with, like you said, AKS over the last couple weeks. One of the things you end up doing there is using kube-control or kube-cuddle, whatever you want to call it, quite a bit to, you're always querying for services, deployments, nodes, pods, things like that. And there's an operator where you can either get your output back from kube-control as JSON, or you can use what they've built in for JSONPaths, which uses a query language that's like what's in JMESPath, but not all the way there, which is like what's in jq, but not all the way there. So my brains been really worked around these things. So I spent, I don't know, maybe like 30-45 minutes in the morning, trying to figure out this one jq query or I would have been happy if it was a JSONPath query or anything else that I could have gotten it into. When you're in Kubernetes, and you're doing something like, say you do kube-control, get nodes or describe nodes, whatever you wanna do to get your data back. Some providers like Microsoft, when they do things like label a node or label a pod with their annotations, they'll go ahead and label them in a way that some JSON parsers, particularly like jq don't like. They will have characters in them that just can't be parsed from a bash command line without a bunch of escaping, or other things otherwise. So something like a forward slash, or an equal sign, just weird things through bash off in general, and then you end up in string processing land and all these kinds of things.

- [Ben] So what you're saying is, Microsoft did not follow conventions around something that has been a standard. Not to bash on Microsoft.

- [Scott] So jq is totally an external tool.

- [Ben] So they don't follow certain standards. Microsoft's not just ignoring standards here.

- [Scott] I can't say that I love that, a million of their annotations have things like forward slashes in them which some parsers treat as a new node and not part of just a key value pair, things like that.

- [Ben] Got it. So is there a standard around how this should all be laid out when you go look at these parsers and stuff, not to go down a rabbit hole?

- [Scott] Well, I mean, it's just JSON. So they should be able to parse JSON. But now then you get into, I'm trying to parse from bash, or just a Linux shell. So whether it was zsh, ksh, or bash, whatever it happened to be, they all have their own rules for string processing from the command line, and then you're trying to figure out the right way to pass strings into things to get it all wired up and where it needs to be. Anyway, I was doing some stuff trying to get down to, I was just trying to write a simple query that retrieved nodes in a node pool. And it gave me the node pool name 'cause it's a cluster with a bunch of different node pools, and then I needed an annotation off of there for the failure domains. So when you create multiple node pools within AKS, you can also take those node pools and push them out to availability zones if you want to. So you can have each node in an AZ. So really the way to figure out where your node sits, is doing something like querying for the failure domain. So it's failure-domain.beta.kubernetes.io/zone, and then that'll give you the name of the zone you're in. And then you can pull out the node name or metadata.labels.agentpool gives you the node name and things like that. So you end up doing all this weird stuff. All I wanted to do was, I wanted. Well, I was first trying to just do a select statement. So with most selects, and you go look at jq's documentation, you're like, all right, select equals this. That makes sense, right? As a nice operator to do equality tax. It also does greater than and less than, but I didn't want to do any of those things, I wanted to do basically a like, or a match, like you might be used to inside of PowerShell. Or you do like, Where clause with a wildcard or something like that.

- [Ben] So, essentially, like in CONTAINS.

- [Scott] Yeah, exactly. But it took 45 minutes for CONTAINS to click in my head. So I was diving through the jq documentation, which isn't like the greatest thing, hey, it's okay. but I'm sitting there like, "All right, so there's not a practical example "for the one thing I wanna do." And I should have just rung you up, 'cause you would have said, "Hey, you're trying to do a CONTAINS," and then I would have just done a find on the operator page, and the function page and said, "Look, there's a CONTAINS function." That will help me a whole bunch and bring it back. So I've been living like this weird nightmare of JSON and CLI and all sorts of fun things like that.

- [Ben] So to bring that full circle that you actually dream about that then last night while you were sleeping on the couch?

- [Scott] Yeah, no, totally had a dream about browsing through the jq documentation. And looking for select and then it was just awesome.

- [Ben] We need to find you some hobbies or we need to get you doing more work so that you're way more tired when you go to bed and you don't dream about it.

- [Scott] Yeah, no, it's, I agree something to do there.

- [Ben] As IT professionals in the Cloud era, sometimes that feels like we don't speak the same language as the rest of the organization. So when stakeholders from finance or other departments start asking about a specific project or teams Azure costs, they don't always realize how much work is involved in obtaining that information, sifting through cluttered CSVs and a complex mess of metadata in order to manually create custom views and reports. It's a real headache. On top of helping you understand and reduce your organization's overall Azure spend, ShareGate overcast, lets you group resources into meaningful cost hubs and map them to real world business scenarios. This way, you can track costs in the way that makes most sense with your corporate structure. Whether it's by-product, business unit, team, or otherwise, it's a flexible, intuitive, and business-friendly way of tracking Azure infrastructure costs. And it's only available in ShareGates overcast. Find out more on ShareGate.comm/ITPro.

- [Scott] So speaking of command line processing, another one that I've run into recently, particularly in the context of working with AKS, and I'm sure I would have run into this someplace else, but it just happened to be that I was playing around with AKS. And one of the things you do when you're spinning up clusters, is clusters need the ability to talk to other resources, and the cluster resource needs the ability to manage its underlying compute. So when you spin up an AKS cluster in Azure, you get one resource, you get the AKS cluster resource, which is what you're actually managing as an Azure resource. And then over in this other resource group that pops up you'll end up with this MC_ resource group and it has a name that shares the same name as your cluster after the underscore. And in that resource group, that MC_ resource group is where your cluster master lives, it's where your API server is, where all your nodes and everything else are. So if I went and did an easy, AKS scale command from the Azure CLI, I'm telling the cluster resource to go talk to the resources in that MC_ resource group and add a new server and spin it up. So that means that the cluster resource itself needs permissions to go and talk and spin up a new VM on my behalf and get it to where it needs to be.

- [Ben] All of this other resource group?

- [Scott] Yep.

- [Ben] Is there a reason it doesn't another resource group or are you getting to that?

- [Scott] No, I mean, it's just the way the service is built out. It was a way to abstract and really have a way to point out that the cluster resource is different than the underlying compute that sits behind it.

- [Ben] Got it. I didn't know 'cause it seemed like having an all-in-one resource group would make this easier. But I didn't know if there was a fundamental reason for that other than just separation.

- [Scott] Well, I mean, there's a lot of stuff there. Because an AKS cluster is load balancers, it's virtual machines and disks, it's VM scale sets, it's public IPs. So it's just a lot of stuff that could be there and couldn't be there based on what you're doing with your cluster. So it is kind of easier to have it managed by the cluster resource, so you don't need to worry about it. Because what happens is, if I delete the cluster resource, it goes and deletes that MC_ resource group for me. So it's not like it's leaving resources around.

- [Ben] So it's a way to separate all those resources out, not muddy up that main resource group, or you may be creating all of this stuff.

- [Scott] Yep, and it's all just built in, and available, and ready to go. So anyway, when you create a cluster, you've got a couple options around cluster identity. So how is the cluster resource going to manage all those other resources? You can go with managed identity or you can go with a service principal. And in some cases, you have to go with a service principal. Like if you're gonna do Azure AD integration, for integrating your cluster roles and your role bindings within your cluster, you're gonna have to use service principals that can talk to applications in Azure AD, like the graph and things like that on your behalf. So you end up spending a lot of time with service principals. And if you go ahead and you just create a raw cluster, and you just do like az, aks create, Microsoft tries to create a service principal by default for you. There's actually a bunch of problems with that. One of the biggest is, at least for me this last week, 50% of the time, cluster creation fails, because it takes four minutes. Up to four minutes for a service principal to propagate through AD 'cause your Azure Active Directory is more geographical than regional. So it takes time for, you create a new user, in this case, a service principal. It takes time for that to replicate through, and you can't use it in the cluster until it's replicated all the way through and where it needs to be. If you're just doing az aks create, and you're getting frustrated that your cluster has failed to create, because they say "AD roll propagation failed", you can either thank the AD team or you can thank the AKS team for not fixing this problem. The AKS team blames the AD team and the AD team someday get the better timing and for AD propagation. But Microsoft fixed it in the portal but nobody creates a cluster through the portal. We all do it through the command line or through the rest API's or ARM templates or something like that. So that basically leads down the path you have to create a service principal. So typically, the way you create a service principal is you just go and do, from the CLI it would be easy az ad sp create and then you would either pass in some scopes or in my case, I just need the service principal. So, --skip-assignment and create a custom service principal. When you create a service principal, you're calling a REST API it through the Azure CLI, and it's gonna return some information to you. So it returns an application ID. It returns a client secret or a password to access that application. And that's your one time and your one chance to go ahead and retrieve that password. So I'm trying to create a script that can spin up clusters very quickly for me as part of a lab. And it's got to do like hundreds of these things. And it's got to be fairly adept in and it's got to work every time you run the script. So I sit in there. So typically, the way I deal with this stuff is I'll start with my bash script locally, and I'll write it all there and I'll be running it against just my subscription. And then usually I'll try and take it up to something like Cloud Shell and figure out like, "Hey, did I miss a check for a piece of software "I need to install or something like that?" And then I'll transition into CSE and get it to where it needs to be. So writing locally, if I do az ad sp create, and I create that service principal, I get one type of password. I get a password that's effectively a GUID. It's formatted just like a GUID. So it's nice, it's easy to parse, it's a string, it has no weird characters, anything in it like that. You go up to Cloud Shell and you run the script. Okay, az ad sp create, you get a GUID back for the password. When I took it and put it in a CSE, and I'm not doing anything different. So custom script extension. So I'm not doing anything different other than calling the same script, probably the only variability there is, I'm installing whatever the latest version of the Azure CLI is when it comes down. But 100% of the time when I run this thing in the CSE, the password is not a GUID, it is a cryptographically-secure password, which is awesome, except it uses every kind of character that you can't pass through the az ad or az aks create command, 'cause it doesn't know how to parse or do URL encoding or anything like that. So things like backticks, single quotes, sometimes you get a double quote in there. So all of those things can't be in the service principal, passwords string, or else now your cluster creation fails because you tried to pass in a bad password to it, which again, is awesome. And of course, I didn't find this out until I'd already been down the path of, "I need the script to go ahead and spin things up." So I ended up having to just do like a big case statement or if statement. Basically what I do is, I create the service principal in a loop and I look at the password. And if the password has any of these characters in it, and certainly better ways to do it, but I was in a time crunch or I could have done like a regex or something like that. Basically, if it has any of these characters in it, just go and reset the password and keep resetting it until you get something that I'm gonna be able to pass through the command that's clean.

- [Ben] Outlook add-ins are a great way to improve productivity and save time in the workplace. And Sperry Software has all the add-ins you'll ever need. The Save as PDF add-in is a best seller and is great for project backups, legal discovery, and more. This add-in saves the email and attachments as PDF files. It's easy to download, easy to install, and Sperry Software's unparalleled customer service is always ready to help. Download a free trial at SperrySoftware.com. S-P-E-R-R-Y-S-O-F-T-W-A-R-E.com and see for yourself how great Save as PDF is. Listeners can get 20% off their order today by entering the code CloudIT. That's CloudIT, C-L-O-U-D-I-T all one word at checkout. Sperry Software, work in email, not on email. I want analytics on this on what the average number of passwords you have to create is before it works.

- [Scott] So from what I've seen, it's like two or three.

- [Ben] That's not bad.

- [Scott] But I couldn't find an open issue or any documentation or anything around it. It's just super annoying that it goes down that path and works in that particular manner. The only way I found it out, like I said, when you do the same command in Cloud Shell, which is running the latest version of the CLI, and you do the same command on my local which is running just to patch releases back. And you do the same command running in the CSE, they all produce different output, and that's super annoying. 'Cause you're calling the same exact REST API. The more annoying thing is, if you just create an AKS cluster, you go down that very first path which I said works like almost some of the time but barely. And you just do az aks create. In that case, the client secret it creates aren't GUIDs, they're just really long strings of numbers. It's ridiculous.

- [Ben] And you can't find anything on why you get all these different variations.

- [Scott] I have not gone digging far enough into the CLI or things like that. I've resigned myself, like I said, just to a big loop.

- [Ben] That doesn't make any sense. Does the REST API actually return the password? Or is it like the different versions of the CLI submitting a password that it's randomly generating?

- [Scott] No, it's the REST API, but the CLI must be calling that REST API in a different way.

- [Ben] Different manners. That's bizarre, because you would, well, I guess, the REST API, so it could have different versions too. I'm thinking that like different versions on a REST API when you call it so they may have updated it. They're calling different versions of that API.

- [Scott] Yes, yes, you you can do weird things like that. So it's just fun times and effectively bash land also, I can definitely create a ton of AKS environments.

- [Ben] Wow, no wonder you were having nightmares last night.

- [Scott] Well, it's the life I lead. And it's all to say that it like it's all just string processing, which is the really annoying thing. Like the jq thing, totally string processing, even this thing with a password, I would have thought. All right, so you're giving me a set of raw characters that now the AKS API is saying it can understand when I'm going to create the cluster. So why don't I just URL or URI encode that secret for you? But then the stupid az aks create command doesn't understand 'cause it's just taking in a raw string. So you end up having to go down this weird path and fight all the tools up and down the stack to get it to where it needs to be.

- [Ben] Otherwise you know, we should do an episode on, I don't know that we've done it. Because I've experienced this a little bit is CLI versus PowerShell. Because I've played with the CLI too. I've done some stuff with the Azure CLI with Office 365 CLI. End of the day, I've been writing PowerShell for so long that I tend to try to revert to PowerShell, because I spend more time fighting with stuff like that, like the whole string processing and all of that in the CLI, than if I just write it in PowerShell where I can have my objects and I can do all my dot paths and do everything else in PowerShell.

- [Scott] We should definitely do it.

- [Ben] CLI versus PowerShell. I think we should have another voice in there too. I fall in the camp, and I'll just let you know where I sit on this one. The Azure team that owns Azure PowerShell completely bastardized Azure PowerShell. They broke a bunch of PowerShell conventions. You mentioned like dot-sourcing and returning objects. Three quarters of the time, you're not getting an object back anyway, you're just getting an object that has a big raw string of JSON in it that you've got to process and do whatever you're gonna do with. And the other thing is, honestly, the CLI is just way more straightforward. Yes, you have to do more with bash or ksh, or csh. You have to know about the shell you're working in. But once you've got the rules for the shell you're working in, it is far, far easier to work with the CLI. And the CLI does things I'm finding more and more, like it used to be that a new piece of functionality would come out and a REST API for Azure. And it would, maybe show up in PowerShell first and then it would be in CLI later. And sometimes it'd be in the CLI first and sometimes it'd be in PowerShell later. Sometimes it's in the portal first, things like that. I'm finding more, and more, and more, that stuff is in the CLI way before it's in PowerShell. And in some cases, it's never in PowerShell, at all. It just doesn't exist. What of the ones we're going through yesterday? You create a new enterprise app in Azure AD. So I create a new app, and I want that app to be able to talk to the graph on behalf of the user that's calling "my app". So to do that, I need to have an admin consent. I want it to be able to talk on behalf of the user, so that's an admin consent within my tenant. So there's a couple ways I can go do that admin consent. I can configure that admin consent through the portal, or I can go and sign into the app as an admin with a GA, or Global Admin or application admin permissions. And I can accept that consent prompt, or I can use the REST API. And I can fire off consent in an automated manner. The portal's annoying, because what if you're creating your app on the fly, and you wanna provide consent on the fly, that's no good. Like in the case of creating an AKS cluster that talks to Azure AD, you actually end up creating two apps. You create one for a server component, which can talk to the graph. And then you can create one that's a client component, which is associated with the cluster. And then the client component is granted a consent to talk to the server component. So you end up with this machination where you wanna do it automatically. Well, because AKS is created so much to CLI, the team that owns AKS or the CLI, or whatever that portion of that module is, there's actually a command in the CLI, where you can grant admin consent from the command line. Yeah, and it's all there, it's just built in, you don't need to mess with the REST API's, anything like that. So a lot of the automation engine that we have at the company that I work for, is built on the back of PowerShell. Because we kind of fall into that same camp, I think where you do. PowerShell is the one way forward. It's the one true path blah, blah, blah. But more and more, so this question came up yesterday and I told somebody on the team, I said, "Well, it's just built into the CLI, "here you go, go run this command." Well, we can't run this command because this particular piece of automation is actually running inside of Azure Automation, and there's no support for the CLI there. It's like, "Okay, well, then we gotta go down "the REST API path or something like that." And the question was, like, "Really? "Why is it in the CLI? "It's in the REST API, it's in the portal, "but it's not in PowerShell." And it's like, it's not in PowerShell 'cause it's not important for it to be in PowerShell. Obviously, but by the people that own those products and wrote that one.

- [Ben] So maybe I need to go play with some of these other JSON parsers and all that. 'Cause so the thing I was doing the other day, I was working with discs on a VM and trying to figure out if a disk and a VM was a managed disk or not managed disk. So we could just blow through a list of 130 servers, figure out which ones still had unmanaged disks. And I was having a heck of a time being able to dig through the JSON that got returned about the VM, to figure out if the disk in there was managed or not, 'cause there's some properties you can pull. And I probably spent like, two or three hours trying to figure out this try to pull it out. And finally, I was like, "This is stupid. "I'm gonna go write PowerShell." And I was able to do it in PowerShell in like 10 or 15 minutes. Arguably, it's probably because I'm much more familiar with PowerShell and I wasn't looking at things like jq to try to figure out and dig through that.

- [Scott] So the thing in there is your comfort level with everything. So if you're just doing the CLI, then certainly invest some time in JMESPath. But in my case, I was doing both Kubernetes so kube-control, plus Azure CLI combined, plus a couple of other things. And when you're doing that, you need another engine, like jq that can just help you with generic parsing across all of those. So I think it's worth learning both jq and JMESPath.

- [Ben] JMESPath, okay.

- [Scott] But to that external voice thing, I bet we could invite the person that you were working on that manage disk problem with 'cause I know he's a big fan of the CLI.

- [Ben] Yes, he is very much a big a fan of the CLI, and he is who I was. He gave me a bunch of the CLI stuff, but we were both having problems figuring out that manage disk.

- [Scott] You know what, we should have just hopped on a call and we could have banged it all together like, you could help me with my stupid select problem, and I could have helped you with yours.

- [Ben] We should. We should ask this set individual to come out and talk about managing Azure with the CLI.

- [Scott] Absolutely. All right look at that. We made it all the way through. We didn't even talk about anything other than bash the whole time.

- [Ben] We didn't. All right, well, now we have plenty of topics for next time. 'Cause we have all kinds of fun stuff we can talk about. So for today though, we will wrap it up and let you get back to writing Azure CLI, and I'm gonna go figure out what I'm gonna do today for work.

- [Scott] All ready, you got it.

- [Ben] All right, enjoy your day. We'll talk to you again next week.

- [Scott] All right, thanks Ben.

- [Ben] If you enjoyed the podcast, go leave us a five star rating on iTunes. It helps to get the word out so more IT pros can learn about Office 365 and Azure. If you have any questions you want us to address on the show or feedback about the show, feel free to reach out via our website, Twitter or Facebook. Thanks again for listening and have a great day.