UCL for FreeBSD
CC Attribution - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Jude, Allan
Most system administrators no longer edit the majority of configuration files by hand, they use automation and configuration management tools like puppet, saltstack, ansible, and the like. Many utilities and daemons in the FreeBSD base system use their own custom configuration file format. While these various different formats are usually accompanied by man pages, they do not lend themselves to automation or programmatic editing. Space and tab delimited files make it harder to extract a specific value, and difficult to edit that value in place, whereas nested key-value pairs are easier to read, and are easily addressed using libUCLs dotted notation. To solve this, I propose teaching the various utilities and daemons in the FreeBSD base system to speak UCL  the Universal Config Language, as implemented by libucl. In addition, I propose adding two small tools to the base system to make the administration of such config files easier for humans and automated scripts.
 Can you hear me? Yes? Good. All right. So I'm going to be talking about UCL, the Universal Config Language, and my goal of converting everything in the previous DET using it. So for anybody that doesn't know who I am, my name is Alan Jude and I've been a previous D-server administrator for about 13 years now. I'm a previous D-doc committer, mostly with focusing on documenting ZFS and Beehive, UCL, Libzo, mostly new stuff. I'm also the co-author of FreeBSD Mastery ZFS, which is a book that just came out, that walks through how to use ZFS, how to use snapshots, how to lay out your file system, etc. For my day job, I built and run the Scale Engine CDN. We do mostly video streaming and HTTP content delivery, although not as big as LimeLite, but anyway. I host two podcasts every week. On Wednesdays, there's BSD Now, where we talk about the latest things that are happening in BSD, new development, and so on and have an interview with the random developer. We've had 90-something of those so far, so we have quite a collection of really good interviews with various people from the FreeBSD and wider BSD community. Then on Thursdays, I do TechSnap, which is a system-in and security podcast that's operating system agnostic. For work, I manage about 100 servers by myself using Puppet, which is unfortunate for me. We have about 35 data centers in 12 different countries that we have a small number of servers sprinkled into. We mostly use ZFS for storing our giant video collections, and ZFS replication for moving it between places. We host the package mirrors for PCBSD, so that's about 500 gigs of stuff that's being used all the time. If you're familiar with how package works, replication on that is very sensitive. If all the files are not perfectly in sync, things happen that you'd rather didn't happen. So why have I randomly decided to start looking at using LibUCL for everything back at EuroBSDCon last year in Sofia? Jordan Hubbard gave the keynote about how we have to adapt FreeBSD for the future. I didn't agree with everything he said. His focus seemed to be more towards little embedded things instead of running entire data centers with FreeBSD. But the recommendation that really hit home with me was that we should get away from having one config file that's completely different for every utility. Because BSD operating system basically came together in pieces over time, and what was the best way to make a config file change over time, every utility has its own config language that is entirely different. Some of them are especially offensive to me like NUSIS log and some other ones like that. So when Jordan was talking about it, he was raving about XML P-Lists that they use at Apple. I really wanted the one true config format not to be XML. So I've looked at some bullet points on my slides. Why it was a good idea to have a universal config file format. We want to be able to have readable and structured config files. The idea of having an include statement in your config file to pull in a snippet from somewhere else, or all the snippets in a subdirectory, is something that I've always really liked. We use it in all of our web servers to separate different stuff. But every utility seems to do that differently, and Brian Drewry grafted includes onto NUSIS log. But what if the config parser could just handle that, and we would have the same system in every utility without actually having to write that code and put it in every utility. Also the idea of overlays. For example, previously ships with a lot of default configuration. For example, NUSIS log rotates all the log files that exist by default. Well, if I want to change that, I have to modify that file and then I have to merge it every time I upgrade. It's just different than how I would like to do it. Also, it's good for when you're auditing your configs to be able to say, well, this is all the defaults here, and then this is what I've changed away from the defaults. So the idea of overlays, basically you have your default configuration, and then on top of that I've layered on our site-wide configuration. That applies to everything. Then on top of that, I've layered on the configuration that's specific to this one machine. So basically being able to set a configuration multiple times and eventually the last one wins. That I thought was something I would like to have. So I found out about UCL, which is written by another free BSD developer, and is already included in the base system as a private library because it's used by PKG. But as a system in, it really speaks to me because it's based on the bind or engineX style config file, right? Key equals value, optionally with the terminating semicolon, and it can do sections like combined and engineX. So it's what every system in is already probably used to doing. Or is the config file they would prefer that everything used. But it can also read JSON, which is something people like to use, although if you've ever tried to write by hand, it's very finicky. But UCL gets away from that by actually making most of those finicky things optional. It's got a very good state machine in it. So if you make an array in JSON, you can't include a comma after the last item, and that's how it knows the array's over. Well, UCL knows the array's over when you close the array. So you can include the trail and comma, which makes a big difference in your config files because when you diff them, your diff is the item you added to the array, not the previous line getting a comma and the next item you added. The other thing is it has syntax sugar stuff. For example, you're specifying the maximum size of a log file in new syslog, well, you can just put 100k. Currently, the new syslog config file assumes you want to rotate your logs in kilobytes. Well, maybe you want to locate them in megabytes or something, but the configuration doesn't tell you what the units are. You have to go and read that in the man page. So being able to just put 100 kilobytes or whatever is handy, but also UCL does time. I want to do this every five minutes or every seven days. In the internal representation, that's just turned into a number of seconds, and that works really well with programming and it handles all the parsing for you. So this is what UCL can look like. I've purposely used a bunch of the different styles all put together because it will take all of them and just be happy. So you can have key equals value, the coding is optional and the terminating semicolon is optional. When you do sections, you can make them look like JavaScript objects with the colon, or you can make them look like n-gen X or bind sections without the colon, or you can actually put an equal sign if you really want to. Then the same, you can define a key in JSON style, or you can do it with an equal sign or whatever. The other thing is it has syntax sugar for booleans. Yes, no, on, off, or true, false, or any mix of them defines the boolean. Or if you want the literal text, the word true then you just quote it. So I've briefly gone over the things that UCL has an advantage over using something like JSON. But the biggest one is that you can put comments in your config file which you cannot do with UCL. So with JSON. UCL? Yes. Yes, you can and you can include, it does globbling, it can do trimode where it's not a failure. If the file you're including doesn't actually exist, it can also do includes over HGVS, but I'm not sure you want to do that. Can it also do conditional stuff? Or you set a few things and then includes in the files that say, well, you use the ones that's globb, otherwise. Not quite. During the Dev Summit, the list of all explained why an if macro doesn't quite work yet. But it does support variables in your config files, but those have to be defined by the program that's going to parse it so we can set a bunch of variables that you can then use inside the config file. But one thing we're looking to add to it is the ability to reference values you set previously. Specifically, in order to convert Jail.conf, it's very handy once you set your Jail's hostname to be able to use the hostname in the path name and things like that. So it's coming along for that, although currently there's not much in the way of conditional stuff. But the main reasons to use UCL over something else is that it's very flexible. Because you can do sections and arrays or just key value pairs or whatever you need to do, you can express the configuration for pretty much any application I can think of using it. Your config file can look as simple as rc.conf, whatever equals whatever, or you can do engineXStyle with nesting and whatever you need to end up doing. It has an includes feature which also supports priorities. So you can say, here's some config and now include this config with a higher priority. So if there's conflicting keys, they overwrite or include it with a lower priority because it's just the defaults. So if there's conflict, the original one still wins rather than what you're including. It also has a mode where if you include or just specify the same value twice with the same priority, it can actually create an array out of them if that's what you want it to do. It can be very terse, if it just key equals value, there's no excess formatting that you would get with XML. So it makes it much easier to read. Yes, while it's easy to read, unlike JSON, it's also easy to write and it allows comments. So for example, CronTab has the format that we've all used, although trying to teach it to classrooms full of people as I've done, you quickly learned that they don't quite understand that to get what they want, they don't put, if you want to run something once an hour, you don't put star one, which will run at every minute only at one o'clock. You have to pick a minute to run it out and then run it every hour or something. Anyway, if you convert CronTab config into something like this for UCL, it ends up actually looking like how you specify a CronTab in Puppet or Salt or any orchestration system, where you specify the keys by their name, but it also allows you to do things like specify the environment more cleanly. In the regular CronTab file, you specify mail to something or path is this, and then you specify some CronTabs, and maybe you override path again, but you can't unset an environment variable very easily. Where as this, you could isolate it more easily. But that's just an example of what a CronTab would look like if it was converted. So, new syslog especially is this one. At the top there, you can see what the original config looks like. The new syslog config file is especially offensive because beyond the end of what you see there, there are optional fields, but because they're optional, they're all overloaded under that one field. So, if the last field starts with a slash, then it's the path to a PID file unless you want to operate on. Or if it doesn't start with a slash, then it is the command you want to invoke in order to reload the service. Then if it's a special character or something, then or if it's a number, then it's the signal that you want to send. It's that one field has different meanings and different contexts and that's not very good. But the biggest problem is you end up with a bunch of these lines in the file, and it's not really possible that it's not very easy to programmatically parse what's actually happening there or to programmatically change it. We're using puppets, so when we want to enable something in RC.com, we use the sysrc command line. We say sysrc, this variable, we get the value, compare that to our manifest. If it's right, if not, we can use sysrc and say programmatically change this without risking working my config file. So we want something like that and I'll get to more on that later. But this is what you would end up looking like if you converted new syslog, which I actually have a patch for on fabricator. You have the file name, the mode, the number of logs you want to keep and so on. But instead of JC being random flags that you have to look up the man page, you just specify different booleans for each thing you want. We can use the include support here. I should have a dot in front of it. You can specify things like try so that it'll not fail if there are no config files in that directory and so on. There's still a question as what to do about the time field in the new syslog. It is a standard way to specify time, but it's not a very intuitive one. So I'd like to expand the concept beyond just replacing a bunch of the config files and having a standard way to deal with the common problems, specifically moving default config files. Right now we ship ETC, Krontab, ETC, new syslog.com, and a bunch of config files in ETC that people modify. It'd be better to move the defaults away. Based on discussion in the working group, some of those should actually probably be moved to user share rather than ETC defaults. But either way, the default should live someplace, and then the file you put in ECC should only be the changes you've locally made away from the defaults. Can you briefly summarize why user share instead of ETC default? I forget. I forget why. It was someone else's recommendation. Yeah, it makes no sense. Oh, okay. It was a Linux guy's recommendation. So yeah. Sorry. What is the question? The question was, I had said that at the Dev Summit working group we had earlier in the week, they had suggested that the defaults not be installed into ETC defaults, but move to user share, like where our example config files go, and the rest of the defaults just be implied, I suppose. And do you remember why? The rationale was because, I don't know if it's an LSB issue or something, but the rationale was that many users of Puppet and Nets or whatever expect that slash EDC is only for files that you put there, and it's only for local configuration that override something else and that anything else you would want. Anything that shifts with the system as a default should go to the user share. That's something that apparently a lot of Linux things have been moving to. I don't know. I like EDC defaults. I think it makes sense. That's the in two places. Yeah. Either be a user share or a default, or it should be an FTP default. Some shouldn't be here and some shouldn't be there. Yeah, exactly. So the proposal is to say, we're going to make it clear that the stuff that you do goes here and the defaults that we should go somewhere else and how you spell the place where the OS puts its defaults. I don't think we really care. We can stick to EDC defaults and Linux for user share. I think sticking to EDC defaults probably makes more sense because some people still partitioned user separately and you want your config files to exist in single user mode. You kind of do. Yeah. So the overwrite feature basically including one file and then including a second file with a higher priority allows you to overwrite the existing config. So for example, new syslog, I like to keep more copies of varlog messages than the default, which is five. I want eight, so I can always go back a week. If I modify the config file for that, then every time I upgrade, I have to merge it and it gets annoying. Whereas with UCL, I would just in my etcnosyslog.com, I would just say messages, curly brace, count equals eight. And I'm only changing that one setting and everything else is inherited from the defaults. So are there ways to modify the inheritance rules? And what if you wanted to completely re-specify the varlog message? Right. That came up in the working group and so we're probably going to add a macro.delete and then a key and it will destroy any previous thing that was there so that you'll be able to re-specify everything. So if by default everything is merged? Yes. Not over-registered? Yes. Yeah. There's some controls for it, but it's... Controlling the recursion of the merges is not really possible at the moment. But I've done quite a bit of work on improving that and giving a little bit more control with it. So, the other one was we ship a bunch of crontabs in ETC crontab. What if I wanted to disable one of those? Currently I'd have to go in there and delete the line or comment it out or something. Whereas if we did something more like we do with package and default package repo, I would just in my crontab file that wherever that ends up living, I could just select a certain crontab and set enable to no and that crontab won't execute anymore. So, yeah, this one deals with the presidencies you just mentioned. So if we have ETC package, frebsd.conf and it's enabled, but then in use a local ETC repose, I've set it enable no, and then in yet a higher priority place somewhere, I've set it back to yes. How do we decide who wins and how does the administrator see who wins? So, we can use the priority key to decide, because the order might just be alphabetical, you might of the includes, you can use the priority number, which currently supports values between 0 and 15, to indicate how high of a priority your changes to the config file are. So you can decide how it's going to end up getting merged. And separately, I've talked about later, a tool that allows you to compile this config and see what the resulting config will look like, like you get with post comp from postfix or test perm from SAMBA. And so, when I originally wrote these slides for AsiaBSDcon, the idea of priorities and includes didn't exist yet, but since then, Vlustavad has actually implemented that and I've actually separately recently added a feature where you can do it for a small block of config rather than, with the original implementation, you only change the priority of each file as you included it, but you can actually change the priority inline in the config file for even just certain sections of the config file now. Interesting one that I haven't run into yet, but I foresee as possible being a problem is the ordering of keys. Most of the apps I've looked at so far, what order you specify configuration for stuff in doesn't really matter, but at some point that there might be something that did. Obviously, firewall rules or something, the order is actually really important and deciding how to preserve that, but I haven't written anything for that yet because I haven't encountered a tool I want to convert yet that requires very specific ordering of the config stuff. Everything we've looked at so far was just priority stuff to decide what the resulting config will look like, where the actual order of the statements in the config will actually change how it's configured because I imagine in most places that's actually a bad design other than things like a firewall where you're being very specific. That's like there's a slide coming up on that because mostly it's done. It's almost finished. So talking about actually doing it, I got the idea originally at EuroBSDCon after hearing the talk by Jordan Hubbard. Month and some later, I was at MeetBSD and I had done a short lightning talk about this idea of, hey, somebody should do this. Then I was stuck in an airport waiting for an airplane and I was like, how hard can it be? So I picked Newsyslog because it's a relatively small piece of code and it's not something that if I broke it, I would get murdered or anything. So I started playing with it and basically it was the first thing in C I had ever tried to write. I read little bits of C to try to understand what was happening. This is the first time I was adding more than an if statement into a program. So I started doing it and now there's a patch on Fabricator for actually converting Newsyslog to use UCL and it's set to be backwards compatible so it will actually read the old config file and only switch to parsing with the new parser if you have a comment at the top of the file that indicates it's the new config file format. And I wrote most of that on an airplane back home from California and then it was a bit rough but over the course of a couple of spare weekends I managed to actually get a working prototype and the best part is it doesn't change any of the existing code, it just added some new stuff and a little hijack to the config parser that says if the first line is this, run the new parser, if it's not, rewind the pointer to the beginning of the file and parse it traditionally. And so that brings up compatibility. You know, if we're going to change the config file format of all these utilities in time for FreeBSD 11 or something, people that upgrade from 10 to 11 are going to be like, you want me to rewrite all my config files? I was like, going to work. So if we use a similar strategy as we did with Newsyslog there of, you know, a sentinel at the top of the file or some way to indicate this is a UCL config file, then we can do that or if it doesn't have that then we fall back to the traditional parsing so people upgrading from 10 to 11 can just keep their existing config files but anybody doing a fresh install of 11 can start with the new format and we can deprecate the old ones in 12 or 13 or something. And the other thing I did as part of the sentinel was include a version number in the config file format because I'm quite sure that the first revision won't be perfect. And so this would allow us to do something like in the Gehli code, they have a little version number and, you know, based on that version number, it picks a different parsing function to read the configuration. Hopefully we don't add too much bloat doing that but. And so this just describes how I changed the Newsyslog startup routine. I already described that to you. So basically what it does is you run the libucl parser on the config file, it handles all the includes and layering and stuff and gives you an object that represents the completed configuration. Then we just use its iterator function to walk over the config file, extract the root keys which there should be one for each config file or a log file that you want to rotate. Then we process the subkeys, you know, oh this log file says compress equals xz or compress equals yes and use the default compressor and so on. And then it executes, it loads that up into the struct that Newsyslog's original config parser used and then it just runs the program as normal. The one feature that it's missing at the moment is a way to specify the defaults. Specifically it'd be really handy to disable compression on all my log files if my log files are on ZFS where it's going to get compressed that way instead. Because in a normal case you have your log file it's being written to ZFS constantly getting compressed and then I'm going to rotate it, read it all and compress it again with BZIP or something. That's basically a waste of disk cycles at that point. So the remaining items to sort out are possibly designing a consistent set of keys to use across config files like the enabled that we talked about for crontab and package but in the working group it seemed that those would be fairly specific to the different use cases and maybe we don't actually need a set of them. But having something like Zolint where we could kind of build up a database of the common names of keys and try to be consistent across our config files probably makes a lot of sense. But we've decided against actually having a set of specific keys that are reserved for free BSD utilities or trying to namespace those so they might not conflict with what an application does specifically. The other problem was developing a universal API for actually reading the lib UCL stuff in to C and I think we actually settled on basically writing a config file schema and a program that will actually take that and generate the C code for you since there's not really a great way to load it directly into the structs or whatever. And so, yeah, the decision there was it probably best to just auto generate the SturComp ladder you'd end up with to load your configuration. And so the other thing is I've written this tool UCLCMD which is a command line interface to allow you to read and set and merge UCL config files so that for example I can instruct puppet to go change the number of log files that we keep. And the other thing that it needs to learn, it mostly does it now, would be the compiling the resulting configuration but based on talk at the Dev Summit it would also be nice if it could exclude the defaults from that so you can see what is different between the resulting configuration and what the original default configuration was. So kind of a version of diff built into it as well. And yes, we had a Dev Summit working group for about three hours earlier this week and quite a bit of interesting stuff resulted from that. Then we looked at other things that we would use UCL. I was wondering, do you have a support for config directory so include? Yeah, so it can do shell lobbying so you can include newsystlog.conf.d slash star.conf and it will include every file in that directory. I don't know if we want to support recursion to go into subdirectories, probably not. Could be a flag. Does anybody have a use case for that? Yeah. You would have to have a lot of config files in order to want to sort them into subdirectories. And hopefully you don't have that many fragments. Well, even if you have a lot of config files and a lot of directories to have multiple levels of directories that can't be expressed. Yeah. So this is a slightly outdated list. We actually built a much larger one at the Dev Summit about what utilities we would like to convert and actually Beehive being at the top of the list. There is a patch already done for that so that you can just say Beehive and the VM name and it reads the config out of your Beehive configuration nest and the VM starts. And it's very expressive. It allows you to do things like on my disks, this disk is read only because it's a CD and so on. Have you considered putting config 8 on that list? No. I have to do that. Think about it. We'll check out the IRC. Okay. Other suggestions we had at the Dev Summit were login.conf and you know, what's that? Is that for TDIWAD? Maybe. Yeah. I can pull up the other slides at the end if there's time where we have the complete list of extra things we targeted. But the iSCSI stuff, most of that's fairly new. The client retained the config format from the original one. So you know, but because it's new now would be a good time to switch it before people adopt the old new config. So iSCSI and CTLD, autoFS, free miss the update. Hopefully we'll just go away with packaging base so we don't have to touch that. Currently, it would mostly probably just parse with UCL as it is, except for in the cases where there are multiple values and that would be a matter of adding some commas or whatever. The jail.conf has some interesting problems. It has key names with dots in them, but it's actually trying to express that as sections. So expanding it into the sections would eliminate the dots and solve most of the problem. The existing problem is the self-referencing variables where I need to include the host name of the jail in the path of the jail. And so that one could be interesting, especially since you'd have to do like late expansion or something. Otherwise, if you override the host name later on, how do you, you know, and should it expand late or early? And so on. So devd, the syntax is almost matching, except in some cases there are two values for one key and that's something that might have to be changed a little bit. Except for the multiple values are not a list. It's slightly more complicated. But yes, in general, yes. And that was one of the outstanding things that Velesovod and I were looking at was implicit expansion of like flags into a list. When we were originally looking at the J and C flags for new syslog, right, for compress and create, we're like, wouldn't it be cool if we could have some way to specify there's a list here that I'm specifying as a string, but I actually want to explode that into separate elements. So the new toys, this is the things you can actually play with. For example, we have UCL CMD, which I've written here. Originally, I wrote it because I want to, I wrote a shell script to start my beehives, but I wanted it to read from a UCL config file. And so I wrote a small little C program that basically was a shell interface to live UCL. And you can basically use it, you run UCL, get, point out a config file and tell it, you know, in JSON dot notation, previousd.url and it goes into that config file and pulls out the value for you. It can't expand that variable there because that's populated by PKG, not by UCL. And so I have some other examples of what you can do with it. For example, if you wanted to change the default package repo, you can use UCL CMD set and point at the file and it will read the current value for the entire file, but then change the my repo dot URL to my package repo instead. But there are some more complex cases that are going to require a bit more code when you want to do things like, you know, indicate that I want to delete this whole sub object and replace it with a new one rather than just overwrite the one specific value so that you can remove other sub keys and so on. I also, I mulled a little bit off a little command line tool called JQ, which is a JSON parser for shell. And so after you get a key, you can pipe it into little operators that are part of the program. And so I've implemented these ones. Each just lets you iterate over all the keys of an object and perform another function on them after. Or you can do pipe keys and it'll give you just the key names without the values or just the values without the key names. Or you can pipe an array into the length function and it will tell you how many objects are in the array. Or you can pipe some object into the type and it'll tell you, you know, this is an object or this is an array or this is a string or whatever. The iterate one is slightly different. It allows you to expand the implicit arrays. If you set the same key multiple times, UCL will create an implicit array. And then the one I m most proud of is recurse, which will take a UCL object and flatten it into shell environment variables, which I think I have an example of. And the four main verbs it supports are get to read a key, set to overwrite a key, merge to take two files and put them together and show you the output, and then remove to remove a section or a key from an existing file. And so yes, the merge operation isn't meant for compiling the configurations. You would just use get for that and it would do the includes automatically. Merge is more for I have this new blob of UCL that I want to append and deal with conflicts into a config file. So it's like set, but instead of setting one key, you can provide a whole fragment that it'll merge into a certain place. So here's an example file I've done up in UCL with plain string key and a sub key and an array and so on, or in this case an array of objects. And if I take that and get root that root value, you can see that is that string there. And we get back that string. But I can do this array here list of things. I can do you know list of things iterate over each item in that and give me just the value and I get the four values that were in my array. Or this is a sub file where basically I've added one new key to the object and I've overwritten the size with a new bigger size. And then when I do merge between the original file and this new file and tell it to output as UCL, I get this new size and the 150k was expanded to bytes and then the new value was appended down at the end. And it didn't change the other stuff. Whereas if you wanted it to get rid of sub key and just populate with a new one, there's not a great way to indicate that yet. But that's just because I'm still working on it. If your UCL file includes another file, that other file is where the values are actually set? With the set the file indicated with my name is F is the one you're actually writing into. UCL has protection against an include of the file you're including or whatever. But because my app uses two separate parsers, you could quite easily shoot your foot off with that. I just wanted to program where it actually sets the value in the original place where it was defined. Yeah, or at least set it later down to normalize the original file. Yeah, like you've got the false file and you've got an overrides. Yeah, currently the limitation of UCL CMD is when you output, it gives you the final version. So all the includes will have been done and you will get one single file with no include statements left in it. It's something we're looking at in the parser about actually being able to preserve the include macros to solve that problem. The set command changes something. Yeah, so use the set command and change something. First it gets all the current configuration, resolves all the includes and turn it into one big flat config file, then applies your set operation. So the set command doesn't actually modify your files. Right, it could but it would overwrite the original file and all the include files would no longer be included but their content of them would be put. So it's a limitation of the parser now that we can't get back the original include statements. We discussed the other working group and John came up with an idea of how to solve it with an AST tree or something. Right? Yeah. So yeah, this tool is fairly rudimentary because it's the first thing I've ever written in C. Yes, so if you targeted the override file, you could easily solve it. And there might be a way I could specify or you could specify from the command line, don't actually evaluate my includes so that my changes happen in the one file and I don't end up obliterating my whole include tree because that's not actually what I would like to do. But some other things, UCL has a schema validation thing but I haven't added that to UCL CMD but that would allow you to make sure your configuration is valid before you deploy it and check for syntax errors but it could actually be expanded such that we ship schemas for things like new syslog.com and be like, oh, you set the mode to something that's not an octal mode that's not allowed or you specified some value that was outside the acceptable range. So interesting ideas we had were something like VIPW to atomically edit a config file and not let you overwrite the config file until you've solved any parsing errors because nobody wants to save a config that has one too many commas in it and breaks everything. There could be some interesting things because live UCL uses M-map, if your config file, your writing ends up being smaller than it was originally and it's in use somewhere that could possibly cause problems, I'm not sure. I don't know that much about that. I just remember reading somewhere in the code that M-map and shrinking files causes fun. But yes, the config compilation, it mostly does it now but you don't have a way to extract the defaults out of that and actually generate a useful diff. But something like post-comp or Samba test parm is on my roadmap so that you'll be able to, especially if you are reading, etcdefaults.beheb.conf, etcbeheb.conf, etcbeheb.conf.d slash star.conf and user local etcbeheb.d star.conf, you're definitely going to be able to want to get what the actual configuration that your program is going to use looks like so you can actually read it. And it might be important for your puppet to be able to see what the resulting configuration is and find the one value that's not right and add yet another file to user local etc, et cetera. And obviously adding recipes or modules or whatever for UCL-CMD to orchestration frameworks like puppet, ansible, and salt, and so on so that everybody can start using it. And once FreeBSD is the easiest operating system to configure, we have more of a case for using it in large deployments. Another idea that came up at AsiaBSDCon when I presented an earlier version of this talk was wouldn't it be nice if we had something like a router has where I can do show running config and it gives me the config of everything on the entire system. So this would basically be a meta file that included all the other files in substances but might even go so far as using the.load macro we've added which allows you to include a file as a multi-line string in the config file where it could actually slurp up your sshd.conf and stick it in as a string for the couple of files that we're not going to convert because they're from an upstream vendor. Another interesting one was being able to generate NV lists from UCL and the other way around. NV lists are a binary representation and being able to get a text version out of that and go back and forth between the two would be really interesting. And the libucl parser and emitter are already there so adding NV lists shouldn't be that hard. And the other one is possibly using NV lists as the internal representation inside your application and then using UCL to the first step. One of the big to do items is adding more regression tests to libucl. I'm a very novice programmer and I've been making a lot of changes so more regression tests would be good. And you know we've broken things a couple of times or unfixed bugs that we fixed so having more of that would be good. And I also have a fairly simple test suite that's just a shell script looping over a bunch of examples for UCL-cmd and I'd like to add more to that as well. Okay. Getting back to the NV list thing. I already have something like that will take UCL for the IOV utility I added this year. Everything is for the NV list so it's very specific to my big file format but it wouldn't be too hard. I don't think it would potentially extend it to more familiar to big file. Oh awesome thank you. I'll have to get that from you. I'm good for about generator loops. I was saying Ryan's talk about SR IOV and we have an additional function and an x number of virtual functions and for each of those virtual functions you may want to set the same. This is for network device and MAC address. And if you talk about say 128 of those virtual functions that you specify it's a lot of typing which potentially could be done with a simple generator rule that says for... Start with this MAC address and add one every time. Yes. And things like that. That would be very interesting yes. That would probably fit something like a UCL command. Yeah. I'm going to add anything else right. Yeah. I'll have to add that to my to-do list. There was a whole discussion of that at the Dev Summit as well. It's like I thought we wanted not to have a Turing complete config file. Maybe we should actually convert new rc.conf and get away from having a Turing complete config file. And at aAGBsdcon this actually raised the question of once our config file is this simple do we need an orchestration system like puppet or salt? Could we build something that just relied on the operating system providing UCL, CMD, and sysrc or whatever and the service command and some sysctls to be able to extract all the information it needs or libzo. It could be able to extract any information it needs from the system and be able to express this configuration back to the system with just a couple of basic tools and you could write a much simpler orchestration system. But if people bought into an existing orchestration system there's no reason it can't work with these tools as well. Like the KernConf? Yeah. Oh yeah. Could be helpful. I guess that more expressive includes than it has now and so. I guess. Right now it is an ugly little hack on the car bumper of a hack. And that's going to be a terrible about it. Right. It would be nice to give a more standard config language. It doesn't fit with the sysadmium language that you're talking to UCL at. But it mostly works. It's the same kind of problem. I need to include these devices. I need to not include them in the devices. How do I orchestrate getting all this together and kind of have a simple little program that takes a config or method you're already used to when you configure your system because a lot of people come to bring you sysadmium and inform this and they want to put in a kernel config stuff. It might not participate in all the services that you're talking about. Right. I was wondering about overhauling the game. I don't like netbies these things because it's about this much more nicely done but it still has all of the same problems. Right. So I'm looking at imagining the current kernel config in my head and we have device blah, device blah. So we either I guess want implicit Booleans. So if you say device blah that sets it to true and later you can set it to false or something. Right. Delete this item. So include this file and then delete this item from the array of devices. Yeah. Or you know I'm compiling this you know the no debug kernel is include the generic and delete these couple of keys that enable debugging or whatever. Right. And lastly if you're somehow not aware of this there's a podcast that you can watch. We've recorded put a few interesting interviews here and we have more every week. And also if you're doing anything interesting or have ever done anything interesting or have a spare 15 minutes we'd love to interview you. We can do it here at the conference after lunch or you can do it via Skype at your convenience. Any other questions. If people are talking about the location of the default config why might why not make a switch at computer stage of the. Possibly. So the question was about storing where to store the defaults and maybe that should just be something people can choose. Yeah. Well to be the default for the default location of the default files. Currently the way I implemented in Beehive is it explicitly loads each defaults beehive dot com. And that's where these locations of every other config file are included to define those. And then obviously in each beehive dot com you can include your own extra directories. But. I don't think it is the default seems like the right place to me because it's that's where configuration goes and that's where the defaults are. There are already lots of defaults there. You mean for where to put the files. Because XZG isn't I don't know it's an option. You guys decide and tell me what to do. I know. Yes that's why I'm just doing it my way unless somebody else has a better idea. Yeah. In particular if you provide the mechanism that says here are but I'm going to tell you a bunch of paths. Now I want you to find a file that is something dot com in the first of the paths or all of them and merge together or whatever. Yes. Correct and that functionality was added for package when you do include you can specify an array of paths and then the. The file you're looking for so you know include paths equals ETC user locally to see or whatever and then include my thing dot com. And it will iterate over all those in order and stop after it finds the first one that matches. If you say glob equals true which is kind of just overriding it but then it will actually include all go search all of the paths and. Yes. And so it'll loop over every one of the paths you specify include all the files and compile a result in configuration out of that. Because that's behavior is against something that I kind of like to choose some cases I want to take the first one. Last one to override the first one right. Right. And so that's where the priority system comes in. Currently you can't do that all at once you would have to include all the ones you want at one level. Then change the default priority and then include do the search again for the files you want to override. You know we can always augment the API people have a work case that makes sense. Any other questions comments ideas things to do things not to do. Okay. Thank you.
 Can you hear me? Yes? Good. All right.  So I'm going to be talking about UCL,  the Universal Config Language,  and my goal of converting everything in  the previous DET using it.  So for anybody that doesn't know who I am,  my name is Alan Jude and I've been  a previous D-server administrator for about 13 years now.  I'm a previous D-doc committer,  mostly with focusing on documenting ZFS and  Beehive, UCL, Libzo, mostly new stuff.  I'm also the co-author of FreeBSD Mastery ZFS,  which is a book that just came out,  that walks through how to use ZFS,  how to use snapshots, how to lay out your file system, etc.  For my day job, I built and run the Scale Engine CDN.  We do mostly video streaming and HTTP content delivery,  although not as big as LimeLite, but anyway.  I host two podcasts every week.  On Wednesdays, there's BSD Now,  where we talk about the latest things that are  happening in BSD, new development,  and so on and have an interview with the random developer.  We've had 90-something of those so far,  so we have quite a collection of really good interviews with  various people from the FreeBSD and wider BSD community.  Then on Thursdays, I do TechSnap,  which is a system-in and security podcast  that's operating system agnostic.  For work, I manage about 100 servers by myself using Puppet,  which is unfortunate for me.  We have about 35 data centers in  12 different countries that we have  a small number of servers sprinkled into.  We mostly use ZFS for storing our giant video collections,  and ZFS replication for moving it between places.  We host the package mirrors for PCBSD,  so that's about 500 gigs of stuff that's being used all the time.  If you're familiar with how package works,  replication on that is very sensitive.  If all the files are not perfectly in sync,  things happen that you'd rather didn't happen.  So why have I randomly decided to start looking at using  LibUCL for everything back at EuroBSDCon last year in Sofia?  Jordan Hubbard gave the keynote about how we have to  adapt FreeBSD for the future.  I didn't agree with everything he said.  His focus seemed to be more towards little embedded things instead  of running entire data centers with FreeBSD.  But the recommendation that really hit home with me was that  we should get away from having  one config file that's completely different for every utility.  Because BSD operating system basically came together in pieces over time,  and what was the best way to make a config file change over time,  every utility has its own config language that  is entirely different.  Some of them are especially offensive to me like  NUSIS log and some other ones like that.  So when Jordan was talking about it,  he was raving about XML P-Lists that they use at Apple.  I really wanted the one true config format not to be XML.  So I've looked at some bullet points on my slides.  Why it was a good idea to have a universal config file format.  We want to be able to have readable and structured config files.  The idea of having an include statement in  your config file to pull in a snippet from somewhere else,  or all the snippets in a subdirectory,  is something that I've always really liked.  We use it in all of our web servers to separate different stuff.  But every utility seems to do that differently,  and Brian Drewry grafted includes onto NUSIS log.  But what if the config parser could just handle that,  and we would have the same system in every utility without  actually having to write that code and put it in every utility.  Also the idea of overlays.  For example, previously ships with a lot of default configuration.  For example, NUSIS log rotates all the log files that exist by default.  Well, if I want to change that,  I have to modify that file and then I have to merge it every time I upgrade.  It's just different than how I would like to do it.  Also, it's good for when you're auditing your configs to be able to say,  well, this is all the defaults here,  and then this is what I've changed away from the defaults.  So the idea of overlays, basically you have your default configuration,  and then on top of that I've layered on our site-wide configuration.  That applies to everything.  Then on top of that, I've layered on the configuration that's specific to this one machine.  So basically being able to set a configuration multiple times and eventually the last one wins.  That I thought was something I would like to have.  So I found out about UCL,  which is written by another free BSD developer,  and is already included in the base system as a private library because it's used by PKG.  But as a system in,  it really speaks to me because it's based on the bind or engineX style config file,  right? Key equals value,  optionally with the terminating semicolon,  and it can do sections like combined and engineX.  So it's what every system in is already probably used to doing.  Or is the config file they would prefer that everything used.  But it can also read JSON,  which is something people like to use,  although if you've ever tried to write by hand,  it's very finicky.  But UCL gets away from that by actually making most of those finicky things optional.  It's got a very good state machine in it.  So if you make an array in JSON,  you can't include a comma after the last item,  and that's how it knows the array's over.  Well, UCL knows the array's over when you close the array.  So you can include the trail and comma,  which makes a big difference in your config files because when you diff them,  your diff is the item you added to the array,  not the previous line getting a comma and the next item you added.  The other thing is it has syntax sugar stuff.  For example, you're specifying the maximum size of a log file in new syslog,  well, you can just put 100k.  Currently, the new syslog config file  assumes you want to rotate your logs in kilobytes.  Well, maybe you want to locate them in megabytes or something,  but the configuration doesn't tell you what the units are.  You have to go and read that in the man page.  So being able to just put 100 kilobytes or whatever is handy,  but also UCL does time.  I want to do this every five minutes or every seven days.  In the internal representation,  that's just turned into a number of seconds,  and that works really well with  programming and it handles all the parsing for you.  So this is what UCL can look like.  I've purposely used a bunch of  the different styles all put together because it will take all of them and just be happy.  So you can have key equals value,  the coding is optional and the terminating semicolon is optional.  When you do sections,  you can make them look like JavaScript objects with the colon,  or you can make them look like  n-gen X or bind sections without the colon,  or you can actually put an equal sign if you really want to.  Then the same, you can define a key in JSON style,  or you can do it with an equal sign or whatever.  The other thing is it has syntax sugar for booleans.  Yes, no, on, off, or true,  false, or any mix of them defines the boolean.  Or if you want the literal text,  the word true then you just quote it.  So I've briefly gone over the things that  UCL has an advantage over using something like JSON.  But the biggest one is that you can put  comments in your config file which you cannot do with UCL.  So with JSON.  UCL? Yes.  Yes, you can and you can include,  it does globbling, it can do  trimode where it's not a failure.  If the file you're including doesn't actually exist,  it can also do includes over HGVS,  but I'm not sure you want to do that.  Can it also do conditional stuff?  Or you set a few things and then includes in the files that say,  well, you use the ones that's globb, otherwise.  Not quite. During the Dev Summit,  the list of all explained why  an if macro doesn't quite work yet.  But it does support variables in your config files,  but those have to be defined by  the program that's going to parse it so we can set  a bunch of variables that you can then use inside the config file.  But one thing we're looking to add to it is  the ability to reference values you set previously.  Specifically, in order to convert Jail.conf,  it's very handy once you set  your Jail's hostname to be able to use  the hostname in the path name and things like that.  So it's coming along for that,  although currently there's not much in the way of conditional stuff.  But the main reasons to use UCL over something else is that it's very flexible.  Because you can do sections and arrays or just key value pairs or whatever you need to do,  you can express the configuration for pretty much  any application I can think of using it.  Your config file can look as simple as rc.conf,  whatever equals whatever,  or you can do engineXStyle with nesting  and whatever you need to end up doing.  It has an includes feature which also supports priorities.  So you can say, here's some config and now  include this config with a higher priority.  So if there's conflicting keys,  they overwrite or include it with a lower priority because it's just the defaults.  So if there's conflict,  the original one still wins rather than what you're including.  It also has a mode where if you include or just specify  the same value twice with the same priority,  it can actually create an array out of them if that's what you want it to do.  It can be very terse,  if it just key equals value,  there's no excess formatting that you would get with XML.  So it makes it much easier to read.  Yes, while it's easy to read,  unlike JSON, it's also easy to write and it allows comments.  So for example, CronTab has the format that we've all used,  although trying to teach it to classrooms full of people as I've  done, you quickly learned that they don't quite understand that to get what they want,  they don't put, if you want to run something once an hour,  you don't put star one,  which will run at every minute only at one o'clock.  You have to pick a minute to run it out and then run it every hour or something.  Anyway, if you convert CronTab config into something like this for UCL,  it ends up actually looking like how you specify a CronTab in  Puppet or Salt or any orchestration system,  where you specify the keys by their name,  but it also allows you to do things like specify the environment more cleanly.  In the regular CronTab file,  you specify mail to something or path is this,  and then you specify some CronTabs,  and maybe you override path again,  but you can't unset an environment variable very easily.  Where as this, you could isolate it more easily.  But that's just an example of what a CronTab would look like if it was converted.  So, new syslog especially is this one.  At the top there, you can see what the original config looks like.  The new syslog config file is especially offensive because beyond the end of what you see there,  there are optional fields,  but because they're optional,  they're all overloaded under that one field.  So, if the last field starts with a slash,  then it's the path to a PID file unless you want to operate on.  Or if it doesn't start with a slash,  then it is the command you want to invoke in order to reload the service.  Then if it's a special character or something,  then or if it's a number,  then it's the signal that you want to send.  It's that one field has different meanings and different contexts and that's not very good.  But the biggest problem is you end up with a bunch of these lines in the file,  and it's not really possible that it's not very easy to  programmatically parse what's actually happening there or to programmatically change it.  We're using puppets, so when we want to enable something in RC.com,  we use the sysrc command line.  We say sysrc, this variable, we get the value,  compare that to our manifest.  If it's right, if not,  we can use sysrc and say programmatically change this without  risking working my config file.  So we want something like that and I'll get to more on that later.  But this is what you would end up looking like if you converted new syslog,  which I actually have a patch for on fabricator.  You have the file name, the mode,  the number of logs you want to keep and so on.  But instead of JC being random flags that you have to look up the man page,  you just specify different booleans for each thing you want.  We can use the include support here.  I should have a dot in front of it.  You can specify things like try so that it'll not fail  if there are no config files in that directory and so on.  There's still a question as what to do about the time field in the new syslog.  It is a standard way to specify time,  but it's not a very intuitive one.  So I'd like to expand the concept beyond just replacing a bunch of the config files  and having a standard way to deal with the common problems,  specifically moving default config files.  Right now we ship ETC, Krontab, ETC, new syslog.com,  and a bunch of config files in ETC that people modify.  It'd be better to move the defaults away.  Based on discussion in the working group,  some of those should actually probably be moved to user share rather than ETC defaults.  But either way, the default should live someplace,  and then the file you put in ECC should only be the changes you've locally made away from the defaults.  Can you briefly summarize why user share instead of ETC default?  I forget. I forget why.  It was someone else's recommendation.  Yeah, it makes no sense.  Oh, okay. It was a Linux guy's recommendation.  So yeah.  Sorry.  What is the question?  The question was, I had said that at the Dev Summit working group we had earlier in the week,  they had suggested that the defaults not be installed into ETC defaults,  but move to user share, like where our example config files go,  and the rest of the defaults just be implied, I suppose.  And do you remember why?  The rationale was because, I don't know if it's an LSB issue or something,  but the rationale was that many users of Puppet and Nets or whatever  expect that slash EDC is only for files that you put there,  and it's only for local configuration that override something else  and that anything else you would want.  Anything that shifts with the system as a default should go to the user share.  That's something that apparently a lot of Linux things have been moving to.  I don't know.  I like EDC defaults.  I think it makes sense.  That's the in two places.  Yeah.  Either be a user share or a default, or it should be an FTP default.  Some shouldn't be here and some shouldn't be there.  Yeah, exactly.  So the proposal is to say, we're going to make it clear that  the stuff that you do goes here and the defaults that we should go somewhere else  and how you spell the place where the OS puts its defaults.  I don't think we really care.  We can stick to EDC defaults and Linux for user share.  I think sticking to EDC defaults probably makes more sense  because some people still partitioned user separately  and you want your config files to exist in single user mode.  You kind of do.  Yeah.  So the overwrite feature basically including one file and then including a second file with a higher priority  allows you to overwrite the existing config.  So for example, new syslog, I like to keep more copies of varlog messages than the default,  which is five.  I want eight, so I can always go back a week.  If I modify the config file for that, then every time I upgrade, I have to merge it and it gets annoying.  Whereas with UCL, I would just in my etcnosyslog.com,  I would just say messages, curly brace, count equals eight.  And I'm only changing that one setting and everything else is inherited from the defaults.  So are there ways to modify the inheritance rules?  And what if you wanted to completely re-specify the varlog message?  Right.  That came up in the working group and so we're probably going to add a macro.delete  and then a key and it will destroy any previous thing that was there  so that you'll be able to re-specify everything.  So if by default everything is merged?  Yes.  Not over-registered?  Yes.  Yeah.  There's some controls for it, but it's...  Controlling the recursion of the merges is not really possible at the moment.  But I've done quite a bit of work on improving that and giving a little bit more control with it.  So, the other one was we ship a bunch of crontabs in ETC crontab.  What if I wanted to disable one of those?  Currently I'd have to go in there and delete the line or comment it out or something.  Whereas if we did something more like we do with package and default package repo,  I would just in my crontab file that wherever that ends up living,  I could just select a certain crontab and set enable to no  and that crontab won't execute anymore.  So, yeah, this one deals with the presidencies you just mentioned.  So if we have ETC package, frebsd.conf and it's enabled,  but then in use a local ETC repose, I've set it enable no,  and then in yet a higher priority place somewhere, I've set it back to yes.  How do we decide who wins and how does the administrator see who wins?  So, we can use the priority key to decide,  because the order might just be alphabetical,  you might of the includes, you can use the priority number,  which currently supports values between 0 and 15,  to indicate how high of a priority your changes to the config file are.  So you can decide how it's going to end up getting merged.  And separately, I've talked about later,  a tool that allows you to compile this config and see what the resulting config will look like,  like you get with post comp from postfix or test perm from SAMBA.  And so, when I originally wrote these slides for AsiaBSDcon,  the idea of priorities and includes didn't exist yet,  but since then, Vlustavad has actually implemented that  and I've actually separately recently added a feature  where you can do it for a small block of config rather than,  with the original implementation, you only change the priority of each file as you included it,  but you can actually change the priority inline in the config file  for even just certain sections of the config file now.  Interesting one that I haven't run into yet,  but I foresee as possible being a problem is the ordering of keys.  Most of the apps I've looked at so far,  what order you specify configuration for stuff in doesn't really matter,  but at some point that there might be something that did.  Obviously, firewall rules or something, the order is actually really important  and deciding how to preserve that,  but I haven't written anything for that yet because I haven't encountered  a tool I want to convert yet that requires very specific ordering of the config stuff.  Everything we've looked at so far was just priority stuff to decide what the resulting  config will look like, where the actual order of the statements in the config will actually  change how it's configured because I imagine in most places that's actually a bad design  other than things like a firewall where you're being very specific.  That's like there's a slide coming up on that because mostly it's done.  It's almost finished.  So talking about actually doing it, I got the idea originally at  EuroBSDCon after hearing the talk by Jordan Hubbard.  Month and some later, I was at MeetBSD and I had done a short lightning talk about this idea of,  hey, somebody should do this.  Then I was stuck in an airport waiting for an airplane and I was like, how hard can it be?  So I picked Newsyslog because it's a relatively small piece of code  and it's not something that if I broke it, I would get murdered or anything.  So I started playing with it and basically it was the first thing in C I had ever tried to write.  I read little bits of C to try to understand what was happening.  This is the first time I was adding more than an if statement into a program.  So I started doing it and now there's a patch on Fabricator for actually converting Newsyslog  to use UCL and it's set to be backwards compatible so it will actually read the old config file  and only switch to parsing with the new parser if you have a comment at the top of the file  that indicates it's the new config file format.  And I wrote most of that on an airplane back home from California and then it was a bit rough  but over the course of a couple of spare weekends I managed to actually get a working prototype  and the best part is it doesn't change any of the existing code, it just added some new stuff  and a little hijack to the config parser that says if the first line is this,  run the new parser, if it's not, rewind the pointer to the beginning of the file and parse it traditionally.  And so that brings up compatibility.  You know, if we're going to change the config file format of all these utilities in time  for FreeBSD 11 or something, people that upgrade from 10 to 11 are going to be like,  you want me to rewrite all my config files? I was like, going to work.  So if we use a similar strategy as we did with Newsyslog there of, you know,  a sentinel at the top of the file or some way to indicate this is a UCL config file,  then we can do that or if it doesn't have that then we fall back to the traditional parsing  so people upgrading from 10 to 11 can just keep their existing config files  but anybody doing a fresh install of 11 can start with the new format and we can  deprecate the old ones in 12 or 13 or something. And the other thing I did as part of the sentinel  was include a version number in the config file format because I'm quite sure that the first  revision won't be perfect. And so this would allow us to do something like in the Gehli code,  they have a little version number and, you know, based on that version number,  it picks a different parsing function to read the configuration.  Hopefully we don't add too much bloat doing that but.  And so this just describes how I changed the Newsyslog startup routine. I already described  that to you. So basically what it does is you run the libucl parser on the config file,  it handles all the includes and layering and stuff and gives you an object that represents  the completed configuration. Then we just use its iterator function to walk over the config file,  extract the root keys which there should be one for each config file or a log file that you want  to rotate. Then we process the subkeys, you know, oh this log file says compress equals xz or  compress equals yes and use the default compressor and so on. And then it executes, it loads that up  into the struct that Newsyslog's original config parser used and then it just runs the program  as normal. The one feature that it's missing at the moment is a way to specify the defaults.  Specifically it'd be really handy to disable compression on all my log files if my log files  are on ZFS where it's going to get compressed that way instead. Because in a normal case you  have your log file it's being written to ZFS constantly getting compressed and then I'm going  to rotate it, read it all and compress it again with BZIP or something. That's basically a waste  of disk cycles at that point. So the remaining items to sort out are possibly designing a  consistent set of keys to use across config files like the enabled that we talked about for  crontab and package but in the working group it seemed that those would be fairly specific to  the different use cases and maybe we don't actually need a set of them. But having something like  Zolint where we could kind of build up a database of the common names of keys and try to be consistent  across our config files probably makes a lot of sense. But we've decided against actually having  a set of specific keys that are reserved for free BSD utilities or trying to namespace those so  they might not conflict with what an application does specifically. The other problem was developing  a universal API for actually reading the lib UCL stuff in to C and I think we actually settled on  basically writing a config file schema and a program that will actually take that and generate the  C code for you since there's not really a great way to load it directly into the structs or whatever.  And so, yeah, the decision there was it probably best to just auto generate the SturComp ladder  you'd end up with to load your configuration. And so the other thing is I've written this tool UCLCMD  which is a command line interface to allow you to read and set and merge UCL config files so that  for example I can instruct puppet to go change the number of log files that we keep. And the other  thing that it needs to learn, it mostly does it now, would be the compiling the resulting  configuration but based on talk at the Dev Summit it would also be nice if it could exclude the  defaults from that so you can see what is different between the resulting configuration  and what the original default configuration was. So kind of a version of diff built into it as well.  And yes, we had a Dev Summit working group for about three hours earlier this week and  quite a bit of interesting stuff resulted from that.  Then we looked at other things that we would use UCL.  I was wondering, do you have a support for config directory so include?  Yeah, so it can do shell lobbying so you can include newsystlog.conf.d slash star.conf  and it will include every file in that directory. I don't know if we want to support recursion  to go into subdirectories, probably not. Could be a flag. Does anybody have a use case for that?  Yeah.  You would have to have a lot of config files in order to want to sort them into subdirectories.  And hopefully you don't have that many fragments.  Well, even if you have a lot of config files and a lot of directories to have multiple  levels of directories that can't be expressed.  Yeah. So this is a slightly outdated list. We actually built a much larger one at the Dev Summit  about what utilities we would like to convert and actually Beehive being at the top of the list.  There is a patch already done for that so that you can just say Beehive and the VM name and it  reads the config out of your Beehive configuration nest and the VM starts.  And it's very expressive. It allows you to do things like on my disks, this disk is read only  because it's a CD and so on.  Have you considered putting config 8 on that list?  No. I have to do that.  Think about it. We'll check out the IRC.  Okay. Other suggestions we had at the Dev Summit were login.conf  and you know, what's that?  Is that for TDIWAD?  Maybe.  Yeah. I can pull up the other slides at the end if there's time where we have the complete list  of extra things we targeted. But the iSCSI stuff, most of that's fairly new. The client retained  the config format from the original one. So you know, but because it's new now would be a good  time to switch it before people adopt the old new config. So iSCSI and CTLD, autoFS,  free miss the update. Hopefully we'll just go away with packaging base so we don't have to touch  that. Currently, it would mostly probably just parse with UCL as it is, except for in the cases  where there are multiple values and that would be a matter of adding some commas or whatever.  The jail.conf has some interesting problems. It has key names with dots in them, but it's actually  trying to express that as sections. So expanding it into the sections would eliminate the dots and  solve most of the problem. The existing problem is the self-referencing variables where I need to  include the host name of the jail in the path of the jail. And so that one could be interesting,  especially since you'd have to do like late expansion or something. Otherwise,  if you override the host name later on, how do you, you know, and should it expand late or early?  And so on. So devd, the syntax is almost matching, except in some cases there are two values for  one key and that's something that might have to be changed a little bit. Except for the  multiple values are not a list. It's slightly more complicated. But yes, in general, yes. And that  was one of the outstanding things that Velesovod and I were looking at was implicit expansion of  like flags into a list. When we were originally looking at the J and C flags for new syslog,  right, for compress and create, we're like, wouldn't it be cool if we could have some way to specify  there's a list here that I'm specifying as a string, but I actually want to explode that into  separate elements. So the new toys, this is the things you can actually play with. For example,  we have UCL CMD, which I've written here. Originally, I wrote it because I want to, I wrote a shell  script to start my beehives, but I wanted it to read from a UCL config file. And so I wrote a  small little C program that basically was a shell interface to live UCL. And you can basically use  it, you run UCL, get, point out a config file and tell it, you know, in JSON dot notation,  previousd.url and it goes into that config file and pulls out the value for you. It can't expand  that variable there because that's populated by PKG, not by UCL. And so I have some other examples  of what you can do with it. For example, if you wanted to change the default package repo,  you can use UCL CMD set and point at the file and it will read the current value  for the entire file, but then change the my repo dot URL to my package repo instead.  But there are some more complex cases that are going to require a bit more code when you want  to do things like, you know, indicate that I want to delete this whole sub object and replace it  with a new one rather than just overwrite the one specific value so that you can remove other sub  keys and so on. I also, I mulled a little bit off a little command line tool called JQ, which is a  JSON parser for shell. And so after you get a key, you can pipe it into little operators that are  part of the program. And so I've implemented these ones. Each just lets you iterate over all the  keys of an object and perform another function on them after. Or you can do pipe keys and it'll  give you just the key names without the values or just the values without the key names. Or you  can pipe an array into the length function and it will tell you how many objects are in the array.  Or you can pipe some object into the type and it'll tell you, you know, this is an object or this  is an array or this is a string or whatever. The iterate one is slightly different. It allows  you to expand the implicit arrays. If you set the same key multiple times, UCL will create an  implicit array. And then the one I m most proud of is recurse, which will take a UCL object and  flatten it into shell environment variables, which I think I have an example of. And the four main  verbs it supports are get to read a key, set to overwrite a key, merge to take two files and put  them together and show you the output, and then remove to remove a section or a key from an existing  file. And so yes, the merge operation isn't meant for compiling the configurations. You would just  use get for that and it would do the includes automatically. Merge is more for I have this new  blob of UCL that I want to append and deal with conflicts into a config file. So it's like set,  but instead of setting one key, you can provide a whole fragment that it'll merge into a certain  place. So here's an example file I've done up in UCL with plain string key and a sub key and an  array and so on, or in this case an array of objects. And if I take that and get root that root value,  you can see that is that string there. And we get back that string. But I can do  this array here list of things. I can do you know list of things iterate over each item in that and  give me just the value and I get the four values that were in my array. Or this is a sub file where  basically I've added one new key to the object and I've overwritten the size with a new bigger size.  And then when I do merge between the original file and this new file and tell it to output as UCL,  I get this new size and the 150k was expanded to bytes and then the new value was appended down  at the end. And it didn't change the other stuff. Whereas if you wanted it to get rid of sub key  and just populate with a new one, there's not a great way to indicate that yet. But that's just  because I'm still working on it.  If your UCL file includes another file, that other file is where the values are actually set?  With the set the file indicated with my name is F is the one you're actually writing into.  UCL has protection against an include of the file you're including or whatever.  But because my app uses two separate parsers, you could quite easily shoot your foot off with that.  I just wanted to program where it actually sets the value in the original place where it was defined.  Yeah, or at least set it later down to normalize the original file.  Yeah, like you've got the false file and you've got an overrides.  Yeah, currently the limitation of UCL CMD is when you output, it gives you the final version.  So all the includes will have been done and you will get one single file with no include statements  left in it. It's something we're looking at in the parser about actually being able to preserve  the include macros to solve that problem.  The set command changes something.  Yeah, so use the set command and change something. First it gets all the current configuration,  resolves all the includes and turn it into one big flat config file, then applies your set operation.  So the set command doesn't actually modify your files.  Right, it could but it would overwrite the original file and all the include files would no longer  be included but their content of them would be put. So it's a limitation of the parser now that we  can't get back the original include statements. We discussed the other working group and  John came up with an idea of how to solve it with an AST tree or something.  Right? Yeah. So yeah, this tool is fairly rudimentary because it's the first thing I've ever written in C.  Yes, so if you targeted the override file, you could easily solve it.  And there might be a way I could specify or you could specify from the command line,  don't actually evaluate my includes so that my changes happen in the one file and I don't  end up obliterating my whole include tree because that's not actually what I would like to do.  But some other things, UCL has a schema validation thing but I haven't added that to UCL CMD  but that would allow you to make sure your configuration is valid before you deploy it  and check for syntax errors but it could actually be expanded such that we ship schemas for things  like new syslog.com and be like, oh, you set the mode to something that's not an octal mode that's  not allowed or you specified some value that was outside the acceptable range.  So interesting ideas we had were something like VIPW to atomically edit a config file  and not let you overwrite the config file until you've solved any parsing errors  because nobody wants to save a config that has one too many commas in it and breaks everything.  There could be some interesting things because live UCL uses M-map, if your config file, your  writing ends up being smaller than it was originally and it's in use somewhere that could  possibly cause problems, I'm not sure. I don't know that much about that. I just remember reading  somewhere in the code that M-map and shrinking files causes fun. But yes, the config compilation,  it mostly does it now but you don't have a way to extract the defaults out of that and actually  generate a useful diff. But something like post-comp or Samba test parm is on my roadmap  so that you'll be able to, especially if you are reading, etcdefaults.beheb.conf, etcbeheb.conf,  etcbeheb.conf.d slash star.conf and user local etcbeheb.d star.conf, you're definitely going  to be able to want to get what the actual configuration that your program is going to  use looks like so you can actually read it. And it might be important for your puppet to be able to  see what the resulting configuration is and find the one value that's not right  and add yet another file to user local etc, et cetera. And obviously adding recipes or modules  or whatever for UCL-CMD to orchestration frameworks like puppet, ansible, and salt, and so on so  that everybody can start using it. And once FreeBSD is the easiest operating system to configure,  we have more of a case for using it in large deployments. Another idea that came up at  AsiaBSDCon when I presented an earlier version of this talk was wouldn't it be nice if we had  something like a router has where I can do show running config and it gives me the config of  everything on the entire system. So this would basically be a meta file that included all the  other files in substances but might even go so far as using the.load macro we've added which  allows you to include a file as a multi-line string in the config file where it could actually  slurp up your sshd.conf and stick it in as a string for the couple of files that we're not going to  convert because they're from an upstream vendor. Another interesting one was being able to generate  NV lists from UCL and the other way around. NV lists are a binary representation and being  able to get a text version out of that and go back and forth between the two would be really  interesting. And the libucl parser and emitter are already there so adding NV lists shouldn't  be that hard. And the other one is possibly using NV lists as the internal representation inside  your application and then using UCL to the first step. One of the big to do items is adding more  regression tests to libucl. I'm a very novice programmer and I've been making a lot of changes  so more regression tests would be good. And you know we've broken things a couple of times  or unfixed bugs that we fixed so having more of that would be good. And I also have a fairly  simple test suite that's just a shell script looping over a bunch of examples for UCL-cmd and  I'd like to add more to that as well. Okay. Getting back to the NV list thing. I already have something  like that will take UCL for the IOV utility I added this year. Everything is for the NV list so  it's very specific to my big file format but it wouldn't be too hard. I don't think it would  potentially extend it to more familiar to big file. Oh awesome thank you. I'll have to get that from you.  I'm good for about generator loops. I was saying Ryan's talk about SR IOV and we have an additional function  and an x number of virtual functions and for each of those virtual functions you may want to set  the same. This is for network device and MAC address. And if you talk about say 128 of those virtual functions  that you specify it's a lot of typing which potentially could be done with a simple generator  rule that says for... Start with this MAC address and add one every time. Yes. And things like that.  That would be very interesting yes. That would probably fit something like a UCL command. Yeah.  I'm going to add anything else right. Yeah. I'll have to add that to my to-do list.  There was a whole discussion of that at the Dev Summit as well. It's like I thought we wanted  not to have a Turing complete config file. Maybe we should actually convert new rc.conf  and get away from having a Turing complete config file.  And at aAGBsdcon this actually raised the question of once our config file is this simple do we need  an orchestration system like puppet or salt? Could we build something that just relied on the operating  system providing UCL, CMD, and sysrc or whatever and the service command and some sysctls to be able  to extract all the information it needs or libzo. It could be able to extract any information it  needs from the system and be able to express this configuration back to the system with just a couple  of basic tools and you could write a much simpler orchestration system. But if people bought into  an existing orchestration system there's no reason it can't work with these tools as well.  Like the KernConf?  Yeah. Oh yeah. Could be helpful. I guess that more expressive includes than it has now and  so.  I guess.  Right now it is an ugly little hack on the car bumper of a hack.  And that's going to be a terrible about it.  Right.  It would be nice to give a more standard config language. It doesn't fit with the sysadmium  language that you're talking to UCL at.  But it mostly works.  It's the same kind of problem. I need to include these devices. I need to not include them in the devices.  How do I orchestrate getting all this together and kind of have a simple little program that takes a config  or method you're already used to when you configure your system because a lot of people come to bring you  sysadmium and inform this and they want to put in a kernel config stuff. It might not participate in all the services that you're talking about.  Right.  I was wondering about overhauling the game. I don't like netbies these things because it's about this  much more nicely done but it still has all of the same problems.  Right.  So I'm looking at imagining the current kernel config in my head and we have device blah, device blah.  So we either I guess want implicit Booleans. So if you say device blah that sets it to true and later you can set it to false or something.  Right.  Delete this item. So include this file and then delete this item from the array of devices.  Yeah. Or you know I'm compiling this you know the no debug kernel is include the generic and delete these couple of keys that enable debugging or whatever.  Right.  And lastly if you're somehow not aware of this there's a podcast that you can watch. We've recorded put a few interesting interviews here and we have more every week.  And also if you're doing anything interesting or have ever done anything interesting or have a spare 15 minutes we'd love to interview you.  We can do it here at the conference after lunch or you can do it via Skype at your convenience.  Any other questions.  If people are talking about the location of the default config why might why not make a switch at computer stage of the.  Possibly. So the question was about storing where to store the defaults and maybe that should just be something people can choose.  Yeah. Well to be the default for the default location of the default files.  Currently the way I implemented in Beehive is it explicitly loads each defaults beehive dot com.  And that's where these locations of every other config file are included to define those.  And then obviously in each beehive dot com you can include your own extra directories.  But.  I don't think it is the default seems like the right place to me because it's that's where configuration goes and that's where the defaults are.  There are already lots of defaults there.  You mean for where to put the files.  Because XZG isn't I don't know it's an option.  You guys decide and tell me what to do.  I know. Yes that's why I'm just doing it my way unless somebody else has a better idea.  Yeah.  In particular if you provide the mechanism that says here are but I'm going to tell you a bunch of paths.  Now I want you to find a file that is something dot com in the first of the paths or all of them and merge together or whatever.  Yes.  Correct and that functionality was added for package when you do include you can specify an array of paths and then the.  The file you're looking for so you know include paths equals ETC user locally to see or whatever and then include my thing dot com.  And it will iterate over all those in order and stop after it finds the first one that matches.  If you say glob equals true which is kind of just overriding it but then it will actually include all go search all of the paths and.  Yes.  And so it'll loop over every one of the paths you specify include all the files and compile a result in configuration out of that.  Because that's behavior is against something that I kind of like to choose some cases I want to take the first one.  Last one to override the first one right.  Right.  And so that's where the priority system comes in.  Currently you can't do that all at once you would have to include all the ones you want at one level.  Then change the default priority and then include do the search again for the files you want to override.  You know we can always augment the API people have a work case that makes sense.  Any other questions comments ideas things to do things not to do.  Okay. Thank you.
UCL_for_FreeBSD_0_359.png UCL_for_FreeBSD_360_419.png UCL_for_FreeBSD_420_515.png UCL_for_FreeBSD_516_575.png UCL_for_FreeBSD_576_623.png UCL_for_FreeBSD_624_671.png UCL_for_FreeBSD_672_719.png UCL_for_FreeBSD_720_839.png UCL_for_FreeBSD_840_887.png UCL_for_FreeBSD_888_935.png UCL_for_FreeBSD_936_1007.png UCL_for_FreeBSD_1008_1055.png UCL_for_FreeBSD_1056_3923.png UCL_for_FreeBSD_3924_5951.png UCL_for_FreeBSD_5952_8651.png UCL_for_FreeBSD_8652_11879.png UCL_for_FreeBSD_11880_13235.png UCL_for_FreeBSD_13236_15587.png UCL_for_FreeBSD_15588_17723.png UCL_for_FreeBSD_17724_19679.png UCL_for_FreeBSD_19680_19727.png UCL_for_FreeBSD_19728_23255.png UCL_for_FreeBSD_23256_30383.png UCL_for_FreeBSD_30384_32819.png UCL_for_FreeBSD_32820_34307.png UCL_for_FreeBSD_34308_34391.png UCL_for_FreeBSD_34392_34667.png UCL_for_FreeBSD_34668_37619.png UCL_for_FreeBSD_37620_39551.png UCL_for_FreeBSD_39552_39719.png UCL_for_FreeBSD_39720_41543.png UCL_for_FreeBSD_41544_45011.png UCL_for_FreeBSD_45012_51287.png UCL_for_FreeBSD_51288_52427.png UCL_for_FreeBSD_52428_53459.png UCL_for_FreeBSD_53460_55151.png UCL_for_FreeBSD_55152_56207.png UCL_for_FreeBSD_56208_56435.png UCL_for_FreeBSD_56436_56531.png UCL_for_FreeBSD_56532_56651.png UCL_for_FreeBSD_56652_56735.png UCL_for_FreeBSD_56736_56903.png UCL_for_FreeBSD_56904_56975.png UCL_for_FreeBSD_56976_57203.png UCL_for_FreeBSD_57204_57515.png UCL_for_FreeBSD_57516_62015.png UCL_for_FreeBSD_62016_65507.png UCL_for_FreeBSD_65508_70475.png UCL_for_FreeBSD_70476_75647.png UCL_for_FreeBSD_75648_81409.png