mandoc: becoming the main BSD manual toolbox - BSDCan 2015 presentation
CC Attribution - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Schwarze, Ingo
The original audio stream of my presentation at BSDCan 2015 in Ottawa (except for the first 30 seconds and the last four minutes; those two chunks failed to record in Ottawa, so i had to re-record them). The associated video stream contains the presentation slides captured off the beamer input by the conference organizers, so video and audio are in sync. Topics are the new man(1), man.conf(5), man.cgi(8); eqn(7) HTML5 and MathML output; UTF-8 improvements, afl(1) audit, -Wunsupp, pod2mdoc(1), a status summary in various operating systems, and possible future directions.
 Welcome to my presentation on Mandok. The topic this year is becoming the main VSD manual toolbox. My name is Ingo Schwarzze, Schwarzze at OpenVSD.org and I have been an OpenVSD developer for the last six years. While I've also contributed in a few other areas, it happened that my main focus became documentation tools. One thing that's always nice for starting a talk is reminding everybody that we are usually standing on the shoulders of giants and the pioneer in this area is really Cynthia Livingstone. You are seeing here on this picture, she designed the main documentation language we are still using today in 1999-1990, the M-Doc language. She implemented that language herself. She translated the whole corpus of BSD manuals from the old man language to the new M-Doc language and in the process she also rewrote all the text that was still encumbered by AT&T copyrights. So all that by one single person. When I talked here about Mandok four years ago, when I first presented the Mandok toolbox, my focus was this is a completely new tool and we have to train it to do real work and you'll see how the beast has matured until now. Okay, so the key point, we are talking about documentation tools. The key point from my point of view about system documentation is really that all documentation should be in one place and in one format, so not have a part of the web and a part in HTML and a part in user share docs and whatnot, one place, one format. That makes it easy to find, easy to read and easy to write and only if it's easy to write there is any chance that it will be correct, complete and concise. That of course puts a particular focus on which system to use. If you want to use one system, the basic markup syntax we are still using today goes back more than 50 years. Jerry Salzer started the Roth-Ranoff markup in 1964. It is unobtrusive, it's diff-friendly, it's easy to hand edit and there are simple tools to produce high-quality output in various formats from them. The basic manual structure goes back to the very first version of Unix to Thompson and Ritchie. The manual language still used in Linux today comes from the famous version 7 Unix, the last resourced Unix version that is publicly available, but the real revolution in documentation languages really was the invention of semantic markup of the M-Doc language in 8990 by Cynthia Livingston, which then got to the world with 4.4 BSD. At about the same time James Clark wrote the GNU implementation of Trof, which even though its GPL software dominated the toolbox world in the BSDs for more than two decades until finally around 2010. We started to introduce the BSD licensed Mandoq toolbox step-by-step into the various BSDs by now. Practically all major BSDs use it, open BSD, free BSD, net BSD, even Illumos has switched to it. Now what is in that toolbox we are talking about from the user perspective it has become really simple. From the user perspective you basically have one user land program that you are calling MAN, the manual viewer. That thing when you call it once there's three things. It finds one manual page in the file system or using a database. You can decide which file to find either by giving its name or by giving a search query. Then in the second step MAN will format the manual page and in the third step it will display it usually using a pager. Even in this very elementary area there's quite some progress since I presented here last year. Last year I said MAN, the manual viewer is out of scope that's not part of the toolbox. Now we have one unified interface for both for the viewer, for MAN, for the formator, for MAN doc and for the search tool apropos. They all take the same options and very new this year 2015 we have a unified and very much simplified configuration file format that I'll show. The toolbox also contains a few auxiliary components. You have a database generation tool, you have a syntax checker, you have a parse tree debugger, you have a format converter from MAN to M doc that is built in, pardon from M doc to MAN that is built in and you have output front ends for various formats like HTML, post-crypt, PDF. So before really starting with the individual topics I give you a very brief overview which topics I'll talk about. I guess I never talked about so many different topics in a single public talk. It's just because so much happened in the last year. And I hope I won't get mired in that swamp of topics. The unified user interface, the new viewer and configuration file format is the first thing, the second is the same for the web, the third is improved formatting of mathematical equations, then improved unicode support, hunting bugs with a fuzzer program, detecting use of unsupported features, converting manual pages from another language from the pearl documentation language to our common M doc language. These are the main subjects, the main things that happened during this year and I will wrap up providing a status in various operating systems and hinting at a few possible future directions. By the way, the pictures I'm using for illustrations are pictures taken by other people along the road of the bicycle tour I did around Southern Ontario just after last year's conference. Okay, so first thing we did, OpenBSD no longer uses the traditional BSD MAN program but an implementation integrated into Mandoch into the formator. The traditional setup was that MAN would fork and execute twice, once to call the formator and then again to page the output. Right now the program finding the files in the file system and formatting is the same program. Now, which is the point in doing that? The point really is to have a unified interface for all the three main front ends, which means that when you call the viewer, when you type MAN, some page, you can use command line options that were traditionally only available to the formator, like saying which warning level you want to see or saying which output format you want. You can now do things like in OpenBSD, not yet in FreeBSD, that will only be available in FreeBSD 12. You can say things like MAN minus T, HTML, just the name of the manual and then pipe it directly to links or something. On the other way around, the search tool now has access to options that traditionally you only had in the viewer. The search tool apropos normally just lists a number of title lines for manuals. You now have options to say I want to see the file names instead or I want to see the command synopsis instead or even I give it a search query but I want to see all the full manual pages in one less right away. So that's quite flexible and you don't need to remember different options for different programs. Besides, it allows a simpler configuration file format which we use for you minor things that I'm not going to name individually except that one of them is quite nice. We have to maintain one less userland program and the traditional BSD MAN is quite old code so it's nice to no longer need to maintain that. There are two other things that we can gain in the future that we have not yet exploited. In particular, many library manuals document not only one but two, three, twenty functions and those typically have multiple entries in the file system hard links. With the new database we can get rid of all those hard links and can get rid of thousands of files in the in the installation and another nice thing that can be done with this it is already implemented it's only not integrated yet in OpenBSD. We can have an interactive chooser so you say apropos something it comes up with a list of programs that match and then you can just choose one of them to open the manual directly without exiting apropos first and typing a new command. Okay any change of a program for a new program will come at a cost. In this case the cost is that database lookup is slightly slower than file system lookup. Then again on my notebook the additional delay for displaying a manual page is on the order of 10 milliseconds. There is another cost when you install a new manual page on the system and you want to find that new manual page with the search tools you have to update the database but the at least the OpenBSD package tools run the required commands automatically for you so you don't need to do it manually and even if you forget it or it doesn't work then the man tool itself will still work you will still see the manuals when you explicitly call for them you just won't find them in the search tool until the weekly make what is run. So there are very little downsides if any compared to the additional features we get. Oh this is a nice one the old configuration file format of the manual viewer this is the list of all the features I identified in the old configuration file format that are completely useless. I'm not going to bore you reading that this to you I'll just pick out one you can configure decompression filters like if I have a G-SIP man page I want to use GUN-SIP on it. Well the old configuration file format allowed you to configure different decompression filters in different sections of the manual so if I have a kernel manual that is G-SIP I want to use this decompression filter if I have a user-length manual this G I prefer the other decompression filter and dozens of them so the format was so complicated that really consistently everybody hated it all both nobody used it even though people had good uses for it some things you want to configure so I came up with a new configuration file format that basically has just two directives the things that people actually need one is you can specify a man pass you give a directory name it takes that directory as a complete tree of manual pages on consistently uses that tree across all the tools and the other directive is specifying output options for example for the terminal saying how wide the terminal is for HTML saying which style sheet you want to link from the generated file for PostScript.pf specifying the paper format things like that that's very easy to use and needs almost no learning a few a bit of the namespace in the new configuration file format has been reserved so once people ask for it I'm planning to implement the following features an alias directive which makes it easier for people using languages like TCL or so to make whole trees more easily accessible with the minus s option then a sections directive that allows people to configure custom sections and to change the search order of sections and the filter directives in case any operating systems are using other compression formats besides GZIP however so far nobody asked for these formats and as long as people don't need them I say kiss keep it stupidly simple I we shouldn't implement anything that people don't actually want so be aware here I really reduced the functionality inside open BSD and people don't yell at me so it's not always bad to make things simpler good so far I talked about viewing manual pages on the command line on the terminal we also have a CGI for viewing them on the web which has basically identical functionality open BSD on the WWW open BSD org website no longer uses the traditional manned CGI Perl script by Wolfram Schneider from free BSD but a manned CGI implementation included in the manned doc toolkit the traditional setup was that manned CGI would fork and execute the systems man command that man command would fork and execute a graph or later man doc but not in HTML mode in terminal mode then the CGI script in Perl with regular expressions not even with a with a library would parse that terminal output manually converted to HTML incredibly ugly the new manned CGI is one single C program yes some people do write CGI programs in C and it links in just the components needed the man doc parses the database client code and the HTML formata code and directly generates clean HTML code with the benefit of providing full semantic search capabilities so just at the command line like at the command line in the web interface you can use search for all the semantic gimmicks one thing that was surprising about this manned CGI was that even though when I started at last year's main hackathon in Yubilana all the underlying functionality was already quite mature and ready there were was about a dozen different components to to tweak so the configuration syntax for this thing had to be tweaked so what I say it is that there were even in the even though we had a good mature code base to really exchange the manned CGI completely a surprisingly large number of small things had to be fixed and adopted so even in a seemingly small project you can sometimes be prepared to have to do a lot of work but in the end it paid off because quite a few of the features we originally implemented for the web manual viewer turned out to be useful for the command line to one of the new command line options derives from that the code for that was originally developed for the web was then reduced for the implementation of the man command I talked about before even the the way the apropos command searches results was originally developed for the web viewer so there were quite a few quite a few benefits but one thing we completely overlooked at first is that even if you are not doing anything with HTTPS or authentication or limited access whenever you do anything on the web you at once get into security issues simply because you are taking untrusted data off the net and you are processing it the processing it itself could be harmful even if it only puts Lord on your server and the output you throw at the user could be harmful to XSS or whatever so at some point we decided that we had to audit the man CGI code for security issues and the way we did that was in three ways first starting with all the untrusted input tracing forward and looking what is this input used for on the other hand starting from the other end locating all the places where the CGI is printing output to the user and tracing backward where is that data coming from is it coming from sources could it be clobbered some way and given that there are two modules that need to be audited the steering program and the formator we also identified all the places where data is transferred from one module to the other and started auditing from that interface into both directions of course all these these tracings hopefully end up in the same code paths but you really don't want to miss any code path so it's good to have a bit of redundancy in such an audit initially almost all the security issues found were reported by Sebastian Marie who by the way in the meantime has become an open BSD developer but I read it all the audit to make sure that nothing was missed and I'm now reasonably confident that we found most things that shouldn't be in a program run on the web so Baptiste Darussin is planning to replace the free BSD man CGI to and he's planning to use this exact code here are a few here is an overview of the security act issues we actually found one important class was unvenuated input unvenuated input in particular in the URI both in the path provided to the URI in the URI and in the query string that we led to two kinds of problems first reading unrelated files from the file system on the server and possibly disclosing content of files that were never intended for display and on the other hand information disclosure in in error messages so even when the program realized okay this is strange I shouldn't be doing that and the error messages might reveal stuff to the to the attacker he shouldn't know and of course the fixes were rejecting absolute paths rejecting ascension to parent directories validating stuff upfront paying attention what we displayed in error messages and the other type of problems were mostly cross-site scripting issues partly due to invalid characters embedded in query strings and partly due to the stuff embedded in manual pages I mean when you run a manual page server you will previously for example also serves manual pages from ports and you don't really know what people put into manual pages in ports so that should not be able to trigger cross-site scripting attacks in your CGI front-end basically all that these CGI these excess s things require getting the encoding right which turned out to be quite tricky because some of the output needs html encoding some needs uri encoding some even needs both so it wasn't only a question of doing the encoding of the at the rice places but doing the right number of encodings and choosing the right encoding at the right places but I guess we figured it out the end one thing that is almost impossible to fix is a regular expression does attacks we prove at on the command line we allow people to search in manual pages using ready regular expressions and we wanted to have the same functionality on the web so people can enter regular expressions into that thing and regular expressions are so powerful that you can't really prevent them from clobbering server resources so the only mitigation we came up is just limiting the total time a CGI process can run and so far nobody has brought the server down I hope it can stay like that if if it doesn't we might have to switch off regular expressions there's probably no better way okay so so much about the various display tools yes you're welcome it's extended regular expressions by the standard routines contained in the C library might be I must admit I'm not really up to date whether those are less easy to exploit that might make sense on the other hand getting on the nerves of people with restricting to be our yeah but it might be better than switching if really we suffer from attacks then we should probably consider that it's a nice nice idea yeah okay so so far about the various viewers now let's get to some things about parsing and actual formatting the main progress last year was made in the area of mathematical equations in manual pages now I admit that the EQN language is not used as much as m doc man and TBL but there are some manual pages containing mathematical equations in particular in xorg and a bit in the mouse libraries the parsing works quite well chris steps the mostly finished that in 2011 but the the formatting was really ugly and we had to apply some paint to that in HTML output chris steps rewrote the output module to generate math ml in the context where he also switched the output to HTML 5 that was actually quite straightforward the parse streak falling out of his EQN parser can be translated one to one to to math ml it's less than 200 lines of code and the the output looks quite beautiful in a graphical browser just look at that on the open BSD website in the open manuals look at a few x manuals containing matrices and so it works quite well terminal output at first sight seems seems harder how do you format mathematical in equations on an ASCII terminal and what GNU EQN does is they try to move elements up a line a down a line and try to draw lines from minus signs and such stuff and the results are just unintelligible it doesn't work at all so I chose a different approach and rewrote the terminal output as a linear textual representation and here you have a few examples of how fractions and matrices and functions look like admittedly that's not pretty but at least you can figure out what it means and that is the main thing for manuals so the status now is that mandoc actually formats equations much better than GNU EQN both on the terminal and for HTML while PostScript and PDF is still the domain of of the full thing of GNU EQN another thing about processing and formatting is a internationalized manuals multi-byte characters now admittedly non-English manuals have a lot of problems they are they are rare they are hard to maintain even if you try to maintain them then they tend to get outdated and once they are outdated they are arguably worse than nothing however it doesn't have at all if in addition to all these problems the tools hinder reading them for that reason Christops quite early implemented basic UTF-8 support but in the same way as it's done in graph it required a pre-processor to transform the UTF-8 input into rough escape sequences and then it required a specific output options to tell the thing okay I want UTF-8 output so in addition to all those problems you also had to when when actually viewing the manual pages you had to take care of these special options now this year I integrated the pre-processor right into mandoc so that the input encoding is automatically detected and I switched the default output mode from T ASCII to T local which means as long as you use POSIX or C local it doesn't make any difference for you but if you have your stuff set up for UTF-8 output anyway then it just works so right now viewing a Japanese or a Russian manual is no longer more difficult than viewing an English manual and I think that is how it should be okay so much for functionality now let's come to the things that don't work because probably all know programming is kind of about getting things wrong this predator here found us a lot of bugs a fuzzing tool is a program that runs another program trying to feed it varying input trying to crash it or hang it and the specific things advertised about the American fuzzy lop fuzzer program are that it does compile time instrumentation of the tested code and has genetic algorithms such that it can discover test cases itself and can execute as many functional code paths as possible with a goal of full functional coverage now getting full functional coverage for terminal output in mandoc on modern PC hardware takes several days of round the clock running but that it's exactly what Jonathan Gray an open BSD developer down in Australia did repeatedly since end of last year and I was a bit surprised that he found more than 40 issues grand total now what were these about a third of them were cases where we assumed that our data structures had certain invariance and these were actually violated there were cases of general environments very violated that way and cases of macro specific ones another third were logic errors that were just arising from excessive complexity of the code in part unavailable because the language design is so complex the the topic here is badly nested block in the m doc language I talked about that four years ago I can't repeat it for time constraints partly because of complexity in the implementation the specific thing that caused these bugs was macro rewinding so you open a block and then don't close it again or you close a block that is not even open all that has to be handled and if you do it that in creative ways then it might crash the program so these were two-thirds invariance and complexity the remaining third where the things you expect to cause security issues like missing input validation buffer overflows use after free so it's not only interesting to look at the causes of bugs but also the severity of the issues isn't one of the things that you're supposed to do is not write your own string parsers right your own string parsers well it's I don't know I mean here but everybody there you know they wrote their own string handling routines and that was the source of all the problems as soon as we adopted open BSD's Lipsy the problems went away well the the routines available in the in the open BSD Lipsy are mostly routines for concatenating and copying strings like StrelCut and StrelCopy but I'm not aware of any open BSD specific string parsing routines so and that yeah but but you are right that the top the top five issues found were really buffer overruns where a string parser parsed the string that was broken in such a creative way that it skipped over the final zero and then if you if you return that parser to the calling code and that calling code might modify the buffer it can even end up as a write buffer overrun okay so kind of one lesson learned here is that I mean this code was written by moderately experienced people in the context of open BSD so you should assume they were aware that there are dangers and still these things were in there and what is particularly interesting is that this wait I'm just losing power the particular interesting thing was that the easier the stuff would have been to avoid the more dire the consequences so if you just if you pay attention from the beginning and then after a certain time audit the code once again looking for the simplest thing like being careful when passing back when passing around pass pointers making sure you don't pass zeros watching out for use after free not forgetting to validate input being careful with arithmetic operations you will probably still find stuff and find things to fix some of the con keep in mind all this in code written with security in mind from the first point it can't be stressed enough that some of the well-known things come up again and again the largest numbers of bugs and absolute numbers were in the most complex code so yes it does pay off to avoid complexity if you can the distribution of the bugs across the various modules was more or less proportional to the size of the modules not not a big surprise but yes it does pay off to keep code small so in this case we were about one dot five serious bugs per thousand lines of code something between half and three serious bugs per line of code is a range that you might expect in this case we had a few aggravating factors in particular the languages we are parsing have no formal definitions that's nothing has really ever been written down clearly cleanly they are not designed according to any strict paradigms but rather evolved historically so the part of the requirements and the design goals weren't known from the start but discovered piecemeal so again and again we had to break existing logic to change existing logic and existing invariance that might have contributed to to to this thing one thing that would have helped tremendously I didn't find the time yet but if you are working on an important project and have the time to spend the effort I would recommend to explicitly specify for all your mayor data structures which invariance you intend to guarantee and then audit your code whether all places changing these data structures actually respect the invariance and say audit your code whether all places reading from the data structures don't assume anything else except those invariance you have explicitly specified in the case of mandoc that would have caught about a third of all those issues which is quite a substantial fraction okay so well broken stuff not only code is broken manuals are broken too now what do we do it about it well we tell the authors we tell the portes for that we have three message levels in mandoc the lowest one is warnings a warning means the author should be aware that the quality of his code could be improved it's clear what he means but it might cause portability problems with older tools or whatever an error means the author has written something but we don't really know what that means it's inconsistent it's it's likely that information gets lost the user doesn't see the full text is intended to see others the the formatting might be completely clobbered that's an error this year I introduced the third level which is called unsupported that's not so much for manual authors but for porters it tells that mandoc has the impression okay this is probably valid code but I know I can't handle that yet so better use graph for formatting this particular manual and there was historically a fourth level and I'm quite proud of finally after five years having gotten rid of it it was called fatal that means you threw a manual at mandoc and it replied no that's so weird I don't give you any output well there's some text in it so it should display that text and we've finally reached that whatever you throw at mandoc you get some output even if it's empty it no longer there's nothing good so in the base system the problem is not the problem of broken manual pages is not really hard if you find a broken manual in the base system you fix it and are done with it but in ports that's not really an option you can try sending patches upstream but not likely that something will happen so good news is that by now after several years of development of mandoc about 95% of ports manuals just work but what about the remaining 5% in open BSD we mark those ports where the manuals don't work with mandoc with a use graph variable in the make file and there are still about 200 such ports these manuals are form a pre-formatted at port build time and the formatted versions are packaged the advantage obviously is that end users get perfect manuals for every port but well from a formatting perspective the content might still be off the inconvenience is that you need support in the ports infrastructure for such a thing mark SP has written that years ago and it works and the porters need to maintain this use graph variable for every single port for that reason to avoid this work free BSD has chosen a different way what they do is the man program doesn't run mandoc right away but first asks mandoc what do you think about this manual page can you deal with that and then if mandoc says you are that looks good it's run again that time for real and if mandoc says no I don't think I don't like that then growth is run instead inconvenience are considerable of course because at the time you run the man command the page has to be parsed twice that costs time and particularly bad in case mandoc doesn't realize it's unfit for the job it's too confident the user gets incomplete it on miss formatted output and on the other hand if mandoc is shy and says no I don't want that even though it could handle it then time is wasted on running graph so that's kind of a trade-off in which way you do it net BSD has a very creative way to handle that if I understand correctly that they just ignore the whole problem and it seems to be good enough for them I don't hear complaints about those probably about 5% of broken ports manuals in net BSD well in the future we might improve it in two ways one way would be to improve low-level Roth support in mandoc and to remove use graph from various open B is deports another way would be to improve the W unsupported logic such that the number of problems in free BSD is reduced and at least nutty so Christian Weisgerber a colleague in open BSD myself hope that these two ways will ultimately converge so that we can go the net BSD way and everything just works with mandoc but we'll still take some time okay at this point I'm through with the mandoc toolbox in the strict sense now I'm talking about one of the companion tools a converter from the pro pot format to the M dog format because in that area we have made quite some progress during the last few months why is pro pot relevant well after the M dog format used in BSD's and the man format used in Linux I guess that's the third most used format for for manuals it is used by Pearl it is used by open SSL it is used by FF M peg by various projects usually these pages are converted to the old man format by the pot to man program which is itself written in pearl the downsides are that you get no semantic searching and that the developers have to learn another formatting language the pro pot language which is less powerful so learning two languages and one is even less powerful than the other doesn't make a lot of sense so we have decided to convert the libre SSL manuals from pro pot to M dog Anthony Bentley has done half that work already last year and it is committed so the lip SSL manuals are done I'm currently working on the lip crypto manuals to gather with a guy from Düsseldorf Max Fillinger for Anthony it was still quite hard because he only had a prototype of the pot to M dog tool and it required a lot of manual post processing I've now improved a lot of details in particular in things like white space and closing punctuation and quoting you might say well that all seems quite minor but you need to keep in mind that the goal is to commit the converted manuals so the generated code must be clean and maintainable because after that developers are going to hand edit it for decades maybe so I'm cleaning it up by hand is quite tedious we are talking about hundreds of manuals here but admittedly when talking about improvements of this pot to M dog converter the conceptual things are even more interesting keep in mind that the pro pot format has no semantic markup it basically only says things about the physical formatting bold and it'll like and so on but in the output in the M dog output you want semantic markup and I've written some heuristics that look at really at the text where there are parent thesis and commerce and blanks and figure out oh this might be a function declaration in particular when it's in the synopsis and add the missing markup on the fly for function types and function names and function arguments and so on and not only that but it also uses hash tables using the ohash library written by mark SP and remembers the names such that when later in the description these function names and argument names reoccur they can be found in the hash tables and the correct macros can be inserted in the text which considerably reduces the amount of manual post processing that needs to be done I released all that last month so it's quite new one thing that is interesting about this is that it is somewhat similar to stuff Eric Raymond has done in the context of of the tool he's using for converting man manuals to dog book how is it called that thing I've forgotten the name but yeah so a dream for the future is that we might use similar logic like the one developed here in the future to extract semantic information from man manuals and enable the semantic searching that we have for mdoc documents for man too but there is no clear concept yet how to do that it's just an observation that the the algorithms needed points extracting these information from various formats be pro pot or man or whatever are quite similar good so let's get to the status in open DSD most of the work was already done last year so there is a lot that was complete by 2014 in particular man dog is the only documentation formata in open BSD base now for almost five years and the search tools had have been switched before last year's conference the main progress here since last year is the new online interface the unified interface for the formata and search tools the switch of the manual viewer to the new implementation and all that is released last month with open BSD 5.7 so even if you install open BSD stable now you get all that part of that will only be available in the next not in the next but in the really a free BSD release after that free BSD 12 but free BSD has made even more progress than open BSD this year last year I could only say okay they have it in base but don't use it right now but is that was so has done tremendous work he has switched the default formata end of last year he's using the unsupported option I explained for ports manuals since March this year he has switched the search tools a week ago the code in free BSD is completely up to date with the latest stable man dog release all that is going to be released with free BSD 11 and the only thing he postpones until free BSD 12 is switching the man implementation because he's sensible enough to say we shouldn't change everything at the same time that might alienate users people might get upset if everything changes at the same time but very impressive progress here in free BSD unfortunately in net BSD and dragonfly almost nothing happened net BSD is using man dog as the default format are longer than free BSD but they don't have semantic search tools they have their own search implementation which is doing full-text search but lacking semantic search and dragonfly is still having it but not using it another system that made impressive progress is Illumos in previous years we often said we often cited Solaris based systems as examples of very old very traditional stuff that didn't even have any M.Doc implementation Illumos has decided to bit by bit translate all their manuals from man to M.Doc so the same thing Cynthia Livingston in BSD did in 1990 they are now doing that too I it will be interesting to see whether they completed in one year like she did I guess not but Garrett Amor switched the default formator in the same commit as he imported M.Doc into the base system so now they have M.Doc and they are using it not the newest version but it seems to just work for them that's the third system grand total who did the the switch and the first non-BSD system in Linux there are surprisingly there are two distributions who are completely relying on M.Doc both are very small ones Alpine Linux and Void Linux but they have everything they have the search tools they use them the manual viewer they have the latest release Alpine Linux was the first non-BSD system ever to use the the M.Doc based man and there are a few others Arch Linux has an official port slackware and crooks have unofficial one but none of the major Linux's really has picked up anything so far even though ports have been available for all of them unofficially and I'm regularly testing on Debian before releases we shall see what happens there yeah okay it would be nice it would somehow fit the Arch Linux philosophy more or less well let's just be patient and if anybody if they need help they should just come to us to Christophe's and myself and try to help them if they need any help other operating systems Minix has it in base but is somehow completely apathetic since five years or something there is a some kind of the user community or several user communities in OS X there are ports available both on homebrew and Mac ports there is even a halfway up-to-date port for for Windows such information is periodically updated on our website so you can see the status also between conferences so the status summary is fully integrated in open BSD alpine Linux void Linux except for man also in free BSD current the default format in net BSD and Illumos it's at least in the base system in free BSD 10 dragonfly and Minix official packages exist for free BSD 9 Arch Linux and PKG source and then there are a few having unofficial packages or outdated packages I'm regularly announcing goals for the future at conferences of those that were announced for were reached this year the replacement of the man CGI the integration of pre-con the switch to local output by default and replacing man in open BSD several things are in progress we are working on the Lib crypto manuals in Libre SSL and improving part to MDoc to facilitate that we are I'm unifying the parses aiming for better Roth support in the future that's a very complicated subject that I really couldn't cover in this talk if I were to talk about that I would have to give a full talk maybe sometime later the unsupported mode is still being improved we want to at some point delete all those redundant hard links in the file system as saying free BSD already does one thing free BSD already did a month ago but I only learned about it yesterday so it's not in the slides yet is that tech info is no longer in free BSD but is that was saying use the tech info to end up utility by Chris steps to convert all the tech info documentation to MDoc I'd like to do that in open BSD too this is a very nice example of free BSD leading the way actually what's a bit stalled is providing help with the man to end of conversions but that can be picked up again it's basically what Chris steps has started with the dog book to end up to it's mentioned two things that have not yet been started at some point I dream of using pot to end up but not for convergence with manual post processing that inside build system instead of running pot to man on the pearl manuals we could run pot to MDoc convert all the pot pearl manuals to MDoc format and then we would get semantic searching in pearl manuals in the open BSD base system for free and another thing one thing that is good about info not talking about all the problems that info has but one thing that is good there is internal linking with the manual pages so long as we don't change the basic way manual pages are built self-contained linear easy to navigate by hand if we get additional options for linking inside manual pages that would help a bit and one idea to do that was functionality similar to C tags that could be integrated into less but that really needs to be worked out to conclude I'd like to thank a few people Chris steps of course the original author of mandoc who again contributed quite some code this year for example the new eq u.m. parser html 5 and martin l output then of course Jonathan Gray who did extensive testing with AFL reporting more than 40 important bugs but he's done was so for tremendous work on previous the system integration and also sending source code patches Christian Weiss Gerber Nadia open BSD for removing use graph from many ports and helping with open BSD porting work Thomas Klausner with net BSD for PKG source maintenance while net BSD is lagging a bit in the base system Thomas is doing excellent work on PKG source Nataniel Koopa Alpine Linux who did system integration there and proved that mandoc can be used as the system manual from turn Linux Paul only shook also Alpine Linux suggested the implementation of man if it weren't for Paul who said oh why don't you implement man and I said well that's a bad idea last year at BSD can I say that's out of scope but then I stepped back and thought well actually why not we already have almost all the code that is needed it just needs to be shuffled a bit this year there were quite a few people who contributed patches and of course again even more people who reported bugs or suggested features
 Welcome to my presentation on Mandok. The topic this year is becoming the main  VSD manual toolbox. My name is Ingo Schwarzze, Schwarzze at OpenVSD.org and I  have been an OpenVSD developer for the last six years. While I've also  contributed in a few other areas, it happened that my main focus became  documentation tools. One thing that's always nice for starting a talk is  reminding everybody that we are usually standing on the shoulders of giants and  the pioneer in this area is really Cynthia Livingstone. You are seeing here on  this picture, she designed the main documentation language we are still  using today in 1999-1990, the M-Doc language. She implemented that language  herself. She translated the whole corpus of BSD manuals from the old man  language to the new M-Doc language and in the process she also rewrote all the  text that was still encumbered by AT&T copyrights. So all that by one single  person. When I talked here about Mandok four years ago, when I first presented  the Mandok toolbox, my focus was this is a completely new tool and we have to  train it to do real work and you'll see how the beast has matured until now.  Okay, so the key point, we are talking about documentation tools. The key point  from my point of view about system documentation is really that all  documentation should be in one place and in one format, so not have a part of the  web and a part in HTML and a part in user share docs and whatnot, one place, one  format. That makes it easy to find, easy to read and easy to write and only if  it's easy to write there is any chance that it will be correct, complete and  concise. That of course puts a particular focus on which system to use. If  you want to use one system, the basic markup syntax we are still using today  goes back more than 50 years. Jerry Salzer started the Roth-Ranoff markup in 1964.  It is unobtrusive, it's diff-friendly, it's easy to hand edit and there are simple  tools to produce high-quality output in various formats from them. The basic  manual structure goes back to the very first version of Unix to Thompson and  Ritchie. The manual language still used in Linux today comes from the famous  version 7 Unix, the last resourced Unix version that is publicly available, but  the real revolution in documentation languages really was the invention of  semantic markup of the M-Doc language in 8990 by Cynthia Livingston, which then  got to the world with 4.4 BSD. At about the same time James Clark wrote the  GNU implementation of Trof, which even though its GPL software dominated the  toolbox world in the BSDs for more than two decades until finally around 2010. We  started to introduce the BSD licensed Mandoq toolbox step-by-step into the  various BSDs by now. Practically all major BSDs use it, open BSD, free BSD, net  BSD, even Illumos has switched to it. Now what is in that toolbox we are talking  about from the user perspective it has become really simple. From the user  perspective you basically have one user land program that you are calling MAN,  the manual viewer. That thing when you call it once there's three things. It  finds one manual page in the file system or using a database. You can decide  which file to find either by giving its name or by giving a search query. Then in  the second step MAN will format the manual page and in the third step it  will display it usually using a pager. Even in this very elementary area there's  quite some progress since I presented here last year. Last year I said MAN, the  manual viewer is out of scope that's not part of the toolbox. Now we have one  unified interface for both for the viewer, for MAN, for the formator, for  MAN doc and for the search tool apropos. They all take the same options and very  new this year 2015 we have a unified and very much simplified configuration  file format that I'll show. The toolbox also contains a few auxiliary  components. You have a database generation tool, you have a syntax checker, you  have a parse tree debugger, you have a format converter from MAN to M doc that  is built in, pardon from M doc to MAN that is built in and you have output  front ends for various formats like HTML, post-crypt, PDF. So before really  starting with the individual topics I give you a very brief overview which  topics I'll talk about. I guess I never talked about so many different topics in  a single public talk. It's just because so much happened in the last year.  And I hope I won't get mired in that swamp of topics. The unified user  interface, the new viewer and configuration file format is the first thing,  the second is the same for the web, the third is improved formatting of  mathematical equations, then improved unicode support, hunting bugs with a  fuzzer program, detecting use of unsupported features, converting manual  pages from another language from the pearl documentation language to our  common M doc language. These are the main subjects, the main things that  happened during this year and I will wrap up providing a status in various  operating systems and hinting at a few possible future directions. By the way,  the pictures I'm using for illustrations are pictures taken by other people along  the road of the bicycle tour I did around Southern Ontario just after last year's  conference. Okay, so first thing we did, OpenBSD no longer uses the traditional  BSD MAN program but an implementation integrated into Mandoch into the  formator. The traditional setup was that MAN would fork and execute twice, once to  call the formator and then again to page the output. Right now the program  finding the files in the file system and formatting is the same program. Now,  which is the point in doing that? The point really is to have a unified  interface for all the three main front ends, which means that when you call  the viewer, when you type MAN, some page, you can use command line options that  were traditionally only available to the formator, like saying which warning  level you want to see or saying which output format you want. You can now do  things like in OpenBSD, not yet in FreeBSD, that will only be available in  FreeBSD 12. You can say things like MAN minus T, HTML, just the name of the  manual and then pipe it directly to links or something. On the other way around,  the search tool now has access to options that traditionally you only had in the  viewer. The search tool apropos normally just lists a number of title lines for  manuals. You now have options to say I want to see the file names instead or I  want to see the command synopsis instead or even I give it a search query but I  want to see all the full manual pages in one less right away. So that's quite  flexible and you don't need to remember different options for different programs.  Besides, it allows a simpler configuration file format which we  use for you minor things that I'm not going to name individually except that  one of them is quite nice. We have to maintain one less userland program and  the traditional BSD MAN is quite old code so it's nice to no longer need to  maintain that. There are two other things that we can gain in the future that we  have not yet exploited. In particular, many library manuals document not only  one but two, three, twenty functions and those typically have multiple entries in  the file system hard links. With the new database we can get rid of all those  hard links and can get rid of thousands of files in the in the installation and  another nice thing that can be done with this it is already implemented it's only  not integrated yet in OpenBSD. We can have an interactive chooser so you say  apropos something it comes up with a list of programs that match and then you  can just choose one of them to open the manual directly without exiting apropos  first and typing a new command. Okay any change of a program for a new program  will come at a cost. In this case the cost is that database lookup is slightly  slower than file system lookup. Then again on my notebook the additional delay  for displaying a manual page is on the order of 10 milliseconds. There is another  cost when you install a new manual page on the system and you want to find that  new manual page with the search tools you have to update the database but the  at least the OpenBSD package tools run the required commands automatically for  you so you don't need to do it manually and even if you forget it or it doesn't  work then the man tool itself will still work you will still see the manuals  when you explicitly call for them you just won't find them in the search tool  until the weekly make what is run. So there are very little downsides if any  compared to the additional features we get. Oh this is a nice one the old  configuration file format of the manual viewer this is the list of all the  features I identified in the old configuration file format that  are completely useless. I'm not going to bore you reading that this to you I'll  just pick out one you can configure decompression filters like if I have a  G-SIP man page I want to use GUN-SIP on it. Well the old configuration file  format allowed you to configure different decompression filters in different  sections of the manual so if I have a kernel manual that is G-SIP I want to  use this decompression filter if I have a user-length manual this G I prefer the  other decompression filter and dozens of them so the format was so complicated  that really consistently everybody hated it all both nobody used it even  though people had good uses for it some things you want to configure so I came  up with a new configuration file format that basically has just two directives  the things that people actually need one is you can specify a man pass you give  a directory name it takes that directory as a complete tree of manual pages on  consistently uses that tree across all the tools and the other directive is  specifying output options for example for the terminal saying how wide the  terminal is for HTML saying which style sheet you want to link from the  generated file for PostScript.pf specifying the paper format things  like that that's very easy to use and needs almost no learning a few a bit of  the namespace in the new configuration file format has been reserved so once  people ask for it I'm planning to implement the following features an  alias directive which makes it easier for people using languages like TCL or so  to make whole trees more easily accessible with the minus s option then a  sections directive that allows people to configure custom sections and to change  the search order of sections and the filter directives in case any operating  systems are using other compression formats besides GZIP however so far  nobody asked for these formats and as long as people don't need them I say  kiss keep it stupidly simple I we shouldn't implement anything that people  don't actually want so be aware here I really reduced the functionality inside  open BSD and people don't yell at me so it's not always bad to make things  simpler good so far I talked about viewing manual pages on the command line on  the terminal we also have a CGI for viewing them on the web which has  basically identical functionality open BSD on the WWW open BSD org website no  longer uses the traditional manned CGI Perl script by Wolfram Schneider from  free BSD but a manned CGI implementation included in the manned doc toolkit the  traditional setup was that manned CGI would fork and execute the systems man  command that man command would fork and execute a graph or later man doc but not  in HTML mode in terminal mode then the CGI script in Perl with regular  expressions not even with a with a library would parse that terminal  output manually converted to HTML incredibly ugly the new manned CGI is  one single C program yes some people do write CGI programs in C and it links in  just the components needed the man doc parses the database client code and the  HTML formata code and directly generates clean HTML code with the benefit of  providing full semantic search capabilities so just at the command line  like at the command line in the web interface you can use search for all  the semantic gimmicks one thing that was surprising about this manned CGI was  that even though when I started at last year's main hackathon in Yubilana all the  underlying functionality was already quite mature and ready there were was  about a dozen different components to to tweak so the configuration syntax for  this thing had to be tweaked so what I say it is that there were even in the  even though we had a good mature code base to really exchange the manned CGI  completely a surprisingly large number of small things had to be fixed and  adopted so even in a seemingly small project you can sometimes be prepared  to have to do a lot of work but in the end it paid off because quite a few of  the features we originally implemented for the web manual viewer turned out to  be useful for the command line to one of the new command line options derives  from that the code for that was originally developed for the web was then  reduced for the implementation of the man command I talked about before even  the the way the apropos command searches results was originally developed for the  web viewer so there were quite a few quite a few benefits but one thing we  completely overlooked at first is that even if you are not doing anything with  HTTPS or authentication or limited access whenever you do anything on the  web you at once get into security issues simply because you are taking  untrusted data off the net and you are processing it the processing it itself  could be harmful even if it only puts Lord on your server and the output you  throw at the user could be harmful to XSS or whatever so at some point we  decided that we had to audit the man CGI code for security issues and the way we  did that was in three ways first starting with all the untrusted input  tracing forward and looking what is this input used for on the other hand  starting from the other end locating all the places where the CGI is printing  output to the user and tracing backward where is that data coming from is it  coming from sources could it be clobbered some way and given that there are two  modules that need to be audited the steering program and the formator we  also identified all the places where data is transferred from one module to  the other and started auditing from that interface into both directions of  course all these these tracings hopefully end up in the same code paths but  you really don't want to miss any code path so it's good to have a bit of  redundancy in such an audit initially almost all the security issues found  were reported by Sebastian Marie who by the way in the meantime has become an  open BSD developer but I read it all the audit to make sure that nothing was  missed and I'm now reasonably confident that we found most things that  shouldn't be in a program run on the web so Baptiste Darussin is planning to  replace the free BSD man CGI to and he's planning to use this exact code here  are a few here is an overview of the security act issues we actually found  one important class was unvenuated input unvenuated input in particular in the  URI both in the path provided to the URI in the URI and in the query  string that we led to two kinds of problems first reading unrelated files  from the file system on the server and possibly disclosing content of files  that were never intended for display and on the other hand information  disclosure in in error messages so even when the program realized okay this is  strange I shouldn't be doing that and the error messages might reveal stuff to  the to the attacker he shouldn't know and of course the fixes were rejecting  absolute paths rejecting ascension to parent directories validating stuff  upfront paying attention what we displayed in error messages and the other  type of problems were mostly cross-site scripting issues partly due to  invalid characters embedded in query strings and partly due to the stuff  embedded in manual pages I mean when you run a manual page server you will  previously for example also serves manual pages from ports and you don't  really know what people put into manual pages in ports so that should not be  able to trigger cross-site scripting attacks in your CGI front-end basically  all that these CGI these excess s things require getting the encoding right  which turned out to be quite tricky because some of the output needs html  encoding some needs uri encoding some even needs both so it wasn't only a  question of doing the encoding of the at the rice places but doing the right  number of encodings and choosing the right encoding at the right places but I  guess we figured it out the end one thing that is almost impossible to fix  is a regular expression does attacks we prove at on the command line we allow  people to search in manual pages using ready regular expressions and we wanted  to have the same functionality on the web so people can enter regular  expressions into that thing and regular expressions are so powerful that you  can't really prevent them from clobbering server resources so the only  mitigation we came up is just limiting the total time a CGI process can run and  so far nobody has brought the server down I hope it can stay like that if if it  doesn't we might have to switch off regular expressions there's probably no  better way okay so so much about the various display tools yes you're welcome  it's extended regular expressions by the standard routines contained in the C  library might be I must admit I'm not really up to date whether those are less  easy to exploit that might make sense on the other hand getting on the nerves of  people with restricting to be our yeah but it might be better than switching if  really we suffer from attacks then we should probably consider that it's a  nice nice idea yeah okay so so far about the various viewers now let's get to some  things about parsing and actual formatting the main progress last year  was made in the area of mathematical equations in manual pages now I admit  that the EQN language is not used as much as m doc man and TBL but there are  some manual pages containing mathematical equations in particular in  xorg and a bit in the mouse libraries the parsing works quite well  chris steps the mostly finished that in 2011 but the the formatting was really  ugly and we had to apply some paint to that in HTML output chris steps rewrote  the output module to generate math ml in the context where he also switched the  output to HTML 5 that was actually quite straightforward the parse streak falling  out of his EQN parser can be translated one to one to to math ml it's less than  200 lines of code and the the output looks quite beautiful in a graphical  browser just look at that on the open BSD website in the open manuals look at a  few x manuals containing matrices and so it works quite well terminal output at  first sight seems seems harder how do you format mathematical in equations on  an ASCII terminal and what GNU EQN does is they try to move elements up a line  a down a line and try to draw lines from minus signs and such stuff and the  results are just unintelligible it doesn't work at all so I chose a different  approach and rewrote the terminal output as a linear textual representation and  here you have a few examples of how fractions and matrices and functions  look like admittedly that's not pretty but at least you can figure out what it  means and that is the main thing for manuals so the status now is that  mandoc actually formats equations much better than GNU EQN both on the terminal  and for HTML while PostScript and PDF is still the domain of of the full thing of  GNU EQN another thing about processing and formatting is a  internationalized manuals multi-byte characters now admittedly non-English  manuals have a lot of problems they are they are rare they are hard to  maintain even if you try to maintain them then they tend to get outdated and  once they are outdated they are arguably worse than nothing however it  doesn't have at all if in addition to all these problems the tools hinder  reading them for that reason Christops quite early implemented basic UTF-8  support but in the same way as it's done in graph it required a pre-processor to  transform the UTF-8 input into rough escape sequences and then it required a  specific output options to tell the thing okay I want UTF-8 output so in  addition to all those problems you also had to when when actually viewing the  manual pages you had to take care of these special options now this year I  integrated the pre-processor right into mandoc so that the input encoding is  automatically detected and I switched the default output mode from T ASCII to T  local which means as long as you use POSIX or C local it doesn't make any  difference for you but if you have your stuff set up for UTF-8 output anyway  then it just works so right now viewing a Japanese or a Russian manual is no  longer more difficult than viewing an English manual and I think that is how  it should be okay so much for functionality now let's come to the  things that don't work because probably all know programming is kind of about  getting things wrong this predator here found us a lot of bugs a fuzzing tool is  a program that runs another program trying to feed it varying input trying  to crash it or hang it and the specific things advertised about the American  fuzzy lop fuzzer program are that it does compile time instrumentation of the  tested code and has genetic algorithms such that it can discover test cases  itself and can execute as many functional code paths as possible with a  goal of full functional coverage now getting full functional coverage for  terminal output in mandoc on modern PC hardware takes several days of round the  clock running but that it's exactly what Jonathan Gray an open BSD developer  down in Australia did repeatedly since end of last year and I was a bit  surprised that he found more than 40 issues grand total now what  were these about a third of them were cases where we assumed that our data  structures had certain invariance and these were actually violated there were  cases of general environments very violated that way and cases of macro  specific ones another third were logic errors that were just arising from  excessive complexity of the code in part unavailable because the language  design is so complex the the topic here is badly nested block in the m doc  language I talked about that four years ago I can't repeat it for time  constraints partly because of complexity in the implementation the specific  thing that caused these bugs was macro rewinding so you open a block and then  don't close it again or you close a block that is not even open all that has  to be handled and if you do it that in creative ways then it might crash the  program so these were two-thirds invariance and complexity the remaining  third where the things you expect to cause security issues like missing input  validation buffer overflows use after free so it's not only interesting to look  at the causes of bugs but also the severity of the issues  isn't one of the things that you're supposed to do is not write your own string parsers  right your own string parsers well it's I don't know I mean  here but everybody there you know they wrote their own string handling routines  and that was the source of all the problems as soon as we adopted open  BSD's Lipsy the problems went away well the the routines available in the in the  open BSD Lipsy are mostly routines for concatenating and copying strings like  StrelCut and StrelCopy but I'm not aware of any open BSD specific string  parsing routines so and that yeah but but you are right that the top the top  five issues found were really buffer overruns where a string parser parsed  the string that was broken in such a creative way that it skipped over the  final zero and then if you if you return that parser to the calling code and  that calling code might modify the buffer it can even end up as a write  buffer overrun okay so kind of one lesson learned here is that I mean this code  was written by moderately experienced people in the context of open BSD so  you should assume they were aware that there are dangers and still these things  were in there and what is particularly interesting is that this wait I'm just  losing power the particular interesting thing was that the easier the stuff would  have been to avoid the more dire the consequences so if you just if you pay  attention from the beginning and then after a certain time audit the code  once again looking for the simplest thing like being careful when passing  back when passing around pass pointers making sure you don't pass zeros  watching out for use after free not forgetting to validate input being  careful with arithmetic operations you will probably still find stuff and find  things to fix some of the con keep in mind all this in code written with  security in mind from the first point it can't be stressed enough that some of  the well-known things come up again and again the largest numbers of bugs and  absolute numbers were in the most complex code so yes it does pay off to  avoid complexity if you can the distribution of the bugs across the  various modules was more or less proportional to the size of the modules  not not a big surprise but yes it does pay off to keep code small so in this  case we were about one dot five serious bugs per thousand lines of code  something between half and three serious bugs per line of code is a range that  you might expect in this case we had a few aggravating factors in particular  the languages we are parsing have no formal definitions that's nothing has  really ever been written down clearly cleanly they are not designed according  to any strict paradigms but rather evolved historically so the part of the  requirements and the design goals weren't known from the start but discovered  piecemeal so again and again we had to break existing logic to change existing  logic and existing invariance that might have contributed to to to this thing  one thing that would have helped tremendously I didn't find the time  yet but if you are working on an important project and have the time to  spend the effort I would recommend to explicitly specify for all your mayor  data structures which invariance you intend to guarantee and then audit your  code whether all places changing these data structures actually respect the  invariance and say audit your code whether all places reading from the data  structures don't assume anything else except those invariance you have  explicitly specified in the case of mandoc that would have caught about a  third of all those issues which is quite a substantial fraction okay so well  broken stuff not only code is broken manuals are broken too now what do we  do it about it well we tell the authors we tell the portes for that we have  three message levels in mandoc the lowest one is warnings a warning means the  author should be aware that the quality of his code could be improved it's clear  what he means but it might cause portability problems with older tools  or whatever an error means the author has written something but we don't really  know what that means it's inconsistent it's it's likely that information gets  lost the user doesn't see the full text is intended to see others the the  formatting might be completely clobbered that's an error this year I  introduced the third level which is called unsupported that's not so much  for manual authors but for porters it tells that mandoc has the impression  okay this is probably valid code but I know I can't handle that yet so better  use graph for formatting this particular manual and there was historically  a fourth level and I'm quite proud of finally after five years having gotten  rid of it it was called fatal that means you threw a manual at mandoc and it  replied no that's so weird I don't give you any output well there's some text in  it so it should display that text and we've finally reached that whatever you  throw at mandoc you get some output even if it's empty it no longer there's  nothing good so in the base system the problem is not the problem of broken  manual pages is not really hard if you find a broken manual in the base system  you fix it and are done with it but in ports that's not really an option you  can try sending patches upstream but not likely that something will happen so  good news is that by now after several years of development of mandoc about  95% of ports manuals just work but what about the remaining 5% in open BSD we  mark those ports where the manuals don't work with mandoc with a use  graph variable in the make file and there are still about 200 such ports these  manuals are form a pre-formatted at port build time and the formatted versions  are packaged the advantage obviously is that end users get perfect manuals for  every port but well from a formatting perspective the content might still be  off the inconvenience is that you need support in the ports infrastructure for  such a thing mark SP has written that years ago and it works and the porters  need to maintain this use graph variable for every single port for that reason to  avoid this work free BSD has chosen a different way what they do is the man  program doesn't run mandoc right away but first asks mandoc what do you think  about this manual page can you deal with that and then if mandoc says you are  that looks good it's run again that time for real and if mandoc says no I don't  think I don't like that then growth is run instead  inconvenience are considerable of course because at the time you run the man  command the page has to be parsed twice that costs time and particularly bad in  case mandoc doesn't realize it's unfit for the job it's too confident the user  gets incomplete it on miss formatted output and on the other hand if mandoc  is shy and says no I don't want that even though it could handle it then time  is wasted on running graph so that's kind of a trade-off in which way you do it  net BSD has a very creative way to handle that if I understand correctly  that they just ignore the whole problem and it seems to be good enough for them  I don't hear complaints about those probably about 5% of broken ports  manuals in net BSD well in the future we might improve it in two ways one way  would be to improve low-level Roth support in mandoc and to remove use  graph from various open B is deports another way would be to improve the W  unsupported logic such that the number of problems in free BSD is reduced and  at least nutty so Christian Weisgerber a colleague in open BSD myself hope that  these two ways will ultimately converge so that we can go the net BSD way and  everything just works with mandoc but we'll still take some time okay at this  point I'm through with the mandoc toolbox in the strict sense now I'm  talking about one of the companion tools a converter from the pro pot format to  the M dog format because in that area we have made quite some progress during  the last few months why is pro pot relevant well after the M dog format  used in BSD's and the man format used in Linux I guess that's the third most  used format for for manuals it is used by Pearl it is used by open SSL it is  used by FF M peg by various projects  usually these pages are converted to the old man format by the pot to man  program which is itself written in pearl the downsides are that you get no  semantic searching and that the developers have to learn another  formatting language the pro pot language which is less powerful so learning  two languages and one is even less powerful than the other doesn't make a  lot of sense so we have decided to convert the libre SSL manuals from  pro pot to M dog Anthony Bentley has done half that work already last year and  it is committed so the lip SSL manuals are done I'm currently working on the  lip crypto manuals to gather with a guy from Düsseldorf Max Fillinger for  Anthony it was still quite hard because he only had a prototype of the pot to  M dog tool and it required a lot of manual post processing I've now improved a  lot of details in particular in things like white space and closing punctuation  and quoting you might say well that all seems quite minor but you need to keep  in mind that the goal is to commit the converted manuals so the generated code  must be clean and maintainable because after that developers are going to hand  edit it for decades maybe so I'm cleaning it up by hand is quite tedious we are  talking about hundreds of manuals here but admittedly when talking about  improvements of this pot to M dog converter the conceptual things are even  more interesting keep in mind that the pro pot format has no semantic markup  it basically only says things about the physical formatting bold and it'll like  and so on but in the output in the M dog output you want semantic markup and I've  written some heuristics that look at really at the text where there are  parent thesis and commerce and blanks and figure out oh this might be a function  declaration in particular when it's in the synopsis and add the missing markup  on the fly for function types and function names and function arguments and so  on and not only that but it also uses hash tables using the ohash library  written by mark SP and remembers the names such that when later in the  description these function names and argument names reoccur they can be found  in the hash tables and the correct macros can be inserted in the text which  considerably reduces the amount of manual post processing that needs to be  done I released all that last month so it's quite new one thing that is  interesting about this is that it is somewhat similar to stuff Eric Raymond  has done in the context of of the tool he's using for converting man manuals  to dog book how is it called that thing I've forgotten the name but yeah so a  dream for the future is that we might use similar logic like the one developed  here in the future to extract semantic information from man manuals and enable  the semantic searching that we have for mdoc documents for man too but there is  no clear concept yet how to do that it's just an observation that the the  algorithms needed points extracting these information from various formats be  pro pot or man or whatever are quite similar good so let's get to the status  in open DSD most of the work was already done last year so there is a lot that  was complete by 2014 in particular man dog is the only documentation formata  in open BSD base now for almost five years and the search tools had have  been switched before last year's conference the main progress here since  last year is the new online interface the unified interface for the formata and  search tools the switch of the manual viewer to the new implementation and  all that is released last month with open BSD 5.7 so even if you install open  BSD stable now you get all that part of that will only be available in the next  not in the next but in the really a free BSD release after that free BSD 12 but  free BSD has made even more progress than open BSD this year last year I could  only say okay they have it in base but don't use it right now but is that was  so has done tremendous work he has switched the default formata end of last  year he's using the unsupported option I explained for ports manuals since March  this year he has switched the search tools a week ago the code in free BSD is  completely up to date with the latest stable man dog release all that is going  to be released with free BSD 11 and the only thing he postpones until free BSD  12 is switching the man implementation because he's sensible enough to say we  shouldn't change everything at the same time that might alienate users people  might get upset if everything changes at the same time but very impressive  progress here in free BSD unfortunately in net BSD and dragonfly almost nothing  happened net BSD is using man dog as the default format are longer than free BSD  but they don't have semantic search tools they have their own search  implementation which is doing full-text search but lacking semantic search and  dragonfly is still having it but not using it another system that made  impressive progress is Illumos in previous years we often said we often  cited Solaris based systems as examples of very old very traditional stuff that  didn't even have any M.Doc implementation Illumos has decided to bit by bit  translate all their manuals from man to M.Doc so the same thing Cynthia Livingston  in BSD did in 1990 they are now doing that too I it will be interesting to see  whether they completed in one year like she did I guess not but  Garrett Amor switched the default formator in the same commit as he imported  M.Doc into the base system so now they have M.Doc and they are using it not the  newest version but it seems to just work for them that's the third system grand  total who did the the switch and the first non-BSD system in Linux there are  surprisingly there are two distributions who are completely relying on  M.Doc both are very small ones Alpine Linux and Void Linux but they have  everything they have the search tools they use them the manual viewer they  have the latest release Alpine Linux was the first non-BSD system ever to use  the the M.Doc based man and there are a few others Arch Linux has an official  port slackware and crooks have unofficial one but none of the major  Linux's really has picked up anything so far even though ports have been  available for all of them unofficially and I'm regularly testing on Debian  before releases we shall see what happens there yeah okay it would be nice it  would somehow fit the Arch Linux philosophy more or less well let's just  be patient and if anybody if they need help they should just come to us to  Christophe's and myself and try to help them if they need any help other  operating systems Minix has it in base but is somehow completely apathetic since  five years or something there is a some kind of the user community or several  user communities in OS X there are ports available both on homebrew and Mac  ports there is even a halfway up-to-date port for for Windows such  information is periodically updated on our website so you can see the status  also between conferences so the status summary is fully integrated in open  BSD alpine Linux void Linux except for man also in free BSD current the default  format in net BSD and Illumos it's at least in the base system in free BSD  10 dragonfly and Minix official packages exist for free BSD 9 Arch Linux and  PKG source and then there are a few having unofficial packages or outdated  packages I'm regularly announcing goals for the future at conferences of those  that were announced for were reached this year the replacement of the man CGI  the integration of pre-con the switch to local output by default and replacing  man in open BSD several things are in progress we are working on the Lib  crypto manuals in Libre SSL and improving part to MDoc to facilitate that we are  I'm unifying the parses aiming for better Roth support in the future that's a  very complicated subject that I really couldn't cover in this talk if I were to  talk about that I would have to give a full talk maybe sometime later the  unsupported mode is still being improved we want to at some point delete all  those redundant hard links in the file system as saying free BSD already does  one thing free BSD already did a month ago but I only learned about it yesterday  so it's not in the slides yet is that tech info is no longer in free BSD but  is that was saying use the tech info to end up utility by Chris steps to convert  all the tech info documentation to MDoc I'd like to do that in open BSD too this  is a very nice example of free BSD leading the way actually what's a bit  stalled is providing help with the man to end of conversions but that can be  picked up again it's basically what Chris steps has started with the dog book to  end up to it's mentioned two things that have not yet been started at some  point I dream of using pot to end up but not for convergence with manual  post processing that inside build system instead of running pot to man on the  pearl manuals we could run pot to MDoc convert all the pot pearl manuals to  MDoc format and then we would get semantic searching in pearl manuals in  the open BSD base system for free and another thing one thing that is good  about info not talking about all the problems that info has but one thing  that is good there is internal linking with the manual pages so long as we  don't change the basic way manual pages are built self-contained linear easy to  navigate by hand if we get additional options for linking inside manual pages  that would help a bit and one idea to do that was functionality similar to C  tags that could be integrated into less but that really needs to be worked out  to conclude I'd like to thank a few people Chris steps of course the  original author of mandoc who again contributed quite some code this year  for example the new eq u.m. parser html 5 and martin l output then of course  Jonathan Gray who did extensive testing with AFL reporting more than 40  important bugs but he's done was so for tremendous work on previous the system  integration and also sending source code patches Christian Weiss Gerber Nadia  open BSD for removing use graph from many ports and helping with open BSD  porting work Thomas Klausner with net BSD for PKG source maintenance while  net BSD is lagging a bit in the base system Thomas is doing excellent work  on PKG source Nataniel Koopa Alpine Linux who did system integration there and  proved that mandoc can be used as the system manual from turn Linux Paul  only shook also Alpine Linux suggested the implementation of man if it weren't  for Paul who said oh why don't you implement man and I said well that's a  bad idea last year at BSD can I say that's out of scope but then I stepped  back and thought well actually why not we already have almost all the code that  is needed it just needs to be shuffled a bit this year there were quite a few  people who contributed patches and of course again even more people who  reported bugs or suggested features
mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_0_2315.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_2316_2663.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_2664_2783.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_2784_6311.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_6312_8327.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_8328_8387.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_8388_8435.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_8436_8483.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_8484_8531.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_8532_8579.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_8580_8627.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_8628_8675.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_8676_8735.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_8736_8783.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_8784_8831.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_8832_8879.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_8880_8927.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_8928_8975.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_8976_11219.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_11220_12191.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_12192_16835.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_16836_18803.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_18804_20363.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_20364_21563.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_21564_23399.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_23400_26135.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_26136_26903.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_26904_28871.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_28872_32939.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_32940_37883.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_37884_38171.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_38172_39299.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_39300_40631.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_40632_43523.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_43524_46355.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_46356_48515.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_48516_50747.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_50748_50819.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_50820_50939.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_50940_55451.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_55452_59267.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_59268_62147.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_62148_64367.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_64368_66479.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_66480_67595.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_67596_70295.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_70296_70643.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_70644_71663.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_71664_75635.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_75636_77243.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_77244_78971.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_78972_79559.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_79560_81443.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_81444_83651.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_83652_84563.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_84564_85391.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_85392_85943.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_85944_90623.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_90624_90755.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_90756_91619.png mandoc__becoming_the_main_BSD_manual_toolbox_-_BSDCan_2015_presentation_91620_92949.png