New Format for Publishing Docs

[dlavigne]

This is a long post. If you are TLDR;, please at least skim through this before complaining about the current doc transition :)

The FreeNAS project has published a User Guide for each major software version since version 8.0.1. The platform for making the documentation has been mediawiki, which allows for collaborative editing of the upcoming version. This makes it easy for anyone to contribute to the documentation as they only need to make a wiki account and mediawiki formatting is fairly quick to learn and well documented. However, this method has several disadvantages:

- the mediawiki was meant to provide an easy way for many users to edit the upcoming version of the Guide; however, since it is findable by search engines, Internet search users were finding documentation that did not necessarily match their version of FreeNAS. FreeNAS is a rapidly moving target wrt to features and the look of the GUI config screens, which is why reading the docs for the installed version is important.

- mediawiki really sucks at creating different outputs; trust me, we've tried and failed at every mediawiki plugin. It just wasn't designed to make decent looking PDFs or versioned HTML. It also doesn't really understand the concept of a Table of Contents and how to use one to generate one cohesive document. The workaround for this was to copy/paste the mediawiki content into an OpenOffice document and use that document to export to other formats. This sucked on many levels. It meant that one person (me) had to track every wiki change and make sure it made it into the OpenOffice document. While OpenOffice does a good job of generating PDFs, there are no polite words to describe what it does when creating HTML. The generated HTML was so bad, we stopped making them after the 8.3.1 release. Not having an easy way to generate HTML brings us back to that Internet search problem. While we can stress that users need to grab the PDF for their version, that's not going to mean anything when any Internet search will happily bring them to the mediawiki page for the upcoming version.

Since April, I have been converting what will be the 9.3 docs from mediawiki to reST format. This is the format used by the FreeNAS API documentation and by many other open source projects. This format offers several advantages:

- it integrates into github, meaning that the docs are included with the FreeNAS source code

- it integrates into the build system, meaning that edits to the docs can be immediately pushed to the doc website

- it automatically generates decent looking HTML and can instantly build decent looking PDFs, EPUBs, and other formats

- the formatting syntax is easy to learn

- moving forward, it solves the Internet search problem. While 9.3 is the current upcoming version, it will be the current version in the next month or so. Once 9.3 is released, we simply have to start another HTML page for the next version of FreeNAS.

It is important to note that we are currently in a transition phase. If you go to doc.freenas.org, it still provides the list of archived docs and indicates their release dates. Remember, unless you are testing nightlies, you still want to use the PDF (and possibly ERRATA) for your release. Since it took nearly 2 months of solid work to migrate the 9.3 docs into properly formatted reST, we simply don't have the man hours to convert all of the existing docs to the new format. This means that they will continue to be available in their current PDF format. For upcoming FreeNAS versions, the "nightly" HTML will be available for the upcoming version and the "published" HTML will be available with each release, starting with 9.3. Moving forward, we will also publish PDFs for each version and will see if other formats are also useful to publish.

For users who are interested in assisting with doc edits, instructions for getting started are at https://github.com/freenas/freenas-docs/blob/master/README.md. Feel free to ping me if you need help getting started. If you have a simple change you would like to see committed you can also ping me through the forums, IRC channel, or doc mailing list and I can commit the edit for you.

 

[gpsguy]

Thanks for the info. I saw the new docs this morning when @warri posted a link to: sphinx.freenas.org


Sent from my phone

 

[dlavigne]

Note that sphinx.freenas.org was recently moved to its new permanent location for the 9.3 docs: docs.freenas.org/9.3.

 

[fracai]

docs.freenas.org is just redirecting to drive.google.com and asking me to authenticate.
sphinx.freenas.org still works.

 

[dlavigne]

Oops, typo. Meant to say doc.freenas.org

 

[scott55]

When may we expect a PDF version of the 9.3 documentation?

 

[dlavigne]

I'm waiting for the docs to stabilize, so most likely early in the new year.

 

[dlavigne]

Also noting that instructions for creating a local copy of the docs are at https://github.com/freenas/freenas/blob/master/docs/userguide/README. Instructions are for the PDF and multi-html formats. We're currently using multi-html format on the doc.freenas.org website as it makes for easier navigation between chapters and reduces the length of the ToC shown in the left sidebar. If you would like to generate a singlehtml (which makes it easier to search for keywords on a single, extremely long webpage, as yes, the current search function is limited on the website), change the HTML build command to:

% sphinx-build -b singlehtml . _build

 

[scott55]

Sounds good. I will build my own as described if I need a copy before the new year. I like to print off and study a hard copy before installing.

 

[mjws00]

Heh. Thought I'd make myself useful and build this pdf. Built a ports jail. Installed all the packages. Cloned from git. Then the fun began.

Code:

>sphinx-quickstart
Error: an existing conf.py has been found in the selected root path.
sphinx-quickstart will not overwrite existing Sphinx projects.

Just decided to ignore it. Maybe we are already initialized.

Code:

>yes '' | gmake
Extension error:
Could not import extension sphinxcontrib.httpdomain (exception: No module named sphinxcontrib.httpdomain)
Makefile:106: recipe for target 'latexpdf' failed
gmake: *** [latexpdf] Error 1

Hmm. That's not good. Google everywhere. Tried to install extension. Really hoping to avoid a deep dive.
Decided I'd comment it out of the config.py and see what broke.

Code:

 ... snip a bunch of verbose output ...
Output written on FreeNAS.pdf (295 pages, 7428166 bytes).
Transcript written on FreeNAS.log.
Makefile:54: recipe for target 'FreeNAS.pdf' failed
gmake[1]: *** [FreeNAS.pdf] Error 1
gmake[1]: Leaving directory '/root/fnbuild/freenas/docs/userguide/_build/latex'
Makefile:106: recipe for target 'latexpdf' failed
gmake: *** [latexpdf] Error 2

Success! At least partially. Output is verbose. Looks like I throw an error. But a nice pdf is built in _build. Title page is separate.

@dlavigne I'm no wizard, but don't mind being a guinea pig. Any hints on resolving the sphinxcontrib extension error? Is it necessary? Is the output always full of warnings and quite verbose?

The html seemed to build pretty nicely, has some warnings but not bad.

Hopefully that is useful to someone and the procedure could be refined a little.

 

[gpsguy]

I tried it over the weekend on a FreeBSD 10.1 VM on ESXi.

I ran into something similar with sphinx-quickstart - I ended up running it a second time and changing an answer or two.

I also encountered a same problem with sphinxcontrib.httpdomain and after a bunch of Google searches, found a reference to "textproc/py-sphinxcontrib-httpdomain", which I installed.

I never got the PDF (can't remember what happened there). My HTML file looked nice with the exception of the missing images (.png).

At the end of the day, I kept my HTML file (such as it was) and blew away the VM. I figured I'd retry the experiment at a later date.

 

[mjws00]

I tried to install that package from ports and it failed. Worked from pkg install when not typo'ed. Doh.
That changed things.

make <target> seems to work except for the pdf

Code:

# make latexpdf
Error expanding embedded variable.
*** [latexpdf] Error code 2

 

[norskman]

I wish someone could write out some simple instructions for the commands to get the latest packages, run them so that I can get a PDF anytime.

I would think that most people would like that as an option

 

[dlavigne]

Yes, you should get a bunch of warnings and it should take 2 tries for the PDF to successfully generate (we're still researching why). As for the contrib error, you have the 3 extra pkgs installed needed for PDF generation?

I've also been thinking of removing the quick-start line but haven't had time yet to test on a fresh install whether or not the build will work without it. If someone has a chance to test, let me know.

 

[dlavigne]

I tried to keep the README non-FreeBSD specific, so users don't think that running FreeBSD is a requirement. But, it didn't make sense to find the pkg names for every distro out there for the software that is needed, hence the FreeBSD package names. If it makes more sense, I can change it to show FreeBSD commands and let non-FreeBSD users figure it out from there... Thoughts?

 

[dlavigne]

Definitely haven't seen that one before. Can you double-check the output of pkg info to make sure that all of the dependent pkgs are installed?

 

[mjws00]

My thought was, everyone would have a portsjail available with FreeNAS... so that's an easy common starting point.

I installed everything on the list, and added textproc/py-sphinxcontrib-httpdomain

Code:

[root@bsd /]# pkg info (snipped)
git-2.2.1                      Distributed source code management tool
py27-sphinx-1.2.3              Python documentation generator
tex-formats-20140525_1         Formats for Basic TeX Engines
tex-dvipsk-5.994               Convert a TeX DVI file to PostScript
gmake-4.1_1                    GNU version of 'make' utility
py27-sphinxcontrib-httpdomain-1.2.1 Sphinx domain for HTTP APIs

For some reason today gmake is working, last night it threw me to that make <target> prompt. I did clone a fresh freenas copy to make sure my actions weren't interfering. I'll build a fresh jail and walkthrough again. I was pretty thorough and have the console logs, but some extra actions may have complicated things. I did manage the pdf both with my hacked config.py, and eventually a fresh default one. But also had hours of runs with no .pdf it moved all the images etc, then crashed out.

Would be really cool if you could just clone the /docs instead of the entire project. This is gigs of downloads and over an hour on a fast connection and machine for a 7MB pdf.

 

[gpsguy]

Just like the Rolling Stones "Shattered" song, "Ain't you hungry for success, success, success, success"...

After seeing mjws00's replies today, I recreated my FreeBSD 10.1 VM tonight and found nirvāṇa. I now have both a HTML and PDF version of the doc's.

I need to take a break for awhile, I'll post my findings later tonight.

 

[depasseg]

I think adding freebsd specific instructions would be helpful. using pkg install was pretty easy, but not being familiar with freebsd took me a while. It would also be nice to see how to create an updated PDF later on.

 

[cyberjock]

Personally, I feel that providing FreeBSD-only commands are just fine. If you don't know FreeBSD and you plan to compile the manual to a PDF yourself you'll probably know what to use.