What have we got?

RISC OS is documented in many different places these days.
 
Originally it was documented through the Programmers Reference Manuals ('PRMs'), a physical set of manuals in multiple volumes for each of the major areas. Over the years these have been augmented by updates in the form of volume 5/5a, application notes, specifications for some of the 'new' components that were never integrated into centralised documentation, and information that was noted by people and submitted to the StrongHelp manuals.
 
The PRMs were released as part of the 'Techie CD', which had a hypertext application which could be browsed on RISC OS. The content was the same as the physical books, with the same page numbers. Later, there were PDFs and HTML versions ¹ produced by RISCOS Ltd, which were more accessible, but had their own problems ².
 
The application notes and various specifications came out of Acorn which included some detailed documentation which covered usage notes and APIs for 'newer' features. These 'newer' features include such things as the URI/URL fetcher stack, nested window manager, redraw manager, and miscellaneous updates that followed on.
 
The StrongHelp manuals for SWIs and the like were originally written by Guttorm Vik, and maintained by myself for a few years, before being passed on to other people - they were never official. However, they included many of the fixes and notes about oddities within the PRM and the different implementations within various RISC OS versions. The StrongHelp manuals were the primary place for developers to refer to APIs whilst writing code, with reference to the PRMs when more detail was required. The main 'SWIs' StrongHelp manual contains links to the pages of the original form of the PRMs to aid in this process of escalating gathering information.
 
There is also a Wiki on the RISC OS Open site, which is very similar to the StrongHelp documentation in its detail and degree of authority.
 
So we have...

• Original PRMs, in printed, Techie CD, PDF and HTML forms.
• Random specifications produced officially and unofficially.
• Random user-sourced cheat-sheet documentation like StrongHelp (and later the RISC OS Open Wiki).

That's the current state of the documentation - it hasn't really changed in the last 15 years, as far as I can see.

What did I want?

Back when I was doing RISC OS work in the dim past, after the RISCOS Ltd days, I felt that this situation was not ideal and needed to be addressed. So I tried to work out what the goal was, and what was problematic.
 
The goal...

• Produce documentation that's viewable on RISC OS and other systems.
• Manage the source as a living document that can be editable by developers, on RISC OS (or other) systems.
• A documentation system that was easy to work with, so that people without experience could produce documentation.

The documentation as delivered by Pace was in FrameMaker format, as that's how Acorn's publications department had managed it. FrameMaker is an Adobe product which was only available for Windows. As supplied, the documentation was in the binary file format, which meant that at any time only one person could work on a chapter ²⁰. We only had a single license for the product, so in any case only one person could work on any part of the documentation at a time.
 
FrameMaker did have textual formats, but these were not easy to work with, and were not easy to bring into an automation system which we were hoping to use. FrameMaker could output to PDF, HTML and PostScript, which was useful to us for physical manuals if we had wanted to produce them in the future. It's a good system if you happen to be producing manuals for large projects through a central publications group (as Acorn had).
 
However, we wanted to be able to work on the documentation ourselves, and to be able to farm out documentation to others outside the company if they had an interest (or as contract work). We hoped that this would encourage others who had had complaints about the documentation to be able to contribute.
 
The problems with this...

• FrameMaker had a steep learning curve for those who hadn't used it before.
• It was important that what was being produced was able to be worked on within RISC OS.
• Not being able to be worked on by multiple people was quite limiting.
• The binary file format meant that changes couldn't easily be managed in the source control system in the same way as the rest of the sources.
• Expecting third parties to purchase the expensive product ¹⁸, just to update documentation, was not reasonable.
• The HTML that FrameMaker produced was not good in RISC OS browsers - hence the transformation work to make it usable.
• RISC OS native products like Ovation or Impression would show a preference and be prejudicial to others.

What does that mean? Well, it implied that the output format would need to be HTML or PDF. I had already decided that FrameMaker was not appropriate, and so working directly in the HTML was an option. Much earlier, I'd used 'HSC' ³ as a tool for generating some simple documentation in HTML format ⁴. This worked well in producing the right sort of documentation but it was a little specialised. Because it was a niche system, I did not feel that it would work well in a wider community. Whatever I used really needed to be able to be used by people without too much knowledge of the system, and for which tooling was widely available and standardised.
 
This is where I began to look at an XML format with transformations through XSLT.
 
DocBook become a thing that people were using, so I had a look at it and ... it's quite generic, a little weird in places ⁶, and had a steep learning curve. But because it was generic, it didn't have any specialised layout for the style that I wanted for the PRM - that would have to be built on top. You'd essentially be writing XML elements that were for DocBook styled to look a particular way.
 
There is documentation for DocBook and you'd be able to produce something useful, but you're still not capturing the essence of the RISC OS documentation in a useful way - I'd become convinced that the benefit in using XML was not so much in the fact that I could use it to produce readable documentation, but that it was capturing structured information about how the APIs worked. That is, not only could you write the XML and get out a page of readable documentation at the end, but that the XML that you wrote could be cross referenced, indexed and even converted to other formats easily.
 
I'd been tinkering with other things at the same time. Thoughts about how modules might become self-documenting were going around my head - if a module could report information about its SWI interfaces in a structured manner, then in-system lookup of APIs, and debug information for tools became possible.
 
Toolbox already had its own format for describing the toolbox SWI interfaces, which were then built into C/ObjAsm files that were used in toolboxlib. If those definitions were built into the module, we could always extract them from a running system. And things like the toolbox test applications (!ResTest) could use toolbox event information from the module to present messages about what was going on.
 
The BTS system ⁷ recorded every piece of information about modules when an abort happened. You could later show that information on another system. If the modules that were running reported their API's parameters, then it would be possible to report the meaning of those recorded registers within a diagnostic dump.

Creating PRM-in-XML

All of these things were pointing to having structured information about interfaces and data structures ⁸.
 
This produced the idea of using an XML representation to hold the structured information in the PRM. Having separate tags to describe SWIs, vectors and events, etc, with information about the register usage being structured within that. This seemed like a workable solution to me, and I set about trying to represent the content of a few pages of the PRMs in an XML format.
 
Initially it was a bit HTML-like, but some elements were culled - most of the elements which were related to formatting were replaced by semantic elements ²¹. The only formatting elements which remain are for stress within the prose, line breaks, and for super-script and sub-script, as these are often needed to explain more maths-heavy sections.
 
As I was working on it, I created the XSLT - the transformation that turns it into HTML - so that it could render the elements using tables. That meant that it worked in the browsers of the time. The transformation is important to make it look presentable, but the point of the XML format is that if you want it to look different, or you want to extract just certain information from the files, you can do so with whatever XML tools you have. And there are a lot of tools for doing that.
 
There are actually a few transformations that were working, to varying degrees:

• There's the HTML transformation I've mentioned, which used tables and inline formatting.
• There's a simple extraction of *Command usage, so you can get simple information that might go into *Help.
• There's a C header file generation that creates constants from the APIs in the file.
• There's an Impression DDF generator, that really didn't work very well.
• There's a rudimentary StrongHelp format generator, which worked reasonably but wouldn't be suitable for the quick-reference use that you would have in StrongHelp.
• There's a specialised HTML transformation that uses frames to link content in a different way. It didn't work very well.

And I needed a way to pull all this together, so I created a set of index/contents pages, which extracted the cross-reference information from the files to provide direct links toeach of the SWIs, events, vectors, etc, just like the equivalent sections in the PRMs did.
 
Although these are the transformations that exist, more can be created with a little effort.

What does it look like?

Here's the definition of a SWI for AMPlayer in the PRM-in-XML format:
 

<swi-definition name="AMPlayer_Pause"
        number="52E02"
        description="Pauses playback"
        irqs="undefined"
        fiqs="enabled"
        processor-mode="SVC"
        re-entrant="no">
 
<entry>
 <register-use number="0">flags :
 <bitfield-table>
  <bit number="0" name="Resume">
  Resumes playback.</bit>
  <bit number="1-30">Reserved, must be 0.</bit>
  <bit number="31">R8 contains the instance handle to which this call
          should be directed.</bit>
 </bitfield-table>
 </register-use>
 <register-use number="8">if bit 31 of R0 set:<br />
             instance handle to direct at, or 0 for
             the base</register-use>
</entry>
 
<use>
<p>This SWI is used to pause or resume playback. When paused, decoding to
the output buffer continues, but at a much reduced rate. There is no sound
output.</p>
 
<p>Pause mode may also be cancelled by stopping. If
<reference type="swi" name="AMPlayer_Stop" /> is used to
cut to the next file, or if a different file is started, pause mode will
continue to be in effect, freezing the new file at the start of the file.
This can be used to ensure that playback starts at the instant of calling
AMPlayer_Pause (as opposed to calling
<reference type="swi" name="AMPlayer_Play" />, which can have a delay while
opening the file etc).</p>
 
</use>
 
<related>
 <reference type="command" name="AMPause" />
 <reference type="swi" name="AMPlayer_Play" />
 <reference type="swi" name="AMPlayer_Stop" />
</related>
 
</swi-definition>

• The definition declares, as attributes of the swi-definition element, what the SWI is and some of the common fields that are described in the PRMs. Because these are attributes with defined meaning, they can be extracted - the contents generation uses this information to create references to the SWIs.
• Then there is a description of the registers on entry to the SWI call (entry). Each of these is described through a register-use, which describes one or more registers. The flags in R0 are described in a bitfield-table which explains what each bit means.
• There could be a description of the registers on exit from the SWI (exit), but this SWI does not have any, so it is omitted.
• There's a longer section of prose which describes what the SWI does and how it is used (use). This includes cross-references to other SWIs or sections as necessary.
• Finally, there's a section for related interfaces, which includes a collection of references to other sections within the documentation.

It's a specialised format and there's a little bit of a learning curve to find the right way to express things, but because you're focusing on what the interface is, rather than how it's presented it is much easier to write. Mostly you know that the produced documentation will be in a consistent format and you cannot accidentally deviate from the style of the PRMs. That can be a bad thing for some use cases but, for consistent and easy to follow documentation, it is important.

What did converting the PRMs involve?

So having decided on a format that can represent the information contained within the PRM, the next step is to turn the chapters of the PRMs into that format. However, it's useful to prove that it will work first by starting to use it for real things. Using it for real APIs and structures meant that it got 'battle hardened' and the error messages improved, and redundant parts stripped or made more useful. Then real conversions began.
 
And that's a lot of work. It's not a process I completed, but it's one that's largely mechanical. Take the HTML PRMs, pass it through a Perl script which turns the patterns of usage into skeletons of structured elements, and then chip away at the many bits that it got wrong until you have a document in the right format. It's tedious and it takes time to do, but the results are XML files that describe the same content as the PRMs. It's important at this stage to not start introducing differences. Ideally you can visually skim two copies of the documents - old and new - and see that, apart from styling, the content is the same.
 
This was a long running process that I embarked on, and got some help from Andrew Hill in creating some of the documents. Take a chapter, convert it, review in comparison to the original and fix. Repeat. Then go and do something else because it's a bit mind numbing ¹⁰.
 
There are bound to be mistakes, but the point is to get it into a format that's easier to manage, and has greater flexibility. Each chapter of the PRMs can be handled in (mostly) isolation. There will always be cross references, but these can be given general targets and we can be reasonably sure that a link checker will be able to verify these later. Because new documentation is all being written in this format, you're not falling behind and having to convert things repeatedly.
 
When the original PRMs were written, my understanding is that the original developers of the OS had written notes about the components to explain the interfaces and interactions. I believe that there were also some interviews done by the documentation group to understand the intent of the parts of the system ⁹. So there was some point of divergence at the point that the documentation and transfer of understanding of the components were made - from the developers to the documentation department.
 
This transfer was necessitated by the large, and quite skilled, job of writing documentation being different to the day to day work of developers. Generally, developers don't write good documentation, and they certainly don't have access to the tools and the knowledge of how to drive those tools well to create a full manual. Or at least they didn't at that time.
 
However, I was never going to get professionals skilled in documenting and publishing tools, so the PRM-in-XML project, as I named it, was going to have to be 'good enough' with developer written documentation. That also meant that the tools needed to be pretty simple to use. So I wrote a RISC OS application that could take a given XML file and turn it into HTML - which is about as simple as it gets. It's just invoking xsltproc on the file in a TaskWindow and showing you the output.

Moving the goal posts

The goals have changed slightly, as the needs become more obvious. Instead of purely wanting to have something that was easy to work with, it should be something that was easy to work into the software development life cycle. That is, as you update a component like a module or tool, the documentation gets updated with it. And the easiest way to do that is to have the documentation sit alongside the component in its source directory.
 
If the multi-stage build process that builds an application (or the whole OS) includes a set of targets for exporting documentation, much as you export the resources for ResourceFS, then you can keep the documentation in the same system. That's where it is most useful and it can be used to construct a set of manuals from the application or system that you're building. Or even just a set of manuals for a section, if you only process the parts you need.
 
By keeping the source with the documentation with the component, you remove that need to transfer knowledge from one place to another at marked points. You can update the documentation as you change the code. Yes that means that adding a new feature is more than just writing code, but if you believed it was just writing code in the first place you were woefully mistaken.
 
And it's not that hard to do. Before the PRM-in-XML began to be used in builds of components (and that was really late on), most components just had plain text files which described the functioning of the components. Usually these were structured as an introduction and description of the component and its interfaces, then a description of the SWIs and services that it provided. They were intentionally structured that way because that eased the transition to the PRM-in-XML format, or to be released directly in that format because it would still be a familiar and useful manner of introducing new features.

Retrospective

So, with an extra 15 years of experience, what do I think about it?
 
I dusted off the PRM-in-XML project last year and started to make it work. The xsltproc tool had become a bit more strict so there were some operations that were broken that I'd had to fix. I cleaned up and exported the style sheets and tools as part of the release of resources with my Build service presentation ²² - because it's used within Pyromaniac's build, and in a number of my other builds that I do.
 
It's even more useful to me now because systems are so much faster - processing a chapter is a sub-second task. And it's still a structured format so the pages that were converted years ago are still useful, and can be converted to other formats if I wanted.
 
Other technologies exist for managing documentation, so is it still the right way to do things? The goals these days, I think, have changed a little...

• Being viewable on other systems is more important now - most users will be using a second device (a tablet or phone, or Windows/Linux/macOS device), so being able to be viewed on such systems is vital.
• Editing in the single format by developers is more important - as the community is more open, it is important to allow anyone to be able to submit documentation changes, and doing so in a similar way to the regular flow of development means that having the documentation alongside the main source is vital.
• Having a community of unskilled developers is a useful commodity for documentation. They can provide better insight into what is useful, and be able to offer better suggestions. And if the format is simple enough that it's relatively obvious what they need to do to work with it, then that is all the better.
• A textual format that is diff-able is even more important for the review process in source control systems such as git, as exposed by GitLab, GitHub, etc. This was important in my original goals but essentially implicit. With the expectation that contributors will want to be able to review changes that happened and submit their own, this is important to the review process.
• The documentation can be worked on by multiple people at once - because it's text, and therefore able to be processed by the standard tools, two people working on a documentation file is very easy to resolve. And because chapters are split between components, the likelihood of conflicting will be low-to-non-existent.
• It's all free - there's no cost in using a the XML processing tools, and therefore you don't have to buy Impression, Ovation, or FrameMaker to work on the documentation.
• Anyone can do it - the tools and docs available to everyone, and anyone can process the content into HTML, or into other formats with some processing.

This last point offers the most flexibility - in that if someone wants to do something fun like produce a documentation browser ¹¹, or in-editor context help ¹², that would be possible, and would grow in usefulness as more sections of the system were documented.
 
What about PDF production? Change the HTML stylesheet to use classes ¹³ and then use CSS media queries to allow the pages to be formatted for printing. Or change the output format to directly output a presentation format like DocBook which is then able to be laid out for printing. This isn't so much a challenge as just another step.
 
Honestly, though, HTML is more flexible than a PDF most of the time, and PDF as a format to transition to a physical manual is a huge waste of time and money ¹⁴. I looked into it for a print version of my Rambles - it would have been about £80 a copy if I remember correctly ¹⁵. If the new PRMs were going to be sold in printed format they'd be much more expensive, for content that you could download yourself and browse on your machine, or online. So, what's the point? Some people might prefer that, but not enough to make a printed version of the manual viable. At least that's my opinion - nothing about the conversions precludes you doing it if that's what you want.
 
EPub is probably more useful, as it would allow for layout on tablets and other mobile devices. And EPub is just HTML ¹⁶ with some metadata around it, so that's easy.

Conclusion

PRM-in-XML offers...

• Structured information for the documentation and APIs.
• The ability to transform into other formats easily - HTML, StrongHelp, PDF, etc.
• Requires only a text editor to work with.
• Easy to read and edit format.
• Easy to process format.
• Easy to get community involvement for creation and review.
• Easy to ensure it remains up to date (if it remains with the source).
• Simple to integrate with build systems.
• Interacts well with modern version control systems like git.
• Zero start up cost.

So I believe that it meets the goals that I set myself, and should be a good solution for the documentation of RISC OS and its interfaces. With an active community it should be relatively easy for people to get involved, whether it be for just simple typos, or more wide scale changes. Similarly, developers can keep their documentation right beside the code that they're working on, ensuring that it stays up to date.
 
A few of the things that I have released include some documentation in the PRM-in-XML format, because for me it's a good solution.
 
Any new things that people write could always be updated to use the PRM-in-XML format if they wished. One way of doing that is to just start out with some basic documentation and as people provide feedback on problems, or where things aren't right, it's easy to update.
 
Of course, updating all the RISC OS documentation to this format takes time and effort. But it doesn't have to be done all at once. Taking on small parts of the effort of documenting components breaks it up into nice chunks, that are relatively easy to do (if a little dull). That's where having an active community is pretty handy - anyone that's got an interest in improving things, can do so.
 
I'll write more in the future about how the PRM-in-XML can be used, but for now I think this covers what I think should be done to create consistent and community managed documentation - whether it be to replace the PRMs, or for your own software.

Resources

I'm working on releasing these resources out on GitHub, but the release from the original Build system talk is available right now and usable. The release contains documentation on how to use the tools and what the format looks like.

• PRM-in-XML stylesheet and tool (124K)
• PRM-in-XML example files (89K)
• Pyromaniac API documentation (48K) - documentation generated from PRM-in-XML sources, including the sources.
• RISC OS LibXML2 port and LibXML2+XMLLint binaries
• RISC OS LibXSLT port and LibXSLT+XSLTProc binaries