HTTPS SSH

Pandoc/Markdown Tools

11 February 2013 v1.0

That's a pretty unimaginative name…but there you are. This set of tools came about because my wife, who works on an iPad at work, wanted a way to convert documents. She's not keen on Terminal so I had to find a way around that. The answer was an SSH connection to a Mac server and a set of tools run by a connection script that provide a menu driven interface for some basic Pandoc/Markdown activities.

Just for clarity I'm not a coder so I'm sure many people will look at these tools and think the coding is long winded or clumsy. However I think they are quite robust and they do the job. If you want to fix some stuff, start by forking the git repository from Bitbucket and making the improvements. Then share it back with us via a pull request.

Warranty and liability is limited in all senses to exactly what I am asking you to pay me for these scripts…which is nothing. Please understand that you use them entirely at your own risk. If you are worried about data loss or anything else then don't use these scripts. They are offered to you on a take it or leave it basis with no representations as to fitness for purpose or anything else. Use at your own risk.

So these tools do three things:

  1. They provide a "menu" driven interface, via the command line, to some key Markdown and Pandoc tasks;
  2. They allow the simple conversion of Markdown docs to docx (or with minor tweaks to other formats) using that menu driven interface;
  3. They really simplify the creation and insertion of Pandoc style tables into Markdown documents.

It's that last task that for me is the make it or break it for Markdown. Unfortunately tables are a fact of life, and creating Pandoc style Markdown tables by hand is a pain.

Prerequisites

So that brings us very nicely to prerequisites.

  1. A 'nix flavoured operating system. This set of scripts work on Mac OS 10.8.2 . I haven't tested them on anything else but they should work on other 'nix variants and maybe under some tool or other to emulate such a variant on Windows.
  2. Pandoc must be installed so go to the excellent Pandoc site and download and install Pandoc. Make sure it can be found from the command line, all the scripts assume it's easily found and don't carry a path to it.
  3. If you want to do tables then you need to install the excellent R package. Don't be daunted by it, you'll never have to interact with it again once you have installed it and installed devtools and pander packages and their prerequisites. It is used as the engine to create the Pandoc style tables.

That's it. So if you don't have those things installed go away and do that before you come back.

Script Location & Structure & Installation

The scripts assume that you have some directory (for discussion we'll call this maindir) within which all of the scripts live and from which those scripts are called. Further the scripts assume that at the root level within that directory is Dropbox/ and that there are directories within Dropbox which you will wish to work in.

If that's not the case in your situation then you will have to modify the scripts to suit your circumstances. Start by searching for 'Dropbox' in the scripts and changing that to suit your circumstances.

The basic principle however holds that the scripts expect to be working in a path below their own location.

The scripts are as follows:

  1. Main.sh which is the script that you call using ./Main.sh or some other variant. This is the only script you should have to interact with directly…thank my wife for that!
  2. GenerateTable.sh which is the script you would choose from the menu in order to generate R tables. It collects the information from you and calls RTable.r.
  3. RTable.r which is an R script to do the actual hard yards on the table creation. You won't see it from the menu and you don't need to interact directly with it.
  4. GenerateWord.sh which is the script you call to convert docs from Markdown to docx. You can modify this to your heart's content to add other conversion formats and other Pandoc niceties.
  5. InsertTableMaster.sh which is the script you call in order to be able to replace markers in Markdown documents with the content of the table documents that you create using GenerateTable.sh. This saves a lot of cutting and pasting.

To install the scripts unzip them and copy them out of the directory they are in and into maindir making sure they are at the root level of maindir. It's assumed that also at the root level of maindir is Dropbox/ so if it's not you have some modifications to do to the scripts.

You may have to set permissions on the scripts with chmod 755 scriptname.sh

Really the nicest way to install and deploy the scripts is to clone the Git repo into maindir you'll end up with your repo at maindir/PandocMarkdownTools. Then create symlinks in maindir that point to the scripts in the repo directory. That way you can update the scripts at any stage by simply shifting into the repo directory and issuing git pull.

Using the Scripts

It's simple really: call Main.sh by issuing ./Main.sh and then choose the script you want from the menu…then just follow the prompts.

Probably the ones that need explaining are the table scripts.

Using the table scripts

There are 2 scripts for working with tables. The first takes a set of tab delimited data and turns it into a basic Pandoc Markdown table. It will ask you for the directory you want to work in, the input and output names and then generate the table of that name. You need tab delimited data in the input file in the form that you want your table to be in. There are a couple of things to note:

  1. These are not standard Markdown tables, they are one of Pandoc's variants and therefore are for use with Pandoc.
  2. If your tables have strings in them with spaces such as Jane Black then when you enter that data you must quote it: "Jane Black" otherwise there'll be trouble.

Other than that it's pretty easy and it generates nice tables.

The problem is that you need to get those tables into your Markdown document out of the individual files that they are in. That's a pain…so there's a script for that too!!

In your main Markdown document - the one that you want the tables to go into you need to insert markers. Those markers need to be on a line of their own and separated from the text above and below by a blank line.

The markers take the form: @filename.extension@. What's important here is that there are no spaces, that the filename.extension bit is the exact name of the file that has the table in it that you want in there, and that the @ characters are tight at either end around that filename. So your file could be called table1.md and your marker would therefore be @table1.md@. This is important because that's the pattern the script looks for, and then it strips that pattern down to find the filename and then imports it into the position of the marker. The file can be named anything you like foo.weirdextension for all it matters, as long as the marker has the @ at either end and matches the filename. The files need to be in the same directory as the document with the markers in it.

This one script lacks a bit of error trapping. So if you have a marker and there isn't a matching file then the marker will be found, the file won't and therefore won't be inserted but the marker will be deleted. So please check carefully.

Packaging

The scripts are in a Git repository which you can find at BitBucket. You can then either clone the repo using your choice of connection method or if you are not that into Git you can download an archive. To find current archives go to the Downloads tab and then choose the Tags tab. You can then choose which version to download and what kind of archive you want to download.

Note

A script called SignedGitTag.sh has found it's way into the repo so I've left it there. You can call it in any repo to automate the process of creating a signed Git tag. It gives you a picture of the log and allows you to name, add the commit hash and add a comment for a signed tag. You'll need to have your GPG keys sorted to allow you to sign the tag. You'll be asked to give the GPG key passphrase during the tag creation. Nothing to do with Pandoc or Markdown but useful in Git.

Note 2

There is another use case for InsertTableMaster.sh which is quite interesting. I'm thinking of creating a branch that deals with this use case.

So the use case is to use InsertTableMaster.sh as a pre-processor to build a single file out of multiple files. You would create an "Book.md" file that had only markers in it, something like:

@Introduction.md@

@Chapert1.md@

@Whatever.md@

@Endmatter.md@

In the same directory as the Book.md file you would have all of the different .md files specified by your markers. Then you would just call ./InserTableMaster.sh before GenerateWord.sh to build your book.

If you modified the GenerateWord.sh script to call InsertTableMaster.sh before it created the docx run then you would end up with a fully built document made up of the various different .md files and in the order you specify.