Pandoc, Pagefind and Make

Recently I’ve refresh my approach to website generation using three programs.

Pandoc does the heavy lifting. It renders all the HTML pages, CITATION.cff (from the projects codemeta.json) and rendering an about.md file (also from the project’s codemeta.json). This is done with three Pandoc templates. Pandoc can also be used to rendering man pages following a simple page recipe.

I’ve recently adopted Pagefind for indexing the HTML for the project’s website and providing the full text search UI suitable for a static website. The Pagefind indexes can be combined with your group or organization’s static website providing a rich cross project search (exercise left for another post).

Finally I orchestrate the site construction with GNU Make. I do this with a simple dedicated Makefile called website.mak.

website.mak

The website.mak file is relatively simple.

  1. #
  2. # Makefile for running pandoc on all Markdown docs ending in .md
  3. #
  4. PROJECT = PROJECT_NAME_GOES_HERE
  6. MD_PAGES = $(shell ls -1 *.md) about.md
  8. HTML_PAGES = $(shell ls -1 *.md | sed -E 's/.md/.html/g') about.md
  10. build: $(HTML_PAGES) $(MD_PAGES) pagefind
  12. about.md: .FORCE
  13.         cat codemeta.json | sed -E 's/"@context"/"at__context"/g;s/"@type"/"at__type"/g;s/"@id"/"at__id"/g' >_codemeta.json
  14.         if [ -f $(PANDOC) ]; then echo "" | pandoc --metadata title="About $(PROJECT)" --metadata-file=_codemeta.json --template codemeta-md.tmpl >about.md; fi
  15.         if [ -f _codemeta.json ]; then rm _codemeta.json; fi
  17. $(HTML_PAGES): $(MD_PAGES) .FORCE
  18.     pandoc -s --to html5 $(basename $@).md -o $(basename $@).html \
  19.         --metadata title="$(PROJECT) - $@" \
  20.         --lua-filter=links-to-html.lua \
  21.         --template=page.tmpl
  22.     git add $(basename $@).html
  24. pagefind: .FORCE
  25.     pagefind --verbose --exclude-selectors="nav,header,footer" --bundle-dir ./pagefind --source .
  26.     git add pagefind
  28. clean:
  29.     @if [ -f index.html ]; then rm *.html; fi
  30.     @if [ -f README.html ]; then rm *.html; fi
  32. .FORCE:

Only the “PROJECT” value needs to be set. Typically this is just the name of the repository’s base directory.

Pandoc, filters and templates

When write my Markdown documents I link to Markdown files instead of the HTML versions. This serves two purposes. First GitHub can use this linking directory and second if you decide to repurposed the website as a Gopher or Gemini resource you don’t linking to the Markdown file makes more sense. To convert the “.md” names to “.html” when I render the HTML I use a simple Lua filter called links-to-html.lua.

  1. # links-to-html.lua
  2. function Link(el)
  3.   el.target = string.gsub(el.target, "%.md", ".html")
  4.   return el
  5. end

The “page.tmpl” file provides a nice wrapper to the Markdown rendered as HTML by Pandoc. It includes the site navigation and project copyright information in the wrapping HTML. It is based on the default Pandoc page template with some added markup for navigation and copyright info in the footer. I also update the link to the CSS to conform with our general site branding requirements. You can generate a basic template using Pandoc.

  1. pandoc --print-default-template=html5

I also use Pandoc to generate an “about.md” file describing the project and author info. The content of the about.md is taken directly from the project’s codemeta.json file after I’ve renamed the “@” JSON-LD fields (those cause problems for Pandoc). You can see the preparation of a temporary “_codemeta.json” using cat and sed to rename the fields. This is I use a Pandoc template to render the Markdown from.

  1. ---
  2. title: $name$
  3. ---
  5. About this software
  6. ===================
  8. $name$ $version$
  9. ----------------
  11. $if(author)$
  12. ### Authors
  14. $for(author)$
  15. - $it.givenName$ $it.familyName$
  16. $endfor$
  17. $endif$
  19. $if(description)$
  20. $description$
  21. $endif$
  24. $if(license)$- License: $license$$endif$
  25. 0$if(codeRepository)$- GitHub: $codeRepository$$endif$
  26. $if(issueTracker)$- Issues: $issueTracker$$endif$
  29. $if(programmingLanguage)$
  30. ### Programming languages
  32. $for(programmingLanguage)$
  33. - $programmingLanguage$
  34. $endfor$
  35. $endif$
  37. $if(operatingSystem)$
  38. ### Operating Systems
  40. $for(operatingSystem)$
  41. - $operatingSystem$
  42. $endfor$
  43. $endif$
  45. $if(softwareRequirements)$
  46. ### Software Requiremets
  48. $for(softwareRequirements)$
  49. - $softwareRequirements$
  50. $endfor$
  51. $endif$
  53. $if(relatedLink)$
  54. ### Related Links
  56. $for(relatedLink)$
  57. - [$it$]($it$)
  58. $endfor$
  59. $endif$

This same technique can be repurposed to render a CITATION.cff if needed.

Pagefind

Pagefind provides three levels of functionality. First it will generate indexes for a full text search of your project’s HTML pages. It also builds the necessary search UI for your static site. I include the search UI via a Markdown document that embeds the HTML markup described at Pagefind.app’s Getting started page. When I invoke Pagefind I use the --bundle-dir option to be “pagefind” rather than “_pagefind”. The reason is GitHub Pages ignores the “pagefind” (probably ignores all directories with ”” prefix).

If you need a quick static web server while you’re writing and developing your documentation website Pagefind can provide that using the --serve option. Assuming you’re in your project’s directory then something like this should do the trick.

  1.     pagefind --source . --bundle-dir=pagefind --serve