new-ryuslash.org/literate-build.org

473 lines
22 KiB
Org Mode
Raw Normal View History

#+title: ryuslash's website's build's files
#+subtitle: Literate configuration of how to build this site
#+options: num:nil prop:t
#+macro: kbd @@html:<kbd>@@$1@@html:</kbd>@@
I'm a big fan of [[file:literate-programming.org][Literate Programming]], so I figured I'd make the builds for my website a literate configuration file as well.
First I need to build the build files. This is the smallest make file that I can think to make to enable me to build the rest out. This make file is duplicated in both this file and the source code repository since I don't know of a way not to.
I specify that =build.mk= file should depend on /this/ file ({{{input-file}}}) and that it is generated by running =org-babel-tangle-file= on it. The =$<= is the first dependency and =$@= is the current target file (whatever =.mk= file we're generating).
#+begin_src makefile-gmake :tangle GNUmakefile
build.mk: literate-build.org
emacs -quick -batch \
-eval "(package-initialize)" \
-load ob-tangle \
-eval "(org-babel-tangle-file \"$<\"))"
#+end_src
I can't use Eldev in this particular recipe because it needs a file with proper package headers and all, and that doesn't get generated until we run this. I'm cheating a little bit because I'm expecting this command to also generate the other files the build needs without explicitly saying so.
After that it's just a matter of including the file I want.
#+begin_src makefile-gmake :tangle GNUmakefile
include build.mk
#+end_src
GNU Make (I don't know about other makes) will see if there is a recipe to make the file it wants to include and will try and run it before trying to include the file. This combined with our =%.mk= target ensures that make will always try to recreate the =build.mk= file when {{{input-file}}} is updated.
* Makefile
:PROPERTIES:
:header-args:makefile-gmake: :tangle build.mk
:END:
This is the actual make file that builds and deploys my site. It's all put into the =build.mk= file and executed from there. The =%.mk= pattern rule thankfully doesn't get recognized as a make target, so the first target define in the included file is assumed to be the default target.
First off I specify the =help= target. This target parses the make files and extracts targets that include some comment on what they do. This target should come first so that it automatically becomes the default target. This way when I run just ~make~ I can see which targets I have available. I got this awesome trick from [[https://victoria.dev/][Victoria Drake]]s article [[https://victoria.dev/blog/how-to-create-a-self-documenting-makefile/][How to create a self-documenting Makefile]].
#+begin_src makefile-gmake
help: ## Show this help
@grep --extended-regexp --no-filename '^[^#].*?\s##\s' $(MAKEFILE_LIST) \
| sort \
| awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-20s\033[0m %s\n", $$1, $$2}'
#+end_src
The =build= target converts everything from whatever source files they are to html and css. The build target has 2 other targets it depends on, not surprisingly =html= and =css=. The =html= and =css= targets don't have any comment because they're not really meant to be executed directly.
#+begin_src makefile-gmake
build: html css ## Build the site and copy it to the staging directory
#+end_src
The =html= target calls Emacs. It depends on the =Eldev= file having been generated to specify any additional dependencies and which package archives should be used and this file will be generated by the =Eldev= target.
#+begin_src makefile-gmake
html: Eldev
@echo "Publishing..."
eldev emacs --quick --batch --load publish.el --funcall org-publish-all
#+end_src
The =css= target does specify its dependencies. This is both an exercise in writing make files (which I generally quite enjoy), and also to make sure that my builds don't take too long unless they actually have to. Ultimately any =.css= file gets created from a =.less= file by calling the =lessc= program. I'm intentionally not using recursive make in this project because it slows make down a lot, and I don't have to manage several make files this way.
#+begin_src makefile-gmake
css: public/assets/css/main.css public/assets/css/tekuti.css public/assets/css/cgit.css
public/assets/css/main.css public/assets/css/tekuti.css public/assets/css/cgit.css: \
src/less/include/common.less \
src/less/include/components.less \
src/less/include/colors.less \
src/less/yoshi.css
public/assets/css/%.css: src/less/%.less
lessc $< $@
#+end_src
The =deploy= target first makes sure that =build= has been executed at least once and then uses =rsync= to upload all of the files. This intentionally doesn't depend on the =build= target so that I can upload whatever I happen to have generated without being forced to rebuild.
#+begin_src makefile-gmake
deploy: ## Deploy the site to live
@[[ -e public/index.html ]] || (echo "Run 'make build' before deploy" && exit 1)
rsync --verbose --checksum --recursive --delete \
--exclude '*~' --exclude '.eldev' --delete-excluded \
public/ ryuslash.org:ryuslash-next/
#+end_src
The =clean= target makes sure that everything that is generated gets cleaned up again. This is important if I need to start with a clean slate.
#+begin_src makefile-gmake
clean: ## Remove all of the build files
@echo "Cleaning up..."
@rm -rvf *.elc
@rm -rvf public
@rm -rvf .org-timestamps
@rm -rvf posts/index.org build.mk Eldev publish.el
#+end_src
The =serve= target is a convenience target for when I'm writing or making modifications to the build and publish processes. It just starts a simple =php= web server in the =public/= directory so that I can easily load it in my browser.
#+begin_src makefile-gmake
serve: ## Run a simple web server to look at the results
@cd public && php -S "localhost:8000"
#+end_src
The =theme= target is another convenience target. I generate the colors for the source code blocks on my site from my Emacs theme. This target exports the colors from my theme so that the code blocks can use them. This file is then included by the less files. There is no good dependency here, because there is no file for the export of my theme to depend on right now, just occasionally I have to run it. It does depend on the =Eldev= file having been generated.
#+begin_note
I keep this particular target around for playing with, but right now it doesn't actually work. Just because even though I load and enable =yoshi-theme= it doesn't appear to apply the theme in a batch session. Seems understandable because no UI actually gets loaded, but that does mean that it can't figure out which faces it sets and it just outputs the colors for the default theme.
#+end_note
#+begin_src makefile-gmake
theme: Eldev ## Generate the theme CSS
eldev emacs --quick --batch --load htmlize --load ox-html \
-eval "(setq org-html-htmlize-output-type 'css)" \
-funcall org-html-htmlize-generate-css \
-load yoshi-theme \
-eval "(enable-theme 'yoshi)" \
-load make-mode \
-eval "(kill-whole-line)" \
-eval "(kill-whole-line)" \
-eval "(goto-char (point-max))" \
-eval "(forward-line -2)" \
-eval "(kill-whole-line)" \
-eval "(kill-whole-line)" \
-eval "(css-mode)" \
-eval "(indent-region (point-min) (point-max))" \
-eval '(write-file "src/less/yoshi.css")'
#+end_src
Finally, as a precaution, I specify that all of the main targets are phony targets. This way if I ever introduce any file with the same name as these targets they will still build and not assume that because the file exists, everything is up-to-date.
#+begin_src makefile-gmake
.PHONY: publish deploy html css help theme
#+end_src
* Eldev
:PROPERTIES:
:header-args:emacs-lisp: :tangle Eldev
:END:
With Eldev I can install dependencies of my [[Publishing project]] locally. Really the only reason I chose to use Eldev is because it is available in the [[file:guix.org][Guix]] repository.
Since this is an actually full-fledged Emacs Lisp file and I don't know what I'm going to be doing in the future, I should be sure to enable lexical binding. Otherwise Emacs defaults to using dynamic binding, and that might cause some surprises in the future, since I'm quite used to using lexical binding everywhere.
#+begin_src emacs-lisp
; -*- lexical-binding: t; -*-
#+end_src
All I'm doing here, really, is enable the various Emacs Lisp Package Archives.
#+begin_src emacs-lisp
(eldev-use-package-archive 'gnu)
(eldev-use-package-archive 'nongnu)
(eldev-use-package-archive 'melpa)
#+end_src
* Publishing project
:PROPERTIES:
:header-args:emacs-lisp: :tangle publish.el
:END:
Before anything else, here too I need to add a file header and enable lexical binding. If I don't add the file header Eldev won't work right.
#+begin_src emacs-lisp
;;; publish.el --- Publish project for ryuslash.org -*- lexical-binding: t; -*-
#+end_src
And then I have to keep a list of required packages so that Eldev can figure out which to download. I also need the Version header to keep Eldev happy.
#+begin_src emacs-lisp :noweb yes
;; Version: 1
;; Package-Requires: (<<required-packages>>)
#+end_src
I install =dockerfile-mode= because some of my posts include examples of docker files. And I use the latest =org= mode package.
#+begin_src emacs-lisp :exports none :tangle no :noweb-ref required-packages :noweb-sep " "
dockerfile-mode org htmlize
#+end_src
Some packages must be loaded to make sure their features can be used by the export process.
#+begin_src emacs-lisp
(require 'dockerfile-mode)
(require 'ob-dot)
(require 'ox-publish)
(require 'ox-rss)
(require 'rainbow-delimiters)
(require 'subr-x)
#+end_src
After that I define a constant for the current root so that it's easy to refer to. I use =defconst= here instead of =defvar= for 2 reasons:
1. It's a constant, and shouldn't be changed during execution.
2. Re-evaluating a =defvar= doesn't do anything[fn:1], but re-evaluating a =defconst= updates the value.
#+begin_src emacs-lisp
(defconst publish-root
(file-name-directory
(or load-file-name
(buffer-file-name)))
"The directory where oni-org was loaded from.")
#+end_src
Keep the timestamp cache (used to determine which pages need to be cleaned up) local. This prevents publishing from relying on global state on my PC and makes it easy to reset everything.
#+begin_src emacs-lisp
(setq org-publish-timestamp-directory
(expand-file-name ".org-timestamps/" publish-root))
#+end_src
I add the =rainbow-delimiters-mode= to the =prog-mode-hook= so that delimiters are highlighted like they are in my usual Emacs sessions.
#+begin_src emacs-lisp
(add-hook 'prog-mode-hook 'rainbow-delimiters-mode)
#+end_src
This requires that I have the =rainbow-delimiters= package installed.
#+begin_src emacs-lisp :exports none :tangle no :noweb-ref required-packages :noweb-sep " "
rainbow-delimiters
#+end_src
Anything that is marked with either the =noexport= or =draft= tag shouldn't be exported.
#+begin_src emacs-lisp
(setq org-export-exclude-tags '("noexport" "draft"))
#+end_src
Don't ask for my confirmation whether babel blocks should be evaluated. If I've added them I want them to run.
#+begin_src emacs-lisp
(setq org-confirm-babel-evaluate nil)
#+end_src
Reset the org-mode style, I don't want the default style interfering with my own.
#+begin_src emacs-lisp
(setq org-html-head-include-default-style nil)
#+end_src
Just add CSS class names to code blocks, I'll make sure that the CSS selectors exist.
#+begin_src emacs-lisp
(setq org-html-htmlize-output-type 'css)
#+end_src
Use HTML5 and all its fancy elements.
#+begin_src emacs-lisp
(setq org-html-html5-fancy t)
(setq org-html-doctype "html5")
#+end_src
** Projects
Here come the actual projects.
#+begin_src emacs-lisp :noweb no-export :noweb-prefix no
(setq org-publish-project-alist `(<<projects>>))
#+end_src
*** Pages
The pages project is the main publishing project. It exports all of the =.org= files except for =README.org= and anything found in the =posts/= directory. The =README.org= is only relevant when you're looking at the source code for this website and =posts/= has its own project.
It also loads the stylesheet into the HTML head, and add a link to my Mastodon account in the postamble, for verification purposes.
#+begin_src emacs-lisp :tangle no :noweb-ref projects
("pages"
:base-directory "."
:base-extension "org"
:publishing-directory "public/"
:recursive t
:exclude ,(rx string-start
(or "posts/"
(and "README.org" string-end)))
:publishing-function org-html-publish-to-html
:html-head "<link rel=\"stylesheet\" href=\"/assets/css/main.css\" type=\"text/css\"/>"
:html-postamble t
:html-postamble-format (("en" "<p class=\"social social-mastodon\">Find me on <a href=\"https://fosstodon.org/@ryuslash\" rel=\"me\">Mastodon</a></p>
<p class=\"date\">Date: %C</p>
<p class=\"creator\">%c</p>")))
#+end_src
*** Posts
The posts project is an experiment right now to see if I can find a way I'm happy with to actually publish my blog through plain old org files. This goes together with the [[RSS]] project. This also calculates a rough estimate of how long the reading time for the page will be and adds it to the preamble of each post.
First I need a few of functions. Calculate how many minutes it would take to read the given buffer. I don't remember where I got this particular algorithm from, and I haven't tried out how accurate it is, but my experience with other websites doing this is that it's generally not very accurate anyway.
#+begin_src emacs-lisp
(defun publish-calculate-reading-time (buffer)
"Calculate the amount of minutes it would take to read the contents of BUFFER."
(with-current-buffer buffer
(max 1 (/ (count-words (point-min) (point-max)) 228))))
#+end_src
I decided to split the calculations and formatting functions, so here is just a simple function that format the given reading time.
#+begin_src emacs-lisp
(defun publish-format-reading-time (time)
"Return a string describing TIME."
(format "Reading time: %d minute%s"
time
(if (= time 1) "" "s")))
#+end_src
And then I need a function to actually generate the preamble. This function ignores the =index.org= file, because that is not a page to read but just and index of my blog posts.
#+begin_src emacs-lisp
(defun publish-generate-post-preamble (_project)
"Generate the preamble for a post with the estimated reading time."
(unless (string= (file-name-nondirectory (buffer-file-name)) "index.org")
(publish-format-reading-time
(publish-calculate-reading-time (current-buffer)))))
#+end_src
Finally the actual project. It publishes files inside the =posts/= directory.
#+begin_src emacs-lisp :tangle no :noweb-ref projects
("posts"
:base-directory "posts/"
:base-extension "org"
:publishing-directory "public/posts/"
:recursive t
:publishing-function org-html-publish-to-html
:html-head "<link rel=\"stylesheet\" href=\"/assets/css/main.css\" type=\"text/css\"/>" ; (ref:html-head)
:html-preamble publish-generate-post-preamble) ; (ref:html-preamble)
#+end_src
At [[(html-head)]] I again load the CSS style sheet for this site. [[(html-preamble)]] calculates the page's reading time and puts it on the page.
*** RSS
The =rss= project works together with the =posts= project. This project very specifically generates and targets a specific file and exports it as an RSS feed. For this I require the =ox-rss= package.
#+begin_src emacs-lisp :exports none :tangle no :noweb-ref required-packages :noweb-sep " "
ox-rss
#+end_src
Before I can define my project I still need to have some helper functions. The first I'll make is the =publish-empty-time=. It's just a dumb little function that takes the current time and subtracts it with itself. This is used as an empty value, so that if compared to another time it'll always be less than the other time.
#+begin_src emacs-lisp
(defun publish-empty-time ()
"Get an empty time value."
(let ((current-time (current-time)))
(time-subtract current-time current-time)))
#+end_src
Next up is =publish-get-latest-modified-time= which takes a list of files and finds the latest time at which any of the files were modified.
#+begin_src emacs-lisp
(defun publish-get-latest-modified-time (files)
"Get the latest modification time of any file from the list FILES."
(car
(last
(sort (mapcar #'org-publish-cache-mtime-of-src files) #'time-less-p))))
#+end_src
I don't know why exactly, but there is a =time-less-p= and a =time-equal-p=, but no =time-greater-p= or any other. Just for clarity I decided to implement ~publish-time>=~ which is just the complement of =time-less-p= (because if the time is not less, it has to be either greater than or equal to).
#+begin_src emacs-lisp
(defun publish-time>= (a b)
"Check if time A is greater than or equal to time B."
(not (time-less-p a b)))
#+end_src
Now a bigger function with a bit more meat to it. This function opens an org file and tries to get some information out of it to construct a headline from it with a link to the actual post. This is used to generate the index of all of the posts. It goes through the file trying to find any headings tagged with the tag =summary= and extracts the contents from the first one. This gives us something to put on the index so it's not just a plain list with links.
#+begin_src emacs-lisp
(defun publish-extract-summary-from-file (file props)
"Extract a summary from FILE.
PROPS is used as an aid in getting the right information from the file."
(format "* %s\n:PROPERTIES:\n:CUSTOM_ID: %s\n:PUBDATE: %s\n:RSS_PERMALINK: %s\n:END:\n\n%s\n\n[[file:%s][Read More]]\n\n"
(car (org-publish-find-property file :title props))
(file-name-nondirectory file)
(format-time-string "[%Y-%m-%d %a %H:%M]" (org-timestamp-to-time (car (org-publish-find-property file :date props))))
(file-name-nondirectory file)
(car (org-map-entries (lambda () (let ((element-data (cadr (org-element-at-point))))
(buffer-substring-no-properties
(map-elt element-data :contents-begin)
(map-elt element-data :contents-end))))
"summary"
(list file)))
(file-name-nondirectory file)))
#+end_src
Finally the last function mushes it all together and actually generates an =index.org= file. I just looks through the =posts/= directory and finds any files that start with a date. It compares the last modification time of the =index.org= and that of the latest modified post, and skips generating if the =index.org= is newer. Then it takes the 30 latest posts and extracts the necessary information from them and writes it all into =index.org=.
#+begin_src emacs-lisp
(defun publish-generate-index (props)
"Generate an index from my posts.
Argument PROPS
."
(let* ((index-file (expand-file-name "posts/index.org"))
(index-generated-time (or (and (file-exists-p index-file)
(org-publish-cache-mtime-of-src index-file))
(publish-empty-time)))
(files (directory-files "posts/" t (rx bos (= 8 digit) "-" (= 4 digit) "-" (one-or-more nonl) (not "~") eos)))
(latest-modification-time (publish-get-latest-modified-time files)))
(if (publish-time>= index-generated-time latest-modification-time)
(message "Not generating index...")
(progn
(message "Generating index...")
(with-temp-buffer
(insert "#+title: ryuslash's blog\n")
(insert "#+options: num:nil\n")
(insert "#+html_link_up: /")
(insert "#+html_link_home: /")
(insert "\n")
(apply 'insert
(mapcar (lambda (file) (publish-extract-summary-from-file file props))
(take 30 (reverse files))))
(write-file "posts/index.org"))))))
#+end_src
With that all out of the way I can finally assemble my project.
#+begin_src emacs-lisp :tangle no :noweb-ref projects
("rss"
:base-directory "posts/"
:base-extension "org"
:rss-extension "xml"
:preparation-function publish-generate-index ; (ref:preparation-function)
:publishing-directory "public/posts/"
:publishing-function (org-rss-publish-to-rss) ; (ref:publishing-function)
:html-link-home "https://ryuslash.org/posts/"
:html-link-use-abs-url t
:section-numbers nil
:exclude ".*" ; (ref:exclude)
:include ("index.org")
:table-of-contents nil)
#+end_src
The [[(preparation-function)]] uses my =publish-generate-index= function to create the =index.org= before trying to turn it into an RSS feed. The [[(publishing-function)]] exports the org file to an RSS feed, this is from the =ox-rss= package. And [[(exclude)]] excludes everything, and then we include the =index.org= again so that it only tries to turn the =index.org= into an RSS feed, not anything else.
*** Assets
This is a simple project that just goes around and copies over any asset files (=.svg=, =.jpg=, etc.) from the source directory into the publish directory.
#+begin_src emacs-lisp :tangle no :noweb-ref projects
("assets"
:base-directory "."
:recursive t
:exclude "^public/"
:base-extension "svg\\|png\\|jpg\\|ico"
:publishing-function org-publish-attachment
:publishing-directory "public/")
#+end_src
*** All
This is a convenience project so that I can publish everything all at once. It also establishes the right order in which to do so. Really the only order is that [[RSS]] has to go before [[Posts]] because the RSS preparation function needs to happen before the =index.org= it generates can be included in the Posts export.
#+begin_src emacs-lisp :tangle no :noweb-ref projects
("all" :components ("pages" "rss" "posts" "assets"))
#+end_src
** File footer
Finally I need to add a file footer, again to keep Eldev happy.
#+begin_src emacs-lisp
(provide 'publish)
;;; publish.el ends here
#+end_src
* Footnotes
[fn:1] You have to evaluate it in a special way. Either with {{{kbd(C-M-x)}}} or {{{kbd(C-u C-x C-e)}}}.
# Local Variables:
# org-src-preserve-indentation: t
# End: