Author Archives: David Thompson

call-with-container: Linux containers for GNU Guix

Containers have become a hot topic. Docker, LXC, Rkt, systemd-nspawn,
etc. are popular and make a lot headlines on technology news websites.
There are even entire conferences dedicated to "containerization."
The next release of GNU Guix, a purely functional package manager,
will feature basic support for Linux containers, dubbed
call-with-container. GNU Guix is written in the Guile Scheme
programming language. Thus, in typical Scheme fashion, we have used
the call-with-* naming convention (like our old friend
call-with-current-continuation, for example) to refer to the
container implementation, which allows the user to apply an arbitrary
Scheme procedure in a new, containerized process. For those familiar
with Guile, it’s like a fancier version of primitive-fork. In
addition to forking via clone(2), call-with-container unshares
resources such as the network, mount, and user namespaces from the
host system and creates a temporary file system in a chroot (though we
actually use pivot_root instead), among other things.

So, given the fast-paced world of Linux container development and the
companies investing huge sums of money into the aformentioned tools,
why add another implementation to the mix? What’s wrong with the
status quo? What can Guix possibly add? Let’s dive into the
motivation behind Guix containers, how they differ from the status
quo, and how they solve issues that other implementations cannot.

The Trouble with Disk Images

The term "container" has in practice come to mean a disk image whose
binaries are run in a lightweight virtualization with kernel
namespaces. Docker and friends work with raw disk images, or layers
of them, as their fundamental data unit. There are several issues
with this approach. This presents issues with regards to disk and
memory usage and file system complexity. Containers typically include
the minimal core software of a GNU/Linux distribution within them,
plus the extra software that is application-specific.

WRITEME: Deduplicating software amongst containers via
content-addressable storage (/gnu/store) and full dependency graph

Imperative vs. Functional – The Dockerfile Problem

WRITEME: Functional > Imperative. Mention issues with imperative
Dockerfiles and how the order of operations influences how well the
cache is utilized. Docker cannot possibly know the details that Guix
knows because Docker defers to other package managers to do things
that are opaque to it.

Guix and GuixSD Containers

WRITEME: Simple, unprivileged containers with ‘guix environment’. Add
example for sharing file systems from the host.

WRITEME: GuixSD containers with ‘guix system container’. Mention the
use of a real init system (dmd) and the full-system configuration
language.

WRITEME: An arbitrary Scheme script that launches a container, showing
that (guix build linux-container) is a generic library to be re-used as
you wish.

Future Work

WRITEME: The future: cgroups, networking, orchestration. Reach out
for contributors.

As an aside, before I bid you farewell, dear reader: I live in the
Boston area and I’m looking for local meetups and conferences that
would welcome a technical presentation about GNU Guix and the benefits
of the functional package and configuration management paradigm. If
you have the connections and I have piqued your interest, please send
an email to davet at gnu dot org.

From the blog dthompson by David Thompson and used with permission of the author. All other rights reserved by the author.

Web Development with Guile Scheme

Guile Scheme is a wonderful general-purpose programming language. I
use it for simple scripting, systems programming, game programming,
and of course, web programming. I think that, despite not having the
wealth of libraries available in Ruby/Java/etc., Guile provides an
excellent environment for web programming thanks to its advanced
features: First-class and higher-order functions, hygienic macros,
pattern matching, quasiquote, and REPL server.

Guile ships with a nice, small set of HTTP modules in the (web)
namespace that expose data types for URIs, HTTP requests, and HTTP
responses. Guile also provides an HTTP client and server. What makes
Guile’s web modules stand out from a lot of other languages is that
they use expressive data types to represent URIs and HTTP headers
rather than treating them as strings. This eliminates an entire class
of bugs and security vulnerabilities caused by mishandling "stringly
typed" data, such as header injection. Using expressive data types
will be a common theme amongst all components of the web programming
tools I will describe below. Now, let’s see how to write a web
application with Guile!

Handling Requests

WRITEME: Status quo: String-based routes

WRITEME: Guile: Pattern Matching

Rendering Responses

WRITEME: Status quo: String-based templating

injection bugs

WRITEME: Guile: SXML and SJSON

Iterative Development

WRITEME: Status quo: Auto-reloading changed files

WRITEME: Guile: REPL server

From the blog dthompson by David Thompson and used with permission of the author. All other rights reserved by the author.

Ruby on Guix

I’ve been working with Ruby professionally for over 3 years now and
I’ve grown frustrated with two of its most popular development tools:
RVM and Bundler. For those that may not know, RVM is the Ruby version
manager and it allows unprivileged users to download, compile,
install, and manage many versions of Ruby instead of being stuck with
the one that is installed globally by your distro’s package manager.
Bundler is the tool that allows developers to keep a version
controlled "Gemfile" that specifies all of the project’s dependencies
and provides utilities to install and update those gems. These tools
are crucial because Ruby developers often work with many applications
that use different versions of Ruby and/or different versions of gems
such as Rails. Traditional GNU/Linux distributions install packages
to the global /usr directory, limiting users to a single version
of Ruby and associated gems, if they are packaged at all. Traditional
package management fails to meet the needs of a lot of users, so many
niche package managers have been developed to supplement them.

Taking a step back, it becomes apparent that dependency isolation is a
general problem that isn’t confined to software written in Ruby: Node
has npm and nvm, Python has pip and virtualenv, and so on. A big
limitation of all these language-specific package managers is that
they cannot control what is outside of their language domain. In
order to use RVM to successfully compile a version of Ruby, you need
to make sure you have the GCC toolchain, OpenSSL, readline, libffi,
etc. installed using the system package manager (note: I’ve seen RVM
try to build prerequisites like OpenSSL before, which I then disabled
to avoid duplication and security issues and I recommend you do the
same.) In order to use Bundler to install Nokogiri, you need to make
sure libxml2 has been installed using the system package manager. If
you work with more than a single language, the number of different
package management tools needed to get work done is staggering. For
web applications, it’s not uncommon to use RVM, Bundler, NPM, Bower,
and the system package manager simultaneously to get all of the
necessary programs and libraries. Large web applications are
notoriously difficult to deploy, and companies hire a bunch of
operations folk like me to try to wrangle it all.

Anyway, let’s forget about Node, Python, etc. and just focus on Ruby.
Have you or someone you work with encountered hard to debug issues and
Git merge conflicts due to a problem with Gemfile.lock? Bundler’s
fast and loose versioning in the Gemfile (e.g. rails >= 4.0)
causes headaches when different users update different gems at
different times and check the resulting auto-generated
Gemfile.lock into the repository. Have you ever been frustrated
that it’s difficult to deduplicate gems that are shared between
multiple bundled gem sets? Have you looked at the RVM home page
and been frustrated that they recommend you to curl | bash to
install their software? Have you been annoyed by RVM’s strange system
of overriding shell built-ins in order to work its magic? I’m not
sure how you feel, dear reader, but my Ruby environments feel like one
giant, brittle hack, and I’m often enough involved in fixing issues
with them on my own workstation, that of my colleagues, and on
production servers.

So, if you’re still with me, what do we do about this? How can we
work to improve upon the status quo? Just use Docker? Docker is
helpful, and certainly much better than no isolation at all, but it
hides the flaws of package management inside an opaque disk image and
restricts the environments in which your application is built to
function. The general problem of dependency isolation is orthogonal
to the runtime environment, be it container, virtual machine, or "bare
metal." Enter functional package management. What does it mean for a
package manager to be functional? GNU Guix, the functional package
manager that I contribute to and recommend, has this to say:

GNU Guix is a functional package management tool for the GNU
system. Package management consists of all activities that relate
to building packages from sources, honoring their build-time and
run-time dependencies, installing packages in user environments,
upgrading installed packages to new versions or rolling back to a
previous set, removing unused software packages, etc.

The term functional refers to a specific package management
discipline. In Guix, the package build and installation process
is seen as a function, in the mathematical sense. That function
takes inputs, such as build scripts, a compiler, and libraries,
and returns an installed package.

Guix has a rich set of features, some of which you may find in other
package managers, but not all of them (unless you use another
functional package manager such as Nix.) Gem/Bundler can do
unprivileged gem installation, but it cannot do transactional upgrades
and rollbacks or install non-Ruby dependencies. Dpkg/yum/pacman can
install all build-time and rumtime dependencies, but it cannot do
unprivileged package installation to isolated user environments. And
none of them can precisely describe the full dependency graph (all
the way down to the C compiler’s compiler) but Guix can.

Guix is written in Guile, an implementation of the Scheme programming
language. The upcoming release of Guix will feature a Ruby build
system that captures the process of installing gems from .gem
archives and a RubyGems import utility to make it easier to write Guix
packages by using the metadata available on https://rubygems.org.
Ruby developers interested in functional package management are
encouraged to try packaging their gems (and dependencies) for Guix.

Now, how exactly can Guix replace RVM and Bundler? Guix uses an
abstraction called a "profile" that represents a user-defined set of
packages that should work together. Think of it as having many
/usr file system trees that can be used in isolation from the
others (without invoking virtualization technologies such as virtual
machines or containers.) To install multiple versions of Ruby and
various gems, the user need only create a separate profile for them:

guix package --profile=project-1 --install ruby-2.2 ruby-rspec-3
# Hypothetical packages:
guix package --profile=project-2 --install ruby-1.9 ruby-rspec-2

A profile is a "symlink forest" that is the union of all the packages
it includes, and files are deduplicated among all of them. To
actually use the profile, the relevant environment variables must be
configured. Guix is aware of such variables, and can tell you what to
set by running the following:

guix package --search-paths --profile=project-1

Additionally, you can also create ad-hoc development environments with
the guix environment tool. This tool will spawn a sub-shell (or
another program of your choice) in an environment in which a set of
specified packages are available. This is my preferred method as it
automagically sets all of the environment variables for me and Guix is
free to garbage collect the packages when I close the sub-shell:

# Launch a Ruby REPL with ActiveSupport available.
guix environment --ad-hoc ruby ruby-activesupport -E irb

In order to make this environment reproducible for others, I recommend
keeping a package.scm file in version control that describes the
complete dependency graph for your project, as well as metadata such
as the license, version, and description:

(use-modules (guix packages)
             (guix licenses)
             (guix build-system ruby)
             (gnu packages)
             (gnu packages version-control)
             (gnu packages ssh)
             (gnu packages ruby))

(package
  (name "cool-ruby-project")
  (version "1.0")
  (source #f) ; not needed just to create dev environment
  (build-system ruby-build-system)
  ;; These correspond roughly to "development" dependencies.
  (native-inputs
   `(("git" ,git)
     ("openssh" ,openssh)
     ("ruby-rspec" ,ruby-rspec)))
  (propagated-inputs
   `(("ruby-pg" ,ruby-pg)
     ("ruby-nokogiri" ,ruby-nokogiri)
     ("ruby-i18n" ,ruby-i18n)
     ("ruby-rails" ,ruby-rails)))
  (synopsis "A cool Ruby project")
  (description "This software does some cool stuff, trust me.")
  (home-page "https://example.com")
  (license expat))

With this package file, it is simple to an instantiate a development
environment:

guix environment -l package.scm

I’m not covering it in this post, but properly filling out the blank
source field above would allow for building development snapshots,
including running the test suite, in an isolated build container using
the guix build utility. This is a very useful when composed with
a continuous integration system. Guix itself uses Hydra as its CI
system to perform all package builds.

As mentioned earlier, one of the big advantages of writing Guix
package recipes is that the full dependency graph can be captured,
including non-Ruby components. The pg gem provides a good example:

(define-public ruby-pg
  (package
    (name "ruby-pg")
    (version "0.18.2")
    (source
     (origin
       (method url-fetch)
       (uri (rubygems-uri "pg" version))
       (sha256
        (base32
         "1axxbf6ij1iqi3i1r3asvjc80b0py5bz0m2wy5kdi5xkrpr82kpf"))))
    (build-system ruby-build-system)
    (arguments
     '(#:test-target "spec"))
    ;; Native inputs are used only at build and test time.
    (native-inputs
     `(("ruby-rake-compiler" ,ruby-rake-compiler)
       ("ruby-hoe" ,ruby-hoe)
       ("ruby-rspec" ,ruby-rspec)))
    ;; Native extension links against PostgreSQL shared library.
    (inputs
     `(("postgresql" ,postgresql)))
    (synopsis "Ruby interface to PostgreSQL")
    (description "Pg is the Ruby interface to the PostgreSQL RDBMS.  It works
with PostgreSQL 8.4 and later.")
    (home-page "https://bitbucket.org/ged/ruby-pg")
    (license license:ruby)))

Note how the recipe specifies the PostgreSQL dependency. Below is the
dependency graph for ruby-pg as produced by guix graph, excluding
the GCC compiler toolchain and other low-level tools for brevity.
Pretty neat, eh?

Given that Guix doesn’t yet have many gems packaged (help wanted), it
can still be advantageous to use it for getting more up-to-date
packages than many distros provide, but in conjuction with Bundler for
fetching Ruby gems. This gets RVM out of your hair whilst creating a
migration path away from Bundler at a later time once the required
gems have been packaged:

cd my-project/
guix environment --ad-hoc ruby bundler libxml2 libxslt # etc.
# A small bash script can be used to make these gem sets.
mkdir .gems
export GEM_HOME=$PWD/.gems
export GEM_PATH=$GEM_HOME:$GEM_PATH
export PATH=$GEM_HOME/bin:$PATH
bundle install

As you’ve seen in the above package snippets, Guix package definitions
are typically very short and rather easy to write yourself. The
guix import gem tool was made to lower the barrier even more by
generating most of the boilerplate code. For example:

guix import gem pry

Produces this Scheme code:

(package
  (name "ruby-pry")
  (version "0.10.1")
  (source
    (origin
      (method url-fetch)
      (uri (rubygems-uri "pry" version))
      (sha256
        (base32
          "1j0r5fm0wvdwzbh6d6apnp7c0n150hpm9zxpm5xvcgfqr36jaj8z"))))
  (build-system ruby-build-system)
  (propagated-inputs
    `(("ruby-coderay" ,ruby-coderay)
      ("ruby-method-source" ,ruby-method-source)
      ("ruby-slop" ,ruby-slop)))
  (synopsis
    "An IRB alternative and runtime developer console")
  (description
    "An IRB alternative and runtime developer console")
  (home-page "http://pryrepl.org")
  (license expat))

One still has to package the propagated inputs if they aren’t yet
available, add the necessary inputs for building native extensions if
needed, and fiddle with the native inputs needed to run the test
suite, but for most pure Ruby gems this gets you close to a working
package quickly.

In conclusion, while support for Ruby in Guix is still in its early
days, I hope that you have seen the benefits that using a
general-purpose, functional package manager can bring to your Ruby
environments (and all other environments, too.) For more information
about Guix concepts, installation instructions, programming interface,
and tools, please refer to the official manual. Check out the
help page for ways to contact the development team for help or to
report bugs. If you are interested in getting your hands dirty,
please contribute. Besides contributions of code, art, and docs,
we also need hardware donations to grow our build farm to meet the
needs of all our users. Happy hacking!

From the blog dthompson by David Thompson and used with permission of the author. All other rights reserved by the author.

Rendering HTML with SXML and GNU Guile

GNU Guile provides modules for working with XML documents called SXML.
SXML provides an elegant way of writing XML documents as s-expressions
that can be easily manipulated in Scheme. Here’s an example:

(sxml->xml '(foo (bar (@ (attr "something")))))
<foo><bar attr="something" /></foo>

I don’t know about you, but I work with HTML documents much more often
than XML. Since HTML is very similar to XML, we should be able to
represent it with SXML, too!

(sxml->xml '(html
             (head
              (title "Hello, world!")
              (script (@ (src "foo.js"))))
             (body
              (h1 "Hello!"))))
<html>
  <head>
    <title>Hello, world!</title>
    <script src="foo.js" /> <!-- what? -->
  </head>
  <body>
    <h1>Hello!</h1>
  </body>
</html>

That <script> tag doesn’t look right! Script tags don’t close
themselves like that. Well, we could hack around it:

(sxml->xml '(html
             (head
              (title "Hello, world!")
              (script (@ (src "foo.js")) ""))
             (body
              (h1 "Hello!"))))
<html>
  <head>
    <title>Hello, world!</title>
    <script src="foo.js"></script>
  </head>
  <body>
    <h1>Hello!</h1>
  </body>
</html>

Note the use of the empty string in (script (@ (src "foo.js"))
"")
. The output looks correct now, great! But what about the other
void elements? We’ll have to remember to use the empty string hack
each time we use one. That doesn’t sound very elegant.

Furthermore, text isn’t even escaped properly!

(sxml->xml "Copyright © 2015  David Thompson <davet@gnu.org>")
Copyright © 2015  David Thompson &lt;davet@gnu.org&gt;

The < and > braces were escaped, but © should’ve been
rendered as &copy;. Why does this fail, too? Is there a bug in
SXML?

There’s no bug. The improper rendering happens because HTML, while
similar to XML, has a bunch of different syntax rules. Instead of
using sxml->xml, a new procedure that is tailored to the HTML
syntax is needed. Introducing sxml->html:

(define* (sxml->html tree #:optional (port (current-output-port)))
  "Write the serialized HTML form of TREE to PORT."
  (match tree
    (() *unspecified*)
    (('doctype type)
     (doctype->html type port))
    ;; Unescaped, raw HTML output
    (('raw html)
     (display html port))
    (((? symbol? tag) ('@ attrs ...) body ...)
     (element->html tag attrs body port))
    (((? symbol? tag) body ...)
     (element->html tag '() body port))
    ((nodes ...)
     (for-each (cut sxml->html <> port) nodes))
    ((? string? text)
     (string->escaped-html text port))
    ;; Render arbitrary Scheme objects, too.
    (obj (object->escaped-html obj port))))

In addition to being aware of void elements and escape characters, it
can also render '(doctype "html") as <!DOCTYPE html>, or
render an unescaped HTML string using '(raw "frog &amp; toad").

Here’s the full version of my (sxml html) module. It’s quite
brief, if you don’t count the ~250 lines of escape codes! This code
requires Guile 2.0.11 or greater.

Happy hacking!

(define-module (sxml html)
  #:use-module (sxml simple)
  #:use-module (srfi srfi-26)
  #:use-module (ice-9 match)
  #:use-module (ice-9 format)
  #:use-module (ice-9 hash-table)
  #:export (sxml->html))

(define %void-elements
  '(area
    base
    br
    col
    command
    embed
    hr
    img
    input
    keygen
    link
    meta
    param
    source
    track
    wbr))

(define (void-element? tag)
  "Return #t if TAG is a void element."
  (pair? (memq tag %void-elements)))

(define %escape-chars
  (alist->hash-table
   '((#" . "quot")
     (#& . "amp")
     (#' . "apos")
     (#< . "lt")
     (#> . "gt")
     (#¡ . "iexcl")
     (#¢ . "cent")
     (#£ . "pound")
     (#¤ . "curren")
     (#¥ . "yen")
     (#¦ . "brvbar")
     ( . "sect")
     (#¨ . "uml")
     (#© . "copy")
     (#ª . "ordf")
     (#« . "laquo")
     (#¬ . "not")
     (#® . "reg")
     (#¯ . "macr")
     (#° . "deg")
     (#± . "plusmn")
     (#² . "sup2")
     (#³ . "sup3")
     (#´ . "acute")
     (#µ . "micro")
     (# . "para")
     (#· . "middot")
     (#¸ . "cedil")
     (#¹ . "sup1")
     (#º . "ordm")
     (#» . "raquo")
     (#¼ . "frac14")
     (#½ . "frac12")
     (#¾ . "frac34")
     (#¿ . "iquest")
     (#À . "Agrave")
     (#Á . "Aacute")
     ( . "Acirc")
     (#Ã . "Atilde")
     (#Ä . "Auml")
     (#Å . "Aring")
     (#Æ . "AElig")
     (#Ç . "Ccedil")
     (#È . "Egrave")
     (#É . "Eacute")
     (#Ê . "Ecirc")
     (#Ë . "Euml")
     (#Ì . "Igrave")
     (#Í . "Iacute")
     (#Î . "Icirc")
     (#Ï . "Iuml")
     (#Ð . "ETH")
     (#Ñ . "Ntilde")
     (#Ò . "Ograve")
     (#Ó . "Oacute")
     (#Ô . "Ocirc")
     (#Õ . "Otilde")
     (#Ö . "Ouml")
     (#× . "times")
     (#Ø . "Oslash")
     (#Ù . "Ugrave")
     (#Ú . "Uacute")
     (#Û . "Ucirc")
     (#Ü . "Uuml")
     (#Ý . "Yacute")
     (#Þ . "THORN")
     (#ß . "szlig")
     (#à . "agrave")
     (#á . "aacute")
     (#â . "acirc")
     (#ã . "atilde")
     (#ä . "auml")
     (#å . "aring")
     (#æ . "aelig")
     (#ç . "ccedil")
     (#è . "egrave")
     (#é . "eacute")
     (#ê . "ecirc")
     (#ë . "euml")
     (#ì . "igrave")
     (#í . "iacute")
     (#î . "icirc")
     (#ï . "iuml")
     (#ð . "eth")
     (#ñ . "ntilde")
     (#ò . "ograve")
     (#ó . "oacute")
     (#ô . "ocirc")
     (#õ . "otilde")
     (#ö . "ouml")
     (#÷ . "divide")
     (#ø . "oslash")
     (#ù . "ugrave")
     (#ú . "uacute")
     (#û . "ucirc")
     (#ü . "uuml")
     (#ý . "yacute")
     (#þ . "thorn")
     (#ÿ . "yuml")
     (#Œ . "OElig")
     (#œ . "oelig")
     (#Š . "Scaron")
     (#š . "scaron")
     (#Ÿ . "Yuml")
     (#ƒ . "fnof")
     (#ˆ . "circ")
     (#˜ . "tilde")
     (#Α . "Alpha")
     (#Β . "Beta")
     (#Γ . "Gamma")
     (#Δ . "Delta")
     (#Ε . "Epsilon")
     (#Ζ . "Zeta")
     (#Η . "Eta")
     (#Θ . "Theta")
     (#Ι . "Iota")
     (#Κ . "Kappa")
     (#Λ . "Lambda")
     (#Μ . "Mu")
     (#Ν . "Nu")
     (#Ξ . "Xi")
     (#Ο . "Omicron")
     (#Π . "Pi")
     (#Ρ . "Rho")
     (#Σ . "Sigma")
     (#Τ . "Tau")
     (#Υ . "Upsilon")
     (#Φ . "Phi")
     (#Χ . "Chi")
     (#Ψ . "Psi")
     (#Ω . "Omega")
     (#α . "alpha")
     (#β . "beta")
     (#γ . "gamma")
     (#δ . "delta")
     (#ε . "epsilon")
     (#ζ . "zeta")
     (#η . "eta")
     (#θ . "theta")
     (#ι . "iota")
     (#κ . "kappa")
     (#λ . "lambda")
     (#μ . "mu")
     (#ν . "nu")
     (#ξ . "xi")
     (#ο . "omicron")
     (#π . "pi")
     (#ρ . "rho")
     (#ς . "sigmaf")
     (#σ . "sigma")
     (#τ . "tau")
     (#υ . "upsilon")
     (#φ . "phi")
     (#χ . "chi")
     (#ψ . "psi")
     (#ω . "omega")
     (#ϑ . "thetasym")
     (#ϒ . "upsih")
     (#ϖ . "piv")
     (# . "ensp")
     (# . "emsp")
     (# . "thinsp")
     (# . "ndash")
     (# . "mdash")
     (# . "lsquo")
     (# . "rsquo")
     (# . "sbquo")
     (# . "ldquo")
     (# . "rdquo")
     (# . "bdquo")
     (# . "dagger")
     (# . "Dagger")
     (# . "bull")
     (# . "hellip")
     (# . "permil")
     (# . "prime")
     (# . "Prime")
     (# . "lsaquo")
     (# . "rsaquo")
     (# . "oline")
     (# . "frasl")
     (# . "euro")
     (# . "image")
     (# . "weierp")
     (# . "real")
     (# . "trade")
     (# . "alefsym")
     (# . "larr")
     (# . "uarr")
     (# . "rarr")
     (# . "darr")
     (# . "harr")
     (# . "crarr")
     (# . "lArr")
     (# . "uArr")
     (# . "rArr")
     (# . "dArr")
     (# . "hArr")
     (# . "forall")
     (# . "part")
     (# . "exist")
     (# . "empty")
     (# . "nabla")
     (# . "isin")
     (# . "notin")
     (# . "ni")
     (# . "prod")
     (# . "sum")
     (# . "minus")
     (# . "lowast")
     (# . "radic")
     (# . "prop")
     (# . "infin")
     (# . "ang")
     (# . "and")
     (# . "or")
     (# . "cap")
     (# . "cup")
     (# . "int")
     (# . "there4")
     (# . "sim")
     (# . "cong")
     (# . "asymp")
     (# . "ne")
     (# . "equiv")
     (# . "le")
     (# . "ge")
     (# . "sub")
     (# . "sup")
     (# . "nsub")
     (# . "sube")
     (# . "supe")
     (# . "oplus")
     (# . "otimes")
     (# . "perp")
     (# . "sdot")
     (# . "vellip")
     (# . "lceil")
     (# . "rceil")
     (# . "lfloor")
     (# . "rfloor")
     (# . "lang")
     (# . "rang")
     (# . "loz")
     (# . "spades")
     (# . "clubs")
     (# . "hearts")
     (# . "diams"))))

(define (string->escaped-html s port)
  "Write the HTML escaped form of S to PORT."
  (define (escape c)
    (let ((escaped (hash-ref %escape-chars c)))
      (if escaped
          (format port "&~a;" escaped)
          (display c port))))
  (string-for-each escape s))

(define (object->escaped-html obj port)
  "Write the HTML escaped form of OBJ to PORT."
  (string->escaped-html
   (call-with-output-string (cut display obj <>))
   port))

(define (attribute-value->html value port)
  "Write the HTML escaped form of VALUE to PORT."
  (if (string? value)
      (string->escaped-html value port)
      (object->escaped-html value port)))

(define (attribute->html attr value port)
  "Write ATTR and VALUE to PORT."
  (format port "~a="" attr)
  (attribute-value->html value port)
  (display #" port))

(define (element->html tag attrs body port)
  "Write the HTML TAG to PORT, where TAG has the attributes in the
list ATTRS and the child nodes in BODY."
  (format port "<~a" tag)
  (for-each (match-lambda
             ((attr value)
              (display #space port)
              (attribute->html attr value port)))
            attrs)
  (if (and (null? body) (void-element? tag))
      (display " />" port)
      (begin
        (display #> port)
        (for-each (cut sxml->html <> port) body)
        (format port "</~a>" tag))))

(define (doctype->html doctype port)
  (format port "<!DOCTYPE ~a>" doctype))

(define* (sxml->html tree #:optional (port (current-output-port)))
  "Write the serialized HTML form of TREE to PORT."
  (match tree
    (() *unspecified*)
    (('doctype type)
     (doctype->html type port))
    ;; Unescaped, raw HTML output
    (('raw html)
     (display html port))
    (((? symbol? tag) ('@ attrs ...) body ...)
     (element->html tag attrs body port))
    (((? symbol? tag) body ...)
     (element->html tag '() body port))
    ((nodes ...)
     (for-each (cut sxml->html <> port) nodes))
    ((? string? text)
     (string->escaped-html text port))
    ;; Render arbitrary Scheme objects, too.
    (obj (object->escaped-html obj port))))

From the blog dthompson by David Thompson and used with permission of the author. All other rights reserved by the author.

Reproducible Development Environments with GNU Guix

If you’re a software developer, then you probably know very well that
setting up a project’s development environment for the first time can
be a real pain. Installing all of the necessary dependencies using
your system’s package manager can be very tedious. To "solve" this
problem, we have resorted to inventing new package managers and
dependency bundlers for pretty much every programming language. Ruby
has rubygems and bundler, Python has pip and virtualenv, PHP has
composer, node.js has npm, and so on. Wouldn’t it be nice to instead
have a single package manager that can handle it all? Enter GNU
Guix
, a purely functional package manager and GNU/Linux
distribution. Using Guix, you can easily create a development
environment for any software project using the guix environment
tool.

guix environment is a new utility added in Guix 0.8, which should
be released in a few weeks. It accepts one on more packages as input
and produces a new shell environment in which all of the dependencies
for those packages are made available. For example, guix
environment emacs
will get you everything you need to build GNU
Emacs from a source release tarball. By default, your $SHELL is
spawned, but you may opt to use any arbitrary shell command instead.
If you’ve used Nix before, you may notice that this sounds exactly
like nix-shell, and that’s because it was what inspired me to
write guix environment.

Now, let’s take a look at an example. One of my hobby projects is
Sly, a game engine written in Guile Scheme. Here’s the relevant Guix
code that will produce a complete development environment:

;;; Copyright (C) 2014 David Thompson <davet@gnu.org>
;;;
;;; Sly is free software: you can redistribute it and/or modify it
;;; under the terms of the GNU General Public License as published by
;;; the Free Software Foundation, either version 3 of the License, or
;;; (at your option) any later version.

(use-modules (guix packages)
             (guix licenses)
             (guix build-system gnu)
             (gnu packages)
             (gnu packages autotools)
             (gnu packages guile)
             (gnu packages gl)
             (gnu packages pkg-config)
             (gnu packages sdl)
             (gnu packages maths)
             (gnu packages image))

;; The development environment needs a tweaked LD_LIBRARY_PATH for
;; finding libfreeimage.
(define freeimage
  (package (inherit freeimage)
    (native-search-paths
     (list (search-path-specification
            (variable "LD_LIBRARY_PATH")
            (directories '("lib")))))))

(package
  (name "sly")
  (version "0.0")
  (source #f)
  (build-system gnu-build-system)
  (inputs
   `(("pkg-config" ,pkg-config)
     ("autoconf" ,autoconf)
     ("automake" ,automake)
     ("guile" ,guile-2.0)
     ("guile-sdl" ,guile-sdl)
     ("guile-opengl" ,guile-opengl)
     ("gsl" ,gsl)
     ("freeimage" ,freeimage)
     ("mesa" ,mesa)))
  (synopsis "2D/3D game engine for GNU Guile")
  (description "Sly is a 2D/3D game engine written in Guile Scheme.
Sly differs from most game engines in that it emphasizes functional
reactive programming and live coding.")
  (home-page "https://gitorious.org/sly/sly")
  (license gpl3+))

You may have noticed that the source field has been set to false.
This is because the package is not for building and installing. It’s
sole purpose is to provide the necessary software needed to build Sly
from a fresh git checkout.

Assuming this code is in a file called package.scm, you can simply
run guix environment -l package.scm to spawn a shell for hacking
on Sly. By default, the environment created is an augmented version
of your pre-existing shell’s environment. This is convenient when you
want to use your installed software in the new environment, such as
git. However, it’s important to make sure that the environment really
does have everything needed for development without relying on any
impurities introduced by your existing environment. Without verifying
this, new developers might be frustrated to find out that the
environment provided to them is incomplete. To verify, pass the
--pure flag and build from scratch.

So, guix environment is a pretty nice way to acquire all the
dependencies you need to work on a project, but there is still a lot
of room for improvement. What if you were working on a web
application that required a running PostgreSQL database, Redis server,
and XMPP server? It would be really great if Guix could handle
setting this up for you, too, a la Vagrant. To do so, guix
environment
could use the existing features of guix system to
spawn a new virtual machine that shared the host system’s package
store and the project’s source tree, and then spawn a shell with your
development environment. I hope to implement this in the
not-too-distant future. Until next time, happy hacking!

From the blog dthompson by David Thompson and used with permission of the author. All other rights reserved by the author.

Guile-2D is now named "Sly"

Guile-2D the a working title for my game engine written in Guile
Scheme. The name has become very limiting since I realized that it
wouldn’t be much extra work to support 3D graphics. After much
indecision, I’ve finally decided on an official name: Sly. I think
it’s a great name. It’s short, easy to type, and slyness is one of
the definitions of “guile”.

In other news:

  • Sly has a new contributor!
    Jordan Russel has written a
    new module
    that provides joystick input support.

  • I have written a
    module for describing animations.

  • I have been slowly working on a scene graph implementation that
    plays well with the functional reactive programming API.

  • As mentioned above, 3D graphics support is on the way! So far, I
    have implemented a perspective projection matrix, a “look at”
    matrix, and a cube primitive.

3D Scene Graph

Check out the Sly source code repository on
Gitorious!

From the blog dthompson by David Thompson and used with permission of the author. All other rights reserved by the author.

Live Asset Reloading with guile-2d

Guile-2d provides a dynamic environment in which a developer can build
a game incrementally as it runs via the Guile REPL. It’s nice to be
able to hot-swap code and have the running game reflect the changes
made, but what about the game data files? If an image file or other
game asset is modified, it would be nice if the game engine took
notice and reloaded it automatically. This is what guile-2d’s live
asset reloading feature does.

The new (2d live-reload) module provides the live-reload
procedure. live-reload takes a procedure like load-texture
and returns a new procedure that adds the live reload magic. The new
procedure returns assets wrapped in a signal, a time-varying value. A
coroutine is started that periodically checks if the asset file has
been modified, and if so, reloads the asset and propagates it via the
signal. Game objects that depend on the asset will be regenerated
immediately.

Here’s some example code:

(define load-texture/live
  (live-reload load-texture))

(define-signal texture
  (load-texture/live "images/p1_front.png"))

(define-signal sprite
  (signal-map
   (lambda (texture)
     (make-sprite texture
                  #:position (vector2 320 240)))
   texture))

load-texture/live loads textures and reloads them when they change
on disk. Every time the texture is reloaded, the sprite is
regenerated using the new texture.

Here’s a screencast to see live reloading in action:

Guile-2d is ever-so-slowly approaching a 0.2 release. Stay tuned!

From the blog dthompson by David Thompson and used with permission of the author. All other rights reserved by the author.

Functional Reactive Programming in Scheme with guile-2d

Last month, the GNU Guile project celebrated the 3rd anniversary of
its 2.0 release with a hacker potluck. Guilers were encouraged to
bring a tasty hack to the mailing list to share with everyone. My
dish was a simple functional reactive programming library.

Functional reactive programming (FRP) provides a way to describe
time-varying values and their relationships using a functional and
declarative programming style. To understand what this means, let’s
investigate a typical variable definition in Scheme. The expression
(define c (+ a b)) defines the variable c to be the sum of
variables a and b at the time of evaluation. If a or
b is assigned a new value, c remains the same. However, for
applications that deal with state that transforms over time, it would
be convenient if we could define c to react to changes in a
and b by recomputing the sum. Contrast this approach with the
more traditional style of modeling dynamic state via events and
callbacks. A lot of programmers, myself included, have written code
with so many callbacks that the resulting program is unmaintainable
spaghetti code. Callback hell is real, but if you accept FRP into
your heart then you will be saved!

By now you’re probably wondering: "What the hell does all this mean?"
So, here’s a real-world example of guile-2d’s FRP API:

(define-signal position
  (signal-fold v+ (vector2 320 240)
               (signal-map (lambda (v)
                             (vscale v 4))
                           (signal-sample game-agenda 1 key-arrows))))

In guile-2d, time-varying values are called "signals". The above
signal describes a relationship between the arrow keys on the keyboard
and the position of the player. signal-sample is used to trigger
a signal update upon every game tick that provides the current state
of the arrow keys. key-arrows is a vector2 that maps to the
current state of the arrow keys, allowing for 8 direction movement.
This vector2 is then scaled 4x to make the player move faster.
Finally, the scaled vector is added to the previous player position
via signal-fold. The player’s position is at (320, 240)
initially. As you can see, there are no callbacks and explicit
mutation needed. Those details have been abstracted away, freeing the
programmer to focus on more important things.

I think it’s helpful to see FRP in action to really appreciate the
magic. So, check out this screencast!

To see all of the juicy implementation details, check out the git
repository
. Thanks for following along!

From the blog dthompson by David Thompson and used with permission of the author. All other rights reserved by the author.

A Cooperative REPL Server for Guile 2.0.10

The next release of GNU Guile, 2.0.10, is to be released "real soon
now". My contribution to this release is the new (system repl
coop-server)
module. This module introduces a useful variant of the
REPL server that I’ve named the "cooperative" REPL server. It is
cooperative because it can be integrated with single-threaded programs
without the thread synchronization issues present in the ordinary REPL
server.

Using delimited continuations and mvars (another new feature coming in
Guile 2.0.10), it was possible to create a REPL server whose clients
all ran in the context of a single thread. The cooperative server
puts the user in control of when it is safe to evaluate expressions
entered at the REPL prompt.

Here’s an example of how to use it:

(use-modules (system repl coop-server))

(define server (spawn-coop-repl-server))

(while #t
  (poll-coop-repl-server server)
  (sleep 1))

That’s all it takes. There are only 2 public procedures!
spawn-coop-repl-server creates a new cooperative REPL server
object and starts listening for clients. poll-coop-repl-server
applies pending operations for the server.

Now that I’ve explained the API, onto the implementation details!
Although the REPLs run in the context of a single thread, there are
other threads needed to make it all work. To avoid thinking too much
about thread synchronization issues, I used the new (ice-9 mvars)
module to provide thread-safe objects for read/write operations. I
used delimited continuations in order to yield control back to the
user from inside the loop.

When the server is spawned, a new thread is created to listen for
client connections. When a client connects, a request to create a new
REPL is put into the server’s "evaluation" mvar, the storage place for
pending server operations. The job of poll-coop-repl-server is to
attempt to take from the evaluation mvar and execute the operation
stored within, if any. In this case, an operation called new-repl
is hanging out along with the client socket. The contents of the mvar
are removed, and a new REPL is started in the main thread.

This new REPL is run within a "prompt", Guile’s implementation of
delimited continuations. The prompt allows the REPL to be paused
temporarily while waiting for user input. Just before the prompt is
aborted, a thunk containing the logic to read user input is stored
within the client’s "read" mvar for yet another thread to use.
Without this additional thread and the use of a prompt, the main
thread would block while waiting for input, defeating the purpose of
the cooperative REPL. The reader thread waits for this thunk to
appear in the read mvar. When it appears, the thunk is applied and
the resulting expression is stored as an eval operation within the
evaluation mvar. When poll-coop-repl-server is called, the REPL
prompt is resumed. The input expression is evaluated in the context
of the main thread, the result is printed, and the process repeats
itself.

That’s all, folks. Thanks for following along. I’m very excited to
use the cooperative REPL server in guile-2d and I hope that others
find it useful as well. Many thanks to Mark Weaver for helping me out
when I got stuck and for all of the helpful code review.

From the blog dthompson by David Thompson and used with permission of the author. All other rights reserved by the author.

Syncing Required Packages in Emacs

I use Emacs on several different computers. To keep my configuration
consistent across all of them, I do what many people do and made the
~/.emacs.d directory a git repository. I don’t like to keep
copies of all of the Elisp extensions that I use, such as paredit and
geiser, in this repository. Instead, I prefer to use package.el
(introduced in Emacs 24) with the MELPA repository. This saves me
from having to manually keep all of the extensions I use up-to-date,
but requires another method to keep useful packages in sync between
computers.

There’s a project called Pallet that solves this problem, but it was
too heavy for my liking. Instead, I wrote a short function that
simply iterates over a list of required packages and installs those
that are not currently installed.

;; Additional packages that I use.
 (setq required-packages
       '(better-defaults
         elfeed
         geiser
         ido-ubiquitous
         js2-mode
         magit
         paredit
         rainbow-delimiters
         smex))

 (defun install-missing-packages ()
   "Install all required packages that haven't been installed."
   (interactive)
   (mapc (lambda (package)
           (unless (package-installed-p package)
             (package-install package)))
         required-packages)
   (message "Installed all missing packages!"))

Now, it’s as easy as typing M-x install-missing-packages RET when
starting Emacs for the first time on a new computer to download all of
the extensions that I need. Note that before calling
install-missing-packages you must have already initialized the
package manager via the package-initialize function. This
approach does require some manual bookkeeping in order to keep the
required-packages list up-to-date with your workflow, but I
haven’t found it to be problematic.

Update: If this solution is too simplistic for you, you should check
out use-package, which reddit user lunayorn pointed out to
me. Thanks!

Check out the comments on reddit.

From the blog dthompson by David Thompson and used with permission of the author. All other rights reserved by the author.