cannam@133: ---
cannam@133: layout: post
cannam@133: title: "Cap'n Proto v0.2: Compiler rewritten Haskell -> C++"
cannam@133: author: kentonv
cannam@133: ---
cannam@133: 
cannam@133: Today I am releasing version 0.2 of Cap'n Proto.  The most notable change: the compiler / code
cannam@133: generator, which was previously written in Haskell, has been rewritten in C++11.  There are a few
cannam@133: other changes as well, but before I talk about those, let me try to calm the angry mob that is
cannam@133: not doubt reaching for their pitchforks as we speak.  There are a few reasons for this change,
cannam@133: some practical, some ideological.  I'll start with the practical.
cannam@133: 
cannam@133: **The practical:  Supporting dynamic languages**
cannam@133: 
cannam@133: Say you are trying to implement Cap'n Proto in an interpreted language like Python.  One of the big
cannam@133: draws of such a language is that you can edit your code and then run it without an intervening
cannam@133: compile step, allowing you to iterate faster.  But if the Python Cap'n Proto implementation worked
cannam@133: like the C++ one (or like Protobufs), you lose some of that: whenever you change your Cap'n Proto
cannam@133: schema files, you must run a command to regenerate the Python code from them.  That sucks.
cannam@133: 
cannam@133: What you really want to do is parse the schemas at start-up -- the same time that the Python code
cannam@133: itself is parsed.  But writing a proper schema parser is harder than it looks; you really should
cannam@133: reuse the existing implementation.  If it is written in Haskell, that's going to be problematic.
cannam@133: You either need to invoke the schema parser as a sub-process or you need to call Haskell code from
cannam@133: Python via an FFI.  Either approach is going to be a huge hack with lots of problems, not the least
cannam@133: of which is having a runtime dependency on an entire platform that your end users may not otherwise
cannam@133: want.
cannam@133: 
cannam@133: But with the schema parser written in C++, things become much simpler.  Python code calls into
cannam@133: C/C++ all the time.  Everyone already has the necessary libraries installed.  There's no need to
cannam@133: generate code, even; the parsed schema can be fed into the Cap'n Proto C++ runtime's dynamic API,
cannam@133: and Python bindings can trivially be implemented on top of that in just a few hundred lines of
cannam@133: code.  Everyone wins.
cannam@133: 
cannam@133: **The ideological:  I'm an object-oriented programmer**
cannam@133: 
cannam@133: I really wanted to like Haskell.  I used to be a strong proponent of functional programming, and
cannam@133: I actually once wrote a complete web server and CMS in a purely-functional toy language of my own
cannam@133: creation.  I love strong static typing, and I find a lot of the constructs in Haskell really
cannam@133: powerful and beautiful.  Even monads.  _Especially_ monads.
cannam@133: 
cannam@133: But when it comes down to it, I am an object-oriented programmer, and Haskell is not an
cannam@133: object-oriented language.  Yes, you can do object-oriented style if you want to, just like you
cannam@133: can do objects in C.  But it's just too painful.  I want to write `object.methodName`, not
cannam@133: `ModuleName.objectTypeMethodName object`.  I want to be able to write lots of small classes that
cannam@133: encapsulate complex functionality in simple interfaces -- _without_ having to place each one in
cannam@133: a whole separate module and ending up with thousands of source files.  I want to be able to build
cannam@133: a list of objects of varying types that implement the same interface without having to re-invent
cannam@133: virtual tables every time I do it (type classes don't quite solve the problem).
cannam@133: 
cannam@133: And as it turns out, even aside from the lack of object-orientation, I don't actually like
cannam@133: functional programming as much as I thought.  Yes, writing my parser was super-easy (my first
cannam@133: commit message was
cannam@133: "[Day 1: Learn Haskell, write a parser](https://github.com/kentonv/capnproto/commit/6bb49ca775501a9b2c7306992fd0de53c5ee4e95)").
cannam@133: But everything beyond that seemed to require increasing amounts of brain bending.  For instance, to
cannam@133: actually encode a Cap'n Proto message, I couldn't just allocate a buffer of zeros and then go
cannam@133: through each field and set its value.  Instead, I had to compute all the field values first, sort
cannam@133: them by position, then concatenate the results.
cannam@133: 
cannam@133: Of course, I'm sure it's the case that if I spent years writing Haskell code, I'd eventually become
cannam@133: as proficient with it as I am with C++.  Perhaps I could un-learn object-oriented style and learn
cannam@133: something else that works just as well or better.  Basically, though, I decided that this was
cannam@133: going to take a lot longer than it at first appeared, and that this wasn't a good use of my
cannam@133: limited resources.  So, I'm cutting my losses.
cannam@133: 
cannam@133: I still think Haskell is a very interesting language, and if works for you, by all means, use it.
cannam@133: I would love to see someone write at actual Cap'n Proto runtime implementation in Haskell.  But
cannam@133: the compiler is now C++.
cannam@133: 
cannam@133: **Parser Combinators in C++**
cannam@133: 
cannam@133: A side effect (so to speak) of the compiler rewrite is that Cap'n Proto's companion utility
cannam@133: library, KJ, now includes a parser combinator framework based on C++11 templates and lambdas.
cannam@133: Here's a sample:
cannam@133: 
cannam@133: {% highlight c++ %}
cannam@133: // Construct a parser that parses a number.
cannam@133: auto number = transform(
cannam@133:     sequence(
cannam@133:         oneOrMore(charRange('0', '9')),
cannam@133:         optional(sequence(
cannam@133:             exactChar<'.'>(),
cannam@133:             many(charRange('0', '9'))))),
cannam@133:     [](Array<char> whole, Maybe<Array<char>> maybeFraction)
cannam@133:         -> Number* {
cannam@133:       KJ_IF_MAYBE(fraction, maybeFraction) {
cannam@133:         return new RealNumber(whole, *fraction);
cannam@133:       } else {
cannam@133:         return new WholeNumber(whole);
cannam@133:       }
cannam@133:     });
cannam@133: {% endhighlight %}
cannam@133: 
cannam@133: An interesting fact about the above code is that constructing the parser itself does not allocate
cannam@133: anything on the heap.  The variable `number` in this case ends up being one 96-byte flat object,
cannam@133: most of which is composed of tables for character matching.  The whole thing could even be
cannam@133: declared `constexpr`...  if the C++ standard allowed empty-capture lambdas to be `constexpr`, which
cannam@133: unfortunately it doesn't (yet).
cannam@133: 
cannam@133: Unfortunately, KJ is largely undocumented at the moment, since people who just want to use
cannam@133: Cap'n Proto generally don't need to know about it.
cannam@133: 
cannam@133: **Other New Features**
cannam@133: 
cannam@133: There are a couple other notable changes in this release, aside from the compiler:
cannam@133: 
cannam@133: * Cygwin has been added as a supported platform, meaning you can now use Cap'n Proto on Windows.
cannam@133:   I am considering supporting MinGW as well.  Unfortunately, MSVC is unlikely to be supported any
cannam@133:   time soon as its C++11 support is
cannam@133:   [woefully lacking](http://blogs.msdn.com/b/somasegar/archive/2013/06/28/cpp-conformance-roadmap.aspx).
cannam@133: 
cannam@133: * The new compiler binary -- now called `capnp` rather than `capnpc` -- is more of a multi-tool.
cannam@133:   It includes the ability to decode binary messages to text as a debugging aid.  Type
cannam@133:   `capnp help decode` for more information.
cannam@133: 
cannam@133: * The new [Orphan]({{ site.baseurl }}/cxx.html#orphans) class lets you detach objects from a
cannam@133:   message tree and re-attach them elsewhere.
cannam@133: 
cannam@133: * Various contributors have declared their intentions to implement
cannam@133:   [Ruby](https://github.com/cstrahan/capnp-ruby),
cannam@133:   [Rust](https://github.com/dwrensha/capnproto-rust), C#, Java, Erlang, and Delphi bindings.  These
cannam@133:   are still works in progress, but exciting nonetheless!
cannam@133: 
cannam@133: **Backwards-compatibility Note**
cannam@133: 
cannam@133: Cap'n Proto v0.2 contains an obscure wire format incompatibility with v0.1.  If you are using
cannam@133: unions containing multiple primitive-type fields of varying sizes, it's possible that the new
cannam@133: compiler will position those fields differently.  A work-around to get back to the old layout
cannam@133: exists; if you believe you could be affected, please [send me](mailto:temporal@gmail.com) your
cannam@133: schema and I'll tell you what to do.  [Gory details.](https://groups.google.com/d/msg/capnproto/NIYbD0haP38/pH5LildInwIJ)
cannam@133: 
cannam@133: **Road Map**
cannam@133: 
cannam@133: v0.3 will come in a couple weeks and will include several new features and clean-ups that can now
cannam@133: be implemented more easily given the new compiler.  This will also hopefully be the first release
cannam@133: that officially supports a language other than C++.
cannam@133: 
cannam@133: The following release, v0.4, will hopefully be the first release implementing RPC.
cannam@133: 
cannam@133: _PS.  If you are wondering, compared to the Haskell version, the new compiler is about 50% more
cannam@133: lines of code and about 4x faster.  The speed increase should be taken with a grain of salt,
cannam@133: though, as my Haskell code did all kinds of horribly slow things.  The code size is, I think, not
cannam@133: bad, considering that Haskell specializes in concision -- but, again, I'm sure a Haskell expert
cannam@133: could have written shorter code._