annotate src/capnproto-git-20161025/doc/_posts/2013-08-12-capnproto-0.2-no-more-haskell.md @ 133:1ac99bfc383d

Add Cap'n Proto source

---
layout: post
title: "Cap'n Proto v0.2: Compiler rewritten Haskell -> C++"
author: kentonv
---

Today I am releasing version 0.2 of Cap'n Proto.  The most notable change: the compiler / code
generator, which was previously written in Haskell, has been rewritten in C++11.  There are a few
other changes as well, but before I talk about those, let me try to calm the angry mob that is
not doubt reaching for their pitchforks as we speak.  There are a few reasons for this change,
some practical, some ideological.  I'll start with the practical.

**The practical:  Supporting dynamic languages**

Say you are trying to implement Cap'n Proto in an interpreted language like Python.  One of the big
draws of such a language is that you can edit your code and then run it without an intervening
compile step, allowing you to iterate faster.  But if the Python Cap'n Proto implementation worked
like the C++ one (or like Protobufs), you lose some of that: whenever you change your Cap'n Proto
schema files, you must run a command to regenerate the Python code from them.  That sucks.

What you really want to do is parse the schemas at start-up -- the same time that the Python code
itself is parsed.  But writing a proper schema parser is harder than it looks; you really should
reuse the existing implementation.  If it is written in Haskell, that's going to be problematic.
You either need to invoke the schema parser as a sub-process or you need to call Haskell code from
Python via an FFI.  Either approach is going to be a huge hack with lots of problems, not the least
of which is having a runtime dependency on an entire platform that your end users may not otherwise
want.

But with the schema parser written in C++, things become much simpler.  Python code calls into
C/C++ all the time.  Everyone already has the necessary libraries installed.  There's no need to
generate code, even; the parsed schema can be fed into the Cap'n Proto C++ runtime's dynamic API,
and Python bindings can trivially be implemented on top of that in just a few hundred lines of
code.  Everyone wins.

**The ideological:  I'm an object-oriented programmer**

I really wanted to like Haskell.  I used to be a strong proponent of functional programming, and
I actually once wrote a complete web server and CMS in a purely-functional toy language of my own
creation.  I love strong static typing, and I find a lot of the constructs in Haskell really
powerful and beautiful.  Even monads.  _Especially_ monads.

But when it comes down to it, I am an object-oriented programmer, and Haskell is not an
object-oriented language.  Yes, you can do object-oriented style if you want to, just like you
can do objects in C.  But it's just too painful.  I want to write `object.methodName`, not
`ModuleName.objectTypeMethodName object`.  I want to be able to write lots of small classes that
encapsulate complex functionality in simple interfaces -- _without_ having to place each one in
a whole separate module and ending up with thousands of source files.  I want to be able to build
a list of objects of varying types that implement the same interface without having to re-invent
virtual tables every time I do it (type classes don't quite solve the problem).

And as it turns out, even aside from the lack of object-orientation, I don't actually like
functional programming as much as I thought.  Yes, writing my parser was super-easy (my first
commit message was
"[Day 1: Learn Haskell, write a parser](https://github.com/kentonv/capnproto/commit/6bb49ca775501a9b2c7306992fd0de53c5ee4e95)").
But everything beyond that seemed to require increasing amounts of brain bending.  For instance, to
actually encode a Cap'n Proto message, I couldn't just allocate a buffer of zeros and then go
through each field and set its value.  Instead, I had to compute all the field values first, sort
them by position, then concatenate the results.

Of course, I'm sure it's the case that if I spent years writing Haskell code, I'd eventually become
as proficient with it as I am with C++.  Perhaps I could un-learn object-oriented style and learn
something else that works just as well or better.  Basically, though, I decided that this was
going to take a lot longer than it at first appeared, and that this wasn't a good use of my
limited resources.  So, I'm cutting my losses.

I still think Haskell is a very interesting language, and if works for you, by all means, use it.
I would love to see someone write at actual Cap'n Proto runtime implementation in Haskell.  But
the compiler is now C++.

**Parser Combinators in C++**

A side effect (so to speak) of the compiler rewrite is that Cap'n Proto's companion utility
library, KJ, now includes a parser combinator framework based on C++11 templates and lambdas.
Here's a sample:

{% highlight c++ %}
// Construct a parser that parses a number.
auto number = transform(
    sequence(
        oneOrMore(charRange('0', '9')),
        optional(sequence(
            exactChar<'.'>(),
            many(charRange('0', '9'))))),
    [](Array<char> whole, Maybe<Array<char>> maybeFraction)
        -> Number* {
      KJ_IF_MAYBE(fraction, maybeFraction) {
        return new RealNumber(whole, *fraction);
      } else {
        return new WholeNumber(whole);
      }
    });
{% endhighlight %}

An interesting fact about the above code is that constructing the parser itself does not allocate
anything on the heap.  The variable `number` in this case ends up being one 96-byte flat object,
most of which is composed of tables for character matching.  The whole thing could even be
declared `constexpr`...  if the C++ standard allowed empty-capture lambdas to be `constexpr`, which
unfortunately it doesn't (yet).

Unfortunately, KJ is largely undocumented at the moment, since people who just want to use
Cap'n Proto generally don't need to know about it.

**Other New Features**

There are a couple other notable changes in this release, aside from the compiler:

* Cygwin has been added as a supported platform, meaning you can now use Cap'n Proto on Windows.
  I am considering supporting MinGW as well.  Unfortunately, MSVC is unlikely to be supported any
  time soon as its C++11 support is
  [woefully lacking](http://blogs.msdn.com/b/somasegar/archive/2013/06/28/cpp-conformance-roadmap.aspx).

* The new compiler binary -- now called `capnp` rather than `capnpc` -- is more of a multi-tool.
  It includes the ability to decode binary messages to text as a debugging aid.  Type
  `capnp help decode` for more information.

* The new [Orphan]({{ site.baseurl }}/cxx.html#orphans) class lets you detach objects from a
  message tree and re-attach them elsewhere.

* Various contributors have declared their intentions to implement
  [Ruby](https://github.com/cstrahan/capnp-ruby),
  [Rust](https://github.com/dwrensha/capnproto-rust), C#, Java, Erlang, and Delphi bindings.  These
  are still works in progress, but exciting nonetheless!

**Backwards-compatibility Note**

Cap'n Proto v0.2 contains an obscure wire format incompatibility with v0.1.  If you are using
unions containing multiple primitive-type fields of varying sizes, it's possible that the new
compiler will position those fields differently.  A work-around to get back to the old layout
exists; if you believe you could be affected, please [send me](mailto:[e-mail]) your
schema and I'll tell you what to do.  [Gory details.](https://groups.google.com/d/msg/capnproto/NIYbD0haP38/pH5LildInwIJ)

**Road Map**

v0.3 will come in a couple weeks and will include several new features and clean-ups that can now
be implemented more easily given the new compiler.  This will also hopefully be the first release
that officially supports a language other than C++.

The following release, v0.4, will hopefully be the first release implementing RPC.

_PS.  If you are wondering, compared to the Haskell version, the new compiler is about 50% more
lines of code and about 4x faster.  The speed increase should be taken with a grain of salt,
though, as my Haskell code did all kinds of horribly slow things.  The code size is, I think, not
bad, considering that Haskell specializes in concision -- but, again, I'm sure a Haskell expert
could have written shorter code._