cannam@133: --- cannam@133: layout: page cannam@133: title: RPC Protocol cannam@133: --- cannam@133: cannam@133: # RPC Protocol cannam@133: cannam@133: ## Introduction cannam@133: cannam@133: ### Time Travel! _(Promise Pipelining)_ cannam@133: cannam@133: cannam@133: cannam@133: Cap'n Proto RPC employs TIME TRAVEL! The results of an RPC call are returned to the client cannam@133: instantly, before the server even receives the initial request! cannam@133: cannam@133: There is, of course, a catch: The results can only be used as part of a new request sent to the cannam@133: same server. If you want to use the results for anything else, you must wait. cannam@133: cannam@133: This is useful, however: Say that, as in the picture, you want to call `foo()`, then call `bar()` cannam@133: on its result, i.e. `bar(foo())`. Or -- as is very common in object-oriented programming -- you cannam@133: want to call a method on the result of another call, i.e. `foo().bar()`. With any traditional RPC cannam@133: system, this will require two network round trips. With Cap'n Proto, it takes only one. In fact, cannam@133: you can chain any number of such calls together -- with diamond dependencies and everything -- and cannam@133: Cap'n Proto will collapse them all into one round trip. cannam@133: cannam@133: By now you can probably imagine how it works: if you execute `bar(foo())`, the client sends two cannam@133: messages to the server, one saying "Please execute foo()", and a second saying "Please execute cannam@133: bar() on the result of the first call". These messages can be sent together -- there's no need cannam@133: to wait for the first call to actually return. cannam@133: cannam@133: To make programming to this model easy, in your code, each call returns a "promise". Promises cannam@133: work much like Javascript promises or promises/futures in other languages: the promise is returned cannam@133: immediately, but you must later call `wait()` on it, or call `then()` to register an asynchronous cannam@133: callback. cannam@133: cannam@133: However, Cap'n Proto promises support an additional feature: cannam@133: [pipelining](http://www.erights.org/elib/distrib/pipeline.html). The promise cannam@133: actually has methods corresponding to whatever methods the final result would have, except that cannam@133: these methods may only be used for the purpose of calling back to the server. Moreover, a cannam@133: pipelined promise can be used in the parameters to another call without waiting. cannam@133: cannam@133: **_But isn't that just syntax sugar?_** cannam@133: cannam@133: OK, fair enough. In a traditional RPC system, we might solve our problem by introducing a new cannam@133: method `foobar()` which combines `foo()` and `bar()`. Now we've eliminated the round trip, without cannam@133: inventing a whole new RPC protocol. cannam@133: cannam@133: The problem is, this kind of arbitrary combining of orthogonal features quickly turns elegant cannam@133: object-oriented protocols into ad-hoc messes. cannam@133: cannam@133: For example, consider the following interface: cannam@133: cannam@133: {% highlight capnp %} cannam@133: # A happy, object-oriented interface! cannam@133: cannam@133: interface Node {} cannam@133: cannam@133: interface Directory extends(Node) { cannam@133: list @0 () -> (list: List(Entry)); cannam@133: struct Entry { cannam@133: name @0 :Text; cannam@133: file @1 :Node; cannam@133: } cannam@133: cannam@133: create @1 (name :Text) -> (node :Node); cannam@133: open @2 (name :Text) -> (node :Node); cannam@133: delete @3 (name :Text); cannam@133: link @4 (name :Text, node :Node); cannam@133: } cannam@133: cannam@133: interface File extends(Node) { cannam@133: size @0 () -> (size: UInt64); cannam@133: read @1 (startAt :UInt64, amount :UInt64) -> (data: Data); cannam@133: write @2 (startAt :UInt64, data :Data); cannam@133: truncate @3 (size :UInt64); cannam@133: } cannam@133: {% endhighlight %} cannam@133: cannam@133: This is a very clean interface for interacting with a file system. But say you are using this cannam@133: interface over a satellite link with 1000ms latency. Now you have a problem: simply reading the cannam@133: file `foo` in directory `bar` takes four round trips! cannam@133: cannam@133: {% highlight python %} cannam@133: # pseudocode cannam@133: bar = root.open("bar"); # 1 cannam@133: foo = bar.open("foo"); # 2 cannam@133: size = foo.size(); # 3 cannam@133: data = foo.read(0, size); # 4 cannam@133: # The above is four calls but takes only one network cannam@133: # round trip with Cap'n Proto! cannam@133: {% endhighlight %} cannam@133: cannam@133: In such a high-latency scenario, making your interface elegant is simply not worth 4x the latency. cannam@133: So now you're going to change it. You'll probably do something like: cannam@133: cannam@133: * Introduce a notion of path strings, so that you can specify "foo/bar" rather than make two cannam@133: separate calls. cannam@133: * Merge the `File` and `Directory` interfaces into a single `Filesystem` interface, where every cannam@133: call takes a path as an argument. cannam@133: cannam@133: {% highlight capnp %} cannam@133: # A sad, singleton-ish interface. cannam@133: cannam@133: interface Filesystem { cannam@133: list @0 (path :Text) -> (list :List(Text)); cannam@133: create @1 (path :Text, data :Data); cannam@133: delete @2 (path :Text); cannam@133: link @3 (path :Text, target :Text); cannam@133: cannam@133: fileSize @4 (path :Text) -> (size: UInt64); cannam@133: read @5 (path :Text, startAt :UInt64, amount :UInt64) cannam@133: -> (data :Data); cannam@133: readAll @6 (path :Text) -> (data: Data); cannam@133: write @7 (path :Text, startAt :UInt64, data :Data); cannam@133: truncate @8 (path :Text, size :UInt64); cannam@133: } cannam@133: {% endhighlight %} cannam@133: cannam@133: We've now solved our latency problem... but at what cost? cannam@133: cannam@133: * We now have to implement path string manipulation, which is always a headache. cannam@133: * If someone wants to perform multiple operations on a file or directory, we now either have to cannam@133: re-allocate resources for every call or we have to implement some sort of cache, which tends to cannam@133: be complicated and error-prone. cannam@133: * We can no longer give someone a specific `File` or a `Directory` -- we have to give them a cannam@133: `Filesystem` and a path. cannam@133: * But what if they are buggy and have hard-coded some path other than the one we specified? cannam@133: * Or what if we don't trust them, and we really want them to access only one particular `File` or cannam@133: `Directory` and not have permission to anything else. Now we have to implement authentication cannam@133: and authorization systems! Arrgghh! cannam@133: cannam@133: Essentially, in our quest to avoid latency, we've resorted to using a singleton-ish design, and cannam@133: [singletons are evil](http://www.object-oriented-security.org/lets-argue/singletons). cannam@133: cannam@133: **Promise Pipelining solves all of this!** cannam@133: cannam@133: With pipelining, our 4-step example can be automatically reduced to a single round trip with no cannam@133: need to change our interface at all. We keep our simple, elegant, singleton-free interface, we cannam@133: don't have to implement path strings, caching, authentication, or authorization, and yet everything cannam@133: performs as well as we can possibly hope for. cannam@133: cannam@133: #### Example code cannam@133: cannam@133: [The calculator example](https://github.com/sandstorm-io/capnproto/blob/master/c++/samples/calculator-client.c++) cannam@133: uses promise pipelining. Take a look at the client side in particular. cannam@133: cannam@133: ### Distributed Objects cannam@133: cannam@133: As you've noticed by now, Cap'n Proto RPC is a distributed object protocol. Interface references -- cannam@133: or, as we more commonly call them, capabilities -- are a first-class type. You can pass a cannam@133: capability as a parameter to a method or embed it in a struct or list. This is a huge difference cannam@133: from many modern RPC-over-HTTP protocols that only let you address global URLs, or other RPC cannam@133: systems like Protocol Buffers and Thrift that only let you address singleton objects exported at cannam@133: startup. The ability to dynamically introduce new objects and pass around references to them cannam@133: allows you to use the same design patterns over the network that you use locally in object-oriented cannam@133: programming languages. Many kinds of interactions become vastly easier to express given the cannam@133: richer vocabulary. cannam@133: cannam@133: **_Didn't CORBA prove this doesn't work?_** cannam@133: cannam@133: No! cannam@133: cannam@133: CORBA failed for many reasons, with the usual problems of design-by-committee being a big one. cannam@133: cannam@133: However, the biggest reason for CORBA's failure is that it tried to make remote calls look the cannam@133: same as local calls. Cap'n Proto does NOT do this -- remote calls have a different kind of API cannam@133: involving promises, and accounts for the presence of a network introducing latency and cannam@133: unreliability. cannam@133: cannam@133: As shown above, promise pipelining is absolutely critical to making object-oriented interfaces work cannam@133: in the presence of latency. If remote calls look the same as local calls, there is no opportunity cannam@133: to introduce promise pipelining, and latency is inevitable. Any distributed object protocol which cannam@133: does not support promise pipelining cannot -- and should not -- succeed. Thus the failure of CORBA cannam@133: (and DCOM, etc.) was inevitable, but Cap'n Proto is different. cannam@133: cannam@133: ### Handling disconnects cannam@133: cannam@133: Networks are unreliable. Occasionally, connections will be lost. When this happens, all cannam@133: capabilities (object references) served by the connection will become disconnected. Any further cannam@133: calls addressed to these capabilities will throw "disconnected" exceptions. When this happens, the cannam@133: client will need to create a new connection and try again. All Cap'n Proto applications with cannam@133: long-running connections (and probably short-running ones too) should be prepared to catch cannam@133: "disconnected" exceptions and respond appropriately. cannam@133: cannam@133: On the server side, when all references to an object have been "dropped" (either because the cannam@133: clients explicitly dropped them or because they became disconnected), the object will be closed cannam@133: (in C++, the destructor is called; in GC'd languages, a `close()` method is called). This allows cannam@133: servers to easily allocate per-client resources without having to clean up on a timeout or risk cannam@133: leaking memory. cannam@133: cannam@133: ### Security cannam@133: cannam@133: Cap'n Proto interface references are cannam@133: [capabilities](http://en.wikipedia.org/wiki/Capability-based_security). That is, they both cannam@133: designate an object to call and confer permission to call it. When a new object is created, only cannam@133: the creator is initially able to call it. When the object is passed over a network connection, cannam@133: the receiver gains permission to make calls -- but no one else does. In fact, it is impossible cannam@133: for others to access the capability without consent of either the host or the receiver because cannam@133: the host only assigns it an ID specific to the connection over which it was sent. cannam@133: cannam@133: Capability-based design patterns -- which largely boil down to object-oriented design patterns -- cannam@133: work great with Cap'n Proto. Such patterns tend to be much more adaptable than traditional cannam@133: ACL-based security, making it easy to keep security tight and avoid confused-deputy attacks while cannam@133: minimizing pain for legitimate users. That said, you can of course implement ACLs or any other cannam@133: pattern on top of capabilities. cannam@133: cannam@133: For an extended discussion of what capabilities are and why they are often easier and more powerful cannam@133: than ACLs, see Mark Miller's cannam@133: ["An Ode to the Granovetter Diagram"](http://www.erights.org/elib/capability/ode/index.html) and cannam@133: [Capability Myths Demolished](http://zesty.ca/capmyths/usenix.pdf). cannam@133: cannam@133: ## Protocol Features cannam@133: cannam@133: Cap'n Proto's RPC protocol has the following notable features. Since the protocol is complicated, cannam@133: the feature set has been divided into numbered "levels", so that implementations may declare which cannam@133: features they have covered by advertising a level number. cannam@133: cannam@133: * **Level 1:** Object references and promise pipelining, as described above. cannam@133: * **Level 2:** Persistent capabilities. You may request to "save" a capability, receiving a cannam@133: persistent token which can be used to "restore" it in the future (on a new connection). Not cannam@133: all capabilities can be saved; the host app must implement support for it. Building this into cannam@133: the protocol makes it possible for a Cap'n-Proto-based data store to transparently save cannam@133: structures containing capabilities without knowledge of the particular capability types or the cannam@133: application built on them, as well as potentially enabling more powerful analysis and cannam@133: visualization of stored data. cannam@133: * **Level 3:** Three-way interactions. A network of Cap'n Proto vats (nodes) can pass object cannam@133: references to each other and automatically form direct connections as needed. For instance, if cannam@133: Alice (on machine A) sends Bob (on machine B) a reference to Carol (on machine C), then machine B cannam@133: will form a new connection to machine C so that Bob can call Carol directly without proxying cannam@133: through machine A. cannam@133: * **Level 4:** Reference equality / joining. If you receive a set of capabilities from different cannam@133: parties which should all point to the same underlying objects, you can verify securely that they cannam@133: in fact do. This is subtle, but enables many security patterns that rely on one party being able cannam@133: to verify that two or more other parties agree on something (imagine a digital escrow agent). cannam@133: See [E's page on equality](http://erights.org/elib/equality/index.html). cannam@133: cannam@133: ## Encryption cannam@133: cannam@133: At this time, Cap'n Proto does not specify an encryption scheme, but as it is a simple byte cannam@133: stream protocol, it can easily be layered on top of SSL/TLS or other such protocols. cannam@133: cannam@133: ## Specification cannam@133: cannam@133: The Cap'n Proto RPC protocol is defined in terms of Cap'n Proto serialization schemas. The cannam@133: documentation is inline. See cannam@133: [rpc.capnp](https://github.com/sandstorm-io/capnproto/blob/master/c++/src/capnp/rpc.capnp). cannam@133: cannam@133: Cap'n Proto's RPC protocol is based heavily on cannam@133: [CapTP](http://www.erights.org/elib/distrib/captp/index.html), the distributed capability protocol cannam@133: used by the [E programming language](http://www.erights.org/index.html). Lots of useful material cannam@133: for understanding capabilities can be found at those links. cannam@133: cannam@133: The protocol is complex, but the functionality it supports is conceptually simple. Just as TCP cannam@133: is a complex protocol that implements the simple concept of a byte stream, Cap'n Proto RPC is a cannam@133: complex protocol that implements the simple concept of objects with callable methods.