cannam@62: --- cannam@62: layout: page cannam@62: title: RPC Protocol cannam@62: --- cannam@62: cannam@62: # RPC Protocol cannam@62: cannam@62: ## Introduction cannam@62: cannam@62: ### Time Travel! _(Promise Pipelining)_ cannam@62: cannam@62: cannam@62: cannam@62: Cap'n Proto RPC employs TIME TRAVEL! The results of an RPC call are returned to the client cannam@62: instantly, before the server even receives the initial request! cannam@62: cannam@62: There is, of course, a catch: The results can only be used as part of a new request sent to the cannam@62: same server. If you want to use the results for anything else, you must wait. cannam@62: cannam@62: This is useful, however: Say that, as in the picture, you want to call `foo()`, then call `bar()` cannam@62: on its result, i.e. `bar(foo())`. Or -- as is very common in object-oriented programming -- you cannam@62: want to call a method on the result of another call, i.e. `foo().bar()`. With any traditional RPC cannam@62: system, this will require two network round trips. With Cap'n Proto, it takes only one. In fact, cannam@62: you can chain any number of such calls together -- with diamond dependencies and everything -- and cannam@62: Cap'n Proto will collapse them all into one round trip. cannam@62: cannam@62: By now you can probably imagine how it works: if you execute `bar(foo())`, the client sends two cannam@62: messages to the server, one saying "Please execute foo()", and a second saying "Please execute cannam@62: bar() on the result of the first call". These messages can be sent together -- there's no need cannam@62: to wait for the first call to actually return. cannam@62: cannam@62: To make programming to this model easy, in your code, each call returns a "promise". Promises cannam@62: work much like Javascript promises or promises/futures in other languages: the promise is returned cannam@62: immediately, but you must later call `wait()` on it, or call `then()` to register an asynchronous cannam@62: callback. cannam@62: cannam@62: However, Cap'n Proto promises support an additional feature: cannam@62: [pipelining](http://www.erights.org/elib/distrib/pipeline.html). The promise cannam@62: actually has methods corresponding to whatever methods the final result would have, except that cannam@62: these methods may only be used for the purpose of calling back to the server. Moreover, a cannam@62: pipelined promise can be used in the parameters to another call without waiting. cannam@62: cannam@62: **_But isn't that just syntax sugar?_** cannam@62: cannam@62: OK, fair enough. In a traditional RPC system, we might solve our problem by introducing a new cannam@62: method `foobar()` which combines `foo()` and `bar()`. Now we've eliminated the round trip, without cannam@62: inventing a whole new RPC protocol. cannam@62: cannam@62: The problem is, this kind of arbitrary combining of orthogonal features quickly turns elegant cannam@62: object-oriented protocols into ad-hoc messes. cannam@62: cannam@62: For example, consider the following interface: cannam@62: cannam@62: {% highlight capnp %} cannam@62: # A happy, object-oriented interface! cannam@62: cannam@62: interface Node {} cannam@62: cannam@62: interface Directory extends(Node) { cannam@62: list @0 () -> (list: List(Entry)); cannam@62: struct Entry { cannam@62: name @0 :Text; cannam@62: file @1 :Node; cannam@62: } cannam@62: cannam@62: create @1 (name :Text) -> (node :Node); cannam@62: open @2 (name :Text) -> (node :Node); cannam@62: delete @3 (name :Text); cannam@62: link @4 (name :Text, node :Node); cannam@62: } cannam@62: cannam@62: interface File extends(Node) { cannam@62: size @0 () -> (size: UInt64); cannam@62: read @1 (startAt :UInt64, amount :UInt64) -> (data: Data); cannam@62: write @2 (startAt :UInt64, data :Data); cannam@62: truncate @3 (size :UInt64); cannam@62: } cannam@62: {% endhighlight %} cannam@62: cannam@62: This is a very clean interface for interacting with a file system. But say you are using this cannam@62: interface over a satellite link with 1000ms latency. Now you have a problem: simply reading the cannam@62: file `foo` in directory `bar` takes four round trips! cannam@62: cannam@62: {% highlight python %} cannam@62: # pseudocode cannam@62: bar = root.open("bar"); # 1 cannam@62: foo = bar.open("foo"); # 2 cannam@62: size = foo.size(); # 3 cannam@62: data = foo.read(0, size); # 4 cannam@62: # The above is four calls but takes only one network cannam@62: # round trip with Cap'n Proto! cannam@62: {% endhighlight %} cannam@62: cannam@62: In such a high-latency scenario, making your interface elegant is simply not worth 4x the latency. cannam@62: So now you're going to change it. You'll probably do something like: cannam@62: cannam@62: * Introduce a notion of path strings, so that you can specify "foo/bar" rather than make two cannam@62: separate calls. cannam@62: * Merge the `File` and `Directory` interfaces into a single `Filesystem` interface, where every cannam@62: call takes a path as an argument. cannam@62: cannam@62: {% highlight capnp %} cannam@62: # A sad, singleton-ish interface. cannam@62: cannam@62: interface Filesystem { cannam@62: list @0 (path :Text) -> (list :List(Text)); cannam@62: create @1 (path :Text, data :Data); cannam@62: delete @2 (path :Text); cannam@62: link @3 (path :Text, target :Text); cannam@62: cannam@62: fileSize @4 (path :Text) -> (size: UInt64); cannam@62: read @5 (path :Text, startAt :UInt64, amount :UInt64) cannam@62: -> (data :Data); cannam@62: readAll @6 (path :Text) -> (data: Data); cannam@62: write @7 (path :Text, startAt :UInt64, data :Data); cannam@62: truncate @8 (path :Text, size :UInt64); cannam@62: } cannam@62: {% endhighlight %} cannam@62: cannam@62: We've now solved our latency problem... but at what cost? cannam@62: cannam@62: * We now have to implement path string manipulation, which is always a headache. cannam@62: * If someone wants to perform multiple operations on a file or directory, we now either have to cannam@62: re-allocate resources for every call or we have to implement some sort of cache, which tends to cannam@62: be complicated and error-prone. cannam@62: * We can no longer give someone a specific `File` or a `Directory` -- we have to give them a cannam@62: `Filesystem` and a path. cannam@62: * But what if they are buggy and have hard-coded some path other than the one we specified? cannam@62: * Or what if we don't trust them, and we really want them to access only one particular `File` or cannam@62: `Directory` and not have permission to anything else. Now we have to implement authentication cannam@62: and authorization systems! Arrgghh! cannam@62: cannam@62: Essentially, in our quest to avoid latency, we've resorted to using a singleton-ish design, and cannam@62: [singletons are evil](http://www.object-oriented-security.org/lets-argue/singletons). cannam@62: cannam@62: **Promise Pipelining solves all of this!** cannam@62: cannam@62: With pipelining, our 4-step example can be automatically reduced to a single round trip with no cannam@62: need to change our interface at all. We keep our simple, elegant, singleton-free interface, we cannam@62: don't have to implement path strings, caching, authentication, or authorization, and yet everything cannam@62: performs as well as we can possibly hope for. cannam@62: cannam@62: #### Example code cannam@62: cannam@62: [The calculator example](https://github.com/sandstorm-io/capnproto/blob/master/c++/samples/calculator-client.c++) cannam@62: uses promise pipelining. Take a look at the client side in particular. cannam@62: cannam@62: ### Distributed Objects cannam@62: cannam@62: As you've noticed by now, Cap'n Proto RPC is a distributed object protocol. Interface references -- cannam@62: or, as we more commonly call them, capabilities -- are a first-class type. You can pass a cannam@62: capability as a parameter to a method or embed it in a struct or list. This is a huge difference cannam@62: from many modern RPC-over-HTTP protocols that only let you address global URLs, or other RPC cannam@62: systems like Protocol Buffers and Thrift that only let you address singleton objects exported at cannam@62: startup. The ability to dynamically introduce new objects and pass around references to them cannam@62: allows you to use the same design patterns over the network that you use locally in object-oriented cannam@62: programming languages. Many kinds of interactions become vastly easier to express given the cannam@62: richer vocabulary. cannam@62: cannam@62: **_Didn't CORBA prove this doesn't work?_** cannam@62: cannam@62: No! cannam@62: cannam@62: CORBA failed for many reasons, with the usual problems of design-by-committee being a big one. cannam@62: cannam@62: However, the biggest reason for CORBA's failure is that it tried to make remote calls look the cannam@62: same as local calls. Cap'n Proto does NOT do this -- remote calls have a different kind of API cannam@62: involving promises, and accounts for the presence of a network introducing latency and cannam@62: unreliability. cannam@62: cannam@62: As shown above, promise pipelining is absolutely critical to making object-oriented interfaces work cannam@62: in the presence of latency. If remote calls look the same as local calls, there is no opportunity cannam@62: to introduce promise pipelining, and latency is inevitable. Any distributed object protocol which cannam@62: does not support promise pipelining cannot -- and should not -- succeed. Thus the failure of CORBA cannam@62: (and DCOM, etc.) was inevitable, but Cap'n Proto is different. cannam@62: cannam@62: ### Handling disconnects cannam@62: cannam@62: Networks are unreliable. Occasionally, connections will be lost. When this happens, all cannam@62: capabilities (object references) served by the connection will become disconnected. Any further cannam@62: calls addressed to these capabilities will throw "disconnected" exceptions. When this happens, the cannam@62: client will need to create a new connection and try again. All Cap'n Proto applications with cannam@62: long-running connections (and probably short-running ones too) should be prepared to catch cannam@62: "disconnected" exceptions and respond appropriately. cannam@62: cannam@62: On the server side, when all references to an object have been "dropped" (either because the cannam@62: clients explicitly dropped them or because they became disconnected), the object will be closed cannam@62: (in C++, the destructor is called; in GC'd languages, a `close()` method is called). This allows cannam@62: servers to easily allocate per-client resources without having to clean up on a timeout or risk cannam@62: leaking memory. cannam@62: cannam@62: ### Security cannam@62: cannam@62: Cap'n Proto interface references are cannam@62: [capabilities](http://en.wikipedia.org/wiki/Capability-based_security). That is, they both cannam@62: designate an object to call and confer permission to call it. When a new object is created, only cannam@62: the creator is initially able to call it. When the object is passed over a network connection, cannam@62: the receiver gains permission to make calls -- but no one else does. In fact, it is impossible cannam@62: for others to access the capability without consent of either the host or the receiver because cannam@62: the host only assigns it an ID specific to the connection over which it was sent. cannam@62: cannam@62: Capability-based design patterns -- which largely boil down to object-oriented design patterns -- cannam@62: work great with Cap'n Proto. Such patterns tend to be much more adaptable than traditional cannam@62: ACL-based security, making it easy to keep security tight and avoid confused-deputy attacks while cannam@62: minimizing pain for legitimate users. That said, you can of course implement ACLs or any other cannam@62: pattern on top of capabilities. cannam@62: cannam@62: For an extended discussion of what capabilities are and why they are often easier and more powerful cannam@62: than ACLs, see Mark Miller's cannam@62: ["An Ode to the Granovetter Diagram"](http://www.erights.org/elib/capability/ode/index.html) and cannam@62: [Capability Myths Demolished](http://zesty.ca/capmyths/usenix.pdf). cannam@62: cannam@62: ## Protocol Features cannam@62: cannam@62: Cap'n Proto's RPC protocol has the following notable features. Since the protocol is complicated, cannam@62: the feature set has been divided into numbered "levels", so that implementations may declare which cannam@62: features they have covered by advertising a level number. cannam@62: cannam@62: * **Level 1:** Object references and promise pipelining, as described above. cannam@62: * **Level 2:** Persistent capabilities. You may request to "save" a capability, receiving a cannam@62: persistent token which can be used to "restore" it in the future (on a new connection). Not cannam@62: all capabilities can be saved; the host app must implement support for it. Building this into cannam@62: the protocol makes it possible for a Cap'n-Proto-based data store to transparently save cannam@62: structures containing capabilities without knowledge of the particular capability types or the cannam@62: application built on them, as well as potentially enabling more powerful analysis and cannam@62: visualization of stored data. cannam@62: * **Level 3:** Three-way interactions. A network of Cap'n Proto vats (nodes) can pass object cannam@62: references to each other and automatically form direct connections as needed. For instance, if cannam@62: Alice (on machine A) sends Bob (on machine B) a reference to Carol (on machine C), then machine B cannam@62: will form a new connection to machine C so that Bob can call Carol directly without proxying cannam@62: through machine A. cannam@62: * **Level 4:** Reference equality / joining. If you receive a set of capabilities from different cannam@62: parties which should all point to the same underlying objects, you can verify securely that they cannam@62: in fact do. This is subtle, but enables many security patterns that rely on one party being able cannam@62: to verify that two or more other parties agree on something (imagine a digital escrow agent). cannam@62: See [E's page on equality](http://erights.org/elib/equality/index.html). cannam@62: cannam@62: ## Encryption cannam@62: cannam@62: At this time, Cap'n Proto does not specify an encryption scheme, but as it is a simple byte cannam@62: stream protocol, it can easily be layered on top of SSL/TLS or other such protocols. cannam@62: cannam@62: ## Specification cannam@62: cannam@62: The Cap'n Proto RPC protocol is defined in terms of Cap'n Proto serialization schemas. The cannam@62: documentation is inline. See cannam@62: [rpc.capnp](https://github.com/sandstorm-io/capnproto/blob/master/c++/src/capnp/rpc.capnp). cannam@62: cannam@62: Cap'n Proto's RPC protocol is based heavily on cannam@62: [CapTP](http://www.erights.org/elib/distrib/captp/index.html), the distributed capability protocol cannam@62: used by the [E programming language](http://www.erights.org/index.html). Lots of useful material cannam@62: for understanding capabilities can be found at those links. cannam@62: cannam@62: The protocol is complex, but the functionality it supports is conceptually simple. Just as TCP cannam@62: is a complex protocol that implements the simple concept of a byte stream, Cap'n Proto RPC is a cannam@62: complex protocol that implements the simple concept of objects with callable methods.