cannam@147: --- cannam@147: layout: page cannam@147: title: Road Map cannam@147: --- cannam@147: cannam@147: # Road Map cannam@147: cannam@147: This is a list of big ideas we'd like to implement in Cap'n Proto. We don't know in what order cannam@147: these will actually happen; as always, real work is driven by real-world needs. cannam@147: cannam@147: ### Language Features cannam@147: cannam@147: * **Inline lists:** Lets you define a field composed of a fixed number of elements of the same cannam@147: type, and have those elements be stored directly within the struct rather than as a separate cannam@147: object. Useful mainly to avoid the need to validate list lengths when the length should always cannam@147: be the same. Also saves a pointer's worth of space. cannam@147: * **Type aliases:** Ability to define a type which is just an alias of some other type, and cannam@147: have it show up as e.g. a `typedef` in languages that support that. (The current `using` cannam@147: keyword is intended only for local use and does not affect code generation.) cannam@147: * **Doc comments:** Harvest doc comments from schema files and use them to generate doc comments cannam@147: on generated code. Also make them available in the compiled schema so that a documentation cannam@147: generator could use them. cannam@147: * **Encapsulated types:** This will allow you to create a hand-written wrapper around a cannam@147: type which will be automatically injected into the generated code, so that you can provide a cannam@147: nicer interface which encapsulates the type's inner state. cannam@147: * **Maps:** Based on encapsulated and parameterized types. cannam@147: cannam@147: ### RPC Protocol Features cannam@147: cannam@147: * **Dynamic schema transmission:** Allow e.g. Python applications to obtain schemas directly from cannam@147: the RPC server so that they need not have a local copy. Great for interactive debugging. cannam@147: * **Three-way introductions (level 3 RPC):** Allow RPC interactions between more than two parties, cannam@147: with new connections formed automatically as needed. cannam@147: * **Bulk and Realtime**: Add features that make it easier to design Cap'n Proto APIs for bulk cannam@147: data transfers (with flow control) and realtime communications (where it's better to drop cannam@147: messages than to deliver them late). cannam@147: * **UDP transport**: Cap'n Proto RPC could benefit from implementing a UDP transport, in order cannam@147: to achieve zero-round-trip three-party introductions and to implement "realtime" APIs (see cannam@147: "bulk and realtime", above). cannam@147: * **Encrypted transport**: Cap'n Proto RPC should support an encrypted transport which uses cannam@147: capability-based authorization (not PKI), can accomplish zero-round-trip three-party cannam@147: introductions (via a pre-shared key from the introducer) and based on modern crypto. TLS is cannam@147: not designed for this, but we don't want to invent new crypto; we intend to build on cannam@147: [libsodium](https://github.com/jedisct1/libsodium) and the cannam@147: [Noise Protocol Framework](http://noiseprotocol.org/) as much as possible. cannam@147: cannam@147: ### C++ Cap'n Proto API Features cannam@147: cannam@147: * **Plain Old C Structs:** The code generator should also generate a POCS type corresponding cannam@147: to each struct type. The POCS type would use traditional memory allocation, thus would not cannam@147: support zero-copy, but would support a more traditional and easy-to-use C++ API, including cannam@147: the ability to mutate the object over time without convoluted memory management. POCS types cannam@147: could be extracted from an inserted into messages with a single copy, allowing them to be cannam@147: used easily in non-performance-critical code. cannam@147: * **Multi-threading:** It should be made easy to assign different Cap'n Proto RPC objects cannam@147: to different threads and have them be able to safely call each other. Each thread would still cannam@147: have an anyschronous event loop serving the objects that belong to it. cannam@147: * **Shared memory RPC:** Zero-copy inter-process communication. cannam@147: * **JSON codec customization:** Extend the JSON library to support customizing the JSON cannam@147: representation using annotations. For example, a field could be given a different name in cannam@147: JSON than it is in Cap'n Proto. The goal of these features would be to allow any reasonable cannam@147: pre-existing JSON schema to be representable as a Cap'n Proto type definition, so that cannam@147: servers implementing JSON APIs can use Cap'n Proto exclusively on the server side. cannam@147: * **LZ4 integration:** Integrate LZ4 compression into the API to further reduce bandwidth needs cannam@147: with minimal CPU overhead. cannam@147: * **Annotations API:** For each annotation definition, generate code which assists in extracting cannam@147: that annotation from schema objects in a type-safe way. cannam@147: cannam@147: ### C++ KJ API Features cannam@147: cannam@147: KJ is a framework library that is bundled with Cap'n Proto, but is broadly applicable to C++ cannam@147: applications even if they don't use Cap'n Proto serialization. cannam@147: cannam@147: * **Fiber-based concurrency:** The C++ runtime's event loop concurrency model will be augmented cannam@147: with support for fibers, which are like threads except that context switches happen only at cannam@147: well-defined points (thus avoiding the need for mutex locking). Fibers essentially provide cannam@147: syntax sugar on top of the event loop model. cannam@147: * **TLS bindings:** Write bindings for e.g. OpenSSL to make it easy to integrate with the KJ cannam@147: I/O framework, Cap'n Proto RPC, and the KJ HTTP library. cannam@147: * **Modern crypto bindings:** A thin wrapper around cannam@147: [libsodium](https://github.com/jedisct1/libsodium) with a nice C++ API, e.g. representing cannam@147: keys using fixed-size, trivially-copyable classes. cannam@147: * **Event loop integrations:** We should provide off-the-shelf integrations with popular event cannam@147: loop libraries, such as libuv, libev, libevent, boost::asio, and others, so that it's easier cannam@147: to use Cap'n Proto RPC in applications that already use another event framework. cannam@147: cannam@147: ### Storage cannam@147: cannam@147: * **ORM interface:** Define a standard interface for capabilities that represent remotely-stored cannam@147: objects, with get, put, publish, and subscribe methods. Ideally, parameterize this interface cannam@147: on the stored type. cannam@147: * **mmap-friendly mutable storage format:** Define a standard storage format that is friendly cannam@147: to mmap-based use while allowing modification. (With the current serialization format, mmap cannam@147: is only useful for read-only structures.) Possibly based on the ORM interface, updates only cannam@147: possible at the granularity of a whole ORM entry. cannam@147: cannam@147: ### Tools cannam@147: cannam@147: * **Schema compatibility checker:** Add a `capnp` command which, given two schemas, verifies cannam@147: that the latter is a compatible upgrade from the former. This could be used as a git hook cannam@147: to prevent submission of schema changes that would break wire compatibility. cannam@147: * **RPC debugger:** Add a `capnp` command which sends an RPC from the command line and prints cannam@147: the result. Useful for debugging RPC servers. cannam@147: cannam@147: ## Quality Assurance cannam@147: cannam@147: These things absolutely must happen before any 1.0 release. Note that it's not yet decided when cannam@147: a 1.0 release would happen nor how many 0.x releases might precede it. cannam@147: cannam@147: * **Expand test coverage:** There are lots of tests now, but some important scenarios, such as cannam@147: handling invalid of invalid input, need better testing. cannam@147: * **Performance review:** Performance is already very good compared to competitors, but at some cannam@147: point we need to break out the profiler and really hone down on the details. cannam@147: * **Security review:** We need a careful security review to make sure malicious input cannot cannam@147: crash an application or corrupt memory. cannam@147: cannam@147: ### Infrastructure cannam@147: cannam@147: Note: These are very large projects. cannam@147: cannam@147: * **JSON-HTTP proxy:** Develop a web server which can expose a Cap'n Proto RPC backend as a cannam@147: JSON-over-HTTP protocol. cannam@147: * **Database:** A fast storage database based on Cap'n Proto which implements the ORM interface cannam@147: on top of the mmap storage format.