123 lines
4.1 KiB
Markdown
123 lines
4.1 KiB
Markdown
# roto
|
|
|
|
Zero-allocation Rust protobuf reader and writer.
|
|
|
|
## Overview
|
|
|
|
Instead of deserializing binary protobuf data into Rust structs, roto scans a message _once_ on
|
|
construction — recording the byte offset of each field — then reads fields on demand directly from
|
|
the original bytes. No heap allocation, no data copying, no full deserialization upfront.
|
|
|
|
Writing works the same way: you provide a fixed buffer and a builder writes fields directly into it,
|
|
returning a slice of the bytes written.
|
|
|
|
## Design
|
|
|
|
`protoc` generates a `CodeGeneratorRequest` message; `protoc-gen-roto` (in
|
|
`src/bin/protoc-gen-roto.rs`) reads this from stdin, generates Rust source files, and writes a
|
|
`CodeGeneratorResponse` to stdout. `protoc` then writes those `.rs` files to disk. The generated
|
|
files are included directly in the crate that uses the protobuffers.
|
|
|
|
## Generated code
|
|
|
|
For each protobuf message roto generates two types:
|
|
|
|
- **Reader struct** `MessageName<'a>` — borrows the original byte slice, zero-copy.
|
|
- **Builder struct** `MessageNameBuilder<'b>` — writes into a caller-provided `&mut [u8]`.
|
|
|
|
Nested message types are placed in a `pub mod message_name { ... }` module (snake_case of the
|
|
parent message name) within the same generated file.
|
|
|
|
## Sample usage
|
|
|
|
Given this proto definition:
|
|
|
|
```proto
|
|
message Hello {
|
|
string hello_world = 1;
|
|
message InnerWorld {
|
|
string thought = 1;
|
|
}
|
|
InnerWorld inner_world = 2;
|
|
}
|
|
```
|
|
|
|
### Reading
|
|
|
|
```rust
|
|
fn parse_proto(data: &[u8]) -> roto::Result<String> {
|
|
// Scan the data once, recording field offsets
|
|
let hello = Hello::new(data)?;
|
|
|
|
// String fields return &str borrowed from the original bytes (zero-copy)
|
|
let hello_world: &str = hello.hello_world()?;
|
|
|
|
// Nested message fields return &[u8]; construct the nested reader from those bytes
|
|
let inner_bytes: &[u8] = hello.inner_world()?;
|
|
let inner_world = hello::InnerWorld::new(inner_bytes)?;
|
|
let thought: &str = inner_world.thought()?;
|
|
|
|
Ok(format!("{} is about {}", hello_world, thought))
|
|
}
|
|
```
|
|
|
|
Fields absent from the binary data return `Err(roto::RotoError::FieldNotFound)`.
|
|
|
|
### Writing
|
|
|
|
Nested messages must be serialized into a scratch buffer first, then embedded as raw bytes in the
|
|
outer builder.
|
|
|
|
```rust
|
|
fn build_proto(buf: &mut [u8]) -> roto::Result<&[u8]> {
|
|
// Serialize the inner message first
|
|
let mut inner_buf = [0u8; 256];
|
|
let inner_bytes = hello::InnerWorldBuilder::builder(&mut inner_buf)
|
|
.thought("some thought")?
|
|
.finish()?;
|
|
|
|
// Build the outer message, embedding the serialized inner bytes
|
|
HelloBuilder::builder(buf)
|
|
.hello_world("some world")?
|
|
.inner_world(inner_bytes)?
|
|
.finish() // returns Result<&'b mut [u8]> — the written portion of buf
|
|
}
|
|
```
|
|
|
|
Builder methods consume `self` and return `Result<Self>`, enabling `?`-based chaining.
|
|
`finish()` returns `Result<&'b mut [u8]>` — a slice of the portion of the buffer that was written.
|
|
|
|
### Repeated fields
|
|
|
|
Repeated fields return a `RepeatedFieldIterator<'a>`. Each item yields `Result<(&[u8], WireType)>`.
|
|
|
|
```rust
|
|
let hello = Hello::new(data)?;
|
|
for item in hello.tags() {
|
|
let (value_bytes, _wire_type) = item?;
|
|
// decode value_bytes according to the expected wire type
|
|
}
|
|
```
|
|
|
|
## Runtime API
|
|
|
|
The core runtime in `src/lib.rs` provides:
|
|
|
|
- `ProtoAccessor<'a>` — scans a message's fields and reads values at recorded offsets.
|
|
- `ProtoBuilder<'a>` — writes fields into a provided `&mut [u8]` buffer.
|
|
- `FieldIterator<'a>` / `RepeatedFieldIterator<'a>` — iterators over fields and repeated fields.
|
|
- `Tag`, `WireType` — protobuf encoding primitives.
|
|
- `read_varint`, `write_varint`, `skip_value` — low-level wire-format helpers.
|
|
- `RotoError`, `Result<T>` — error type and alias.
|
|
|
|
## High-level design
|
|
|
|
On construction (`MessageName::new(data)`), the generated reader struct iterates the binary once
|
|
using `FieldIterator` and records the byte offset of each field's tag. Subsequent field accesses
|
|
call `ProtoAccessor::get_value_at(offset)` — no re-scanning. For repeated fields, the start and
|
|
end offsets of the field range are recorded to bound iteration efficiently.
|
|
|
|
## Literature
|
|
|
|
https://protobuf.dev/programming-guides/encoding/
|