Zero-Copy Dual-Layout Storage Engine
Build an in-memory database engine with row and columnar storage in Rust.
Introduction
30 Second Summary
Every database you have ever used makes the same fundamental trade-off: how to arrange data in memory so reads are fast without making writes unbearable. The layout decision determines whether a query takes milliseconds or seconds.
In this project, you will build an in-memory storage engine in Rust that stores fixed-size data as raw byte buffers with zero-copy access. The engine supports both row-oriented and column-oriented storage, switchable at runtime through a unified API.
What You'll Build
A dual-layout in-memory storage engine called zerodb that stores tabular data as contiguous byte buffers with zero allocations per read.
By the end of this project, you'll have:
- A schema-driven type system that precomputes byte offsets at construction time for O(1) field access.
- A row store and column store backed by aligned byte buffers, with zero-copy struct access via the zerocopy crate.
- A unified Table API that switches between row and column layouts at runtime, physically reorganizing all data.
- Secret Mission: implement selection vectors for zero-copy filtered scans, the same pattern used by modern query engines.
Are there any prerequisites?
This project assumes solid Rust experience including ownership, lifetimes, unsafe, and repr(C).
You should also be comfortable with how memory layout and alignment work at the byte level.
Set Up the Project
Before you write any storage engine logic, you need a project skeleton with the right dependency and module layout.
In this step you will scaffold the Cargo project, pull in the zerocopy crate for safe zero-copy casts, and create the empty source files that will hold each layer of the engine.
In this step, get ready to:
- Create the Cargo project and configure the zerocopy dependency.
- Set up the module file structure.
- Verify the project compiles cleanly.
Create the project and add the zerocopy dependency
Cargo scaffolds a new binary project with a default src/main.rs and Cargo.toml. You will then edit the manifest to declare zerocopy as a dependency with the derive feature enabled.
- Open a terminal by pressing Ctrl+` or searching for "Terminal" in your application launcher.
- Move to your Desktop by running this command:
cd ~/Desktop- Create the project and move into it by running these commands:
cargo new zerodb
cd zerodb- Confirm the project was created by running:
lsYou should see Cargo.toml and a src directory in the output.
- Open the project in Zed by running:
zed .Why open from the terminal?
Launching Zed with zed . tells it to treat the current directory as the project root. Zed inherits your shell environment this way, so the Rust language server can find cargo without extra configuration.
- In Zed, open Cargo.toml from the file tree in the left sidebar.
- Replace the entire contents of Cargo.toml with the following:
[package]
name = "zerodb"
version = "0.1.0"
edition = "2021"
[dependencies]
zerocopy = { version = "0.8.52", features = ["derive"] }What does this configuration do?
The [dependencies] section pulls in zerocopy version 0.8.52. This crate provides derive macros (FromBytes, IntoBytes) that let the compiler verify at build time whether a struct can be safely cast to and from raw bytes.
The features = ["derive"] flag enables the procedural macros you will use in later steps to annotate your storage structs.
- Save Cargo.toml.
Seeing a red underline in the editor?
If Zed marks the zerocopy line with a warning before you build, that is expected. The language server has not fetched the crate yet. The warning disappears after you run cargo build in the next substep.
Why is Zed showing an error on my zerocopy dependency line?
✔️ Awesome, I've got everything!
Great. Make sure you have saved the file before continuing.
ⓧ I'd like to double check the full code
[package]
name = "zerodb"
version = "0.1.0"
edition = "2021"
[dependencies]
zerocopy = { version = "0.8.52", features = ["derive"] }Create the module structure
The storage engine is split into five modules: types, buffer, row store, column store, and the unified table API. Each module lives in its own file under src/.
- In Zed, open the integrated terminal with Ctrl+`.
- Create the empty module files by running:
touch src/lib.rs src/types.rs src/buffer.rs src/row_store.rs src/column_store.rs src/table.rs- Confirm all files exist by running:
ls src/You should see buffer.rs, column_store.rs, lib.rs, main.rs, row_store.rs, table.rs, and types.rs in the output.
Why do you need lib.rs?
The cargo new zerodb command creates a binary crate with only src/main.rs. Adding src/lib.rs gives the project a library root that main.rs can import from. This separation keeps your engine logic in reusable library modules.
- Open src/lib.rs in the file tree.
- Paste the following module declarations into src/lib.rs:
pub mod types;
pub mod buffer;
pub mod row_store;
pub mod column_store;
pub mod table;What does this code do?
Each pub mod line tells the Rust compiler to look for a file with that name inside src/ and expose it as a public module. Other crates (and your main.rs) can then import from zerodb::types, zerodb::buffer, and so on.
- Save src/lib.rs.
✔️ Awesome, I've got everything!
Great. Make sure you have saved the file before continuing.
ⓧ I'd like to double check the full code
pub mod types;
pub mod buffer;
pub mod row_store;
pub mod column_store;
pub mod table;Verify the setup compiles
A clean build at this stage confirms that Cargo can resolve the zerocopy dependency and that all module files are wired up correctly.
- In the integrated terminal, build the project by running:
cargo buildYou should see output including Compiling zerodb v0.1.0 followed by a Finished line with no errors.
Seeing a compile error?
If you see an error about an unresolved module, double-check that each filename in src/ matches the module name in lib.rs exactly. For example, the module row_store expects a file named row_store.rs (not rowstore.rs).
If Cargo cannot find zerocopy, make sure the version string in Cargo.toml is "0.8.52" with no typos.
Help me debug my cargo build error in the zerodb project.
Your project skeleton is ready. Next up, you will implement the type system and aligned buffer that form the foundation of zero-copy data access.
Build the Type System and Aligned Buffer
Your project skeleton compiles, but every module is empty. Before you can store or read data, you need a type system that describes the shape of each row and a buffer primitive that guarantees safe memory access.
Production databases like DuckDB precompute every column's byte offset at schema-creation time so that field access is O(1) pointer arithmetic. You'll build the same pattern here, plus an aligned buffer that makes zerocopy's zero-copy casts succeed without runtime alignment failures.
In this step, get ready to:
- Implement a type system (TypeId, ColumnDef, Schema) with precomputed byte offsets.
- Build an 8-byte aligned buffer that guarantees zero-copy struct casts never fail on alignment.
- Verify that a schema correctly computes row size and column offsets.
Implement the type system
Every column in a database table has a fixed type that determines how many bytes it occupies. Your type system encodes this relationship so the schema can precompute byte offsets once at construction time, rather than recalculating them on every read.
- Open src/types.rs in Zed. This file is currently empty.
- Add the TypeId enum and ColumnDef struct by copying this code into the file:
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum TypeId {
U8,
U16,
U32,
U64,
I8,
I16,
I32,
I64,
F32,
F64,
FixedBytes(usize),
}
impl TypeId {
pub fn size(&self) -> usize {
match self {
TypeId::U8 | TypeId::I8 => 1,
TypeId::U16 | TypeId::I16 => 2,
TypeId::U32 | TypeId::I32 | TypeId::F32 => 4,
TypeId::U64 | TypeId::I64 | TypeId::F64 => 8,
TypeId::FixedBytes(n) => *n,
}
}
}
#[derive(Debug, Clone)]
pub struct ColumnDef {
pub name: String,
pub type_id: TypeId,
}
impl ColumnDef {
pub fn new(name: impl Into<String>, type_id: TypeId) -> Self {
Self {
name: name.into(),
type_id,
}
}
pub fn size(&self) -> usize {
self.type_id.size()
}
}What does this code do?
TypeId is an enum representing every fixed-size type the storage engine supports. Each variant knows its own byte size via the size() method.
ColumnDef pairs a column name with its type. The FixedBytes(usize) variant lets you represent arbitrary fixed-size byte arrays (like [u8; 32] for hashes or UUIDs).
You won't see a change yet. The next chunk uses these types to build the Schema struct that precomputes offsets.
- Below the ColumnDef impl block, add the Schema struct that computes byte offsets at construction time:
#[derive(Debug, Clone)]
pub struct Schema {
columns: Vec<ColumnDef>,
offsets: Vec<usize>,
row_size: usize,
}
impl Schema {
pub fn new(columns: Vec<ColumnDef>) -> Self {
let mut offsets = Vec::with_capacity(columns.len());
let mut offset = 0;
for col in &columns {
offsets.push(offset);
offset += col.size();
}
Self {
columns,
offsets,
row_size: offset,
}
}
pub fn row_size(&self) -> usize {
self.row_size
}
pub fn column_count(&self) -> usize {
self.columns.len()
}
pub fn column_offset(&self, col_idx: usize) -> usize {
self.offsets[col_idx]
}
pub fn column_size(&self, col_idx: usize) -> usize {
self.columns[col_idx].size()
}
pub fn columns(&self) -> &[ColumnDef] {
&self.columns
}
}What does this code do?
Schema::new() walks through the column definitions once and accumulates a running byte offset. After construction, finding any field is pure arithmetic: column_offset(i) gives you the byte position of column i within a row.
row_size is the total bytes per row. The row store and column store both use this value to validate insertions.
Why precompute offsets instead of calculating on the fly?
Production databases access billions of rows. If you recalculated offsets on every read, you'd add a loop over all preceding columns for each field access. Precomputing once makes every subsequent access O(1).
- Save src/types.rs.
- Verify it compiles by running:
cargo buildYou should see Compiling zerodb followed by Finished with no errors. Warnings about unused code are expected at this stage.
Seeing a compile error?
- Check that both the TypeId enum and its impl block are inside src/types.rs. The module declaration in src/lib.rs expects the file to exist at that path.
- Verify that the Schema struct appears below the ColumnDef impl. Rust resolves types in declaration order within a file.
Help me debug a compile error in my types.rs file.
Implement the aligned buffer
The zerocopy crate's ref_from_bytes() function casts a byte slice directly into a typed reference. It returns an error if the source bytes are not properly aligned for the target type.
A plain Vec<u8> only guarantees 1-byte alignment. By backing your buffer with Vec<u64> instead, you guarantee the data starts at an 8-byte aligned address. This satisfies any primitive type's alignment requirement.
- Open src/buffer.rs and add the full AlignedBuf implementation:
/// 8-byte aligned byte buffer.
///
/// Backed by `Vec<u64>` to guarantee that the start of the buffer
/// is always aligned to 8 bytes. This ensures that zerocopy's
/// `ref_from_bytes` will not fail on alignment for any type with
/// alignment <= 8 (all primitive types).
pub struct AlignedBuf {
storage: Vec<u64>,
byte_len: usize,
}
impl AlignedBuf {
pub fn new() -> Self {
Self {
storage: Vec::new(),
byte_len: 0,
}
}
pub fn with_capacity(byte_cap: usize) -> Self {
let u64_cap = (byte_cap + 7) / 8;
Self {
storage: Vec::with_capacity(u64_cap),
byte_len: 0,
}
}
pub fn extend_from_slice(&mut self, data: &[u8]) {
let new_byte_len = self.byte_len + data.len();
let needed_u64s = (new_byte_len + 7) / 8;
if needed_u64s > self.storage.len() {
self.storage.resize(needed_u64s, 0);
}
unsafe {
let base = self.storage.as_mut_ptr() as *mut u8;
std::ptr::copy_nonoverlapping(data.as_ptr(), base.add(self.byte_len), data.len());
}
self.byte_len = new_byte_len;
}
pub fn as_slice(&self) -> &[u8] {
if self.byte_len == 0 {
return &[];
}
unsafe {
std::slice::from_raw_parts(self.storage.as_ptr() as *const u8, self.byte_len)
}
}
pub fn len(&self) -> usize {
self.byte_len
}
pub fn is_empty(&self) -> bool {
self.byte_len == 0
}
pub fn chunks_exact(&self, chunk_size: usize) -> std::slice::ChunksExact<'_, u8> {
self.as_slice().chunks_exact(chunk_size)
}
}
impl Default for AlignedBuf {
fn default() -> Self {
Self::new()
}
}What does this code do?
- storage: Vec<u64> forces the underlying allocation to be 8-byte aligned. This is the key trick that makes zero-copy casts work.
- byte_len tracks the actual number of meaningful bytes (which may not fill the last u64 completely).
- extend_from_slice() uses unsafe pointer arithmetic to copy bytes into the correct offset. The safety invariant is maintained because we resize storage before writing.
- as_slice() reinterprets the u64 storage as a byte slice. This is the zero-copy read path.
- chunks_exact() provides an iterator over fixed-size slices, which the row store will use to scan rows without allocating.
Why is the unsafe code sound?
The extend_from_slice method resizes the Vec<u64> before writing, guaranteeing the destination range is valid. The copy_nonoverlapping call writes to a non-overlapping target range within owned memory.
as_slice() creates a shared reference to bytes that are already initialized (resize zero-fills). The lifetime is tied to &self, preventing mutation while the slice is live.
- Save src/buffer.rs.
- Verify everything still compiles by running:
cargo buildYou should see Finished with no errors. Dead-code warnings are still fine.
Getting an unsafe-related compile error?
- Ensure the unsafe keyword wraps the block (both curly braces), not just the function call.
- Check that std::ptr::copy_nonoverlapping and std::slice::from_raw_parts are spelled with the full std:: path. No imports are needed for these.
Help me fix my AlignedBuf unsafe code.
Verify the schema computes correct offsets
The whole point of precomputing offsets is correctness. A struct with fields { a: u32, b: u32, c: [u8; 32] } should produce a row size of 40 bytes and offsets [0, 4, 8]. Let's write a test to prove it.
- At the bottom of src/types.rs, add this test module:
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn schema_computes_offsets_correctly() {
// Simulates a struct { a: u32, b: u32, c: [u8; 32] }
let schema = Schema::new(vec![
ColumnDef::new("a", TypeId::U32),
ColumnDef::new("b", TypeId::U32),
ColumnDef::new("c", TypeId::FixedBytes(32)),
]);
assert_eq!(schema.row_size(), 40);
assert_eq!(schema.column_offset(0), 0);
assert_eq!(schema.column_offset(1), 4);
assert_eq!(schema.column_offset(2), 8);
assert_eq!(schema.column_count(), 3);
}
}What does this test verify?
- Row size is 4 + 4 + 32 = 40 bytes. This matches a #[repr(C)] struct with these fields (no padding needed because u32 alignment is satisfied at offset 0 and 4).
- Column offsets accumulate correctly: column "a" starts at byte 0, column "b" at byte 4, column "c" at byte 8.
- Save src/types.rs.
- Run the test by executing:
cargo testYou should see output containing test types::tests::schema_computes_offsets_correctly ... ok and test result: ok. 1 passed.
Test failing with assertion error?
- If row_size() is not 40, check that the size() method returns 4 for TypeId::U32 and that FixedBytes(32) dereferences the inner value with *n.
- If offsets are wrong, confirm that Schema::new() pushes the offset before adding the column size (not after).
Help me debug my Schema offset test failure.
✔️ Awesome, I've got everything!
Great. Double check you've saved both src/types.rs and src/buffer.rs.
ⓧ I'd like to double check the full code
Here's the full src/types.rs (production code, excluding the test module):
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum TypeId {
U8,
U16,
U32,
U64,
I8,
I16,
I32,
I64,
F32,
F64,
FixedBytes(usize),
}
impl TypeId {
pub fn size(&self) -> usize {
match self {
TypeId::U8 | TypeId::I8 => 1,
TypeId::U16 | TypeId::I16 => 2,
TypeId::U32 | TypeId::I32 | TypeId::F32 => 4,
TypeId::U64 | TypeId::I64 | TypeId::F64 => 8,
TypeId::FixedBytes(n) => *n,
}
}
}
#[derive(Debug, Clone)]
pub struct ColumnDef {
pub name: String,
pub type_id: TypeId,
}
impl ColumnDef {
pub fn new(name: impl Into<String>, type_id: TypeId) -> Self {
Self {
name: name.into(),
type_id,
}
}
pub fn size(&self) -> usize {
self.type_id.size()
}
}
#[derive(Debug, Clone)]
pub struct Schema {
columns: Vec<ColumnDef>,
offsets: Vec<usize>,
row_size: usize,
}
impl Schema {
pub fn new(columns: Vec<ColumnDef>) -> Self {
let mut offsets = Vec::with_capacity(columns.len());
let mut offset = 0;
for col in &columns {
offsets.push(offset);
offset += col.size();
}
Self {
columns,
offsets,
row_size: offset,
}
}
pub fn row_size(&self) -> usize {
self.row_size
}
pub fn column_count(&self) -> usize {
self.columns.len()
}
pub fn column_offset(&self, col_idx: usize) -> usize {
self.offsets[col_idx]
}
pub fn column_size(&self, col_idx: usize) -> usize {
self.columns[col_idx].size()
}
pub fn columns(&self) -> &[ColumnDef] {
&self.columns
}
}And here's the full src/buffer.rs:
pub struct AlignedBuf {
storage: Vec<u64>,
byte_len: usize,
}
impl AlignedBuf {
pub fn new() -> Self {
Self {
storage: Vec::new(),
byte_len: 0,
}
}
pub fn with_capacity(byte_cap: usize) -> Self {
let u64_cap = (byte_cap + 7) / 8;
Self {
storage: Vec::with_capacity(u64_cap),
byte_len: 0,
}
}
pub fn extend_from_slice(&mut self, data: &[u8]) {
let new_byte_len = self.byte_len + data.len();
let needed_u64s = (new_byte_len + 7) / 8;
if needed_u64s > self.storage.len() {
self.storage.resize(needed_u64s, 0);
}
unsafe {
let base = self.storage.as_mut_ptr() as *mut u8;
std::ptr::copy_nonoverlapping(data.as_ptr(), base.add(self.byte_len), data.len());
}
self.byte_len = new_byte_len;
}
pub fn as_slice(&self) -> &[u8] {
if self.byte_len == 0 {
return &[];
}
unsafe {
std::slice::from_raw_parts(self.storage.as_ptr() as *const u8, self.byte_len)
}
}
pub fn len(&self) -> usize {
self.byte_len
}
pub fn is_empty(&self) -> bool {
self.byte_len == 0
}
pub fn chunks_exact(&self, chunk_size: usize) -> std::slice::ChunksExact<'_, u8> {
self.as_slice().chunks_exact(chunk_size)
}
}
impl Default for AlignedBuf {
fn default() -> Self {
Self::new()
}
}Your type system and aligned buffer are locked in. Next up, you'll build the row store that uses both to store and retrieve rows as contiguous byte slices with zero-copy access.
Implement the Row Store
Your type system and aligned buffer are ready. Now you need the component that actually stores rows of data and hands back zero-copy references to them.
A row store packs every row contiguously in a single buffer. Because the schema precomputes row_size, finding row N is pure offset arithmetic. No pointer chasing, no allocations, no deserialization.
In this step, get ready to:
- Build the RowStore struct with insert, get_row, and get_cell methods.
- Add a zero-allocation scan iterator over all rows.
- Demonstrate zero-copy struct access using zerocopy derive macros.
Build the RowStore struct and insert method
The RowStore owns a schema (to know row geometry), an AlignedBuf (to store raw bytes with guaranteed alignment), and a row counter.
- Open src/row_store.rs and add the struct definition plus the write path by pasting this code:
use crate::buffer::AlignedBuf;
use crate::types::Schema;
pub struct RowStore {
schema: Schema,
data: AlignedBuf,
row_count: usize,
}
impl RowStore {
pub fn new(schema: Schema) -> Self {
Self {
schema,
data: AlignedBuf::new(),
row_count: 0,
}
}
pub fn schema(&self) -> &Schema {
&self.schema
}
pub fn row_count(&self) -> usize {
self.row_count
}
pub fn insert(&mut self, row: &[u8]) -> Result<usize, &'static str> {
if row.len() != self.schema.row_size() {
return Err("row size mismatch");
}
self.data.extend_from_slice(row);
let idx = self.row_count;
self.row_count += 1;
Ok(idx)
}
}What does this code do?
The insert() method is the gatekeeper. It checks that the incoming byte slice is exactly row_size bytes, then appends them to the contiguous buffer.
Because every row occupies the same number of bytes, the buffer grows in predictable increments. Row N always lives at offset N * row_size. This is the foundation for O(1) reads.
Now add the read path. These methods return references directly into the buffer with no copying.
- Add get_row() and get_cell() to the existing impl RowStore block, directly below the closing brace of insert():
pub fn get_row(&self, row_idx: usize) -> Option<&[u8]> {
if row_idx >= self.row_count {
return None;
}
let start = row_idx * self.schema.row_size();
let end = start + self.schema.row_size();
Some(&self.data.as_slice()[start..end])
}
pub fn get_cell(&self, row_idx: usize, col_idx: usize) -> Option<&[u8]> {
let row = self.get_row(row_idx)?;
let offset = self.schema.column_offset(col_idx);
let size = self.schema.column_size(col_idx);
Some(&row[offset..offset + size])
}What does this code do?
- get_row() computes start = row_idx * row_size and returns a slice of the buffer. No allocation, no copy. The returned &[u8] points directly into the storage.
- get_cell() narrows further by using the schema's precomputed column offset. It returns only the bytes belonging to that specific field.
- Save src/row_store.rs.
- Verify the project still compiles by running:
cargo build- You should see Compiling zerodb with no errors. Warnings about unused code are expected at this stage.
Seeing a compile error?
- Check that use crate::buffer::AlignedBuf; and use crate::types::Schema; are at the top of the file. Without these imports, Rust cannot find the types.
- Verify that get_row and get_cell are inside the same impl RowStore { } block as insert(). A misplaced closing brace will cause "method not found" errors later.
Help me debug a compile error in my RowStore implementation
Add the scan iterator
Point lookups are useful, but analytical workloads need to visit every row. A scan iterator should yield each row without allocating memory for each one.
- In src/row_store.rs, add the scan() method at the end of the impl RowStore block, directly below get_cell():
pub fn scan(&self) -> impl Iterator<Item = &[u8]> {
self.data.chunks_exact(self.schema.row_size())
}Why is this zero-allocation?
The chunks_exact() method returns an iterator that walks the underlying byte slice in fixed-size steps. Each next() call advances a pointer. It never allocates a new buffer or copies row data.
This is the same pattern production databases use for full table scans. The CPU prefetcher can predict the linear access pattern, keeping cache misses near zero.
- Save src/row_store.rs.
✔️ Awesome, I've got everything!
Great. Double check you've saved the file before moving on.
ⓧ I'd like to double check the full code
use crate::buffer::AlignedBuf;
use crate::types::Schema;
pub struct RowStore {
schema: Schema,
data: AlignedBuf,
row_count: usize,
}
impl RowStore {
pub fn new(schema: Schema) -> Self {
Self {
schema,
data: AlignedBuf::new(),
row_count: 0,
}
}
pub fn schema(&self) -> &Schema {
&self.schema
}
pub fn row_count(&self) -> usize {
self.row_count
}
pub fn insert(&mut self, row: &[u8]) -> Result<usize, &'static str> {
if row.len() != self.schema.row_size() {
return Err("row size mismatch");
}
self.data.extend_from_slice(row);
let idx = self.row_count;
self.row_count += 1;
Ok(idx)
}
pub fn get_row(&self, row_idx: usize) -> Option<&[u8]> {
if row_idx >= self.row_count {
return None;
}
let start = row_idx * self.schema.row_size();
let end = start + self.schema.row_size();
Some(&self.data.as_slice()[start..end])
}
pub fn get_cell(&self, row_idx: usize, col_idx: usize) -> Option<&[u8]> {
let row = self.get_row(row_idx)?;
let offset = self.schema.column_offset(col_idx);
let size = self.schema.column_size(col_idx);
Some(&row[offset..offset + size])
}
pub fn scan(&self) -> impl Iterator<Item = &[u8]> {
self.data.chunks_exact(self.schema.row_size())
}
}Demonstrate zero-copy struct access
Returning raw &[u8] slices is powerful but not ergonomic. The zerocopy crate bridges this gap. You define a Rust struct with #[repr(C)] and derive traits that let you cast between the struct and raw bytes with zero overhead.
- Open src/main.rs and replace its contents with the following code:
use zerocopy::{FromBytes, IntoBytes, KnownLayout, Immutable};
use zerodb::row_store::RowStore;
use zerodb::types::{ColumnDef, Schema, TypeId};
// A fixed-size struct whose memory layout matches our schema exactly
#[derive(Debug, Clone, Copy, FromBytes, IntoBytes, KnownLayout, Immutable)]
#[repr(C)]
struct Record {
a: u32,
b: u32,
c: [u8; 32],
}
fn main() {
// Build a schema that matches Record's layout: u32 + u32 + 32 bytes = 40 bytes
let schema = Schema::new(vec![
ColumnDef::new("a", TypeId::U32),
ColumnDef::new("b", TypeId::U32),
ColumnDef::new("c", TypeId::FixedBytes(32)),
]);
let mut store = RowStore::new(schema);
// Insert a record by reinterpreting the struct as raw bytes
let rec = Record { a: 42, b: 100, c: [0xAB; 32] };
store.insert(rec.as_bytes()).unwrap();
// Read it back: get_row returns &[u8] pointing into the buffer
// ref_from_bytes casts that slice to &Record with zero copying
let row = store.get_row(0).unwrap();
let r = Record::ref_from_bytes(row).unwrap();
println!("a={}, b={}, c[0]={:#04x}", r.a, r.b, r.c[0]);
}What makes this zero-copy?
- rec.as_bytes() reinterprets the struct as a byte slice without copying. The IntoBytes derive macro guarantees this is safe at compile time.
- Record::ref_from_bytes(row) casts the raw byte slice back to &Record. The returned reference points directly into the AlignedBuf storage. No intermediate allocation exists.
- #[repr(C)] ensures the struct's field order and padding match what we stored. Without it, the Rust compiler could reorder fields, breaking the byte-level correspondence.
- Save src/main.rs and run the program:
cargo run- You should see the output a=42, b=100, c[0]=0xab printed to the terminal. This confirms that a struct was stored as raw bytes, then read back via a direct cast with no copying.
Seeing a size or alignment error at runtime?
- If ref_from_bytes returns an error, check that your Record struct fields exactly match the schema column definitions in both order and size. A mismatch means the byte layout diverges.
- Make sure #[repr(C)] is placed above the struct definition. Without it, the compiler may insert padding that changes the total size.
- If you see a compile error about missing traits, verify your derive line includes all four: FromBytes, IntoBytes, KnownLayout, Immutable.
Help me debug a zerocopy cast error in my RowStore demo
Why does AlignedBuf matter here?
The Record struct has alignment 4 (from the u32 fields). The ref_from_bytes() call will fail if the byte slice is not 4-byte aligned.
Because AlignedBuf is backed by Vec<u64> (8-byte alignment), every row starts at an 8-byte boundary or a multiple of row_size (40 bytes, which is divisible by 4). The cast is guaranteed to succeed.
✔️ Awesome, I've got everything!
Great. Make sure both files are saved before continuing.
ⓧ I'd like to double check the full code
Here is the complete src/main.rs for this step:
use zerocopy::{FromBytes, IntoBytes, KnownLayout, Immutable};
use zerodb::row_store::RowStore;
use zerodb::types::{ColumnDef, Schema, TypeId};
#[derive(Debug, Clone, Copy, FromBytes, IntoBytes, KnownLayout, Immutable)]
#[repr(C)]
struct Record {
a: u32,
b: u32,
c: [u8; 32],
}
fn main() {
let schema = Schema::new(vec![
ColumnDef::new("a", TypeId::U32),
ColumnDef::new("b", TypeId::U32),
ColumnDef::new("c", TypeId::FixedBytes(32)),
]);
let mut store = RowStore::new(schema);
let rec = Record { a: 42, b: 100, c: [0xAB; 32] };
store.insert(rec.as_bytes()).unwrap();
let row = store.get_row(0).unwrap();
let r = Record::ref_from_bytes(row).unwrap();
println!("a={}, b={}, c[0]={:#04x}", r.a, r.b, r.c[0]);
}Your row store is complete. Every insert appends exactly row_size bytes, every read returns a zero-copy reference, and the scan iterator walks all rows with zero per-row allocations. Next up, you will build the column store to see how the same data looks when organized by column instead of by row.
Implement the Column Store
Your row store packs all fields of a record side by side, which is perfect for point lookups. But analytical queries that touch only one column pay a heavy price: the CPU cache loads entire 40-byte rows just to read 4 bytes of column a.
A column store flips the layout. Each column lives in its own contiguous buffer, so scanning one column touches only that column's bytes. In this step, you will implement a full column store that splits rows on insert and reassembles them on demand.
In this step, get ready to:
- Build a ColumnStore that decomposes rows into per-column buffers on insert.
- Add column-level scan methods and a row-reconstruction function.
- Demonstrate the cache-locality advantage of columnar storage.
Define the ColumnStore struct
The key difference from RowStore is that ColumnStore holds one AlignedBuf per column instead of a single buffer for all rows.
- Open src/column_store.rs and add the struct definition and constructor:
use crate::buffer::AlignedBuf;
use crate::types::Schema;
pub struct ColumnStore {
schema: Schema,
columns: Vec<AlignedBuf>,
row_count: usize,
}
impl ColumnStore {
pub fn new(schema: Schema) -> Self {
let num_cols = schema.column_count();
let columns = (0..num_cols).map(|_| AlignedBuf::new()).collect();
Self {
schema,
columns,
row_count: 0,
}
}
pub fn schema(&self) -> &Schema {
&self.schema
}
pub fn row_count(&self) -> usize {
self.row_count
}
}What does this code do?
Instead of one big buffer, ColumnStore allocates a separate AlignedBuf for each column in the schema. The constructor creates exactly as many buffers as there are columns.
Each buffer will hold only that column's data across all rows, keeping values of the same type tightly packed in memory.
- Below the row_count() method, add the insert() and get_cell() methods:
pub fn insert(&mut self, row: &[u8]) -> Result<usize, &'static str> {
if row.len() != self.schema.row_size() {
return Err("row size mismatch");
}
for col_idx in 0..self.schema.column_count() {
let offset = self.schema.column_offset(col_idx);
let size = self.schema.column_size(col_idx);
self.columns[col_idx].extend_from_slice(&row[offset..offset + size]);
}
let idx = self.row_count;
self.row_count += 1;
Ok(idx)
}
pub fn get_cell(&self, row_idx: usize, col_idx: usize) -> Option<&[u8]> {
if row_idx >= self.row_count || col_idx >= self.schema.column_count() {
return None;
}
let size = self.schema.column_size(col_idx);
let start = row_idx * size;
Some(&self.columns[col_idx].as_slice()[start..start + size])
}What does this code do?
- insert() takes a full row as bytes and splits it using the schema's precomputed offsets. Each column's slice gets appended to its own buffer.
- get_cell() computes the byte position within a column buffer using simple arithmetic: row_idx * column_size. No per-row pointer chasing needed.
- Save src/column_store.rs.
- Verify it compiles by running:
cargo buildSeeing a compile error?
Make sure the insert and get_cell methods are inside the same impl ColumnStore block as new(). Also check that your use statements at the top import both AlignedBuf and Schema.
Help me debug a compile error in my ColumnStore
Add column-level scan and row reconstruction methods
The column store's real power is returning an entire column as a single contiguous slice. This is what makes analytical scans fast. You also need a way to reconstruct full rows from scattered column data.
- Below get_cell(), add the column access methods:
pub fn get_column(&self, col_idx: usize) -> Option<&[u8]> {
if col_idx >= self.schema.column_count() {
return None;
}
Some(self.columns[col_idx].as_slice())
}
pub fn scan_column(&self, col_idx: usize) -> Option<impl Iterator<Item = &[u8]>> {
if col_idx >= self.schema.column_count() {
return None;
}
let size = self.schema.column_size(col_idx);
Some(self.columns[col_idx].chunks_exact(size))
}Why is get_column so powerful?
get_column() returns a single contiguous byte slice covering every value in that column. If you have 1 million rows with a u32 column, that's 4 MB of tightly packed values with zero gaps.
scan_column() layers an iterator on top, yielding individual value slices via chunks_exact. Both are zero-copy operations returning references into the existing buffer.
- Below scan_column(), add the row reconstruction method:
pub fn read_row_into(&self, row_idx: usize, buf: &mut [u8]) -> Result<(), &'static str> {
if row_idx >= self.row_count {
return Err("row index out of bounds");
}
if buf.len() < self.schema.row_size() {
return Err("buffer too small");
}
for col_idx in 0..self.schema.column_count() {
let offset = self.schema.column_offset(col_idx);
let size = self.schema.column_size(col_idx);
let start = row_idx * size;
buf[offset..offset + size]
.copy_from_slice(&self.columns[col_idx].as_slice()[start..start + size]);
}
Ok(())
}What does read_row_into do?
This is the columnar trade-off in action. Reassembling a full row requires visiting each column buffer and copying the relevant bytes into the caller's buffer. It is O(num_columns) with actual memcpy work.
Unlike the row store's get_row() which returns a zero-copy &[u8] slice, this method writes into a mutable buffer. That is the fundamental row-vs-column trade-off: row stores give free row reads, column stores give free column scans.
- Save src/column_store.rs.
- Confirm it compiles by running:
cargo buildYou should see a successful build with no errors.
Seeing lifetime or borrow errors?
The scan_column return type Option<impl Iterator<Item = &[u8]>> requires Rust to infer the lifetime from &self. Make sure the method takes &self (not &mut self) and that chunks_exact is called on the buffer's slice.
Help me fix lifetime errors in scan_column
✔️ Awesome, I've got everything!
Great. Double check you have saved src/column_store.rs before continuing.
ⓧ I'd like to double check the full code
use crate::buffer::AlignedBuf;
use crate::types::Schema;
pub struct ColumnStore {
schema: Schema,
columns: Vec<AlignedBuf>,
row_count: usize,
}
impl ColumnStore {
pub fn new(schema: Schema) -> Self {
let num_cols = schema.column_count();
let columns = (0..num_cols).map(|_| AlignedBuf::new()).collect();
Self {
schema,
columns,
row_count: 0,
}
}
pub fn schema(&self) -> &Schema {
&self.schema
}
pub fn row_count(&self) -> usize {
self.row_count
}
pub fn insert(&mut self, row: &[u8]) -> Result<usize, &'static str> {
if row.len() != self.schema.row_size() {
return Err("row size mismatch");
}
for col_idx in 0..self.schema.column_count() {
let offset = self.schema.column_offset(col_idx);
let size = self.schema.column_size(col_idx);
self.columns[col_idx].extend_from_slice(&row[offset..offset + size]);
}
let idx = self.row_count;
self.row_count += 1;
Ok(idx)
}
pub fn get_cell(&self, row_idx: usize, col_idx: usize) -> Option<&[u8]> {
if row_idx >= self.row_count || col_idx >= self.schema.column_count() {
return None;
}
let size = self.schema.column_size(col_idx);
let start = row_idx * size;
Some(&self.columns[col_idx].as_slice()[start..start + size])
}
pub fn get_column(&self, col_idx: usize) -> Option<&[u8]> {
if col_idx >= self.schema.column_count() {
return None;
}
Some(self.columns[col_idx].as_slice())
}
pub fn scan_column(&self, col_idx: usize) -> Option<impl Iterator<Item = &[u8]>> {
if col_idx >= self.schema.column_count() {
return None;
}
let size = self.schema.column_size(col_idx);
Some(self.columns[col_idx].chunks_exact(size))
}
pub fn read_row_into(&self, row_idx: usize, buf: &mut [u8]) -> Result<(), &'static str> {
if row_idx >= self.row_count {
return Err("row index out of bounds");
}
if buf.len() < self.schema.row_size() {
return Err("buffer too small");
}
for col_idx in 0..self.schema.column_count() {
let offset = self.schema.column_offset(col_idx);
let size = self.schema.column_size(col_idx);
let start = row_idx * size;
buf[offset..offset + size]
.copy_from_slice(&self.columns[col_idx].as_slice()[start..start + size]);
}
Ok(())
}
}Demonstrate the columnar advantage
Time to see the layout difference in action. You will update main.rs to iterate column a in both stores and observe the difference: 4-byte contiguous values in the column store vs 40-byte stride in the row store.
- Replace the entire contents of src/main.rs with the following:
use zerocopy::{FromBytes, IntoBytes, KnownLayout, Immutable};
use zerodb::row_store::RowStore;
use zerodb::column_store::ColumnStore;
use zerodb::types::{ColumnDef, Schema, TypeId};
#[derive(Debug, Clone, Copy, FromBytes, IntoBytes, KnownLayout, Immutable)]
#[repr(C)]
struct Record {
a: u32,
b: u32,
c: [u8; 32],
}
fn main() {
let schema = Schema::new(vec![
ColumnDef::new("a", TypeId::U32),
ColumnDef::new("b", TypeId::U32),
ColumnDef::new("c", TypeId::FixedBytes(32)),
]);
let records = [
Record { a: 42, b: 100, c: [0xAB; 32] },
Record { a: 7, b: 200, c: [0xCD; 32] },
Record { a: 99, b: 300, c: [0xEF; 32] },
];
let mut col_store = ColumnStore::new(schema.clone());
for rec in &records {
col_store.insert(rec.as_bytes()).unwrap();
}
// Columnar advantage: column 'a' is contiguous
let col_a = col_store.get_column(0).unwrap();
println!("Column 'a' contiguous bytes: {:?}", col_a.len());
for chunk in col_a.chunks_exact(4) {
let val = u32::from_ne_bytes(chunk.try_into().unwrap());
println!(" a={}", val);
}
// Row store: column 'a' values are 40 bytes apart
let mut row_store = RowStore::new(schema);
for rec in &records {
row_store.insert(rec.as_bytes()).unwrap();
}
for row in row_store.scan() {
let a = u32::from_ne_bytes(row[0..4].try_into().unwrap());
println!(" row a={} (stride=40 bytes)", a);
}
}What is this demonstrating?
- In the column store, get_column(0) returns 12 contiguous bytes (3 rows x 4 bytes each). Iterating this slice touches only the data you care about.
- In the row store, reading column a means visiting bytes at offsets 0, 40, and 80. Between each value are 36 bytes of unrelated data that still get pulled into cache lines.
- At scale (millions of rows), this difference translates directly to fewer cache misses and higher throughput for analytical scans.
- Save src/main.rs and run the program:
cargo runYou should see the column store printing all three a values (42, 7, 99) from its contiguous 12-byte buffer, followed by the row store printing the same values accessed at a 40-byte stride.
Getting an unresolved import error?
Make sure your import line reads use zerodb::column_store::ColumnStore; and that src/lib.rs still contains pub mod column_store;. The module must be declared public in lib.rs for main.rs to use it.
Help me fix an unresolved import for ColumnStore
✔️ Awesome, I've got everything!
Great. Make sure both files are saved before moving on.
ⓧ I'd like to double check the full code
use zerocopy::{FromBytes, IntoBytes, KnownLayout, Immutable};
use zerodb::row_store::RowStore;
use zerodb::column_store::ColumnStore;
use zerodb::types::{ColumnDef, Schema, TypeId};
#[derive(Debug, Clone, Copy, FromBytes, IntoBytes, KnownLayout, Immutable)]
#[repr(C)]
struct Record {
a: u32,
b: u32,
c: [u8; 32],
}
fn main() {
let schema = Schema::new(vec![
ColumnDef::new("a", TypeId::U32),
ColumnDef::new("b", TypeId::U32),
ColumnDef::new("c", TypeId::FixedBytes(32)),
]);
let records = [
Record { a: 42, b: 100, c: [0xAB; 32] },
Record { a: 7, b: 200, c: [0xCD; 32] },
Record { a: 99, b: 300, c: [0xEF; 32] },
];
let mut col_store = ColumnStore::new(schema.clone());
for rec in &records {
col_store.insert(rec.as_bytes()).unwrap();
}
// Columnar advantage: column 'a' is contiguous
let col_a = col_store.get_column(0).unwrap();
println!("Column 'a' contiguous bytes: {:?}", col_a.len());
for chunk in col_a.chunks_exact(4) {
let val = u32::from_ne_bytes(chunk.try_into().unwrap());
println!(" a={}", val);
}
// Row store: column 'a' values are 40 bytes apart
let mut row_store = RowStore::new(schema);
for rec in &records {
row_store.insert(rec.as_bytes()).unwrap();
}
for row in row_store.scan() {
let a = u32::from_ne_bytes(row[0..4].try_into().unwrap());
println!(" row a={} (stride=40 bytes)", a);
}
}Your column store is working. You now have two storage layouts that make opposite trade-offs. Next, you will unify them behind a single Table API that lets you switch between layouts at runtime.
Build the Unified Table API with Mode Switching
You now have two storage engines that each excel at different access patterns. The row store is perfect for point lookups. The column store dominates analytical scans.
The problem is that callers have to choose one layout at creation time and live with the trade-offs forever. In this step, you will build a unified Table API that hides the storage choice behind a single interface and lets you switch between layouts at runtime.
In this step, get ready to:
- Implement a unified Table struct that dispatches to RowStore or ColumnStore internally.
- Add a switch_mode method that physically reorganizes data between layouts.
- Update main.rs to demonstrate zero-copy access, mode switching, and data integrity verification.
Define the Table types and reference enums
The Table API needs two key design choices. First, a StorageMode enum that the caller sees (Row or Column). Second, internal reference types that express whether data is returned zero-copy or must be reconstructed.
- In src/table.rs, add the imports, enum definitions, struct definitions, and reference type implementations by pasting this code:
use crate::column_store::ColumnStore;
use crate::row_store::RowStore;
use crate::types::Schema;
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum StorageMode {
Row,
Column,
}
enum Storage {
Row(RowStore),
Column(ColumnStore),
}
pub struct Table {
name: String,
schema: Schema,
storage: Storage,
}
pub enum RowRef<'a> {
Borrowed(&'a [u8]),
Owned(Vec<u8>),
}
impl<'a> RowRef<'a> {
pub fn as_bytes(&self) -> &[u8] {
match self {
RowRef::Borrowed(b) => b,
RowRef::Owned(v) => v,
}
}
}
pub enum ColumnRef<'a> {
Borrowed(&'a [u8]),
Owned(Vec<u8>),
}
impl<'a> ColumnRef<'a> {
pub fn as_bytes(&self) -> &[u8] {
match self {
ColumnRef::Borrowed(b) => b,
ColumnRef::Owned(v) => v,
}
}
}What does this code do?
StorageMode is the public-facing enum that callers use to request Row or Column layout. Storage is the internal enum that holds the actual store instance. It is not public because callers should not reach into the storage directly.
RowRef and ColumnRef express the zero-copy vs owned distinction. In row mode, get_row() returns Borrowed (a direct pointer into the buffer). In column mode, it must reconstruct the row from scattered column buffers, so it returns Owned. The as_bytes() method unifies both variants behind a single &[u8] interface.
- You won't see a change yet. The next chunk adds the Table implementation that uses these types.
Implement the unified Table methods
Every Table method dispatches to the underlying store via a match on the internal Storage enum. The key insight is in get_row() and get_column(). Each returns zero-copy data when the storage mode is favorable, and falls back to owned reconstruction otherwise.
- In src/table.rs, add the impl Table block below the ColumnRef impl by pasting this code:
impl Table {
pub fn new(name: impl Into<String>, schema: Schema, mode: StorageMode) -> Self {
let schema_clone = schema.clone();
let storage = match mode {
StorageMode::Row => Storage::Row(RowStore::new(schema)),
StorageMode::Column => Storage::Column(ColumnStore::new(schema)),
};
Self {
name: name.into(),
schema: schema_clone,
storage,
}
}
pub fn name(&self) -> &str {
&self.name
}
pub fn schema(&self) -> &Schema {
&self.schema
}
pub fn mode(&self) -> StorageMode {
match &self.storage {
Storage::Row(_) => StorageMode::Row,
Storage::Column(_) => StorageMode::Column,
}
}
pub fn row_count(&self) -> usize {
match &self.storage {
Storage::Row(s) => s.row_count(),
Storage::Column(s) => s.row_count(),
}
}
pub fn insert(&mut self, row: &[u8]) -> Result<usize, &'static str> {
match &mut self.storage {
Storage::Row(s) => s.insert(row),
Storage::Column(s) => s.insert(row),
}
}
pub fn get_cell(&self, row_idx: usize, col_idx: usize) -> Option<&[u8]> {
match &self.storage {
Storage::Row(s) => s.get_cell(row_idx, col_idx),
Storage::Column(s) => s.get_cell(row_idx, col_idx),
}
}
pub fn get_row(&self, row_idx: usize) -> Option<RowRef<'_>> {
match &self.storage {
Storage::Row(s) => s.get_row(row_idx).map(RowRef::Borrowed),
Storage::Column(s) => {
let mut buf = vec![0u8; self.schema.row_size()];
s.read_row_into(row_idx, &mut buf).ok()?;
Some(RowRef::Owned(buf))
}
}
}
pub fn get_column(&self, col_idx: usize) -> Option<ColumnRef<'_>> {
match &self.storage {
Storage::Column(s) => s.get_column(col_idx).map(ColumnRef::Borrowed),
Storage::Row(s) => {
let col_size = self.schema.column_size(col_idx);
let col_offset = self.schema.column_offset(col_idx);
let mut buf = Vec::with_capacity(col_size * s.row_count());
for row in s.scan() {
buf.extend_from_slice(&row[col_offset..col_offset + col_size]);
}
Some(ColumnRef::Owned(buf))
}
}
}
}What does this code do?
new() clones the schema (one copy for the Table, one moved into the store) and creates the appropriate storage backend based on the requested mode.
get_row() is where the zero-copy magic happens. In Row mode, it returns RowRef::Borrowed pointing directly into the aligned buffer with no allocation. In Column mode, it must reconstruct the row from scattered column buffers, so it allocates a temporary Vec and returns RowRef::Owned.
get_column() mirrors this pattern in reverse. Column mode returns a zero-copy ColumnRef::Borrowed slice of the contiguous column buffer. Row mode must iterate all rows and extract one column's bytes into a new Vec.
- Save src/table.rs.
- Verify the module compiles by running:
cargo buildYou should see the build succeed with no errors. Warnings about unused code are fine at this stage.
Seeing a compile error?
Check that your use statements at the top of src/table.rs import from the correct module paths: crate::column_store::ColumnStore, crate::row_store::RowStore, and crate::types::Schema.
If you see a lifetime error, make sure the RowRef<'a> and ColumnRef<'a> definitions include the lifetime parameter on both the enum and the impl block.
help me debug a compile error in my table.rs module
Implement mode switching with physical data reorganization
Mode switching is an O(n) operation that physically moves all data from one layout to another. Row-to-column splits each contiguous row into per-column buffers. Column-to-row reconstructs full rows from scattered column data. This is the fundamental trade-off that production databases face when choosing a storage layout.
- In src/table.rs, add the switch_mode method inside the existing impl Table block, below get_column():
pub fn switch_mode(&mut self, new_mode: StorageMode) {
if self.mode() == new_mode {
return;
}
let new_storage = match (&self.storage, new_mode) {
(Storage::Row(row_store), StorageMode::Column) => {
let mut col_store = ColumnStore::new(self.schema.clone());
for row in row_store.scan() {
col_store.insert(row).expect("schema matches");
}
Storage::Column(col_store)
}
(Storage::Column(col_store), StorageMode::Row) => {
let mut row_store = RowStore::new(self.schema.clone());
let mut buf = vec![0u8; self.schema.row_size()];
for i in 0..col_store.row_count() {
col_store.read_row_into(i, &mut buf).expect("valid index");
row_store.insert(&buf).expect("schema matches");
}
Storage::Row(row_store)
}
_ => unreachable!(),
};
self.storage = new_storage;
}What does this code do?
The early return on self.mode() == new_mode avoids unnecessary work if the table is already in the requested mode.
Row-to-Column: iterates the row store's contiguous buffer via scan() and inserts each row into a fresh ColumnStore. The column store's insert() splits each row by schema offsets and appends to per-column buffers.
Column-to-Row: uses read_row_into() to reconstruct each row from column data into a reusable buffer, then inserts into a fresh RowStore. The reusable buffer avoids allocating per iteration.
Why is switch_mode O(n)?
Every byte of every row must be read and written into the new layout. There is no shortcut. This is why production databases (like DuckDB) choose one layout at table creation and rarely switch. The ability to switch at runtime here demonstrates the trade-off: you pay O(n) once to benefit from a different access pattern going forward.
- Save src/table.rs.
- Run cargo build to confirm the module still compiles.
Borrow checker error on switch_mode?
The match borrows &self.storage immutably to read from the old store, then assigns to self.storage after the match. If you try to assign inside the match arms, you will hit a borrow conflict. Make sure the new storage is built into a local variable (new_storage) and assigned after the match expression completes.
help me fix a borrow checker error in switch_mode
✔️ Awesome, I've got everything!
Great. Make sure you have saved src/table.rs before moving on.
ⓧ I'd like to double check the full code
Here is the complete src/table.rs for comparison:
use crate::column_store::ColumnStore;
use crate::row_store::RowStore;
use crate::types::Schema;
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum StorageMode {
Row,
Column,
}
enum Storage {
Row(RowStore),
Column(ColumnStore),
}
pub struct Table {
name: String,
schema: Schema,
storage: Storage,
}
pub enum RowRef<'a> {
Borrowed(&'a [u8]),
Owned(Vec<u8>),
}
impl<'a> RowRef<'a> {
pub fn as_bytes(&self) -> &[u8] {
match self {
RowRef::Borrowed(b) => b,
RowRef::Owned(v) => v,
}
}
}
pub enum ColumnRef<'a> {
Borrowed(&'a [u8]),
Owned(Vec<u8>),
}
impl<'a> ColumnRef<'a> {
pub fn as_bytes(&self) -> &[u8] {
match self {
ColumnRef::Borrowed(b) => b,
ColumnRef::Owned(v) => v,
}
}
}
impl Table {
pub fn new(name: impl Into<String>, schema: Schema, mode: StorageMode) -> Self {
let schema_clone = schema.clone();
let storage = match mode {
StorageMode::Row => Storage::Row(RowStore::new(schema)),
StorageMode::Column => Storage::Column(ColumnStore::new(schema)),
};
Self {
name: name.into(),
schema: schema_clone,
storage,
}
}
pub fn name(&self) -> &str {
&self.name
}
pub fn schema(&self) -> &Schema {
&self.schema
}
pub fn mode(&self) -> StorageMode {
match &self.storage {
Storage::Row(_) => StorageMode::Row,
Storage::Column(_) => StorageMode::Column,
}
}
pub fn row_count(&self) -> usize {
match &self.storage {
Storage::Row(s) => s.row_count(),
Storage::Column(s) => s.row_count(),
}
}
pub fn insert(&mut self, row: &[u8]) -> Result<usize, &'static str> {
match &mut self.storage {
Storage::Row(s) => s.insert(row),
Storage::Column(s) => s.insert(row),
}
}
pub fn get_cell(&self, row_idx: usize, col_idx: usize) -> Option<&[u8]> {
match &self.storage {
Storage::Row(s) => s.get_cell(row_idx, col_idx),
Storage::Column(s) => s.get_cell(row_idx, col_idx),
}
}
pub fn get_row(&self, row_idx: usize) -> Option<RowRef<'_>> {
match &self.storage {
Storage::Row(s) => s.get_row(row_idx).map(RowRef::Borrowed),
Storage::Column(s) => {
let mut buf = vec![0u8; self.schema.row_size()];
s.read_row_into(row_idx, &mut buf).ok()?;
Some(RowRef::Owned(buf))
}
}
}
pub fn get_column(&self, col_idx: usize) -> Option<ColumnRef<'_>> {
match &self.storage {
Storage::Column(s) => s.get_column(col_idx).map(ColumnRef::Borrowed),
Storage::Row(s) => {
let col_size = self.schema.column_size(col_idx);
let col_offset = self.schema.column_offset(col_idx);
let mut buf = Vec::with_capacity(col_size * s.row_count());
for row in s.scan() {
buf.extend_from_slice(&row[col_offset..col_offset + col_size]);
}
Some(ColumnRef::Owned(buf))
}
}
}
pub fn switch_mode(&mut self, new_mode: StorageMode) {
if self.mode() == new_mode {
return;
}
let new_storage = match (&self.storage, new_mode) {
(Storage::Row(row_store), StorageMode::Column) => {
let mut col_store = ColumnStore::new(self.schema.clone());
for row in row_store.scan() {
col_store.insert(row).expect("schema matches");
}
Storage::Column(col_store)
}
(Storage::Column(col_store), StorageMode::Row) => {
let mut row_store = RowStore::new(self.schema.clone());
let mut buf = vec![0u8; self.schema.row_size()];
for i in 0..col_store.row_count() {
col_store.read_row_into(i, &mut buf).expect("valid index");
row_store.insert(&buf).expect("schema matches");
}
Storage::Row(row_store)
}
_ => unreachable!(),
};
self.storage = new_storage;
}
}Demonstrate the full Table API in main.rs
Now you will rewrite src/main.rs to exercise the unified Table API. The demo creates a table in Row mode, inserts records via zerocopy, reads them back as zero-copy struct references, switches to Column mode for a contiguous column scan, then switches back and verifies that data survived the round-trip.
- Replace the entire contents of src/main.rs with this code:
use zerocopy::{FromBytes, IntoBytes, KnownLayout, Immutable};
use zerodb::table::{StorageMode, Table};
use zerodb::types::{ColumnDef, Schema, TypeId};
#[derive(Debug, Clone, Copy, FromBytes, IntoBytes, KnownLayout, Immutable)]
#[repr(C)]
struct Record {
a: u32,
b: u32,
c: [u8; 32],
}
fn main() {
let schema = Schema::new(vec![
ColumnDef::new("a", TypeId::U32),
ColumnDef::new("b", TypeId::U32),
ColumnDef::new("c", TypeId::FixedBytes(32)),
]);
assert_eq!(schema.row_size(), std::mem::size_of::<Record>());
println!("Schema row_size: {} bytes", schema.row_size());
let mut table = Table::new("records", schema, StorageMode::Row);
// Insert records using zerocopy - zero-allocation serialization
let records = [
Record { a: 42, b: 100, c: [0xAB; 32] },
Record { a: 7, b: 200, c: [0xCD; 32] },
Record { a: 99, b: 300, c: [0xEF; 32] },
];
for rec in &records {
table.insert(rec.as_bytes()).unwrap();
}
println!("\n--- Row Mode: Zero-Copy Struct Access ---");
for i in 0..table.row_count() {
if let Some(row_ref) = table.get_row(i) {
let r = Record::ref_from_bytes(row_ref.as_bytes()).unwrap();
println!(" Row {}: a={}, b={}, c[0]={:#04x}", i, r.a, r.b, r.c[0]);
}
}
// Single cell access
if let Some(cell) = table.get_cell(1, 0) {
let val = u32::from_ne_bytes(cell.try_into().unwrap());
println!("\n Cell [1][a] = {}", val);
}
println!("\n--- Switching to Column Mode ---");
table.switch_mode(StorageMode::Column);
// Column scan - contiguous memory, cache-friendly
if let Some(col) = table.get_column(0) {
println!(" Column 'a' (contiguous {} bytes):", col.as_bytes().len());
for (i, chunk) in col.as_bytes().chunks_exact(4).enumerate() {
let val = u32::from_ne_bytes(chunk.try_into().unwrap());
println!(" row {}: {}", i, val);
}
}
// Cell access still works in column mode
if let Some(cell) = table.get_cell(0, 2) {
println!("\n Cell [0][c]: first_byte={:#04x}, len={}", cell[0], cell.len());
}
println!("\n--- Switch Back to Row Mode ---");
table.switch_mode(StorageMode::Row);
// Verify data integrity after round-trip
if let Some(row_ref) = table.get_row(2) {
let r = Record::ref_from_bytes(row_ref.as_bytes()).unwrap();
println!(" Row 2 after round-trip: a={}, b={}, c[0]={:#04x}", r.a, r.b, r.c[0]);
}
println!("\nDone. All data accessed with zero per-row allocations.");
}What does this code do?
The demo exercises the full lifecycle. table.insert(rec.as_bytes()) serializes the struct to bytes with zero allocation (zerocopy's IntoBytes). Record::ref_from_bytes(row_ref.as_bytes()) casts the raw byte slice back to a typed &Record reference pointing directly into the storage buffer.
After switching to Column mode, get_column(0) returns a ColumnRef::Borrowed because column 0's data is contiguous in the column store. The final switch back to Row mode and the integrity check proves that data survives the Row-to-Column-to-Row round-trip without corruption.
- Save src/main.rs.
- Run the program by executing:
cargo runYou should see output showing the schema size, row-mode reads with correct values (a=42, a=7, a=99), the column scan in Column mode, and the round-trip verification printing Row 2 with a=99, b=300, c[0]=0xef.
Seeing a runtime panic or wrong values?
If ref_from_bytes returns an error, check that the assert_eq!(schema.row_size(), std::mem::size_of::<Record>()) line passes. A mismatch means the schema and struct have different sizes, which makes zero-copy casts impossible.
If you see import errors like "unresolved import", make sure the first line reads use zerodb::table::{StorageMode, Table} and that StorageMode and Table are marked pub in src/table.rs.
help me debug my zerodb Table API output
✔️ Awesome, I've got everything!
Great. Make sure both src/table.rs and src/main.rs are saved.
ⓧ I'd like to double check the full code
Here is the complete src/main.rs for comparison:
use zerocopy::{FromBytes, IntoBytes, KnownLayout, Immutable};
use zerodb::table::{StorageMode, Table};
use zerodb::types::{ColumnDef, Schema, TypeId};
#[derive(Debug, Clone, Copy, FromBytes, IntoBytes, KnownLayout, Immutable)]
#[repr(C)]
struct Record {
a: u32,
b: u32,
c: [u8; 32],
}
fn main() {
let schema = Schema::new(vec![
ColumnDef::new("a", TypeId::U32),
ColumnDef::new("b", TypeId::U32),
ColumnDef::new("c", TypeId::FixedBytes(32)),
]);
assert_eq!(schema.row_size(), std::mem::size_of::<Record>());
println!("Schema row_size: {} bytes", schema.row_size());
let mut table = Table::new("records", schema, StorageMode::Row);
// Insert records using zerocopy - zero-allocation serialization
let records = [
Record { a: 42, b: 100, c: [0xAB; 32] },
Record { a: 7, b: 200, c: [0xCD; 32] },
Record { a: 99, b: 300, c: [0xEF; 32] },
];
for rec in &records {
table.insert(rec.as_bytes()).unwrap();
}
println!("\n--- Row Mode: Zero-Copy Struct Access ---");
for i in 0..table.row_count() {
if let Some(row_ref) = table.get_row(i) {
let r = Record::ref_from_bytes(row_ref.as_bytes()).unwrap();
println!(" Row {}: a={}, b={}, c[0]={:#04x}", i, r.a, r.b, r.c[0]);
}
}
// Single cell access
if let Some(cell) = table.get_cell(1, 0) {
let val = u32::from_ne_bytes(cell.try_into().unwrap());
println!("\n Cell [1][a] = {}", val);
}
println!("\n--- Switching to Column Mode ---");
table.switch_mode(StorageMode::Column);
// Column scan - contiguous memory, cache-friendly
if let Some(col) = table.get_column(0) {
println!(" Column 'a' (contiguous {} bytes):", col.as_bytes().len());
for (i, chunk) in col.as_bytes().chunks_exact(4).enumerate() {
let val = u32::from_ne_bytes(chunk.try_into().unwrap());
println!(" row {}: {}", i, val);
}
}
// Cell access still works in column mode
if let Some(cell) = table.get_cell(0, 2) {
println!("\n Cell [0][c]: first_byte={:#04x}, len={}", cell[0], cell.len());
}
println!("\n--- Switch Back to Row Mode ---");
table.switch_mode(StorageMode::Row);
// Verify data integrity after round-trip
if let Some(row_ref) = table.get_row(2) {
let r = Record::ref_from_bytes(row_ref.as_bytes()).unwrap();
println!(" Row 2 after round-trip: a={}, b={}, c[0]={:#04x}", r.a, r.b, r.c[0]);
}
println!("\nDone. All data accessed with zero per-row allocations.");
}Secret mission
Selection Vectors for Zero-Copy Filtering
Your storage engine can insert and scan, but real query engines never copy data just to filter it. DuckDB and DataFusion use selection vectors to express 'which rows matched?' as a tiny list of indices, keeping the actual data untouched in place. Can you add this pattern to zerodb?
Clean Up Your Resources
Clean Up Your Resources
This project runs entirely locally with no cloud resources or paid services. There are no ongoing costs to worry about.
Decide whether to keep your project files for future extensions or delete them entirely.
Resources you used:
- The zerodb project folder on your Desktop (contains all source code and compiled artifacts).
- Cargo's downloaded crate cache for the zerocopy dependency (stored in ~/.cargo/registry).
Keep everything running
No action needed. Choose this if you plan to extend the storage engine with file persistence, a write-ahead log, or buffer pool pages.
- Your zerodb folder stays on your Desktop.
- Cargo's crate cache remains in ~/.cargo/registry for faster future builds.
Pause - I'll come back to this later
Since this is a local-only project with no running processes, there is nothing to pause. Your files sit on disk and consume no CPU or memory when idle.
- Optionally remove the compiled artifacts to free disk space by running:
- Your source code remains intact and ready for cargo build when you return.
cd ~/Desktop/zerodb
cargo cleanWhat does cargo clean do?
This removes the target/ directory containing compiled artifacts. Your source code and Cargo.toml are untouched.
Delete - I don't want to use this again
Remove all project files and free up disk space completely.
- Delete the project folder by running:
rm -rf ~/Desktop/zerodb- Optionally clear the Cargo crate cache for zerocopy by running:
cargo cache -r zerocopy 2>/dev/null || echo "Cache cleared with project deletion"Is the crate cache worth clearing?
Cargo's global cache is shared across all your Rust projects. If you use zerocopy in other projects, keep the cache. If this was your only Rust project, deleting the project folder is enough. The cache is small and Cargo manages it automatically.
That's a Wrap!
That's a Wrap!
Nice work. You just built a dual-layout in-memory storage engine from scratch in Rust, complete with zero-copy struct access and runtime layout switching.
You've learned how to:
- Build a schema-driven type system that precomputes byte offsets at construction time, enabling O(1) field access into raw byte buffers without per-row parsing.
- Implement row-oriented and column-oriented storage backed by 8-byte-aligned buffers, with zero-copy struct access via zerocopy::FromBytes that returns references directly into the storage buffer.
- Create a unified Table API that switches between row and column layouts at runtime, physically reorganizing data in O(n) to optimize for different access patterns.
- Secret Mission: Implemented selection vectors for zero-copy filtered scans, the same pattern used by DuckDB and DataFusion to avoid materializing intermediate results.
Ready to quiz yourself?