Build a Constrained LLM File REPL

Introduction

30 Second Summary

Plain-English computer requests feel convenient until a model produces malformed output or reaches beyond the folder you meant to change. Safe local automation needs firm boundaries before any filesystem action happens.

In this project, you will build a terminal REPL in Rust that converts plain-English requests into filesystem commands inside sandbox/. Candle runs Phi-3 locally while llguidance removes tokens that would violate the JSON Schema.

What You'll Build

Your finished REPL turns a request like create a file called notes.txt into visible JSON plus a safely executed change inside sandbox/.

By the end of this project, you'll have:

• A repeatable mask lab that proves raw argmax can prefer a forbidden token while constrained selection chooses an allowed one.
• A CPU-only inference run that loads the quantized model locally and prints a complete schema-conforming JSON command.
• An interactive sandboxed filesystem REPL that lists directories, creates files, changes folders, and prints text without invoking a shell.
• Secret Mission: Add a --diagnostics switch that reveals the allowed-token count plus the selected token at every decoding step.

Are there any prerequisites?

You should be comfortable writing Rust on Linux with Rust 1.87 or newer already installed.

Internet access is required because the first real-model run downloads a 2.39 GB GGUF file.

Before We Start

This first checkpoint locks in the inference-time constraint behind the filesystem command router you are building: llguidance excludes JSON-invalid tokens before Candle selects one. Generate-validate-retry allows invalid output to exist before rejection, repair, or another attempt.

Set Up the Rust Project

The decoder eventually connects Candle token logits to llguidance masks. Mismatched crate releases can hide that mechanism behind compiler errors.

Cargo gives the project one reproducible dependency graph. You will create a Rust 2024 binary in Visual Studio Code. Cargo will then verify the pinned build.

In this step, get ready to:

• Create the schema-fs-repl file structure in Visual Studio Code.
• Pin the Rust 2024 package metadata plus dependencies.
• Confirm the project with cargo check.

Create the project structure

A Cargo binary keeps its manifest beside the src directory. Empty module files establish the boundaries that later steps fill with decoding logic.

• Use your Linux application launcher to start Visual Studio Code.
• Use the folder-opening control on the welcome screen to display your Desktop.
• Use the folder-creation control in the chooser to create schema-fs-repl.
• Select schema-fs-repl as the folder that Visual Studio Code opens.

You will see schema-fs-repl as the project root in the file sidebar.

• Use the folder-creation control in the file sidebar to create src inside schema-fs-repl.
• Confirm that src appears beneath the project root.

• Use the file-creation control in the file sidebar to create Cargo.toml directly inside schema-fs-repl.
• Use the file-creation control in the file sidebar to create src/main.rs inside src.

The file sidebar now shows Cargo.toml beside src. It also shows main.rs inside src.

• Use the file-creation control to create src/decoder.rs.
• Use the file-creation control to create src/model.rs.

You will see decoder.rs plus model.rs inside src.

• Use the file-creation control to create src/sandbox.rs.
• Use the file-creation control to create src/schema.rs.

Good. The workspace now has the exact file boundaries used by the decoder, model loader, sandbox executor, and command schema.

Pin the manifest and module map

The manifest locks every crate to the release used by this project. The module declarations connect each source file to the binary crate.

• Select Cargo.toml in the file sidebar.
• Replace its contents with the pinned manifest below:

[package]
name = "schema-fs-repl"
version = "0.1.0"
edition = "2024"
rust-version = "1.87"
publish = false

[dependencies]
anyhow = "=1.0.95"
candle = { package = "candle-core", version = "=0.11.0" }
candle-transformers = "=0.11.0"
hf-hub = "=0.5.0"
llguidance = "=1.8.0"
serde = { version = "=1.0.217", features = ["derive"] }
serde_json = { version = "=1.0.138", features = ["preserve_order"] }
toktrie_hf_downloader = "=1.8.0"

What Does This Manifest Do?

• The [package] section defines the binary's identity plus compiler requirements.
• The exact dependency pins keep the decoding crates on the releases used by the complete project.
• The publish = false setting keeps this learning project out of a package registry.

• Save Cargo.toml.
• Confirm that the editor shows no unsaved indicator beside Cargo.toml.

Seeing a TOML Problem?

• Confirm that Cargo.toml sits directly inside schema-fs-repl.
• Compare every quote plus brace against the manifest above.
• Keep the leading equals sign in every exact version pin.

Help me compare my Cargo manifest with the required pinned version.

Rust compiles a source module after the crate declares it. The four declarations in main.rs make the empty module files part of this binary.

• Select src/main.rs in the file sidebar.
• Replace its contents with the module declarations below:

mod decoder;
mod model;
mod sandbox;
mod schema;

fn main() {}

How Are the Modules Connected?

Each mod declaration connects a matching file inside src to the binary. The empty main() function gives Cargo a valid program to check.

• Save src/main.rs.
• Confirm that all four module declarations remain visible above main().

Seeing a Missing Module?

• Confirm that every module file sits directly inside src.
• Compare each file name with its matching mod declaration.
• Remove any accidental file extension added after .rs.

Help me match my Rust module declarations to the files inside src.

Check the pinned build

The first dependency check can take several minutes while Cargo resolves the pinned crates. A quiet stretch during compilation is normal.

• Use Visual Studio Code's top menu to create an integrated terminal.

Before you run the check, do you expect the empty module files to stop Cargo from checking the crate?

• Check the project from the schema-fs-repl root by running:

cargo check

What Does This Command Check?

Cargo checks the package plus its dependencies without producing the final executable. This catches dependency conflicts and source-layout mistakes before the decoding code arrives.

Cargo finishes checking schema-fs-repl without an error. That is the setup hurdle cleared.

Did Cargo Stop With an Error?

• Confirm that the terminal is inside the schema-fs-repl project root.
• Confirm that your device has internet access for the crate download.
• Compare Cargo.toml plus src/main.rs with the complete files below.

Help me diagnose why cargo check fails in my pinned Rust project.

✔️ Awesome, I've got everything!

Your pinned project now passes Cargo's dependency check. Save every open file before moving on.

ⓧ I'd like to double check the full code

Compare your complete Cargo.toml with this reference.

[package]
name = "schema-fs-repl"
version = "0.1.0"
edition = "2024"
rust-version = "1.87"
publish = false

[dependencies]
anyhow = "=1.0.95"
candle = { package = "candle-core", version = "=0.11.0" }
candle-transformers = "=0.11.0"
hf-hub = "=0.5.0"
llguidance = "=1.8.0"
serde = { version = "=1.0.217", features = ["derive"] }
serde_json = { version = "=1.0.138", features = ["preserve_order"] }
toktrie_hf_downloader = "=1.8.0"

This manifest contains the complete package metadata plus every dependency required at this checkpoint.

Compare your complete src/main.rs with this reference.

mod decoder;
mod model;
mod sandbox;
mod schema;

fn main() {}

This file declares every module while keeping the binary entry point minimal.

The following module files should have zero content:

• Keep src/decoder.rs empty.
• Keep src/model.rs empty.
• Keep src/sandbox.rs empty.
• Keep src/schema.rs empty.

Your pinned Rust project now checks cleanly. Next, you will make unconstrained argmax choose a token that the schema forbids.

Make Unconstrained Argmax Fail

Your Rust project now resolves every pinned dependency. That stable foundation lets you focus on the decoding behavior instead of dependency drift.

A model assigns a score to every possible token. Raw argmax chooses the highest score even when that token violates the JSON Schema for your command router. In this step, you will make that failure visible.

In this step, get ready to:

• Define the JSON command schema with matching Rust variants.
• Compute the first grammar mask with the Phi-3 tokenizer.
• Compare raw argmax with mask-filtered argmax.

Define the command contract

The command contract limits model output to five action shapes. Each anyOf branch has a distinct action value. The matching Rust enum gives each valid object a typed destination.

• Select src/schema.rs in the VS Code Explorer sidebar.
• Add the schema imports and TOOL_SCHEMA by pasting this code:

use anyhow::Result;
use llguidance::{api::TopLevelGrammar, toktrie::TokEnv, Matcher, ParserFactory};
use serde::Deserialize;

pub const TOOL_SCHEMA: &str = r#"
{
  "x-guidance": { "whitespace_flexible": false },
  "anyOf": [
    {"type":"object","properties":{"action":{"const":"text"},"message":{"type":"string","maxLength":200}},"required":["action","message"],"additionalProperties":false},
    {"type":"object","properties":{"action":{"const":"list_directory"},"path":{"type":"string","minLength":1,"maxLength":80}},"required":["action","path"],"additionalProperties":false},
    {"type":"object","properties":{"action":{"const":"change_directory"},"path":{"type":"string","minLength":1,"maxLength":80}},"required":["action","path"],"additionalProperties":false},
    {"type":"object","properties":{"action":{"const":"create_directory"},"path":{"type":"string","minLength":1,"maxLength":80}},"required":["action","path"],"additionalProperties":false},
    {"type":"object","properties":{"action":{"const":"create_file"},"path":{"type":"string","minLength":1,"maxLength":80},"content":{"type":"string","maxLength":200}},"required":["action","path","content"],"additionalProperties":false}
  ]
}
"#;

What does this schema control?

• The action constants distinguish the five command shapes.
• The required arrays prevent incomplete command objects.
• The additionalProperties setting blocks unexpected fields.
• The whitespace_flexible setting keeps generated JSON compact.

• Save src/schema.rs.
• Check the schema declaration by running this command in the VS Code terminal:

cargo check

What does this check prove?

Cargo checks the new imports and schema declaration against the pinned dependencies. A successful finish proves the schema text is valid Rust source.

Seeing a schema compile error?

Compare the raw string markers around the schema. The opening marker must match the closing marker exactly.

Check every double quote inside the JSON object. Raw Rust strings keep those quotes unchanged.

Ask for help with the schema syntax.

The schema describes valid JSON. Serde provides the matching Rust type that receives a completed object.

• Append the Command enum below TOOL_SCHEMA by pasting this code:

#[derive(Debug, Deserialize)]
#[serde(tag = "action", rename_all = "snake_case", deny_unknown_fields)]
pub enum Command {
    Text { message: String },
    ListDirectory { path: String },
    ChangeDirectory { path: String },
    CreateDirectory { path: String },
    CreateFile { path: String, content: String },
}

How does the enum match the schema?

• The internal action tag selects a Rust variant.
• The snake case rule maps list_directory to ListDirectory.
• The unknown-field rule preserves the schema boundary during deserialization.

• Save src/schema.rs.
• Check the typed command definition by running:

cargo check

What should you see?

Cargo finishes checking schema-fs-repl without an error. Warnings about unused code are expected because the decoder does not use the enum yet.

Enum attributes not compiling?

Check that both attributes sit directly above pub enum Command.

Confirm each variant ends with a comma.

Ask for help with the enum attributes.

The final schema helper turns TOOL_SCHEMA into an llguidance parser. The parser uses the same tokenizer environment that later interprets model token IDs.

• Append new_matcher below the Command enum by pasting:

pub fn new_matcher(tok_env: &TokEnv) -> Result<Matcher> {
    let schema = serde_json::from_str(TOOL_SCHEMA)?;
    let grammar = TopLevelGrammar::from_json_schema(schema);
    let mut factory = ParserFactory::new_simple(tok_env)?;
    factory.quiet();
    Ok(Matcher::new(factory.create_parser(grammar)))
}

What does the matcher contain?

• The parsed schema becomes a top-level grammar.
• The parser factory binds that grammar to the supplied TokEnv.
• The matcher tracks which token IDs remain legal as generation advances.

• Save src/schema.rs.
• Check the matcher construction by running:

cargo check

What should you see?

Cargo finishes without an error. Your project now has one command contract shared by grammar masking and Rust deserialization.

Matcher construction failing?

Check that TopLevelGrammar comes from the api module in the import at the top of src/schema.rs.

Confirm new_matcher receives a borrowed TokEnv.

Ask for help with the parser setup.

✔️ Awesome, I've got everything!

Your command schema and matcher factory are ready. Double check that you saved src/schema.rs.

ⓧ I'd like to double check the full code

Compare your file with the complete src/schema.rs reference below.

use anyhow::Result;
use llguidance::{api::TopLevelGrammar, toktrie::TokEnv, Matcher, ParserFactory};
use serde::Deserialize;

pub const TOOL_SCHEMA: &str = r#"
{
  "x-guidance": { "whitespace_flexible": false },
  "anyOf": [
    {"type":"object","properties":{"action":{"const":"text"},"message":{"type":"string","maxLength":200}},"required":["action","message"],"additionalProperties":false},
    {"type":"object","properties":{"action":{"const":"list_directory"},"path":{"type":"string","minLength":1,"maxLength":80}},"required":["action","path"],"additionalProperties":false},
    {"type":"object","properties":{"action":{"const":"change_directory"},"path":{"type":"string","minLength":1,"maxLength":80}},"required":["action","path"],"additionalProperties":false},
    {"type":"object","properties":{"action":{"const":"create_directory"},"path":{"type":"string","minLength":1,"maxLength":80}},"required":["action","path"],"additionalProperties":false},
    {"type":"object","properties":{"action":{"const":"create_file"},"path":{"type":"string","minLength":1,"maxLength":80},"content":{"type":"string","maxLength":200}},"required":["action","path","content"],"additionalProperties":false}
  ]
}
"#;

#[derive(Debug, Deserialize)]
#[serde(tag = "action", rename_all = "snake_case", deny_unknown_fields)]
pub enum Command {
    Text { message: String },
    ListDirectory { path: String },
    ChangeDirectory { path: String },
    CreateDirectory { path: String },
    CreateFile { path: String, content: String },
}

pub fn new_matcher(tok_env: &TokEnv) -> Result<Matcher> {
    let schema = serde_json::from_str(TOOL_SCHEMA)?;
    let grammar = TopLevelGrammar::from_json_schema(schema);
    let mut factory = ParserFactory::new_simple(tok_env)?;
    factory.quiet();
    Ok(Matcher::new(factory.create_parser(grammar)))
}

Expose the raw argmax failure

The Phi-3 tokenizer assigns the token IDs used by the grammar mask. Keeping tokenization and masking inside one TokEnv keeps those IDs aligned.

The lab gives one forbidden token the highest synthetic logit. Raw argmax only compares scores, so the grammar has no influence over that choice.

• Select the empty src/decoder.rs file in the VS Code Explorer sidebar.
• Add the decoder imports and raw argmax helper by pasting:

use anyhow::{ensure, Context, Result};
use candle::{Device, Tensor};
use llguidance::toktrie::{SimpleVob, TokEnv};

use crate::schema::new_matcher;

fn argmax(logits: &[f32]) -> Result<u32> {
    logits.iter().enumerate().filter(|(_, score)| !score.is_nan())
        .max_by(|left, right| left.1.total_cmp(right.1))
        .map(|(index, _)| index as u32)
        .context("logits contained no selectable value")
}

How does raw argmax choose?

The helper scans every non-NaN score. It returns the token ID with the largest value without consulting a grammar mask.

• Save src/decoder.rs.
• Check the raw selector by running:

cargo check

What should you see?

Cargo finishes checking the project. An unused-function warning is expected until the lab calls argmax.

Argmax helper not compiling?

Check the closure patterns inside filter and max_by.

Confirm the function returns Result<u32>.

Ask for help with the iterator chain.

The first mask identifies every token that can legally begin a command object. The lab dynamically finds an ID outside that set, so the demonstration does not depend on a hard-coded vocabulary position.

• Append the first version of run_mask_lab below argmax by pasting:

pub fn run_mask_lab(tok_env: &TokEnv) -> Result<()> {
    let target_json = r#"{"action":"list_directory","path":"."}"#;
    let target_tokens = tok_env.tokenize(target_json);
    ensure!(!target_tokens.is_empty(), "target JSON produced no tokens");
    let mut first_matcher = new_matcher(tok_env)?;
    let first_mask = first_matcher.compute_mask()?;
    let first_target = target_tokens[0];
    ensure!(first_mask.is_allowed(first_target), "target token is not schema-valid");
    let vocab_size = tok_env.tok_trie().vocab_size();
    let forbidden = (0..vocab_size as u32).find(|token| !first_mask.is_allowed(*token))
        .context("could not find a forbidden token for the demonstration")?;
    let mut synthetic = vec![-100.0_f32; vocab_size];
    synthetic[first_target as usize] = 10.0;
    synthetic[forbidden as usize] = 100.0;
    let tensor = Tensor::from_vec(synthetic, vocab_size, &Device::Cpu)?;
    let logits = tensor.to_vec1::<f32>()?;
    let raw = argmax(&logits)?;
    println!("Raw argmax: {} allowed={}", tok_env.tok_trie().token_dbg(raw), first_mask.is_allowed(raw));
    Ok(())
}

How is the failure constructed?

• The known-good JSON supplies a legal first token.
• The first mask supplies the legal token set.
• The dynamically selected forbidden token receives a logit of 100.0.
• The legal target receives a lower logit of 10.0.

• Save src/decoder.rs.
• Check the lab function by running:

cargo check

What should you see?

Cargo finishes without an error. The lab can now construct a mask and a synthetic Candle tensor.

Mask lab not compiling?

Confirm new_matcher is public in src/schema.rs.

Check that the synthetic vector length uses vocab_size.

Ask for help with the lab setup.

The binary entry point loads the tokenizer environment before calling the lab. That same environment supplies tokenization, masking, and debug decoding.

• Select src/main.rs in the VS Code Explorer sidebar.
• Replace its current contents with this lab entry point:

mod decoder;
mod model;
mod sandbox;
mod schema;

use anyhow::Result;
use decoder::run_mask_lab;
use llguidance::toktrie::TokEnv;
use model::TOKENIZER_REPO;

fn main() -> Result<()> {
    let tok_env: TokEnv = toktrie_hf_downloader::tok_env_from_name(TOKENIZER_REPO)?;
    run_mask_lab(&tok_env)
}

What does the entry point do?

The downloader resolves the tokenizer named by TOKENIZER_REPO and constructs a TokEnv. The entry point passes that environment into the mask lab.

• Save src/main.rs.

Before you run the lab, which token do you think raw argmax chooses when a forbidden ID has the highest score?

• Run the falling-short selector with this command:

cargo run --release -- lab

What should you see?

The first release build may take several minutes while Cargo compiles the dependencies. The tokenizer files may also download during this run.

The output prints a raw argmax token followed by allowed=false. Raw argmax has selected the highest logit even though the schema forbids that token.

Lab does not reach raw argmax?

Confirm your device has internet access for the tokenizer download.

Run the command from the schema-fs-repl project root.

Ask for help with the first lab run.

✔️ Awesome, I've got everything!

Your entry point runs the tokenizer-backed lab. Double check that you saved src/main.rs.

ⓧ I'd like to double check the full code

Compare your file with the complete src/main.rs reference below.

mod decoder;
mod model;
mod sandbox;
mod schema;

use anyhow::Result;
use decoder::run_mask_lab;
use llguidance::toktrie::TokEnv;
use model::TOKENIZER_REPO;

fn main() -> Result<()> {
    let tok_env: TokEnv = toktrie_hf_downloader::tok_env_from_name(TOKENIZER_REPO)?;
    run_mask_lab(&tok_env)
}

Filter selection through the mask

You have now seen the intended shortfall. A constrained selector only compares logits whose mask bits are set, so a forbidden token never enters the competition.

• Return to src/decoder.rs in the VS Code editor.
• Insert constrained_argmax between argmax and run_mask_lab by pasting:

fn constrained_argmax(logits: &[f32], mask: &SimpleVob) -> Result<u32> {
    ensure!(logits.len() >= mask.len(), "logit vocabulary is smaller than the llguidance mask");
    let mut best: Option<(u32, f32)> = None;
    mask.iter_set_entries(|index| {
        let score = logits[index];
        if !score.is_nan() && best.is_none_or(|(_, best_score)| score > best_score) {
            best = Some((index as u32, score));
        }
    });
    best.map(|(token, _)| token).context("the grammar allowed no selectable token")
}

How does constrained argmax differ?

• The vocabulary-size check prevents indexing beyond the logits vector.
• The set-entry iterator visits only token IDs permitted by the mask.
• The highest permitted score becomes the selected token.

• Save src/decoder.rs.
• Check the constrained selector by running:

cargo check

What should you see?

Cargo finishes without an error. The selector is ready to receive the first grammar mask.

Constrained selector not compiling?

Confirm SimpleVob appears in the llguidance tokenizer import.

Check that the closure receives index from iter_set_entries.

Ask for help with mask iteration.

The final lab runs both selectors against the same logits. That shared input isolates the mask as the only difference between their choices.

• Replace the current run_mask_lab function with this final version:

pub fn run_mask_lab(tok_env: &TokEnv) -> Result<()> {
    let target_json = r#"{"action":"list_directory","path":"."}"#;
    let target_tokens = tok_env.tokenize(target_json);
    ensure!(!target_tokens.is_empty(), "target JSON produced no tokens");
    let mut first_matcher = new_matcher(tok_env)?;
    let first_mask = first_matcher.compute_mask()?;
    let first_target = target_tokens[0];
    ensure!(first_mask.is_allowed(first_target), "target token is not schema-valid");
    let vocab_size = tok_env.tok_trie().vocab_size();
    let forbidden = (0..vocab_size as u32).find(|token| !first_mask.is_allowed(*token))
        .context("could not find a forbidden token for the demonstration")?;
    let mut synthetic = vec![-100.0_f32; vocab_size];
    synthetic[first_target as usize] = 10.0;
    synthetic[forbidden as usize] = 100.0;
    let tensor = Tensor::from_vec(synthetic, vocab_size, &Device::Cpu)?;
    let logits = tensor.to_vec1::<f32>()?;
    let raw = argmax(&logits)?;
    let constrained = constrained_argmax(&logits, &first_mask)?;
    println!("Raw argmax: {} allowed={}", tok_env.tok_trie().token_dbg(raw), first_mask.is_allowed(raw));
    println!("Constrained argmax: {} allowed={}", tok_env.tok_trie().token_dbg(constrained), first_mask.is_allowed(constrained));
    Ok(())
}

What changed in the lab?

The lab now passes the same logits through constrained_argmax. It prints the mask status of both selected token IDs for a direct comparison.

• Save src/decoder.rs.

Before you rerun the lab, do you expect the forbidden token to survive mask-filtered selection?

• Run the completed comparison with this command:

cargo run --release -- lab

What should you see?

The raw argmax line ends with allowed=false. The constrained argmax line ends with allowed=true.

You have caught the failure at its source. The mask changes token selection before an invalid token can enter the parser.

Both selectors show the same status?

Check that the raw selector receives the complete logits vector.

Confirm the constrained selector receives first_mask.

Ask for help comparing the two paths.

✔️ Awesome, I've got everything!

Your lab now demonstrates both selection paths. Double check that you saved src/decoder.rs.

ⓧ I'd like to double check the full code

Compare your file with the complete src/decoder.rs reference below.

use anyhow::{ensure, Context, Result};
use candle::{Device, Tensor};
use llguidance::toktrie::{SimpleVob, TokEnv};

use crate::schema::new_matcher;

fn argmax(logits: &[f32]) -> Result<u32> {
    logits.iter().enumerate().filter(|(_, score)| !score.is_nan())
        .max_by(|left, right| left.1.total_cmp(right.1))
        .map(|(index, _)| index as u32)
        .context("logits contained no selectable value")
}

fn constrained_argmax(logits: &[f32], mask: &SimpleVob) -> Result<u32> {
    ensure!(logits.len() >= mask.len(), "logit vocabulary is smaller than the llguidance mask");
    let mut best: Option<(u32, f32)> = None;
    mask.iter_set_entries(|index| {
        let score = logits[index];
        if !score.is_nan() && best.is_none_or(|(_, best_score)| score > best_score) {
            best = Some((index as u32, score));
        }
    });
    best.map(|(token, _)| token).context("the grammar allowed no selectable token")
}

pub fn run_mask_lab(tok_env: &TokEnv) -> Result<()> {
    let target_json = r#"{"action":"list_directory","path":"."}"#;
    let target_tokens = tok_env.tokenize(target_json);
    ensure!(!target_tokens.is_empty(), "target JSON produced no tokens");
    let mut first_matcher = new_matcher(tok_env)?;
    let first_mask = first_matcher.compute_mask()?;
    let first_target = target_tokens[0];
    ensure!(first_mask.is_allowed(first_target), "target token is not schema-valid");
    let vocab_size = tok_env.tok_trie().vocab_size();
    let forbidden = (0..vocab_size as u32).find(|token| !first_mask.is_allowed(*token))
        .context("could not find a forbidden token for the demonstration")?;
    let mut synthetic = vec![-100.0_f32; vocab_size];
    synthetic[first_target as usize] = 10.0;
    synthetic[forbidden as usize] = 100.0;
    let tensor = Tensor::from_vec(synthetic, vocab_size, &Device::Cpu)?;
    let logits = tensor.to_vec1::<f32>()?;
    let raw = argmax(&logits)?;
    let constrained = constrained_argmax(&logits, &first_mask)?;
    println!("Raw argmax: {} allowed={}", tok_env.tok_trie().token_dbg(raw), first_mask.is_allowed(raw));
    println!("Constrained argmax: {} allowed={}", tok_env.tok_trie().token_dbg(constrained), first_mask.is_allowed(constrained));
    Ok(())
}

Your lab now proves that unrestricted logits can prefer an invalid token. Next, you will repeat masked selection token by token until the grammar accepts a complete JSON command.

Apply the llguidance Token Mask

Your mask lab already exposed the failure: raw argmax can favor a token that the schema forbids. That proof gives you a clear target for the complete decoding loop.

Now llguidance must protect every decoding step. Each parser transition produces a new SimpleVob mask for the next token.

In this step, get ready to:

• Confirm that constrained argmax compares only schema-allowed logits.
• Advance the matcher after every selected token.
• Decode the accepted token IDs into a typed command.

Inspect the allowed-token boundary

Your existing constrained_argmax() function is the boundary between unrestricted logits and schema-safe selection. Its search visits only token indexes whose mask bits are set.

• Switch back to src/decoder.rs in VS Code.
• Locate the existing constrained_argmax() function shown below.
• Trace the callback passed to mask.iter_set_entries().

fn constrained_argmax(logits: &[f32], mask: &SimpleVob) -> Result<u32> {
    ensure!(logits.len() >= mask.len(), "logit vocabulary is smaller than the llguidance mask");
    let mut best: Option<(u32, f32)> = None;
    mask.iter_set_entries(|index| {
        let score = logits[index];
        if !score.is_nan() && best.is_none_or(|(_, best_score)| score > best_score) {
            best = Some((index as u32, score));
        }
    });
    best.map(|(token, _)| token).context("the grammar allowed no selectable token")
}

How Does Constrained Argmax Work?

A set bit in SimpleVob means the grammar permits that token in the current parser state. The callback never receives a forbidden token index.

The highest logit wins within this permitted set. A forbidden token cannot win even when its raw score is larger.

Build the token-by-token loop

The tokenizer repository gives every stage the same vocabulary. This keeps token IDs aligned from masking through decoding.

• Switch back to the empty src/model.rs file from earlier.
• Add the shared tokenizer repository constant by copying this line:

pub const TOKENIZER_REPO: &str = "microsoft/Phi-3-mini-4k-instruct";

Why Use One Tokenizer Source?

The Phi-3 tokenizer defines the token IDs used by the matcher. The same source also reconstructs text from selected IDs.

The TOKENIZER_REPO constant prevents different application modes from choosing mismatched vocabularies.

• Save src/model.rs.
• Confirm that the shared tokenizer source resolves by running:

cargo check

What Should I See?

Cargo finishes checking schema-fs-repl without an error. This confirms that src/main.rs can import TOKENIZER_REPO from src/model.rs.

Tokenizer Constant Not Found?

Confirm that the constant is inside src/model.rs. Check that the name is exactly TOKENIZER_REPO.

Save the file before running Cargo again. Unsaved code remains invisible to the compiler.

Help me debug the missing tokenizer constant.

A single mask protects one token choice. The complete loop recomputes the mask after each parser transition.

• Return to src/decoder.rs.
• Add Command beside new_matcher in the schema import so it matches this line:

use crate::schema::{new_matcher, Command};

Why Import Command?

The completed loop produces a JSON string first. The Command type gives Serde the Rust enum that the finished string must match.

• Find the existing println! call that reports the constrained argmax result.
• Insert the complete decoding loop directly below that call by copying this code:

    let mut matcher = new_matcher(tok_env)?;
    let mut generated = Vec::new();
    for target in target_tokens {
        let mask = matcher.compute_mask()?;
        ensure!(mask.is_allowed(target), "known-good token was rejected");
        let mut values = vec![-100.0_f32; vocab_size];
        values[target as usize] = 10.0;
        let tensor = Tensor::from_vec(values, vocab_size, &Device::Cpu)?;
        let selected = constrained_argmax(&tensor.to_vec1::<f32>()?, &mask)?;
        matcher.consume_token(selected)?;
        generated.push(selected);
    }
    ensure!(matcher.is_accepting()?, "grammar did not reach an accepting state");
    let json = tok_env.tok_trie().decode_str(&generated);
    let command: Command = serde_json::from_str(&json)?;
    println!("Schema-conforming JSON: {json}");
    println!("Parsed command: {command:?}");
    println!("Grammar accepting: true");

How Does the Loop Stay Valid?

• The new matcher starts at the beginning of the command grammar.
• Each compute_mask() call returns the legal tokens for the current parser state.
• The synthetic values make the next known-good token the strongest legal choice.
• The consume_token() call advances parser state before the next mask is computed.
• The generated vector stores each selected token ID in order.
• The decode_str() method reconstructs the compact JSON command.
• The serde_json::from_str() call converts the completed JSON into Command.

• Save src/decoder.rs.

Before you run the completed decoder, do you expect the matcher to accept after every known-good token has been consumed?

• Run the completed mask loop by using the lab command:

cargo run --release -- lab

What Should I See?

You should see Schema-conforming JSON: {"action":"list_directory","path":"."} after the earlier argmax comparison.

The parsed command line confirms that the completed JSON maps to the tagged Rust enum. The final line reports Grammar accepting: true.

You have closed the mask loop. Every selected token is legal at the parser state where it is chosen.

Missing the Completed JSON?

If compilation points to Command, confirm that the schema import includes it beside new_matcher.

If the matcher rejects a known-good token, check that compute_mask() runs before constrained_argmax() for every loop iteration.

Help me debug the constrained decoding order.

Expose and verify lab mode

An explicit mode keeps this deterministic experiment separate from the real-model modes added later. The entry point also rejects unknown modes instead of silently running the wrong path.

• Switch back to src/main.rs.
• Replace its contents with this mode-aware entry point:

mod decoder;
mod model;
mod sandbox;
mod schema;

use anyhow::{bail, Result};
use decoder::run_mask_lab;
use llguidance::toktrie::TokEnv;
use model::TOKENIZER_REPO;

fn main() -> Result<()> {
    let args: Vec<String> = std::env::args().collect();
    let mode = args.get(1).map(String::as_str).unwrap_or("lab");
    let tok_env: TokEnv = toktrie_hf_downloader::tok_env_from_name(TOKENIZER_REPO)?;
    match mode {
        "lab" => run_mask_lab(&tok_env),
        other => bail!("unknown mode {other:?}; use lab"),
    }
}

What Does the Entry Point Do?

• The args vector captures the mode supplied after Cargo's argument separator.
• The mode value defaults to lab when no mode is supplied.
• The match expression sends the supported mode to run_mask_lab().
• The remaining branch reports unsupported mode names.

• Save src/main.rs.

Before the final run, which output lines would prove that masking, parsing, and typed deserialization all succeeded?

• Verify the explicit lab mode by running:

cargo run --release -- lab

What Should I See?

The output identifies a raw argmax token with allowed=false. It identifies the constrained token with allowed=true.

You should then see Schema-conforming JSON: {"action":"list_directory","path":"."}, a parsed Command variant, and Grammar accepting: true.

Lab Mode Does Not Compile?

Confirm that src/main.rs imports bail beside Result.

Check that the supported mode is spelled lab in the match expression.

Help me debug the lab-mode entry point.

The reference below shows the complete project state for this checkpoint.

✔️ Awesome, I've got everything!

Great work. Double check that src/decoder.rs, src/main.rs, and src/model.rs are saved.

ⓧ I'd like to double check the full code

Compare each file with these complete checkpoint versions.

[package]
name = "schema-fs-repl"
version = "0.1.0"
edition = "2024"
rust-version = "1.87"
publish = false

[dependencies]
anyhow = "=1.0.95"
candle = { package = "candle-core", version = "=0.11.0" }
candle-transformers = "=0.11.0"
hf-hub = "=0.5.0"
llguidance = "=1.8.0"
serde = { version = "=1.0.217", features = ["derive"] }
serde_json = { version = "=1.0.138", features = ["preserve_order"] }
toktrie_hf_downloader = "=1.8.0"

What to Compare in Cargo.toml

The package still uses Rust edition 2024 with Rust 1.87. Every dependency remains pinned to the checkpoint version.

use anyhow::{ensure, Context, Result};
use candle::{Device, Tensor};
use llguidance::toktrie::{SimpleVob, TokEnv};

use crate::schema::{new_matcher, Command};

fn argmax(logits: &[f32]) -> Result<u32> {
    logits.iter().enumerate().filter(|(_, score)| !score.is_nan())
        .max_by(|left, right| left.1.total_cmp(right.1))
        .map(|(index, _)| index as u32)
        .context("logits contained no selectable value")
}

fn constrained_argmax(logits: &[f32], mask: &SimpleVob) -> Result<u32> {
    ensure!(logits.len() >= mask.len(), "logit vocabulary is smaller than the llguidance mask");
    let mut best: Option<(u32, f32)> = None;
    mask.iter_set_entries(|index| {
        let score = logits[index];
        if !score.is_nan() && best.is_none_or(|(_, best_score)| score > best_score) {
            best = Some((index as u32, score));
        }
    });
    best.map(|(token, _)| token).context("the grammar allowed no selectable token")
}

pub fn run_mask_lab(tok_env: &TokEnv) -> Result<()> {
    let target_json = r#"{"action":"list_directory","path":"."}"#;
    let target_tokens = tok_env.tokenize(target_json);
    ensure!(!target_tokens.is_empty(), "target JSON produced no tokens");
    let mut first_matcher = new_matcher(tok_env)?;
    let first_mask = first_matcher.compute_mask()?;
    let first_target = target_tokens[0];
    ensure!(first_mask.is_allowed(first_target), "target token is not schema-valid");
    let vocab_size = tok_env.tok_trie().vocab_size();
    let forbidden = (0..vocab_size as u32).find(|token| !first_mask.is_allowed(*token))
        .context("could not find a forbidden token for the demonstration")?;
    let mut synthetic = vec![-100.0_f32; vocab_size];
    synthetic[first_target as usize] = 10.0;
    synthetic[forbidden as usize] = 100.0;
    let tensor = Tensor::from_vec(synthetic, vocab_size, &Device::Cpu)?;
    let logits = tensor.to_vec1::<f32>()?;
    let raw = argmax(&logits)?;
    let constrained = constrained_argmax(&logits, &first_mask)?;
    println!("Raw argmax: {} allowed={}", tok_env.tok_trie().token_dbg(raw), first_mask.is_allowed(raw));
    println!("Constrained argmax: {} allowed={}", tok_env.tok_trie().token_dbg(constrained), first_mask.is_allowed(constrained));

    let mut matcher = new_matcher(tok_env)?;
    let mut generated = Vec::new();
    for target in target_tokens {
        let mask = matcher.compute_mask()?;
        ensure!(mask.is_allowed(target), "known-good token was rejected");
        let mut values = vec![-100.0_f32; vocab_size];
        values[target as usize] = 10.0;
        let tensor = Tensor::from_vec(values, vocab_size, &Device::Cpu)?;
        let selected = constrained_argmax(&tensor.to_vec1::<f32>()?, &mask)?;
        matcher.consume_token(selected)?;
        generated.push(selected);
    }
    ensure!(matcher.is_accepting()?, "grammar did not reach an accepting state");
    let json = tok_env.tok_trie().decode_str(&generated);
    let command: Command = serde_json::from_str(&json)?;
    println!("Schema-conforming JSON: {json}");
    println!("Parsed command: {command:?}");
    println!("Grammar accepting: true");
    Ok(())
}

What to Compare in decoder.rs

Check the Command import first. Then compare the mask loop from new_matcher() through the three final output lines.

mod decoder;
mod model;
mod sandbox;
mod schema;

use anyhow::{bail, Result};
use decoder::run_mask_lab;
use llguidance::toktrie::TokEnv;
use model::TOKENIZER_REPO;

fn main() -> Result<()> {
    let args: Vec<String> = std::env::args().collect();
    let mode = args.get(1).map(String::as_str).unwrap_or("lab");
    let tok_env: TokEnv = toktrie_hf_downloader::tok_env_from_name(TOKENIZER_REPO)?;
    match mode {
        "lab" => run_mask_lab(&tok_env),
        other => bail!("unknown mode {other:?}; use lab"),
    }
}

What to Compare in main.rs

Confirm that the entry point reads the mode before loading the tokenizer environment. The match expression must support lab.

pub const TOKENIZER_REPO: &str = "microsoft/Phi-3-mini-4k-instruct";

What to Compare in model.rs

This file contains the shared TOKENIZER_REPO constant at this checkpoint.

The src/sandbox.rs file remains empty at this checkpoint.

use anyhow::Result;
use llguidance::{api::TopLevelGrammar, toktrie::TokEnv, Matcher, ParserFactory};
use serde::Deserialize;

pub const TOOL_SCHEMA: &str = r#"
{
  "x-guidance": { "whitespace_flexible": false },
  "anyOf": [
    {"type":"object","properties":{"action":{"const":"text"},"message":{"type":"string","maxLength":200}},"required":["action","message"],"additionalProperties":false},
    {"type":"object","properties":{"action":{"const":"list_directory"},"path":{"type":"string","minLength":1,"maxLength":80}},"required":["action","path"],"additionalProperties":false},
    {"type":"object","properties":{"action":{"const":"change_directory"},"path":{"type":"string","minLength":1,"maxLength":80}},"required":["action","path"],"additionalProperties":false},
    {"type":"object","properties":{"action":{"const":"create_directory"},"path":{"type":"string","minLength":1,"maxLength":80}},"required":["action","path"],"additionalProperties":false},
    {"type":"object","properties":{"action":{"const":"create_file"},"path":{"type":"string","minLength":1,"maxLength":80},"content":{"type":"string","maxLength":200}},"required":["action","path","content"],"additionalProperties":false}
  ]
}
"#;

#[derive(Debug, Deserialize)]
#[serde(tag = "action", rename_all = "snake_case", deny_unknown_fields)]
pub enum Command {
    Text { message: String },
    ListDirectory { path: String },
    ChangeDirectory { path: String },
    CreateDirectory { path: String },
    CreateFile { path: String, content: String },
}

pub fn new_matcher(tok_env: &TokEnv) -> Result<Matcher> {
    let schema = serde_json::from_str(TOOL_SCHEMA)?;
    let grammar = TopLevelGrammar::from_json_schema(schema);
    let mut factory = ParserFactory::new_simple(tok_env)?;
    factory.quiet();
    Ok(Matcher::new(factory.create_parser(grammar)))
}

What to Compare in schema.rs

The schema and tagged enum remain unchanged. The decoder now consumes their existing new_matcher() and Command definitions.

Your synthetic decoder now proves that schema constraints control every token choice. Next, you will replace the synthetic logits with output from a local Phi-3 model.

Replace Synthetic Logits with Phi-3

Your synthetic loop has proved that Candle logits can pass through an llguidance mask before token selection. Now the same bridge needs logits from Phi-3.

A real model adds a prompt prefill that produces the first logits. Each constrained token then returns to the model at the next position.

In this step, get ready to:

• Load the quantized Phi-3 model on the CPU.
• Generate constrained commands from real model logits.
• Run one natural-language request through the complete inference path.

Load the quantized Phi-3 model

A quantized GGUF model stores compressed weights that Candle can run on your CPU. The PhiModel wrapper keeps those weights beside the device used for every tensor.

• In src/model.rs, replace the current contents with these model definitions:

use std::fs::File;

use anyhow::{Context, Result};
use candle::{quantized::gguf_file, Device, Tensor};
use candle_transformers::models::quantized_phi3::ModelWeights;
use hf_hub::{api::sync::Api, Repo, RepoType};

pub const TOKENIZER_REPO: &str = "microsoft/Phi-3-mini-4k-instruct";
const MODEL_REPO: &str = "microsoft/Phi-3-mini-4k-instruct-gguf";
const MODEL_FILE: &str = "Phi-3-mini-4k-instruct-q4.gguf";
const MODEL_REVISION: &str = "main";

pub struct PhiModel {
    inner: ModelWeights,
    device: Device,
}

What do these definitions prepare?

• The repository constants identify the tokenizer repository and the quantized model file.
• The MODEL_REVISION constant pins the download to the repository's main revision.
• The PhiModel struct holds Candle's quantized ModelWeights.
• The device field keeps tensor creation aligned with the model.

• Save src/model.rs.
• Confirm the model definitions compile by running:

cargo check

What should you see?

Cargo checks the project without reporting a compilation error. Warnings about unused model definitions are expected until the loader uses them.

Model definitions not compiling?

Check that src/model.rs contains each import exactly once. Confirm that the repository constants remain above PhiModel.

Compare the Candle import paths with the code block above. A missing quantized_phi3 segment prevents Rust from finding ModelWeights.

Help me diagnose the model definition compile failure.

• In src/model.rs, add this implementation below the PhiModel struct:

impl PhiModel {
    pub fn load() -> Result<Self> {
        println!("Resolving {MODEL_REPO}/{MODEL_FILE} from Hugging Face Hub...");
        let api = Api::new()?;
        let repo = Repo::with_revision(MODEL_REPO.to_string(), RepoType::Model, MODEL_REVISION.to_string());
        let model_path = api.repo(repo).get(MODEL_FILE)?;
        println!("Loading model from {}", model_path.display());
        let device = Device::Cpu;
        let mut file = File::open(&model_path)
            .with_context(|| format!("failed to open {}", model_path.display()))?;
        let content = gguf_file::Content::read(&mut file)
            .map_err(|error| error.with_path(&model_path))?;
        let inner = ModelWeights::from_gguf(false, content, &mut file, &device)?;
        Ok(Self { inner, device })
    }

    pub fn forward(&mut self, input: &Tensor, position: usize) -> candle::Result<Tensor> {
        self.inner.forward(input, position)
    }

    pub fn device(&self) -> &Device {
        &self.device
    }
}

How does the model loader work?

• The synchronous Hugging Face Hub client resolves the model file into its local cache.
• The GGUF reader loads the quantized file metadata and tensors.
• The ModelWeights::from_gguf() call constructs Phi-3 on Device::Cpu.
• The forward() method exposes Candle's model pass to the decoder.

• Save src/model.rs.
• Confirm the complete loader compiles by running:

cargo check

What should you see now?

Cargo checks the new loader without a compilation error. The command does not download the model because PhiModel::load() has not run.

Loader still not compiling?

Confirm the implementation sits below the closing brace of the PhiModel struct. Check that forward() returns Candle's tensor result type.

Help me fix the PhiModel loader.

✔️ Awesome, I've got everything!

Your src/model.rs file now has a CPU model loader and a forward-pass wrapper.

ⓧ I'd like to double check the full code

Compare your complete src/model.rs file with this version.

use std::fs::File;

use anyhow::{Context, Result};
use candle::{quantized::gguf_file, Device, Tensor};
use candle_transformers::models::quantized_phi3::ModelWeights;
use hf_hub::{api::sync::Api, Repo, RepoType};

pub const TOKENIZER_REPO: &str = "microsoft/Phi-3-mini-4k-instruct";
const MODEL_REPO: &str = "microsoft/Phi-3-mini-4k-instruct-gguf";
const MODEL_FILE: &str = "Phi-3-mini-4k-instruct-q4.gguf";
const MODEL_REVISION: &str = "main";

pub struct PhiModel {
    inner: ModelWeights,
    device: Device,
}

impl PhiModel {
    pub fn load() -> Result<Self> {
        println!("Resolving {MODEL_REPO}/{MODEL_FILE} from Hugging Face Hub...");
        let api = Api::new()?;
        let repo = Repo::with_revision(MODEL_REPO.to_string(), RepoType::Model, MODEL_REVISION.to_string());
        let model_path = api.repo(repo).get(MODEL_FILE)?;
        println!("Loading model from {}", model_path.display());
        let device = Device::Cpu;
        let mut file = File::open(&model_path)
            .with_context(|| format!("failed to open {}", model_path.display()))?;
        let content = gguf_file::Content::read(&mut file)
            .map_err(|error| error.with_path(&model_path))?;
        let inner = ModelWeights::from_gguf(false, content, &mut file, &device)?;
        Ok(Self { inner, device })
    }

    pub fn forward(&mut self, input: &Tensor, position: usize) -> candle::Result<Tensor> {
        self.inner.forward(input, position)
    }

    pub fn device(&self) -> &Device {
        &self.device
    }
}

What should match?

Check the repository values and the CPU device first. Confirm that all three methods remain inside the same PhiModel implementation.

Drive constrained decoding with real logits

Prompt prefill gives Candle the request and produces logits for the first generated token. The existing mask bridge then restricts each selection before the parser consumes it.

• In src/decoder.rs, replace everything above fn argmax with this updated model bridge:

use anyhow::{bail, ensure, Context, Result};
use candle::{DType, Device, Tensor};
use llguidance::toktrie::{SimpleVob, TokEnv};

use crate::{model::PhiModel, schema::{new_matcher, Command}};

const MAX_NEW_TOKENS: usize = 160;

What changes in the decoder?

• The DType import lets the decoder convert model logits into F32 values.
• The PhiModel import connects the existing mask logic to real model inference.
• The MAX_NEW_TOKENS limit prevents an unfinished generation from running forever.
• The bail import returns an error when the grammar does not finish within that limit.

• Save src/decoder.rs.
• Confirm the updated decoder imports compile by running:

cargo check

What should you see?

Cargo checks the project without a compilation error. Warnings about the unused model bridge disappear after you add the generation function.

• In src/decoder.rs, add generate_command() below run_mask_lab() by copying this code:

pub fn generate_command(model: &mut PhiModel, tok_env: &TokEnv, request: &str) -> Result<(String, Command)> {
    let prompt = format!("<|user|>\nYou are a filesystem command router. Return exactly one compact JSON object.\nAllowed shapes:\n{{\"action\":\"text\",\"message\":\"...\"}}\n{{\"action\":\"list_directory\",\"path\":\"relative path\"}}\n{{\"action\":\"change_directory\",\"path\":\"relative path\"}}\n{{\"action\":\"create_directory\",\"path\":\"relative path\"}}\n{{\"action\":\"create_file\",\"path\":\"relative path\",\"content\":\"...\"}}\nUse only relative paths and use . for the current directory.\nRequest: {request}<|end|>\n<|assistant|>\n");
    let prompt_tokens = tok_env.tokenize_special(&prompt);
    ensure!(!prompt_tokens.is_empty(), "prompt produced no tokens");
    let input = Tensor::new(prompt_tokens.as_slice(), model.device())?.unsqueeze(0)?;
    let mut logits = model.forward(&input, 0)?.squeeze(0)?.to_dtype(DType::F32)?;
    let mut matcher = new_matcher(tok_env)?;
    let mut generated = Vec::new();
    for _ in 0..MAX_NEW_TOKENS {
        let mask = matcher.compute_mask()?;
        if matcher.is_stopped() { break; }
        let values = logits.to_vec1::<f32>()?;
        let selected = constrained_argmax(&values, &mask)?;
        matcher.consume_token(selected)?;
        generated.push(selected);
        if matcher.is_stopped() { break; }
        let position = prompt_tokens.len() + generated.len() - 1;
        let input = Tensor::new(&[selected], model.device())?.unsqueeze(0)?;
        logits = model.forward(&input, position)?.squeeze(0)?.to_dtype(DType::F32)?;
    }
    if !matcher.is_stopped() && !matcher.is_accepting()? {
        bail!("generation reached {MAX_NEW_TOKENS} tokens before the grammar completed");
    }
    let json = tok_env.tok_trie().decode_str(&generated);
    let command = serde_json::from_str(&json).with_context(|| format!("schema-constrained output did not map to Command: {json}"))?;
    Ok((json, command))
}

How does real constrained generation work?

• The prompt uses Phi-3's chat markers and lists the permitted command shapes.
• The tokenize_special() call preserves the special chat tokens during tokenization.
• The first forward() call prefills the model with the complete prompt.
• Each loop computes a fresh mask before constrained_argmax() selects a token.
• Each selected token returns to Candle at its autoregressive position.
• The completed token sequence becomes JSON before Serde converts it into Command.

• Save src/decoder.rs.
• Confirm the real inference loop compiles by running:

cargo check

What should you see now?

Cargo checks generate_command() without a compilation error. This confirms that the Candle tensor shapes and llguidance mask types connect successfully.

Decoder loop not compiling?

Confirm that DType appears in the Candle import. Check that PhiModel appears in the crate import.

Check the position calculation carefully. It uses the prompt length plus the number of generated tokens minus one.

Help me troubleshoot the real-logit decoder loop.

✔️ Awesome, I've got everything!

Your src/decoder.rs file now supports the synthetic lab and real Phi-3 generation.

ⓧ I'd like to double check the full code

Compare your complete src/decoder.rs file with this version.

use anyhow::{bail, ensure, Context, Result};
use candle::{DType, Device, Tensor};
use llguidance::toktrie::{SimpleVob, TokEnv};

use crate::{model::PhiModel, schema::{new_matcher, Command}};

const MAX_NEW_TOKENS: usize = 160;

fn argmax(logits: &[f32]) -> Result<u32> {
    logits.iter().enumerate().filter(|(_, score)| !score.is_nan())
        .max_by(|left, right| left.1.total_cmp(right.1))
        .map(|(index, _)| index as u32)
        .context("logits contained no selectable value")
}

fn constrained_argmax(logits: &[f32], mask: &SimpleVob) -> Result<u32> {
    ensure!(logits.len() >= mask.len(), "logit vocabulary is smaller than the llguidance mask");
    let mut best: Option<(u32, f32)> = None;
    mask.iter_set_entries(|index| {
        let score = logits[index];
        if !score.is_nan() && best.is_none_or(|(_, best_score)| score > best_score) {
            best = Some((index as u32, score));
        }
    });
    best.map(|(token, _)| token).context("the grammar allowed no selectable token")
}

pub fn run_mask_lab(tok_env: &TokEnv) -> Result<()> {
    let target_json = r#"{"action":"list_directory","path":"."}"#;
    let target_tokens = tok_env.tokenize(target_json);
    ensure!(!target_tokens.is_empty(), "target JSON produced no tokens");
    let mut first_matcher = new_matcher(tok_env)?;
    let first_mask = first_matcher.compute_mask()?;
    let first_target = target_tokens[0];
    ensure!(first_mask.is_allowed(first_target), "target token is not schema-valid");
    let vocab_size = tok_env.tok_trie().vocab_size();
    let forbidden = (0..vocab_size as u32).find(|token| !first_mask.is_allowed(*token))
        .context("could not find a forbidden token for the demonstration")?;
    let mut synthetic = vec![-100.0_f32; vocab_size];
    synthetic[first_target as usize] = 10.0;
    synthetic[forbidden as usize] = 100.0;
    let tensor = Tensor::from_vec(synthetic, vocab_size, &Device::Cpu)?;
    let logits = tensor.to_vec1::<f32>()?;
    let raw = argmax(&logits)?;
    let constrained = constrained_argmax(&logits, &first_mask)?;
    println!("Raw argmax: {} allowed={}", tok_env.tok_trie().token_dbg(raw), first_mask.is_allowed(raw));
    println!("Constrained argmax: {} allowed={}", tok_env.tok_trie().token_dbg(constrained), first_mask.is_allowed(constrained));

    let mut matcher = new_matcher(tok_env)?;
    let mut generated = Vec::new();
    for target in target_tokens {
        let mask = matcher.compute_mask()?;
        ensure!(mask.is_allowed(target), "known-good token was rejected");
        let mut values = vec![-100.0_f32; vocab_size];
        values[target as usize] = 10.0;
        let tensor = Tensor::from_vec(values, vocab_size, &Device::Cpu)?;
        let selected = constrained_argmax(&tensor.to_vec1::<f32>()?, &mask)?;
        matcher.consume_token(selected)?;
        generated.push(selected);
    }
    ensure!(matcher.is_accepting()?, "grammar did not reach an accepting state");
    let json = tok_env.tok_trie().decode_str(&generated);
    let command: Command = serde_json::from_str(&json)?;
    println!("Schema-conforming JSON: {json}");
    println!("Parsed command: {command:?}");
    println!("Grammar accepting: true");
    Ok(())
}

pub fn generate_command(model: &mut PhiModel, tok_env: &TokEnv, request: &str) -> Result<(String, Command)> {
    let prompt = format!("<|user|>\nYou are a filesystem command router. Return exactly one compact JSON object.\nAllowed shapes:\n{{\"action\":\"text\",\"message\":\"...\"}}\n{{\"action\":\"list_directory\",\"path\":\"relative path\"}}\n{{\"action\":\"change_directory\",\"path\":\"relative path\"}}\n{{\"action\":\"create_directory\",\"path\":\"relative path\"}}\n{{\"action\":\"create_file\",\"path\":\"relative path\",\"content\":\"...\"}}\nUse only relative paths and use . for the current directory.\nRequest: {request}<|end|>\n<|assistant|>\n");
    let prompt_tokens = tok_env.tokenize_special(&prompt);
    ensure!(!prompt_tokens.is_empty(), "prompt produced no tokens");
    let input = Tensor::new(prompt_tokens.as_slice(), model.device())?.unsqueeze(0)?;
    let mut logits = model.forward(&input, 0)?.squeeze(0)?.to_dtype(DType::F32)?;
    let mut matcher = new_matcher(tok_env)?;
    let mut generated = Vec::new();
    for _ in 0..MAX_NEW_TOKENS {
        let mask = matcher.compute_mask()?;
        if matcher.is_stopped() { break; }
        let values = logits.to_vec1::<f32>()?;
        let selected = constrained_argmax(&values, &mask)?;
        matcher.consume_token(selected)?;
        generated.push(selected);
        if matcher.is_stopped() { break; }
        let position = prompt_tokens.len() + generated.len() - 1;
        let input = Tensor::new(&[selected], model.device())?.unsqueeze(0)?;
        logits = model.forward(&input, position)?.squeeze(0)?.to_dtype(DType::F32)?;
    }
    if !matcher.is_stopped() && !matcher.is_accepting()? {
        bail!("generation reached {MAX_NEW_TOKENS} tokens before the grammar completed");
    }
    let json = tok_env.tok_trie().decode_str(&generated);
    let command = serde_json::from_str(&json).with_context(|| format!("schema-constrained output did not map to Command: {json}"))?;
    Ok((json, command))
}

What should match?

Keep the synthetic mask lab unchanged. Confirm that generate_command() reuses the same constrained_argmax() function.

Add the one-shot inference mode

A one-shot mode gives you the smallest possible real-model test. It loads Phi-3 once and prints both the constrained JSON and its parsed Rust command.

• In src/main.rs, replace the complete file with this mode dispatcher:

mod decoder;
mod model;
mod sandbox;
mod schema;

use anyhow::{bail, Result};
use decoder::{generate_command, run_mask_lab};
use llguidance::toktrie::TokEnv;
use model::{PhiModel, TOKENIZER_REPO};

fn main() -> Result<()> {
    let args: Vec<String> = std::env::args().collect();
    let mode = args.get(1).map(String::as_str).unwrap_or("lab");
    let tok_env: TokEnv = toktrie_hf_downloader::tok_env_from_name(TOKENIZER_REPO)?;
    match mode {
        "lab" => run_mask_lab(&tok_env),
        "once" => {
            let request = if args.len() > 2 { args[2..].join(" ") } else { "list the current directory".to_string() };
            let mut model = PhiModel::load()?;
            let (json, command) = generate_command(&mut model, &tok_env, &request)?;
            println!("JSON: {json}");
            println!("Parsed: {command:?}");
            Ok(())
        }
        other => bail!("unknown mode {other:?}; use lab or once"),
    }
}

What does the new mode do?

• The lab branch keeps the synthetic demonstration available.
• The once branch joins the remaining arguments into one request.
• The branch loads PhiModel before calling generate_command().
• The terminal prints the schema-constrained JSON.
• The terminal also prints the parsed Command variant.

• Save src/main.rs.

✔️ Awesome, I've got everything!

Your src/main.rs file now supports both lab and once modes.

ⓧ I'd like to double check the full code

Compare your complete src/main.rs file with this version.

mod decoder;
mod model;
mod sandbox;
mod schema;

use anyhow::{bail, Result};
use decoder::{generate_command, run_mask_lab};
use llguidance::toktrie::TokEnv;
use model::{PhiModel, TOKENIZER_REPO};

fn main() -> Result<()> {
    let args: Vec<String> = std::env::args().collect();
    let mode = args.get(1).map(String::as_str).unwrap_or("lab");
    let tok_env: TokEnv = toktrie_hf_downloader::tok_env_from_name(TOKENIZER_REPO)?;
    match mode {
        "lab" => run_mask_lab(&tok_env),
        "once" => {
            let request = if args.len() > 2 { args[2..].join(" ") } else { "list the current directory".to_string() };
            let mut model = PhiModel::load()?;
            let (json, command) = generate_command(&mut model, &tok_env, &request)?;
            println!("JSON: {json}");
            println!("Parsed: {command:?}");
            Ok(())
        }
        other => bail!("unknown mode {other:?}; use lab or once"),
    }
}

What should match?

Confirm that the tokenizer loads before mode selection. Check that only the once branch loads the real model.

Before the first model load

The first real-model run downloads the 2.39 GB GGUF file. The quiet period comes from network transfer and model loading.

Total runtime memory depends on your machine. The synthetic mask lab remains available if the model cannot fit.

• Close other memory-heavy applications before starting the first run.

Before you run this, which command shape do you expect Phi-3 to choose for the request?

• Run the one-shot request from the schema-fs-repl project folder with this command:

cargo run --release -- once "list the current directory"

What should you see?

The first run resolves the model through Hugging Face Hub and loads it on the CPU. After generation finishes, you see a complete JSON object followed by its parsed Command variant.

The exact JSON can vary with model logits. Its structure still matches one branch of your filesystem command schema.

Model download or inference failed?

If the download stops, confirm your network connection has enough free disk space for the 2.39 GB model. Run the same command again so the cache can resolve the file.

If the model cannot load in memory, close other applications. You can continue using the completed mask lab on this machine.

Help me diagnose the Phi-3 download or CPU inference failure.

That is the real inference bridge working. Phi-3 now supplies every logit while llguidance controls which token can win.

Your application can now turn a natural-language request into a schema-conforming Rust command. Next, you will execute those typed commands inside a path-checked filesystem sandbox.

Execute Commands in a Sandbox REPL

Your Candle decoder now turns a plain-English request into a valid Command. That command needs a controlled place to act before it can touch the filesystem.

A filesystem sandbox keeps every action under sandbox/. It rejects paths that escape its canonical root.

You will connect the typed handlers to a REPL. The model stays loaded between requests.

In this step, get ready to:

• Protect filesystem access with canonical path checks.
• Execute the five typed command variants through Rust filesystem APIs.
• Run a persistent REPL for repeated sandboxed requests.

Protect the sandbox boundary

Path traversal can turn an innocent-looking request into access outside your project. The sandbox prevents that by checking every path against one canonical root.

• In the schema-fs-repl project from earlier, select src/sandbox.rs in the VS Code Explorer sidebar.
• Replace the empty file with this path-protection foundation:

use std::{fs, path::{Component, Path, PathBuf}};
use anyhow::{bail, ensure, Context, Result};
use crate::schema::Command;

pub struct Sandbox { root: PathBuf, cwd: PathBuf }

impl Sandbox {
    pub fn new(root: impl AsRef<Path>) -> Result<Self> { fs::create_dir_all(root.as_ref())?; let root = root.as_ref().canonicalize()?; Ok(Self { root, cwd: PathBuf::new() }) }
    pub fn cwd_display(&self) -> String { if self.cwd.as_os_str().is_empty() { ".".to_string() } else { self.cwd.display().to_string() } }
    fn resolve(&self, input: &str) -> Result<PathBuf> { let relative = Path::new(input); ensure!(!relative.is_absolute(), "absolute paths are not allowed"); for component in relative.components() { match component { Component::Normal(_) | Component::CurDir => {}, _ => bail!("path traversal is not allowed"), } } let target = self.root.join(&self.cwd).join(relative); let mut existing = target.as_path(); while !existing.exists() { existing = existing.parent().context("path has no existing ancestor")?; } let canonical_existing = existing.canonicalize()?; ensure!(canonical_existing.starts_with(&self.root), "path escapes the sandbox"); Ok(target) }
}

How Does the Boundary Work?

• The Sandbox::new() method creates the sandbox root when needed.
• The canonicalized root becomes the boundary for every request.
• The cwd field stores the current directory as a relative path.
• The resolve() method rejects absolute paths.
• The component check rejects parent-directory traversal.
• The nearest existing ancestor catches escapes through symbolic links before a missing target is created.

• Save src/sandbox.rs.
• Check that the path-protection foundation compiles by running:

cargo check

What Does This Check Prove?

Cargo checks the new module against the existing Command type. A successful check proves that the sandbox boundary compiles before execution logic is added.

Cargo finishes checking schema-fs-repl without an error. Warnings about unused sandbox code are expected until the REPL uses it.

Seeing a Sandbox Compile Error?

Check that both use lines appear above Sandbox. Confirm that the final brace closes the impl Sandbox block.

Compare the path component names carefully. Rust treats Component::CurDir as a different identifier from other capitalizations.

Help me fix the sandbox path-protection code.

Execute typed filesystem commands

Schema-constrained JSON becomes a Rust Command before execution. Each handler calls filesystem APIs directly.

No handler invokes a shell. The enum decides which limited operation the application can perform.

• In src/sandbox.rs, find the final closing brace of the impl Sandbox block.
• Insert this execute() method immediately before that brace:

    pub fn execute(&mut self, command: Command) -> Result<()> { match command { Command::Text { message } => println!("{message}"), Command::ListDirectory { path } => { let target = self.resolve(&path)?.canonicalize()?; ensure!(target.is_dir(), "list target is not a directory"); let mut entries = fs::read_dir(target)?.map(|entry| entry.map(|value| value.file_name().to_string_lossy().into_owned())).collect::<std::io::Result<Vec<_>>>()?; entries.sort(); if entries.is_empty() { println!("(empty)"); } else { for entry in entries { println!("{entry}"); } } }, Command::ChangeDirectory { path } => { let target = self.resolve(&path)?.canonicalize()?; ensure!(target.is_dir(), "change target is not a directory"); self.cwd = target.strip_prefix(&self.root)?.to_path_buf(); println!("cwd: {}", self.cwd_display()); }, Command::CreateDirectory { path } => { let target = self.resolve(&path)?; fs::create_dir_all(&target)?; println!("created directory: {}", target.display()); }, Command::CreateFile { path, content } => { let target = self.resolve(&path)?; let parent = target.parent().context("file path has no parent")?; fs::create_dir_all(parent)?; fs::write(&target, content)?; println!("created file: {}", target.display()); } } Ok(()) }

What Does This Method Do?

• The Text variant prints the model's message.
• The ListDirectory variant sorts the entries before printing them.
• The ChangeDirectory variant updates the relative cwd only after validating a directory.
• The CreateDirectory variant creates a validated directory path.
• The CreateFile variant writes content through Rust's filesystem API.

• Save src/sandbox.rs.
• Check that every typed handler compiles by running:

cargo check

What Does This Check Prove?

This check verifies that every Command variant has a matching handler. It also confirms that each filesystem result can flow through anyhow::Result.

Cargo finishes without an error. Your application now has typed handlers for every schema branch.

Handler Match Not Compiling?

Make sure the method sits inside the existing impl Sandbox block. A method placed after the final brace becomes invalid top-level code.

Check each variant name against src/schema.rs. The handler names must match the enum exactly.

Help me fix the typed sandbox handlers.

✔️ Awesome, I've got everything!

Your sandbox now validates paths before executing any typed filesystem command.

ⓧ I'd like to double check the full code

Compare your complete src/sandbox.rs file with this version.

use std::{fs, path::{Component, Path, PathBuf}};
use anyhow::{bail, ensure, Context, Result};
use crate::schema::Command;

pub struct Sandbox { root: PathBuf, cwd: PathBuf }

impl Sandbox {
    pub fn new(root: impl AsRef<Path>) -> Result<Self> { fs::create_dir_all(root.as_ref())?; let root = root.as_ref().canonicalize()?; Ok(Self { root, cwd: PathBuf::new() }) }
    pub fn cwd_display(&self) -> String { if self.cwd.as_os_str().is_empty() { ".".to_string() } else { self.cwd.display().to_string() } }
    fn resolve(&self, input: &str) -> Result<PathBuf> { let relative = Path::new(input); ensure!(!relative.is_absolute(), "absolute paths are not allowed"); for component in relative.components() { match component { Component::Normal(_) | Component::CurDir => {}, _ => bail!("path traversal is not allowed"), } } let target = self.root.join(&self.cwd).join(relative); let mut existing = target.as_path(); while !existing.exists() { existing = existing.parent().context("path has no existing ancestor")?; } let canonical_existing = existing.canonicalize()?; ensure!(canonical_existing.starts_with(&self.root), "path escapes the sandbox"); Ok(target) }
    pub fn execute(&mut self, command: Command) -> Result<()> { match command { Command::Text { message } => println!("{message}"), Command::ListDirectory { path } => { let target = self.resolve(&path)?.canonicalize()?; ensure!(target.is_dir(), "list target is not a directory"); let mut entries = fs::read_dir(target)?.map(|entry| entry.map(|value| value.file_name().to_string_lossy().into_owned())).collect::<std::io::Result<Vec<_>>>()?; entries.sort(); if entries.is_empty() { println!("(empty)"); } else { for entry in entries { println!("{entry}"); } } }, Command::ChangeDirectory { path } => { let target = self.resolve(&path)?.canonicalize()?; ensure!(target.is_dir(), "change target is not a directory"); self.cwd = target.strip_prefix(&self.root)?.to_path_buf(); println!("cwd: {}", self.cwd_display()); }, Command::CreateDirectory { path } => { let target = self.resolve(&path)?; fs::create_dir_all(&target)?; println!("created directory: {}", target.display()); }, Command::CreateFile { path, content } => { let target = self.resolve(&path)?; let parent = target.parent().context("file path has no parent")?; fs::create_dir_all(parent)?; fs::write(&target, content)?; println!("created file: {}", target.display()); } } Ok(()) }
}

What Should Match?

Confirm that resolve() validates every path before the handlers read or create anything. The final brace must close the single impl Sandbox block.

Run the persistent REPL

A REPL keeps Phi-3 in memory while you enter repeated requests. Each completed JSON command passes into the same sandbox instance.

• Select src/main.rs in the VS Code Explorer sidebar.
• Replace its contents with the REPL-enabled entry point below:

mod decoder;
mod model;
mod sandbox;
mod schema;

use std::io::{self, Write};
use anyhow::{bail, Result};
use decoder::{generate_command, run_mask_lab};
use llguidance::toktrie::TokEnv;
use model::{PhiModel, TOKENIZER_REPO};
use sandbox::Sandbox;

fn main() -> Result<()> {
    let args: Vec<String> = std::env::args().collect();
    let mode = args.get(1).map(String::as_str).unwrap_or("lab");
    let tok_env = toktrie_hf_downloader::tok_env_from_name(TOKENIZER_REPO)?;
    match mode {
        "lab" => run_mask_lab(&tok_env),
        "once" => { let request = if args.len() > 2 { args[2..].join(" ") } else { "list the current directory".to_string() }; let mut model = PhiModel::load()?; let (json, command) = generate_command(&mut model, &tok_env, &request)?; println!("JSON: {json}"); println!("Parsed: {command:?}"); Ok(()) }
        "repl" => run_repl(tok_env),
        other => bail!("unknown mode {other:?}; use lab, once, or repl"),
    }
}

fn run_repl(tok_env: TokEnv) -> Result<()> {
    let mut model = PhiModel::load()?;
    let mut sandbox = Sandbox::new("sandbox")?;
    let stdin = io::stdin();
    println!("Schema-constrained filesystem REPL. Enter exit to quit.");
    loop { print!("fs:{}> ", sandbox.cwd_display()); io::stdout().flush()?; let mut request = String::new(); if stdin.read_line(&mut request)? == 0 { break; } let request = request.trim(); if request.eq_ignore_ascii_case("exit") { break; } if request.is_empty() { continue; } match generate_command(&mut model, &tok_env, request) { Ok((json, command)) => { println!("JSON: {json}"); if let Err(error) = sandbox.execute(command) { eprintln!("Action blocked: {error:#}"); } } Err(error) => eprintln!("Generation failed: {error:#}"), } }
    Ok(())
}

How Does the REPL Stay Safe?

• The run_repl() function loads the model once.
• The loop displays the current relative sandbox directory in its prompt.
• Each request passes through generate_command() before execution.
• The sandbox.execute() method receives only a deserialized Command.
• Blocked actions print an error without closing the interactive loop.
• Entering exit ends the process cleanly.

• Save src/main.rs.

✔️ Awesome, I've got everything!

Your entry point now supports the existing modes plus the persistent REPL.

ⓧ I'd like to double check the full code

Compare your complete src/main.rs file with this version.

mod decoder;
mod model;
mod sandbox;
mod schema;

use std::io::{self, Write};
use anyhow::{bail, Result};
use decoder::{generate_command, run_mask_lab};
use llguidance::toktrie::TokEnv;
use model::{PhiModel, TOKENIZER_REPO};
use sandbox::Sandbox;

fn main() -> Result<()> {
    let args: Vec<String> = std::env::args().collect();
    let mode = args.get(1).map(String::as_str).unwrap_or("lab");
    let tok_env = toktrie_hf_downloader::tok_env_from_name(TOKENIZER_REPO)?;
    match mode {
        "lab" => run_mask_lab(&tok_env),
        "once" => { let request = if args.len() > 2 { args[2..].join(" ") } else { "list the current directory".to_string() }; let mut model = PhiModel::load()?; let (json, command) = generate_command(&mut model, &tok_env, &request)?; println!("JSON: {json}"); println!("Parsed: {command:?}"); Ok(()) }
        "repl" => run_repl(tok_env),
        other => bail!("unknown mode {other:?}; use lab, once, or repl"),
    }
}

fn run_repl(tok_env: TokEnv) -> Result<()> {
    let mut model = PhiModel::load()?;
    let mut sandbox = Sandbox::new("sandbox")?;
    let stdin = io::stdin();
    println!("Schema-constrained filesystem REPL. Enter exit to quit.");
    loop { print!("fs:{}> ", sandbox.cwd_display()); io::stdout().flush()?; let mut request = String::new(); if stdin.read_line(&mut request)? == 0 { break; } let request = request.trim(); if request.eq_ignore_ascii_case("exit") { break; } if request.is_empty() { continue; } match generate_command(&mut model, &tok_env, request) { Ok((json, command)) => { println!("JSON: {json}"); if let Err(error) = sandbox.execute(command) { eprintln!("Action blocked: {error:#}"); } } Err(error) => eprintln!("Generation failed: {error:#}"), } }
    Ok(())
}

What Should Match?

Confirm that the repl match arm passes the tokenizer environment into run_repl(). The interactive loop must keep one model instance and one sandbox instance alive.

Before you run the REPL, do you expect the model to create hello.txt beside the project files or inside sandbox/?

• Start the persistent REPL by running:

cargo run --release -- repl

What Happens When the REPL Starts?

The application loads the cached model into CPU memory. It also creates the persistent sandbox/ directory when needed.

The model load can take a little while on CPU, so the printed loading messages show that the process is still moving. When it is ready, you will see a prompt beginning with fs:.>.

• Create the demonstration file by entering this request at the REPL prompt:

create a file called hello.txt containing hello

What Does This Request Prove?

The model generates a schema-conforming create_file object. The typed handler resolves its path under the sandbox before writing the content.

You will see a compact JSON object with action set to create_file. The next line confirms a path ending in sandbox/hello.txt.

• List the sandbox contents by entering this request at the next REPL prompt:

list the current directory

What Does the Listing Confirm?

The model generates a list_directory command for the current relative path. The handler reads that validated directory before printing its sorted entries.

You will see the constrained JSON command followed by hello.txt in the directory listing. That file exists inside the local sandbox/ directory.

• Exit the REPL cleanly by entering exit.

The REPL closes without unloading and reloading the model between your two requests. You now have the complete path from constrained local inference to protected filesystem execution.

REPL Not Creating the File?

Confirm that you started the repl mode from the schema-fs-repl project. The ready prompt should begin with fs:.>.

If the terminal prints Generation failed:, re-enter the creation request exactly as shown above. If it prints Action blocked:, check that the generated path is relative.

Help me troubleshoot the sandbox REPL.

Secret mission

Add Per-Token Decoding Diagnostics

Your REPL already produces safe schema-conforming commands. Add an optional diagnostics mode that exposes each grammar mask. You will see the allowed-token count plus the chosen token without changing command behavior.

Clean Up Your Resources

Clean Up Your Resources

Your local project has no ongoing charges. Choose the option below that matches what you want to keep.

Resources you used:

• The local schema-fs-repl folder containing your Rust source files.
• The Cargo build artifacts stored inside schema-fs-repl.
• The sandbox/ directory containing files from your REPL demo.
• The cached Microsoft Phi-3 tokenizer data downloaded through Hugging Face Hub.
• The cached Phi-3-mini-4k-instruct-q4.gguf model file.

Keep everything running

No action needed. Choose this if you are still testing requests or extending the REPL.

• Keep the schema-fs-repl folder ready for future development.
• Keep sandbox/ to preserve the files created during your demo.
• Keep the cached model file to avoid another 2.39 GB download.
• Keep the cached tokenizer data for future runs.

Pause - I'll come back to this later

Stop the active REPL to free its memory. All local files remain available for your next session.

• Enter exit at the fs: prompt.
• Wait for your normal terminal prompt to return.
• Leave schema-fs-repl on disk for later.

The model process is now stopped. No server or paid resource continues running.

Delete - I don't want to use this again

Remove all project resources to start fresh.

Deleting local files is permanent. Confirm each target before running a deletion command.

• Use your file manager to locate the schema-fs-repl folder.
• Record its full path here: full path to schema-fs-repl.
• Delete the project folder by running this command:

rm -rf "[[PROJECT_FOLDER_PATH="full path to schema-fs-repl"]]"

This removes the source files. It also removes the build artifacts plus the entire sandbox/ directory.

• Inspect the directory that previously contained schema-fs-repl.

You should no longer see the project folder.

Project Folder Still Present?

• Confirm that the recorded path ends with schema-fs-repl.
• Check that your account has permission to delete the folder.

Help me safely diagnose why my local schema-fs-repl folder was not deleted.

The cached model file uses 2.39 GB. Only remove cached Phi-3 files when no other local project uses them.

Confirm the Cache Paths

Cache locations vary by local configuration. Delete only entries that clearly belong to the model repositories used in this project.

• Confirm that the model path ends with Phi-3-mini-4k-instruct-q4.gguf.
• Confirm that the tokenizer cache entry belongs to microsoft/Phi-3-mini-4k-instruct.

• Find the full model path in the earlier terminal line beginning with Loading model from.
• Record the full model file path here: full path to Phi-3-mini-4k-instruct-q4.gguf.
• Search your home directory for the cached tokenizer entry associated with microsoft/Phi-3-mini-4k-instruct.
• Record the full tokenizer cache entry path here: full path to the cached Phi-3 tokenizer entry.
• Remove the two confirmed cache entries by running these commands:

rm -f "[[PHI3_GGUF_PATH="full path to Phi-3-mini-4k-instruct-q4.gguf"]]"
rm -rf "[[PHI3_TOKENIZER_CACHE_PATH="full path to the cached Phi-3 tokenizer entry"]]"

These commands remove the selected GGUF file. They also remove the selected tokenizer cache entry.

• Search your home directory again for Phi-3-mini-4k-instruct-q4.gguf.
• Inspect the parent cache location that held full path to the cached Phi-3 tokenizer entry.

You should no longer find either selected cache entry. A future real-model run downloads any missing data again.

Cache Entry Still Present?

• Confirm that each recorded path points to the intended Phi-3 cache entry.
• Check that no process is currently using the cached model file.

Help me safely remove the remaining Phi-3 cache entry.

Nice Work!

Nice Work!

Excellent work! Your CPU-only Rust REPL now turns plain-English requests into schema-conforming filesystem commands.

You learned how to:

• Built a Candle-based CPU inference path for quantized Phi-3 that produces token logits locally.
• Exposed a schema-invalid raw argmax choice. Applied an llguidance SimpleVob mask to select an allowed token before sampling. Advanced the grammar after each selected token until it accepted complete JSON.
• Deserialized completed JSON into a tagged Command enum with Serde. Executed typed filesystem operations through sandbox path checks without invoking a shell.
• Secret Mission: Added optional per-token decoding diagnostics for every constrained choice. The diagnostics reveal the step number, allowed-token count, selected token ID, plus the tokenizer debug representation. The final JSON remains unchanged.

Ready to quiz yourself?