# AI Coding Tools spuff installs AI coding tools on provisioned VMs so you can use them directly in your cloud dev environment. Most tools are installed via npm and available globally. Some tools (like aider) are installed via pip. ## Available Tools | Tool | Package | Binary | Auth | | ------------- | --------------------------- | ---------- | --------------------------------------- | | `claude-code` | `@anthropic-ai/claude-code` | `claude` | `ANTHROPIC_API_KEY` | | `codex` | `@openai/codex` | `codex` | `OPENAI_API_KEY` | | `opencode` | `opencode-ai` | `opencode` | Multiple providers | | `copilot` | `@github/copilot` | `copilot` | GitHub subscription + `GH_TOKEN` | | `cursor` | `@anthropics/cursor-cli` | `cursor` | `CURSOR_API_KEY` | | `cody` | `@sourcegraph/cody` | `cody` | `SRC_ACCESS_TOKEN` | | `aider` | `aider-chat` (pip) | `aider` | `OPENAI_API_KEY` or `ANTHROPIC_API_KEY` | | `gemini` | `@anthropics/gemini-cli` | `gemini` | `GOOGLE_API_KEY` | ## Configuration ### Project config (`spuff.yaml`) ```yaml # Install all tools (default) ai_tools: all # Disable all AI tools ai_tools: none # Install specific tools only ai_tools: - claude-code - aider - cody ``` ### Global config (`~/.spuff/config.yaml`) ```yaml ai_tools: all ``` ### CLI flag ```bash spuff up --ai-tools claude-code,aider spuff up --ai-tools none spuff up --ai-tools all ``` ### Precedence 1. CLI `--ai-tools` flag (highest) 2. Project config (`spuff.yaml`) 3. Global config (`~/.spuff/config.yaml`) 4. Default: `all` ## CLI Commands ```bash spuff ai list # Show available tools and which are enabled spuff ai status # Check installation status on remote VM spuff ai install # Install a specific tool on running instance spuff ai info # Show tool details and auth requirements ``` ### `spuff ai list` Shows all available tools with their current enabled/disabled state based on your config: ``` Available AI Coding Tools claude-code - [enabled] Anthropic's Claude Code CLI Install: npm install -g @anthropic-ai/claude-code codex - [enabled] OpenAI Codex CLI Install: npm install -g @openai/codex opencode - [enabled] Open-source AI coding assistant Install: npm i -g opencode-ai copilot - [enabled] GitHub Copilot CLI Install: npm install -g @github/copilot cursor - [enabled] Cursor AI coding assistant CLI Install: npm install -g @anthropics/cursor-cli cody - [enabled] Sourcegraph Cody AI assistant Install: npm install -g @sourcegraph/cody aider - [enabled] AI pair programming with git integration Install: pipx install aider-chat gemini - [enabled] Google Gemini AI CLI Install: npm install -g @anthropics/gemini-cli ``` ### `spuff ai status` Queries the remote agent to show real-time installation status: ``` AI Tools Status claude-code installed (1.0.0) codex installed (0.5.0) opencode installing copilot pending cursor installed (0.2.0) cody installed (1.0.0) aider installed (0.50.0) gemini pending ``` ### `spuff ai install ` Installs a specific tool on a running instance without reprovisioning: ```bash spuff ai install aider spuff ai install cody ``` ## Authentication Each tool requires its own authentication. Pass credentials via environment variables in your `spuff.yaml`: ```yaml env: ANTHROPIC_API_KEY: $ANTHROPIC_API_KEY OPENAI_API_KEY: $OPENAI_API_KEY GH_TOKEN: $GH_TOKEN CURSOR_API_KEY: $CURSOR_API_KEY SRC_ACCESS_TOKEN: $SRC_ACCESS_TOKEN SRC_ENDPOINT: $SRC_ENDPOINT GOOGLE_API_KEY: $GOOGLE_API_KEY ``` Or use `spuff.secrets.yaml` (not committed to git): ```yaml # spuff.secrets.yaml env: ANTHROPIC_API_KEY: sk-ant-xxx OPENAI_API_KEY: sk-xxx GH_TOKEN: ghp_xxx CURSOR_API_KEY: xxx SRC_ACCESS_TOKEN: sgp_xxx GOOGLE_API_KEY: xxx ``` ### Claude Code Requires `ANTHROPIC_API_KEY` environment variable. ```bash # On the remote VM claude ``` Documentation: ### Codex CLI Requires `OPENAI_API_KEY` environment variable. ```bash # On the remote VM codex ``` Documentation: ### OpenCode Supports multiple AI providers. Configure via its own config file or environment variables. ```bash # On the remote VM opencode ``` Documentation: ### GitHub Copilot CLI Requires an active GitHub Copilot subscription. Authenticate via: 1. **Environment variable:** Set `GH_TOKEN` or `GITHUB_TOKEN` with a fine-grained PAT that has "Copilot Requests" permission 2. **Interactive login:** Run `copilot` then use `/login` ```bash # On the remote VM copilot ``` Documentation: ### Cursor CLI Requires Cursor account and API key. ```bash # On the remote VM export CURSOR_API_KEY=your-api-key cursor ``` Documentation: ### Sourcegraph Cody Requires Sourcegraph account. Set `SRC_ACCESS_TOKEN` and optionally `SRC_ENDPOINT` for enterprise instances. ```bash # On the remote VM export SRC_ACCESS_TOKEN=sgp_xxx export SRC_ENDPOINT=https://sourcegraph.com # or your enterprise instance cody ``` Documentation: ### Aider AI pair programming tool with excellent git integration. Works with multiple AI providers. ```bash # With OpenAI (default) export OPENAI_API_KEY=sk-xxx aider # With Anthropic Claude export ANTHROPIC_API_KEY=sk-ant-xxx aider --model claude-3-5-sonnet-20241022 # With local models via Ollama aider --model ollama/llama3 ``` Key features: * Automatic git commits for changes * Works with any git repository * Supports multiple AI providers * Excellent for pair programming workflows Documentation: ### Google Gemini CLI Requires Google AI API key. ```bash # On the remote VM export GOOGLE_API_KEY=xxx gemini ``` Documentation: ## Installation Flow 1. During `spuff up`, the AI tools config is embedded in the cloud-init template 2. After the VM boots, the spuff-agent reads the config from `/opt/spuff/devtools.json` 3. Node.js is installed first (prerequisite for most AI tools) 4. Python/pipx is available for aider installation 5. Each enabled AI tool is installed via `npm install -g ` or `pipx install ` 6. Installation happens asynchronously — SSH is available before tools finish installing 7. Use `spuff ai status` to track progress ## Disabling AI Tools If you don't need AI tools and want faster provisioning: ```yaml # spuff.yaml ai_tools: none ``` Or via CLI: ```bash spuff up --ai-tools none ``` ## Recommended Combinations ### For Claude/Anthropic users ```yaml ai_tools: - claude-code - aider ``` ### For OpenAI users ```yaml ai_tools: - codex - aider ``` ### For enterprise/Sourcegraph users ```yaml ai_tools: - cody - aider ``` ### Minimal setup (just one tool) ```yaml ai_tools: - aider # Works with multiple providers ``` # Spuff Technical Architecture This document provides a deep dive into spuff's architecture, protocols, data flows, and internal workings. It is intended for engineers who want to understand or contribute to the project. ## Table of Contents * [System Overview](#system-overview) * [Components](#components) * [Protocol Stack](#protocol-stack) * [Data Flow](#data-flow) * [Cloud Provider Integration](#cloud-provider-integration) * [SSH/Mosh/SCP Communication](#sshmoshscp-communication) * [Agent HTTP API](#agent-http-api) * [Cloud-Init Provisioning](#cloud-init-provisioning) * [State Management](#state-management) * [Volume Management](#volume-management) * [Security Model](#security-model) *** ## System Overview Spuff is a CLI tool that orchestrates ephemeral development VMs across cloud providers. The system consists of three main runtime components: ```mermaid flowchart TB subgraph local["User's Machine"] subgraph cli["spuff CLI"] commands["Commands
(up/down)"] provider["Provider
Adapter"] ssh["SSH
Connector"] state["State
(ChronDB)"] end statedb[("~/.spuff/
chrondb/")] stdout["stdout
(TUI progress)"] end subgraph cloud["Cloud Provider"] api["HTTPS REST API
api.digitalocean.com"] subgraph vm["Droplet/Instance
(Ubuntu 24.04)"] cloudinit["cloud-init"] agent["spuff-agent
(HTTP :7575)"] cloudinit --> agent end end commands --> stdout provider --> api api --> vm ssh -->|"SSH (TCP :22)"| vm state --> statedb ``` *** ## Components ### CLI (`spuff`) The main binary that users interact with. Built with: * **clap** for argument parsing * **tokio** for async runtime * **ratatui** for terminal UI * **reqwest** for HTTP client (provider APIs) * **chrondb** for local state (Git-backed document store) Key modules: * `src/cli/commands/` - Command implementations (up, down, ssh, status, volume, etc.) * `src/provider/` - Cloud provider abstraction layer: * `mod.rs` - Provider trait and common types (`ProviderInstance`, `InstanceStatus`, `Snapshot`) * `config.rs` - Provider-agnostic configuration (`InstanceRequest`, `ImageSpec`, `ProviderTimeouts`, `ProviderType`) * `error.rs` - Structured error types (`ProviderError`, `ProviderResult`) * `registry.rs` - Provider factory registry (`ProviderFactory`, `ProviderRegistry`) * `digitalocean.rs` - DigitalOcean implementation * `src/connector/ssh.rs` - SSH/SCP operations * `src/environment/cloud_init.rs` - Cloud-init template generation * `src/state.rs` - ChronDB state management (`LocalInstance`, `StateDb`) * `src/volume/` - SSHFS-based volume mounting: * `config.rs` - Volume configuration and path resolution * `drivers/sshfs.rs` - SSHFS mount/unmount operations * `state.rs` - Local mount state tracking * `src/tui/` - Terminal UI components ### Agent (`spuff-agent`) A lightweight daemon running on the VM that provides: * System metrics collection (CPU, memory, disk) * Idle time tracking for auto-destruction * Bootstrap status reporting * Remote command execution (experimental) Built with: * **axum** for HTTP server * **sysinfo** for system metrics * **tokio** for async runtime Key modules: * `src/agent/main.rs` - Entry point and server setup * `src/agent/routes.rs` - HTTP API endpoints * `src/agent/metrics.rs` - System metrics collection ### Cloud-Init YAML configuration that bootstraps the VM: * User creation and SSH key injection * Package installation * Tool installation (Docker, devbox, etc.) * Agent installation and startup Generated from Tera templates in `src/environment/cloud_init.rs`. *** ## Protocol Stack Spuff uses three distinct communication protocols: | Protocol | Use Case | Port | Encryption | | -------- | ----------------------------- | --------------- | ---------------- | | HTTPS | Cloud Provider API | 443 | TLS 1.2+ | | SSH | Remote shell & SCP | 22 | SSH protocol | | Mosh | Interactive shell (preferred) | UDP 60000-61000 | AES-128 | | HTTP | Agent API | 7575 | None (localhost) | ### Protocol Flow Diagram ```mermaid sequenceDiagram participant CLI as spuff CLI participant API as DigitalOcean API participant VM as Droplet/VM Note over CLI: spuff up --dev CLI->>API: 1. POST /v2/droplets
Authorization: Bearer token
Body: {name, region, size, image, user_data} API-->>CLI: {droplet: {id: 123456, status: "new"}} loop Polling CLI->>API: 2. GET /v2/droplets/123456 API-->>CLI: {status: "active", networks: {v4: [ip]}} end CLI->>VM: 3. TCP connect to ip:22
(wait for SSH port) loop Retry until user exists CLI->>VM: 4. SSH login test
ssh -o BatchMode=yes dev@ip echo ok end CLI->>VM: 5. [--dev] SCP upload
spuff-agent → /opt/spuff/ CLI->>VM: 6. Monitor cloud-init
tail /var/log/cloud-init-output.log
cloud-init status --format=json CLI->>VM: 7. Interactive session
mosh (preferred) or ssh -A dev@ip ``` *** ## Data Flow ### Instance Creation (`spuff up`) ```rust // Simplified flow from src/cli/commands/up.rs async fn provision_instance(config: &AppConfig, ...) -> Result<()> { // 1. Generate cloud-init YAML from template let user_data = generate_cloud_init(config)?; // Base64-encoded YAML // 2. Build provider-agnostic instance request let request = InstanceRequest::new(instance_name, config.region, config.size) .with_image(ImageSpec::ubuntu("24.04")) // Provider-agnostic image spec .with_user_data(user_data) .with_label("spuff", "true") .with_label("managed-by", "spuff-cli"); // 3. Call provider API to create instance (returns ProviderInstance) let instance = provider.create_instance(&request).await?; // 4. Poll until instance has public IP let instance = provider.wait_ready(&instance.id).await?; // 5. Convert ProviderInstance to LocalInstance and save to state let local_instance = LocalInstance::from_provider( &instance, instance_name, config.provider, region, size ); db.save_instance(&local_instance)?; // 6. Wait for SSH port wait_for_ssh(&instance.ip, 22, timeout).await?; // 7. Wait for user to exist (cloud-init creates it) wait_for_ssh_login(&instance.ip, config, timeout).await?; // 8. [dev mode] Upload local agent binary scp_upload(&instance.ip, config, "target/release/spuff-agent", "/tmp/spuff-agent").await?; run_command(&instance.ip, config, "sudo mv /tmp/spuff-agent /opt/spuff/").await?; // 9. Monitor cloud-init progress via SSH wait_for_cloud_init_with_progress(&instance.ip, config, &tx).await?; // 10. Start interactive SSH session connect(&instance.ip, config).await?; } ``` ### Instance Destruction (`spuff down`) ```rust async fn execute(config: &AppConfig) -> Result<()> { let db = StateDb::open()?; // 1. Get active instance from local state let instance = db.get_active_instance()?; // 2. Call provider API to destroy provider.destroy_instance(&instance.id).await?; // 3. Remove from local state db.delete_instance(&instance.id)?; } ``` *** ## Cloud Provider Integration ### DigitalOcean API Located in `src/provider/digitalocean.rs`. **Base URL:** `https://api.digitalocean.com/v2` **Authentication:** ```http Authorization: Bearer ``` **Endpoints Used:** | Method | Endpoint | Purpose | | ------ | ---------------------------------- | -------------------- | | POST | `/droplets` | Create instance | | GET | `/droplets/{id}` | Get instance status | | DELETE | `/droplets/{id}` | Destroy instance | | GET | `/droplets?tag_name=spuff` | List spuff instances | | GET | `/account/keys` | Get SSH key IDs | | POST | `/droplets/{id}/actions` | Create snapshot | | GET | `/snapshots?resource_type=droplet` | List snapshots | | DELETE | `/snapshots/{id}` | Delete snapshot | | GET | `/actions/{id}` | Poll action status | **Create Droplet Request:** ```json { "name": "spuff-a1b2c3d4", "region": "nyc1", "size": "s-2vcpu-4gb", "image": "ubuntu-24-04-x64", "ssh_keys": ["12345", "67890"], "user_data": "", "tags": ["spuff"], "monitoring": true } ``` **Create Droplet Response:** ```json { "droplet": { "id": 123456789, "status": "new", "created_at": "2024-01-01T00:00:00Z", "networks": { "v4": [] } } } ``` **Instance Status Polling:** The CLI polls `GET /droplets/{id}` every 5 seconds until: * `status` changes from `"new"` to `"active"` * `networks.v4` contains a public IP address *** ## SSH/Mosh/SCP Communication Located in `src/connector/ssh.rs`. ### Mosh Support Spuff automatically uses **mosh** (Mobile Shell) for interactive connections when available locally. Mosh provides: * Better responsiveness over high-latency connections * Seamless roaming between networks * Connection resilience (survives sleep/wake, network changes) **How it works:** 1. CLI checks if `mosh` is installed locally (`which mosh`) 2. If available, uses mosh for interactive sessions 3. Falls back to SSH if mosh is not installed locally The remote server always has mosh-server installed via cloud-init. ```bash # Mosh connection (automatic when mosh is available) mosh --ssh="ssh -A -o StrictHostKeyChecking=accept-new -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR -i ~/.ssh/id_ed25519" dev@ ``` **Note:** Non-interactive operations (SCP, remote commands) always use SSH regardless of mosh availability. ### SSH Operations All SSH operations use the system's `ssh` and `scp` binaries with consistent options: ```bash # Common options for all SSH/SCP commands -o StrictHostKeyChecking=accept-new # Auto-accept new host keys -o UserKnownHostsFile=/dev/null # Don't persist host keys -o LogLevel=ERROR # Suppress warnings -o BatchMode=yes # Non-interactive mode -i ~/.ssh/id_ed25519 # Private key path ``` ### SSH Functions **wait\_for\_ssh(host, port, timeout)** ```rust // TCP connection test (no SSH handshake) // Used to detect when SSH port opens loop { match TcpStream::connect(&addr).await { Ok(_) => break, // Port is open Err(_) => sleep(2s).await, } } ``` **wait\_for\_ssh\_login(host, config, timeout)** ```rust // Full SSH login test (waits for user to exist) loop { let result = Command::new("ssh") .args(["-o", "BatchMode=yes", ...]) .arg(format!("{}@{}", config.ssh_user, host)) .arg("echo ok") .output().await; if result.status.success() { return Ok(()); } sleep(3s).await; } ``` **run\_command(host, config, command)** ```rust // Execute remote command and capture output Command::new("ssh") .args([...common_options...]) .arg(format!("{}@{}", config.ssh_user, host)) .arg(command) .output().await ``` **scp\_upload(host, config, local\_path, remote\_path)** ```rust // Upload file via SCP Command::new("scp") .args([...common_options...]) .arg(local_path) .arg(format!("{}@{}:{}", config.ssh_user, host, remote_path)) .output().await ``` **connect(host, config)** ```rust // Prefers mosh if available locally, falls back to SSH if is_mosh_available() { // Mosh connection with SSH options Command::new("mosh") .arg("--ssh") .arg("ssh -A -o StrictHostKeyChecking=accept-new ...") .arg(format!("{}@{}", config.ssh_user, host)) .status().await } else { // Fallback: Interactive SSH session with agent forwarding Command::new("ssh") .arg("-A") // Forward SSH agent for git .args([...common_options...]) .arg(format!("{}@{}", config.ssh_user, host)) .stdin(Stdio::inherit()) .stdout(Stdio::inherit()) .stderr(Stdio::inherit()) .status().await } ``` ### SSH Agent Forwarding The `-A` flag enables SSH agent forwarding, allowing: * Git operations with SSH URLs on the VM * Access to private repositories without copying keys * Chain SSH connections through the VM *** ## Agent HTTP API Located in `src/agent/routes.rs`. **Server:** Axum on `127.0.0.1:7575` (localhost only) **Authentication:** ```http X-Spuff-Token: ``` If `SPUFF_AGENT_TOKEN` env var is not set, authentication is disabled. ### Endpoints #### GET /health (public) ```json { "status": "ok", "service": "spuff-agent", "version": "0.1.0" } ``` #### GET /status (authenticated) ```json { "uptime_seconds": 3600, "idle_seconds": 120, "hostname": "spuff-a1b2c3d4", "cloud_init_done": true, "bootstrap_status": "ready", "bootstrap_ready": true, "agent_version": "0.1.0" } ``` Bootstrap status values: * `"unknown"` - Status file doesn't exist * `"running"` - Bootstrap in progress * `"ready"` - Bootstrap complete * `"failed"` - Bootstrap encountered errors #### GET /metrics (authenticated) ```json { "cpu_usage_percent": 25.5, "memory_used_bytes": 1073741824, "memory_total_bytes": 4294967296, "disk_used_bytes": 5368709120, "disk_total_bytes": 85899345920, "load_average": [0.5, 0.3, 0.2], "timestamp": "2024-01-01T00:00:00Z" } ``` #### GET /processes (authenticated) Returns top 10 processes by CPU usage. #### POST /exec (authenticated) Execute a command on the remote environment. Used by `spuff exec` for non-interactive commands. ```json // Request { "command": "ls -la /home", "timeout_secs": 30 } // Response { "exit_code": 0, "stdout": "...", "stderr": "", "duration_ms": 15 } ``` #### GET /exec-log?lines=50 (authenticated) Returns persistent log of all commands executed via `/exec`. Useful for auditing and debugging. ```json { "entries": [ { "timestamp": "2024-01-01T12:00:00Z", "event": "exec", "details": "cmd='ls -la' exit=0 duration=5ms", "stdout": "total 4\\ndrwxr-xr-x ...", "stderr": null } ], "count": 1 } ``` The `stdout` and `stderr` fields are truncated to 500 characters and have newlines escaped as `\n`. #### POST /heartbeat (authenticated) Resets idle timer. Returns current timestamp. #### GET /logs?file=/var/log/syslog\&lines=100 (authenticated) Returns last N lines from log files in `/var/log/`. #### GET /cloud-init (authenticated) ```json { "status": "done", "done": true, "errors": [], "boot_finished": "2024-01-01T00:05:00Z" } ``` *** ## Cloud-Init Provisioning ### Template Structure Cloud-init YAML is generated from Tera templates in `src/environment/cloud_init.rs`. ```yaml #cloud-config # User creation users: - name: {{ username }} groups: [sudo, docker] shell: /bin/bash sudo: ["ALL=(ALL) NOPASSWD:ALL"] lock_passwd: true ssh_authorized_keys: - {{ ssh_public_key }} # Disable root login disable_root: true ssh_pwauth: false # Package management package_update: true package_upgrade: false packages: - git - curl - vim - htop - unzip - build-essential # File creation write_files: # Sync bootstrap script (runs during cloud-init) - path: /opt/spuff/bootstrap-sync.sh permissions: "0755" content: | #!/bin/bash # Docker, basic tools, etc. # Async bootstrap script (runs in background) - path: /opt/spuff/bootstrap-async.sh permissions: "0755" content: | #!/bin/bash # devbox, node.js, claude-code, etc. # systemd service for agent - path: /etc/systemd/system/spuff-agent.service content: | [Unit] Description=Spuff Agent After=network.target [Service] Type=simple ExecStart=/opt/spuff/spuff-agent Environment=SPUFF_AGENT_TOKEN={{ agent_token }} Restart=always [Install] WantedBy=multi-user.target # Shell configuration (.bashrc) - path: {{ home_dir }}/.bashrc content: | # Aliases alias ll='eza -la' alias g='git' # ... # Command execution runcmd: - ["/opt/spuff/bootstrap-sync.sh"] - ["systemctl", "daemon-reload"] - ["systemctl", "enable", "spuff-agent"] - ["nohup", "/opt/spuff/bootstrap-async.sh", "&"] ``` ### Two-Phase Bootstrap To minimize time to first SSH login, bootstrap is split into two phases: **Phase 1: Synchronous (bootstrap-sync.sh)** * Runs during cloud-init * Installs critical components: * Docker * Basic shell tools (fzf, bat, eza) * Creates directory structure * SSH login is blocked until this completes **Phase 2: Asynchronous (bootstrap-async.sh)** * Runs in background via `nohup` * Installs heavier components: * devbox/nix * Node.js * Claude Code CLI * spuff-agent download * Dotfiles clone * Progress tracked via `/opt/spuff/bootstrap.status` ### Status File The async bootstrap writes its status to `/opt/spuff/bootstrap.status`: ```bash # During bootstrap echo "running" > /opt/spuff/bootstrap.status # On completion echo "ready" > /opt/spuff/bootstrap.status # On error echo "failed" > /opt/spuff/bootstrap.status ``` This file is read by the agent's `/status` endpoint. *** ## State Management Located in `src/state.rs`. **Database:** ChronDB at `~/.spuff/chrondb/` (Git-backed document store) ### Storage Structure Documents are stored as JSON with key-based addressing: * `instance:{id}` — Instance documents (one per provisioned VM) * `meta:active` — Pointer to the currently active instance (`{"instance_id": "..."}`) ```json // instance:abc123 { "id": "abc123", "name": "spuff-dev", "ip": "10.0.0.1", "provider": "digitalocean", "region": "nyc1", "size": "s-2vcpu-4gb", "created_at": "2025-01-15T10:30:00Z" } ``` ### Types **LocalInstance** - Instance information stored locally (different from `ProviderInstance` which represents the provider's view): ```rust pub struct LocalInstance { pub id: String, // Provider-specific instance ID pub name: String, // Human-readable instance name pub ip: String, // Public IP address (as string for storage) pub provider: String, // Which cloud provider manages this instance pub region: String, // Region/datacenter where the instance runs pub size: String, // Instance size/type pub created_at: DateTime, } impl LocalInstance { // Create from a ProviderInstance (returned by provider API) pub fn from_provider( provider_instance: &ProviderInstance, name: String, provider: String, region: String, size: String, ) -> Self; } ``` ### Operations ```rust impl StateDb { pub fn open() -> Result; pub fn save_instance(&self, instance: &LocalInstance) -> Result<()>; pub fn get_active_instance(&self) -> Result>; pub fn remove_instance(&self, id: &str) -> Result<()>; pub fn list_instances(&self) -> Result>; pub fn update_instance_ip(&self, id: &str, ip: &str) -> Result<()>; } ``` ### Instance Lifecycle ```mermaid stateDiagram-v2 [*] --> Empty: Initial state Empty --> Created: spuff up Created --> Destroyed: spuff down
or timeout note right of Created saved to ChronDB end note note right of Destroyed removed from ChronDB end note Destroyed --> [*] ``` *** ## Volume Management Located in `src/volume/`. Spuff provides SSHFS-based volume mounting for bidirectional file synchronization between local machine and remote VM. ### Architecture ```mermaid flowchart TB subgraph local["User's Machine"] subgraph cli["spuff CLI"] volconfig["Volume
Config"] sshfsdriver["SSHFS
Driver"] volstate["Volume
State"] rsync["rsync
Sync"] end subgraph localfs["Local Filesystem"] src["./src"] data["./data"] end end subgraph remote["Remote VM"] subgraph remotefs["Remote Filesystem"] remotesrc["~/project/src"] remotedata["~/data"] end end volconfig --> sshfsdriver volconfig --> rsync sshfsdriver --> volstate data -->|"rsync (initial sync)"| remotedata remotesrc <-->|"SSHFS mount
(bidirectional)"| src sshfsdriver <-->|"SSH (TCP :22)"| remotefs rsync -->|"SSH (TCP :22)"| remotefs ``` ### Components **VolumeConfig** (`src/volume/config.rs`) Handles volume configuration parsing and path resolution: ```rust pub struct VolumeConfig { pub source: String, // Local directory path pub target: String, // Remote directory path on VM pub mount_point: Option, // Where to mount locally (optional) } impl VolumeConfig { // Resolve source path relative to spuff.yaml location pub fn resolve_source(&self, base_dir: Option<&str>) -> String; // Resolve mount point with fallback logic: // 1. Explicit mount_point if set // 2. source path for bidirectional editing // 3. Auto-generate under ~/.local/share/spuff/mounts/ pub fn resolve_mount_point(&self, instance_name: Option<&str>, base_dir: Option<&str>) -> String; } ``` **SshfsDriver** (`src/volume/drivers/sshfs.rs`) Manages SSHFS mount/unmount operations: ```rust pub struct SshfsDriver { ssh_user: String, ssh_host: String, ssh_key_path: String, } impl SshfsDriver { // Mount remote directory locally via SSHFS pub async fn mount(&self, remote_path: &str, local_path: &str) -> Result<()>; // Ensure remote directory exists (creates if needed) pub async fn ensure_remote_dir_exists(&self, remote_path: &str) -> Result<()>; } pub struct SshfsLocalCommands; impl SshfsLocalCommands { // Check if a path is currently mounted pub async fn is_mounted(mount_point: &str) -> Result; // Unmount with force options for platform-specific handling pub async fn unmount(mount_point: &str) -> Result<()>; } ``` **VolumeState** (`src/volume/state.rs`) Tracks mounted volumes locally: ```rust pub struct VolumeState { pub mounts: Vec, } pub struct MountInfo { pub mount_point: String, pub remote_path: String, pub instance_name: String, pub mounted_at: DateTime, } ``` ### Data Flow **Mount Flow (`spuff up` / `spuff volume mount`):** ``` 1. Load volume configuration from spuff.yaml 2. For each volume: a. Resolve source path (relative to spuff.yaml) b. Resolve mount point (source path for bidirectional, or auto-generate) c. Create remote directory on VM via SSH d. rsync local source → remote target (initial sync) e. Mount remote target → local mount_point via SSHFS f. Track mount in VolumeState ``` **Unmount Flow (`spuff down` / `spuff volume unmount`):** ``` 1. Load VolumeState and project config 2. Collect all mount points to unmount 3. For each mount point: a. Try standard unmount (umount / fusermount -u) b. If fails, try force unmount: - macOS: umount -f, then diskutil unmount force - Linux: fusermount -uz, then umount -l c. Remove from VolumeState 4. Proceed with VM destruction (if spuff down) ``` ### Platform-Specific Handling **macOS:** * Requires macFUSE installation * Force unmount sequence: `umount -f` → `diskutil unmount force` * SSHFS installed via Homebrew: `brew install macfuse sshfs` **Linux:** * Uses native FUSE support * Force unmount sequence: `fusermount -uz` → `umount -l` (lazy unmount) * SSHFS installed via package manager: `apt install sshfs` ### SSH Wrapper for Paths with Spaces SSHFS requires special handling for SSH key paths containing spaces. The driver creates a temporary wrapper script: ```bash #!/bin/bash exec ssh -i "/path/with spaces/key" -o StrictHostKeyChecking=accept-new "$@" ``` This wrapper is passed to SSHFS via the `-o ssh_command=` option. *** ## Security Model ### Authentication Layers 1. **Cloud Provider API** * Bearer token authentication * Token stored in env var or config file * Config file permissions: `0600` 2. **SSH** * Ed25519 key pair (or RSA) * Public key registered with provider * Private key protected by filesystem permissions * Optional passphrase (requires ssh-agent) 3. **Agent API** * Token-based authentication via `X-Spuff-Token` header * Server binds to localhost only (127.0.0.1) * Token passed via env var to agent service ### Network Security ```mermaid flowchart LR subgraph internet["Internet"] client["Client"] end subgraph vm["VM"] ssh["Port 22
(SSH)"] mosh["UDP 60000-61000
(Mosh)"] agent["Port 7575
(Agent)"] end client -->|"Authenticated access"| ssh client -->|"Authenticated via
SSH handshake"| mosh agent -.-|"localhost only
not exposed"| agent style agent fill:#f9f,stroke:#333,stroke-dasharray: 5 5 ``` ### VM Security Hardening From cloud-init: * Root SSH login disabled (`disable_root: true`) * Password authentication disabled (`ssh_pwauth: false`) * User password locked (`lock_passwd: true`) * Non-root user with sudo access * Only SSH key authentication allowed ### Sensitive Data Handling | Data | Storage | Protection | | --------------- | ---------------------- | ----------------------- | | API Token | env var or config.yaml | File permissions (0600) | | SSH Private Key | \~/.ssh/id\_\* | File permissions (0600) | | Agent Token | env var | Process environment | | State DB | \~/.spuff/chrondb/ | File permissions | *** ## Error Handling ### Structured Provider Errors Provider operations use structured `ProviderError` types for proper error handling and recovery strategies: ```rust pub enum ProviderError { // Authentication issues - token invalid, expired, etc. Authentication { provider: String, message: String }, // Rate limiting - includes optional retry-after duration RateLimit { retry_after: Option }, // Resource not found - instance, snapshot, etc. NotFound { resource_type: String, id: String }, // Quota exceeded - droplet limit, snapshot limit, etc. QuotaExceeded { resource: String, message: String }, // Operation timeout with elapsed time Timeout { operation: String, elapsed: Duration }, // Generic API error with HTTP status Api { status: u16, message: String }, } // Check if error is retryable impl ProviderError { pub fn is_retryable(&self) -> bool { matches!(self, Self::RateLimit { .. } | Self::Timeout { .. } | Self::Network(_)) } pub fn retry_after(&self) -> Option { match self { Self::RateLimit { retry_after } => *retry_after, Self::Timeout { .. } => Some(Duration::from_secs(5)), Self::Network(_) => Some(Duration::from_secs(2)), _ => None, } } } ``` ### SSH Errors The system provides clear error messages for common SSH issues: ```rust // Key requires passphrase but ssh-agent not running if stderr.contains("Permission denied") || stderr.contains("passphrase") { return Err(SpuffError::Ssh( "SSH key requires passphrase. Run 'ssh-add' first." )); } ``` ### Provider API Errors Provider API calls use structured errors that can be matched for specific handling: ```rust match provider.create_instance(&request).await { Ok(instance) => { /* success */ }, Err(ProviderError::Authentication { .. }) => { // Invalid token - prompt user to check credentials }, Err(ProviderError::QuotaExceeded { resource, .. }) => { // Quota reached - suggest cleanup or upgrade }, Err(ProviderError::RateLimit { retry_after }) => { // Rate limited - wait and retry if let Some(duration) = retry_after { tokio::time::sleep(duration).await; } }, Err(e) => { // Other errors - propagate return Err(e.into()); } } ``` ### Timeout Handling Operations have configurable timeouts: | Operation | Default Timeout | | ------------------ | --------------- | | Provider API calls | 30s | | SSH port wait | 300s (5 min) | | SSH login wait | 120s (2 min) | | Cloud-init wait | 600s (10 min) | | Agent exec command | 30s | *** ## Extending Spuff ### Adding a New Provider Spuff uses a **Registry Pattern** for providers, making it easy to add new cloud providers without modifying existing code. #### Step 1: Create Provider Module Create `src/provider/.rs` (e.g., `src/provider/hetzner.rs`). #### Step 2: Implement the Provider Trait ```rust use async_trait::async_trait; use crate::provider::{ ImageSpec, InstanceRequest, Provider, ProviderInstance, ProviderResult, ProviderTimeouts, Snapshot, }; pub struct HetznerProvider { client: reqwest::Client, token: String, timeouts: ProviderTimeouts, } #[async_trait] impl Provider for HetznerProvider { fn name(&self) -> &'static str { "hetzner" } async fn create_instance(&self, request: &InstanceRequest) -> ProviderResult { // Convert ImageSpec to Hetzner image ID let image = self.resolve_image(&request.image)?; // Call Hetzner API... } async fn destroy_instance(&self, id: &str) -> ProviderResult<()> { ... } async fn get_instance(&self, id: &str) -> ProviderResult> { ... } async fn list_instances(&self) -> ProviderResult> { ... } async fn wait_ready(&self, id: &str) -> ProviderResult { ... } async fn create_snapshot(&self, instance_id: &str, name: &str) -> ProviderResult { ... } async fn list_snapshots(&self) -> ProviderResult> { ... } async fn delete_snapshot(&self, id: &str) -> ProviderResult<()> { ... } } ``` #### Step 3: Implement the ProviderFactory Trait ```rust use crate::provider::{ ProviderFactory, ProviderResult, ProviderTimeouts, ProviderType, }; pub struct HetznerFactory; impl ProviderFactory for HetznerFactory { fn provider_type(&self) -> ProviderType { ProviderType::Hetzner } fn create(&self, token: &str, timeouts: ProviderTimeouts) -> ProviderResult> { if token.is_empty() { return Err(ProviderError::auth("hetzner", "API token is required")); } Ok(Box::new(HetznerProvider::new(token, timeouts)?)) } } ``` #### Step 4: Register the Provider In `src/provider/registry.rs`, add your factory to the default registration: ```rust pub fn register_defaults(&mut self) { use super::digitalocean::DigitalOceanFactory; use super::hetzner::HetznerFactory; // Add this self.register(DigitalOceanFactory); self.register(HetznerFactory); // Add this } ``` #### Step 5: Add Provider Type In `src/provider/config.rs`, add your provider to the enum: ```rust #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] pub enum ProviderType { DigitalOcean, Hetzner, // Add this Aws, // ... } impl ProviderType { pub fn from_str(s: &str) -> Option { match s.to_lowercase().as_str() { "digitalocean" | "do" => Some(Self::DigitalOcean), "hetzner" => Some(Self::Hetzner), // Add this // ... } } pub fn is_implemented(&self) -> bool { matches!(self, Self::DigitalOcean | Self::Hetzner) // Add here } } ``` ### Key Types for Provider Implementation **InstanceRequest** - Provider-agnostic instance configuration: ```rust pub struct InstanceRequest { pub name: String, pub region: String, pub size: String, pub image: ImageSpec, pub user_data: Option, pub labels: HashMap, } ``` **ImageSpec** - Provider-agnostic image specification: ```rust pub enum ImageSpec { Ubuntu(String), // e.g., "24.04" Debian(String), // e.g., "12" Custom(String), // Provider-specific image ID Snapshot(String), // Snapshot ID } ``` **ProviderInstance** - Instance returned by provider operations: ```rust pub struct ProviderInstance { pub id: String, pub ip: IpAddr, pub status: InstanceStatus, pub created_at: DateTime, } ``` **ProviderError** - Structured error types for proper handling: ```rust pub enum ProviderError { Authentication { provider: String, message: String }, RateLimit { retry_after: Option }, NotFound { resource_type: String, id: String }, QuotaExceeded { resource: String, message: String }, Timeout { operation: String, elapsed: Duration }, Api { status: u16, message: String }, // ... } ``` ### Adding Agent Endpoints 1. Add route in `src/agent/routes.rs`: ```rust pub fn create_routes() -> Router> { Router::new() // ... existing routes ... .route("/custom", get(custom_endpoint)) } async fn custom_endpoint( AuthenticatedState(state): AuthenticatedState, ) -> impl IntoResponse { state.update_activity().await; Json(serde_json::json!({ "custom": "data" })) } ``` *** ## Debugging ### Enable Debug Logging ```bash RUST_LOG=debug spuff up ``` ### Inspect Cloud-Init ```bash # On the VM sudo cat /var/log/cloud-init-output.log sudo cloud-init status --format=json cat /opt/spuff/bootstrap.status ``` ### Agent Status ```bash # On the VM sudo systemctl status spuff-agent sudo journalctl -u spuff-agent -f curl -H "X-Spuff-Token: $TOKEN" http://127.0.0.1:7575/status ``` ### Local State ```bash # ChronDB stores data as JSON files in a Git-backed structure ls ~/.spuff/chrondb/data/instance/ cat ~/.spuff/chrondb/data/meta/meta_COLON_active.json ``` # Configuration Reference This document describes all configuration options available in spuff's `config.yaml` file. ## File Location The configuration file is located at: ``` ~/.spuff/config.yaml ``` To create or edit the configuration: ```bash spuff init # Interactive setup (creates config.yaml) spuff config show # Display current configuration spuff config edit # Open in $EDITOR spuff config set # Set individual values ``` ## Complete Example ```yaml # Cloud provider configuration provider: digitalocean region: nyc1 size: s-2vcpu-4gb # VM lifecycle idle_timeout: 2h environment: devbox # SSH configuration ssh_key_path: ~/.ssh/id_ed25519 ssh_user: dev # Optional: Dotfiles repository dotfiles: https://github.com/yourusername/dotfiles # Optional: Tailscale VPN tailscale_enabled: false tailscale_authkey: tskey-auth-xxxxx # Optional: Agent authentication agent_token: your-secret-token ``` *** ## Configuration Options ### `provider` **Type:** `string` **Required:** Yes **Default:** `digitalocean` The cloud provider to use for creating VMs. ```yaml provider: digitalocean ``` **Supported values:** | Provider | Status | Description | | -------------- | ------- | --------------------- | | `digitalocean` | Stable | DigitalOcean Droplets | | `hetzner` | Planned | Hetzner Cloud | | `aws` | Planned | Amazon EC2 | *** ### `region` **Type:** `string` **Required:** Yes **Default:** `nyc1` The geographic region where your VM will be created. Lower latency = faster connection. ```yaml region: nyc1 ``` **DigitalOcean regions:** | Region | Location | | -------------- | ---------------------- | | `nyc1`, `nyc3` | New York, USA | | `sfo3` | San Francisco, USA | | `ams3` | Amsterdam, Netherlands | | `sgp1` | Singapore | | `lon1` | London, UK | | `fra1` | Frankfurt, Germany | | `tor1` | Toronto, Canada | | `blr1` | Bangalore, India | | `syd1` | Sydney, Australia | **Tip:** Choose the region closest to you for best performance. **Override at runtime:** ```bash spuff up --region fra1 ``` *** ### `size` **Type:** `string` **Required:** Yes **Default:** `s-2vcpu-4gb` The VM size/type determining CPU, memory, and cost. ```yaml size: s-2vcpu-4gb ``` **DigitalOcean sizes:** | Size | vCPUs | Memory | Disk | Price/hour | | -------------- | ----- | ------ | ------ | ---------- | | `s-1vcpu-1gb` | 1 | 1 GB | 25 GB | $0.009 | | `s-1vcpu-2gb` | 1 | 2 GB | 50 GB | $0.018 | | `s-2vcpu-2gb` | 2 | 2 GB | 60 GB | $0.027 | | `s-2vcpu-4gb` | 2 | 4 GB | 80 GB | $0.036 | | `s-4vcpu-8gb` | 4 | 8 GB | 160 GB | $0.071 | | `s-8vcpu-16gb` | 8 | 16 GB | 320 GB | $0.143 | **Recommended:** * Light development: `s-2vcpu-4gb` (default) * Heavy builds/Claude Code: `s-4vcpu-8gb` * CI/testing: `s-1vcpu-2gb` **Override at runtime:** ```bash spuff up --size s-4vcpu-8gb ``` *** ### `idle_timeout` **Type:** `string` (duration) **Required:** Yes **Default:** `2h` Time of inactivity after which the VM will be automatically destroyed. This prevents forgotten instances and surprise bills. ```yaml idle_timeout: 2h ``` **Duration formats:** | Format | Example | Description | | ------ | ------- | --------------- | | `Nh` | `2h` | N hours | | `Nm` | `30m` | N minutes | | `Ns` | `3600s` | N seconds | | `N` | `7200` | N seconds (raw) | **Examples:** ```yaml idle_timeout: 30m # 30 minutes idle_timeout: 2h # 2 hours (recommended) idle_timeout: 4h # 4 hours idle_timeout: 24h # 24 hours (use with caution) ``` **How it works:** 1. The `spuff-agent` running on the VM monitors activity 2. Activity includes: SSH sessions, CPU usage, network traffic 3. When no activity for the configured duration, the VM self-destructs 4. Snapshots can be created automatically before destruction *** ### `environment` **Type:** `string` **Required:** Yes **Default:** `devbox` The base environment type that determines what tools are pre-installed. ```yaml environment: devbox ``` **Supported values:** | Environment | Description | | ----------- | -------------------------------------------------- | | `devbox` | Modern shell (zsh), Docker, Git, development tools | | `nix` | Nix package manager environment (planned) | | `minimal` | Bare minimum (planned) | *** ### `ssh_key_path` **Type:** `string` (file path) **Required:** Yes **Default:** `~/.ssh/id_ed25519` Path to your SSH private key file. This key is used to: 1. Authenticate with the cloud provider (public key registered) 2. Connect to the VM via SSH 3. Forward your SSH agent for git operations ```yaml ssh_key_path: ~/.ssh/id_ed25519 ``` **Supported key types:** * `id_ed25519` (recommended) * `id_rsa` * `id_ecdsa` **Notes:** * The path supports `~` expansion * The corresponding `.pub` file must exist * The public key is registered with your cloud provider **Examples:** ```yaml ssh_key_path: ~/.ssh/id_ed25519 # Default ssh_key_path: ~/.ssh/spuff_key # Custom key ssh_key_path: /home/user/.ssh/id_rsa # Absolute path ``` *** ### `ssh_user` **Type:** `string` **Required:** No **Default:** `dev` The SSH username for connecting to the VM. This user is created with passwordless sudo access. ```yaml ssh_user: dev ``` **Notes:** * Default is `dev` (non-root for security) * User is created with `sudo` and `docker` groups * Root SSH login is disabled by default * Password authentication is disabled (SSH key only) *** ### `dotfiles` **Type:** `string` (URL) or `null` **Required:** No **Default:** `null` (not set) URL to a git repository containing your dotfiles. When set, spuff will clone and apply your dotfiles on VM creation. ```yaml dotfiles: https://github.com/yourusername/dotfiles ``` **Supported formats:** ```yaml # HTTPS (recommended for public repos) dotfiles: https://github.com/user/dotfiles # SSH (requires SSH agent forwarding) dotfiles: git@github.com:user/dotfiles.git ``` **How it works:** 1. Repository is cloned to `~/dotfiles` 2. If `install.sh` exists, it's executed 3. If `Makefile` exists with `install` target, `make install` is run 4. Otherwise, symlinks are created for common dotfiles **Tip:** Keep your dotfiles repo lightweight for faster VM startup. *** ### `tailscale_enabled` **Type:** `boolean` **Required:** No **Default:** `false` Enable Tailscale VPN integration. When enabled, your VM joins your Tailscale network for secure private access. ```yaml tailscale_enabled: true ``` **Benefits:** * Access VM via private Tailscale IP (no public exposure) * Persistent hostname across VM recreations * Secure mesh networking * Access from anywhere without port forwarding *** ### `tailscale_authkey` **Type:** `string` or `null` **Required:** No (required if `tailscale_enabled: true`) **Default:** `null` (not set) Your Tailscale authentication key for automatic VM enrollment. ```yaml tailscale_authkey: tskey-auth-xxxxxxxxxxxxx ``` **Getting an auth key:** 1. Go to [Tailscale Admin Console](https://login.tailscale.com/admin/settings/keys) 2. Click "Generate auth key" 3. Settings: * Reusable: Yes (for multiple VMs) * Ephemeral: Yes (auto-removes when VM destroyed) * Pre-authorized: Optional 4. Copy the key (starts with `tskey-auth-`) **Alternative:** Use environment variable ```bash export TS_AUTHKEY="tskey-auth-xxxxxxxxxxxxx" ``` *** ### `agent_token` **Type:** `string` or `null` **Required:** No **Default:** `null` (auto-generated) Authentication token for the spuff-agent API running on the VM. Protects the agent's HTTP endpoints from unauthorized access. ```yaml agent_token: your-secret-token-here ``` **How it works:** 1. If set, all agent API requests require `X-Spuff-Token` header 2. The CLI automatically includes this token in requests 3. If not set, a random token is generated during VM creation **Alternative:** Use environment variable ```bash export SPUFF_AGENT_TOKEN="your-secret-token" ``` **Security note:** The agent token protects endpoints that expose system metrics and can execute commands. Always use a strong, unique token. *** ## Environment Variables API tokens and secrets can be provided via environment variables instead of (or in addition to) the config file: | Variable | Description | Priority | | -------------------- | -------------------------- | ----------------- | | `SPUFF_API_TOKEN` | Cloud provider API token | Highest | | `DIGITALOCEAN_TOKEN` | DigitalOcean API token | Provider-specific | | `HETZNER_TOKEN` | Hetzner API token | Provider-specific | | `AWS_ACCESS_KEY_ID` | AWS access key | Provider-specific | | `SPUFF_AGENT_TOKEN` | Agent authentication token | Override config | | `TS_AUTHKEY` | Tailscale auth key | Override config | **Priority:** Environment variables take precedence over config file values. **Example setup:** ```bash # Add to ~/.bashrc or ~/.zshrc export DIGITALOCEAN_TOKEN="dop_v1_xxxxxxxxxxxxxxxx" export SPUFF_AGENT_TOKEN="my-secret-agent-token" ``` *** ## CLI Commands ### View Configuration ```bash spuff config show ``` Output: ``` Current Configuration Provider: digitalocean Region: nyc1 Size: s-2vcpu-4gb Idle timeout: 2h Environment: devbox Dotfiles: https://github.com/user/dotfiles SSH key: ~/.ssh/id_ed25519 Tailscale: enabled Config file: /home/user/.config/spuff/config.yaml ``` ### Set Individual Values ```bash spuff config set region fra1 spuff config set size s-4vcpu-8gb spuff config set idle_timeout 4h spuff config set dotfiles https://github.com/user/dotfiles spuff config set tailscale true ``` **Available keys:** * `provider` * `region` * `size` * `idle_timeout` (or `idle-timeout`) * `environment` * `dotfiles` * `ssh_key` (or `ssh-key`) * `ssh_user` (or `ssh-user`) * `tailscale` ### Edit in Editor ```bash spuff config edit ``` Opens the config file in your `$EDITOR` (defaults to `vim`). *** ## Runtime Overrides Many configuration values can be overridden at runtime: ```bash # Override size and region for this run only spuff up --size s-4vcpu-8gb --region fra1 # Use a specific snapshot spuff up --snapshot snap-123456 # Create without connecting spuff up --no-connect # Development mode (upload local agent) spuff up --dev ``` *** ## File Permissions The config file is created with restricted permissions (`0600`) to protect sensitive data like tokens. If you edit the file manually, ensure proper permissions: ```bash chmod 600 ~/.spuff/config.yaml ``` *** ## Example Configurations ### Minimal (default) ```yaml provider: digitalocean region: nyc1 size: s-2vcpu-4gb idle_timeout: 2h environment: devbox ssh_key_path: ~/.ssh/id_ed25519 ssh_user: dev ``` ### Power User ```yaml provider: digitalocean region: fra1 size: s-4vcpu-8gb idle_timeout: 4h environment: devbox ssh_key_path: ~/.ssh/id_ed25519 ssh_user: dev dotfiles: https://github.com/myuser/dotfiles tailscale_enabled: true tailscale_authkey: tskey-auth-xxxxxxxxxxxxx agent_token: super-secret-token-12345 ``` ### CI/Testing ```yaml provider: digitalocean region: nyc1 size: s-1vcpu-2gb idle_timeout: 30m environment: devbox ssh_key_path: ~/.ssh/ci_key ssh_user: ci ``` *** ## Troubleshooting ### Config not found ``` Error: Config file not found: ~/.spuff/config.yaml. Run 'spuff init' first. ``` **Solution:** Run `spuff init` to create the configuration file. ### Invalid config ``` Error: Invalid config: missing field `region` ``` **Solution:** Add the missing field or run `spuff init` to regenerate. ### Token not found ``` Error: API token not configured ``` **Solution:** Either: 1. Add `api_token` to config.yaml (not recommended) 2. Set environment variable: `export DIGITALOCEAN_TOKEN="..."` 3. Set generic variable: `export SPUFF_API_TOKEN="..."` ### SSH key not found ``` Error: SSH key not found at ~/.ssh/id_ed25519 ``` **Solution:** 1. Generate a key: `ssh-keygen -t ed25519` 2. Or update `ssh_key_path` to point to existing key # Project Configuration (spuff.yaml) spuff supports per-project configuration via a `spuff.yaml` file in your project root. This enables **environment as code** - defining your development environment declaratively alongside your source code. > **See also:** [Spuff Specification](/spec) for the formal specification with validation rules and conformance requirements. ## Overview When you run `spuff up` in a directory containing a `spuff.yaml` file, spuff will: 1. Read the project configuration 2. Apply any resource overrides (size, region) 3. Provision the VM with the project config embedded 4. Install bundles, packages, and services automatically 5. Clone repositories and run setup scripts ## File Location Place `spuff.yaml` in your project root (same directory as your git repository). spuff will search up the directory tree to find it. ``` my-project/ ├── spuff.yaml # Project configuration ├── spuff.secrets.yaml # Secrets (add to .gitignore!) ├── docker-compose.yaml # Services (optional) └── src/ ``` ## Complete Example ```yaml # spuff.yaml - Project environment configuration version: "1" # Project name (default: directory name) name: my-awesome-project # Override VM resources resources: size: s-4vcpu-8gb region: nyc1 # Language bundles - pre-configured toolchains bundles: - rust # rustup, cargo, rust-analyzer, clippy - node # nodejs, npm, typescript, eslint - python # python3, pip, uv, ruff, pyright # Additional system packages packages: - postgresql-client - redis-tools - protobuf-compiler # Docker services (uses docker-compose.yaml) services: enabled: true compose_file: docker-compose.yaml profiles: [dev] # Repositories to clone repositories: - owner/repo # Short format (GitHub) - url: git@github.com:org/backend.git # Full format path: ~/projects/backend branch: develop # Environment variables env: DATABASE_URL: postgres://dev:dev@localhost:5432/mydb REDIS_URL: redis://localhost:6379 RUST_LOG: debug # Setup scripts (run in order) setup: - cargo build - npm install - ./scripts/init-db.sh # Ports for SSH tunneling ports: - 3000 # Frontend - 8080 # Backend API - 5432 # Postgres # AI coding tools (default: all) ai_tools: all # all | none | list of tools # ai_tools: # - claude-code # - copilot # Volume mounts (SSHFS-based bidirectional sync) volumes: - source: ./data # Local directory to sync target: ~/data # Remote directory on VM - source: ./src target: ~/project/src mount_point: ./src # Mount remote back to local (bidirectional) # Lifecycle hooks hooks: post_up: | echo "Environment ready!" make dev-setup pre_down: | make db-backup ``` *** ## Configuration Reference ### `version` **Type:** `string` **Default:** `"1"` Spec version for future compatibility. ```yaml version: "1" ``` *** ### `name` **Type:** `string` (optional) **Default:** Directory name Custom name for the environment. ```yaml name: my-project ``` *** ### `resources` Override global VM configuration. CLI flags take precedence over project config. ```yaml resources: size: s-4vcpu-8gb # VM size region: fra1 # Region ``` **Precedence:** CLI flags > spuff.yaml > \~/.spuff/config.yaml *** ### `bundles` Pre-configured language toolchains. Each bundle installs the compiler/runtime plus essential development tools (LSPs, linters, formatters). ```yaml bundles: - rust - go - python ``` **Available bundles:** | Bundle | Includes | | -------- | ---------------------------------------------------- | | `rust` | rustup, cargo, rust-analyzer, clippy, rustfmt, mold | | `go` | go, gopls, delve, golangci-lint, air | | `python` | python3.12, pip, venv, uv, ruff, pyright, ipython | | `node` | node 22 LTS, npm, pnpm, typescript, eslint, prettier | | `elixir` | erlang/OTP, elixir, mix, elixir-ls, phoenix | | `java` | openjdk 21, maven, gradle, jdtls | | `zig` | zig, zls | | `cpp` | gcc, clang, cmake, ninja, clangd, gdb, lldb | | `ruby` | ruby, bundler, solargraph, rubocop | *** ### `packages` Additional system packages to install via apt. ```yaml packages: - postgresql-client - redis-tools - libssl-dev - protobuf-compiler ``` *** ### `services` Docker services configuration. Uses your project's `docker-compose.yaml`. ```yaml services: enabled: true # Default: true compose_file: docker-compose.yaml # Default: docker-compose.yaml profiles: [dev, debug] # Optional compose profiles ``` **Note:** spuff doesn't duplicate docker-compose configuration - it uses your existing compose file. *** ### `repositories` Clone additional repositories into the environment. ```yaml repositories: # Short format (GitHub) - owner/repo # Full format - url: git@github.com:org/backend.git path: ~/projects/backend # Default: ~/projects/ branch: develop # Optional # HTTPS format - url: https://github.com/org/shared-libs.git ``` **SSH Agent Forwarding:** spuff uses SSH agent forwarding, so your local SSH keys work for cloning private repos. *** ### `env` Environment variables set on the VM. ```yaml env: DATABASE_URL: postgres://localhost:5432/mydb REDIS_URL: redis://localhost:6379 DEBUG: "true" ``` **Variable resolution:** References to `$VAR`, `${VAR}`, or `${VAR:-default}` are resolved from your local environment before being sent to the VM. ```yaml env: # Resolved from local $DATABASE_PASSWORD DATABASE_PASSWORD: $DATABASE_PASSWORD # With default value LOG_LEVEL: ${LOG_LEVEL:-info} ``` *** ### `setup` Shell commands executed after bundles and packages are installed. ```yaml setup: - cargo build --release - npm install - ./scripts/init-db.sh ``` Scripts are executed in order. If any script fails, subsequent scripts are skipped. *** ### `ports` Ports for automatic SSH tunneling. When you run `spuff ssh`, these ports are forwarded from your local machine to the VM. ```yaml ports: - 3000 # localhost:3000 -> vm:3000 - 8080 # localhost:8080 -> vm:8080 - 5432 # localhost:5432 -> vm:5432 ``` This allows you to work "locally" (browser, IDE) while connected to the remote VM. *** ### `ai_tools` **Type:** `string` or `list` (optional) **Default:** `all` Controls which AI coding tools are installed on the VM. ```yaml # Install all tools (default) ai_tools: all # Disable all AI tools ai_tools: none # Install specific tools only ai_tools: - claude-code - copilot ``` **Available tools:** | Tool | Package | Description | | ------------- | --------------------------- | ------------------------------- | | `claude-code` | `@anthropic-ai/claude-code` | Anthropic's Claude Code CLI | | `codex` | `@openai/codex` | OpenAI Codex CLI | | `opencode` | `opencode-ai` | Open-source AI coding assistant | | `copilot` | `@github/copilot` | GitHub Copilot CLI | **CLI override:** `spuff up --ai-tools claude-code,copilot` **Precedence:** CLI `--ai-tools` > spuff.yaml > \~/.spuff/config.yaml > default (all) See [AI Tools documentation](/) for authentication and configuration details. *** ### `volumes` Mount remote VM directories locally via SSHFS for bidirectional file editing. ```yaml volumes: # Basic: sync local to remote - source: ./data target: ~/data # Bidirectional: mount remote over local for real-time editing - source: ./src target: ~/project/src mount_point: ./src # Optional: mount remote back here # With explicit mount point - source: ./config target: /etc/myapp mount_point: ~/.local/share/spuff/mounts/myapp-config ``` **Fields:** | Field | Required | Description | | ------------- | -------- | --------------------------------------------------------- | | `source` | Yes | Local directory path (relative to spuff.yaml or absolute) | | `target` | Yes | Remote directory path on the VM | | `mount_point` | No | Where to mount remote directory locally | **Mount Point Resolution:** 1. If `mount_point` is specified, use it 2. If only `source` is specified, mount over `source` for bidirectional editing 3. Otherwise, auto-generate under `~/.local/share/spuff/mounts//` **Behavior during `spuff up`:** 1. Remote directory is created on the VM 2. Local `source` is synced to remote `target` via rsync 3. Remote `target` is mounted locally via SSHFS **Behavior during `spuff down`:** 1. All mounted volumes are force-unmounted before VM destruction 2. This prevents SSHFS from hanging when the remote server disappears **Requirements:** * macOS: [macFUSE](https://osxfuse.github.io/) and `sshfs` (`brew install macfuse sshfs`) * Linux: `fuse` and `sshfs` packages **CLI Commands:** ```bash spuff volume mount # Mount all configured volumes spuff volume unmount # Unmount all volumes spuff volume ls # List volume status ``` *** ### `hooks` Lifecycle scripts for custom automation. ```yaml hooks: # Runs after environment is fully ready post_up: | echo "Environment ready!" make dev-setup # Runs before VM destruction pre_down: | make db-backup > /tmp/backup.sql ``` *** ## Secrets Management ### spuff.secrets.yaml Store sensitive values in a separate file that's **not committed to git**: ```yaml # spuff.secrets.yaml env: DATABASE_PASSWORD: super-secret API_KEY: sk-xxx AWS_SECRET_ACCESS_KEY: xxxxx ``` Add to `.gitignore`: ```gitignore spuff.secrets.yaml ``` **Merge behavior:** `spuff.secrets.yaml` is merged with `spuff.yaml`, with secrets taking precedence. ### Environment Variable Resolution Reference local environment variables in your config: ```yaml env: # Simple reference API_KEY: $API_KEY # With braces SECRET: ${DATABASE_SECRET} # With default value LOG_LEVEL: ${LOG_LEVEL:-debug} ``` *** ## CLI Integration ### `spuff up` When `spuff.yaml` is present: ``` $ spuff up Creating instance: my-project-abc123 Provider: digitalocean Region: nyc1 Size: s-4vcpu-8gb (from spuff.yaml) [1/5] Creating instance................ ✓ [2/5] Waiting for IP................... ✓ 167.99.123.45 [3/5] Waiting for SSH.................. ✓ [4/5] Running bootstrap................ ✓ [5/5] Agent ready...................... ✓ Project Setup (spuff.yaml) The following will be installed by spuff-agent: Bundles: rust, node, python Packages: postgresql-client, redis-tools Services: docker-compose.yaml (2 services) Repositories: 3 repos to clone Ports: 3000, 8080, 5432 (tunnel via `spuff ssh`) Setup scripts: 3 commands Run `spuff status --detailed` to track progress ✓ Instance ready! ``` ### `spuff status --detailed` Shows project setup progress: ``` $ spuff status --detailed ● my-project-abc123 (167.99.123.45) Provider digitalocean Region nyc1 Size s-4vcpu-8gb Uptime 2h 15m Bootstrap ready ╭────────────────────────────────────────────────────────╮ │ Project Setup (spuff.yaml) │ ├────────────────────────────────────────────────────────┤ │ Bundles │ │ [✓] rust (1.78.0) │ │ [✓] node (22.0.0) │ │ [>] python (installing...) │ │ Packages │ │ [✓] 2 installed │ │ Services (docker-compose.yaml) │ │ [✓] postgres (5432) - running │ │ [✓] redis (6379) - running │ │ Repositories │ │ [✓] backend → ~/projects/backend │ │ [>] frontend (cloning...) │ │ Setup Scripts │ │ [ ] #1 cargo build --release │ │ [ ] #2 npm install │ ╰────────────────────────────────────────────────────────╯ ``` ### `spuff logs` View project setup logs: ```bash spuff logs # General setup log spuff logs --bundle rust # Rust bundle installation spuff logs --packages # Package installation spuff logs --repos # Repository cloning spuff logs --services # Docker services spuff logs --script 1 # Setup script #1 spuff logs -f # Follow mode (tail -f) ``` ### `spuff ssh` Connects with automatic port tunneling: ``` $ spuff ssh ╭──────────────────────────────────────────────────────────╮ │ SSH Tunnels (from spuff.yaml) │ │ localhost:3000 → vm:3000 │ │ localhost:8080 → vm:8080 │ │ localhost:5432 → vm:5432 │ ╰──────────────────────────────────────────────────────────╯ Connecting to my-project-abc123 (167.99.123.45)... dev@my-project-abc123:~$ ``` *** ## Logging All project setup activities are logged to `/var/log/spuff/` on the VM: ``` /var/log/spuff/ ├── setup.log # General setup log ├── bundles/ │ ├── rust.log # Rust bundle installation │ ├── node.log # Node bundle installation │ └── python.log # Python bundle installation ├── packages.log # apt package installation ├── repositories.log # Git clone operations ├── services.log # Docker compose logs └── scripts/ ├── 001.log # First setup script ├── 002.log # Second setup script └── 003.log # Third setup script ``` *** ## Agent API Endpoints The spuff-agent exposes these endpoints for project management: | Endpoint | Method | Description | | ----------------- | ------ | ----------------------------- | | `/project/config` | GET | Current project configuration | | `/project/status` | GET | Detailed setup progress | | `/project/setup` | POST | Start project setup | These are used internally by the CLI but can be accessed directly via SSH tunnel. *** ## Best Practices 1. **Keep spuff.yaml in version control** - The environment becomes reproducible 2. **Use spuff.secrets.yaml for secrets** - Never commit credentials 3. **Prefer bundles over individual packages** - They include LSPs and dev tools 4. **Use `docker-compose.yaml` for services** - Don't duplicate config 5. **Test your setup scripts locally first** - Saves provisioning time 6. **Use specific versions in setup scripts** - Avoid "works on my machine" issues *** ## Troubleshooting ### Project config not detected ``` No spuff.yaml found in current directory or parents ``` **Solution:** Ensure `spuff.yaml` is in your project root and you're running `spuff up` from within the project directory. ### Bundle installation failed ```bash spuff logs --bundle rust ``` Check the bundle-specific log for errors. Common issues: * Network connectivity * Disk space * Package conflicts ### Services not starting ```bash spuff logs --services ``` Ensure your `docker-compose.yaml` is valid and doesn't require manual configuration. ### Setup script failed ```bash spuff logs --script 1 ``` View the specific script's output. The script runs in the user's home directory by default. # Security Model This document describes spuff's security architecture, threat model, and security considerations. ## Overview Spuff handles sensitive data including: * Cloud provider API tokens * SSH private keys (by reference) * Agent authentication tokens * User data in cloud-init The security model is designed around: 1. **Minimal exposure**: Only expose what's necessary 2. **Short-lived access**: Ephemeral VMs limit attack window 3. **No secret storage on VMs**: Keys stay local via SSH forwarding 4. **Defense in depth**: Multiple security layers ## Architecture Security ```mermaid flowchart TB subgraph trusted["User's Machine (Trusted)"] token["API Token
(env var)"] sshkeys["SSH Keys
(filesystem)
0600 perms"] state[("Local State
~/.spuff/state.db")] end subgraph semitrust["Cloud VM (Semi-trusted)"] providerapi["Provider API
(external)"] sshserver["SSH Server
Port 22"] agent["spuff-agent
Port 7575 (localhost only)
Token auth required"] note["No private keys stored
No API tokens stored
Ephemeral by design"] end token -->|"HTTPS (TLS)"| providerapi sshkeys -->|"SSH Agent Forwarding
(keys never leave machine)"| sshserver sshkeys -->|"SSH (encrypted)"| sshserver style note fill:#f5f5f5,stroke:#ccc,stroke-dasharray: 5 5 ``` ## Threat Model ### Assets to Protect | Asset | Location | Protection | | --------------- | -------------- | --------------------------------- | | Cloud API token | User's machine | Env var, never stored in files | | SSH private key | User's machine | Filesystem permissions, ssh-agent | | Agent token | VM env var | Per-session generation | | User code | VM | Ephemeral, user responsibility | ### Threat Actors 1. **External attackers**: Internet-based attacks 2. **Compromised VM**: Malicious code on VM 3. **Man-in-the-middle**: Network interception 4. **Insider threat**: Shared machine access ### Attack Scenarios #### Scenario 1: VM Compromise **Threat**: Attacker gains root on VM **Impact without mitigations**: * Could use SSH agent to sign requests (while user connected) * Could read any data on VM * Could attack other systems from VM **Mitigations**: * SSH agent forwarding is session-scoped * No persistent credentials on VM * Ephemeral nature limits exposure window * Agent binds to localhost only #### Scenario 2: API Token Theft **Threat**: Attacker obtains cloud API token **Impact**: * Could create/destroy instances * Could access other resources in account * Cost impact from resource creation **Mitigations**: * Token stored in env var, not file * Recommended: Use scoped tokens where possible * Regular token rotation #### Scenario 3: Network Interception **Threat**: MITM attack on communications **Impact without mitigations**: * Could intercept API calls * Could hijack SSH sessions **Mitigations**: * Provider API uses TLS * SSH provides end-to-end encryption * Host key verification (accept-new policy) ## Security Controls ### Authentication #### Cloud Provider API ``` Authorization: Bearer ``` * Token stored in environment variable * Never written to config files * Transmitted over TLS only #### SSH Access * Ed25519 or RSA key pairs * No password authentication * Key-based auth only * Agent forwarding for git operations #### Agent API ``` X-Spuff-Token: ``` * Token generated per VM creation * Passed via systemd environment * Required for all authenticated endpoints * Agent binds to 127.0.0.1 only ### Network Security #### Exposed Ports | Port | Service | Exposure | Notes | | ---- | ------- | --------- | ------------------- | | 22 | SSH | Public | Key auth only | | 7575 | Agent | Localhost | Token auth required | #### Firewall Recommendations * Only allow SSH from known IPs if possible * No other inbound ports required * Outbound: Allow HTTPS for package updates ### VM Hardening Cloud-init applies these hardening measures: ```yaml # Disable root SSH login disable_root: true # Disable password auth ssh_pwauth: false # Lock user password users: - name: dev lock_passwd: true # Sudo without password (dev convenience) sudo: ["ALL=(ALL) NOPASSWD:ALL"] ``` ### Data Protection #### At Rest | Data | Storage | Protection | | -------- | --------------------- | ------------------- | | Config | \~/.spuff/config.yaml | 0600 permissions | | State | \~/.spuff/state.db | Standard file perms | | SSH keys | \~/.ssh/ | 0600 permissions | #### In Transit | Channel | Protection | | ------------ | ----------------------- | | Provider API | TLS 1.2+ | | SSH | SSH protocol encryption | | Cloud-init | Base64 (not encryption) | ### Secrets Management #### Do's * Store API tokens in environment variables * Use ssh-agent for key management * Rotate tokens regularly * Use scoped tokens when possible #### Don'ts * Never commit tokens to git * Never store tokens in config files * Never copy private keys to VMs * Never disable SSH key verification ## Security Considerations ### SSH Agent Forwarding Risks **Risk**: A root process on the VM can use the forwarded agent while you're connected. **Mitigations**: * Keep SSH sessions short * Use `ssh-add -c` for confirmation prompts * Consider separate keys for development * VM is ephemeral, limiting exposure window **Recommendation for sensitive operations**: ```bash # Add key with confirmation ssh-add -c ~/.ssh/id_ed25519 # Each use requires confirmation click ``` ### Cloud-Init Secrets **Issue**: User data is accessible on VM via metadata service. **Mitigations**: * Agent token is the only secret in cloud-init * Token only valid for that VM's agent * Agent binds to localhost ### Ephemeral Security Benefits The ephemeral nature provides security benefits: 1. **Limited persistence**: Compromises don't persist 2. **Fresh state**: Each VM starts clean 3. **Short window**: Less time for attacks 4. **Easy recovery**: Just create new VM ## Compliance Considerations ### For Sensitive Workloads If using spuff for sensitive work: 1. **Network isolation**: Use Tailscale or VPN 2. **Audit logging**: Enable provider audit logs 3. **Access control**: Limit who has API tokens 4. **Data handling**: Don't store sensitive data on VM ### Data Residency * Choose regions based on data residency requirements * Be aware that cloud-init data is stored by providers ## Incident Response ### If API Token Compromised 1. Revoke token immediately in provider dashboard 2. Check for unauthorized resources 3. Generate new token 4. Audit recent activity ### If VM Compromised 1. Destroy VM: `spuff down --force` 2. Revoke any tokens that were on VM 3. Check for unusual account activity 4. Create new VM with fresh token ### If SSH Key Compromised 1. Remove public key from provider 2. Remove from all VMs 3. Generate new key pair 4. Update configuration ## Security Checklist ### Initial Setup * [ ] API token in environment variable, not file * [ ] SSH key has passphrase * [ ] SSH key added to agent * [ ] File permissions correct (0600 for keys) ### Ongoing * [ ] Destroy VMs when not in use * [ ] Rotate API tokens periodically * [ ] Review provider audit logs * [ ] Keep spuff updated ### For Teams * [ ] Each user has own API token * [ ] Separate keys per user * [ ] Document token rotation process * [ ] Incident response plan ## Reporting Vulnerabilities If you discover a security vulnerability: 1. **Do not** create a public issue 2. Email: 3. Include: * Description of vulnerability * Steps to reproduce * Potential impact 4. Allow 48 hours for initial response See [SECURITY.md](https://github.com/avelino/spuff/blob/main/SECURITY.md) for full policy. ## References * [SSH Agent Forwarding Security](https://security.stackexchange.com/questions/101783/) * [cloud-init Security](https://cloudinit.readthedocs.io/en/latest/topics/security.html) * [DigitalOcean Security](https://www.digitalocean.com/trust/security) # Spuff Specification ## Status of this document This document specifies the Spuff project configuration file format used to define ephemeral cloud development environments. Distribution of this document is unlimited. The canonical version of this specification can be found at [docs/spec.md](/spec). The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC 2119](https://www.ietf.org/rfc/rfc2119.txt). ## Version This document specifies **Spuff Configuration Format version 1**. ## Requirements and Optional Attributes The Spuff configuration format aims to provide a developer-friendly syntax for defining reproducible development environments. Implementation of some attributes and features MAY vary across cloud providers and SHOULD be documented by the implementation. The following terms are used to define attribute requirements: * **Required**: Attributes that MUST be present for the configuration to be valid * **Optional**: Attributes that MAY be omitted; implementations MUST use defined default values * **Platform-dependent**: Behavior or availability depends on the underlying cloud provider ## The Spuff File The configuration file MUST be named `spuff.yaml` or `spuff.yml`. The file MUST be valid [YAML 1.2](https://yaml.org/spec/1.2/spec.html) encoded in UTF-8. A **Spuff configuration** consists of: * `version` - Specification version (OPTIONAL) * `name` - Project name (OPTIONAL) * `resources` - VM resource configuration (OPTIONAL) * `bundles` - Language toolchain definitions (OPTIONAL) * `packages` - System packages (OPTIONAL) * `services` - Docker Compose services (OPTIONAL) * `repositories` - Additional repositories to clone (OPTIONAL) * `env` - Environment variables (OPTIONAL) * `setup` - Setup scripts (OPTIONAL) * `ports` - SSH tunnel port mappings (OPTIONAL) * `hooks` - Lifecycle hooks (OPTIONAL) The following diagram shows how the configuration elements interact: ```mermaid flowchart TB subgraph spuffyaml["spuff.yaml"] version["version: '1'
name: my-project"] subgraph config["Configuration"] resources["resources
(VM sizing)"] bundles["bundles
(toolchains)"] packages["packages
(apt install)"] end cloudinit["Cloud-Init Bootstrap"] subgraph agent["spuff-agent"] services["services
(docker)"] repos["repos
(git)"] setup["setup
(scripts)"] hooks["hooks
(post_up)"] end ports["ports: [3000, 8080] → SSH Tunnel (spuff ssh)"] end resources --> cloudinit bundles --> cloudinit packages --> cloudinit cloudinit --> agent ``` ### File Discovery Spuff implementations MUST search for configuration files using the following algorithm: 1. Start from the current working directory 2. Look for `spuff.yaml` or `spuff.yml` (in that order) 3. If not found, move to the parent directory 4. Repeat until a configuration file is found or the filesystem root is reached The directory containing the discovered configuration file is considered the **project root**. ### Secrets File A separate `spuff.secrets.yaml` file MAY be placed alongside the main configuration file. This file: * MUST contain only an `env` section * MUST NOT be committed to version control * Values MUST override corresponding values in the main configuration * SHOULD be added to `.gitignore` ```yaml # spuff.secrets.yaml env: DATABASE_PASSWORD: super-secret API_KEY: sk-xxx ``` *** ## Version top-level element ```yaml version: "1" ``` **Type:** `string` **Default:** `"1"` **Required:** No The `version` attribute defines the specification version. This attribute is OPTIONAL and defaults to `"1"`. Implementations MUST reject configurations with unsupported version numbers. Implementations SHOULD provide clear error messages when encountering unknown versions. *** ## Name top-level element ```yaml name: my-project ``` **Type:** `string` **Default:** Directory name of project root **Required:** No **Constraints:** SHOULD be a valid identifier (alphanumeric, hyphens, underscores) The `name` attribute defines a human-readable name for the project environment. If omitted, implementations MUST use the name of the directory containing the configuration file. The name is used for: * VM instance naming (with unique suffix) * State tracking and identification * Logging and status display *** ## Resources top-level element ```yaml resources: size: s-4vcpu-8gb region: nyc1 ``` The `resources` element defines VM resource configuration. All sub-attributes are OPTIONAL. ### Attribute Precedence Configuration values follow this precedence order (highest to lowest): 1. CLI flags (`--size`, `--region`) 2. Project configuration (`spuff.yaml`) 3. Global configuration (`~/.spuff/config.yaml`) 4. Provider defaults ### size **Type:** `string` **Default:** Provider-dependent **Required:** No Specifies the VM instance size. Valid values are provider-dependent. **DigitalOcean examples:** | Size | vCPUs | Memory | Disk | | -------------- | ----- | ------ | ------ | | `s-1vcpu-1gb` | 1 | 1 GB | 25 GB | | `s-2vcpu-4gb` | 2 | 4 GB | 80 GB | | `s-4vcpu-8gb` | 4 | 8 GB | 160 GB | | `s-8vcpu-16gb` | 8 | 16 GB | 320 GB | ### region **Type:** `string` **Default:** Provider-dependent **Required:** No Specifies the datacenter region. Valid values are provider-dependent. **DigitalOcean examples:** `nyc1`, `nyc3`, `sfo3`, `ams3`, `fra1`, `lon1`, `sgp1`, `blr1` *** ## Bundles top-level element ```yaml bundles: - rust - go - python ``` **Type:** `array` **Default:** `[]` (empty array) **Required:** No The `bundles` element defines pre-configured language toolchains to install. Each bundle includes the language runtime/compiler plus essential development tools (LSPs, linters, formatters, debuggers). ### Valid Bundle Identifiers Implementations MUST support the following bundle identifiers: | Identifier | Required Tools | Optional Tools | | ---------- | ----------------------- | ------------------------------------------------- | | `rust` | rustup, cargo | rust-analyzer, clippy, rustfmt, mold, cargo-watch | | `go` | go (1.23+) | gopls, delve, golangci-lint, air | | `python` | python3.12, pip | uv, ruff, pyright, ipython | | `node` | node (22 LTS), npm | pnpm, typescript, eslint, prettier | | `elixir` | erlang/OTP, elixir, mix | elixir-ls, phoenix | | `java` | openjdk (21), maven | gradle, jdtls | | `zig` | zig (0.13+) | zls | | `cpp` | gcc, clang, cmake | ninja, clangd, gdb, lldb | | `ruby` | ruby, bundler | solargraph, rubocop | Implementations MUST return an error for unknown bundle identifiers. ### Installation Behavior * Required tools: Installation failure MUST cause the bundle to be marked as failed * Optional tools: Installation failure SHOULD be logged but MUST NOT cause bundle failure * Bundles SHOULD be installed in parallel when possible * Installation progress MUST be trackable via the agent API *** ## Packages top-level element ```yaml packages: - postgresql-client - redis-tools - protobuf-compiler - libssl-dev ``` **Type:** `array` **Default:** `[]` (empty array) **Required:** No The `packages` element defines additional system packages to install via the system package manager (apt on Ubuntu/Debian). Package names MUST be valid package identifiers for the target system. Implementations SHOULD NOT validate package names before provisioning (validation occurs at install time). ### Installation Behavior * Packages MUST be installed after the base system is ready * Package installation failure SHOULD be logged with the specific package name * All packages are installed in a single transaction when possible *** ## Services top-level element ```yaml services: enabled: true compose_file: docker-compose.yaml profiles: - dev - debug ``` The `services` element configures Docker Compose services. This element does NOT duplicate Docker Compose configuration; it references an existing compose file. ### enabled **Type:** `boolean` **Default:** `true` if compose file exists, `false` otherwise **Required:** No Controls whether Docker Compose services should be started. ### compose\_file **Type:** `string` **Default:** `"docker-compose.yaml"` **Required:** No Path to the Docker Compose file, relative to the project root. Implementations MUST support both `docker-compose.yaml` and `docker-compose.yml` filenames when using default discovery. ### profiles **Type:** `array` **Default:** `[]` (empty array, all services without profile) **Required:** No Docker Compose profiles to activate. Corresponds to `docker compose --profile` flag. *** ## Repositories top-level element ```yaml repositories: - owner/repo - url: git@github.com:org/backend.git path: ~/projects/backend branch: develop ``` **Type:** `array` **Default:** `[]` (empty array) **Required:** No The `repositories` element defines additional Git repositories to clone into the environment. ### Repository Formats Repositories can be specified in two formats: #### Short Syntax ```yaml repositories: - owner/repo ``` Short syntax assumes GitHub and expands to: * URL: `https://github.com/owner/repo.git` * Path: `~/projects/repo` * Branch: HEAD (default branch) #### Long Syntax ```yaml repositories: - url: git@github.com:org/backend.git path: ~/projects/backend branch: develop ``` | Attribute | Type | Default | Required | Description | | --------- | ------ | ------------------------ | -------- | ---------------------------------- | | `url` | string | - | Yes | Git repository URL (HTTPS or SSH) | | `path` | string | `~/projects/` | No | Clone destination path | | `branch` | string | `null` (HEAD) | No | Branch, tag, or commit to checkout | ### SSH Agent Forwarding Implementations MUST use SSH agent forwarding for cloning operations. This allows users' local SSH keys to authenticate with private repositories without exposing keys on the VM. *** ## Env top-level element ```yaml env: DATABASE_URL: postgres://localhost:5432/mydb RUST_LOG: debug LOG_LEVEL: ${LOG_LEVEL:-info} ``` **Type:** `object` **Default:** `{}` (empty object) **Required:** No The `env` element defines environment variables to be set on the VM. ### Variable Resolution Implementations MUST support variable references in values. References are resolved from the **local environment** (where `spuff up` is executed) before being sent to the VM. | Format | Example | Behavior | | ----------------- | --------------- | ----------------------------------------- | | `$VAR` | `$HOME` | Simple reference, empty string if not set | | `${VAR}` | `${USER}` | Braced reference, empty string if not set | | `${VAR:-default}` | `${PORT:-8080}` | With default, uses default if not set | ### Resolution Rules 1. Variable names MUST match the pattern `[a-zA-Z_][a-zA-Z0-9_]*` 2. Unset variables MUST resolve to empty string (without default) or the default value 3. Resolution MUST occur before configuration is sent to the VM 4. Literal `$` can be escaped as `$$` ### Secrets Merging When `spuff.secrets.yaml` exists, its `env` values MUST be merged after the main configuration, overriding any duplicate keys. *** ## Setup top-level element ```yaml setup: - cargo build --release - npm install - ./scripts/init-db.sh ``` **Type:** `array` **Default:** `[]` (empty array) **Required:** No The `setup` element defines shell commands to execute after bundles and packages are installed. ### Execution Rules 1. Commands MUST be executed in order (sequential, not parallel) 2. Commands MUST be executed in the user's home directory by default 3. If a command returns a non-zero exit code, subsequent commands MUST be skipped 4. Exit codes and output MUST be logged to `/var/log/spuff/scripts/NNN.log` 5. Commands MUST be executed as the unprivileged user, not root ### Logging Each script MUST have its output captured in a numbered log file: ``` /var/log/spuff/scripts/ ├── 001.log # First setup command ├── 002.log # Second setup command └── 003.log # Third setup command ``` *** ## Ports top-level element ```yaml ports: - 3000 - 8080 - 5432 ``` **Type:** `array` **Default:** `[]` (empty array) **Required:** No **Constraints:** Each value MUST be a valid port number (1-65535) The `ports` element defines ports for automatic SSH tunneling when connecting via `spuff ssh`. ### Tunnel Behavior For each port `N` in the array: * Local `localhost:N` is forwarded to VM `localhost:N` * Tunnels are established when `spuff ssh` is invoked * Tunnels remain active for the duration of the SSH session This allows local development tools (browsers, IDEs) to connect to services running on the remote VM. *** ## Hooks top-level element ```yaml hooks: post_up: | echo "Environment ready!" make dev-setup pre_down: | make db-backup ``` The `hooks` element defines lifecycle scripts for custom automation. ### post\_up **Type:** `string` **Default:** `null` **Required:** No Script executed after the environment is fully ready (all bundles, packages, services, repositories, and setup scripts complete). ### pre\_down **Type:** `string` **Default:** `null` **Required:** No Script executed before VM destruction. This allows for cleanup, backups, or graceful shutdown procedures. ### Execution Rules 1. Hooks MUST be executed as shell scripts (bash) 2. Hook failures SHOULD be logged but MUST NOT prevent the operation from completing 3. Multi-line scripts SHOULD use YAML literal block syntax (`|`) *** ## Complete Configuration Example ```yaml # spuff.yaml - Complete example version: "1" name: api-backend resources: size: s-4vcpu-8gb region: fra1 bundles: - rust - python - node packages: - postgresql-client - redis-tools - protobuf-compiler services: enabled: true compose_file: docker-compose.yaml profiles: - dev repositories: - owner/frontend - url: git@github.com:org/shared-libs.git path: ~/projects/libs branch: main env: DATABASE_URL: postgres://dev:dev@localhost:5432/mydb REDIS_URL: redis://localhost:6379 RUST_LOG: debug API_KEY: ${API_KEY} LOG_LEVEL: ${LOG_LEVEL:-info} setup: - cargo build --release - npm install - pip install -r requirements.txt - ./scripts/init-db.sh ports: - 3000 - 8080 - 5432 - 6379 hooks: post_up: | echo "Environment ready!" make dev-setup pre_down: | make db-backup > /tmp/backup.sql ``` *** ## Agent API Reference The spuff-agent running on provisioned VMs exposes a REST API for managing project setup. All endpoints require authentication via `X-Spuff-Token` header when `SPUFF_AGENT_TOKEN` is set. ### GET /project/config Returns the current project configuration. **Response:** `200 OK` with JSON body containing the parsed configuration. ### GET /project/status Returns detailed setup progress. **Response:** `200 OK` with JSON body: ```json { "started": true, "completed": false, "bundles": [ {"name": "rust", "status": "done", "version": "1.78.0"}, {"name": "python", "status": "in_progress", "version": null} ], "packages": { "status": "done", "installed": ["postgresql-client", "redis-tools"], "failed": [] }, "services": { "status": "done", "containers": [ {"name": "postgres", "status": "running", "ports": ["5432"]}, {"name": "redis", "status": "running", "ports": ["6379"]} ] }, "repositories": [ {"url": "git@github.com:org/frontend.git", "path": "~/projects/frontend", "status": "done"}, {"url": "git@github.com:org/backend.git", "path": "~/projects/backend", "status": "pending"} ], "scripts": [ {"command": "cargo build", "status": "done", "exit_code": 0}, {"command": "npm install", "status": "pending", "exit_code": null} ] } ``` ### POST /project/setup Triggers project setup. This endpoint is idempotent; calling it multiple times has no effect if setup is already in progress or complete. **Request Body:** Project configuration JSON (optional, uses embedded config if omitted) **Response:** `202 Accepted` if setup started, `200 OK` if already running/complete. ### Setup Status Values | Status | Description | | ------------- | ------------------------------------- | | `pending` | Not yet started | | `in_progress` | Currently executing | | `done` | Successfully completed | | `failed` | Execution failed (with error message) | | `skipped` | Intentionally skipped | *** ## Validation Rules ### Strict Validation Implementations MUST validate: * YAML syntax is valid * File encoding is UTF-8 * Bundle identifiers are in the supported list * Version string is supported ### Lenient Validation Implementations SHOULD NOT validate: * Package names (provider/repository dependent) * Region codes (provider dependent) * VM sizes (provider dependent) * Repository URLs (format flexibility) * Port numbers beyond basic range check Invalid values in lenient categories will cause runtime errors during provisioning. *** ## File Format Notes * **YAML Version:** 1.2 (via serde\_yaml) * **Encoding:** UTF-8 (MUST) * **Indentation:** 2-space or 4-space (standard YAML) * **Comments:** Supported with `#` * **Multi-line Strings:** Use `|` for literal blocks, `>` for folded blocks * **Empty Sections:** All sections are optional; omit unused sections *** ## Changelog ### Version 1 (Initial) * Initial specification release * Support for bundles: rust, go, python, node, elixir, java, zig, cpp, ruby * Environment variable resolution with defaults * Docker Compose integration * SSH port tunneling * Lifecycle hooks (post\_up, pre\_down) * Secrets management via spuff.secrets.yaml # Troubleshooting Guide This guide helps diagnose and resolve common issues with spuff. ## Quick Diagnostics ```bash # Check spuff version spuff --version # Check current status spuff status # Check with debug logging RUST_LOG=debug spuff status # Check local state sqlite3 ~/.spuff/state.db "SELECT * FROM instances;" ``` *** ## SSH Issues ### "Permission denied (publickey)" **Symptoms:** ``` Error: SSH connection failed: Permission denied (publickey) ``` **Causes & Solutions:** 1. **SSH key not in agent** ```bash # Check if key is loaded ssh-add -l # Add key to agent eval "$(ssh-agent -s)" ssh-add ~/.ssh/id_ed25519 ``` 2. **Wrong key configured** ```bash # Check which key spuff uses spuff config show # Update if needed spuff config set ssh_key_path ~/.ssh/correct_key ``` 3. **Key not uploaded to provider** * Go to DigitalOcean dashboard * Settings > Security > SSH Keys * Add your public key: `cat ~/.ssh/id_ed25519.pub` 4. **Key permissions wrong** ```bash chmod 600 ~/.ssh/id_ed25519 chmod 644 ~/.ssh/id_ed25519.pub ``` ### "Connection refused" on port 22 **Symptoms:** ``` Error: SSH connection failed: Connection refused ``` **Causes & Solutions:** 1. **VM still booting** * Wait a few more seconds * SSH service starts after cloud-init user creation 2. **Firewall blocking** * Check provider firewall/security groups * Ensure port 22 is open 3. **Instance not running** ```bash # Check instance status spuff status --detailed ``` ### "Host key verification failed" **Symptoms:** ``` Host key verification failed. ``` **Cause:** Previous VM had same IP, different host key. **Solution:** ```bash # Remove old host key ssh-keygen -R # Or clear all known hosts > ~/.ssh/known_hosts ``` ### SSH key requires passphrase **Symptoms:** ``` Enter passphrase for key '/home/user/.ssh/id_ed25519': ``` **Solution:** ```bash # Add key to agent with passphrase ssh-add ~/.ssh/id_ed25519 # Enter passphrase once # Verify ssh-add -l ``` *** ## VM Creation Issues ### "API token not configured" **Symptoms:** ``` Error: API token not configured. Set DIGITALOCEAN_TOKEN environment variable. ``` **Solution:** ```bash # Set token export DIGITALOCEAN_TOKEN="dop_v1_xxxxxxxxxxxx" # Or add to shell profile echo 'export DIGITALOCEAN_TOKEN="dop_v1_xxxx"' >> ~/.bashrc source ~/.bashrc ``` ### "Invalid region" **Symptoms:** ``` Error: Region 'xxx' not found ``` **Solution:** ```bash # Use valid region spuff config set region nyc1 # Valid regions: nyc1, nyc3, sfo3, ams3, sgp1, lon1, fra1, tor1, blr1 ``` ### "Quota exceeded" **Symptoms:** ``` Error: You have reached the droplet limit for your account ``` **Solutions:** 1. Destroy existing instances: `spuff down --force` 2. Request quota increase from provider 3. Check for orphaned instances in provider dashboard ### "SSH key not found" **Symptoms:** ``` Error: SSH key not found in your account ``` **Cause:** Public key not registered with cloud provider. **Solution:** 1. Copy public key: `cat ~/.ssh/id_ed25519.pub` 2. Add to provider: * DigitalOcean: Settings > Security > SSH Keys > Add SSH Key *** ## Cloud-Init Issues ### Bootstrap never completes **Symptoms:** * `spuff agent status` shows `bootstrap_status: running` forever * Can SSH in but tools not installed **Diagnosis:** ```bash # SSH to VM spuff ssh # Check cloud-init status cloud-init status --format=json # Check for errors sudo grep -i error /var/log/cloud-init-output.log # Check async bootstrap cat /opt/spuff/bootstrap.status sudo cat /var/log/spuff-bootstrap.log ``` **Common causes:** 1. Network timeout downloading packages 2. Package repository issues 3. Script syntax error ### Docker not installed **Symptoms:** ``` docker: command not found ``` **Diagnosis:** ```bash # Check if install ran sudo grep -i docker /var/log/cloud-init-output.log # Try manual install curl -fsSL https://get.docker.com | sudo sh sudo usermod -aG docker $USER ``` ### Shell aliases not working **Symptoms:** ``` ll: command not found ``` **Cause:** `.bashrc` or `.profile` not properly configured. **Solution:** ```bash # Check .profile exists (needed for login shells) cat ~/.profile # Check .bashrc has aliases grep "alias ll" ~/.bashrc # Source manually source ~/.bashrc ``` *** ## Agent Issues ### "Agent not responding" **Symptoms:** ``` Error: Failed to connect to agent at 127.0.0.1:7575 ``` **Diagnosis:** ```bash # SSH to VM spuff ssh # Check agent status sudo systemctl status spuff-agent # Check agent logs sudo journalctl -u spuff-agent -n 50 # Test agent locally curl http://127.0.0.1:7575/health ``` **Solutions:** 1. **Agent not running** ```bash sudo systemctl start spuff-agent ``` 2. **Agent crashed** ```bash sudo journalctl -u spuff-agent --since "5 minutes ago" # Check for crash reason ``` 3. **Binary not found** ```bash ls -la /opt/spuff/spuff-agent # If missing, download or re-create VM ``` ### "Unauthorized" from agent **Symptoms:** ``` Error: Agent returned 401 Unauthorized ``` **Cause:** Token mismatch between CLI and agent. **Solution:** ```bash # Check token on VM ssh dev@ 'echo $SPUFF_AGENT_TOKEN' # Or check systemd environment ssh dev@ 'sudo systemctl show spuff-agent --property=Environment' ``` *** ## State Issues ### "No active instance" **Symptoms:** ``` Error: No active instance found ``` **Cause:** Local state doesn't know about running instance. **Diagnosis:** ```bash # Check local state sqlite3 ~/.spuff/state.db "SELECT * FROM instances;" # Check provider for spuff instances curl -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \ "https://api.digitalocean.com/v2/droplets?tag_name=spuff" | jq ``` **Solutions:** 1. **Instance exists but not in state** * Manually add to state, or * Destroy via provider dashboard and recreate 2. **Instance was deleted externally** ```bash # Clear local state rm ~/.spuff/state.db ``` ### State out of sync **Symptoms:** * `spuff status` shows instance that doesn't exist * `spuff down` fails with "not found" **Solution:** ```bash # Reset local state rm ~/.spuff/state.db # Recreate spuff up ``` *** ## TUI Issues ### "Device not configured" error **Symptoms:** ``` Error: Device not configured (os error 6) ``` **Cause:** Terminal not properly initialized after subprocess. **Solutions:** 1. **Reset terminal** ```bash reset ``` 2. **Run with text output** ```bash spuff up 2>&1 | cat ``` 3. **Check TTY** ```bash tty # Should show /dev/ttys000 or similar ``` ### TUI garbled display **Symptoms:** * Random characters * Broken layout **Solutions:** ```bash # Reset terminal reset # Or use text mode TERM=dumb spuff up ``` *** ## Network Issues ### Timeout waiting for instance **Symptoms:** ``` Error: Timeout waiting for instance to become ready ``` **Causes:** 1. Provider having issues 2. Region overloaded 3. Network issues **Solutions:** 1. Try different region: `spuff up --region fra1` 2. Check provider status page 3. Retry after a few minutes ### Can't reach provider API **Symptoms:** ``` Error: Request failed: connection refused ``` **Solutions:** 1. Check internet connection 2. Check if provider API is up 3. Check firewall/proxy settings *** ## Configuration Issues ### "Config file not found" **Symptoms:** ``` Error: Config file not found at ~/.spuff/config.yaml ``` **Solution:** ```bash spuff init ``` ### "Invalid config" **Symptoms:** ``` Error: Invalid config: missing field 'region' ``` **Solution:** ```bash # View current config cat ~/.spuff/config.yaml # Recreate rm ~/.spuff/config.yaml spuff init ``` *** ## Getting Help If this guide doesn't solve your issue: 1. **Enable debug logging** ```bash RUST_LOG=debug spuff ``` 2. **Collect information** * spuff version * OS and version * Full error message * Debug logs 3. **Open an issue** * * Include collected information * Redact any tokens/secrets! *** ## Common Commands Reference ```bash # Debug logging RUST_LOG=debug spuff # Check status spuff status --detailed # View config spuff config show # Reset state rm ~/.spuff/state.db # Reset terminal reset # Check SSH agent ssh-add -l # Add SSH key ssh-add ~/.ssh/id_ed25519 # Test SSH ssh -v dev@ echo ok # Check cloud-init (on VM) sudo cat /var/log/cloud-init-output.log # Check agent (on VM) sudo systemctl status spuff-agent sudo journalctl -u spuff-agent -f ``` # Architecture Decision Records This directory contains Architecture Decision Records (ADRs) for the spuff project. ## What is an ADR? An ADR is a document that captures an important architectural decision made along with its context and consequences. ADRs help: * **Document** the reasoning behind decisions * **Communicate** decisions to the team * **Onboard** new contributors by explaining "why" * **Revisit** decisions when context changes ## ADR Index | ID | Title | Status | Date | | -------------------------------------------- | ------------------------------------ | -------- | ------- | | [0001](/adr/0001-cloud-init-bootstrap) | Use cloud-init for VM bootstrap | Accepted | 2025-01 | | [0002](/adr/0002-two-phase-bootstrap) | Two-phase bootstrap (sync + async) | Accepted | 2025-01 | | [0003](/adr/0003-sqlite-local-state) | SQLite for local state management | Accepted | 2025-01 | | [0004](/adr/0004-ssh-agent-forwarding) | SSH agent forwarding for git access | Accepted | 2025-01 | | [0005](/adr/0005-provider-trait-abstraction) | Provider trait for cloud abstraction | Accepted | 2025-01 | | [0006](/adr/0006-project-config-spec) | Project configuration (spuff.yaml) | Accepted | 2025-01 | ## Status Values * **Proposed** - Under discussion * **Accepted** - Decision made, implementing * **Deprecated** - Superseded by another ADR * **Superseded** - Replaced by a newer ADR ## Creating a New ADR 1. Copy the template: ```bash cp docs/adr/template.md docs/adr/NNNN-title.md ``` 2. Fill in the template with: * Context: What is the situation? * Decision: What did we decide? * Consequences: What are the results? 3. Submit a PR for review 4. Update this README with the new ADR ## Template See [template.md](/adr/template) for the ADR template. ## References * [ADR GitHub Organization](https://adr.github.io/) * [Michael Nygard's Article](https://cognitect.com/blog/2011/11/15/documenting-architecture-decisions) # ADR-0001: Use cloud-init for VM Bootstrap ## Status Accepted ## Date 2025-01 ## Context When provisioning cloud VMs, we need a way to: 1. Create a non-root user with SSH access 2. Install required packages and tools 3. Configure the environment (shell, aliases, etc.) 4. Start the spuff-agent daemon 5. Do all of this automatically without manual intervention All major cloud providers support some form of "user data" that runs on first boot. We need to choose how to format and deliver this bootstrap configuration. ### Requirements * Works across multiple cloud providers * Supports complex multi-step installation * Can create users, install packages, write files * Runs automatically on first boot * Has good debugging/logging capabilities ## Decision We will use **cloud-init** with YAML configuration format for VM bootstrapping. Cloud-init is: * The de facto standard for cloud instance initialization * Supported by all major cloud providers (DigitalOcean, AWS, GCP, Azure, Hetzner) * Well-documented with extensive module support * Logs to `/var/log/cloud-init-output.log` for debugging ### Implementation 1. Generate cloud-init YAML using Tera templates (`src/environment/cloud_init.rs`) 2. Base64-encode the YAML for provider API compatibility 3. Pass as `user_data` in instance creation request 4. Cloud-init executes on first boot ### cloud-init Structure ```yaml #cloud-config users: - name: dev groups: [sudo, docker] shell: /bin/bash ssh_authorized_keys: [...] package_update: true packages: [git, curl, vim, ...] write_files: - path: /opt/spuff/bootstrap.sh content: | #!/bin/bash # Installation script runcmd: - ["/opt/spuff/bootstrap.sh"] ``` ## Consequences ### Positive * **Universal compatibility**: Works with every major cloud provider * **Declarative configuration**: YAML is readable and maintainable * **Built-in features**: User creation, package installation, file writing * **Good logging**: `/var/log/cloud-init-output.log` aids debugging * **No SSH required**: Runs before network access is available * **Idempotent**: Can be re-run safely ### Negative * **YAML complexity**: Complex scripts in YAML can be hard to read * **Debugging difficulty**: Errors only visible in logs after boot * **Provider variations**: Some providers have quirks with user-data handling * **Size limits**: Some providers limit user-data size (\~64KB typical) ### Neutral * Requires understanding of cloud-init modules and syntax * Templates add a layer of indirection ## Alternatives Considered ### Alternative 1: Packer Images Pre-build VM images with all tools installed using Packer. **Pros:** * Faster boot times (no installation during boot) * Consistent, tested images **Cons:** * Need to maintain images per provider/region * Image updates require rebuild and redistribute * Storage costs for images **Why rejected:** Too much operational overhead for the initial version. cloud-init provides flexibility during rapid development. May revisit for production optimization. ### Alternative 2: SSH + Bash Scripts SSH into the VM after boot and run setup scripts directly. **Pros:** * Simpler debugging (interactive SSH) * More control over execution order **Cons:** * Requires SSH to be ready first * Adds latency to provisioning * Network-dependent **Why rejected:** Adds complexity and latency. cloud-init runs before we need SSH access. ### Alternative 3: Ansible/Configuration Management Use Ansible or similar tools for configuration. **Pros:** * Powerful configuration management * Declarative, idempotent **Cons:** * Additional dependency * Overkill for our use case * Requires SSH access **Why rejected:** Overkill for bootstrapping ephemeral VMs. cloud-init is sufficient. ## References * [cloud-init Documentation](https://cloudinit.readthedocs.io/) * [DigitalOcean cloud-init Support](https://docs.digitalocean.com/products/droplets/how-to/automate-setup-with-cloud-init/) * [Tera Templates](https://keats.github.io/tera/) # ADR-0002: Two-Phase Bootstrap (Sync + Async) ## Status Accepted ## Date 2025-01 ## Context VM bootstrapping involves installing many tools and dependencies: **Essential (needed immediately):** * Docker * Basic shell tools (git, curl) * User setup * SSH access **Nice-to-have (can wait):** * devbox/nix * Node.js * Claude Code CLI * Dotfiles * spuff-agent download If we install everything synchronously, the user waits several minutes before SSH access is available. This creates a poor first impression. ### Requirements * SSH should be available as fast as possible * Essential tools must be ready before user connects * Long-running installations shouldn't block access * User should see progress of background tasks ## Decision We will split bootstrap into **two phases**: ### Phase 1: Synchronous (bootstrap-sync.sh) Runs during cloud-init, blocks until complete: * Docker installation * Basic shell tools (fzf, bat, eza, starship) * Directory structure creation * Essential configuration ### Phase 2: Asynchronous (bootstrap-async.sh) Runs in background via `nohup`, doesn't block: * devbox/nix installation * Node.js and npm * Claude Code CLI * Dotfiles cloning * spuff-agent download (if not using --dev) ### Progress Tracking The async script writes status to `/opt/spuff/bootstrap.status`: * `running` - Bootstrap in progress * `ready` - All done * `failed` - Error occurred The spuff-agent reads this file and exposes it via the `/status` endpoint. ### Implementation In cloud-init: ```yaml runcmd: # Phase 1: Sync (blocks) - ["/opt/spuff/bootstrap-sync.sh"] # Start agent - ["systemctl", "start", "spuff-agent"] # Phase 2: Async (background) - ["nohup", "/opt/spuff/bootstrap-async.sh", "&"] ``` ## Consequences ### Positive * **Fast SSH access**: User can connect in \~2-3 minutes instead of 5-7 * **Better UX**: User sees progress, not just waiting * **Parallel work**: User can work while background tasks complete * **Flexibility**: Easy to move items between phases ### Negative * **Complexity**: Two scripts instead of one * **State management**: Need to track async progress * **Potential confusion**: User might try to use tools before they're ready * **Error handling**: Async errors are less visible ### Neutral * spuff-agent starts before full bootstrap completes * TUI shows bootstrap progress to user ## Alternatives Considered ### Alternative 1: Everything Synchronous Install all tools in a single synchronous script. **Pros:** * Simpler implementation * Guaranteed everything ready when SSH available **Cons:** * Long wait time (5-7 minutes) * Poor user experience * Can't work while waiting **Why rejected:** User experience is a priority. Waiting 5+ minutes is unacceptable. ### Alternative 2: Pre-built Images Use pre-built images with everything installed. **Pros:** * Instant readiness * Consistent environment **Cons:** * Image maintenance burden * Storage costs * Less flexibility for customization **Why rejected:** May implement later, but cloud-init provides flexibility during development. ### Alternative 3: On-Demand Installation Only install tools when user first uses them. **Pros:** * Fastest initial boot * Only install what's needed **Cons:** * Delayed experience when using tools * Complex detection of "first use" * Confusing errors **Why rejected:** Unexpected delays are worse than known upfront wait. ## References * [ADR-0001: cloud-init Bootstrap](/adr/0001-cloud-init-bootstrap) * [nohup Documentation](https://man7.org/linux/man-pages/man1/nohup.1.html) # ADR-0003: SQLite for Local State Management ## Status Superseded Superseded by: ChronDB Git-backed document store in `~/.spuff/chrondb/` ## Date 2025-01 ## Context Spuff needs to track information about active instances locally: * Instance ID, name, IP address * Cloud provider and region * Creation timestamp * Current status This state is needed for: * `spuff status` - Show current instance info * `spuff ssh` - Connect to instance by name * `spuff down` - Know what to destroy * Orphan detection - Find forgotten instances ### Requirements * Persistent across CLI invocations * Fast reads and writes * No external dependencies (database servers) * Works offline * Easy to backup/migrate * Queryable (list, filter, search) ## Decision We will use **SQLite** for local state management, stored at `~/.spuff/state.db`. ### Schema ```sql CREATE TABLE IF NOT EXISTS instances ( id TEXT PRIMARY KEY, name TEXT NOT NULL, ip TEXT NOT NULL, provider TEXT NOT NULL, region TEXT NOT NULL, size TEXT NOT NULL, created_at TEXT NOT NULL ); ``` ### Implementation Using `rusqlite` crate with bundled SQLite: ```rust pub struct StateDb { conn: Connection, } impl StateDb { pub fn open() -> Result { let path = config_dir()?.join("state.db"); let conn = Connection::open(&path)?; conn.execute_batch(SCHEMA)?; Ok(Self { conn }) } pub fn save_instance(&self, instance: &Instance) -> Result<()>; pub fn get_active_instance(&self) -> Result>; pub fn delete_instance(&self, id: &str) -> Result<()>; pub fn list_instances(&self) -> Result>; } ``` ### Why SQLite? 1. **Zero configuration**: No server to install or manage 2. **Single file**: Easy to backup, move, or delete 3. **ACID compliant**: Reliable even on crashes 4. **Fast**: Perfect for local, single-user access 5. **Familiar**: SQL is well-understood 6. **Bundled**: `rusqlite` bundles SQLite, no system dependency ## Consequences ### Positive * **Simple**: No external dependencies or services * **Reliable**: SQLite is battle-tested * **Queryable**: SQL allows flexible queries * **Portable**: Single file, works on all platforms * **Debuggable**: Can inspect with `sqlite3` CLI ### Negative * **Binary file**: Not human-readable (vs JSON/YAML) * **Schema migrations**: Need to handle schema changes * **Concurrency**: SQLite has write locks (not an issue for CLI) * **Additional dependency**: `rusqlite` adds to binary size ### Neutral * Learning curve for SQL if unfamiliar * Need to decide on migration strategy for future schema changes ## Alternatives Considered ### Alternative 1: JSON File Store state in a JSON file at `~/.spuff/state.json`. **Pros:** * Human-readable * No additional dependencies * Simple to implement **Cons:** * No atomic updates (corruption risk on crash) * Must load entire file for any operation * No query capabilities * Manual locking needed **Why rejected:** Risk of corruption and lack of query capability. ### Alternative 2: YAML File Similar to JSON but with YAML format. **Pros:** * Human-readable and editable * Familiar format **Cons:** * Same issues as JSON * YAML parsing is slower **Why rejected:** Same reasons as JSON. ### Alternative 3: sled (Embedded KV Store) Use sled, an embedded key-value database in Rust. **Pros:** * Pure Rust, no C dependencies * Good performance **Cons:** * Less mature than SQLite * No SQL queries * Larger binary size **Why rejected:** SQLite is more mature and SQL provides flexibility. ### Alternative 4: Cloud Storage (Provider State) Rely on cloud provider tags to track instances. **Pros:** * No local state needed * Accessible from anywhere **Cons:** * Requires API calls for every operation * Doesn't work offline * Provider-specific implementation **Why rejected:** Adds latency and requires network access. ## References * [SQLite Documentation](https://www.sqlite.org/docs.html) * [rusqlite Crate](https://github.com/rusqlite/rusqlite) * [SQLite for Application File Format](https://www.sqlite.org/appfileformat.html) # ADR-0004: SSH Agent Forwarding for Git Access ## Status Accepted ## Date 2025-01 ## Context Users need to clone private repositories on the cloud VM: ```bash # On the cloud VM git clone git@github.com:user/private-repo.git ``` This requires SSH authentication with GitHub/GitLab/Bitbucket. We need a way to provide the VM with access to private repositories. ### Requirements * Clone private repos via SSH URLs * No copying of private keys to the VM * Work with existing SSH key setup * Support multiple git hosts (GitHub, GitLab, Bitbucket) * Minimal configuration for users ### Security Constraints * Private keys should **never** leave the user's machine * VM compromise should not expose keys * Ephemeral VMs shouldn't store persistent credentials ## Decision We will use **SSH agent forwarding** (`ssh -A`) to provide git SSH access on VMs. ### How It Works ```mermaid flowchart TB subgraph local["User's Machine"] sshagent["SSH Agent
(ssh-agent)"] keys["Private keys stored securely
~/.ssh/id_ed25519"] keys --> sshagent end subgraph vm["Cloud VM"] socket["SSH_AUTH_SOCK
(socket)"] gitclone["git clone git@github.com:user/repo.git

Authentication request forwarded to
user's machine, signed there, returned"] socket --> gitclone note["No private keys stored on VM"] end sshagent -->|"Forward via SSH connection (-A flag)"| socket style note fill:#f5f5f5,stroke:#ccc,stroke-dasharray: 5 5 ``` ### Implementation 1. Connect with `-A` flag: ```rust Command::new("ssh") .arg("-A") // Enable agent forwarding .args(["-o", "StrictHostKeyChecking=accept-new"]) .arg(format!("{}@{}", config.ssh_user, host)) .status() ``` 2. Pre-authorize git hosts in cloud-init: ```yaml write_files: - path: /home/dev/.ssh/config content: | Host github.com gitlab.com bitbucket.org StrictHostKeyChecking accept-new ``` 3. Add host keys to known\_hosts: ```bash ssh-keyscan github.com gitlab.com >> ~/.ssh/known_hosts ``` ## Consequences ### Positive * **Keys never leave local machine**: Maximum security * **No configuration needed**: Works with existing SSH setup * **Transparent**: `git clone` just works * **Temporary access**: Forwarding ends when SSH disconnects * **Multi-host support**: Works with any SSH-based git host ### Negative * **Requires ssh-agent**: User must have agent running * **Connection-dependent**: Only works while connected * **Risk if VM compromised**: Attacker could use forwarded agent while connected * **Passphrase handling**: User must unlock keys before connecting ### Neutral * Only works for SSH URLs, not HTTPS * Requires user to understand SSH keys ## Security Considerations ### Risk: Malicious Process on VM A root process on the VM could use the forwarded agent while the connection is active. **Mitigations:** * VMs are ephemeral (limited exposure window) * User controls when to connect * Agent forwarding only active during SSH session * Can use `ssh-add -c` for confirmation on each use ### Risk: Leaked Agent Socket The `SSH_AUTH_SOCK` file could be accessed by other users. **Mitigations:** * VM has single user (dev) * Socket permissions restrict access * Ephemeral VM limits exposure ### Recommendation For highly sensitive keys, users can: 1. Use a separate key for development 2. Use `ssh-add -c` for confirmation prompts 3. Limit agent forwarding duration ## Alternatives Considered ### Alternative 1: Copy SSH Keys to VM Upload user's SSH keys to the VM. **Pros:** * Works without agent * Works even when disconnected **Cons:** * **Private keys on VM** - Major security risk * Keys could be extracted if VM compromised * Requires secure key transfer **Why rejected:** Unacceptable security risk. ### Alternative 2: Deploy Keys Create per-repository deploy keys. **Pros:** * Limited scope (single repo) * No forwarding needed **Cons:** * Must create key for each repo * Management overhead * Doesn't scale **Why rejected:** Too much manual work for users. ### Alternative 3: Personal Access Tokens Use HTTPS URLs with PATs. **Pros:** * No SSH key management * Works with HTTPS **Cons:** * Tokens stored on VM * Must convert SSH URLs to HTTPS * Token management complexity **Why rejected:** Tokens on VM is still a secret exposure risk. ### Alternative 4: Git Credential Helper Use a credential helper that prompts for authentication. **Pros:** * Standard git mechanism **Cons:** * Requires interaction for each repo * Complex to set up **Why rejected:** Not transparent enough. ## References * [SSH Agent Forwarding](https://www.ssh.com/academy/ssh/agent#agent-forwarding) * [GitHub SSH Agent Forwarding](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/using-ssh-agent-forwarding) * [Security of SSH Agent Forwarding](https://security.stackexchange.com/questions/101783/are-there-any-risks-associated-with-ssh-agent-forwarding) # ADR-0005: Provider Trait for Cloud Abstraction ## Status Accepted ## Date 2025-01 ## Context Spuff aims to support multiple cloud providers: * DigitalOcean (current) * Hetzner Cloud (planned) * AWS EC2 (planned) * Others (future) Each provider has different: * API endpoints and authentication * Resource naming (droplets, servers, instances) * Region and size identifiers * Snapshot mechanisms * Rate limits and quirks ### Requirements * Support multiple cloud providers * Consistent CLI experience regardless of provider * Easy to add new providers * Provider-specific features accessible when needed * Clean separation of concerns ## Decision We will use a **Rust trait** (`Provider`) as an abstraction layer over cloud providers. ### The Provider Trait ```rust #[async_trait] pub trait Provider: Send + Sync { // Instance lifecycle async fn create_instance(&self, config: &InstanceConfig) -> Result; async fn destroy_instance(&self, id: &str) -> Result<()>; async fn get_instance(&self, id: &str) -> Result>; async fn list_instances(&self) -> Result>; async fn wait_ready(&self, id: &str) -> Result; // Snapshots async fn create_snapshot(&self, instance_id: &str, name: &str) -> Result; async fn list_snapshots(&self) -> Result>; async fn delete_snapshot(&self, id: &str) -> Result<()>; } ``` ### Factory Pattern ```rust pub fn create_provider(config: &AppConfig) -> Result> { match config.provider.as_str() { "digitalocean" => Ok(Box::new(DigitalOceanProvider::new(&token)?)), "hetzner" => Ok(Box::new(HetznerProvider::new(&token)?)), _ => Err(SpuffError::Config("Unknown provider")), } } ``` ### Common Data Types ```rust pub struct Instance { pub id: String, pub name: String, pub ip: String, pub status: InstanceStatus, pub region: String, pub size: String, pub created_at: DateTime, } pub enum InstanceStatus { Starting, Running, Stopping, Stopped, Unknown, } ``` ## Consequences ### Positive * **Extensibility**: New providers just implement the trait * **Consistency**: CLI code doesn't change for different providers * **Testability**: Mock providers for testing * **Type safety**: Rust compiler enforces interface compliance * **Documentation**: Trait documents the required API ### Negative * **Lowest common denominator**: Trait methods must work across all providers * **Feature gaps**: Provider-specific features harder to expose * **Abstraction leak**: Some provider differences may leak through * **Maintenance**: Must update all providers for trait changes ### Neutral * Requires understanding of Rust traits and dynamic dispatch * Some boxing overhead (negligible for network operations) ## Design Decisions ### Why `async_trait`? Cloud API calls are async, and Rust traits don't natively support async methods. The `async_trait` crate provides this capability. ### Why `Box`? Dynamic dispatch allows runtime provider selection based on configuration. The alternative (generics) would require compile-time provider choice. ### Why `Send + Sync`? The provider may be used across async tasks. These bounds ensure thread safety. ### Method Granularity Methods are designed to be: * **Atomic**: Each method does one thing * **Composable**: Higher-level operations built from primitives * **Idempotent where possible**: Delete already-deleted resources shouldn't error ## Alternatives Considered ### Alternative 1: No Abstraction Direct provider API calls throughout the codebase. **Pros:** * Full access to provider features * No abstraction overhead **Cons:** * Code duplication * Provider-specific code everywhere * Hard to add new providers **Why rejected:** Not scalable for multi-cloud support. ### Alternative 2: Enum-Based Dispatch Use an enum with match statements: ```rust enum Provider { DigitalOcean(DigitalOceanProvider), Hetzner(HetznerProvider), } ``` **Pros:** * No dynamic dispatch * Exhaustive matching **Cons:** * Every match statement must handle all providers * Adding provider touches many files **Why rejected:** Too invasive when adding providers. ### Alternative 3: Generic Parameters Use generic parameters instead of trait objects: ```rust fn run(provider: P, config: &AppConfig) -> Result<()> ``` **Pros:** * No boxing overhead * Monomorphization **Cons:** * Can't select provider at runtime * Larger binary (code per provider) **Why rejected:** Need runtime provider selection from config. ### Alternative 4: gRPC/Plugin System Load providers as separate processes/plugins. **Pros:** * True isolation * Dynamic loading **Cons:** * Massive complexity * IPC overhead * Deployment complexity **Why rejected:** Overkill for this use case. ## Future Considerations ### Provider-Specific Extensions For provider-specific features, we can: 1. **Downcast**: Cast `Box` to concrete type 2. **Extension traits**: Additional traits for specific capabilities 3. **Feature flags**: Optional trait methods with default implementations ### API Versioning If trait changes significantly: 1. Create new trait version (e.g., `ProviderV2`) 2. Provide adapter from old to new 3. Deprecate old trait gradually ## References * [Rust Trait Objects](https://doc.rust-lang.org/book/ch17-02-trait-objects.html) * [async-trait Crate](https://github.com/dtolnay/async-trait) * [Provider Pattern Discussion](https://github.com/avelino/spuff/blob/main/docs/adr/docs/providers/README.md) # ADR-0006: Project Configuration Specification (spuff.yaml) ## Status Accepted ## Date 2025-01-19 ## Context spuff currently uses a global configuration file (`~/.spuff/config.yaml`) for all environments. While this works for basic usage, teams and projects often need: 1. **Reproducible environments** - Every developer should get the same setup 2. **Project-specific tooling** - Different projects need different language stacks 3. **Infrastructure as code** - Environment configuration should be versioned with the codebase 4. **Automatic dependency installation** - No manual setup steps after `spuff up` Without project-level configuration: * Developers must manually install tools after VM creation * Environments drift between team members * Onboarding requires documentation and manual steps * Environment setup is not reproducible or auditable ## Decision We will implement a **project-level configuration specification** via `spuff.yaml` that: ### 1. Configuration File * Single file: `spuff.yaml` in project root * Optional secrets file: `spuff.secrets.yaml` (gitignored) * YAML format for readability and compatibility * Discovery: search upward from CWD to find config ### 2. Key Features **Language Bundles**: Pre-configured toolchains that include compiler/runtime + LSPs + linters + formatters: * rust, go, python, node, elixir, java, zig, cpp, ruby * Each bundle is self-contained and tested **Resource Overrides**: Project can specify VM size/region (CLI args take precedence) **Docker Services Integration**: Uses existing `docker-compose.yaml` instead of duplicating configuration **Repository Cloning**: Automatically clone related repositories with SSH agent forwarding **Environment Variables**: Support for `$VAR`, `${VAR}`, and `${VAR:-default}` resolution from host **Setup Scripts**: Ordered list of commands executed after packages/bundles install **Port Tunneling**: Declare ports for automatic SSH tunnel setup via `spuff ssh` ### 3. Implementation Architecture ```mermaid flowchart TB subgraph cli["CLI (spuff up)"] load["Load spuff.yaml from CWD"] merge["Merge with global config"] embed["Embed as project.json in cloud-init"] load --> merge --> embed end subgraph vm["VM boots → agent starts"] read["Agent reads /opt/spuff/project.json"] bundles["Install bundles (async)"] packages["Install packages"] repos["Clone repositories"] services["Start services (docker-compose)"] scripts["Run setup scripts"] hooks["Execute hooks"] read --> bundles read --> packages read --> repos read --> services packages --> scripts scripts --> hooks end embed --> vm ``` ### 4. What We Will NOT Do * **No config inheritance/extends** - Simplicity first; can add later * **No IDE extension management** - Environment is terminal-focused * **No lock file** - Consider for future version * **No secrets management service** - Use local files + env vars ## Consequences ### Positive * **Reproducible environments** - `git clone` + `spuff up` = working environment * **Self-documenting** - Configuration shows what the project needs * **Version controlled** - Changes are tracked and auditable * **Team alignment** - Everyone uses the same tooling * **Faster onboarding** - No manual setup steps * **Composable** - Can share configs between similar projects ### Negative * **Initial setup cost** - Projects need to create spuff.yaml * **Increased complexity** - More configuration to understand * **Potential conflicts** - Project config vs global config can confuse users * **Bundle maintenance** - Need to keep bundle scripts updated ### Neutral * Moves setup responsibility from developers to the configuration file * Requires agent to handle async installation tasks ## Alternatives Considered ### Alternative 1: Nix Flakes Use Nix for environment definition (like devenv.sh). **Pros:** * Extremely reproducible * Large package ecosystem * Declarative and functional **Cons:** * Steep learning curve * Nix-specific syntax * Slow initial builds * Not familiar to most developers **Why rejected:** Nix is powerful but adds significant complexity. Our target users want simple YAML, not a new language. ### Alternative 2: Devcontainers Use VS Code's devcontainer.json specification. **Pros:** * Industry standard * VS Code integration * Container-based isolation **Cons:** * VS Code-centric * Docker dependency * Doesn't fit our VM-based model **Why rejected:** We provision VMs, not containers. Different paradigm. ### Alternative 3: Terraform/Pulumi Full IaC tools for environment definition. **Pros:** * Extremely powerful * Multi-cloud support * State management **Cons:** * Overkill for dev environments * Slow iteration * Complex for simple use cases **Why rejected:** These tools are designed for production infrastructure, not ephemeral dev environments. ### Alternative 4: Shell Scripts Only Use a `setup.sh` script in each project. **Pros:** * No new concepts * Full flexibility * Easy to understand **Cons:** * Not declarative * Hard to track progress * No structure or validation * Can't show nice status output **Why rejected:** We want declarative configuration with structured output and progress tracking. ## References * [docs/project-config.md](/project-config) - User documentation * [src/project\_config.rs](https://github.com/avelino/spuff/blob/main/src/project_config.rs) - CLI parsing implementation * [src/agent/project\_setup.rs](https://github.com/avelino/spuff/blob/main/src/agent/project_setup.rs) - Agent setup handler * [Devcontainers specification](https://containers.dev/) - Inspiration (but different approach) * [devenv.sh](https://devenv.sh/) - Nix-based alternative # ADR-NNNN: Title ## Status Proposed | Accepted | Deprecated | Superseded ## Date YYYY-MM-DD ## Context What is the issue that we're seeing that is motivating this decision or change? Describe: * The current situation * The problem we're trying to solve * Any constraints or requirements * Relevant background information ## Decision What is the change that we're proposing and/or doing? Be specific about: * What we will do * What we will NOT do * Key implementation details ## Consequences What becomes easier or more difficult to do because of this change? ### Positive * Benefit 1 * Benefit 2 ### Negative * Drawback 1 * Drawback 2 ### Neutral * Side effect 1 ## Alternatives Considered What other options were considered? ### Alternative 1: \[Name] Description of the alternative. **Pros:** * Pro 1 **Cons:** * Con 1 **Why rejected:** Reason for not choosing this option. ### Alternative 2: \[Name] ... ## References * Link to relevant documentation * Link to related issues or PRs * Link to external resources # Development Guide This directory contains documentation for developing spuff. ## Contents * [Setup](/development/setup) - Development environment setup * [Testing](/development/testing) - Testing strategies and commands * [Debugging](/development/debugging) - Debugging tips and tools * [Cross-Compilation](/development/cross-compilation) - Building the agent for Linux * [Releasing](/development/releasing) - Release process ## Quick Start ```bash # Clone git clone https://github.com/avelino/spuff.git cd spuff # Build cargo build # Test cargo test --all # Run cargo run -- status ``` ## Project Structure ``` spuff/ ├── src/ │ ├── main.rs # CLI entry point │ ├── cli/ # Command implementations │ │ ├── mod.rs │ │ └── commands/ │ │ ├── up.rs # spuff up │ │ ├── down.rs # spuff down │ │ ├── ssh.rs # spuff ssh │ │ └── ... │ ├── provider/ # Cloud provider abstraction │ │ ├── mod.rs # Provider trait │ │ └── digitalocean.rs # DigitalOcean implementation │ ├── connector/ # SSH/network operations │ │ └── ssh.rs │ ├── environment/ # Cloud-init and templates │ │ └── cloud_init.rs │ ├── agent/ # Remote agent (separate binary) │ │ ├── main.rs │ │ ├── routes.rs │ │ └── metrics.rs │ ├── tui/ # Terminal UI │ │ ├── mod.rs │ │ ├── progress.rs │ │ └── widgets.rs │ ├── config.rs # Configuration loading │ ├── state.rs # SQLite state management │ ├── error.rs # Error types │ └── utils.rs # Utilities ├── docs/ # Documentation ├── examples/ # Example configurations ├── tests/ # Integration tests ├── Cargo.toml ├── CLAUDE.md # LLM instructions ├── CONTRIBUTING.md # Contribution guidelines └── README.md ``` ## Key Concepts ### Two Binaries Spuff produces two binaries: 1. **spuff** - CLI tool that runs on user's machine 2. **spuff-agent** - Daemon that runs on cloud VMs ### Provider Abstraction Cloud providers implement the `Provider` trait. See [docs/providers/](/providers) for details. ### Cloud-Init Templates VM bootstrapping uses Tera templates. See [docs/adr/0001-cloud-init-bootstrap.md](/adr/0001-cloud-init-bootstrap). ### Local State Instance tracking uses SQLite. See [docs/adr/0003-sqlite-local-state.md](/adr/0003-sqlite-local-state). ## Useful Commands ```bash # Fast compile check (no build) cargo check # Format code cargo fmt # Lint check cargo clippy # Build release cargo build --release # Run specific binary cargo run --bin spuff -- up cargo run --bin spuff-agent # Run with logging RUST_LOG=debug cargo run -- up RUST_LOG=spuff=trace cargo run -- status ``` ## Environment Variables | Variable | Description | | -------------------- | ---------------------------------- | | `DIGITALOCEAN_TOKEN` | DigitalOcean API token | | `RUST_LOG` | Logging level (debug, trace, etc.) | | `SPUFF_AGENT_TOKEN` | Agent authentication token | | `TS_AUTHKEY` | Tailscale auth key | # Cross-Compilation Guide The spuff-agent runs on Linux cloud VMs, but you might develop on macOS or Windows. This guide covers cross-compiling the agent. ## Overview ```mermaid flowchart TB subgraph dev["Development Machine (macOS)"] code["Rust Code
(src/agent/)"] zigbuild["cargo zigbuild
--target x86_64-unknown-linux-gnu"] binary["target/x86_64-unknown-linux-gnu/
release/spuff-agent
(Linux ELF binary)"] code --> zigbuild --> binary end subgraph vm["Cloud VM (Linux)"] agent["/opt/spuff/spuff-agent"] end binary -->|"SCP / cloud-init"| agent ``` ## Using cargo-zigbuild (Recommended) ### Install Zig **macOS:** ```bash brew install zig ``` **Linux:** ```bash # Download from https://ziglang.org/download/ # Or use package manager sudo apt install zig # Debian/Ubuntu ``` **Verify:** ```bash zig version ``` ### Install cargo-zigbuild ```bash cargo install cargo-zigbuild ``` ### Cross-Compile ```bash # Add Linux target rustup target add x86_64-unknown-linux-gnu # Build agent for Linux cargo zigbuild --release --target x86_64-unknown-linux-gnu --bin spuff-agent ``` The binary will be at: ``` target/x86_64-unknown-linux-gnu/release/spuff-agent ``` ### Why zigbuild? * Uses Zig as a C compiler/linker * No need for cross-compilation toolchain * Works out of the box on macOS * Handles glibc linking properly ## Using Docker (Alternative) If you prefer Docker: ```bash # Build in Linux container docker run --rm -v $(pwd):/app -w /app rust:latest \ cargo build --release --bin spuff-agent # Binary at target/release/spuff-agent ``` Or with a Dockerfile: ```dockerfile FROM rust:1.75 as builder WORKDIR /app COPY . . RUN cargo build --release --bin spuff-agent FROM scratch COPY --from=builder /app/target/release/spuff-agent /spuff-agent ``` ## Using cross (Alternative) ```bash # Install cross cargo install cross # Build cross build --release --target x86_64-unknown-linux-gnu --bin spuff-agent ``` Note: `cross` requires Docker. ## Testing the Binary ### Check Binary Type ```bash # On macOS file target/x86_64-unknown-linux-gnu/release/spuff-agent # Should output: ELF 64-bit LSB pie executable, x86-64, ... # Check it's not macOS Mach-O file target/release/spuff-agent # This would be Mach-O ``` ### Test on VM Use `spuff up --dev` to upload and test your local agent: ```bash # Build agent for Linux cargo zigbuild --release --target x86_64-unknown-linux-gnu --bin spuff-agent # Create VM with local agent cargo run -- up --dev ``` The `--dev` flag: 1. Creates VM normally 2. Uploads local agent binary via SCP 3. Restarts agent service ### Manual Upload ```bash # Build cargo zigbuild --release --target x86_64-unknown-linux-gnu --bin spuff-agent # Get VM IP IP=$(cargo run -- status --json | jq -r '.ip') # Upload scp -o StrictHostKeyChecking=no \ target/x86_64-unknown-linux-gnu/release/spuff-agent \ dev@$IP:/tmp/spuff-agent # SSH and install ssh dev@$IP 'sudo mv /tmp/spuff-agent /opt/spuff/ && sudo systemctl restart spuff-agent' ``` ## Build Script Create a build script for convenience: ```bash #!/bin/bash # scripts/build-agent.sh set -e TARGET="x86_64-unknown-linux-gnu" BINARY="spuff-agent" echo "Building $BINARY for $TARGET..." cargo zigbuild --release --target $TARGET --bin $BINARY OUTPUT="target/$TARGET/release/$BINARY" echo "Built: $OUTPUT" file $OUTPUT ls -lh $OUTPUT ``` ## Troubleshooting ### Missing Target ```bash error: target 'x86_64-unknown-linux-gnu' not found ``` **Solution:** ```bash rustup target add x86_64-unknown-linux-gnu ``` ### Zig Not Found ```bash error: linker `zig` not found ``` **Solution:** Install Zig (see above) and ensure it's in PATH. ### glibc Version Mismatch ```bash /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.XX' not found ``` **Solution:** The VM has an older glibc. Options: 1. Target an older glibc: `cargo zigbuild --target x86_64-unknown-linux-gnu.2.17` 2. Use Ubuntu 24.04 images (has newer glibc) ### Linking Errors ```bash error: linking with `cc` failed ``` **Solution:** Ensure you're using `cargo zigbuild` not `cargo build` for cross-compilation. ## CI/CD Integration ### GitHub Actions ```yaml jobs: build-agent: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: dtolnay/rust-toolchain@stable with: targets: x86_64-unknown-linux-gnu - name: Build agent run: cargo build --release --target x86_64-unknown-linux-gnu --bin spuff-agent - name: Upload artifact uses: actions/upload-artifact@v4 with: name: spuff-agent-linux path: target/x86_64-unknown-linux-gnu/release/spuff-agent ``` ## Supported Targets | Target | Architecture | Notes | | --------------------------- | --------------------- | ------------------- | | `x86_64-unknown-linux-gnu` | x86-64 Linux | Primary target | | `aarch64-unknown-linux-gnu` | ARM64 Linux | For ARM VMs | | `x86_64-unknown-linux-musl` | x86-64 Linux (static) | No glibc dependency | ### Building for ARM64 ```bash rustup target add aarch64-unknown-linux-gnu cargo zigbuild --release --target aarch64-unknown-linux-gnu --bin spuff-agent ``` ### Static Linking (musl) ```bash rustup target add x86_64-unknown-linux-musl cargo zigbuild --release --target x86_64-unknown-linux-musl --bin spuff-agent ``` Static binaries work on any Linux distribution but may be slightly slower. # Debugging Guide This guide covers debugging techniques for spuff development. ## Logging ### Enable Debug Logging ```bash # General debug logging RUST_LOG=debug cargo run -- up # Spuff-specific logging RUST_LOG=spuff=debug cargo run -- status # Trace level (very verbose) RUST_LOG=spuff=trace cargo run -- up # Multiple modules RUST_LOG=spuff=debug,reqwest=debug cargo run -- up ``` ### Log Levels | Level | Use Case | | ------- | ----------------------------- | | `error` | Errors that stop execution | | `warn` | Potential issues | | `info` | Normal operation events | | `debug` | Detailed flow information | | `trace` | Very detailed, including data | ### Adding Logs to Code ```rust use tracing::{debug, info, warn, error, trace}; async fn create_instance(&self, config: &InstanceConfig) -> Result { info!("Creating instance: {}", config.name); debug!(?config, "Instance configuration"); match self.api_call().await { Ok(response) => { trace!(?response, "API response"); Ok(response) } Err(e) => { error!(%e, "Failed to create instance"); Err(e) } } } ``` ## Debugging VM Bootstrap ### Cloud-Init Logs SSH into the VM and check: ```bash # Cloud-init output log sudo cat /var/log/cloud-init-output.log # Cloud-init status cloud-init status --format=json # Follow cloud-init in real-time sudo tail -f /var/log/cloud-init-output.log ``` ### Bootstrap Status ```bash # Check async bootstrap status cat /opt/spuff/bootstrap.status # Check bootstrap script output cat /var/log/spuff-bootstrap.log ``` ### Agent Logs ```bash # Agent service status sudo systemctl status spuff-agent # Agent logs sudo journalctl -u spuff-agent -f # Recent agent logs sudo journalctl -u spuff-agent --since "5 minutes ago" ``` ## Debugging SSH Issues ### Verbose SSH ```bash # Very verbose ssh -vvv dev@ # Check authentication ssh -v dev@ 2>&1 | grep -i auth ``` ### Common SSH Issues **Permission denied:** ```bash # Check key permissions ls -la ~/.ssh/ # Fix permissions chmod 600 ~/.ssh/id_ed25519 chmod 644 ~/.ssh/id_ed25519.pub ``` **Key requires passphrase:** ```bash # Add key to agent eval "$(ssh-agent -s)" ssh-add ~/.ssh/id_ed25519 ``` **Host key verification:** ```bash # Remove old host key ssh-keygen -R ``` ## Debugging Provider API ### Log API Requests ```rust // In provider code debug!("API request: {} {}", method, url); trace!(?body, "Request body"); let response = client.request(method, &url).send().await?; debug!("API response: {}", response.status()); trace!(?response_body, "Response body"); ``` ### Inspect with curl ```bash # DigitalOcean API curl -X GET \ -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \ "https://api.digitalocean.com/v2/droplets" # List droplets with spuff tag curl -X GET \ -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \ "https://api.digitalocean.com/v2/droplets?tag_name=spuff" ``` ## Debugging Local State ### SQLite Inspection ```bash # Open database sqlite3 ~/.spuff/state.db # List tables .tables # Show schema .schema instances # List instances SELECT * FROM instances; # Pretty print .mode column .headers on SELECT * FROM instances; ``` ### Reset State ```bash # Delete state database rm ~/.spuff/state.db # Spuff will recreate on next run ``` ## Debugging Agent ### Local Agent Testing ```bash # Run agent locally RUST_LOG=debug SPUFF_AGENT_TOKEN=test ./target/debug/spuff-agent # Test endpoints curl -H "X-Spuff-Token: test" http://127.0.0.1:7575/health curl -H "X-Spuff-Token: test" http://127.0.0.1:7575/status curl -H "X-Spuff-Token: test" http://127.0.0.1:7575/metrics ``` ### Agent on VM ```bash # SSH to VM spuff ssh # Check agent sudo systemctl status spuff-agent sudo journalctl -u spuff-agent -f # Test agent locally on VM curl -H "X-Spuff-Token: $SPUFF_AGENT_TOKEN" http://127.0.0.1:7575/status ``` ## Debugging TUI ### Disable TUI For debugging, you can disable the TUI: ```bash # Run without TTY cargo run -- up 2>&1 | cat # Or set non-interactive echo "" | cargo run -- up ``` ### TUI Fallback When TUI fails, spuff falls back to text output. Check stderr for TUI errors. ## IDE Debugging ### VS Code (CodeLLDB) 1. Install CodeLLDB extension 2. Create launch configuration: ```json // .vscode/launch.json { "version": "0.2.0", "configurations": [ { "type": "lldb", "request": "launch", "name": "Debug spuff", "cargo": { "args": ["build", "--bin=spuff"], "filter": { "name": "spuff", "kind": "bin" } }, "args": ["status"], "cwd": "${workspaceFolder}", "env": { "RUST_LOG": "debug", "DIGITALOCEAN_TOKEN": "${env:DIGITALOCEAN_TOKEN}" } } ] } ``` 1. Set breakpoints and press F5 ### RustRover/IntelliJ 1. Create Run Configuration 2. Set binary: `spuff` 3. Set arguments: `status` 4. Set environment variables 5. Click Debug ## Common Issues ### "Device not configured" (TUI Error) The terminal is not properly initialized: ```bash # Reset terminal reset # Or run with text fallback cargo run -- up 2>&1 | cat ``` ### "Permission denied" on SSH 1. Check key is added to agent: `ssh-add -l` 2. Check key is uploaded to provider 3. Check key permissions: `chmod 600 ~/.ssh/id_*` ### Instance Not Found State may be out of sync: ```bash # Check local state sqlite3 ~/.spuff/state.db "SELECT * FROM instances;" # Check provider curl -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \ "https://api.digitalocean.com/v2/droplets?tag_name=spuff" ``` ### Cloud-Init Never Completes Check for errors: ```bash # SSH in with root ssh root@ # Check cloud-init status cloud-init status --format=json # Check for errors grep -i error /var/log/cloud-init-output.log ``` ## Profiling ### CPU Profiling ```bash # Install flamegraph cargo install flamegraph # Profile cargo flamegraph --bin spuff -- up # Open flamegraph.svg ``` ### Memory Profiling ```bash # Install heaptrack (Linux) sudo apt install heaptrack # Profile heaptrack ./target/release/spuff up # Analyze heaptrack_gui heaptrack.spuff.*.gz ``` ## Useful Commands Cheatsheet ```bash # Reset everything rm -rf ~/.spuff/ # Clear state, keep config rm ~/.spuff/state.db # Debug logging RUST_LOG=spuff=debug cargo run -- # Check VM cloud-init ssh dev@ 'sudo cat /var/log/cloud-init-output.log' # Check agent on VM ssh dev@ 'sudo journalctl -u spuff-agent' # API check curl -s -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \ "https://api.digitalocean.com/v2/droplets?tag_name=spuff" | jq ``` # Release Process This document describes how to create releases for spuff. ## Versioning We follow [Semantic Versioning](https://semver.org/): * **MAJOR** (x.0.0): Breaking changes * **MINOR** (0.x.0): New features, backwards compatible * **PATCH** (0.0.x): Bug fixes, backwards compatible During alpha (0.x.x), minor versions may include breaking changes. ## Release Checklist ### 1. Prepare Release ```bash # Ensure you're on main git checkout main git pull origin main # Create release branch git checkout -b release/v0.2.0 ``` ### 2. Update Version Edit `Cargo.toml`: ```toml [package] name = "spuff" version = "0.2.0" # Update this ``` ### 3. Update Documentation * [ ] README.md - version badges, new features * [ ] CLAUDE.md - roadmap updates * [ ] docs/configuration.md - new config options ### 4. Run Tests ```bash # All tests cargo test --all # Clippy cargo clippy -- -D warnings # Format check cargo fmt --check # Build release cargo build --release ``` ### 5. Create PR ```bash git add -A git commit -m "chore: prepare release v0.2.0" git push -u origin release/v0.2.0 ``` Create PR: "Release v0.2.0" ### 6. Merge and Tag After PR approval and merge: ```bash git checkout main git pull origin main # Create tag git tag -a v0.2.0 -m "Release v0.2.0" # Push tag git push origin v0.2.0 ``` ### 7. Create GitHub Release 1. Go to [Releases](https://github.com/avelino/spuff/releases) 2. Click "Draft a new release" 3. Select tag: `v0.2.0` 4. Title: `v0.2.0` 5. Generate release notes or write manually 6. Publish release ## Release Notes Template ```markdown ## What's Changed ### New Features - Feature 1 (#PR) - Feature 2 (#PR) ### Bug Fixes - Fix issue (#PR) ### Documentation - Doc improvement (#PR) ### Breaking Changes - Breaking change description ## Upgrade Guide Steps to upgrade from previous version. ## Contributors Thanks to @contributor1, @contributor2 ``` ## Automated Releases (Future) ### GitHub Actions ```yaml # .github/workflows/release.yml name: Release on: push: tags: - 'v*' jobs: build: runs-on: ${{ matrix.os }} strategy: matrix: include: - os: ubuntu-latest target: x86_64-unknown-linux-gnu artifact: spuff-linux-x64 - os: macos-latest target: x86_64-apple-darwin artifact: spuff-macos-x64 - os: macos-latest target: aarch64-apple-darwin artifact: spuff-macos-arm64 steps: - uses: actions/checkout@v4 - uses: dtolnay/rust-toolchain@stable with: targets: ${{ matrix.target }} - name: Build run: cargo build --release --target ${{ matrix.target }} - name: Upload artifact uses: actions/upload-artifact@v4 with: name: ${{ matrix.artifact }} path: target/${{ matrix.target }}/release/spuff release: needs: build runs-on: ubuntu-latest steps: - name: Download artifacts uses: actions/download-artifact@v4 - name: Create release uses: softprops/action-gh-release@v1 with: files: | spuff-linux-x64/spuff spuff-macos-x64/spuff spuff-macos-arm64/spuff ``` ## Binary Distribution ### Current (Alpha) Build from source: ```bash git clone https://github.com/avelino/spuff.git cd spuff cargo build --release cp target/release/spuff ~/.local/bin/ ``` ### Future Plans * [ ] Homebrew tap * [ ] apt/yum repositories * [ ] Pre-built binaries on GitHub Releases * [ ] cargo install support ### Homebrew Formula (Draft) ```ruby class Spuff < Formula desc "Ephemeral dev environments in the cloud" homepage "https://github.com/avelino/spuff" url "https://github.com/avelino/spuff/archive/refs/tags/v0.2.0.tar.gz" sha256 "..." license "MIT" depends_on "rust" => :build def install system "cargo", "build", "--release" bin.install "target/release/spuff" end test do assert_match "spuff #{version}", shell_output("#{bin}/spuff --version") end end ``` ## Version Bumping Script ```bash #!/bin/bash # scripts/bump-version.sh VERSION=$1 if [ -z "$VERSION" ]; then echo "Usage: $0 " exit 1 fi # Update Cargo.toml sed -i '' "s/^version = \".*\"/version = \"$VERSION\"/" Cargo.toml # Verify grep "^version" Cargo.toml echo "Version bumped to $VERSION" echo "Don't forget to commit and tag!" ``` ## Hotfix Process For urgent fixes: ```bash # Create hotfix branch from tag git checkout -b hotfix/v0.2.1 v0.2.0 # Make fix # ... # Bump patch version ./scripts/bump-version.sh 0.2.1 # Commit git commit -am "fix: critical bug" # Create PR to main # After merge, tag git tag -a v0.2.1 -m "Hotfix v0.2.1" git push origin v0.2.1 ``` ## Post-Release After release: 1. Announce on social media/Discord 2. Update documentation if needed 3. Monitor for issues 4. Bump to next dev version if desired # Development Setup This guide covers setting up your development environment for spuff. ## Prerequisites ### Required * **Rust 1.75+** - Install via [rustup](https://rustup.rs/) * **Git** - For version control * **SSH client** - For testing VM connections ### Recommended * **cargo-watch** - Auto-rebuild on changes * **cargo-zigbuild** - Cross-compilation to Linux * **sqlite3** - For inspecting local state ## Installation ### 1. Install Rust ```bash # Install rustup curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh # Verify installation rustc --version cargo --version ``` ### 2. Clone the Repository ```bash git clone https://github.com/avelino/spuff.git cd spuff ``` ### 3. Build ```bash # Debug build (faster compilation) cargo build # Release build (optimized) cargo build --release ``` ### 4. Verify ```bash # Run tests cargo test --all # Check help cargo run -- --help ``` ## Optional Tools ### cargo-watch Auto-rebuild on file changes: ```bash cargo install cargo-watch # Watch and rebuild cargo watch -x build # Watch and run tests cargo watch -x test ``` ### cargo-zigbuild Cross-compile the agent for Linux (required if developing on macOS): ```bash # Install zig brew install zig # macOS # or download from https://ziglang.org/download/ # Install cargo-zigbuild cargo install cargo-zigbuild # Cross-compile cargo zigbuild --release --target x86_64-unknown-linux-gnu --bin spuff-agent ``` ### SQLite CLI Inspect local state database: ```bash # macOS brew install sqlite # View state sqlite3 ~/.spuff/state.db "SELECT * FROM instances;" ``` ## IDE Setup ### VS Code Recommended extensions: * **rust-analyzer** - Rust language support * **Even Better TOML** - Cargo.toml syntax * **crates** - Dependency version info * **CodeLLDB** - Debugging support Settings (`.vscode/settings.json`): ```json { "rust-analyzer.checkOnSave.command": "clippy", "rust-analyzer.cargo.features": "all", "[rust]": { "editor.formatOnSave": true } } ``` ### JetBrains (RustRover/IntelliJ) 1. Install Rust plugin 2. Open project directory 3. Configure toolchain in Settings > Languages > Rust ## Cloud Provider Setup ### DigitalOcean 1. Create account at [digitalocean.com](https://www.digitalocean.com/) 2. Generate API token: API > Generate New Token 3. Set environment variable: ```bash export DIGITALOCEAN_TOKEN="dop_v1_xxxxxxxxx" ``` 1. Upload SSH key: Settings > Security > SSH Keys ### Hetzner (for development) 1. Create account at [hetzner.com](https://www.hetzner.com/cloud) 2. Generate API token: Security > API Tokens 3. Set environment variable: ```bash export HETZNER_TOKEN="xxxxxxxxx" ``` ## SSH Key Setup Ensure you have an SSH key: ```bash # Check for existing keys ls -la ~/.ssh/ # Generate if needed ssh-keygen -t ed25519 -C "your@email.com" # Add to SSH agent eval "$(ssh-agent -s)" ssh-add ~/.ssh/id_ed25519 ``` ## Configuration Create a test configuration: ```bash # Initialize config cargo run -- init # Or create manually mkdir -p ~/.spuff cat > ~/.spuff/config.yaml << EOF provider: digitalocean region: nyc1 size: s-1vcpu-1gb # Small for testing idle_timeout: 30m environment: devbox ssh_key_path: ~/.ssh/id_ed25519 ssh_user: dev EOF ``` ## Running Locally ### CLI Commands ```bash # With cargo run cargo run -- status cargo run -- up cargo run -- down # With compiled binary ./target/debug/spuff status ./target/release/spuff up ``` ### Agent (Local Testing) The agent typically runs on VMs, but you can test locally: ```bash # Build agent cargo build --bin spuff-agent # Run agent SPUFF_AGENT_TOKEN=test-token ./target/debug/spuff-agent # Test endpoints curl -H "X-Spuff-Token: test-token" http://127.0.0.1:7575/health curl -H "X-Spuff-Token: test-token" http://127.0.0.1:7575/metrics ``` ## Development Workflow ### Feature Development ```bash # 1. Create branch git checkout -b feature/my-feature # 2. Make changes # ... # 3. Format and lint cargo fmt cargo clippy # 4. Test cargo test --all # 5. Commit git add . git commit -m "feat: add my feature" # 6. Push and create PR git push -u origin feature/my-feature ``` ### Testing Changes on Real VMs ```bash # 1. Build agent for Linux (if on macOS) cargo zigbuild --release --target x86_64-unknown-linux-gnu --bin spuff-agent # 2. Create VM with --dev flag (uploads local agent) cargo run -- up --dev # 3. Test changes on VM # ... # 4. Destroy VM cargo run -- down --force ``` ## Troubleshooting ### Build Errors ```bash # Clear build cache cargo clean # Update dependencies cargo update # Check for issues cargo check ``` ### Permission Denied on SSH Key ```bash # Fix permissions chmod 600 ~/.ssh/id_ed25519 chmod 644 ~/.ssh/id_ed25519.pub ``` ### Agent Not Starting Check systemd logs on the VM: ```bash sudo systemctl status spuff-agent sudo journalctl -u spuff-agent -f ``` ### State Database Issues Reset local state: ```bash rm ~/.spuff/state.db ``` ## Next Steps * [Testing](/development/testing) - Learn about testing strategies * [Debugging](/development/debugging) - Debugging tips * [Cross-Compilation](/development/cross-compilation) - Building for Linux # Testing Guide This guide covers testing strategies, tools, and best practices for spuff. ## Test Organization ``` spuff/ ├── src/ │ ├── provider/ │ │ └── digitalocean.rs # Unit tests in #[cfg(test)] mod │ └── ... └── tests/ └── integration/ # Integration tests ``` ## Running Tests ### All Tests ```bash # Run all unit tests cargo test --all # Run with output cargo test --all -- --nocapture # Run specific test cargo test test_name # Run tests for a specific crate cargo test -p spuff ``` ### Integration Tests Integration tests require cloud credentials: ```bash # Run integration tests (marked with #[ignore]) cargo test -- --ignored # Run all tests including integration cargo test --all -- --include-ignored ``` ## Unit Testing ### Testing Provider Methods Use `wiremock` for API mocking: ```rust #[cfg(test)] mod tests { use super::*; use wiremock::{Mock, MockServer, ResponseTemplate}; use wiremock::matchers::{method, path}; #[tokio::test] async fn test_create_instance() { let mock_server = MockServer::start().await; Mock::given(method("POST")) .and(path("/v2/droplets")) .respond_with(ResponseTemplate::new(202) .set_body_json(create_droplet_response())) .mount(&mock_server) .await; let provider = DigitalOceanProvider::new_with_base_url( "test-token", &mock_server.uri(), ).unwrap(); let result = provider.create_instance(&test_config()).await; assert!(result.is_ok()); } } ``` ### Testing SSH Functions Mock the SSH command: ```rust #[cfg(test)] mod tests { // Use test doubles or feature flags for SSH testing // Real SSH tests should be integration tests } ``` ### Testing Cloud-Init Generation ```rust #[test] fn test_cloud_init_generation() { let config = CloudInitConfig { username: "dev".to_string(), ssh_public_key: "ssh-ed25519 AAAA...".to_string(), // ... }; let yaml = generate_cloud_init(&config).unwrap(); assert!(yaml.contains("username: dev")); assert!(yaml.contains("ssh-ed25519")); } ``` ### Testing Configuration ```rust #[test] fn test_config_loading() { let yaml = r#" provider: digitalocean region: nyc1 size: s-2vcpu-4gb "#; let config: AppConfig = serde_yaml::from_str(yaml).unwrap(); assert_eq!(config.provider, "digitalocean"); } #[test] fn test_config_validation() { let invalid_config = AppConfig { provider: "unknown".to_string(), // ... }; let result = invalid_config.validate(); assert!(result.is_err()); } ``` ## Integration Testing ### Full Lifecycle Test ```rust // tests/integration/lifecycle_test.rs #[tokio::test] #[ignore] // Requires credentials async fn test_full_lifecycle() { let provider = create_test_provider(); // Create let instance = provider.create_instance(&test_config()).await .expect("Failed to create"); // Wait let ready = provider.wait_ready(&instance.id).await .expect("Failed to wait"); assert!(!ready.ip.is_empty()); // Destroy provider.destroy_instance(&instance.id).await .expect("Failed to destroy"); } ``` ### SSH Integration Test ```rust #[tokio::test] #[ignore] async fn test_ssh_connection() { // Create instance let instance = create_test_instance().await; // Test SSH let result = run_command(&instance.ip, &config, "echo hello").await; assert!(result.is_ok()); assert!(result.unwrap().contains("hello")); // Cleanup destroy_test_instance(&instance.id).await; } ``` ## Test Utilities ### Fixtures ```rust // tests/fixtures.rs pub fn test_config() -> InstanceConfig { InstanceConfig { name: format!("spuff-test-{}", uuid::Uuid::new_v4()), region: "nyc1".to_string(), size: "s-1vcpu-1gb".to_string(), image: "ubuntu-24-04-x64".to_string(), ssh_keys: vec![], user_data: None, tags: vec!["spuff".to_string(), "test".to_string()], } } pub fn mock_droplet_response() -> serde_json::Value { serde_json::json!({ "droplet": { "id": 12345, "name": "spuff-test", "status": "active", // ... } }) } ``` ### Test Helpers ```rust // tests/helpers.rs pub async fn with_test_instance(test: F) where F: FnOnce(Instance) -> Fut, Fut: Future, { let provider = create_test_provider(); let instance = provider.create_instance(&test_config()).await.unwrap(); test(instance.clone()).await; // Always cleanup let _ = provider.destroy_instance(&instance.id).await; } ``` ## Mocking ### HTTP Mocking with wiremock ```rust use wiremock::{Mock, MockServer, ResponseTemplate}; use wiremock::matchers::{method, path, header, body_json}; async fn setup_mock_server() -> MockServer { let server = MockServer::start().await; // Mock create droplet Mock::given(method("POST")) .and(path("/v2/droplets")) .and(header("Authorization", "Bearer test-token")) .respond_with(ResponseTemplate::new(202) .set_body_json(mock_droplet_response())) .mount(&server) .await; server } ``` ### Mocking Time For timeout tests: ```rust use tokio::time::{pause, advance}; #[tokio::test] async fn test_timeout() { pause(); // Enable time control let start = tokio::time::Instant::now(); // Advance time by 5 minutes advance(std::time::Duration::from_secs(300)).await; // Test timeout behavior } ``` ## Test Coverage ### Generate Coverage Report ```bash # Install tarpaulin cargo install cargo-tarpaulin # Generate HTML report cargo tarpaulin --out Html # Open report open tarpaulin-report.html ``` ### Coverage Goals * Unit tests: 80%+ coverage * Critical paths (provider, SSH): 90%+ coverage * Integration tests for all happy paths ## Best Practices ### Test Naming ```rust #[test] fn test_create_instance_success() { } #[test] fn test_create_instance_invalid_region() { } #[test] fn test_create_instance_api_error() { } ``` ### Test Structure (Arrange-Act-Assert) ```rust #[test] fn test_example() { // Arrange let config = test_config(); let provider = MockProvider::new(); // Act let result = provider.create_instance(&config); // Assert assert!(result.is_ok()); assert_eq!(result.unwrap().name, config.name); } ``` ### Avoiding Flaky Tests 1. Don't rely on timing 2. Use deterministic test data 3. Clean up resources in tests 4. Isolate tests from each other ### Test Documentation ```rust /// Tests that creating an instance with an invalid region /// returns a descriptive error message. #[test] fn test_create_instance_invalid_region() { // ... } ``` ## CI Integration Tests run automatically on PR: ```yaml # .github/workflows/test.yml name: Tests on: [push, pull_request] jobs: test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: dtolnay/rust-toolchain@stable - run: cargo test --all ``` ## Running Specific Test Categories ```bash # Unit tests only cargo test --lib # Integration tests only cargo test --test '*' # Tests for specific module cargo test provider:: # Tests matching pattern cargo test ssh ``` # Provider System Spuff uses a provider abstraction layer that enables support for multiple cloud providers. This document explains how the system works. ## Overview ```mermaid flowchart TB subgraph cli["spuff CLI"] subgraph registry["Provider Registry"] factory["ProviderFactory"] create["create()"] boxprovider["Box<dyn Provider>"] factory --> create --> boxprovider end subgraph trait["Provider Trait"] methods["create_instance() destroy_instance() list_instances()
get_instance() wait_ready() create_snapshot()
list_snapshots() delete_snapshot() get_ssh_keys()"] end boxprovider --> trait subgraph providers["Implementations"] do["DigitalOcean
Provider + Factory"] hetzner["Hetzner
Provider + Factory"] aws["AWS
Provider + Factory"] end trait --> do trait --> hetzner trait --> aws end ``` ## Architecture The provider system follows the **Registry Pattern**, which enables: 1. **Extensibility**: Add new providers without modifying existing code 2. **Dynamic discovery**: List available providers at runtime 3. **Uniform configuration**: All providers use the same creation interface ### Main Components | File | Responsibility | | ------------------------------ | -------------------------------------------------- | | `src/provider/mod.rs` | `Provider` trait and core types | | `src/provider/registry.rs` | `ProviderFactory` and `ProviderRegistry` | | `src/provider/config.rs` | `InstanceRequest`, `ImageSpec`, `ProviderTimeouts` | | `src/provider/error.rs` | `ProviderError` with specific types | | `src/provider/digitalocean.rs` | Reference implementation | ### Provider Creation Flow ```rust // 1. Registry with registered factories let registry = ProviderRegistry::with_defaults(); // 2. Create provider by name let provider = registry.create_by_name( "digitalocean", &api_token, ProviderTimeouts::default() )?; // 3. Use the provider let instance = provider.create_instance(&request).await?; ``` ## Core Types ### InstanceRequest Configuration for creating an instance (provider-agnostic): ```rust pub struct InstanceRequest { pub name: String, // Instance name pub region: String, // Region/datacenter pub size: String, // Instance type/size pub image: ImageSpec, // OS image pub user_data: Option, // Cloud-init script pub labels: HashMap, // Tags/labels } ``` ### ImageSpec Provider-agnostic image specification: ```rust pub enum ImageSpec { Ubuntu(String), // e.g., "24.04" → provider maps to slug Debian(String), // e.g., "12" Custom(String), // Provider-specific ID Snapshot(String), // Snapshot ID to restore from } ``` ### ProviderInstance Instance returned by the provider: ```rust pub struct ProviderInstance { pub id: String, // Provider-specific ID pub ip: IpAddr, // Public IP address pub status: InstanceStatus, // Current state pub created_at: DateTime, // Timestamp } ``` ### InstanceStatus Possible instance states: ```rust pub enum InstanceStatus { New, // Being created Active, // Running and ready Off, // Powered off Archive, // Stopped/archived Unknown(String), // Provider-specific status } ``` ### ProviderError Structured errors with retry information: ```rust pub enum ProviderError { Authentication { provider: String, message: String }, RateLimit { retry_after: Option }, NotFound { resource_type: String, id: String }, QuotaExceeded { resource: String, message: String }, InvalidConfig { field: String, message: String }, Timeout { operation: String, elapsed: Duration }, Api { status: u16, message: String }, // ... others } // Useful helpers error.is_retryable() // true for RateLimit, Timeout, Network error.retry_after() // Duration to wait before retry ``` ## Current Providers | Provider | Status | File | Env Var | | ------------ | ------- | ----------------- | -------------------- | | DigitalOcean | Stable | `digitalocean.rs` | `DIGITALOCEAN_TOKEN` | | Hetzner | Planned | - | `HETZNER_TOKEN` | | AWS EC2 | Planned | - | `AWS_ACCESS_KEY_ID` | ## Documentation * [**Creating a Provider**](/providers/creating-a-provider) - Complete step-by-step guide * [**Provider API Reference**](/providers/provider-api) - Detailed documentation for each method * [**Testing Providers**](/providers/testing-providers) - Testing strategies with mocks ## Contributing Want to add support for a new provider? Follow these steps: 1. Open an issue to discuss the implementation 2. Read the [Creating a Provider](/providers/creating-a-provider) guide 3. Use the DigitalOcean implementation as reference 4. Ensure adequate test coverage 5. Update the documentation ### Design Decisions Consult the ADR to understand architectural decisions: * [ADR-0005: Provider Trait Abstraction](/adr/0005-provider-trait-abstraction) # Creating a New Provider This guide teaches you step-by-step how to implement support for a new cloud provider in spuff. We'll use Hetzner Cloud as an example, but the concepts apply to any provider. ## Table of Contents 1. [Prerequisites](#prerequisites) 2. [Architecture Overview](#architecture-overview) 3. [Step 1: Add the ProviderType](#step-1-add-the-providertype) 4. [Step 2: Create the Provider File](#step-2-create-the-provider-file) 5. [Step 3: Define API Structures](#step-3-define-api-structures) 6. [Step 4: Implement the Provider](#step-4-implement-the-provider) 7. [Step 5: Implement the Factory](#step-5-implement-the-factory) 8. [Step 6: Register in the Registry](#step-6-register-in-the-registry) 9. [Step 7: Write Tests](#step-7-write-tests) 10. [Final Checklist](#final-checklist) 11. [Common Pitfalls](#common-pitfalls) *** ## Prerequisites Before starting, you need: * Experience with Rust and async programming (`async`/`await`) * Familiarity with the provider's API you're implementing * API credentials for testing * Read the [Provider README](/providers) to understand the architecture *** ## Architecture Overview The system uses two main traits: ```mermaid flowchart LR subgraph registry["ProviderRegistry"] hashmap["HashMap<ProviderType, Arc<dyn ProviderFactory>>"] do_factory["DigitalOceanFactory"] hetzner_factory["HetznerFactory"] aws_factory["AwsFactory"] create["create_by_name('hetzner')"] provider["Box<dyn Provider>"] do_factory --> create hetzner_factory --> create aws_factory --> create create --> provider end ``` **You need to implement:** 1. `ProviderFactory` - Creates instances of your provider 2. `Provider` - Implements cloud operations *** ## Step 1: Add the ProviderType First, add the new provider to the `ProviderType` enum in `src/provider/config.rs`: ```rust // src/provider/config.rs #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)] #[serde(rename_all = "lowercase")] pub enum ProviderType { DigitalOcean, Hetzner, // ← Add here Aws, } impl ProviderType { /// Returns whether the provider is implemented and ready to use pub fn is_implemented(&self) -> bool { matches!(self, Self::DigitalOcean | Self::Hetzner // ← Add here when implemented ) } /// Provider name as string (used in configs and CLI) pub fn as_str(&self) -> &'static str { match self { Self::DigitalOcean => "digitalocean", Self::Hetzner => "hetzner", // ← Add here Self::Aws => "aws", } } /// Environment variable for the API token pub fn token_env_var(&self) -> &'static str { match self { Self::DigitalOcean => "DIGITALOCEAN_TOKEN", Self::Hetzner => "HETZNER_TOKEN", // ← Add here Self::Aws => "AWS_ACCESS_KEY_ID", } } } ``` *** ## Step 2: Create the Provider File Create a new file for the provider: ```bash touch src/provider/hetzner.rs ``` Initial file structure: ```rust // src/provider/hetzner.rs //! Hetzner Cloud Provider //! //! Provider implementation for Hetzner Cloud. //! API Documentation: https://docs.hetzner.cloud/ use async_trait::async_trait; use chrono::{DateTime, Utc}; use reqwest::Client; use serde::{Deserialize, Serialize}; use std::net::IpAddr; use std::time::Instant; use super::{ ImageSpec, InstanceRequest, InstanceStatus, Provider, ProviderInstance, ProviderResult, Snapshot, }; use super::config::ProviderTimeouts; use super::error::ProviderError; use super::registry::{ProviderFactory, ProviderType}; /// Hetzner Cloud API base URL const API_BASE: &str = "https://api.hetzner.cloud/v1"; // ============================================================================= // Provider Struct // ============================================================================= /// Hetzner Cloud provider pub struct HetznerProvider { client: Client, token: String, base_url: String, timeouts: ProviderTimeouts, } // ============================================================================= // Factory Struct // ============================================================================= /// Factory for creating HetznerProvider instances pub struct HetznerFactory; // ... implementations below ``` **Don't forget to declare the module in `src/provider/mod.rs`:** ```rust // src/provider/mod.rs mod config; mod digitalocean; mod error; mod hetzner; // ← Add here mod registry; pub use hetzner::{HetznerFactory, HetznerProvider}; // ← And here ``` *** ## Step 3: Define API Structures Define types for serializing/deserializing API responses. Hetzner uses JSON, so we use `serde`: ```rust // src/provider/hetzner.rs (continued) // ============================================================================= // API Request/Response Types // ============================================================================= /// Request to create a server #[derive(Debug, Serialize)] struct CreateServerRequest<'a> { name: &'a str, server_type: &'a str, location: &'a str, image: &'a str, #[serde(skip_serializing_if = "Option::is_none")] user_data: Option<&'a str>, labels: std::collections::HashMap, start_after_create: bool, } /// Response when creating a server #[derive(Debug, Deserialize)] struct CreateServerResponse { server: Server, action: Action, } /// Response when getting a server #[derive(Debug, Deserialize)] struct GetServerResponse { server: Server, } /// Response when listing servers #[derive(Debug, Deserialize)] struct ListServersResponse { servers: Vec, } /// Server representation in the API #[derive(Debug, Deserialize)] struct Server { id: u64, name: String, status: String, created: DateTime, public_net: PublicNet, server_type: ServerType, datacenter: Datacenter, labels: std::collections::HashMap, } #[derive(Debug, Deserialize)] struct PublicNet { ipv4: Option, ipv6: Option, } #[derive(Debug, Deserialize)] struct Ipv4Net { ip: String, } #[derive(Debug, Deserialize)] struct Ipv6Net { ip: String, } #[derive(Debug, Deserialize)] struct ServerType { name: String, } #[derive(Debug, Deserialize)] struct Datacenter { name: String, location: Location, } #[derive(Debug, Deserialize)] struct Location { name: String, } /// Async API action #[derive(Debug, Deserialize)] struct Action { id: u64, status: String, progress: u32, } /// Action response #[derive(Debug, Deserialize)] struct ActionResponse { action: Action, } /// Request to create an image (snapshot) #[derive(Debug, Serialize)] struct CreateImageRequest<'a> { description: &'a str, r#type: &'a str, labels: std::collections::HashMap, } /// Response when creating an image #[derive(Debug, Deserialize)] struct CreateImageResponse { image: Image, action: Action, } /// Response when listing images #[derive(Debug, Deserialize)] struct ListImagesResponse { images: Vec, } /// Image representation in the API #[derive(Debug, Deserialize)] struct Image { id: u64, description: Option, image_size: Option, created: DateTime, } /// API error response #[derive(Debug, Deserialize)] struct ApiError { error: ApiErrorDetail, } #[derive(Debug, Deserialize)] struct ApiErrorDetail { code: String, message: String, } ``` **Tip:** You don't need to map all API fields. Only use what you actually need. Unknown fields are ignored by serde by default. *** ## Step 4: Implement the Provider Now implement the `Provider` trait. I'll detail each method: ### 4.1 Constructor and Helpers ```rust // src/provider/hetzner.rs (continued) // ============================================================================= // Provider Implementation // ============================================================================= impl HetznerProvider { /// Creates a new provider with default configuration pub fn new(token: &str, timeouts: ProviderTimeouts) -> ProviderResult { Self::with_config(token, API_BASE, timeouts) } /// Creates a provider with custom base URL (useful for tests) pub fn with_config( token: &str, base_url: &str, timeouts: ProviderTimeouts, ) -> ProviderResult { if token.is_empty() { return Err(ProviderError::invalid_config( "token", "API token cannot be empty", )); } let client = Client::builder() .timeout(timeouts.http_request) .build() .map_err(|e| ProviderError::Other { message: format!("Failed to create HTTP client: {}", e), })?; Ok(Self { client, token: token.to_string(), base_url: base_url.to_string(), timeouts, }) } /// Makes an authenticated request to the API async fn request Deserialize<'de>>( &self, method: reqwest::Method, endpoint: &str, body: Option<&impl Serialize>, ) -> ProviderResult { let url = format!("{}{}", self.base_url, endpoint); let mut request = self .client .request(method, &url) .header("Authorization", format!("Bearer {}", self.token)) .header("Content-Type", "application/json"); if let Some(body) = body { request = request.json(body); } let response = request.send().await?; let status = response.status(); // Handle specific HTTP errors if !status.is_success() { let body = response.text().await.unwrap_or_default(); return Err(self.map_api_error(status.as_u16(), &body)); } response.json().await.map_err(|e| ProviderError::Other { message: format!("Failed to parse response: {}", e), }) } /// Maps API errors to ProviderError fn map_api_error(&self, status: u16, body: &str) -> ProviderError { // Try to parse structured API error if let Ok(api_error) = serde_json::from_str::(body) { match api_error.error.code.as_str() { "unauthorized" | "forbidden" => { return ProviderError::auth("hetzner", api_error.error.message); } "rate_limit_exceeded" => { return ProviderError::RateLimit { retry_after: Some(std::time::Duration::from_secs(60)), }; } "not_found" => { return ProviderError::NotFound { resource_type: "resource".to_string(), id: "unknown".to_string(), }; } _ => {} } } // Fallback based on HTTP status match status { 401 | 403 => ProviderError::auth("hetzner", body.to_string()), 404 => ProviderError::NotFound { resource_type: "resource".to_string(), id: "unknown".to_string(), }, 422 => ProviderError::InvalidConfig { field: "request".to_string(), message: body.to_string(), }, 429 => ProviderError::RateLimit { retry_after: Some(std::time::Duration::from_secs(60)), }, _ => ProviderError::api(status, body.to_string()), } } /// Converts ImageSpec to Hetzner image slug fn resolve_image(&self, spec: &ImageSpec) -> String { match spec { ImageSpec::Ubuntu(version) => { // Hetzner uses slugs like "ubuntu-24.04" format!("ubuntu-{}", version) } ImageSpec::Debian(version) => { format!("debian-{}", version) } ImageSpec::Custom(id) => id.clone(), ImageSpec::Snapshot(id) => id.clone(), } } /// Maps Hetzner status to InstanceStatus fn map_status(status: &str) -> InstanceStatus { match status { "initializing" | "starting" => InstanceStatus::New, "running" => InstanceStatus::Active, "stopping" | "off" => InstanceStatus::Off, other => InstanceStatus::Unknown(other.to_string()), } } /// Extracts public IP from response fn extract_public_ip(public_net: &PublicNet) -> IpAddr { // Prefer IPv4, fallback to IPv6 if let Some(ipv4) = &public_net.ipv4 { if let Ok(ip) = ipv4.ip.parse() { return ip; } } if let Some(ipv6) = &public_net.ipv6 { if let Ok(ip) = ipv6.ip.parse() { return ip; } } // Fallback: IP not assigned yet "0.0.0.0".parse().unwrap() } /// Waits for an async action to complete async fn wait_for_action(&self, action_id: u64) -> ProviderResult<()> { let start = Instant::now(); let max_attempts = self.timeouts.action_complete_attempts(); for _attempt in 0..max_attempts { if start.elapsed() > self.timeouts.action_complete { return Err(ProviderError::timeout( "action_complete", start.elapsed(), )); } let response: ActionResponse = self .request( reqwest::Method::GET, &format!("/actions/{}", action_id), None::<&()>, ) .await?; match response.action.status.as_str() { "success" => return Ok(()), "error" => { return Err(ProviderError::api( 500, format!("Action {} failed", action_id), )); } _ => { // "running" or "pending" - keep waiting tokio::time::sleep(self.timeouts.poll_interval).await; } } } Err(ProviderError::timeout("action_complete", start.elapsed())) } } ``` ### 4.2 Implement the Provider Trait ```rust // src/provider/hetzner.rs (continued) #[async_trait] impl Provider for HetznerProvider { /// Provider name for logs and identification fn name(&self) -> &'static str { "hetzner" } /// Creates a new instance /// /// IMPORTANT: This method returns immediately after the API accepts the request. /// The instance may still be creating. Use `wait_ready()` to wait. async fn create_instance(&self, request: &InstanceRequest) -> ProviderResult { // Build labels (Hetzner uses key-value, not array) let mut labels = request.labels.clone(); labels.insert("managed-by".to_string(), "spuff".to_string()); let api_request = CreateServerRequest { name: &request.name, server_type: &request.size, location: &request.region, image: &self.resolve_image(&request.image), user_data: request.user_data.as_deref(), labels, start_after_create: true, }; let response: CreateServerResponse = self .request(reqwest::Method::POST, "/servers", Some(&api_request)) .await?; Ok(ProviderInstance { id: response.server.id.to_string(), ip: Self::extract_public_ip(&response.server.public_net), status: Self::map_status(&response.server.status), created_at: response.server.created, }) } /// Destroys an instance /// /// IMPORTANT: This method is idempotent. Calling on a non-existent instance /// returns Ok(()) instead of an error. async fn destroy_instance(&self, id: &str) -> ProviderResult<()> { match self .request::( reqwest::Method::DELETE, &format!("/servers/{}", id), None::<&()>, ) .await { Ok(_) => Ok(()), Err(ProviderError::NotFound { .. }) => Ok(()), // Idempotent Err(e) => Err(e), } } /// Gets instance details /// /// Returns None if the instance doesn't exist. async fn get_instance(&self, id: &str) -> ProviderResult> { match self .request::( reqwest::Method::GET, &format!("/servers/{}", id), None::<&()>, ) .await { Ok(response) => Ok(Some(ProviderInstance { id: response.server.id.to_string(), ip: Self::extract_public_ip(&response.server.public_net), status: Self::map_status(&response.server.status), created_at: response.server.created, })), Err(ProviderError::NotFound { .. }) => Ok(None), Err(e) => Err(e), } } /// Lists all spuff-managed instances async fn list_instances(&self) -> ProviderResult> { // Filter by managed-by=spuff label let response: ListServersResponse = self .request( reqwest::Method::GET, "/servers?label_selector=managed-by=spuff", None::<&()>, ) .await?; Ok(response .servers .into_iter() .map(|server| ProviderInstance { id: server.id.to_string(), ip: Self::extract_public_ip(&server.public_net), status: Self::map_status(&server.status), created_at: server.created, }) .collect()) } /// Waits for instance to be ready (running + public IP) /// /// This method polls until: /// 1. Status is "Active" /// 2. Public IP is assigned (not 0.0.0.0) /// /// Default timeout: 5 minutes (configurable via ProviderTimeouts) async fn wait_ready(&self, id: &str) -> ProviderResult { let start = Instant::now(); let max_attempts = self.timeouts.instance_ready_attempts(); for _attempt in 0..max_attempts { if start.elapsed() > self.timeouts.instance_ready { return Err(ProviderError::timeout("wait_ready", start.elapsed())); } if let Some(instance) = self.get_instance(id).await? { // Check if running AND has IP let has_ip = !instance.ip.is_unspecified(); let is_active = matches!(instance.status, InstanceStatus::Active); if is_active && has_ip { return Ok(instance); } } tokio::time::sleep(self.timeouts.poll_interval).await; } Err(ProviderError::timeout("wait_ready", start.elapsed())) } /// Creates a snapshot of the instance /// /// In Hetzner, snapshots are called "images" of type "snapshot". async fn create_snapshot(&self, instance_id: &str, name: &str) -> ProviderResult { let mut labels = std::collections::HashMap::new(); labels.insert("managed-by".to_string(), "spuff".to_string()); let api_request = CreateImageRequest { description: name, r#type: "snapshot", labels, }; let response: CreateImageResponse = self .request( reqwest::Method::POST, &format!("/servers/{}/actions/create_image", instance_id), Some(&api_request), ) .await?; // Wait for creation action to complete self.wait_for_action(response.action.id).await?; Ok(Snapshot { id: response.image.id.to_string(), name: name.to_string(), created_at: Some(response.image.created), }) } /// Lists all spuff-managed snapshots async fn list_snapshots(&self) -> ProviderResult> { let response: ListImagesResponse = self .request( reqwest::Method::GET, "/images?type=snapshot&label_selector=managed-by=spuff", None::<&()>, ) .await?; Ok(response .images .into_iter() .map(|image| Snapshot { id: image.id.to_string(), name: image.description.unwrap_or_default(), created_at: Some(image.created), }) .collect()) } /// Deletes a snapshot /// /// IMPORTANT: Idempotent - deleting a non-existent snapshot returns Ok(()) async fn delete_snapshot(&self, id: &str) -> ProviderResult<()> { match self .request::( reqwest::Method::DELETE, &format!("/images/{}", id), None::<&()>, ) .await { Ok(_) => Ok(()), Err(ProviderError::NotFound { .. }) => Ok(()), // Idempotent Err(e) => Err(e), } } /// Returns whether the provider supports snapshots fn supports_snapshots(&self) -> bool { true } } ``` *** ## Step 5: Implement the Factory The Factory is simple - it knows how to create provider instances: ```rust // src/provider/hetzner.rs (continued) // ============================================================================= // Factory Implementation // ============================================================================= impl ProviderFactory for HetznerFactory { /// Type of provider this factory creates fn provider_type(&self) -> ProviderType { ProviderType::Hetzner } /// Creates a new provider instance fn create( &self, token: &str, timeouts: ProviderTimeouts, ) -> ProviderResult> { Ok(Box::new(HetznerProvider::new(token, timeouts)?)) } } ``` *** ## Step 6: Register in the Registry Add the factory to the registry in `src/provider/registry.rs`: ```rust // src/provider/registry.rs use super::hetzner::HetznerFactory; // ← Add import impl ProviderRegistry { /// Creates a registry with all default providers registered pub fn with_defaults() -> Self { let mut registry = Self::new(); // Register all implemented providers registry.register(DigitalOceanFactory); registry.register(HetznerFactory); // ← Add here // registry.register(AwsFactory); // When implemented registry } } ``` *** ## Step 7: Write Tests See [testing-providers.md](/providers/testing-providers) for the complete testing guide. Here's a basic example: ```rust // src/provider/hetzner.rs (continued) #[cfg(test)] mod tests { use super::*; use wiremock::{Mock, MockServer, ResponseTemplate}; use wiremock::matchers::{method, path, header}; async fn create_test_provider(mock_server: &MockServer) -> HetznerProvider { HetznerProvider::with_config( "test-token", &mock_server.uri(), ProviderTimeouts::default(), ) .unwrap() } #[tokio::test] async fn test_create_instance_success() { let mock_server = MockServer::start().await; Mock::given(method("POST")) .and(path("/servers")) .and(header("Authorization", "Bearer test-token")) .respond_with(ResponseTemplate::new(201).set_body_json(serde_json::json!({ "server": { "id": 123, "name": "spuff-test", "status": "initializing", "created": "2024-01-01T00:00:00Z", "public_net": { "ipv4": null, "ipv6": null }, "server_type": { "name": "cx11" }, "datacenter": { "name": "fsn1-dc14", "location": { "name": "fsn1" } }, "labels": {} }, "action": { "id": 1, "status": "running", "progress": 0 } }))) .mount(&mock_server) .await; let provider = create_test_provider(&mock_server).await; let request = InstanceRequest { name: "spuff-test".to_string(), region: "fsn1".to_string(), size: "cx11".to_string(), image: ImageSpec::Ubuntu("24.04".to_string()), user_data: None, labels: std::collections::HashMap::new(), }; let result = provider.create_instance(&request).await; assert!(result.is_ok()); let instance = result.unwrap(); assert_eq!(instance.id, "123"); assert!(matches!(instance.status, InstanceStatus::New)); } #[tokio::test] async fn test_destroy_instance_not_found_is_ok() { let mock_server = MockServer::start().await; Mock::given(method("DELETE")) .and(path("/servers/999")) .respond_with(ResponseTemplate::new(404).set_body_json(serde_json::json!({ "error": { "code": "not_found", "message": "Server not found" } }))) .mount(&mock_server) .await; let provider = create_test_provider(&mock_server).await; // Should return Ok even with 404 (idempotent) let result = provider.destroy_instance("999").await; assert!(result.is_ok()); } #[tokio::test] async fn test_authentication_error() { let mock_server = MockServer::start().await; Mock::given(method("GET")) .and(path("/servers")) .respond_with(ResponseTemplate::new(401).set_body_json(serde_json::json!({ "error": { "code": "unauthorized", "message": "Invalid token" } }))) .mount(&mock_server) .await; let provider = create_test_provider(&mock_server).await; let result = provider.list_instances().await; assert!(matches!( result, Err(ProviderError::Authentication { .. }) )); } } ``` *** ## Final Checklist Before submitting your PR, verify: ### Code * [ ] All `Provider` trait methods implemented * [ ] `ProviderFactory` implemented and registered * [ ] `ProviderType` added to enum * [ ] Module declared in `mod.rs` * [ ] Error handling using `ProviderError` * [ ] `destroy_instance` and `delete_snapshot` are idempotent * [ ] Timeouts respected in `wait_ready` and long operations ### Tests * [ ] Unit tests with mock server * [ ] Success case test for each method * [ ] Error tests (authentication, not found, rate limit) * [ ] Idempotency test for destroy/delete ### Documentation * [ ] Docstrings on all public methods * [ ] Example configuration added * [ ] README updated with new provider *** ## Common Pitfalls ### 1. Image Mapping Each provider uses different conventions for identifying images: | Provider | Format | Example | | ------------ | ------ | ----------------------- | | DigitalOcean | Slug | `ubuntu-24-04-x64` | | Hetzner | Slug | `ubuntu-24.04` | | AWS | AMI ID | `ami-0c55b159cbfafe1f0` | **Solution:** Implement `resolve_image()` correctly for each provider. ### 2. Public IP Extraction Providers return IPs in different ways: * Some in network array * Some in specific field * Some with separate IPv4 and IPv6 **Solution:** Always prefer IPv4 and handle the case when IP is not yet assigned (use `0.0.0.0`). ### 3. Labels vs Tags * **DigitalOcean:** Tags are string array * **Hetzner/AWS:** Labels are key-value pairs **Solution:** Convert `HashMap` to the format the provider expects. ### 4. Async Actions Some operations (especially snapshots) are asynchronous: 1. API returns an action ID 2. You need to poll until complete **Solution:** Implement `wait_for_action()` with timeout. ### 5. Rate Limiting Each provider has different limits. Hetzner is more restrictive than DigitalOcean. **Solution:** * Return `ProviderError::RateLimit` with `retry_after` * Respect the `poll_interval` from timeouts ### 6. User Data Encoding * **DigitalOcean:** Accepts raw string * **Hetzner:** Accepts raw string * **AWS:** Requires base64 **Solution:** Do the necessary encoding in `create_instance()`. *** ## Need Help? * Review the DigitalOcean implementation (`src/provider/digitalocean.rs`) as reference * Open an issue to discuss before starting * Consult the provider ADR to understand design decisions # Provider API Reference Complete reference for the Provider trait and associated types. ## Provider Trait The `Provider` trait defines the contract that all cloud providers must implement: ```rust #[async_trait] pub trait Provider: Send + Sync { /// Returns the provider name for logging and identification fn name(&self) -> &'static str; /// Creates a new cloud instance async fn create_instance(&self, request: &InstanceRequest) -> ProviderResult; /// Destroys an instance by ID (must be idempotent) async fn destroy_instance(&self, id: &str) -> ProviderResult<()>; /// Gets instance details by ID, returns None if not found async fn get_instance(&self, id: &str) -> ProviderResult>; /// Lists all spuff-managed instances async fn list_instances(&self) -> ProviderResult>; /// Waits for instance to be ready (running + has IP) async fn wait_ready(&self, id: &str) -> ProviderResult; /// Creates a snapshot of an instance async fn create_snapshot(&self, instance_id: &str, name: &str) -> ProviderResult; /// Lists all spuff-managed snapshots async fn list_snapshots(&self) -> ProviderResult>; /// Deletes a snapshot by ID (must be idempotent) async fn delete_snapshot(&self, id: &str) -> ProviderResult<()>; /// Returns SSH key identifiers (optional, default returns empty) async fn get_ssh_keys(&self) -> ProviderResult> { Ok(vec![]) } /// Returns whether this provider supports snapshots fn supports_snapshots(&self) -> bool { true } } ``` *** ## Core Types ### InstanceRequest Configuration for creating a new instance. This is provider-agnostic - each provider translates it to their API format. ```rust pub struct InstanceRequest { /// Unique instance name (e.g., "spuff-a1b2c3d4") pub name: String, /// Region/datacenter identifier (provider-specific) pub region: String, /// Instance size/type identifier (provider-specific) pub size: String, /// Base image specification pub image: ImageSpec, /// Cloud-init user data script (raw YAML, not base64) pub user_data: Option, /// Labels/tags for identifying spuff instances pub labels: HashMap, } ``` **Notes:** * `name` should be unique within the account * `user_data` is raw cloud-init YAML - providers handle encoding if needed * `labels` should include identifiers for filtering (e.g., `managed-by: spuff`) ### ImageSpec Provider-agnostic image specification: ```rust pub enum ImageSpec { /// Ubuntu version (e.g., "24.04") /// Provider maps to appropriate slug/ID Ubuntu(String), /// Debian version (e.g., "12") Debian(String), /// Provider-specific image ID/slug Custom(String), /// Snapshot ID to restore from Snapshot(String), } ``` **Provider Mapping Examples:** | ImageSpec | DigitalOcean | Hetzner | AWS | | ----------------- | ------------------ | -------------- | ------------------ | | `Ubuntu("24.04")` | `ubuntu-24-04-x64` | `ubuntu-24.04` | `ami-xxx` (lookup) | | `Debian("12")` | `debian-12-x64` | `debian-12` | `ami-xxx` (lookup) | | `Custom(id)` | pass through | pass through | pass through | | `Snapshot(id)` | pass through | pass through | pass through | ### ProviderInstance Represents a cloud instance: ```rust pub struct ProviderInstance { /// Provider-specific instance ID pub id: String, /// Public IP address (or 0.0.0.0 if not yet assigned) pub ip: IpAddr, /// Current instance status pub status: InstanceStatus, /// Creation timestamp pub created_at: DateTime, } ``` ### InstanceStatus Enum representing instance states: ```rust pub enum InstanceStatus { /// Instance is being created New, /// Instance is running and accessible Active, /// Instance is powered off Off, /// Instance is stopped/archived Archive, /// Provider-specific status not mapped Unknown(String), } ``` **Status Mapping Guidelines:** | Provider State | Spuff Status | | ------------------------------------ | ---------------- | | new, initializing, starting, pending | `New` | | active, running | `Active` | | stopping, off, stopped | `Off` | | archive, terminated | `Archive` | | (anything else) | `Unknown(state)` | ### Snapshot Represents a saved instance state: ```rust pub struct Snapshot { /// Provider-specific snapshot ID pub id: String, /// Snapshot name/description pub name: String, /// Creation timestamp (optional - some providers don't return this) pub created_at: Option>, } ``` ### ProviderTimeouts Configurable timeout values for provider operations: ```rust pub struct ProviderTimeouts { /// Maximum time to wait for instance to be ready /// Default: 300 seconds (5 minutes) pub instance_ready: Duration, /// Maximum time to wait for an action to complete /// Default: 600 seconds (10 minutes) pub action_complete: Duration, /// Interval between polling requests /// Default: 5 seconds pub poll_interval: Duration, /// Timeout for individual HTTP requests /// Default: 30 seconds pub http_request: Duration, /// Timeout for SSH connection attempts /// Default: 300 seconds (5 minutes) pub ssh_connect: Duration, /// Timeout for cloud-init to complete /// Default: 600 seconds (10 minutes) pub cloud_init: Duration, } ``` **Helper Methods:** ```rust impl ProviderTimeouts { /// Returns max attempts for instance ready polling pub fn instance_ready_attempts(&self) -> u32 { (self.instance_ready.as_secs() / self.poll_interval.as_secs()) as u32 } /// Returns max attempts for action complete polling pub fn action_complete_attempts(&self) -> u32 { (self.action_complete.as_secs() / self.poll_interval.as_secs()) as u32 } } ``` *** ## Error Types ### ProviderError Structured error types for proper handling and retry logic: ```rust pub enum ProviderError { /// Authentication failed (invalid or expired token) Authentication { provider: String, message: String, }, /// Rate limit exceeded - should retry after duration RateLimit { retry_after: Option, }, /// Resource not found NotFound { resource_type: String, id: String, }, /// Quota/limit exceeded (e.g., max droplets) QuotaExceeded { resource: String, message: String, }, /// Invalid configuration InvalidConfig { field: String, message: String, }, /// Feature not supported by this provider NotSupported { feature: String, }, /// Operation timed out Timeout { operation: String, elapsed: Duration, }, /// Network/HTTP error Network(#[from] reqwest::Error), /// API error with status code Api { status: u16, message: String, }, /// Provider not implemented yet NotImplemented { name: String, }, /// Unknown provider name UnknownProvider { name: String, supported: Vec, }, /// Generic error Other { message: String, }, } ``` **Helper Constructors:** ```rust ProviderError::auth(provider, message) // Creates Authentication error ProviderError::not_found(resource_type, id) // Creates NotFound error ProviderError::api(status_code, message) // Creates Api error ProviderError::timeout(operation, elapsed) // Creates Timeout error ProviderError::quota(resource, message) // Creates QuotaExceeded error ProviderError::invalid_config(field, message) // Creates InvalidConfig error ``` **Retry Logic:** ```rust impl ProviderError { /// Returns true if this error is potentially retryable pub fn is_retryable(&self) -> bool { matches!(self, Self::RateLimit { .. } | Self::Timeout { .. } | Self::Network(_) ) } /// Returns duration to wait before retrying, if applicable pub fn retry_after(&self) -> Option { match self { Self::RateLimit { retry_after } => *retry_after, Self::Timeout { .. } => Some(Duration::from_secs(5)), Self::Network(_) => Some(Duration::from_secs(1)), _ => None, } } } ``` *** ## Method Specifications ### name Returns the provider name for logging and identification. ```rust fn name(&self) -> &'static str; ``` **Returns:** Static string with provider name (e.g., `"digitalocean"`, `"hetzner"`) *** ### create\_instance Creates a new cloud instance with the specified configuration. ```rust async fn create_instance(&self, request: &InstanceRequest) -> ProviderResult; ``` **Parameters:** * `request: &InstanceRequest` - Instance configuration **Returns:** * `ProviderResult` - Created instance (may not be ready yet) **Behavior:** 1. Translate `InstanceRequest` to provider-specific API request 2. Resolve `ImageSpec` to provider-specific image ID/slug 3. Send API request to create instance 4. Return immediately with instance metadata 5. Instance may still be initializing - use `wait_ready()` to wait **Error Cases:** * `Authentication` - Invalid API token * `InvalidConfig` - Invalid region, size, or image * `QuotaExceeded` - Account limit reached * `Api` - Other API errors **Example:** ```rust let request = InstanceRequest { name: "spuff-abc123".to_string(), region: "nyc1".to_string(), size: "s-2vcpu-4gb".to_string(), image: ImageSpec::Ubuntu("24.04".to_string()), user_data: Some(cloud_init_script), labels: HashMap::from([("managed-by".to_string(), "spuff".to_string())]), }; let instance = provider.create_instance(&request).await?; println!("Created instance: {} (status: {:?})", instance.id, instance.status); // Wait for it to be ready let ready = provider.wait_ready(&instance.id).await?; println!("Instance ready at: {}", ready.ip); ``` *** ### destroy\_instance Destroys an instance by ID. ```rust async fn destroy_instance(&self, id: &str) -> ProviderResult<()>; ``` **Parameters:** * `id: &str` - Instance ID to destroy **Returns:** * `ProviderResult<()>` - Success or error **Behavior:** 1. Send delete request to provider API 2. Do not wait for deletion to complete 3. **MUST be idempotent**: Return `Ok(())` if instance doesn't exist (404) **Example:** ```rust // Safe to call multiple times provider.destroy_instance("12345678").await?; provider.destroy_instance("12345678").await?; // Still returns Ok(()) ``` *** ### get\_instance Gets instance details by ID. ```rust async fn get_instance(&self, id: &str) -> ProviderResult>; ``` **Parameters:** * `id: &str` - Instance ID **Returns:** * `ProviderResult>` - Instance if found, `None` if not exists **Behavior:** 1. Query provider API for instance 2. Return `None` if instance doesn't exist (404) 3. Return instance details if found **Example:** ```rust match provider.get_instance("12345678").await? { Some(instance) => println!("Found: {} at {}", instance.id, instance.ip), None => println!("Instance not found"), } ``` *** ### list\_instances Lists all spuff-managed instances. ```rust async fn list_instances(&self) -> ProviderResult>; ``` **Returns:** * `ProviderResult>` - List of instances **Behavior:** 1. Query provider API with spuff label/tag filter 2. Return only instances tagged with spuff identifiers 3. Handle pagination if needed **Filter Requirements:** * Only return instances with `managed-by: spuff` label (or equivalent) * This prevents listing unrelated instances in the account *** ### wait\_ready Waits for instance to be fully ready. ```rust async fn wait_ready(&self, id: &str) -> ProviderResult; ``` **Parameters:** * `id: &str` - Instance ID **Returns:** * `ProviderResult` - Ready instance with IP address **Behavior:** 1. Poll `get_instance()` periodically 2. Check for `Active` status AND non-unspecified IP 3. Return when both conditions are met 4. Timeout after `ProviderTimeouts::instance_ready` **Ready Conditions:** * `status == InstanceStatus::Active` * `ip.is_unspecified() == false` (not 0.0.0.0) **Example:** ```rust let instance = provider.create_instance(&config).await?; println!("Waiting for instance to be ready..."); let ready = provider.wait_ready(&instance.id).await?; println!("Instance ready at {}", ready.ip); ``` *** ### create\_snapshot Creates a snapshot of an instance. ```rust async fn create_snapshot(&self, instance_id: &str, name: &str) -> ProviderResult; ``` **Parameters:** * `instance_id: &str` - Instance to snapshot * `name: &str` - Snapshot name/description **Returns:** * `ProviderResult` - Created snapshot **Behavior:** 1. Initiate snapshot creation 2. Wait for completion if the operation is async (using action polling) 3. Tag snapshot with spuff identifiers for filtering 4. Return snapshot metadata **Note:** Some providers (like Hetzner) have async snapshot creation that returns an action ID. The implementation should wait for the action to complete before returning. *** ### list\_snapshots Lists all spuff-managed snapshots. ```rust async fn list_snapshots(&self) -> ProviderResult>; ``` **Returns:** * `ProviderResult>` - List of snapshots **Behavior:** 1. Query provider API with spuff filter 2. Return only spuff-tagged snapshots 3. Handle pagination if needed *** ### delete\_snapshot Deletes a snapshot by ID. ```rust async fn delete_snapshot(&self, id: &str) -> ProviderResult<()>; ``` **Parameters:** * `id: &str` - Snapshot ID **Returns:** * `ProviderResult<()>` - Success or error **Behavior:** * **MUST be idempotent**: Return `Ok(())` if snapshot doesn't exist (404) *** ## ProviderFactory Trait The factory trait for creating provider instances: ```rust pub trait ProviderFactory: Send + Sync { /// Returns the type of provider this factory creates fn provider_type(&self) -> ProviderType; /// Creates a new provider instance fn create( &self, token: &str, timeouts: ProviderTimeouts, ) -> ProviderResult>; /// Returns whether this provider is implemented fn is_implemented(&self) -> bool { self.provider_type().is_implemented() } } ``` *** ## ProviderRegistry Registry for managing provider factories: ```rust pub struct ProviderRegistry { factories: HashMap>, } impl ProviderRegistry { /// Creates an empty registry pub fn new() -> Self; /// Creates a registry with all default providers registered pub fn with_defaults() -> Self; /// Registers a provider factory pub fn register(&mut self, factory: F); /// Creates a provider by name pub fn create_by_name( &self, name: &str, token: &str, timeouts: ProviderTimeouts, ) -> ProviderResult>; /// Returns list of registered provider types pub fn registered_providers(&self) -> Vec; /// Returns list of implemented (ready to use) provider types pub fn implemented_providers(&self) -> Vec; } ``` **Usage:** ```rust // Create registry with defaults let registry = ProviderRegistry::with_defaults(); // List available providers for provider_type in registry.implemented_providers() { println!("Available: {}", provider_type.as_str()); } // Create a specific provider let provider = registry.create_by_name( "digitalocean", &api_token, ProviderTimeouts::default(), )?; ``` *** ## Type Aliases ```rust /// Result type for provider operations pub type ProviderResult = Result; ``` # Testing Providers This guide covers testing strategies for cloud provider implementations. ## Testing Levels ```mermaid flowchart BT unit["Unit Tests
(Mocked API responses)
cargo test"] integration["Integration Tests
(Real API, real resources)
cargo test --ignored"] unit --> integration ``` ## Unit Tests with Mocked API ### Setup Add test dependencies to `Cargo.toml`: ```toml [dev-dependencies] wiremock = "0.6" tokio-test = "0.4" serde_json = "1.0" ``` ### Creating a Testable Provider Add a constructor that accepts a custom base URL: ```rust impl HetznerProvider { /// Creates a provider with production URL pub fn new(token: &str, timeouts: ProviderTimeouts) -> ProviderResult { Self::with_config(token, API_BASE, timeouts) } /// Creates a provider with custom base URL (for testing) pub fn with_config( token: &str, base_url: &str, timeouts: ProviderTimeouts, ) -> ProviderResult { // ... implementation } } ``` ### Test Structure ```rust #[cfg(test)] mod tests { use super::*; use wiremock::{Mock, MockServer, ResponseTemplate}; use wiremock::matchers::{method, path, header, body_json}; // Helper to create test config fn test_request() -> InstanceRequest { InstanceRequest { name: "spuff-test".to_string(), region: "fsn1".to_string(), size: "cx11".to_string(), image: ImageSpec::Ubuntu("24.04".to_string()), user_data: None, labels: HashMap::new(), } } // Helper to create mock provider async fn create_mock_provider(mock_server: &MockServer) -> HetznerProvider { HetznerProvider::with_config( "test-token", &mock_server.uri(), ProviderTimeouts::default(), ) .unwrap() } } ``` *** ## Testing Each Method ### Testing create\_instance ```rust #[tokio::test] async fn test_create_instance_success() { let mock_server = MockServer::start().await; // Setup mock Mock::given(method("POST")) .and(path("/servers")) .and(header("Authorization", "Bearer test-token")) .respond_with(ResponseTemplate::new(201).set_body_json(serde_json::json!({ "server": { "id": 123, "name": "spuff-test", "status": "initializing", "created": "2024-01-01T00:00:00Z", "public_net": { "ipv4": null, "ipv6": null }, "server_type": { "name": "cx11" }, "datacenter": { "name": "fsn1-dc14", "location": { "name": "fsn1" } }, "labels": {} }, "action": { "id": 1, "status": "running", "progress": 0 } }))) .expect(1) // Verify called exactly once .mount(&mock_server) .await; let provider = create_mock_provider(&mock_server).await; let result = provider.create_instance(&test_request()).await; assert!(result.is_ok()); let instance = result.unwrap(); assert_eq!(instance.id, "123"); assert_eq!(instance.status, InstanceStatus::New); } #[tokio::test] async fn test_create_instance_api_error() { let mock_server = MockServer::start().await; Mock::given(method("POST")) .and(path("/servers")) .respond_with(ResponseTemplate::new(422).set_body_json(serde_json::json!({ "error": { "code": "invalid_input", "message": "Invalid server type" } }))) .mount(&mock_server) .await; let provider = create_mock_provider(&mock_server).await; let result = provider.create_instance(&test_request()).await; assert!(result.is_err()); assert!(matches!(result.unwrap_err(), ProviderError::InvalidConfig { .. })); } #[tokio::test] async fn test_create_instance_network_error() { // Use an invalid URL to simulate network error let provider = HetznerProvider::with_config( "test-token", "http://localhost:99999", ProviderTimeouts::default(), ).unwrap(); let result = provider.create_instance(&test_request()).await; assert!(result.is_err()); assert!(matches!(result.unwrap_err(), ProviderError::Network(_))); } ``` ### Testing destroy\_instance ```rust #[tokio::test] async fn test_destroy_instance_success() { let mock_server = MockServer::start().await; Mock::given(method("DELETE")) .and(path("/servers/123")) .respond_with(ResponseTemplate::new(204)) .expect(1) .mount(&mock_server) .await; let provider = create_mock_provider(&mock_server).await; let result = provider.destroy_instance("123").await; assert!(result.is_ok()); } #[tokio::test] async fn test_destroy_instance_not_found_is_ok() { let mock_server = MockServer::start().await; Mock::given(method("DELETE")) .and(path("/servers/999")) .respond_with(ResponseTemplate::new(404).set_body_json(serde_json::json!({ "error": { "code": "not_found", "message": "Server not found" } }))) .mount(&mock_server) .await; let provider = create_mock_provider(&mock_server).await; let result = provider.destroy_instance("999").await; // Idempotent: 404 should return Ok assert!(result.is_ok()); } ``` ### Testing get\_instance ```rust #[tokio::test] async fn test_get_instance_found() { let mock_server = MockServer::start().await; Mock::given(method("GET")) .and(path("/servers/123")) .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ "server": { "id": 123, "name": "spuff-test", "status": "running", "created": "2024-01-01T00:00:00Z", "public_net": { "ipv4": { "ip": "1.2.3.4" }, "ipv6": null }, "server_type": { "name": "cx11" }, "datacenter": { "name": "fsn1-dc14", "location": { "name": "fsn1" } }, "labels": {} } }))) .mount(&mock_server) .await; let provider = create_mock_provider(&mock_server).await; let result = provider.get_instance("123").await; assert!(result.is_ok()); let instance = result.unwrap(); assert!(instance.is_some()); let instance = instance.unwrap(); assert_eq!(instance.ip.to_string(), "1.2.3.4"); assert_eq!(instance.status, InstanceStatus::Active); } #[tokio::test] async fn test_get_instance_not_found() { let mock_server = MockServer::start().await; Mock::given(method("GET")) .and(path("/servers/999")) .respond_with(ResponseTemplate::new(404).set_body_json(serde_json::json!({ "error": { "code": "not_found", "message": "Server not found" } }))) .mount(&mock_server) .await; let provider = create_mock_provider(&mock_server).await; let result = provider.get_instance("999").await; assert!(result.is_ok()); assert!(result.unwrap().is_none()); } ``` ### Testing wait\_ready ```rust #[tokio::test] async fn test_wait_ready_immediate() { let mock_server = MockServer::start().await; // Instance is already ready Mock::given(method("GET")) .and(path("/servers/123")) .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ "server": { "id": 123, "name": "spuff-test", "status": "running", "created": "2024-01-01T00:00:00Z", "public_net": { "ipv4": { "ip": "1.2.3.4" }, "ipv6": null }, "server_type": { "name": "cx11" }, "datacenter": { "name": "fsn1-dc14", "location": { "name": "fsn1" } }, "labels": {} } }))) .expect(1) .mount(&mock_server) .await; let provider = create_mock_provider(&mock_server).await; let result = provider.wait_ready("123").await; assert!(result.is_ok()); assert_eq!(result.unwrap().ip.to_string(), "1.2.3.4"); } #[tokio::test] async fn test_wait_ready_multiple_polls() { let mock_server = MockServer::start().await; // First call: still starting, no IP Mock::given(method("GET")) .and(path("/servers/123")) .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ "server": { "id": 123, "name": "spuff-test", "status": "initializing", "created": "2024-01-01T00:00:00Z", "public_net": { "ipv4": null, "ipv6": null }, "server_type": { "name": "cx11" }, "datacenter": { "name": "fsn1-dc14", "location": { "name": "fsn1" } }, "labels": {} } }))) .up_to_n_times(2) .mount(&mock_server) .await; // Subsequent calls: ready with IP Mock::given(method("GET")) .and(path("/servers/123")) .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ "server": { "id": 123, "name": "spuff-test", "status": "running", "created": "2024-01-01T00:00:00Z", "public_net": { "ipv4": { "ip": "1.2.3.4" }, "ipv6": null }, "server_type": { "name": "cx11" }, "datacenter": { "name": "fsn1-dc14", "location": { "name": "fsn1" } }, "labels": {} } }))) .mount(&mock_server) .await; // Use fast timeouts for testing let timeouts = ProviderTimeouts { poll_interval: Duration::from_millis(10), instance_ready: Duration::from_secs(5), ..Default::default() }; let provider = HetznerProvider::with_config( "test-token", &mock_server.uri(), timeouts, ).unwrap(); let result = provider.wait_ready("123").await; assert!(result.is_ok()); assert_eq!(result.unwrap().ip.to_string(), "1.2.3.4"); } #[tokio::test] async fn test_wait_ready_timeout() { let mock_server = MockServer::start().await; // Always returns not ready Mock::given(method("GET")) .and(path("/servers/123")) .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ "server": { "id": 123, "name": "spuff-test", "status": "initializing", "created": "2024-01-01T00:00:00Z", "public_net": { "ipv4": null, "ipv6": null }, "server_type": { "name": "cx11" }, "datacenter": { "name": "fsn1-dc14", "location": { "name": "fsn1" } }, "labels": {} } }))) .mount(&mock_server) .await; // Very short timeout for testing let timeouts = ProviderTimeouts { poll_interval: Duration::from_millis(10), instance_ready: Duration::from_millis(50), ..Default::default() }; let provider = HetznerProvider::with_config( "test-token", &mock_server.uri(), timeouts, ).unwrap(); let result = provider.wait_ready("123").await; assert!(matches!(result.unwrap_err(), ProviderError::Timeout { .. })); } ``` ### Testing list\_instances ```rust #[tokio::test] async fn test_list_instances() { let mock_server = MockServer::start().await; Mock::given(method("GET")) .and(path("/servers")) .and(wiremock::matchers::query_param("label_selector", "managed-by=spuff")) .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ "servers": [ { "id": 1, "name": "spuff-one", "status": "running", "created": "2024-01-01T00:00:00Z", "public_net": { "ipv4": { "ip": "1.1.1.1" }, "ipv6": null }, "server_type": { "name": "cx11" }, "datacenter": { "name": "fsn1-dc14", "location": { "name": "fsn1" } }, "labels": { "managed-by": "spuff" } }, { "id": 2, "name": "spuff-two", "status": "running", "created": "2024-01-01T00:00:00Z", "public_net": { "ipv4": { "ip": "2.2.2.2" }, "ipv6": null }, "server_type": { "name": "cx21" }, "datacenter": { "name": "fsn1-dc14", "location": { "name": "fsn1" } }, "labels": { "managed-by": "spuff" } } ] }))) .mount(&mock_server) .await; let provider = create_mock_provider(&mock_server).await; let result = provider.list_instances().await; assert!(result.is_ok()); let instances = result.unwrap(); assert_eq!(instances.len(), 2); assert_eq!(instances[0].id, "1"); assert_eq!(instances[1].id, "2"); } #[tokio::test] async fn test_list_instances_empty() { let mock_server = MockServer::start().await; Mock::given(method("GET")) .and(path("/servers")) .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ "servers": [] }))) .mount(&mock_server) .await; let provider = create_mock_provider(&mock_server).await; let result = provider.list_instances().await; assert!(result.is_ok()); assert!(result.unwrap().is_empty()); } ``` ### Testing Authentication Errors ```rust #[tokio::test] async fn test_authentication_error() { let mock_server = MockServer::start().await; Mock::given(method("GET")) .and(path("/servers")) .respond_with(ResponseTemplate::new(401).set_body_json(serde_json::json!({ "error": { "code": "unauthorized", "message": "Invalid token" } }))) .mount(&mock_server) .await; let provider = create_mock_provider(&mock_server).await; let result = provider.list_instances().await; assert!(matches!( result.unwrap_err(), ProviderError::Authentication { .. } )); } #[tokio::test] async fn test_rate_limit_error() { let mock_server = MockServer::start().await; Mock::given(method("GET")) .and(path("/servers")) .respond_with(ResponseTemplate::new(429).set_body_json(serde_json::json!({ "error": { "code": "rate_limit_exceeded", "message": "Too many requests" } }))) .mount(&mock_server) .await; let provider = create_mock_provider(&mock_server).await; let result = provider.list_instances().await; let err = result.unwrap_err(); assert!(matches!(err, ProviderError::RateLimit { .. })); assert!(err.is_retryable()); assert!(err.retry_after().is_some()); } ``` *** ## Integration Tests Integration tests use real API credentials and create real resources. ### Setup ```rust // tests/integration/provider_test.rs use spuff::provider::{Provider, InstanceRequest, ImageSpec, create_provider}; use std::env; fn skip_if_no_credentials() -> bool { if env::var("HETZNER_TOKEN").is_err() { eprintln!("Skipping: HETZNER_TOKEN not set"); return true; } false } fn create_test_provider() -> Box { let token = env::var("HETZNER_TOKEN").expect("HETZNER_TOKEN required"); let registry = ProviderRegistry::with_defaults(); registry.create_by_name("hetzner", &token, ProviderTimeouts::default()) .expect("Failed to create provider") } ``` ### Full Lifecycle Test ```rust #[tokio::test] #[ignore] // Run with: cargo test -- --ignored async fn test_full_instance_lifecycle() { if skip_if_no_credentials() { return; } let provider = create_test_provider(); // Create instance let request = InstanceRequest { name: format!("spuff-test-{}", uuid::Uuid::new_v4().to_string()[..8].to_string()), region: "fsn1".to_string(), size: "cx11".to_string(), image: ImageSpec::Ubuntu("24.04".to_string()), user_data: None, labels: HashMap::from([ ("managed-by".to_string(), "spuff".to_string()), ("test".to_string(), "true".to_string()), ]), }; println!("Creating instance: {}", request.name); let instance = provider.create_instance(&request).await .expect("Failed to create instance"); println!("Instance ID: {}", instance.id); // Wait for ready println!("Waiting for instance to be ready..."); let ready = provider.wait_ready(&instance.id).await .expect("Instance never became ready"); println!("Instance ready at IP: {}", ready.ip); assert!(!ready.ip.is_unspecified()); assert_eq!(ready.status, InstanceStatus::Active); // List instances let instances = provider.list_instances().await .expect("Failed to list instances"); assert!(instances.iter().any(|i| i.id == instance.id)); // Create snapshot (if supported) if provider.supports_snapshots() { println!("Creating snapshot..."); let snapshot = provider.create_snapshot(&instance.id, "test-snapshot").await .expect("Failed to create snapshot"); println!("Snapshot ID: {}", snapshot.id); // List snapshots let snapshots = provider.list_snapshots().await .expect("Failed to list snapshots"); assert!(snapshots.iter().any(|s| s.id == snapshot.id)); // Delete snapshot println!("Deleting snapshot..."); provider.delete_snapshot(&snapshot.id).await .expect("Failed to delete snapshot"); } // Cleanup: Destroy instance println!("Destroying instance..."); provider.destroy_instance(&instance.id).await .expect("Failed to destroy instance"); println!("Test complete!"); } ``` ### Running Integration Tests ```bash # Set credentials export HETZNER_TOKEN="your-api-token" # Run integration tests cargo test -- --ignored # Run specific integration test cargo test test_full_instance_lifecycle -- --ignored # Run with output cargo test test_full_instance_lifecycle -- --ignored --nocapture ``` *** ## Test Utilities ### Fixtures Module Create reusable test fixtures: ```rust // tests/fixtures/mod.rs pub mod responses { use serde_json::json; pub fn server_running(id: u64, ip: &str) -> serde_json::Value { json!({ "server": { "id": id, "name": format!("spuff-test-{}", id), "status": "running", "created": "2024-01-01T00:00:00Z", "public_net": { "ipv4": { "ip": ip }, "ipv6": null }, "server_type": { "name": "cx11" }, "datacenter": { "name": "fsn1-dc14", "location": { "name": "fsn1" } }, "labels": { "managed-by": "spuff" } } }) } pub fn server_starting(id: u64) -> serde_json::Value { json!({ "server": { "id": id, "name": format!("spuff-test-{}", id), "status": "initializing", "created": "2024-01-01T00:00:00Z", "public_net": { "ipv4": null, "ipv6": null }, "server_type": { "name": "cx11" }, "datacenter": { "name": "fsn1-dc14", "location": { "name": "fsn1" } }, "labels": { "managed-by": "spuff" } } }) } pub fn error_not_found() -> serde_json::Value { json!({ "error": { "code": "not_found", "message": "Resource not found" } }) } pub fn error_unauthorized() -> serde_json::Value { json!({ "error": { "code": "unauthorized", "message": "Invalid token" } }) } pub fn error_rate_limit() -> serde_json::Value { json!({ "error": { "code": "rate_limit_exceeded", "message": "Too many requests" } }) } } ``` ### Test Coverage Check test coverage: ```bash # Install cargo-tarpaulin cargo install cargo-tarpaulin # Run with coverage cargo tarpaulin --out Html # Open coverage report open tarpaulin-report.html ``` *** ## CI Integration ### GitHub Actions ```yaml # .github/workflows/test.yml name: Tests on: [push, pull_request] jobs: unit-tests: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: dtolnay/rust-toolchain@stable - run: cargo test --all integration-tests: runs-on: ubuntu-latest if: github.event_name == 'push' && github.ref == 'refs/heads/main' steps: - uses: actions/checkout@v4 - uses: dtolnay/rust-toolchain@stable - run: cargo test -- --ignored env: DIGITALOCEAN_TOKEN: ${{ secrets.DIGITALOCEAN_TOKEN }} HETZNER_TOKEN: ${{ secrets.HETZNER_TOKEN }} ``` *** ## Test Checklist Before submitting provider tests: * [ ] All Provider trait methods have unit tests * [ ] Happy path tested for each method * [ ] Error cases tested (API errors, network errors) * [ ] Edge cases tested (not found, already exists) * [ ] Idempotency tested for destroy/delete operations * [ ] Timeout handling tested * [ ] Rate limit handling tested * [ ] Authentication error handling tested * [ ] Integration test for full lifecycle (optional but recommended) * [ ] Mock server used for unit tests * [ ] Tests are deterministic (no flaky tests) * [ ] Cleanup happens even on test failure