Observability

Dark Factory captures structured logs, trace IDs, per-step artifacts, and historical analytics for every run. This page covers the full telemetry surface — from raw log files to the web dashboard — so you can debug failed issues, audit agent decisions, and track cost trends over time.

Run Directories

Every run writes its output to a timestamped directory under ~/.godark/runs/. This is the canonical source for all per-run data.

~/.godark/runs/
  <owner>/
    <repo>/
      <YYYYMMDD-HHMMSS>/
        run.json                  # Run metadata, dependencies, rate-limit state
        debug.log                 # Run-level structured log
        waves/
          1.json, 2.json, ...     # Per-wave execution records
        issues/
          <N>/
            debug.log             # Issue-level log
            recon.json            # Step artifacts (see Artifacts Reference)
            planner.json
            implement.json
            dialogue.json
            failure-analysis.json
            ...

Note: there is currently no log rotation or retention policy. Runs accumulate without bound — see Known Gaps.

Logs

Logs are written at two levels — one debug.log for the entire run, and one per issue. Both use structured JSON via Go's slog.JSONHandler, with colored text also streamed to stdout.

Log Locations

Logger Variants

• Full — writes to both file and stdout with ANSI colors
• File-only — structured JSON to disk, no terminal output
• Append — appends to an existing log (used for retries)

Format

Each log line is a JSON object with timestamp, level, message, and structured fields. Terminal output adds ANSI highlighting for verdicts and timeouts.

{"time":"2026-04-15T14:32:01Z","level":"INFO","msg":"step completed","step":"implement","issue":42,"cost_usd":0.18,"duration_s":34.2}

Tracing

Every issue gets a UUID trace ID generated at the start of the agent loop. That ID propagates through every step result, verification, and outcome record, making it possible to follow a single issue's entire decision path.

Propagation Path

Querying Traces

The godark trace CLI supports three output modes:

# Text summary (default)
godark trace 42

# Full JSON output
godark trace 42 --json

# Detailed step-by-step view
godark trace 42 --detail

# Query by trace ID directly
godark trace abc12def-...

Surface Points

• TUI — shows the first 8 characters of the trace ID on issue completion
• Dashboard — renders the full trace ID in the issue detail header with a copy button
• SQLite — trace_id columns on both step_results and issue_outcomes

Analytics Database

The analytics database at ~/.godark/stats.db is a SQLite file created at orchestrator startup. It stores run metadata, per-issue outcomes, and per-step cost/duration records. Stats are written post-run and failures are non-fatal — a broken analytics write never corrupts a run.

Cost & Duration Tracking

• Per-step: cost_usd, duration_seconds, started_at, finished_at
• Resources: peak_memory_bytes, cpu_nanoseconds (Phase 30)
• Aggregation: Aggregate() computes cost totals, duration stats, flag frequencies, retry stats
• Trends: ComputeTrends() produces historical line-chart data
• Per-issue cost: IssueCostUSD() sums all step cost fields

Outcome Values

CLI Commands

Four commands cover the main observability use cases. See CLI Reference for full flag documentation.

godark trace 42 --detail

godark analyze --repo owner/repo --since 2026-01-01

godark report --since 2w --format markdown

godark status --port 9090

TUI & Dashboard

Terminal UI (Bubble Tea)

The TUI renders a live issue table during godark run with spinner-based progress, stage transitions, trace IDs, and per-issue cost/retry counters. Disable with --no-tui for plain-text output.

Web Dashboard

godark status serves an HTML dashboard powered by HTMX for partial page updates. Features include:

• Paginated log viewer
• Run list with filterable outcomes
• Issue detail view with trace ID, dialogue timeline, and step costs
• Copy-to-clipboard trace IDs

Progress Events

Both surfaces consume a shared event stream. Current event types:

Artifacts Reference

Each issue directory contains step-level JSON artifacts capturing the full agent output at every stage. Prompts are also stored in step_results.prompt in SQLite (capped at 32 KB).

Step Outputs

Analysis Artifacts

Wave Tracking

Wave execution records are stored in waves/<N>.json — one file per dependency-resolution wave, capturing which issues ran in parallel and their outcomes.

Failure Analysis

When issues fail, Dark Factory captures detailed failure data to help you understand what went wrong and where.

Failure Artifacts

Retries

Each retry cycle produces separate step records named retry-N and retry-N-quality-review. These are stored both as JSON files in the issue directory and as rows in step_results.

Resilience

Analytics writes are non-fatal by design. If the stats database is unavailable or a write fails, the run continues normally — run data is never corrupted by an analytics failure.

Known Gaps

The observability surface is mature but has known limitations. These are tracked for future development.