FRIDAY Phase 4: An Arena for Comparing Model Choices

This note describes the Phase 4 direction of FRIDAY, my independent personal-agent work: choosing a model for the task, making the budget visible and keeping a record of what happened. The aim was task-fit per dollar, not a claim that a local model could match every hosted model. The name FRIDAY is retained here as the name used in this build note; the project overview is FRIDAY.

The problem: a default model is a tax

The default model in an agent workflow can become a habit. But a short reply and a codebase audit need different amounts of context and different kinds of judgment. Sending both to the same model can obscure that difference.

Some requests are much simpler than others: reading a setting, formatting an answer or making a small edit. Those are candidates for a smaller or local model, provided the model can actually complete the task and use the required tools.

What "Arena" actually does

The Arena module sits in front of the orchestrator. The design treats routing as an auction among candidates, using estimates that can be inspected before selecting one.

Candidates can include local Ollama checkpoints and hosted providers. The bid schema represents the estimates used to compare them; these estimates are not measurements of a task that has not run yet:

Bid(
    model_id: str,
    estimated_tokens_in: int,
    estimated_tokens_out: int,
    estimated_cost_usd: float,
    self_reported_confidence: float,  # 0.0 – 1.0
    latency_p50_ms: int,
)

The scoring idea combines confidence, historical fit for a task class and estimated cost. Its quality depends on the evidence behind those inputs. A score needs explicit treatment of missing history and zero-cost candidates; self-reported confidence alone is not a reliable basis for choosing.

The cost-aware brain

The bidding alone was not enough. A confidence estimate can be wrong, and the cheapest attempted call may not be the cheapest completed task. The budget guard sits above the routing decision.

The budget guard does three things:

1. Gives each task a configurable spending ceiling.
2. Reconsiders candidates when a bid exceeds that ceiling, making the budget-versus-capability trade-off visible.
3. Accounts for retry costs against a daily budget, with a local-only mode as a possible boundary when that budget is exhausted.

A routing budget is a useful control, but it is not a guarantee against all spending. Provider limits, concurrent calls and failed attempts still need to be accounted for. A local-only route is useful only if the local model can complete the task.

Scheduler + dashboard

Two less glamorous parts of the Phase 4 design make the workflow inspectable:

• Scheduler — cron-style, but tasks are stored in SQLite with idempotency keys. The intent is to prevent recurring work from duplicating an unfinished run.
• Dashboard — a tiny localhost web view. Every Arena auction, every budget hit, every task outcome. Searchable. The purpose is to make routing and prompt problems visible enough to investigate.

Until you can see the routing decisions, it is difficult to know what the system is doing. An inspectable record is more useful than a reassuring label.

What's next: Phase 5

Voice interaction and a browser worker were directions considered for a later phase. Both introduce new action boundaries: a spoken instruction and a form submission have consequences beyond generating text. Those capabilities need their own evaluation and controls; they are not presented here as released features.

Memory, routing, budgets, scheduling and visibility are the foundations. The useful next step is to establish what they can reliably do before expanding the interfaces.