The Hidden Costs of Orchestrating AI Coding Agents: A Deep Dive into Straight Up AI’s Control Plane Sustainability

Straight Up AI, a consultancy specializing in artificial intelligence solutions, has revealed significant insights into the operational costs and sustainability of its internal control plane designed to orchestrate coding agents. The analysis, spanning seven weeks of development cycles, highlights a critical challenge for organizations leveraging advanced AI tools: ensuring that the efficiency gains do not come at an unsustainable financial or temporal cost. The company’s findings underscore the importance of rigorous cost-benefit analysis, even for promising internal technologies, as they scale.

The Genesis of the Control Plane and the Cost Imperative

At the core of Straight Up AI’s operation is a sophisticated internal control plane. This system governs the autonomy granted to coding agents, a decision influenced by several factors, including the complexity of the task, the agent’s demonstrated reliability, and crucially, the cost associated with development. As a small consultancy, Straight Up AI relies on client allocations for much of its work. However, when operating internally, the control plane is powered through a Claude Max account, which incurs a monthly subscription fee of approximately £200. This fixed cost means that any "bad decision making" by the agents translates directly into wasted time, and by extension, increased operational expenditure.

The realization that the control plane’s long-term viability hinged on understanding its true operational cost prompted a detailed investigation. "Developing it further without reviewing its sustainability is dangerous," stated a spokesperson for Straight Up AI. "Nothing tells us whether we have built something that we could genuinely afford to run." This led to a comprehensive analysis of seven weeks of the control plane’s activity, aiming to quantify its real-world expenses and identify areas for optimization.

Quantifying Utilization: The API Cost Discrepancy

The analysis, conducted between July 15 and September 4, examined 44 distinct development cycles. To translate the control plane’s token usage into a comparable API bill, Straight Up AI employed a specific conversion methodology. Every token was equated to input-token equivalents, with cached reads valued at 0.1x, cached writes at 1.25x, and output tokens at a significant 5x multiplier.

The results were stark: Straight Up AI’s current utilization, if billed at standard API rates, would cost an astounding 22 times more. This significant markup has profound implications for scalability. As the consultancy grows, this escalating cost quickly becomes untenable, approaching the equivalent of a mid-level engineer’s annual salary. This suggests that while the control plane offers significant automation benefits, its current configuration is not economically sustainable for widespread, high-volume application without substantial adjustments.

Adversarial Review: A Double-Edged Sword of Quality Assurance

Beyond the direct financial implications, the analysis also scrutinized the temporal efficiency of the control plane. Several phases of operation were identified as being unnecessarily slow. The prime suspect was the system’s reliance on adversarial review, a process that deploys a secondary coding agent to scrutinize every code commit or increment. While the intuition pointed towards this being a bottleneck, the underlying reasons were more nuanced than initially assumed.

A comparative analysis revealed a 1.19x increase in the number of agents dispatched for adversarial review compared to implementation tasks. More critically, adversarial review accounted for a 69% higher cost in "cost units" and a 71% increase in "agent-hours." Individually, review agents consumed 1.12 million units and operated for 7.8 minutes on average, compared to 825,000 units and 5.6 minutes for implementation agents.

The overhead isn’t attributed to a single, exceptionally expensive review agent. Instead, it stems from the sheer volume of reviewers deployed. With 26% of these reviewers requesting changes, a subsequent remediation agent is dispatched, triggering another adversarial review cycle. This recursive process mirrors the challenges of human code review: if solutions aren’t found within two or three iterations, further code-writing might not be the answer. Adversarial review, in this context, risks leading the agents into a "rabbit hole" of iterative fixes rather than prompting a broader, structural reassessment of the code’s design. This can lead to a situation where agents are focused on micro-optimizations rather than higher-level architectural improvements.

The Control Plane Architecture: A Centralized Arbiter

The control plane itself operates with a central "controller" agent and numerous "worker" agents. The controller is the primary interface for user interaction and is responsible for a suite of critical functions, including planning, orchestrating worker execution, and arbitrating the truthfulness of progress reports. It is the only agent that remains active throughout the entire job execution.

This architectural choice addresses a key limitation in simpler "plan and build" setups where a single agent might both narrate progress and advance the plan. In such scenarios, inaccuracies in progress statements can cascade into flawed subsequent increments. The controller’s role as an independent arbiter prevents this by ensuring that the implementer’s assertions are validated. The "execute-plan" contract establishes a single source of truth, with all subsequent operations inheriting this rule.

The controller’s continuous operation is a significant factor in the cost analysis. Its sustained engagement means it must maintain awareness of the entire job’s state, contributing to its substantial resource consumption. Once an increment is implemented and verified, it proceeds to the adversarial review stage.

The Mechanics of Adversarial Review

The adversarial review process is intentionally designed as a robust gatekeeper. A fresh agent is provided with the repository root and the specific diff scope (comparing the head versus the base commit). Its explicit mandate is to identify failure scenarios. To ensure a rigorous and unbiased evaluation, the reviewer is prohibited from directly editing code. If a logic or type error is detected, the agent must generate a test case to substantiate its findings. The output of this process is an "adversarial report," which is then fed into a remediator agent tasked with implementing the necessary fixes.

The decision to delegate this crucial gatekeeping function to an expensive coding agent (specifically, a Fable 5 high model) is deliberate. The rationale is that a cheaper review with lower recall could lead to widespread failures cascading throughout the system. By scoping the review strictly to the changed code within the diffs, the system avoids exploring transient issues or known project risks, focusing the review on the immediate modifications.

Unpacking the Costs: Where the Money Truly Goes

The investigation into the control plane’s expenses yielded a surprising revelation: the most expensive component of the system is not the agents that write code, but the controller itself. The controller’s cost surpasses the combined expenditure on implementation, planning, and review. This finding challenged initial assumptions that long-session context management might be the primary cost driver.

While the system was designed with per-commit compaction to mitigate context bloat, the analysis revealed that this mechanism was not being implemented reliably. Over the 44 analyzed controllers, only 34 compaction events occurred across 25,878 controller API calls, averaging roughly 760 calls per compaction. Alarmingly, only 12 out of the 44 controllers actually compacted their context at all during the observed period.

The Contextual Conundrum: Bloat and Reliability Concerns

The controllers’ context windows grew almost linearly with session duration, indicating a fundamental issue with context management. The largest observed context window reached an astonishing 996,659 tokens. This linear growth points to a failure in the intended compaction strategy, leading to an ever-expanding memory footprint for the controller.

Further breakdown of API calls revealed that controllers accounted for 30.7% of all calls, consuming a staggering 58.7% of cache-read tokens. Their median context per call stood at a substantial 312,000 tokens, dwarfing the 101,000 tokens per call for worker agents.

The primary cost driver, therefore, was not the act of writing code but the continuous re-reading and maintenance of session state. This state accumulation was exacerbated by a misapplication of the compaction strategy. While individual workers were compacting their context upon opening (averaging 38,000 tokens) and closing (averaging 103,000 tokens), the controller was not performing this crucial function. Consequently, the controller ended up holding the state of every worker within its memory. This not only drove up costs but also posed a significant threat to system reliability, as the degradation of performance with saturated context windows is a well-documented limitation of large language models.

An Honest Look at the Controller’s Internal Operations

To address the escalating costs and reliability concerns, a deeper examination of the controller’s internal workings was initiated, with the planning system initially identified as a likely culprit. The planning process involves a layered approach: an architecture specification, a detailed technical specification, an implementation scope, and finally, a build plan comprising numbered tasks. The controller reads all of these documents. While this granular execution path provides robust oversight, it was initially suspected that maintaining this extensive ledger throughout the run conflicted with effective compaction.

However, measurement revealed that these specification documents, when analyzed against an average controller increment of 312,000 tokens, constituted only about 2% of the overhead. This indicated that the planning chain, while important for maintaining LLM focus and generating solutions, was not the primary source of contextual bloat. In fact, these well-defined scopes are considered essential for clear evaluation criteria and ensuring agents are held accountable to specific objectives.

The True Culprit: The Controller’s Own Output

The analysis pointed to a different, more self-inflicted source of controller bloat: its own generated output. The largest single component contributing to the controller’s context was its "tool arguments." A significant quarter of these arguments consisted of dispatch briefs used for assigning tasks to agents. Across the analyzed corpus, 698 such briefs were generated, with a median length of approximately 1,390 tokens and a longest exceeding 6,000 tokens.

While the initial planning process for these briefs is a one-time cost, the generated briefs themselves are persisted and re-read for every subsequent increment until the run concludes. This persistence is unnecessary, as the briefs are inherently scoped to the specific increments they are designed to address. Consequently, the system was accumulating a growing stack of briefs rather than treating them as ephemeral task descriptors and marking completion against the overarching build plans.

This issue was further compounded by two deliberate design decisions within the internal contract:

1. Controller Validation of Worker Results: The controller is mandated to validate every worker result. This necessitates the controller evaluating all state, requiring the evidence to be persistently stored within its context.
2. Task Tracking and Rendering: To facilitate human review, the controller is prompted to re-render the task list on every state change of an agent. This means the task list, along with its associated metadata, must be maintained and updated throughout longer sessions, naturally leading to an increase in its size.

These combined decisions result in a coding agent that accumulates metadata without shedding it, causing its context to grow with every completed increment by its worker agents.

The Path Forward: Strategies for Sustainability

The critical need for cost and reliability improvements led Straight Up AI to propose three key changes to mitigate the identified issues and ensure the long-term sustainability of their control plane:

1. Pass Briefs by Reference, Not by Value: Drawing an analogy from low-level programming, the concept of passing objects by reference rather than by value can be applied. Instead of creating and passing entire, expensive objects, pointers or references to these objects can be utilized, allowing for downstream reuse. The run record, currently a persistent state incremented on every turn, can be managed such that the controller only needs to hold the most recent state and a pointer to it. This recursive application ensures that all previous states are appropriately handled before a decision is made, drastically reducing the controller’s metadata burden.

2. Compact at Increment Boundaries: Once an increment reaches a terminal state, its implementation transcript no longer influences subsequent decisions. The controller can effectively rehydrate from the build plan and run record, which serve as the definitive sources of truth. This strategy is projected to reduce context overhead from an average of 65,000 to 487,000 tokens per turn to a more manageable sawtooth pattern that resets with each increment. This could potentially bring a typical controller turn closer to 100,000 tokens, a significant reduction from the measured 360,000 tokens.

3. Scope the Reviewer by Attempt, Not by Increment: The initial review of an increment should encompass the entire change. However, subsequent reviews, post-remediation, should focus only on the remediation diff and any outstanding findings. This approach, akin to effective human code review, ensures that an increasingly narrow set of in-scope items are processed through the review funnel. This is expected to reduce the ratio of reviewers to executors within the system.

Conclusion: A Sustainable Future for AI Orchestration

Straight Up AI has developed a system that has become indispensable for both client delivery and internal development. However, the current operational model is not financially sustainable, relying on the current pricing disparity between Claude Max and API access, which could shift unpredictably. The company’s focus has now shifted from merely proving the system’s efficacy to ensuring its long-term affordability.

By addressing the issue of context bloat, Straight Up AI believes it can achieve a more sustainable operational model. The underlying architecture is deemed sound, with the identified problems stemming from the management of increment state within the global run context. Rectifying this will simultaneously reduce costs and minimize the propensity for agents to follow incorrect execution paths.

The investigation’s findings underscore the importance of conducting analysis without preconceived notions and maintaining an honest assessment of operational realities. The initial hypotheses regarding compaction and the planning chain as primary cost drivers proved incorrect. Instead, the controller’s own generated output, particularly the persistent dispatch briefs and task tracking metadata, emerged as the dominant factor in its contextual bloat. This comprehensive self-assessment provides a clear roadmap for optimizing the control plane, ensuring that the power of AI orchestration can be harnessed responsibly and sustainably. The success of these proposed changes will be closely watched by other organizations navigating the complex landscape of AI agent management and cost optimization.