Understanding the Rising Costs of AI Agents Over Time

In the rapidly evolving landscape of AI, the focus has often been on the capabilities of AI systems, such as how long it takes them to complete tasks compared to humans. However, a crucial aspect that is often overlooked is the cost associated with these AI tasks. Over the past seven years, AI systems have seen exponential growth in both model size and the number of tokens generated per task. While efficiencies have been found, the cost of achieving peak performance has likely been increasing exponentially.

If AI agents can complete tasks three times longer each year and costs increase by the same factor, the cost relative to human labor remains stable. However, if costs rise faster than task durations, AI becomes less competitive with human labor. This could mean that while AI performance improves, it might not be economically viable, akin to the high-cost, high-performance nature of Formula 1 racing.

The 'hourly' cost of AI agents is a key metric, calculated by dividing the financial cost of using a large language model (LLM) by the human-equivalent task duration. For instance, Claude 4.1 Opus can complete tasks that take human engineers two hours, and its cost is divided by this duration to find its hourly rate. Opinions on these costs vary, with some assuming stable costs despite longer tasks, implying a declining hourly rate, while others see costs rising exponentially.

The METR benchmark data provides insights into these costs, although it does not directly estimate the cost of achieving performance due to the extensive compute used to ensure models reach their performance plateau. The METR chart for GPT-5 shows performance increases with cost, with human performance as a baseline. AI models show diminishing returns, with costs rising steeply as tasks lengthen.

By adding lines of constant hourly cost to the chart, we can identify each model's 'sweet spot'—the point where diminishing returns begin. For example, human engineers cost about $120 per hour, while AI models range from $0.40 to $40 per hour at their sweet spots. However, costs can be much higher near the plateau, with some models exceeding human costs.

There are discrepancies in cost estimates, particularly for OpenAI models, suggesting potential inaccuracies. The analysis also introduces the concept of a 'saturation point,' where further cost increases yield minimal time horizon gains. This analysis reveals a correlation between task duration and cost, indicating that peak performance may soon be prohibitively expensive.

The findings suggest that while AI capabilities are advancing, the economic feasibility of these advancements lags behind. Real-world applications may not keep pace with the METR time-horizon trend due to rising costs. Further analysis by METR could provide more clarity, especially by addressing model cost discrepancies and plotting hourly costs against release dates.