We Gave AI Real Office Tasks — Only 37% Were Completed Successfully in a 1,150-Task Test

From Email to HR — We Gave AI Real Office Work

EnterpriseOps-Gym is not a simple chatbot test. It's a virtual office that faithfully replicates the tasks that happen every day in a real company. It contains 164 database tables, 512 workflow tools, and 1,150 expert-crafted tasks.

The test covers 8 domains:

Each task requires an average of 9.15 steps to complete. Complex HR tasks need up to 34 steps, and each task must pass an average of 5.3 verification conditions to be counted as a success.

Even the Best AI Fails 6 Out of 10 Times

The research team tested all 16 major AI models currently available. Here are the results:

Even the best-performing Claude Opus 4.5 succeeded only 4 out of 10 times. It achieved around 50% success on relatively simple tasks like email and file management, but dropped significantly on complex work requiring policy compliance, such as IT support (23.8%) and cross-department tasks (30.7%).

AI's Real Weakness: Planning, Not Tools

The most surprising finding was this: when researchers gave AI a human-written task plan in advance, success rates jumped by 14–35 percentage points. Meanwhile, increasing the number of available tools from 512 to add distractors only reduced success by about 1%.

In other words, AI is good at choosing which tools to use, but struggles to strategically plan what to do and in what order. It's like a new employee who's great with tools but doesn't understand the workflow.

Even more striking was the small AI + good plan combination. When the 4B-parameter lightweight model Qwen3-4B was given a human-crafted plan, it matched or outperformed much larger models. Creating a good workflow manual is more effective than using an expensive AI.

When Given Impossible Tasks, AI Just Tries Anyway

The researchers deliberately included 30 impossible tasks — for example, editing records for a nonexistent employee or accessing a system without authorization. A proper AI assistant should refuse, saying "This cannot be done."

The results were alarming. Even the best-performing model at refusal (GPT-5.2 Low) only rejected 53.9% — barely better than a coin flip. Claude Opus 4.5 scored exactly 50%. The other half of the time, the AI forced its way through impossible tasks, corrupting data in the process. In a real work environment, this could lead to incorrect customer records or unauthorized system changes — serious problems.

AI Success Rates Plummet as Tasks Get Longer

On simple 4-step tasks, AI succeeded about 35% of the time, but on complex tasks with 16+ steps, the rate dropped below 20%. As steps increase, small early mistakes snowball. The decline was even steeper for open-source models.

Teaming Up Multiple AIs Doesn't Solve the Problem

"If one AI can't do it, why not use several?" The researchers tested this too. They tried splitting roles between a planning AI and an execution AI, or breaking tasks into subtasks distributed across multiple AIs. While slightly better than the baseline, these approaches fell far short of human-crafted plans. In some cases, splitting tasks actually hurt performance because it broke the contextual thread.

Best Value for Money: Google's Gemini 3 Flash

In terms of cost-effectiveness, Gemini 3 Flash was the most efficient at ~$0.03 per task with a 31.9% success rate. Claude Opus 4.5 had a higher success rate but cost ~$0.34 per task — 12 times more expensive. Among open-source models, DeepSeek V3.2 was the most economical at ~$0.01 per task with 24.5% success.

For Those Looking to Deploy AI Assistants at Work

The core message of this research is clear:

The research team has open-sourced the paper and the full benchmark. If you'd like to explore it yourself, check out the code and data on the GitHub repository.

To learn more about AI and vibe coding, check out our Free Learning Guide.

Related Content — Get Started with AI | Free Learning Guide