DeskCraft is a new benchmark designed to evaluate desktop GUI agents, such as GPT-5.4, on complex professional workflows within creative software like video and 3D creation tools. It focuses on long-horizon tasks requiring over 50 steps and formalizes human-in-the-loop collaboration with mid-turn and post-turn interaction protocols. Evaluations of 18 agents, including GPT-5.4, showed persistent failures in workflow delivery and proactive clarification.
This benchmark highlights existing limitations of AI agents, even advanced models like GPT-5.4, in handling complex, multi-step professional creative software workflows. For video production and post-production studios exploring AI integration, this indicates that current agents are not yet capable of fully autonomous, end-to-end task execution, especially when human guidance or clarification is needed. The next stage of development will likely focus on improving agents' ability to manage long task sequences and engage proactively with human operators, making human-AI collaboration protocols a critical area to watch for improved efficiency gains.
Add StreamingMeme as a preferred source on Google to see more of our streaming news at the top of your Search results.
Add as preferred source