AM-Bench: how AI agents behave when faced with incomplete or ambiguous tasks, by model and harness

Disclaimer: the material received is only the arXiv abstract page for the paper, not the full text. This analysis is therefore limited to what the abstract says and does not go into methodology, results tables or specific figures by model or harness.

The paper, titled “AM-Bench: Understanding Agents under Task Underspecification across Models and Harnesses,” was submitted to arXiv on October 13, 2026 (category cs.AI). Its authors are Kaiyu Huang, Yiting Li, Angyang Gu, Pinjia He, Minchen Yu and Qingjiang Shi.

The starting point is that ambiguous or incomplete specifications make it harder for language-model-based agents both to carry out a task and to clarify it. According to the authors, how agents respond to these information gaps is poorly characterized, both across different models and across different “harnesses,” that is, the execution environments that wrap the model.

AM-Bench is built from 48 tasks adapted from Terminal-Bench 2.0 and Harness-Bench. Each task includes a fully specified reference and six variants covering ambiguous or missing goals, context and constraints. The authors evaluate every combination of five models and four harnesses, for a total of 20,160 runs.

The results described in the abstract point to different execution deficits depending on the type of gap:

• Gaps in the goal make it harder for the task to be completed. • Gaps in the context increase token consumption. • Gaps in the constraints weaken compliance, though they do not necessarily prevent the task from being completed.

In addition, across all three domains, missing information reduces success more than ambiguous information does.

Another notable finding is that these effects change how systems compare. Advantages observed with complete information do not necessarily hold when data is missing, and rankings depend on the model × harness combination and on the evaluation criterion chosen: completion, success or consistency across repeated runs.

Finally, based on the clarification requests observed and on equivalent fully specified runs, the authors estimate that asking for clarification could recover unfinished work but would leave most constraint violations unresolved.

From this they draw two recommendations: evaluate model and harness jointly under different information conditions, and clarify outstanding requirements in a targeted way before executing actions that depend on them. To support future research, they release an AM-Bench project repository, whose address does not appear in the text received.

Overall, the paper suggests that measuring agents only on fully specified tasks may give a misleading picture of their real-world performance. Assessing its conclusions would require reviewing the full text, which could not be analyzed here.