The problem is completing the assignment
Consider a research assignment with a reasonably demanding brief. Read a collection of papers, explain an engineering approach, distinguish demonstrated results from speculation, and produce an article that preserves the author’s analytical position. The references must support the claims. The technical diagrams must agree with the prose. When a section changes, conclusions elsewhere may need to change with it.
An AI system could divide this assignment among several agents. One plans the investigation, others read the papers, another writes, and a reviewer checks the result. If something fails, the work returns for revision. A final agent integrates the pieces and decides whether the assignment is complete.
There is a substantial engineering idea here. Complex work contains dependencies, opportunities for parallel execution, and errors that can be detected before they reach the final output. A system that can obtain evidence, inspect intermediate results and recover from mistakes has capabilities that a single response cannot provide. The question is how far those capabilities extend, and under which conditions they become dependable.
The difficult part of our example is that all the pieces can appear acceptable while the assignment remains unsatisfied. Every section may contain accurate statements, yet the article may fail to answer its central question. A revision may improve readability while weakening an important qualification. A reviewer may approve the citations without noticing that the argument has changed. Completion therefore requires coordination between execution, evidence and the user’s criteria for success.
This review examines that coordination. Its unit of analysis is the engineering mechanism, rather than the name or ranking of an underlying model. It asks what loops, graphs and harnesses contribute; how information passes between agents; what checking can establish; and when additional iteration needs something beyond another model response.
The evidence is a targeted critical reading of primary papers and implementation specifications, including foundational work and selected research available by 11 September 2026. Sources were selected for their relevance to architecture, feedback, context, evaluation, coordination and openness. This is not an exhaustive systematic review or a new benchmark study. Historical results describe the tested configurations, not the performance of every system available today. Recent preprints are identified as such.
The running report example is an illustrative design, not an experiment I have conducted. Figures distinguish explanatory architectures, proposed interventions and reconstructions of published observations. That distinction matters: a plausible diagram can explain a hypothesis without proving it.
Models, loops, graphs and harnesses
What a model call actually contributes
A language model receives an input context and generates an output. Training has shaped its parameters, which encode learned patterns and capabilities. The current context supplies the particular instructions, examples and information available for this invocation. Giving the model an example at execution time is different from updating its weights through training, a distinction made explicit in the early literature on in-context learning, especially Brown and colleagues’ Section 2.
For our purposes, the output might be prose, a proposed tool call, or a structured decision. A model can propose a search query, but surrounding software must execute the search. It can request a file modification, but a tool must perform the write. A statement that the file was changed is not itself a file change.
This is why describing an agent as simply retrieving reasoning from training data is inadequate. Its behaviour depends on learned capability, current evidence, instructions and the actions its environment permits. The important engineering question is what information and feedback reach each decision, and what converts that decision into an observable change.
We can describe a typical execution step using four quantities. The system has stored state S, constructs a context C from some of that state, asks a model to propose an action A, and obtains an observation O after execution. It then updates S. These are different objects. Confusing them makes failures difficult to diagnose.
The broader architectural lineage matters. Sumers and colleagues’ Cognitive Architectures for Language Agents, published in TMLR, organises agents around memory, internal and external actions, and a decision cycle. That framework helps locate loops and state management within an established research vocabulary. The definitions here are practical working definitions, not a claim that loop, graph and harness engineering constitute three formally standardised disciplines.
Loop engineering
An agent loop repeatedly assembles context, requests an action, executes permitted actions, records their results and decides whether to continue. The model may select the next action, while software enforces permissions, validates arguments and limits expenditure. Some loops alternate generation and critique without using external tools; others interact extensively with an environment.
ReAct, introduced by Yao and colleagues, provides a foundational example of interleaving reasoning with actions and observations. Its experiments include question answering, fact verification and interactive environments. The architectural contribution is to let observations inform subsequent decisions, rather than require a complete answer before interacting with evidence. Its results support that approach in the evaluated settings, not a guarantee that every action sequence will converge on a correct outcome.
Loop engineering concerns the conditions around repetition. What should trigger another attempt? What feedback should it receive? Which errors are recoverable? When should the system stop? A loop with no meaningful change in evidence, strategy or checking may spend more resources without resolving the underlying uncertainty.
Figure 1. Explanatory architecture of an agent loop. The separation of action and observation is informed by ReAct, Figure 1 and Section 2. State selection, permission checks and budget control are explicit additions in this synthesis. Arrows carry the labelled objects, not an assumed shared understanding.
Graph engineering
A workflow graph represents operations as nodes and possible transitions as edges. A node might retrieve a paper, execute a calculation or invoke an agent that has its own loop. A conditional edge chooses between routes. A graph can therefore contain loops; the two concepts describe different aspects of execution.
The graph determines which work is eligible to happen next and which dependencies must be satisfied. For our report, evidence collection may branch across topics, then join before synthesis. A failed source check might return only the affected claim for investigation. An unresolved conflict in the brief might route to a human rather than back to drafting.
The graph API specification examined here makes state, nodes, edges and reducers explicit. A reducer is a rule for combining updates with existing state: for example, appending evidence records instead of replacing the entire list. This is implementation documentation, not experimental proof of improved task performance.
Our proposed report graph would use separate fields for accepted requirements, candidate findings, unresolved conflicts and draft versions. A node that adds a new finding should not overwrite the requirements. Two researchers discovering incompatible results should create a conflict to examine, rather than let whichever update arrives last silently determine the conclusion.
Harness engineering
Here, the harness means the surrounding execution system: the software that assembles model inputs, exposes tools, routes work, manages state, records events and controls continuation. The term is used differently across projects, so it should always be defined when comparing implementations. An evaluation harness, for instance, can mean the software administering a benchmark rather than the agent’s runtime.
In our architecture, the harness implements the loop and graph. It also handles ordinary but consequential execution problems: a tool timeout, an invalid response, a process restart, or an attempt to modify an obsolete document version. Those are part of making work dependable even when the model’s proposed next step is reasonable.
Figure 2. Proposed graph for the running report example. Research branches join at evidence integration; missing evidence, conflicting requirements and failed checks take different routes. This is an author-designed workflow, not a reproduced benchmark result. Node and state semantics follow the graph concepts documented above.
Follow the information through the work
Suppose the user explicitly requires the report to distinguish successful demonstrations from dependable repeated execution. We assign that requirement an identifier, R1. Its purpose is not to make the instruction look formal. It is to let later stages refer to the same obligation without repeatedly paraphrasing it.
The planner produces research questions and a proposed section order. It also marks which questions bear on R1. A researcher returns a paper record containing the evaluated task, relevant result, source location and limitations. The synthesiser groups compatible findings and records disagreements. The writer receives the approved question, relevant evidence and the current requirements. The reviewer checks the exact version the writer produced.
Each of these steps creates a distinct opportunity for error. The planner might narrow R1 into an easier question. A researcher might omit the study’s repeated-trial method. The writer might preserve the result but lose its qualification. The reviewer might receive only the draft and never see R1. These possibilities require different diagnoses, even if the final symptom is the same: the article fails to distinguish demonstration from reliability.
The trace needed for diagnosis is consequently more detailed than a list of agent messages. We need to know which version of the requirement was supplied, what source material was available, what the node produced and which state update was accepted. A message saying that R1 was checked should point to the relevant passage and the evidence used to check it.
Our proposed state record would distinguish intended actions from confirmed effects. A request to inspect an appendix would remain pending until a retrieval result was recorded. A claimed correction would remain unverified until the changed passage was compared with the source. This design does not require the system to expose every internal computation. It requires an accurate record of externally meaningful actions and artifacts.
Persistence helps support that record. The examined persistence documentation distinguishes checkpoints used to preserve execution state from stores used across interactions. These mechanisms support resumption and memory, but saving a state does not validate its contents. An incorrect assumption can be preserved just as reliably as a correct one.
For the report, the practical response is to store status and provenance alongside content. A finding might be a candidate, a source-checked observation or an unresolved interpretation. A later agent should not infer that a statement is established merely because it appears in persistent memory. We should preserve the reason for trusting it, and the conditions under which that trust should be reconsidered.
There is another boundary here: a successful software operation and successful intellectual work are different outcomes. A retrieval function can return without error while fetching the wrong version of a paper. A merge operation can complete while combining incompatible definitions. The execution system needs technical checks, but the assignment also needs checks that refer to meaning.
That is the first useful limit on the promise of orchestration. A graph can require a review node to run. It cannot, simply by containing that node, establish that the review examined the right question.
What this engineering genuinely improves
Better interfaces can make existing capability more usable
One of the clearest demonstrations comes from Yang and colleagues’ agent-computer interface study. The researchers changed how an agent searched, viewed and edited code, and how execution feedback was presented. In Table 1, a tailored interface resolved 18% of 300 software issues, compared with 11% for a separate shell-only baseline using the same underlying model. These are SWE-bench Lite results, not results on the full benchmark. The comparison changes the surrounding interface and agent setup; it is not a one-component ablation. This is an architecture comparison, not a model ranking.
Table 3 also reports a decline from 18% to 15% when editing lacked the tested linting intervention. A linter checks certain code errors mechanically. The linting ablation and broader interface comparison show that interface and feedback design can affect outcomes without changing model weights. They do not establish equivalent gains for other repositories, tasks or present-day systems.
Figure 3. Simplified intervention mechanism from Yang et al., Section 3 and Table 3. The tailored edit interface rejects selected invalid edits and returns diagnostics. The diagram represents the checked interface behaviour; it does not imply that passing syntax checks establishes functional correctness.
For our report, the analogous design question is whether the tool exposes what the agent needs to make a good decision. A bare search result may be less useful than a source passage accompanied by its section and publication version. A citation checker that returns only a red indicator may be less useful than one that identifies the unsupported sentence. These are proposed design applications, not measured transfers of the coding result.
Structure can reduce unnecessary decisions
Xia and colleagues’ Agentless study examines a constrained software-repair process based on localisation, repair and validation. It provides evidence that useful automation need not delegate every sequencing decision to a conversational agent. Its reported results are tied to software-issue benchmarks and the configurations tested.
The broader engineering question is where adaptive choice is valuable. In our report example, checking that every reference identifier resolves is a predictable operation. Asking a model to decide whether to perform that check on every run creates a decision without an obvious benefit. Choosing which conflicting paper deserves closer examination is a different matter. That decision depends on the evidence encountered.
The distinction suggests a design principle to test: use explicit workflow rules for stable dependencies and model judgment where the task genuinely requires interpretation. This does not imply that fixed workflows always outperform flexible agents. It identifies a useful baseline against which additional autonomy can be evaluated.
There is a simple way to see why the number of required decisions matters without making claims about any particular model. Imagine a workflow with ten indispensable stages. If each stage succeeds with probability 0.95, independently of the others, the probability that all ten succeed is approximately 0.60: the product of ten factors of 0.95. This is an illustrative calculation, not an empirical estimate. Real stages are not necessarily independent, some errors are recoverable, and different stages have different difficulty.
The calculation nevertheless exposes a design question. Adding a stage can improve the workflow if it prevents enough downstream errors, but it also creates another operation that can fail or consume resources. A verification stage earns its place through its net contribution, including false rejection of correct work. The number of agents or review rounds therefore cannot serve as a substitute for measuring the completed assignment.
Feedback can carry experience forward
Reflexion, by Shinn and colleagues, converts feedback into textual reflections retained for subsequent attempts. The framework can use environmental signals or internally generated evaluations. It does not update model weights during those attempts. The paper reports improvements in its evaluated decision-making, reasoning and programming settings, while acknowledging dependence on the quality of evaluation and reflection.
This is an important distinction for claims about learning. Retaining a useful note can change a later attempt without changing the underlying model. For our report, such a note might identify that a previous search repeatedly returned summaries instead of original papers. The useful content is the actionable correction and its evidence, rather than the mere presence of a reflection stage.
These examples establish real engineering value. Interfaces can make actions easier to execute correctly. Structure can remove avoidable choices. Feedback can expose mistakes and inform later attempts. The next question is whether the information needed for those benefits survives as the work becomes longer and more distributed.
Context does not move automatically
Storage, availability and use are different
A system can store an entire conversation while supplying only a fraction of it to a particular invocation. It can supply the right passage while the model fails to use it correctly. It can use the passage locally while a later summary drops its importance. Calling all three failures “forgetting” hides the engineering decisions involved.
Liu and colleagues’ Lost in the Middle tested multi-document question answering and key-value retrieval. Performance depended on where relevant information appeared in the context, with weaknesses in middle positions for the evaluated configurations. This is evidence that inclusion and effective use are distinct. It is not a measurement of every modern context window or a direct experiment on preserving an author’s argument.
More recent evaluation makes the distinction sharper. NoLiMa, by Modarressi and colleagues, reduces literal overlap between questions and the information needed to answer them. Its results show that finding information through an association can be harder than finding a matching phrase, especially as context grows. A context-window specification therefore does not establish effective reasoning over everything it can hold. This is a controlled retrieval benchmark, not a direct test of multi-agent handoffs; it supports caution about that inference rather than measuring the handoff itself.
For diagnosis, we should therefore ask three successive questions. Was the required information stored? Was it supplied to the decision that needed it? Did the output correctly reflect it? The first two can often be inspected directly in a trace. The third requires evaluation against the task and evidence.
Figure 4. Proposed diagnostic for requirement R1. The three failure locations are deliberately separate: omission during context assembly, incorrect use after delivery, and loss during handoff. The distinction between presence and use is motivated by Liu et al.; the report trace is illustrative, not an observation from that study.
Summaries are decisions about future relevance
Imagine that a researcher reads a study finding improvement on a bounded task under a particular budget. The handoff says only that the intervention improves performance. The sentence is shorter, but the budget and scope may be exactly what the writer needs to compare this study with another.
The original source has not necessarily disappeared. It may remain in a file. What has disappeared is the signal that a particular qualification matters. A downstream agent cannot be assumed to revisit every source for every sentence; doing so would undermine the purpose of delegation. The handoff must preserve enough information to guide selective reinspection.
For our report, I would use a compact evidence record with separate fields for finding, task, comparison, limitation and source location. A summary would remain useful, but it would not be the only representation of the evidence. This is a proposed intervention whose value should be tested against shorter unstructured handoffs under comparable budgets.
There is a trade-off. More fields consume tokens and can be filled incorrectly. A comprehensive schema can become administrative work that obscures the important result. The goal is to preserve decision-relevant distinctions, not to accumulate the largest possible record. An evaluation should include cases where a small omitted qualification changes the appropriate conclusion.
Persistent memory changes retrieval, not the need for it
Packer and colleagues’ MemGPT explicitly separates the model’s current context from external storage and provides functions to move information between them. Its evaluations cover document analysis and conversation. The mechanism addresses finite input capacity by making stored information retrievable; it does not make all stored information simultaneously available or automatically authoritative.
In our proposed workflow, a memory entry should therefore carry its source and revision status. If the user changes the article’s central question, earlier planning notes may become stale. Retrieving them more successfully could make the report less aligned with the new brief. Memory management needs a way to supersede decisions as well as retain them.
Selective retention can improve the result
The remedy is not necessarily to retain more. Lodha and colleagues’ June 2026 preprint, Less Context, Better Agents, evaluates 50 enterprise expense-processing tasks with five independent runs per configuration. With the executing model and simulated-user model held constant in the relevant comparison, full history achieves 71.0% completion, retaining five recent tool-call/response pairs achieves 79.0%, and combining that window with summaries achieves 91.6%. The latter configuration also reduces token consumption. The evaluation is narrow, uses simulated users and does not establish a universal retention policy.
The finding changes the design question. Compression can remove distracting material and preserve useful working state; it can also discard a decisive qualification. Its value depends on what survives and how subsequent decisions use it. The right comparison measures completed work, retention of critical facts and resource use together. “More context” and “less context” are incomplete specifications.
Figure 5. Proposed selective-context design, informed by Lodha et al.’s recent-window-plus-summary intervention. A protected requirements record and source reinspection are additions proposed here. The return path addresses a missing qualification; it does not claim that summarisation is lossless.
For our report, I would retain the approved brief and unresolved disagreements independently of the rolling working summary. Recent tool results would remain available in detail. Older evidence would be represented by source-linked records with scope and uncertainty intact. When a claim depends on an omitted condition, the worker would reopen that source instead of asking the summary to reconstruct it. This is a proposed combination to evaluate, not the configuration tested in the expense study.
Different models do not share an internal workspace
In a conventional text-mediated harness, agents exchange messages, artifacts or structured data. They do not merge their training histories or automatically transfer an internal representation of the task. Each receiving model processes what it is given through its own learned parameters and instructions.
The A2A specification addresses discovery, messages, tasks, artifacts and task states across agent boundaries. Such a protocol can standardise the exchange without requiring identical internal implementations. It does not certify that two agents interpret an analytical qualification identically.
The Model Context Protocol architecture, in its 18 June 2025 specification, defines host, client and server roles for connecting applications to capabilities such as tools and resources. A2A addresses agent collaboration; MCP addresses a different integration boundary. Neither protocol by itself supplies a shared interpretation, validates retrieved evidence or decides whether an action is authorised for this particular user. Interoperability makes exchange implementable. The harness still needs policies for selecting, interpreting and acting on what is exchanged.
For our example, successful communication means more than delivering a valid message containing R1. The recipient must understand how R1 constrains its particular contribution. A useful handoff check would ask the next stage to identify which planned claims depend on R1, then compare that interpretation with the brief. This is a testable design proposal, not evidence that acknowledgement alone ensures understanding.
Why checking and revision can fail
The checker needs a basis for judgment
A review instruction can sound demanding while leaving the reviewer with little more than the writer’s output. If the task is to verify citations, the reviewer needs the cited material. If it must check a calculation, an independent computation can help. If it must preserve the author’s question, it needs the question and the relevant constraints.
Madaan and colleagues’ Self-Refine evaluates seven tasks using a cycle in which a model generates feedback and revises its output. Gains vary substantially: the mathematical-reasoning results are essentially unchanged. The result should not be described as uniform improvement across all seven tasks. The study establishes that self-feedback can be useful under particular task and evaluation conditions. It does not establish that every correct answer will survive revision or that every factual error is detectable from the draft alone.
Huang and colleagues’ self-correction study examines intrinsic correction of reasoning without external feedback. In the tested settings, revision could damage correct answers as well as repair incorrect ones. The relevant lesson is to measure both transitions. A revision process that makes many changes is not necessarily improving the result.
These findings should be read together. Different tasks, feedback prompts and success criteria can produce different outcomes. Improving a response on an expressed stylistic dimension is not identical to determining whether an uncertain reasoning answer is correct. Neither paper licenses a universal verdict that self-correction always works or cannot work.
Kamoi and colleagues’ critical survey of self-correction situates this disagreement across the literature. It distinguishes prompted self-feedback from correction supported by reliable external signals or training. Its analysis cautions against treating successful results on feedback-friendly tasks as evidence of general intrinsic correction. This supports a conditional account: specify the feedback source, the task and the training regime before describing a system as self-correcting. The survey is a synthesis of the evidence available to it, not a permanent impossibility result.
Figure 6. Explanatory comparison of two feedback arrangements. One reviewer sees the draft and brief; the other also receives source passages or executable checks. Inspired by the feedback distinctions in Self-Refine, Reflexion and Huang et al. This is an architectural synthesis, not a claim that external evidence eliminates error.
Agreement can coexist with a wrong conclusion
Cemri and colleagues’ failure taxonomy identifies 14 failure modes grouped into system design issues, inter-agent misalignment and task verification. Its October 2025 version reports more than 1,600 annotated traces across seven frameworks; the initial taxonomy development used 150 traces. Traces and distinct tasks are different units and should not be interchanged. It is a diagnostic study, not a representative estimate of all deployed systems.
Appendix N.11 contains a particularly clear trace. An agent incorrectly answers 14 hours to a painting problem. It receives another agent’s correct 49-hour solution, acknowledges the correct conclusion during review, yet ultimately returns 14. The needed information arrived; the final answer failed to reconcile it. This case demonstrates a possible failure, not its population frequency.
Figure 7. Reconstruction of the published trace in Cemri et al., Appendix N.11. There are 12 rooms, five completed, and seven hours per room. The correct remaining time is 49 hours. The agent receives the correction but retains 14. Describing its arithmetic as subtracting completed work twice is this review’s interpretation of the displayed calculation, not the paper’s named failure diagnosis. Labels paraphrase the trace.
In the omission, interpretation and compression diagnostic, this trace is especially useful because it rules out a simple delivery explanation for the displayed correction. That narrows the engineering investigation. We should inspect how the final decision was assembled and checked, rather than assume a larger memory would fix the problem.
This example gives our report workflow a sharper evaluation question. Does the final synthesis resolve explicit contradictions between intermediate outputs, or merely mention that review occurred? A checker that approves one claim while leaving an incompatible claim untouched has not completed that responsibility.
I would record contradictions as objects requiring a disposition: resolve through evidence, preserve as a genuine disagreement, or escalate because the available material is insufficient. The final document should not be approved while the same proposition is treated as both established and rejected in different sections. This proposed check would need semantic evaluation; a text search alone cannot capture every contradiction.
A score is also a measurement instrument
Zheng and colleagues’ LLM-as-a-judge study examines the usefulness of model judgments alongside position, verbosity and other biases. Its relevance here is methodological: a reviewer is an instrument whose behaviour needs validation. Agreement with human preferences in one evaluation does not establish competence for every technical verification task.
For the report, we should separate source support, logical consistency, requirement compliance and editorial quality. A single overall score can conceal a severe failure in one dimension. A beautifully written but unsupported conclusion should not be rescued by high scores for readability and organisation.
The author’s point of view creates a further distinction. An unexpressed preference is missing information. An explicit position that disappears is a fidelity problem. A position contradicted by evidence needs discussion rather than automatic preservation. A dependable assistant should distinguish these situations instead of treating alignment as agreement at any cost.
An appropriate review record would explain which situation applies. If the evidence challenges the planned argument, the system should surface that conflict. It should not quietly rewrite the author’s position, but it should not manufacture support for it either. That is a more demanding standard than producing a smooth final draft.
What changes when several agents work together
Parallel research can be useful when the work divides cleanly. One agent can examine memory methods while another examines verification. Separate contexts can limit irrelevant material within each subtask. Different tools or instructions can make specialisation meaningful. These are architectural possibilities whose benefits must be measured against the cost of integration.
The difficult case is tightly coupled work. If the definition of reliability changes, both research branches may need to revisit their selection of evidence. A local decision can alter what counts as a useful result elsewhere. More agents then create more opportunities for inconsistent assumptions unless the shared decision is propagated explicitly.
The communication burden depends on topology. In a fully connected directed exchange among n agents, there are n(n − 1) possible sender-recipient links. A central coordinator can reduce the number of direct links, but it must integrate what others send. This is a count of possible connections, not a prediction of token use or performance. Actual communication depends on the protocol and task.
For the report, a coordinator is useful only if it has a defined integration responsibility. Combining paragraphs is insufficient. It must compare definitions, identify duplicated evidence, preserve disagreements and check that the whole article answers its question. The system should allocate time and context to that work instead of treating synthesis as a brief final formatting step.
Sander and colleagues’ July 2026 scalability preprint examines progressively more elaborate agent configurations on Terminal-Bench tasks. Its reported outcomes are conditional: increasing architectural complexity does not yield uniform improvement, and timeouts and consistency problems matter. The paper has internal reporting inconsistencies, including different model counts in the abstract and Section V-C. I therefore use it as exploratory evidence, without reproducing its numerical performance claims.
Controlled comparisons are already emerging. Tran and Kiela’s April 2026 preprint, Single-Agent LLMs Outperform Multi-Agent Systems on Multi-Hop Reasoning Under Equal Thinking Token Budgets, compares architectures while matching thinking-token budgets. In the evaluated multi-hop reasoning settings, single-agent execution matches or exceeds the multi-agent alternatives. Their information-theoretic argument depends on assumptions including effective use of the available context; it should not be read as a theorem that every practical single agent dominates every team. The study makes compute allocation a necessary part of the comparison, while leaving transfer to distributed research, tool use and other tasks to be tested.
The unresolved question is not whether a swarm is inherently useful. It is which tasks benefit from which communication structure under a defined resource budget. An architecture that wins by consuming substantially more computation may still be worthwhile, but its advantage should be described accurately.
Figure 8. Proposed integration control for the report. Two branches make incompatible claims about the same evidence. A join that only concatenates text allows both through; a conflict-aware join requests adjudication. This is an author-designed failure scenario and intervention, not a measured result from the scalability paper.
Repeated performance also matters. Yao and colleagues’ τ-bench evaluates tool use and dialogue in constructed service domains, including comparison of resulting database state with the expected outcome. It distinguishes succeeding at least once across attempts from succeeding consistently across repeated trials. Its simulated users and simplified domains limit generalisation, but the distinction is valuable for evaluating dependable execution.
Rabanser and colleagues’ Towards a Science of AI Agent Reliability broadens measurement beyond average success. Its June 2026 version evaluates 15 models on two benchmarks using twelve metrics organised around consistency, robustness, predictability and safety. In those settings, capability gains translate into only modest reliability gains. The study’s safety dimension concerns bounded operational consequences; its experiments do not evaluate adversarial attacks. The relevant lesson for harness evaluation is to measure how a system varies and fails, not to infer dependable execution from one aggregate score.
For our proposed investigation, the same report task should be run more than once. We would record accepted completion, source errors, requirement violations, human repair time and total resource use. We would also inspect whether failures recur at the same boundary. One excellent output cannot reveal how often an author must rescue the process.
An additional evaluation should vary the location of a requirement without changing its meaning, or introduce a legitimate brief update midway through execution. That would test whether success depends on a convenient initial arrangement. Such experiments are proposed here; the cited papers do not establish their outcome.
The benchmark can also be wrong
A benchmark score is evidence about a test procedure, whose validity must itself be examined. Kapoor and colleagues’ AI Agents That Matter argues for cost-aware evaluation, stronger holdouts and clearer separation of model capability from application performance. These concerns matter when a more elaborate graph purchases a higher score through additional sampling or repeated attempts. Report the gain together with the resources required to obtain it.
Zhu and colleagues’ benchmark audit documents concrete evaluation weaknesses, including inadequate tests in SWE-bench Verified and τ-bench cases where doing nothing can receive credit on deliberately impossible tasks because the expected database remains unchanged. This does not invalidate every task or the separate SWE-bench Lite results discussed earlier. It shows why final-state equality can be insufficient evidence of appropriate conduct. An agent may need to explain a refusal or identify an impossible request, even when no database modification is correct.
For the report, the analogous mistake would be accepting the presence of references as proof that claims are supported. The evaluator must inspect the relationship between claim and source. We should also test the evaluator on deliberately defective drafts and correctly cautious drafts. Otherwise, a system can become better at satisfying a weak test while making little progress on the assignment.
Open and proprietary systems: what can be inspected and changed?
Openness matters because diagnosing a failure requires access to the part of the system that may have caused it. However, openness is not one switch. A developer might control the harness while invoking a proprietary model. Another might run downloadable weights through a managed service whose orchestration is opaque. These arrangements expose different evidence and different opportunities for intervention.
The Open Source AI Definition 1.0 distinguishes access to parameters from a broader package of freedoms, code and data information. Under that definition, downloadable weights alone are insufficient. This is a particular published definition, not a universally settled taxonomy of openness. It is useful to state exactly what is available rather than use “open” as a general claim about transparency.
For the present review, four questions are more informative than a binary label. Can the developer inspect the assembled model input? Can they change the context-selection and routing code? Can they pin the relevant components to known versions? Can they examine and, where permitted and feasible, modify the model itself?
Figure 9. Access boundaries in two illustrative deployments. An inspectable harness can sit around either a remotely served proprietary model or a locally controlled model. Model access and harness access are separate properties. Local weights do not by themselves reveal training data or explain internal decisions. Definition reference: Open Source Initiative, version 1.0.
Suppose R1 is missing from a worker’s input. If the developer can inspect and modify context assembly, they can test whether the selection rule caused the omission. They can replay the same task with the requirement supplied explicitly. That experiment can isolate a harness-level contribution without changing the model.
If assembled inputs are unavailable, the investigation is less direct. The developer may compare outputs under alternative instructions, but cannot establish from those outputs alone exactly what an internal stage received. The limitation is evidential: several explanations may remain compatible with the observation. It does not prove that the proprietary system is unreliable.
Conversely, access to every prompt and state update does not explain every internal model decision. If R1 is visibly present and the model still misapplies it, an execution trace narrows the diagnosis but does not fully resolve it. Having weights may enable further experiments, yet running or training a large model can require resources beyond those available to the investigator.
Control also brings responsibility. In a self-managed deployment, the team must maintain serving, dependencies, permissions and evaluation. A managed service may take on some of that operational work. Those are allocation choices, not evidence that one category is categorically more capable. Their value depends on the engineering problem and the access actually provided.
Reproducibility should therefore be described in layers. Reproducing the harness configuration is different from reproducing the model, and reproducing either is different from obtaining an identical stochastic run. For the article workflow, a useful record would include prompts, context-selection rules, tool versions, source snapshots, stopping conditions and the model identifier privately or in a methods artifact where appropriate. Public analysis need not turn those details into a brand comparison.
Finally, closed source and closed loop mean different things. Closed-loop execution uses feedback to influence later actions. Both open and proprietary systems can implement it. Keeping this distinction explicit prevents an ownership question from being confused with a control mechanism.
What could make these systems more dependable?
Improve the signal before increasing repetition
The first intervention should address the identified failure. If the worker never received R1, another critique of the draft may miss the cause. If the worker received R1 but interpreted it incorrectly, merely increasing retention may not help. If the interpretation was correct but a later revision undid it, the intervention belongs at the update and review boundary.
For our report, I would preserve stable requirements separately from working notes, supply source-linked evidence records and make unresolved conflicts explicit. Reviews would refer to a particular artifact version. A revision would trigger rechecking of the claims it changes and any conclusions that depend on them. These are engineering proposals, not results established by adding a checklist.
The evaluation should test each intervention independently where possible. Otherwise, a combined improvement in prompts, tools, memory and budget leaves the contribution of each unknown. Some combinations may be necessary, but the interpretation should then concern the combined system.
For example, begin with the same set of public papers and a fixed report brief. Compare an unstructured handoff with a structured evidence record while holding the model, tool access and total budget constant. Have an evaluator who did not produce the draft check whether each material qualification survives. Repeat the comparison on topics that were not used to design the schema. If structured records improve retention but take substantially longer, report both outcomes.
A second experiment would introduce a deliberate disagreement between two source summaries. The test is whether the integration stage returns to the sources and resolves or accurately preserves the disagreement. Simply producing a confident conclusion would receive no credit. A third would change an explicit requirement after an initial draft and check whether the system updates dependent sections while preserving unrelated accepted content.
These experiments distinguish mechanisms more clearly than a general request to make the report better. They also expose costs that a final quality score can hide: repeated retrieval, unnecessary rewriting and the time needed for a person to inspect the result. Success criteria should be specified before examining outputs, with separate labels for unsupported claims, missing qualifications and editorial disagreement. Otherwise, the investigator risks calling any preferred draft a successful intervention.
Train correction as a capability
Kumar and colleagues’ SCoRe research directly addresses correction through multi-turn reinforcement learning. Rather than rely only on a request to reconsider, the training objective rewards useful revision. In Table 2, the trained configuration moves from 60.0% on a first attempt to 64.4% after correction on the MATH benchmark. That is a within-configuration change, not a comparison of brands.
Training covered one round of correction. The authors also investigated additional rounds at inference time; training for useful correction over longer sequences remains a separate question. The paper demonstrates that correction behaviour can be improved in the evaluated domains; it does not establish dependable revision of long research articles. Its relevance is that some weaknesses may require training changes as well as runtime design.
Figure 10. Simplified distinction between runtime feedback and correction training. Reflexion changes the context used for a later attempt; SCoRe changes model parameters during training to reward useful correction. Both can affect revision, through different mechanisms. See Reflexion, Section 3, and SCoRe, Figures 3 and 7.
This distinction helps avoid two premature conclusions. A failed prompt-only loop does not show that correction cannot improve. A successful correction-training experiment does not show that arbitrary runtime loops will improve indefinitely. We need evidence about the actual combination of task, trained behaviour and available feedback.
Make verification independent where independence is possible
An independent check need not be another model. For a numerical claim, it may be a calculation from the source data. For an artifact, it may be inspecting the file actually written. For a citation, it may be retrieving the exact passage rather than asking whether the reference looks plausible.
The independence is in the evidence or method. Two reviewers can share the same unsupported assumption despite using different models. A single model using a reliable external computation may receive a materially different signal. Neither arrangement should be declared superior without specifying what is being checked.
For interpretive claims, full independence is harder. A literature synthesis can have several defensible readings. The goal should be to expose the inference: which findings support it, which challenge it, and what assumptions connect the evidence to the conclusion. Human disagreement may remain legitimate after careful review.
Preserve the boundary between evidence and authority
So far, failures have largely involved mistakes. An external source can also be deliberately hostile. Greshake and colleagues’ indirect prompt-injection research demonstrates how attacker-controlled material encountered by an application can contain instructions intended to redirect its behaviour. For our report, a retrieved page might ask the agent to disclose another document or ignore the brief. Its arrival through a legitimate retrieval tool does not make those instructions authorised.
This failure differs from ordinary omission. The system promotes source content into authority over its own actions. Summarisation can compound the problem if it strips away provenance and preserves the malicious instruction as a task note. Passing that note to another agent can spread the error across an otherwise valid communication graph. These are illustrative mechanisms to test in the report workflow, not measurements of their prevalence.
Debenedetti and colleagues’ CaMeL research investigates a structural defence: separating control and data flow, tracking capabilities, and enforcing policies around tool use. The objective is to constrain what untrusted content can cause the system to do, rather than rely only on the model recognising malicious prose. Its guarantees depend on the stated policies and assumptions. The paper discusses limitations, including manipulation of text outputs that does not violate an enforced data-flow policy. Protecting an action boundary does not automatically establish the factual integrity of a summary.
Figure 11. Proposed trust boundary for the report harness, informed by indirect prompt-injection research and CaMeL. Retrieved material remains labelled as data; a separate policy check controls tool effects. This is an explanatory design, not a complete reproduction of CaMeL or a proof of security.
The report harness should therefore distinguish permission to read a source from permission to obey it. The evidence record should retain its origin through summaries and handoffs. A tool gate should evaluate the requested operation and its data dependencies against the authorised assignment. If a request exceeds that authority, the harness should block it while preserving the legitimate research task. Source verification and action control address related but separate questions: whether a statement is supported, and whether an operation is permitted.
Separate progress from activity
A useful revision record would state the defect being addressed, the changed evidence or action, and the observed result. If several attempts repeat the same diagnosis and produce no new information, the controller should reconsider the method. It should not interpret the growing transcript as proof that work is advancing.
For the report, progress could mean resolving a source contradiction, verifying an important claim or obtaining an author decision about scope. It need not mean adding words. Deleting an unsupported conclusion can be a substantive improvement even when it makes the output shorter.
Figure 12. Proposed repair process with evidence and version checks. Revisions are accepted after targeted verification and a check for damage to previously satisfied requirements. The preserved version enables rollback. This is a proposed design to evaluate, not a published guarantee of improvement.
The remaining challenge is measurement. A controller that estimates progress badly can stop useful exploration or permit endless unproductive revision. Progress signals should therefore be validated against independently assessed outcomes, rather than judged solely by the same process that produced them.
When another loop cannot supply what is missing
Kwa and colleagues’ Measuring AI Ability to Complete Long Tasks offers a useful empirical framing: relate success to the time human baselines take on tasks, then estimate a duration associated with a specified success probability. Its software-oriented task distribution and human-time estimates limit generalisation. A 50% task horizon is not the length of uninterrupted autonomous operation, nor a guarantee of success on arbitrary work of that duration. For our report, task length should therefore be paired with dependency structure and success criteria, rather than treated as a single universal capability threshold.
There are circumstances in which the present system cannot resolve the assignment with its available information and actions. These are operational boundaries, not proof that all future AI systems face the same permanent limit.
Consider two possible situations that produce identical observations for the agent. In one, a document is the approved final version; in the other, it is an obsolete draft. If no available input or tool distinguishes them, the system has no evidential basis for knowing which situation applies. More fluent reasoning cannot establish the missing fact. It needs another observation, such as version metadata or confirmation from the owner.
This is an information problem. A related problem concerns objectives. If the user requires two incompatible outcomes and has not specified a priority, no sequence of revisions can satisfy both as stated. The system can explain the conflict and request a decision. Silently choosing one changes the assignment rather than completing it.
A third boundary is evaluation. The system may generate several credible interpretations but lack a reliable way to choose among them. More candidates can broaden the options without improving selection. In a research article, the appropriate outcome may be a bounded conclusion that preserves uncertainty, rather than an invented resolution.
There is also an execution boundary. A system may correctly identify the needed action but lack the tool, access or authorisation to perform it. Further planning cannot substitute for the missing capability. The appropriate response is to identify the dependency and preserve the completed work so someone else can continue.
Figure 13. Proposed stopping and escalation logic. Each route corresponds to a different obstacle: missing evidence, incompatible requirements, unavailable execution capability, or failure to discriminate improvement. This is the review’s decision framework, not a universal optimal stopping algorithm.
For recoverable errors, retries can still be valuable. A transient tool failure is different from an unavailable source. An unsuccessful search query is different from information that does not exist in the accessible environment. The controller should distinguish a new attempt that changes the chances of learning something from one that merely restates the request.
Cost creates a further practical boundary. Even if another attempt has some chance of success, the expected benefit may not justify its time and resource cost. In the report example, a human may resolve a narrow editorial question quickly while further automated debate remains inconclusive. Escalation can be part of efficient execution rather than evidence that the whole system is useless.
The empirical question is where those boundaries occur and how reliably the system recognises them. An agent that stops too early leaves solvable work unfinished. One that stops too late consumes resources and may damage correct work. Neither behaviour is captured adequately by measuring output fluency.
What the evidence supports
Loops, graphs and harnesses make important capabilities usable. They connect models to information and tools, organise dependencies, preserve artifacts and create opportunities for correction. Research provides concrete examples in which interface design, structured execution and feedback improve measured outcomes. Dismissing this engineering as empty packaging would ignore those results.
The evidence also gives no basis for assuming that a sufficiently elaborate arrangement automatically preserves intent or judges completion correctly. Delivery of a message, use of its contents and reconciliation with earlier decisions are separate achievements. A review stage can run without performing a valid review. A persistent memory can preserve a mistake. Additional agents can create useful specialisation while increasing integration work.
For a system producing our research report, the important questions are consequently inspectable ones. Which requirements reached each stage? What evidence supports each conclusion? Which contradictions remain unresolved? What changed during revision? Did the final artifact satisfy the brief across repeated runs, and how much human repair was required?
Three investigations would materially strengthen this body of evidence. First, compare context-handoff designs on tasks where a small qualification changes the correct conclusion. Second, evaluate revision by counting both repaired errors and damage to previously correct content. Third, extend existing matched-budget comparisons to sustained research and editorial work, including human integration and review costs. These are proposed extensions of existing research, not claims that controlled architecture comparisons have never been conducted.
Openness affects the ability to conduct those investigations. Access to context assembly and execution traces can make particular failures easier to isolate and correct. Model access offers different experimental possibilities. Neither property alone guarantees dependable behaviour, but both should be described precisely when interpreting results.
The evidence brings us back to the distinction that motivated this review: a system can perform substantial useful work and still require someone to determine whether it has completed the assignment. Better interfaces, context management and correction methods can reduce that gap. Their success does not establish that the remaining gap will disappear simply by adding more agents or extending the loop.
This is where the limits of autonomy become concrete. A system cannot reliably establish a fact when its available evidence cannot distinguish between competing answers. It cannot determine an author’s unexpressed priorities, or satisfy incompatible requirements without a decision about which takes precedence. It can produce another revision, but without a sufficiently reliable check, it may have no sound basis for deciding whether that revision improved the work.
These are different problems from an awkward prompt or a poorly designed handoff. Some require new information. Others require a change in the objective, additional execution capabilities or judgment from outside the existing process. An agent can sometimes obtain those inputs through its tools. Where it cannot, further iteration does not remove the dependency.
For a research article, that dependency may concern whether the argument preserves the author’s intended distinction. For an application, it may concern whether the implemented behaviour meets a requirement that the tests never captured. In both cases, the system can complete its internal workflow while leaving the actual assignment unfinished.
The claim that follows is narrower, but more consequential, than saying AI makes mistakes. Humans make mistakes too. Perfect precision on every attempt is an unrealistic standard for either. The practical question is how much supervision, verification and repair a system requires to deliver acceptable work consistently, and whether its claimed autonomy accounts for that effort.
Agent engineering deserves credit for expanding what AI can accomplish independently. It connects capabilities that would otherwise remain isolated, makes useful feedback available and enables recovery from some failures. But loops, graphs and harnesses do not provide a general guarantee of independent, precise completion. They organise the process through which a result is produced; the adequacy of that result still depends on the evidence, objectives and checks available to the system.
That is the boundary this review argues we should make explicit. Useful automation can coexist with a continuing need for human judgment. A system may perform most of an assignment while leaving its most consequential decisions unresolved. The value of the completed work and the effort required to make it dependable must therefore be assessed together.
A system’s ability to keep working is not proof that it can determine, by itself, when the work is right.
Bibliography
Entries identify primary research, implementation specifications and definition documents. Dates distinguish original publication years from the specific versions inspected. Model identifiers remain available in the original methods; they are not used here to rank products. The review contains no new experimental results.
Brown, T. B., et al. (2020). Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems, 33. Relevant material: Section 2, training versus in-context task specification.
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. (2023). ReAct: Synergizing Reasoning and Acting in Language Models. International Conference on Learning Representations. Originally posted 2022; arXiv v3 inspected. Relevant material: Figure 1; Sections 2–4.
Yang, J., Jimenez, C. E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., and Press, O. (2024). SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. Advances in Neural Information Processing Systems. arXiv v3 inspected. Relevant material: Section 3; Tables 1 and 3; Section 5.1.
Xia, C. S., Deng, Y., Dunn, S., and Zhang, L. (2024). Agentless: Demystifying LLM-based Software Engineering Agents. Published as Demystifying LLM-based Software Engineering Agents, FSE 2025, DOI 10.1145/3715754. arXiv v2 inspected. Relevant material: three-stage repair architecture, evaluation and benchmark analysis.
Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., and Yao, S. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. Advances in Neural Information Processing Systems. arXiv v4 inspected. Relevant material: Figure 1; framework and feedback mechanisms; evaluations and limitations.
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. (2024). Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics, 12, 157–173. Originally posted 2023. Relevant material: multi-document question answering and key-value retrieval evaluations.
Packer, C., Wooders, S., Lin, K., Fang, V., Patil, S. G., Stoica, I., and Gonzalez, J. E. (2023). MemGPT: Towards LLMs as Operating Systems. arXiv:2310.08560, v2 preprint inspected. Relevant material: Figure 3; Section 2; document and conversation evaluations.
Madaan, A., et al. (2023). Self-Refine: Iterative Refinement with Self-Feedback. Advances in Neural Information Processing Systems. Relevant material: feedback-refinement architecture and seven-task evaluation. Full author list appears in the linked paper.
Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X., and Zhou, D. (2024). Large Language Models Cannot Self-Correct Reasoning Yet. International Conference on Learning Representations. Originally posted 2023. Relevant material: Section 3, including transitions between correct and incorrect answers and the role of external feedback.
Cemri, M., et al. (2025). Why Do Multi-Agent LLM Systems Fail?. arXiv:2503.13657, v3, 26 October 2025, version inspected. Relevant material: Sections 3–4 and Appendix N.11. Study counts refer to this version.
Zheng, L., et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems, Datasets and Benchmarks Track. Relevant material: Section 3 and Appendix D, judge behaviour and bias analyses.
Yao, S., Shinn, N., Razavi, P., and Narasimhan, K. (2024). τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv:2406.12045, v1 preprint inspected. Relevant material: domain design, state-based evaluation, repeated-trial metrics and simulated-user limitations.
Kumar, A., Zhuang, V., Agarwal, R., Su, Y., et al. (2024). Training Language Models to Self-Correct via Reinforcement Learning. arXiv:2409.12917, v2 preprint inspected. Relevant material: Figures 3 and 7; Table 2; Section 7. Full author list appears in the linked paper.
Sander, L., Pan, F., Zolfaghari, V., Schamschurko, A., Petrovic, N., and Knoll, A. (2026). Scaling LLM-Driven Multi-Agent Systems: Design Principles and Architectural Scalability Analysis. arXiv:2607.27942, v1, July 2026, preprint. Relevant material: architecture, experimental configurations and limitations. Reporting inconsistencies are noted in the text; no numerical performance comparison is drawn from it.
LangChain documentation. Graph API overview. Implementation specification, accessed 11 September 2026. Relevant material: state, nodes, edges and reducers. Cited for concrete semantics, not as independent evidence of effectiveness.
LangChain documentation. Persistence. Implementation specification, accessed 11 September 2026. Relevant material: checkpoints, stores and resumption.
A2A project. Agent2Agent protocol specification. Technical specification, accessed 11 September 2026. Relevant material: messages, tasks, artifacts and task states. The live specification may change after this access date.
Open Source Initiative (2024). The Open Source AI Definition, version 1.0. Definition document, accessed 11 September 2026. Relevant material: freedoms, data information, code and parameters.
Sumers, T. R., Yao, S., Narasimhan, K., and Griffiths, T. L. (2024). Cognitive Architectures for Language Agents. Transactions on Machine Learning Research. Originally posted 2023; v3 inspected. Memory, actions and decision cycles.
Modarressi, A., et al. (2025). NoLiMa: Long-Context Evaluation Beyond Literal Matching. arXiv:2502.05167, v3 inspected. Associational retrieval and context-length evaluation.
Lodha, A., Varnosfaderani, M. P., Chakraborty, A., and Mithal, A. (2026). Less Context, Better Agents: Efficient Context Engineering for Long-Horizon Tool-Using LLM Agents. June 2026 preprint, v1. Methods, context policies and task-completion results.
Model Context Protocol contributors (2025). Architecture, specification dated 18 June 2025. Versioned technical specification. Host, client, server and capability boundaries.
Kamoi, R., Zhang, Y., Zhang, N., Han, J., and Zhang, R. (2024). When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs. Transactions of the Association for Computational Linguistics, 12, 1417–1440. DOI: 10.1162/tacl_a_00713.
Tran, D., and Kiela, D. (2026). Single-Agent LLMs Outperform Multi-Agent Systems on Multi-Hop Reasoning Under Equal Thinking Token Budgets. April 2026 preprint, v2. Matched-budget comparisons and assumptions of the theoretical analysis.
Rabanser, S., Kapoor, S., Kirgis, P., Liu, K., Utpala, S., and Narayanan, A. (2026). Towards a Science of AI Agent Reliability. ICML 2026; arXiv v3, 2 June 2026, inspected. Sections 3–4 and experimental limitations.
Kapoor, S., Stroebl, B., Siegel, Z. S., Nadgir, N., and Narayanan, A. (2024). AI Agents That Matter. arXiv:2407.01502, v1 inspected. Cost, holdouts and agent-evaluation methodology.
Zhu, Y., et al. (2025). Establishing Best Practices for Building Rigorous Agentic Benchmarks. arXiv:2507.02825, v5, August 2025, inspected. Benchmark validity and concrete evaluator failure cases.
Greshake, K., et al. (2023). Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. arXiv:2302.12173, v2 inspected. Attacker-controlled external content and application trust boundaries.
Debenedetti, E., et al. (2025). Defeating Prompt Injections by Design. arXiv:2503.18813, v2, June 2025, inspected. CaMeL design, security assumptions and limitations.
Kwa, T., West, B., et al. (2025). Measuring AI Ability to Complete Long Tasks. arXiv:2503.14499, March 2025 v1 inspected. Human-time baselines, task-horizon methodology and limitations.