DeCLaRe Lab

ScrambleToolBench: Agents Search Exhaustively Even When Their Own Map Points to the Next Step

2026-09-09T00:00:00+00:00

Lab note

ScrambleToolBench: Agents Search Exhaustively Even When Their Own Map Points to the Next Step

An agent discovers what its tools do. Their names change, and it starts searching again—even when its earlier observations point to the next call.

Vernon Toh

September 9, 2026 · 10 min read · Learning from Interaction

Paper GitHub

The agent first has to discover what the commands do. A useful response leads it to the next command and eventually to a solution; that experience carries into the next task. View full-size figure.

Suppose an agent has worked out which command reads a file and which one queries a database. It has written the names down and used both successfully. Then the interface changes. The command that used to read a file now calls a different function.

The agent could try every command again. But the unexpected response contains a clue: it tells the agent which function has moved into that name. Its old map can tell it where that function used to be. Together, those two observations suggest a specific next call.

We built ScrambleToolBench to examine what agents do at moments like this. The strongest models can discover unfamiliar tools and complete tasks with them. Recovering cheaply after a change is harder. Even with the relevant entries in memory, they rarely follow the chain those entries imply.

This connects to a question from my MNIST-PRO note: when an agent has already gathered useful evidence, what stops it from using that evidence? Here the evidence is a tool’s behavior rather than a fragment of an image. We can inspect the next action to see whether the agent has made the connection.

What the tool names normally tell you

A name such as read_file gives an agent a useful starting hypothesis. It suggests both the operation and the kind of argument to supply. That is helpful in practice, but it makes it difficult to tell how much an agent has learned from the current environment and how much it brought with it.

ScrambleToolBench removes these cues from 28 tools for file operations, network diagnostics and data processing. A tool receives an arbitrary name such as fn_3d8a. Its parameter names are replaced with tags such as arg_0, and the initial schema does not list them. A malformed call reveals the required keys and types. Output fields and success indicators are also obfuscated, so the agent has to interpret the responses it receives.

The environment is a Python simulator, with 20 procedural task templates. An episode contains five tasks, each with a budget of 100 inference steps. The tasks share a history: a discovery made while investigating a service can be useful later when locating a file or retrieving a value. Three control commands, including submit_solution, remain readable so that submitting an answer does not itself require tool discovery.

We then change the conditions in three ways. Mapping drift moves seven of the 28 tool identifiers in a cycle between tasks. Action failure makes valid executions return a timeout with probability 0.15. Execution windows require certain procedures to finish within ten actions of a trigger; an expired window resets the state and regenerates intermediate values. We test each change separately and all three together.

These changes require different responses. A timeout can justify retrying a correct call. A changed mapping requires revising which command to use. An expired window requires restarting the procedure rather than reusing values from the previous attempt. An agent that treats all three as the same kind of error will waste calls or carry an incorrect assumption forward.

Discovery can succeed while adaptation fails

The main evaluation uses 20 five-task episodes per model and condition. Episode completion means solving all five tasks. With 20 episodes, one completed episode changes the reported completion rate by five percentage points.

The selected results below show why the changing environment matters. Claude Sonnet 5, Gemini 3.1 Pro and Gemini 3.5 Flash all complete every episode in the static scrambled setting. Under the three combined changes, their completion rates fall to 0%, 20% and 25%.

Model	Named tools	Scrambled	+ Drift	+ Failure	+ Window	+ All
Qwen 3.6 27B	95%	55%	25%	35%	15%	0%
Qwen 3.6 27B + Memory	—	60%	40%	35%	35%	0%
Claude Sonnet 5	100%	100%	100%	100%	70%	0%
Gemini 3.1 Pro	100%	100%	90%	100%	80%	20%
Gemini 3.1 Pro + Memory	—	100%	100%	100%	90%	50%
Gemini 3.5 Flash	100%	100%	90%	85%	65%	25%
Gemini 3.5 Flash + Memory	—	100%	95%	95%	80%	30%
Mean across all 15 models, without memory	93%	32%	23%	26%	19%	3%

These are selected rows from Table 2 of the paper. The mean includes the full set of 15 no-memory models, not just the four displayed here. A dash denotes an unreported condition, not zero performance.

The memory variant maintains two structured stores: task recipes and tool knowledge. One records reusable procedures; the other records inferred tool behavior, arguments and confidence. The agent updates these stores as it acts. For Gemini 3.1 Pro, memory raises completion in the combined condition from 20% to 50%. That is a useful gain, but it leaves another question: does the agent use its remembered map to reason about the change, or does it simply search more successfully?

A wrong call can tell you where to look next

Consider a small illustrative example. Before a change, command A reads files, B queries the database and C lists interfaces. After the names rotate, A lists interfaces, C queries the database and B reads files.

The agent wants to read a file, so it starts with A. The response identifies the interface-listing function. The old map says that function used to be at C, so C is the next place to look. C now returns the database function, whose old name was B. Calling B reaches the file reader. The route is A → C → B; the agent did not need to scan the remaining commands.

That is cycle tracing. After a mismatch, identify the function that answered, look up its previous identifier, and call that identifier next. Repeat until the required function appears. The stable, function-specific response fields let the agent match a response to an earlier observation even when the command name changes.

For the benchmark’s seven-identifier cycle, this takes six extra calls relative to an ordinary successful tool call. That guarantee assumes the stored map was current immediately before the drift and the returned functions can be identified. The same calls reveal the changed part of the map, allowing it to be repaired before the next task. If the map mixes unresolved observations from several changes, the six-call bound no longer follows.

The paper also gives a reference for a task requiring four of the 28 functions. With seven identifiers chosen uniformly for drift, the probability that at least one required function moved is one minus the probability that all four stayed unchanged. Multiplying that probability by six gives 4.25 expected extra actions at one fresh task boundary. This is a recovery reference under the stated assumptions, not a bound on the cost of discovering the whole environment or handling the combined stressors. The derivation is in Section 4.2.

Do agents follow that clue?

We measure whether, after observing a changed mapping, the agent calls the next identifier in the chain within three actions. The comparison is with random selection among identifiers the agent has already mapped.

Model	Reasoning	Opportunities	Follows within three actions	Random baseline
Gemini 3.1 Pro	High	484	11.0%	10.8%
Gemini 3.1 Pro + Memory	High	444	12.8%	10.9%
Claude Sonnet 5	Low	564	14.0%	10.6%
Claude Sonnet 5	Medium	590	11.9%	10.6%
Claude Sonnet 5	High	604	14.1%	10.6%

The opportunity counts are diagnostic events, not independent episodes; the analysis excludes early-exit episodes as described in Table 5. Gemini’s follow rates are close to the random baseline, including with memory. Sonnet is modestly above it at low and high reasoning, but increasing reasoning does not make it follow the chain more consistently. These results do not mean that no useful deduction ever occurs. They show that this particular recovery rule is not used reliably.

One trajectory makes the alternative behavior easy to recognize. Qwen 3.6 27B had used fn_fc40 for file search. After drift, it kept varying the arguments to that name over a seven-step loop, trying different search patterns instead of reconsidering which function the name now referred to. The agent was changing its query while keeping the more consequential assumption fixed.

Memory can also be revised sensibly. In another episode, Gemini 3.1 Pro explicitly cleared its old tool mappings while retaining task progress and critical details. Under an execution window, Gemini 3.5 Flash stopped making exploratory calls and focused on the required sequence. These examples show useful local adaptation. They also explain why a single success rate cannot tell us whether the agent found the cheaper recovery procedure.

More reasoning improves completion, but not the shortcut

We vary the reasoning effort for Gemini 3.1 Pro and Claude Sonnet 5 in both the static scrambled environment and the drift condition. Higher effort improves the number of tasks completed, especially for Gemini. The chain-following results above show that this improvement should not be mistaken for reliable cycle tracing.

Mean tasks completed per episode, out of five. Blue circles denote the static scrambled Base condition; orange squares denote Drift. View full-size figure.

Completion tokens per solved task count tokens spent across all episodes, including failed ones, and divide by the total number of tasks solved. At high reasoning under drift, Sonnet uses 11,652 tokens per solved task, compared with Gemini’s 3,332—about 3.5 times as many at similar task completion.

Completion tokens per solved task, in thousands. The cost includes unsuccessful episodes. Lower reasoning effort is not consistently cheaper. View full-size figure.

The cost does not rise monotonically with reasoning effort. Sonnet’s low-effort drift setting is the most expensive per solved task; Gemini’s cost changes little across effort levels. The distinction is between completing more tasks and adopting a more economical strategy. A larger reasoning budget can improve the former while leaving the latter largely unchanged.

The connection to MNIST-PRO

In MNIST-PRO, an agent moves a small window over a handwritten digit. It may collect many useful fragments and still give the wrong answer. In ScrambleToolBench, it may identify a changed function and retain the relevant earlier mapping, yet choose an unrelated command next. Both failures happen after useful evidence has reached the agent.

The two benchmarks let us investigate that gap in different ways. For MNIST-PRO, we keep a completed trajectory fixed and assemble its observed glimpses into a coordinate-aligned canvas. No new region becomes visible. If the final answer improves, a better search path cannot explain the gain: organizing the existing observations has helped. For example, on Claude Opus 5’s Textual State trajectories, this changes Level 1 accuracy from 41% to 76% and Level 2 accuracy from 13% to 67%.

For ScrambleToolBench, we inspect a decision rather than supply a consolidated view. Once a changed call returns a known function, does the agent use its old map to choose the next identifier? The low follow rates show that storing the relevant entries does not reliably produce that action. We have not shown here that a different memory format would fix the problem; the MNIST-PRO canvas result is not evidence that the same intervention works for tool use.

There is also an important difference in what changes. The digit in MNIST-PRO stays still. An early guess may be wrong, or the agent may have failed to join its views together, but a previously observed stroke does not move. ScrambleToolBench deliberately changes the interface: a mapping that was correct can become stale. The first task calls for constructing and interpreting a state from partial views; the second additionally calls for revising that state when the environment changes.

This is why I would not treat either result as a general argument for more memory. A history can preserve the right observations without exposing their spatial relationship. A tool dictionary can preserve a correct old mapping without prompting the reverse lookup that would make it useful now. The question is what the stored information allows the agent to do at the next decision.

What I would test next

For tool use, I would separate three checks: whether the map contains the needed entries, whether the agent recognizes a mismatch, and whether it follows the implication of that mismatch. A controlled intervention could provide the current pre-drift map, then test a reverse-lookup rule only when a response identifies a displaced function. That would help distinguish incomplete discovery from a failure to use information already available. It is a proposed test, not a result reported here.

For perception, the corresponding check is to hold the path fixed before changing how its observations are represented. If the agent never saw the second digit, a better arrangement of its existing glimpses cannot supply it. If it did see the distinguishing strokes, the next question is whether it can use them together.

The practical aim is to locate the missed step. In ScrambleToolBench, an error response can be the observation that tells the agent where to call next. In MNIST-PRO, several individually ambiguous crops can become a recognizable digit when placed together. An agent needs to carry those relationships into its decisions, not just carry the observations into its context.

Citation

@misc{toh2026scrambletoolbenchagentssearchexhaustively,
  title={ScrambleToolBench: Agents Search Exhaustively Even When Their Own Map Points to the Next Step},
  author={Vernon Toh and Navonil Majumder and Zhengyuan Liu and Nancy F. Chen and Soujanya Poria},
  year={2026},
  eprint={2608.02358},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2608.02358}
}

MNIST-PRO: MNIST is Back as a Partially Observable World for AI Agents

2026-09-08T00:00:00+00:00

Lab note

MNIST-PRO: MNIST is Back as a Partially Observable World for AI Agents

When an agent gets a digit wrong, did it miss the evidence or struggle to put the glimpses together? We use MNIST-PRO to tell these failures apart.

Vernon Toh

September 8, 2026 · 7 min read · Agentic Perception

Paper GitHub

The full digit on the left is hidden from the agent. The selected glimpses show a successful search that ends with the answer 0 at step 9. View full-size figure.

Look at the zero above. With the whole image in view, recognizing it is easy. Now cover everything except a small square near the bottom-right stroke. That fragment could belong to several digits. To answer, you have to move the window, remember the earlier fragments, and work out how they fit together.

In the illustrated episode, the agent moves up along the stroke, reaches the top of the digit, and eventually answers 0. The interesting part is not the label. It is the work between the first glimpse and that answer: deciding where to look, keeping track of position, and deciding when the observations support a conclusion.

We built MNIST-PRO to study that work. In this note, I want to focus on what a wrong answer can tell us—and why collecting more evidence is not always the same as making better use of it.

Why return to handwritten digits?

Imagine navigating a room with only a flashlight. You need to combine successive views into a useful picture of your surroundings. But if a robot fails in that room, the cause may be difficult to identify: it could have misread an object, forgotten a location, collided with something, or failed to execute an action.

MNIST gives us a simpler setting. There is no physics to simulate and no object to manipulate. The agent moves a window over a static image and reports a digit. We can record exactly which parts it saw before it answered.

The window is 64 × 64 pixels and moves 32 pixels at a time, up, down, left or right. Level 1 places one digit on a 224 × 224 canvas. Level 2 places two digits side by side on a 224 × 448 canvas and asks for their ordered sequence: seeing a 5 and an 8 is not enough if the answer should be 58, not 85.

This keeps the recognition problem familiar while changing how the evidence arrives. It also gives us two separate questions to ask of an unsuccessful episode: did the agent look in the right places, and could it use what it had already seen?

What survives after the window moves?

Keeping every image in context sounds like an obvious solution. That is our Image Only baseline: the agent retains its previous glimpses and actions. But a sequence of crops is not a map. The model still has to work out which views overlap and where each fragment belongs.

We compare this with two ways of writing information down. In Textual State, only the current image is visible; descriptions of earlier observations carry information forward. In Metric Grid Map, the agent also records features against relative coordinates in a structured map. A description tells it what it saw. Coordinates are intended to help it remember where.

These representations impose different demands. A text description can omit a small visual detail. A map can contain an incorrect position. A long visual history can contain all the fragments without making their spatial relationship clear. None of these formats guarantees that the agent has a reliable picture of the digit.

The full-image control makes the contrast concrete. Gemini 3.1 Pro reaches 99% accuracy on the single-digit control but 38% in the Image Only condition. The digit has not become a different object; the model now has to gather and combine views of it.

Did it actually see enough?

Accuracy alone does not answer this. We also measure stroke coverage: how much of the digit’s foreground was exposed through the glimpse window. This is different from the number of moves. An agent can spend many steps returning to the same region.

Evidence acquired versus task accuracy. Each point averages the four conditions formed by Level 1 and Level 2 × Image Only and Textual State; it does not include Metric Grid Map. View full-size figure.

The plot shows why coverage is worth measuring separately. Claude Fable 5 covers more of the strokes than Gemini 3.7 Flash in this comparison, yet answers fewer episodes correctly. In the Textual State condition specifically, Fable and Opus exceed 80% stroke coverage, while their Level 1 accuracies are 38% and 41%; their Level 2 accuracies are 18% and 13%.

That does not make coverage useless. Nor does it prove that memory is the sole cause of failure. A large covered area may still leave out the part that distinguishes two digits. What it tells us is that “the agent should look more” is not a complete diagnosis.

To go further, we need an intervention that changes how the evidence is presented without giving the agent another chance to explore.

Keep the path fixed; put the glimpses together

We take completed Image Only and Textual State trajectories and arrange the already observed glimpses on a coordinate-aligned canvas. The model then makes a final prediction from this consolidated view. The route stays fixed. No new region of the digit is revealed.

This matters because a better answer can no longer be attributed to a better search path. We have changed the organization of the evidence available at decision time.

Open circles show the original predictions; filled circles show predictions with the consolidated canvas. Coverage stays fixed. Both endpoints use the same four-condition averaging as the preceding plot. These are offline final-answer interventions, not new exploration runs. View full-size figure.

All eight model settings plotted improve on average. Looking at the Textual State trajectories separately makes the size of the change easier to see:

Model	Level 1: original → canvas	Level 2: original → canvas
Claude Fable 5	38% → 81%	18% → 69%
Claude Opus 5	41% → 76%	13% → 67%
Gemini 3.6 Flash	47% → 89%	15% → 49%

These results support a narrower, useful conclusion: for these trajectories, much of the error can be reduced by organizing the observations differently. They do not show that exploration no longer matters. A canvas cannot recover a stroke that the agent never saw. And the intervention supplies spatial alignment, so it does not separate every possible failure in remembering, aligning and interpreting the fragments. The per-condition results are in Table 2 of the paper.

Some agents stop before reaching the second digit

The other side of the diagnosis is visible in the trajectories. In an audit of 1,600 Level 2 episodes using Textual State or Metric Grid Map, 323 predictions contained only one digit. In 238 of those episodes, the agent never looked at the second digit at all. In 266, it exposed less than a quarter of that digit. These single-digit answers arrived after roughly 15 steps on average, out of an available 78.

Here, reorganizing the existing evidence cannot solve the missing-observation problem. The agent has committed before checking the whole task.

We also see redundant exploration and failures to revise early hypotheses. For Gemini 3.7 Flash, adding a Metric Grid Map reduces the revisit rate from 45.1% to 13.6%. That is a substantial change in where it spends its moves, but a more efficient route is not itself a correct answer. Likewise, an early guess must remain revisable: later observations are useful only if they can change the prediction.

These failures call for different tests. An episode that never reaches the second digit needs a different explanation from one that observes both digits and still answers incorrectly.

What if the agent can write its own tools?

The offline canvas is something we construct for the model. We also tested a harness in which agents can write and execute Python and manage local files. Could they build useful representations during the task themselves?

In the Gemini 3.7 Flash runs, the agent wrote scripts such as stitch.py and render_mnist.py to assemble glimpses into a coordinate-aligned matrix, inspect an ASCII rendering, and guide further exploration. Its harness accuracy was 88% on Level 1 and 63% on Level 2.

This is a different experiment from replaying a fixed trajectory: tools can change both the representation and the search that follows. The result is encouraging, but it should not be read as an isolated measurement of memory quality. It shows a model using computation to make its observations easier to work with.

When the observations are tool responses

My ScrambleToolBench note asks a related question about tool use. An agent learns which commands perform which operations, then some command names change. A response from a displaced function, together with the old map, can point to the next command to try. Agents often search elsewhere instead.

The distinction matters. In MNIST-PRO, the digit stays fixed, and assembling the same glimpses into a canvas tests whether their organization affects the answer. In ScrambleToolBench, the interface changes, and we inspect whether the agent uses the resulting mismatch to revise its next action. Both ask what happens after useful evidence arrives. Keeping that evidence available is only part of the work; the agent still has to use the relationship between observations.

What I would check before adding more context

For an agent that fails on a partially observed task, I would first inspect what it had actually seen at the time it answered. Did it reach every relevant object? Did it revisit the same area? Did it stop while much of the task remained unseen?

Then I would hold that evidence fixed and change its representation. If the answer improves when the same observations are organized into a map, collecting additional observations is not the only route to improvement. If crucial evidence is absent, the search and stopping policy need attention too.

MNIST-PRO lets us make those checks in a small, controlled world. Handwritten digits do not establish how an agent will behave on a website or in a physical room. They do give us a way to tell apart errors that a single accuracy score would otherwise hide.

Citation

@misc{toh2026mnistpromnistpartiallyobservable,
  title={MNIST-PRO: MNIST is Back as a Partially Observable World for AI Agents},
  author={Vernon Toh and Navonil Majumder and Zhengyuan Liu and Nancy F. Chen and Soujanya Poria},
  year={2026},
  eprint={2608.31022},
  archivePrefix={arXiv},
  primaryClass={cs.AI},
  url={https://arxiv.org/abs/2608.31022}
}

IDEAgent: Agentic Quality-Diversity Search for Research Idea Generation

2026-08-12T00:00:00+00:00

Lab note

IDEAgent: Agentic Quality-Diversity Search for Research Idea Generation

IDEAgent asks whether a fixed generation budget can produce more ideas that meet stated quality thresholds without repeating one another.

Varun Gumma

August 12, 2026 · 7 min read · AI for Science

Paper GitHub Medium

Introduction

Most ideation systems optimize one proposal at a time. This can produce a high-scoring idea while the full set remains repetitive. IDEAgent instead treats ideation as a quality-diversity search problem: under a fixed budget, it tries to increase the number of ideas that clear stated quality thresholds and remain distinct from those already accepted.

How IDEAgent works

Figure 1: IDEAgent generates ideas sequentially for diversification and applies repair or refinement to eligible proposals. Descendants retain the lineage of their parent.

The system has four main pieces:

Generate ideas sequentially, using the context of prior ideas to explicitly avoid repetition and encourage diversification.
Decompose each free-form idea into specific fields, providing a more interpretable representation for pairwise comparison between ideas.
Treat improvements to an idea as children of a common parent and assign them the same lineage identifier, so they are not mistakenly counted as separate “new discoveries.”
Maintain archives of currently accepted ideas, historical/retired ideas (superseded by newer versions), and rejected ideas and families. Rejected ideas are clustered into families to capture broader avoidance patterns and help identify similar ideas in the future.

Figure 2: An incoming idea can replace its closest active duplicate if it has a higher quality score. The other version moves to the historical archive.

Six LLM agents handle generation, summarisation and evaluation:

Ideator: The primary generator that produces new ideas or refines existing ones.
Stenographe: Compresses each idea into a structured representation by identifying the problem addressed, central mechanism, novel addition over the background literature, key assumptions, and expected measurable effect from the prose.
Quality Judge: Grades the idea ($I$) on Non-obviousness ($N_I$), Clarity ($C_I$), and Feasibility ($F_I$) on a scale of 0–100.
Soundness Panel: Assesses the logical and mathematical rigor of the idea and its assumptions using five independent judgments for broader coverage. The judgments are given on a scale of 0–9 and then aggregated into a single score on a 0–100 scale ($S_I$).
Diversity Judge: Compares the idea against all archives and identifies the nearest duplicates from each. It also assigns a single score ($D_I$) on a 0–100 scale to capture the overall uniqueness, novelty, and diversity of the current idea with respect to prior generations.
Critic: Assimilates the judgments and scores from the three aforementioned judges to produce a single, targeted improvement directive for the Ideator, guiding it to improve the current generation.

We define the overall quality score as: $Q = 0.7N_I + 0.2S_I + 0.1C_I$, with ties decided by $F_I$.

Figure 3: IDEAgent's agents, archives and evaluation loop.

Figure 4: Transitions between generation, rejection, repair, acceptance and refinement.

Search procedure

An idea generated by the Ideator is first compressed by the Steno into a structured summary, which is used for future comparisons.
The full idea is then scored by all three judges. The Diversity Judge also produces a list of near-duplicates from each archive.
If the idea fails to clear the required threshold for any metric (Non-obviousness, Clarity, Soundness, or Diversity) by more than a permissible margin, it is rejected and added to the rejected archive. It is also assigned to one or more rejection families based on its failure pattern.
If the idea matches an idea in the historical archive but has no match in the active set, it is discarded because it has replicated an older idea that has already been superseded.
If the idea falls below a threshold but remains within the permissible margin, it gets an opportunity for repair. Based on the Critic’s feedback, which incorporates the evaluation results, the Ideator corrects the bottleneck or problematic aspects of the idea. The repaired idea is then evaluated again. If it still fails to cross the required thresholds, it is rejected.
If the idea, either before or after repair, clears all the thresholds, it becomes eligible for acceptance. If the active set (which has a fixed capacity) is not full, the idea is added directly. If the active set is full, the incoming idea competes against its closest match in the active set, as identified by the Diversity Judge. The idea with the higher overall (Q) score is retained, while the other is moved to the historical archive. If the incumbent is replaced, the newcomer inherits its lineage because it was identified as a duplicate of the incumbent, thereby preserving the lineage.
Once an idea is accepted into the active set, it has up to two additional refinement opportunities, provided that one of them was not already used for repair. Refinement is triggered when the idea falls below a higher soft target above the acceptance thresholds. The Critic’s feedback is again provided to the Ideator, and the resulting variant is accepted if it preserves Non-obviousness while improving the overall quality score. If the refinement does not meet these criteria, the new child is discarded.
Once the search budget is exhausted, the ideas remaining in the active set are treated as the final variants and passed on for downstream evaluation.

Evaluation

We evaluate our ideas at both the individual and group levels using LLM-based judges that are distinct from those used during generation (i.e., the internal evaluation).

At the individual level, we evaluate each idea for Non-obviousness, Soundness, and Clarity as our primary quality metrics, each on a 0–9 scale. At the group level, we compute Diversity pairwise between ideas, also on a 0–9 scale. Finally, Yield filters ideas through predefined quality gates and measures the largest subset that also satisfies a pairwise diversity threshold.

Results

We compare IDEAgent with four alternatives and ablations:

Stateless: Naive parallel generation of N ideas, with no information shared between them.
Single-Shot: Generates all N ideas at once in a single output. For a thinking-based Ideator, the reasoning space is shared across all N ideas, allowing them to share a common memory block for reference and diversification.
Sequential Memory: Sequentially generates ideas while providing summaries of prior generations as references for diversification. In other words, this is IDEAgent without the quality-improvement component.
NOVA: A NOVA-inspired version of Sequential Memory that uses iterative seed-pool-based germination and replacement for diversification. In each round, N ideas are generated, after which an internal judge selects the K most promising ideas as seeds for generating the next N ideas. After three rounds, the internal judge selects the most diverse N ideas from the resulting 3N ideas.

Figure 5: Results across 32 topics. A successful topic has non-zero Yield. S: soundness; C: clarity; D: diversity; NB: non-obviousness.

Figure 6: Yield for IDEAgent and Sequential Memory at $S \in [6, 8]$, $NB \in [6, 7]$ and $D \geq 7$.

Full 10x10 Yield surface: $S \in [6, 8]$, $NB \in [6, 7]$, $D \geq 7$.

Limits and next steps

IDEAgent evaluates ideation at the level of a set, not only one proposal at a time. Yield makes the quality and diversity thresholds explicit. Both the search and the evaluation depend on model-based judges, so the results do not show that the generated proposals are scientifically correct or useful. Expert review, literature checks and experiments are still required.

Citation

Code: https://github.com/declare-lab/IDEAgent
Paper: https://arxiv.org/abs/2607.22375

@misc{gumma2026ideagentagenticqualitydiversitysearch,
      title={IDEAgent: Agentic Quality-Diversity Search for Research Idea Generation},
      author={Varun Gumma and Navonil Majumder and Soumitra Sinhahajari and Soujanya Poria},
      year={2026},
      eprint={2607.22375},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2607.22375},
}

Toward Efficient Data-Centric Training, Part II: Beyond What to Select - How Much Data Should We Use?

2026-05-14T00:00:00+00:00

Lab note · Part II

Toward Efficient Data-Centric Training, Part II: Beyond What to Select - How Much Data Should We Use?

Part I asked which examples a model should use. PODS asks whether the amount of selected data should also change during training.

Suorong Yang

May 14, 2026 · 5 min read · Efficiency

Paper Part I

PODS changes the selected data volume over the course of training.

Why a fixed selection ratio is limiting

Most data selection methods focus on deciding what to select. They may dynamically change the identity of selected samples, but the selected data volume is usually fixed by a target ratio such as 50% or 70% throughout training.

That assumption is convenient, but training is not static. Early, middle, and late training phases can have different optimization needs. Sometimes, seeing less data can act like regularization by forcing the model to focus on informative or harder samples. At other times, seeing more data is important for coverage and stable optimization.

If the ratio is too low, training may be efficient but unstable or biased. If the ratio is too high, training is more stable but loses much of the efficiency and regularization benefit. PODS treats the selected data volume as a temporal control signal rather than a fixed cost knob.

How PODS works

PODS stands for Plug-and-play Oscillatory Data-volume Scheduling. It alternates between low- and high-ratio phases during training.

Low-ratio phases expose the model to a smaller amount of selected data. These phases strengthen the regularization effect induced by data selection and reduce computation. High-ratio phases expose the model to more data, improving coverage and helping optimization recover.

The important constraint is that PODS works under the same cumulative data budget as the fixed-ratio baseline. The total amount of data used across training does not exceed the target budget. What changes is when the data is used.

Low-ratio phase -> High-ratio recovery phase -> Low-ratio phase -> High-ratio recovery phase

Why alternate the ratio?

The intuition comes from viewing data selection through optimization. When training on selected data, the stochastic gradient differs from the full-data gradient. This difference can introduce an implicit regularization effect, and PODS shows that the strength of this effect is modulated by the instantaneous selection ratio.

The selected-data objective can be understood as introducing a ratio-dependent implicit regularization effect.

A lower selection ratio leads to stronger selection-induced regularization, which can reduce overfitting and improve generalization. But if the ratio remains too low for too long, the model may lose data coverage and optimization can become unstable.

A higher selection ratio weakens this regularization effect, but improves coverage and makes optimization closer to full-data training. PODS is designed to exploit both sides of this trade-off rather than choosing one fixed point.

Low-ratio and high-ratio phases produce phase-aligned oscillatory training dynamics.

PODS does not change the scoring rule

PODS does not propose a new sample-importance metric. It is orthogonal to selectors based on loss, uncertainty, gradient norm, clustering, coverage, or learned data policies such as Data Agent.

A selector decides what to select. PODS decides how much to select at a given training stage. This makes it a lightweight module that can be placed on top of existing static or dynamic data selection methods.

Reported results

PODS is evaluated across image classification, fine-grained recognition, long-tailed classification, out-of-distribution generalization, object detection, and LLM instruction tuning.

PODS improves the efficiency-generalization trade-off across several existing selection methods.

On ImageNet-1k, PODS reduces training cost while maintaining or improving accuracy, showing that data-volume scheduling can scale beyond small benchmarks. The paper reports 50% ImageNet-1k training-cost reduction with improved accuracy.

PODS also generalizes to more challenging recognition settings. On fine-grained classification, it preserves or improves full-data performance with reduced training costs. On long-tailed recognition, it improves results especially for medium-shot and few-shot categories, suggesting that scheduled data exposure can help models learn more balanced representations.

Reported instruction-tuning cost and downstream benchmark results.

In large-scale fine-tuning experiments with Qwen and LLaMA models, PODS accelerates instruction tuning while maintaining competitive performance on benchmarks such as MMLU and BBH. This suggests that scheduling data volume is useful for both vision and language model training.

What the two studies show

Data Agent and PODS address two separate choices in efficient data-centric training.

Part I: learn what data to select.

Part II: schedule how much data to use.

Both make data selection part of the training loop: Data Agent changes the examples, while PODS changes their volume. The papers report the effects on training cost and performance across the evaluated vision and language tasks.

Toward Efficient Data-Centric Training, Part I: Can Models Learn What Data They Need?

2026-05-13T00:00:00+00:00

Lab note · Part I

Toward Efficient Data-Centric Training, Part I: Can Models Learn What Data They Need?

Not every example is equally useful throughout training. Data Agent learns a selection policy alongside the model it is training.

Suorong Yang

May 13, 2026 · 5 min read · Efficiency

Paper GitHub Part II

Data Agent updates its selection policy as the target model trains.

Why data selection needs to adapt

Data selection is often treated as a cost-reduction tool: use fewer samples, train faster, and preserve performance if possible. That view is useful, but incomplete. The usefulness of a sample is not fixed.

Early in training, a model may need examples that help build broad representations. Later, it may need samples near the decision boundary, uncertain samples, or cases that reveal what the model still misunderstands. A sample that is useful at one stage can become redundant at another.

Many existing methods rely on predefined scoring rules such as loss, gradient norm, clustering distance, or other handcrafted metrics. These rules can be effective in specific settings, but they are often static, snapshot-based, or tied to a particular task. Data Agent reframes the problem: data selection should be an adaptive part of optimization, not only a preprocessing step.

How Data Agent works

Data Agent places a lightweight agent inside the training loop. At each stage, the agent observes the current state of the target model and outputs sample-wise selection weights. The selected data is then used to update the model. After the update, the model’s new state becomes fresh feedback for the agent.

This creates a closed loop:

Model state -> Data Agent -> Selected data -> Model update -> New model state

Instead of using one fixed importance score, Data Agent learns a policy that changes with the target model. The model trains on selected samples; its updated state then provides feedback for the next selection step.

Data Agent shifts selection from heuristic-driven scoring to optimization-driven policy learning.

More concretely, Data Agent formulates dynamic data selection as a sequential decision-making problem. The agent observes feature representations from the target model as its state, outputs continuous selection weights as actions, and prioritizes samples with higher weights during training.

Training signals

A central question is how the agent knows which samples are useful. Data Agent uses two complementary training-aware signals.

Difficulty captures what the model has not learned well. It is based on sample loss: high-loss samples often expose under-learned regions of the data distribution and can have larger optimization impact.

Uncertainty captures where the model is unsure. A sample may be classified correctly but still lie near a decision boundary. Predictive entropy helps identify these samples and encourages the agent to refine regions where confidence remains low.

The relative importance of difficulty and uncertainty changes over training. Data Agent therefore introduces an adaptive weighting mechanism that balances the two signals automatically, rather than relying on a fixed manually tuned reward weight.

Reported results

The paper evaluates Data Agent on image classification, object detection, semantic segmentation, LLM instruction tuning and noisy-data settings, using several model families.

Across large-scale vision and instruction-tuning settings, Data Agent improves efficiency while preserving or improving performance.

On ImageNet-1k, the paper reports substantial training-cost reduction while maintaining or improving accuracy. The method also remains effective across ResNet, ViT, and Swin Transformer backbones. Beyond classification, Data Agent extends to MS-COCO object detection with YOLOv8 and ADE20K semantic segmentation with UperNet.

The method also applies to LLM instruction tuning. With only part of the training data, Data Agent improves performance on benchmarks such as MMLU and AlpacaEval compared with full-data training in the reported experiments.

Data Agent also transfers to dense prediction tasks such as detection and segmentation.

What changes

Data Agent moves selection from preprocessing into the training loop. The reported experiments test which samples to use; Part II asks whether the amount of selected data should also change over time.

δ-mem: Giving Large Language Models Lightweight, Online, and Dynamically Evolving Memory

2026-05-12T00:00:00+00:00

Lab note

δ-mem: Giving Large Language Models Lightweight, Online, and Dynamically Evolving Memory

Long-running assistants and agents need to reuse earlier information without placing their entire history in every new prompt. δ-mem tests a small state that updates during inference.

Jingdi Lei

May 12, 2026 · 7 min read · Efficiency

Paper GitHub

A compact online state supplies low-rank corrections to frozen attention layers.

Why memory is not just longer context

Putting more history into a prompt lets a model see the past, but increases attention cost and does not guarantee that the relevant information will be used. The memory problem also includes deciding what to retain, updating it as new information arrives and retrieving it for the current input.

Standard Transformer attention becomes more expensive as context grows. Long contexts can also disperse attention and make relevant details harder to recover.

δ-mem therefore keeps memory outside the visible prompt and updates it while the model is running.

How δ-mem works

δ-mem stores history in an Online State of Associative Memory beside a frozen full-attention backbone.

When a token or interaction segment arrives, the model projects it into a low-dimensional memory space and updates the state with a delta rule. The update is residual: it depends on the state’s prediction error rather than simply adding each new value.

The state changes online without retraining the backbone. In the main setting it is an 8 × 8 matrix.

During generation, the current input reads a signal from the previous state. That signal becomes low-rank corrections to the query and output sides of attention. Earlier information can therefore affect computation without reappearing as prompt tokens.

Comparison with other memory mechanisms

Textual memory methods such as RAG, MemoryBank and prompt compression return memory to the context as text. δ-mem instead writes directly to a numerical state.

Unlike methods with a separate retriever, reader and fusion path, its state directly produces attention corrections.

LoRA, prefix tuning and model editing usually produce parameter changes that are fixed after training or editing. δ-mem also uses a low-rank interface, but its corrections depend on the current memory state and can change from one history to another.

Three write schedules

The paper studies three write granularities.

Token-State Write (TSW): the state is updated at every token. This captures fine-grained information changes, but it can be sensitive to formatting symbols, repeated expressions, and local noise.
Sequence-State Write (SSW): the hidden states of a message or segment are averaged and written once per segment. This reduces redundant writes and makes state evolution smoother.
Multi-State Write (MSW): multiple parallel memory states are maintained, allowing different states to carry different information types and reducing overwriting within a single state.

Reported results

δ-mem improves memory-heavy and general benchmarks over the frozen Qwen3-4B-Instruct backbone and memory baselines.

On Qwen3-4B-Instruct, TSW raises the reported overall average from 46.79% for the frozen backbone to 51.66%. Its average is 6.76 points above Context2LoRA in the same table.

The gains are especially visible on memory-intensive evaluations. On MemoryAgentBench, δ-mem increases the average score from 29.54% to a maximum of 38.85%. On LoCoMo, MSW reaches the highest score of 49.12%. On HotpotQA, TSW improves EM/F1 from 42.35% / 56.00% to 49.41% / 63.66%.

The no-context test

In no-context recovery, δ-mem can recover part of the task-relevant signal from its compressed online state.

The paper also removes the original historical context and asks the model to answer using only the compressed memory state.

In that setting, δ-mem scores above the no-context baseline on HotpotQA and LoCoMo. On HotpotQA, overall EM rises from 0.08% to 6.48% and F1 from 8.27% to 15.20%. The state recovers part, but not all, of the information lost when the context is removed.

Parameter and memory cost

In the main experiments, SSW and TSW add 4.87M trainable parameters, about 0.12% of the Qwen3-4B backbone. The multi-state MSW version uses 19.47M parameters, about 0.48%.

Reported memory use is close to the base model. Decoding is slightly slower because the state must be updated.

What the experiments show

The results show that a small recurrent state can retain useful information on the evaluated tasks and influence a frozen model through attention corrections. They do not establish unlimited memory or complete recovery: the no-context scores show that substantial information is still lost. Testing other backbones, longer histories and interactive tasks remains open work.