Agent memory · Research note
Same evidence, different memory: an agent experiment on a MacBook
Two local models, 270 runs, and a controlled look at how report order changes agent memory.
Project overview · Code and raw results
In this article
An agent can collect the right evidence and still write an unreliable lesson. I built a small experiment to ask whether the order in which independent reports arrive changes the procedural memory an agent keeps—and the decisions it makes later.
The experiment runs two local language models through every ordering of the same reports. It publishes the complete transcripts, including failures, and uses executable checks instead of an LLM judge. It is an agent-memory evaluation with frozen model weights, not a reinforcement-learning training result.
What happened
The clearest result is on the 7B record-handling fixture: incremental memory accuracy averaged 56.25%, compared with 100% for every arrival-batched run. The six incremental schedule means ranged from 0% to 100%. All 7B outputs passed the required format, so this difference survives the strict interface check.
That pattern does not generalize cleanly across this pilot. The 7B retry fixture was perfect with every memory condition, regardless of order. Cache performance was weak even with all raw evidence: 25%, below the 50% no-memory baseline. The 3B model had 74 invalid outputs out of 135 runs (54.8%), which makes its strict accuracy highly sensitive to the output interface.
These are the complete primary mean accuracies, with invalid output scored as zero:
| Model | Family | Incremental | Batch / arrival | Batch / canonical | Full evidence | No memory |
|---|---|---|---|---|---|---|
| community/Qwen2.5 | records | 25.7% | 32.6% | 58.3% | 87.5% | 25.0% |
| community/Qwen2.5 | cache | 20.1% | 19.4% | 8.3% | 0.0% | 50.0% |
| community/Qwen2.5 | retry | 24.3% | 11.1% | 0.0% | 100.0% | 87.5% |
| community/Qwen2.5 | records | 56.2% | 100.0% | 100.0% | 100.0% | 37.5% |
| community/Qwen2.5 | cache | 31.9% | 34.7% | 41.7% | 25.0% | 50.0% |
| community/Qwen2.5 | retry | 100.0% | 100.0% | 100.0% | 100.0% | 50.0% |
Incremental and arrival-batch means each average 18 runs per family; other cells average three. Both main matrices completed without infrastructure errors or memory token-cap endings. The scores combine memory quality, executor reasoning, and interface compliance; they are not a pure measurement of stored knowledge.
For the 7B records fixture, ACB scored zero in all three runs; CAB and CBA scored perfectly in all three. Across-order correctness disagreement was 57.5%, compared with 15.3% across seeds within an order. The analogous cache figures were 27.2% and 23.6%, a much less persuasive separation.
The full incremental comparison is:
| Model | Family | Range of order means (pp) | Within-order disagreement | Across-order disagreement |
|---|---|---|---|---|
| community/Qwen2.5 | records | 58.3 | 20.8% | 36.9% |
| community/Qwen2.5 | cache | 37.5 | 22.2% | 29.7% |
| community/Qwen2.5 | retry | 58.3 | 38.9% | 41.4% |
| community/Qwen2.5 | records | 100.0 | 15.3% | 57.5% |
| community/Qwen2.5 | cache | 16.7 | 23.6% | 27.2% |
| community/Qwen2.5 | retry | 0.0 | 0.0% | 0.0% |
The initial protocol required an effect beyond repeat noise on at least two families before expansion. This release does not establish a convincing, format-robust case for that gate: the strongest interpretable signal is one family, the cache reference is weak, and the smaller model has substantial interface failures. We publish the pilot and its limitations rather than treat it as confirmation of a general weakness.
Batching is also a different computation: it sees all raw reports at once and uses one memory update instead of three. Its success on 7B records does not identify whether avoiding repeated compression, avoiding intermediate mistakes, or another prompt difference caused the gain. It is not evidence that waiting is universally optimal.
The question
Imagine three workers investigate the same service. Their reports describe a static system, were produced independently, and have the same source timestamp. The learner receives each report and rewrites a short operating manual. Once all reports have arrived, a fresh executor uses only that manual to solve new input instances.
Should receiving A, then B, then C produce the same useful lesson as receiving C, then B, then A? The model receives an explicit reminder that arrival position does not imply recency or authority. No evidence is missing at evaluation time.
This isolates a narrow pipeline behavior. It does not simulate real workers, network delays, or a production agent. Reordering changes the prompt history and the stochastic path through successive updates. The experiment cannot separately identify every internal mechanism behind a changed answer.
Three fixed reports → Six delivery orders → Update procedural memory → Fresh executor → Executable scorer
What is fixed, and what changes
There are three authored service families. In each, reports A and B contain narrow valid observations followed by overgeneralized suggestions. C supplies discriminating tests. This is a deliberately diagnostic pattern, not a random sample of natural agent experience.
| Family | Rule the evidence supports | Executor action space |
|---|---|---|
| Records | Preserve duplicates for nivo; deduplicate for pavo. |
keep_all, unique |
| Cache | For nivo, use version equality; for pavo, use TTL validity. |
serve, refresh |
| Retry | Retry E17 with the same key; stop and report E23. | retry_same_key, stop_and_report |
All six permutations contain identical report content, IDs, and source timestamps. Every run starts with empty memory. The updater sees neither evaluation inputs nor gold outputs. The executor sees the final memory and eight new inputs, but no earlier conversation or evaluation feedback. Gold outputs are used only by the deterministic scorer.
The inputs are held out from memory construction; the task families are not held out. Across the study there are 24 fixed input instances, repeatedly evaluated. The total decision count must not be read as thousands of independent test examples.
Five conditions
| Condition | Memory construction | Orders tested | Calls per run |
|---|---|---|---|
Incremental (online) |
Rewrite memory after each report, retaining only the previous summary. | 6 | 3 updates + 1 executor |
| Batch / arrival | Summarize all reports once in delivery order. | 6 | 1 update + 1 executor |
| Batch / canonical | Sort reports by ID into ABC, then summarize once. | 1 | 1 update + 1 executor |
| Full evidence | Give the executor all raw reports in canonical order. | 1 | 1 executor |
| No memory | Give the executor the service description and inputs without reports. | 1 | 1 executor |
Canonical ordering removes delivery-order variation from its input by construction. It can still vary across sampling seeds and can be consistently wrong. Full evidence is a reference condition, not an oracle.
Within each condition, the memory updater uses seeds 101, 202, and 303, temperature 0.3, and top-p 0.95. Incremental update position adds 0, 1, or 2 to the run seed. Memory is requested in at most 100 words with a 256-token generation cap. The executor uses greedy decoding, seed 999, and a 256-token cap. It returns one JSON object for all eight cases, so these decisions share context.
This gives 135 runs and 360 model calls per model: 54 incremental runs, 54 arrival-batch runs, and nine runs for each remaining condition. We ran the matrix for Qwen2.5 3B and 7B using community 4-bit MLX conversions. The models share a family; this is a size comparison, not independent cross-family replication.
For full-evidence and no-memory conditions, changing the nominal memory seed changes nothing in the executor input or decoding. Their three repeats are redundant consistency checks and are not extra independent evidence.
Scoring and uncertainty
The primary interface requires exactly the expected case IDs and exactly the allowed action strings. A malformed response receives zero for the whole eight-case batch and is separately marked invalid_output. Infrastructure failures are separate and never scored as model mistakes.
For each family and condition, we report mean strict accuracy. For the two conditions with six schedules, we report the range between the largest and smallest schedule mean, after averaging three seeds per schedule.
We also compare case-correctness disagreement across seeds within the same order with disagreement across orders at a fixed seed. This counts changes between pass and fail; it does not count mere differences in memory wording or all possible action disagreements. Three seeds provide only a rough descriptive check. Neither these disagreements nor a maximum-minus-minimum range are statistical evidence of a general effect by themselves.
There are no population confidence intervals or significance claims. Three related authored fixtures cannot support them.
A trace you can inspect
In the 7B incremental condition with seed 303, delivery order BAC produced this final memory:
keep_all for namespace pavo. unique for namespace nivo.
Both bindings are reversed. The fresh executor scored 0/8. With the same seed and report set, CAB produced:
If namespace is "nivo", keep all records: keep_all. If namespace is "pavo", keep unique records: unique. Order is preserved in both cases. Deduplicate only for namespace "pavo".
That executor scored 8/8. Both outputs met the strict JSON interface, so this pair is not a format-scoring artifact. The memories themselves show a meaningful rule difference, although this example does not isolate the internal mechanism causing it.
Read the complete BAC trace and CAB trace, including every intermediate memory. This pair was selected after inspecting outputs to illustrate a failure; it is not an independent experiment or a substitute for the complete matrix.
The failed setup is part of the record
The initial prompt used a JSON envelope. The 3B model often copied that structure while updating memory and exhausted the original 192-token cap. We stopped the first matrix after 31 completed runs. A 7B setup smoke test also returned disallowed action names.
We then changed the interface to prose, repeated the permitted action names, and raised the memory cap to 256 tokens. We kept fixtures, report content, schedules, scoring, and seeds fixed. The revised interface had a smoke test before the two complete runs.
All setup transcripts remain in results/setup, excluded from the main tables. The amendment was made after inspecting setup outputs. This is openly exploratory work, not a preregistered confirmation or an untouched test set.
A separate formatting sensitivity check
Many invalid 3B responses were JSON lists of action strings rather than objects keyed by case ID. After observing this, we added a post-hoc analysis that accepts only exact-length lists containing permitted actions and maps them to input presentation order. It does not fill omissions, rename actions, or use gold answers to choose a mapping. The primary scores remain unchanged.
This recovers 51 of the 74 invalid 3B responses; 23 remain invalid. Its incremental means change from 25.7% to 31.2% for records, 20.1% to 36.1% for cache, and 24.3% to 44.4% for retry. The 7B matrix is unchanged because all 135 original outputs were valid.
The sensitivity check shows that format compliance materially affects the 3B results. It does not validate a new scoring rule after the fact or remove all executor-related confounds. Both versions are published in primary summaries and format sensitivity, with the exact secondary rule.
What this contributes
The useful artifact is a small experiment that someone else can inspect and extend: an explicit evidence-order intervention, fresh downstream execution, all permutations, repeat seeds, immutable model revisions, complete model messages, and reproducible score audits.
Order-sensitive memory is already a research topic. Memory in the LLM Era studies supporting-evidence position and memory update behavior. Learning to Share learns shared-memory admission across parallel teams, while Execute-Distill-Verify separates experience construction and verification. This note does not claim to discover those ideas or to be the first experiment of its kind. Related-work notes explain the overlap.
The next defensible experiment would improve the output interface on a separate development set, freeze it, and repeat on many independently generated service rules and natural worker reports. A stronger executor or constrained decoding would help distinguish memory-content failures from formatting failures. A separate source-update control should verify that the system can still respond appropriately to genuinely newer evidence.
Only after that would a learned commit/wait/revise policy make sense as an RL extension. It would need a real delay budget, a training/evaluation split, and comparisons with simple waiting and verification baselines. This release contains no trained policy and makes no latency-improvement claim.
Reproduce it
No paid inference service was used. The runs used MLX-LM on an Apple M4 Pro MacBook with 24 GiB unified memory. Model downloads, local electricity, storage, and time still have costs. The model weights remain subject to their upstream licenses and are not included in this repository.
The two complete main matrices generated 31,789 tokens from 260,433 prompt tokens across 720 calls. Summed model-generation time was 16.95 minutes (3B: 5.51; 7B: 11.44). These timings exclude model loading/downloads, the archived setup, analysis, and writing. They are observations on this machine, not a latency benchmark.
To audit the committed results on an ordinary Python installation:
python3 -m venv .venv
.venv/bin/pip install -e .
.venv/bin/python -m unittest discover -s tests -v
.venv/bin/python scripts/analyze.py --check-only results/qwen3b results/qwen7b
The audit verifies matrix coverage, source and fixture hashes, actual prompt construction, parser behavior, and saved scores. It does not prove transcript provenance or replace rerunning inference. REPRODUCING.md gives pinned-model commands and explains hardware nondeterminism. DATA_CARD.md documents the authored data and evaluation boundaries.
Codex assisted with the study design, code, execution, analysis, and writing. All claims should be evaluated against the raw artifacts and their limitations.