MemGUI-RL

01 / Overview

Where the GRPO recipe breaks on policy-managed memory

The verifier scores a ConAct response on four components (format, action type, action parameters, folding range); the folds it scores are mostly single-step. Both facts get in the way of group-relative policy optimisation.

GRPO applied naively to ConAct, and what FARPO changes — (a) GRPO applied naively to ConAct; (b) the two changes FARPO makes.

Observation 1 Heterogeneous rewards need an explicit comparison scale.

Aggregating the components before group normalisation gives each one an effective weight proportional to its within-group spread. The format component is saturated (group mean ≈ 0.999) while the folding component is not (0.33 at initialisation), so the weights written into the objective are not the weights the update obeys.

Observation 2 Memory compression needs controlled exposure.

A span fold compresses several past steps and is the operation that shrinks the context; span folds are 22.7% of the annotated folds, and the majority step folds are almost always already correct. Under natural sampling the share of prompt groups that still carry a learning signal falls from about 70% to 23–38% within one epoch, and the policy collapses to step-only folding (deep-fold rate on held-out states 17.7% → 0.2%).

FARPO Two mechanisms, one per observation.

Within a prompt group, each component is standardised separately (as in GDPO) and only then weighted, which makes the advantage group-centred and scale-invariant. Across prompts, every span fold is kept and ⌈N_span/ρ⌉ step folds are subsampled, so span exposure is one explicit ratio ρ; the released model uses ρ = 9 (deep-fold rate 92.5%).

02 / Results

MemGUI-Bench and MobileWorld

MemGUI-Bench: 128 long-horizon tasks, three attempts, Pass@k and information-retention rate (IRR). MobileWorld GUI-only: 117 tasks in a different environment and application set, one attempt, no MobileWorld data in training.

From the same start

Policy	P@1	P@3	IRR	MobileWorld
Qwen3-VL-8B-Instruct (backbone)	9.4	20.3	15.1	11 / 117 · 9.4%
MemGUI-8B-SFT (RL start)	23.4	35.9	30.2	21 / 117 · 17.9%
GRPO recipe (scalar advantage, natural corpus)	23.4	39.1	35.7	18 / 117 · 15.4%
+ reward decoupling	29.7	45.3	38.5	20 / 117 · 17.1%
+ folding-aware sampling, ρ = 9 (= FARPO, MemGUI-8B-RL)	30.5	48.4	39.0	23 / 117 · 19.7%
Qwen3-VL-235B-Thinking (reported)	24.2	46.9	30.0	14.5%

MemGUI-Bench columns in %, 128 tasks; MobileWorld GUI-only solved tasks and success rate. Every row is one training run evaluated once.

Pass@3 of MemGUI-8B-SFT and MemGUI-8B-RL by application count and task type — Pass@3 of the SFT and RL policies by the number of applications a task spans and by task type. The 115 memory-intensive tasks carry the whole gain (31.3 → 45.2%); the 13 standard tasks are unchanged.

2.4× / 2.6× the backbone. Pass@3 20.3 → 48.4 and IRR 15.1 → 39.0 for the complete SFT + FARPO recipe; MobileWorld success doubles (9.4 → 19.7%). MemGUI-8B-RL is above every end-to-end model except the 235B ConAct agent.
Gains where memory matters. By golden-step difficulty, +29.2 Pass@3 on Easy and +9.6 on Medium tasks over SFT; on the Easy tier the 8B policy matches the zero-shot 235B ConAct agent (68.8%). On the Hard tier (41+ golden steps) SFT and RL are within two tasks of each other: only 12% of the RL prompts lie beyond step 40 of their trajectory, and the dominant failure there is not what the fold verifier scores. By application count, +10.7 / +14.3 / +14.7 on one-, two- and three-application tasks.
Folding exposure is bimodal. Across seven ratios the trained policy folds spans almost always or almost never, and ρ decides which; ρ = 9 is best on the development set and among the two best ratios on MobileWorld.

03 / Method

The ConAct loop and the FARPO training step

Training is offline: a prompt is an annotated state of MemGUI-3K, the policy samples G = 5 responses, and an annotation-based verifier scores each one. No device rollouts are needed during training; task success is measured afterwards on the benchmarks.

Overview of the FARPO pipeline — **One FARPO iteration.** Folding-aware sampling keeps every span fold and subsamples step folds, so span exposure is one ratio ρ. Per prompt, the rollout policy samples G ConAct responses that the factorised verifier scores on four components. FARPO standardises each component within the group before weighting, whitens the advantages over the batch and updates the policy with dual-clipped PPO and a KL penalty to the SFT policy.

Training dynamics of the stepwise ablation

Reward-decoupled group advantages

Each verifier component is centred and scaled within its prompt group, gated when its group variance is zero, and only then weighted (w = 0.1 / 0.4 / 0.4 / 0.1 for format, type, parameters, folding). A one-standard-deviation change in any component moves the advantage by the same amount, which the scalar recipe does not guarantee. Added to the GRPO recipe on the same natural corpus, it gives +6.2 Pass@3 and +2.8 IRR, and memory-argument match on the held-out states rises stepwise along the ladder (63.1 → 72.3 → 74.7 → 75.9).

Deep-fold rate over training and the development sweep over the ratio rho

Folding-aware sampling

Span folds are rare and step folds are already solved by the SFT policy, so under natural sampling the share of prompt groups with zero advantage grows from 25–33% to 62–77% over an epoch; with ρ = 9 it stays at 13–15%. The deep-fold rate on held-out states shows the two modes: natural corpora drive it to 0.2%, folding-aware corpora above 90%. At deployment MemGUI-8B-RL folds about 22 steps per span (SFT: 11) and writes 1.69 memory entries per attempt (SFT: 1.26).

04 / Case study

One task, five policies

MemGUI-Bench task 091: find the price and star rating of two cameras on Amazon, look up the founding year and founder of each maker on Wikipedia, and send all eight facts in one message. First attempts of the five policies of the ablation; the bars show the folded action history at the end of each run, with memory writes and repeated-action loops marked.

Case study on MemGUI-Bench task 091: five policies — Qwen3-VL-8B-Instruct alternates between "now I need the rating" and "now I need the price" for 35 scrolls, keeps no memory and sends no message. MemGUI-8B-SFT stores all eight facts but folds one step at a time and runs out of budget at step 65; the GRPO recipe spends 50 of 81 actions scrolling and stores four facts. Reward decoupling without folding-aware sampling collects all eight facts into five memory entries but returns them as an answer instead of sending the message. MemGUI-8B-RL closes each sub-task with a span fold, stores the facts as four memory entries, ends its 80 steps with a two-line history and sends the message.

05 / Release

Code, model, data and evaluation logs

All artefacts are released under anonymous accounts for the review period. MemGUI-8B-SFT, MemGUI-3K and the ConAct interface come from the MemGUI-Agent release; MemGUI-Bench and MobileWorld are used with their public harnesses.

CodeFARPO trainer on top of veRL: reward-decoupled estimator, folding-aware sampler, ConAct verifier, and the launch scripts of every reported run. ModelMemGUI-8B-RL, the FARPO (ρ = 9) checkpoint after 100 steps from MemGUI-8B-SFT; bf16 safetensors with tokenizer and processor. Training dataThe annotated MemGUI-3K states in the RL prompt format (73,807 training and 8,296 held-out states), with the split used in the paper. Evaluation logsTrajectories and judge outputs of every MemGUI-Bench and MobileWorld session behind the paper's rows: a compact tier per session and the complete tier with all screenshots and model requests.