A team from the OmniAI group at Zhejiang University has released the EmbodiedMemory-Bench (EMem-Bench) benchmark. This tool is designed to test the ability of embodied artificial intelligence agents to form memory based on long interaction histories and subsequently act based on that memory. The article about this was posted on arXiv about three weeks ago and attracted attention from Chinese media last weekend. On the Hugging Face platform, the dataset has received over 32,000 downloads as of October 18th.
The problem that the benchmark solves is well-known from the experience of domestic robots: an agent sees a book on a coffee table, performs other tasks, and then, when asked to bring it, starts searching in all rooms or continues pulling a drawer that it has already found closed.
The team manually analyzed 100 failed trajectories obtained from Gemini-3-Flash and Qwen3-VL-32B models on the EB-ALF and EB-Hab datasets from EmbodiedBench. It was found that 35% of failures were related to the lack of subtle visual cues, 31% to the inability to record states identified during interaction, 18% to ignoring changes in the environment, and 7% to not utilizing past experience.
EMem-Bench transforms these gaps into four families of tasks: Passive Observation, Dynamic Tracking, Interaction Failure, and Experience Generalization. Each of the 2,554 executable episodes contains a time-ordered multimodal history followed by a new task. The agent must find the corresponding evidence and then navigate, open containers, pick up, and place objects in the simulation until the target state is reached. Scenes from AI2-THOR, which take place in one room, are combined into multi-room houses, and distracting but executable interactions are inserted between the key cue and the task. The benchmark covers 1,118 scenes and has undergone automated testing and human review.
Under full context conditions, the Gemini-3-Flash model demonstrated the best result with an average success rate of 64.2%, followed by Gemini-2.5-Pro with 60.8%, Qwen3.6-27B with 58.2%, and GPT-5.4 with 56.8%. The article notes that all open-source models, except one, showed a result below 45%.
The team also developed an external memory system called Embodied-Memorizer (EMem), which segments experience into scene, spatial, and event memories. Using GPT-5.4-mini as a base, EMem increased the average success rate from 35.3% to 58.9%, which is 21.1 percentage points higher than the next best memory system, TeleMem. The EMem-8B model, fine-tuned on policy, achieved 46.0%, which is 20.6 percentage points higher than its base model Qwen3-VL-8B. Simply increasing the number of history images did not help: success dropped from 36.6% to 26.2%, while the number of input tokens per episode increased from 41,770 to 206,489.
The authors emphasize that all results obtained are in simulation and require further testing on real robots facing sensor noise and execution errors.