3D Gaussian Splatting as spatial memory for robots (GSMem)

By Turing Post · Wed, 14 Oct 2026 21:16:20 +0000.

This Turing Post guide, part of its week devoted to agentic memory, tackles one question: how can a robot remember a world it can no longer see? According to the newsletter, as AI moves from the screen into the physical world, spatial intelligence becomes essential. The robot is part of its environment: it moves, changes position and interacts with objects. But its camera only captures what is in front of it, so it needs a 3D map of the space that it can update and return to.

The newsletter's TL;DR says 3D Gaussian Splatting reconstructs scenes from multiple views using millions of small 3D primitives. Systems such as GSMem turn those reconstructions into a queryable spatial memory, allowing the robot to revisit, update and reinterpret places it has already observed. Combined with additional physical models, Gaussian representations can also be used for simulation. However, remembering a scene is not the same as understanding how it behaves.

The episode's contents include: why a robot needs 3D memory; what 3DGS is and how it represents the world; how the original 3DGS works; the GSMem and Gaussian-Particle approaches; how splatting becomes Gaussian memory; how the robot can understand and search its memory; whether it can decide what to keep, update and query; and whether Gaussian memory can help robots understand physics.

Why does a robot need 3D memory? The newsletter offers a household scenario: a robot enters a kitchen, sees a mug on the table and turns toward the sink. The mug leaves its field of view, but the robot still needs to know where it is. A short history of frames can help, but the camera sees the world frame by frame, and the mug does not stop existing when the robot turns. To go back for it, the robot must remember where it was and how to get there. A 3D memory links separate observations into a space the robot can return to, and preserves what things look like, where they are and how the surroundings fit together.

The text adds that what matters now may not have mattered a minute ago. A robot may walk past a shelf without noticing a set of keys; if memory only stores recognized objects, those keys would leave no searchable trace. Keeping the captured scene gives it another chance to find them when someone asks, by examining it from another angle within the limits of what was actually observed. Memory also needs to be correctable: small errors in motion estimation accumulate, and recognizing a familiar place helps realign the map and recover the robot's position. The section concludes that seeing something, understanding its relevance and returning to it can happen at three different moments, and memory is what connects them.

What is 3D Gaussian Splatting? The newsletter notes that it was introduced in August 2023 by Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler and George Drettakis, of Inria, Université Côte d'Azur and Max-Planck-Institut für Informatik, in the paper "3D Gaussian Splatting for Real-Time Radiance Field Rendering". It defines three terms: splatting is a rendering technique in 3D graphics that projects 3D data elements onto a 2D screen; a radiance field describes how light and color appear at different positions and viewing directions, and rendering a view from a position not present in the original photos is called novel view synthesis; and 3DGS connects both ideas to turn ordinary photos or videos into a photorealistic digital copy of a space, viewable from new angles in real time.

3DGS represents the scene as many soft, overlapping blobs called Gaussians. Each has a center in space, a shape and an appearance. Its contribution is greatest at the center and fades gradually outward, and it can stretch, flatten and rotate like an ellipsoid to fit different parts of the scene. The properties of each Gaussian, according to the newsletter, are: position (XYZ coordinates of the center), scale (how far it extends along each local axis), rotation (its orientation), opacity (how much it contributes to the image), color/appearance (how it looks from different views) and, optionally, learned features (semantic or task-relevant information added by extensions of standard 3DGS). By combining millions of these small clouds with varying levels of opacity, they form a complete and highly realistic image.

The rest of the guide (how the original 3DGS works, GSMem, Gaussian-Particle approaches, searching and updating the map, and the link to physical simulation) is reserved for Premium subscribers, so the newsletter does not provide those details. In addition, Turing Post mentions a video of the week on the evolution of Yann LeCun's JEPA (H-JEPA, DeepJEPA, RoboJEPA) and links to the previous episode, "How World Models Work – From Dreamer and JEPA to Waymo and Atlas".