Machine learning researchers build a better world model
Machine learning researchers have developed a technique that allows world models to make better predictions about how objects will behave.
Key points
- "We prove that the fraction of the disturbance's effect that this rule gets wrong shrinks with the square of the time step, and that no method using the same ordinary data can do better than this rate in the worst case." Counterfactual world models have three likely uses, according to Balestriero.
- "When an attempt fails because of something the robot could not see, such as an unexpected load, a current or a weak motor, the counterfactual world model lets it estimate which action would have worked in that same situation," he explained.
- Balestriero said that the counterfactual world model experiments were done using off-the-shelf encoders (DINOv2 and MAE) without any fine-tuning and only a small predictor trained on top.
The technique that the researchers developed is not specific to JEPAs – it can be applied to any world model that predicts the average next state under any action.
Additional details
- "The world model predicted where the boat would end up after steering left, and it actually ended up two meters further downstream," he explained.
- But a JEPA world model produces an average of outcomes and would not be able to anticipate counterfactual scenarios – events that didn't happen.
- Creating a counterfactual world model from that starting point would only require minor additions.
- These models could provide better direction to robots if they were able to make inferences about possible actions without being trained for all the possible outcomes.
- To answer 'where would the boat be had I steered right?', we take the world model's prediction for steering right and add the same two meters.
How this summary was made: extractive summary — built from the sources' own reporting. It is drawn only from the reporting of the sources listed below — read the originals for full detail. How we summarize →
Advertisement
Every article behind this summary — 1 article from 1 outlet
Get stories like this in your inbox.
Subscribe — it's freeOne free email at 6am — your topics, deduplicated, five minutes. No account, no paywall.