Talking to Your Self-Driving Car: LLMs Could Personalize Autonomous Driving

The idea of handing the wheel to an autonomous system still makes many people uneasy. But what if you could simply tell your car to drive more calmly or to hurry up, just as you might ask a human driver? Research from TU Delft in the Netherlands is exploring exactly that, aiming to make autonomous vehicles more adaptable to passenger preferences through natural language.

In a recent study, researchers integrated a large language model (LLM) with a motion planning controller. The model interprets casual requests like ‘I’m late, let’s move faster’ and translates them into adjustments of certain driving parameters—such as speed, smoothness, and turning agility—without jeopardizing safety. This is achieved by having the LLM rate the importance of various criteria the controller uses to evaluate potential trajectories, effectively tweaking the car’s behavior within safe bounds set by the engineers.

What sets this approach apart is the human-in-the-loop element. Before any changes are applied, the system tells the passenger in plain language what adjustments it plans to make, and asks for confirmation. Users can then approve or further refine the settings interactively. This not only prevents misunderstandings but also keeps control in the passenger’s hands, much like directing a taxi driver.

Simulation tests in the nuPlan environment showed that the system correctly modified driving behavior for prompts ranging from ‘I feel carsick’ to ‘I’m in a hurry.’ While the integration of LLMs into vehicle control faces challenges such as response time and safety guarantees, isolating the language model from the core planner helps ensure that even if it errs, nothing unsafe happens. This research points toward a future where autonomous cars aren’t just smart, but also personal.