Microsoft Tests ChatGPT's Ability to Control Robots (microsoft.com) 35
"We extended the capabilities of ChatGPT to robotics," brags a blog post from Microsoft's Autonomous Systems and Robotics research group, "and controlled multiple platforms such as robot arms, drones, and home assistant robots intuitively with language."
They're exploring how to use ChatGPT to "make natural human-robot interactions possible... to see if ChatGPT can think beyond text, and reason about the physical world to help with robotics tasks." We want to help people interact with robots more easily, without needing to learn complex programming languages or details about robotic systems. The key challenge here is teaching ChatGPT how to solve problems considering the laws of physics, the context of the operating environment, and how the robot's physical actions can change the state of the world.
It turns out that ChatGPT can do a lot by itself, but it still needs some help. Our technical paper describes a series of design principles that can be used to guide language models towards solving robotics tasks. These include, and are not limited to, special prompting structures, high-level APIs, and human feedback via text.... In our work we show multiple examples of ChatGPT solving robotics puzzles, along with complex robot deployments in the manipulation, aerial, and navigation domains....
We gave ChatGPT access to functions that control a real drone, and it proved to be an extremely intuitive language-based interface between the non-technical user and the robot. ChatGPT asked clarification questions when the user's instructions were ambiguous, and wrote complex code structures for the drone such as a zig-zag pattern to visually inspect shelves. It even figured out how to take a selfie! We also used ChatGPT in a simulated industrial inspection scenario with the Microsoft AirSim simulator. The model was able to effectively parse the user's high-level intent and geometrical cues to control the drone accurately....
We are excited to release these technologies with the aim of bringing robotics to the reach of a wider audience. We believe that language-based robotics control will be fundamental to bring robotics out of science labs, and into the hands of everyday users.
That said, we do emphasize that the outputs from ChatGPT are not meant to be deployed directly on robots without careful analysis. We encourage users to harness the power of simulations in order to evaluate these algorithms before potential real life deployments, and to always take the necessary safety precautions. Our work represents only a small fraction of what is possible within the intersection of large language models operating in the robotics space, and we hope to inspire much of the work to come.tics to the reach of a wider audience. We believe that language-based robotics control will be fundamental to bring robotics out of science labs, and into the hands of everyday users.
ZDNet points out that Google Research and Alphabet's Everyday Robots "have also worked on similar robotics challenges using a large language models called PaLM, or Pathways Language Model, which helped a robot to process open-ended prompts and respond in reasonable ways."
They're exploring how to use ChatGPT to "make natural human-robot interactions possible... to see if ChatGPT can think beyond text, and reason about the physical world to help with robotics tasks." We want to help people interact with robots more easily, without needing to learn complex programming languages or details about robotic systems. The key challenge here is teaching ChatGPT how to solve problems considering the laws of physics, the context of the operating environment, and how the robot's physical actions can change the state of the world.
It turns out that ChatGPT can do a lot by itself, but it still needs some help. Our technical paper describes a series of design principles that can be used to guide language models towards solving robotics tasks. These include, and are not limited to, special prompting structures, high-level APIs, and human feedback via text.... In our work we show multiple examples of ChatGPT solving robotics puzzles, along with complex robot deployments in the manipulation, aerial, and navigation domains....
We gave ChatGPT access to functions that control a real drone, and it proved to be an extremely intuitive language-based interface between the non-technical user and the robot. ChatGPT asked clarification questions when the user's instructions were ambiguous, and wrote complex code structures for the drone such as a zig-zag pattern to visually inspect shelves. It even figured out how to take a selfie! We also used ChatGPT in a simulated industrial inspection scenario with the Microsoft AirSim simulator. The model was able to effectively parse the user's high-level intent and geometrical cues to control the drone accurately....
We are excited to release these technologies with the aim of bringing robotics to the reach of a wider audience. We believe that language-based robotics control will be fundamental to bring robotics out of science labs, and into the hands of everyday users.
That said, we do emphasize that the outputs from ChatGPT are not meant to be deployed directly on robots without careful analysis. We encourage users to harness the power of simulations in order to evaluate these algorithms before potential real life deployments, and to always take the necessary safety precautions. Our work represents only a small fraction of what is possible within the intersection of large language models operating in the robotics space, and we hope to inspire much of the work to come.tics to the reach of a wider audience. We believe that language-based robotics control will be fundamental to bring robotics out of science labs, and into the hands of everyday users.
ZDNet points out that Google Research and Alphabet's Everyday Robots "have also worked on similar robotics challenges using a large language models called PaLM, or Pathways Language Model, which helped a robot to process open-ended prompts and respond in reasonable ways."
*dont show Robocop or terminator to chatgpt* (Score:5, Funny)
Just saying.
Re: (Score:3)
Blue screen of death? (Score:3)
I hope it will not take it literally...
Re: (Score:2)
I hope it will not take it literally...
Note, robot dog end effectors may not update as expected.
Re: Blue screen of death? (Score:1)
Re: (Score:2)
Or the Bill Gatus of Borg meme
list of games to teach it / have it play (Score:2)
falken's maze
black jack
gin rummy
hearts
bridge
checkers
chess
poker
fighter combat
guerrilla engagement
desert warfare
air-to-ground actions
theaterwide tactical warfare
theaterwide biotoxic and chemical warfare
global thermonuclear war
Re: (Score:2)
PUBG :-)
Re: (Score:1)
Speech is not intelligence (Score:5, Insightful)
Neuropsychologists know this ... that the ability to generate plausible speech does not equal intelligence or competence.
Language is very superficially convincing though. It's easy to be fooled by it, unless you are very skeptical and probe for actual understanding.
It's gonna be awhile before I want a jumped up autocorrect to be operating a robot anywhere around me ...
Re:Speech is not intelligence (Score:5, Funny)
Exactly. Just watch any US talk show if you're not convinced.
Re: (Score:2)
While I totally agree with your first sentence, after than I think you need to pause and reflect.
Once the learning that ChatGPT engages in isn't limited to language, then we don't KNOW how well it will do. I've got opinions, but that's all they are. I think we need to solve the presentation of fantasy as reality before it's reasonable to hook it up to a robot. But perhaps I'm wrong. Perhaps it needs to encounter reality to know the difference between fantasy and fact.
I expect that this is an extremely i
Re: (Score:2)
Re: (Score:3)
Once the learning that ChatGPT engages in isn't limited to language, then we don't KNOW how well it will do.
This is easy. Any situation where the AI can operate by interpolating what has previously been observed, it will perform well. Because that's what these neural networks do.
Any situation where the AI needs an understanding of what it is doing, it will perform poorly. Because it has no underlying concept of what it is doing, it just selects words (or motions, or whatever).
Re: (Score:2)
Re: Speech is not intelligence (Score:2)
Re: (Score:1)
Re: (Score:3, Informative)
Speech is not intelligence. Neuropsychologists know this... that the ability to generate plausible speech does not equal intelligence or competence.
Citation needed. I only did one semester of neuroscience and computational neuroscience at college, so I'm hardly an expert, but I didn't pick up from it anything even near a hint of what you're saying.
"How can I know what I think till I see what I say?" (Auden).
What I DID pick up from my neuroscience studies, though, is that the human mind does its decision-making unconsciously within tens or hundreds of milliseconds, and the conscious part of our mind only follows along later, and also has a good ability
Re: (Score:2)
Speech is not intelligence. Neuropsychologists know this... that the ability to generate plausible speech does not equal intelligence or competence.
Citation needed.
"The ability to speak does not make you intelligent." -- Qui Gon Jinn (Star Wars Episode 1: The Phantom Menace)
Re: (Score:2)
A technology that can convert speech input into a list of basic tasks to get something done + the ability to use cameras and other sensors to recognize objects and their position + the ability to control motors to change the position of things in the physical world = the next BIG thing. This is bigger than the internet. Rosie the robot big. Do my laundry, make my dinner from scratch, clean my house big. Assemble mobile phones and all other consumer items,
"think" and "reason" (Score:2)
Re: (Score:2)
Lets give a psychotic artificial Idiot extra-strength opposable thumbs, what can possibly go wrong??
Re: (Score:2)
I guess some researchers don't know what these words mean. ChatGPT is perfectly happy to tell you that 2 + 3 = 6.
Given some "complaints" about recent elections, I'm guessing also: s/researchers/voters/
Sounds vaguely familiar (Score:2)
The key challenge here is teaching ChatGPT how to solve problems considering the laws of physics, the context of the operating environment, and how the robot's physical actions can change the state of the world.
Isn't this pretty close to Asimov's Three Laws? Might as well incorporate them now, in the deveopment stage. With luck, this will prevent us from having robot overlords.
Oh boy (Score:4, Funny)
bsg (Score:2)
Just don't ask the robot... (Score:2)
... about its "feelings". Who knows how it'll react.
Terrible code (Score:2)
I've seen this plot (Score:2)
Computer: "What is it that you desire?"
Me: "Peace on Earth and good will to all."
Computer: "Yes sir!" [wikipedia.org]
Re: (Score:1)