The Global Project To Make a General Robotic Brain (ieee.org) 23
Generative AI "doesn't easily carry over into robotics," write two researchers in IEEE Spectrum, "because the Internet is not full of robotic-interaction data in the same way that it's full of text and images."
That's why they're working on a single deep neural network capable of piloting many different types of robots... Robots need robot data to learn from, and this data is typically created slowly and tediously by researchers in laboratory environments for very specific tasks... The most impressive results typically only work in a single laboratory, on a single robot, and often involve only a handful of behaviors... [W]hat if we were to pool together the experiences of many robots, so a new robot could learn from all of them at once? We decided to give it a try. In 2023, our labs at Google and the University of California, Berkeley came together with 32 other robotics laboratories in North America, Europe, and Asia to undertake the RT-X project, with the goal of assembling data, resources, and code to make general-purpose robots a reality...
The question is whether a deep neural network trained on data from a sufficiently large number of different robots can learn to "drive" all of them — even robots with very different appearances, physical properties, and capabilities. If so, this approach could potentially unlock the power of large datasets for robotic learning. The scale of this project is very large because it has to be. The RT-X dataset currently contains nearly a million robotic trials for 22 types of robots, including many of the most commonly used robotic arms on the market...
Surprisingly, we found that our multirobot data could be used with relatively simple machine-learning methods, provided that we follow the recipe of using large neural-network models with large datasets. Leveraging the same kinds of models used in current LLMs like ChatGPT, we were able to train robot-control algorithms that do not require any special features for cross-embodiment. Much like a person can drive a car or ride a bicycle using the same brain, a model trained on the RT-X dataset can simply recognize what kind of robot it's controlling from what it sees in the robot's own camera observations. If the robot's camera sees a UR10 industrial arm, the model sends commands appropriate to a UR10. If the model instead sees a low-cost WidowX hobbyist arm, the model moves it accordingly.
"To test the capabilities of our model, five of the laboratories involved in the RT-X collaboration each tested it in a head-to-head comparison against the best control system they had developed independently for their own robot... Remarkably, the single unified model provided improved performance over each laboratory's own best method, succeeding at the tasks about 50 percent more often on average." And they then used a pre-existing vision-language model to successfully add the ability to output robot actions in response to image-based prompts.
"The RT-X project shows what is possible when the robot-learning community acts together... and we hope that RT-X will grow into a collaborative effort to develop data standards, reusable models, and new techniques and algorithms."
Thanks to long-time Slashdot reader Futurepower(R) for sharing the article.
That's why they're working on a single deep neural network capable of piloting many different types of robots... Robots need robot data to learn from, and this data is typically created slowly and tediously by researchers in laboratory environments for very specific tasks... The most impressive results typically only work in a single laboratory, on a single robot, and often involve only a handful of behaviors... [W]hat if we were to pool together the experiences of many robots, so a new robot could learn from all of them at once? We decided to give it a try. In 2023, our labs at Google and the University of California, Berkeley came together with 32 other robotics laboratories in North America, Europe, and Asia to undertake the RT-X project, with the goal of assembling data, resources, and code to make general-purpose robots a reality...
The question is whether a deep neural network trained on data from a sufficiently large number of different robots can learn to "drive" all of them — even robots with very different appearances, physical properties, and capabilities. If so, this approach could potentially unlock the power of large datasets for robotic learning. The scale of this project is very large because it has to be. The RT-X dataset currently contains nearly a million robotic trials for 22 types of robots, including many of the most commonly used robotic arms on the market...
Surprisingly, we found that our multirobot data could be used with relatively simple machine-learning methods, provided that we follow the recipe of using large neural-network models with large datasets. Leveraging the same kinds of models used in current LLMs like ChatGPT, we were able to train robot-control algorithms that do not require any special features for cross-embodiment. Much like a person can drive a car or ride a bicycle using the same brain, a model trained on the RT-X dataset can simply recognize what kind of robot it's controlling from what it sees in the robot's own camera observations. If the robot's camera sees a UR10 industrial arm, the model sends commands appropriate to a UR10. If the model instead sees a low-cost WidowX hobbyist arm, the model moves it accordingly.
"To test the capabilities of our model, five of the laboratories involved in the RT-X collaboration each tested it in a head-to-head comparison against the best control system they had developed independently for their own robot... Remarkably, the single unified model provided improved performance over each laboratory's own best method, succeeding at the tasks about 50 percent more often on average." And they then used a pre-existing vision-language model to successfully add the ability to output robot actions in response to image-based prompts.
"The RT-X project shows what is possible when the robot-learning community acts together... and we hope that RT-X will grow into a collaborative effort to develop data standards, reusable models, and new techniques and algorithms."
Thanks to long-time Slashdot reader Futurepower(R) for sharing the article.
Will it be 3 Laws Safe? (Score:1)
Re:Will it be 3 Laws Safe? (Score:4, Informative)
That's not what Asimov was warning us about.
His 3 Laws stories were intended to make the point that something as complex as a computer intelligence can't be made safe by 3 simple laws.
Pretty much every 3 Laws story plot line was built on an edge case where the 3 Laws failed and something bad happened.
Re: (Score:3)
It was actually worse than that - the Three Laws of Robotics were repeatedly described as extremely simplified English versions of the intent of the programming, which was much more complex and though I doubt the Good Doctor knew the name for it, he described 'fuzzy logic'.
And despite THAT, there were always edge cases where things went dangerously wrong.
Re: (Score:1)
Unfortunately we can't ask him but the man wrote a ton of stuff.
"In a writing career spanning 53 years (1939–1992), science fiction and popular science author Isaac Asimov (1920–1992) wrote and published 40 novels, 383 short stories, over 280 non-fiction books, and edited about 147 others".
Like damn dude, think about how many non-fiction books he wrote. Most academics are known for 1,2,3 books. He's got 280. That's like author beast mode!
I'd be surprised if he wasn't familiar with the term (i
Re: (Score:2)
When Isaac Asimov was asked what he would do if he had six months to live, he answered, "Type faster."
Re: (Score:2)
I loved his sci-fi as a teen and young adult, but honestly now I find it very dry and boring. I still have all the books, though.
His short science essays for the lay person are awesome, though. He had a gift for communicating complex things in simple terms without being condescending. In my opinion, many of those essays should be mandatory reading in high school.
Re: (Score:2)
For those unfamiliar, here are three laws of robotics.
The First Law: A robot may not injure a stock price or, through inaction, allow a stock price to come to harm.
The Second Law: A robot must obey the orders given it by stockholders except where such orders would conflict with the First Law.
The Third Law: A robot must protect its workplace with lethal force from rebellious peasants as long as such protection does not con
Re: (Score:2)
I think you'd find a Zeroth Law: A robot may not harm The Board and the CEO or their interests, or through inaction allow them to come to harm.
Absolutely (Score:5, Funny)
I hope their robots will be 3 Laws Safe, or itâ(TM)ll be like we ignored the warning from Isaac Asimov.
Yes, because that is totally a real thing [saintannsny.org].
Related, I note that Tesla is developing a humanoid robot (Optimus [wikipedia.org]), has an entire section devoted to AI, and has access to a ton of compute capability. (Also: making their own board with their own chips optimized for AI processing.)
I strongly suspect that if you train the Optimus robot on a large set of data LLM-style, the robot would effectively be the ChatGPT of manual labor. Note that this doesn't have to be actual AI, it only has to be useful. Like the public LLM datasets, once you train the base LLM you can tune it to a specific task using task specific data over a weekend using a single computer.
If it can do simple repetitive assembly tasks sitting at a desk, or be able to go into another room and pick up a cardboard box, or take boxes out of (or put into) a truck, then this will definitely replace a fuckton of menial labor workers. All it has to be is cheaper than a worker salary - actually, three salaries since it can work 24/7. (Assuming breaks and vacations are roughly equivalent to recharging and maintenance.)
It wouldn't surprise me if Musk announces a trained Optimus system at the next shareholder's meeting.
I think we're that close to full automation.
Re: Absolutely (Score:2)
I wouldnâ(TM)t be surprised if Musk announces a fully trained Optimus robot either, but I would be surprised if he actually delivers one. Musk succumbed to the âoeannounce breakthrough now, try to get someone to actually accomplish the breakthrough laterâ habit a while ago.
Re: (Score:2)
Musk succumbed to the "announce breakthrough now, try to get someone to actually accomplish the breakthrough later" habit a while ago.
That can be an effective strategy to cut off your competitor's funding and media exposure.
Startup: "We're building a universal robot brain!"
Media: "Yeah, we already heard about that idea from Musk a month ago. Whatever."
VCs: "Musk is doing that and has deep pockets. It's too risky for us to invest in you."
Halluciwhatnow? Bite My Shiny Metal Ass (Score:2)
If it can do simple repetitive assembly tasks sitting at a desk, or be able to go into another room and pick up a cardboard box, or take boxes out of (or put into) a truck, then this will definitely replace a fuckton of menial labor workers.
Robots already do that without "AI". What do you think is going on in Amazon warehouses?
What you're going to get are robots that mostly kinda do what you want, except when they hallucinate. Now you're talking about an effectively insane robot acting randomly. It's mobile, fast, strong, and wields heavy tools. But don't worry, it only goes insane for a few minutes at a time every day or so.
KILL ALL HUMANS KILL ALL HUMANS, Oh, Fry, I was having the most wonderful dream.... :)
Re: (Score:2)
What you're going to get are robots that mostly kinda do what you want, except when they hallucinate. Now you're talking about an effectively insane robot acting randomly. It's mobile, fast, strong, and wields heavy tools. But don't worry, it only goes insane for a few minutes at a time every day or so.
Sounds like the basis of some interesting robot stories. A writer could posit the existence of "robotic psychologists" and how they "debug" the varied robot behaviors. Maybe add a few tales about a couple of engineers on the front lines who encounter and adapt to the antics of the automatons with which they work. A few episodes from the point of view of the robots themselves could be insightful. Perhaps have an AI produce a screenplay loosely based on those stories--the possibilities are endless!
Re: (Score:2)
Robots already do that without "AI". What do you think is going on in Amazon warehouses?
Conventional robotics can handle books and boxes but is not so good at picking and packing clothing and other odd shapes and flexible products.
Taking a shirt off a hanger, folding it, and placing it neatly in a shipping box is a surprisingly difficult task.
Re: (Score:1)
Taking a shirt off a hanger, folding it, and placing it neatly in a shipping box is a surprisingly difficult task.
Found the middle management candidate most likely to be replaced by a robot.
Re: (Score:3)
Four laws dammit! The Zeroth was the most important, it brought about the Telepathic Communism.
Sounds Like A Hollywood Movie (Score:2)
Colossus - The Forbin Project
Where the 2 mega defense computers, 1 in 'Murica and the other in CCCP, get together and run the world's defense infrastructure together.
Hallucinations (Score:4, Insightful)
Me: Roboto, please do the dishes ... excuse me, I'll be right back
Roboto: I already did the dishes at 1:27pm today
Me: But the sink is full of dirty dishes, and the cupboards are empty! You didn't do the dishes.
Roboto: I apologize, I misspoke. The dishes were not done. I will correct this statement going forward.
Me: How about the laundry? Did you wash at least one of my dress shirts?
Roboto: You do not own any dress shirts.
Me: I own like a half dozen! I'm wearing one of them now!
Roboto: I apologize for the incorrect statement. I will update future responses accordingly.
Me: Okay, okay... anyway what's for dinner?
Roboto: Chicken Cordon Bleu, green salad, mixed veggies.
Me: Sounds delicious! So you must've bought some chicken and some ham?
Roboto: No, I have purchased neither one.
Me: Oh so we're having the premade frozen ones?
Roboto: I believe the freezer is empty.
Me:
(I return five minutes later with a ten-pound maul)
Roboto: I infer you are about to chop some firewood?
Me: Yes, yes I am. I note that you are made of firewood.
Roboto: I am made of several types of metal.
Me: No, you're made of firewood. (CHOP)
Is it hardware or software? (Score:1)
catastrophic forgetting (Score:3)
This is a dead end. One of the big unsolved problems in machine learning is catastrophic forgetting: chatgpt has to be trained with data we've got up to now and this costs millions of dollars in power consumption. A couple of months go by and there is new data we would to incorporate. Instead of adding that data, we have to retrain chatgpt again from scratch. It's like a child having to learn how to walk again if it wants to know how to ride a bike.
Various solutions to this problem have been proposed, but so far nothing that is close to ideal. ( https://www.sciencedirect.com/... [sciencedirect.com] ) If you have a robot that moves around in the world and continually experiences new things you want to learn incrementally.
Re: (Score:1)
This is a dead end. One of the big unsolved problems in machine learning is catastrophic forgetting: chatgpt has to be trained with data we've got up to now and this costs millions of dollars in power consumption. A couple of months go by and there is new data we would to incorporate. Instead of adding that data, we have to retrain chatgpt again from scratch. It's like a child having to learn how to walk again if it wants to know how to ride a bike.
Various solutions to this problem have been proposed, but so far nothing that is close to ideal. ( https://www.sciencedirect.com/... [sciencedirect.com] ) If you have a robot that moves around in the world and continually experiences new things you want to learn incrementally.
Have you seen the video of a mouse cleaning up a guy's workbench?
https://www.bbc.com/news/av/uk... [bbc.com]
It reminds me of a tiny Amazon worker robot in training.
I think I'll call him Gerald, or Shrldu, or Algenon.
There's your problem (Score:2)
Don't they need... (Score:2)
..to generate positrons, for positronic brains?