Robotic Hand Translates Speech into Sign Language

usermilk writes "Robot educators Keita Matsuo and Hirotsugu Sakai have created a robot hand that translate the spoken word into sign language for the deaf. From the article: 'A microchip in the robot recognizes the 50-character hiragana syllabary and about 10 simple phrases such as "ohayo" (good morning) and sends the information to a central computer, which sends commands to 18 micromotors in the joints of the robotic hand, translating the sound it hears into sign language.'"

How to sign First Post?

Good lord! I imagine the Japanese language [wikipedia.org] with its 1945+ character alphabet is hard enough to learn; learning Japanese sign language must really suck.

You know what would really spoil those deaf kids is, instead of a robot doing sign language, a robot that shows images or words based on what a speaker says. I know, I know; creating a robot to do this is a feat within itself and impressive in its own right, but perhaps there are better ways of communicating with a robot if it can already perform more than adequate speech recognition.

Re:How to sign First Post?

Uhhh, isn't sign language universal? I thought it didn't depend on the spoken language. I might be wrong, of course :).

Re:How to sign First Post?

I believe it's a joke... but just to point out, there's ASL and BSL and...

Re:How to sign First Post?

Unfortunately, no [wikipedia.org] ;-)

Universal Sign Language

Unfortunately, no ;-)
Yup, much the same as how, unfortunately, no one's come up with a universal spoken or written language. Gosh, let alone trying to get a universal programming language...

Re:How to sign First Post?

Good lord! I imagine the Japanese language with its 1945+ character alphabet is hard enough to learn; learning Japanese sign language must really suck.

The relationship between a language & sign language does not work like that.

From the wikipedia sign language page [wikipedia.org]

A common misconception is that sign languages are somehow dependent on oral languages, that is, that they are oral language spelled out in gesture, or that they were invented by hearing people

and

On the whole, deaf sign languages are independent of oral languages and follow their own paths of developmental. For example, British Sign Language and American Sign Language are quite different and mutually unintelligible, even though the hearing people of Britain and America share the same oral language.

You know what would really spoil those deaf kids is, instead of a robot doing sign language, a robot that shows images or words based on what a speaker says.

That doesn't really sound like a robot, but speech recognition software connected to a teleprompter (or monitor)

Re:How to sign First Post?

Japanese has a whole bunch of kanji, but the various words in the language can be formed from a much smaller (hiragana, mentioned in TFA) character set that represents the various syllables in the words. These syllables are always pronounced consistently,

Re:How to sign First Post?

I know nothing about Japanese sign language, and practically nothing about American sign language, but I believe American sign language shares a similarity to written Japanese in that there are signs for common words most any competent signer knows (simila

Scutters!

They only need to put it on wheels and it can become a scutter.

Additional warning:
Do not let this robot pat you on the back whilst near the top of the stairs.

Over Kill?

Call me culturally insensitive but, why not simply translate speech to text?

Text is not the same

Since sign languages are different all over the world, I don't know if there is the same problem in Japan, but:

American Sign Language is not English (American or other).

Thus, translating speech to ASL would reach people that that understand ASL but don't r

Signing, but not reading

After reading a couple of replies to my parent post, I was thinking about people that might understand signing, but not read or hear.

It is my understanding that children can learn to sign before they can learn to read. (In fact hearing children can learn

Re:Over Kill?

Call me culturally insensitive but, why not simply translate speech to text?

Because signing is the native 'tongue' for most deaf people - and it is easier for them to communicate using sign language (over text) - just as its easier for you to understand speech (over text).

Basically - the same reason that some British TV (and undoubtedly many other channels around the world) have a signer translating the news rather then scrolling text.

Visibility?

A good sign language interpreter can read signs from a fair distance, well across a board room at least. How far away can the PDA be before you stop being able to read the text on the screen?

Does it also distinguish...

... Does it also distinguish between different 'dialects' in sign language?
I seem to recall that sign languages differ between countries, same as 'natural' language.

However this is really great for deaf people.

Picture of robotic hand

signing "I'll be back" [collectoybles.com.au]

More Useful As Software

Would this not be more useful as software, able to render simple 3d hands with low microprocessor overheads, and preferably available for mobile phones and PDAs?

Deaf people could carry a PDA, and when they need to find out what someone is saying, they can hold the PDA up like a microphone, and watch the screen, assuming the translation is at least reasonable accurate...
Of course they could lipread too but some find that harder than others, and this could also be used eventually to cross language barriers?

I imagine it's extremely hard to lipread a foreign language.

it's about form factor

Would this not be more useful as software?

Yes but not nearly as intimidating. Who's going to get their lunch money taken -- deaf kid with a PDA, or deaf kid with a giant robot hand?

Amy Pretty

Tickle... Amy... Tickle

http://www.imdb.com/title/tt0112715/ [imdb.com]

I've gotta hand it to these scientists

The robotic hand was shown at a two-legged robot tournament

So it's not just a hand, but a hand with two legs!

This gives a whole new meaning

This gives a whole new meaning to the phrase "talk to the hand"

The article makes no sense

...that can covert spoken words and simple phrases into sign language...

Ignoring the spelling, this implies that it has speech recognition. It converts SPOKEN words.

...recognizes the 50-character hiragana syllabary and about 10 simple phrases such as "ohayo" (good morning)...

Hiragana is the Japanese phonetic alphabet, so it READS. Huh? Now, does it read only 10 simple phrases, or does it read anything plus it recognizes 10 simple audio phrases. I guess the breakthrough is the hand articulation and the idea, not the rest.

Universal Signs

Can you flip off someone in Japanese? Give them the OK sign? Give them a stop with the full palm?

It would be interesting to know how these motions translate, if at all.

Recognziing Sign Language

There's a researcher [gallaudet.edu] at Gallaudet working on the other side of this equation with a system designed to recognize sign langauge, which seems like a much harder problem.

ASL isn't like English in that there are always specific words- a lot of it has to do with spacial context (where in the signing space the sign was made) and a whole class of signs that don't translate directly into words (they're hand shapes which can translate into an event or a description of an object or set of objects).

And, as the research page shows, facial expressions and even facial movements can be part of a sign.

Of course, this is American Sign Language, Japanese Sign Language may be very different.

I bet deaf people hate it

From my observation, much of the "color" or entertainment value in signed conversations comes not from the movement of the hands but the expressions on the face etc. combined with the movements. Clearly, this robot is still nowhere near being capable of the same range of expression as a human being. As a simple test, I'd like to see the robot tell a joke and get the same laughs as a proficient human signer telling the same joke...