Become a fan of Slashdot on Facebook

 



Forgot your password?
typodupeerror
Transportation Security Software Hardware Technology

You Can Trick Self-Driving Cars By Defacing Street Signs (bleepingcomputer.com) 272

An anonymous reader quotes a report from Bleeping Computer: A team of eight researchers has discovered that by altering street signs, an adversary could confuse self-driving cars and cause their machine-learning systems to misclassify signs and take wrong decisions, potentially putting the lives of passengers in danger. The idea behind this research is that an attacker could (1) print an entirely new poster and overlay it over an existing sign, or (2) attach smaller stickers on a legitimate sign in order to fool the self-driving car into thinking it's looking at another type of street sign. While scenario (1) will trick even human observers and there's little chance of stopping it, scenario (2) looks like an ordinary street sign defacement and will likely affect only self-driving vehicles. Experiments showed that simple stickers posted on top of a Stop sign fooled a self-driving car's machine learning system into misclassifying it as a Speed Limit 45 sign from 67% to 100% of all cases. Similarly, gray graffiti stickers on a Right Turn sign tricked the self-driving car into thinking it was looking at a Stop sign. Researchers say that authorities can fight such potential threats to self-driving car passengers by using an anti-stick material for street signs. In addition, car vendors should also take into account contextual information for their machine learning systems. For example, there's no reason to have a certain sign on certain roads (Stop sign on an interstate highway).
This discussion has been archived. No new comments can be posted.

You Can Trick Self-Driving Cars By Defacing Street Signs

Comments Filter:
  • Easy (Score:2, Flamebait)

    You set up snipers in strategic locations across town to cover every and all traffic sign; and you shoot the fucker who dares get even close to it.

    • Yeah, in other shocking news, removing stop signs and shooting out stop lights can cause accidents!

      • Yeah, in other shocking news, removing stop signs and shooting out stop lights can cause accidents!

        But what if the stop light draws first?

        • by epyT-R ( 613989 )

          Then you can label the traffic light a gun nut and ignore it.

        • Yeah, in other shocking news, removing stop signs and shooting out stop lights can cause accidents!

          But what if the stop light draws first?

          George Lucas will make new ones that don't.

      • Misleading title (Score:5, Informative)

        by Dutch Gun ( 899105 ) on Tuesday August 08, 2017 @08:05PM (#54970381)

        A better title would be, "Researchers fool Google's TensorFlow library in laboratory tests".

        As it turns out, they did NOT test this against actual self-driving vehicle image recognition, but a generic deep neural network library. This seemed obvious, as there are still no commercially available fully autonomous vehicles, but I skimmed the paper to confirm it.

        There was another issue I noticed as well. They resized all their training images down to 32x32 pixels. I admit I'm no expert in neural networks, but this seems like it would greatly favor the ability to fool classification algorithms. Maybe someone more knowledgeable can correct me if I'm off base here. Still, my suspicion seems to be confirmed by this little gem:

        "Our final classifier accuracy was 91% on the test dataset."

        So, their baseline algorithm only worked properly slightly better than 9/10 times. Should we believe that this represents the state of the art that will be applied in actual self-driving vehicles? It seems like the researchers didn't even have a highly robust classifier from the start.

        I believe the merits of the paper lie in demonstrating this as a theoretical concern, but this should in no way be construed to represent a definitive threat against actual vehicle systems. You can't necessarily blame the researchers for the crappy headline, of course, as the title is "Robust Physical-World Attacks on Machine Learning Models". But I wouldn't necessarily rate this as the most robust research I've ever seen either.

        • Re:Misleading title (Score:4, Interesting)

          by ShanghaiBill ( 739463 ) on Tuesday August 08, 2017 @10:03PM (#54971139)

          I believe the merits of the paper lie in demonstrating this as a theoretical concern

          But that is important, because without this research, the teams of professional engineers designing SDCs would have never even considered that a traffic sign could be smudged or obscured by a tree branch.

    • Re:Easy (Score:5, Funny)

      by PopeRatzo ( 965947 ) on Tuesday August 08, 2017 @07:06PM (#54969881) Journal

      You set up snipers in strategic locations across town to cover every and all traffic sign; and you shoot the fucker who dares get even close to it.

      Here in Texas, they just shoot the traffic sign and skip the middleman. Because freedom has to be irrigated by the blood of patriotic drivers in self-driving cars. Or something. I don't remember the exact quote, but it's in the Second Amendment or the Bible, I'm pretty sure.

      https://s-media-cache-ak0.pini... [pinimg.com]

      • Here in Texas, they just shoot the traffic sign and skip the middleman.

        Way back in 1982 or so I drove from Tampa to Miami and went across the southern, west to east segment of Highway 75 [wikipedia.org] in Florida -- I think it's known as Alligator Alley [wikipedia.org]. Anyway, the road is basically straight all the way across Florida. Along the way were signs that said "Unlawful to discharge firearms within 1/2 mile of road". The signs had all been shot several times.

        • Re:Easy (Score:4, Informative)

          by 93 Escort Wagon ( 326346 ) on Tuesday August 08, 2017 @08:19PM (#54970511)

          My wife and I drove from Seattle to Anchorage back in the late 1980s - her sister had gotten married, and we went up to meet her husband and his family. Not long after we crossed from Canada into Alaska, we started noticing that pretty much every road sign had been shot multiple times. It got worse, the further into Alaska we travelled. Along the stretch of highway that heads down the peninsula towards Anchorage, many of the signs had so many bullet holes that they were unreadable.

          After meeting my (now ex-) brother-in-law and his friends, I ceased to be surprised at the state of the road signs - instead, I wondered why none of them had thought of destroying the signposts using automatic weapons.

    • by Luthair ( 847766 )

      I pointed it out on yours, but both cover a topic we looked at 5-months ago [slashdot.org].

      I'm not sure why bleepingcomputers is posted at all - every single story is submitted by an anonymous user, pretty hard to accept its not someone from the site spamming Slashdot.

  • Better solution (Score:5, Insightful)

    by cunina ( 986893 ) on Tuesday August 08, 2017 @06:59PM (#54969827)
    Why not just have a geospatial database of signs that self-driving cars access? Then it won't matter what's on the sign, or if the sign even physically exists. Why is anti-stick coating the solution that "researchers" suggest?
    • Your idea is awesome.

      On a practical basis however, it sounds like a cluster-fuck. Just think about that for a little while.

    • by arth1 ( 260657 )

      Why not just have a geospatial database of signs that self-driving cars access? Then it won't matter what's on the sign, or if the sign even physically exists. Why is anti-stick coating the solution that "researchers" suggest?

      For one thing, there's a need for temporary signs.
      And the sign has to physically exist for everything that isn't a self-driving car.

      • by Qzukk ( 229616 )

        And the sign has to physically exist for everything that isn't a self-driving car

        And the humans driving won't be fooled at all by fake signs put up by trolls (mostly because humans ignore most of the signs anyway).

      • And, as a human driver, you always look at every sign that you see every day? You don't ever become blind to the stop sign at the end of the street you live on? Or perhaps you do use an internal database for the vast majority of your travels?

        Self-driving cars will synthesize situation awareness from many sources including their previous experiences and the experiences of all the other vehicles on the road contributing to the database.

        The physical stop sign won't rule. When it becomes obscured by the bush gr

        • Self-driving cars will synthesize situation awareness from many sources including their previous experiences and the experiences of all the other vehicles on the road contributing to the database.

          Self-driving cars will need to synthesize situation awareness from many sources including their previous experiences at least as well as a reasonably good human driver, but I doubt the current algorithms are anywhere close, and getting them there is going to take a lot of time and money.

      • Why not just have a geospatial database of signs that self-driving cars access? Then it won't matter what's on the sign, or if the sign even physically exists. Why is anti-stick coating the solution that "researchers" suggest?

        For one thing, there's a need for temporary signs.
        And the sign has to physically exist for everything that isn't a self-driving car.

        This.

        You have to be really carefully how you design this. The self-driving car that refuses to see a stop sign on an interstate is going to absolutely love construction zones.

    • And how frequently will this database be updated? And how frequently are downloads of the updates required? Have you noticed that work crews generally erect signs during the day? And that drivers are generally required to follow the sign once it is in place? That means that the database must be updated as soon as the workers declare the sign to have been installed and all cars must download the update immediately because more than one crew could exist in a city and cars are everywhere.

      Or perhaps the downloa

    • So every sign will have to be accurate and up to date with the database, at all times, across the entire country? Further, you'll have two masters now. What should a car do if/when it encounters a conflict? Should it stop and hand back control, use the database and ignore all signs, or use the signs as posted? All options are messy, other than making sure HAL is as good as a human at reading damaged and defaced signs.

      Once you ask these dumb things to navigate back roads, or poorly maintained hellscapes

      • ...Basically this self driving frenzy is likely to go the way of the VR hype. It will be awesome tech that only a few will shell out money for, and even fewer will make use of.

        I hope you're right, but I think at some point it's going to be mandated. Government, especially in the US, is rapidly accelerating both the degree and the granularity of the control it has over its own citizens. And since Joe Average is a sucker for the 'because safety' and 'because security' BS arguments, once the tech is mature and reliable, self-driving cars will be embraced with open arms. Then law enforcement, along with the rest of the authoritarian power-trippers, will have their most compelling wet

    • There is no way that would work in my city. They can't even keep their own gis map up to date, no way a map like that would be current.
    • my 1st thought years ago was pranking cars by jumping out in front of them. Crazy to risk it; however, when it becomes predictably safe...

      Next thought was some radio nerds experimenting with broadcasting signals towards cars.

    • by mikael ( 484 )

      Roadworks are always putting up crazy signs, traffic cones, and all sorts of obstructions:

      http://www.inspirational-quote... [inspiratio...-stuff.com]

      https://s-media-cache-ak0.pini... [pinimg.com]

    • by AmiMoJo ( 196126 )

      That would be a great idea if the data were available.

      Local government authorities know where most of their signs are, and could provide updates when things change. Mapping companies would love to get hold of that data stream, but it's damn near impossible. The local government authorities want them to pay for the data, and they all negotiate separately. Even if they agree, there is no legal requirement for the data to be accurate or timely so at best you might notice they suck and sue them for breech of co

  • But, the edge cases will become increasingly troublesome as they move from prototype into widespread use

    Road signs are commonly missing, rotated, shot, stolen or defaced

    I love the idea of autonomous vehicles. I wrote autonomous vehicle software for a major auto manufacturer. This shit is hard

    • Road signs are commonly missing,

      I feel like a missing stop sign is a problem regardless if your brain is squishy or silicon. In fact there is an unmarked 4 way stop near my office. There is a crash there about once every 2-3 months.

      • by arth1 ( 260657 )

        In fact there is an unmarked 4 way stop near my office. There is a crash there about once every 2-3 months.

        If it's unmarked, it's not a 4-way stop. No marking means "yield to the right". Too many people have become accustomed to all intersections being marked to remember the basic rules.

        • by hawguy ( 1600213 )

          In fact there is an unmarked 4 way stop near my office. There is a crash there about once every 2-3 months.

          If it's unmarked, it's not a 4-way stop. No marking means "yield to the right". Too many people have become accustomed to all intersections being marked to remember the basic rules.

          It's not just "yield to the right", it's yield to oncoming traffic, yield to the car that gets there first, and then (maybe) yield to the right.

          Some states (like Arizona) treat an uncontrolled intersection as a 4 way stop, which is the only sensible thing to do.

          • by arth1 ( 260657 )

            It's not just "yield to the right", it's yield to oncoming traffic

            No, it's not. Oncoming traffic won't cross your path unless they turn left, in which case they have you on their right, and must yield.

            yield to the car that gets there first

            At least imprecise. If a car has entered the intersection and cannot reasonably be expected to stop before entering your projected path, you have to yield to it, but for a different reason - you're not allowed to cause an accident by intent or negligence. But that doesn't mean the other driver hasn't broken the rules by not yielding to you.

            Some states (like Arizona) treat an uncontrolled intersection as a 4 way stop, which is the only sensible thing to do.

            Many countries have mainly unmar

    • Road signs are commonly missing, rotated, shot, stolen or defaced

      Or, like around here, just plain wrong because it costs money to change them and the government doesn't have the cash.

  • It just occurred to me today: will self-driving cars be smart enough to pull over for cops and fire trucks? If so, does that mean all you have to do to get them out of your way is flash some lights for a bit?
    • AFIAK some emergency vehicles already have equipment to switch traffic lights. So it's not much of a jump they can signal autonomous vehicles to pull over.
      • That switch is a pulsing light that triggers the traffic signal to change the cycle in the same manner as a pedestrian push button or in ground sensor loop might. It just forces the priority of the change so that the normal green-to-yellow-to-red change starts now instead of a bit later. They are not exactly difficult to fake out. Putting them on every car on the road would be a terrible idea, or not. As long as I am in my manual operated vehicle, having one of those would be quite enjoyable at times. "Hey

    • If you honk with the right timing, the car will even send bitcoins to your wallet.

    • by hackel ( 10452 )

      Once we've eliminated idiot human drivers, cars will no longer need to pull over for emergency vehicles, unless it's a single-lane road. They will simply talk to each other and coordinate priority access. They will be able to clear a space in real-time, like a bubble surrounding the emergency vehicle.

      In the meantime, if emergency vehicles aren't already transmitting some kind of signal that can be picked up by autonomous vehicles, that would make me worried. They certainly shouldn't be relying on visual

    • by Qzukk ( 229616 )

      If so, does that mean all you have to do to get them out of your way is flash some lights for a bit?

      Works on human drivers too. I guess that means humans aren't ready to be driving yet.

  • by Cerlyn ( 202990 ) on Tuesday August 08, 2017 @07:08PM (#54969903)

    "...there's no reason to have a certain sign on certain roads (Stop sign on an interstate highway)."

    What about here? [goo.gl] (Cross Island Parkway, New York USA, Exit 31)

    Stop signs often do appear on highway entry ramps, especially where they are short. This is true in construction areas, as well as on some older entrance ramps around New York City.

    Technically this is a 50 MPH (~80 km/h) Parkway and not an Interstate, but rather than randomly searching the area this was the first that came to mind.

    • Can someone please tell me what I'm supposed to do at this intersection?

      Seeing this sign it looks like it wants me to take the onramp and then come to a stop and give way before going? But looking at the road markings it appears to be a giveway-merge where I just match the speed of the traffic and then join in.

      Forget driversless cars, let's start by making it clear for drivers first. I mean went back and forward along that road, none of the cars appeared to stop for anything.

      • by Cerlyn ( 202990 )

        There is not enough aligned ramp space there for you to see if there is any oncoming traffic on the main road before you have to collide with it. Extending the ramp would require expanding the bridge at its terminus.

        So they expect you to stop, look backwards for a gap, and accelerate hard to get into it. Which can get quite tricky at hours when there is a lot of traffic.

        At other intersections you may have to quickly get on before the on-/off-ramp gets you off again, or slow down prior to sharp curves on

  • Octagon? (Score:5, Insightful)

    by im_thatoneguy ( 819432 ) on Tuesday August 08, 2017 @07:08PM (#54969909)

    What horrifically terrible machine learning algorithm sees a red octagon and thinks it's a black and white rectangular speed limit sign? How is the visual machine learning matrix so bad that a triangular yellow sign would be registered as a stop sign?

    Do they not train the machine learning algorithms with color images? Considering you can rely on 1-2 seconds of latency for a sign there is no reason to use the same sort of low latency machine learning algorithms used for pedestrian identification or road lines.

    • >"What horrifically terrible machine learning algorithm sees a red octagon and thinks it's a black and white rectangular speed limit sign?"

      +1 THANK YOU!
      I was wondering the same thing. I mean, I know visual AI is complicated, but it is a FREAKING RED OCTAGON!!! What freaking chance does freaking self-driving technology have if it can't freaking deal with something that freaking simple???

      Freak!!

    • The current state of "AI" is pretty damn crude, and the resulting "trained" system cannot be debugged as such. As best I can tell it is akin to shoving in data, desired behavior, and pressing "optimize". So while we would like to think there is thought and reasoning going on, there is not. If the algorithm has been poorly designed or trained there is no telling how it will react to data that is dissimilar to the training data. Graffiti is pretty random, which is easy to figure out for a human, apparentl

    • by AmiMoJo ( 196126 )

      The machine probably discards colour information, so that it can work at night when colour is either not available or inaccurate.

  • For example, there's no reason to have a certain sign on certain roads (Stop sign on an interstate highway).

    Except when you do, like when there's construction or accidents, and a guy stands there with a stop sign.

  • I can see it now: a company puts up a billboard with a red octagon containing their brand of motor oil, and the car gets thirsty.
    It has begun!

  • by hawguy ( 1600213 ) on Tuesday August 08, 2017 @07:28PM (#54970051)

    For example, there's no reason to have a certain sign on certain roads (Stop sign on an interstate highway).

    Except during road construction when a signman holds up a "stop" sign and the self-driving car says "You're not fooling me! There are no stop signs on freeways, and even your 15mph speed limit sign is fake, my database says the speed limit here is 75mph. See ya!"

    • And they will the do that 55 on I-294 when all others are doing 75+.

    • by hackel ( 10452 )

      In these situations, obviously the workers would need to be using a transmitter to broadcast updated road speed information in a standardised format to all vehicles. Isn't this just common sense?

      • by hawguy ( 1600213 )

        In these situations, obviously the workers would need to be using a transmitter to broadcast updated road speed information in a standardised format to all vehicles. Isn't this just common sense?

        If you're going to standardize every construction site in america and give them transmitters that every car listens to, why not just put long distance RFID tags on every street sign and avoid the need to use faulty image recognition in the first place? Just because humans need to use vision to read signs doesn't mean cars should.

        They could even be cryptographically signed with the sign's meaning and location/direction to prevent a prankster from moving a 70mph freeway speed limit sign to a residential stre

      • I laughed out loud at this one. People keep mowing down the markers at a construction site near me and the workers can't even be bothered to set them back up. No way in hell are they going to be fiddling with a device putting these markers into a virtual database.
  • by pubwvj ( 1045960 ) on Tuesday August 08, 2017 @07:31PM (#54970081)

    You can trick humans by defacing street signs... So... What else is new? This is a "no-duh!"

    • by epyT-R ( 613989 )

      It's a LOT harder to trick a human than it is to trick a computer.

      • by Qzukk ( 229616 )

        Yeah, the first step is to get the human to look at the sign in the first place.

      • Eh... computers will learn the tricks, future generations of machines tend to become immune to the tricks the first generation fell for, heck it is very quick and easy to educate an entire generation of systems to the specific trick that the first generation fell for. Meanwhile there are still new generation humans, vulnerable to the "nigerian prince" exploit.
  • by DatbeDank ( 4580343 ) on Tuesday August 08, 2017 @07:33PM (#54970101)

    Instead of a car making horrific errors in judgment, why not have it safely pull over and say, "I'm lost, please ask for directions."

    Better yet, set it up so the female voice pulls over and asks for help and the male voice just keeps going until it thinks it reached the destination.

    • by epyT-R ( 613989 )

      Better yet, just remove the damn computer and let the human drive it himself.

    • Better yet, set it up so the female voice pulls over and asks for help and the male voice just keeps going until it thinks it reached the destination.

      No, the male voice would keep driving around in circles while insisting it wasn't lost.

  • I like how this is written like it is a surprise. Did people really think that autonomous vehicles actually thought about the signs?
  • by WindBourne ( 631190 ) on Tuesday August 08, 2017 @07:41PM (#54970169) Journal
    Seriously, these machines should be using signs to augment mapping info.
    In addition, the feds should come up with a SINGULAR approach on how to put up secured temporary local notifications.
    Perhaps a digital form of NOTAMs.
  • Why are self-driving cars reading signs in the first place? Seriously, don't we have all of this information available digitally? It makes no sense for them to even be attempting to read the signs. If the car needs to travel into an area where we don't have digital information available, it should require manual control. This is just silly.

  • Researchers say that authorities can fight such potential threats to self-driving car passengers by using an anti-stick material for street signs

    Spend tons of money covering signs with sticker-proof material and you are again defeated by spray paint and stencils. Or by magnetic graffiti! This is not the most efficient way of thinking to remedy this problem.

  • Have a very attractive lady(s) walk on the side of the road. I guarantee there will eventually be a smashup. Most men are suckers that way. I've had multiple close calls due to such "distractions". Plus, it's not illegal to arrange such, unlike sign tampering.

    Hmmm, let's see if bot-cars are distracted by R2D2 in lingerie.

  • AI is stupid.

    News at 11.

  • Signs vary widely between countries.

    Here in New Zealand a stop sign is alway accompanied by a yellow line and the word "STOP" painted on the road at the intersection. Give Way signs are either unmarked or have white lines with a triangle on the road.

    I assume that means if the sign is damaged, you always know the difference between a stop sign controlled intersection and a regular give way intersection.

  • Every new Tesla car (including Model 3) has the full "Hardware 2" platform for self-driving, and even when it's not being used for self-driving it's on and watching the world. Tesla has said that it is already using "fleet learning" to map out roads. This blog post is talking about how radar has problems but is still useful for self-driving, and they are working around the problems:

    When the car is approaching an overhead highway road sign positioned on a rise in the road or a bridge where the road dips un

  • "For example, there's no reason to have a certain sign on certain roads (Stop sign on an interstate highway)."

    I can think of at least two places on I-15 which have a stop sign directly on the interstate, and one on I-40.

  • Give me a stencil and some paint, and I can trick YOU by defacing street signs.

    The only difference here is that idiots don't need the stencils.

    To quote a famous idiot, FAKE NEWS.

  • On the kind of mapped terrain where self-driving cars currently mix with manual traffic, most of the information on traffic signs can be coded into the cars' database, such as speed limits on each stretch of road and the location of no-passing zones and crosswalks. Self-drivers must be able to recognize sudden and temporary control changes, such as for construction, weather damage, and police operations. If someone tries to spoof signs in one of these areas or do something like cover up a Stop sign with a p

fortune: cpu time/usefulness ratio too high -- core dumped.

Working...