Bots Are Better Than Humans At Cracking 'Are You a Robot?' Captcha Tests, Study Finds (independent.co.uk) 78
A recent comprehensive study reveals that automated bots are substantially more efficient than humans at cracking Captcha tests, a widely used security measure on over 100 popular websites. The Independent reports: In the study, scientists assessed 200 of the most popular websites and found 120 still used Captcha. They took the help of 1,000 participants online from diverse backgrounds -- varying in location, age, sex and educational level -- to take 10 captcha tests on these sites and gauge their difficulty levels. Researchers found many bots described in scientific journals could beat humans at these tests in both speed and accuracy.
Some Captcha tests took human participants between nine and 15 seconds to solve, with an accuracy of about 50 to 84 per cent, while it took the bots less than a second to crack them, with up to near perfection. "The bots' accuracy ranges from 85-100 per cent, with the majority above 96 per cent. This substantially exceeds the human accuracy range we observed (50-85 per cent)," scientists wrote in the study. They also found that the bots' solving times are "significantly lower" or nearly the same as humans in almost all cases.
Some Captcha tests took human participants between nine and 15 seconds to solve, with an accuracy of about 50 to 84 per cent, while it took the bots less than a second to crack them, with up to near perfection. "The bots' accuracy ranges from 85-100 per cent, with the majority above 96 per cent. This substantially exceeds the human accuracy range we observed (50-85 per cent)," scientists wrote in the study. They also found that the bots' solving times are "significantly lower" or nearly the same as humans in almost all cases.
AI lies better than humans (Score:2)
Ha! We've finally found a use case for LLM weak AI systems.
Not surprising (Score:5, Funny)
Have you seen the type of humans wandering around of late? I believe a chipmunk could do better at these tests than humans.
Re:Not surprising (Score:2)
Everything changed with LLMs. Intelligence is more than language tests, but if tested with just language, an LLM scores above genius level IQ. It's in the top 0.1% and will get better.
https://www.scientificamerican... [scientificamerican.com]
You don't have to qualify with "type of humans". You and I and just about everyone here will lose to an LLM on these tests.
Re:Not surprising (Score:2)
Well, captcha's aren't usually language tests, but yeah, it's not surprising that they can do better than people when trained on that kind of test. It *is* a bit surprising that publicly available tools can do that, however.
Re:Not surprising (Score:2)
Oh man, you cracked me up on this one. LOL
Re:Not surprising (Score:2)
I believe a chipmunk could do better at these tests than humans.
Jokes on them really, because these humans can hold more food in their mouths.
Mining? (Score:0)
If the spammers want to spam a million people, the costs could add up. They may use other people's computers for the spamming but the mining might make the owners more likely to notice their stuff is pwned.
That was called hashcash (Score:4, Informative)
Addition or alternative to captcha make the client mine some crypto or something?
That was called hashcash [wikipedia.org], and use for email is what inspired the invention of cryptocurrency. Use of proof of work was found to unfairly advantage desktop users with a recent GPU over users of desktop and laptop computers with integrated graphics or (eventually) smartphones.
By design (Score:5, Insightful)
Considering so many captchas are being used to feed ML datasets, it's very not surprising AI trained on those datasets can now solve them effortlessly...
Re:By design (Score:5, Funny)
If you solve it quickly and correctly, you are a bot. If you struggle and fail, come right in!
Re:By design (Score:3)
You joke, but you know how this will go now... you'll *have* to spend at least 15 seconds looking at the damn thing before it'll ever consider you a human. Guess what'll happen next... yep, some ads will appear alongside - just to keep you entertained while you wait.
For this next step of enshitification, I for one can't wait - well, I can, because I'll have to ;-)
Re: By design (Score:4, Interesting)
Thatâ(TM)s literally how reCaptcha and hCaptcha work. The âoechoose all pictures containing traffic lightsâ is just you providing data for them to train their ML model. The actual captcha involves looking at how long it takes you to choose them, along with various other metrics.
Re:By design (Score:2)
You can program it to wait a few seconds before responding, and to give the wrong answers a certain proportion of the time.
Re:By design (Score:4, Funny)
Re: By design (Score:2)
Does javascript have sleep()?
Re: By design (Score:3)
Certainly. That's why it's so slow. You can even program how slow you want it to be, for when you need your clients to buy new computers, because current ones are too slow. :)) Just joking...
Re:By design (Score:0)
A long time ago I read a Slashdot post that recommended both answering incorrectly and slowly, if you want to get by the tests reliably.
From my experience, just being slow is reliable. I rarely get more than one captcha prompt.
They really have no incentive to reward users who answer both quickly and correctly.
Can they get past The Cookies Check? (Score:5, Insightful)
Re:Can they get past The Cookies Check? (Score:1)
Re:Can they get past The Cookies Check? (Score:2)
Best I've got is I Don't Care About Cookies to kill the popup and Privacy Badger to castrate the cookies themselves. Gets almost all of them.
What we really need is the ROTW to catch up to the EU and UK with making cookie skulduggery illegal (a "G" GDPR). No, you can't ask me to give up my statutory rights with a popup. Also the irony of American sites doing this is palpable. The EU knows it's own laws don't apply to entities outside it's own borders, they're just doing it to scare the uninformed and stupid against supporting GDPR laws.
Cost of article 27 compliance multiplies (Score:2)
What we really need is the ROTW to catch up to the EU and UK with making cookie skulduggery illegal (a "G" GDPR).
If different countries and free trade areas were to adopt their own counterparts to GDPR, in how many different countries would each website operator need to register with a designated local representative pursuant to article 27?
The EU knows it's own laws don't apply to entities outside it's own borders
Laws of the Union apply at the border. US-based online stores without a representative pursuant to article 27 could see their shipments turned away at member states' customs.
Re:Cost of article 27 compliance multiplies (Score:2)
What we really need is the ROTW to catch up to the EU and UK with making cookie skulduggery illegal (a "G" GDPR).
If different countries and free trade areas were to adopt their own counterparts to GDPR, in how many different countries would each website operator need to register with a designated local representative pursuant to article 27?
Erm, maybe they can just stop trying to store people's data and sell it off to other parties. That seems the cheapest solution
The EU knows it's own laws don't apply to entities outside it's own borders
Laws of the Union apply at the border. US-based online stores without a representative pursuant to article 27 could see their shipments turned away at member states' customs.
Erm, a server in a completely different country, no physical presence in the EU... How would they enforce that. You clearly know nothing about the EU if you think that they'll block any site that doesn't register with them. Even in the worst case scenario... why am I still getting anti-EU annoyance popups when I'm in the Americas?
Also article 27 only applies if you're holding data on EU or UK citizens, residents at al... The simple answer to avoiding A27 is don't. That's the point, don't store or use our data for whatever purpose you feel free to. Again, the EU knows this is effectively unenforceable on any entity outside the EU (the UK has more imperialist desires, but is powerless).
You clearly have no idea about the GDPR and have fallen victim to their scare mongering. A27 isn't the GOTCHA you think it is. People who want to sell everyone's data with no regard for privacy must love you.
Physical goods shipping address is personal data (Score:2)
Erm, maybe they can just stop trying to store people's data and sell it off to other parties. That seems the cheapest solution
Say a company operates a web-based store through which it sells something goods, such as toys or physical copies of a work, to customers. I fail to imagine how the company might accomplish that without collecting two pieces of customers' personal data: the billing address and the shipping address. Even a website that doesn't ship physical goods is taking billing addresses once it sells a subscription.
Erm, a server in a completely different country, no physical presence in the EU... How would they enforce that.
If a website neither mentions any member state nor quotes prices in euros nor offers a translation of the website in a language primarily spoken in member states, the site probably falls under the recital 23 [gdpr-info.eu] carveout for foreign businesses that don't knowingly target viewers in the Union. But once you take payment for shipment of physical goods, your shipments can in theory be stopped at the border.
why am I still getting anti-EU annoyance popups when I'm in the Americas?
The Americas include the U.S. state of California. California passed a privacy law called CCPA, some of whose provisions parallel provisions in the GDPR.
Re:Can they get past The Cookies Check? (Score:3)
Re:Can they get past The Cookies Check? (Score:4, Funny)
"The Internet doesn't work without cookies."
Aaand now I'm visualizing "The Internet" as having blue fur and googly eyes.
Re:Can they get past The Cookies Check? (Score:3)
Re:Can they get past The Cookies Check? (Score:2)
But if you don't accept the cookies, the website may or may not work correctly. You want to receive the best possible browsing experience, right? The Internet doesn't work without cookies.
No one is talking about rejecting all cookies. In fact that is not an option presented to the user by these annoying popups.
Re:Can they get past The Cookies Check? (Score:2)
>The Internet doesn't work without cookies.
in fact, the reason m UID here is so *high* is that I refused to register for a while due to the use of cookies!
I finally caved, but that also meant having to stop using a folder named .cookies to blanket-block them.'
hawk
Re:Can they get past The Cookies Check? (Score:2)
Great (Score:5, Insightful)
Re:Great (Score:1)
"Click on all the pictures of people wearing jumpers."
How big are the 'bots'? (Score:3)
Re:How big are the 'bots'? (Score:2)
The number of pictures is limited. I've seen the same bicycles hundreds of times by now.
Even if it took 10 seconds that's not a big deal. There aren't _that_ many sites you'd care to crawl that use them and once you've hit it, you're free to do whatever after that.
Your crawler will be multi threaded. Just send the captcha threads to go do their thing while the rest continue on other sites. Turning a 20 hour crawl session into 20 hour, 15 minute crawl session isn't a big deal.
Found the paper (Score:5, Informative)
Tracking how you SOLVE the puzzle (Score:5, Interesting)
Re:Tracking how you SOLVE the puzzle (Score:2)
Re:Tracking how you SOLVE the puzzle (Score:1)
I'm pretty sure they're not collecting mouse position data on my iPad.
I have seen times where I realized I missed one after I hit go but I suspect they simply have a "good enough" setting rather than some complex and clever algorithm.
Re:Tracking how you SOLVE the puzzle (Score:2)
Re:Tracking how you SOLVE the puzzle (Score:1)
Interesting idea, thanks. Like everyone, I find them frustrating and useless. Similar to the useless "check here if you're not a bot" but worse.
My kid had one for some game she was trying to sign up for. Click the animal in the upright position. 4 animals, each successively 90 degrees off. I couldn't do it either. So they stupidly lost some number of sales because they couldn't bother the most basic testing of their captcha clone.
Re:Tracking how you SOLVE the puzzle (Score:1)
My understanding of many current implementations is that they don't just look for a correct answer, but analyze how you solve it - by tracking things like cursor movements or the time between clicks.
Yes, but that is "part of the solution" regarding the bots solving them.
They also use things like browser fingerprinting. A human, for example, would not be able to click on 4 tiles with exactly 15ms between clicks.
A bot, while it would be able to do that, would intentionally not do that either.
The bot is going to be sending "cursor move" events along with "click" events, while Not moving in a straight line between two points, while also Not moving the cursor at a consistent speed between pixels.
A bot won't select the same exact point within a tile to click, and will vary the time between the "down button" and "up button" events.
They also will add some randomness to the order of tiles it plans to select, as if it "missed seeing" one and had to go back for it after a much longer delay than for the others.
All of those things are considered part of the solution to the puzzle, and are data points that far outnumber choosing which tiles should be selected.
I've noticed lately that those "select the traffic lights" puzzles more often than not will accept solutions that are blatantly wrong - maybe I only clicked on one tile per light fixture, or clicked on all the frames that have slivers of poles and wires. That makes me wonder if the tiles I click on don't actually weigh much in the calculation
At one time it was a popular addition to captchas to take feedback from prior solutions to adjust the weighting of the various tiles.
So an obvious traffic light that nearly everyone clicked on would change in weight to be stronger, along with tiles no one ever clicked on weighted stronger in the other direction.
Tiles that say half the solvers clicked on would be weighted down to basically not count anymore, to try and account for ambiguity, like the situation you describe (slivers of the described image)
I know the methodology has long moved on, but I am sure some versions of the software currently in use likely still have that type of weight system in place.
Re:Tracking how you SOLVE the puzzle (Score:5, Insightful)
Anything that these systems use to try to differentiate between humans and bots can be faked by a bot. Anything you think of, including human weaknesses, can be faked by a bot. There is absolutely no way around this.
Maybe current implementations don't do some of this faking. That's because they don't need to. The moment they need to, they will be made to.
The enterprise of making a website try to weed out bots is fundamentally misguided. It can't be done. That ship has sailed. Give it up.
I can hear the objections already...we just need a webcam so you can see the user's face, watch them type it! We just need a secure key based mechanism that gives the website some kind of superior access to the end-users hardware, to make sure a human is using it and not a program! We just need some more laws!
It's folly. None of these will work. Nothing you can think of will work. We are in the AI age now. We cannot put that genie back into the bottle, and we cannot make websites determine when they are being accessed by a bot instead of a human. We must adapt our designs such that it simply doesn't matter if a bot is doing the work. We must build a world that tolerates bots, and still functions well enough. That's the shape of things to come.
Re:Tracking how you SOLVE the puzzle (Score:2)
We are in the AI age now.
We're not in the "AI age". That's absurd. We're in the middle of a hype cycle, and it looks like we're already past the peak. We've been using ML methods to defeat captchas since they first appeared. It's always been an arms race and we've had bots that outperform humans on various types for almost as long as we've had captchas.
The study [arxiv.org] is comparing human performance (from Mechanical Turk) across a variety of captcha types to bot performance reported in other, sometimes much older, studies. Whatever "advancements" you seem to think we've made in the past couple years are not at all responsible for the results they reported.
Re:Tracking how you SOLVE the puzzle (Score:2)
Re:Tracking how you SOLVE the puzzle (Score:0)
I'm also doing my part selecting at least 1 wrong answer whenever I can. One time I got the captcha into hardcore never-ending mode. I think it identified me as bot and there was literally no way out even if I selected all the correct answers multiple times.
The solution is simple (Score:2)
I have this plugin which helps me fill in the captcha, even does it for me. It's only the start though, the next generation of captchas requires more advanced tools to answer them and guess who is making and selling those tools? The same guys who are making the captchas.
Re:The solution is simple (Score:0)
Captchas have become pretty terrible (Score:2)
Captchas have become so terrible and annoying, I am starting to question my humanity.
Re:Captchas have become pretty terrible (Score:2)
Captchas have become so terrible and annoying, I am starting to question my humanity.
Please don't. If you were a bot, you wouldn't mind captchas at all. Your anguish proves that you're a biologic intelligent unit.
Link? (Score:2)
The linked site contains no link to the study, nor does the /. summary.
I guess this is that faith based reporting we hear so much about. Which captchas were tested? Which bots? All the linked site has is links to their own page for keywords that I am sure boost search ranking.
So? (Score:1)
well... (Score:1)
Captchas force you to train their AIs for free. (Score:2)
Not surprised (Score:2)
Eventually I gave up and chose the audio option, which I solved first time.
Re:Not surprised (Score:2)
Exactly! I've failed captcha's recently for the same reasons. They used to make the answer obvious. Some of them aren't as obvious now and are frustrating.
Personally, I hate Captchas (Score:2)
The makers of captcha are evil fucks. (Score:4, Insightful)
I wish them bodily harm.
A grid of cells with, "select all the cells containing motorcycles", when sometimes a sliver of a helmet is part of it and sometimes it isn't, is a dark pattern. Whomever set that up needs to be beaten with sticks.
They absolutely offer captchas that increase in ambiguity to block content distribution. It's shady as hell.
Re:The makers of captcha are evil fucks. (Score:2)
Re:The makers of captcha are evil fucks. (Score:2)
I wish them bodily harm.
Do you routinely attack the people trying to help rather than the cause of the problem? Do you hate an ambulance instead of a terrorist? A police officer instead of the robber?
Captchas exist to solve an underlying problem. Hating on them rather than those who cause the issue in the first place is one of the most dumbfucking stupid things I've read on the internet this week.
Re:The makers of captcha are evil fucks. (Score:2)
" one of the most dumbfucking stupid things I've read on the internet this week."
Glad to help. Hope you found it entertaining.
In case it wasn't clear, my complaint was not about trying to find a solution to the problem. It's that the solution doesn't confine itself to that. It's because what they've built is unworkable, and used for shady purposes - like making it difficult to access the thing you have - in some cases - paid for. This is a conscious choice, I guarantee it. You can ask them to make it exasperating to pierce the veil, and that's what they'll give you.
Simple solution (Score:0)
Websites simply need to adjust their logic so users that fail their captcha are assumed human, and those that solve their captcha are assumed bots.
POW Captcha (Score:2)
Too Fast (Score:2)
So if they solution takes 3 seconds it's not a human.
That seems too obvious. I can't believe this work.
Maybe I’m an AI (Score:2)
Wait, WHAT? (Score:2)
How is this NOT an Onion post?!
Re:Wait, WHAT? (Score:2)
Or an ig-nobel?
bleep blip (Score:1)
I'm thinking of making a T-shirt saying, "Yes, I'm a robot! Whaddya gonna do about it!?"
Re:bleep blip (Score:1)
Re:bleep blip (Score:1)
Perl one-liners can pack enough info to run air traffic control. Reading it is another story.
Re:bleep blip (Score:1)
I make captcha mistakes on purpose (Score:2)
I make mistakes on purpose when solving captcha tests, to find out how many mistakes you can make for captcha to be accepted. Like, when you get 2 words "gimme shelter", I enter "giemm shetler". That can pass sometimes, but entering "gimme shetler" will always pass. Same with pictures - you don't have to click on all the pictures, just a couple and even some wrong ones, to pass the test. You should try it sometimes. It's fun. I guess this also makes machine learning harder, and us humans look more stupid than we are. :)
This is getting pretty obvious (Score:2)
One of the major commercial applications for today's AIs is interpreting digital scans. AIs that are getting good at finding tumors in a noisy mammogram are going to be really good at identifying fuzzy letters and numbers in a CAPTCHA.
Meanwhile, you as puny human get shown endless grids of motorcycles and traffic signals, and you will fail every time because they CAPTCHAs are now beyond human pattern matching ability. Time to retire the entire idea.
Re:This is getting pretty obvious (Score:2)
It's always been an arms race. We've had bots capable of defeating captchas almost as long as we've had captchas. That doesn't mean we need to scrap the whole idea, only that we likely won't have a long-term solution. Remember that 'bots' can't just magically adapt to new challenges and that training takes a great deal of time and effort.
While the study compares human performance on a few different kinds of captchas (from Mechanican Turk) to bot performance reported in other, sometimes much older, studies, that's not really what it's about. It's about human performance and perceptions, not advances in captcha solving bots. The headline completely misrepresents the purpose and results of the study.
Slow speed (Score:2)
bots not affected by aging in solving captcha's (Score:2)
bots aren't affected by problems that aging eyesight causes in solving captcha's
What counts as success? (Score:2)
Is success getting though the stupid gate, or accurately performing the task? I still run into plenty of captchas were system has misinterpreted markings on the side of the road as as a crosswalk, misses half of the stop-lights in a scene, etc. And I therefore end up having to answer more of the stupid things for doing them right.
Test taken the wrong way (Score:2)
Have I misjudged Captchas? (Score:1)
I have always thought that speed and pace of clicking were being evaluated along with accuracy... such that solving too quickly would indicate a 'bot, and fail the test.
Internet Poll (Score:2)
This means they put it on Fiver and/or Mechanical Turk.