Slashdot is powered by your submissions, so send in your scoop


Forgot your password?
Security Software Hardware IT

Most Security Products Fail To Perform 99

An anonymous reader writes "Nearly 80 percent of security products fail to perform as intended when first tested and generally require two or more cycles of testing before achieving certification, according to a new ICSA Labs report that details lessons gleaned from testing thousands of security products over 20 years. Across seven product categories core product functionality accounted for 78 percent of initial test failures. For example, an anti-virus product failing to prevent infection and for firewalls or an IPS product not filtering malicious traffic. Rounding out the top three is the startling finding that 44 percent of security products had inherent security problems. Security testing issues range from vulnerabilities that compromise the confidentiality or integrity of the system to random behavior that affects product availability."
This discussion has been archived. No new comments can be posted.

Most Security Products Fail To Perform

Comments Filter:
  • by mjwx ( 966435 ) on Monday November 16, 2009 @09:53AM (#30114588)
    Maybe they're nervous?

    I mean you put them under a lot of pressure to perform and chastise them harshly when they fail to meet your expectations.

    Perhaps you should mix them a nice drink, use some mood lighting and tell them you love them once in a while. It's not just about you after all.
    • Re: (Score:3, Funny)

      by slimjim8094 ( 941042 )

      Security devices can't get it up?

      Of course not - many security devices require you to get it up before you can even install them.

      • Re: (Score:1, Insightful)

        by Anonymous Coward

        Porn, Government, and Security Devices...They promise so much, and deliver so little.

      • As my basketball coach said, "You gotta get it up to get it in."
    • Re: (Score:3, Interesting)

      You mean after the all the claims they made? After all they said they'd keep us safe from? After how sure they made us feel in their ability? After all the charm, and the cajoling, and the expenses, and the hassle? After all they promised, now that they can't live up to even our most basic expectations, you're telling me that we're the ones at fault?

      They can't perform, but now we're the ones who have to change? We're the ones who have to clean all the laundry, and be careful around strangers, and lock up fo

    • Remove the word "security" from the above sentence and replace with "software", and you'll have a good view of the industry.

  • This just in! (Score:4, Insightful)

    by L4t3r4lu5 ( 1216702 ) on Monday November 16, 2009 @09:53AM (#30114590)
    New devices and software may have bugs which affect performance. Patches may be required for correct performance when exposed to unexpected conditions.

    Is security software supposed to be automagically immune to human error? Or is this another "Coders aren't employing secure coding practices" piece I've been reading for well over 3 years. "Validate your inputs" "check loops exit under all circumstances" etc etc. Woo. Insightful this ain't.
    • Re: (Score:1, Insightful)

      by Herkum01 ( 592704 )

      New devices and software may have bugs which affect performance. Patches may be required for correct performance when exposed to unexpected conditions.

      Companies (in general) would rather polish turds than expend the energy to make a good product.

    • Re:This just in! (Score:5, Insightful)

      by mcgrew ( 92797 ) * on Monday November 16, 2009 @10:18AM (#30114808) Homepage Journal

      Woo. Insightful this ain't.

      Mods, please don't mod that uninsightful coment "insightful". Having a defect in a device I've bought has been extremely rare, buying anything from toasters to TV sets to video cards that just don't work is unheard of. Don't talk to me about the "complexity" of writing software, you think you car is simple?

      If your software is buggy your company is incompetent. Period. We as customers shoud stop putting up with defective products and beta sofware that's been rolled out as a "finished product." If I find your software doesn't perform, I should get my money back.

      People, can we please stop putting up with incompetents' excuses? After a quarter of a century of putting my up with your crap software I'm getting a little tired of it.

      • Re:This just in! (Score:4, Insightful)

        by Thanshin ( 1188877 ) on Monday November 16, 2009 @10:40AM (#30115076)

        you think you car is simple?

        Car analogy to the rescue!

        Let's imagine you're a car builder capable of building cars with the current expected quality.

        Let's now imagine your competition builds and sells defective cars for half your costs. For whatever reason, the buyer will buy the half cost faulty car and then repair it until it finally works, rather than buying your "perfect on release" car.

        What do you do?

        • Re: (Score:3, Informative)

          by mcgrew ( 92797 ) *

          Let's now imagine your competition builds and sells defective cars for half your costs

          So if that would work, why hasn't anyone done it? The answer is simple -- car buyers are smarter than people who buy software. Also, it's a lot easier to patch a program than to recall a defective car.

          And cars have warrantees. I'd like to see warrantees on software.

          Also, see the AC who responded to your comment, he said a few things I was going to.

          • Your answer leads to the answer to the post I was answering (which was my intention).

            Changing the software quality paradigm isn't a responsibity of the producers, it's the buyers who must start asking for quality and paying for it, as they are who create the market in the first place.

      • Re: (Score:3, Funny)

        by L4t3r4lu5 ( 1216702 )
        Here's a quote you might like: I reject your reality, and substitute my own! - Mythbusters

        Half of me thinks you're being sarcastic, but the other half is concerned that you think companies actually want to pay for something good, and that PHBs don't impose stupid deadlines to rush projects out of the door because competitors are building the same product.

        You want to know which projects are going to be bug-free at realease? Hurd, Duke Nukem: Forever, and the Phantom console.
      • Re:This just in! (Score:5, Insightful)

        by RichardJenkins ( 1362463 ) on Monday November 16, 2009 @11:17AM (#30115616)

        Your car may be complex, but it has relatively few ways for the user to interact with, and is likely always used in the same environment, and fundamentally the same to most every other car on the road. It's been done. Lots.

        This goes doubly for your TV and even more for your toaster.

        Are you saying software bugs needn't exist because mechanical and electrical engineering can be done so well? That's asinine.

        And last I checked, most cars can still crash.

        • Once again we come to the weak point in the system.

          Idiot in, idiot out. PEBSWAC (Steering Wheel)
        • by mcgrew ( 92797 ) *

          And last I checked, most cars can still crash.

          Like software, blame the device driver... only with cars it usually is the driver.

      • Re: (Score:3, Insightful)

        There's a big difference between software and hardware my friend. The first of which is safety: when a TV or Car blow up or otherwise severely malfunction it is not tolerated and therefore companies that make those products have much different cycles of testing and engineering (Waterfall development cycles). Software on the other hand has much more leniency for most fields since it has the capability of being continually improved and has a tendency to be rushed through development with that in mind (Spiral
        • by rgviza ( 1303161 )

          This is the tip of the iceburg. At least software engineers don't burn people's houses down and kill people with a bug then deny it's their fault.

          I got news for you: nobody's perfect.

        • Re: (Score:3, Insightful)

          by mcgrew ( 92797 ) *

          If your starter goes out a week after buying a new car, there's no safety issue but you're not likely to buy that brand of car again. Any auto manufacturer with shoddy manufacturing and design won't be in business long, unlike software.

      • by rgviza ( 1303161 )

        You should become an engineer. You sound like you would be a perfect candidate for the job with your quality conscious attitude.

      • I have to applaud your comparison to the car or even the airline sector. If any of those sectors had the same failure rate as the software sector, we would be walking with about 50% less population and an accepted death rate much higher then we do now.

        People allow this crap to happen, and still line the pockets of the M$ corporate types. The day we all say no to gas price hikes by banding together and stop buying gas for a full week, like the email sells you to do....then we will see gas prices drop like a

        • by mcgrew ( 92797 ) *

          I agree with you, except for this:

          The day we all say no to gas price hikes by banding together and stop buying gas for a full week, like the email sells you to do....then we will see gas prices drop like a hot potato.

          Gasoline prices are artificially influenced by the prices of gasoline futures. A little earlier this year gas prices started dropping slightly because of oversupply and decreased demand, then promptly jumped $.25 per gallon in less than a week because it looked like the recession may be ending

          • Do not confuse the gas prices being the way they are because of what have done compared to what the gas prices are because of what the economy is and what the gas companies think we can afford.
            A study has been done by the likes more intelligent then me, that have calculated the holding tanks capacity for storing the fuel the saudis have, and how much is actually coming out of there....

            If we were all to just stop gasing for a week, and I mean all of north america, their biggest client....they would such an o

      • Re: (Score:1, Insightful)

        by Anonymous Coward

        > Don't talk to me about the "complexity" of writing software, you think you car is simple?

        Security hardware/software has to constantly deal with new attacks. Your car doesn't have to deal with anything that hasn't existed for ah hundred years already, and yet a street thug can still be walking away with your radio in under two minutes. (Or driving away the whole car, given slightly more time).

        > If your software is buggy your company is incompetent. Period. We as customers shoud stop putting up with d

        • by mcgrew ( 92797 ) *

          Yes, there are recalls, but I've never had a car I've owned recalled and I doubt many people have had more than one recall in their life. I wasn't just talking about security software and security vulns, but software and bugs in general.

          Computer programming has been around since 1843 [], the gasoline powered automobile [] wasn't invented until 1806, less than fifty years earlier.

      • by syousef ( 465911 )

        Having a defect in a device I've bought has been extremely rare, buying anything from toasters to TV sets to video cards that just don't work is unheard of. Don't talk to me about the "complexity" of writing software, you think you car is simple?

        I guess you've never heard of factory recalls?

        Video cards []

        Toasters []

        • by mcgrew ( 92797 ) *

          Yes, there are product recalls, but they are relatively rare. I've never had a product I owned recalled, but except for games I don't think I've ever bought a piece of software that didn't get at least one bug patch.

          Yes, airplanes crash, but that's rare too.

      • Why sell a product if you're not ready to make it work? I'm looking at you, Symantec! When Endpoint Protection first rolled out it was so buggy it couldn't do anything right. It's as though nobody bothered to do any testing at all. And those guys had the nerve to collect money for it.

    • by gclef ( 96311 )

      No, but there is a certain level of irony (or at least amusing superposition) when a security product has security vulnerabilities...after all, getting your company hacked because you put in security controls isn't the way one anticipates these things happening. To use a car analogy, it's as if belting your seatbelt led to *more* people dying.

    • New devices and software may have bugs which affect performance. Patches may be required for correct performance when exposed to unexpected conditions.

      I read the article.

      Nowhere does it mention that these are new products. Only that they're newly used within any given company.

      It also mentions that patching to fix problems is a problem in itself, with 20% of products failing to accept patches properly.

    • Re: (Score:1, Informative)

      by Anonymous Coward

      I think we're all missing something in the article summary:

      when ***FIRST TESTED***

      Read the rest of that sentence, too. "Two or more cycles of testing before achieving certification". That means that it hasn't been released yet.

      When was the last time you coded something and it not only compiled the first time out, but worked perfectly? Was it entitled "Hello World"?

      Sorry, but this article is not news.

  • by Anonymous Coward

    Verizon is just trying to proof the relevancy of their so-called 'security' tests. They do not really perform any security test at all. Please, stop posting these marketing messages. And puleaszze, stop this semi-bullshit measures such as 44%, 78% ...

  • by Afforess ( 1310263 ) <> on Monday November 16, 2009 @09:59AM (#30114652) Journal
    There is no such thing as security. You can become more secure, but never absolutelysecure. Security is a process, not a product. The moment we realize this, most of these problems go away.

    Instead of looking for the "silver bullet" in the form of a anti-virus software, you should be using anti-virus in conjunction with Firewalls, the latest patches for your OS, and safe browsing habits. After all, I would bet that 9/10 viruses come in the form of human error rather than the case of a malicious hacker trying to force entry to your system.
    • 'Security is a process' is a tired cliche and a cop-out. What you are saying is: Our firewall doesn't quite work, but if you buy all these other pieces of shit that also don't quite work and implement all these procedures and unplug your network cables, then you'll have a good system...
      • Re: (Score:3, Insightful)

        by TwistedGreen ( 80055 )
        No, what he's saying is that a single security solution will <i>never</i> work 100%. You're right, the only magic bullet is to unplug your network cables, but that's not going to happen. That's why you need multiple lines of defense combined with informed usage policies.
  • by fuzzyfuzzyfungus ( 1223518 ) on Monday November 16, 2009 @10:00AM (#30114654) Journal
    The TSA has issued a press release calling their performance "In line with industry standard private sector security solutions"...
  • by Anonymous Coward on Monday November 16, 2009 @10:02AM (#30114670)
    This report is not good news. While ICSA is promoting the need for certified security products, it may do more to convince security managers that they've been getting ripped off. This is what Larry Walsh writes in his blog: []
  • by Dr. Evil ( 3501 ) on Monday November 16, 2009 @10:18AM (#30114804)

    This all sounds like security certification speak.

    Among the recommendations from the article: "Use certified products. While certification can never eliminate risk, it substantially reduces risk by ensuring that products meet objective, publicly vetted criteria."

    This shouldn't be on Slashdot. We all know that the best software tools are FOSS, subject to the most rigourous testing and peer review. "Certified Products" are a black box with a "Trust us" next to a logo for a "Limited Liability Coproration."

    The article should be lumped in with the Gartner reports and marketing materials.

    • Re: (Score:1, Insightful)

      by Anonymous Coward

      Yes and NT4 got EAL rating with a bunch of qualifications.

      A whole industry of 'certifiers' has sprung up to make money off clients who can then paint gold stars on their products - just like wine or cigars.

      Everyone is missing the point: The vendor is proactive and up to it - or they drag the chain about timely patches. If they are 'one monthers' On this score, BSD beats MS.

    • I've been involved in certifying a firewall to meet ICSA requirements. Let me say that it can only be a good thing to take into account what certifications the product has before using it. This includes FOSS and commercial.

      While it's nice that you can review the source of FOSS tools, that gives you no guarantee that the tools are configured appropriately and securely. If you are in an organisation that requires a verifiable degree of security (or as management sees it: level of risk) then using certified

      • by Dr. Evil ( 3501 )

        FOSS tools are disadvantaged when it comes to certification because certification is expensive, time consuming and resists changes in the project. On the other hand, for-profit vendors are disadvantaged when it comes to security because scrutiny is limited and the motive changes from quality to profitability.

  • by jonaskoelker ( 922170 ) < minus city> on Monday November 16, 2009 @10:21AM (#30114844)

    This highlights a point you may very well know already, but allow me to restate it:

    People (at least people who program computers) haven't really figured out how to write secure code.

    Well, what do I mean by secure code? Code that is 100% secure against a particular well-specified threat, or several of these. I.e. "only users logged in as root on the local console can [...]; users accessing the database through the web interface can't [...].", or "no TCP flow will cause the $OS network stack to crash", or [etc.].

    This article is merely the observation that even when people write code that has a security function, they can't magically do better than everybody else.

    Also, I'd like to advocate the viewpoint that security is a system property. You can't apt-get install security. Putting a firewall in front of a flaky app (especially a flaky proprietary app) is not going to work well: if you need code to detect whether a packet is evil or not, why don't you put that code in the application, so you don't have three competing vendors waste time trying to be the best flaky-packet-handler for $APP?

    Oh well, I guess you can ship sooner. Also, if the original developers of $APP can't get the don't-be-flaky right, we might need something to stand in front.

    (I hope this is more coherent than my feeling of well-being would suggest I'm able to make it)

    • Re: (Score:3, Insightful)

      by maxume ( 22995 )

      It isn't just the knowing, there is also the bothering. For instance, buffer overflows and SQL injection are some of the most commonly exploited flaws in programs, and the prevention of both is well understood.

  • Is anyone here suprised by the fact that security isn't something anyone can buy?
  • The most common source of security problems is poor user interfaces. These can't easily be fixed by third-party products. A ludicrous password policy, for example, which makes people write their passwords on post-it notes because they can't remember them, is a good example. ActiveX allowing untrusted code to run with full privileges with a single button press was another example. UAC and SELinux also suffer from this; the UI is so bad that people often just disable them.

    The other cause of security pro

    • by mcgrew ( 92797 ) *

      A ludicrous password policy, for example, which makes people write their passwords on post-it notes because they can't remember them, is a good example

      If your office door has a physical lock on it, a postit note isn't insecure. My office has no door so I keep passwords written down, in my wallet, disguised as something else. At home it's a list of sites and passwords written down and laying on the computer desk.

      I'd like to know why there's the "change your password monthy" rule? It seems to me that a rule l

      • Re: (Score:3, Informative)

        by TheRaven64 ( 641858 )

        If your office door has a physical lock on it, a postit note isn't insecure

        And the cleaner, being paid minimum wage, won't be tempted to make a couple of years' salary selling the password to an unscrupulous competitor? Depends on your market and how well you vet your staff...

        I'd like to know why there's the "change your password monthy" rule?

        Cargo cultism. This one actually used to make sense, but was copied by people who didn't understand it. Passwords are stored encrypted. To reduce CPU load, they used to use very simple hashing / encryption algorithms. A month was about as long as you could guarantee that a copied password file would rema

        • Re: (Score:3, Interesting)

          by Eevee ( 535658 )

          Close but no cigar. You change passwords periodically in order to limit damage. If your password is discovered by someone, then they can only exploit it until the next password change. Guess what...if you keep the same password forever, it can be exploited forever.

          Yes, there are many circumstances in which the damage from a compromised password happens immediately after the compromise. But there are times when the damage is ongoing; consider a rival company monitoring the progress of a new product via email

  • by Anonymous Coward

    Change "most security" to "most products" fail to perform.

    Software is generally poorly written, is not held to any product standards, comes with "NO WARRANTY", "NO FITNESS FOR A PARTICULAR PURPOSE" and contains "KNOWN DEFECTS".

    It's like a new car coming with two flat tires, and you happily paying for it.

    It's time we hold software to some decent standards.

  • You cannot buy security and you cannot buy love.

  • Yeah, I am a bitter vet, and I am so damn happy I got out of that shit world called 'security'.

    People were just too dumb, they always wanted to buy products to "make them safe", while they almost never wanted to invest into training, procedures, policies, etc.

    Guess they're happy now.

  • So, a certification vendor says certification is necessary, based on statistics produced in-house. Subtext: security product vendors need to buy the services of the certification vendor. It might be true, or it might be bias. Hardly news.

  • The customer for 'Security Products' is some buyer typically disconnected from the nuts-and-bolts of security!

    A bunch of mid-to-upper level people sit in a room and talk about 'security.' They don't understand it, but the like/need the idea of it so they can come off as believable to their customer. Better still the clicky-pointy-GUI and report generation features *really* feed the TPS beast. They talk past each other and pass reports around. Perception! Perception! Perception!

    The finance industry is th

  • by petes_PoV ( 912422 ) on Monday November 16, 2009 @11:29AM (#30115808)
    The article paints a negative picture, when in fact the opposite is true: testing works! When we test stuff we find the bugs, fix them and re-test. After a few iterations the tests are passed. What's wrong with that? As someone who's done a *lot* of testing in the past it sounds to me like the process works.

    If the testing process didn't find any problems and passed a product on the firsat attempt, I'd be more suspicious of the tests than of the product - not that I'd buy the product, either.

  • Every security product "fails." It is impossible to prevent all threats. The point of security is to reduce the risk of compromise. There will always be some risk.

    If an antivirus product stops now viruses at all, then it's a failure. If it lets some through but stops others, then it is actually a success because it reduces risk.

  • Security isn't a 'product' that you can bolt on. Security is something that has to be built in from the ground up. A primary function being irrevociable auditing of all activity on the system. How you can design a 'security product' that doesn't accuratly log activity beggers belief. These 'products' sound like the typical management process of covering their arses with certificates.

    'Incomplete or inaccurate logging [] of who did what and when accounted for 58 percent of initial failures'
  • ... we should point the finger at the criminals that write viruses and otherwise break computers.

    They write viruses to "get around" current virus protection. Now if you have a tool that works, and a criminal circumvents it, how does that make the tool faulty? It wasn't faulty when it was written, what makes it faulty now?

    Are the software engineers supposed to be able to predict the future? What constitutes a tool that works?

    Why don't we hold police responsible for not predicting murders and fireman fires?


  • As some folks know, a lot of physical security products don't really work, either; they give us a false feeling of safety when in fact there is little or no actual benefit. We've got half of America's cities lit up like Christmas trees at night now, burning who knows how many tons of coal every year to do it, but have all those street lights and backyard security lights really made us safer? Some people got a whole lot richer in that process, though.

    Another even more striking example close to home: my city

  • Mod me down, but seriously, SSL, DNSSEC, and so many things for "security" are just junkware, introducing their own bugs and problems while making things excessively bloaty. Noticed how many vulnerabilities there have been in SSL alone lately? It's scary and this really needs to be rethought.

  • Security is a practice, not a product.

"Wish not to seem, but to be, the best." -- Aeschylus