Please create an account to participate in the Slashdot moderation system

 



Forgot your password?
typodupeerror
×
Portables (Apple) Hardware Apple

Consumer Reports Updates Its MacBook Pro Review (consumerreports.org) 246

Reader TheFakeTimCook writes: Last month, the new MacBook Pro failed to receive a purchase recommendation from Consumer Reports due to battery life issues that it encountered during testing. Apple subsequently said it was working with Consumer Reports to understand the results, which it said do not match its "extensive lab tests or field data." According to an article from Consumer Reports, Apple has since concluded its work, and says it learned that Consumer Reports was using a "hidden Safari setting" which triggered an "obscure and intermittent bug" that led to inconsistent battery life results. With "normal user settings" enabled, Apple said Consumer Reports "consistently" achieved expected battery life. Apple stated: "We learned that when testing battery life on Mac notebooks, Consumer Reports uses a hidden Safari setting for developing web sites which turns off the browser cache. This is not a setting used by customers and does not reflect real-world usage. Their use of this developer setting also triggered an obscure and intermittent bug reloading icons which created inconsistent results in their lab. After we asked Consumer Reports to run the same test using normal user settings, they told us their MacBook Pro systems consistently delivered the expected battery life." Apple said it has fixed the Safari bug in the latest macOS Sierra beta seeded to developers and public testers this week.
This discussion has been archived. No new comments can be posted.

Consumer Reports Updates Its MacBook Pro Review

Comments Filter:
  • affectionately termed AppleCore
  • Meh... (Score:5, Insightful)

    by Anonymous Coward on Tuesday January 10, 2017 @01:02PM (#53642753)

    ... battery life wasn't really the straw that broke the proverbial camel's back anyway.

  • By not disabling the cache Safari will just reload the web page from disk, instead of downloading it all over wifi. In normal use you don't sit around reloading the same page all day, you surf to different web sites, so caching extends battery life to unrealistic levels.

    • by TWX ( 665546 )
      I suppose it matters if, as someone browses multiple pages of a website, how much re-downloading of common components of the pages is going on. If I visit the first page for a site to retrieve all of the graphics-intensive formatting stuff, then as I browse thirty more pages on that same site I do not have to re-download that stuff because it's cached then that could make for a difference.
      • by AmiMoJo ( 196126 )

        Sure, it's a great help in real life, but a big problem for repeatable testing and A/B comparison.

        Disabling cache does seem like the fairest option. While in real life you might get somewhat better results, you won't get any worse results.

        It certainly isn't fair to allow reliance on unrealistic caching to inflate battery life.

        • Re: (Score:2, Troll)

          by Just Some Guy ( 3352 )
          But they disabled cache by putting it in developer mode. Which turns out has its own bugs that Apple said they've now fixed, but that's kind of beside the point: they weren't running Safari in a configuration that an end user would ever have found themselves in. I'm in dev mode right now so I can test some stuff on my laptop. You very well might be too. But my non-techie wife will never run in dev mode, so she'd never see the behavior that Consumer Reports experienced.
          • by AmiMoJo ( 196126 )

            What it boils down to is that we can't know how much effect this fix will have until Consumer Resorts repeats the test.

            Maybe it will fix it, maybe this is like the signal strength bug that was supposed to stop you holding it wrong.

          • by saloomy ( 2817221 ) on Tuesday January 10, 2017 @02:19PM (#53643347)
            Sounds to me (with the limited info in the summary) that developer mode was constantly reloading page icons?

            Their use of this developer setting also triggered an obscure and intermittent bug reloading icons which created inconsistent results in their lab

            That would obviously chew up battery life downloading icons over and over again, chewing up CPU cycles to refresh the icon on the window, and chew up wifi power because its easy to see how a stuck loop re-downloading icons could cycle thousands or millions of times.

            Good for Consumer Reports for sticking to their guns, seeing an issue, reporting the issue, and forcing Apple to fix it. It's obscure sure. But many developers probably use Safari Developer Mode to work on their projects, and this will help them.

            • Good point, glad it was reported and fixed. Please mod parent up.
            • But many developers probably use Safari Developer Mode to work on their projects, and this will help them.

              Yes, but those developers don't get their recommendations from Consumer Reports. That magazine's audience would never have encountered that bug.

              Obligatory car analogy: say they're testing a Ford Focus. They disable its antilock brakes so that a professional driver can get its best-case dry pavement stopping distance. Along the way, the find an OBD-II bug that causes the brakes to take twice as long to stop the car. They report the bad results instead of the normal, expected values. Yes, their test was correct! It found a bug that needs to be fixed. However, the only people who would ever see that bug are the exact ones who'd notice something was wrong and be able to troubleshoot it. You and I aren't ever going to disable our antilock brakes, even if a test engineer might.

              I think that's kind of what happened here. Again, yes, they legit found a bug. My problem with it is that they reported the buggy results instead of the actual ones that a normal non-developer would see. A developer would notice their battery draining in a fourth the expected time and that it only happened when they were debugging in Safari, so they probably wouldn't even be significantly affected by the bug.

              • My problem with it is that they reported the buggy results instead of the actual ones that a normal non-developer would see

                So by your reasoning, if a car manufacturer accidentally made a bug which caused the engine to cheat on diesel emissions tests, it's actually the EPA's fault for not designing their test to more accurately mimic how people use their cars in real life?

                Everyone seems to be trying to spin this for or against Apple / Consumer Reports. The no-spin version is that CR was using an industr

      • by Rob Y. ( 110975 )

        Disabling the cache seems reasonable for testing the battery life of one PC vs another, but not to produce a real-world battery life statistic. To some extent, all tests of this sort produce 'performance in testing mode' stats that differ from real life performance, but shouldn't some attempt be made to measure the real life values as well. I normally do something like

        1. wipe the cache
        2. perform the test
        3. perform the test again, assuming the cache is now improving the results.

        Some mathematical combinatio

        • by Rob Y. ( 110975 )

          I guess they could have been doing something like this - and the Safari bug would've still skewed the results enough to matter, though.

        • by jon3k ( 691256 )
          I don't see it as that big of a deal. I bet a large percentage of our average web use is visiting the same sites frequently (ie Slashdot, CNN, Facebook, Gmail, etc). I'd be interested to see what percentage of average users page loads result in cache hits.
      • If I visit the first page for a site to retrieve all of the graphics-intensive formatting stuff, then as I browse thirty more pages on that same site I do not have to re-download that stuff because it's cached then that could make for a difference.

        That depends on a particular user's browsing habits. You might "browse thirty more pages on" Slashdot while reading stories and writing comments but hit only one document on a site when reading each story's featured article. Or you might "browse thirty more pages on" a web search engine while performing queries but hit only one document on a site when reading each result.

    • By not disabling the cache Safari will just reload the web page from disk, instead of downloading it all over wifi.

      Yes, that is the definition of a cache...

      In normal use you don't sit around reloading the same page all day

      You don't? Are you seriously saying you do not visit several sites multiple times in a day?

      Not to mention, lets say some sites you only go to ever so often - say Amazon, I go to a few times a month. A cache is still useful there for many of the page components and CSS files do not change

      • by AmiMoJo ( 196126 )

        In normal use you don't sit around reloading the same page all day

        You don't? Are you seriously saying you do not visit several sites multiple times in a day?

        No, I'm saying I don't sit around reloading the same page all day. I might reload a number of pages, but most of my browsing is to new content. Sure, a lot of the CSS and images and the like will be cached, but that's quite different from the whole page being cached entirely.

        Clearly Apple is trying to minimize network traffic, because wifi uses a lot of energy. That's a good thing to do, a perfectly reasonable optimization, but it does distort automated test results. To make tests repeatable and fair Consum

        • Comment removed based on user account deletion
          • Unless you were visiting using Lynx, it's hard to believe that you didn't benefit from having all the CSS and image files that make up every page on Slashdot cached.

            I can't speak to the CSS because that is normally invisible, but I can speak to "image files that make up every page". They are invisible here. A banner at the top, a couple of small icons at the bottom, but other than that -- what images?

            When I visit a /. page I explicitly refresh the page because I want to see everything that is new. If I wanted to see the same things over and over I'd just print the page and tack it up on the wall.

            • Comment removed based on user account deletion
              • There's more than you think.

                I used about:config to disable image loading (preferences.image.default = 2) and I see only one tiny difference between the previous version of this page and the one I see now: the two little 64x64 icons at the bottom of the page attached to the previous and next stories are gone. That's all.

                Oh, and the "Slashdot" logo is gone. That's not very large compared to the page itself, which is useless if I use a cached version.

    • by TheFakeTimCook ( 4641057 ) on Tuesday January 10, 2017 @01:23PM (#53642935)

      By not disabling the cache Safari will just reload the web page from disk, instead of downloading it all over wifi. In normal use you don't sit around reloading the same page all day, you surf to different web sites, so caching extends battery life to unrealistic levels.

      Read, then post:

      Disabling the Cache did more than just cause Reloading each time. Apparently, it ALSO triggered an intermittent bug in Safari that caused REPEATED loading of "icons" from the page.

      Apparently, THAT is what burned the battery. Very similar to a "runaway process", like I (and others) originally postulated.

      So, you can safely remove that extra layer of tinfoil, Hater.

    • By not disabling the cache Safari will just reload the web page from disk, instead of downloading it all over wifi. In normal use you don't sit around reloading the same page all day, you surf to different web sites, so caching extends battery life to unrealistic levels.

      No, you don't reload the same page constantly, but you usually visit several pages within the same site. There's no reason you'd want those images to reload every time. I saw somewhere that the average user only visits 5 different sites per day. That was a few years back, so it has probably changed since then, but I wouldn't think it's too dramatic.So it seems to me that caching extends battery life to expected levels.

    • By not disabling the cache Safari will just reload the web page from disk, instead of downloading it all over wifi. In normal use you don't sit around reloading the same page all day, you surf to different web sites, so caching extends battery life to unrealistic levels.

      On your normal web site, the actual page content is a pretty small part of the overall download. As images, stylesheets, and javascript files are cached the only thing that loads from page to page is the actual HTML. I develop using Rails, and we have an asset system now that allows us to tell the browser to cache the non-HTML assets for a year, so they'll hopefully never be reloaded. If using standard jquery and such, you can use a CDN that'll have the same sorts of policies to promote caching.

      Turning o

    • Comment removed based on user account deletion
    • By not disabling the cache Safari will just reload the web page from disk, instead of downloading it all over wifi. In normal use you don't sit around reloading the same page all day, you surf to different web sites, so caching extends battery life to unrealistic levels.

      Really? In normal use, people visit a very small subset of sites very frequently. I'd wager the majority just waste the day on facebook alone.

      You, in all likelihood, visited this site twice already, once for the home page, and once to get to this article. Not to mention whatever happens after you submitted your post. Turning off cache would be far *less* realistic

      Besides, it's not like Apple invented web caching just to inflate their battery performance. It exists in all browsers for a very good reason. An

    • by hey! ( 33014 )

      Not if you count downloading popular javascript libraries like JQuery. That's 258 KBytes/page refresh right there. A lot of ad-related stuff follows you around too.

    • by MikeMo ( 521697 )
      They actually put Safari in developer mode. While it did achieve the desired results of setting it for no caching, the real issue was that it also triggered a bug that only exists in developer mode.
  • Inaccurate headline (Score:5, Informative)

    by celeb8 ( 682138 ) <celeb8&gmail,com> on Tuesday January 10, 2017 @01:15PM (#53642853)
    They did not update their review, they posted that they may.
    • by Calydor ( 739835 )

      But did they post that as an update to the review, or in a separate article?

    • by Altrag ( 195300 )

      They have updated it now to note that a fix has been produced and they're waiting to re-test with the fix (that part hasn't been done yet I guess.)

    • They did not update their review, they posted that they may.

      They said that they were in the process of doing a thorough re-testing, and will update their review when complete.

  • by phantomfive ( 622387 ) on Tuesday January 10, 2017 @01:16PM (#53642871) Journal
    So that's what you need to do to get Apple to fix your bug. A couple years I found a bug in their version of sqlite3, it stopped accepting international characters (chinese, japanese, specifically). I tried to submit a bug via their bug reporting console, and I got this error message [imgur.com]. So I sent an email to the address listed there, explaining the situation, and I got a response,
    "Please report that through our bug console." The console was still broken.
  • by bjdevil66 ( 583941 ) on Tuesday January 10, 2017 @01:17PM (#53642881)
    .. - one that returns a USB and HDMI ports as well.

    If there's a setting to make the dynamic touchbar a static set of function keys I'm used to using, vs. having to wonder what buttons/functionality is there at any given point in time, that would be helpful as well.
    • If there's a setting to make the dynamic touchbar a static set of function keys I'm used to using, vs. having to wonder what buttons/functionality is there at any given point in time, that would be helpful as well.

      I kind of think they are going to add that Option, if it is not there already. But right now, you can do it [apple.com], at least sort of...

    • Just hold down the Fn key. I think that was the default on previous Macs anyway, since the brightness and volume settings were performed by pressing the top row without holding down Fn.

      • Yes because it's awesome having to press Fn-Ctrl-F12 instead of Ctrl-F2 a few hundred times a day.

        You can force the function keys, but you have to add each application in the control panel one by one.. there is no global setting.
        • Give it a while, someone will create a utility to do the one-by-one adding for you. Maybe you could do that yourself and release it to the world. Either way, it will probably not be a problem long-term.

          • I'm not a big fan of installing many little third party toolets to make an expensive machine do what you want, as important as it seems to be on OSX. I hope Apple sees their way to doing it.
            • installing many little third party toolets to make an expensive machine do what you want

              That's the Unix way.

              • No, the unix way is to be able to configure it yourself. It might be in a text file, but it's usually possible. I can't think of any time I've had to install a tool to configure the keyboard in any enterprise unix I am familiar with. In linux installing gconf-editor makes such things more convienent for gnome but you could do it in the gome settings with a text editor. That's it, one tool that I know of and you're done. Why would I compare OSX to a free operating system anyway, I would expect the free
                • I said Unix - OS X is based on a different Unix OS and I'm not comparing it to Linux.

                  A lot of things you take for granted on the CLI are based on bash scripts using all sorts of tiny scripts (sed, awk, grep). Even major packages work this way.

                  Either way, Apple is well-known for enforcing their preferred way of using a device (you're holding it wrong). If you are wanting to do something different, there's a long-established history of being forced to go the third-party route. If you're already using a Mac

                  • I was vaguely aware of it but I was only a very rarely mac user before and I thought I just wasn't understanding it and was just trying to get by. Now that I have a mac I'm spending a lot more time searching on things and finding out that, no, I wasn't missing anything; that's really all there is to OSX.
  • by LifesABeach ( 234436 ) on Tuesday January 10, 2017 @01:39PM (#53643047) Homepage
    What is this Hidden Safari Setting?
  • Does Firefox have this bug out of the box? I only use Firefox and the battery on my mac only lasts 4 hours. Tested three times now.
    • Does Firefox have this bug out of the box? I only use Firefox and the battery on my mac only lasts 4 hours. Tested three times now.

      Probably because the actual bug is in a Framework ALSO used by FireFox.

  • by Chris Mattern ( 191822 ) on Tuesday January 10, 2017 @02:04PM (#53643247)

    Sounds more like Apple's updating CR's review. Not much in there about what CR thinks about all this.

  • I've tried using that keyboard several times and it's just doesn't make the cut. If they would have left the thickness they could have had better keyboard travel and better battery life.

I'd rather just believe that it's done by little elves running around.

Working...