Recently, Google came out with the new Pixel 2 devices with highest ever smartphone camera ranking on DxOMark of overall rating of 98! From the past 2-3 years DxOMark is one of the point to rate a smartphone camera and is getting used to second-from-second.
So DxOlabs measures and rates a bunch of different cameras, sensors, lenses and some DSLR’s all kind of stuff, and they have an entire separate section dedicated to smartphone cameras.
If you the DxOMark’s site, you’ll see this hierarchy of smartphone cameras, each of them with high number rating. This is somewhat accurate but also a bit misleading, using one single number to describe and to encompass all the complexities that go into a smartphone camera is a bit too broad, it’s just a little bit crazy in my opinion.
So DxO breaks it down into photo and video and then bricks each of those down into a bunch of sub categories. So, if you scroll down far enough in Dxo smartphone camera view underneath the rating, you’ll see the breakdown of the subsets scores for photos and for videos. Pixel 2 for example got a 99 for photos and 96 for video. But you notice these numbers clearly aren’t an average of the sub scores, so they have this weighting system essentially their own algorithm for combining all of the subs cores into overall score. So, it’s that score that gets published and plastered all over the internet. You’d have to actually go to DxoMark’s site and actually scroll all the way down and read through the review for that. And if you did, you would find that Dxo does some pretty legit, really solid testing. They try to be scientific and objective wherever they possibly can.
They have the bunch of indoor sets, controlled environments for testing things like noise performance, sharpness, color reproduction and detail in the exact same situation for every phone, so it’s repeatable every time. For testing bokeh with portrait mode they have a foreground subject and a background that’s a certain distance away. They measure auto focus speed and shutter speed lag with moving subjects the whole deal and they also have a bunch of slightly less exact but still pretty telling outdoor tests for things like high dynamic range and seeing color casts etc. And essentially what they do is they take the new image from the smartphone camera and match it with the closest one on a scale with what’s called an image quality ruler. They have an existing scale with a bunch of different levels of noise for example and they match the new sample with where it falls on that scale. So that’s how they turn a qualitative measurement into a quantitative value. So, DxO is smart and to make more easy to headline they summarized everything into one big overall score, which like I mentioned is not the average of the sub scores, So, we can assume they’re weighted roughly in that order with exposure and contrast being the most important, auto focus and color being almost important and this decision of the order that they weight things is subjective. The order that they decide to rank these and the amount of weight that they put into each one is entirely up to them and that’s why you shouldn’t put all your purchase decision into just the overall score, you should look past the overall rating. Compiling everything into a overall sore can be misleading because different characteristics matter to different people, some people take a lot of low light photos or take a lot of photos of people, a lot of selfies, or lot of landscapes shots.
Here’s the perfect example, the Galaxy Note 8 got an overall score of 94, the Pixel 2 got an overall score of 98, you take a lot of portrait photos, which one you going to pick? You’re going to pick the Galaxy Note 8, you gonna throughout the overall score and pick the one that has a dedicated second camera for portrait photos. If you look at the photos sub scores, it got a way higher score for zoom and bokeh, but these are pretty new things and so they are not really weighted this heavily so they really as much of an effect on that overall score and as a result the phone that you would pick because of it’s better for you like got a lower overall sore on the DxOMark scale.
At honestly most of these smartphone cameras are great when you talk about things like contrast and color and dynamic range to varying degrees obviously, but the top ones are all pretty good, the bigger difference comes with the different modes with things like portrait mode and long exposure and panorama stitching and all that, those are what makes bigger difference to the user experience, especially when you see a phone like Pixel 2 doing portrait mode mostly within the software versus with a second camera.
Now here’s another reason you may want to look past the overall score. DxO Labs is a consulting company as a testing company. For a fee they will work with the smart phone manufacturers before their phones comes out to create a better camera. So, they will provide their testing software, their hardware and their testing methodology so that you can calibrate your sensor and your image processing pipeline to produce better photos, but also kind of to just do better on their tests. It’s pretty easy to look at this and think they were just working with you to get a higher score.
Another quick point, with the Pixel 2 getting a 98 you’re probably wondering, what’s the best score they have ever given to any camera? That would be 108! And that was the sensor score for the Red Helium 8K with the super 35 sensor.
So, in summary, DxOmark does real testing, YES!! But an overall number to describe everything that goes into a camera is a little too broad, YES!! And then DxO also does consulting to improve cameras based on their own algorithms and testing, so maybe take those numbers with the grain of salt. And the score is just happened to be around 100 right now but that’s purely a coincidence, they are not out of 100.
So, we have learned here is ratings are nice clean even number to point to one being better than the other, but if you actually want to make a purchase decision you got to look a little bit deeper than the DxoMark rating, plus they’re all pretty incredible anyway.