We’re stepping a little out of our comfort zone here but UL (you might know them as Futuremark), the company who makes numerous benchmarks for various platforms including the popular 3DMark software, got in touch to inform us that yet another mobile phone company has been caught cheating and artificially inflating their benchmarking scores.
Naturally we hate this kind of misrepresentation as much as anyone else, especially since there are a significant number of consumers who will purchase a device purely based on their ranking. If left unchecked consumers will ultimately pay the price and so we felt it is important to pass this info on. Share it far and wide people and stop your friends and relatives from being ripped off.
The full release from UL is listed below:
After Huawei was caught unfairly boosting benchmark scores last month, we were contacted by reviewers from Tech2 who had similar concerns about Oppo smartphones.
Extensive testing by Tech2 found that the flagship Oppo Find X smartphone produced artificially high and misleading benchmark scores.
UL has clear rules for manufacturers that govern how a platform can interact with its benchmarking software. The purpose of these rules is to ensure that users get benchmark results that accurately reflect the true performance of the device for apps and games.
After reviewing Tech2’s benchmark results, and conducting testing in our own lab, we have decided to delist the affected models and remove them from our performance rankings.
Which models have been delisted?
Based on Tech2’s testing data and reporting, we have delisted the Oppo Find X. Based on our own testing, we have delisted the Oppo F7.
The Oppo Find X was ranked #4 in our list of the Best Smartphones for 3DMark Sling Shot Extreme performance. It now appears unranked, and without a score, at the bottom of our rankings. 3DMark scores from delisted devices should not be used to compare models.
Why have you delisted these models?
Each model was tested with the public version of 3DMark, available from Google Play, and a private version of 3DMark that is not available to the public or manufacturers.
We found that the scores from the public 3DMark app were up to 41% higher than the scores from the private app, even though the tests are identical.
The difference in scores tells us that the devices are simply recognizing the 3DMark app by name rather than adapting to the type of work in the test.
This kind of detection and optimization is forbidden by our rules for manufacturers.
We contacted Oppo with our findings. Oppo admits that the devices are detecting the public version of the app,
“When we detect that the user is running applications like games or 3D Benchmarks that require high performance, we allow the SoC to run at full speed for the smoothest experience.
Oppo explained how its devices manage power and performance for other apps that are not recognised,
“For unknown applications, the system will adopt the default power optimization strategy…After the user has not actively operated for 5 to 10 seconds, the device limits the system performance to 70% to 80% of the maximum performance (according to different platforms)…When there is a user operation, it will immediately cancel the performance limit, to ensure that the user experience is not affected.”
In practice, this means that it is possible to improve performance in most apps by continually tapping on the screen. However, this has no effect on the public 3DMark app. Nor do we consider such tapping an accurate way to benchmark a device due to the variability in performance that it introduces.
Fortunately, Oppo also said that it is looking to improve on this approach for the benefit of its users.
“At the same time, we are working on upgrading the system, and strive to distinguish between the requirements of undetected apps or the subjective needs of users.”
Under our rules, optimizing performance by detecting heavy computation demands is allowed. Simply detecting the benchmark app by name is not. A device must run the benchmark as if it were any other application.
We’re committed to creating benchmarks you can trust
- 3DMark is protected by rules for manufacturers.
- 3DMark has public documentation that explains what each test measures and how the scores are calculated.
- Our Best Smartphones list only shows publicly available models. We never list or leak unverified scores from pre-release hardware.
