A Timely Discovery: Examining Our AMD 2nd Gen Ryzen Results
by Ian Cutress & Ryan Smith on April 25, 2018 11:15 AM ESTLast week, we published our AMD 2nd Gen Ryzen Deep Dive, covering our testing and analysis of the latest generation of processors to come out from AMD. Highlights of the new products included better cache latencies, faster memory support, an increase in IPC, an overall performance gain over the first generation products, new power management methods for turbo frequencies, and very competitive pricing.
In our review, we had a change in some of the testing. The big differences in our testing for this review was two-fold: the jump from Windows 10 Pro RS2 to Windows 10 Pro RS3, and the inclusion of the Spectre and Meltdown patches to mitigate the potential security issues. These patches are still being rolled out by motherboard manufacturers, with the latest platforms being first in that queue. For our review, we tested the new processors with the latest OS updates and microcode updates, as well as re-testing the Intel Coffee Lake processors as well. Due to time restrictions, the older Ryzen 1000-series results were used.
Due to the tight deadline of our testing and results, we pushed both our CPU and gaming tests live without as much formal analysis as we typically like to do. All the parts were competitive, however it quickly became clear that some of our results were not aligned with those from other media. Initially we were under the impression that this was as a result of the Spectre and Meltdown (or Smeltdown) updates, as we were one of the few media outlets to go back and perform retesting under the new standard.
Nonetheless, we decided to take an extensive internal audit of our testing to ensure that our results were accurate and completely reproducible. Or, failing that, understanding why our results differed. No stone was left un-turned: hardware, software, firmware, tweaks, and code. As a result of that process we believe we have found the reason for our testing being so different from the results of others, and interestingly it opened a sizable can of worms we were not expecting.
An extract from our Power testing script
What our testing identified is that the source of the issue is actually down to timers. Windows uses timers for many things, such as synchronization or ensuring linearity, and there are sets of software relating to monitoring and overclocking that require the timer with the most granularity - specifically they often require the High Precision Event Timer (HPET). HPET is very important, especially when it comes to determining if 'one second' of PC time is the equivalent to 'one second' of real-world time - the way that Windows 8 and Windows 10 implements their timing strategy, compared to Windows 7, means that in rare circumstances the system time can be liable to clock shift over time. This is often highly dependent on how the motherboard manufacturer implements certain settings. HPET is a motherboard-level timer that, as the name implies, offers a very high level of timer precision beyond what other PC timers can provide, and can mitigate this issue. This timer has been shipping in PCs for over a decade, and under normal circumstances it should not be anything but a boon to Windows.
However, it sadly appears that reality diverges from theory – sometimes extensively so – and that our CPU benchmarks for the Ryzen 2000-series review were caught in the middle. Instead of being a benefit to testing, what our investigation found is that when HPET is forced as the sole system timer, it can sometimes a hindrance to system performance, particularly gaming performance. Worse, because HPET is implemented differently on different platforms, the actual impact of enabling it isn't even consistent across vendors. Meaning that the effects of using HPET can vary from system to system, as well as the implementation.
And that brings us to the state HPET, our Ryzen 2000-series review, and CPU benchmarking in general. As we'll cover in the next few pages, HPET plays a very necessary and often very beneficial role in system timer accuracy; a role important enough that it's not desirable to completely disable HPET – and indeed in many systems this isn't even possible – all the while certain classes of software such as overclocking & monitoring software may even require it. However for a few different reasons it can also be a drain on system performance, and as a result HPET shouldn't always be used. So let's dive into the subject of hardware timers, precision, Smeltdown, and how it all came together to make a perfect storm of volatility for our Ryzen 2000-series review.
242 Comments
View All Comments
lenghui - Thursday, April 26, 2018 - link
Sounds like it's time to dig out that good old stopwatch from storage.haplo602 - Thursday, April 26, 2018 - link
And when MS changes HPET default to forced if detected ... then you are screwed again and have to retest ...You are at a dead-end actually. You are switching to HPET off (effectively) because it highly favors Intel in a few benchmarks yet AMD is mostly unaffected. Will you change that if the tables turn in the future again ?
Come on ... it is more about consistency. Your HPET forced mode definitely highlighted an issue with Intel chips yet instead of hitting on Intel about the issue you are changing your settings ...
Maxiking - Thursday, April 26, 2018 - link
"when MS changes HPET default to forced if detected"" if games start utilizing more cores "
It is a though job being an AMD fan these days, just "ifs" and "whens" all the time.
Maxiking - Thursday, April 26, 2018 - link
***tough***. damn you, autocorrecteva02langley - Thursday, April 26, 2018 - link
He's entirely right. HPET is an issue on Intel shoulder as of now. How can we be sure that without HPET on, Intel benchmarks are accurate?Also, you cannot turn off something in the BIOS that is supposed to be on, as mention by Intel, just because you want to give the crown to one manufacturer or the other. By the way, we are talking about 1080p benchmarks with a 1080 GTX. 60 Hz is irrelevant since a RX 580 can render it, it leaves only 1080p @ 144Hz.
Also, what about new games since these results seem to be linked to old games?
You don't get 40% more performance by switching something on and off in the BIOS. If it does, than something needs to be fixed.
RafaelHerschel - Thursday, April 26, 2018 - link
Well, I once disable my boot drive in the BIOS and experienced a 100% slow down. I'm tempted to agree with you and feel that something needs to be fixed so that my system works without a boot drive, but other people don't seem to agree. Opinions...Then there is the time when I disabled my NVDIA GPU in the BIOS of my laptop. Massive performance drop in games... Not good. Sad. Needs fixing.
jor5 - Thursday, April 26, 2018 - link
tl;dr - "We completely screwed up our review and made a show of ourselves - but we're not apologising"AndrewJacksonZA - Thursday, April 26, 2018 - link
You don't analyze data much in your day job, do you?SkyBill40 - Friday, April 27, 2018 - link
What would be more accurate than your statement is something like:"We found that our previous data contained some pretty significant inaccuracies and to be thorough, we're re-testing and improving our testing methodologies as a whole and explaining as such at length for the sake of transparency.Thanks for being patient with us."
OrphanageExplosion - Thursday, April 26, 2018 - link
What I don't understand is how the original gaming data ended up being published at all.The faulty results that were published were entirely at odds with the data supplied by AMD itself (which we've all seen - even prior to the reviews dropping, if you've been following the leaks). Surely if Ryzen 2000 was so much faster than Coffee Lake, they would have been shouting this from the rooftops - gaming is, after all, one of the few weaknesses Ryzen has. AMD's no-doubt massaged results were a ton more accurate than Anandtech's - madness.
Not only that but the Anand results showed a massive increase over Ryzen 1000 - which simply isn't feasible for what is effectively a mild refresh. Meanwhile, the results also showed Ryzen 5 handily beating an 8700K... surely you must have realised that something wasn't right at that point? Utterly baffling and calls into question your approach generally.
This is a major hit to credibility. I mean, if you're going to publish a CPU 'deep dive', surely you need to actually analyse the data and be ready to question your results rather than just hitting the publish button?