Skin ADV

Some hours ago, during a discussion on Twitter, it came to light that a lot of people don’t understand the importance of a complete and clear review methodology.

Even geeks, engineers, journalists seem to be unable to see how bad an incomplete testing method could be. We know that in scientific fields a rigorous methodology is needed. According to scientific studies, a test shall be capable of being replicated: “Reliability is the degree of consistency of a measure. A test will be reliable when it gives the same repeated result under the same conditions” (Source).

To do this, we have to know exactly how the experiment was done. About a hardware review, we have to know every single hardware and software component was used, and which version of it. Thanks to these informations, if we want, we could be able to replicate the review tests and we could be able to confirm the achieved results.

So, during that Twitter discussion, someone questioned my results of this review, where I tested the power consumption of the Intel Core i9-9900K @ stock setting. As you can see, I meticulously described the configuration and the methodology I used. Also, I used default settings for everything: “out-of-the-box settings”. I did that in order to see how the PC of "average Joe" perform. We all know that average Joe purchases the pc, puts the plug and pushes the power button. Stop.

People complained that I had to change BIOS options in order to use a lower Power Limit 2, according to Intel specifications. The problem is that Intel itself allowed Asus, Gigabyte, MSI and other manufacturers to use that setting on high end main boards: PL2 @ Auto (About 210W). According to Intel, it’s not overclock: “The values on the data sheet for 1.52 V and 138 A are the maximum values that can be used on the Intel® processor” (Source). So, main boards manufacturers didn’t violate the specification. They just use the max PL2 value allowed.

But this is not the problem.

The problem concerns the methodology of the tests. These people, if they think my results are wrong, could remake the same tests with the same hardware and the same software. Despite that, they attacked B&C saying that other sites are more reliable. It could be. We absolutely don’t think B&C is the best site. However, I think B&C is the site with the best and more clear methodology tables. Here there are some examples.

 

Some Intel Core i9-9900 reviews (I can't list every 9900K review made)

 

Site Anandtech Bits'n'Chips Guru3D TechPowerup TechSpot TomsHardware
Mainboard Yes Yes No Yes Yes Yes
BIOS Yes Yes Yes Yes No No
CPU Cooler Yes Yes No No No Yes
Thermal compound No No No No No No
Memory Yes Yes No Yes Yes Yes
Memory Timings No Yes Yes Yes No No
SSD/HDD Yes Yes No Yes No Yes
GPU Yes Yes Yes Yes Yes Yes
PSU Yes Yes Yes Yes No Yes
Operating System Yes Yes Yes Yes No Yes
Version OS Yes Yes No Yes No No
Build OS No Yes No No No No
Driver Chipset No Yes No No No No
Driver GPU No Yes No Yes No No
Software Yes Yes Yes Yes Yes Yes
Software version Yes Yes No No No Yes

 

As you can see, B&C tests can be replicated without problems. We don’t fear about that. On the other hand, we could have a problem if we want to replicate the tests of other sites.

In conclusion, we think that review sites have to improve their methodology, in order to create more reliable reviews. Currently, the reviews are far far away from being considered “scientific”.

Please note: It's not an attack to the other sites. B&C is a small site, attended in our spare time, so even we want to read some "scientific" reviews. We are end users in first place, not journalists. ;)

Delicious
Submit to Digg
StumbleUpon
Gian Maria Forni
Autore: Gian Maria Forni
Esperto in:
Esperto di mercati e CPU
Sebbene sia laureato in Lettere e Filosofia, indirizzo Storia Contemporanea, e scriva per quotidiani e riviste di tale settore, ha sempre avuto la passione per l'informatica ed ha collaborato quale moderatore in importanti forum del settore
Notizie postate da Gian Maria ForniArticoli postati da Gian Maria Forni
2990
news
212
articoli
    

Ultime dal forum



[Official Thread] AMD GPU Navi RX5000 (673 messaggi)
Ultimo messaggio di: Blobay (16/09/2019 20:16)

[Official Topic] AMD "Zen" (2514 messaggi)
Ultimo messaggio di: Fottemberg (16/09/2019 19:41)

Nuova configurazione (11 messaggi)
Ultimo messaggio di: Masciale (10/09/2019 12:22)

Che state ascoltando in questo momento? (32 messaggi)
Ultimo messaggio di: Sasha (08/09/2019 22:23)

Contenitore di news varie (353 messaggi)
Ultimo messaggio di: Fottemberg (06/09/2019 12:11)

Una barzelletta allunga la vita (6 messaggi)
Ultimo messaggio di: Masciale (05/09/2019 21:22)

Problema NAS - aMule (17 messaggi)
Ultimo messaggio di: Bivvoz (04/09/2019 20:24)

Le recensioni e gli articoli che pubblicheremo sul nostro portale (69 messaggi)
Ultimo messaggio di: Fottemberg (03/09/2019 11:27)

Il meglio dal forum



[Guida] all'overclock di AMD Ryzen.
Il thread per spiegare ai novizi come overclockare le CPU Ryzen, per raccogliere pareri e per scambiarci suggerimenti.

[Official Topic] AMD APU Kabini, Temash, Kaveri e... Carrizo.
Discutiamo le ultime novità sulle APU AMD basate su microarchitettura Bulldozer e Jaguar.

[Waiting for] AMD GPU Polaris & Vega (ex-Arctic Islands).
Discutiamo le ultime novità sulle GPU Polaris di AMD, senza dimenticarci di Vega!

[Official Topic] Nvidia Pascal GPU.
Discutiamo le ultime novità sulle GPU Pascal di NVIDIA e su quanto ci aspetta in futuro!

[Official Topic] AMD "Zen".
Discutiamo le ultime novità sulla microarchitettura x86 Zen di AMD e le CPU/APU in arrivo.

Creative Sound Blaster e Windows 10/8.1/8/7/Vista.
Dedicato a tutti i possessori delle schede audio di Creative Labs!

B&C e la questone "degli articoli prezzolati".
Perché B&C è un portale diverso?