B&C: Hi Oliver, first of all, thank you for this interview. Can you explain to our readers what your position and roles are at Futuremark?

OB: Thank you for the opportunity. My role is President at Futuremark.

I spend most of my time meeting with hardware and software companies around the world. We are fortunate to have some of the biggest names in technology in our Benchmark Development Program. BDP members include AMD, Dell, HP, Imagination Technologies, Intel, NVIDIA, Microsoft, Qualcomm, Samsung and many others. 

We work with our BDP members to design, develop and test our benchmarks. I spend time with their product managers and engineers talking about the future direction of PC and mobile device technology. You simply cannot create relevant benchmarks without this level of expert insight and close cooperation.

B&C: 3DMark is the most popular benchmark for the PC market, but in the Mobile market Futuremark is a new entry. AnTuTu is the main rival: today it's the most used software to benchmark Smartphones and Tablets. What do you offer more than AnTuTu?

OB: Futuremark has been making benchmarks for mobile platforms since 2004, though these have mainly been professional tools for our business customers in manufacturing industries.   

3DMark came to Android in April and quickly passed 250,000 downloads. More importantly, we've seen many press reviewers at websites both big and small choose 3DMark as their preferred gaming benchmark for mobile devices. And of course, 3DMark has been widely adopted at all levels of the mobile industry, from top tier manufacturers and network providers down to component suppliers and resellers. 

If we are a little late to the party, it's because we chose not to create a standalone benchmark for each mobile platform. A key factor in our decision to make 3DMark cross-platform was the large number of emails we got from editors and reviewers who were desperate, not only for a decent mobile benchmark they could trust, but for a benchmark that would let them compare Windows, Android and iOS devices directly. 

We do not release benchmarks in a beta state, since people rely on the accuracy and consistency of the scores from day one. So if it takes us a longer to create a new benchmark we're fine with that as we know we are doing it properly: by working with the technology leaders in the industry, by having a cooperative and transparent development process, and by doing the right amount of testing before releasing to the public. 

This is not a commentary on others who may be doing similar work. It is simply the process we follow to produce the best benchmarks for our customers and the market at large.

B&C: Today we know that AnTuTu uses two different compilers to support ARM and x86 SoCs (GCC and ICC, respectively), and also we know that the results from ARM and x86 SoCs are not comparable. What compiler do you use for your Mobile software? And why? Are the results from ARM and x86 SoCs comparable?

OB: Yes, you absolutely can compare 3DMark scores from Windows, Android, iOS and Windows RT devices.  

We design our benchmarks to measure real-world performance by testing the device as a whole. Performance differences caused by development tools, such as compilers, are inherent features of the platforms and should be included in real-world comparisons.  

We call 3DMark "the gamer's benchmark" because it shows how well your PC or mobile device is able to handle the demands of a graphically intense game. When we create 3DMark tests, we use the same tools and techniques commonly used by game developers, for example DirectX on Windows and OpenGL ES 2.0 on Android. Likewise, we use Microsoft's compiler on Windows and GCC on Android, since they are widely adopted by developers on those platforms.

B&C: Recently Samsung has cheated at AnTuTu with its latest Galaxy smartphones, and AnTuTuLabs changed quickly the ICC implementation in the revision 3.3 of AnTuTu. The interview with VR-Zone seems to be a veiled attack to AnTuTu about these instances. But Futuremark was accused to ganged up with some manufacturers too. Do you remember the 3DMark-nVidia Gate way back in 2003, for example? You confirmed in these times: "nVidia's driver design is an application-specific optimization and not a cheat". This sentence made a fuss. Seems that it's a common practise cheating at benchmarks. Are you worried about it? Samsung, or another company, could cheat at your Mobile benchmarks too.

OB: At Futuremark we take the integrity of our benchmarks very seriously. Every part of our business and development process is designed to ensure that our benchmarks are impartial and can be trusted by industry, press and home users. 

Every company wants to "win" at 3DMark, but I don't believe any of the companies we work with would risk outright cheating to improve their 3DMark scores. They would certainly be caught and no company needs that kind of negative publicity - your example proves how long people remember.  

Besides, our development process makes it practically impossible for any BDP member to manipulate the benchmark in their favour. All BDP members are involved throughout development and receive regular builds for testing. The benchmark source code is also available for them to review at any time. If one company's results looked out of line, the other members would quickly and loudly speak up! 

There is a strong incentive for manufacturers to improve their 3DMark scores by improving their drivers. The key point is that those optimisations must benefit all applications and not target our benchmarks specifically. 

A benchmark is only valid if the hardware performs all of the work without skipping or modifying the content in any way. We have a strict driver approval policy that states what is and isn't allowed. This policy is clearly communicated to the member companies of our Benchmark Development Program and is also available on our website. 

We test every major WHQL driver release to make sure that it does not unfairly influence or manipulate the 3DMark tests or scores. 3DMark will only give you a valid result if you are using a driver that Futuremark has tested and approved. When you use 3DMark you can be sure that the scores are fair.

B&C: Smartphones and Tablets will have more and more cores, like the PC CPUs. Is it essential to have so many cores within these devices, in your opinion? Or is it only a Marketing choice?

OB: The unstoppable march of progress, right? Computational power goes up and costs go down. More cores can also reduce power consumption and improve battery life. The mobile space is incredibly competitive and product life spans are short. Manufacturers can experiment and test new designs. If there is a killer mobile app made possible by that extra power, and consumers demand it, then why not?

B&C: In this interview, Tamás Miklós, Lead Developer of AIDA64, says that the documentation for the developers is inadequate in the PC industry (and we agree with him). Do you have the same problem in the Mobile industry? ARM and Qualcomm support effectively the development community. Do Intel and nVidia the same?

OB: I'm sorry, but I don't know what the quality of documentation and support is like for other developers. We work closely with those companies and are fortunate to have direct access to their engineers and developers when we need it.

B&C: By the way, what do you think about the HWBOT's decision to no longer accept Windows 8 submissions? Which is the main responsible for this bug, in your opinion? Intel or Microsoft?

OB: I think it is worth repeating that the exploit has no practical benefit for manufacturers or gamers. It is not something you could stumble upon by accident, nor does it serve any purpose other than to create a false benchmark score.  

The good news is that we already have a way to detect this issue and we'll be including it in our next update. In the meantime, there is no reason to stop benchmarking on Windows 8 and no cause to doubt 3DMark scores from reviews or your own system.

B&C: A way back, to review hardware components, IT magazines and sites used to utilize only benchmark suites like Business Winstone, CPUMark and PCMark. Today the reviewers and the final users prefer to test their components by the real applications (for example video game and compression software integrated benchmarks). In the near future the benchmark suites like 3DMark could disappear?

OB: I am so glad you asked me this question! It's good to use a variety of tools for benchmarking, but it's important to understand their strengths and weaknesses.

If you only want to know how game X runs on your system, the sensible thing is to run game X. However, it's not always as simple as that.  

Let's say you build a new PC and want to test its gaming chops. So you fire up your game of choice and start playing. How do you know if what you are seeing is normal? How do you know if your hardware is performing correctly? If your game has a built-in benchmark, where do you go to compare your results?  

Many games cannot be used reliably as impartial benchmarks since hardware vendors make game-specific optimisations to their drivers. If you unknowingly base your upgrade decision on a game that is heavily optimised for one vendor's hardware, you might miss out on the best performing choice for other games.  

With 3DMark you can see how your system performs under consistent and easily repeated test conditions. It's like testing your system with a hundred different games and averaging the results, only faster, easier and cheaper.  

3DMark shows you how your score compares with other people with the same hardware. You can see right away if there is something wrong with your setup. We have millions of 3DMark results in our database so it's easy to see how your score would improve by changing your graphics card or processor.  

You can compare 3DMark scores from different brands of hardware without concern because the test content is the same in every case, something we guarantee with our Driver Approval Policy. And if you want cross-platform comparisons, then a dedicated benchmark like 3DMark is really the only way to go.

B&C: 3DMark Vantage is rarely used by the reviewers and the enthusiast community: can you tell us what is wrong with this benchmark?

OB: I don't think there is anything wrong with 3DMark Vantage. Since its release in 2008, however, PC game engines have largely jumped from DirectX 9 straight to DirectX 11. There is not much demand for a native DirectX 10 benchmark for gaming PCs today.  

For modern mobile devices with capable hardware, I recommend using the Cloud Gate benchmark in our latest 3DMark. Cloud Gate uses DirectX 11 feature level 10, the modern equivalent of DirectX 10.

B&C: When you write a benchmark how do you choose the features to test?

OB: The hardest part of creating a benchmark is designing the workloads so that the test is representative of real-world applications, such as games in the case of 3DMark.  

It all starts with something we call the Technical Specification Proposal. This document describes our plan for the benchmark in great detail. The Spec for 3DMark on Android ran to 25 pages, for example. The Technical Specification Proposal details the features of the engine, the rendering techniques and post-processing effects that will be used, how scores will be calculated and so on. 

The Spec is shared with our BDP members for their comments and input. This is how we ensure that the benchmark reflects the current capabilities of hardware while anticipating the advances expected in the coming years. 

Once the Technical Specification is agreed upon, development begins. Most of the development work is focused on creating the engine and supporting technology, followed by a lot of tuning and balancing to ensure that the benchmark provides accurate and relevant insights into hardware performance.

B&C: By the way, your CPU benchmarks test the Intel and AMD architectures (SMT vs CMT) in the same way? Or did you implement a specific test for each one?

OB: We run the same binary on all CPUs. There are no separate code paths for different architectures. In general, we don't do vendor-specific optimisations since that would go against the aim of producing neutral benchmarks.

B&C: Thank you Oliver, it has been a pleasure talking with you.

OB: Thank you.

Pubblicato in: Articolo, Software
Tags: 3dmark , android , antutu , baltuch , benchmark , futuremark , mobile , windows

Autore: Gian Maria Forni
Esperto in: Esperto di mercati e CPU

Sebbene sia laureato in Lettere e Filosofia, indirizzo Storia Contemporanea, e scriva per quotidiani e riviste di tale settore, ha sempre avuto la passione per l'informatica ed ha collaborato quale moderatore in importanti forum del settore

2166 news	126 articoli

     Questo indirizzo email è protetto dagli spambots. E' necessario abilitare JavaScript per vederlo.