user20008: I just read this question and it made me wonder: when setting up such a game/comparison - shouldn't you try to level the computing capacity field? Or at least limit how much energy each "player" is allowed to use (and then let each team optimize within that limit). Otherwise, isn't it just a matter of building the biggest computer? 

user19877: It really comes down to the purpose of the event.
If it is a championship that is trying to determine who wrote the best software, and not just the strongest hardware/software combination, then you want to even up the hardware as best as possible so the software is the determining factor. The winner determines the best software.
If it is a demonstration or exhibition match, like the AlphaZero-Stockfish match, or the Kasparov-Deep Blue match years ago, and it is all about just crushing the opponent and winning at all costs, all bets are off, and you can expect to see massive, and outrageous, hardware builds.

user17425: Yes, you are right.  So computer competitions will have rules about compute to keep things level.  For example the chess.com computer championship  uses this specification:
CPU Engines:

CPUs: 2 x Intel Xeon Platinum 8168 @ 2.70 GHz 33 MB L3   
Cores: 48 physical (96 logical)   
RAM: 256GB DDR4-2666 ECC Registered RDIMM  
SSD: 2x Crucial MX300 (1TB) in RAID1   
OS: Windows Server 2016

GPU Engines:

GPU: 4x Tesla V100 (64 GB GPU memory) 
CPU: Intel(R) Xeon(R) CPU
E5-2686 v4 @ 2.30 GHz Cores: 16 physical (32 virtual) 
RAM: 256 GB


user20872: There are a number of reasonable things you could set as constant, which is probably why they have experts determine the rules in a fundamentally arbitrary but hopefully fair fashion. Energy is one, but some other options would be:

Hardware cost.
FLOPS (floating point operations per second). 

I suspect (although my info is limited) hardware cost and energy would help neural network neural based devices, while a flops based metric would benefit classical engines, because GPUs tend to provide great FLOPS/watt and FLOPS/$ -- metrics that CPUs sacrifice for the ability to go (relatively) quickly through branchy logic. 
An argument against using a simple metric is that it might induce behavior that doesn't advance the state of the field in an interesting direction. If you ask for performance/watt or performance/$, people start asking questions like "what about an FPGA or ASIC," and eventually it becomes a competition in brute force and low level tuning, rather than interesting new ideas. Plus people wouldn't be able tinker with the engines at home if they needed specific devices to do so. 
It is better to have experts in the field push toward interesting results, IMO. Although "energy limited on commodity hardware" might be fun. 

user15223: You absolutely should, which is why the two most important computer chess tournaments (CCC and TCEC) both make all competitors play on the "same" hardware. "Same" in inverted commas because the recent advent of neural-network engines like AlphaZero, which run best on GPUs, meant that it's no longer fair to play on exactly the same hardware for everyone.
You can see the result of not using equal hardware from the 2019 WCCC results, which Komodo won. Komodo is almost a hundred elo behind Stockfish when run on equal hardware (and Stockfish doesn't play in the WCCC, because it's not a commercial engine and doesn't have full-time staff to attend the event).
The situation right now is traditional engines run on the same CPUs, NN engines run on the same GPUs, and whenever traditional engines play against NN engines, they run on consensus fair hardware. "Fair" here is a very controversial word, because there are several different measures of what is fair that I'm glossing over. Suffice to say unless you are a hardware expert yourself, I'd just trust the admins of CCC and TCEC to use something that's fair.
Edit: to illustrate the impact of hardware, here's the list of hardware the competitors played at in the 2019 WCCC. Note Jonny ran on 1200 (!) cores, and it just barely edged out Ginkgo, which was running on 1/20 of the cores. In TCEC Season 14, the two engines played with the same hardware in Division 1, and Ginkgo comfortably finished ahead of Jonny in spite of multiple crashes (which happen because Ginkgo doesn't run well at 43 cores; they result in auto-losses).
You can also see that these games were played is Division 1, not even Division Premier. Many of the strongest engines in the world simply don't participate in WCCC. I don't know the exact reasons why, but I'm confident hardware is a major issue for participants. In turn, this lack of participation is partly why the community don't take the WCCC very seriously.

20008: <p>I just read <a href="https://chess.stackexchange.com/questions/28605/knowing-that-alphazero-beat-stockfish-8-28-wins-0-losses-and-72-draws-what">this question</a> and it made me wonder: when setting up such a game/comparison - shouldn't you try to level the computing capacity field? Or at least limit how much energy each "player" is allowed to use (and then let each team optimize within that limit). Otherwise, isn't it just a matter of building the biggest computer? </p>
 19877: <p>It really comes down to the purpose of the event.</p>

<p>If it is a championship that is trying to determine who wrote the best software, and not just the strongest hardware/software combination, then you want to even up the hardware as best as possible so the software is the determining factor. The winner determines the best software.</p>

<p>If it is a demonstration or exhibition match, like the AlphaZero-Stockfish match, or the Kasparov-Deep Blue match years ago, and it is all about just crushing the opponent and winning at all costs, all bets are off, and you can expect to see massive, and outrageous, hardware builds.</p>
 17425: <p>Yes, you are right.  So computer competitions will have rules about compute to keep things level.  For example the <a href="https://www.chess.com/news/view/announcing-the-new-computer-chess-championship" rel="nofollow noreferrer">chess.com computer championship</a>  uses this specification:</p>

<p>CPU Engines:</p>

<ul>
<li>CPUs: 2 x Intel Xeon Platinum 8168 @ 2.70 GHz 33 MB L3   </li>
<li>Cores: 48 physical (96 logical)   </li>
<li>RAM: 256GB DDR4-2666 ECC Registered RDIMM  </li>
<li>SSD: 2x Crucial MX300 (1TB) in RAID1   </li>
<li>OS: Windows Server 2016</li>
</ul>

<p>GPU Engines:</p>

<ul>
<li>GPU: 4x Tesla V100 (64 GB GPU memory) </li>
<li>CPU: Intel(R) Xeon(R) CPU</li>
<li>E5-2686 v4 @ 2.30 GHz Cores: 16 physical (32 virtual) </li>
<li>RAM: 256 GB</li>
</ul>
 20872: <p>There are a number of reasonable things you could set as constant, which is probably why they have experts determine the rules in a fundamentally arbitrary but hopefully fair fashion. Energy is one, but some other options would be:</p>

<ul>
<li><p>Hardware cost.</p></li>
<li><p>FLOPS (floating point operations per second). </p></li>
</ul>

<p>I suspect (although my info is limited) hardware cost and energy would help neural network neural based devices, while a flops based metric would benefit classical engines, because GPUs tend to provide great FLOPS/watt and FLOPS/$ -- metrics that CPUs sacrifice for the ability to go (relatively) quickly through branchy logic. </p>

<p>An argument against using a simple metric is that it might induce behavior that doesn't advance the state of the field in an interesting direction. If you ask for performance/watt or performance/$, people start asking questions like "what about an FPGA or ASIC," and eventually it becomes a competition in brute force and low level tuning, rather than interesting new ideas. Plus people wouldn't be able tinker with the engines at home if they needed specific devices to do so. </p>

<p>It is better to have experts in the field push toward interesting results, IMO. Although "energy limited on commodity hardware" might be fun. </p>
 15223: <p>You absolutely should, which is why the two most important computer chess tournaments (<a href="https://www.chess.com/computer-chess-championship" rel="nofollow noreferrer">CCC</a> and <a href="https://tcec-chess.com/#/" rel="nofollow noreferrer">TCEC</a>) both make all competitors play on the "same" hardware. "Same" in inverted commas because the recent advent of neural-network engines like AlphaZero, which run best on GPUs, meant that it's no longer fair to play on exactly the same hardware for everyone.</p>

<p>You can see the result of not using equal hardware from the 2019 WCCC results, which Komodo won. Komodo is <a href="https://ccrl.chessdom.com/ccrl/4040/" rel="nofollow noreferrer">almost a hundred elo</a> behind Stockfish when run on equal hardware (and Stockfish doesn't play in the WCCC, because it's not a commercial engine and doesn't have full-time staff to attend the event).</p>

<p>The situation right now is traditional engines run on the same CPUs, NN engines run on the same GPUs, and whenever traditional engines play against NN engines, they run on consensus fair hardware. "Fair" here is a very controversial word, because there are several different measures of what is fair that I'm glossing over. Suffice to say unless you are a hardware expert yourself, I'd just trust the admins of CCC and TCEC to use something that's fair.</p>

<p><strong>Edit</strong>: to illustrate the impact of hardware, here's the list of hardware the competitors played at in the <a href="https://www.chessprogramming.org/WCCC_2019#Participants" rel="nofollow noreferrer">2019 WCCC</a>. Note Jonny ran on 1200 (!) cores, and it just barely edged out Ginkgo, which was running on 1/20 of the cores. In <a href="https://www.tcec-chess.com/archive.html?season=14&amp;div=1&amp;game=1" rel="nofollow noreferrer">TCEC Season 14</a>, the two engines played with the same hardware in Division 1, and Ginkgo comfortably finished ahead of Jonny in spite of multiple crashes (which happen because Ginkgo doesn't run well at 43 cores; they result in auto-losses).</p>

<p>You can also see that these games were played is Division 1, not even Division Premier. Many of the strongest engines in the world simply don't participate in WCCC. I don't know the exact reasons why, but I'm confident hardware is a major issue for participants. In turn, this lack of participation is partly why the community don't take the WCCC very seriously.</p>
