Match Deep Hiarcs 15.2 vs Deep Hiarcs 15.3

[Mag]

Match Deep Hiarcs 15.2 x64 - Deep Hiarcs 15.3 x64

Match result: 8 - 5 - 87 (100 games)



The match was played on the following equipment:

processor - AMD Ryzen Threadripper 2990WX 3.0 GHz
hard drive - Samsung 970 EVO Plus NVMe M.2 SSD 2 TB
memory ram - Kingston HyperX Fury 128 GB DDR4 2400MHz CL15
graphics card - Two graphics cards - ASUS Tuf GeForce RTX 3080Ti Gaming LHR 12GB
monitor - Nec MultiSync EA245WMi
keyboard - A4Tech KV-300H Slim
mouse - DeathAdder Essential
operating system - Microsoft Windows 10 Professional 64 bit

During the match I was using the book Balsa_270423 - 5 minutes + 3 seconds. I have used 6-men Syzygy table bases. The match was played on the GUI Cutochess 1.2 by Guenther Simon.

Setting GUI Cutochess 1.2 by Guenther Simon

Tournament type:
Round Robin=On

Rounds:
Rounds=100
Save unfinished games=On
Swap sides=On

Games:
Time Control:
Time=5,00
Increment=3,00sec

Opening suite:
PGN/EPD file=D:/Book/Balsa_270423.pgn
Depth=16 plies
Opening book=Random

Draw adjudication:
Move number=35
Move count=5
Score=15 cp

Resign adjudication:
Move count=5
Score=1000 cp
Two-Sided=On

Game Length:
Limit=Off

Tablebase adjudication=On

Setting the engines during the game.

Deep Hiarcs 15.2 x64

Threads = 16
CPU Usage = 100
Swindle Opponent = Off
Detect Fortress = Off
Syzygy Tablebases = On
Syzygy Proble Depth = 2
Syzygy Proble Limit = 6
Syzygy 50 Moves Rule = On
Hash = 8192
Retain Hash = On
Position Learning = On
Combinations = On
Use Null Move Pruning = On
Playing Style = Aggressive

Deep Hiarcs 15.3 x64

Threads = 16
CPU Usage = 100
Swindle Opponent = Off
Detect Fortress = Off
Syzygy Tablebases = On
Syzygy Proble Depth = 2
Syzygy Proble Limit = 6
Syzygy 50 Moves Rule = On
Hash = 8192
Retain Hash = On
Position Learning = On
Combinations = On
Use Null Move Pruning = On
Playing Style = Aggressive

A short comment on the match - as you can see, the previous version of the program is better than the new one. I have attached all the games of the match.

Regards
Mag

[Tibono2]

Hi, thanks for the experimental data.
Indeed an interesting indicator but, TMHO, you cannot draw a conclusion from such a 121 match. More variety of opponents would make more sense.
Kind rgds,
Tibono

[Harvey Williamson]

Hi, thanks for the experimental data.
Indeed an interesting indicator but, TMHO, you cannot draw a conclusion from such a 121 match. More variety of opponents would make more sense.
Kind rgds,
Tibono

I agree the match is too short. I would prefer the test with default settings - playing style Active not Aggressive

[Mag]

I agree the match is too short. I would prefer the test with default settings - playing style Active not Aggressive

There is no need to play a longer match. The engines are perfectly aligned, there is no need to change to Active - because it will not help.

[Carl Bicknell]

I agree the match is too short. I would prefer the test with default settings - playing style Active not Aggressive

There is no need to play a longer match. The engines are perfectly aligned, there is no need to change to Active - because it will not help.

Ofc you need a longer match, it's basic maths

[Tschens]

If there would be a significant difference in Elo, 100 games would be more than enough.

[Fran66]

Hi,

Is 15.2 stronger than 15.3?

[Mark Uniacke]

I agree the match is too short. I would prefer the test with default settings - playing style Active not Aggressive

There is no need to play a longer match. The engines are perfectly aligned, there is no need to change to Active - because it will not help.

I doubt that. When two engines are close in strength 100 games is completely inadequate and often gives misleading results, play at least 40K games and even then it's often not enough to be sure. Statistically your 100 games gives less than 60% confidence which is not much better than tosing a coin!

You might feel you are 100% confident your little test is correct but the fact is real confidence is obtained from mathematical analysis based on the results under equal conditions over many thousands of games. The normal standard for deciding one engine is stronger than another is having at least a 95% confidence the result could not have happened by chance. That is why for each engine change I run between 20,000 and 140,000+ games under precise conditions. Even then sometimes more games are required.

H15.3 was played against H15.2 before release over 140K games from different unique positions with reverse colours under equal conditions in multiple matches and proved to be stronger, in fact on every machine out of the 15 machines where a match was run between H15.3 and H15.2 resulted in a match win by H15.3, cumulatively by well over 1000 games. Statistically I am confident H15.3 is slightly stronger than H15.2 based on my tests.

YMMV given different conditions and you may prefer H15.2 but I think H15.3 et al is the better engine and gives more detailed analysis.

[Fran66]

Very Interesting, Mark. It is clear then that this match is not useful to draw any conclusions.


Best regards!

[gads]

I agree the match is too short. I would prefer the test with default settings - playing style Active not Aggressive

There is no need to play a longer match. The engines are perfectly aligned, there is no need to change to Active - because it will not help.

I doubt that. When two engines are close in strength 100 games is completely inadequate and often gives misleading results, play at least 40K games and even then it's often not enough to be sure. Statistically your 100 games gives less than 60% confidence which is not much better than tosing a coin!

You might feel you are 100% confident your little test is correct but the fact is real confidence is obtained from mathematical analysis based on the results under equal conditions over many thousands of games. The normal standard for deciding one engine is stronger than another is having at least a 95% confidence the result could not have happened by chance. That is why for each engine change I run between 20,000 and 140,000+ games under precise conditions. Even then sometimes more games are required.

H15.3 was played against H15.2 before release over 140K games from different unique positions with reverse colours under equal conditions in multiple matches and proved to be stronger, in fact on every machine out of the 15 machines where a match was run between H15.3 and H15.2 resulted in a match win by H15.3, cumulatively by well over 1000 games. Statistically I am confident H15.3 is slightly stronger than H15.2 based on my tests.

YMMV given different conditions and you may prefer H15.2 but I think H15.3 et al is the better engine and gives more detailed analysis.

Hi Mark, can you please share the rating improvement figure from your extensive testing? Thanks.

[Mark Uniacke]

The tests showed about +3 Elo.

[herO]

The tests showed about +3 Elo.

Now it makes even more sense. If the difference in strength between two engines is very small, it's indeed necessary to conduct a large number of games under strictly defined conditions, which must be the same for both engines. If the new engine is 3 ELO stronger than the previous version, then it's clear that 100 games are really not enough. The result of user Mag only proves statistical deviation, so his test practically shows nothing. His test is suitable for chess engines where their strength is not approximately the same. For example, I can conduct a chess tournament between Deep Hiarcs 15.3 and Chessmaster, and after 100 games, we can really say which of the chess engines is stronger because Hiarcs will win the vast majority of games and draw in exceptional cases.

[Peter Grayson]

The tests showed about +3 Elo.

under strictly defined conditions, which must be the same for both engines.

A key element of testing to achieve some meaningful results and information between two engines is to ensure both engines play the same opening with both colours. At least then if differing results are achieved for a particular opening line, a comparison at the move deviation can be made and perhaps with knowledge of how the engine should work an understanding of what happened may prove useful. Thousands of different openings where only one engine plays the opening line offers no real comparison of performance. Let alone 100 games where the opening lines were not matched.

With the HIARCS engine attempting to maintain a human like characteristic to its play then the use of established ECO lines with perhaps some of the more preferred variations are likely to provide more information in openings matched paired games.

Peter