CCRL update (3rd November 2007)

[Graham Banks]

The November 3rd update of the CCRL Rating Lists and Statistics is now available for viewing at:
http://www.computerchess.org.uk/ccrl/4040/

The list gets updated periodically during the week and these updates can be viewed here:
http://www.computerchess.org.uk/ccrl/4040.live/
Please be aware that no game downloads are available from this live link.

The links to the various rating lists can be found just beneath the default Best Versions list.
For example there is a 32-bit Single CPU list.

Our standard testing is at 40 moves in 40 minutes repeating while our current blitz testing is at both 40 moves in 4 minutes repeating and 40 moves in 12 minutes repeating, all adjusted to the AMD64 X2 4600+ (2.4GHz).

Currently active testers in our team are:
Graham Banks, Ray Banks, Shaun Brewer, Kirill Kryukov, Dom Leste, Tom Logan, Andreas Schwartmann, Charles Smith, George Speight, Chris Taylor, Chuck Wilson, Gabor Szots and Martin Thoresen.

A big thanks to all testers as usual for their efforts this week.


40/40 Notes

There currently 84,286 games in our 40/40 database.

Many engines on our list have few games and in many cases their ratings are likely to fluctuate (markedly for some) until a lot more games are played. Therefore no conclusions should be drawn about their strength yet.
To illustrate this point, when an engine has 200 games played, the error margin is still approximately +-40 ELO, after 500 games +-25 ELO, after 1000 games +-17 ELO and even after 2000 games there is a +-13 ELO error margin!
This of course highlights the importance of looking at other rating lists that are also available in order to draw comparisons and get a more accurate overall picture.


4CPU 64-bit Engines

Rybka 2.3.2a is over 50 ELO stronger than Zappa Mexico.

After 308 games, Deep Shredder 11 lies 30 points further back in third spot.

Naum 2.2 and Hiarcs 11.1 are fourth and fifth strongest respectively in this category.

The remaining well tested engines in order of rating are Loop M1-T, Deep Fritz 10, Deep Junior 10, Deep Sjeng 2.7 and Glaurung 1.2.1.

We have started testing Deep Fritz 10.1 and early signs reflect the strength increase reported by others.


2CPU Engines

Some of the engines here are in the process of getting further games, but the order of strength is almost identical to that described in the 4CPU notes, except that we have not started testing either Deep Shredder 11 or Deep Sjeng 2.7 in this category yet.


Single CPU Engines

After 233 games, Shredder 11 has slotted straight into second spot just ahead of Zappa Mexico, but still well over 100 ELO behind Rybka 2.3.2a.
The 64-bit version of Rybka 2.3.2a seems to have a good edge over the 32-bit version.

Loop M1-T has only 106 games, but is currently level with the well tested Toga II 1.3.1.
Both are slightly ahead of Hiarcs 11.1, Naum 2.2, Fritz 10 and Fruit 051103.
Hiarcs 11.1 and Fruit 051103 seem to be stronger than the later Hiarcs 11.2 and Fruit 2.3.1 versions.

Strelka 1.8, Spike 1.2 Turin, Deep Sjeng 2.7 and Junior 10 are the next group of engines.
Junior 10 seems to be stronger than the later Junior 10.1.

Ktulu 8.0, Glaurung 2 epsilon/5, Bright 0.2b (private) and Chess Tiger 2007.1 are 40-50 ELO further back.

Delfi 5.2 seems to be on a par with the top CM settings, SmarThink 1.00, Alaric 707 and Movei 0.08.438.

We haven't started testing Scorpio 2.0 yet.


Free Single CPU Engines

Rybka 1.0 64-bit is still the top free engine ahead of Toga II 1.3.1 and Fruit 051103.

Strelka 1.8 and Spike 1.2 Turin come in next, well ahead of Glaurung 2 epsilon/5 and Naum 2.0 which are likewise well ahead of Scorpio 1.91.
We have yet to start testing Scorpio 2.0.

Delfi 5.2, Alaric 707, Movei 0.08.438, SlowChess Blitz WV2.1, Zappa 1.1, WildCat 7, Pro Deo 1.2 and List 512 are within a 30 ELO point range of each other.

We test a very extensive range of amateur engines through our Amateur Championship divisions (32-bit 1CPU) plus other tournaments, all of which can be followed in our public forum.

Our aim is of course to ensure that all engines lower on our lists get 200+ games.


Blitz Notes

There are currently 191,220 games in our 40/4 database.

The 40/4 update is usually done separately to our 40/40 update.
The latest ratings can be found at one of the following links:
http://computerchess.org.uk/ccrl/404/
http://computerchess.org.uk/ccrl/404.live/

An enormous amount of work goes into the blitz list and it is well worth a visit.


FRC Notes

There are currently 21,500 games in the FRC 40/4 database.

Ray tests only those engines that can play FRC through the Shredder Classic GUI.
If engine authors have a new and stable version of their engine that will run under this GUI, they should contact Ray if they wish to see it tested.

Shredder 11 has now taken over second spot behind Rybka 2.3.2 FRC (private), an impressive 80 ELO ahead of the closely bunched Hiarcs 11.1, Naum 2.2, Fruit 051103 and Loop 10.32f.

For FRC the best list to look at is the pure list.
http://www.computerchess.org.uk/ccrl/404FRC/


Stats/Presentation Notes

The LOS stats to the right hand side of each rating list are "likelihood of superiority" stats. They tell you the likelihood in percentage terms of each engine being superior to the engine directly below them.

A list of games played this week per engine can be found in the update thread in the CCRL public forum, accessible through the link given at the top of this post.

All games are available for download through the link given at the top of this post. They can be downloaded by engine or by month.
ELO ratings are now saved in all game databases for those engines that have 200 games or more.

Clicking on an engine name will give details as to opponents played plus homepage links where applicable.
Clicking on an engine name now also gives you a ratings history graph for that engine over time (a bit further down the page). The green line is the actual rating. The red lines are the upper and lower error bars, and the blue line represents the number of games.

Custom lists of engines can be selected for comparison.

An openings report page (link at bottom of index page) lists the number of games played by ECO codes with draw percentage and White win percentage. Clicking on a column heading will sort the list by that column.
Games can now be downloaded by ECO code.

[Steve Maughan]

Great work! So Stefan has made huge progress with Shredder, gaining about 100 ELO - amazing!!

Steve

[Harvey Williamson]

Great work! So Stefan has made huge progress with Shredder, gaining about 100 ELO - amazing!!

Steve

Hi Steve,

Welcome to our forum!

Yes it looks like a nice jump for shredder 11 which is very good news for Computer Chess.

Best Wishes,

Harvey

[Graham Banks]

Great work! So Stefan has made huge progress with Shredder, gaining about 100 ELO - amazing!!

Steve

Cheers Steve and yes, Shredder 11 does seem to be a nice improvement for Stefan.

Did you notice that Monarch has 300+ games on our 40/40 list?
I hope that they're of some use to you.

Regards, Graham.

[Steve Maughan]

Graham,

Did you notice that Monarch has 300+ games on our 40/40 list?
I hope that they're of some use to you.

Oh Yes! I'm flattered that you've tested Monarch. I noticed in the past that it gets some airtime on CCRL. Hopefully I'll have some time in December to do some more work on it. There is loads to do.

Cheers,

Steve

[Steve B]

Graham,

Did you notice that Monarch has 300+ games on our 40/40 list?
I hope that they're of some use to you.

Oh Yes! I'm flattered that you've tested Monarch. I noticed in the past that it gets some airtime on CCRL. Hopefully I'll have some time in December to do some more work on it. There is loads to do.

Cheers,

Steve

Hi Steve
i came across an old 8 pg. pamphlet entitled "Evaluating Chess Computer Programmes..A New Approach"
dated Friday 10 November 1989 by a Steve Maughan
what drew my attention to the pamphlet was that the Saitek Stratos was used in the analysis

would this have been written by you perhaps?
Best
Steve