HCE Pro 1.2.1 Import multiple files

[GoneBallistics]

I tried something and got "strange" results.
Created a pgn file, then File | Import.
Seeing that I could select multiple files, I selected 1500 pgns.
After a long import, the file remains empty.
I waited some more time to make sure it was not saving.
What's wrong?
What I was expecting or what the program did?

[Ruyter]

Hi,

i have tried with 11 pin files without problems.
Have you the same problem with 11 or 10 pin files ?
Do you have the same problem if you create a hce file and not a pgn file ?

[GoneBallistics]

Hi,

i have tried with 11 pin files without problems.
Have you the same problem with 11 or 10 pin files ?
Do you have the same problem if you create a hce file and not a pgn file ?

Tried with 10 pgn files, imported into the pgn database without any problem.
Tried against an hce database with 1500 files, ends quickly saying 0 games imported.
Looks like some error condition escaped through error management.

[Ruyter]

I have downloaded 50 pgn files from web and duplicate it x times to have 1600 pgn files.
I have created a new pgn file and the select all pgn files to import them.
All is fine and at the end I have a lot of games.
with HCEPro I found all games in one file.
I open the disk file (Finder for Mac) and the size file is 0.
Then I close HCE PRO and then HCE Pro save the pgn file and finally I have a 2GB size file saved.
Same for you ?

[GoneBallistics]

I have downloaded 50 pgn files from web and duplicate it x times to have 1600 pgn files.
I have created a new pgn file and the select all pgn files to import them.
All is fine and at the end I have a lot of games.
with HCEPro I found all games in one file.
I open the disk file (Finder for Mac) and the size file is 0.
Then I close HCE PRO and then HCE Pro save the pgn file and finally I have a 2GB size file saved.
Same for you ?

Nope.
In my case it remained empty.

[Ruyter]

is it possible you share your pgn file with some cloud or other platform ?
you use windows or osx ?

[Mark Uniacke]

I cannot reproduce this, it would be useful if you could narrow it down, could the HCE file be in a read-only location.

What are the sizes in games of the respective PGN files?

Importing 1500 PGN files is probably not that common, I wonder if one of the files has an issue, are they all valid PGNs?

Could you zip the PGNs and upload and then email us the download link so we can try to reproduce, if we can reproduce it can be fixed.

Can you send a Help->Report the report a problem info with config and log?

Have you tried importing 750 files to see if that works? Isolating a file or set of files where the issue exists that would be excellent.

[GoneBallistics]

is it possible you share your pgn file with some cloud or other platform ?
you use windows or osx ?

The output file is empty.
The input files are twic 1-1500.
I'm going to less files (1000, 500, etc)

[GoneBallistics]

is it possible you share your pgn file with some cloud or other platform ?
you use windows or osx ?

The output file is empty.
The input files are twic 1-1500.
I'm going to less files (1000, 500, etc)

Found what happened.
Made a few attempts loading blocks and went down to the single file causing the problem.
The first twics were txt files with games in the middle, so I cleaned up all what is not pgn code.
File twic61 didn't contain pgn code, apaprently, so the text remained.
If you try and import it you get a msgbox "Error: the database 'twic0061' has incorrect format."
Apparently with multiple files this notification is lost.
If you import the files with Chessbase they are imported without any problem.
If you want the file I can send it, but any text file should be fine to replicate the problem.

[Mark Uniacke]

Thanks, that makes sense if the pgns were corrupt.

If you use hold the Alt key (or Cmd on Mac) and select the Help->Show application log... menu item, it will list the first offending file,
e.g. 00:45:22 Database: [Warning] the database 'C:/Users/admin/Documents/HIARCS Chess/Databases/corrupt.pgn' has incorrect format.

Given one file is incorrect and corrupt it would seem sensible not to try to import the other files regardless to avoid corrupting the user database.

[Ruyter]

Hi,

nice to learn the happy end.

[GoneBallistics]

Thanks, that makes sense if the pgns were corrupt.

If you use hold the Alt key (or Cmd on Mac) and select the Help->Show application log... menu item, it will list the first offending file,
e.g. 00:45:22 Database: [Warning] the database 'C:/Users/admin/Documents/HIARCS Chess/Databases/corrupt.pgn' has incorrect format.

Given one file is incorrect and corrupt it would seem sensible not to try to import the other files regardless to avoid corrupting the user database.

I learned something: I had no idea that Alt + Help gave those additional informations.
The problem I see I get the line number for the errors but there is no file name.
Apparently it works as expected only with a single file.

09:49:19 Progress: start operation 0 "" (100 steps)
09:49:19 PGN: [Warning] PGN parse error in game 1 (line 1): no header
09:49:19 PGN: [Warning] PGN parse error in game 9 (line 152): no header
09:49:19 PGN: [Warning] PGN parse error in game 11 (line 173): no header

I was importing 100 files.

[mrudolf]

I learned something: I had no idea that Alt + Help gave those additional informations.
The problem I see I get the line number for the errors but there is no file name.
Apparently it works as expected only with a single file.

09:49:19 Progress: start operation 0 "" (100 steps)
09:49:19 PGN: [Warning] PGN parse error in game 1 (line 1): no header
09:49:19 PGN: [Warning] PGN parse error in game 9 (line 152): no header
09:49:19 PGN: [Warning] PGN parse error in game 11 (line 173): no header

I was importing 100 files.

As far as I understand, there are a few issues:

1. One buggy import file cancels the import. Not sure if this is correct behaviour or whether we should import other files. You can argue both ways, so I believe HCE Pro behaviour is reasonable.
2. The message for incorrect format is easy to miss.
3. The log doesn't contain file names when importing multiple files. (BTW. these are just warnings, they do not stop import).

Is that correct?

[GoneBallistics]

As far as I understand, there are a few issues:

1. One buggy import file cancels the import. Not sure if this is correct behaviour or whether we should import other files. You can argue both ways, so I believe HCE Pro behaviour is reasonable.
2. The message for incorrect format is easy to miss.
3. The log doesn't contain file names when importing multiple files. (BTW. these are just warnings, they do not stop import).

Is that correct?

Completely agree with 1, the behaviour is perfectly reasonable.
About 2, the message doesn't appear at all, or perhaps it's removed so quickly that I can't see it: in single file mode the msgbox is clearly visible.
ABout 3, the log is not much helpful.
Not a big problem, I'm going to join the file as I did before, and the check them: the problem arised becaus I wanted to try the multiple files import, which is a new function apparently.
The problem is I trusted too much the original files.

[mrudolf]

Then I close HCE PRO and then HCE Pro save the pgn file and finally I have a 2GB size file saved.

I would add one more thing: please don't use PGN for large files unless you really really need it (so, mostly, for export to another program).
PGN is not a format suitable for large databases: it takes a lot of space and changing one move in a single game requires a complete rewrite.

Use native database and export to PGN if necessary. Your data will be safer and everything will be faster.