I've literally built this exact thing over the past few months before I read this post. I didn't think about using Brier score, but I have the "predict a move a player this strength would make given the position and time on the clock", the "how likely is it that white wins / black wins / draws" given position, ratings, and time on the clock and "predict how much time the player would think given the position, rating, and times on the clock"
I haven't packaged it up into a full product yet, but hit me up if you want to try it out.
I'd love to try it out and curious your plans on productization. It has a lot of value just as a game explorer where you start from the opening position, since there are raging debates about what the best practical opening play is at different strength levels and without that tool nobody knows what they're talking about. After that there can be a PGN explorer and then more sophisticated tools which automatically collate the results of different values in the same position to make something resembling commentary but it's extremely useful even without that and having a human mess with it to figure out what's going on.
A slightly different approach, I think I would go for a prediction of, given this position and this player rating, what move will they make. Then you could use that to deduce, this move is technically worse, but it can only be defeated by a 3000-level move, and it is very likely to beat a 2600.
The question of how to train an LLM to eg explain to me when I should push my h pawn positionally seems tougher. You can at least get a corpus of when it’s actually correct and when it isn’t. Then maybe fine tune the llm for it?
I've literally built this exact thing over the past few months before I read this post. I didn't think about using Brier score, but I have the "predict a move a player this strength would make given the position and time on the clock", the "how likely is it that white wins / black wins / draws" given position, ratings, and time on the clock and "predict how much time the player would think given the position, rating, and times on the clock"
I haven't packaged it up into a full product yet, but hit me up if you want to try it out.
I'd love to try it out and curious your plans on productization. It has a lot of value just as a game explorer where you start from the opening position, since there are raging debates about what the best practical opening play is at different strength levels and without that tool nobody knows what they're talking about. After that there can be a PGN explorer and then more sophisticated tools which automatically collate the results of different values in the same position to make something resembling commentary but it's extremely useful even without that and having a human mess with it to figure out what's going on.
How can I contact you? I don't want to post my email address here where it can be scraped, but you can DM me on twitter at @thomasj02
A slightly different approach, I think I would go for a prediction of, given this position and this player rating, what move will they make. Then you could use that to deduce, this move is technically worse, but it can only be defeated by a 3000-level move, and it is very likely to beat a 2600.
The question of how to train an LLM to eg explain to me when I should push my h pawn positionally seems tougher. You can at least get a corpus of when it’s actually correct and when it isn’t. Then maybe fine tune the llm for it?