Then give the black box points depending on the certainty it predicts the right hand.
Lets say the black box narrow the possible hands down to: AA (30%), AK (45%), and KK (25%).
If the hand turns out to be AK then the black box gets 0.45 points.

Ок, I got it.
And if the prediction is wrong? It can be zero points or some negative score. If negative, how much?

I also think about giving more points for right guessing in big pots and less for the small ones...

Lets say the black box narrow the possible hands down to: AA (30%), AK (45%), and KK (25%).
If the hand turns out to be AK then the black box gets 0.45 points.

The points could either be collected at each street or at the end of the hand.

Evaluate the black box over 1000 hands and add the points together. Make some changes to the black box and compare it's new score with the old score.

I'm wondering about multiplying by the winnings/losses at showdown. An error associated with a win is something you want to reward, and an error associated with a loss is something you want to punish.

You probably don't want to do it that way. That would be equivalent to minimizing the absolute error, which means the maximum likelihood model would perform better than the true model in some cases. For example, consider a situation where given your features the opponent has AA (51%) and KK (49%). If you use your method, then the model that predicts AA (100%) would receive an expected score of 0.51. However, the model that predicts the distribution perfectly would receive 0.51*0.51 + 0.49*0.49 = 0.2601 + 0.2401 = 0.5002.

Instead, you should use the squared error. That way, if your model predicts AA (100%), it would receive the score of 0.51*(0*0) + 0.49*(-1*-1) = 0.49. The true model would get 0.51*(-0.49*-0.49) + 0.49*(-0.51*-0.51) = 0.122451 + 0.127449 = 0.2499. You then evaluate the model on its ability to minimize the mean squared error.

That assumes of course that you have uniform samples, or at least enough samples that the bias is small. If you have a biased dataset that you use for evaluation, then the MSE will be biased as well. Since you're assuming you have perfect information, you can just sample uniformly over the holecards and that should be an unbiased sample.

That wasn't really in the scope of his system though, was it?

I also think about giving more points for right guessing in big pots and less for the small ones...

All of this assumes you have some payoff function that can look ahead into the future and say, given complete info of the game, this is what you'll win. You could use HS or W% but I think those are going to be pretty poor predictors of the true payoffs of the hand given the opponent actions.

Whereas I propose to compare
predicted cards with actual cards weighted with actual payoff

Oh, I see. So you just want to use the payoff as the error. Seems reasonable, and you could still just do mean squared payoff error as the loss function.

I'm a little curious though about the credit assignment problem. If you have a hand history with N opponent actions, do you break those N actions into N samples and give them equal weight in your evaluation? Seems like what you might want to do is randomly sample a hand history, then randomly sample a single action point to prevent bias towards showdowns.

No. I'm proposing that the error is calculated from the actual cards and the predicted probabilities, but it is somehow (I haven't figured out the details) weighted by the winnings/losses actually incurred.

Initially, I proposed to use pot size as weight. Not winning or losses - if we lost, but predictor worked well it is still good predictor.