Dominion: the dangers of simulation

Disclaimer, because I feel like I need one of these:

Some people have put a lot of time and effort into writing software that simulates Dominion games, and I’m about to be pretty critical of their work. Those same people have made lots of meaningful contributions to the community including, but not limited to, having their software used to help playtest actual Dominion cards before they are published. Dominion simulations do have some value and I’m questioning some of it; this article is meant to focus on the applications of Dominion simulations that I find misleading.

And it should go without saying that I have nothing against these people personally. I think these people are nice people. I want to be very clear that I’m not trying attack anyone’s character, but I am going to be critical of some ideas that they have worked hard to support if I don’t think they are valid and sound.

OK that’s enough for the disclaimer, let’s get to the good stuff.

1. What is the simulator trying to tell me?

You may have seen some statements out there like “Jack Big Money beats Mountebank Big Money”, so you come across a game with Jack and Mountebank, and you play Jack BM and your opponent plays Mountie BM and you lose anyways! Hey! What happened? Simulation is worthless!

Not quite. There are many, many things wrong with this; let’s get to them:

First, the statement “Jack Big Money beats Mountebank Big Money” is pretty misleading. What it means is that a simulation matched two bots against each other, and the Jack bot had a higher winrate than the Mountie bot. This does not mean that Jack will always win a game against Mountebank — sometimes the strategy that’s favored won’t win because you have bad luck. So maybe you just got unlucky. Great! You now have a reason to get really salty and whine a lot, and tell your opponent about how you are still better than them even though you lost! This is great sportsmanship!

Not so fast. Even if your strategy was actually favored, maybe you should be a little more gracious in defeat as a start; let’s put that Haterade aside for a minute. But also, YMYOSL. This is a great time to look critically at your own play and figure out if you could have done something better…

“Better? I already picked a strategy that’s favored! I should win! What more do you want from me?”

Second, just because Jack BM is favored against Mountebank BM doesn’t mean it’s the best thing you can do. A strategy that incorporates both Jack and Mountie will do even better — you’ll be more favored. Maybe this game you just lost could have been won if you played a better strategy that would win a higher percentage of games against what your opponent was doing.

This is exactly what the spirit of YMYOSL is (You Make Your Own Shuffle Luck) — the player who blames bad luck for his loss, even if he isn’t wrong in doing so, can miss out on a chance to get better at Dominion by refusing to look critically at his own play.

Anyways, this is an illustration of how simulator results can be interpreted incorrectly. Just be careful to not take simulator results to mean anything more than what they’re saying, and realize in a real game of Dominion there will be many more factors at play than what the simulation results have taken into account. In fact, enough factors that the simulation results themselves are often not going to be useful anymore…

2. Attempting to apply simulation results to your play

I’ll say this up-front: simulation results can be pretty accurate for Big Money games with minimal player interaction. In fact, because of simulators, the game has effectively been solved if you use no kingdom cards! They call this “best strategy” BMU, or Big Money Ultimate. It knows just how to build and Duchy Dance and observes PPR (the Penultimate Province Rule, which only actually applies with zero kingdom cards), which represents the extent of player interaction in the game with no kingdom cards.

This goes back to a criticism I often hear for Dominion — it’s just multiplayer solitaire. My response to that criticism is that if you think Dominion is just multiplayer solitaire, UR DOIN IT RONG. Player interaction is in every single game of Dominion and if you ignore it, you’re just going to lose more. This is actually true of any game with player interaction.

The problem with simulators is that it’s really hard to account for player interaction when you’re making the bots that are inputs to the simulator. It’s theoretically possible but it would be extremely laborious to actually do, and there’s basically no way to ever know that you’re finished. Why?

Let’s say I want to match up two different strategies on a kingdom of ten given cards to decide which one is better (has a higher winrate). I program my two bots and let it fly. Boom. Done. I got a number, Bot A beats Bot B 55%-40%-5%(tie), post it on the internet! QED.

What I’ve shown here is that I am slightly better at programming a bot for strategy A than for strategy B. How do I know that I did a good job with either one? This number would only be valid if I could know somehow that both bots are playing perfectly, not only in general but also perfectly in reaction to the other bot’s play. This is clearly impossible, but let’s try to get as close as we can and then we can realize how far away we are from this.

I’ll work on refining Bot B’s play to get its winrate up. Maybe I’ll try adding in a couple of tricks that deny resources to Bot A. Ha! If I just pile the villages super-fast and then play Big Money I can beat Bot A most of the time because Bot A will not adapt to the fact that its deck won’t work with only 3 villages. Maybe Bot A is vulnerable to an attack on the board, so when Bot B starts buying that attack, Bot A does worse, mostly because it doesn’t adapt to being attacked.

So you go and work on Bot A — play around these new tricks Bot B is trying to pull. The winrate swings back and forth as you go back and forth, just seeing what works well and what doesn’t. But the goal is to create one Bot A that accounts for everything and one Bot B that also does. The bots get refined and refined, and in theory the swings in winrates will get smaller and smaller until it settles around somewhere. Sweet, I think we figured it out.

But how can you know that you’ve thought of everything? How can you know that there aren’t weaknesses in your bots that you just haven’t found yet? How can you know that you’re building as quickly as possible? Try everything imaginable? I mean, that’s about it. So as much work as we did to get here, we still don’t know if we’re there, we can only assume that we’re getting closer. This process is so difficult and laborious because it’s the only real way to account for the interaction between players in a game of Dominion, using a tool that isn’t particularly good at it.

So theoretically it’s possible to optimize a bot and get a number from a simulator. I suppose that if we take a cue from the scientific method we could come up with a peer review process: whenever someone states simulator results, they should include the bots they used and invite people to improve upon each of the bots until a consensus is reached. I feel like if this doesn’t happen, simulation results shouldn’t be taken seriously (or at least not any more seriously than someone chiming in and saying that they feel like it’s different).

I’ll say this again so I can bold it, because it’s kind of the whole point of this article. Simulation results without the bots used to generate them shouldn’t be taken seriously. It’s the scientific equivalent of stating your results without any method, data, or sources; if you want to get better at Dominion, I feel like this is the least you can do to make sure your information is legitimate.

I’ll even go further and say that I have very rarely seen any simulation results posted where any attempt was made to account for player interaction; and the attempts that I have seen made are almost never even close to being rigorous enough to get accurate results. The single exception to this is what I mentioned before, about the case where there are no kingdom cards. What this means is that any simulation results that cover matches with any player interaction at all can’t be considered accurate. (Man I hope I can change this statement some day, but it reflects the current state of the way simulation results are generated and presented.)

I should be a little more specific about this. I think the margin of error for Big Money vs. Big Money matchups is small enough that you can get some value out of them, and by “Big Money” I’m including most definitions of “slog” as well. Let’s just say that if each bot is assuming the end of the game will come from the same endgame condition (Provinces or the same three piles) and neither bot aims to get more than one Province on a turn, you can take the simulation results somewhat seriously. If this is not the case, you cannot assume the numbers your simulator is giving you are accurate.

3. What are simulation results good for?

I’ve been pretty harsh so far on simulation results, and for pretty good reason IMHO. On the other hand, they are out there and there is some value in using simulation tools, you just have to be really, really careful about the conclusions you draw (namely, that you don’t draw any conclusions). Think of it as a good first step to the conversation about how to play better.

For example: let’s take a certain matchup between Bot A, which plays Militia Big Money, vs. Bot B, which plays Ironworks/Gardens. I suspect that Bot B is winning this matchup a lot of the time if you just use the canned bots for these two strats, so let’s try and improve Bot A and see what we can learn about playing against rushes. You might find that greening a bit earlier than normal increases the winrate. You might find that choosing other big money enablers besides Militia will give better results against a rush which doesn’t care about discard attacks. You can try all sorts of things to see what gives you better results.

Taking this first step is a good start to the conversation. Maybe the rush can play differently to account for these changes, so we have no idea which strategy is best yet — but remember that’s not really an attainable goal. We have learned something about how to get better at Dominion — that we suspect these methods of playing around a rush could be good. Maybe it’s not, maybe the rush can easily play around some of them, but the simulator results nudged us in this direction so it’s probably more promising.

This is something that leads to getting better at Dominion, and it had next to nothing to do with the numbers the simulator output. We can learn about ways to optimize certain strategies with a quick-and-dirty way to get an idea of how effective they are. Now we know which ones to investigate further. Simulation results aren’t about the numbers, it’s about the steps you take to increase those numbers. It’s not about the “what,” it’s about the “why” — it’s not the destination, it’s the journey.

Tables that compare winrates of strategies against each other, in an attempt to determine which strategy has the most raw power are great examples of a misuse of simulator results. There is no attempt to modify each bot to play the matchups well, and it’s all about the numbers. This is more of an exercise of which bots are easier to program (and can do things that the other bots aren’t likely to account for or need to play around) than which strategies are actually best in a game of Dominion. If you wanted to answer the question of which strategy has the most raw power in Dominion, I don’t think simulation is the best way to get there. I also don’t think it’s a question worth answering, but that’s a different topic.

This article is a great example of using simulation results to get better at Dominion. Yeah, the big money bot they played against probably wasn’t adjusted to play well against this engine so the numbers probably can’t be trusted, but the narrative in this article shows you how this deck works and why, and it gives you a great starting point for building the deck if you’re having trouble coming up with it yourself. There is more to playing this deck than the simulator lets on, so don’t stop trying to get better because you think the simulator figured it out for you, but it’s a great first step.

Leave a Reply