Losing by my own design
I could open my first machine learning model and watch it think. Built on years of StarCraft knowledge, readable and inspectable.
Then holdout day: 16.9% accuracy on four classes. Random guessing scores 25. My weeks of hand-built theory lost to a coin flip.
You have a piece of code like that. Something you’ve put so much time designing, and you feel really clever in its creation.
You think it’s the heart of your system. You’re probably wrong.
My bot was getting wrecked by cheese I knew that. The reactions were right, but the detection was too late.
I’d spent weeks hand wiring the Bayesian network I used to teach my bot gut feeling to get my bot to predict the cheese coming. Every edge carried my own theory of the game.
Incredibly proud of its design, but I tell ya, there lies the trap.
The performance was abysmal, even after hours tweaking inputs and priors. Leaving me with salty frustration.
While on the hunt for a solution, the suggestion of using an off the shelf library came up.
So there I was: patch the design I loved, or delete it all for a boring off-the-shelf library that understood nothing about my game.
But did I want the bot to win? Or did I want to lose by my design?
If you’ve been in this spot before you know when your pride is blocking you from seeing what the results are telling you.
A refactor later, I wired up a Gradient Boost Classifier to take over the core part of the system.
67.7% on the same holdout set. The library that understood nothing about StarCraft read my own replays four times better than my theory ever did.
My pride took the hit. The bot got the win. Once the numbers were on the table, the decision made itself.
⌨️ Next Commit
Take a second look at one of your custom solutions and benchmark it against a battle tested alternative.
If you felt resistance reading that, then we’re making some progress 😉
May the Bugs Be Ever In your Favour🪲
Email Preference:
Unsubscribe | Update your profile | 600 1st Ave, Ste 330 PMB 92768, Seattle, WA 98104-2246