Developing a bot is always a trade-off between time invested and time saved in the end. In this post, I'll discuss my experience developing a bot for Steem Monsters.
The whole Steem ecosystem never piqued my interest until this spring. That's when I've gotten into Steemit and the lovely ssg-community. After a month or so I developed my first bot, a curation bot for Steemit. While watching this bot do its thing I noticed some of the Steem dapps. Since I've played a couple of fantasy card games in my youth Steem Monsters piqued my interest pretty quickly. Even though at first I thought it was just another Cryptokitties clone.
First impressions can deceive. The game was surprisingly fun.
Making of a Bot
I've had a couple of lectures on reinforcement learning and general artificial intelligence at University, so I had a rough idea of how to make a gameplay bot. But still, applying all this theory to an actual game is a different story. I finally settled for a Monte Carlo Tree Search (MCTS) as the base of the bot.
The main issue with the MCTS is that you need a simulation of the game which you want to play. The faster the simulation the better. The algorithm selects one team member at a time, after each selection it fills the remaining slots with random team members and simulates a match. By doing this the algorithm scores each partial team and should find the optimal team against an enemy team selection.
Simulation
The simulation was the most coding and trial and error intensive part. Evaluating matches step by step to check whether or not they were both doing the same thing. There are many nuances in the game in turn order,´e.g. edge cases where some monsters have the same speed. In this case, magic goes first, followed by ranged and then melee. Otherwise, the game decides randomly who goes first.
Another nuance which you usually don't encounter in normal gameplay is the fatigue mechanic. After turn 20 one more damage is applied to each monster after each round.
After finishing the first version, the simulation computed ~80% of the matches correctly. I don't think it will get much better than that, because there are many chance mechanics in place. For example, a series of unlucky dodges can turn the tide of a game. Even just one missed poison or a dodge can decide a game. One can work with expectancy values in this case, but then you just get the average outcome of the match. And an unlucky dodge is not the average outcome but an outlier.
For those of you who are interested, I pushed the code onto GitHub where you can check it out.
The Bot
After finishing the first decent Simulation I moved onto the bot. The actual mechanism is pretty straight forward, but the devil's in the detail. It all starts with the team selection which the bot should be evaluated against. Do you just look at the last 5 teams of the enemy? The last 5 winning teams? Do you use the bot's team history as well? Maybe just evaluate against the most successful teams of the season? It's not an easy question to answer. I settled for the last 5 winning teams with the same ruleset of both my bot and the opponent.
But once you go down that route you come across another issue: rewarding the bot. After each step, the Monte Carlo Tree Search evaluates the team through the simulation against the "opponent" team selection. The tricky part is how to reward each win. You could reward each team equally with 1 point for a win, 0 points for a draw and -1 point for a loss. But this leaves out a lot of information about how good the team performed. E.g. is a loss where the team almost wiped out the whole enemy team better than a loss, where the hit points of the enemy are untouched? Should a win which took 25 turns give equal reward than one which occurred only after 5 turns? Many of those things depend on each other and should be used in a good reward function.
After some playing around I came up with this function. It takes most of those things into account.
First Result
The bot has been running with many different settings over the last couple days and I noticed the fewer splinters it can choose from, the better the result usually gets. The dragon splinter adds a lot of confusion, that's why I omitted them for now.
With every splinter available (except dragon) the bot reaches a win rate of 65% until its deck limits it's progress. Then the win rate falls to 50% and it stagnates +-100 points. With my last deck that was around Silver II, with an upgraded deck I'm currently at Gold II. Not too bad for the amount of work.
If the opponent always uses the same team the bot is very good at finding optimal counters. For example this match :
Example match
https://steemmonsters.com?p=battle&id=708e8c4e9cdccf5246a68c049b65c97b7fc254fa&ref=aicu. The enemy uses a pretty strong team with double sneak and a high level legendary. Cerberus with retaliate devastates the double sneak team and tanks the back. The second tank mitigated the damage with dodge and shield from the lord of darkness after the first was defeated.
Discussion
I'm surprised by the performance of such a simple approach. I've talked with other people who actively develop bots for Steem Monsters and I've seen similar numbers from various approaches. There might just be a good portion of luck involved in some matches.
After working on this bot I can definitely say two things about this all:
- That my bot is still far from perfect.
There is still room for improvement. I'll work on those approaches in the upcoming weeks. One field which looks promising would be the use of machine learning as a heuristic to guide the team selection towards promising nodes more quickly. - To play Steem Monsters in the higher leagues is an expensive hobby!
Some Beta Pack Porn
Afterwards, I was curious if it's worth your money to open 100 Beta packs with two brilliant potions (gold foil and legendary) and I got lucky! One Gold Chromatic Dragon and one Gold Hydra! Gotta keep those two around. Two gold foil legendaries. Although having opened a couple beta packs before, I don't think that's the norm. Even with those two potions.
Closing Word
You can see for yourself how my bot is doing under the account "aicu" on Steem Monsters.
And one more thing: I thought long and hard whether I should make the code for the bot public on GitHub as well. For now, I leave it private, because it would be too easy for someone to build a bot army with that and ruin the game. Even with a win ratio of just 0.5 to 0.75. I'd be interested on your thoughts down below on that topic. Should I make the bots code public as well?
The creation of bots to automate actions has always fascinated me as a concept. It takes a progressive yet analytical mind ot be able to do that and you have my respect.
I noticed the generous @aicu bot vote and investigated where it originated to find you. So glad I did as your skill and involvement in STEEM is legit.
Thanks again for what you do an Well Met!
Thanks for you kind words. Automation is truly a fascinating topic. And the advances in the field of machine learning, or pattern recognition if you want to call it by its name are fascinating.
Aicu is one of those projects which I should put more work in as well. It's still pretty basic and there are many ways of improving it.
And it seems my bot likes your content I hope you attract the attention of larger curation networks soon :)
I like to scan my steemworld profile constantly for people who support so that I can reciprocate. I am not sure what intelligence you have programmed in so that it can find my stories and blogging but I am sure glad you did.
Thanks again and I look forward to learning more about programming and the adaptive learning you can create with code.
Posted using Partiko iOS
I did a quick writeup on that, the tldr:
The bot looked at 10000 posts which were 50% posts which were selected by large curation networks and 50% other posts. Through some machine learning techniques it extracted patterns which identify high quality posts.
It's interesting to see your numbers. I've got a few bugs in my bot since I haven't spent time debugging since I got it working, but I think I have around 50% win rate when it's working, but I haven't tracked it.
My approach to choosing a team is based on historical top matches choosing a random one for the rule set. Probably not the best method, but I at least it isn't too easy to predict for other bots.
I'm not sure in regards to making the code public. There is definitely a likelihood that more people will make bot armies, but they already exist to some extent.
Thanks for the post 👍
Posted using Partiko Android
Hey, I think our bots ran into each other a couple of times :D My bot had some difficulties defeating yours if I remember correctly.
And that there are already bot armies is pretty likely. Some players have pretty suspicious names. E.g. variations of name + number. I think it would help by making the game turn-based like Hearthstone. That would make it much more difficult to make a bot.
Congratulations @bauloewe! You have completed the following achievement on the Steem blockchain and have been rewarded with new badge(s) :
You can view your badges on your Steem Board and compare to others on the Steem Ranking
If you no longer want to receive notifications, reply to this comment with the word
STOPTo support your work, I also upvoted your post!
Vote for @Steemitboard as a witness to get one more award and increased upvotes!