Making Enemy AI
Welcome to the second dev blog post for The Day We Fall! I’ve been working on the technical side of the game a lot lately, so I’ve been in the mood to talk about something technical. So, if you’ll humor me, let’s have a post about the biggest technical part of all: making the enemy’s AI. I’m going to talk specifically about how I made The Day We Fall’s turn-based combat AI, but the general principles apply to essentially all game AI. I’ll also give an update on how the game is coming further down towards the end.
First, just a refresher for how The Day We Fall’s combat works (you can read about it in much more detail in the previous Combat dev blog post). The player starts off their turn with a certain amount of Reaction Time (RT). They can then spend their RT on any number of actions (e.g. moving, fighting, healing, etc…). Once they run out of RT, their turns ends, and the enemy’s turn begins. The enemy’s turn works the exact same way. The enemy has a certain amount of RT, and has to determine which actions to spend it on. The way it determines which action to take is through its AI.
So, let's talk about how the AI makes that call.
The AI itself is one single loop that can be broken down into three stages. First, it tries to figure out which actions (e.g. healing, shooting the player, etc…) are even possible for the enemy to do. Second, for each action it could do, it tries to figure out which action would be the best to do. Third, it selects and actually does that action. After the action is executed, it runs a series of checks, and usually restarts back at the top and goes again (hence the loop). The real meat of the AI is in those three stages, and they’re each discussed in more detail below.
First Stage: Determine Which Actions Are Possible
There are many reasons that an action simply couldn’t be executed (including, for example, not having the item required to do that action). To weed out actions that aren’t worth evaluating further, the very first thing the AI does is check if there are any actions the enemy at this moment absolutely cannot do. To do this, it looks at each theoretically possible action and runs a series of checks for conditions that would rule that action out. At the end of the checks, it sets a bool (a true/false variable) for each action to store whether or not that action can be done. This bool will be checked later to determine if an action needs to be considered in more detail.
To determine if the enemy can fire a gun at the player, the following checks are run:
First, it checks if the enemy even has a gun. Through code, the AI pulls open the enemy’s inventory, checks the category of each item (which I manually assign to each new item as I make them) to see if it’s in the Gun category, and very simply returns a Yes response if anything in the inventory is, and No if nothing is. If none are, it stores that firing a gun is not possible (by setting the Fire Gun Bool variable to False). It then doesn’t even bother doing the rest of these checks, and instead moves onto evaluating if the next action in the list is possible. If, however, something in the inventory was a gun, it moves onto the next check.
Second, after it has determined if the enemy even has a gun, it checks to see if there’s enough ammo currently in the gun to fire a full shot. The AI grabs the gun out of the enemy’s inventory and checks how much ammo is currently loaded into it. It then grabs how much ammo that gun needs to fire a full shot. Most guns will only require one bullet, but burst-fire guns may take three. It compares how many bullets are in the gun to how many bullets are needed to fire a full shot. If the enemy doesn’t have enough bullets in their gun, the enemy can’t fire a full shot, and the AI notes that by setting the Fire Gun Bool to False. If, however, there are enough bullets in the gun, it has passed another check, and the AI moves onto the final check.
Third, if the enemy has a gun and that gun has enough ammo to be fired, the last thing to check is whether or not the enemy has enough time (RT) to fire this gun. The AI grabs the gun out of the enemy’s inventory again, tells it to calculate how much RT a character with the stats that the enemy has would need to fire it, and brings that number back to the main AI loop. It then checks to see if the enemy’s current RT is greater than or equal to the amount of RT the enemy will need to fire this gun. If the enemy’s RT is too low, it can’t fire the gun at this point, so it flags the Fire Gun Bool as False and moves on. If, however, the enemy does have enough RT, the enemy has now passed all of the checks needed to determine if they can fire a shot. The Fire Gun Bool is now flagged as True, and the loop continues onto checking the next possible action. It will keep doing this until it has made a judgment for every possible action.
Second Stage: Evaluate and Rank Each Action
After the first stage is complete, the enemy AI has a list of every action the enemy could theoretically do at this time. Now what the AI needs to figure out is which of those options would be the best to do. This is the meatiest part of the AI, and is what most directly determines how the enemy behaves.
I decided to build the AI in a way that was intuitive to me. I figured there would be a) some situations in which I would want that action to definitely execute, b) some situations in which I would want that action to definitely not execute, and c) some times when it would depend on the situation. Therefore, I gave the AI essentially three chunks for each action, which it goes through in turn:
First, it checks if the action fits my pre-assigned criteria for immediately executing. If it does, the loop stops in its tracks, and goes ahead and executes that action. This is only used in extreme situations, such as reloading when a healthy enemy’s only weapon is a gun and he’s run out of ammo.
In certain situations, an action may be possible, but extremely ill-advisable. For example, the enemy probably shouldn’t reload if they already have a full clip. For each action, the AI runs a series of checks to see if it would be wasteful or self-defeating to perform this action. If so, it prevents this action from occurring. It prevents that in code by setting this action’s Priority Value to 0. Priority Values are discussed more below.
Calculate Priority Value
Most of the time, however, there are no extreme situations in which an action is desperately needed, nor circumstances in which actions definitely shouldn’t occur. Most of the time it will depend on the situation. To determine which action should be the priority, the AI uses a formula to calculate a priority value for each action. These priority values are then compared later on, with the highest winning. Comparing these priority values is how the AI determines what the best action to take would be. Each action has a different formula. I also introduce an element of randomness into these equations to keep the enemy from doing the exact same thing if the player reloads the game and tries the battle again.
If we’ve flagged Healing as a possible action, all we’ve really done so far is note that the enemy has a healing item as well as the RT needed to use it. The AI doesn’t know yet if it would be actually smart to heal at this time. To figure that out, it goes through each of the following, in order:
My personal judgment is the only time I want the enemy to definitely heal is when the enemy is critically injured and near death. Therefore, when the AI gets to this section, it checks the enemy’s current HP and looks to see if it is under 10% of its max. If it is, the immediate execution component triggers, and the AI immediately tells the enemy to use its healing item to heal. The AI doesn’t even consider other actions after this point. The AI loop just runs any maintenance it needs to do and then restarts. If, however, the enemy’s health is greater than 10%, the AI moves onto checking the Prevent Action component.
Any situations in which I would want the enemy to definitely not heal can be examined here. The only time I think the enemy should absolutely not heal is when they are already at full health. When the AI gets to this section, it checks the enemy’s current HP and looks to see if it is equal to the enemy’s max possible HP. If it is, the code sets the Healing Priority Value to 0. Later on, this will prevent this action from being selected for execution. The loop then skips the Calculate Priority Value section (which would overwrite what I just assigned) and moves onto analyzing the next possible action. If, however, the enemy is not at full health, then the AI knows at this point that the enemy is neither critically wounded nor completely healthy. Since the enemy can only do a certain number of actions in a turn, the AI needs some information to decide how important it would be to heal in this specific situation. It gathers this information in the Calculate Priority Value section below.
Calculate Priority Value
Usually, it’s tough to look at just one action and decide if that’s the right thing for the enemy to do. Sometimes performing this action would be very important, but another action could be even more important still. To help the AI compare actions and make a decision, I use priority values. Priority values are numerical values calculated for each action that provide a relative measure of an action’s usefulness to the enemy. For example, a priority value of 50 would illustrate that that action is less important than an action with a priority value of 90. In the Calculate Priority Value section, unlike the Immediate Execution and Prevent Action sections, the AI does not make a definite decision at this point whether or not to execute the action. It simply calculates the priority value and stores it for later comparison after priority values have been calculated for every other action as well.
The trick is coming up with a formula that both represents what’s important about the situation and is useful for comparison to other values. Intuitively, I believe it is more important for the enemy to heal the more injured they are. The formula I’ve decided to use for this is therefore:
Here’s the explanation piece by piece:
First, the AI grabs the enemy’s current and max HP, and divides them. This returns what percentage health the enemy is at. For example, if the enemy currently has 5 HP whereas they started with 20 HP, this part of the formula returns 0.25 (i.e. 25%).
In the first part of the formula, the lower the enemy’s health, the lower the percentage returned. Because ultimately I want the lower the enemy’s health, the higher the priority value for healing, I need to invert that result. I therefore subtract the result of the first part of this formula from 1. This makes it so that the less HP the enemy has, the higher the result of this part of the formula. So, in this example, subtracting 0.25 from 1 gives us 0.75.
I like to keep my priority values roughly scaled to a 0-100 range. Since the previous part of the formula could range from 0 to 1, multiplying the result by 100 gives it a minimum of 0 and a maximum of 100. In this case, multiplying 0.75 by 100 returns 75. Keeping the priority values scaled similarly is necessary to make comparing them informative.
Finally, because I give the player the ability to instantly restart any battle in this game, I felt I needed to do something that would prevent difficult battles from being accomplishable by just brute force memorization. I didn’t want the player to simply memorize what an enemy does in certain situations and win by playing a meta game. To help prevent this, I introduce some randomness into the priority value calculations. In code, I have the AI select a random number from -30 to 30. The AI then adds that result to this action’s current priority value. If the random number it selected is negative, this will subtract from the previous result, whereas if it is positive, this will add to it. So, if the AI selects a random number of -25, that would drop our priority value from 75 to 50. If it selects a random number of +25, however, that raises it all the way to 100. Having the AI simply select a different random number during each calculation helps ensure each battle will be a little different.
The AI does this for every action until it has calculated a priority value for every possible action that the enemy could take. Presuming the AI loop didn’t get cut short by an Immediate Execution component, it then continues to the third stage.
Third Stage: Select Action To Execute
In most cases, the enemy AI loop will determine priority values for every action without triggering an Immediate Execution trigger. If so, the third step is gathering all priority values and comparing them to figure out which action to execute.
First of all, I want to make sure there is at least one action is worth taking. The AI runs a check to see if any priority value is above 0. If so, it continues, but if not, it just ends its turn right there and then. A slightly more technical note here is that when I subtract a random number from a priority value in the second stage, I clamp the minimum value that can result at 1 rather than 0, so that the action still remains under consideration at this stage.
The AI then gathers the list of actions that had priority values above 0. The AI looks at all of the remaining priority values, compares them all against each other, and, very simply, picks the highest one. The AI then triggers whichever action the highest priority value corresponded to. That is the action that the enemy undertakes.
In the event of a tie between two or more priority values, I have a manually ranked list of actions the enemy should give preference to. The highest ranked tied priority value on this list is the one that gets selected. So, in the event of a tie, the AI simply takes each tied value, finds their ranking on the list, and the one closest to Rank 1 gets picked. The list is a little arbitrary, but it keeps my computer from exploding.
Let’s say a full-health enemy is in range to strike the player with a melee weapon. In this particular situation, the Healing Priority Value would be assigned to be 0 (since the enemy is at full health), let’s say Stepping Backward (which would pull the enemy out of melee range) has a calculated priority value of 5, and that there is a tie between taking the Butaroid pharmaceutical (a melee damage-boosting buff) and striking the player with a Melee Weapon, both of which have calculated priority values of 90. So:
Healing Priority Value: 0 Step Backward Priority Value: 5 Butaroid Priority Value: 90 Melee Priority Value: 90
First, the AI checks to see if any possible actions have priority values that are above 0. It notes that the enemy has three possible actions it could reasonably perform, and pulls them out for further analysis. However, Healing, which had a priority value of 0, gets left behind. So now the AI is just choosing between whether to Step Backward, take Butaroid, or swing the Melee Weapon.
The AI next tries to determine which action to take by selecting the action with the highest priority value. Step Backwards, which has a lower priority value, now gets dropped from consideration. It notes that there are in fact two highest priority values: the tie between Butaroid and Melee. It now needs to break that tie.
The AI then consults the ranked list of actions I made to determine which it should select. In this list, I decided if there’s ever a tie between Butaroid and Melee, I would like the enemy to take the Butaroid, because it makes more sense to me to take the damage buff before rather than after the enemy starts dealing damage. Therefore, in my list, I assigned Butaroid to Rank 7, and Melee slightly lower at Rank 8. The AI goes down this ranked list, and sees that Butaroid is closer to Rank 1. Therefore, the AI selects Butaroid as the action the enemy should take.
The AI then does everything it needs to do to make the enemy perform the Use Butaroid action. Once the action is complete, the AI just needs to do a few brief final checks in the Post-Execution section, discussed below.
Once the action completes, the AI has a small amount of maintenance it needs to do. First, it checks to see if the action it took just killed the player (i.e. if the player’s HP now equals 0). If it did, it triggers the Game Over screen as well as the prompts to reload and try again, then terminates. If it didn’t kill the player, it then checks if the enemy’s turn is over (i.e. if its RT equals 0). If it is, it runs some code to end its turn, refills the player’s RT, and transfers control back to the player so they can start their turn.
If both those checks pass, however, then the enemy is free to continue combat and take another action. The AI then resets all priority values, loops itself back to the First Stage, and the whole process begins again.
So that’s the basics of how enemy AI works. Hopefully that was interesting to someone. Anyways, here’s a few updates on how the game’s coming.
Current state of the game
Room-to-room transitions are done. That was the last big implementation I felt I needed to have the basic framework of a game. Now the player can enter a new screen, begin a fight, transition to combat, have combat fully complete, transition back to the overworld, proceed to the next room, and repeat. That feels pretty nice.
The Escape button on the player’s inventory now works. Each enemy has a minimum distance (using the Step Forward and Step Backward buttons) you’ll need to be away from them in order to Escape. Usually that distance is pretty small for melee-only enemies and territorial animals, and much larger for gun-based enemies and predators. If you reach that distance, you’ll be able to Escape and end the fight. When that happens, you are automatically booted back into the overworld and moved outside of the enemy’s detection cone. I need to do something to prevent the player from being able to use most of their turn on long-range sniper shots, Escape right before their turn ends, and then re-engage and get another full turn. I’m thinking about making it so that if you fight an enemy you’ve already fought, that enemy will be “expecting you,” and therefore will always go first in any subsequent combats. I’m still thinking about it, though.
Reloading works as well. That was the last of the player-side actions I needed to implement, which means player-side combat is now 100% functional. All buttons and a prototype model of every class of combat item is up and running. There are still a few flavor features to add and bugs to stomp, and I’m sure I’ll make a few last-minute design changes as we go along, but for now I’m pretty happy with where things are.
Healing events work. I’ve never seen another game do these quite the same way, and I think it’s pretty neat. A lot of RPGs do them like Fallout, where the game just checks your Medical skill value against another number and returns a yes/no judgment. I didn’t like that if the player doesn’t have enough to pass the check then they just can’t do it. I wanted make it possible for the player to succeed even with a lower medical skill, just that it’s harder. But I didn’t want any RNG (e.g. 25% chance to succeed with a low skill, 75% to succeed with a high skill); especially since the player could just reload and try again. So here’s what I came up with. When the player comes across an NPC that requires healing, clicking on them brings up a special inventory screen. On the right side of the screen appears an HP Needed To Heal number, which will vary for each wounded NPC. For example, someone with a sprained ankle might only require 5 HP to heal, but someone with a serious gunshot wound might require 50 HP to heal. The player can then place medical items into the special inventory, and the HP (still) Needed To Heal number updates in real time as the player places items into the screen. Here’s where the player’s medical skill comes in. The player has a Medical Effectiveness skill in their stat list. The amount that healing items heal the player goes up dramatically with your Medical Effectiveness skill, and that carries over to items placed in this special inventory. So a player with a high medical skill could fully heal a wounded NPC with possibly only a single item, while a character with a low medical skill may need to use nearly all of their healing supplies to help; but they can still do so if they want to. The fact that healing items are relatively rare I think also turns what used to be just a stat check into a bit of a moral choice. Especially if there’s more than one injured person on screen.
Enemies can now begin combat as well as players. Previously, the player clicking on an enemy was the only way for combat to begin. Now each enemy has their own invisible detection radius. If the player moves too close to them, the enemy will automatically detect the player and begin combat. Depending on the enemy and player’s Instincts stat, the enemy may go first in combat if they detect the player in this manner. One of the things I can use this for is ambushes.
I’ve got the player’s movement controller working, as well as placeholder frames for stutter-frame animation, though I still don’t know if that’s the way I’m going to go for animation. Fluid Flash-style animation is more professional, but it’s probably prohibitively expensive. Since the art style in this game has a realistic feel rather than cartoon-y, I don’t know if I can get fluid animations that would sufficiently look and move like actual people without a lot of dough. Stutter-frame is expensive too, since every frame has to be hand-made, but I could potentially minimize that with blurred transition frames. There’s also the fact that stutter-frame is a risk visually. It could either end up looking janky and low-effort, or genuinely add to the feel of the game like I think it does in games like The Cat Lady. I feel like I probably wouldn’t know until it all comes together in the end. My current plan is to commission a test before committing one way or the other.
Alright, that’s pretty much it for this post! I’m a little farther ahead than I thought I would be technically, but I’m making up for that by being a little farther behind than I hoped to be story-wise. I’ve still made some progress, though. I’ve got a general bullet-pointed outline for every branch in the game, and I’ve got a Story Design Document all set up with templates for every room, enemy, NPC, combat event, healing event, dialogue event, etc… just waiting for me to start filling out. I’m mainly at this point trying to fill a few final gaps in the puzzle that none of the current pieces seem to fit, and I keep getting distracted by the technical stuff for which holes are easier to fill in.
I’m also now working with a great musician for the soundtrack for this game. He just sent over an unmixed draft for the game’s main theme, which I think sounds fantastic. We have some neat stuff planned for music that dynamically changes with the environment, so I might talk about the music more in a dedicated future post.
Alright, that’s it for me. Thanks for reading, and I’ll see you in the next post!