The best thing about game design is the fact that you need to be proficient in many areas of software engineering. E.g. For example, when you develop a multiplayer online game, it is not enough to think about how to write your net code. It is also important to think about how to match your player and how to make sure your servers are available to everyone around the globe at low latency. AWS was used by us to create a global, on-demand server infrastructure called reaktron.
Matching Slow - Using Fast
For reaktron, we run a matchmaking service that is accessed by players looking for adversaries to play with. When this server receives a request, it matches players to form a group of regionally close players and starts an instance of the actual game server which is then used by the matched group to play on. The following image illustrates how this works.
1. Players contact the matchmaking server to find them a game. 2. The matchmaking server matches the players to groups and spawns a game server for each group in their region.
3. The matchmaking server waits for the game server to be started and returns the IP and port to the players.
4. The groups connect to their servers.
So, while the matchmaking server needs to be running 24 hours a day, 7 days a week, the game server instances are only started on demand. When a game ends, the game server instance shuts itself down. Also, latency with the game server instances is critical for the in-game experience while high latency during matchmaking won't be noticed. This means game servers have to run regional, matchmaking servers can run anywhere.
Cloud Hosted Game Servers
By hosting our game servers with AWS, we achieve three things:
1. You can deploy a global, scalable, and low-latency infrastructure, without having to rent servers from different providers around the globe
2. You shouldn't manage hundreds of servers
3. Only pay when someone plays our games
Since the matchmaking server can run anywhere, we will focus on how to get the game server running on AWS.
To run the game server Docker images, we opted in for AWS Elastic Compute Cloud (ECS) using Fargate. ECS is Amazon's solution to run Docker containers. Fargate is a special agent that starts and stops containers. It has its pros and cons.
Fargate isn't available in every AWS region, but in many.
- With Fargate, you avoid running your own Docker daemon on an AWS EC2 instance (this is a big advantage since the EC2 instance would have to run 24/7)
Fargate containers cannot be run with AWS Elastic Block Storage, (EBS).
Our idea was to use the AWS SDK within the matchmaking server to dynamically start instances of the game server with ECS/Fargate. To be able to do this, we first had to set some things up:
- Each region has its own ECS cluster that will run the containers
- Fargate has an ECS task description that explains how to start containers
- We built the Docker image of our game server and pushed it into an AWS Elastic Container Registry (ECR)
Hint: AWS Cloud Formation is recommended for setting these things up. This will automate the creation and maintenance of uniform stacks in different regions.
After implementing and running our first version of the container-spawning matchmaking server, we noticed that starting the game servers took too long. Our players couldn't wait for their matches to start for two minutes. Two main reasons were found for it.
1. Since our game is Steam-based, the game server instances had to run a Steam update on startup which took about 40 seconds. Updating Steam on each launch is necessary because the game server instances don't get re-used.
2. Fargate took a while to get our Docker Image from Registry before it was able to go ahead and start the container.
Optimizing Fargate Container Starting Times
To address the latter, we first stripped down our game server Docker image as much as possible. Second, we decided to use regional ECRs instead of just one ECR. So when the matchmaking server starts a game server in London, Fargate now also pulls the docker image from a London-based ECR instead of the central international one which speeds up things quite a bit.
We created AWS Simple Storage Service S3 (S3) Buckets for all regions. To fix the slow steam update, we built a simple service that updates steam, pushes new files to all S3 buckets and then shuts off itself. Fargate is configured to run this service every day. We also dockerized the service. We then updated the game servers to download the steam files from the S3 bucket in their region instead of running a steam update themselves. This brought the actual game server starting time down from 40 to about 3 seconds. However, there will be some Fargate overhead you cannot get rid of.
We ended up with Fargate taking about 30 seconds to start our game servers, which is acceptable. This time varies a bit, depending on whether the cluster is "hot" or "cold" when starting a container.
Data Retention from Short-Lived Container Arrangements
As our game servers store the played levels, we created another S3 bucket to which the game servers upload their data when a match has ended. Since this is not time-critical anymore, a shared global S3 bucket can be used, as opposed to the regional buckets which have to be used to download the steam files when the game server starts up. While it would be better to use all the S3 buckets and an EBS volume, Fargate isn't supporting that at the time of writing so we will have to resort to S3.
Low Latency, Low Cost, Low Maintenance
Our takeaway is that you can dynamically spawn game server instances using AWS ECS Fargate as long as your game server is not too large. Our game server docker image is about 70 MB in size. WEBARCHIVE AWS Cloud Formation can be used for describing and rolling out the server landscape in different areas. We are happy with the latency of the game servers and we only pay for the time users play our game. Also, we don't have to maintain our own docker daemons or clusters.