Hi Folks! I’m J Brian, and I lead the core technology efforts for the Crucible team. One of the things our team is responsible for is the game’s performance.
Generally speaking, performance refers to how responsive and smooth the game feels. Problems with performance, however, can take a lot of different forms and have a lot of different causes. Let’s start out by talking through some of the key components of a game’s performance and what we’ve been working on since May. After that, we’ll talk about some of our next steps as we move forward into a brighter, more performant future!
Almost every player knows that more frames per second (fps) is better. Framerate problems can take the form of low sustained framerate, or they can show up as “stutters” (a single frame that takes much longer than previous frames, which feels jarring). Our goal is that on machines that meet our recommended system specifications, using the default “high” graphic settings, Crucible runs at a constant 60 fps. On higher end hardware we should run much faster, and on the “minimum spec” hardware with default “low” settings we should run at a constant 30fps at least. Currently we are at 60fps on the recommend spec for the majority of frames, but too frequently drop below 60fps for periods of time or single bad frames.
In development, we usually look at “frametime” rather than framerate. In order to hit 60fps, each frame must take around 16.5 milliseconds or less. Our work revolves around reducing the time it takes to complete simulation and rendering of a single frame. We’ve done over a dozen micro optimizations for framerate, and overall we’ve reduced frametime by more than 15% on the recommended specification machines since launch, crossing the critical threshold where the majority of our frames take less than the 16.5 millisecond target. Most of these improvements came from either making the core systems and engine code that executes our game logic and rendering more efficient or making better use of idle central processing unit (CPU) cores in parts of our code that were unnecessarily bound to using only one CPU core at a time.
When your client (the copy of the game running on your machine) and the server share the same state of the game world accurately, we call them synchronized—they “agree” on the state of the world. A desync is when the client and server disagree about what the state of the world is for a given point in time. When this happens, the server issues a correction to the client, forcing it to adopt the server’s truth of the state of the world (the server is always the “authority” on the state of the world). This correction causes the world to warp unnaturally in some way from your perspective. You or another player might “rubber band” to a different place, an ability you thought did or didn’t happen might take effect or cancel, or some other gameplay value might change in a way that looks and feels unnatural. If desyncs happen for several frames in a row it feels especially bad and can cause framerate and bandwidth problems.
The realities of internet traffic and low-latency network synchronization models mean we will never reduce these to zero, but we can improve them. We found a number of situations where a single desync due to a dropped internet packet or a logic error could result in a chain of dozens of desyncs in a row. Fixing several of those issues has reduced our overall desync rate by more than half of what it was at launch.
When the game uses close to the limits of your computer’s system random-access memory (RAM) or video RAM, performance suffers badly, especially if other programs and systems are using some resources as well. Using less memory ensures we are less likely to approach those limits, keeps performance smooth, and could eventually allow us to run on a wider range of hardware.
We set a goal of reducing our memory usage from launch by 3 GB (which is a lot, considering we run on machines with 8 GB of RAM!), and as of our release on August 12, we have already reduced memory usage by 2.6 GB. The most difficult aspect of this work was implementing much better measurement and reporting code in our engine to give developers detailed data on what exactly was using all of the memory. Armed with accurate data about what systems and content were using what memory, more than half a dozen key areas of the game were optimized in both code and content to reduce memory usage dramatically without sacrificing the quality of the game experience. This work significantly improves the experience of playing the game on the lower end of our supported hardware range.
Most people’s internet connections fluctuate over the course of a match. When the game needs to send or receive more information than your connection can support at any given moment, information is lost and your game suffers desyncs and high latency. Reducing the bandwidth used by the game reduces the frequency of these problems.
One of the key optimizations we’ve made on this front so far dealt with a few key systems that were using much more bandwidth than necessary to synchronize their data.
Now that we’ve gone through the four factors—framerate, desyncs, memory, and bandwidth—that make up most of the player experience of performance, how do we know if our changes are working?
Without consistent measurement of our performance characteristics, it’s very difficult to know if we are making progress! We’ve been measuring performance carefully throughout development, and we continue to refine our ability to understand all performance scenarios across the game. There are three primary ways we track how performance changes over time:
We’ve improved our test automation, which automatically tests and gathers detailed data on performance of the game dozens of times per day, to more consistently and correctly describe performance. This data allows us to quickly catch when new problems are introduced and verify the effectiveness of new optimizations.
We supplement the automated data with daily manual performance runs, where our Quality Assurance (QA) team plays the game with special tools and performance recording code to ensure we have data from real game scenarios that might be missed by our automated test (which tests exactly the same conditions every time by design).
Finally, we track high level performance data from real world games on an internal web page to compare what players are actually experiencing to our expectations from internal testing.
Now that we’ve covered what contributes to performance quality, the state of each of those for attributes in Crucible, and how we measure them, there’s just one last topic for today: what comes next! Here’s some of what we’re working on right now, broken down by category:
In our internal tests on the recommended spec machine, we still find around 10% of our frames take 19 ms (remember that we’re shooting for 16.5ms or less), and 1% of our frames take around 23 ms. Going forward, we expect to make further use of additional CPU cores, which will reduce our baseline frame time, especially on higher end PCs with more CPU cores. We’ll also be doing some deeper analysis of the situations that result in “bad frames” to better address their specific causes. And finally, we’ll work on better balancing the load of the work that needs to be done on any single frame.
We eventually aim to support a consistent 144 fps on higher end hardware. In the short to medium term, we’re focused primarily on supporting a consistent rate of 60 fps on machines matching our recommended specifications, and the gains there will immediately contribute to higher fps on higher end hardware as well.
Over the next couple of months, we’ll be working to reduce the current rate of desyncs by more than half, which will make them rare and should have a noticeable impact on further improving overall game feel, especially in heavy combat situations. These optimizations will come from improving the prediction logic in cases where existing prediction is necessary but fragile, as well as removing some unnecessary prediction logic altogether.
As I mentioned previously, the goal we’ve set ourselves is to reduce Crucible’s memory usage by 3GB, and we’ve already gone down by about 2.6 GB. Through a few more planned code and content optimizations (specifically relate to how we load geometry into memory), we believe we can complete that goal within the next month.
In our next release, we will use some fast compression technology to further reduce our bandwidth usage by a whopping 30%. With that, we will be consistently meeting our goal of using less than 300 kbit/second of download bandwidth (and most of the time using far less than that).
At that point, we’ll mostly be done optimizing bandwidth for the time being, but we will take a look at how much we can improve the upload bandwidth used by the game. We use less upload overall than download, but some people have connections that are more upload constrained than download, so it can be worth further optimizing here if we find some good opportunities.
Thank you so much for reading along and for playing Crucible! We know that performance is critical to the experience of a competitive multiplayer shooter. We will never be fully satisfied, and will always be looking for new ways to improve performance in the game. If you are passionate about performance and have notes about specific detailed game situations that seem to have issues, we’d love to hear from you—the “#bugg reports” channel on Discord is a great place to share that.