Matchmaking-Improvements

Hey Champions! JShredz here.

We know that matchmaking has been a hot topic in the community. So, as promised on Friday, we’re going to share some details on how the system works today, what immediate improvements we’re rolling out, and what some of our long-term plans look like to build something bigger and better.

First things first.

Matchmaking is hard.

So let’s talk about it a bit.

What is Skill-Based Matchmaking?

Skill-Based Matchmaking is the process by which we assign teams for a match of Predecessor. When you hit the “Play” button, a message is sent to our Matchmaking Service with information about which region and Game Mode you’re trying to play, how many people are in your party, and your MMR (Matchmaking Rating). MMR is the number our system uses to try to assess your skill and experience in Predecessor based on your past wins and losses against stronger and weaker opponents.

Everyone starts at the same baseline MMR value, and from there your number constantly adjusts as you win and lose games, as the system attempts to find you the fairest matches for your skill level. You gain more MMR for beating opponents with a higher MMR than you, and drop more when you lose to opponents with a lower MMR than you. Eventually you’ll end up at a point where you’re winning about as often as you’re losing against players of a similar MMR.

But isn’t it hard to assign a single number to your skill level?

Absolutely! 

This is one of the great unsolved challenges in competitive gaming, especially for a team- and role-based game like Predecessor. Even the best athletes in traditional sports will rarely score exactly their average points per game, and video games are no different. 

Competitive performance in Predecessor is a mixture of natural skill, experience, your comfort with a particular team role or hero, the playstyles of teammates and opponents, your mood, your ability to communicate with others, your desire for intense competition or silly fun, or even how much sleep you got the night before! Assigning a single number to your chances of winning a match will likely never be an exact science, but the goal of our matchmaking system is to get close enough to create a fair, balanced game with every single match.

So does it all work?

Well, yes and no. 

The Glicko Matchmaking Rating system is a tried-and-tested method for assessing the likelihood of a team winning against any given set of opponents. Variations of this system are used in many of your favourite online competitive games. 

We currently use our own modified system based on Glicko, and we’ve found that we can predict a team’s actual probability of winning a match with over 93% accuracy for the vast majority of all Predecessor matches.  

Predictive ability does start to tail off for larger gaps in MMR between teams and wider MMR ranges within a match, but since this system works very well when teams are evenly balanced, our current focus is on improving that team balance, which will also improve the predictive value of the MMR rating. As soon as we’ve implemented most of the major improvements we’ll be talking about below, we’ll evaluate a potential move to other MMR systems to bring our predictive accuracy even higher.

This means that the matchmaking problems we’ve been experiencing are not with the MMR calculations, but instead with the process of matching players with a similar MMR into fair teams. 

When you queue up for a match, the system begins by searching for players and parties closest to your MMR value. As your ticket moves through the queue and the matchmaker starts to assemble a potential game of 10 players, it navigates the difficult tradeoff between holding out for teammates who are more similar to your skill level and ensuring that you’re able to find a match in a reasonable amount of time. 

While some players might advocate for longer matchmaking times and slower MMR range expansion in exchange for matches with a narrower range of MMR, there is a practical limit to how balanced you can make matches given a certain population size and amount of time players are willing to wait before giving up. We made a conscious decision at the beginning of Early Access to bias towards faster matchmaking times to ensure players can play more matches of Predecessor while we continue to develop the game, and while we still believe that to be the right course, we acknowledge that the quality bar for match fairness has suffered as a result. 

We’re sorry, and we’re working hard to fix it.

So, what are the issues?

Well there's a couple.

Grab your net, we’re going on a bug hunt.

First and foremost, we need to discuss a significant bug that up until recently had silently plagued our matchmaking system. 

The matchmaking process can be broadly split into two distinct stages: 

1. The service finds 10 players with comparable skill levels based on similar MMR.
2. It balances those players into two fairly-matched teams of 5. 

Occasionally, the matchmaker would overwrite the MMR value for a player with a bogus default value while trying to update the “matchmaking request ticket” and pass it between those two matchmaking steps.  When that MMR value was overwritten for even a single player, the entire process of rebalancing teams to make sure those 10 players were evenly sorted was thrown out the window, and players were randomly grouped into two teams. 

Sometimes the random sorting was reasonably fair, but often a match that could have been nearly perfectly balanced if appropriately mixed ended up with differences of several hundred MMR, with all the higher-MMR players on one side and the lower-MMR players on the other.  

What made this bug particularly nasty to catch was that our own Predecessor game clients and servers continued reporting all the correct MMR values to our own data pipeline, while that bug lived deep within thousands of lines of external backend code. 

Thanks to the collaboration between our own internal teams and the matchmaking provider, we were ultimately able to track down the bad code and develop a fix. This change was rolled out by the matchmaking provider less than two weeks ago (on July 12th), and the improvement to match fairness was significant enough that we thought it was a good time to write up this blog and talk about our progress and plans.

Inflation hurts

Beyond that significant team-sorting bug, we’ve also been slowly combatting something we’ve been calling MMR Inflation. 

Every player started at the same initial MMR when we launched Early Access, but over time the average player MMR has shifted several hundred points above where new players begin their Predecessor journey. Having the player base average somewhere above the initial MMR value is natural, as a player who wins more frequently is more likely to keep playing Predecessor, while a player that struggles consistently is less likely to play as often. 

However, much of this inflation was the result of accidental introduction of “extra” MMR into the system due to an error in the initial version of the post-match MMR adjustment calculation for parties. We were able to detect and fix this error, but while the improper calculations were out in the world quite a few players were moved into MMR brackets that were well above their skill level.

The system is still in the process of rebalancing. Once players reach their correct MMR and the extra inflation is fully corrected, match quality will improve and queue times for new players will decrease.

3’s a crowd, 4’s a party

The Party Multiplier has been another hot topic in the community. 

Our data has shown a strong correlation between party size and the odds of winning a match at a given MMR, where larger groups of partied players gain a significant advantage over ungrouped players of the same level.

This is due to a number of factors including faster and more accurate communication via platforms like Discord, advantages in coordinated role or hero selection, and plain old team chemistry that comes from playing with your friends.

With the goal of compensating for these advantages, we rolled out a Party MMR Multiplier system that inflates your team’s MMR for matchmaking purposes as a function of how large your party is. Essentially, a bigger party gains a bigger “boost” to their MMR to find stronger opponents that we believe makes for a fairer fight.

We’ve found this initial version of the system to be extremely effective at balancing out parties of 2 and 3 in most MMR ranges, but the MMR adjustment for parties of 4 and 5 has proven to be too large for teams of newer players and not large enough for very skilled parties. We’ve been developing a new Party Multiplier method that we’ll break down further in our Future Plans section below, and we’ll be rolling that out in the upcoming V0.9.1 game update.

Practical limits

Lastly and as you might expect, there are regional and time-of-day population limitations that we need to design around. 

While we’d love to configure our systems to make the absolute best matches during peak weekend evening hours, the reality is that we also need the system to function at 3am on a Tuesday in our smaller regions. 

Within any single region, the population difference between the busiest and quietest hours can be a factor of 10 times or more, and building systems that work well at both ends of that extreme will remain a challenge even as we grow our player base into console launch and beyond. 

We also need to set a limit to how high or low an individual’s MMR can go to ensure players at either end of the MMR spectrum are still able to find matches without unreasonable queue times, even though it does mean a loss in some predictive accuracy compared to the estimates of third-party sites using the Predecessor API. 

There is a constant balance between matchmaking queue times, matchmaking quality, and player latency (“ping”) as determined by the number of matchmaking regions, and optimising all three of those factors takes time and experimentation.

Developing and Testing

To tackle something as complicated as matchmaking, we put together a task force of developers from our Game Design, Analytics, and Backend Engineering teams. 

Turning player feedback on matchmaking quality into meaningful solutions first required a stronger understanding of the system as it previously existed, so we began by extracting and analysing data on every aspect of matchmaking quality we could find, including team-to-team average MMR differences, the range of MMR values within a match, post-game survey results, queue duration, queue cancellation rates, and the effects of party composition on all of the above. We were able to get direct access to the codebase of the matchmaking tool we were using, worked with the matchmaking provider’s Engineering team, and collaborated with outside experts that have helped design and improve the matchmaking systems of a number of massively successful competitive multiplayer games.

Once we felt we had a comprehensive understanding of our problems, the next step was designing and testing solutions. Our team recorded many hours-long stretches of real Predecessor matchmaking requests and built a simulation testing platform from scratch where we could replay exact copies of those requests through different iterations of the matchmaker. We can run these long simulations with many simultaneous configurations and analyse the data after each set, allowing our team to continuously test hypotheses and iterate quickly without affecting the live Predecessor game services.

New and Improved

As mentioned above, we’re thrilled to announce that the first improvements have begun rolling out. With the primary team rebalancing bug addressed on July 12, the median MMR difference between teams (the blue line on the graph below) has dropped from 60 MMR points to less than 20, taking expected win rates from 55% for the higher-ranked team to nearly completely even. In the worst 10% of matches (the red line), that difference has gone from 160 points and an 80% expected win rate for the higher team to 70 MMR and a 57% expected win for the favoured team. These improvements aren’t sufficient to satisfy our goals, but this change represents a big step forward for overall match quality.

To combat MMR Inflation, we ran a database change of everyone’s MMR to move those stats closer to the starting value. 

The ranking of everyone’s MMR relative to the rest of the player base was unaffected, but by shifting everyone back down by a proportional factor and tightening the distribution, we’ve been able to cut average matchmaking times in half without meaningfully affecting the group of players you’re most likely to be matched with.  This improvement in matchmaking time also allows us more room to configure stricter MMR matching requirements without creating excessively long queues, a process which we’ve already begun working on.

The last immediate improvement is the unveiling of our new Party MMR adjustment with Version 0.9.1. This is a formula that accounts for both the party size and the party’s average MMR when calculating the adjustment factor, and is scaled to better align with the match data we’ve collected, recognizing the reality that parties with high average MMR are far more likely to leverage their additional coordination than parties of less experienced players. This should be much more forgiving for newer players, but might increase queue times for large parties of our best players as the system attempts to compensate for their advantage. Our backend team has specifically built this version of the Party MMR system to be far more customizable, allowing us to quickly react and adjust for the MMR and team size combinations that fall out of scale with our desired 50% win rate.

Future Plans

The improvements to team mixing, queue times, and party adjustments are a great start, but we’ve got much bigger plans in store. 

The simulations we’ve run have yielded a number of promising configuration changes that we’re incorporating into an upcoming release, including a change to reduce the number of players that are accidentally pulled into matches well outside of their actual MMR range. We also plan on exploring matching like-for-like party sizes against each other, but our early simulations show that its simplest form may actually lead to worse overall match fairness by restricting the number of available match partners, so there’s more development and experimentation to be done before we pull that trigger.

We’re developing new systems to allow us to dynamically adjust matchmaking configurations to better accommodate time-of-day and regional population requirements, including the ability to adjust parameters through both daily and weekly cycles. These improvements will allow us to adapt to geo-specific requirements by loosening parameters to favour matchmaking times during off-peak hours and tightening match quality during busier stretches.

Last and largest of all, alongside the improvements for the existing matchmaking system our team has been developing and configuring an entirely new matchmaker. This new system necessitates major backend changes to our service architecture in addition to significant development and tuning work, so we don’t have a release date yet, but we’ll be putting it through the same simulations as the current matchmaker and will roll the new system out once we’re convinced it outperforms the current model.

Thanks as always to everyone that’s supported Predecessor throughout the Early Access journey. Your feedback continues to be invaluable as we develop this game into everything we know it can be. 

Please reach out to me on Discord with any comments, suggestions or concerns whenever you have them.

Additionally tune in to TheBeardedWolverine’s PredCast livestream on Wednesday, July 26th at 7:30pm ET where I’ll be on to discuss all these changes and more.

~ JShredz, Senior Manager, Live Production and Game Operations