I'm a pretty casual MTGO player (read: losing player), big part of my engagement is enjoying following the competitive events. I can understand not displaying the full league data as it is kind of a privacy breach if you can look up someones exact results over time on MTGO, but not having full challenge data feels so silly when it's still available, just tedious to gather.
I'm not sure if non-5-0 league decklists were supposed to be visible (though indeed you could query them). Adding to your point, instead of reworking leagues to curate 5-0 lists why not make only 5-0 lists visible? The privacy concern here would be much more diminished since you're only publishing results that were already at risk of being published through the 'curated' system.
More technically, the bottleneck for retrieving match wins for a league run was from obtaining a
loginplayeventcourseid value that represents a player's league result, which you can only get from querying league decklists. To rectify this, you only need to restrict the decklists queryable for leagues, which also restricts all other league results. Compared to how league pages worked before, this also lowers the initial page-load time as the only lists queried (or possibly even scanned by a database) are the same 5-0 lists to be displayed.
[...] While I would personally love all the league data even just returning it to the state it was pre-API state that we had for months would be reasonable.
Roll-back to pre-API feels best, even if slightly disappointing as I was really looking forward to seeing the full data.
Just to clarify, the API was rolled out with the re-work of decklists way back on December 13th. There hadn't been any significant refinements to the API since the first few months it was rolled out. A creator had only recently discovered the API a few weeks ago (
reference) which created some confusion about this in the Twitter circles.
At the very least I want to see full challenge data return. This is data that can be ascertained as-is through watching replays, so why hide it and make things harder?
Besides replays, you can also recover most of this information from just the standings of the tournament. The event standings themselves give you a great deal of information on tiebreakers and pairings, which still gives you a sizeable snapshot of the metagame even with only top-32 or x-1 decklists known. For larger tournaments, you'll often have users who scout the top 32 or 64 players to get decklists or archetype names and post them on Reddit. This was pretty much the status quo when decklists were posted a day later over half a year ago, and even older than that, when MTGO Dailies and other events were first hidden
almost a decade ago.
However, this introduces another issue with the data -- we only know a tiny chunk of the top chunk of decklists, which means we can't observe a substantial bottom half of a deck's performance. This introduces a sizeable amount of survivorship bias, which combined with less total data means that the released results are quite misleading. There are a lot of emergent effects from this kind of bias, some of which are also helpful for reducing this bias, but for now I'll only cover the most visible effects to a player below by using a recent example.
A Trip Down Memory Lane
To illustrate this, we can compare the full Modern OTJ metagame (pre-MH3) with and without full event results:
The first thing to notice is that we now know only a vanishingly small amount of decklists in the Top 32 or x-1 or better results. From the full results (left) to truncated (right), we go from 6,048 players to 570 players (less than 10% of the original dataset). In terms of matches, we go from a total of 30,424 matches to only 520; that's roughly 65 matches per week from what was originally 3,800 (only 2% of the original dataset). Additionally, we go from 92 unique archetypes to only 36.
We can also notice that both the top decks' metagame shares and winrates change quite substantially, especially for less popular decks. While we have a relatively comparable idea of what the top 5 decks are in both examples, we have relatively little information about the less popular archetypes. What's worse is that this creates an artificial 'gap' between archetypes metagame shares that grows exponentially, hurting the appeal of less played strategies that fall under the tail-end of the distribution of decks.
In both cases we observe what appears like two fairly solved metagames when looking left to right. We also still observe this same behavior on shorter timetables of only a couple weeks, which still presents a similar pool rankings for the top few decks. What this comparison shows is which strategies are hurt the most, which are the lesser-played decks that fall under much of the rest of the metagame.
This is much more observable when you flip the rankings of the two graphics to instead reflect the Top 32/x-1 or better results:
The second set of graphics show this to be a self-reinforcing pattern as truncating decklists increasingly widens the gap in the metagame shares presented without any player intervention needed. In other words, this is an effect that happens just from hiding the lower distribution of results. But players still use these truncated results to strategize for the next few or several tournaments, which turns into a feedback loop where only the top few decks make it past both "player-selection" and the tournaments themselves.
To better contextualize this point, below is a simulation that shows how winrates change with a top-32 cut of decklists:
As only a small subset of the top decks remain salient features of the metagame after truncating results to top-32, this prevents break-out decks from further developing the metagame, dropping their playrate. What's worse is that truncating the lower distribution of results also inflates the winrates of the most played strategies (as shown in the graphic above), which further widens the gap in winrates presented. The more these gaps increase, the less likely it is for players to pick up other less played strategies and more likely it is for this loop to continue.
We've also observed this effect several times over the last several years after a large paper tournament publishes results. For example, we saw this happen both before and after Pro Tour Lord of the Rings nearly a year ago, where break-out decks from the Pro Tour broke a feedback loop in Modern from the past few weeks of MTGO results. For long stretches of time, the most played decks were synonymous with the most performing. We also saw the same pattern about a week ago at Pro Tour Modern Horizons 3, where the Modern metagame flipped from beating Ruby Storm to beating Nadu for the first time (also with only a partial albeit outdated view of MTGO results).
We can see that hiding this data doesn't actually slow down solving a metagame but instead hides the fact that the metagame isn't solved -- that there exists a wider pool of competitive strategies that can compete. Furthermore, it not only hides the diversity of other viable strategies in the metagame but stagnates their development by keeping break-out strategies out of results. Inevitably this leads to what players perceive as "solved" metagames much sooner than they should actually exist, though more critically, metagames are "solved" with less archetypes involved and thus less moving pieces to offset the balance.
An Open Letter to Daybreak
The information used to produce the above graphics brings us to is the least common denominator of a tournament: standings -- information needed for the competitive integrity of a tournament -- and decklists, which raises the question of why this data hiding is needed in the first place. With data hiding in place, players can still find the top several performing decks and observe metagames being "solved" in roughly equal or even faster periods of time, but what they can't observe is the wider pool of strategies the game offers to them. What I argue here is that it is not data analysis that stagnates metagames but data hiding.
I encourage Daybreak and the MTGO team to relay the same sentiment our community expresses to WotC, and to reconsider this change using the feedback in this thread to improve the API and continue to improve the game experience for players. With a game as dynamic and ever-changing as Magic: The Gathering, this decision reduces the pool of interesting cards and strategies to explore and interesting games of Magic accessible to many players, neither of which is an outcome the game creators intend.
