Online Player Rankings Thread

[kevindola]

First,

I want to say thanks for all the organizers for spending their time and effort to put all these events on, and on using a rating system at all. I find it fun and added value to the experience. I also know that they are constantly improving the online gaming process and making the experience more enjoyable (such as adding the log of updated games to track when games are being played) I hope any of my comments didn’t come off as complaining as I just wanted to ask some questions about the system to try and get a better understanding of how it is achieving its ratings and point out what I perceive as concerns with how it evaluates people based on amount of games played.
Now to address some comments and add some final input. Everyone has agreed that the greater number of games played DOES positively affect your overall ranking (for those who have a positive rating). This is true to a point.

Originally Posted by Kinseth

It eventually works itself out, the only explanation for why some players with worst win % are ranked higher than others is because some players may have played harder schedules.

I think everyone believes this, however the concern is the ‘eventually works itself out over time’ part. The argument is that it takes too long for the 4-5 game average per tournament that our players are seeing for equally skilled player’s ratings to equalize. And by extension, those who have a worse skill but have more games played will have a higher rating than those with more skill and less game played….at least until they hit the level of a low confidence variance.

Originally Posted by Kinseth

Kinseth, Filthy, Bengi, Awesome unleashed are already in because they won an event. Therefore the rankings affect the other organizers(Dok, Mega, Xorlof). Dok clearly has earned his way no matter how you dice it. So therefore we are taking a system that now can exclude Mega and Xorlof at this point. I don't see how this system now benefits the Organizers, when it puts two of them at jeopardy.

I don’t believe the organizers intentionally are manipulating a system to ensure they get in. However, this system does benefit people who have played longer because they have some rating advantage over equally skilled players who have played less to some degree (I am fairly certain that no one is refuting that sentence). Therefore, it easy to conclude that the people who were participating in tournament in their inception (the organizers) inherently have some kind of advantage based on a system that is influenced by games played. (I assume the bottom end of the spectrum is full of ‘bad’ players who have played a lot of game as opposed to 0-3 players)

Originally Posted by Foudzing

The Xbox live system is not appropriate at all for our community with very few players and very few games played.
That's what I feel when I look at the rankings.

I feel very strongly that this is root cause of the issues revolving around the critiques of the rating system being used. I have little evidence other than analyzing the W/L ratios and games played, but I did find this stating convergence taking on average 12 games and this comment from here

Originally Posted by XBoX FAQ

Overall, we observed in our experiments that the sum of these effects leads to an increase by a factor of 2 - 3 in the numbers of games necessary per gamer.

I infer from those statements that convergence or minimization of confidence can take between 12-36 games. So my conclusion is that it takes too long for the fringe players to get the accurate separation in rankings based on skill over games played. My worry is that we do not get enough games in during the year to have the variance factor minimalized.

Originally Posted by dok

So, given that an official format and qualifying structure wasn't announced until late August, what are you actually suggesting here? Are you suggesting we should only use season 9 and season 10 and ignore everything else?

The reasonable conclusion would be not to have an EOY championship for 2013, however I don't think anyone wants that, and I am not purporting we use any other system than what has been proposed. I just wanted to voice my concerns.

I am very curious to see what the top 25 looks like using only the method described earlier this year using a points method based on playoff performance. And then comparing that list to what we currently have.

Though, I am not asking for that because it would open another can of worms on which seasons to use and how to evaluate seasons which did not use a playoff system.

In conclusion, I would like to add that Megasilver has no sense of how to play vipers and couldn't frenzy himself out of a paper bag with 12 stacked Lodins. So I can't fathom how he made the top 25, other than to conclude there is some kind of variable that increases rating based on slithering figures used.

Spoiler Alert!

[MegaSilver]

Originally Posted by kevindola

In conclusion, I would like to add that Megasilver has no sense of how to play vipers and couldn't frenzy himself out of a paper bag with 12 stacked Lodins. So I can't fathom how he made the top 25, other than to conclude there is some kind of variable that increases rating based on slithering figures used.

Spoiler Alert!

Be very glad I saw that as a joke quickly.

[dok]

Interesting stuff as usual, kevindola. Just a couple quick comments:

Originally Posted by kevindola

I assume the bottom end of the spectrum is full of ‘bad’ players who have played a lot of game as opposed to 0-3 players)

Actually, the bottom end of the rankings (which I am fairly adamant that we will never post, since it just seems like pointless public shaming) is mostly 0-3 and 0-4 players. Very few players who consistently lose keep playing and build up a large record. There are a few exceptions, but not many. The bottom four in mean ranking are a collective 5-38, but the next seven players are all 0-3 or 0-4.

Also, a low variance improves your conservative skill estimate whether you're bad or good, so those bottom four in mean rankings are not the bottom four in the Trueskill rankings.

Originally Posted by kevindola

I am very curious to see what the top 25 looks like using only the method described earlier this year using a points method based on playoff performance. And then comparing that list to what we currently have.

Though, I am not asking for that because it would open another can of worms on which seasons to use and how to evaluate seasons which did not use a playoff system.

You know, to be honest I'm curious too. Not curious enough to go through and figure it out, but curious. Honestly, I doubt it would be very different.

Originally Posted by kevindola

In conclusion, I would like to add that Megasilver has no sense of how to play vipers and couldn't frenzy himself out of a paper bag with 12 stacked Lodins. So I can't fathom how he made the top 25, other than to conclude there is some kind of variable that increases rating based on slithering figures used.

At least we can agree on something.

[Kinseth]

Better post coming

[MegaSilver]

It's average what? Rank of the opponent?

[Kinseth]

Record, those numbers are wins and losses.

[vegietarian18]

This is a quick (kind of) list of what the Top 16 would be under the playoff point system. If anyone thinks I screwed up their score, let me know. This was done on my phone so it's quite possible I missed something. I started from Season 5 like it was said the points would, but gave auto qualifications to older tournament winners.

1. Sparkle- 20 points
2. Wanderer 17 or 20 points pending finals
3. Foudzing 12 points
4. vegie's dad 10 points
5/8. Toogwick 7 or 10 points pending finals
5/6. Dok 9 points
6/7. Infectedsloth 9 points
8. vegietarian18 8 points
9. Robber 6 points
10. Dysole 5 points
11. Greygnarl 5 points
12. Kinseth 5 points
13. LegendaryIrishSword 5 points
14. Filthy the Clown 3 points (qualifies on Season 1 win)
15. Soundwarp 2 points (qualifies on Season 4 win)
16. awesomeunleashed 1 point (qualifies on Season 2 win)

It is pretty similar to the current Top 16.

[dok]

Interesting stuff.

FWIW, when the organizer group was started, we started numbering our seasons from there, but we retroactively recognize the "First Online Heroscape Tournament" as "Season -1", and the "Monday Night Googlescape League" as "season 0". So Kinseth and Bengi are both also considered former champs.

[vegietarian18]

Hmm, Bengi being in makes things more complicated. He has 3 points, so he would be #14. It seems unfair to kick out one of the 5 point players, so I'd say play-in between Soundwarp and AU, but hope that Robber/someone else can't play. .

This system rewards streakier players; that is people who have one good run. People like LunaticFringe, capsocrates, etc. that don't play many seasons were only one point out of making it, Robber made it on a single good run, but in contrast some of the consistently good players like Deroche are out. I guess it's an okay way of doing it.

(Really, the only reason I was so frustrated earlier is because I THOUGHT this was going to be the system, then it was changed on me after I stopped putting in the same effort for two seasons, and now I'm almost out of the playoffs in the other format. The current system is fine, it just would be a lot better if it was announced sooner/only pulled from recent seasons.)

[MegaSilver]

Originally Posted by vegietarian18

This system rewards streakier players; that is people who have one good run. People like LunaticFringe, capsocrates, etc. that don't play many seasons were only one point out of making it, Robber made it on a single good run, but in contrast some of the consistently good players like Deroche are out. I guess it's an okay way of doing it.

Yeah, that's not fair, is it?

This is why the system we are using is better. I understand why you could/would/was frustrated though.

[awesomeunleashed]

Originally Posted by vegietarian18

(Really, the only reason I was so frustrated earlier is because I THOUGHT this was going to be the system, then it was changed on me after I stopped putting in the same effort for two seasons, and now I'm almost out of the playoffs in the other format. The current system is fine, it just would be a lot better if it was announced sooner/only pulled from recent seasons.)

That's perfectly understandable, although I'm not sure why you would put in less effort once you thought you were in. FWIW, it's not like we knew that this was going to be the system much earlier than you did; I'm pretty sure we announced it very soon after we had finalized the decision (Edit: About 11 days later).

[Kinseth]

I'd like to illustrate a point about Win Loss records not painting the whole picture. If all things were being equal, everyone's opponents would have a .500 record when they do not play against you.

Let's take for instance Foudzing and compare him to Megasilver.

Foudzings opponents records(- the games they played foudzing) are 405-293. This makes sense because Foudzing is often in the playoffs and facing better players in those playoff rounds who generally have above .500 records.

Looking further into the records and separate opponents based on when Foudzing Wins, when he loses and compare that to his average.(Not including the actual record of the game having been played in these totals.)

Foudzing Wins vs Opponent - 10.3(w) 8.5(L)
Foudzing's Opponent wins - 18.4(w) 10.7(L)
Average win/loss of opponent - 12.65(w) 9.15(L)

Based on this, we can see when Foudzing wins it is against an opponent that generally has 54% win percentage. When he loses its against an opponent who has the winning % of .63%

Overall his opponents winning % is .58, if all things were being equal, his opponents should be .500

Let's look at Megasilver, his opponents have a record of 570-418. This is almost the same winning % as Foudzing, yet Megasilver clearly has a worse overall record than foudzing. How can his opponents win % be so high? The answer is that early on he faced Bengi/Dok/Kinset/Filthy etc at the beginning and thus the system recognizes this. Let's look a little further at the breakdown.

Megasilver win vs opponent - 12.65(w) - 9.55(L)
Megasilvers opponents win - 14.4(w) - 10.3(L)
Average win/loss of opponent - 13.5(w) 9.95(L)

When Megasilver wins, it is vs a player that has ~56% winning percentage, which means his wins are not "Cheap" wins. The system can tell that he isn't just rolling over cupcakes(If this was the case his opponents win % would likely be lower than 50%). After that we can see that his average losses are to opponents are 58%. This its generally to more of the better players.

The win loss % of his opponents is clearly higher than .500, and thus cannot be completely taken at face value. Not all win losses are equal, just the same as College football teams and the strength of their schedule.

While it is nice to look at the Win/Loss for a quick snapshot, it doesn't tell the true story without diving in and looking at the #'s.