Ok, so I need a stat's expert to double check me here, as I'm not entirely confident I'm not overlooking something in this idea (and no I haven't worked through any of the math yet):

(see http://en.wikipedia.org/wiki/Estimating_sample_sizes#Estimating_sample_sizes)

By using Cohen's d and the sample size (# of games played) of the data we can get a power level for the statistic. We can then use that power level to weight the normalization.

For small sample sizes with small variations in the data the power level remains small and thus the normalization would have little effect on the score. For larger sample sizes and for larger variance from the expected result (equally balanced), the power level would increase, thereby increasing the weight of the normalization.

For example, look at the table of sample sizes in that article and note the sample sizes and corresponding power levels. Using something like this, it would seem that normalization would have a significant effect only where the data was statistically relevant.

While I do agree with Guaroz that the performance ratings currently aren't that adequate for properly determining imbalance, they are a starting point. I do think that some tweaking of the performance rating would likely be required however, and agree that SC totals in the draw/survive would be nice to have tracked as well.

Also good rated players could give low rated countries a chance, because a win would score much more, and a loss wouldn't hurt their rating that much.

And because of this the stats would get better each time.
At the moment no one like to take bad-rated countries in a choose-your-country game.
This would be no problem anymore.

I second that

Btw have you considered other game point systems like bowling, fantasy football,?
and if you are arguing about perfomances of each country then do you propose to sit down and analyse every country of every variant?

maybe we should split the variants in 4 -5 categories perhaps according to diffiuclty or equality of map (Chaos)

Analysis of each variant can be done "on the fly". I think nobody wants todo this by hand.
Because we have all games already in our database we can assign everybody a rating based on his already played games.

That is interesting. I didn't realize you could use the existing database to create the initial rating. I thought we were going to start over. I like that.

By the way, the Elo rating system was named after Arpad Elo, for Chess, I think the Ghost-rating system was made by TheGhostMaker on Webdip. I think this system should not be named after them, it should be named after you: Oli-Rating, for vdip.

Hey Leif, I didn't mean THAT sophisticated! I think that any number of games between 15 and 20-25 is enough, because each Country would have at least 15 SC's-records for its Average. Let's keep it simple!

"SC totals in the draw/survive would be nice to have tracked as well. "

Well, Leif, I didn't mean "only in the draw/survive", it's in every game! Draws and survives were just examples.

You could say for Eliminations likewise arguments I said for Draws and Survivals:
- in PPSC, an eliminated Country is like a survive with 0 SCs.
- in WTA, while for a PLAYER a defeat is a defeat, for a COUNTRY there's defeat and defeat. After a game ended with a solo, for our stat a Country that ended with 0 SC has a different potential than a country that ended with 16 SC and maybe fought until the end for the win. While their players had the same result (defeat), the countries didn't have the same performance.

Even easier for Wins. If a player won, then his country ended with many SCs: just put that number into the stat. Did a player win a Classic with only 6 SCs just because everyone else voted concede in 1902 (or left/resigned)? The current rating-system hugely overvalues his Country's potential (and hugely undervalues the others), totally screwing that Variant's stat. Unless it has an enormous sample.
Average-SCs system would be much less mistaken, even with a small sample.

Corner cases are much better handled. Think of games abnormally ending too soon (because of a Victory Condition ridicolously low, or 4 players deciding to draw because the other 3 CDed very soon, or a strange "massive concede", or whatever...). Such games would heavily damage Countries' stats in the W/D/S/E system, while they'd affect Countries' SC-stat just a little.


So the point of what I said in my previous post about Country-rating is that we need a stat thinking like a Country. Stop thinking like a Player!
W/D/S/E may be good for Player's rating, because it reflects how he behaved in a game.
What reflects Country's potential (in a stat) is "with how many SCs the Country ended the game", no matter if its player resigned/won/conceded/survived or whatever.

(well, it matters a little, obviousely. But... say you miss last 2 turns of a Classic and go CD -resigned- when you got 15 or 8 SCs. So they play the last turn, the game ends, and your Country ends up with... say... 13 or 6 SCs. Now the current system records a "Eliminated" for your Country, because it mixes up players and countries, while the AverageSCs-system would record "13" or "6". Much less mistake! Not Country's fault if its player resigned! Or if he conceded... It's COUNTRY-Rating: we must think like a Country!)

Another idea would be to value a country on it's pot-share.
A country that just manages to gain 5 SCs, but always manages to reach a draw might be a better value than a country that gains more SC, but gets eliminated more often...
---
I have s rough idea how to handle this without the different country-values and will try to code this the next few days. Than we can try to add a parameter for country-performance and see how the rating differs...
---
And IMHO we should name this Vrating... :-)
---
Also this will not replace DPoints. Usually after enough games played your rating exactly represents your skill. Usually there is not much to gain if you win more games. The top players can't get their rating any higher, usually it goes down.
With the DPoints you always get something for your stats if you play well in your game and it's more fun this way, even if it's not an accurate measure of your performance...

Guaroz,

I agree with you that our current performance rankings likely do not always adequately capture the approximate performance of the country, however I proposed using them as a starting point (partially to see what the impact of normalizing based on performance had on the rankings). Seeing as we currently don't have the SC count statistics you propose (and I likewise believe would be more useful), I figured starting with the current performance values would be better than nothing in the mean time.

Just out of curiosity, I cranked the SC count calculations by hand for the Africa variant (one of the more imbalanced variants that I'm familiar with that has a manageable number of nations/games to do the stats by hand, but enough games played that the stats should have some statistical significance.)

The 'country performance' statistics come out as follows for the 11 finished games:
Country............Total SC's........Avg. SC's/Game......Avg. % of map (63).....Avg. % of win (33)
Morocco...........274..................24.90909091...........39.54%.......................75.48%
Madagascar.....112..................10.18181818...........16.16%.......................30.85%
DRC.................98....................8.909090909...........14.14%.......................27.00%
Ethiopia............84....................7.636363636...........12.12%.......................23.14%
South Africa.....69....................6.272727273...........9.96%.........................19.01%
Mali..................32....................2.909090909...........4.62%.........................8.82%
Egypt...............20....................1.818181818...........2.89%..........................5.51%
Nigeria.............8......................0.727272727...........1.15%..........................2.20%

Country.....SC Results in each of the 11 games:
Morocco...........35....22....13....35....15.....0....32....35....33....33....21
Madagascar.....15.....5.....33.....4......0......6....12.....9......9.....19.....0
South Africa......0......0......0......0.....16.....6....10.....3......0......0.....34
Ethiopia...........11......0......0......1......0.....36....0.....15....11....10.....0
DRC..................0.....28.....0.....23....20.....0.....9......1.....14.....0......3
Egypt................0......7......1......0......12....0.....0......0.......0......0.....0
Nigeria..............1......1......0......0.......0.....0.....0......0......0......1......5
Mali...................1......0....16......0.......0....15....0......0......0......0......0


For comparison here are the current performance statistics:
Country........Solos...Draws....Survivals Eliminated....Performance*
Morocco..........5..........3...............3.................0...................8.45
Madagascar....1..........2...............6.................2...................2.82
South Africa....1..........2...............2.................6...................2.45
Ethiopia..........1..........0...............6.................4...................1.91
DRC................0.........2...............3.................6...................1.18
Egypt..............0..........2...............1.................8...................1
Nigeria............0..........0...............4.................7...................0.36
Mali.................0..........0...............3.................8...................0.27
*Performance = (15 x Solos + 5 x Draws + 1 x Survivals) / Games


It is interesting to note that Mali's 3 survives (1SC, 16SC's, 15SC's) make it's performance 4 times better (in this sample set) than Nigeria's 4 survives (1SC, 0SC's,1SC,5SC's -- the extra SC in Nigeria's total was from a resign currently counted as an elimination).

Additionally, Mali's 2 substantial survives even indicate that it currently outperforms Egypt, despite Egypt having 2 draws (7 SC's and 12 SC's) and 1 survive (1SC).

Lastly Congo's performance leaps to 3rd place from 5th place when its 28 SC resign (resulting in a draw of that game by remaining players) is included in the statistics and not brushed aside as an elimination.


Now, however, comparing the results between the two sets of statistic from an overall perspective, the general weakness or power of a country is still more or less valid between the stats.

Morocco is the clear front runner in both cases.

Madagascar, Ethiopia, DRC, and South Africa are all middle of the pack and probably fairly evenly balanced, though they shift within this grouping between the two cases.

Mali, Egypt and Nigeria are all low performers in both results.

Thus while the SC count statistics may be more accurate representations of true performance, the current stats are still relatively decent indicators and could be used for my suggested normalization without additional work, (but also later replaced with this more accurate stat of country performance, I would propose).

Just a few thoughts, and obviously I just examined a single case so I don't know that all variants follow this pattern, but I doubt it'd be extremely different in most cases.

Egypt being ELIMINATED on the WW4 map in 23 games out of 27 means nothing. Aside from the fact that 27 games is a ridiculously small sample to make any concliusions on whatsoever. Of those 23 eliminations I can think of at least six where the sole reason for elimination was 100% player performance related.

Gtting fixated on the perceived and unproven strengths and weaknesses of starting positions is absurd minutae that measures NOTHING of what goes on during the game. There are so many other worthy elements and achievements to track and consider. Why on earth would you complicate it with tthis base nonsense?

Leif,
I think there is some value in what you are proposing, but I think we have to keep in mind that Migrane looks very similar to Africa, both in the SC count and the performance rating. If it wasn't obviously symmetrical, we might be debating that Migrane was an extremely imbalanced variant.

I played as Egypt in one of the two times I played that map. I very rarely look at stats, and by judging the map fresh in that game I perceived myself as having a great starting position. It requires work, as there was lots of neighbours to talk and different angles to consider. But isn't that why we play the game, for these types of challenges?

It seems on the forum in every thread people are complaining about how weak various powers are compared to their opponents on the map. This game is about diplomacy and negotiation, so I have no sympathy when people complain about their draw. Man up, play smart and take it to the endgame.

Gunboats might be a slightly different story, but even I think should mostly be a non-factor. Once these maps all get more use and players get more experience even they should get close to parity, I think.

Ihis community is fixated on the issue of imbalanced positions. This entire assertion is utter rubbish. Did some mean bully come take away all your toys when you were children. At what point does player respsonibility and accountability come into effect for you?

The standard Diplomacy map is as perfectly balanced it can be, and has been paly tested hundreds of thousands if not millions of times. If it were imbalanced in any way, it would be clear as to which nation was weak, and the game would have never become so popular. The common assumption is that Italy and/or Austria are weak nations to draw, but this is inaccurate and flawed. And getting all geeked up on statistics proves nothing. The beauty of this game is that even the very best of players drawn into the commonly percieve strongest starting positions can be beaten on any given day based upon the complex interaction of all of the other players. There is no way to qualify what occurs in the process of diplomacy that statistics can track. How do you convert the emotions of trust, and betrayal, or their applications to strategic and tactical tactical skill and skill into math? It's impossible.

@Lief: for the average supply center count, in the result of a draw, since the pot is split equally, the supply centers should be divided equally. This should correct the problem of Egypt having a 7 sc draw decreasing its average.

Also I agree with Ruffhaus, when a country is really imbalanced, I reflect that in my diplomacy, in the Africa map for example, I would be less likely to ally with Morrocco because of all those solos, I would probably start an early game alliance against him and he would get killed.

This game is about diplomacy, you have to convince people to ally with you, and that is more important then any starting position.

It seems people feel I am advocating that the un-normalized Oli-rankings wouldn't be fair because of imbalances on the maps. This isn't the case.

What I am proposing is that we do these standard unmodified Oli-Rankings (or Auth-Ratings as an alternative moniker?) and simply provide an additional rating alongside that is normalized based on a country performance stat. Those who feel that imbalances are significant can flaunt their normalized rating, while those who feel that imbalances are of no import can champion their unadjusted ranking. (Admittedly this really only provides fuel for the argument 'unbalanced vs skill', but that argument will be around no matter what you do, just as the gunboat vs full press argument will survive to perpetuity).

My own interest is more curiosity than anything, along with providing ideas to meet the desire of as many in the community as possible, and being able to capture as much information as possible for design purposes.

I too feel that maps with imbalances are not only still playable from all positions, but with the right diplomacy and a bit of luck, winnable from all positions. Not only that they can present a fun challenge, and providing incentives to pick up ugly positions, perceived or real, in 'choose your country' games is not a bad thing, and can help correct the perceptions that countries such as Egypt in WWIV and Morocco in Africa have gotten.



fasces,

in regards to: "for the average supply center count, in the result of a draw, since the pot is split equally, the supply centers should be divided equally"

I would agree from a player perspective, however from a country perspective as Guaroz indicated (and as I tried to present the stats above), being part of a draw with 1 SC doesn't indicate the same performance as being part of the draw with 1 SC short of the win (32 in the case of Morocco in column 7 above). From a player perspective I would sometimes count a 1 SC draw position more of a personal victory than a 32 SC failed solo attempt.


Ruffhaus,

"Did some mean bully come take away all your toys when you were children. At what point does player respsonibility and accountability come into effect for you?"

No need to be attacking us, lay off a bit!! I am simply throwing ideas out that can provide more data and meet more people's expectations and desires for a ratings system with the least amount of unnecessary work possible. I agree that some people complain overmuch about their draw on maps, but in my opinion, this sort of normalized ranking would likely provide more evidence that such players are simply less skilled diplomats/strategists/tacticians, and would still be in the bottom of the rankings.

"The standard Diplomacy map is as perfectly balanced it can be." Agreed it has more than enough play testing behind it to indicate it's balance, but many of the variants we have implemented here do not have that play testing behind them, and either intentionally or unintentionally still have imbalances. More data simply allows us to examine the playability of the variants and possibly correct any real imbalances should the community desire to do so. Surely you aren't advocating that all our variants are perfectly balanced are you?

Before this goes any farther, I'd like to point out that there are really two discussions here on the issue of balance.
1. Player performance and overall player rankings.
In this case, I personally care little about map imbalance as the good players will rise to the top regardless of whether imbalances are 'corrected for' or not. They will do decently from any position and will tend to win more often despite the odds against them (whether it being ganged up on because a country perceived as powerful and/or the player is known to be a strong or when playing 'disadvantaged' from a 'poor' starting position).

2. Play testing data and variant design/modification.
Here the balance issue is actually a real issue, and one that some (many?) of us care about. Ideally if a designer intends for balance to be more or less even on their variant, he/she would probably like some data to know how close they got to their intended desire. More data here doesn't hurt. I know I would want this sort of data when I jump into the design community once I finish my thesis.


Hope all of that clarifies where I'm coming from.

Just to confuse the issue even further, (and I'm by no means advocating we pursue these anytime soon), it would also be interesting to check a player's stats by variant, or to know who the top ranked player are by variant (thus we'd need a separate rating tracked for each variant). Players could then figure out which variants they are best at and which they need to work on, and players could also be able to boast at being the king of a particular variant.

I only note all this because the numbers would be interesting to me (and likely to others), and I want to capture these ideas in the post before I forget them.

I am by no means suggesting we pursue any of this now before the vRating is in place.

Also, just wanted to clarify...

Oli,

I believe you mentioned that you intend to calculate the rating based on all historical games so people will get a current rating based on all their past performance, not a starting from now and moving forward rating, right? I'm assuming that means your code would be filtering through the games in the order of completion and adjusting the score 'through time' game by game to catch up to the current time?

I ask because, as I understand the Elo rating system, the score you get at completion of a game is essentially time-dependent relative to other games. So, for instance if a new player beat a seasoned player early on, then after a year or so of many more games, played that player again and beat them exactly the same a second time, the score gained from the first win would be significantly more than the second win when they were at comparable ratings.

If this is indeed how you plan to approach the ratings, then it should be fairly simple to also look at ratings over any given period of time (last month, last year, all of 2010, etc.) simply by adding a date range function, no? (And, webdip could implement it over their massive data set, and such code could be used to test multiple rating systems over a particular data set to compare them).

I'm not sure how you would go about this, it seems a pretty complex piece of work to step through all the games in the order they were played to calculate the current scores based on all past play, so if you know already how to do this (or have an idea), I'd be very interested in the algorithm (or code).

I can create the rating by checking the outcome of each game one by one starting with the first game. So the rating should reflect this quite well (just misses the CDs).

Also the results should be fairly equal, as the rating applies to all players the same way.

I absolutely agree with LS in regards to additionally making stats also available on a per-variant basis. Good idea!

So will this be including 1v1s? Because it could be argued that they should be in a separate category... 1v1s contain no diplomacy, so you could get a very high V-Rating with no negotiating skills (even gunboat has a decent amount of negotiation with supports - 1v1s have none at all). Plus, they can be finished much faster, so 1v1 players would be at an advantage.

Sorry if this has been covered - I skim-read this thread because there are so many enormous posts...

@Captainmeme:
I don't think that this is an issue, as there is a hard limit for the rating.
After enough games the rating will represent your skill.
If you perform in a given game like your skill it won't get any better. Example: If you win 10:10 games, your rating won't go up if you win again, as this is the expected result. From there it only can go down if you loose a game. (That's why I want to keep the DPoints, as it gets boring if you play well and gain no more points)

I agree with Leif about @checking a player's stats by variant,or to know who the top ranked player are by variant (thus we'd need a separate rating tracked for each variant) something I argued from time to time.

Oli, when you are ready -test with the rating maybe you should post 2-3 examples perhaps one of a committed player (ex. Leif, Ruffhaus, fasces, me, you etc) one of a high CD, on from the top 20 rank and one from the top 80 rank? so we can argue a bit more :)

Yes, for sure.
The system will allow for different algorithms and a ranking for "global" and each variant.
So it should be easy to make a stats-page for all users with the different stats and compare the results of the different algorithms.

Oli,

Are you using the elo/ghostrating system I described?

I ask because the hard limit is only theoretical. To reach the theoretical limit, a person would have to win every game against every player an infinite number of times. A better description would be that the limit gets harder and harder.

Asymptotic approach toward the limit.. you never reach it and your rating can theoretically keep improving but it can never really reach, and certainly cannot exceed the 'limit'

Also, thanks for tackling this Oli, really looking forward to it!

Right, Asymptotic approach = harder & harder. :-)

I'm really looking forward to it also.
Thanks for proposing this improvement, Oli.

Oli:
But surely 1v1s would still have an impact on the V-rating? For example, I have about 50 finished games. Due to my participation in the VDip Cup, I'd hazard a guess that 20 or so of those are 1v1s. I'm a lot better at actual Diplomacy than 1v1s, as negotiation is my main strong point, so technically if the ratings were split I would have a high Diplomacy V-Rating and a fairly low 1v1 V-Rating; however, combining them into one rating gives a false impression for both, showing me as better in 1v1s than I am but worse in diplomacy than I am.

What I'm trying to say is that 1v1s are a different game type to all other variants; enough to give them a separate rating.

I don't want to make this hard for you, but I do think this is important.