Login Register WSPC 2017 • Schedule • Participation • Location • WSC • WPC • Contact •Search •Recent





Sticky LMI Players' Rating System
   LMI Tests -> Monthly Sudoku and Puzzle Tests238 posts • Page 4 of 10 • 1 2 3 4 5 6 7 8 9 10
Administrator
Subject: New LMI Players Rating System @ 2011-05-11 7:42 PM (#4373 - in reply to #4268) (#4373) Top


200010010020
motris - 2011-04-26 9:45 PM

I'm sure I'm not the only one who will be interested to learn the methodology behind the system as it is a real "world leader board" these days.

We had been working over the last couple of months to come up with a new revamped LMI Players Rating System. And, we are happy to share the details (including the rating calculation mechanism) of the new system with everyone.

The details of the rating system have been captured in a pdf. You can either download it or view it. And, feel free to discuss the ratings in this thread.

As for the new rating list, it will be published after MAYnipulation, for both Sudoku and Puzzles.
neerajmehrotra
Subject: RE: New LMI Players Rating System @ 2011-05-11 11:37 PM (#4375 - in reply to #4373) (#4375) Top




Posts: 325
10010010020
Location: India
Administrator - 2011-05-11 7:42 PM

motris - 2011-04-26 9:45 PM

I'm sure I'm not the only one who will be interested to learn the methodology behind the system as it is a real "world leader board" these days.

We had been working over the last couple of months to come up with a new revamped LMI Players Rating System. And, we are happy to share the details (including the rating calculation mechanism) of the new system with everyone.

The details of the rating system have been captured in a pdf. You can either download it or view it. And, feel free to discuss the ratings in this thread.

As for the new rating list, it will be published after MAYnipulation, for both Sudoku and Puzzles.


The V2.0 of rating system looks interesting but needs thorough discussion. I request all the active players of LMI to please comment to make this system more robust.
Kudos to Rakesh Rai for designing the algorithm. I think it takes care of all the variables required for a proper rating system.

Edited by neerajmehrotra 2011-05-11 11:39 PM
debmohanty
Subject: RE: LMI Players' Rating System @ 2011-05-12 7:34 AM (#4377 - in reply to #1357) (#4377) Top



Test Co-ordinator

1000500100100100202020
Location: India
As Neeraj mentioned, this system looks like covering all variables, although at the cost of being little complex.
It would help if we can show 3 different cases in action ( players getting advantage, players being penalized ) with some numbers.
rakesh_rai
Subject: RE: LMI Players' Rating System @ 2011-05-13 6:45 PM (#4390 - in reply to #4377) (#4390) Top



Support Administrator, Converse Variations (SM 15/16) Author

Posts: 665
500100202020
Location: India
debmohanty - 2011-05-12 7:34 AM

As Neeraj mentioned, this system looks like covering all variables, although at the cost of being little complex.
It would help if we can show 3 different cases in action ( players getting advantage, players being penalized ) with some numbers.
The image shows the working of ratings using a few fictitious players:

- Players A, B and E have played very few tests and are, therefore, penalized. The level of penalty depends on the overall weight of tests played. As they play more regularly, the level of penalty will reduce and ultimately go away.
- Player D has played average number of tests. The rating takes weighted average of NS from all participated tests. This player does not get any benefit nor is he/she penalized.
- Player C is a regular player. This player gets the benefit of only his best performances being considered for ratings.

debmohanty
Subject: RE: LMI Players' Rating System @ 2011-05-13 7:26 PM (#4391 - in reply to #1357) (#4391) Top



Test Co-ordinator

1000500100100100202020
Location: India
Am I correct in saying

For a player who starts playing at LMI
1) he needs to play 4 tests before he gets the ranking he deserves? After 3 tests his penalty will be very less though
2) to get benefit of being a regular player, he has to play more than 7 tests consecutively. (after 7 tests K=6.4), and if he misses some tests, it takes longer

Edited by Rohan Rao 2011-05-13 8:14 PM
debmohanty
Subject: RE: LMI Players' Rating System @ 2011-05-13 7:31 PM (#4392 - in reply to #1357) (#4392) Top



Test Co-ordinator

1000500100100100202020
Location: India
Also we discussed few posts back in this thread that we'll add an option for players who would want to consider the results to be rated.
It will be implemented starting MAYnipulation

This is how it will look like


Do we still remove 0-scores after that?
purifire
Subject: RE: LMI Players' Rating System @ 2011-05-13 7:33 PM (#4393 - in reply to #4392) (#4393) Top



Odd Even & Twisted Classics (SM 16/17) Author

Posts: 441
1001001001002020
Location: India
debmohanty - 2011-05-13 7:31 PM

Also we discussed few posts back in this thread that we'll add an option for players who would want to consider the results to be rated.
It will be implemented starting MAYnipulation

This is how it will look like


Do we still remove 0-scores after that?


If someone checks the box allowing the score to be considered then I would say even zero scores should be considered.

Rishi
MellowMelon
Subject: Re: LMI Players' Rating System @ 2011-05-13 7:52 PM (#4394 - in reply to #4393) (#4394) Top


Fillomino-Fillia 2 Author

100
Location: USA
There's a slight typo in the image by Rakesh Rai: Player E's base rating should read 1000 instead of 775 (although the final calculation is correct). Wonder who he was based off of? :P

I think considering the 0-scores is okay if you clearly say that near the "Start" button and the check box.


I don't know if I like the current weighting system. For one thing, the simulation. Relative to a 738 rating Player C has had a lot of dismal recent performances in that simulation. I suppose the ratings would eventually reflect that if it continued, but perhaps the "penalty" for those performances should kick in sooner. The fact that they never take effect if he picks his game back up is a feature I'm undecided about.

Another related issue is the following case of my own design: two regular players F and G.
-- F gets four scores around 700, then tanks for a bit and gets four scores around 500, then improves and gets four scores around 900.
-- G is consistently improving. He gets four scores around 500, then four scores around 700, then four scores around 900.
If I understand how the weighted average is calculated correctly (all of this may be moot if not), player F gets the higher rating here, because his 500s that are thrown out have a higher weight so the 900s get emphasized more in the calculation. In my opinion G's performance warrants the better rating.

Both of these issues would be fixed if the weighted average divided by the total weights of the most recent U tests, instead of the weights of the highest scoring tests. But this has its own issue in that it is a very harsh penalty on a recent bad performance. You would not want a test of weight 1 thrown out in this method. Not sure what a fix would be.
rakesh_rai
Subject: RE: LMI Players' Rating System @ 2011-05-13 8:15 PM (#4395 - in reply to #4391) (#4395) Top



Support Administrator, Converse Variations (SM 15/16) Author

Posts: 665
500100202020
Location: India
debmohanty - 2011-05-13 7:26 PM

Ami I correct in saying

For a player who starts playing at LMI
1) he needs to play 4 tests before he gets the ranking he deserves? After 3 tests his penalty will be very less though
Yes. For normal players, who do not author/test any tests, this would be true. For authors/testers, there can be cases where playing 3 tests may be enough. And, they are justified to get the benefits of a reduced N.
2) to get benefit of being a regular player, he has to play more than 7 tests consecutively. (after 7 tests K=6.4), and if he misses some tests, it takes longer
Yes. In order for the player's poor performances to be ignored from rating calculations, about 7 tests out of 12 would be needed. The author/tester logic applies here too.
rakesh_rai
Subject: RE: LMI Players' Rating System @ 2011-05-13 8:19 PM (#4396 - in reply to #4393) (#4396) Top



Support Administrator, Converse Variations (SM 15/16) Author

Posts: 665
500100202020
Location: India
purifire - 2011-05-13 7:33 PM

debmohanty - 2011-05-13 7:31 PM

Also we discussed few posts back in this thread that we'll add an option for players who would want to consider the results to be rated.
It will be implemented starting MAYnipulation

Do we still remove 0-scores after that?

If someone checks the box allowing the score to be considered then I would say even zero scores should be considered.
I agree that 0 scores should be considered (manipulated) for ratings from now on. But, since this is the first test after the change being implemented, we'd take a call after the test depending on how the players have adapted to the change.

Edited by rakesh_rai 2011-05-13 8:19 PM
rakesh_rai
Subject: Re: LMI Players' Rating System @ 2011-05-13 9:39 PM (#4397 - in reply to #4394) (#4397) Top



Support Administrator, Converse Variations (SM 15/16) Author

Posts: 665
500100202020
Location: India
MellowMelon - 2011-05-13 7:52 PM

There's a slight typo in the image by Rakesh Rai: Player E's base rating should read 1000 instead of 775 (although the final calculation is correct).
Yes. Its corrected now.

Wonder who he was based off of? :P
These are all fictional players. Any resemblances may at best be coincidental.

For one thing, the simulation. Relative to a 738 rating Player C has had a lot of dismal recent performances in that simulation. I suppose the ratings would eventually reflect that if it continued, but perhaps the "penalty" for those performances should kick in sooner. The fact that they never take effect if he picks his game back up is a feature I'm undecided about.
We deliberated if we should keep 12 tests or 8 tests. Ultimately we decided for a longer duration. So the ratings will be based on all performances during this period. And, as mentioned earlier too, regular players are entitled to some benefits - they can afford to have a few bad days, for example. And, I would view player C example as the system allowing regular players to recover too.

Also, these ratings should reflect the whole 12-month period without being too volatile. One bad or good performance should not shake up the the ratings.

Another related issue is the following case of my own design: two regular players F and G.
-- F gets four scores around 700, then tanks for a bit and gets four scores around 500, then improves and gets four scores around 900.
-- G is consistently improving. He gets four scores around 500, then four scores around 700, then four scores around 900.

If I understand how the weighted average is calculated correctly (all of this may be moot if not), player F gets the higher rating here, because his 500s that are thrown out have a higher weight so the 900s get emphasized more in the calculation. In my opinion G's performance warrants the better rating.
Both F and G would get a rating of 822 in this case. But next month, F's 700 and G's 500 go out of the calculations. So G will have a better rating. And so on.

Both of these issues would be fixed if the weighted average divided by the total weights of the most recent U tests, instead of the weights of the highest scoring tests. But this has its own issue in that it is a very harsh penalty on a recent bad performance. You would not want a test of weight 1 thrown out in this method. Not sure what a fix would be.
I get your point and I agree that the ratings are a little slow to reflect the recent performances, but the weights are still an improvement over what we had so far.

Edited by rakesh_rai 2011-05-13 9:40 PM
debmohanty
Subject: RE: LMI Players' Rating System @ 2011-05-16 6:13 AM (#4424 - in reply to #4392) (#4424) Top



Test Co-ordinator

1000500100100100202020
Location: India
debmohanty - 2011-05-13 7:31 PM

Also we discussed few posts back in this thread that we'll add an option for players who would want to consider the results to be rated.
It will be implemented starting MAYnipulation

This is how it will look like


Do we still remove 0-scores after that?

This didn't seem to work. Either the check box was too small or the purpose of it was not clear.
Most of the players having zero score still have the check box selected.
rakesh_rai
Subject: RE: LMI Players' Rating System @ 2011-05-16 11:06 AM (#4427 - in reply to #4424) (#4427) Top



Support Administrator, Converse Variations (SM 15/16) Author

Posts: 665
500100202020
Location: India
debmohanty - 2011-05-16 6:13 AM

This didn't seem to work. Either the check box was too small or the purpose of it was not clear.
Most of the players having zero score still have the check box selected.
So we'll exclude all zero scores from ratings...
debmohanty
Subject: RE: LMI Players' Rating System @ 2011-05-17 4:48 PM (#4437 - in reply to #4427) (#4437) Top



Test Co-ordinator

1000500100100100202020
Location: India
rakesh_rai - 2011-05-16 11:06 AM

debmohanty - 2011-05-16 6:13 AM

This didn't seem to work. Either the check box was too small or the purpose of it was not clear.
Most of the players having zero score still have the check box selected.
So we'll exclude all zero scores from ratings...

To put in perspective, here are the numbers -
Out of 116 players who started the test, exactly 7 marked that their results shouldn't be considered for ratings. (1 of them got non-zero score)
Of the remaining 109 players, 32 got zero scores.
rakesh_rai
Subject: Re: LMI Players' Rating System @ 2011-05-17 6:34 PM (#4438 - in reply to #1357) (#4438) Top



Support Administrator, Converse Variations (SM 15/16) Author

Posts: 665
500100202020
Location: India
Updated LMI Puzzle Ratings after MAYnipulation (May 2011 LMI puzzle test), and LMI Sudoku Ratings after the April 2011 LMI sudoku test are now available.

The ratings are based on the new logic shared earlier. Four players find a place in the Top 10 in both lists - motris, deu, nikola and misko.

Overall 487 players (from 45 countries) are included in the sudoku ratings and 425 (from 44 countries) in the puzzle ratings.
kiwijam
Subject: Re: LMI Players' Rating System @ 2011-05-18 4:20 AM (#4441 - in reply to #1357) (#4441) Top




Posts: 120
10020
Location: New Zealand
Rakesh, good work on the ratings list.
I can see I have a rating number now, and like most of us I'd like to improve it! But I don't know the numbers that were used to calculate it (like my last 12 scores), or the numbers for other puzzlers with similar ratings (these are the people I want to beat in the monthly test to move higher).
Is it possible to have a link to a full table also (like the 'pink' table of examples you published above)?
rakesh_rai
Subject: Re: LMI Players' Rating System @ 2011-05-18 12:00 PM (#4442 - in reply to #4441) (#4442) Top



Support Administrator, Converse Variations (SM 15/16) Author

Posts: 665
500100202020
Location: India
kiwijam - 2011-05-18 4:20 AM

I can see I have a rating number now, and like most of us I'd like to improve it! But I don't know the numbers that were used to calculate it (like my last 12 scores), or the numbers for other puzzlers with similar ratings (these are the people I want to beat in the monthly test to move higher).
If you click on your name, you go to your profile page that contains scores in all tests that you have taken part. These are the numbers based on which your ratings are calculated. Note that your scores in TVC are not included for ratings. You can see the history of scores for any player this way.

Is it possible to have a link to a full table also (like the 'pink' table of examples you published above)?
We have shared the computation logic already. But we feel that a full table of numbers used during ratings calculation can be cumbersome and complex, so it may not be needed online.

But, since you have asked, your normalized scores in the three tests are 601, 670 and 723. And you have played only 3 tests so far. Although the weighted average rating (based on these tests) is 665, you are penalized by a small factor to arrive at your final rating which is 651 currently. Here is the 'pink' table for you:



(kiwijam.png)



Attachments
----------------
Attachments kiwijam.png (10KB - 0 downloads)
debmohanty
Subject: Re: LMI Players' Rating System @ 2011-05-18 12:06 PM (#4443 - in reply to #4442) (#4443) Top



Test Co-ordinator

1000500100100100202020
Location: India
May be we should enhance the Profile page to display the pink table and related computations. It needs some amount of work. So no promises right now.
euklid
Subject: Re: LMI Players' Rating System @ 2011-05-21 9:11 PM (#4467 - in reply to #4443) (#4467) Top




Posts: 27
20
Location: Austria
I would love to see "my pink table" in my profile page! And I think I speak for many puzzlers here. :-)

Have fun, Stefan
Administrator
Subject: Re: LMI Players' Rating System @ 2011-05-24 2:11 PM (#4496 - in reply to #4467) (#4496) Top


200010010020
euklid - 2011-05-21 9:11 PM

I would love to see "my pink table" in my profile page! And I think I speak for many puzzlers here. :-)

Have fun, Stefan
The profile page has the pink table now.
euklid
Subject: Re: LMI Players' Rating System @ 2011-05-25 1:06 AM (#4507 - in reply to #4496) (#4507) Top




Posts: 27
20
Location: Austria
My Puzzle Rating is not correct. The weighted average of my NS should be 707, my rating is 724, though.

Are you sure that you have implemented the weighted average of the NS correctly? I arrive at approx. 724 when I compute a non-weighted(!) average of my results, including the EvergreensI-result which should have weight zero by now.

In the pink table I would love to see the EvergreensI data (NS,...) also. It has weight 0 of course but the data is available and interesting nevertheless.

Thanks,
Stefan
rakesh_rai
Subject: Re: LMI Players' Rating System @ 2011-05-25 10:36 AM (#4508 - in reply to #4507) (#4508) Top



Support Administrator, Converse Variations (SM 15/16) Author

Posts: 665
500100202020
Location: India
euklid - 2011-05-25 1:06 AM

My Puzzle Rating is not correct. The weighted average of my NS should be 707, my rating is 724, though.

Are you sure that you have implemented the weighted average of the NS correctly? I arrive at approx. 724 when I compute a non-weighted(!) average of my results, including the EvergreensI-result which should have weight zero by now.
Thanks for pointing out this discrepancy. There are no problems with the implementation; however, we did find that your Twist score was completely missed in the ratings calculation. The ratings will be corrected today.

In the pink table I would love to see the EvergreensI data (NS,...) also. It has weight 0 of course but the data is available and interesting nevertheless.
OK. We'll include the ratings data (PS/RS/NS) for zero weight tests as well.

Edited by rakesh_rai 2011-05-25 10:37 AM
rakesh_rai
Subject: Re: LMI Players' Rating System @ 2011-05-25 12:12 PM (#4509 - in reply to #4508) (#4509) Top



Support Administrator, Converse Variations (SM 15/16) Author

Posts: 665
500100202020
Location: India
rakesh_rai - 2011-05-25 10:36 AM

In the pink table I would love to see the EvergreensI data (NS,...) also. It has weight 0 of course but the data is available and interesting nevertheless.
OK. We'll include the ratings data (PS/RS/NS) for zero weight tests as well.
This is done. Now you can see your normalized scores for even those tests which are not included in current ratings (like EG1).
rakesh_rai
Subject: Re: LMI Players' Rating System @ 2011-05-25 8:10 PM (#4514 - in reply to #4508) (#4514) Top



Support Administrator, Converse Variations (SM 15/16) Author

Posts: 665
500100202020
Location: India
rakesh_rai - 2011-05-25 10:36 AM

euklid - 2011-05-25 1:06 AM

My Puzzle Rating is not correct. The weighted average of my NS should be 707, my rating is 724, though.

Are you sure that you have implemented the weighted average of the NS correctly? I arrive at approx. 724 when I compute a non-weighted(!) average of my results, including the EvergreensI-result which should have weight zero by now.
Thanks for pointing out this discrepancy. There are no problems with the implementation; however, we did find that your Twist score was completely missed in the ratings calculation. The ratings will be corrected today.
The Puzzle Ratings have now been updated. Also, the sudoku ratings have been updated to rectify one error in Sampler Platter scores.
euklid
Subject: Re: LMI Players' Rating System @ 2011-05-26 1:52 AM (#4516 - in reply to #4514) (#4516) Top




Posts: 27
20
Location: Austria
One more request of minor importance. Is it possible to mark those players' names in the result tables who have opted not to be included in the LMI rating?

MaYnipulation has 83 participants according to result table but 82 participants according to the LMI rating. If the one player were marked somehow (e.g. asterix at his name), everybody could calculate his LMI rating for himself and "verify" the correctness of the official LMI rating. :-)
238 posts • Page 4 of 10 • 1 2 3 4 5 6 7 8 9 10
Jump to forum :
Search this forum
Printer friendly version