|
|
Posts: 199
 
Location: United States | Instant grading actually came about because of the competing desires to fairly rank fast solvers with running tests that many can finish. I think a good LMI test should have at least five finishers, ideally more, meaning it is quite likely that someone is done with over thirty minutes left. Before "Claim Bonus" there was a long wait to just check and check and check work since last submit would equal the final time. Actually handing in is a huge improvement, but now makes the meta choice of how long to check far too heavy in weight when you want a test to be long enough for many other solvers. Instant grading solves this imbalance, just as quick grading at WPC playoffs do, to emphasize solving speed over everything else.
Where we disagree is the view a typo is different from say not marking the last mine in a minesweeper puzzle. In both cases I think the solver is effectively done and slightly inaccurate so they are treated the same. Because you can't run a perfect test to tell them apart, and self reporting will be unfair if not all solvers know to use the system or only half of typos can be identified as such, it is certainly fairest if not most comforting to treat wrong as wrong. The solver gets an X and then proves they know the right answer by getting to resubmit. This is much better than an organizer guessing each and every time.
Edited by motris 2012-07-26 6:07 AM
|
| |
|
2021 World Sudoku+Puzzle Convention Organizer
Posts: 1780
 
Location: India | motris - 2012-07-26 6:05 AM
Where we disagree is the view a typo is different from say not marking the last mine in a minesweeper puzzle. In both cases I think the solver is effectively done and slightly inaccurate so they are treated the same. Because you can't run a perfect test to tell them apart, and self reporting will be unfair if not all solvers know to use the system or only half of typos can be identified as such, it is certainly fairest if not most comforting to treat wrong as wrong. The solver gets an X and then proves they know the right answer by getting to resubmit. This is much better than an organizer guessing each and every time.
I agree that a typo is similar to say one mine not put in. I don't think thats what Para means by a solving error. The valid example of where a solving error is helped by instant grading is say when you're done with 60% of a puzzle and you've narrowed it down to two possible paths towards the solution. Now maybe narrowing it down to one requires some difficult logic which you can avoid by quickly trying one path, clicking submit, and if its wrong, trying the other. In this case, just a -5 isn't really enough. Of course going back and solving it correctly will ideally take more time than say the last mine or a typo in the answer key. The suggestion of either second penalty being higher, or of the penalty increasing per second after the first wrong submission, reduces the possibility of someone using this instant grading as a means of solve-checking rather than answer-checking.
Take Palmer's 2 errors for example - He himself has said his solves were mostly instinctive and the instant grading helped as a checker. If the penalty conditions weren't so friendly I feel he may have taken more time to check and been more cautious, and so may not have had as much of a runaway performance. |
| |
|
Posts: 199
Location: United States | I've been using the definition of solving error as "what is on the paper would not get points at a WPC." So one mine off is different from a typo which is only an error in sending along you perfect paper solution. A serious puzzle error is certainly different, and even in your hypothetical case I would hope a 15 point (1.5 minute) penalty would be high enough. I think only on my last puzzle, where I did not check that all pentominoes were in and had a tweak to make did I feel I was using the checking in a way a five point penalty was worth it as I would be finished. |
| |
|
2021 World Sudoku+Puzzle Convention Organizer
Posts: 1780
Location: India | motris - 2012-07-26 9:37 PM
I've been using the definition of solving error as "what is on the paper would not get points at a WPC." So one mine off is different from a typo which is only an error in sending along you perfect paper solution. A serious puzzle error is certainly different, and even in your hypothetical case I would hope a 15 point (1.5 minute) penalty would be high enough. I think only on my last puzzle, where I did not check that all pentominoes were in and had a tweak to make did I feel I was using the checking in a way a five point penalty was worth it as I would be finished.
If I remember correctly, there was a rule in the last WPC that if the organizers/judges felt that a puzzle was 90% correct, a percentage of the points would be awarded. So while a one-mine-off error may be slightly different from answer key errors, it is however very different from a serious puzzle error. The thing is, while you can't differentiate between answer-key/one-mine-off in terms of time required for correcting it, you can do that for a major solving error. We are trying to bring the online tests as much to the level of an offline event as possible, so at least where the differentiation is possible, I think it should be implemented. Thats why the penalty increasing per second after the first incorrect submission till the correct submission(and (no penalty/just the base penalty) if it is never correctly submitted) makes sense to me. To your end of keeping a 15 point penalty, we can keep the base penalty low, and make sure it increases at a fair rate. |
| |
|
Posts: 199
Location: United States | The 90% rule was simply to granting time bonus to the rest of the round, even with the mistake. It has never allowed, to my knowledge, puzzle points directly. (We created the rule for WSC 5 and Hungarian organizers used again last fall.) |
| |
|
2021 World Sudoku+Puzzle Convention Organizer
Posts: 1780
Location: India | Directly or indirectly, the point I'm trying to make is that concessions were made when the puzzle was almost correct. Concessions weren't made when the puzzle was mostly wrong. So simply, returning to the topic of penalty, a system that provides less penalty for a 90% correct puzzle or an answer entry, and more penalty for a major solving error should be encouraged as opposed to a flat penalty for all kinds of errors. It may not be perfect, but all I'm saying is its closer to a perfect evaluation than the current system, whether the current system be -5 for all errors, or -15 for all errors. |
| |
|
Posts: 315
Location: The Netherlands | I don't think the WPC playoff is an accurate measure to compare this competition to. The WPC playoff is a single round that decides the winner of the competition. It was introduced to make the competition more exciting and add to the spectator element. Palmer himself said he didn't feel like he was the best overall puzzler of the last WPC, as Ulrich outscord in him rounds where he managed to do the best he could. So for judging the overall performance of solvers, a playoff system is not accurate.
This competition seems more like the regular rounds of a WPC. where you have a bunch of variating rounds and the overall score of those rounds determines a ranking. Especially the long Varia rounds at the WPC. You get a lot of puzzles and you try to solve as many as possible and get points accordingly. Making mistakes in those rounds can cost you points, but in the long run a single mistake won't cost you that much as you can make it up.
Sometimes this can happen in a championship as it did to me in Brazil. I missed a single bridge in a Hashiwokakero puzzle in the first round. It was a carelessness by me, caused by me wanting to be extra fast as people were finishing. I lost a lot of bonus in that round. The other rounds weren't very well made to make up that loss. I therefor lost any chance of getting a good finish.
But this situation wont occur in the LMI ranking as all scores are normalised. So all 12 tests that count towards you rating are comparably scored, and no one test will be worth far more than others. So over 12 tests, making a mistake in a single test won't hurt the overall assessment on the LMI ranking. It will only hurt the LMI ranking if you constantly keep making mistakes in your tests. But then that should be something you have to work on and improve. It shouldn't be something you want to fix through the scoring system.
Not making mistakes is a part of puzzling. You don't want to take that out by changing the scoring system. It's comparable to the starting gun in athletics, swimming or speedskating. If you don't react well to the starting gun, you can lose the race and the gold. You might have gone the fastest of everyone, but because you reacted 2 tenths of a second slower than the rest, you miss out on the gold by 1 tenth of a second. So is not making mistakes part of puzzling and making mistakes should cost you points.
Just to make clear why I think the bonus shouldn't get too big in instant grading is this. The penalty was set at approximately 30 seconds of solving time. Except that is only accurate if you solve for 1000 points in 100 minutes. Palmer solved for 1600+ points in 100 minute. So those 5 points are worth less than 20 seconds of solving time. So if he had spent 25 seconds checking, noticed his mistake and fixed it, he would have scored lower than now submitting and then fixing his mistake. But he did beat the 30 seconds by which you defined a penalty as. So this means the better you do, the less time an error is worth. So 30 seconds of solving time in penalty should really have been .5% of the total score. Or to make it more general ([30/"total time of the test in seconds"]*100)%.
I think that a test shouldn't be beaten by more than 25% of the total time. 30 minutes bonus on a 120 minute test should be maximum. 40 minutes on a 100 minute test is just too much. Optimally the top time on a 120 minute test would be 100 minutes for the best solvers in my opinion. |
| |
|
Posts: 199
Location: United States | Except his penalty was no longer five points per mistake. It was five points times time correction or about 8.5 points. That is still the right proportion of score, .5% of score per mistake as you say. I'm pretty sure penalty was in the prefactor. If not it should have been.
|
| |
|
Posts: 199
Location: United States | Maybe the right system, to be fair as Gareth wants for the range of solvers, fast and slow, is a set percentage of final score? If every penalty was 1% or 1.5% of final score (therefore the exact value of a minute or so for that solver) would one bother to check more? Probably. But unlike a flat fifteen points this doesn't excessively subtract from a solver who completes only five or ten of twenty puzzles. |
| |
|
Posts: 3
Location: India | I went to the Monthly Sudoku Contest but the puzzle is not showing inspite of clicking on the Reset button and pressing the 'Control' button while left clicking on the mouse. What should I do? |
| |
|
2021 World Sudoku+Puzzle Convention Organizer
Posts: 1780
Location: India | sudokufan - 2012-07-29 8:52 PM
I went to the Monthly Sudoku Contest but the puzzle is not showing inspite of clicking on the Reset button and pressing the 'Control' button while left clicking on the mouse. What should I do?
I assume you mean the Beginners' contest, since the puzzle tests don't have an interface. The puzzles will show only when you've clicked "Start Classics"/"Start Variants" button, and then you will have 40 minutes to solve and submit the answers. As of now there is no record of you starting the test.
|
| |
|
Posts: 16
Location: United Kingdom | Should the password be visible, or the test be non-passworded now? I wanted to have another look at it. |
| |
|
Location: India | BohemianCoast - 2012-08-19 8:27 PM
Should the password be visible, or the test be non-passworded now? I wanted to have another look at it. Typically the passwords are removed from the pdfs after the test, but it wasn't in this case.
Password for the pdf : Daf3gIte_New
Edited by debmohanty 2012-08-19 8:46 PM
|
| |
WSPC 2017 • Schedule • Participation • Location • WSC • WPC • Contact •Search •Recent
Logirace — LMI July Puzzle Test — 21st - 23rd July
