5 replies [Last post]
beta_helix's picture
User offline. Last seen 13 hours 9 min ago. Offline
Joined: 05/09/2008
Groups: None

As many of you noticed, Puzzle 445: CASP9 Refinement Puzzle 8 - Double Start with constraints
had to be closed and reposted due to errors in the starting models.

These errors weren't immediately apparent, but were noticed and reported by many players: http://fold.it/portal/node/990139

By the time we finally noticed the problem, fixed it and closed this puzzle, one day had gone by.

Because the puzzle was supposed to be up for ~7 days and it was worth 100 points, the closed puzzle was awarded a maximum of 15 points (since 100/7 =~ 15) for the 93 players that played it.

This was discussed at our weekly Foldit meeting today and this policy will be used for any future cases where we have to close a puzzle early.

Ideally, we will catch any problems early (we usually try to monitor chat & the puzzle comments and feedback right after a new puzzle is posted) but sometimes it takes a while to find out that a new puzzle is useless.

We welcome any feedback on this and will gladly take it into account for future puzzles, but we will not change the points given on any previous puzzles.

mimi's picture
User offline. Last seen 2 years 43 weeks ago. Offline
Joined: 11/17/2008
Groups: Contenders
Feedback as requested

I think that I am probably in a minority, but when I read the first comments about there being problems with this puzzle I expected it to be closed and restarted fairly quickly.
For that reason I didn't even bother working on it.
I didn't expect that any points would be awarded for it - especially since the hope was that it would be possible to carry over solutions from the original puzzle to its replacement.
My recollection was that when previous puzzles had been restarted no points were awarded.

I think that your decision regarding awarding points is perfectly reasonable - even if I did miss out on the opportunity to get any :)

It is good to have a policy on this for future reference but I think it would also be good if you could define how long a puzzle can be up before you award points if you have to close it - just to avoid future arguements

beta_helix's picture
User offline. Last seen 13 hours 9 min ago. Offline
Joined: 05/09/2008
Groups: None
Thank you for your feedback, mimi.

Yes, the goal of this is to lay out the policy so that it will be clear for future puzzles.

The main thing is that if a problem is caught early, we can close that puzzle quickly, repost it and let you use the previous solutions. This has been the case for most reposted puzzles, since all the time spent was not wasted because you can still use those solutions.

The first puzzle that was closed and received points was 423: H2N2 Flu Design Puzzle 1, http://fold.it/portal/node/989767.

The reason for this was that the original puzzle had such a bad error in it that most old solutions were invalid (since the target protein was mutatable), hence a small amount of points were awarded for those who worked on that puzzle. Luckily that problem was caught pretty quickly.

This last case was tricky again because of the bug that makes it so that players can't load in previous solutions (http://fold.it/portal/node/990089) and the fact that it took so long to diagnose, fix and repost.

To sum up (and answer your question, Mimi):

Any puzzle closed within the first few hours will not receive any points (provided players can still use old solutions).
Any puzzle that has to be reposted where old solutions are useless (or cannot be loaded) will receive a small amount of points: minimum of 1 point and points go up the longer a puzzle remained up: ~(total points/number of days up)

Again, hopefully we will catch these problems sooner rather than later and this loading previous solutions bug will be fixed in the next update to 'main'.

B_2's picture
User offline. Last seen 6 years 32 weeks ago. Offline
Joined: 11/29/2008
Groups: None
I was in the same place as

I was in the same place as mimi

Saw comments on IRC about a new puzzle popping up (some time after 12 PM PDT on Sunday) and shortly afterwards comments about the puzzle being broken, and then a feedback was posted around early evening with details of all the problems.

I really don't think it's fair to reward those who just tried to rack up points on a broken puzzle while others were ignoring it expecting it to be pulled.

Joined: 06/17/2010
I was offline this time

but imo points should be given for all those who want/need work on that puzzle anyway.
It is so small amount of points so I think no one get hurts and those who work on it are happy.

beta_helix's picture
User offline. Last seen 13 hours 9 min ago. Offline
Joined: 05/09/2008
Groups: None
Thank you for your feedback, B_2 and Rav3n_pl

I see how frustrating it would be if you heard about problems early on and therefore didn't play it.

If you ever see (or hear talk about) a major problem with a puzzle that will surely require a reposting, but haven't noticed a feedback or puzzle comment about it, please feel free to PM me directly so we can make sure to fix the problem as quickly as possible.


Developed by: UW Center for Game Science, UW Institute for Protein Design, Northeastern University, Vanderbilt University Meiler Lab, UC Davis
Supported by: DARPA, NSF, NIH, HHMI, Amazon, Microsoft, Adobe, Boehringer Ingelheim, RosettaCommons