10 replies [Last post]
Joined: 12/06/2008
Groups: Contenders

Bruno's post about Gold Players in History at https://fold.it/portal/node/2003115 got me in the mood to revive my old analytical skills and look at some data from this game.

FoldIt has been around since 2007, and has had hundreds of thousands of players come and go. The leaderboards suggest that there are less than three thousand currently active players. One such question I had was "who is still playing, after all these years?"

I mined the data of the top five hundred players (by hand... you'd be amazed how complicated that is) and determined that we have one player active from 2007 (waves to spvincent!),

twenty-seven still here that joined in 2008:

LastAndroid
Diderot
Bearpaw
IHGreenman
lynnai
deathbat_87
jobo0502
Steven Pletsch
dd-2
Angus
theonlyperuvian
Pazithi
Mike Cassidy
CatSari
ivalnic
pruneau_44
Franco Padelletti
aendgraend
nellasdim
aznarog
heather-1
justjustin
SKSbell
BootsMcGraw
Migi
Dolichwier
phi16

ten from 2009:

infjamc
Amphimixus
heyubob
knotartist
CharlieFortsConscience
Enzyme
strong_base
Bletchley Park
xabxs
drjr

178 from the 2010's, and 284 that joined just this year.

A spreadsheet with the join date of the current top 500 players is at:

https://gofile.io/d/kweE6T

Not all five hundred people have played, recently, particularly those who joined in 2020. One conclusion I have made from this study: we have a dismally low player retention rate.

Should we open a discussion in another post about why this is and what we can do about it?

donuts554's picture
User offline. Last seen 7 hours 56 sec ago. Offline
Joined: 06/20/2019
Groups: Go Science
My thoughts for improvements on Foldit about your question

I think we should open a discussion in another post about why this is and what we can do about it. Here are my thoughts about your question, and I will try to make my thoughts as complete and concise as possible.

I think this is because they think that Foldit is not as fun as other games, and that they think that folding up a repetitive sequence of abstract shapes and cartoon proteins and squiggles is uninteresting and boring, and that I have heard Foldit beginners say that the Foldit tutorials are "frustrating" and that people are doing "random" things to solve the tutorials. Also, the Foldit beginners say that the tutorials are very hard to do, as many players ask for help in Global chat and/or the Foldit feedback and Foldit forums to get help to solve the tutorials.

I think that Foldit should be more relatable to other less abstract games, and be more relatable to things in the visual real world, such as humans, food, trees, magic, animals, spacesuits, racecars, candy, strings, puzzle pieces, spikes, pipes, fish, swords, pebbles, money, spoons, cars, ores, and ovens. If the proteins were more relatable, then they would be more interested in the game, as it is similar to other popular games. Also, folding up proteins repetitively can be boring, as you have to follow mostly the same steps in order to get a high score. The people
who joined Foldit in 2020 wanted to help make a real coronavirus vaccine in real life, as they are motivated by the danger of the coronavirus, and they are not that interested in abstract proteins in general. They wanted to know whether Foldit would actually make a difference in the real world, and whether their protein will actually get made into a vaccine.

I think it is like global warming. They know its important, and it has a large impact, but several people don't care about global warming as it is something that is not able to be seen in their everyday lives, and "not seeing global warming happen" in their everyday lives makes them think that global warming is "less often", more "rare" and less important.

I think that the same thing happens for proteins. They know proteins are important, it has a large impact, but several people don't care about proteins as it is something that is not able to be seen in their everyday lives, and "not seeing proteins happen" in their everyday lives makes them think that proteins are "less often", more "rare", and less important.

The limitations of Rosetta and Foldit being in perpetual beta are a significant factor in making people not "see proteins happen". I think that Rosetta is a protein modeling and potential energy calculating tool, not a thermodynamic, protein animatic kinetic energy modeling tool. This limitation by using a scientific analytical instrument to be used in a "game" is what makes this game not interesting, and make a dismally low player retention rate. "Logical scientific analytics" and "everyday fun games" do not match. The two phrases have very different moods and tones to them.

|------|
[UNLESS]... --- \/\\/ ----


I think people need to be able to actually "see proteins happen", doing what they are supposed to do, binding and interacting with other molecules, transporting, storing breaking molecules apart, forming molecular bonds, flowing around, moving, in action, not just standing by itself. This orderly action and movement would make proteins more lively, and what they truly are.

This movement would be more interesting to people, because the people would know that seeing moving squiggles in action would be more important than stationary squiggles in action, as in real life, people think animals are more like living things than plants, and that plants, which grow, are more like living things than rocks. They think that moving people are more useful than people who are not moving. Hunters used to think moving food is more useful than stationary food. People think that a ball moving around in a sports stadium is more interesting than a ball not moving around.

This innate human instinct to be more interested in moving things should be used in Foldit.

However, this kind of dynamic movement can not be implemented in Rosetta, so Foldit has to make use of what is available in Rosetta.

The processes modeled in logical scientific analytics are what make up and constitute the everyday lives and games. I think that the action that occurs, no matter how simple and small, can be combined to make a very fun game. The tools such as wiggle, the moving score, shake, remix, more little things like the moving stripes in hydrogen and cysteine bonds, the flickering of the bands, the varying conversations in the chat, the pulsing of the heavy bondable atoms, are what make the game interesting thus so far.

I think this action may be further developed and made more lively and more interesting by:

  • instead of the normal wiggle and remix, making the protein appear more lively by moving it into a slight random position, which is not shown on the score, and then making it into a its normal position (like the normal wiggle or remix)repeatedly, like the small waves of the water in a swimming pool. The random position would be only for visual purposes, and not be used as an actual position.
  • making the score flash green and show a green up arrow when it goes up, and making the score flash red and show a red down arrow when it goes down, like a game of traffic light.
  • making the sidechains move more when shaking, such as being in random different rotamers and then stopping at the normal stable position (like the normal shake), which is similar to a fidget spinner, a dreidel, or a Wheel of Fortune prize wheel
  • making conversations in the Global chat and Office Hours more lively by talking about interesting things and fun facts about proteins and talking about interesting strategies on how to get a high score from the developers, like training as a team for a sports tournament. (this could be of a lower priority because many popular games do not use chat)
  • making the lines in the hydrogen and cysteine bonds move around more like waves, or with sparkles around the bonds, and adding more special effects, such as sparkles and spinning, to the bands and bondable atoms. This would also be only for visual purposes, like glitter and string and glue in arts and crafts.

There are several other ways to make Foldit more lively, but the limitations of Rosetta might not make them possible.

Also, the tutorials should be really redesigned and be made more easier for new players, by showing more tips about how to finish the puzzle and to reset the puzzle if you worked on it for a long time. This is explained more in the "Better introduction/explanation needed for tutorials" feedback, at https://fold.it/portal/node/993594 .

Finally, Foldit should also take advantage of the randomness that Foldit beginners prefer to do to get a high score. People like to do things with randomness, from the randomness of rolling dice, to the randomly generated world of 3-D sandbox games, to the randomness in casinos. Me and Susume think that tutorial puzzles should have multi-start puzzles with similar real-life proteins from different species, so that there are small variations in between the proteins. These seemingly random small variations can show to Foldit beginners how a small "random" difference in the sequence of amino acids can affect a protein's shape.

Joined: 09/24/2012
Groups: Go Science
Some side questions / suggestions

-once kept, quite many players show fidelity to the game. Is it common with other games?

-boring, frustrating: could it be linked to machine power ?

-childs are not very interested: would it be usefull to create a "child" version ?

-random actions are not very interesting for science: should there be some tutorials with "guides" in order to develop hability to move parts of protein to a good shape ?

-progressivity: should there be a second type of tutorials, more funny (less utilitarian) where young players are invited to try to "copy" a small real (eg. poison) protein, replacing the score by a figure showing the effectiveness of the protein or poison ? The goal is to show the relation of a protein and it's ultimate contribution to something real.

I could imagine a Coronavirus "comics" picture getting worse when the binding protein gets more points.

(it's a gadget but I would personally find it funny to see a poisoned animal, a recovering person or whatever that makes me "feel" that what I'm doing is linked to real life aspirations). Using comics. Even, for fun (players like "bad" motivations, a sense of "immorality" in game), you could optionally change the "reward", e. g. with a rising mountain of gold coins or dollars you win if you get closer to the high score. Speaking of "immorality": when you use wiggle or recipes instead of hand fold, something says you that you are "cheating" in a way. It can be fun to deliberatily choose for facility.

Or something "unusefull" from science perspective, like graphics showing how mutch total time you or your team used hand folding (moving) versus recipes/tools.

Joined: 12/06/2008
Groups: Contenders
Concise?

Your definition of "concise" is very different from mine.

joshmiller's picture
User offline. Last seen 13 hours 23 min ago. Offline
Joined: 09/08/2017
Groups: Foldit Staff
Wow, cool data!

Hey Boots,

Thanks for putting that data together! Really interesting stuff, and I really appreciate it!

Yes, Foldit's retention rate is abysmal. This is why I'm dedicating my entire PhD to fixing it ;) I'll be happy to share findings with the players as my work progresses. We unfortunately don't have many developer resources for making progress fast on this, but I have a good sense of what needs to change, it's just a matter of time to make it happen. Happy to talk about this more in a PM or vet chat (just say Josh and it should ping me).

robgee's picture
User offline. Last seen 9 hours 49 min ago. Offline
Joined: 07/26/2013
Propz to the elders

Thanks Boots. Propz to you and the elders.

Joined: 07/20/2017
Groups: None
If Foldit really works

If Foldit really works everyone will be out of a job!

Also, get rid of the leaderboard. It breaks the game.

Foldit is marketed to gamers but is not actually a videogame, it is an interface. It's like claiming that drafting applications are videogames. They are not, they are interfaces for engineering. It looks like a video game to the untrained eye, but it is not. Fundamental flaw here.

Replace scoring with a voting system. People submit folds and the "scientists" vote on community solutions at the end of a round. The highest voted solution is the "top score" on the leaderboard. A player can click this top solution and discover how and WHY it's scored well solution and how to improve or fold something similar. That way we actually get somewhere as a community.

Those YouTube videos of the CoV19 solutions are disappointing to say the least. The top 10 scores were just players folding up literal random crap that had nothing to do with the problem described. I couldn't believe it. Nobody wants to play a game like that. You guys publish this too, to like make sure at the smart gamers who find it are immediately turned off of it. Anybody who has played more than 3 videogames knows to stay very far away from that kind of stink. Not that Foldit stinks, it just fits a model that reeks like garbage videogame, and when people smell that, they will run. Don't get me wrong Foldit is great, but people don't know that - they have to discover it.

Foldit's learning curve isn't even a curve. It's a pile of squiggles.

Like oh yes come waste your time chasing a score for science but even if you achieve a top Rosetta score, it's still not useful at all. Takes about 20 minuets of that and the tutorials before everyone nopes out. The only game I have ever played where you can get a top score on a leaderboard, but you still "lose" (although I understand there is no "losing" under the idea of premise for Foldit.) because it wasn't the right score. Again, nobody is going to play a game like that, and they don't.

There seems to be a lot of big egos around score and you are attracting specific types of personalities by attaching a leaderboard system in a video game to real world tangible results. Of the tiny amount of people you're attracting to Foldit only a very small portion stick around which seem to be almost exclusively players who want top points and top scores, not working or viable solutions. And surprise what to we have, 8 or 9 solutions with a great Rosetta scores, but not even remotely useful for the problem described.

Foldit keeps asking for help with things but then you keep seeing News articles about how that problem is already solved. Foldit is giving off the impression that it's community isn't achieving anything useful. The internet is FULL to the brim of articles about Corona vaccines. Lol like there's absolutely nothing we will be folding up in the game and cant or isn't done better in a lab or so it feels that way.

The only people who can play Foldit properly are biomed students and most of them are probably busy eating ramen noodles and calculating their massive debt loads rather than folding up cutesie lil proteins in their non existent spare - time but that's just me.

Foldit is not organized in a way which clearly communicates a community driven goal or objectives to a player. Currently it's fold up random things with some vague guideline. There are filters, but those aren't useful to the average player.

Personally i'm leaving Foldit due inability for the relevant parties to provide quality examples or guidelines to a sought solution. That's my number 1 issue. Most recent CoronaVirus puzzle is an excellent example. There's all this lore about some spike protein? that's apparently missing or this or that. I don't know. What I do know is that the info provided is not enough, and i'm not willing to waste my time folding random things until the folks asking for me to invest personal time and energy into a solution can properly articulate what they are asking for.

Foldit is a great idea but a terrible video game. The only saving grace it has with player retention is the idea they are "helping the scientists" but then Foldit fundamentally fails to provide that to the player by enabling an environment where players are folding without understanding what they should be trying to fold but their score is going up, leading to confusion and then a big let down when it is explained that scoring is essentially irrelevant

Get rid of scoring. No more points. Then the silly point chasers will either leave or learn to fold with an interest in community. Foldit ISNT ABOUT GETTING HIGH SCORES. It's about finding solutions to problems. That is the number one huge huge flaw with Foldit. It's the points. People are chasing points, not solutions.

I apologize if I seem crass or abrasive in my thoughts here but there's emotions involved. I thought what I was doing was helping. I didn't know exactly how or what but I thought and I believed that I was helping. I learned now that I was wasting my time.

Also as of late there are students doing Labs with Foldit and some of them have been questioning the legitimacy of the learning capacity of what Foldit has to offer, I honestly hope to gosh that the good people shelling out 50K plus for student loans have access to better learning resources than Foldit.

Joined: 09/29/2016
Groups: Gargleblasters
A lot of valid and personally relatable opinions, Neon!

I wholly agree on a lot of what Neon says, as unfortunate as it may be to hear. :\
However, on the flip side, some of the feedback given there is also --at least in my opinion and observation-- based on a slightly incomplete picture of how Foldit has, is, and will function.

For example, yes, the scoring system is flawed at present... but not to a point that merits its removal from the game. Here's my take on why:
1) The system functions quite well in most puzzles. It's when you get into certain ones that things get hazy, BUT that is only a recent development. Up until June or so, the Objectives we had did not include BUNS, DDG, SASA, or SC (aka New Metrics). So we really only had "Core Existence" and "Ideal Loops" to guide us. Unfortunately, back then the lack of New Metrics gave us a similar false impression: High Score must inherently mean Best Science... but that wasn't the case. The New Metrics are our guiding light now, and this is a very good thing and a much needed addition to Foldit, and its scoring system!

2) The New Metrics are, well, New. They're still working out the kinks, bugs, issues, and making them so we have a more refined and easier to understand objective: "Make this go up = Good!", simple lay terms, no science knowledge needed. The new combo objective that was attempted to be released with 1916 was a step in that direction, since it combined two less straight forwards metrics of SASA and SC. But trust me when I say that I feel your pain when it comes to the challenge here. Many of us have been dealing with their pitfalls since day 1 of their introduction back in June (with BUNS), and the headache they initially caused. They've been ironed out now, thanks to OUR (community's) feedback on them. If we just decided to quit, that wouldn't help anyone. Now the BUNS objective is not the same nightmare it once was.

3) It's important to know that the points are actually a real thing in Foldit, not some arbitrary fabrication by the devs in an effort to satisfy the reward center of our brain. No, the points are literally the very same metrics that drive Rosetta and any other protein software. They've just been turned into a layman friendly format that's easy to understand and follow. It actually works out to something like "Rosetta Energy Value + 8000" and +/-300 depending on the puzzle, but in general it's roughly that. The difference being that these puzzles with Binders and Locked segments, are pretty much always going to be a "cropped protein" that has lots of bits and pieces removed in order to make it run on our computers. Which means that some segments will have iiiiiinsanely negative scores and they need to be offset further than what the standard +8000 can account for. (Which is why the Linker puzzle scores into the 30K range whereas our binders are in the 12-15K)
Point being: It's inherently going to be flawed to some degree, in large part due to it being a simulation which is sacrificing some accuracy for speed; however, that shouldn't be taken as though the lack of accuracy means your solutions aren't valid.

4) Foldit has, and will continue to be, a curious beast. For all its shortcomings it's still able to allow people like you and I, who know next to nothing about BioChem, to come in and create legitimate and viable protein solutions.... if you are willing to put a little bit of time and effort into it. There are small things that can be learned from chatting with players, reading the Wiki, or reading outside science material, which can shed beneficial light on what you can do that will make folding seem far less "random".
Personally, this is the route I've taken. I intentionally don't burden myself with the vast amount of things I COULD learn... Why? Because that's the beauty of what we everyday-people have that the higher educated have lost (no offense scientists lol): creativity with a dash of ignorance, sprinkled with a gamer's intuition and bake at 400F for 7 days. Voila, a protein that, if it scores high enough (top 100 on the Scoreboard) is quite likely to fold in the lab, and is a design that the scientists wouldn't have thought of. Why not? Their knowledge of science actually hinders their ability to make things that defy what they know works. There's a quote by David Baker (whose lab, you could say, is behind Foldit coming to be) in a journal from back in 2010, that he mentions that he's horrible at Foldit and isn't able to get very high on the scoreboards, whereas his 12yr old son is able to walk circles around him in that regard -- not because the scoring system is 'broken', but because we non-formally trained players are unconsciously willing to try the absurd!

5) The scoreboards themselves... They're just here for our enjoyment. And yes, it's indeed what helps drive us in a lot of ways, by fueling our natural desire for competition and to be the dominant species :P NUT it doesn't mean that's what everyone aims for! I have it IN ME to consistently be in the Top 15... but I don't anymore because of the same reason that you quit: that in these Binder puzzles, the highest score isn't legitimately the best solution at present time -- that part is key, since as the bonuses get adjusted, it'll once again be that highest=best. So instead of quitting, I decided I'll just chase the Metrics. Lowest BUNS count, lowest DDG, highest SASA, and highest SC.
But that's where that "little bit extra science knowledge" comes in, since you can easily attain all of those with a purely HydroPHOBIC protein, but that won't do well in the lab. So that's where the Score comes back into play. I need to still make a protein that has a score of around say 13,000 for a Corona Binder in addition to those really good Metrics! Sure, this puts me in roughly 50th place, since the high score is 15,000... But for now while their Bonuses get refined, that has generally meant that my protein's metrics will more than likely be exceeding those of what the proteins in the top 20 have. (based on the screenshots we share after a puzzle expires)


Anyways, I've typed a big enough wall of words, so I'll end it there.
Neon, I genuinely hope you'll reconsider leaving! Staying and continually sharing your feedback is just as valuable to the project as making 'viable solutions', since the latter is not possible if the program isn't able to provide the necessary environment to facilitate that, and so letting them know where we have troubles accomplishing those goals is what will make everything better in the long run... :)

infjamc's picture
User offline. Last seen 1 hour 43 min ago. Offline
Joined: 02/20/2009
Groups: Contenders
You have a valid point that

You have a valid point that the solutions with the highest scores are not necessarily the ones that have the most scientific value. But in that case, it seems to me that the better solution would be to revamp the scoring system-- so that selfish interests would align with scientific goals. For example, maybe the new metrics (DDG, SASA, SC, etc.) should be given more weight?

infjamc's picture
User offline. Last seen 1 hour 43 min ago. Offline
Joined: 02/20/2009
Groups: Contenders
Addendum: Okay, looks like I

Addendum:

Okay, looks like I have just duplicated Formula350's Point #5. In that case, I'd like to suggest going one step further...What if we go one step further by elevating the new metrics to hard "minimum requirements"? For example:

  • Perhaps a solution should not be accepted until each metric is above / below a certain threshold? This would discourage score-chasing somewhat. (Obviously, the hard point is finding a good balance between avoiding discouraging newer players and ensuring the "scientific quality" of the solutions.)

  • Another possibility is to implement additional penalties for known artifacts of the scoring system. For example, since helices are known to score better than sheets (all else being equal), a penalty could be applied to all-helix designs (unless there is a scientifically-valid reason to prefer them, e.g. "they are more likely to fold up correctly").
Joined: 09/24/2012
Groups: Go Science
The learing curve

I estimated that the learning curve was about 2 years or 200 puzzles in order to reach the current top 15-20 soloist players, or the >50 who have a little bit more chance to "contribute to science than beginners playing random.

From this point of view, beginner's race is a good incentive to keep beginners on line during this long process.

Achievments are good rewards as well.

There might be other parallel competitions or rewards to keep players during the 2 years learning curve.

From the point of view of "contributing to science", evolver's race seems to be a way for beginners to have a short term "contributing to science" sense or reward.

One reward was "authored a paper". In recent policy of publication, there is more chance to be rewarded here: should it be reactivated ?

Sitemap

Developed by: UW Center for Game Science, UW Institute for Protein Design, Northeastern University, Vanderbilt University Meiler Lab, UC Davis
Supported by: DARPA, NSF, NIH, HHMI, Amazon, Microsoft, Adobe, RosettaCommons