6 replies [Last post]
axcho's picture
User offline. Last seen 7 years 31 weeks ago. Offline
Joined: 12/18/2007
Groups: None

Thought I'd share with you the research paper I wrote based on my work on Foldit, Designing and Evaluating Player Learning in Human Computation Games. The first half is about the theory I've extracted, about the design considerations important for learning games, and the second half is about the mistakes we (I) made in adjusting the design. Some statistics are made public there.

Here's the abstract:

"We present a new approach to the design of effective training systems for participants in human computation projects. When the computation involved is not only too complex for computers to solve, but too specialized even for an untrained human to handle, there must be an effective and enjoyable way for would-be participants to learn the skills necessary to contribute meaningful results to the project. Our work on the protein structure prediction game Foldit indicates that player learning is most strongly affected by a small set of key design components, whose effect on player understanding and ability can be evaluated through both formal and informal data collection."

Read it.

Feedback welcome. :) Let me know if you have any questions.

Joined: 12/14/2008
long blog post ^^

and one month old :(

I will give a few thoughts for certain points. What has jumped in my mind *while reading*.

"[8]. People like figuring things out."
100% true. A game is no fun when there is nothing to find out.
"And the more hints given to players to 'help' them complete each puzzle, the less assurance there is that they have actually learned the skills being tested, rather than simply having followed instructions without understanding."
Test puzzles: e.g. a one-move sidechain change: one right position, 4 wrong - can the player do the right one on first try? (if not, restart)

"The purpose of such an introductory sequence is to ensure that each player gains a certain conceptual understanding of the problem,"
more explanations. *why* should I move a sidechain? answer like: In this puzzle we are learning about sidechains. A sidechain is (short) and with moving it in the right place, the protein can be packed more compact. Let's see if you can do it!
|I alread wrote it somewhere a time ago: why and what it is what I am moving? were my biggest qeustion as i started.

"how well a player understands what characterizes a good solution"
Yeah, the why ^^

"Feedback is the design component that has the most impact on a player's understanding of a game's goals."
thump up

"the player is expected to experiment, to try out different actions in different ways"
Half-OT: Tool to bend amino acids easily (like a 3D strength regulator) - I was never able to do this as I wanted so I'm stuck with rebuild.

"indicating which goals were affected and whether they were improved or worsened by the player's action."
Really? I never watch the "bit of text"
That is the reason why I suggested the improvement of the currend ctr+i tool. Better watch on single scores.

"In order to encourage a more balanced understanding of the goals involved, we modified one of the unused visualizations such that it could be inserted into the existing puzzles easily."
Which one?

"Progression deals with the way new skills and concepts are introduced and how the player learns based on previous knowledge."
idea: one puzzle through the tutorial - one tutorial level, go to "learning puzzle", back to one tutorial puzzle, back to the (changed) tutorial puzzle - would also introduce restore very best etc. at this training puzzle at the appropriate time.

"finding ways to use existing skills and knowledge for new purposes or problems. Progression ensures that this process remains smooth and enjoyable."
training puzzle, too

"whether because the challenge itself is too difficult"
the flippin sheets ^^ It took me 3 days nnd 5 tries to get it. I mostly took the wrong sheet ^^ btw: I have written a solution to 2 puzzles in the wiki where players often get stuck.

"where all the instructional text was moved into a sequence of text bubbles that would point to specific parts of the protein or the interface, instead of having all the text appear at once in one area of the screen. Though the effect on the player experience was significant, the change hardly registers in the statistics."
Yes, it was a way better feeling.

-----------------------
2 points: 1. I really like the idea of the training puzzle. I have just finished a tutorial for another game and used a do-it-like-there method in some cases. (Perhaps thats why I got the idea now)

2. The team should show more concepts. e.g. I heard for a long time now that a new interface is planned, but have never seen a bit of it.
Having a concept shown, a team member there to answer and a bit advertising in the chat would give a big amount of feedback from the players. (Have you ever asked e.g. in a thread here the players about their problems with the tutorials?)
At the end, you have to do it like the players want it (or they wil go away), so you can ask them from the beginning on ;) And of coure: Never let a programmer test his UI. It may fit him, but not 99% of the people ;) I know cases of programmers blindness from own experience ^^

axcho's picture
User offline. Last seen 7 years 31 weeks ago. Offline
Joined: 12/18/2007
Groups: None
Just to clarify, the theory

Just to clarify, the theory section is not meant to be a description of what we did well in Foldit - in fact, most of what exists in Foldit violates these principles, to its detriment. This is more a description of what Foldit *should* be like.

"more explanations. *why* should I move a sidechain? answer like: In this puzzle we are learning about sidechains. A sidechain is (short) and with moving it in the right place, the protein can be packed more compact. Let's see if you can do it!"

I agree. Tell the player why it would be a good idea to do something (whether explicitly or implicitly), make sure they know what tools are at their disposal, and they'll do it, by trial and error if necessary, with much more motivation than if you tell them how without telling why.

"Really? I never watch the "bit of text"
That is the reason why I suggested the improvement of the currend ctr+i tool. Better watch on single scores."

Yes, there is a huge amount of room for improvement in the feedback in Foldit. The "bit of text" is close to worthless, though players do notice it. I have a few ideas, some of which I've probably mentioned in other threads.

"idea: one puzzle through the tutorial - one tutorial level, go to "learning puzzle", back to one tutorial puzzle, back to the (changed) tutorial puzzle - would also introduce restore very best etc. at this training puzzle at the appropriate time."

This is a cool idea, kind of like the way World of Goo Corporation mode works in World of Goo. Certainly worth trying.

"the flippin sheets ^^ It took me 3 days nnd 5 tries to get it. I mostly took the wrong sheet ^^ btw: I have written a solution to 2 puzzles in the wiki where players often get stuck."

The register shift puzzle is just poorly designed. We are well aware of how unforgiving and unintuitive that puzzle is and how many people get stuck on that one. However, fixing it is easier said than done, especially when all the preceding puzzles are poorly designed as well. We'll find a better solution eventually though. :)

"1. I really like the idea of the training puzzle. I have just finished a tutorial for another game and used a do-it-like-there method in some cases. (Perhaps thats why I got the idea now)"

What game? Can you post a link?

"2. The team should show more concepts. e.g. I heard for a long time now that a new interface is planned, but have never seen a bit of it.
Having a concept shown, a team member there to answer and a bit advertising in the chat would give a big amount of feedback from the players."

Yes, there has been a concept animation for the new interface, and I've been really wanting to show it off to you all ever since I first saw it. It's really cool, at least to me. :) However, if I remember correctly the team was hesitant to show the concept this soon. I can ask again though.

Joined: 12/14/2008
other game

What game? Can you post a link?
You think too much of this ;)
It's only text, as this is the only thing that is possible in-game. So it mostly runs on the line of "As you have done it there, right clicking also works here". And I have to go over the texts one time more.

But the game (an UFO/X-COM Clone) is here: http://xforce-online.de/ German based, there is only (after I finshed) an english tutorial, the GalWar and a small, not worked-on gameset that are not German. Of course, you can start your own one ^^
If you really want to have a look, wait a day for me to proofread ;) But this is the place where my opinions and experiences on making a game come from for 80%.

Joined: 05/19/2009
Groups: Go Science
Why do you give all new

Why do you give all new players who play the same version the same levels?
That seems a huge waste in data that you could gather by not having three variants for each level and randomly giving players either level A, level B or level C.

Not all feedback has to be consciously noticed. "Getting it" is often a very inutitive process and maybe the bit of text could be aided by sound effects.
Every time that the game displays "Protein breaks apart" it could play a sounded to inform the player about the event.

axcho's picture
User offline. Last seen 7 years 31 weeks ago. Offline
Joined: 12/18/2007
Groups: None
An interesting idea. If we

An interesting idea. If we were building the system again we might do it that way. I'm not sure what it would take to change the system to support it now though.

Goal-specific sound effects are definitely a good idea.

Joined: 05/19/2009
Groups: Go Science
That's more or less the way

That's more or less the way google works.
When they didn't know which shade of blue is the right one, they run tests with 40+ different shades of blue and the one that effects the metrics the best gets choosen.

Sitemap

Developed by: UW Center for Game Science, UW Institute for Protein Design, Northeastern University, Vanderbilt University Meiler Lab, UC Davis
Supported by: DARPA, NSF, NIH, HHMI, Amazon, Microsoft, Adobe, RosettaCommons