Introducing Foldit Metrics

Foldit Metrics are a new kind of Objective. They will appear in the Objectives dropdown, under the score panel at the top of the screen.

Just like normal Objectives, Metrics calculate useful properties of your solution, and can award bonuses that boost your score. However, Metrics are different from other Objectives in that they are much slower to compute.

Normally we like to ensure that Foldit can calculate your score (the base Foldit score plus all Objectives) in less than 30 milliseconds -- brief enough that it appears to your brain as "immediate." That way Foldit can constantly update your score in real time as you fold your protein.

However, some kinds of protein calculations simply can’t be completed in that time. Anything that takes more than about 100 milliseconds would cause a noticeable delay, and Foldit gameplay would become frustratingly “choppy” as the scoring struggles to keep up with your folding. We’ve developed Foldit Metrics as a way to handle these slow calculations without interrupting regular gameplay.

Our latest devprev update includes support for Metrics, and we’ve posted a non-competitive puzzle for devprev users to try out the new features. After some time in devprev, we will release Metrics in a main update so we can start using them in our Science Puzzles.

Puzzles with Metrics will behave a little differently than other puzzles. Below we describe the Metrics features and discuss the new challenges they bring to Foldit gameplay.

Hand-folding with Metrics

Since Metrics are too slow to compute in real time, Foldit runs them in the background. Whenever you make a substantial change to your solution, the Metrics will start calculating in the background, while the rest of Foldit continues to respond to mouse clicks and keystrokes.

Until the calculation completes, your score at the top of the screen will be greyed out and will not update. When all Metric calculations are completed, the score will update and regain its usual color.

You can continue folding your solution while the Metrics are calculating. When the Metrics finish, the calculations will automatically restart for your latest solution. Note that the Metrics will skip over any intermediate solutions, so you don’t have to worry about accumulating a backlog of Metrics to slog through. If you want to see the Metrics for your current solution, you can just stop folding, and the Metrics should catch up in a second or two.

If you don’t want to wait a second or two for score updates, you can disable Metrics while you are hand folding. While a Metric is disabled, your score will update in real time like in regular puzzles, but the score will be invalid. To trigger a one-time Metric calculation while it is disabled, click the “Run” button next to the Metric.

Using recipes with Metrics

Existing recipes will ignore the new Metrics by default. You can run any normal recipe in a Metrics puzzle, and it should run just as fast as in any other puzzle.

This comes with an important caveat:
Existing Lua functions like current.GetScore do not include Metrics bonuses.

That means that the value returned by current.GetScore may not match the competitive score at the top of your screen. And the value returned by creditbest.GetScore may not match your competitive score on the Foldit leaderboards. Recipes will need to be modified to support Metrics.

In order to get your competitive score in a recipe, you will need to add together the value of current.GetScore and metric.GetBonusTotal. But be careful -- accessing Metric bonuses in a recipe can drastically increase the recipe’s run time! Every time you access a Metric bonus in a recipe, the recipe stops to wait for the Metric to compute.

Metrics are distinct from filters in Foldit recipes, and have separate Lua functions. Functions like filter.DisableAll will have no effect on Metrics, and filter.GetNames will not return the names of any Metrics. Our first release includes three new Lua functions for Metrics:

metric.GetNames
Parameters: none
Return type: table
Description: Returns a table containing the names of all metrics in the puzzle.

metric.GetBonus
Parameters: string name [only names of metrics are recognized, others produce Lua errors]
Return type: number
Description: Triggers the (slow) computation of the named metric. Blocks computation of the script until the metric is finished computing, then returns the metric score.

metric.GetBonusTotal
Parameters: none
Return type: number
Description: Triggers the (slow) computation of all metrics. Blocks computation of the script until all metrics are finished computing, then returns the sum of all metric scores.

Learning to play with Metrics

It will take some time for us to figure out the best way to use Metrics in Foldit. We think that they will help us produce better solutions in Science Puzzles, but this has to be balanced with gameplay and fair competition in Foldit.

Compared to the base Foldit score, Metrics are much slower to compute, but the good news is that we don’t think they need to be calculated as frequently. Although we’d like to strive for solutions with decent Metrics, we don’t necessarily want to grind away at them to squeeze out tiny gains.

Likewise, we don’t want to invest too much importance in Metrics. The Foldit base score is still our primary tool for judging solutions, although we know from lab experiments that some Metrics have informative thresholds.

For example, we’ve seen that most successful binder designs tend to have a shape complementarity (SC) Metric > 0.60. However, it’s not clear that increasing SC beyond this threshold is helpful, and we certainly don’t want to sacrifice other design features (like a well-packed, hydrophobic core) for good SC.

With this in mind, we’ll be starting with Metrics that award a flat bonus at a threshold value. For example, we may award a set bonus for a binder with SC of at least 0.60, but you will not get a bigger bonus for increasing SC further than that. Once you find an initial solution that comfortably meets the threshold, we hope that you can turn off the Metric and only check it periodically while you optimize other features of your solution.

Beyond that, we’re not sure about the best strategies for folding with Metrics! Scientists traditionally use them to weed out poor designs from big batches, but never spend time tweaking those designs to improve their Metrics. This is an experiment and we don’t know where it will lead.

We’ll be counting on players for feedback about what works and what doesn’t. Please don’t hesitate to leave us feedback or suggestions, or to ask questions in the comments below!

Devprev users can check out the new Metrics now in the [DEVPREV] LCB1 Binder with Metrics puzzle.

( Posted by  bkoep 102 685  |  Thu, 10/01/2020 - 09:02  |  6 comments )
1
Joined: 12/06/2008
Groups: Contenders
"Substantial change"?

Please define "substantial change".

If I am flipping a sidechain to find its optimal position, the score will change dramatically with each new rotamer. Am I going to have to wait for the metrics to recalculate with each attempt?

If I am rebuilding/remixing a portion of the protein, the score will change dramatically with each new conformation. Am I going to have to wait for the metrics to recalculate with each attempt?

bkoep's picture
User offline. Last seen 22 hours 47 min ago. Offline
Joined: 11/15/2012
Groups: Foldit Staff
Any change to protein structure

Yes, both of these scenarios (flipping sidechains and rebuilding backbone) change the structure of the protein, and can affect the Metrics of your solution. So, if you want to know how these changes affect your Metrics score, you will have to wait for the Metrics to recalculate.

Our intention is that you would disable Metrics if you are doing lots of hand-folding and aren't worried about Metrics. For example, in a binder design puzzle you might disable the Metrics at the start while you construct your binder in isolation from the target; afterward you might enable the Metrics while you dock your folded binder against the target to develop an interface with good DDG, SASA, and SC.

Some Foldit actions do not change the protein structure, and these will not trigger the Metrics to recalculate. Examples are adding/removing bands, freezing/unfreezing parts of your solution, or assigning secondary structure.

jeff101's picture
User offline. Last seen 19 min 31 sec ago. Offline
Joined: 04/20/2012
Groups: Go Science
How will metrics affect wiggle?

Wiggle often takes a long time to settle.
How will metrics work with wiggle?

bkoep's picture
User offline. Last seen 22 hours 47 min ago. Offline
Joined: 11/15/2012
Groups: Foldit Staff
Wiggle ignores Metrics

That means Wiggle will not be any slower on Metrics puzzles. But Wiggle might diminish your Metrics in order to improve the base Foldit score, and in some cases this could result in a net loss of points.

This can lead to a sticky situation where you might have an incentive to NOT Wiggle your solution (since it causes you to lose a Metrics bonus). This is worrisome because, from a scientific perspective, it is always important to Wiggle a solution. Please let us know if you run into this problem in Metrics puzzles!

Wiggle reduces strain in your solution. A Wiggled solution is more physically realistic than an un-Wiggled solution, regardless of its Metrics. If you have a solution where Wiggle causes your Metrics bonus to go down, it means there is some strain in your solution, and that strain is a bigger problem than poor metrics.

spdenne's picture
User offline. Last seen 4 days 16 hours ago. Offline
Joined: 10/01/2011
Groups: Void Crushers
One problem with hand

One problem with hand folding, and wiggling, is that the score at the top of the screen does not change till after you stop wiggling. I use that changing score to decide when to stop wiggling. Instead of disabling the score update, can we see the score without the metrics component added, like is displayed (at a reduced update frequency) as the current score in the score board?

Joined: 05/09/2008
Groups: None
Thanks spdenne

This is now implemented in main!
https://fold.it/portal/node/2010602

Thanks again for your suggestion.

User login
Download links:
  Windows    OSX    Linux  
Windows
(7/8/10)
OSX
(10.12 or later)
Linux
(64-bit)

Are you new to Foldit? Click here.

Are you a student? Click here.

Are you an educator? Click here.
Social Media


Search
Only search fold.it
Other Games: Mozak
Recommend Foldit
Topics
Top New Users
Sitemap

Developed by: UW Center for Game Science, UW Institute for Protein Design, Northeastern University, Vanderbilt University Meiler Lab, UC Davis
Supported by: DARPA, NSF, NIH, HHMI, Amazon, Microsoft, Adobe, RosettaCommons