Introducing Foldit Metrics
Foldit Metrics are a new kind of Objective. They will appear in the Objectives dropdown, under the score panel at the top of the screen.
Just like normal Objectives, Metrics calculate useful properties of your solution, and can award bonuses that boost your score. However, Metrics are different from other Objectives in that they are much slower to compute.
Normally we like to ensure that Foldit can calculate your score (the base Foldit score plus all Objectives) in less than 30 milliseconds -- brief enough that it appears to your brain as "immediate." That way Foldit can constantly update your score in real time as you fold your protein.
However, some kinds of protein calculations simply can’t be completed in that time. Anything that takes more than about 100 milliseconds would cause a noticeable delay, and Foldit gameplay would become frustratingly “choppy” as the scoring struggles to keep up with your folding. We’ve developed Foldit Metrics as a way to handle these slow calculations without interrupting regular gameplay.
Our latest devprev update includes support for Metrics, and we’ve posted a non-competitive puzzle for devprev users to try out the new features. After some time in devprev, we will release Metrics in a main update so we can start using them in our Science Puzzles.
Puzzles with Metrics will behave a little differently than other puzzles. Below we describe the Metrics features and discuss the new challenges they bring to Foldit gameplay.
Hand-folding with Metrics
Since Metrics are too slow to compute in real time, Foldit runs them in the background. Whenever you make a substantial change to your solution, the Metrics will start calculating in the background, while the rest of Foldit continues to respond to mouse clicks and keystrokes.
Until the calculation completes, your score at the top of the screen will be greyed out and will not update. When all Metric calculations are completed, the score will update and regain its usual color.
You can continue folding your solution while the Metrics are calculating.
When the Metrics finish, the calculations will automatically restart for your latest solution. Note that the Metrics will skip over any intermediate solutions, so you don’t have to worry about accumulating a backlog of Metrics to slog through. [CORRECTION: Metrics will continuously calculate in the background. When a calculation is complete, it will be permanently associated with the solution in case you want to go back.] If you want to see the Metrics for your current solution, you can just stop folding, and the Metrics should catch up in a second or two. If you don’t want to wait a second or two for score updates, you can disable Metrics while you are hand folding. [CORRECTON: You can disable Metrics while you are hand folding but your score will keep updating without the metric score while the metrics are running either way.] While a Metric is disabled, your score will update in real time like in regular puzzles, but the score will be invalid. To trigger a one-time Metric calculation while it is disabled, click the “Run” button next to the Metric.
Using recipes with Metrics
Existing recipes will ignore the new Metrics by default. You can run any normal recipe in a Metrics puzzle, and it should run just as fast as in any other puzzle.
This comes with an important caveat:
Existing Lua functions like current.GetScore do not include Metrics bonuses.
That means that the value returned by current.GetScore may not match the competitive score at the top of your screen. And the value returned by creditbest.GetScore may not match your competitive score on the Foldit leaderboards. Recipes will need to be modified to support Metrics.
In order to get your competitive score in a recipe, you will need to add together the value of current.GetScore and metric.GetBonusTotal. But be careful -- accessing Metric bonuses in a recipe can drastically increase the recipe’s run time! Every time you access a Metric bonus in a recipe, the recipe stops to wait for the Metric to compute.
Metrics are distinct from filters in Foldit recipes, and have separate Lua functions. Functions like filter.DisableAll will have no effect on Metrics, and filter.GetNames will not return the names of any Metrics. Our first release includes three new Lua functions for Metrics:
Return type: table
Description: Returns a table containing the names of all metrics in the puzzle.
Parameters: string name [only names of metrics are recognized, others produce Lua errors]
Return type: number
Description: Triggers the (slow) computation of the named metric. Blocks computation of the script until the metric is finished computing, then returns the metric score.
Return type: number
Description: Triggers the (slow) computation of all metrics. Blocks computation of the script until all metrics are finished computing, then returns the sum of all metric scores.
Learning to play with Metrics
It will take some time for us to figure out the best way to use Metrics in Foldit. We think that they will help us produce better solutions in Science Puzzles, but this has to be balanced with gameplay and fair competition in Foldit.
Compared to the base Foldit score, Metrics are much slower to compute, but the good news is that we don’t think they need to be calculated as frequently. Although we’d like to strive for solutions with decent Metrics, we don’t necessarily want to grind away at them to squeeze out tiny gains.
Likewise, we don’t want to invest too much importance in Metrics. The Foldit base score is still our primary tool for judging solutions, although we know from lab experiments that some Metrics have informative thresholds.
For example, we’ve seen that most successful binder designs tend to have a shape complementarity (SC) Metric > 0.60. However, it’s not clear that increasing SC beyond this threshold is helpful, and we certainly don’t want to sacrifice other design features (like a well-packed, hydrophobic core) for good SC.
With this in mind, we’ll be starting with Metrics that award a flat bonus at a threshold value. [NOTE: We are also trying out metrics that award increasing bonuses UP TO a threshold]. For example, we may award a set bonus for a binder with SC of at least 0.60, but you will not get a bigger bonus for increasing SC further than that. Once you find an initial solution that comfortably meets the threshold, we hope that you can turn off the Metric and only check it periodically while you optimize other features of your solution.
Beyond that, we’re not sure about the best strategies for folding with Metrics! Scientists traditionally use them to weed out poor designs from big batches, but never spend time tweaking those designs to improve their Metrics. This is an experiment and we don’t know where it will lead.
We’ll be counting on players for feedback about what works and what doesn’t. Please don’t hesitate to leave us feedback or suggestions, or to ask questions in the comments below!
Devprev users can check out the new Metrics now in the [DEVPREV] LCB1 Binder with Metrics puzzle.( Posted by bkoep 91 885 | Thu, 10/01/2020 - 09:02 | 6 comments )