A/B split testing for Recipes

Case number:845813-987271
Topic:Game: Tools
Opened by:ChristianK
Opened on:Sunday, January 24, 2010 - 21:53
Last modified:Saturday, July 13, 2013 - 03:42

When creating a recipe there's a problem that you really don't know whether you should put 10, 15 or 20 into a function like set_behavior_clash_importance.
It would be nice to be able to write a recipe that randomly chooses one of the three numbers every times the recipe runs and afterwards uploads information about the score change that the recipe produced and the time that the script took to complete.
This would allow evidence based fine tuning of the parameters of recipes.

(Sun, 01/24/2010 - 21:53  |  6 comments)

Joined: 05/19/2009
Groups: Go Science

Ah, I'm sorry for multiposting. It somehow didn't show at first.

Joined: 09/18/2009
Groups: SETI.Germany

So, a list of ALL possible variables and their range would be nice.

Normally, "real" random numbers are possible in LUA (as far as random numbers are general possible on a computer anyway), but unfortunately, the LUA standard library is not implemented into the game at this time.

Themarquis and me use pseudo-random-numbers.

Anser here if you want to know more.

Joined: 05/19/2009
Groups: Go Science

Of course it's possible to generate pseudo random numbers by writing a bit of code.

Here the problem is however not only about generating random numbers but also about actually measuring results of a script.
At the moment a script can't write a text file or upload data to the cloud and therefore it's nearly impossible to compare different values.

Joined: 09/18/2009
Groups: SETI.Germany

We need file access by scripts.

Joined: 05/19/2009
Groups: Go Science

The problem with full file access is that it might give us the possibility to blast the sequence in a real database.
When we get puzzles where the 3D structures is already known in uniprot a script that looks up the real sequence has an unfair advantage.

It's a huge problem in bioinformatics to make sure that you don't use information of the 3D structure of a protein that's already known to make a prediction about the 3D structure of the same protein.
Therefore the information exchange between the script and the rest of the world has to be limited a bit.

At the same time it's crucial to actually use the information that gets produced by running the script to improve the script for the use in future puzzles.
It would also nice to gather the information from all uses of the script whether the script runs on the computer of the author of the script or whether it runs on the computer of someone who downloaded it via the shared recipe site.

spmm's picture
User offline. Last seen 7 weeks 5 days ago. Offline
Joined: 08/05/2010
Groups: Void Crushers
Status: Open » Closed
Type: Suggestion » Bug

not doable


Developed by: UW Center for Game Science, UW Institute for Protein Design, Northeastern University, Vanderbilt University Meiler Lab, UC Davis
Supported by: DARPA, NSF, NIH, HHMI, Amazon, Microsoft, Adobe, Boehringer Ingelheim, RosettaCommons