network disruption can permanently disconnect client

Case number:671071-993031
Topic:Game: Other
Opened by:Bletchley Park
Status:Open
Type:Bug
Opened on:Monday, June 25, 2012 - 12:00
Last modified:Saturday, March 5, 2016 - 10:46

Whenever there is a network disruption, also causing chat disconnects for which feedback 985890 exists, the in-game display of scores and most importantly the *uploading of scores to the server* is interrupted. One may have a best scoring client without connection to the server. If the game ends in that state, that best scoring client's score is not recognized. Also, all intermediary model states are not sent to the server in that case, harming the science part.

So, not only should a client detect it is not connected anymore, it should actively try to repair that connection, you could make that chat reconnect from feedback 985890 optional then.

(Mon, 06/25/2012 - 12:00  |  51 comments)


Joined: 05/19/2009
Groups: Contenders

I just experienced this on 584. As I just found out now, after hours of processing, it disconnected while in handfoling phase. In my client all scores of others are wrong, on the server I have 0 points, whereas in reality I'm over 11700 points in the client. Nothing updates.

Had this puzzle ended I would have had no points.

upping priority.

Joined: 05/19/2009
Groups: Contenders

This also means that none of the interim solutions were sent to the server and science data got lost.
As this can go unnoticed for quite some time, and many clients could suffer from this, the project suffers because of lost data. Upping priority to 1.

Most likely the network disruption is caused by a time-out in network traffic or the inability to access the DNS server.

Madde's picture
User offline. Last seen 11 weeks 1 day ago. Offline
Joined: 05/29/2008
Groups: Void Crushers

I had the same problem a few times in the last few weeks. The first time I noticed it only because I ran two tracks of the same puzzle and the clients' scoreboards were different.
Win7 64bit - DevPrev - I didn't use ingame chat, only IRC - 2-4 clients running

tamirh's picture
User offline. Last seen 6 years 10 weeks ago. Offline
Joined: 05/11/2012

assigning

Joined: 05/19/2009
Groups: Contenders

I'll give this a bump because it was not fixed in the latest devprev client. Having a button to reconnect is not good enough, nor is having to click the indicator light. The client should detect by itself and without user intervention that it is not connected anymore. I have clients running unattended for days at times.

One way to do this is to expect a heartbeat for every so-many results sent to the server.
No heartbeat = also no chat.

This could be a simple small datapacket from the server to the client saying 'thsnk you, got it'.

tamirh's picture
User offline. Last seen 6 years 10 weeks ago. Offline
Joined: 05/11/2012

The client does check periodically if it can connect to the Foldit servers to send scores. If it can't then the network indicator will turn red. Clicking the indicator will only recheck if you don't want to wait for the automatic check to happen again.

If at some point your network comes back, it will again try and send scores.

Are you seeing this not happen? When your network goes down and then it comes back are you still seeing scores not being sent? Is it showing green but your scores are not being sent?

Joined: 05/19/2009
Groups: Contenders

How periodically is periodically ? I rebooted my router and watched how long it would take for the client to respond. After a minute or so it detected the outage, but then I waited for several minutes for the client to detect the presence again and it did not until I clicked the indicator light.

Joined: 05/19/2009
Groups: Contenders

Also, the chat that gets interrupted will not automatically reconnect. I found only after trying to post this morning that I was disconnected, so I missed all comments that were posted overnight.

Joined: 05/19/2009
Groups: Contenders

I had another instance today, all lights remain red, no matter how long you wait.

tamirh's picture
User offline. Last seen 6 years 10 weeks ago. Offline
Joined: 05/11/2012

The chat will not auto-reconnect, only the server connection reconnects to send scores.

If you have the 'reduce bandwidth' option turned on, the max time should be 5 minutes. If you don't, then the max time should be 1 minute. If you are seeing the network status stay off for longer than this after you have regained network connection, could you post your log.txt from when that happened?

EDIT: Nevermind, I think I've found the issue.

Joined: 11/05/2010
Groups: None

Knock on wood, I'm having the best internet I've had since first starting Foldit over 18 months ago, at least 3 days without interruption and still using free wifi, however I am most painfully aware of this problem. Highly recommend BP's or similar fix to affect active and non-active clients. Internal chat has often not been necessary or even desired in some cases due to lack of BW, etc. Non-active clients are, however, of prime concern, as they are where the work gets done assuming one runs multiple clients and I believe most of us do.

Joined: 05/19/2009
Groups: Contenders

The log file is over 900 Mb large. I have searched for errors and found these:

(multiple of this one)
SRVR_THRD error: Transferred a partial file: Error contacting server. (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:86)

and

(multiple of this one)
SRVR_THRD error: Bad bool format for Online: ''. (..\..\..\rosetta_source\src\interactive\util\Dictionary.cc:247)

and a little while back in the file:
Error: Could not parse path/filename from: "puzzle_993229_time_1346509784.ir_solution"
Error creating temporary file falling back to unsafe direct write of save data
Error reading upload_tmp_992988_1339998308.ir_solution; expected SOLN, got .

Error reading upload_tmp_992988_1339998308.ir_solution; expected SOLN, got .

Error: Could not parse path/filename from: "upload_tmp_993229_1346509789.ir_solution"
Error creating temporary file falling back to unsafe direct write of save data
SRVR_THRD starting solution upload...
SRVR_THRD finished solution upload. took 2.55562 seconds.
SRVR_THRD starting solution data upload...
Autosaved puzzles/0000993229/0000108474/t01_00_added/autosave.ir_solution.
Autosaved puzzles/0000993229/0000108474/t01_00_added/autosave-best.ir_solution.
Autosaved puzzles/0000993229/0000108474/t01_00_added/autosave-recentbest.ir_solution.
Autosaved puzzles/0000993229/0000108474/t01_00_added/autosave-creditbest.ir_solution.
Autosaved puzzles/0000993229/0000108474/t01_00_added/autosave-best-improver.ir_solution.
Autosaved puzzles/0000993229/0000108474/t01_00_added/autosave-recentbest-improver.ir_solution.
Autosaved puzzles/0000993229/0000108474/t01_00_added/autosave-creditbest-improver.ir_solution.
Tool on_action_complete called

before that, thousands and thousands of lines:
Tool on_action_complete called

before that:
core.optimization.LineMinimizer: Inaccurate G! step= 3.8147e-008 Deriv= -2.15721 Finite Diff= 0.550606
core.scoring.NeighborList: Minimization stats: 15 score/deriv cals, 0 narrow-from-wide updates, 0 full updates.
delta_score: -3.58341e-010
Playing sound: 4
Tool on_action_complete called
core.optimization.LineMinimizer: Inaccurate G! step= 3.8147e-008 Deriv= -1.46649 Finite Diff= 0.285826
core.scoring.NeighborList: Minimization stats: 15 score/deriv cals, 0 narrow-from-wide updates, 0 full updates.
delta_score: 3.31056e-010
Playing sound: 4
Tool on_action_complete called

Joined: 05/19/2009
Groups: Contenders

The reconnect lights green/red will not reconnect when clicked after the connection was lost and instead crash the client.

Joined: 09/21/2011
Groups: Void Crushers

After the power outage my clients say they are connected but scores are not updating inside the clients.

Joined: 09/21/2011
Groups: Void Crushers

Seperate problem, see feedback about it. Has nothing to do with reconnecting.

Joined: 05/19/2009
Groups: Contenders
Status: Closed » Open

This bug still exists. Clients do NOT automatically reconnect after a network disruption. This has caused a lot of frustration over lost scores and lost positions. And even if you click the GREEN light after it lost connection, the client will not always properly reconnect. Often it will hang.

As I am the one originally opening this feedback I appreciate if I am also the one closing it, after I verify that it actually works, thanks.

Joined: 05/19/2009
Groups: Contenders

I just found out that my client that had been painstakingly gathering points on puzzle 787 had not updated the server for days while the status light was bright and shiny green. Effectively it is a mute client. Leaving me at position 96 in the rankings where I should have been in the 40s. This bug has been brought forward so many times and for so long now that I am no longer willing to spend my electric bill on non scoring puzzles.

I am contemplating to boycott foldit until this gets fixed.

Joined: 05/19/2009
Groups: Contenders

Same is true for 788, listed as 80th place, where I should be 36th.

Joined: 04/19/2009

BP - you are right.

I noticed in the last few minutes that my score hadn't been updated in my supposedly connected client. I opened another one and went to the puzzle and the track - still no update on the website.

I finally killed the supposedly connected client and reopened another. Finally the website updated.

It made no difference in my rank, but I was wondering how many folders did get caught in the same issue, with bad results. Sorry you were one of them.

Joined: 09/24/2012
Groups: Go Science

I've this problem every day with a Windows laptop. If I want my score updated on the server (or if I want to share), I must close the client, open it again, then take my best solution. From there, server is updated and I can share without hanging. I live with it but it's not very pleasant. ;(

RicGray's picture
User offline. Last seen 1 year 31 weeks ago. Offline
Joined: 11/15/2012
Groups: ricg test group

Next time this happens, could you send me your log.txt file?

Thanks

beta_helix's picture
User offline. Last seen 2 hours 53 min ago. Offline
Joined: 05/09/2008
Groups: None

We are trying to find out what is causing this frustrating issue.
Although UW has had some bad server issues the past few weeks, the status light should at least reflect that!

This has been a tough one to diagnose, but we hope to have a resolution for you soon.

We apologize for the obvious grief this has caused, and we thank you very much for your patience on this one.

gitwut's picture
User offline. Last seen 33 weeks 4 days ago. Offline
Joined: 05/18/2012
Groups: Contenders

I have also noticed this problem on puzzle 788. My high score is 9894. The website scores show 9890.

I saved a copy of the 9894 solution as upload-for-myself. If I open a client to puzzle 988 on different computers without loading anything, it shows the wrong score (9890). After I load my upload-for-myself solution the client shows 9894 but the website does not update the score.

I've done an upload-for-scientists of my solution: gw9894.358 solo

RicGray's picture
User offline. Last seen 1 year 31 weeks ago. Offline
Joined: 11/15/2012
Groups: ricg test group

When this occurs, without running any other clients, can you upload your log.txt file to SkyDrive at : http://sdrv.ms/1fjAqPP

You can use the following account to upload:
username : FolditUpload@live.com
password : TreeFrog

gitwut's picture
User offline. Last seen 33 weeks 4 days ago. Offline
Joined: 05/18/2012
Groups: Contenders

Unfortunately, I'm unable to replicate the problem since I shared a copy of it to scientists. Now it load as an evolver solution which already has a higher score than 9894. Isn't that a bug? I can understand why sharing it with group would cause it to become an evolver solution, but not if shared to scientists.

Joined: 05/19/2009
Groups: Contenders

+ Two years old and still a bug

Joined: 09/24/2012
Groups: Go Science

It's a little bit better now. But still problematic. For safety, I always open a new client in order to get credited or to share or change puzzle or track.

It seems that one client is ok (I suppose the first one).
For the other 2 clients:
-I can always stop recipe and save.
-I can always recover my saves when opening a new client.
-With the new client, everything works well until network cut (but this is now the client that work, the former good one can turn problematic)
-Some times (not always), I even can download to group, or load another solution, or change track
-It happens that I must wait a wile before it works

What does not work (= hang):

-to do many operations. e.g. save, edit the save (name or description), save (hang) or share (hang) or load (hang) or refresh list (hang). Or change track once, credited best, change track (hang)
-at least one of the 3 clients hangs immediately after saving

Sorry that I did not send a log yet. I'm so used with this problem (and busy with CASP) that I do everything fast and I forget to mention it.

Joined: 05/19/2009
Groups: Contenders

It cannot be a little bit better. It is either fixed or not. it is not.
Opened on: Monday, June 25, 2012 - 12:00
Getting tired of this demotivating issue.

Joined: 09/19/2011
Groups: None

I am currently having this issue. Per RicGray's suggestion, I was going to upload my log.txt file to Foldit's SkyDrive (now named OneDrive), but Microsoft wouldn't allow me to upload (even with the correct Username/Password). It's ok though, my log file is >6GB, and it would have taken forever. Isn't 6GB a bit large for the log file? I think I'll delete it, but I'll keep a copy in case the devs want to look at it.

Joined: 05/19/2009
Groups: Contenders

revived yet again, unfortunately...

Joined: 05/19/2009
Groups: Contenders

I think http://fold.it/portal/node/996649 describes a similar, if not identical, issue.

Joined: 05/19/2009
Groups: Contenders

I like to bring this bug to your attention again.
It is still present.
- connect client
- interrupt network traffic between router and external internet
- wait a few minutes
- reconnect internet
- client light will be green, yet...
- create new track, light is still green
- client hangs

Joined: 09/24/2012
Groups: Go Science

I tried to send a lock to the above mentioned account, but it's impossible (microsoft ask for a code that only you can receive).
I'll try to send it by e-mail to the same mail.

Joined: 05/19/2009
Groups: Contenders

I think you meant a log file ? You had me puzzled Bruno :)

Joined: 05/19/2009
Groups: Contenders

Time to draw attention again to this, still unsolved, very very annoying bug.

I have a lot of clients open and hung. All lights are green, nothing automatically reconnects, all hang when trying to reconnect or interact with the server.

Joined: 09/21/2011
Groups: Void Crushers

This is happening a lot lately. That a client loses the connection and it get never fixed.
I thought this was fixed a year or so ago. Sad that it is back. Just today it happened again.

Joined: 05/19/2009
Groups: Contenders

Please find attached the log file of two clients, both are not sending scores to the server and both are having reconnect isues with the chat. On both clients, the lights are red, and remain red. The chat lights turn grey and remain grey, the reconnect button cannot be used anymore after it is pressed.

It was discussed with Timo and Susume in veteran chat, via my separate irc client that worked well at the same time as the clients were having issues, that this is not just my issue and this is happening frequently. Susume has no trouble, Timo does. Susume runs windows 7 and so do I, Timo runs windows 8.

Joined: 05/19/2009
Groups: Contenders

trying again... this is a zipped file with .zip replaced by .png

spmm's picture
User offline. Last seen 33 weeks 15 hours ago. Offline
Joined: 08/05/2010
Groups: Void Crushers

Susume is in the US and BP and Timo are in Europe - not that I know anything about networks but dropped packets in prime time are not unusual and may be unnoticeable to normal users but cause problems in foldit. Just a suggestion

Susume's picture
User offline. Last seen 2 days 23 hours ago. Offline
Joined: 10/02/2011

I see a lot of lines in BP's log file that read "Error contacting server using URL: https://fold.it/comm/foldit_ops" - as Timo has pointed out, it would be useful to include the http return code when logging these errors, to help debug the connection issue.

On the IRC side, I find the following in BP's log:
ghosting: Bletchley Park
Notice : Nick Bletchley isn't registered.

Maybe there is a problem trying to ghost a username with a space in it. On top of that, the IRC portion of the client gets stuck (lights grey, reconnect button visible but disabled, unable to type in chat box) rather than handling the error well. Timo van der Laan also has spaces; maybe ghosting is failing there as well.

Joined: 05/19/2009
Groups: Contenders

You are brilliant ! Thank you Susume, you may have nailed the issue right there.

Susume's picture
User offline. Last seen 2 days 23 hours ago. Offline
Joined: 10/02/2011

Bletchley Park shows up in chat as BletchleyPark, so stripping out spaces in the ghost command may make it work. Another special case is all-numeric user names: player 01010011111 shows up in chat as u01010011111, so to ghost an all-numeric nick the client probably needs to add a "u" on the front.

That does not address the failure to update scores between client and server but may help with the chat reconnect button.

spmm's picture
User offline. Last seen 33 weeks 15 hours ago. Offline
Joined: 08/05/2010
Groups: Void Crushers

Duh maybe foldit should explicitly stop people putting spaces into their usernames, always been a bit surprised they work without 'the system' putting a %20 or similar instead of the space.
bruno K would appear to have the same problem.
As long time players special name changing dispensation may be required to set up new accounts for them.

spmm's picture
User offline. Last seen 33 weeks 15 hours ago. Offline
Joined: 08/05/2010
Groups: Void Crushers

Duh maybe foldit should explicitly stop people putting spaces into their usernames, always been a bit surprised they work without 'the system' putting a %20 or similar instead of the space.
bruno K would appear to have the same problem.
As long time players special name changing dispensation may be required to set up new accounts for them.

jflat06's picture
User offline. Last seen 21 hours 41 min ago. Offline
Joined: 09/29/2010
Groups: Window Group

Thanks for tracking that down, Susume! Foldit generates IRC friendly nicknames when you sign up. If your name is IRC friendly to begin with, they are identical. This is why it was only showing up for some people. The ghost command was using the get_nick function instead of the get_irc_nick function. This should fix the problem for people with different IRC nicks.

Joined: 09/24/2012
Groups: Go Science

May be linked to my name with a space but what happens now is the following:

When disconnected, the client is able to reconnect for sharing solutions only, but my score is never credited unless I open a new session.

So I could go very high - disconnected - and share.

I could be soloist credited 0 and evolvers credited 9000 from the shared solution.

Should I shange my name replacing the space by _ ?

Joined: 04/24/2014
Groups: None

If you want to do that just drop me a PM. I'll fix it for ya. :)

Joined: 09/24/2012
Groups: Go Science

Changing my name to Bruno_Kestemont didn't solve the problem....

LociOiling's picture
User offline. Last seen 2 hours 59 min ago. Offline
Joined: 12/27/2012
Groups: Beta Folders

Back to the connection issue topic, we've had a couple of recent server outages and slowdowns.

After one such incident, I noticed that some of my clients would hang or stop responding when I attempted to upload a solution. Other clients were able to upload and continue, even though they had been through the same outage.

It seems like the clients that stopped responding had the "disable network timeouts" option checked on the initial logon screen. The clients that kept going had it unchecked. (I had it checked or unchecked more or less at random, since I'm not sure exactly what it does.)

Chat with Mike Cassidy indicated that checking "disable network timeouts" may help if the server is running slowly, but it seems like it may have a downside after a server outage.

I've unchecked "disable network timeouts" on all my clients, so we'll see what happens Next Time.

In the mean time, perhaps this is an avenue to investigate in resolving this issue. Or perhaps someone could at least tell us what this option does and when it should be used.

Joined: 09/24/2012
Groups: Go Science

Unchecking "disable network timeouts" does not solve the problem.

I've a bad wifi connection and disruptions occur several times a day. This makes me regularilly loose ranks when I'm not available to reconnect manually (=open a new client with the same track) just before the end of a puzzle.

Sitemap

Developed by: UW Center for Game Science, UW Institute for Protein Design, Northeastern University, Vanderbilt University Meiler Lab, UC Davis
Supported by: DARPA, NSF, NIH, HHMI, Amazon, Microsoft, Adobe, RosettaCommons