Multiple networking issues causing all the problem

[psyrox2k]

First let me reference all my posts pointing to the same core issue.
http://www.xsyon.com/forum/showthread.php/5748-Can-t-stay-connected
http://www.xsyon.com/forum/showthread.php/5744-Loading-Entities
http://www.xsyon.com/forum/showthread.php/5699-Random-crashes..

In all my posts I've come to the conclusion loading entities is slow due to low download speeds, it goes extra slow when the connection stops receiving all-together, and then picks back up (perhaps error recovery but it takes a lonnnng time)

In game I see huge lag spikes before a disconnect, sometimes up to 2 minutes of no bandwidth moving at all, then I get a sudden spike of data like 34k/s for about 15 seconds where everything in the world catches up, text floods real fast etc etc. Then the game runs normal again at the 3 - 5k/s normal bandwidth. As soon as commands stop working, or typing doesn't register, I see my bandwidth meter drop to nothing, and sometimes this disconnects me to login screen.
The fact that this is intermittent, and doesn't effect everyone at once, generally means a hardware issue, or configuration issue on one of your many WAN IPs.


Now to strengthen my statements above, I ran an ICMP to my own ISPs gateway, with 0 packet loss, 100% packets sent and received which mean "I DID NOT LOSE INTERNET CONNECTION, to any server except yours"
I also ran into the following error messages several times while trying to post this, from your forums.

Service Temporarily Unavailable

The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.

---------------------------------------------------------------
Server error
The website encountered an error while retrieving http://www.xsyon.com/forum/showthread.php/5748-Can-t-stay-connected. It may be down for maintenance or configured incorrectly.
Here are some suggestions:
Reload this web page later.
HTTP Error 500 (Internal Server Error): An unexpected condition was encountered while the server was attempting to fulfill the request.


I hope that's enough proof to at least start looking at networking/configuration issues, if anyone has some REAL input to back this up would be much appreciated =).

[digdug]

I just bought this game today, I'm very excited to play.
Servers aren't UP?

singleplayer?

I'm sure they're doing everything they can to fix whats going on, I just hope I dont hafta wait all night

[psyrox2k]

The game is cool just a few issues need to be worked out at the moment.
Servers are up...
Single Player doesn't exist...
As with most/all? mmorpgs, the communication sucks and we have no idea what they are looking into, or what they are fixing =/. More communication is always good, even if it's bad news.

[ColonelTEE3]

There's enough abbreviations i don't know and words i don't know in this guys post that i believe he knows what he's talking about.

Devs should probably listen to him.

[psyrox2k]

lol, they are common abbreviations for networking, I'm just looking for someone to at least look at it as a possibility. I've seen similar behavior with major ISPs I had to argue with before they looked at their own network. The poor guys ran new lines, fiber all kinds of stuff they spent 2 months and it was a misconfigured router lol...
Basically my theory on this one is they have multiple IPs on their WAN.
In lamens terms, an IP is a house and the WAN is street. They have a whole freakin apartment complex with lots of IPs, but it's still going to the WAN or the street ;).

One of those many IPs they own is misdirected (in my theory) or routed wrong, so all those packets that should go somewhere, get dropped into nowhere, which creates a biggggg lag spike when your computer isnt' getting updates from their server anymore. Then error correction is like yo I'm not getting any data and I'm gonna disconnect soon, but their server does quality control and pushes the same data at a super fast speed (to catchup for the time you didn't get any data) down another IP that IS pushing to the proper route, and it stays on that route for a while so you go around lag free, and yer like ya this is schweet no more lag they fixed it!!! and then like 2 hours later you get that bad link again and lag all to hell.

That's my idea in theory ;) I'm probably wrong but it's worth looking at.

[JCatano]

What address are you tracing?

[psyrox2k]

I'm not tracing any address, as I'm not sure what IP pool they use, although it wouldn't be hard to find. I just noticed when I get the lag spikes, and disconnect issues others are talking about, I'm seeing my network activity drop to almost nothing on a bandwidth monitor I run, yet at the same time I see downspeed drop (meaning the Xsyon server isn't sending me any data), I can still ping other internet addresses just fine. I also ran a bandwidth test during the time I had issues, to verify it wasn't my end, and it wasn't.

I'm not saying Xsyon HAS bandwidth issues, I'm saying the problem I'm seeing seems to be data from the Xsyon servers stops completely, which causes the lagspikes, and in some cases disconnects. If it doesn't disconnect, I get a sudden burst of data, which is catching up from the lack of data to resync the game, and then all is well again.

I'm basically trying to provide enough information and hard evidence as possible to help them diagnose the issues so I can play =).

Tonight things ran awesome after I started posting stuff, probably a coincidence though =O.

[JCatano]

173.208.85.178 is the one someone PM'd me.

[caldrin]

That traces to hypernia server hosting company.. They do have a site in chicargo by the looks so i guess its them they use.

haha in the us their premium dedicated server costs $395 but click the uk and that goes upto $500 nice..

[xyberviri]

when your logged in run netstat -a -f and you will see usually the very last connection is the xsyon server when nothing else is using the internet on my pc.
I almost always trace to the Xsyon server in under 80ms.

I dont believe its a bandwidth issue i think its something to do with there world server

[blackzilla]

lol, they are common abbreviations for networking, I'm just looking for someone to at least look at it as a possibility. I've seen similar behavior with major ISPs I had to argue with before they looked at their own network. The poor guys ran new lines, fiber all kinds of stuff they spent 2 months and it was a misconfigured router lol...
Basically my theory on this one is they have multiple IPs on their WAN.
In lamens terms, an IP is a house and the WAN is street. They have a whole freakin apartment complex with lots of IPs, but it's still going to the WAN or the street ;).

One of those many IPs they own is misdirected (in my theory) or routed wrong, so all those packets that should go somewhere, get dropped into nowhere, which creates a biggggg lag spike when your computer isnt' getting updates from their server anymore. Then error correction is like yo I'm not getting any data and I'm gonna disconnect soon, but their server does quality control and pushes the same data at a super fast speed (to catchup for the time you didn't get any data) down another IP that IS pushing to the proper route, and it stays on that route for a while so you go around lag free, and yer like ya this is schweet no more lag they fixed it!!! and then like 2 hours later you get that bad link again and lag all to hell.

That's my idea in theory ;) I'm probably wrong but it's worth looking at.

I already made this conclusion almost two months ago when I monitored major packet loss.

[psyrox2k]

Thanks for also noting that you are seeing packet loss, and to the other post, I never said it was a bandwidth issue, I said it stops sending data which can happen with a misconfiguration, or faulty hardware. You can have tons of bandwidth but if it's sending it to a dead link, or thru faulty hardware, it gets dropped.

As far as that command, that will show you what you are currently connected to, but that IP you are connecting to, maybe just a gateway to a network of IPs that are setup for qaulity control (like say you have 5 100mbit connections, if connection 1, 2 , and 3 are at around 75% utilization, it rolls over the other connections to connection 4, and 5, to keep bandwidth from maxing out on a specific device). This is probably similar to their setup, as you need tons of bandwidth on the headend to feed 10,000 people connecting who are each asking for 5k/s constantly, that's 50,000k/s their server has to deal with in output in this scenerio or 500mbit total at it's peak =) roughly.

Anyways none of that matters, if they are having packet loss, it's either a configuration error, faulty hardware, or a problem with whatever phone company or isp is feeding the data farm hosting their server, and it can be found with some simple tests.

However everything has been running a lot smoother since last night, and this morning. Either that means less people were on, and it showed, or they fixed something =)

[niccoli00]

I think you're making a large assumption that multiple machines are being used.

In the end, you'll be connecting to 1 IP address, that is possibly being load balanced to multiple machines.

Loss of packets does not always happen because of a network issue, which is something else to keep in mind. I haven't had any real issues, other than combat for a while now. I haven't bothered to do a sniffer trace or anything to see what the traffic looks like, but if you insist on trying to show something, I'd start there. ICMP is only useful to an extent, to help show there might be an issue, then you should dig deeper. Also a sniffer trace will help show what IP you are connecting to, and I would not be surprised to find it is one IP. HOPEFULLY not the same as the website. :P

[psyrox2k]

I never said multiple machines I said multiple IPs, and that's not and assumption. Everytime you connect you will notice you are talking to a different IP. They also host from Steadfast.net not Hyperion, unless those companies are one in the same, or they switched recently. I also checked their update section which clearly stated they had bandwidth issues (which is why we noticed packet loss) which should be fixed now and appears to be. I never said I ICMP'd their servers, that's not even legal.... I said I said ICMP'd "MY" ISPs Gateway to prove I still had connection during packet loss, which is legal to my knowledge to ICMP my own bandwidth all I want, as I'm paying for it, and I didn't do it long enough for it to be an issue, maybe 10 minutes while I was testing my disconnect. In any event they also mentioned some issues with other people's lag on slower connection causing lag for the rest of us, which would be code optimization they are working out.

This post was just to help them diagnose in the first place but everyone seems to think everything I post is opinionated. I clearly stated the difference between the facts and my opinions, read between the lines =)

Anyways I think they got it under control now, so I'm done posting unless I see more problems arise that need attention.

Added after 11 Hours 14 minutes:

---------------------------------------------------------------------------------------------
Bumping this up to let you guys know after the latest patch loading entities seems to be pushing 4 - 8k/s again. Yesterday and today before the patch loading entities was pushing 250 - 300k/s and i was loading in under a minute instead of 10 minutes. There seems to be some sort of bandwidth? issue still, or intermittent problem with quality control not pushing the correct amount of data? 250k/s for loading entities is fine it loads fast like that, 4k/s is not =O