Determining your latency to battle servers

[PhatE]

Hey guys,

Figured I'd leave an article for you all that will help determine where the latency issues are with the new battle servers.

I went over this a little bit when I streamed Friday but I'll give a more detailed explanation here.

TL:DR - if you're a beginner at networking or haven't thought about it before then this article might confuse you a little to say the least. If that's the case then feel free to skip to the bottom and there are the tools used to determine whether you're going to be experiencing lag throughout playing.

Types of lag that you were seeing before and after the battle server implementation.

A bit of background for those who aren't very familiar with either method of how dawn of war was/is working.

Peer to Peer - A decent system for the most part, works by both hosts(you and the player) attempting to establish a connection to each other, agree on some parameters, after that phase players then finish the connection establishment and you can play the game.

Maintaining a healthy connection was the hard part, many contributing factors such as people torrenting (getting the latest porn), complexity of Network Address Translation (NAT) on some devices, lack of Quality of Service (QoS) cause shared households to suffer the most on cheaper connections such as ADSL, security technologies (firewall/AV products such as zone alarm, Avira) that were doing Stateful Packet Inspection (SPI) or even Deep Packet Inspection (DPI) causing user's machines to be relatively taxed system resources wise, device buffers being filled and not clearing after previous connections had become stale, ISP related problems with routing and not taking optimal paths, some people were no doubt using proxy services for privacy reasons however those would be very few.

Relic realised however that this system was flawed for the simple reason that people are on other ends of the planet so sessions are a bit less reliable than what they wanted. Therefore the addition to lag grace was introduced to help assist and allow the connection to 'catch up' and both players resumed exactly where the connection began to trail off.

To be fair, no system is without flaw so they made due with what they had. On the other side of the coin, as of recent (and past) blunders they haven't exactly boosted many player's confidence with their ability to provide a better experience.

For the most part this system worked. Both players had the exact same experience even though at times it was a little crappy having constant lag grace due to bad connection establishments and maintaining them.

Battle servers - In theory the superior choice. In practice however, it sucks (as of writing this post) but it's not entirely Relic's fault.

Operation is something like this (paraphrasing press releases from Relic and various articles);

Player boots up the game, Dawn of War sends a [SYN] packet to a battle server address, the server responds with a [SYN,ACK], Dawn of War Receives the [SYN,ACK] and then responds back with an [ACK] and the connection is made. This means that both ends can start sending packets/traffic/information to one another. This is a lower level description of what happens (basics of TCP operations).

To put the above paragraph into simpler terms, dawn of war tries to connect to a battle server, once successful you're good to go.

Here's where it gets a little fuzzy of what exactly the server does but from what I can tell, the server does all the connection handling and holds onto a few ID's that are assigned to the hosts (being you the players). It then receives the commands that you're all sending and goes 'Great! I got those, now it's time to multicast them out to all players' or unicast them for 1 vs 1.

FYI, multicast is sending traffic to a specific set of hosts and unicast is sending it to one host. There are probably more things going on in the backend so the likelihood of unicast being used is slim.

Through this method everyone gets the same data at relatively the same time.

Two major factors that are affecting those outside the USA;

Latency - this is quite obvious, all network applications suffer from this. It's just the nature of the beast. Not even the speed of light is fast enough for everything to work instantly due to the massive degree of factors between point A and point B which ruin this (read above). In terms of the battle server implementation the player either isn't sending their commands fast enough for the server to process them in time meaning that you see command lag (abilities won't activate, units move with delays, etc). I know there are some who aren't quite familiar with why those outside USA have issues.

It's due to distance. A simple example is imagine 2 track runners one starts at 20 meters but the other has to start at 100 meters. Those who get to the finish line get some ice cream. Both runners really want the icecream but only one person gets served at a time. This example is perfect to understand that everything in networking gets queued at some stage.

Server under load (CPU/high interface utilisation) - to be put simply, if the servers getting slaughtered by the trillions of packets that get sent to it every minute (yes, trillions given who owns that server) then it can't process your commands in time and send them off. Given that these servers are being rented from Amazon there's a high chance that these servers are constantly being pummelled to their eventual deaths. It also depends how many have been rented (factoring in redundancy and load balancing) and whether they've been purposed for battle server services.

I'm more inclined to go with option 1. Reason for this is because amazon is fucking massive, and their cloud is fucking massive as well. The data centres that they have to house their many sold products are amazing. Then again it could definitely be a combination of both but not at a 1:1 ratio.

Now after all that garbage you may ask, 'PhatE, you handsome, well-endowed, hunk of a man. What does this all mean?'

It means that now, with a certain set of skills and tools, you can determine where the issue is and perhaps help relic at the same time and give them some feedback as to where the fault is. It could even be the server you're connecting to and simply rebooting the game might select you a different one thereby enhancing your playing experience.

Below is really just basic troubleshooting. I could do a more advanced version of this but at this stage it's unnecessary (plus I have to think of a more advanced version)

TCPview from Microsoft http://technet.microsoft.com/en-us/sysi ... 97437.aspx

This is where you can explicitly find out what the server you're connecting to is. For this particular example it's decided to connect me to 54.86.28.86, there are a few more which I've gotten prior to this particular case and this one is by far the worst.

http://postimg.org/image/txyig4t6h/

Next up is a great tool which is 'ping' from cmd (Windows), terminal (MAC, Ubuntu), Konsole (kubuntu)

http://postimg.org/image/62lc2r40h/

the -t option will make the ping last forever on Windows until you stop it manually like I have at the bottom.

These pings were surprisingly ok since I'm watching 1080p twitch. However give it more than the 10 seconds I did it for the purpose of this thread, more accurate results are obtained that way.

To wrap up the article, give this a try if you're planning on playing the game and want to avoid the frustration of command lag and the like. An ideal latency I've found is 280ms or below. Anything higher then you start to see more issues.

Either way this isn't intended to fix anything but really to try and narrow down where the issues could be.

[holyoke]

Thank you for your contribution, even thoufh I didn't uderstand much

[Ace of Swords]

Cool, will do this later.

[Myrdal]

That's a lot of words, might as well add a section for disabling delayed acks and nagle's algorithm as these can add to the latency (they are by default enabled and for a reason, they improve throughput and reduce overhead). That's assuming they don't use udp, but it doesn't look like they are.

Here's where it gets a little fuzzy of what exactly the server does

Yes, there's a very apparent lack of info on these things. I was hoping someone had done some investigation especially considering they've been active for a while now in coh2. Anyway, it would be useful if you could add links to the info you've found in your post.

Two major factors that are affecting those outside the USA;

I'd like add another, shitty code (relic approved!). I have around 110ms to the us east coast, my load time and performance in general is probably better than average. So why am I getting lag so often, and even consistently each time a game starts by about 5-10 seconds? That's clearly not latency or server load issues, it's just broken code. This is a good thing though, assuming relic can get their shit together and produce decent code rather than the current mess that has "alpha" written all over it.

Btw, that multicast bit is wrong. Tcp has no support for multicast, it's strictly point to point.

[PhatE]

I'll edit this a few more times as I find more details, start probing dawn of war community

You're correct it is a lot of words and can probably do with a clean up at the minimum. I threw this together whilst it was on my mind at the time.

I'm highly doubtful it's UDP as well.

I understand that TCP is point to point. It wouldn't make sense otherwise. Perhaps it was the way I worded it that led to the confusion but I wasn't trying to imply that it was otherwise. I guess it was just a term I brought over from routing that I thought would fit the scenario. A more appropriate term would have been 'relay' traffic. But yes, I agree with that

http://blogs.companyofheroes.com/2014/0 ... e-servers/

"With the new system players will be connected directly to a Battle Server, which will relay game information between players. If one player has a poor connection, only their game will be affected."

I'll have to dig through to find what I've read here and there on the subject of battle servers. Some of them have been inside steam news and others from brief gaming news articles.

I'd like add another, shitty code (relic approved!)

hahah this is something I hadn't really considered. My experience with Relic's software practices is quite limited but now that you mention it from what they have been producing it hasn't been very reassuring. But yes this is a very plausible reason as well. I had asked one or two people in the US if they had similar issues to the ones that I was seeing and they had said 'not really'. I will say though that's nowhere near a credible sample size.

Funnily enough from what I get from TCPview we connect to a server that listens on https which is interesting to say the least. I had thought that would be only for the steam overlay but by the looks of where the connection goes to we do infact connect via HTTPS. Unless TCPview is leading me in the wrong direction when it comes to this deduction.

[Myrdal]

I had asked one or two people in the US if they had similar issues to the ones that I was seeing and they had said 'not really'. I will say though that's nowhere near a credible sample size.

People in the US would be less likely to notice these issues. They can afford poor netcode adding considerable delay because they have such a short access time to begin with. Relic ditching tcp in favour of a udp based protocol would help a lot. So would adding servers in different regions but that also means splitting the playerbase between these regions. Not really feasible until a widely successful dow3.

Funnily enough from what I get from TCPview we connect to a server that listens on https which is interesting to say the least. I had thought that would be only for the steam overlay but by the looks of where the connection goes to we do infact connect via HTTPS. Unless TCPview is leading me in the wrong direction when it comes to this deduction.

Yep, I usually see 2 https connections open but they don’t seem to be used by the game protocol (that run on port 2702x). They're probably used for chat, matchmaking, leaderboards and statistics instead but I may be wrong.