Random Q

[snoopy]

Just a curiosity:

Are you guys thinking of making the network architecture p2p as Drak suggests?

If so: the idea of writing anti-cheat for p2p in a world ruled by movement/rotation limitation seems like an interesting problem to me... I wonder if I could throw some ideas around - maybe something revolving around a server-side replay of a client journal, looking for 1. physical impossibilities, 2. probability tracking - basically a tracking measure of mistakes & timing, 3. player profiling - basically a tracking measure of a particular player's tendencies & style.

I'm thinking - if you wrote good code to statistically handle those three items via the replay of a client journal, you could probably do a pretty good job of rooting out cheaters. 1. Would eliminate the really blatant ones. 2. Could eliminate the ones using deterministic coding (too much predictability). 3. Could help to catch the ones where something "takes over" when a threat's present...

Hopefully you could cut out all but some sort of a really well coded "autopilot" - and even then, 3. should allow you to develop "profiles" which you can blacklist.

[Krom]

A good way to defeat most cheats is to use ASLR in the memory, and have the program regularly scan its memory for irregularities.

[Avder]

Are there going to be multiple architectures available? Peer to Peer is good for pickup games and LANs, but it would be good to have client-server available too.

[dunkelza]

A lot of the server infrastructure stuff is still being decided. IIRC, they are trying to balance a lot of different concerns.

If you are interested, there are a number of discussions about server types, server locations, P2P, etc. on the D:U forums:
http://descendentstudios.com/community/

[Jeff250]

A good way to defeat most cheats is to use ASLR in the memory, and have the program regularly scan its memory for irregularities.

How would ASLR help? The cheat tool is running on the same machine as the game, so it can see the game's memory layout.

[Krom]

How would ASLR help? The cheat tool is running on the same machine as the game, so it can see the game's memory layout.

If the games memory layout is organized differently every time, it means fixed offsets won't work so someone would have to use pointer searches and the like to find the same data through reloads. It adds one major hurdle that will chase off most newbies with a memory editor.

[sigma]

In multiplayer Descent 3 definitely has a cheaters. They are trying to deceive me? F*ck, I am a cheater with years of experience in several online games, if I see a cheater.
I can make invisible cheat. But why? Just so I have the game statistics 250/30? It does not give me pleasure. Guys, I won a very cool cheaters even if I do not turn on the cheat.

[Jeff250]

How would ASLR help? The cheat tool is running on the same machine as the game, so it can see the game's memory layout.

If the games memory layout is organized differently every time, it means fixed offsets won't work so someone would have to use pointer searches and the like to find the same data through reloads. It adds one major hurdle that will chase off most newbies with a memory editor.

It's just security theater. Calling EnumProcessModules() is not a major hurdle! This isn't what ASLR is for.

[Drakona]

It sounds pretty server-based from everything I've heard so far. But we'll see.

I have some technical suggestions that may get most of the benefit of p2p without the security drawbacks. The main one is:

Spawn shots on the server for each player with a delay that corresponds to their ping to the server. E.g., if Lotharbot is shooting at me, and I have a 200 ms ping to the server, it waits 200 ms to spawn the shot, tags it "Drakona", and that's the shot that (from the servers POV) can hit me. From a latency perspective, this is client-side hits, or as close to it as makes very little difference; from a security perspective, it's server-side hits. I think it would be a good compromise.

That doesn't solve server-induced lag, but it would take most of the pain out of server-side hits.

I haven't lofted the suggestion yet. It's an easy thing to do after the fact, and I'm waiting to see what they do. I'm still hoping for an optional P2P mode.

[Verran]

UE4 does not have native P2P support. Descendant Studios would have to build an implementation into UE4 on their own.

P2P is on the UE4 roadmap, marked under 'Wishlist/Backlog', with 180 votes so far; go vote!
https://trello.com/c/o7mHCkGK/263-p2p-support

[Foil]

Spawn shots on the server for each player with a delay that corresponds to their ping to the server.

Not a bad idea. [ It goes without saying, this would have to be done strictly server-side, since clients should still see their shots spawn instantly. ]

That said, I'm not sure I agree that doing it "after the fact" would work well. It would be a noticeable change for players who have a good sense of timing for leading their shots. A change like that could be frustrating for some players, particularly if the gameplay "feel" has already been somewhat established.

[snoopy]

Spawn shots on the server for each player with a delay that corresponds to their ping to the server.

Not a bad idea. [ It goes without saying, this would have to be done strictly server-side, since clients should still see their shots spawn instantly. ]

That said, I'm not sure I agree that doing it "after the fact" would work well. It would be a noticeable change for players who have a good sense of timing for leading their shots. A change like that could be frustrating for some players, particularly if the gameplay "feel" has already been somewhat established.

I think the idea is that you only do it for purposes of server-side hit validation. The goal of the effect is to synchronize the server side shots individually to the defensive perspective of each player.

That being said: I'm not sure that it would have the desired effect, drak. In fact, I think it might exaggerate the ping effect (it takes X seconds for the location of the enemy to come to my machine from the arbitrator, X seconds for you reaction to get back the arbitrator, and then you're going to further penalize me by delaying my reaction by another X seconds?) By the logic, you might want to advance the spawn of my shots by 2X? (Also, what about varying pings?)

[Tebo]

Spawn shots on the server for each player with a delay that corresponds to their ping to the server.

Not a bad idea. [ It goes without saying, this would have to be done strictly server-side, since clients should still see their shots spawn instantly. ]

That said, I'm not sure I agree that doing it "after the fact" would work well. It would be a noticeable change for players who have a good sense of timing for leading their shots. A change like that could be frustrating for some players, particularly if the gameplay "feel" has already been somewhat established.

I think the idea is that you only do it for purposes of server-side hit validation. The goal of the effect is to synchronize the server side shots individually to the defensive perspective of each player.

That being said: I'm not sure that it would have the desired effect, drak. In fact, I think it might exaggerate the ping effect (it takes X seconds for the location of the enemy to come to my machine from the arbitrator, X seconds for you reaction to get back the arbitrator, and then you're going to further penalize me by delaying my reaction by another X seconds?) By the logic, you might want to advance the spawn of my shots by 2X? (Also, what about varying pings?)

I think that's an inherent problem of client server, the effective ping to another player becomes the sum of the ping of both players instead of the ping between them. With Drakona's method high ping players still have it harder to aim, but they don't have it harder to dodge, instead whoever is shooting at them has it more difficult. Looks like this would approach the ping fairness of P2P.
Identifying who is being shot at is probably the big difficulty here, sometimes I'm not even sure myself who I'm shooting at exactly

Another nice method, which unfortuantely is not symmetrical like this one, would be to on one hand do client side prediction of the shots fired at oneself, on the other hand show the shots one fires only once they get broadcasted by the server.
In that way your own shots are displayed in the same timeframe as the ships you are shooting at and in which the collisions are done on the server. The shots appearing delayed only reflects what is actually happening on the server and makes it easier to adjust for ones own ping (in D3 permissible C/S does this; unfortunately the shots get fired in the direction the server thinks your ship is pointing, instead of where you're actually aiming...).
With client side prediction ping still puts you at a disadvantage when doding, but at least if you do dodge it should not be counted as a hit. That is unless your position packages get lost/delayed, but I don't think there is a way around that besides client side collision detection.

I'd like to see P2P support, since it would allow fair transatlantic games, but I don't think they will have the time to implement and thoroughly test it if it isn't already supported by UE4.
By the way, a disadvantage of P2P I haven't seen mentioned is that the upload bandwith required grows with the number of players. This could become a problem in big games with most people having an asymmetric connection.

[Lothar]

It should have exactly the desired effect from a "dodges happen client-side" perspective.

The idea is that, for any shot packed the server receives, it sends out "a shot has happened" immediately to all clients (which they will receive in ping ms, or perhaps ping/2 ms depending on whether you're tracking one-way or two-way ping; either way let's call it [P]). So the client will see each shot at a delay of [P] just because that's how lagged they are. If the server creates a duplicate shot at a delay of [P] on its end, and only uses that shot for collision detection with the client at ping [P], then both the server and the client will be seeing as close to the same thing as we can manage. (Note that the server can actually spawn multiple copies of the same shot at [P1], [P2], ..., [Pn] for each player in the area based on their individual pings.)

It won't be exactly like the pure p2p dodging we're used to from D1 and D2, but it'll be much closer than the crap we put up with in D3, where you could see a shot miss by multiple shiplengths but the server decided it hit because you were playing on an overseas server.

(Remember, Descent is more about dodging than aiming. Yes, this means you have to lag-lead your shots even if you have a great server ping. But it also means drastically reduced "I know I dodged that", which means rewarding dodging skill, which means we get great players like LoNi.)

[Tebo]

(Note that the server can actually spawn multiple copies of the same shot at [P1], [P2], ..., [Pn] for each player in the area based on their individual pings.)

Very good point, I missed that.
There will still be problems with loss and changes in latency, but looks like this is as good as it's going to get for dodging and putting everybody on equal footing without P2P.

With regard to aiming/lag-lead, I still think it would make sense to only show the own shots once they are fired on the server (question than only is, which instance if you have one for each player in proximity). That should give a better feel for the lag and make leading the right amount easier. Of course people will probably complain about the delay, but really it's there either way.
It would also stop accusations of cheating when shots don't hit because the packets never made it to the server (instead the complaints would be about weapons not firing...).

[Drakona]

That said, I'm not sure I agree that doing it "after the fact" would work well. It would be a noticeable change for players who have a good sense of timing for leading their shots. A change like that could be frustrating for some players, particularly if the gameplay "feel" has already been somewhat established.

Ah. Yeah, I mean, it's not something as deep as p2p itself, i.e., it's not something that has to be thought about at an architectural stage. From a technical perspective, it's a solution that's equally easy to adopt at any time.

Obviously I'd mention it (if it's still appropriate) as soon as I have a good look at multiplayer, which will be long before most players see the game. I'm expecting to have the "make it as p2p as possible" fight somewhere between proving grounds and beta.

Identifying who is being shot at is probably the big difficulty here, sometimes I'm not even sure myself who I'm shooting at exactly

Well, a naive approach (doing it for everyone) would multiply the number of shots by the number of players . . . which still isn't that big a number, in modern computer terms. The obvious clever thing to do to make it more efficient is only create a lag-delayed shot when a shot comes within a cube or so of someone.

the effective ping to another player becomes the sum of the ping of both players instead of the ping between them.

Yeah. That's the server-induced lag I was talking about that this doesn't help with. I have some bad ideas for what to do about that, but no good ones.

For example, you can try similar approach: you can send position packets directly to the other guy, and the server can adjust you forward or backwards in time to simulate what p2p conditions looked like for the purpose of collision detection. The information is all there to do it that way, but that is a lot of moving parts and complication in server code, and it still has to trust you for what your connection to each other is.

You could make position and shots p2p and hits client-side, and let the server handle everything else with the world (like doors and kills and stuff). That makes cheating a lot more technically feasible, but as above -- the server theoretically has sufficient information to detect it, and players won't be anonymous.

I would be much more likely to go down either of these paths than I expect the DU guys to be, though. To me, shaving 25 or 50 ms off a connection is worth an insane amount of work . . . I kinda doubt they share those priorities. But we'll see.

So anyway. Yeah, I have some bad ideas for server-induced lag. I'll keep thinking about it and see if I come up with any good ones, but it's a harder problem. Fortunately, the client-side hits is the bigger of the two pain points, and that's the easier one to fix in a clever way. I'm really hoping I can sell that, that'd be an easy thing to do and a big win.

I'd like to see P2P support, since it would allow fair transatlantic games, but I don't think they will have the time to implement and thoroughly test it if it isn't already supported by UE4. By the way, a disadvantage of P2P I haven't seen mentioned is that the upload bandwith required grows with the number of players. This could become a problem in big games with most people having an asymmetric connection.

Yeah. This is already a problem sometimes with 8 players. There are ways to mitigate it. For example, you can slow the update rate way down for people who aren't close to you and who you can't see, or even fall back on C/S in those cases.

It doesn't help if you have 32 people in a single battle, but I'm thinking you might expect performance problems there for other reasons, and at any rate, you'd never have 32 people in a single battle for very long, and I somehow doubt lag would made it very high on the list of things causing chaos in that scenario.

With regard to aiming/lag-lead, I still think it would make sense to only show the own shots once they are fired on the server

That's certainly an approach you could take, in the 'telling you about it' class of strategies instead of the 'fixing it' class. Other things could be done along those lines -- for example, lag lead indicators on targets. Personally, I think that approach would drive me crazy. I fire a lot more shots than I observe hits, so breaking the verisimilitude on the former would bother me more.

[Sirius]

One minor complication with latency-offsetting shots on the server is that it only works correctly if the latency is predictable, which generally means it has to be consistent. If your ping is unsteady it might still break WYSIWYG, but generally, on a good-quality connection (regardless of where in the world unless the internet there genuinely is crap, not just far away) the difference should be relatively minor and it's certainly better than nothing at all.

[Lothar]

One minor complication with latency-offsetting shots on the server is that it only works correctly if the latency is predictable, which generally means it has to be consistent. If your ping is unsteady it might still break WYSIWYG

Right; I don't think you can fix an unstable connection in a CS model. Or any network model, really. Someone is going to have to deal with the instability -- whether it's unstable leading or unstable dodging. In that case, I'd rather have it be the person whose connect is unstable who feels it, so the incentives line up right.

Even so, "the server uses your average ping to determine shots" is going to feel a lot closer to WYSIWYG than "the server determines shots as if you had a zero ping" (the traditional model).

[Sirius]

Another potential complicating factor: my knowledge of how Unreal Engine does networking is pretty out-of-date, but in the old days it used something of an "object-oriented" approach; you had entities you wanted synchronized, you tagged them appropriately, and the networking stack would take care of the rest. Makes it really easy to implement netplay, but the flexibility in that model is a little questionable; can you fiddle with how that synchronization is done?

But, as I said, my knowledge is out-of-date. As long as there's a way to clone objects (there should be) and selectively synchronize them to particular clients (the main question), the rest is just paperwork.

[Jeff250]

What happens if someone artificially inflates their pings to the server? Do they get extra time to dodge players' shots?

[snoopy]

What happens if someone artificially inflates their pings to the server? Do they get extra time to dodge players' shots?

Maybe yes and maybe no. If you trick the system and just hold the outgoing traffic then I can see an advantage. If you have a longer ping in both directions, then your notification about the shot just gets delayed, too. (In other words: a "low tech" approach that just slows down the whole connection isn't likely to make too much of a difference. A "high tech" approach that only slow down the outgoing traffic is likely to give you a dodging advantage.

My idea is to genuinely let each player arbitrate for themselves... and have a statistical way to judge the integrity of the arbitration by auditing a blow-by-blow account coming from each client. I think this doesn't end up work too well in games where pointing and shots have infinite rates of change... but in a world where all changes are bound the possibilities also become bound, making them audit-able.

[Tebo]

Yeah. That's the server-induced lag I was talking about that this doesn't help with. I have some bad ideas for what to do about that, but no good ones.

Definitely no easy problem, I was thinking about distributed servers so everybody can have a low ping server for interactions with players that are close latency-wise, but that's obviously another bad idea...

I would be much more likely to go down either of these paths than I expect the DU guys to be, though. To me, shaving 25 or 50 ms off a connection is worth an insane amount of work . . . I kinda doubt they share those priorities. But we'll see.

Agreed on the worth of reducing latency, but I understand if the priorities are elsewhere. I'd much rather have a new Descent that draws plenty of players than one that flops but has perfect netcode.

Yeah. This is already a problem sometimes with 8 players. There are ways to mitigate it. For example, you can slow the update rate way down for people who aren't close to you and who you can't see, or even fall back on C/S in those cases.

It doesn't help if you have 32 people in a single battle, but I'm thinking you might expect performance problems there for other reasons, and at any rate, you'd never have 32 people in a single battle for very long, and I somehow doubt lag would made it very high on the list of things causing chaos in that scenario.

Yeah, the problem can be addressed to an extent. In a way it's similar to how the CS problem of concentrating all the upstream on one server can be addessed. I just brought it up because doing that increases complexity again and takes time to implement. Unfortunately for us UE4 already has CS support, but apparently not P2P.

With regard to aiming/lag-lead, I still think it would make sense to only show the own shots once they are fired on the server

That's certainly an approach you could take, in the 'telling you about it' class of strategies instead of the 'fixing it' class. Other things could be done along those lines -- for example, lag lead indicators on targets. Personally, I think that approach would drive me crazy. I fire a lot more shots than I observe hits, so breaking the verisimilitude on the former would bother me more.

If the ping is constant enough I think one can get used to it fairly quickly. After all the brain is already compensating a human reaction time of comparable size. It's probably been five years since I coincidentally played on a D3 server running permissible CS. From what I recall it felt weird at first, but I got used to it surprisingly fast. Than again maybe my ping was low, I don't remember. Might be worth testing that again with a few experienced player on a server running at a high packet rate (so the shots go more or less where aimed) and see what they think.
I don't like the idea of having a lag lead indicator, but the nice thing about these methods is that they are all client side and could be made customizable
I'm all for exploring them, since there are limits to what can be done in the 'fixing it' department.

What happens if someone artificially inflates their pings to the server? Do they get extra time to dodge players' shots?

Maybe yes and maybe no. If you trick the system and just hold the outgoing traffic then I can see an advantage. If you have a longer ping in both directions, then your notification about the shot just gets delayed, too. (In other words: a "low tech" approach that just slows down the whole connection isn't likely to make too much of a difference. A "high tech" approach that only slow down the outgoing traffic is likely to give you a dodging advantage.

My idea is to genuinely let each player arbitrate for themselves... and have a statistical way to judge the integrity of the arbitration by auditing a blow-by-blow account coming from each client. I think this doesn't end up work too well in games where pointing and shots have infinite rates of change... but in a world where all changes are bound the possibilities also become bound, making them audit-able.

Whether incoming or outgoing traffic is delayed doesn't matter, only the sum is relevant (as long as the delay is not content dependent). There is no way to tell on which way how much of the lag occurs without having an independent synchronisation mechanism (like a GPS based clock on client and server).

From the point of view of a player with increased ping there is no more time to dodge. Ideally they get the weapon information at such a time that an immediate reaction by them would arrive back at the server at exactly the time the server creates the weapon for collision detection, so it's like they did the collision detection themself.
However the weapon fired at the player will be less accurate, because the firing player had to lead more to compensate for the delayed firing. In turn the player with increased ping also needs to lead more, so it's still fair for anarchy type mods, unless the ping is changed deliberately depending on the tactical situation.
A problem would arise for mods like CTF, where being harder to hit at the cost of having more trouble aiming could be an advantage.

[Jeff250]

I'm not asking what would happen if I increased my actual network latency to the server. I'm asking, what if I artificially delayed responding to pings from the server, but delayed nothing else.

Maybe yes and maybe no. If you trick the system and just hold the outgoing traffic then I can see an advantage. If you have a longer ping in both directions, then your notification about the shot just gets delayed, too. (In other words: a "low tech" approach that just slows down the whole connection isn't likely to make too much of a difference. A "high tech" approach that only slow down the outgoing traffic is likely to give you a dodging advantage.

The "highest tech" solution, the one that I'm suggesting, would be to only slow down outgoing ping replies. Packets I send concerning my movement, shots, and so on would all go out without any delay.

My idea is to genuinely let each player arbitrate for themselves... and have a statistical way to judge the integrity of the arbitration by auditing a blow-by-blow account coming from each client. I think this doesn't end up work too well in games where pointing and shots have infinite rates of change... but in a world where all changes are bound the possibilities also become bound, making them audit-able.

Worst case, you have everyone involved record a demo, and then someone could manually try to observe inconsistencies. Doing it programatically becomes challenging, but I suppose the fact that there is already some precedence for such tools used on Descent 3 servers in real time means that it could be done with a journal.

[Tebo]

I'm not asking what would happen if I increased my actual network latency to the server. I'm asking, what if I artificially delayed responding to pings from the server, but delayed nothing else.

Ideally each packet would carry timing information, since it doesn't take much space, but yes if the measured ping was increased artifically it would give an advantage to that player. The shots used for collision would lag behind those the player sees.

[Jeff250]

Even if I had to include timing information in all of my outgoing packets, I could just include outdated information to trick the server into thinking my ping is high.

[Sirius]

I imagine that would be a significant help in straight-line stuff since you can make the incoming shots land way behind you. Less helpful in corridors or other situations that might have more randomly-placed shots since you're basically not getting an accurate view of where things are going anymore. It's also tough to dodge homing weapons if you don't know exactly where they are; if you have to time dodges late, that's as difficult as having to time them early (like you get in D3 on a 200+ ms connection).

[Jeff250]

It's a legitimate question of whether this would be a net help or a net hurt. But consider this simple modification: suppose the cheat tool delays delivering incoming packets to the game so that in the game you were seeing what actually was on the server but the cheat tool immediately plays a warning sound to give you, for example, a half second warning before someone fires an MD shot or what have you. I think that that can only be a net help. And I think that people who are actually good at this game could come up with much better ways of exploiting this.

Another concern I have is that, at best, you'd be estimating the time from the server to the client as the ping time divided by two, which could be problematic in some situations, because no matter how rock star your home ISP is, in general, routing over the internet isn't symmetric. At least with the traditional client-server model it's easy to understand that what happens on the server happens before you see it, but once you try to correct for this and these approximations become wrong for *whatever* reason, then you can have things happening on the server before you see it or after you see it, all depending on which server you're connecting to, in a way that would be difficult to diagnose and that I suspect would be difficult for players to account for or even understand.

[Lothar]

at best, you'd be estimating the time from the server to the client as the ping time divided by two, which could be problematic in some situations

... but which I suspect would be closer to accurate than "zero" in nearly all circumstances. In a C/S model like D3's, the server treated you as though you could see and dodge the shots on the server at the moment they were fired on the server, and treated any delay in your movement as just you sucking at dodging. In a partial-ping-predictive model, we treat ships as though they can see and dodge shots [P] later, and use [P] as an adjustment to try to make the client and server see as close to the same thing as we can. Which, yeah, can definitely cause problems if there's instability, but problems that are no worse than you already have in D3.

[Sirius]

Ultimately, though, this is why full peer-to-peer is the best solution; but I'd still rather play with fake peer-peer or client-side hit detection on C/S than just go with the D3-style model. D:U cannot be a serious competitive game with that.

[sigma]

I'm just wondering how can it be that my account has been banned twice Descent: Underground and as Sol Contingency, What's going on?
In my opinion, the developers have done everything possible and impossible, that I did not want to play a new game.

[Lothar]

There's an old saying: "if everywhere you go you smell dog poop, check your shoes."

What that means is if you find yourself having problems in a bunch of different places, instead of thinking the problem is with other people, you should consider the possibility that you're bringing the problem with you. Instead of telling others they're wrong, ask them what you can do differently.

[sigma]

In my opinion, they had no right to ban me. We will deal first day? It is disrespectful to me. They can shove up your ass even the cool game. They do not respect the players, they have no respect for people. If you want, you can play it. I will not play it. They made me angry and now they are not worthy of my attention.

[Pumo]

They have been very respectful to me and other users, so it may have something to do with how each of us behave.

IMHO, Descendent Studios forum is a very friendly place.
Just a matter of opinion I guess?

And by the way, you're not banned from the SolC forums, why you keep saying that?¡!?

[sigma]

I always respect other administrators are in multiplayer. But no admin will not allow to dictate the conditions, if it is not a complete moron.
Understand finally, you can not dictate the terms of experienced players. You just make me laugh