The DBB needs your help ....

KoolBear · Post by **KoolBear** » Fri Mar 04, 2011 5:51 pm

I have talked with Diedel and started talking to Zico about game trackers. I have also been talking to Ryujin about mirroring his D3 tracker here o the DBB.

Diedel is willing to update the XL code to allow for more information to be displayed by the DBB. BUT I am not a programmer and really need some help here.

Below is the current tracker that D2X-XL has been using.

I would like for this program to write out a file that that I could in turn read it into the front page of the DBB through an iframe or some such method. I realize in it's current state there might not be much to work with but surely you gurus can work something out.

Wouldn't it be nice to have a place to see any version of Descent beig played and also to join said game!

Code: Select all

# Author: Sebastian "VEX-Munk" Starosielec © 2005
# Contact: sebastian.starosielec@ruhr-uni-bochum.de
# License: GNU Public License #!/usr/bin/perl
# Server Program
use IO::Socket::INET;
use Data::Dumper;

# Config
my $TIMEOUT;
my $LISTENPORT;

$TIMEOUT = 30;
$LISTENPORT = 9424;

# Create a new socket
$InSocket = new IO::Socket::INET->new(LocalPort=>$LISTENPORT,Proto=>'udp');
print ">> Server Program <<\n";
%Tracker=();

# Keep receiving messages from client
while(1) {
	$InSocket->recv($Message,8);
	print "Incoming Message: ", $Message , "\n";
	if ($Message =~ /^S/) {					# incoming Message from Server: store Message in List
		print "Message from Server\n";
		$key = $InSocket->peeraddr() . chr(int($InSocket->peerport() / 256)) . chr($InSocket->peerport() % 256);
		$Tracker{$key} = time();
		print Dumper(\%Tracker);
	} elsif ($Message =~ /^R/) {	# incoming Message from Client: read Message from List
		print "Message from Client\n";

		# Updating Tracker list, delete old Tracker Entries
		while ($key = each(%Tracker)) {
			print "Differences ", $key, ": ", time() - $Tracker{$key}, "\n";
			if (time() - $Tracker{$key} > $TIMEOUT) { #Timeout ?
				delete $Tracker{$key};
			}
		}

		# Creating Response
		$Count = keys(%Tracker);
		print "Count: ", $Count, "\n";
		$Response = "S" . chr($Count);
		while ($key = each(%Tracker)) {
			$Response .= $key;
		}		
		$Response .= "\x00" x (512 - length($Response));
		
		print "Length Response: ", length($Response), "\n";
		$InSocket->send($Response);		
	}
}

Jeff250 · Post by **Jeff250** » Sat Mar 05, 2011 12:58 am

Do you have any high level documentation of the protocol or some example input? If I were to take this on, I would rewrite it in something that's not perl.

Do you have any preference for the text file? JSON for instance (php can easily suck this up)? Maybe we should just write code that talks to the database...

Sirius · Post by **Sirius** » Sat Mar 05, 2011 1:47 am

This tracker seems to only keep track of a list of server IP addresses and ports - it doesn't retain any data on the game itself. To show a list of games that would be viewable on a website, you'd need to have it also take on some of the abilities of a client.

I have to agree with Jeff that it may not be worth retaining this script for the sake of it... most effective solution would probably be a daemon (perl, PHP, whatever, doesn't matter much) that runs on the same server and has write access to the appropriate SQL database.

Ryujin · Post by **Ryujin** » Sat Mar 05, 2011 8:03 am

Haha, KB. Seems like you and I have the same goal regarding game tracking. I happened to ask Zico last year about providing tracking info so DescentValhalla.com could display games for all Descent's in one place. I know he has been planning on it but we're dependent on them to get their end to produce the game info before we can take advantage of it on our sites. I just received a note from Zico that he is planning to start working on the tracker several months down the road.

I think Descent is going to have some snazzy modern features this time next year. I'd love to be able to browse DescentValhalla, DBB, whatever and see games going on in any version of Descent.

fliptw · Post by **fliptw** » Sat Mar 05, 2011 12:23 pm

If the protocol is sane, it would be trivial to implement. I used packet inspection to reverse engineer the protocol for quake 2/3(which was all text for information).

Its not like you can't interrogate servers for game information, A DB would be nice, not really required.

KoolBear · Post by **KoolBear** » Sat Mar 05, 2011 1:27 pm

Jeff250/fliptw/Sirus,
I am not a programmer so I am not sure what needs to be done. I am wide open to suggestions and maybe a team leader is needed that knows how to get it done, someone capable and willing to work with Diedel and Zico and Ryujin with his knowledge of D3 It would be nice if what was created could be used by any site so that it's not lost to the community. Diedel has already committed to implementing something in D2X-XL for integration here, and any site that wants to add it. The main thing is that a person should be able to see and launch D1 D2 or D3 from their browser.

Jeff,

Diedel has this webpage ...
http://www.descent2.de/d2x-tracker.html#basics But again he is committed in revamping the information shared for our purpose, I am sure if we had the bugs already worked out Zico could/would do the same or something very similar hopefully..

I attached this script because it is what is available to us right now other than Ryujin's Site for D3 I am not connected to it nor do I think it is the ticket, but it is a tool to start you gurus thinking.

Ryujin,
Exactly, the last time I came back before the to move to RC server I was trying to get it working, I had trackers setup on koolbear.com and one running on DescentBB.net but as usual life tackeld me and ended up getting another toe amputated a few years back

so what litte focus I could summon went into keeping my kids moving forward and had no time for my fun Descent..

On a different note for some reason I don't think I have been getting notices of the responses to this thread

fliptw · Post by **fliptw** » Sat Mar 05, 2011 2:04 pm

I havent' really be paying much attention to what D2X's have been doing, but some questions:

Zico or Diedle have a tracker running currently?
Does KB want a listing of current games, or does he want a tracker the DBB?

KoolBear · Post by **KoolBear** » Sat Mar 05, 2011 2:32 pm

fliptw wrote:I havent' really be paying much attention to what D2X's have been doing, but some questions:

Zico or Diedle have a tracker running currently?
Does KB want a listing of current games, or does he want a tracker the DBB?

Diedel does have an INGAME tracker currently running and functional, he interfaces with the perl script I show above. Zico is willing to change his in the near future.

It would be nice if the DBB could read the output from the tracker(s) and display it on the DBB so people could join a game that is in process IF that game is designated "open"

Jeff250 · Post by **Jeff250** » Sat Mar 05, 2011 3:50 pm

So probably what we want then is to not reinvent our own tracker but to just read in whichever Diedel etc. are already using? Do we know the hostname and port for Diedel's?

The protocol seems simple enough from the documentation, although the endianness of the IP addresses isn't clear--I would assume little-endian for consistency with the port? The tracker as-is leaves something to be desired though, since it seems to only track ip addresses and port, nothing about the game itself, such as name, who is in it, or what level is being played. I don't know how satisfying a list of IP addresses and ports would be for someone to look at when barfed out as HTML.

Still, the easiest way to implement this would be to just write some php to suck in Diedel's tracker on each request, assuming he doesn't mind the traffic. If we implemented it this way, we would want it to be on its own page though, since this implementation would increase the latency of any page we spliced the php directly into. If we wanted it visible on, say, the front page though, we could still use an iframe or AJAX.

I could hack up some mockup code for the above implementation in php when I get home this evening. If we wanted to implement our own tracker, then maybe this would still be the easiest way of doing it--we would just run the perl code as is and point the php code locally. But with this case, probably the more robust way would be to have the php talk to the tracker through a database though.

Sirius · Post by **Sirius** » Sat Mar 05, 2011 4:20 pm

That sounds right. The above Perl script is the tracker - wherever it might be hosted (and assuming nobody plans to replace it?) - the website plugin would basically just query the tracker (whether through a file the tracker writes to if it's on the same machine, or by acting as a client), and from there the servers that it references. The tracker is honestly the easy part - afterward you need to poll the servers using the D2X-XL and D1/2X-Rebirth (when available) formats.

akula65 · Post by **akula65** » Sat Mar 05, 2011 4:59 pm

I'll say up front that I am a single player-type player myself, but I was wondering if those of you contemplating a tracker have considered more general trackers that already exist. As an example, Ronald Mercer has been running and developing Qtracker for many years:

http://www.qtracker.com/

Descent 3 is listed as one of the supported games in a really long list:

http://www.qtracker.com/phpBB3/viewtopic.php?f=4&t=1265

and there is an SDK for adding additional games (like DXX-Rebirth or D2X-XL perhaps):

http://www.qtracker.com/developers.php

It apparently has a visibility utility that can be run on a server for games that don't have master servers:

http://www.qtracker.com/download-sectio ... ion=0,2,12

Perhaps there are other such general game tracking programs that are worth considering as well. Is it imperative that the DBB and other sites display the tracking info and act as an interface, or would a more modern and more general Kali-like substitute (like Qtracker) be more useful to players?

Qtracker does seem to have shortcomings. One example is that it is PC-centric. There is no version or port of the tracker for Mac or Linux systems, and people have tried and failed to get it to work in Wine:

http://www.qtracker.com/phpBB3/viewtopic.php?f=4&t=2447

KoolBear · Post by **KoolBear** » Sat Mar 05, 2011 6:05 pm

All,

I emailed the creator of the tracker above and here was his response, Jeff250 maybe it'll make sense to you.

[quote="Sebastian "VEX-Munk" Starosielec"]
Hi Michael,

Thanks for your interest in the tracker file.
From reading the file (don't recall much, it's been a long time since then), the tracker only saves IP+port of the servers. Of course you can modify the tracker programm to also include additional data. Make an additional hash $Data{$key}
which will store the additional data. The tracker does not have to know much about this data, it will store it as it is received from the server, and send it without modification or interpretation to the clients (= your website). The only information the tracker needs is the length of that data it has to store and recall - it helps a lot if this data is of fixed size (say 1kB, enough for 128 bytes for each of the 8 players).

Second comment is on compatibility: The protocol tokens used by the tracker are "R" and "S" (line 24 and 29). Leave those sections untouched, and for the data-included tracker use other single-byte tokens (i.e. "r" and "s"), expanding the if/elsif structure beyond line 51.
This way, the tracker will be fully backward compatible, old clients and servers can use the new tracker without modification.

Ah, and I see the tracker file still contains a lot of debug code (all those print lines). Those can be safely removed. They only produce a lot of garbage on the console when run...

For website integration, this is actually a quite different topic. The basic steps are: contacting the tracker and retrieving data, then interpreting the data based on the agreed data structures, and display to the website user. This would be a project for itself.

Unfortunatly I don't have much time helping you modifying and testing the tracker code. But please ask any questions about the code you have. It doesn't contain much of voodoo (in fact that was one of my first perl programs, so no fancy card tricks in there). The choice for perl was simply for the built-in hash capability.

Glad to see people are still interested in D2 though, I had to cut my Descent activity down with my job. Descent was always my motivation to try new things in programming, and by that learning a great deal of different techniques.

Good luck with your project,
Sebastian[/quote]

flip · Post by **flip** » Sat Mar 05, 2011 8:06 pm

There may be some possibilities with YANG. It's a free client like Kali that lauches UDP games. It has specific support for Rebirth already built in. Maybe you can just take the information directly from YANG.

EDIT: Forgot link: http://yang-online.com/

Jeff250 · Post by **Jeff250** » Sun Mar 06, 2011 1:19 am

KoolBear wrote:All,

I emailed the creator of the tracker above and here was his response, Jeff250 maybe it'll make sense to you.

He basically confirmed everything that we had already suspected and suggested how we should retain backwards compatibility if we introduce new data fields.

To add new data fields, we will need to talk to Diedel at some point about this so that his client will send this data to begin with and decide how. I guess the way his client gets this data was by contacting the hosts mentioned in the tracker themselves for the information? This seems like a bad idea for our php code though.

To query Diedel's tracker:

Code: Select all

$ echo -n 'R' | nc -q1 -u 87.106.88.224 9424

Everything seems to be big-endian, which seems to contradict Diedel's documentation.

I rewrote the tracker in python because I think perl is icky.

This should be equivalent:

Code: Select all

#!/usr/bin/env python

import sys
import time
from struct import pack
from socket import socket, inet_aton, AF_INET, SOCK_DGRAM

TIMEOUT = 30
PORT = 9424
MAX_SERVERS = 85

games = {}

def handle_server(s, addr):
    print str(addr)
    games[addr] = time.time()

def handle_client(s, addr):
    now = time.time()
    for server in [server for server, last_update in games.iteritems()
            if now - last_update > TIMEOUT]:
        del games[server]
    count = pack('B', min(len(games), MAX_SERVERS))
    body = ''.join(inet_aton(host) + pack('!H', port)
            for host, port in games.iterkeys())[:MAX_SERVERS]
    response = 'S' + count + body + '\0' * (len(body) - 510)
    s.sendto(response, addr)

def main():
    s = socket(AF_INET, SOCK_DGRAM)
    s.bind(('0.0.0.0', PORT))
    handlers = {'S': handle_server, 'R': handle_client}
    while True:
        msg, addr = s.recvfrom(1)
        handlers.get(msg, lambda s, addr: None)(s, addr)

if __name__ == '__main__':
    main()

Here's some initial php code for sucking in the tracker information:

Code: Select all

<?php

$HOST = '127.0.0.1';
$PORT = 9424;
$TIMEOUT = 10;
$MAX_SERVERS = 85;

$s = socket_create(AF_INET, SOCK_DGRAM, SOL_UDP) or
    die("<p>Error creating socket.</p>\n");
socket_connect($s, $HOST, $PORT) or
    die("<p>Error connecting to tracker.</p>\n");
socket_send($s, 'R', 1, 0) or
    die("<p>Error sending request to tracker.</p>\n");
socket_recv($s, $buf, 512, MSG_WAITALL) or
    die("<p>Error receiving response from tracker.</p>\n");
$buf[0] == 'S' or die("<p>Illegal response from tracker.</p>\n");
$unpacked = unpack('Ccount', $buf[1]);
$count = min($unpacked['count'], $MAX_SERVERS) or
    die("<p>No games.</p>\n");
$servers = str_split(substr($buf, 2), 6);
echo "<table>\n";
echo "<tr><th>Host</th><th>Port</th></tr>\n";
for ($i = 0; $i < $count; ++$i) {
    echo "<tr>";
    $server = $servers[$i];
    $unpacked = unpack('Nhost/nport', $server);
    $host = long2ip($unpacked['host']);
    $port = $unpacked['port'];
    echo "<td>$host</td><td>$port</td>";
    echo "</tr>\n";
}
echo "</table>\n";

Hmm, are there any phpBB plugins for syntax coloring? Although I think that php would still be just as fugly.

fliptw · Post by **fliptw** » Sun Mar 06, 2011 1:56 am

This will likely be running on a Windows VPS besides the DBB - don't make richard install anything besides whats there now.

IIRC, tho this is nearing 15 years ago, you could get more info on a host from inside d2...paaacket sniffffer?

karx11erx · Post by **karx11erx** » Sun Mar 06, 2011 3:48 am

Hi there,

Right now the D2X-XL tracker functions like telephone dial switch. Every host using the tracker sends a single byte message to it saying "I am a game host". The tracker stores the IP address of the message's sender in its table of game hosts. Clients (people looking for a host) send a single byte message to the tracker saying "Please send me your current table of hosts". Once the clients have received that table with IP addresses of available game hosts, they periodically send a "Give me your game info" message directly to the game hosts. This behavior is part of Descent's networking protocol (and hence not D2X-XL specific). Descent itself makes sure data endianness is handled properly (MSB machines convert data to be sent to LSB and received data to MSB).

If I remember correctly, the host table size is exactly 512 bytes, which is plenty given the actual amount of game hosts these days.

IP and port number endianness is not an issue - this is handled by the OS.

you can find all information about the D2X-XL tracker interface here: http://www.descent2.de/d2x-tracker.html

I can see several approaches to retrieving detailled game info via a tracker.

Approaches

The tracker is enhanced and will not only store the IP address of available game hosts, but also a well defined, relevant subset of their game parameters. As D2X-XL has introduced quite a bunch of new settings that are gameplay relevant. I would make a sound proposition of the required data structure design.
The tracker stays as it is, and once your service has the hosts table, it directly queries the hosts themselves for their game info.
1. Your service directly uses the present networking interface of Descent and has to properly interpret the game information it gets.
2. Your service uses the present networking interface of Descent, but add a new message type for this particular purpose will be added and a game info data structure that is as concise as possible while giving all relevant information will be designed for it.

Data format

As far as I can tell, your tracking service will have to be expanded, as D2X-XL offers a bunch of additional multiplayer relevant game settings (smoke, target indicators, etc. etc.)

There is already a useful data format your tracking service uses and which XL/R can support. Utilizing it would require moderate effort on your side for XL specific extensions.
Your service directly uses the Descent game info data format and XL's extensions to it. This would put all the burden on you.
A distinct game info data format will be defined that will require you to make greater changes in your tracking service or even branch to completely different handling of data formats.

Evaluation

(1.)
Advantage:

The easiest way to query the data for you

Disadvantage:

The tracker will require changes. Depending on the size of the game info, the number of game hosts returned in a single response packet of the tracker may be drastically reduced, requiring a more complicated protocol
XL/R will require changes (minor changes for XL, as it already knows how to interface with the tracker)

(2.1)
Advantage:

Neither Zico nor I would need to change the existing code (-> no inconsistencies due to different implementations, no need for coordination, particularly when the need of changes should arise)
The tracker wouldn't need any changes and either lose storage capacity or would need to use a more complicated protocol in case there is so much data that it needs to be sent in several packets

Disadvantage:

You would have to interpret the game data Descent sends, which is quite a bit.

(2.2)
Advantage:

The tracker wouldn't need any changes and thus neither lose storage capacity nor would need to use a more complicated protocol in case there is so much data that it needs to be sent in several packets
The game info data would be tailored to the actual requirement and hence be rather sleek and easy to handle

Disadvantage:

The existing XL/R code would need to be expanded (the changes would be moderate and easy to handle)
You would have to query all game hosts after having retrieved their addresses.

Conclusion

(1) While this may seem attractive at the first glance, it actually may not be. The problem is that right now a tracker response packet isn't all that big and you may not be able to store too many game host info records in it, requiring you to do extra data management and networking stuff, with all the problems of lost and out of sync UDP packets. A decision can only be made here after the final game info data design is available.

(2.1) Doesn't really make sense. The game info data Descent uses is too bloated for the purpose aimed at here, and probably a PITA to interpret for you.

(2.2) This solution looks quite attractive to me. It keeps the tracker communication of XL/R simple, and will minimize problems with packet loss, as you will only lose a single game host's information in case of networking problems, and not a big bunch as you would when a tracker response gets lost. Such a case could be easily detected using a timeout, and game info requests to such game hosts could be repeated.

The required game info data handling would be needed to be implemented in XL/R anyway, regardless of whether solution (1) or (2.2) are used.

Your tracking service might need an extra level of indirection by first having to query a tracker and then each game host the tracker returns, but this looks very straightforward to me.

Creating an HTML snippet that can be loaded into a tracker display page (e.g. using jQuery) wouldn't be a big thing. If need be, I can implement that. I would create some C/C++/C# code you could add to your tracking service (it is coded in C/C++, isn't it?)

fliptw · Post by **fliptw** » Sun Mar 06, 2011 8:24 am

Wouldn't it be more sane to use TCP packets for tracker related functions?

karx11erx · Post by **karx11erx** » Sun Mar 06, 2011 9:21 am

Imo given the current functionality UDP is the right protocol. UDP basically is intended for single messages and an optional confirmation. That's what the tracker does now. Losing one packet doesn't hurt, at least not if the tracker is polled in a regular, short interval (like 15 to 30 secs). Imo TCP would be total overkill here, as it provides a constant connection intended for streamed data transfer. Keeping a TCP connection open all the time would be impractical, and establishing it would be more effort than exchanging single UDP packets.

The same would be true if we chose approach (2.2). A game info record provided host by host would easily fit in a single UDP packet, even regarding conservative MTU values.

Krom · Post by **Krom** » Sun Mar 06, 2011 10:08 am

Aren't most game trackers UDP for that reason?

karx11erx · Post by **karx11erx** » Sun Mar 06, 2011 10:16 am

Maybe. UDP is simpler to implement, too. Sending is "fire and forget", receiving is "let's look whether something has arrived - if yes, nice - if no, never mind".

karx11erx · Post by **karx11erx** » Sun Mar 06, 2011 11:00 am

As I think of it, if we chose (2.2), the game hosts could send the game info as HTML snippet already. I would just need to know the format. This only makes sense if the format changes very rarely - I think this is the case.

I will start a list of required information here. Since I have to look into the details of multiplayer relevant D2X-XL specific settings, I will have to complete the list later on. If something is missing, please just create a new post, copy the list, and add the missing items at its bottom (we will do the sorting when the list is complete).

Game info items:

- host IP and Port (IP:Port) (1)
- host name (2)
- Mission name
- Game type (Coop, Anarchy, CTF, CTF+, Entropy, Monsterball, Darkness)
- Status (open, launching, game in progress)
- Current players
- Max. players
- D2X-XL specific info

(1) Can be omitted

zico · Post by **zico** » Sun Mar 06, 2011 12:41 pm

Actually I like the 2.2 idea - just because the tracker can stay as simple as it is (really love how "kiss" karx' tracker works).

But so the hosts need to send their game info so it can be parsed. My primary question would be "where" they send this info to? Directly to the DBB?
I'd personally vote they also send the info to the tracker (or at least where the tracker is located) and the tracker does transform the game info to a readable format and stores it into a HTML table everyone can parse.

My vote goes to this since then all hosts can send their data in the usual "lite info" format instead of a slightly bigger clear type format. Takes off load from the hosts.
And having all this info gathered on the trackers gives other parties (clans, etc) the opportunity to do the same thing we do here now without maybe modificating the actual games.
(Also if the table format needs to change, only the tracker would need an update - not the games).

Hope I could explain correctly what I have in mind here...

karx11erx · Post by **karx11erx** » Sun Mar 06, 2011 1:21 pm

zico,

your post doesn't really take into account how the tracker and the Descent game info communication works. Also proposing to have the tracker store additional information is contradictory to saying (2.2) sounds like a good idea. The hosts will also not suffer from dramatically increased network load due to game info requests.

(2.2) would actually use a behavior very similar to the way clients query a game host for game info already. My proposition is to add another message id and response data format for tracker game info requests. Since these requests may ideally come every 60 seconds (or less frequently), this would cause as good as no extra load on a game host. During a typical multiplayer sessions, dozens of packets are sent to each participant every second. Having to process an extra request every 60 sec (or even every 30 secs) would have next to no impact.

If you look at the game info currently sent to clients, you will quickly realize that you can easily derive a subset of information a tracker would need there.

The lite game info response is not suitable for our purposes here: On the one side it contains unnecessary information (speaking off the top of my head), on the other side it doesn't comprise D2X-XL specific data (which Rebirth would have to report as well, just everything being signalled as disabled).

Now to the actual procedure:

1. TDS (Tracker display service, DBB/Ryujin) will query tracker(s) for available game hosts.
2. TDS will send a "tracker game info" request to each game host in the list(s) returned by the tracker(s).
3. Each game host will send a "tracker game info" response back to the address it had received the related request.

That is very simple and straightforward, and I could even point out the code areas where this would have to be handled by Still ... cough

, Rebirth. Doing it this way would allow 3rd parties to use the same scheme just as well.

Ryujin has already told me that this is the way he queries D3 game info, which adds another good reason to handle it this way.

Ideally the game hosts will send the tracker game info in HTML format that can be injected into the tracker display using Ajax (e.g. jQuery).

zico · Post by **zico** » Sun Mar 06, 2011 2:42 pm

Okay I guess I either spoke explained it wrong or I oversee something:

The current way how the tracker works - just exchanging IPs I think is fine as it is.

My suggestion would only be to let hosts send their actual game info to the track "as well". Not to exchange this with other clients - only so the tracker can create a human readable table everyone can parse.
This info can also sent even less frequent - only if the actual game info would change (player amount changes, game status changes).
Again, not speaking of Clients querying this info. And no, it's not exactly like 2.2 - that's right. As far as I understand how you explained 2.2 it would reverse the way how game info is delivered: not the tracker asking for the game info but the hosts delivering new/updated game info if it's actually needed. (effectively eliminating the second disadvantage your described for 2.2)

karx11erx · Post by **karx11erx** » Sun Mar 06, 2011 3:03 pm

If a host only sends an update when needed the service wouldn't work right.

The tracker uses a timeout at which game hosts are deleted from the table. That means as long as game host is available it has to constantly announce its availability to the tracker by sending the "I am here" message. Otherwise it would disappear from the tracker's host list (e.g. when a game is in progress), and players coming late wouldn't be able to join as they wouldn't see the game host anymore.

Same goes true for the TDS.

In our case, a system where a client repeats to query for game info in a reasonable time interval looks like the most robust system. As I said before, the increased network load argument isn't really one here. I also did not propose to have the game hosts autonomously send game info to the TDS (how would they know of its presence and address in the first place?). I wrote that the TDS would ask the game hosts for game info, just as players wishing to join an open game do, and the game hosts would respond with that info (I am writing this the third time now; please make sure you really have not been able to clarify questions you have by diligently reading the above posts).

TDS --[request host info]--> Tracker --[respond with host info]--> TDS --[request game info from each host]--> Hosts --[respond with game info]--> TDS

zico · Post by **zico** » Sun Mar 06, 2011 3:09 pm

Of course for timeouts the tracker would need to "connect" the IPs to the game info. A Host STILL frequently sends the regular "I am here" packets. But the game info it only sends if it's needed. The tracker should be able to associate the "I am here"-packets with the game info - by comparing the IPs.
If there are no regular "I am here" signals, the game info is dropped from the table as well.

But of course you are right - querying the hosts should not be a big thing either...
And additionally I think if we have a common format for the table, it does not matter in which way it works - just how the table is formatted...

karx11erx · Post by **karx11erx** » Sun Mar 06, 2011 3:14 pm

I am for keeping things as simple as possible, therefore

- don't change the tracker as it need not be
- have the TDS query the game hosts it retrieved from any trackers, because that's the way it works right now for D3
- have the game hosts send a game info packet whenever they're asked for, because that requires the least extra code and programmed intelligence, and adds negligable network load

To be dead sure the TDS doesn't shut a game host down by constant game info requests, you can use a timer and only send a response every X seconds (X >= 30)

zico · Post by **zico** » Sun Mar 06, 2011 3:27 pm

That is now the first point I was asking for in my first post here.
1) IF the tracker handles the game info, formats it you gain flexbility: Other services do query the tracker and get some readable contents - independent on how the actual game works. If your game changes the way how game info is handled internally, only the tracker will need an update, not all services querying the game info.

2) Dozends of other TDS can query game infos - never overstressing the hosts (which might be done by intention to kill games). Of course you can block too frequent requests. But being requested too frequently is not even necessary if the game info is stored on the tracker already.

So yeah I am proposing flexbility and decentralization. IF there is a problem with that (besides it's just something DIFFERENT) that's fine. It's just that I think it's very sub-optimal to let services query thousands (theoretically) games. Having readable and uncomplicated game info on the tracker I think is something everyone could benefit from - not just speaking about the dramatically reduced traffic for the TDS (reading one simple formatted file vs. interpreting game-specific data from possibly thousands of games).

karx11erx · Post by **karx11erx** » Sun Mar 06, 2011 3:49 pm

If the game info format changes it will inevitably affect all services querying it, be it from the tracker or directly. Or how would you want to remove the changes once the info crosses the tracker?

If on the other side you chose a format that can be easily expanded w/o changing everything, TDS not knowing of any changes can still extract the information they understand.

That's why a system independent format needs to be chosen, e.g. HTML or XML. This would even allow game hosts to add info specific to them while others who do not offer certain features simply omit it.

So the way to stay flexible is to use a flexible format, not pass everything through a tracker.

Finally, we should stay down to earth. We are not coding a tracker service for the Duke Nukem Forever. We will never have thousands of game hosts and dozens of TDS. It makes absolutely no sense to design a system with otherworldly requirements in mind here. The system to be created here has certain preconditions (small userbase, few TDS, rather static game info). Making wild assumptions that will never come true is not a good argument.

zico · Post by **zico** » Sun Mar 06, 2011 3:58 pm

One example where you can make changes a tracker can handle is if you add a new game mode. The tracker can make a text string out of the bit or whatever you might use internally to specify the mode. The TDS only needs to read the text string.
Of course not ALL changes can be covered by the tracker, but it's significant enough in my eyes.

Besides this I feel this becomes less important compared to the fact that every TDS should query all possible hosts... I cannot see any reason for that. If there is any argument FOR this way I'd be happy to hear it.

Jeff250 · Post by **Jeff250** » Sun Mar 06, 2011 3:59 pm

karx11erx wrote:IP and port number endianness is not an issue - this is handled by the OS.

you can find all information about the D2X-XL tracker interface here: http://www.descent2.de/d2x-tracker.html

What do you mean by the OS will handle it? Are you talking about network byte order? Your documentation seems incorrect--it says that the port is little-endian, whereas in practice both the IP address and port are sent big-endian.

karx11erx wrote:it is coded in C/C++, isn't it?

No.

karx11erx wrote:(2.1) Doesn't really make sense. The game info data Descent uses is too bloated for the purpose aimed at here, and probably a PITA to interpret for you.

Agreed.

karx11erx wrote:(1) While this may seem attractive at the first glance, it actually may not be. The problem is that right now a tracker response packet isn't all that big and you may not be able to store too many game host info records in it, requiring you to do extra data management and networking stuff, with all the problems of lost and out of sync UDP packets. A decision can only be made here after the final game info data design is available.

karx11erx wrote:(2.2) This solution looks quite attractive to me. It keeps the tracker communication of XL/R simple, and will minimize problems with packet loss, as you will only lose a single game host's information in case of networking problems, and not a big bunch as you would when a tracker response gets lost. Such a case could be easily detected using a timeout, and game info requests to such game hosts could be repeated.

We can handle out of order UDP packets easily enough by introducing an incrementing sequence id. We'd only be half-reinventing TCP.

But even more easily, we could also just assume that packets arrive in order, since we only send them out what... every 20 seconds? If packets sent out every 20 seconds are arriving out of order, then we have bigger problems on our hands...

Does anyone know how modern games do trackers? Games that don't inherit mid-90's IPX baggage?

I find 1) the most attractive at the moment. I don't know why everyone is so hesitant to modify the tracker. It's what.. only 30 lines of python or perl? It's probably the easiest thing to modify at the moment.

Since the (# of Web sites intending to output tracker information) > (# of trackers), it makes sense to me to centralize the game information in the tracker itself instead of requiring Web sites to reinvent a secondary tracker to track something that the primary tracker could easily track itself.

zico · Post by **zico** » Sun Mar 06, 2011 4:11 pm

Since the (# of Web sites intending to output tracker information) > (# of trackers), it makes sense to me to centralize the game information in the tracker itself instead of requiring Web sites to reinvent a secondary tracker to track something that the primary tracker could easily track itself.

Amen. Actually this sounds as it pokes around in my head.

karx11erx · Post by **karx11erx** » Sun Mar 06, 2011 4:30 pm

Jeff,

As far as IP addresses go: You will not have to convert them. For port numbers use the htons macro/function.

You guys are all trying to make the issue more complicated than it is. Actually I had written my above evaluation to clarify a lot of things before a lengthy discussion would start about it.

I wonder who will implement this. My current impression is it will be Ryujin, and he has made his preference pretty clear, and they are congruent with mine. Neither he nor me is going to make this any more complicated, and the simplest way to go is to leave the tracker as it is, don't add a second, different tracker handling to the TDS, and keep the changes to Descent as simple as possible. Btw, if there'd be changes to the game info, and the tracker would have to handle it, you would have to change the tracker as well as all TDS using it, and everybody running the tracker would have to update it. The way I'd prefer, only the Descent versions requiring a change would expand the game info data, and the TDS could be upgraded to be able the new data, or simply ignore it.

Jeff250 · Post by **Jeff250** » Sun Mar 06, 2011 5:21 pm

Does anyone know if Ryujin knows about this thread? I feel like that if he's involved, then he should have a chance to respond.

karx11erx wrote:As far as IP addresses go: You will not have to convert them. For port numbers use the htons macro/function.

I already know that you're sending it in network byte order (big-endian), but your documentation says you're sending it little-endian.

karx11erx wrote:Saying the tracker would have to be reinvented for each website providing a TDS is just nonsense and has nothing to do with my propositions.

You mean this?

karx11erx wrote:Ideally the game hosts will send the tracker game info in HTML format that can be injected into the tracker display using Ajax (e.g. jQuery).

You can't do this with AJAX, since it isn't cross-domain. An iframe solution isn't satisfying either. For one, we would need a separate one for each game. Since javascript isn't cross-domain, Web sites wouldn't be able to control the presentation. And the game shouldn't be in charge of the presentation. I'm also not comfortable with a player being able to inject arbitrary HTML into an iframe on the DBB.net. This is a huge security problem. Probably not a XSS risk, but, for instance, a player could perform a DOS attack on users' browsers by inserting javascript that's caught in an infinite loop.

karx11erx · Post by **karx11erx** » Sun Mar 06, 2011 5:31 pm

Then my documentation is wrong. So what? My tracker works.

Write the HTML snippet to a folder on the server and load them all with Ajax. Nothing cross domain here. I don't know how the TDS produces its output, but it might also put the game info HTML directly into the HTML it produces, if it produces any.

You have not been able to convince me. I think I will just wield my 30 years of software development and engineering experience and 7 years of D2X-XL coding superpower, put the "I designed the tracker" wildcard on top of it and do what I think is best here.

Ryujin has been informed about this thread by PM a while ago already.

Jeff250 · Post by **Jeff250** » Sun Mar 06, 2011 5:54 pm

karx11erx wrote:Then my documentation is wrong. So what? My tracker works.

Could you fix it?

karx11erx wrote:Write the HTML snippet to a folder on the server and load them all with Ajax.

Are you proxying each request for game information through the Web server? OK, so technically this wouldn't be reinventing a tracker since you aren't maintaining state, but it's dangerously close. After all, the first enhancement anyone would want to make is to cache requests... basically, reinventing a tracker.

In any case, you're bleeding responsibilities. The DBB.net shouldn't have to know how to contact players for basic game meta-information. That's a tracker's job. And Descent shouldn't know what HTML is. That's the Web server's job.

Jeff250 · Post by **Jeff250** » Sun Mar 06, 2011 6:26 pm

And if we inject arbitrary HTML directly into the DBB.net, then there *is* a XSS vulnerability. A malicious player could e.g. steal users' session cookies. At least use something like JSON that we can trivially parse and sanitize and control the presentation of...

Sirius · Post by **Sirius** » Sun Mar 06, 2011 7:58 pm

Regarding the possibility of using bogus TDSes to DoS-attack game servers, you can do this already by emulating a client, fetching the IP of the server you want from the tracker, and then spamming it to oblivion. Bringing in an additional kind of service will do nothing to increase the prospects of such a kind of attack.

The only thing that arguably could reduce it is if you move all querying functionality to be handled between the clients and the tracker (i.e. game hosts report ALL pertinent information to the tracker, which stores ALL of it, and reports everything directly to the client without the client having to poll game servers in turn). That obviously requires a rewrite not only of the existing tracker but the game client/server code as well. (Not just for D2X-XL but also for Rebirth, as it will most likely currently assume game info is passed directly from host to client as well. The D2X-XL tracker is the minimal possible change to that process.)

P.S. - That still doesn't remove the possibility of DoS attacks because to be able to join a server, at some point a client has to be able to fetch its IP from the tracker. Once you have the IP, that's all you need; spamming it with info requests won't do the same thing but you can spam it with something else, or even just random data.

Of course there is a temptation to take on a "Rolls-Royce" approach and make an ultra-modern tracker system, but I would suggest you also consider what the community really needs before committing to something that, after all, somebody has to do some actual work on at some point.

Jeff250 · Post by **Jeff250** » Sun Mar 06, 2011 10:57 pm

Ah, but that wasn't the kind of DOS attack I was concerned with. You're right--there's not much you can do about the kind you're describing. By DOS attack, I meant the player being able to insert arbitrary HTML into an iframe. For instance, if the player inserted javascript that gets caught in an infinite loop, this can freeze certain browsers, basically denying the ability for users to use their browsers. On others, it can at least make your lap very warm. But if you can inject arbitrary javascript, with enough creativity, you can bring any browser down to a screeching halt one way or another.

Of course, now that the situation has regressed to inserting arbitrary HTML directly into the page, not a cross-domain iframe, we have XSS issues to deal with, like someone injecting javascript to steal DBB session cookies to hijack users' accounts. Not to mention people just inserting goatse or some such.

I doubt that I will work on any code in this direction due to the smell, but my suggestion to any future implementers is to agree on d2x-xl emitting some subset of HTML that can be parsed with an XML parser. Whatever you do, do NOT inject untrusted HTML directly into your page. But this whole thing is ridiculous, because Descent isn't a Web server and HTML isn't a data interchange format. There's a reason why this seems more problematic than it needs to be.

fliptw · Post by **fliptw** » Sun Mar 06, 2011 11:58 pm

or, instead of markup: key-value pairs in plaintext...