The DBB needs your help ....

[Jeff250]

Any data format that is parsable would be an improvement. Even an XML-compliant subset of HTML would theoretically work, but you still have to ask yourself, is a document markup language really the best way to exchange data...

[karx11erx]

I am not interested in arguing with people who have no real part in this but only try to put themselves in the limelight with arguments that are none.

"DoS" attacks on Descent trackers. Sure, Descent is such an attractive target - it's user count is huuuuge, isn't it? We will also never have dozens of TDS - I think more than two or three would already be a gross exaggeration.

Sending preformatted game info isn't hard, it would enforce a unified TDS layout, and the TDS would require next to coding effort to display that stuff.

So lets keep this reasonable.

I will make a clear decision now:

I will not have the D2X-XL tracker be messed around with. It will not start to store additional game info. I cannot hinder you to change the perl script, as I didn't create it, but I will not support that.

I will only support (2.2): TDS queries the trackers, and then the game hosts returned one by one. The game hosts will reply with game info in some format. I would prefer XML or HTML, since this means that additional information that some Descent version adds can simply be ignored, but other options are possible.

All this stuff about DoS attacks, and dozens and dozens of TDS querying hundreds and thousands of trackers is so much off the wall that I don't know whether I should laugh or cry about it. Game hosts can prevent being locked down by game info queries by only replying every 30 secs or so (I wrote that already).

I am tired by this discussion now. Too many futile arguments.

[KoolBear]

This has a been a great exchange of ideas!

Thank you everyone for your participation.

I think it time I select a leader to head this project. We needed your input and we have it. Give us a day or so to get a team leader to work the details with Diedel and Zico.

Kindest Regards,
Mike

[Jeff250]

I am not interested in arguing with people who have no real part in this but only try to put themselves in the limelight with arguments that are none.

At the very least, we still need to discuss a protocol for dxx-rebirth. I assume that we haven't scared zico off.

"DoS" attacks on Descent trackers. Sure, Descent is such an attractive target - it's user count is huuuuge, isn't it? We will also never have dozens of TDS - I think more than two or three would already be a gross exaggeration.

Read my post again. You can DOS a user's browser or steal their session cookies via XSS. This affects everyone who posts on the DBB.

Sending preformatted game info isn't hard, it would enforce a unified TDS layout, and the TDS would require next to coding effort to display that stuff.

Aside from the above security concern, another problem with html is that (being unparsable in the general case) it precludes anyone from using this in non-HTML documents. For instance, someone might want to track games in an RSS feed. I also don't see unified presentation as a feature. It's a bug. Web sites want to have customize how things are presented. Why should the player sending the data control how it's presented?

-----

Aside: when people talk about DOS attacks, you shouldn't just think about overwhelming someone with network traffic. It's any attack intended to deny someone from a service. For instance, not all buffer overflow vulnerabilities can be used for remote code execution. But you could possibly still use them for DOS attacks. One way is to overwrite addresses outside the buffer to point to unpaged memory, causing the service to crash when it attempts to read/write/jump to them. In the case we're concerned with with untrusted html, it could be javascript caught in an infinite loop, freezing someone's browser. But obviously session stealing is an even more grave concern.

[karx11erx]

Jeff,

you are constantly talking from theory just to show yourself off. All your arguments fail to see the reality of Descent being totally unimportant and unattractive for any attacks. You cannot even tell whether it would be hard to parse HTML game info because there is no HTML game info design around right now.

I want to achieve something, and not talk talk talk and prove how great I am.

You are giving me the dire impression that you want to take this over and try to exclude me from this and have your own solution with Zico.

To make this clear: D2X-XL has never needed websites tracking it. It has its tracker service, and everybody who wants to see what games are going on only needs to fire up D2X-XL and launch the netgame browser, where he will have all the info he needs.

[Jeff250]

I think this was added in an edit?

The game hosts will reply with game info in some format. I would prefer XML or HTML, since this means that additional information that some Descent version adds can simply be ignored, but other options are possible.

XML is fine. Whatever we choose, it just needs to be parsable.

I will only support (2.2): TDS queries the trackers, and then the game hosts returned one by one.

I really don't understand the disadvantages of 1). Modifying the tracker is trivial, and we can follow the author's advice to maintain backwards compatibility if we care about that. We have to modify the d2x variants, but we would have to modify them anyways. I also don't see the difficulty in using variable length packets. I mean, we're already talking about using xml.

Why not have the tracker track more meta-information? It already has to communicate with the clients anyways.

New stuff...

you are constantly talking from theory just to show yourself off. All your arguments fail to see the reality of Descent being totally unimportant and unattractive for any attacks.

I can recall at least one instance of a DBB account being hijacked in the past. I don't know if the mechanism was XSS, but that doesn't really matter.

[karx11erx]

Oh, really, one hijacked account in ... how many years? One? Not one thousand or so?

/me bashes head against wall

I am trying to do Koolbear a favor because he has asked me for it and because I appreciate him very much, but I am slowly getting turned off by your attitude here.

To make this clear: I will have the "project lead" and final say here, or it will not happen with D2X-XL. I don't need a wiseguy student with no real industry experience trying to prove how much smarter than me he is here, and that's how you are coming across, sorry for the open words.

[Jeff250]

I'm going to wait until Koolbear assigns a delegate before trying to discuss this any further. People can PM me of course.

[KoolBear]

Thank you gentlemen this is number one one my list! Well list of things to do on the Internet and for the Descent Community!


On another note ... Ryujin and I have made some progress on the D3 side this weekend

Mike

[karx11erx]

Subject: The DBB needs your help ....

I'm going to wait until Koolbear assigns a delegate before trying to discuss this any further. People can PM me of course.

Jeff,

if you are going to implement this on KB's side, we'll have to talk with each other. Sorry for being a bit impatient in this matter, but there's a lot of pressure in my life right now, and I don't have much time for this.

Just one request: Try to keep things as simple as possible, and don't pull out the big DoS attack club and "dozens of TDS are querying thousands of servers" stuff.

The current situation is:

- attacks are extremely unlikely because Descent is a microscopically small part of the inet
- There are very few game hosts and TDS

I am still not convinced of having the tracker handle the game info. Functionally the game info belongs to the game host. With my proposition, everything would always fit into one UDP frame. With the others, you'd need to start and mess around with a more complicated communication protocol.

If you need help with a two phased game info query, I could support you there.

Before deciding whether HTML game info is complicated to handle, we first need to design it. For that, we need a list of the required informations (see above).

[Jeff250]

if you are going to implement this on KB's side, we'll have to talk with each other. Sorry for being a bit impatient in this matter, but there's a lot of pressure in my life right now, and I don't have much time for this.

I don't think anyone has officially decided if I would do it yet. I like writing code, and this seems like a neat project, so I jumped earlier at writing some mockup code. (For any future implementers who aren't me and anyone else, any code I post in this thread is in the public domain.) I understand your urgency, but let's have a fast discussion instead of no discussion. Whatever we implement we will probably be stuck maintaining for a while. And we can't really predict who else might want to use this in the future, so we want it to be nice for them too.

Just one request: Try to keep things as simple as possible, and don't pull out the big DoS attack club and "dozens of TDS are querying thousands of servers" stuff.

In my defense, I never actually talked about those. Sirius did, but it was just because he thought I was talking about them. I mentioned DOS attacks, and since most people mean what you quoted above when they mention DOS attacks, there was a miscommunication, and he thought that I was talking about them. I am not concerned about those kinds of DOS attacks. I think we can write this off as a series of miscommunications.

The current situation is:

- attacks are extremely unlikely because Descent is a microscopically small part of the inet
- There are very few game hosts and TDS

For XSS, the attack base would actually be the DBB surfers, so our attractiveness for attacking would probably be more how many people are surfing the DBB. With even one malicious player on the tracker, they could attack anyone who views that game on the tracker on the DBB.

I don't know if I'm qualified to answer the question of how popular a Website needs to be before worrying about security problems becomes practical. On one hand, for years before we upgraded to phpbb3, we ran on security-bug-ridden code without too much happening. On the other hand, we did experience at least one account hijacking, although I don't think anyone identified the mechanism. IIRC, it seemed to be from someone inside the community too, since using the hijacked account, the attacker made a post that seemed personal in nature. So unfortunately I'd be more worried about inside attacks than attracting attention from someone outside. Maybe even people reading this discussion.

And maybe attacks is the wrong way of looking at it too. There could be simple pranks where someone sends a goatse img tag through to us. That sort of thing would probably be more common.

You mentioned XML as a possible alternative to HTML, and that sounds like a good alternative. Maybe with the XSS vulnerability no one would ever exploit it, but it seems like as long as we can circumvent this security problem, why not! The nice thing about XML is that almost every language comes with robust built-in libraries for parsing it, so no one would have to write their own parsers. For instance, in php, we could easily suck it in and then output our own HTML. We could probably write an ad hoc parser for the HTML if you restricted yourself to only using a well-defined subset (or re-use an XML parser if the HTML is XML-conforming), but in my experience ad hoc parsers for non-trivial languages tend to have subtle bugs. With XML, we have full control over the markup language, i.e., we can make up our own tags to delineate data, but with HTML, we would have to deal with whatever is built into HTML for labeling data, like maybe using the class attribute.

I don't think XML would be any more difficult to implement than HTML in D2x. Web-server-side, most languages have some easy API's to e.g. parse the XML into an "element tree" data structure, so with one line of code, you would have immediate access to all data. We would have to write code to then output our own HTML code, but the way that I see it, we *get* to output our own HTML code.

Aside from the technical concerns, even if someone wrote their own ad hoc parsers or re-used an XML parser to parse generated HTML, I feel like by using HTML we're boxing this API into only being used by Web servers serving HTML. We can't predict who might be interested in using this API for what reasons in the future, and it might not have anything to do with HTML. For example, someone might want a D2x RSS feed. I mean, the more I think about it, I might even be interested in writing one.

I am still not convinced of having the tracker handle the game info. Functionally the game info belongs to the game host. With my proposition, everything would always fit into one UDP frame. With the others, you'd need to start and mess around with a more complicated communication protocol.

I'll admit that this is the issue that I feel least strongly about, but I still think that we should have the tracker track basic game info.

How much information would we need to track? I think we would only need to track a few things, like maybe game name, mission name, level, and # of players? Do these strings have fixed size limits in Descent? I'd argue that if people want more information than this for any particular game, then they can still interrogate that host, but for purposes of just listing any games being played on a Website/RSS feed/etc., then we probably wouldn't need much more than this.

At some point, I should sleep tonight, so I probably won't be able to respond to your thoughts until the evening U.S. time.

[karx11erx]

XML looks good to me.

Why not have the tracker keep game info?

D2X-XL has a lot of additional game info. I don't want to blow the tracker data up unnecessarily, and I don't want to have D2X-XL need to get a ton of info from a tracker it doesn't need everytime it queries it for game hosts. I also do not want having to handle multiple UDP packets in a tracker response, nor do I want to switch to TCP.

If we follow (2.2), all info would fit into single UDP frames, and we'd have one UDP packet per query.

I also do not see a lot of different TDS spring up, all with their specific requirements to what data they need in which format.

[Ryujin]

Wow! This conversation BLEW up in here while I was out having some semblance of a life.

I personally believe in K.I.S.S. and even more so when it comes to code. I think we could discuss implementation for days. I think we all know what the requirements are and I believe I saw key decisions made on Diedel's part. Great! I say let's run with those decisions best we can. It's his call on providing tracking information for us to consume. Now, as for my part in this, I've wanted since the beginning to be able to provide useful and fast game trackers for all versions of Descent on DescentValhalla.com. That is my goal. I am willing to attempt to consume the tracking information in whatever fashion it's given to me. I am not familiar with the programming side of the network stack and UDP queries from .NET but I will certainly give it a shot and I hope that those in the know can help with some basic sample code (as Diedel already mentioned he might be able to provide). However, if it's a matter of generating XML and getting an XML response, then it's cakework for me. If it's HTML, I'm sure I could do that too. Just let us know what the format of the request and response is and I'll generate the request and pull the response info and generate a webpage.

This is an old game that few play anymore. I don't think we should spend anymore time on this than necessary. Let's just get up the essentials and go from there. Diedel and eventually zico, let us know when you have some code samples and/or the request/response info worked out for us to consume.

[fliptw]

Any markup language is overkill for this.

it doesn't need to be any more complicated than this:

Code: Select all

key:a,b,x;

[karx11erx]

Ryujin,

the question whether to have the tracker handle the game info or not is more important to me than the format in which to send the game info. Your saying my proposition (2.2) is how you are handling D3 game info right now confirms my wish/decision to chose that path.

To quickly get an idea how to use UPD, download and examine SDL_net and/or D2X-XL's network/udp_interface.cpp. It is pretty straight forward.

Regarding the game info message format I am open to propositions.

[Jeff250]

Sneaking in a quick post...

Any markup language is overkill for this.

it doesn't need to be any more complicated than this:

Code: Select all

key:a,b,x;

Probably, but then we need to come up with rules for how to handle e.g. semicolons in the data, etc. I don't know if anyone feels like discussing that at this point, but I wouldn't be against it. But at least everyone already knows what XML is and has functions to parse it.

D2X-XL has a lot of additional game info. I don't want to blow the tracker data up unnecessarily, and I don't want to have D2X-XL need to get a ton of info from a tracker it doesn't need everytime it queries it for game hosts. I also do not want having to handle multiple UDP packets in a tracker response, nor do I want to switch to TCP.

If we wanted to retain backwards compatibility (would we?), we would probably have to create a new client packet request for the tracker anyways, so we could keep the one that just returns IP address/port pairs in one fragment, and add a new one that returns the list with some basic game information. So you could pick if you wanted the extra information or not from the tracker.

My proposal isn't that the tracker track *all* game information. I agree that that would quickly get out of hand. But what if it tracks just a few things, like game title, mission title, level, and # of players. Couldn't that fit in a UDP fragment sent to the tracker every 20 seconds or so? Basically, just track the basic information that anyone would ever want to display in a list. If people still want more, then they can interrogate the hosts,

[Jeff250]

Thinking about this more... I still don't feel 100% about 2.2), but I feel a lot better than I did before. If you want to go with it, then that seems like a fair compromise.

[Ryujin]

Ryujin,

the question whether to have the tracker handle the game info or not is more important to me than the format in which to send the game info. Your saying my proposition (2.2) is how you are handling D3 game info right now confirms my wish/decision to chose that path.

To quickly get an idea how to use UPD, download and examine SDL_net and/or D2X-XL's network/udp_interface.cpp. It is pretty straight forward.

Regarding the game info message format I am open to propositions.

Cool, I'll leave that decision up to you. I don't care if I get the info from the tracker, the game, or a combination of both. Just let me know how I can do it and I'll do it. I trust you'll dictate the best approach for us that will be hopefully easy enough to consume and won't negatively affect the performance of the games themselves.

Thanks a lot for your help.

[KoolBear]

Ryujin,

the question whether to have the tracker handle the game info or not is more important to me than the format in which to send the game info. Your saying my proposition (2.2) is how you are handling D3 game info right now confirms my wish/decision to chose that path.

To quickly get an idea how to use UPD, download and examine SDL_net and/or D2X-XL's network/udp_interface.cpp. It is pretty straight forward.

Regarding the game info message format I am open to propositions.

Cool, I'll leave that decision up to you. I don't care if I get the info from the tracker, the game, or a combination of both. Just let me know how I can do it and I'll do it. I trust you'll dictate the best approach for us that will be hopefully easy enough to consume and won't negatively affect the performance of the games themselves.

Thanks a lot for your help.

Perfectly and Elegantly stated.

[karx11erx]

Performance of the games isn't an issue here. You could however tell me what format you'd like to have the info in. I could e.g. send an HTML snippet with every information in a table with a single row and several columns (one column per property). You could easily insert that into some HTML formatting statements of your own.

But that's just one idea.

[Jeff250]

Ack, didn't we already settle this?

Unfortunately, I can't get behind this idea of data format equivocation that people seem to be espousing here. Some data formats are secure; others are not. Some are parsable; others are not. Having a secure data format is important. Having a data format with robust, ubiquitous libraries who all parse the format consistently is important. XML/JSON/CSV/any-actual-data-interchange-format fit all of these criteria. Preformatted HTML fits none of these critieria; it was never designed as a data interchange format. You can choose bad data formats, and it does make a difference. So while we are deciding which to choose, let's make it a good one!

[Ryujin]

My first choice would be XML and second choice CSV as I'm familiar with both. I like XML because it's common and as Jeff mentioned most languages have parsers built in for it. Although it does have some overhead, I think it's negligible in the case of Descent I and II tracking needs. Other query tools out there output in XML as well.

But again, your call.

[Jeff250]

XML would be my first preference too.

[KoolBear]

XML would be my first preference too.

XML would be mine too this works well with RSS feeds to I assume, which the DBB now has BTW.

The whole goal of this exercise for me is to ASSURE that the community has the tools in place to make it easy for old and new players to get set up and in a game. And that whatever we do remains in the public domain, it would be nice to have an exclusive but we need these tools to be in multiple places and should be interlinked when possible so if one of the tracking sites is down people will know where the alternate sites are.

Ok,my thoughts as shallow as they are...

Assuming Diedel get's back online this evening do we stand in agreement?

Diedel's Option 2.2 in XML format? Which is similar and data and format to what D3 currently provides(?)

[fliptw]

XML is great when the context needs to be transmitted, but that isn't needed for a game tracker.

we might as well go full SOAP...

[Jeff250]

I get your point, but I feel like I'm already at my wit's end trying to convince people why preformatted HTML is a bad idea. I'm content with XML, unless everyone else wants to spend more time discussing this.

[karx11erx]

Preformatted HTML was just one proposition to make a start.

[fliptw]

The method that is the least verbose would be the most beneficial. all the tracker cares about is tracking games; it cares not for how it looks.

[karx11erx]

The method that is the least verbose would be the most beneficial. all the tracker cares about is tracking games; it cares not for how it looks.

Are you going to code it?

[fliptw]

I can code it, jeff can code it, Ryujin can code it. Given a good specification, any competent programmer can code it.

[Sirius]

The point of XML is that it's easy to parse (which for these purposes is probably more important than being concise). That makes it a useful mechanism for extending the system in future in case, e.g., you want Ryujin's tracker to display additional data while remaining backward-compatible, and don't want to force yourself into contortions to do so.

It's really not a bad idea at all for custom data.

[Lo]

Hi there,

I think it is a great thing what you are doing - one tracker for all kinds of Descent - cool!

Sorry if I have over-read it by now, but: Would you mind keeping it extensible in a way that future Descent clones might also announce their games on this tracker? Also, once it is finished, could someone please be so kind to write down a little specification (i.e. what byte sequences need to be sent and when)?

Greetings,
Lo

[karx11erx]

I can code it, jeff can code it, Ryujin can code it. Given a good specification, any competent programmer can code it.

The question was "will you code it", not "can you code it".

There is no point in discussing this with a half a dozen people if the actual job is done by one or two.

Sirius has a point, and that's why I would chose XML to.

So I am gonna make a decision now: Game info will be submitted in XML format (I will deliver a specification later this week, or maybe the weekend). Ryujin wants it, Jeff wants it, I want it, Sirius thinks it is a good a idea. That's enough votes, I believe.

[Ryujin]

Great news! We're on our way.

[fliptw]

Next point:

SQL for the backing store? If its going to he hosted alongside the DBB-which is load balanced-would make set up and maintenance there a bit easier in the long run.

[Sirius]

This for the DescentBB's tracker display system, or a tracker itself? I guess if the DBB were to run a tracker, and you wanted to show the contents on the DBB, you'd have access to a SQL database from the tracker - and actually that's probably easier to work with (especially if you have load-balancing and possible concurrency issues, not sure to what extent this is the case though) from PHP.

Assuming PHP will be used. Who owns the DBB-side work again? The TDS is sounding like a custom phpBB extension, but it depends on the maintainer ultimately.

[Jeff250]

If we use 2.2), then I think that this is the sanest way we can do it:

* Generate HTML that uses AJAX to proxy the request through a PHP script to the tracker to load the list of ip address/port pairs
* For each ip address/port pair returned:
* Use AJAX to individually but simultaneously proxy each request through a PHP script to the ip address/port pair to load the game information

Flip, like Sirius asked, are you suggesting SQL for our tracker display system, or the tracker itself?

The tracker proper just stores the result in a dict in RAM.
If we felt it necessary to use a backing store for the DBB to, say, cache results, then I'd once again argue that 2.2) stops making sense and we should use 1), because we'd be essentially inventing a needless "secondary tracker."

[fliptw]

The tracker, I am working under the assumption that the tracker will be hosted alongside the dbb.

[Jeff250]

Aren't we just talking to Diedel's?

[fliptw]

I thought a new tracker is being made.