Fix for Linux Loki Checksum bug

[Xamindar]

I can't seem to be able to specify an ip and connect when this thing is running. Is there a way around this?

[Munk]

Yes you can. But the result of the Direct-IP-Server will be mixed within the servers that come from the trackers.

You can use the following bug/feature:
Enter your IP and Port in the input field. In the meantime, the tracker list will show up.
After the tracker list had shown, click on \"scan for local games\". This will only show you local games (in LAN). Surely their aren't any, so this will flush the whole list (for about 20 seconds).
Now press 'Enter' in the input field (or click on 'search for games at address'. Then your specified server will show up.

Sorry that this is a bit confusing. It's an artefact I had to build in because Descent can't handle server lists with more than 12 servers very well.

Someone asked how this gamebrowser is able to show internet games. Well ok:
When Descent tries to find gameservers in LAN, it sends a \"is anyone here serving descent?\"-Message to the network broadcast address (255.255.255.255). Usually this Message will be routed to all computers in LAN. If a gameserver is running somewhere in LAN and gets this message, it responds to the sender with \"my name is VEX-Server, I'm running on IP 123.45.67.89, my mission is Ascent ...\" (you get the idea).
Your Internet Service provider will not route those broadcast messages, as this would mean to send it to all those billions machines on the internet. Thats why this gamebrowser does not work with internet servers.
So it's just a problem of routing.

Fortunatly, there are servers in the internet how know where descent3 gameservers are running. They are called \"trackers\". One of them is http://d3.descent.cx.
Now, what d3fix does is: with its packet filter it can detect when the \"is anyone here\"-broadcast message is send from your gamebrowser. When this happens, it connects to the d3.descent.cx-tracker and gets a list of running internet gameservers. It then creates identical \"is anyone here\"-messages which are then send to these internet gameservers (knowing there IP).
The gameserver will respond that message to the origin containing his server details. Descent receives these responses, and displays them in the gamebrowser serverlist.

[Xamindar]

Thanks Munk, I'll do that. I just had a frustrating time last night trying to connect to my friend. At first I loaded d3fixd with D3 which showed all the internet games and then tried putting his ip in. I couldn't find it in the list so I reloaded the game without d3fixd but then I couldn't get ANY games to show up, whether my friend's or any other internet game when I typed it in the ip field. It was very strange.

[Munk]

It is maybe so, that when you kill (signal KILL, STOP) the d3fixd, it cannot clear the kernels ip filter rules.

try \"# iptables -t mangle -L\" to show the relevant filter rules used by d3fixd. If there are some lines containing port 2092, and d3fixd is _not_ running (i.e. crashed or killed), then you should run \"# iptables -t mangle -F\", which will flush the mangle table. Otherwise all packets addressed to the descent port 2092 will remain in the ip packet queue and wait there forever since there is no d3fixd which handles them.

Of course, if you just close the d3fixd (signal INT) by hitting CTRL-C or calling \"kill\" without an explicit signal, it should clean up by himself, as he prints \"stopping filter.

[Xamindar]

Ok thanks for the info. I had thought of something like that but didn't bother investigating it because I didn't have iptables installed. So I just installed it and will check it out if this happens again.

[Xamindar]

Hey Munk, D3 Retribution still gives levels don't match errors. I was trying to join a coop game. How can I go about adding the retribution levels to d3fixd?

[Munk]

Sorry for my late response. This forum might be stucked. This thread never bubbles up in the thread list when a new post arrives, nor does it show the correct date of last post (just shows \"Sat Mar 25, 2006 2:30 pm\"). The mail notify doesn't work also... so there's no real chance to see when something is posted here.

I'm sorry Xamindar, D3-Retribution or Mercenary does not run on VEX-Servers. Don't know why it's not running - so no checksums for that mission.

[iten]

Amazing work on this.

I installed this a while ago, but only a couple weeks ago got it working--it turned out that there was a version incompatibility with the perl Storable module. It may or may not be because I have a 64-bit system, but either way the checksum file didn't work for me.

I found a way to convert it, though, so if anyone has any problems mentioning \"Storable,\" try downloading this checksum file instead.

http://carolnrob.net/~jol/checksums

Thanks to Munk for his great work.

[Munk]

I was wondering if this could be used, or sometime in the future be used, to run a linux server that does allows other windows games to connect to it.

There's a version ready which can run on a linux gameserver to allow all those windows-users to join.
Unfortunatly since my ubuntu upgrade, my perl compiler broke. So what I can give you is the perl script - no binary.
If you are brave enough to install a approx. dozen of perl modules by hand, you could try it. Just PM me for instructions.

[FunkyStickman]

Munk, did we ever tell you how much we appreciate this? You're awesome. I haven't gotten to play D3 in a few weeks (real life sucks, don't it?) but I'm planning on upgrading my main system, and when I do, I'll be in the mines almost every night!

Thanks again.

[MoonSire]

Hi.

Did you manage to get the linux server fix to work? I would really appreciate it if you could upload it, me and a couple of friends have recently discovered descent3 and we play it at least an hour a day

Thanks in advance!
/MoonSire

[Munk]

You can try the 2.0 version with server support by running it in perl:

http://munk.vex-server.de/d3fix/d3fixd.pl

I have no working perl compiler anymore, so basically you will need to install the perl modules by hand.

[Do_Checkor]

Uhmmm. Sticky?

[MD-2389]

I would sticky it, but it seams the board is having issues again.

edit This thread is linked in the bugfixes announcement thread.

[MoonSire]

Hi.

Thanks for your tool! It's really good, but I need the checksum generator program, I have some levels that I would like to add.

/MoonSire

[Munk]

Well, I've written a small perl script you can run to add checksums for new missions to the database.

http://munk.vex-server.de/d3fix/d3fixadd.pl

(you will need perl and have to install the Storable module: http://search.cpan.org/CPAN/authors/id/ ... .16.tar.gz)

How to add new checksums of a mission to the database: (it's a bit tricky)

0. Make sure d3fixd is not running, by killing it gently (with SIGTERM).

1. Find or setup a Windows Descent3 Server (or Mac) running that specific mission.

2. Fire up d3fixd without sending it to background (by omitting the \"&\"), so you can see it's output in the console.

3. Start your Loki Descent and try to connect to that server. If you get the \"Levels Don't Match\" error message, quit (or minimize) Loki to see at the output of d3fixd. It will tell you the catched checksum by the message \"read Checksum 12345\". This is the checksum as it is reported from the windows server.

4. Open a second console, and run d3fixadd.pl with \"d3fixadd.pl 12345 +1\".
It will alter the d3fixd checksum database to report a new checksum to your Loki client which is 1 higher (hence the \"+1\").
You do not need to restart d3fixd, it will load the new checksums by itself.

5. Try again connecting with Loki to the Gameserver. Hopefully you can now connect.
If you cannot, repeat step 4 with a different adjustment (i.e. try \"-1\", \"+2\", \"-2\", ...).
The adjustments to the checksums are typically small (up to about +/- 4), but smaller adjustmends have higher probability of success, so start with the lower ones.

6. If you can connect, well done!... But: This checksum will only work for this specific ship you selected. So you need to do it again with Pyro-GL, Phoenix, Magnum-AHT and Black-Pyro.
Both, Checksum and Adjustment will change for each ship, so after changing the ship you need a look at d3fixd to get the new checksum, and then try new adjustments to it. (first try as adjustment +0, maybe you do not need one).

7. If you cannot find a suitable adjustment, ask a friend to setup a Loki Descent 3 Server with the same mission. Connect to it, and look at the output of d3fixd. It will now report (as you connect to a Linux Descent3 Server) the Linux checksum. The Difference <Linux Checksum> - <Windows Checksum> is the adjustment you need to specify to d3fixadd.pl.

8. Finally if you managed to get all adjustments for the specific mission and for all 4 ships (or at least the 3 basic ships without Black-Pyro), you can mail me <munk@vexclan.de> your checksum database (located at /usr/local/share/d3fix/checksums), and I will put it on my website.


P.S.: Maybe step 7 is the fastest way to get to the checksums, but you need lots of ressources, (since you cannot run loki client and loki server at the same time on the same machine).

P.P.S.: You should run d3fixadd.pl as root user, since it needs to modify /usr/local/share/d3fix/checksums, which is write-protected. Also for d3fixd to reload his checksums, it needs to send SIGHUP to d3fixd. Since d3fixd is owned by root, only root can do that.

[FunkyStickman]

Munk, thanks again for your hard work. I'll try to set aside some time to try this sometime this week. I've even got a spare Linux server to work with, so should be fun!

[The Lion]

Why is it impossible to run a loki server and a loki client on the same comp?

[Munk]

because both try to allocate the udp listening port 2092.
Loki didn't implement an option to change that to another port.

[The Lion]

Ok, but still... it looks like a simple LD_PRELOAD job to me.

[MoonSire]

Thank you so much!! Your work is very appreciated!!

[The Lion]

Ok, but still... it looks like a simple LD_PRELOAD job to me.

Job done btw.

[MoonSire]

@TheLion

woohoo! Thank you! So many talanted people and all willing to help! This really made my day! Thanks!

[The Lion]

You're welcome.

It was an easy job though, as I expected.

[Arcalyth]

Sorry for the bump, but can anyone mirror the fix? Munk's site 404s and I'd love to play something other than the campaign!

[Munk]

I'm ashamed it's 404, i have no clue how that happened (VEX changed servers twice since the release).

Searching the web I found the perl script
http://odf.sourceforge.net/downloads/descent3/d3fixd.pl
but you will also need the checksum database.

Things on my todo list (after finishing studies in a month) is a better version of that fix based on LD_PRELOAD hacks instead of IP mangling. It would not need any root privileges...

If you can wait a few weeks, I would need some help in testing.

[flip]

http://www.filehosting.org/file/details ... les.tar.gz

http://www.filehosting.org/file/details ... 1.6.tgz.gz

[Arcalyth]

Thanks flip. Now to get it working... having trouble getting IPTables::IPv4 to install properly >_<

Munk, I'd be happy to assist you in testing. Gmail: Arcalyth
Go ahead and send me some mail when you're ready.

[flip]

NP, over the last 6 years I think I downloaded everything Descent related I could find on the web

[Arcalyth]

I haven't been able to get d3fixd working because IPTables::IPv4 module refuses to install. Since I'm impatient, I figured I'd try to write a python version since the iptables module for python works. To do so, I need the original checksums file. I tried to use Storable to deserialize the one in the .tar.gz, but the new version of Storable can't read it and the old one that you linked on Page 2 won't compile either.

So, do you have the original non-serialized checksums?

[flip]

No, it doesn't look like it, just those 2.

[akula65]

If you are referring to this link ( http://munk.vex-server.de/d3fix/checksums ), there is one instance of it in the Wayback Machine ( http://archive.org/ ) in 2007.

[Jeff250]

I'm taking a stab at this bug from the other direction by hacking at the linux binary's x86/x87 machine code. The following python script can be applied to both the statically and the dynamically linked linux binaries to reconcile the "sqrt" bug:

Code: Select all

# obsolete, see post below

It looks like that when loading the *.dlv file, at some point D3 calculates the normal vectors for a series of surfaces, presumably walls and such, but I haven't dug deep enough yet to figure out what for sure. It then normalizes these vectors to unit length, which is when sqrt() is applied. The windows binary returns the result of this sqrt() in "long double", i.e., in an x87 80-bit precision register, whereas the linux binary first rounds the result to "float" by popping from an x87 register to a 32-bit precision stack location, then pushing back into the x87 register. The checksum is somehow a function of these unit-length normal vectors. The above patch removes the linux binary's rounding of the sqrt() result.

I've tested this patch on a number of popular levels. The results are below. The first Y/N is whether the checksums agreed before the above patch. The second Y/N is whether they agreed after the patch.

abend 2: N Y
ascent: N Y
burning indika 3: Y Y
halcyon: Y Y
kata 1.2: N Y
pyroglyphic: Y Y
skybox: Y Y
stadium: Y Y
subway dancer: N N

The good news is that this patch hasn't introduced any regressions in my testing. Moreover, it reconciles the checksums for a number of levels. Subway Dancer, however, still produces disparate checksums, and I suspect some others untested may as well.

This means that the "sqrt" bug wasn't the only bug.

Is there some reason why Subway Dancer might still be affected, i.e., does it use some feature that perhaps the other levels don't? I've already tried stripping its *.gam file and other customizations from its *.mn3 (neither of which actually appear to even be checksummed...), and it still fails. It may just be hitting some other rounding issue...

Note to programmers: don't use floating point arithmetic for anything where an '==' comparison might matter. Unless your compiler is C99 conforming (AFAIK Microsoft's isn't), where the compiler rounds floating point arithmetic is undefined.

Maybe something simpler than tracking down the other discrepancy(s) is to just jump around the checksum check all together. Would the community care if I did this? It could theoretically be used by linux users for cheating, but I think there are already a dozen easier ways of doing that. The checksum is still a nice sanity check to make sure you aren't accidentally using the wrong version of a level though...

[Munk]

Good work, very nice !

I know for sure that the selected player ship is included into the checksum. Subway dancer uses modified game content, which could also be included.

[Jeff250]

I've found where the checksums are kept in memory. Both keep it as a 32-bit int stored in statically allocated memory: Windows keeps it at 0x5a6648, whereas linux keeps it at 0x82c0484. Knowing this makes it a lot easier to debug this issue. I've also found where the checksums are checked, but I don't think we will need to disable that, as I've found another floating point rounding discrepancy that accounts for the mismatch I was still seeing.

This discrepancy was the opposite case: Windows rounded from 80-bit to 32-bit precision when linux didn't. The fix for this discrepancy resolves the mismatch with, e.g., Subway Dancer. The following script can be used to patch either the statically or the dynamically linked linux binaries, and it's also comprehensive, including the patch from my earlier post:

Code: Select all

#!/usr/bin/env python
# Patches either the statically or the dynamically linked D3 Loki executables
# to resolve multiplayer level checksum mismatches with the Windows version
# By Jeff250

PATCHES = [
    (
        [
            (0, '\x55\x89\xe5\x83\xec\x18\x8b\x45\x08\xd9\x00\xd9\x40\x04\xd9\xc9\xd8\xc8\xd9\xc9\xd8\xc8\xde\xc1\xd9\x40\x08\xd8\xc8\xde\xc1\xd9\xfa\xd9\x5d\xfc\xd9\x45\xfc\x89\xec\x5d\xc3'),
        ],
        [
            (33, '\x90\x90\x90\x90\x90\x90'),
        ]
    ),
    (
        [
            (0, '\x55\x89\xe5\x83\xec\x3c\x57\x56\x53\x8b\x75\x0c\xd9\xee\x31\xc9\x39\xf1\x7d\x76\x8b\x5d\x10\x8d\x41\x02\x99\xf7\xfe\x8b\x7d\x14\x0f\xbf\x04\x53\x8d\x04\x40\x8d\x04\x87\x50\x8b\x7d\x10\x8d\x59\x01\x89\xd8\x99\xf7\xfe\x0f\xbf\x04\x57\x8b\x55\x14\x8d\x04\x40\x8d\x04\x82\x50\x0f\xbf\x04\x4f\x8d\x04\x40\x8d\x04\x82\x50\x8d\x45\xf4\x50\xdb\x7d\xd0\xe8'),
            (4, '\x83\xc4\x10\xdb\x6d\xd0\xd8\xd1\xdf\xe0\x80\xe4\x05\x74\x18\xdd\xd8\x8b\x45\xf4\x8b\x55\x08\x89\x02\x8b\x45\xf8\x89\x42\x04\x8b\x45\xfc\x89\x42\x08\xeb\x02\xdd\xd9\x89\xd9\x39\xf1\x7c\x8a\xdc\x1d'),
            (4, '\xdf\xe0\x80\xe4\x45\x80\xfc\x01\x74\x06\xb0\x01\xeb\x04\x89\xf6\xb0\x00\x8d\x65\xb8\x5b\x5e\x5f\x89\xec\x5d\xc3'),
        ],
        [
            (83, '\xd9\x5d\xd0'),
            (8, '\xd9\x45\xd0'),
        ],
    ),
]

def patch(path):
    try:
        with open(path, 'rb') as f:
            buf = f.read()
    except IOError:
        print '[-] Cannot read %s' % path
        exit(1)
    print '[+] Read %s' % path
    patched = list(buf)
    for i, (before, after) in enumerate(PATCHES):
        first_skip, first_find = before[0]
        start = 0
        found = 0
        while True:
            start = buf.find(first_find, start + first_skip) - first_skip
            if start < 0:
                break
            offset = start + first_skip + len(first_find)
            for skip, find in before[1:]:
                offset += skip
                offset = buf.find(find, offset, offset + len(find))
                if offset < 0:
                    break
                offset += len(find)
            if offset >= 0:
                for skip, replace in after:
                    start += skip
                    patched[start:start + len(replace)] = replace
                    start += len(replace)
                print '[+] Offset #%d to patch found at %#x' % (i + 1, start)
                found += 1
            else:
                start += 1
        if found < 1:
            print '[-] Cannot find offset #%d to patch' % (i + 1)
            exit(1)
    backup_path = path + '.old'
    try:
        with open(backup_path, 'wb') as f:
            f.write(buf)
    except IOError:
        print '[-] Cannot save backup to %s' % backup_path
        exit(1)
    print '[+] Saved backup to %s' % backup_path
    try:
        with open(path, 'wb') as f:
            f.write(''.join(patched))
    except IOError:
        print '[-] Cannot write patches to %s' % path
        exit(1)
    print '[+] Wrote patch to %s' % path

if __name__ == '__main__':
    import sys
    if len(sys.argv) != 2:
        sys.stderr.write('Usage: %s EXECUTABLE\n' % sys.argv[0])
        exit(1)
    patch(sys.argv[1])

Note: This patch will fail if you've already patched using the script from my earlier post--restore the backup first!

This should now be ready for mass consumption. All levels that I've tested now produce consistent checksums. If anyone finds levels that mismatch despite using the above patch, please report them.

edit: Attached script for download below.
edit #2: Cleaned up python, but patch itself is unchanged.
edit #3: Fixed for statically linked binary.

[theAntiBob]

Been trying to patch descent3 and descent3.dynamic with d3-loki-checksum-patch.py.txt (renamed to not txt)
descent3.dynamic patches fine (or says it does) but doesn't launch when called, before or after being patched. just does this:

Code: Select all

: /usr/local/games/descent3/descent3$ ./descent3.dynamic 
./descent3.dynamic: error while loading shared libraries: libSDL-1.2.so.0: cannot open shared object file: No such file or directory
:/usr/local/games/descent3$ locate libSDL-1.2.so.0
/usr/lib/x86_64-linux-gnu/libSDL-1.2.so.0
/usr/lib/x86_64-linux-gnu/libSDL-1.2.so.0.11.4
:/usr/local/games/descent3$ sudo ~/Downloads/descent3/./d3-loki-checksum-patch.py descent3.dynamic 
[sudo] password for theantibob: 
[+] Read descent3.dynamic
[+] Offset #1 to patch found at 0x20e327
[+] Offset #2 to patch found at 0x134608
[+] Saved backup to descent3.dynamic.old
[+] Wrote patch to descent3.dynamic
:/usr/local/games/descent3$ ./descent3.dynamic
./descent3.dynamic: error while loading shared libraries: libSDL-1.2.so.0: cannot open shared object file: No such file or directory

do i need a symlink somewhere?
when attempting to patch descent3, this happens:

Code: Select all

$ sudo ./d3-loki-checksum-patch.py /usr/local/games/descent3/descent3
[+] Read /usr/local/games/descent3/descent3
[+] Offset #1 to patch found at 0x211a77
[-] Cannot find offset #2 to patch

any idea why this won't patch?
it looks like other people have had success with this.

i just wanna play!

btw, the loki port works flawlessly for me, with the exception of not being able to render in 1920x1080. 1600x1200 works well enough on this display so it's still MUCH better than through wine!

[Jeff250]

when attempting to patch descent3, this happens:

Code: Select all

$ sudo ./d3-loki-checksum-patch.py /usr/local/games/descent3/descent3
[+] Read /usr/local/games/descent3/descent3
[+] Offset #1 to patch found at 0x211a77
[-] Cannot find offset #2 to patch

Oops, I've fixed the script in my earlier post to again work with the statically linked binary.

descent3.dynamic patches fine (or says it does) but doesn't launch when called, before or after being patched. just does this:
[...]
do i need a symlink somewhere?

The dynamically linked binary didn't work before the patch either, right? In order to get the dynamically linked binary to work, you need to hunt down the dependencies for it. For instance, right now it's complaining about not having libsdl, so you'll need to get that. (On Ubuntu, "sudo apt-get install libsdl1.2debian:i386".)

[theAntiBob]

Thanks for the update! Both descent3 and descent3.dynamic were successfully patched.
Works great and I can FINALLY play d3 online... (not like there's really anyone out there to play) I do have a stash of win98 pentiumIII boxes that may get used for a lanparty, though.

The dynamically linked binary didn't work before the patch either, right? In order to get the dynamically linked binary to work, you need to hunt down the dependencies for it. For instance, right now it's complaining about not having libsdl, so you'll need to get that. (On Ubuntu, "sudo apt-get install libsdl1.2debian:i386".)

No, it wasn't working before... did:

Code: Select all

$sudo apt-get install libsdl1.2debian:i386 libsmpeg0:i386

but now it just gives me sig11. I'm not really too concerned.

Thanks for all your help and effort...

and thank you, SpiffyBruin for letting me be theAntiBob again.

[Jeff250]

but now it just gives me sig11. I'm not really too concerned.

descent3.dynamic tends to not play nice on newer systems with newer versions of the libraries it links against. The Outrage or Loki folks probably assumed something about an API that they weren't supposed to and that got changed at some point. But like you say, there's no reason to be concerned about it as long as the statically linked binary works. One caveat with the statically linked binary is that it has a version of libsdl built in that is hardcoded to use OSS, the old linux sound system, so you need to run it through the appropriate emulation layer for sound to work. For pulseaudio:

Code: Select all

padsp /usr/local/games/Descent3/descent3 -n # and rest of args

For straight up ALSA:

Code: Select all

aoss /usr/local/games/Descent3/descent3 -n # and rest of args

I suspect you already figured this out, but I'm just documenting it here for whoever else might read this thread someday.