Want to create automatic converter from demo files to videos

[AlexanderBorisov]

Basically want to make a kind of Rebirth "plugin" (in fact, a function) that will play the demo into the video file instead of the screen. I need this because I have a lot of Descent 2 demos to convert (maybe 100 hours worth), and obviously converting them with (F)CRAPS or similar software one-by-one will be very long and tedious process. Another possible benefits are, such a plugin won't block my computer (as it can be run as low priority task) and can be potentially much faster.
How do you think, is it worth trying? From the brief look, I've run into the following problems, greatly appreciate if you could help me with some of them.
1) Compiling Rebirth under Windows environment. I've read the instructions in the project folder (INSTALL.txt) , still it seems that they are for people who already have their environment set up and have all necessary dependencies.
I installed SCons 3.0 and Python 2.7 but running SCons,gives syntax error in SConstruct file, line 17
print "%s: %s" % (program.program_message_prefix, msg). And I have a feeling, that paths to compiler, linker and scons.bat are not set up correctly by scons installer, too.
2) Is there a way to create and use VS project for compiling Rebirth?
3) Could not find where rendering and demo playing functions are located. The project seems to have no documentation, and file names are not self explaining, too. Maybe if I had VS project it would not be a big issue.
4) Can you point me to the right direction WRT to writing sound into the video and syncing it with video frames (which API to use, some code examples). For example smth like creating a window from a soundtrack and a static picture. Also about how to convert Rebirth sound format to anything that could be written to the video.

I am an experienced software engineer (and have no problems understanding other's code), but had very little experience with video codecs, and NO experience with audio (except using Beep function on my old ZX-spectrum 25 years ago ). Mostly was working with various math, machine learning algorithms, low level optimization and stuff like this. Of course did basic GUI and know OpenGL well enough.

[Sirius]

There is currently no Visual Studio-based method for compiling Rebirth that I know of (if you find one let me know - there's something I would really like to debug). CMake could probably fix that, but Rebirth doesn't use it, and I'm not sure one way or the other whether VS's support for Linux is close to the same stuff needed to build Rebirth.

It might be possible to do the same thing with XL, though - depending how picky you are about look and feel (you can tweak visual settings at least) - since its Windows version actually does compile with Visual Studio. Don't know whether its navigability is better or worse - it's heavily rewritten in a lot of areas and uses C++ instead of C.

[AlexanderBorisov]

I found that Rebirth has more problems with VS compilation than just the absence of VS project (or CMake). In fact it seems to have VS2017 project in contrib\broken-vs2017 (however I have only 2010 on my machine and cannot test how "broken" it actually is ). It also requires compiler that supports some C++11 or even C++14 features not supported in VS (only in gcc and clang), and SDL has the same requirements, too... Maybe I can try to download the latest VS and try with it, but still SDL and PhysFS need to be compiled somehow..
And it's a pity Rebirth developers don't use CMake, it is so easy to use and so common in open source community.

And basically I am a Windows guy (although familiar with coding and debugging in Linux, and with CMake of course). I always found Linux IDEs and debuggers just don't work for me, especially when it concerns low-level optimization stuff. Or any complex debugging. Just spent a lot of time working around IDE and debugger problems and limitations to do things otherwise trivial in Windows environment. And observed people near me doing the same and still telling me Linux beats everything. Don't know anything about VS support for linux. I always assumed there is none. I am now more inclined towards using autohotkey to script and automate demo conversion, as Krom suggested in Rebirth forum section.

And concerning XL, I don't like how the demos look like there; too dark, seems that it has aspect ratio changed, and like my aim is shaking constantly, which I don't observe in Rebirth. BTW I found the functions that record a demo and render the screen, but it is still no use until I could compile Rebirth.

[Krom]

You might also want to try asking in the actual DXX-Rebirth forums what all they use to compile it. Apparently they only support free compilers (which is why visual studio is not targeted).

https://forum.dxx-rebirth.com/

[AlexanderBorisov]

Thanks, will try it too, I still cannot get activation e-mail from their forum, probably tomorrow. BTW VS Community Edition is now free, so it is no excuse for them anyway...

[Sirius]

The C++11/14 stuff rubs me the wrong way, since I've looked at the 0.60 Rebirth code and as far as I could see 90%+ of it was just procedural C in .cpp files. I couldn't see anything that used modern C++, never mind esoteric compiler features.

Now that doesn't mean they're not there, or that it's an easy fix, but it does make me wonder just how much of a blocker this really is versus just "not interested in supporting VC++".

But that's just frustration speaking... my hope is that if I actually talked to them it would make more sense.

[AlexanderBorisov]

Yeah, I am also a bit tired of this C++11 hype on my current and previous jobs. People pushing it everywhere. Saying it can make ANY code better. Concerning Rebirth I don't see how it may be important or even blocker issue, if the old game used plain C for everything in many years, from what I saw in the code it was only using some keywords like "auto" and "constexpr" that are in no way useful except for C++ purists. I guess, maybe they also use some cross platform threads, file system and time functions, but I haven't seen any. What i've seen was pretty much plain C, too. That is another piece of frustrated speaking And I still got no reply from them even about the forum registration issue, so probably will get to my backup plan with AutoHotkey. Yes it will require a bit more effort, but perfectly doable.

[Sirius]

There are things about C++11/14 I see the use for, e.g. auto comes in handy for iterators for complex types, and some things like that. But even then, MSVC supports all of that now. The things it doesn't support typically amount to edge cases, which usually have viable workarounds.

(Much later) Edit: Am looking at the VS2017 project and thinking... I do wonder how far off this is. It might be worth poking at, although I'll likely run into the same issues the original author did. Still. There might be ways to resolve them if one is sufficiently determined...

[AlexanderBorisov]

Well yes, auto can be handy but should not be so critical for Rebirth because most of the code is so old and in plain C. And in old times we just used typedefs for those complex types and it was ok. Anyway I wonder what are REAL issues with VS2017 project? I think they can come from dependencies, i.e SDL and PhysFS, too. Hope I will get my hands on most recent VS and try myself later, but first I need to finish the script for demo recording.
Anyway keep me updated if you will have any success with VS build. BTW we had the same problems for Windows compilation support in our work, but with much larger project with a lot of dependencies, everyone kept telling it is impossible to compile in Windows and even more impossible to support it, but the whole problem was solved in several days after it became really necessary. And there were no serious problems with maintaining it afterwards. But at least we had CMake build system, and I think it could be even better solution for Rebirth than fixing this VS project. Of course one needs to fix compilation errors related to "esoteric" compiler features, still.

[Sirius]

There was actually a reasonable description in the related commit comment - gave the following known issues with some examples:

- Visual Studio 2017's C99 preprocessor support, like those of predecessor versions, is deficient regarding variadic macros, causing it to reject valid constructs with confusing error messages.
- Visual Studio 2017 sometimes disallows inner classes access to private typedef symbols in the containing class. Both gcc and clang permit this and the standard says this should work.
- Visual Studio 2017's cl.exe crashes processing some files. Including "segment.h" is sufficient to crash cl.exe.

I suspect it's possible to correct for these with some code changes. Found out I didn't have VS2017 installed on this machine though (been running old versions I guess ) so will have to do that first. My attempt will be on Retro 1.4 (based on 0.58.1) though, since that's what is actually played online these days.

[AlexanderBorisov]

Ok, I believe 2 is not hard to fix (just make those typedefs non-private or even global; I hope there are very few). Yes I know "true C++ people" will curse you for that but who cares...
3 : should not be hard to find what line of segments.h causes the crash by bisection. Unfortunately VS is still prone to "Internal compiler error" crash in cl.exe, and this can only be solved by finding the line that causes it and changing it. Once I had to change normal int assignment to memcpy . But in new VS it happens VERY rarely, in fact.
1 : Do they really need those variadic macros at all? Seems if there is a lot of them, it can cause problems. Otherwise it should be easy, too.
I am more concerned about SDL and PhysFS, which can use a lot of funky C++ stuff (the first one is as I understand, is OpenGL/graphics/sound library, second one is a kind of cross-platform file system wrapper), believe the game code in itself should have very little of it. Still interested in compiling Rebirth, because I have some minor (non-game breaker) but annoying bugs in Rebirth that I'd like to fix. But, remember, even if we will manage to fix the VS project for the game, still need VS projects for SDL and PhysFS (and I don't think just binaries will suffice; most often you need all dependencies to be compiled with the same compiler in order to work), and every time they will add files to Rebirth you'll have to update it, too.
About VS2017, I heard it is a decent version, with full C++11/14 support (including variadic macros and templates, of course there can be some quirks it still does not support), and it is faster and has less bugs than it's predecessors, and has a working built in profiler. And the UI is not different from VS 2013 (that I am using now).
And what is Retro 1.4? I mean what are main differences from Rebirth 0.58.1 (that is in fact the version I use)?

[Sirius]

Unfortunately I'm not sure where a changelist is. I can list the stuff I remember though - and it's mostly multiplayer-related.

• Switched netplay back to mostly peer-to-peer (though it still uses the host as the entry point into a game and a proxy where peer-to-peer isn't possible due to routing issues)
• Introduced a framecap on homing weapon tracking to better emulate "classic" behavior
• Reintroduced position packets on weapon shots to ensure the trajectory is 100% consistent between players
• Option for ship color changes for better visibility and/or fairness (both players blue for 1v1; more distinctive ship 6/7 colors for FFA)
• Options to multiply primary/secondary weapon spawns for larger games, or to reduce missile counts to 2/4/6 of each type
• Additional respawn behavior options - half-second invulnerability, "preview" mode (you see the location you're going to spawn in before you do)
• Custom ship color combinations (wing/body colors configurable separately)
• (D2) Option to restore D1 weapon behavior for spreadfire/fusion
• (D2) Option to spawn with pre-equipped afterburners
• (D1-only for now) Observer mode

[AlexanderBorisov]

Ok, I finished the demo to vid converter in as an AHK script running OBS Studio free recording and streaming software. Works like a charm! And still no response from the Rebirth forum...Hope I will get to fixing Rebirth VS project those weekends.

About Retro build that you are using, by
"Introduced a framecap on homing weapon tracking to better emulate "classic" behavior"
do you mean Retro has easier or harder homers then Rebirth 0.58? Because in my version in D1X 0.58.1 you can hardly dodge them on Insane; maybe with extreme trichording (by that I mean : going nearly perpendicular to the missile at the moment it approaches) you can dodge the first one from the volley, but others turn around and hit you from the back. But this is useless most of the time, even if you could dodge the volley another one is already on it's way to you. In D2X 0.58.1, it was easier to dodge homers without afterburner and it was practical (i. e because you don't need 90 deg angle to missile course and maybe 30-45 deg was enough, so you could dodge and fight back). Still not a piece of cake, missiles turning 90 deg or more and hitting you from the back were a real threat. I am asking because i've seen a lot of threads on the forum where people tell that homers are too easy to dodge.. Maybe they are just not playing on Insane..

About those D1 spreadfire and fusion, also still cannot get the point, because spreadfire was not a very useful weapon even if D1 (if you had plasma and quad lasers), and Fusion is not useful in D2 because robots are either too fast or have too much HP. Only case when I used it in D2 is against big groups of melee robots, to save energy. Maybe it is useful in multiplayer, I don't know.

[ShivanHunter]

Those differences in Rebirth are intentional. Descent 1 had a (buggy) much higher turn rate and a (also buggy, I think) larger tracking/acquisition cone which is why missiles could appear around a corner and instantly turn towards you. Descent 2 fixed/nerfed this behavior, which is the main reason for differences between D1X-Rebirth and D2X-Rebirth.

I think Retro keeps the original D1 behavior in D1 and the D2 behavior in D2 though I'm not certain.

The bug that specific change is talking about is different though: The old homers used weird code that ended up doing things, like, every fourth frame - so "just" using realistic values in a frame-independent simulation (like the rest of the game is) wouldn't really recreate the classic behavior. The solution was to have homing be determined by a separate simulation, locked at 30FPS and emulating the actual old behavior at that FPS, running alongside the rest of the game.

The effects of this fix are a bit subtler but say you fire a homing missile in D2 and watch its camera view - you'll see it sort of snapping towards its target in increments rather than following a smooth motion. This is the new homing code at work and it produces a curve that most resembles the old D1 behavior.

Homing rate itself is not affected by difficulty, although missile velocity is, so it may be somewhat easier on easier difficulties just because you have more time to dodge.

>Fusion is not useful in D2 because robots are either too fast or have too much HP

Yes, that's actually the point of that change - Fusion damage was halved in D2 making it a rather pathetic weapon. Spreadfire projectiles were also made much slower, and the laser damage was nerfed. The D1 values for these weapons are just better and more balanced. Personally I'm restoring the D1 values for my D2 mapset using a HAM file.

[AlexanderBorisov]

I thought that homers behavior in Rebirth 0.58.1 is already frame independent, maybe I am mistaken... Not sure how it can be checked.
About Fusion, even if it had D1 strength I still cannot imagine how one can fight against fast robots - ITD's, ITSC, BPERs with it on Insane (they dodge then swarm and kill you while you recharge), and also against Sidearms, Seekers, Lou guards (that all have HP similar to D1 red hulk as I remember, need maybe 3 shots even with original fusion, so they also kill you while you recharge). What I am trying to say is that even with more powerful fusion and spreadfire you won't have a lot of chances to use them in D2, because of different robot behavior. Unless you design your mission in some special way..
In D1, of course, I used Fusion myself a lot. And I also agree it shouldn't be nerfed in D2, maybe then it will become useful in some situations.

[ShivanHunter]

You're half right - the fusion is not the weapon for D2's swarming robots (which circle around you rather than bunching up like in D1). It actually is decent against fast robots (or other players) since the fusion projectiles are quite fast and have large hitboxes, making them harder to dodge.

Retro is by/for the multiplayer crowd who are mostly very high-skilled old-schoolers - the fact is, if they say the fusion needs to be denerfed because it's actually useful at 60 dmg and that there's a difference between the 9-dmg-160-speed spreadfire and the 10-dmg-200?-speed spreadfire, they're probably right.

[AlexanderBorisov]

Ok, I don't object that de-nerfed Fusion and Spreadfire can change balance in multiplayer for good (although I do not play MP myself and cannot comment on this); but in SP I am yet to see the mission where it would be useful. During my play through many D2 missions (stock campaign, Vertigo, TEW, Obsidian, LL, Enemy Vignettes) on Insane no death no save I never used spreadfire except for cold starts where I had nothing better; used fusion in 3 fights total in TEW and EV; and even with D1 power I think I would not use spreadfire, and probably will just use Fusion a few times more but it would not become weapon of choice like in D1.

[Sirius]

Yes, it's mainly a multiplayer thing. Spreadfire doesn't see that much use in single-player/co-op, but in multi it's heavily used because it's fast and creates a scattered pattern that is more difficult to (mentally) track and thus evade. It's sort of a compromise weapon - not as fast or accurate as Vulcan, but doesn't use ammo which may be limited, and not as high DPS as Plasma or lasers, but will reach the target before either of them. It can be used at basically any common range, which isn't generally true for other weapons - Vulcan is very nearly a superweapon but can still be outclassed at close range.
D2 Spreadfire was not able to do this, mostly because it's too slow - the damage reduction is a more minor factor.

Fusion was not used in multiplayer in D2 because its damage was no longer high enough to make up for the downsides (it requires very real skill to aim it). That's basically the problem with it. It carries over into single-player to a smaller extent in that Fusion has always been good for blowing up tight clusters of robots, and this restores that functionality. However it is also true that robots in D2 tend to be faster, and it's less common to find opportunities to use it than it was in D1.

Retro homing missiles are easier to dodge in D1, harder in D2 - compared to Rebirth homing missiles. The Rebirth implementation is, in a manner, frame-independent - the turning code is scaled to the frame length, which on its surface seems reasonable. However, it smooths the curve of the missile's flight path, which creates very real differences in how easy the missile is to dodge. It ultimately means you need to be closer to the missile to break out of its lock, which obviously reduces the margin for error since if you're too close, you hit it.

[AlexanderBorisov]

Retro homing missiles are easier to dodge in D1, harder in D2 - compared to Rebirth homing missiles. The Rebirth implementation is, in a manner, frame-independent - the turning code is scaled to the frame length, which on its surface seems reasonable. However, it smooths the curve of the missile's flight path, which creates very real differences in how easy the missile is to dodge. It ultimately means you need to be closer to the missile to break out of its lock, which obviously reduces the margin for error since if you're too close, you hit it.

Still not clear what is the difference to Retro, is turn rate of homers different to Rebirth? From the above I see that Rebirth using slightly different (but different to what ? DOS or Rebirth?) algorithm of adjusting the trajectory (more smooth), but it is still not clear how it works in Retro, and also you said Rebirth implementation reduces the margin for error (thus contradicting the first statement that Retro homers are harder to dodge in D2). Can you explain?

About compiling Rebirth code, I am a bit stuck, VS 2017 doesn't not install on my system because I don't have SP1 installed, so I need to either upgrade or reinstall my system. Still hope to do it soon.

[Sirius]

D2 homing missiles had a subtle difference that I don't really remember that kind of crippled them in comparison to D1. Somehow the effects of the Rebirth "smoothing" were flipped because of it. I couldn't figure out where the analysis thread was - that gave better information.

Retro uses the original D1/D2 algorithms, but throttles them so they only run at 25 or 30 "fps" (equivalent to typical machines in 95/6 - or a rough guess at that).

[AlexanderBorisov]

At last I got to Windows compilation of Rebirth. In short, things are bad. I installed VS2017, and started with the "contrib/broken-vs2017" project. After installing SDL and PhysFS and fixing missing/wrong header references, I've got to that variadic macro stuff. First of it was easy - printf style function for error reporting implemented via variadic macros (you surely cannot do it without variadic macros do you?). And yes, even VS2017 does not compile it.

Then there was a big set of macros that have smth to do with storage management (maybe error reporting too?) for main game data structures. Well, I was able to partially understand what it is only by comparing with original DOS Descent code. It is absolutely crazy stuff, recursive (maybe depth 4 or more) variadic macros without any comments what they do or meaningful names; I see such thing for 2nd time in my life, the first case was one guy in our team, his biggest "achievement" was multidimensional matrix(=tensor) multiplication function heavily using recursive macros that compiled for an hour (I later rewritten it so it worked faster and compiled instantly); hopefully he was fired quickly enough so haven't done much harm. So my guess those macros should be either eliminated completely or simplified for it to compile; unfortunately it is hard to do because I cannot understand their function and also it is about the most important game data (so it will need a lot of testing afterwards to make sure it works).