TV Recorder - how hard?

[snoopy]

Alright, so I have and use mythtv right now. It's pretty nice... very feature-full and has lots of capability and all that jazz.

But, I have a few things against it:

1. It's really geared toward being run on a dedicated box... I'm running it on my PC
2. I don't particularly like the front end interface
3. The back end has been randomly crashing lately (I've updated recently, so maybe the issue has been fixed)
4. It's a bit heavy on resources & forces me to run an Apache/mySQL server

I've looked around and haven't had any success finding what I'd really like... but there are some things that are close, that may be adapted, such as WebVCR+ (last updated in 2004). Here's what I'd like:

1. A web-based interface similar to mythweb that will:
1a. Display listings downloaded from an XML source
1b. Allow for searching functions
1c. Allow the user to specify programs to record, either individually or by more advanced filtering (such as by series, by specific channel, etc.)
1d. (bonus - probably not initially implemented) Feature the ability to secure the page using https & username/password authentication
1e. (bonus - probably not in initial implementation) allow the user to stream either recorded programs or live TV

2. A back-end that will:
2a. Record flagged programs via an external command/program (invoke a script, passing it appropriate parameters)
The thought is to avoid having any actual video/tuner/etc implementation - the program will simply invoke other programs of user choice to accomplish the task. This user- script will probably include an automatic re-encode to cut down on disk usage. It could also include commercial skipping, or really anything that the user would like.
2b. Prioritize between overlapping requests based on user controllable priority numbers (priority number configurable via web interface)
2c. Ideally, run without mySQL. If mySQL in necessary, keep the table usage to a minimum (program configuration saved in a text file, not the table)
2d. Keep track of duplicates, and don't re-record previously recorded things unless over-ridden, solve conflicts by picking between multiple times for the same program.
2e. Run as a limited-rights user, so that security is preserved.

Basically, I want the nice recording & scheduling features that mythtv + mythweb has, minus the extra fluff that I don't use (most of the frontend) and the big, heavy mySQL tables, and plus the ability to automatically re-encode and eliminate the original, big, dump. (With mythtv I can automatically re-encode, but I can't get rid of the original.) There's a code-base out there from mythtv and from webvcr+ that will probably get me a good portion of the way there.. I'd just need to make sure that I cover my legal bases (get permission to use their code) and then make sure I adapted it to my needs appropriately. I'm also looking to segregate the TV recording portion from the playback portion, so I increase my flexibility in media-center usage increases.

Anyways, all that to ask:

How big of a project do you think that this would become? Do you think that it would be something that I could handle alone, in a relatively short amount of time (say a couple months) if I'm working on it part-time in the evenings & on weekends, or do you think it's big enough that I should just live with mythtv and put up with what I don't like about it (or try harder at finding work-arounds for it)?

[Isaac]

It for an app that sounds cool. For a light weight single purpose OS, it would be epic.

@2c.
I noticed when my text file "save files" are long my app can start to lag. Even when using good primary and foreign key concepts the text files still must be read completely before being queried. But if they're short, why not. Text files are user friendly. On the other hand, I've been experimenting with MySQL5.1 for a while. It seems very fast. You think it's slow?

[snoopy]

I don't know that I'd say slow... more like I just don't want to run any more services than I have to.

The main mythtv/mySQL problem that I have is mythtv puts EVERYTHING on its table, and mythtv really has a lot of functionality that I don't use.

[EDIT] In a large sense, this is very close to a duplicate of what mythtv already does, just with me stripping off some of the extra junk that I don't care about. It may very well be just a waste of my time to try to do this. I haven't given a whole lot of thought to adding capability for multiple capture card/tuners... which might start to get the thing to the point where I should just stick to mythtv which already works.

[Jeff250]

The backend is probably the riskiest part--is there such an external program that does everything you need it to regarding capture and encoding?

[snoopy]

The backend is probably the riskiest part--is there such an external program that does everything you need it to regarding capture and encoding?

Yes. Between mplayer and ffmpeg I can do it all - I just need something to invoke them at the right time with the right parameters.

[Jeff250]

It sounds like you'll need at least two components, the Web application and then another daemon to poll the database and to handle launching mplayer/ffmpeg instances.

If you don't want a separate database server, you can use sqlite, but whatever you use for storage, you'll want something that's fast and robust to concurrent access. Don't roll your own storage backend. Video files themselves you can and should be kept directly on the file system though.

For the Web application stack, apache + modwsgi / sqlite / python + django should be a good fit. For the daemon, python again, using this code to daemonize:
http://code.activestate.com/recipes/278 ... ython-way/

You could also substitute apache with something more lightweight if desired.

If you think that the actual capture and encoding is the *easy* part in this project, then I don't envision anything else being too terribly difficult (not to say that unexpected problems won't come up), but even best case it'll still take a lot of time to bang all of this out.

[snoopy]

Nice. Thanks.

I'm probably going to sit on this for a while. I'm in school for another couple months, and I shouldn't start a project like this until I'm out of that.

In the mean time, I'm going to think about it a bit more. In a sense, I'll be duplicating a lot of code... you could almost call this a fork from mythtv (If I do it) with slightly different goals... mostly just less grandiose goals... but I guess this is how the billion little projects that are out there come into being.

I think that when (if?) I do tackle this I'll start by talking to someone at mythtv and get their okay to adapt their code to my purposes.

[EDIT] If I wanted to allow the project to grow, I'd want to try to account for the coordination of multiple capture cards... even across multiple machines... which in turn might require me to move back into the heavier client/server database model, meaning probably mySQL. Either that, or I guess I could use something like SSH to issue commands to other machines.

[Jeff250]

Yeah, multimachine would require a database server like MySQL.

Technically, you don't need permission from the MythTV developers to fork their project, but it's still a good idea to hit them up on pointers on how to start out whatever changes you want to make, *especially* if you want to maintain the possibility of the changes ever being merged. Legally though, if you only use this software for your own personal use, then you're under no obligations, and if you copy/distribute it, then all you have to do is respect MythTV's open source copyright license.

[snoopy]

I just downloaded webvcr+.

While it's old, it also looks like it accomplishes a good bit of of what I'm looking for. It may actually make more sense to see if I can take over the project and just give it some sprucing up. At first glance, it appears to accomplish the needed things without a backend... more investigating to come...

[snoopy]

I looked at webVCR+. It might have a few (small) things that I'll want to borrow, but for the most part not.

1. It handled configuration on the web interface. I purposely want to make all of the configuration happen in a text file, because I specifically don't want people with access to the web interface to have control over the program's settings, I only want them to have control over usage. I.E. I want to let them schedule recordings, but I don't want them to be able to say what command accomplishes a recording.

2. It handled the backend portion by scheduling a cron process to run every minute. I want to have an actual process running that the user can kill/start/stop/etc.

3. It choked on the downloading the listings (with good reason... I could address this, but already had some stuff against the program)

It's got the basic idea of what I want to do, but the implementation isn't on the level of what I want it to be.

Jeff, you seem to indicate that multimachine would require a database server. I'm not sure that it'd be absolutely necessary, though maybe it'd be the smart way to do it: I think, if you had ssh & your eternal programs properly configured, you could make your "master" machine go out over ssh and directly invoke the other machine to record & then copy the recorded material over.

The advantages to this configuration: I could use SQlite instead of needing a client/server architecture (lighter weight). The slave machines wouldn't need to run my daemon

The disadvantages: The remote machine needs to be properly configured (could be handled by a script). SSH would need to be configured and running on all machines. The master machine would need to ping the slave machines on a regular basis to verify that they were still available to be controlled.

The best option might actually be a hybrid solution where the slave machines have their own deamon & record command configuration and are configured to listen to some port for instructions. The master machine would then be configured to query the slave.. the slave would respond with its recording capabilities & from there the master would regularly ping the slave for presence & hand it recording & encoding tasks as necessary.

I'll have to think about storage a bit more, too. Distributed storage, where the slave machines store their own recordings would probably mean more storage space, but would also make accessing the data a bit more painful. The idea is that my program will dump the re-encoded video files and be done, so it probably makes the most sense to have it to use a central storage location.

[snoopy]

One more question/thought:

Mythweb is written in php. Am I really going to want to port it over to python? Maybe the web stack should stay with php, especially if anything that I'm going to do is going to be possible to contribute to mythweb... what do you think?

[Jeff250]

Yeah, using MythTV's language is definitely better if you want to fork it. (And php isn't as icky as it used to be...)

eff, you seem to indicate that multimachine would require a database server. I'm not sure that it'd be absolutely necessary, though maybe it'd be the smart way to do it: I think, if you had ssh & your eternal programs properly configured, you could make your "master" machine go out over ssh and directly invoke the other machine to record & then copy the recorded material over.

Copying the file over is half of it, but then you would want some way of letting your application "know" that the file was done and copied over. Adding (or updating) a row in a database pointing to the file (and containing other metadata) is a very natural way of doing this.

[snoopy]

(So much for not working on this)

I'm trying to figure a way to avoid mySQL and use SQlite while still maintaining some sort of a slave process on remote machines.

I think I have an idea, I just don't know how heavy/slow/ineffective it would be:

My idea is to have each slave request a few specific pages from the web server periodically.

1. Each slave can request a "ping" page - essentially telling web server that it's still there & based on the server's response, it can then:
2. Request a "schedule" page - by which means the master passes a task list to the slave
3. Request a "start action" page when it starts something on the schedule
4. Request a "end action" page when it finishes doing the action
5. Request an "upload file" page to initiate a file transfer up to the master.


It'd probably be safe to have 1. happen as infrequently as every 15 minutes or so, and the interval could be configured. All of the others would only happen when they needed to, since the "ping" would queue the schedule transfer and the schedule would queue the rest.

[Jeff250]

So you basically want the daemon to get information from the database via HTTP GET's and make updates to the database via HTTP POST's. There'd be overhead doing this versus the daemon getting information from and making updates to the database directly, but not enough to be an issue. This means more work for you too, since you still have to write the code to update the database but now also extra code to handle the corresponding HTTP requests.

Are you still planning on forking MythTV? I think your top concern should be whether MythTV can be easily reshaped to work like this (and if it can't, whether you want to start from scratch).

[snoopy]

I don't think so anymore.

Freevo has a python web server implementation from which I might borrow some code, but a lot of this is going to turn into a re-write. I also may borrow tidbits from mythweb, but probably not more than very small snippets of code.

I've got ideas about how I want the web interface to look and feel, which differs some from both mythtv and freevo. I never actually managed to get the freevo web interface going, so I don't know how it really looked and felt outside of screenshots available on the web. My concept is to have everything built around a single page to which you add filters to parse down results.

I created a project page: http://sourceforge.net/projects/yawpvr/ ... =directory. There's an initial commit on the SVN with some ideas/plans. As things look, the "web interface" is expected to really be the guts of the system (though maybe it should be more properly segregated into a portion that serves up pages and a portion that acts as a scheduling master), and the slaves aren't really going to do *all that much.* The idea at this point is that the slaves will be handed literally command lines to run at certain times, and that all of the coordination, parsing, etc. will be handled for them.

I'm also trying to plan this to operate in a very generic manner.... so it could be adapted to operate in environments other than TV scheduling. The basic structure that I'm trying to build is a task master that reads a schedule and hands out tasks based on desired actions to be done to that schedule.

[fliptw]

The basic structure that I'm trying to build is a task master that reads a schedule and hands out tasks based on desired actions to be done to that schedule.

so... a web interface to a task scheduler, like cron?

[snoopy]

Yes.... except that it needs to be able to automatically pull listings from a source, schedule favorites based on the listings, handle conflict resolution for the favorite scheduling, and handle multiple slave machines.

The slaves *could* be run via cron, but I think I'd rather make it run its own process.... programs that mess with cron kinda annoy me because they're a lot more annoying to *really* stop. I want the user to be able to kill the process and have it stay dead until they want to bring it back. (I've seen offending programs that don't even bother to clean up after themselves in cron when you uninstall)

Also, I'd like this to be as close to cross-platform as possible & hooking into windows scheduler as opposed to cron would mean separate code while a python process could be run on either platform without as many problems.

[fliptw]

You basically making a task scheduler for a single resource - anything wider in scope(re: more generalized) and it does become a scheduler like cron with a fancier interface.

If you want it to be cross platform, then you are pretty much required to write code for each platform - the true key is proper modularization and abstraction.

that being said, task schedulers are a well-solved problem, using them in your project is a good idea.

[snoopy]

It looks like http://packages.python.org/APScheduler/ may be a useful piece.

[Jeff250]

That could be your best bet. Cron has a lot of warts.

[snoopy]

Posted a new commit..... One record module is mostly written...

If you have linux, a DVB card, and a few dependencies you can play around with it a bit.

[snoopy]

Alright, I have a question. I have made some pretty good progress, but I'm still trying to _fully_ get my head around classes and such. I think maybe I have a case where a class would be appropriate, but I guess I haven't done it because I don't understand them well enough yet. More specifically, I don't quite get the whole "self" part yet, anyways here's my module:

Code: Select all

import subprocess
import sys
import logging
import os
import re
import shutil
import threading

# This is a module for YAWDVR
# These modules should contain number of specific elements, as seen in
# this file.
# Make sure that this file follows proper python syntax.
# This module runs on each slave with parameters passed by the server.

######
# The action may be described by an ID string.
# This is the ID for the action that will be seen in the web interface.
# The default value is the filename.
# This matters for time-dependent actions. Actions with the same
# ID and parameter will always be conflicting.
# "parameter" is an interger passed to functions in the module
# that repesent the different ways that this module can be used.
# The meaning of the parameter needs to be defined in the init
# function.  An example: in this module, each parameter will be
# associated with a channel file and a DVB device number.
# The first parameter will always be 0.

ID = 'DVB Record'

# The type can either be 'time-dependent' or 'time-independent'
# Time dependant scripts will be started at a specific time and will
# carry a duration parameter, So they end at a specific time.
# They typically will not receive an input file to operate upon, but
# will produce an output file.

# Time independent scripts are run at a convenient time.  They typically
# receive an input file to operate upon, and will either modify the
# file or produce different output file(s).
# The webserver/backend will associate one time-dependant and multiple
# time-independant action, run in a chain.

# Default value = 'time-independent'

type = 'time-dependent'

# The above are two things that the server needs to know about this action.

# Things that don't need to pass on to the server

# Initialize some variables that the functions will use 
extprog = 'mplayer'

checkopt = '-list-options'

checkstr = 'dvb'

baseopt = ' -dumpstream'


# Init function for the action.  This function is run on the
# Slave machine and needs to accomplish a few main things:
#
# First, it needs to do whatever is necessary to make the run
# function work.  For example: this module needs there to be
# a channels.conf file for mplayer to know how to find channels.
# The initialization function creates one of these tables per
# slave parameter
#
# Second, it needs give the server certain information
# One thing should be sources associated.
# Another thing should be a specific listings source.
# Also, the server needs to know the type and ID.
# exiting with -1 as a return means that it failed.

def init(parameter, interactive):
    '''Initiate mplayer dvb record parameters'''
    # Check to make sure that mplayer is on the machine, compiled with
    # DVB Support
    p = subprocess.Popen([extprog, checkopt], stdout=subprocess.PIPE)
    stdout = p.stdout.read()
    if stdout.find(checkstr) == -1:
        logging.info(extprog + ' command with dvb support not found') 
        return(1)

    # Run a dvb scan and output to ~/.mplayer/channels.conf.$parameter$
    # I'm making this run each time this function is run, since
    # one of the reasons for runnings this may be channel
    # problems.
    #TODO: This is totally linux-specific.  Make this work
    #TODO cross-platform.
    #TODO code to pull the table from the config file or
    # query the user for to location of the file, hinting
    # so they can get it right.
    #
    #TODO: remove this hard-code once above is done.
    #TODO: note that this needs to be set per parameter
    freqtable = '/usr/share/dvb/atsc/us-ATSC-center-frequencies-8VSB'
    
    conf = os.path.expanduser('~/.mplayer/channels.conf.') + str(parameter)
    
    #TODO: update this to handle multiple parameters.
    #TODO: The adapter/frontend/demux needs to be set
    #TODO: per parameter, not hard-coded.
    adapter = '0'
    frontend = '0'
    demux = '0'
    
    #TODO Update this so it isn't just hard-coded so it's cross-platform
    scancmd = 'scan'
    # Perform the scan and dump the conf file.
    logging.info('scanning channels using' + freqtable)
    print('scanning channels using' + freqtable)
    print('(This will take a few minutes.)')

    scan = subprocess.Popen(
                            [scancmd, '-a ' + adapter, '-f ' + frontend,
                             '-d ' + demux, freqtable, '>'],
                             stdout=subprocess.PIPE,
                             stderr=subprocess.PIPE)
    chanstr = scan.stdout.read()
    chaninfo = scan.stderr.read()

    # By convention, we are going to define the channels by
    # number.  This provides a consistent string that can
    # be used in both the listings and mplayer. To accomplish
    # this, the channel info in the conf file needs to be
    # re-mapped.

    if chanstr != '':
        chans = {} 
        # Parse the stderr output to create the channel list
        inflines = chaninfo.split('\n')
        for l in inflines:
            if 'Channel number' in l:
                # Find the index of the channel number and channel name
                numidx = l.split().index('number:') + 1
                nameidx = l.split().index('Name:') + 1
                # Grab the channel number
                chan = l.split() [numidx].strip('.')
                # Change the : in the channel name to a -
                chan = chan.replace(':','-')
                # Grab the channel name
                name = ' '.join(l.split() [nameidx:])
                chans [chan] = name.strip('\'')
        
        # Rewrite chanstr to use the channel numbers rather than names.
        # This relies on matching channel names.
        # Note that if multiple channels have the same name, they will
        # Be matched in order - 1st with 1st 2nd with 2nd, etc.
        conflines = chanstr.split('\n')
        confout = []
        # Iterate through the channel list generated above 
        for k, v in chans.iteritems():
            # Iterate through the items in the conf file string
            for l in conflines:
                # If the two match (for the first match)
                if v in l:
                    line = l.split(':')
                    # Change the conf's channel name to the channel num
                    line[0] = k
                    # Add it to a list of valid conf lines
                    confout.append(':'.join(line))
                    # Consume the old conf line
                    conflines.remove(l)
                    break

        confstr = '\n'.join(confout)

        # Write the channel conf file
        with open(conf, 'w') as chanconf:
            chanconf.write(confstr)
            logging.info('channels found and written to' +  conf)
            print('channels found and written to' +  conf) 
    else:
        logging.info('no channels returned by scan, using existing file')
        logging.info('scan return:' + chaninfo)
        print('no channels returned by scan, using existing file')
        print('scan return:' + chaninfo)

    # Preparing to return the known channel numbers to the server.
    chanlist = []
    try:
        with open(conf, 'r') as chanconf:
            for line in chanconf:
                chanlist.append(line.split(':')[0])
    except(IOError):
         logging.info('Conf file does not exist. Make sure that you have specified')
         logging.info('the correct device and that it is accessible to the user')
         print('Conf file does not exist. Make sure that you have specified')
         print('the correct device and that it is accessible to the user')


    #TODO:
    #The function also needs to return the appropriate listings
    #module.  The listings module will:
    #1. have an init function to also make sure it can run.
    #2. have a match function that will match names returned
    #   by this module to the appropriate listings.
    #3. have a grab function that will actually go out and get
    #   the listings.
    #This listings module will only run on the server.
    listings = 'listings.xmltv'
    #TODO this shouldn't be hard-coded. The user should pick
    # from valid options.
    
    # Create a dictionary of items to return to the server
    serverret = {__name__ + '.' + str(parameter) + '.ID': ID,
                 __name__ + '.' + str(parameter) + '.type': type,
                 __name__ + '.' + str(parameter) + '.listings': listings,
                 __name__ + '.' + str(parameter) + '.sources': chanlist}
    
    return(serverret)


# The main run function follows.
# This is the function that will be invoked the majority of the time.
# The web server will pass details of invoking this to the slave
# The slave will in turn invoke this function to perform the recording.
# Note: it is up to the slave to invoke this when the asked by the
# server, and correctly calculate the necessary duration. 

# The function should return an indication of success or failure
# 0 indicates success, -1 indicates failure.

def run(source, duration, parameter, outfile):
    '''Dump a DVB stream using mplayer to a TS file'''
    #TODO cd to savedir so the file dumps in the correct place
    # Mplayer doesn't support the conf filename option, so copy the appropriate
    # parameter conf file to channels.conf
    conf = os.path.expanduser('~/.mplayer/channels.conf.') + str(parameter)
    dest = os.path.expanduser('~/.mplayer/channels.conf')
    shutil.copy(conf,dest)

    # Set up the command line for mplayer
    dvbopt = ' -dvbin card=' + str(parameter)
    outopt = ' -dumpfile ' + outfile
    channel = ' dvb://' + source
    runcmd = extprog + baseopt + dvbopt + outopt + channel
    logging.info('Starting Mplayer DVB record of channel ' + source)
    logging.debug(runcmd)
    print(runcmd)

    # Mplayer doesn't stop when using the -endpos option
    # Instead, use threading to spawn mplayer, and then kill
    # it after duration. duration should be specified in seconds.

    class Command(object):
        def __init__(self, cmd):
            self.cmd = cmd
            self.process = None

        def go(self, timeout):
            def target():
                self.process = subprocess.Popen(self.cmd.split())
                self.process.communicate()

            thread = threading.Thread(target=target)
            thread.start()

            thread.join(timeout)
            if thread.is_alive():
                self.process.terminate()
                thread.join()

    command = Command(runcmd)
    command.go(timeout=duration)

    logging.info('Finished Mplayer DVB record')
    
    # Check to see if the file exists and isn't empty
    if os.path.isfile(outfile) and os.stat(outfile).st_size != 0:
        logging.debug('record appears to have succeeded, file size ' +
                      str(os.stat(outfile).st_size) + 'bytes')
        return(0)
    else:
        logging.debug('Problem recording detected')
        return(-1)

With my limited testing, it's worked.

Now, how it'll be used: The init function will only be run while everything's being set up, so when the program is started with the 'setup' flag. The run function will be invoked at the appropriate time for it to record. Since the run function "consumes" the system resource of the capture card, it can only have one instance running per card - so on my PC that has only one capture card, it'll only run one at a time, with 'parameter' set to one. Some people have multiple tuner cards in their PC's, though - mplayer supports up to 4. So, theoretically, up to four instances of this function could run simultaneously, each with its own 'parameter' (1, 2, 3, 4).

Does it make sense to make this into a class? I'm thinking that maybe it does, but I want your ideas. (Maybe I'm just avoiding having to really sit down and get my head around class implementation.)

Also, the threading portion in the run function is a copy from something I found online that I don't completely understand... I'm thinking that maybe it's got too much for my application, and I don't think that I really need to implement it as a class, but I need to research threading and such to understand what I need to do. Would you guys simplify the code there, or just leave it as a class?

[Jeff250]

The Command class looks like something that could be combined into its own function, as the way you're using it here the class part isn't useful, since you never do anything like calling go() multiple times on the same instance.

What were you thinking of turning into a class? Maybe a CaptureCard class that kept track of capture card information with its own init/run/etc. methods?

In general, the code looks good. Some suggestions:
You concatenate runcmd together into a single string just to split it later. It seems simpler to just keep it as a list of strings the entire time. And safer: what happens if, e.g., outfile has spaces in its path?
Instead of os.stat to get file size, you can just use os.path.getsize().
Some showing of an "accent": return(foo) -> return foo, except(Exception) -> except Exception.

Looks like you're making steady progress.

Don't feel like you need to commit to anything. It's nice to have the complete foresight of what should be a class/function/etc. in the beginning, but it's common in a project to have to "refactor" the code throughout its development into (or out of) classes/functions/etc. once things become more clear.

[Jeff250]

I'm also afraid of mplayer possibly swallowing the SIGTERM signal and not terminating (due to a bug or whatever) that you might not want to take down your service. You might want a timeout on the second .join() too that then calls .kill() on the process if the timeout is hit (i.e., if the thread is still alive).

[snoopy]

Some showing of an "accent": return(foo) -> return foo, except(Exception) -> except Exception.

I don't quite understand what you're getting at.

I'll rework the command class. I'm going to have to give it some thought/consideration/etc. as I go.... I haven't gotten to the core/daemonized part of the system yet... and I haven't taken a good look into the task scheduler yet either.... It may make sense to create a utility function/class that serves not only the module above, but also other modules that generally are going to need to start some long-lasting task, move on to other things, and then come back and either stop & kill the task, or just eventually wait for a return from the task when it decides that it's finished. From the 5 minutes of reading that I've done on threading, I think I'll probably go for the thread-safe queue method, once I figure out what the method is.

I think I'm finding that a functional style of programming comes more easily than an object-oriented style. It's probably a matter of getting more used to the object oriented style, though.

I hear you on the stop signal from mplayer & killing it. I will implement, but think I'll put a couple's second's worth of delay between the two. Currently, mplayer receives the signal and responds to it, but does a couple things before it actually dies....

[Jeff250]

Or maybe even more than two seconds if you think the cleanup might ever take longer than that.

If you're still looking at APScheduler, I would try to get as much functionality for free from that as possible.

I don't quite understand what you're getting at.

Here I just mean you can remove the extraneous parens. These sorts of idiosyncrasies are often called "accents" because they are often caused by someone coming from a language where the idiosyncrasy, e.g., the parens, were required there. It's obviously not a big deal though.

[snoopy]

I am looking at APScheduler... one of my litmus tests for the "maturity" or "quality" of certain dependencies is whether it's in the official Arch repositories or if you have to go get it from AUR... APScheduler is in the AUR so I was wondering about avoiding it. The more I think about it though, I think that I can realistically make the recording/encoding backend consist of passing scheduled tasks to AP Scheduler:

Populate AP Scheduler with actions for the machine to do,
Also, pass it a "reschedule" task that makes any necessary changes to the existing tasks scheduled for some interval. I'd make the interval configurable, though if you schedule things ahead of time, you shouldn't need an interval of anything less then something like 15 minutes.

In terms of timing and error catching:
I'd have any idle slaves "check in" with the server some 5-10 seconds after another slave was supposed to start recording... that gives the tasked slave a chance to fail, report its failure to the server, have the server re-shuffle things, and then pass the task to a different slave.

I'm also planning on using XMLtv which is in the Arch AUR...

[Jeff250]

I've never used APScheduler, but I think being hosted on python.org means it can't be too shoddy. If you do use it, please let us know what you think of it.

[snoopy]

Will do.

I think I really (maybe) get the class thing now.

I did a bunch of work on that module tonight, to implement a bunch of my todo stuff. In the process, I thought that it'd be nice to be able to run some of the init stuff simultaneously - the channel scans take a good 3-4 minutes... if I'm running them in serial, assuming that I have the maximum of four, it's going to take painfully long. When I started about what's involved in each scan, there are a lot of unique things: Each scan gets its own device name, its own output conf file, its own frequency table.... and then I understood why to make classes. By making a scan class, I can create four different objects that each internally keep track of all of their own details. Once I've got my four objects, I can loop through them starting them up & splitting them into their own threads, and then loop through waiting for them all to finish before moving on with the script.

It was a beautiful moment.

Now I just need to finish the updating of the function code, and get myself another/more DVB cards so I can genuinely test this with more than one parameter.

This particular module is turning into quite a bit bigger of a beast than I was expecting it to. I was hoping that it'd be a sort of "template" for other people to imitate to implement using other external programs.

I think I may very well split it into a few modules & move some stuff (such as the threading) over to another module, so I can keep the external program implementation as simple as possible, giving others a better chance to adding/adapting it for their own personal needs.

[snoopy]

I tried reworking the Command portion.

As far as I can tell, it needs to be a class. The process object needs to be accessible outside of the target function - so that it can be sent the terminate signal. If the object is created outside of the target function, Popen immediately invokes it, so then you don't get a chance to thread it.... so you have to create it as a class, so you can create it when you split off the thread, and still access it from the main thread.

I guess you'd call it an inheritance issue?

[Jeff250]

Oh, I didn't anticipate that issue. That makes the class approach look more attractive. Or maybe something like this?

Code: Select all

def run(cmd, timeout):
    process = subprocess.Popen(cmd.split())

    thread = threading.Thread(target=lambda: process.communicate())
    thread.start()

    thread.join(timeout)
    if thread.is_alive():
        process.terminate()
        thread.join()

[snoopy]

I'll give it a test.

I need to give this another look anyways...

I was getting weird behavior earlier, and didn't understand it (for a bit) but I think that I now do... which may render all of this moot.

When I was trying to rework this, I was trying to invoke mplayer via subprocess.Popen, and I was using the default stdout=None, stderr=None.... and it kept giving me these weird results where it was indicating failure immediately... I finally solved the issue by going back to the class, and adding a process.communicate().

Anyways, all that to say - I think the subprocess.Popen may already thread out, and the communicate method is what actually pulls it back to the main thread. Likewise, if you pipe one of the outputs, it starts the external thread... but the external thread pauses until you start to read the pipe... essentially delaying the main thread for the duration that the external program is open.

I need to try just doing this:

Code: Select all

process = subprocess.Popen([cmd])
sleep duration
process.terminate()
sleep 3
try:
    process.kill()
except:
    logging.debug('process already dead')

Or something of the sort. (I need to check subprocess' documentation, maybe there's method to check if a object is still running.)

[Jeff250]

You can check if process.poll() is None.