DescentBB

Does anyone know of a free (open-source would be nice) compression/archive format that would allow me to decompress from memory rather than just from disk? I need to be able to decompress small pieces of the archive at a time. It seems that zlib requires the entire archive to be in memory at once, not just a piece of it; bzip2 is like gzip in that it only compresses one file; RAR and ACE have free decompression libraries but don't support decompressing from memory, only from disk.

This is for a network-based program in which the client downloads a rather large (1000 MB uncompressed, 400 MB when supercompressed with WinRAR) directory structure from a server in small chunks, decompressing one while downloading the next. I need a format that will allow the program to do just that.

Actually, every one you listed should be decompressable in parts, and from memory... investigate the APIs more. Oh, and ZLib is almost the same thing as GZip, you need to couple either with an archive format of some kind to store multiple files. Either make up your own (filename+length should suffice, perhaps keep the table at the beginning of the file instead) or use something simple and standard like TAR, which should let you unpack-as-you-go easily.

You might also take a peek at www.7-zip.org - free opensource archiving system. I haven't played too much with it, but it shows promise... especially check out http://www.7-zip.org/sdk.html

I was actually poking around SourceForge and came upon 7-Zip. I downloaded the source and was very confused by the way it works... ten tons of C++ classes. Hopefully this LZMA SDK will give me what I need in a much simpler package.

ZLib is great, been using it for years. Actually I wrote a set of codec classes and one of them uses ZLib under the hood for doing just that ... compress/decompress from memory.