I created a small batch file that calls kzip and 7zip with parameters set for maximum compression (I hope). The final zip is then mixed using zipmix.
May be it will be useful to people on this forum, and your comments are welcome.
I also wish to thank Ken Silverman for his nice programs (and games :-) )
@ECHO OFF
REM Programs' settings. Change the pathes to matche your installation.
REM kzip parameters:
set kBinary=kzip.exe
set kRecursive=/r
REM set kVerbose=/v
REM set kBlockSplit=/b128
set zipmix=zipmix.exe
REM 7zip parameters
set sBinary=7za.exe
set sCompressionLvl=9
set sPasses=15
set sFastBytes=258
REM checking command line parameters
if a%1==a goto help
if b%2==b goto help
REM compressing files
%kBinary% %kRecursive% %kVerbose% %kBlockSplit% %TEMP%\%1k *.*
if not errorlevel 0 goto kzipError
%sBinary% a -tzip -mx=%sCompressionLvl% -mpass=%sPasses% -mfb=%sFastBytes% %TEMP%\%17.zip -r %2\*
if not errorlevel 0 goto 7zipError
%zipmix% %TEMP%\%1k %TEMP%\%17 %1
if not errorlevel 0 goto zipmixError
REM cleaning up
DEL %TEMP%\%1k.zip %TEMP%\%17.zip
pause
goto End
:kzipError
echo kzip exited with errorlevel %errorlevel%
goto :end
:7zipError
echo 7zip exited with errorlevel %errorlevel%
goto end
:mixzipError
echo mixzip exited with errorlevel %errorlevel%
goto end
:help
echo Usage:
echo %~nx0 ^<archive_name^> ^<directory_to_compress^>
goto End
:end
krick at
You should check out ZipMax (at the bottom of this page)...
http://www.clrmame.com/download.htm
ZipMax is a windows application that will re-compress a zip file with multiple command line zip archivers (up to 255, I think) and then build a new zip file using the best compressed parts from each archive.
This is the relevant part of my current zipmax.ini that makes 5 passes using different kzip settings, then merges them into one...
I've run lots of tests using pretty much every other zip compatible compressor I can get my hands on and kzip is the best 99%+ of the time.
You can sometimes get a few bytes more compression using BJWFlate (available on the same download page as ZipMax) but it is SLOOOOOOOOOOW.
DeflOpt is useful to run as a final step as it often shaves of a few more bytes and doesn't take long to run.
Let me know if you need some more info or sample ini files for other compressors.
fred01 at
Thank you, very interesting tool indeed. I will try it this weekend :-)
You say that kzip yields the best compression 99% of the time vs. other
krick at
fred01 said
Thank you, very interesting tool indeed. I will try it this weekend :-)
You say that kzip yields the best compression 99% of the time vs. other
fred01 at
krick said
I even compiled my own custom version of 7Zip that allow maximum compression settings: -mfb=258 -mpass=255
WoW! :shock: Which version?
krick said
However, after doing extensive testing, running 5 passes of KZIP usually gives the best results in the least time.
I have a lot of zipmax.ini files for other compressors that I can share if you're interested in playing.
I copied your settings for kzip and testing them now. From what you say I wonder i there is any use to test 7zip with options lower than -mpass=15 -mfb=258, or any other compressor (aside BJWFlate :roll: ).
krick said
ZipMax has a logging feature so you can see which compressor was the best for each file in the archive.
Yes, very nice tool indeed. It can even run DeflOpt on every zip before merging. As of now, I only miss two features: ability to specify the temp folder (would like to use windows default at least) and drop folders for compression in addition to zip files.
I often have to compress entire directory trees and would like to skip the initial zip creation step, as this just creates a useless compress-decompress action. May be I should try to ask the author about that.
krick said
Also, for further reading, here's an old thread on the BYOAC forums...
I even compiled my own custom version of 7Zip that allow maximum compression settings: -mfb=258 -mpass=255
WoW! :shock: Which version?
Version 3.12, just the command line exe, not the whole GUI version.
I compiled it back in Sept of 2004.
You can grab it here if you're interested in playing with it...
http://3feetunder.com/files/7za_custom.exe
The changes that I made were suggested by Ben Jos Walbeehm on this page...
http://www.clrmame.com/bjwflate.htm
fred01 said
I copied your settings for kzip and testing them now. From what you say I wonder i there is any use to test 7zip with options lower than -mpass=15 -mfb=258, or any other compressor (aside BJWFlate :roll: ).
I'm not sure. Like I said, I did a LOT of testing. I re-compressed an entire MAME rom set (which, at the time, was nearly 6000 zip files). I ran a gauntlet of about 15 compressors, including some old versions of PKZIP that couldn't even handle long filenames. I think it took over 2 months to finish, running 24 hours a day on a P4 3GHz machine with 2GB of memory. Of course, the majority of the time was eaten up by BJWFlate and 7Zip running with extreme settings.
After I was done, I discovered KZIP so I recompressed the whole batch a second time using just KZIP with the 5 passes that I posted above. Would you believe that KZIP was still able to make almost every zip smaller?!? Plus, it only took a couple of days instead of months. I don't even bother with the other compressors anymore. I don't see the point.
fred01 said
Yes, very nice tool indeed. It can even run DeflOpt on every zip before merging.
I've never tried that. I only run it as the final step on the merged zip.
How do you run it on every zip?
fred01 said
As of now, I only miss two features: ability to specify the temp folder (would like to use windows default at least) and drop folders for compression in addition to zip files.
I often have to compress entire directory trees and would like to skip the initial zip creation step, as this just creates a useless compress-decompress action. May be I should try to ask the author about that.
I wouldn't bother bugging Roman about ZipMax. He doesn't want to support it any more. I was working with a guy on the BYOAC forums to develop a version that would take advantage of hyperthreading and multi-core CPUs but Roman didn't want anything to do with it. I think CLRMAMEPRO keeps him plenty busy.
You can download the latest source and play with it yourself if you like. It shouldn't be too hard to read the temp directory from the ini file. Not sure about dragging and dropping directory trees. Seems like a lot of work for not much gain. There's already other tools that do that job much better. In fact, why not compress your directory trees using the "no compression" setting first, that should be faster. Then drop them on ZipMax.
If you make any changes to the source, let me know.
fred01 at
I was trying to make a quoted answer, but there ware just to many interconnected things, so I settled to do it
krick at
fred01 said
...zipmax support was not the priority of the author. I have seen that some other people played with the source. Have their and your changes been ever integrated into a source (a beta)? I understand that a real multithreading support is not trivial (as I read people were trying to launch several copies of zipmax from one path but on different machines to maximize the speed). On the other hand, just launching each compressor in its own thread on one machine should be manageable. The number of thread can be probably determined via some system call and/or be forced from the ini file. Of course one should not start more threads than CPUs.
Also all this theory look nice on paper, but I'm not a C/C++ programmer and so I will not risk myself in to the source code. :? I will rather follow your suggestion and write a batch file that will store the directory into a zip w/o compression and then run zipmax on it (fortunately for me, it accepts command line parameters :-) )
The code to run multiple machines on a shared folder of zips across a network (a ZIP farm?) is already in the version available on the CLRMAMEPRO website. It was actually a somewhat minor change to the code.
As for true multi-threading on a single machine, there have been a few experiments done resulting in a beta executable or two that were never distributed.
The guy doing most of the work on the multi-threaded version added a class to the ZipMax source that queued up zipping jobs and handed them out as processors became available. He had an INI setting that you could modify based on the number of CPUs (or virtual hyperthreading CPUs) that you had.
On the other hand, I experimented with another approach where the CPUs were treated as a limited resource. I just spun off all the threads at once and had them immedately go to sleep. Then after a certain amount of time, the threads each wake up, check to see if a CPU is available and go back to sleep if there isn't one available.
Eventually, if I get some time, maybe I'll dig up the source from those aborted experiments and see what I can do with it.
fred01 said
krick said
How do you run it on every zip?
The reason is explained on the http://www.clrmame.com/bjwflate.htm (Compression sizes, second paragraph). As I understood, DeflOpt may sometimes
fred01 at
krick said
Eventually, if I get some time, maybe I'll dig up the source from those aborted experiments and see what I can do with it.
Interesting. :-) Please tell you you make it available for download and testing.
From your description of the threading method, I was wandering if it would make sense for a terminating thread to send some sort of
Zardalu at
fred01 said
Also, for further reading, here's an old thread on the BYOAC forums...
I'm sorry, but that is wrong. Nobody found my pages on archive.org. I decided to remove my pages. And then Roman e-mailed me. And I sent him my pages through e-mail. I gave him permission to use my pages. And I gave him permission to host them.
Yes, I might get into a "tiff", but Roman got those pages from me, not from anyone else.
Ben Jos.
Zardalu at
krick said
http://www.clrmame.com/bjwflate.htm
Sorry, krick, but you were wrong... on that other forum... I deleted those pages myself... no on found them again... Roman asked me... and I gave him my pages...
I lost interest... in MAME, BJWFlate, ZipMax, and so on... I spent well over a YEAR zipmaxing things. On my own 3, 4, 5, more computers... on a lot of others.
Yes, I deleted them because of some... hmm... "tiffs"?... but nobody used archive.org to get them back. I sent my pages directly to Roman and he agreed to host them even if I didn't want to host them anymore.
fred01 at
Zardalu said
I'm sorry, but that is wrong. Nobody found my pages on archive.org. I decided to remove my pages. And then Roman e-mailed me. And I sent him my pages through e-mail. I gave him permission to use my pages. And I gave him permission to host them.
Yes, I might get into a "tiff", but Roman got those pages from me, not from anyone else.
Ben Jos.
Thank you for this clarification, and thank you even more for coming back to work on DeflOpt as you say the other thread: http://jonof.edgenetwork.org/forum/viewtopic.php?p=6174#6174
Will be patently, but eagerly expecting the new version.
krick at
Zardalu said
Sorry, krick, but you were wrong... on that other forum... I deleted those pages myself... no on found them again... Roman asked me... and I gave him my pages...
... but nobody used archive.org to get them back. I sent my pages directly to Roman and he agreed to host them even if I didn't want to host them anymore.
Actually, I did find your site on archive.org. I found it and posted it on the CLRMAME forums. Here's my post from 09/30/2004...
I'm guessing that my bringing the page to his attention prompted Roman to contact you and ask for your page so he could host it instead of linking to the archive.org page.
Zardalu at
I think by then, Roman had already given up on ZipMax. Pretty much everything that had been added to ZipMax after I discovered it had been added by me. Because ZipMax was a GREAT program (and still is), but it just lacked a few more things I wanted from it. And then a few more. So I added all those. I had a ZipMax.ini with more than 200 entries. I wrote DeflOpt and needed the functionality of running DeflOpt on something BEFORE things were merged back into one zip file.
Actually, the impression I got when I contacted Roman the first time about ZipMax was that he had already lost interest then. So I developed it to a stage where I was happy with it. Added logging, post steps, increased the maximum, and so on. Interesting that you mentioned uppercase and long filenames... my own ZipMax.ini called programs to deal with those. PKZIP 2.04g... 8+3 filenames... I wrote a program to deal with that. To be able to use those old 8.3 programs and change the filenames inside those to long filenames. Frontends. Frontends too for GUI only programs. To make them usable for command line programs. Frontends that would fire up the GUI, then hit all the buttons and fill in all the fields automatically so that they would work like command line programs...
I could release those programs if anyone is still interested. Wrote them for my own use, but then lost interest in MAME...
Ben Jos.
krick at
Zardalu said
I think by then, Roman had already given up on ZipMax. Pretty much everything that had been added to ZipMax after I discovered it had been added by me. Because ZipMax was a GREAT program (and still is), but it just lacked a few more things I wanted from it. And then a few more. So I added all those. I had a ZipMax.ini with more than 200 entries. I wrote DeflOpt and needed the functionality of running DeflOpt on something BEFORE things were merged back into one zip file.
Actually, the impression I got when I contacted Roman the first time about ZipMax was that he had already lost interest then. So I developed it to a stage where I was happy with it. Added logging, post steps, increased the maximum, and so on. Interesting that you mentioned uppercase and long filenames... my own ZipMax.ini called programs to deal with those. PKZIP 2.04g... 8+3 filenames... I wrote a program to deal with that. To be able to use those old 8.3 programs and change the filenames inside those to long filenames. Frontends. Frontends too for GUI only programs. To make them usable for command line programs. Frontends that would fire up the GUI, then hit all the buttons and fill in all the fields automatically so that they would work like command line programs...
I could release those programs if anyone is still interested. Wrote them for my own use, but then lost interest in MAME...
Ben Jos.
Yeah, Roman is definitely not interested in ZipMax anymore. I worked with one of the users from BYOAC (MonitorGuru) and we spent some time cleaning up and simplifying the ZipMax code with intentions of making it multi-threaded. The hardest part about simplifying the code was figuring out how to treat the internal compressor in a generic way so that the external zip and internal zip code could be merged. I'm still not happy with the way it's done. I was actually thinking about having ZipMax call itself with a command line switch that made it act like an external command line zipper. This approach also has the benefit of allowing varied command line options instead of the default internal compression settings.
MonitorGuru had already added an option so that you could run multiple copies of ZipMax at the same time on different machines sharing a common pool of zip files. Then he put together a test version that actually worked pretty well on hyperthreaded/multi-processor/multi-core sytems but Roman really wasn't interested. We planning on releasing the source ourselves eventually if Roman didn't want to host it but then something happened on the BYOAC forums between MonitorGuru and Howard Casto that really pissed off MonitorGuru and he "quit the scene" as they say.
MonitorGuru's approach was to have the main thread queue up zip jobs and start them as processors became available. On the other hand, I did a few experiments where I spun off a (sleeping) thread for each job and had them wake up and check a CSemaphore object (guarding the CPU resources) to see if a CPU is available and go back to sleep if there is not.
The complicated part of my approach is pausing/aborting the queued zip jobs when the user clicks pause or closes zipmax. I never got around to working that part out but I'm thinking that I'll take a stab at it eventually. I'm thinking that I'll just have to create some global flags that the threads will have to check when they wake up. On the surface, it doesn't seem too hard.
As for the long filename issue, I wrote some code (inside zipmax) that renamed all the files (to sequential numbers) after initially unzipping the archive, ran all the external zippers, merged the best of the archives, and then attempted to rename the files in the final archive. I never did get the final rename it to work correctly and I'm not sure why. I suspect that I was using the ZipArch library wrong in some way.
In reality, external wrappers are probably the way to go but I thought it would be cool if I could get it working generically for all external zippers from within ZipMax.
I'd definitely be interested to in any utilities you've written.
Zardalu at
krick said
I'd definitely be interested to in any utilities you've written.
BJWzipFE is available from http://www.walbeehm.com/download. It was designed to make any of the old DOS zippers (using 8+3 uppercase filenames) work like they now support long filenames. Run it without any command line arguments to get a fairly detailed explanation of its usage. As for using it with ZipMax, here are two lines from an old ZipMax.ini I used a long time ago to make it work for ZipMax:
BJWzipFE is available from http://www.walbeehm.com/download. It was designed to make any of the old DOS zippers (using 8+3 uppercase filenames) work like they now support long filenames. Run it without any command line arguments to get a fairly detailed explanation of its usage. As for using it with ZipMax, here are two lines from an old ZipMax.ini I used a long time ago to make it work for ZipMax:
I've run into a problem with ZipMax and I wanted to know what you thought.
I have a mame rom zip file where one of the internal file names ends with a period. This is perfectly allowable in unix and apparently also within the zip format, however this isn't legal in windows. When zipmax unzips the archive before re-zipping, windows throws away the period at the end of the filename. When it is done zipping and goes to merge the archives, it gets an error because it cannot find the original file by name in the archives.
There needs to be some way to re-name every internal file to something short, say a sequentially increasing number, before unzipping and then rename them back when finished to avoid this problem. I had briefly looked at adding this functionality into zipmax itself but I had a hell of a time getting the ziparc library's renaming functions working without crashing or corrupting the archive.
This is the file in question...
http://files.3feetunder.com/gt103ab.zip
Zardalu at
krick said
... ends with a period. This is perfectly allowable in unix and apparently also within the zip format, however this isn't legal in windows. <...> windows throws away the period at the end of the filename.
Hmm... interesting... . I knew that Windows throws away a trailing dot. I didn't expect the mamedevs to keep an "incompatible" filename. But, then again, they don't really seem to care about any non-MAME programs that can be used for MAME-related things.
It's an interesting problem. I can think of a few solutions, but off the top of my head, I can't think of a way of solving it "elegantly" yet. I'm not familiar with the ziparc library, other than that I had to download it to compile it in when I made improvements to ZipMax. (For my own programs, I use my own zip library.) I'll give it some more thought.
BTW, I wouldn't call it "illegal". It's just a Windows quirk/feature. And I only say that half-jokingly because "we" use it all the time. We type "run" instead of "run.bat" or "run.com" or "run.exe" and so on. And Windows has more of those quirks related to file extensions. IIRC, the FindFirstFile() function in Windows also seems to think that "*.png" matches "*.png1" and "*.pngwhatever" and so on and that "*.*" matches anything without a dot in it too. And, even closer to this problem, if you want to specify a file without an extension, you have to use "*."...
In the mean time, check my ZipRen and Zip Manager programs on http://www.walbeehm.com/download. Both of those can rename files within zip files without uncompressing/recompressing.
Ben Jos.
counting_pine at
Zardalu said
And Windows has more of those quirks related to file extensions. IIRC, the FindFirstFile() function in Windows also seems to think that "*.png" matches "*.png1" and "*.pngwhatever" and so on and that "*.*" matches anything without a dot in it too.
I suspect that's because files like "thisisnotapngfile.png1" would get truncated to something like "THISIS~1.PNG" in 8.3 form. Windows tends to be happy with either form.
If this particularly annoys you, I think it might be possible to disable the filesystem using 8.3 form in the registry, but I couldn't guarantee that it wouldn't have adverse effects elsewhere.