I was trying to rezip some zipfiles with kzip to make them smaller.
Everything was going fine until I tried to rezip the contents of this file...
http://3feetunder.com/files/primglex.zip
...using these settings...
kzip.exe /r /y /b128
I understand that kzip is slower than other zip tools but I let it run for nearly 2 days and it still hadn't finished. Other zip archives of similar size were finished much quicker and certainly didn't take days. I'm thinking that there must be a bug and it's stuck in a loop or something.
I compared the contents of other archives that I was able to rezip successfully and the only difference I can find is that primglex.zip contains files that are exactly 4.00 MB in size. Maybe there is a subtle bug that is triggered only when files are EXACTLY 4,194,304 bytes. I've seen similar bugs happen before in other programs.
I've skipped this particular archive for the moment and I'm moving on to other larger archives. If I find any others it gets stuck on, I'll post it here.
Awesoken at
Yes, this is a bug in both KZIP and PNGOUT. In PNGOUT, you can press ESC to quit after the next pass (allowing it to continue). KZIP has no such feature (yet). I've seen this happen before but I've been too lazy to do anything about it. I can say for sure that it has nothing to do with the file size being an exact power of 2. My algorithm continues to do passes while it thinks the next pass will be smaller. Certain files make it fall into infinite loops. "pg1rom0l.ic6" happens to be one of them. All other files in your archive work fine. If you want to have fun, you can run KZIP in verbose mode (/v) and watch it repeating the same pass forever:
er... so... what are the chances of this bug getting fixed?
I hope you can fix it because I think KZIP is awesome. I used to think BJWFlate was the king, but KZIP has stolen the crown.
Would it help to have more files that cause it to exhibit this behaviour?
krick at
I was also curious about the "Block split threshold" parameter.
Is there an upper limit on this value?
Awesoken at
The block split threshold is a double floating point variable, so there really isn't a limit. While the upper limit is in theory infinity, usually anything above 5000 will result in no split blocks. You can enable verbose mode (/v) to see how many blocks were actually created. FYI: /b0 is an alias for /b999999...
counting_pine at
How does your deflate algorithm decide when to not do another pass? Would it be sufficient to check whether two consecutive passes generate the same Huffman tables?
Anonymous at
I found another archive with the same problem.
One of the files is hanging when I try to rezip using
kzip.exe /r /y /b128
Here's a link to the archive if you want it for testing...
http://3feetunder.com/files/rockn.zip
Maybe if I can find enough of these files, we can figure out some sort of commonality that triggers the bug.
Awesoken at
I fixed the infinite pass bugs in PNGOUT and KZIP today. Thanks to everyone who sent me test files - they were very helpful!
As a bonus, I made PNGOUT and KZIP always save the smallest pass. Sometimes, a previous pass is smaller than the last pass (because the Huffman table itself varies in size). The bad news is these files will now require an extra pass. The good news is the result will always be smaller if this pass is necessary.
Next, we need JonoF to re-compile the Linux / Mac ports : )
counting_pine at
Cool, I've tested the files with kzip /b128 and it now manages both of them. I haven't tested PNGOUT because I don't have a PNG file that triggers the bug, but I assume it works OK.
I can't get PNGOUT or KZIP to always save the smallest pass though. One file I tested gave me the result:
Ah... I think I know why that's happening. Would you mind sending me a URL to that image? : )
counting_pine at
Here's the image in question (108KB).
If you run pngout /v on it, the smallest pass is 10 bytes smaller than the final pass, which is the biggest difference I've seen.
I posted my results from running PNGOUT with the /kp switch, so the same thing will happen if you decompress the IDAT into a file and then kzip it.
Awesoken at
Ok, fixed. PNGOUT and KZIP should now save the smallest pass.
BTW, nice image : ) My guess is "(x,y)e=exp(1);(y/e-x)%(x/e+y)" although the spectrum colors are slightly off.
counting_pine at
Yes, it works in both KZIP and PNGOUT now (although in this case, since it doesn't save any bytes, the final pass is unnecessary:).
Awesoken said
My guess is "(x,y)e=exp(1);(y/e-x)%(x/e+y)" although the spectrum colors are slightly off.
Thought you might recognise it :)
I think that's the gist of it... Let me see if I can find the code...
I think it was just a simple experiment, trying to cache a couple of functions of t. The spectrum was probably off because it was an older version of EvalDraw (hence the "globs"). I can't remember for the life of me why I have a screenshot of it...
krick at
Awesoken said
Thanks to everyone who sent me test files - they were very helpful!
No, thank YOU for spending your valuable time time fixing the bug.
I'm sure that I speak for KZIP fans everywhere when I say thank you.
callmeace at
Thanks from me too Ken, it's been a pleasure to use some of your excellent utilities, with my favourite being KZip :)