DeflOpt thread

[Its`about]

If you take a look at Ken's code for just about ANY program he has written,

Truth be told I wouldn't mind looking at kzip

I appreciate your response. I use DeflOpt a lot, though I don't like having to type wine before every command I hope that someday you manage to find a way of making it work with limited time, via your suggested method or otherwise.

Edit:
I misread what you said, removed part of my post that didn't make sense.

[Zardalu]

I could probably port DeflOpt to Linux pretty easily. But I'm not sure if I'm ready to make its source code publicly available. I'm NOT looking for someone to SELL DeflOpt, mind you. I'm just a bit paranoid about releasing my source code. I have had way too many bad experiences in the past. People stealing my code (not only programs, but also web pages) and releasing my code as if THEY had written it. I HATE thieves. My code is my own. Good or bad. I do NOT borrow or steal from anyone else. I think Ken can attest to that.

Ben Jos.

[TPS]

I would like to request returning the old behavior of DeflOpt 1.x (via a switch or whatever) to at least consider files of any extension.  I often have files that are renamed from .gz, .zip, or .png to something else (mostly proprietary extensions) that I would like to optimize via script w/o either having to rename & then remember the old extension or making temporary hardlinks (if supported).

Thanks for a great utility,
TPS =)

[Zardalu]

I would like to request returning the old behavior of DeflOpt 1.x (via a switch or whatever) to at least consider files of any extension.

The reason I removed that is because 1.x dealt with ZIP files only. 2.x added PNG and GZIP support. In order to add the old behaviour, I would have to add code that tries to recognise the file format...

Still, I'll keep your request in mind.

Ben Jos.

[Zardalu]

I would like to request returning the old behavior of DeflOpt 1.x (via a switch or whatever) to at least consider files of any extension.

I'm considering again...

I just found out that .kmz files are simply .kml (Keyhole Markup Language) files that have been zipped (PKZIP-compatible). While it is a lot easier for me to add code that treats a .kmz extension the same way as DeflOpt treats a .zip extension, the .kmz thing has made me realise that making DeflOpt smarter (i.e. making it try to determine the file format by examining the actual file instead of simply looking at the extension) will make DeflOpt be able to deal with future extensions that are really just existing formats a lot better. So the "any extension request" has now gotten a higher priority for me. I'm still worried, though... . People could specify "*" in the root directory with the recursive switch on, and DeflOpt would scan/examine/parse every single file on the entire drive...

Meanwhile, does anyone know of any more extensions that are simply new extensions for old formats?

Also, I have received a request by email to have DeflOpt be able to keep the original date/time while rewriting files. This will definitely be a feature of the next version.

Ben Jos.

[fred01]

Meanwhile, does anyone know of any more extensions that are simply new extensions for old formats?

J/R/WAR (Java); ODT/S/P/G/F/B/M (OpenOffice.org documents), OTG/P/S/T (OpenOffice.org templates), SXD/W (Sun XML Draw/Writer) also BAU, SOB, STW, for various resources and DAT but this will be probably too generic; CRF(Thief1/2 resources); PK3/4 (Quake3/4); KPZ (Komodo Project Zipped); XPI (Mozilla/Chrome theme/extensions); WMZ (Windows Media Player Skins);

I'm still worried, though... . People could specify "*" in the root directory with the recursive switch on, and DeflOpt would scan/examine/parse every single file on the entire drive...

How about showing a warning if the list contains more than 100(1000) items, similar to the way it is used in bash when autocompleting?

[ace_dent]

Howdy!

It's great to see DeflOpt continue to develop. I have stumbled across a problem with large images (generally photos), where the resulting png cannot be displayed in Paint Shop Pro X (PSP)... it just appears as 100% black. While I'm happy to accept this might be poor compliance on the part of PSP, I thought it might be worth further investigation in case there is some bug lurking there! I have isolated a minimal test case that can be downloaded here. Details:

0. BMP 300x1 px, 8bpp grayscale (although only 17 unique colors), 1,378 bytes.
1. -> OptiPNG, PNG : 8bpp grayscale, 214 bytes. (IDAT 157bytes). Views OK.
2. -> DeflOpt: 8bpp grayscale, 213 bytes. (IDAT 156bytes). Will *NOT* display in PSP...

OptiPNG.exe v0.5.5 (28-Jan-2007)
DeflOpt.exe v2.06 (28-Apr-2007)

Cheers,
Andrew

[Martin]

Meanwhile, does anyone know of any more extensions that are simply new extensions for old formats?

.svgz (gzipped SVG)

[Zardalu]

How about showing a warning if the list contains more than 100(1000) items, similar to the way it is used in bash when autocompleting?

Sounds good, but it would cause problems when people use scripts that are meant to run unattendedly. And, sure, I could add another switch to not display warnings, but I think I'll just assume the user knows what he is doing. After all, DeflOpt is a command line program and people who know how to use a command line generally are not newbies. ; )  Reminds me of a quote saying "Make a program fool-proof and only a fool will want to use it." And I don't want to spend my valuable time writing code that protects people who don't know what they're doing against themselves. If I did that, I would never get around to writing really useful things. : P

It's great to see DeflOpt continue to develop. I have stumbled across a problem with large images (generally photos), where the resulting png cannot be displayed in Paint Shop Pro X (PSP)... it just appears as 100% black. While I'm happy to accept this might be poor compliance on the part of PSP, I thought it might be worth further investigation in case there is some bug lurking there! I have isolated a minimal test case that can be downloaded here. Details:

Thanks, Andrew. Always happy to see feedback from you. I downloaded the file and I'll look into it.

OK... you guys have convinced me... . I've received a lot of requests both here and through email to reinstate 1.x behaviour. Meaning that DeflOpt will scan files regardless of extension. And seeing how many other extensions there are that really are just GZIP, PNG, or ZIP files, and realising that many more could be created every day, I'll add a switch to have DeflOpt be "smart" and scan files first to see if they are in a supported format and then treat them accordingly. Using this switch of course will make DeflOpt slower, but I'll just assume the user knows what he's doing when specifying "/a". New switches will be "/a" and "/d". I hope to find the time to code those changes soon.

Ben Jos.

[Zardalu]

I have stumbled across a problem with large images (generally photos), where the resulting png cannot be displayed in Paint Shop Pro X (PSP)... it just appears as 100% black. While I'm happy to accept this might be poor compliance on the part of PSP, I thought it might be worth further investigation in case there is some bug lurking there!

I have analysed the files you made available. The only thing I could find that might cause PSP to have a problem is the following:

The compressed data does not use any distance codes. In TestCase1.png, two distance codes, distance code 0 and distance code 1, are defined and both are given a length of 1, but neither one is actually used. In TestCase2.png, since the deflate specification requires at least one distance code to be defined, DeflOpt defines one distance code, distance code 0, and gives it a length of 0. According to the deflate specification, this is perfectly valid. The specification at http://www.gzip.org/zlib/rfc-deflate.html states (bold added by me):

A code length of 0 indicates that the corresponding symbol in the literal/length or distance alphabet will not occur in the block, and should not participate in the Huffman code construction algorithm given earlier. If only one distance code is used, it is encoded using one bit, not zero bits; in this case there is a single code length of one, with one unused code. One distance code of zero bits means that there are no distance codes used at all (the data is all literals).

While I don't have PSP, I have tried a handful of viewers and editors and none of those had any problem with TestCase2.png. This of course does not prove there's nothing wrong with that PNG, but all this makes me think that that PNG is perfectly valid and that PSP has a bug related to this.

If you run DeflOpt in verbose mode (/v), for TestCase1.png, it will show:

Code:

  Blocktype  :             2  (Dynamic trees; L-codes: 257; D-codes: 2)

while for TestCase2.png, it will show

Code:

  Blocktype  :             2  (Dynamic trees; L-codes: 257; D-codes: 1)

I am wondering if the PNG files you have problems with in PSP all have "D-codes: 1".

Ben Jos.

[Zardalu]

I have just released DeflOpt 2.07. Get it from http://www.walbeehm.com/download/. The changes:
- /a option added ("scan all files")
- /d option added ("preserve date and time")

See the included DeflOpt.txt for more information.

Ben Jos.

[Awesoken]

DeflOpt defines one distance code, distance code 0, and gives it a length of 0. According to the deflate specification, this is perfectly valid.

Damn, I could have saved you some time. Shame on me for being lazy and silent. I went through exactly the same process with PNGOUT. The /mincodes option of PNGOUT adds a distance code if none exist. The source of the problem is in Zlib 1.2.1, which had a bug where it would report an error if the stream had no distance codes - even if it was legal according to the official spec. A few months later, somebody noticed PNGOUT was generating images that didn't work on certain mobile phones. Apparently, making it store at least 2 distance codes seemed to fix it. Apparently, these phones are even buggier than Zlib 1.2.1. You are welcome to copy my /mincodes option name.. as I'm sure it'll make it easier for everyone.

[Zardalu]

Damn, I could have saved you some time. Shame on me for being lazy and silent.

I could have saved myself some time too. I read about the "cell phone problem" here, but only skimmed through those posts. If I had paid more attention, I would have known what your "mincodes" option meant. I don't know if I am going to add something to DeflOpt to deal with it, though. I am reluctant because (1) Zlib 1.2.1 was buggy and those cell phones are buggy, in the sense that they do not conform to the specification. I feel it's up to "them" to fix the problem and up to users to upgrade to versions that don't have those bugs (and hoping the problem won't even exist anymore after that), and (2) because, similar to what you mentioned about your PNGOUT, my DeflOpt was meant to optimise size. Why make it do something sub-optimally just because someone else did not conform to the specification?

I think I'm a bit different from you, Ken, in that I am mainly programming tools that I find useful myself and as long as they work for me, it will take a lot for me to make them useful for others. I think you already know that about me, Ken, but I just thought I'd make it explicit. I'm not programming to become famous or anything. I just program things that I find useful myself and if others find my code useful too, great. If not, hmm... well... maybe... ; )

Most programmers are like me, I guess. They program for their own use. Ken is an exception in at least two ways: (1) He is the best programmer I have ever encountered. (2) He puts in a lot of effort to make his programs work for everyone (including me!).

Ben Jos.

[Zardalu]

I have had some requests for a Linux version of DeflOpt, both here and through email. I thought a Linux port would be relatively straightforward, but then I tried to compile DeflOpt using gcc (or, rather, g++) under Cygwin and saw that there were a lot more problems than I thought.

I hate source code that has #ifdef statements all over the place (because the more portable you make something, the less readable it tends to become), so I rewrote some things so that there are only very few #ifdef statements. I'm not entirely finished yet, but most of the work is done now.

I am looking for someone who has a Linux box and is willing to test a native Linux version of DeflOpt. Of course, that person would get the source code of DeflOpt and my Makefile for it to compile it with g++. Preferably someone that has some knowledge of makefiles and programming under Linux. That person could then host the executable (but not the source) or I could host the executable myself.

Since I do not check this forum very often, please reply to me by email if interested. And read DeflOpt.txt first to see how to email me, because it's very likely if you don't do that, I will never see your email. ; )

Thanks!
Ben Jos.

[Zardalu]

Oh... one more thing... in lots of places, the DeflOpt code assumes "little-endian-ness", so a Linux executable compiled for a big-endian CPU would not work.

Ben Jos.