Various qsort(3) callers pass incomplete comparison functions
Platform: SunOS x 5.9 Generic sun4u sparc SUNW,Ultra-5_10
Library: libvorbis-1.0 (vorbis1_0_public_release)
Arch: sparc-sun-solaris2.9
When calling qsort(3), the comparison function MUST return either -1, 1 or 0.
The following files have "optimised" functions that do not return 0 for
equal-valued arguments:
lib/lsp.c
lib/psy.c
lib/sharedbook.c
vq/vqgen.c
The result under Solaris 9 (possibly other versions) is that qsort will call the
function with out-of-bounds arguments when encountering equal-valued
comparisons. oggenc, for example, SEGVs in psy.c[apsort] immediately upon
invocation.
The attached patch tweaks all potentially offending comparison functions, and
Works-For-Me.
/taso