somehow I still did not get it, why my c64 colors look more nice under java than under cocoa.
here a java result:

and a cocoa result, using the same function to find the distance between two colors:

ok, the c64 was very pixelish and I did miss the 320x200 by 40 pixels yet (very few time to code at home due to that small worm on the images;-), but anyways, the java version looks better a this time. have to play more with the NSBitmapImageRep class I think...
this is the method I use for getting the distance between the colors, basically found the weighting stuff in the net, missed it in my first approach of building a simple 3d room over R G B and doing some vector length calculations:
-(CGFloat) colourDistance:(NSColor*)c1 second:(NSColor*)c2
{
CGFloat distance = DBL_MAX;
CGFloat rmean = ([c1 redComponent] + [c2 redComponent]) /2.0;
CGFloat r = [c1 redComponent] - [c2 redComponent];
CGFloat g = [c1 greenComponent] - [c2 greenComponent];
CGFloat b = [c1 blueComponent] - [c2 blueComponent];
CGFloat weightR = 2.0 + rmean/256.0;
CGFloat weightG = 4.0;
CGFloat weightB = 2.0 + (255.0-rmean)/256.0;
distance = sqrt(weightR*r*r + weightG*g*g + weightB*b*b);
return distance;
} same one under java.
will play around more these days...