Is it just me but the sprite compression actually makes the sprite file bigger?
I've tried different graphic file types but it just seems larger when the whole game is compiled.
Graphics format doesn't matter, the format of the graphics data saved in an AGS game is independent of the original files' formats.
Since RLE is a VERY simple compression algorithm, it may help reducing data size when the data is simple (in the case of graphics, especially when the images have large areas of flat colours), however for "complicated" graphics the help is possibly minimal (and because of overheads, in worst case may even make the size bigger), so if RLE compression doesn't work well with your sprites just turn that off.