I'm pretty sure spdif or optical is limited to 5.1 in 48hz/16bits compressed in either Dolby or DTS (no 5.1 FLAC since it is lossless) and stereo in 96hz/24bits.

I think that is why you can only hear the core DTS.

Going through hdmi will not have this limitations...