You're right, this is something I wasn't aware of since I've always been using custom GUIs for my dialog options. I just checked it in my own game, and it seems that the default dialog options display does antialias while if you use your own transparent GUI it doesnt.
I assume this is something CJ could implement because nothing can animate while the dialog options are displayed.