Ok, I've got the t-shirt,
Spoiler
and I've tried to use it on Ollie. I've exhausted all dialogue options with him beforehand. Still, doesn't work.
I even checked the walkthrough, and it says to give it to Ollie. Why won't it give it to Ollie? Is there something I'm supposed to do first? Maybe to the T-Shirt?
No matter what, it keeps saying 'I don't think he'll want that.'
Think about it from a different angle... Pib's job is not to solve the problems of everyone he meets. His job is to solve his own problems (or at least those of his Master's). Put yourself in Pib's shoes. Why would you want to give away an item unless you knew you could something in return?
If you think about the situation, Ollie is guarding the gate to the mansion, and has sole control over it. You need a reason to go through that gate. Perhaps you can find out something more about the mansion by talking to other people around.
I found it. I realized what happened was, in a previous game I'd gone through all that dialogue, but had to go back to an earlier saved game because of the whole quill pen thing... thus, I didn't talk to that other guy again...