Regarding the term "adventure game" - strangely enough, in Russia the term "Quest"(or "kvest" if you want the correct pronounciation
) is more common, and it's used to describe specifically "classic" adventure games. And they have their own shelf in the stores, usually next to the casual games and audio books. So, at least here, they are definitely already targeted at a different audience than the mainstream action games.
But they are still games, maybe much more passive, but games. Monopoly is a game too, even though it's not as action-oriented as, for example, football.
And although the story is a big part of adventure games, interaction is what defines them. And what is more interactive than pointing and clicking?
I'd also have to agree with Snarky that low-res graphics might be even a bigger turn-off for the non-gaming, story-oriented audience, than for the hardcore gamers. Indie-gamers as a rule love pixels, book readers don't. I wouldn't write off the appeal of low-res graphics purely on nostalgia, but it's sort of an acquired taste. Personally, I dislike high-res adventure games - they usually seem lifeless, static and empty to me, and I hated the Special edition of Monkey Island, but that's almost exactly the same thing as hating the low-res pixelly stuff, only the other way around. It's just a matter of taste.