Yeah, I get what you're saying - I know that (most) manga is deliberately designed to push the reader through the page as fast as possible - the average reading time for a tankoubon (the collected volume) is right around 20 minutes. So the visual language is designed to speed you along, unlike American-style comics which are really designed to slow you down a bit and to look at the art, and I think that's part of what makes people not get it; that just doesn't work with their ... brainstyle? reading style? enjoyment?

Of course, there's other reasons, too - I know people who just can't read right-to-left because the left-to-right is so ingrained, and then there are people who can't get past the character designs and really prefer more realistic art.

(Sorry, I can start lecturing on manga at the slightest provocation. I should have a warning sticker. XD)