Firstly, this is going to be longish, I care too much and am too inexperienced with writing short (you may wanna skip to the end...), but i hope it all rings true in your heart as it does mine. That said, I'm also a complete agnostic, so don't read that comment about the heart and think that I'm gonna try and prove vegetarianism is right, because I can't... yet! :D
Secondly, I think you have assumed (definitely wrongly in some cases) that people don't care about the vegetables :)
To the point...
Many people would be vegetarian because they see killing animals as wrong, what you're saying is that by eating vegetables, they show that they don't care about the vegetables. If someone turns around and says that they in fact don't care about the vegetables, then it simply means that they are acting on their own perceptions. People will care far greater about something that they can see and feel (which is why many people would not be able to kill their own food, but happily buy it in a supermarket). It comes down to something called confrontational suffering. 
Confrontationaly suffering is simply when you are confronted with suffering (some nice rearrangeing of words gone into that definition :P)... ask yoursef, would you pay alittle money (like £2 or $3) to stop me from killing (in a cruel way) a kitten or chicken in front of your eyes? Many people would pay, but happily pay money to do similar things to animals (by buying meat), simply because in the different scenarios are different levels of confrontational suffering.
suffering infront of you, makes you feel shocked and (hopefully ) upset, you hopefully feel that you want to prevent it. But when the suffering is hidden and you don't realize it, it may aswell not exist and there is no thought of preventing it.
So how does this relate to vegetables I hear you ask... well, if you think about it, we all find it much easier to relate to animals suffering than vegetabes suffering. (To say that vegetables don't suffer, because they don't have nervous systems as a possible reason, is to say that our current science has ALL the answers, if you believe it does, than this will not apply to you or the vegetarian in question.) So we are confronted (by way of feeling) with more suffering when we see a cow being cut than when we see an onion being cut. Hence we want to prevent the cow's suffering, but will struggle to appreciate that the onion may be suffering too.
To sum this up alittle... the vegetarian that doesn't care about the vegetables, is just acting on a level of confrontational suffering with which he is comfortable. He is comfortable cutting an onion because he can't relate to it, so there is no (except his own form the evil onion juices!) confrontational suffering. Whilst when an animal is suffering, he feels suffering and is uncomfortable with it.
I'd like to take this moment to stress the theory that everyones level of comfort with confrontational suffering is different, for example a vegetarian only needs to think about animals suffering, whilst for some people it needs to be infront of their eyes, others don't give a hoot at all, and some may need to only think about the suffering of vegetables.
So if a vegetarian does care, or feels that the possiblity of vegetables suffering exists and should be prevented where possible, it is NOT hypocritical to stop eating animals but continue eating (eating more in fact) vegetables. Simply because, an animal needs to eat plant mass to survive . I'm not too sure on facts, but say a cow eats 10Kg for every 1Kg of meat that you get out of it , than you have an option of eating 1Kg of vegetables to be full, or 1Kg of cow to be full. Eating the cow means you killed a cow, and 10Kg of vegetables, so eating only vegetables prevents unnecessary suffering. Of course you can argue that if a vegetarian really cared, he'd commit suicide, but that aside, being vegetarian is just minimizing the combined suffering of animals and vegetables together. (which is why I said you may have assumed wrongly that a vegetarian has no moral objection with eating vegetables, because in truths, that is where we may actually prevent the most suffering... if vegetables even suffer that is.)
 I think I might be the first to coin this phrase!! But the idea I'm sure has been around, hence the "something called..." isn't exactly accurate.. meh.
 No bias intended... honest :D Just saying that society needs people to care, so hopefully people do...
 I don't particularly want to get into something like "so we should eat giraffes, cos they don't kill plants, but kinda harvest trees", so I'll just leave it saying that in todays meat production, the majority of animals kill plants opposed to harvest them, simply cos of intensive farming methods.
 hmmm, is it obvious I'm vegetarian yet? :D yea, facts there are probably totally wrong, but it's a fact that food energy from plants gets wasted through animals.
yea, sorry if there's any bias... but I hope that it answers the question "do vegetarians see it morally wrong to eat vegetables" better than the previous guy. In essence, if they don't, it is due to a personal level of confrontational suffering. If they do, then they try to minimize the vegetables' suffering by being vegetarian.
Also, I know that there are things that I didn't answer, so just give me a message if there is anything on your mind on the matter. If there is anything that is really wrong, also message me, 'cos I'd hate to accidentally mislead someone. Also, this is mostly my own thought, or atleast independently arrived, so I'm not saying it's right, just another possible way of explaining the whole vegetarian chestnut!
Cheers, and Sonke out!
Date of comment: Tue, Mar 24th 2009