Human beings are inherently judgmental. We judge books based on their covers all the time, despite how often we're told not to. For better or worse, it's just a part of how the human brain is wired. Understanding that fact is necessary for being a good customer and a living, breathing human being. It's impossible to ever completely turn off that part of how we think.
That being said, there's a purpose behind my bringing this up. As retail employees, we're constantly being told to judge customers without judging them. To determine exactly how to overcome a customer's objections to what we're trying to sell, and to judge just what we will be able to sell them. My old boss used to tell a story about how he made a bunch of money helping a guy buy a home theater system. He made the sale, my boss insisted, because he didn't judge the customer on his looks. That customer was a farmer, straight from selling the last of his harvest. Dressed in dirty jeans and a ratty shirt, other salesmen assumed he was too poor to afford what he was shopping for. My boss, however, just went over and helped him out, and in the end made a nice commission off everything he sold.
Now, I have no idea how much of that story was true and how much was hyperbole. What I can tell you is that we frequently are forced to judge our customers based on appearance, despite being told not to. If someone walks into a store dressed like a crack addict looking for a fix, security will see that and follow that person around regardless of whether or not its true. On the other hand, if someone walks into a store in a suit and tie, others will think you have money and just came from an office job, regardless of how true it may or may not be.
For women, it's worse, because it's a double-edged sword. We should have the right to dress how we choose without being judged on it. It is inherently unfair that a woman can't go anywhere without thinking about how men will react to how she's dressed. I enjoy wearing black and purple, and favor goth fashion. That doesn't mean I wear it to work, because I understand it is not appropriate attire for the job I'm doing. Unfortunately, not all of my coworkers accept that it's necessary to dress for your job, not expect your job to accommodate your dress.
It's inevitable that if you are out in public, people will judge you based on your attire. So I say use it. If you are worried you might get robbed, dress like you have no money. If you're going to work somewhere that requires you to bend and lift, don't wear something that will get in the way of that. If you're going to a job interview, dress professionally instead of just going in your pajamas. Tailor your appearance for your circumstances and plan for specific reactions.
And never, ever shop for patio blocks in six-inch stilettos.