We often ask "why is this so expensive?" When we should more often ask "why is the other stuff so cheap?"
There are practical answers to these questions, but, as you laid put, there are deeper considerations as well. What things do we give up for the sake of cheaper costs and sexier businesses? Things that are harder to quantify and plot on a graph.
The guy may have liked your shirts but he didn't share your vision or core principles. The price simply exposed that and helped determine that he isn't in your target demographic.