First of all, when people use disrespectful terms to refer to temple garments, it is entirely appropriate for you to kindly ask them to show more respect, since the garments are sacred to us.
Also, you may want to point out that members and clergy in many other religions wear specific articles of clothing to represent their personal faith or their official responsibility, so the fact that our religious practice includes special clothing is really not unusual.
To explain the significance of temple garments, you can say that they are simple, modest underclothing given to adult members of the Church as part of special ceremonies in our temples. In these ceremonies, we commit ourselves to live the way Jesus Christ would have us live, and the garments are a constant physical reminder of this personal, spiritual commitment. In this way, garments can help protect us against temptation and evil.