Well, it says, "nor as being lords over those entrusted to you, but being examples to the flock." (1 Peter 5:3) So, if leaders are to be "examples", why would they dress in a way to show they are so different than us? Examples are ones doing what is for all of us to follow, I consider.
But ones understand that the priestly robes are symbolic, including with different colors for different religious times and events. For example, purple can mean royalty or mourning. As Jesus people, we all are royal, His "royal priesthood" (1 Peter 2:9). So, His "examples" are for all of us.