Around 60% of transgender citizens in the US have avoided using a public restroom out of fear of harassment or abuse. 12% of the trans community reported a receiving verbal abuse in a public restroom within the last 12 months, and 1% were physically attacked. 
Public restrooms are not safe places for the trans community. Trans individuals are routinely subject to verbal abuse and, in some cases, physical violence when they are forced to use the restroom of their sex at birth. Giving trans people the opportunity to use the restroom they feel safer in is an important step in reducing the harassment of the transgender population. Introducing prohibitive restroom laws are likely to exacerbate the situation. It would invite the public to play “gender police” and legitimise their confrontation of trans citizens in restrooms, essentially making public restrooms a more uncertain place for the trans community.
It isn’t just transgendered citizens who experience harassment in restrooms. Ethnic minorities also experience an increased level of harassment and violence in public restrooms, but nobody is suggesting that they be allowed to choose which restroom they use as a matter of safety.
Transgender people have the right to feel safe when using a public restroom. Prohibiting them using the bathroom that aligns to their gender identity robs them of that right.
Transgender people have the right to feel safe providing they don't put others at risk. Allowing transgender individuals to use the restroom of their expressed gender would put others at risk.