Imagine there was a society in which blue eyed people are referred to with blee/bler pronouns, and green eyed people are referred to with glee/gler pronouns, and one day someone from that society saw a brown eyed person and had no idea whether to categorise them as blee/bler or glee/gler
Yeah, we have all kinds of labels that are not arbitrary, but also also not 100% accurate in all cases. They only tend to be a problem when people try to treat them as absolutes.