Biological sex is a category that exists only because it is tied to a specific function, that function being reproduction. And reproduction involves a binary set of prerequisites (sperm and egg) and it's from those prerequisites that we get the categories of male and female.

In other words, biological sex can just be thought of as a "reproductive mode." That means that regardless of any chromosomal variations, sterility/infertility, or secondary sex characteristics, the vast, vast majority of people have a reproductive mode (functional or not). And of that there are only two.