In animals, biological sex is defined in purely binary terms, but it's possible for many kinds of animals (though not mammals) to have both. I don't see why this should be controversial. This is also the case in most plants.
There are some organisms like fungi which don't have "megagametes and microgametes", the umbrella terms used in biology for gametes that function like eggs and sperm respectively, but we generally don't refer to those as being categorized into "sexes" at all. Fungi have "mating types", for example, which aren't related to the kinds of gametes they make but the combinations in which they are and aren't capable of fertilizing.
The second problem, referenced in the last part of the quote (and the one we have been banging on about) is that the definition of gametes being applied to entire human organisms verges on nonsense. For example, it makes the vast majority of the human race sexless, since they are not currently producing either ova or sperm. Going to secondary sexual characteristics is... well why not just go there in the first place? Ah, that's right, we wouldn't find a binary without the imperative that gamete definitions apply to entire human beings.
I do not understand why you find this so hard to grasp. We don't "appl[y]" "the definition of gametes" "to entire human organisms", and we don't consider people sexless if they aren't producing sperm or eggs.
Sex in biology is defined in terms of developmental pathways. Not secondary sexual characteristics, but primary sexual characteristics as explicitly pertaining to the
development of gonads toward the capacity of producing either sperm or eggs.
ETA: This is also why having ambiguous genitalia due to DSDs is no longer considered to make an individual's sex indeterminate. Having, say, a penis is a collateral consequence of the male developmental pathway in the vast majority of cases, but it doesn't directly bear on the development of gonads.