Why do people need to keep tabs on people's TTC efforts?

My brother figured out that I suck at TTC. I didn't know that he knew, and I'm very upset about it. Now I don't know who all knows about me.

My brother blurted out, "So are you having kids or what?" Dh was defensive, and said, "We're working on it." My brother made the usual comments that people say: "Just relax." Oh, and the classic: "Just adopt." Wait, wait, don't forget the self-righteous religious perspective: "It's great that you can't, because then you can be missionaries." Oh, I'm still so upset about this, I can't repeat any more of that conversation.

I asked him via e-mail how he knew this, and he said that I told him. It appears that he took a comment I made years ago about my PCOS (and how it might make having kids difficult in the future), and thought that I was already TTC since, well, I guess since we got married. Which we weren't. So I tried to clarify things for him, yet muddle it enough that he won't be sure if we are, in fact, TTC.

Because I need to regain some control. I feel like I've totally lost control over all of this, because I don't even know anymore who knows what about me.

Now that I rewrote what he said to dh about the missionary thing, I got angry all over again. Not only was that a stupid thing to say, I don't know why he bugged dh about this if he knew we were having problems.

So I just sent this to him: I know people bug you and [your wife] all the time about babies, and people bug us, too. I wish people would consider not asking other people questions about when and if they want kids, because it's hurtful to people who can't have them, or who've had multiple losses (like [dh's sister-in-law]), or who don't want them (because they're scared to admit that they'd rather not have kids). Makes me wonder why people need to know so bad.

I didn't want to tear a strip off of him (because he is a great guy), but I hope that he understands what I'm trying to say.