Not quite the same, but I was going to work with kids in my
  early 20s. I did for a bit: volunteered full time at a cerebral
  palsy center for a solid year. Loved it. They offered me a job.
  I was so excited. But, I didn't take the deadlines for medical
  checkup and stuff seriously enough. I figured there was leeway.
  Nope. No leeway. I had plenty of time to do everything and
  didn't, and on the deadline date, I was missing 1/2 of what I
  needed to give them. Got called in, and that was that. *and* I
  couldn't go back to volunteering because of the possibility of
  retribution. It was a law - Believe it or not, that's an issue:
  if someone gets fired they can cause problems, and being a
  delicate population, the laws are justifiably very cautious in
  that area, so I couldn't blame them. I was heartbroken for days
  and kicking myself. Some of the parents asked me to work with
  their kids on the weekends and stuff but I just couldn't even
  think about that. I was too emotional to think straight. After
  getting over the sadness of it, I thought about other careers
  but then I looked down at my hand. No wedding ring. No picture
  of a girlfriend in my wallet to show people. At that time I was
  inbetween dating, idealistic about "saving the world" in my own
  fashion and didn't want the relationship turmoil. I though, 'How
  does this appear? Single white male in his early 20s, wants to
  work with children". Shit. I knew I'd have to fight against
  skeptical eyes, murmurs and whispers, little comments and such.
  Not from all or most, because most people aren't judgemental.
  But, a few ALWAYS are - and the fear of those few made me decide
  to scrap the idea altogether and pursue other work. I don't
  regret it, not really. It's the kind of work I could pursue at
  any point in life. I had other skills, more profitable, less
  idealistic, and I pursued those instead. But I felt bad for
  those who were all idealistic with good hearts and no fall back.
  I didn't fear conforming to all the stereotypical fears: I'm not
  that way. But I feared matching _some_ of the characteristics,
  enough for people to jump to conclusions. Wasn't worth the
  strain. It can be an uphill battle when you don't conform to
  stereotypes and people desperately expect you to. Sometimes the
  battle isn't worth it.