It gets me annoyed too. I think it's annoying because you sat
  there, LISTENED to what they were saying, _paid attention_,
  thought about it, gave it your consideration, used your brain
  power on it...

  ...and then... at the very end.... *all that work* gets
  dismissed out of hand with "just" sayin'.

  rendering all that hard work you did, for naught because *they*
  don't care about what they're sayin', but YOU did. So,
  consequently by not caring about what they were saying, they
  just said that THEY aren't going to care about what you have to
  say about it.