I've discovered that one of the people following me is Really Actually Blind and therefore needs alt text, so I'm making it a point to add it though.
However, I've found that a lot of times, it's necessary to "explain the joke" in order to properly write alt text. And explaining a joke is like dissecting a frog; you understand it better, but you've killed it in the process.