I'm generally a set collector, so my answer is more speculation than anything else. But my favorite players are often just that because of their personal qualities, along with their prowess on the field. Ryne Sandberg, Lou Brock, Nolan Ryan, Ernie Banks, Roberto Clemente, Anthony Rizzo, Lee Smith, Greg Maddux. They all seem like good guys. If I learned that one did something terrible, I'd keep their cards, but if I was a player collector, I wouldn't go after that player anymore, because that player wouldn't be a favorite anymore. (To be honest, I always found the player collectors just a bit...creepy/obsessive/something. I once thought of putting all Lou Brock cards on the wall, but it felt just a bit too...shrine-ish for my tastes.)
I also think the cards are history. When I was hunting a 1941 Play Ball, there were no Cubs, so I looked for the beautiful, affordable card I could find. Turned out it was Pinky Higgins. And it turns out he was one of the biggest reasons the Red Sox were the last team to intergrate. But all my pre-Topps cards are on my shelf with personally made labels containing interesting facts. And Pinky's label includes the phrase "Strongly opposed integrated baseball." So I guess just put an asterisk on him. And I'd never try to collect all his cards.
Similarly, I (sadly) just cannot listen to my Bill Cosby record collection anymore.
Good topic.
v3