Think Jrue Holiday/Marcus smart. Those guys never go guard KD from the jump, but what's made them special defenders is they survive when they get switched onto a forward like that. Or a big. They're primary guard defenders, but they're disruptors in terms of creating turnovers, they put tremendous ball pressure on lead guards and you can't take advantage of them on a switch. That's what makes an elite defender, not the very basic "Well, can he guard this position for 30 MPG?"
Hell. We've seen Keon do that plenty too. I distinctly remember several possessions where he got switched on Lauri and totally clamped him. And man on man defense. Does that mean I want Keon guarding Lauri every play? Well, maybe, but it's far more important that you have switchable defenders that can play passing lanes/have active hands to create turnovers, etc.
Carter to me has all those tools to be a special defender at the NBA level. And for a team that's pretty much been a bottom feeder defensively for it's entire existence, that to me is a bigger need than pretty much anything.