People often confuse being able to jump with being an all around athlete. Williams can jump, and if he's a PF his mobility is good. If he were big enough and strong enough to just be a fulltime NBA PF, you would have no problems with his athleticism. But if he's got to guard a swingman type, its not so good. if you ask any Minnesota fan they'll tell you he can't stay in front of anybody.
Jared Dudley, Earl Clark and Draymond Green, Matt Barnes, Gallenari, Danny Granger , Caron Butler, Tayshanue Prince, Deng are all SF's who are considered solid-good defenders not one of them has the first step or explosion of Derrick and none are reall all that mobile and he's similar size/bigger than all of them. Also which one of those guys could he not defend?
Current starting SF's he could defend in terms of foot speed: Korver, Jeff Green, Wilson Chandler/Gallo, Deng, Dudley, Aminu (not a threat really), Earl Clark, Ariza, Paul Pierce, Josh Smith, Caron Butler, Marion, Tobias Harris and even the guys who has footspeed most are not good ballhandlers anyway such as Corey Brewer, Gerald Green and literally very few of the current SF's can overpower him like either
He's obviously not going to be able to defend the likes of KD,Durrant, Melo straight up but if they are on no one can, I have a hard time finding that many players his size/athletic ability would not allow him to defend and those guys he can't no one else can, I think people are thinking of players like Lebron, George, Melo , Gay (been terrible) and automatically assume the SF position is stack with elite ball handlers/athletes when really there's a lot of slow footed vets and guys with little to no handle.
Guys 1 on 1 most likely can't defend @ SF
Lebron, Gay, Melo, George, Batum, Hayward, Turner