Jared Dudley, Earl Clark and Draymond Green, Matt Barnes, Gallenari, Danny Granger , Caron Butler, Tayshanue Prince, Deng are all SF's who are considered solid-good defenders not one of them has the first step or explosion of Derrick and none are reall all that mobile and he's similar size/bigger than all of them. Also which one of those guys could he not defend?
Current starting SF's he could defend in terms of foot speed: Korver, Jeff Green, Wilson Chandler/Gallo, Deng, Dudley, Aminu (not a threat really), Earl Clark, Ariza, Paul Pierce, Josh Smith, Caron Butler, Marion, Tobias Harris and even the guys who has footspeed most are not good ballhandlers anyway such as Corey Brewer, Gerald Green and literally very few of the current SF's can overpower him like either
He's obviously not going to be able to defend the likes of KD,Durrant, Melo straight up but if they are on no one can, I have a hard time finding that many players his size/athletic ability would not allow him to defend and those guys he can't no one else can, I think people are thinking of players like Lebron, George, Melo , Gay (been terrible) and automatically assume the SF position is stack with elite ball handlers/athletes when really there's a lot of slow footed vets and guys with little to no handle.
Foot speed is great if they are racing. Defense is about mobility, lateral quickness, positioning. To date, he hasn't demonstrated the ability to use these skills to defend SF. Not really sure what you are arguing as it's not like he hasn't tried playing SF at times over the last 2+ seasons.