Mike James is a good defender, it's just in Toronto they didn't play much defense, played uptempo, and Mike had to spend more energy on offense because of his role. He's doing conditioning to improve his shape and energy for the season this summer. To enhance his game more and prepare, for that possible spot.
Whereas with the Heat/Pistons/Rockets, he was a top defender there. Why? Defense was stressed on all those teams, and had defensive-minded coaches. Along with the make-up of those teams where he had a more limited role. Perfect example being in Detroit for LB, he and Hunter were spark guards off the bench who put pressure on the ball and hit some shots. Key to Detroit's championship run and that great bench from 03-04.
Mike James established himself with the Pistons (above), Rockets (deadline trade, much needed bench spark, mavs series), and then of course the Raptors (all-around guard). Steady and well up to this summer.
Go ask on Heat/Pistons/Rockets boards about James, and they'll say similar. Shame I'm not on my regular comp. yet, could show a bunch of other opinions I found on him.