Both of these players strike as role players rather than potential starters.
I was originally going to say Justin Jackson because like others on this thread he does have 3-and-D potential, however his defensive rating last season ranked last amongst his teammates, whereas Mason ranked a middle of the road eighth. When you consider from beyond the arc, Jackson shot .308 for the season, whereas Mason shot .360 and for the first two months was hitting over 40% of his three point attempts. From a points per game to per 36 perspective, Mason went from 7.9 to 15.1 (7.2 rise), while Jackson went from 6.7 to 10.8 (a 4.1 rise). So Mason based on per 36 almost doubled his points and it is also worth noting he almost doubled his assists per game.
So who would I retain based on their rookie seasons?
I would keep Frank Mason. He did a fair job leading the second unit and proved that he could facilitate, shoot and score. On the defensive end he has enough quickness and tenacity to prove useful, not a lock down defender by any stretch, but useful. Now Justin Jackson has better size and length but he needs to bulk up, improve his shooting, and become a smarter and better defender. So considering that this is a either or situation I think Mason has a better chance at maximizing his strengthens than Jackson because he did show more as a rookie, granted he did regress as the season went on but that is not unusual for a rookie player. Furthermore I feel that Mason has a sixth man ceiling, with his floor being a decent enough second unit point guard, while Jackson is going to need to really improve his shooting and defense if he's going to hit his 3-and-D off the bench small forward ceiling, otherwise he's going to be a so-so rotational small forward.