Durability and availability have made up the bulk of the “dirt” KINGS fans throw at him. That, and the childish, unprofessional behavior displayed by team Bagley. Which, again, isn’t speculative nonsense.
But as far as I can tell, the most common complaint by far Kings fans make about Bagley basically boils down to "he's a terrible basketball player who has no value and has no hope of even developing value". And while that may be a bit of hyperbole for some folks' position, I think it's dead-on many other folks' position.
I wouldn't be so adamant about defending Bagley if the major complaints were durability and unprofessionalism - the first being a clear issue thus far and the second being relatively subjective (and, to be fair, at least partially excusable by the way the fanbase has treated him). But I think the "no value" position is pretty wrong and the "no hope of developing value" position is absolute nonsense.