The head of the UN's inquiry into the 2014 Gaza war has quit amid allegations of "malicious attacks", prompting Israeli Prime Minister to demand the investigation being called off.