A Malaysian Islamic court on Sunday jailed four people for three years for defying a court order to "repent" and attempting to renounce the religion,