Almost elusive permutation groups