Defects engineering has been a widely used approach for developing low thermal conductivity materials. Also, for some materials (e.g., IV-VI compound), considerably large amount of vacancies was often observed in single crystalline samples. The atomic vacancy in dilute case was assumed as a point defect that can cause the Rayleigh scattering with its rate proportional to the fourth power of phonon frequency. Due to the large dependence of scattering rate on phonon frequency, a vacancy was considered to only weakly suppress the mode thermal conductivity of low frequency acoustic phonons. We show that this does not hold for IV-VI materials. We compare the phonon scattering by a single vacancy in group IV (diamond and Si) and IV-VI (PbTe and GeTe) crystals using an exact ab initio Green’s function approach. The phonon-vacancy scattering rates in diamond and Si follow the well-known fourth-power dependence on phonon frequency. However, PbTe and GeTe exhibit phonon scattering rates that have much weaker frequency dependence. As a result, a vacancy scatters low frequency acoustic phonons more strongly in PbTe and GeTe than in Si and diamond. This unusual behavior in PbTe and GeTe originates from the significant changes of interatomic force constants that extend up to the distance of 10 angstrom from the vacancy site. Therefore, a vacancy in IV-VI materials cannot be assumed as a point defect but its finite size should be considered for phonon scattering. In contrast, a vacancy in Si and diamond changes interatomic force constants only up to 3 angstrom range. Our study provides new insights on the phonon spectral contribution to thermal transport when vacancies exist in IV-VI materials and will help to develop low thermal conductivity materials by engineering vacancies.