Removal of the materials during thin-wall milling leads to obvious change of the modal parameters of the thin-walled workpiece. This article proposes a new method for designing a passive damper to suppress the chatters in thin-wall milling by considering the workpiece’s in-process varying modal parameters. The modal parameters of the workpiece during cutting process is theoretically derived by including the influence of material removal, and the design principle of equal peaks is used to calculate the optimal parameters of the passive damper. Based on the derivation, the passive damper is then designed and manufactured. A series of milling experiments are conducted for the cases with and without the designed passive damper. Experimental observations show that the passive damper has good performance in suppressing chatters and improving the stability of the cutting system.