We measured the resonance spectra of edge magnetoplasmon (EMP) oscillations in a two-dimensional (2D) electron system located on a liquid-helium surface below 1,1 K. Systematic measurements of the resonance frequency and the damping rate as a function of the lateral confinement electric field strength shows clear evidence of the oscillation mode transformation. A pronounced change corresponding to the mode transformation was observed in the damping rate. When 2D electrons are confined in a strong lateral electric field, the damping is weak. As the lateral confinement electric field is reduced below a certain threshold value, an abrupt enhancement of the damping rate is observed. We hypothesize that the weak damping mode in the strong lateral confinement electric field is the compressive density oscillation of the electrons near the edge (conventional EMP) and the strong damping mode in the weak confinement field is the coupled mode of conventional EMP and the boundary displacement wave (BDW). The observation of the strong damping in the BDW-EMP coupled mode is a manifestation of the nearly incompressible feature of strongly interacting classical electrons, which agrees with earlier theoretical predictions.