Surface roughness is one of the most important parameters of surface quality and a difficult technical issue in glass polishing, especially for traditional polishing. In this paper, the coupled algorithm of FEM/SPH has been used to simulate the deformation of brittle K9 glass in traditional polishing. The influences of polishing particle size and insert depth on surface roughness are analyzed in detail. Then, experiment is carried out on a empty set 100 mm flat K9 mirror with three sorts of particle, ceria abrasive particle with 1.2, 1.6 and 2 mu m. Simulation and experiment results show that surface roughness of brittle glass has direct relationship with particle size during traditional polishing process. The surface roughness is better as the particle size is smaller.