A new adaptive optics (AO) system for controlling the mode profile of a diode-laser-pumped Nd:YAG solid laser has been set up in our laboratory. A 19-element piezoelectric deformable mirror (DM), which is used as the rear mirror of the solid-state laser, is controlled by a genetic algorithm (GA). To improve the system convergence rate, the GA optimizes the first 10 orders of Zernike mode coefficients rather than optimize 19 voltages on the DM. The transform matrix between the 19 voltages and the first 10 orders of Zernike mode coefficients is deduced. Comparative numerical results show that the convergence speed and the correction performance of the AO system based on optimizing Zernike mode coefficients is far better than that of optimizing voltages. Moreover, experimental results showed that this AO system could change TEM10, TEM11, and TEM20 transverse modes into a TEM00 mode successfully. (C) 2007 Optical Society of America.