In this paper, we provide a simple approach to identify and estimate group structure in panel models using the M-estimation. We consider both linear and nonlinear panel models where the regression coefficients are heterogeneous across groups but homogeneous within a group and the group membership is unknown to researchers. The main result of the paper is that under certain assumptions, our estimation and classification method is consistent even if one uses an incorrect number of groups as long as this number is not underestimated. Conditions under which estimation of groups and regression coefficients are consistent and asymptotically normal are also provided in the paper. Monte Carlo simulations are conducted to examine the finite sample properties of the M-estimation. Findings in the simulation confirm our theoretical results in the paper. Application to labor force participation also highlights the necessity to take into account of individual heterogeneity and group heterogeneity.