Confident prediction is highly relevant in machine learning; for example, in applications such as medical diagnoses, wrong prediction can be fatal. For classification, there already exists procedures that allow to not classify data when the confidence in their prediction is weak. This approach is known as classification with reject option. In the this paper, the authors provide new methodology for this approach. Predicting a new instance via a confidence set, they ensure an exact control of the probability of classification. Moreover, they show that this methodology is easily implementable and entails attractive theoretical and numerical properties.