A classifier is said to possess confidence if it has several possibly overlapping classification regions, and each region is guaranteed to cover a particular class with a specified probability. In this talk, I will first introduce the general framework of classification with confidence. Next, I will highlight the relation between classification with reject options and classification with confidence, which is related to my previous work on multicategory classification with reject and refine options. It turns out that classification with confidence is one way to achieve classification with reject and refine options, and vice versa. This motivates the current proposed work. In particular, I have studied a new support vector classifier, which, with high probability, simultaneously satisfies two properties: (1) the probability that it makes Type I and Type II errors is controlled; and (2) asymptotically it has the smallest ambiguity region among all the classifiers that satisfy property (1). Future work includes the generalization to other convex surrogate loss functions and the multicategory classification with confidence problem. In additional, more computationally efficient algorithms will be developed.