Distance-weighted discrimination (DWD) is a modern margin-based classifier with an interesting geometric motivation. It was proposed as a competitor to the support vector machine (SVM). The authors greatly advance the current DWD methodology and its learning theory. They propose a novel thrifty algorithm for solving standard DWD and generalized DWD. Furthermore, they formulate a natural kernel DWD approach in a reproducing kernel Hilbert space and then establish the Bayes risk consistency of the kernel DWD by using a universal kernel such as the Gaussian kernel.