Vanishing / exploding gradient problem often arises in training deep neural networks. One way to make the neural network more stable is to replace the weight matrices Unitary / Orthogonal matrices. But this comes with the cost of a more expensive training cost. We will study some applications and methods of training in the literature.