We propose a mathematical model based on probability theory to optimize COVID-19 testing by a multi-step batch testing approach with variable batch sizes. This model and simulation tool dramatically increase the efficiency and efficacy of the tests in a large population at a low cost, particularly when the infection rate is low. The proposed method combines statistical modeling with numerical methods to solve nonlinear equations and obtain optimal batch sizes at each step of tests, with the flexibility to incorporate geographic and demographic information. In theory, this method substantially improves the false positive rate and positive predictive value as well. We also conducted a Monte Carlo simulation to verify this theory. Our simulation results show that our method significantly reduces the false negative rate. More accurate assessment can be made if the dilution effect or other practical factors are taken into consideration. The proposed method will be particularly useful for the early detection of infectious diseases and prevention of future pandemics. The proposed work will have broader impacts on medical testing for contagious diseases in general.