In this paper, we proposed a discrete sorting optimization approach for uplink channel of Massive Multiple Input, Multiple Output (MIMO) system. The algorithm includes users (UEs) sorting before QR decomposition (QRD) and sorting-reduced (SR) K-best detector for 48x64 MIMO uncoded systems. Simulation results show that detector losses is about 1dB to a Maximum Likelihood (ML) detection in low detector complexity.The UEs sorting is required to sort diagonal elements of the R matrix in ascending order to avoid error propagation in multi-user (MU) scenario. Fast and low complexity method of online discrete optimization is used to find the loss function minimum. Sorting tracking is proposed, so that the pre-sorted interpolated R matrix is used for further sorting optimization, resulting in low sorting complexity. The proposed sorting demonstrates huge performance gain compare to a common power-based one. Simulation results in 5G QuaDRiGa channel are presented.SR-K-best detector is a variant of K-best detector. The SR-K-best with (K,S,p) parameters results in significant losses in scenarios with high correlated users, therefore we proposed a new structure (K,S,p,v,q) of the SR-K-best algorithm and a new discrete optimization method to increase performance. Discrete stochastic optimization was done offline in QuaDRiGa channel to find optimal (K,S,p,v,q) parameters for fixed detector structure.