As one of the matrix factorization models, concept factorization (CF) achieved promising performance in learning data representation in both original feature space and reproducible kernel Hilbert space (RKHS). Based on the consensuses that 1) learning performance of models can be enhanced by exploiting the geometrical structure of data and 2) jointly performing structured graph learning and clustering can avoid the suboptimal solutions caused by the two-stage strategy in graph-based learning, we developed a new CF model with self-expression. Our model has a combined coefficient matrix which is able to learn more efficiently. In other words, we propose a CF-based joint structured graph learning and clustering model (JSGCF). A new efficient iterative method is developed to optimize the JSGCF objective function. Experimental results on representative data sets demonstrate the effectiveness of our new JSGCF algorithm.