In this paper, we introduce the Eigen-Factor (EF) method, which estimates a planar surface from a set of point clouds (PCs), with the peculiarity that these points have been observed from different poses, i.e. the trajectory described by a sensor. We propose to use multiple Eigen-Factors (EFs) or different planes' estimations, that allow to solve the multi-frame alignment over a sequence of observed PCs. Moreover, the complexity of the EFs optimization is independent of the number of points, but depends on the number of planes and poses. To achieve this, a closed-form of the gradient is derived by differentiating over the minimum eigenvalue with respect to poses, hence the name Eigen-Factor. In addition, a time-continuous trajectory version of EFs is proposed. The EFs approach is evaluated on a simulated environment and compared with two variants of ICP, showing that it is possible to optimize over all point errors, improving both the accuracy and computational time. Code has been made publicly available.