Semiconductor quantum dots (QDs) exhibit exciting photophysical properties for a wide variety of applications in the field of energy conversion, lighting, and more recently quantum communication. The photophysics underpinning these applications at ambient conditions strongly depends on atomistic details of their dynamic surface. Neutral organic ligands used in surface passivation play a critical role in determining the structural and optoelectronic properties of these QDs. Small sizes and irregular atomic arrangements make these QD surface-ligand interfaces challenging to explore at the atomistic level. Here, we combine several computational simulation techniques to study thermally induced geometrical fluctuations in stable cadmium selenide (CdSe) QDs with and without ligand passivation. We find that structural fluctuations of surface atoms significantly depend on passivating molecules. The bulky and strongly binding ligands such as phosphine oxides induce a higher extent of interfacial dynamics. Though these stoichiometric QDs do not possess any permanent in-gap states, significant thermal distortion of QD-ligand interfaces can induce fluctuating defectlike states near band edges. Such vibronic dynamics in these QDs also modifies band-edge state positions on sub-picosecond timescale, impacting their functional properties. Our results further suggest that primary amine ligands are optimal choices for QD passivation. These insights may be helpful to make design principles for screening and optimizing passivating ligands best suited for various QD applications, such as for quantum communication technologies.