Fluorescent defects have opened up exciting new opportunities to chemically tailor semiconducting carbon nanotubes for imaging, sensing, and photonics needs such as lasing, single photon emission, and photon upconversion. However, experimental measurements on the trap depths of these defects show a puzzling energy mismatch between the optical gap (difference in emission energies between the native exciton and defect trap states) and the thermal detrapping energy determined by application of the van 't Hoff equation. To resolve this fundamentally important problem, here we synthetically incorporated a series of fluorescent aryl defects into semiconducting single-walled carbon nanotubes and experimentally determined their energy levels by temperature-dependent and chemically correlated evolution of exciton population and photoluminescence. We found that depending on the chemical nature and density of defects, the exciton detrapping energy is 14-77% smaller than the optical gap determined from photoluminescence. For the same type of defect, the detrapping energy increases with defect density from 76 to 131 meV for 4-nitroaryl defects in (6,5) single-walled carbon nanotubes, whereas the optical gap remains nearly unchanged (<5 meV). These experimental findings are corroborated by quantum-chemical simulations of the chemically functionalized carbon nanotubes. Our results suggest that the energy mismatch arises from vibrational reorganization due to significant deformation of the nanotube geometry upon exciton trapping at the defect site. An unexpectedly large reorganization energy (on the order of 100 meV) is found between ground and excited states of the defect tailored nanostructures. This finding reveals a molecular picture for description of these synthetic defects and suggests significant potential for tailoring the electronic properties of carbon nanostructures through chemical engineering.