The stability of the axisymmetric solitary waves of the Gross-Pitaevskii (GP) equation is investigated. The implicitly restarted Arnoldi method for banded matrices with shift-invert is used to solve the linearized spectral stability problem. The rarefaction solitary waves on the upper branch of the Jones-Roberts dispersion curve are shown to be unstable to axisymmetric infinitesimal perturbations, whereas the solitary waves on the lower branch and all two-dimensional solitary waves are linearly stable. The growth rates of the instabilities on the upper branch are so small that an arbitrarily specified initial perturbation of a rarefaction wave at first usually evolves towards the upper branch as it acoustically radiates away its excess energy. This is demonstrated through numerical integrations of the GP equation starting from an initial state consisting of an unstable rarefaction wave and random non-axisymmetric noise. The resulting solution evolves towards, and remains for a significant time in the vicinity of, an unperturbed unstable rarefaction wave. It is shown however that, ultimately (or for an initial state extremely close to the upper branch), the solution evolves onto the lower branch or is completely dissipated as sound. It is shown how density depletions in uniform and trapped condensates can generate rarefaction waves, and a simple method is suggested by which such waves can be created in the laboratory.