Polymer fibers, including natural silk and synthetic fibers, exhibit universal viscoelastic response. On stretching below yield, they show logarithmic stress decay. On unloading fibers with a glassy amorphous phase, the stress recovers. A simple phenomenological model accurately describes data from independent mechanical experiments and provides insights into the micro structural origins of the fiber response. Counter to intuition, the model indicates that it is the crystalline regions, rather than the amorphous glass, that deform first on stretching fibers at high strain rates. On holding a stretched fiber, stress decays as a consequence of relaxations in amorphous regions. Finally, unloading the fiber transfers stress from the amorphous to crystalline regions resulting in stress recovery. Model parameters correlate well with the fiber microstructure. Crystal and amorphous moduli from the model match those from x-ray diffraction. Activation energies for the temperature dependence of the peak relaxation time are similar to those reported in the literature. Thus, a simple model that invokes only crystal-amorphous coexistence can successfully model the mechanical response of a wide variety of polymer fibers.