When an aqueous dispersion of negatively charged colloids, cationic polymer and crosslinker is frozen and the polymer is allowed to crosslink in the frozen state, we obtain a self-standing macroporous composite foam. This material is soft, despite the high concentration of colloids and yet is remarkably elastic to large compressive strains. In these macroporous composites, the pore walls comprise colloidal particles held within a crosslinked polymer network. Here, we investigate the effect of varying colloid- polymer interactions in pre-fabricated macroporous composites on their microstructure and mechanical properties. During preparation of the composite, cationic polymer adsorbs on the negatively charged colloids. We tune the surface charge of particles embedded in a macroscopic monolith by immersing the composites in water maintained at different pH. In this way, we tune polymer-particle interactions in the composite. We observe a sudden increase in interparticle distance and swelling of composite when pH is decreased below the particle's isoelectric point. Correspondingly, we observe reduction in Young's and shear moduli, compression strength and macroscopic energy dissipation. We did not observe any pH dependent changes in pure polymer sponges (prepared by ice templating and crosslinking a polymer solution that does not contain colloidal particles). Therefore, the pH dependent structural and mechanical property changes arise from the composite structure rather than purely from the crosslinked polymer. We believe that the reduction in mechanical stiffness when the polymer-particle interaction becomes repulsive is because of the reduction in interfacial contacts between particle and polymer. Therefore, the mechanical stiffness of ice templated composite is strongly influenced by interactions between the polymer and particle surface.