The thermal conductivity (κ) of graphene dramatically decreases once supported on a substrate, hindering its use for thermal management. To clarify the underlying mechanisms, we investigate the κ of graphene on amorphous SiO2 by using molecular dynamics with particular attention to the graphene-substrate topography. Our analysis reveals that the suppression in κ increases with the nonuniformity of the forces acting on graphene, which tends to increase as the substrate-surface roughness and graphene conformity increase. Our findings highlight the importance of the interfacial morphology on κ and can provide guidance on the design of substrates to improve thermal transport through graphene.