Learning with Structured Priors for Robust Robot Manipulation