Advancements in technology unlock an increasing number of ways to tailor farming practices to particular farm characteristics. For example, farming data collected from various sensors can guide customization of fertilizers and enable data-driven improvement over time. This is a typical Reinforcement Learning (RL) problem as crop quality depends on a range of factors in a complex and often obscure way. For RL to be integrated into the process of fertilizer customization, it is crucial that the techniques are robust. An aspect of robustness is the capability to detect and adjust for model errors. This entails both model selection and parameter estimation. How to achieve both objectives simultaneously while maintaining provably good performance is an open problem that we intend to address.