It does make you much better at formulating problems. In fact, from my experience as a mathematician and watching others do math, the vast majority of good problems comes after someone has solved a problem and then asked, "what if we change this instead?"