Earlier you fail, it's better. Start design in the ideation phase, even before setting defined goals or commitments. Thinking for visual designs provides enough space to think deeply about complexities and limitations and a better perspective of solving the problems.
Consider it as a value addition, as we turn an idea to actionable one, which helps in faster validation. Design Sprints
suggests getting feedbacks from initial prototypes.