In this talk, we will review some known results on the link between reachability analysis and optimal control problems in the deterministic setting. We will show how this link can be used to get a constructive way for the characterization of the epigraph of the value function of constrained control problems, without assuming any controllability assumption. Some extensions of these results to the stochastic setting will be also discussed.