We introduce an iterative procedure for constructing subsolutions of deterministic or stochastic optimal control problems in discrete time with continuous state space. The procedure generates a non-decreasing sequence of subsolutions, giving true lower bounds on the minimal costs (value function). In each step, state trajectories starting from a fixed initial point are computed which serve to update the current subsolution. The values of the subsolutions at the initial point can be shown to converge from below to the minimal costs. Joint work with Paul Dupuis and Hui Wang (both Brown University).