Optimal control and Reinforcement Learning (RL) deal both with sequential decision-making problems, although they use different tools. We have investigated the connection between these two research areas and in this talk, I will present the results of my thesis. In the first part, I will discuss an optimal control problem with uncertain dynamics showing how this formulation can describe what happens during some RL algorithms. In particular, I will present some convergence results for the value function and for the optimal controls. In the second part, I will propose a new online algorithm dealing with LQR problems where the state matrix A is unknown. Joint works with M. Falcone, M. Palladino and A. Pacifico.