You should do what you want to do, in this case, it seems like you want to do the double degree in mathematics and physics. You can always change after giving it a try. It is much better to do something and then regret it, rather than regret not doing it and wondering "what if".