This.
And you're leading him on if you're "politely denying him".
Unless you want to? o.o
Tell him you don't want to loose his friendship.
Be assertive.
It might be awkward at first and he might not want to see you much, but that's the only way that he can get over it.
It might hurt him...