Mercy, sir, all ya had to say was no. [This insistence on being so formal is oddly uncomfortable - he's just trying to make conversation.]

I'm afraid I can't go handin' out pardons when you've done nothin' particularly wrong, so let's leave it at "never mind all that," if that works for ya.