No, that's not it - I respect him a lot. I don't want to mess up his place.
At least not without his permission.
Other options:
no subject
No, that's not it - I respect him a lot. I don't want to mess up his place.
At least not without his permission.