I think the guilt comes from wanting to please everybody, which is practically impossible. Some people argue, a no should not be accompanied by an explanation. it's simply a no/not interested.