“I never bring something that the hostess can serve right then because it disrupts their meal,” said Garten