"It’s hard to know who your aunts and uncles are when your mom’s a part of a large family," Jinger acknowledged