Ahh that makes sense, with more people involved it’s harder to go by. I use Fossify Contacts, and it has an option to save contacts but to make them invisible to other apps. Perhaps that might help you somehow, I’m not sure