Saying “I don’t care” is a bit rude, but its fairly normal to find some excuse and leave the conversation. Go get a refill or drinks, or say hi to someone else and join their conversation.