The Wannsee Conference didn’t contain Hitler, but did include many higher ups making the explicit plans for the Shoah. It wasn’t Hitler’s first plan, deportation was, but any reasonable understanding of his long term plans would require eventual extermination. He was invading places he had sent Jewish people to.

Also non-Jewish victims didn’t get the attempted deportation. Disabled Germans, criminals, communists, queer people, etc faced brutal prison labor and eventually extermination.

Ultimately, Hitler didn’t give orders in a way where you can say “he ordered this to happen”. He made his desires clear, and let the party vie for his favor. Had they not orchestrated the Shoah, party members would have been replaced with those who would