By installing or running the "team viewer" program you gave the guy control over your computer using his keyboard and mouse.
After you entered the password to open your wallet, the wallet is unprotected and he could then use his keyboard and mouse to click buttons in your wallet to make a payment to his address.
I did not do this,
He did it
how did it happen?
by installing or running team-viewer, you gave him control over your computer.
Everything happened so fast.. Nobody knew my password or even my wallet key
He didn't need to know your password because he tricked you into entering your password when you opened your wallet.
How did my wallet send the initiate the transaction by itself?
Your wallet didn't do this itself, the scammer used his keyboard and mouse in the same way you would. Probably he used tools to speed up the typing so that it happened too fast for you to see all the details.