Nope, don't worry. The nodes you have connected only know that your IP address has downloaded the blockchain, but the transactions you'll make in the future won't contain information specific about it.
The main threat to your privacy are contact points between the blockchain and real life (when you make a purchase and pay with Bitcoin, when you buy at an exchange that know who you are, when you publish an address in a forum...) because that puts your name next to an address someone could eventually follow.
When you make a transaction without Tor you could also compromise your privacy, but it is a long shot because nodes don't store the IP addresses that sent the transactions (and there is nothing in the blockchain either). There is still a chance because someone could modify a node to record the IP addresses and, if you connected to that node, it would record your IP address and your Bitcoin address together. By making that rogue nodes highly available they would increase the chance that you connected to them. There was at least one case of a company doing this in the past that has been discovered, but they said it was something like and experiment.
If you want to learn more about Bitcoin and privacy you can start at:
www.bitcoinisnotanonymous.com