If this is true, when I make a 64 character hexadecimal private key I need only provide randomness for the first 40 characters (40 x 4 bits per character = 160). The remaining 24 characters can be zeros.
No, that's not correct at all. If someone knew that your private key contained 96 zero bits, their search space would be drastically reduced. 256-bit private keys are used because they are the minimum considered sufficient to provide the level of ECDSA security required. 160-bits can be used in the hash function because hash functions get more security per bit than ECDSA does.
You could, however, use 160-bits as a seed to generate a 256-bit ECDSA private key. For example, you could use a SHA-256 hash of a 160-bit seed as your private key with no loss of security. You can even go down to 128-bits with no significant loss of security.