Taproot benefits everyone who uses it, but it has the most benefit to those who use complex scripts and multisigs. Since that group is a small portion of users of Bitcoin, that may be why some people are saying it is useful only to that small part of the population. However I would not characterize such statements as meaning Taproot is not useful to anyone else, nor interpret those statements as meaning Taproot is detrimental to anyone else.
The vast majority of users use single key addresses. These are the typical P2PKH and P2WPKH addresses. At this point, we are almost as simple as it gets for single key things - provide a signature and a pubkey and that's it. While Taproot brings improvements to single key things in the form of Schnorr signatures and a pay to pubkey scheme, the vast majority of users are not going to experience a whole lot of difference between P2WPKH and P2TR. The user experience is going to be almost the same - you click on a button to get a new address, people send money to that address, and then you can click on another button to send Bitcoin elsewhere. The only tangible difference is that it will be slightly cheaper to spend P2TR outputs and slightly more expensive to send to P2TR outputs. Even then, the difference is not that big so many users may not even notice it.
However for users who make multisigs or complex scripts, Taproot is a huge boon. All those multisigs and complex scripts can be really compressed down when spending. Instead of having to put large scripts into the blockchain, in the typical "everyone agrees" case, spending a Taproot output will make the transaction way smaller which really saves on fees. It is a significant improvement to those workflows because much less money is spent on transaction fees. There is also the added benefit that such transactions look just like single key transactions which is good for privacy. For those who have very large multisigs or very large scripts, the difference in fees will be noticed.
Even when it comes to using Taproot script paths, only the used parts are revealed in the transaction. This both reduces the transaction size and increases privacy by not revealing unused script paths. This is also a huge improvement.
Thus Taproot provides most of its benefits to users who make multisigs or complex scripts. This is a small part of the users of Bitcoin. It is not incorrect that Taproot is directed towards improving the experience of scripts and multisig. And since not many people use scripts and multisig, it is not incorrect to say that Taproot only affects a small part of the population.
However this does not mean that we shouldn't want Taproot. This doesn't mean that Taproot is bad. And people who state these things are not necessarily being misleading, lying, or don't want Taproot. They are stating what Taproot is going to be used for and the people that it provides the most benefit to.
In the context of the discussions I believe you are referring to, the argument is not that we shouldn't get Taproot, but rather that it isn't so important that we need to have it ASAP. Taproot provides a benefit to all of the people who will use it, but for most users, this benefit is not significant. We don't need it right now so there is no need to have so much urgency in trying to get Taproot. We want Taproot and it would be nice to have it now, but we shouldn't let the feeling of "I want it now" to get in the way of developing a safe way to activate it that can also be used in future soft forks.