When confirming that an address entered on a site/app against what I expect it to be, I often check the first and last 3-4 characters to see if they match.
This is a good practice, it will help protect you against attacks which do things like highjack your clipboard data, to paste an attacker's address instead of your intended recipient's address.
Relevant info: bitcoin addresses are base58check encoded, so the last few characters of an address are deterministically generated, using the beginning of the address as input. Altering just one character at the beginning (or end) of an address will make the checksum invalid. So an attack which alters just one character of an address will not work (even if this sort of attack didn't steal funds, an attacker could force you to burn funds otherwise).
If an attacker can find a way to just outright present you with a malicious address, then checking digits won't be of use. But for the sake of your question we'll ignore those situations, and just consider how easily an attacker could create a 'copy-cat' address, in order to thwart any quick references to the legitimate payment address.
Generating a bitcoin address has a very low cost, so it is possible for attackers to generate many, many addresses quite easily. Only a small percent of those addresses will be similar to the desired output, and every extra character you verify increases an attacker's costs of creating a copycat address by 58x (since there are 58 characters used in base58check address encoding). So we see that by verifying enough characters, the attack becomes computationally infeasible, as the attacker would have to check an impossibly large number of addresses before finding one that matches.
As an example with numbers: near the end of Mastering Bitcoin: Chapter 4, Andreas highlights that vanity addresses can be dangerous, in that an attacker can make an address that also begins with the same vanity characters. The attacker can then substitute their address for the legit one, and since it is a vanity address this substitution may be less obvious than usual (ie, the user sees that the start of the address matches, and proceeds without checking the end).
Could an attacker feasibly generate an address that matches at the beginning and end, but is different in the middle?
If you verify enough characters, then the attacker could probably not generate a copycat address within the time constraints of the attack vector. The attacker would not be able to start brute-forcing the address until the payment address has been created (and intercepted), and then the time would run out once a user submits payment. In many real-world situations, this time period may only be minutes (or perhaps hours) long. Using the numbers in Andreas' book, we can see that checking 4 characters on the start and end of an address should be sufficient to thwart any attacker that only has <1 hour to generate a copycat address.