In an ideal world, we would want all programs to be auditable/verifiable(we can verify every execution path to be secure and correct). But that's far from reality, when you download a binary from a source(for example, Bitcoin Core), you have the following choices.
1) Trust the developers of the binary on their claim "binary works as intentded"
2) Verify the source yourself (It's open source for this reason) and build your software yourself.
All PR's to Bitcoin Core are visible to everyone, so everyone will be able to detect any harmful PR. An effective attack would involve bribing all people who actively watch the repository.
The best solution review the code yourself. If this attack really gives you nightmares, reviewing code yourself is the only way in which you don't have to trust anyone.
A more realistic solution is what happens today, you can rely upon hunderds of people who keep a watch on bitcoin repository to report if there are any backdoors in code.
EDIT: As pointed out by Peter Wuille, Bitcoin Core has a deterministic build process. You can repeat the build procedure for any release yourself, and verify that the resulting binary is identical to the release binary which you downloaded. This means that you don't necessarily have to trust just developers that the binary is created from publicly audited source code.