Wouldn't it be saver to generate a mnemonic passphrase manually by just taking random words from a lookup table?
No, it would not be. This method may even be insecure if you are actually doing it manually.
A BIP 39 mnemonic, if generated properly, is secure. It is just as secure if you randomly choose words too. This is because both methods require a Cryptographically Secure Random Number Generator. For BIP 39, you use a CSRNG to generate a 256 bit random number and split it up in order to produce the mnemonic. There's an additional checksum that makes the mnemonic represent a 264 bit number.
In your method, you generate 24 random 11-bit numbers (because there are 2048 words in the wordlist) and use each number to choose the word to use. While this is more random, you lose the checksum. Furthermore, it's only 8 bits more random, which is not much more random. Either way, the mnemonic is sufficiently secure, not because of the method used, but because a CSRNG was used to generate the numbers.
However if you actually do it manually, then it is likely insecure. If you are generating the numbers yourself (i.e. without the help of a CSRNG), then your seed is not going to be sufficiently random enouhg. Humans are a notoriously bad source of entropy, but you want your mnemonic to have good entropy. So doing this manually is not a very good idea.
Although you could use a 6 sided die to randomly generate your numbers, but that may not necessarily be a good source of entropy either. It would also require a few hundred rolls in order to have a sufficient number of bits.
a wallet creator can can accidentally or on purpose use deterministic seeds for creating the mnemonic pass phrase
They could, but it would be noticed very quickly. The wallet is open source and people do actually audit the source code. People would notice if a change went in that modified its random number generation.
but also using ledger nano s - so there is not really a way on verifying that the threat above is not implemented on the HW wallet, correct?
Not necessarily. Most of the Ledger Nano S's firmware is open source. And the seed is generated using from the secure element which has a True RNG which is AIS-31 Class PTG.2 compliant. You can check that the firmware calls the correct functions in the Secure Element SDK to use the TRNG for seed generation.
The Secure Element's firmware is closed source unfortunately, however I don't think Ledger can modify that firmware. I believe they only get a binary blob for the firmware from the chip's manufacturer, STMicroelectronics.