The Bitcoin P2P protocol uses its own serialisation format for the transferred data. Is this serialisation considered as more secure against DoS attacks than JSON?
I do not know if there are any comparisons of the security difference between Bitcoin's P2P messages and JSON, but the binary format is certainly more compact than JSON.
I think it would be reasonable to assume that there are security implications, but security likely was not the major concern when the P2P format was initially defined. Rather ease of implementation and message size were the likely concerns.
A binary format is simply smaller and more compact than JSON strings. Strings take up a lot of space, especially strings for data that represent bytes. Furthermore, it takes more logic and time to convert binary data to a string, then convert it back from a string once received. Any string encoding for binary data is not going to be as efficient as the binary itself. For data that the user is never going to see, and with a protocol that isn't text based, using JSON is not reasonable.
Lastly, while security concerns likely were not behind this decision, there certainly are security implications for choosing a binary format over JSON. Variable length string parsing has traditionally been a source of security problems in many different software. As JSON is string based, there is the potential for anyone implementing something that uses JSON to make a mistake that introduces a serious vulnerability in their string parsing. So a binary format would be better in that regard.