Why are Bitcoin raw transactions in hexadecimal format? I am looking for scientific articles / books on this topic and I can't find anything, so my question is: why are they in hexadecimal and can there be any article on this topic?
Because unlike addresses, hex raw transactions aren't meant to be seen or used by end-users. Hex is easy to encode and decode (every two character in hex represents one byte*, whereas in base64 one character is 3/4 byte and Base58 is log(58)/log(256) = 0.7322476244 bytes) so it is convenient if you want to debug your implementation.
*: which means if you can find an erroneous byte in the transaction that you can CTRL+F for that byte in the serialized hex transaction and change it since unlike Base58 or Base64 an encoded character depends on the part of only one byte and never two neighboring bytes. Hex is the highest base with that property. Binary (1 bit divides 8 bits), Quaternary (2 bits divide 8 bits), not octal (3 bits don't divide 8 bits), and hexadecimal (4 bits is the largest divisor of 8 that isn't 8) are other bases with that property, so hex is the most efficient one among them.
Or simply, it was created as an internal tool which didn't need the complexity of a new format that provides space efficiency or avoiding similar characters (or checksum, even though hex doesn't depend on checksum) that is user-friendly.
From Bitcoin QT 0.7.0 release notes:
Added raw transaction API. See https://gist.github.com/2839617
I considered parsing/writing BIP 10 format for raw transactions, but decided that reading/writing BIP 10 format should happen at a higher level and not in the low-level RPC calls. So 'raw transactions' are simply hex-encoded into JSON strings, and encoding/decoding them is just a couple of lines of already-written-and-debugged code.
Why raw Bitcoin transaction are in hex format?
They are not.
So far as I know and have read.
Bitcoin raw format
According to https://bitcoin.org/en/developer-reference
Bitcoin transactions are broadcast between peers in a serialized byte format, called raw format. It is this form of a transaction which is SHA256(SHA256()) hashed to create the TXID and, ultimately, the merkle root of a block containing the transaction—making the transaction format part of the consensus rules.
and https://bitcoin.org/en/glossary/serialized-transaction defines Serialized Transaction, Raw Transaction as
Complete transactions in their binary format; often represented using hexadecimal. Sometimes called raw format because of the various Bitcoin Core commands with “raw” in their names
So the form transmitted over the wire is not composed of ASCII characters 0-9A-F (hexadecimal) but of numbers and other values in a normal binary form.
Software engineers like to use a hexadecimal encoding when writing and reading documentation. This hexadecimal encoding is simply one of many equally-valid visual-representations (binary, octal, ...). It just happens to be convenient and familiar to software engineers.
To take a specific example, the first part of a transaction is a version number. This could have the value 2. It is of data type int32_t, so it is 32 binary bits in length and represents a signed integer. I can write the value as 2 (decimal) or as 00000000000000000000000000000010 (binary) or as 00000002 (hex) but these are all just visual representations, none of them have any special meaning, you can't say that the value 2 is not transmitted. You could inspect and interpret that data using a tool that shows octal or binary or decimal or base 17 or any other arbitrary representation.