what happens if size of a transaction is larger than maximum block size?
Then the transaction can never be included in a block.
There is no mechanism for dividing it up into multiple blocks, or anything like that. Any block that includes it would be invalid, since the block would become too large for the network's rules.
This may be of interest: Block ...
Technically, this is possible. However, it would be a hardfork, so all users would have to upgrade. In such a large ecosystem that exists around Bitcoin today, this would be rather disrupting.
Such change would also have unclear security implications. Miners would be naturally eager to accept as many transactions as possible, so a spammer may gradually push ...
Nick Odell answers the original question:
The maximum transaction size is the size of the block. Source.
Transactions larger than 100 kilobytes (including witness at a 75% discount rate) are non-standard. Source 1. Source 2.
so the case that you are describing will not occur as transactions lager than the blocksize will be sorted out.
Modern Bitcoin Core nodes don't need to store the whole block chain (although they do need to download it all), which may solve the problem with your VPS. See this answer for more information about block file pruning.
The total amount of information in the block chain can't be reduced without making it impossible for new nodes to validate historic ...
In cases like VPS it seems that the storage on plans rarely increases much despite HDD size increasing nearly ten-fold since the beginning of 2009.
Depending on how you use the node there are several options to reduce storage requirements. Here are a few:
Enable pruned mode on the node.
Use an SPV client like Electrum in place of Bitcoin Core.