what happens if size of a transaction is larger than maximum block size?
Then the transaction can never be included in a block.
There is no mechanism for dividing it up into multiple blocks, or anything like that. Any block that includes it would be invalid, since the block would become too large for the network's rules.
This may be of interest: Block ...
Nick Odell answers the original question:
The maximum transaction size is the size of the block. Source.
Transactions larger than 100 kilobytes (including witness at a 75% discount rate) are non-standard. Source 1. Source 2.
so the case that you are describing will not occur as transactions lager than the blocksize will be sorted out.