Is there a way to know in above setup which input address sent which amount to output address?
I've looked into the transaction algorithm and my conclusion is that it is all mostly random matching, am I right?
My goal is to show bitcoin transfers between addresses (ideally ignoring the transaction in the middle) to achieve something like this
That is not possible.
Bitcoin doesn't require that each input be associated with one or more specific outputs. The only requirement is that the sum of inputs is greater than, or equal to, the sum of outputs.
No nodes attempt to infer the sort of associations you describe. It is not needed. No one cares. It is irrelevant to Bitcoins purposes.
When creating such a transaction, A node will simply sum the required outputs, then gather sufficient inputs for that total. The choice of inputs need not be influenced by the specific individual outputs at all. Many wallets choose inputs based on privacy concerns. There is absolutely nothing to be gained by attempting to align each input amount with one or more individual output amounts. Any excess is dealt with by inserting a change output somewhere in the list of outputs.