I wrote a bitcoin transaction parser in python and it flips out when transactions have more than 255 inputs or outputs. Some transactions have their number of inputs/outputs represented by a 1 byte hex string, others are represented by more. How do I know (and how do I program my parser to know) whether a tx's representation of inputs/outputs is 1 byte or more?
The number of inputs/outputs is encoded as a "var_int" or "variable length integer".
Start by looking at the first byte. If it's 252 or less, then that's it, that's your number (like you've been using), there are no more bytes to read.
If the first byte is 253, there are 2 more bytes following it, which are a 16 bit little endian integer. 254 means 4 bytes following it (5 in total), which is a 32 bit integer. 255 means 8 bytes follow, i.e. a 64 bit integer.
You can read more here: https://en.bitcoin.it/wiki/Protocol_documentation#Variable_length_integer