It's the only boundary the taproot/MAST limitation of 128 levels of the tree, ie. 2^128?
MAST isn't really relevant here. As the Ark vtxo trees are just trees of sequential transactions. In theory they can be arbitrarily large.
Some things are to take into account though:
More round participants means more interactivity. Since all participants in the round have to provide signatures for the round to be successful, more participants means higher likelihood a round will fail and will have to be restarted, so the longer a round will take, time-wise. Note that ASPs can limit the number of participants in a round if needed and even run multiple rounds in parallel.
More vtxo tree leaves means a higher cost for unilateral exit. The number of transactions a user has to broadcast (and pay fees for) for a unilateral exit is
nbeing the number of leaves in the vtxo tree. While this scales logarithmically, which is pretty great, it can still become a significant burden for users to participate.
When broadcasting multiple transactions for a unilateral exit, probably using a fee strategy like CPFP (child pays for parent), mempool policy puts an effective limit on the size of the exit transaction package. Currently, Bitcoin Core has a descendent limit of 25 transactions. Current work on cluster mempool will probably change this limitation and replace it with a cluster size limit (IIRC of 100). The way either of these limits affect Ark exit transactions is subtle and depends on how many people are exiting from the same Ark tx at the same time.
Eventually, I think the latter consideration will be the realistic limiting factor on realistic Ark vtxo tree size.
For example, using pre-signed transactions (clArk, i.e. covenant-less Ark) or using OP_CTV, this effectively limits the number of nodes in the entire tree*. With a limit of 25, that would mean a tree with up to 24 leaves and with a cluster limit of 100, that would mean 99 leaves.
However, with more powerful introspection opcodes**, the intermediate leaves could be fee-bumped so that this limit no longer holds.
*: In fact if only one leaf of the tree can confirm, it can make space for other branches. So let's say all nodes except the leaves is the limit. Meaning that for
n leaves, there has to be space for
n + 1 txs in the mempool.
**: like OP_TXHASH, OP_CHECKTXHASHVERIFY, OP_TX, direct introspection opcodes (or CAT+CSFS)