Impact of asset tokens on transaction fees?

Ok, so from what I understand, when you get a lot of incoming rvn transactions into your wallet, it gets messy, raising the information cost of an outgoing transaction, and thus the fees. How do asset token transactions factor into this? Do incoming transactions in XYZ make outgoing RVN transactions more expensive, or is the added information cost only relevant when you have to send XYZ out?