In the past 5 years of using Matrix, I have received exactly 2 direct spam messages, and seen maybe 5-10 in public rooms. (There have been none in my private chats, of course.) If you’re seeing much more, I guess it must depend on how you use it and what rooms you join.