Just note that "your metadata" actually extends to less things than any XMPP and Matrix homeserver can see.
Signal knows your number, when you last connected, whether your account should be discoverable by phone # , how many devices you have linked, when you created your account, and whether anyone can use sealed sender or not.
If for some reason a message doesn't arrive, they store the encrypted data until it can be sent.
Meanwhile your average matrix homeserver has a full copy of chat history and more (including any other homeserver in a room) not including encrypted contents but including:
For XMPP, its much of the same.