Self hosted sentry - node storage for production

nodestorage - this is a peculiar one; the default option on self-hosted sentry was/is to use postgres which is a terrible solution due to how much it bloats the database. Sentry, even on very small instances, produces a lot of records there and you will be pretty much forced into using pg_repack to reclaim the space (which might have its own downside - a major one - which is that pg_repack cannot recover after crashing due to running out of disk space - it leaves tons of hard-to-clean-up low-level postgres bloat on disk).

Fortunately nowadays you can enable S3 storage. Fairly easy if you use this helm chart - but you have to be aware of a few things:

• Sentry will do a significant amount of API calls to the S3 instance you’re using (obviously)
• you can try a self-hosted S3 alternative, but one major trap here is running out of inodes (something which I completely ignored at first, honestly speaking) - because sentry will write thousands and thousands of tiny files (‘raw events’) to the disk

After trying rustfs which was a Very Bad Choice - as it just writes files to disk as is (I also learned about fsGroupChangePolicy: OnRootMismatch lol) - I went with seaweedfs - which actually encapsulates data in its “volumes”.

Without going into details - a “volume” is just three files (dat, idx, vif) on disk which is great for sentry - as you can write thousands of tiny files coming from sentry into such a volume without blowing up inodes out of proportion.

I did a simple “all in one” seaweedfs deployment on my cluster and pointed sentry to it and it’s been a godsend so far. No more postgres crisis, no more bloated hundred-GB postgres volume and I don’t have to worry if the AWS bill will blow up because of the amount of API calls.