Better integrate self-hosted into our software development process

[chadwhitacre]

Self-hosted installations account for approximately 10% of the ~85,000 organizations we count as Sentry users, and, increasingly, these are a source of major new SaaS accounts. Sentry has a commitment to maintaining self-hosted as a usable packaging of Sentry for low-volume and proof-of-concept deployments, as well as a blueprint for high-availability deploys.

Self-hosted sits at the end of Sentry's development pipeline:

We participate in the Backend and Infrastructure TSCs, but we don't have enough process around ensuring that feature development or operational changes to Sentry make their way into self-hosted in an orderly fashion. This has resulted in close calls and a pile-up of work to digest:

• Bump default Postgres from 9.6 to 14 self-hosted#1610
• Include Session Replay in Self Hosted self-hosted#1873
• Add Profiling self-hosted#1838
• Enable Issue Platform for self-hosted self-hosted#2269
• Add Spike Protection self-hosted#2122
• Deprecate sessions self-hosted#2200
• Add Crons self-hosted#1993
• Track geo of self-hosted instances self-hosted#2713
• React to Hybrid Cloud changes self-hosted#1869
• Add Code Coverage Insights self-hosted#2174

What's more, the Sandbox is downstream of Self-hosted, and we definitely want new features to land in the Sandbox, probably at GA.

The outputs of this epic are:

• Alignment with EPD leadership on expectations, documented in Notion.
• Training for EMs/ICs.
• Probably rewrite the test suite in Python (or at least clean it up a lot and comment it).
• At least one of the above specific work items completed.

In order to kick this off in Q4, we'd like to invest in two areas.

• Improve self-hosted Sentry initialization to run <5 minutes in VM's/CI self-hosted#2528
• Land initial groundwork for better self-hosted testing infrastructure self-hosted#2529

┆Issue is synchronized with this Jira Epic by Unito

[chadwhitacre]

Team changes, less bandwidth, need to shed scope to stay afloat. Dropping this from Q1. Likely revisit after First Class Single Tenant🔒 lands, will aim to follow suit.

[chadwhitacre]

We had a suggestion to use one of the headless browsers for integration tests, forget exactly which one was suggested but one of them would be good.

[chadwhitacre]

https://pptr.dev/

[BYK]

pptr.dev

playwright.dev

[chadwhitacre]

Right! GOOG vs. MSFT

[BYK]

@chadwhitacre not just that. Puppeteer locks you into Chrome whereas you can test even on Safari with Playwright.

[prom3theu5]

Not to mention playwright supports multiple concurrent "pages" or sessions in test runs
Its pretty good :)

[azaslavsky]

Paraphrasing a discussion we had offline:

I think we're pretty close to building out a generic turn-key cloud VM that provisions and sets up a fresh cloud instance with self-hosted at a given SHA downloaded installed and ready to go. If we install VSCode server on here, I think we could use VSCode remote to develop/debug on this instance? 3 reasons why I think this is desirable:

1. We use the same architecture for development as our services when they run; no more apple-silicon warnings from docker. Biggest benefit imo.
2. Easy to spin up new instances on command, and keep work isolated. I feel like self-hosted instances develop “cruft” fairly quickly, making a full-restart necessary kinda often. This takes a long time.
3. Multiple dev environments one could easily switch between. These would not collide with one another.

Biggest downside is potential latency (VSCode has been great for remote dev like this in the past for me, but idk), the setup being finicky, and cost.

[chadwhitacre]

Example fail. :-/

[aldy505]

A few new feature tickets:

• Support new Beta User Feedback feature in self-hosted self-hosted#2646
• Add Developer Metrics self-hosted#2698