Recommended architecture for Parca deployment

[gites]

I want to ask about architecture recommendations for Parca deployment.

I want to deploy Parca into multi-namespace and multi-application (go, java) environment.
My environment size is:

• ~30 k8s nodes
• ~1500 pods
• ~750 go pods (pprof), ~150 per namespace
• ~750 java pods

I'm planing to deploying Parca in the following way:

- go-namespace1
  - parca-scraper ( deployment with single replica running in scrape mode only and sending data to the parca-server)
[...]
- go-namespaceN
  - parca-scraper ( deployment with single replica running in scrape mode only and sending data to the parca-server)
- java-namepsace1
[...]
- java-namepsaceN
- managment-namepsace
  - parca-server (deployment with multiple replicas, S3 as persistent storage)
  - parca-agent (demonset) 

My questions:

1. Do you consider my plan for Parca deployment as good or should I change something?
2. Any recommendations about resource configurations for parca-scraper / parca-server / parca-agent? I guess it my by different for every environment but maybe there is something I can start with.
3. Does it make sens to run multiple replicas of parca-scraper scraping the same endpoints? Will the data be somehow deduplicated on the parca-server level? Running single replica seems fine for now, but I'm curious about that.
4. How parca-server handles situation where multiple scrapers are writing to multiple servers (scraper1-> server1, scraper2->server2) and the persistent storage (S3) is enabled? How long it will take for data written to server1 to be available for server2 for querying ?
5. When configuring cross-namespace service for scrapers (and agents) should I use a headless k8s svc or the "normal" one? I see the that official Helm chart is using the "normal" one.
6. Does it make sens to split parca-server into separate deployments for GRPC writes and Web UI for users ? For example multiple replicas for GRPC writes and single instance for Web UI (so all the metadata and cache from previous queries is always there). Will that make user experience/overall performance better in any way?
7. Any other ideas or suggestions ?

[brancz]

1. If this is all one Kubernetes cluster, I recommend running one Parca Server for everything, that also does the scraping.
2. The only one that we can be sure about here is parca-agent, we at Polar Signals recommend to our customers to run it with:

resources:
  requests:
    cpu: 10m
    memory: 100Mi
  limits:
    cpu: 100m
    memory: 500Mi

And we've actually gotten even better so that we're likely to lower tighten these a little more soon, but we haven't experimented with that yet. The scraper should be similar but it depends a bit on the throughput. The Parca server is very hard to say as it depends on the load you put onto it, when using the Parca Agent and no scraping you can expect roughly 10Gb of data per core per month for storage. We generally recommend only resorting to scraping if you absolutely must, as it tends to produce way more data than necessary (as a starting point the Go pprof profilers sample at 100Hz whereas Parca Agent samples at 19Hz).

3. Theoretically this could make sense, but unfortunately, most profilers don't allow this and it would be prohibitively expensive on the workload. For example, the Go pprof HTTP endpoints for CPU profiling will return a 500 error when running concurrently. The good news here is it's also not necessary, profiling data is a bit more forgiving if there is a small gap compared to metrics for example. As statistically things still work out if profiling is done continuously over time. It's not meant for high-fidelity alerting or audit logs.
4. This is controlled by the --storage-active-memory flag. It takes however long it takes to fill up that amount of memory as that's the trigger to flush data to object storage. This is also why I would always recommend running one large Parca server.
5. That depends on the purpose, a regular Parca server should definitely have a ClusterIP type service, whereas the scraper and agents should only have headless ones.
6. It's a bit more complicated than that. Making Parca distributed is very hard, and it's essentially what we've built at Polar Signals Cloud and even though we've been working on that for close to 2 years we're not fully done yet in every aspect. I recommend either running one very large Parca server, or we can get you access to Polar Signals Cloud.

I hope this clears some things up!