VIP11: Shared Memory revamp

Closed: VSC implemented, SLT WONTFIX

Bugwash decision, see https://github.com/varnishcache/varnish-cache/issues/2830:

• dynamic VSCs are done

• dynamic SLTs would require more effort than justified by the benefit: Using a fixed tag (like SLT_VCL_Log) and a prefix (as std.log does) should be sufficient for all use cases currently on the table - and the performance benefit of dynamic SLTs would be marginal in comparison

Synopsis

Remodelling of Shared Memory

Why?

To make it possible for VMODS til add counters and VSL records

To make shm file self-describing across versions etc.

How?

VSC counters will get a unique numeric ID like SLT tags.

In both cases the name is no longer unique, since a newer version of a VMOD may redefine a counter or SLT tag.

Descriptions of the VSC counters and SLT tags, including the numeric-ID -> name translation is stored in a .JSON file next to the SHMLOG file.

The MGT process recreates this .JSON file for every worker process restart.

The worker process appends to the JSON structure, both for its own VSC/SLT and for VMODS additions, but create/write/renames a new JSON file so that the primary filename always leads to consistent JSON.

When VMODs are loaded, they register their SLT and VSC requirements with the worker process along with the JSON descriptions that goes with them. (This should happen from generated code, see below).

VSL_tag_e becomes "struct VSL_tag { unsigned id };" to allow allocation of new ids for VMODS.

VSC + SLT described in .RST files (.vcc for VMOD) and python spits out C structs & register code (.h + .c) including the JSON descriptions.