[target-allocator] targets get dropped when using relabel-config Filter with serviceMonitor

[moh-osman3]

In #1127 a new CRD field called filterStrategy was added. However, the relabel-config option drops all targets in the case of using a serviceMonitor that uses the hashmod action in the relabel_config. The resulting label will be matched with the environment variable $(SHARD), but in the prehook no environment variable substitution is taking place.

       {
        "action": "hashmod",
        "modulus": 1,
        "regex": "(.*)",
        "replacement": "$1",
        "separator": ";",
        "source_labels": [
          "__address__"
        ],
        "target_label": "__tmp_hash"
      },
      {
        "action": "keep",
        "regex": "$(SHARD)",
        "replacement": "$1",
        "separator": ";",
        "source_labels": [
          "__tmp_hash"
        ]
      }

What I expected

filterStrategy: relabel-config to work properly with serviceMonitors. i.e. targets are filtered properly according to prometheus relabel_config

What actually happened

I observe serviceMonitors running in my cluster but no jobs discovered when I set filterStrategy: relabel-config and run

$ curl http://lightstep-collector-targetallocator:80

When checking logs I notice that all targets are dropped.