[Microsoft Entra ID Entity Analytics] Adding custom options such as department do not take affect. #39419
Comments
andrewkroh
added
the
Team:Security-Service Integrations
|
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations) |
|
@efd6 would you mind taking a look at this one when you get a chance, please? See community Slack thread between Nic and Andrew for additional context: https://elasticstack.slack.com/archives/CNRTGB9A4/p1711751406843169?thread_ts=1711141962.535729&cid=CNRTGB9A4 |
|
Yeah, I was talking to Nic about this last week. |
|
@nicpenning What is the version of filebeat that you have installed on the agent host? I've taken a look at the code and the behaviour and everything looks like it is doing what it is expected to do. The custom fields feature was added in v8.13.0, so I'm wondering if you are running an older filebeat. Details of the investigationThere is no config state logging in the input, so add this… diff --git a/x-pack/filebeat/input/entityanalytics/internal/kvstore/manager.go b/x-pack/filebeat/input/entityanalytics/internal/kvstore/manager.go
index 97fa70dc6e..362fcda560 100644
--- a/x-pack/filebeat/input/entityanalytics/internal/kvstore/manager.go
+++ b/x-pack/filebeat/input/entityanalytics/internal/kvstore/manager.go
@@ -5,6 +5,8 @@
package kvstore
import (
+ "fmt"
+
v2 "github.com/elastic/beats/v7/filebeat/input/v2"
"github.com/elastic/elastic-agent-libs/config"
"github.com/elastic/elastic-agent-libs/logp"
@@ -38,6 +40,9 @@ func (m *Manager) Init(grp unison.Group, mode v2.Mode) error {
// Create makes a new v2.Input using the provided config.C which will be
// used in the Manager's Configure function.
func (m *Manager) Create(c *config.C) (v2.Input, error) {
+ var cm map[string]any
+ err := c.Unpack(&cm)
+ logp.L().Infow("MANAGER CREATE", "ucfg_config", cm, "configure", fmt.Sprintf("%p", m.Configure), "error", err)
inp, err := m.Configure(c)
if err != nil {
return nil, err
diff --git a/x-pack/filebeat/input/entityanalytics/provider/azuread/azure.go b/x-pack/filebeat/input/entityanalytics/provider/azuread/azure.go
index 30514352eb..0d96bed769 100644
--- a/x-pack/filebeat/input/entityanalytics/provider/azuread/azure.go
+++ b/x-pack/filebeat/input/entityanalytics/provider/azuread/azure.go
@@ -12,6 +12,7 @@ import (
"time"
"github.com/google/uuid"
+ "github.com/kortschak/utter"
v2 "github.com/elastic/beats/v7/filebeat/input/v2"
"github.com/elastic/beats/v7/libbeat/beat"
@@ -566,9 +567,16 @@ func (p *azure) publishDevice(d *fetcher.Device, state *stateStore, inputID stri
// configure configures this provider using the given configuration.
func (p *azure) configure(cfg *config.C) (kvstore.Input, error) {
+ logp.L().Infow("AZUREAD CONFIGURATION")
var err error
- if err = cfg.Unpack(&p.conf); err != nil {
+ var m map[string]any
+ err = cfg.Unpack(&m)
+ logp.L().Infow("AZUREAD CONFIGURATION", "ucfg_config", m, "error", err)
+
+ err = cfg.Unpack(&p.conf)
+ logp.L().Infow("AZUREAD CONFIGURATION", "config", p.conf, "error", err)
+ if err != nil {
return nil, fmt.Errorf("unable to unpack %s input config: %w", Name, err)
}
@@ -584,6 +592,7 @@ func (p *azure) configure(cfg *config.C) (kvstore.Input, error) {
// New creates a new instance of an Azure Active Directory identity provider.
func New(logger *logp.Logger) (provider.Provider, error) {
+ logp.L().Infow("AZUREAD NEW")
p := azure{
conf: defaultConf(),
}
@@ -592,7 +601,7 @@ func New(logger *logp.Logger) (provider.Provider, error) {
Type: FullName,
Configure: p.configure,
}
-
+ logp.L().Infow("AZUREAD", "state", utter.Sdump(p))
return &p, nil
}
diff --git a/x-pack/filebeat/input/entityanalytics/provider/azuread/fetcher/graph/graph.go b/x-pack/filebeat/input/entityanalytics/provider/azuread/fetcher/graph/graph.go
index 01d2d70702..751a6fd6cd 100644
--- a/x-pack/filebeat/input/entityanalytics/provider/azuread/fetcher/graph/graph.go
+++ b/x-pack/filebeat/input/entityanalytics/provider/azuread/fetcher/graph/graph.go
@@ -301,6 +301,7 @@ func (f *graph) addRegistered(ctx context.Context, device *fetcher.Device, typ s
// It will automatically handle requesting a token using the authenticator attached
// to this fetcher.
func (f *graph) doRequest(ctx context.Context, method, url string, body io.Reader) (io.ReadCloser, error) {
+ logp.L().Infow("AZUREAD DO REQUEST", "method", method, "url", url)
req, err := http.NewRequestWithContext(ctx, method, url, body)
if err != nil {
return nil, fmt.Errorf("unable to create request: %w", err)
@@ -334,6 +335,10 @@ func New(cfg *config.C, logger *logp.Logger, auth authenticator.Authenticator) (
return nil, fmt.Errorf("unable to unpack Graph API Fetcher config: %w", err)
}
+ var m map[string]any
+ err := cfg.Unpack(&m)
+ logp.L().Infow("AZUREAD GRAPH CONFIGURATION", "ucfg_config", m, "graph_config", c, "error", err)
+
client, err := c.Transport.Client()
if err != nil {
return nil, fmt.Errorf("unable to create HTTP client: %w", err)
@@ -384,9 +389,12 @@ func New(cfg *config.C, logger *logp.Logger, auth authenticator.Authenticator) (
func formatQuery(query []string, dflt string) string {
if len(query) == 0 {
+ logp.L().Infow("AZUREAD DEFAULT QUERY", "query", dflt)
return dflt
}
- return "$select=" + strings.Join(query, ",")
+ q := "$select=" + strings.Join(query, ",")
+ logp.L().Infow("AZUREAD USER-CONSTRUCTED QUERY", "query", q)
+ return q
}
// newUserFromAPI translates an API-representation of a user to a fetcher.User.and build a filebeat for drop-in. Upload to agent… and confirm… This gives the following log (lead up — shows that AZURE DEFAULT is used in the first and third cases, and AZUREAD USER-CONSTRUCTED QUERY is used in the second with the expected result) compare with defaults (lead up — shows that AZURE DEFAULT is used in all cases) Comparison of requests ---
+++
@@ -1,5 +1,5 @@
{
- "@timestamp": "2024-05-05T22:52:28.364Z",
+ "@timestamp": "2024-05-05T22:56:33.389Z",
"component": {
"binary": "filebeat",
"dataset": "elastic_agent.filebeat",
@@ -19,5 +19,5 @@
"message": "AZUREAD DO REQUEST",
"method": "GET",
"service.name": "filebeat",
- "url": "https://example.com/v1.0/users/delta?%24select%3DaccountEnabled%2CuserPrincipalName%2Cmail%2CdisplayName%2Csurname%2CjobTitle%2CofficeLocation%2CmobilePhone%2CbusinessPhones%2Cdepartment"
+ "url": "https://example.com/v1.0/users/delta?%24select%3DaccountEnabled%2CuserPrincipalName%2Cmail%2CdisplayName%2CgivenName%2Csurname%2CjobTitle%2CofficeLocation%2CmobilePhone%2CbusinessPhones"
} |
|
Wow, what analysis! I believe I was running 8.12.2 for the Elastic Agent / Filebeat. Since I am at 8.13.2 for the agent now I will see if there any changes. Tha k you for the detailed review. I will get back to you in the next 24 hours. |
|
Alright, so my agent is running 8.13.2 with the integration version of 1.1.1. I still do not see the department field anywhere in the docs. Will this data be in the 24h full sync and/or the 15m changes update? I have looked across both. I can confirm that the URI contains department when looking at the Graph API logs. So it may be possible that even though the field is being added to the query, the response is not including that data. I did check with another user in the org to use the same query that the integration uses and they did not receive department data in their request. This is leading me to believe that this is an issue on the Graph side. I will see if I can replicate this query information outside of Filebeat. This is my config: I can also confirm that the Filebeat.exe that exists and running via the process ID that correlates (I did status --output full) with the integration is 8.13.2. |
|
Update - When running this query with encoded characters, department does not return: Running the same query decoded, department does come back: My guess right now is that the encoding of the URI is part of the issue here. |
|
Update - We have high suspicion that the %3D (which is = in hex) is breaking the query: |
|
@nicpenning, thanks. That means that the original default queries do not work either, and are presumably just either coincidentally the same as what we say we are asking for, or do not actually match. Moving this to beats. |
|
Will this be slated for 8.13.4? Or must I wait for a minor release of 8.14? |
|
Currently in 8.14.1 with version 1.1.1 of the Microsoft Entra ID Entity Analytics. The department does not exists for the regular 15 minute syncs, and every since I have upgraded this agent, the 24 hour sync has not happened. Will the department show in all user entity data or just the syncs? I am adjust the sync time to 5 minutes to see if I can catch any errors or issues since it has been almost 36 hours since the upgrade of the agent. |
|
Confirmed that no full sync occurs since after I have upgraded from 8.13.4 to 8.14.1.
|
|
Update, last night the sync did occur at 20:50 (8:50 PM CST). Unsure why there was such a long gap between syncs. Also, it appears that the department information has arrived in the form of these fields:
|
I added - department as an option to the Entra ID integration but it did not work on the 15 minute checks or the full 24 hour sync.
I did not see any issues in the error logs but it is a busy agent with 20+ integrations.
Am I missing something?
Conversation here: https://elasticstack.slack.com/archives/CNRTGB9A4/p1711751406843169?thread_ts=1711141962.535729&cid=CNRTGB9A4
Also, if I removed fields from the list and only supplied one field, nothing changes, and the same default fields come in as said.
The text was updated successfully, but these errors were encountered: