[TopBeat 1.2.3.] Can't access PID, even with correct rights.

[csuka]

Original thread here: https://discuss.elastic.co/t/server-2012-r2-some-metrics-not-send/51403

• Version: TopBeat 1.2.3
• Operating System: Windows Server 2012 R2 Standard (VMware host)
• Steps to Reproduce:
• Installed TopBeat via Powershell with Administrator Rights (right click, run as admin).
• In processes, it is seen that TopBeat is owned by SYSTEM. Topbeat is also executed as SYSTEM.
• The error in topbeat.log with debug level:

2016-05-31T11:00:13+02:00 DBG Skip process pid=0: error getting process state for pid=0: OpenProcess fails with The parameter is incorrect.
2016-05-31T11:00:13+02:00 DBG Skip process pid=4: error getting process state for pid=4: OpenProcess fails with Access is denied.
2016-05-31T11:00:13+02:00 DBG Windows is interactive: false

For convience, the full image:

[asednev]

This issue also exists in Topbeat 1.3.1

[tsg]

It would be useful to know what processes are not captured. The low PIDs numbers make me think they are system processes.

@asednev To clarify, you get the error for some of the processes but most of the processes are captured?

[asednev]

In my case none of the processes were captured.

[tsg]

What Windows version do you have? You start the Powershell session with "Run as Administrator", right?

[asednev]

Windows Server 2012 R2 version 6.3 (build 9600)

btw, I discovered that older version Topbeat 1.2.3 works well on the same VM.

[asednev]

Topbeat 1.3.1 I ran as windows service running under "Local System" account and also from command prompt "Run As Administrator". Neither worked.

Topbeat 1.2.3 mysteriously works.

[andrewkroh]

To help debug the issue, could you use SysInternals - Process Explorer to examine the permissions of the running process.

I suspect the interesting privilege might be the SeDebugPrivilege.

This security policy could be related. https://technet.microsoft.com/en-us/library/cc976527.aspx?f=255&MSPPError=-2147217396

[andrewkroh]

There is one place in gosigar where it requests PROCESS_ALL_ACCESS permission but the operation that it performs, GetProcessMemoryInfo, requires only PROCESS_QUERY_INFORMATION and PROCESS_VM_READ. gosigar should only request the permissions that it requires. Not sure if this would address the problem at hand, but in any case it's something that should be changed.

In other parts of gosigar it requests PROCESS_QUERY_INFORMATION, but in Windows Vista and newer there is a more limited privilege we can request called PROCESS_QUERY_LIMITED_INFORMATION. This can be used for GetProcessTimes, GetProcessImageFileName, and GetExitCodeProcess. It cannot be used for OpenProcessToken.

[andrewkroh]

I haven't figured out what OS settings are causing the issue for these users. Some feedback is required (see my earlier request for SysInternals data).

I have made the changes mentioned in my previous comment. See elastic/gosigar#50. We can put together a build of Topbeat/Metricbeat with these changes and see if the issue persists.

[monicasarbu]

Closing for now. @csuka Feel free to re-open the issue if the error persists.

[andrewkroh]

@csuka @asednev Would you be able to test the snapshot build of Metricbeat containing my changes on a machine that was having the "Access is denied" problem for all processes. BTW Metricbeat is replacing Topbeat in 5.0.

Access is denied is expected for some system processes, specifically the csrss.exe processes. And "incorrect parameter" is expected for PID 0. From the OpenProcess docs:

If the specified process is the System Process (0x00000000), the function fails and the last error code is ERROR_INVALID_PARAMETER. If the specified process is the Idle process or one of the CSRSS processes, this function fails and the last error code is ERROR_ACCESS_DENIED because their access restrictions prevent user-level code from opening them.

[csuka]

@andrewkroh Destroyed the VM on which it was failing.
Will check if I can run it somewhere else, will take a while.

[asednev]

@andrewkroh, I tried metricbeat-6.0.0-alpha1-SNAPSHOT-windows-x86_64 and it still threw same errors:

2016/10/26 16:44:33.521779 helper.go:276: ERR Error getting process details pid=91076: error getting process mem for pid=91076: OpenProcess
fails with Access is denied.

2016/10/26 16:44:33.527793 helper.go:276: ERR Error getting process details pid=702776: error getting process mem for pid=702776: OpenProces
s fails with Access is denied.

I see system metrics logged to Elasticsearch but none of the process metrics.

[andrewkroh]

@asednev Thanks for testing. Can you check in the process explorer what the names of those processes are? Are you seeing any events in Elasticsearch where metricset.name:process?

[asednev]

I don't see any events with metricset.name:process in Elasticsearch. Only cpu, memory, filesystem, and network.

Actually the list of PID in the output was rather long. I only copied 2 as an example. I went through the list of PID, these are the processes that I'm looking to monitor - our application services.

[andrewkroh]

Sounds like the changes I made didn't help. In the metricbeat download you used, can you tell me what the commit has is inside the .build_hash.txt file. Just want to confirm that the version has my changes.

Are you aware of any local or group security policies that were tightened on this machine? The reason I ask is that it would be nice to be able to reproduce this locally so I don't have to take up your valuable time.

Could you run Sysinternals - ProcExp (like I did in #1897 (comment)) and check the privileges of Metricbeat?

There is one other change that I think could help. It basically involves escalating the Metricbeat processes access token to include the SeDebugPrivilege, like what is described in https://stackoverflow.com/questions/2932461/windows-vista-win7-privilege-problem-sedebugprivilege-openprocess.

[asednev]

I compared the list of privileges between machine where Metricbeat can get list of processes and machine where Metricbeat can't get the list of processes. The list of privilages is the same. Also, I will reiterate that Topbeat 1.2.3 can get process-level metrics on both of these machines.

[asednev]

Also, it's worth mentioning that machine where Metricbeat can't get process-level metrics is production machine that was hardened by InfoSec team. I'm not entirely sure what exact steps go into hardening.

[andrewkroh]

@asednev I added code locally to metricbeat that tries to add SeDebugPrivilege to the process' access token. It's just test code at the moment and should print "AdjustTokenPrivileges success" to stdout at start. This version also has a bit more info in the error messages that will help identify the call that failed due to "Access is denied".

https://s3.amazonaws.com/beats-files/metricbeat.exe

If you are willing to try this binary and report the output this would be very helpful. Thanks

[asednev]

Voila, that fixed the problem. SeDebugPrivilege is Enabled in ProcessExplorer and I can see process-level metrics in Elasticsearch.

[andrewkroh]

Awesome, thanks for the help. I will work on cleaning up my code and put out a PR to address the problem. I'll post here once the changes are available in the regular snapshot build.

[asednev]

Thank you @andrewkroh. I'm glad you were able to come out of this with a fix.

[andrewkroh]

The snapshot build now has the fix for this if you'd like to test it out. https://beats-nightlies.s3.amazonaws.com/index.html?prefix=winlogbeat/

If you look in the log output there will be some lines containing "SeDebugPrivilege" that provide some feedback. In your case, @asednev, you should see "SeDebugPrivilege is now enabled.".