Allow metricbeat to accept additional windows counters within the yaml config

[trevorndodds]

A handy feature would be to allow metricbeat to access the windows performance counters this will allow metricbeat to collect data from a wide range of windows metrics.

This would be similar to the Telegraf windows counter plugin.
https://github.com/influxdata/telegraf/tree/master/plugins/inputs/win_perf_counters

[AnderssonPeter]

Any news on this currently i have a powershell script that dumps counters to a CSV file and then send it with a file beat, but i would prefer if i could skip the powershell part.

[ruflin]

@AnderssonPeter No work has started on this. It would be interesting if you could share your script to get a deeper understanding on the metrics you are using.

[AnderssonPeter]

It's a work in progress so im not 100 sure it does what i want it to.

#Version 0.2

#Todo:
#* Notify on fail
#Function to get Sql Server Counters
function Get-SqlServerData()
{
    $Data = get-counter ($SqlServerCounterPrefix + ":Buffer Manager\Buffer cache hit ratio"), 
                        ($SqlServerCounterPrefix + ":Buffer Manager\Page life expectancy"), 
                        ($SqlServerCounterPrefix + ":Access Methods\Page splits/sec")
    #$Data
    $TransformedData = $Data.CounterSamples | Select-Object -Property Path, CookedValue

    $object = New-Object psobject

    $object | Add-Member -NotePropertyName 'Timestamp' -NotePropertyValue $Data.Timestamp
    foreach ($row in $TransformedData)
    {
        $path = $row.Path
        $name = $path.Substring($path.LastIndexOf("\") + 1)
        $object | Add-Member -NotePropertyName $name -NotePropertyValue $row.CookedValue
    }
    $object
}


#Parameters
$SqlServerCounterPrefix = '\MSSQL$MSSQL_2008'
$Type = "SQLServerStatistics"
$Data = Get-SqlServerData
$Path = "Z:\Temp\PowershellTest\"
$AddTimeStamp = $false
$NumberOfDaysToKeepFiles = 7
$FileExtension = "csv"

#Variables (do not change)
$Date = Get-Date -format yyyy-MM-dd
$Timestamp = Get-Date
$FilenameRegex = "^" + $Type + "_(?<Date>\d{4}-\d{2}-\d{2})(?:\(\d\))?\." + $FileExtension + "$"
$Suffix = ''
$Counter = 0
$Done = $false

if ($AddTimeStamp -eq $true)
{
    $Data | ForEach-Object {
        $_ | Add-Member -NotePropertyName 'Timestamp' -NotePropertyValue $Timestamp
    }
}

#Try to export file if it fails (the headers have changed) add a (number)
while($Done -eq $false -and $Counter -le 9)
{
    Try
    {
        $Filename = $Type + "_" + $Date + $Suffix + "." + $FileExtension
        Write-Host "Trying to write $Filename"
        $FilePath = $Path + $Filename
        $Data | Export-Csv -Path $FilePath -Delimiter ";" -NoTypeInformation -Append
        $Done = $true
    }
    Catch [Exception]
    {
        Write-Host "Failed to create file " + $_.Exception.GetType().FullName, $_.Exception.Message
        $Counter++
        $Suffix = '(' + $Counter + ')'
    }
}

#Notify if we failed
if ($Done -eq $false)
{
    #Todo: Notify that we failed
}

#Cleanup
$Files = Get-ChildItem $Path -Filter ("*." + $FileExtension)
$Files | Foreach-Object {
    $FilePath = $_.FullName
    $Filename = [System.IO.Path]::GetFileName($FilePath)

    $Match = [regex]::Match($Filename, $FilenameRegex)

    Write-Host $FilePath
    Write-Host $Filename

    if ($Match.Success -eq $true)
    {
        Write-Host $Match.Groups["Date"].Value
        $FileDate = [datetime]::ParseExact($Match.Groups["Date"].Value, "yyyy-MM-dd", $null)
        $TimeSince = New-TimeSpan -Start $FileDate -End (Get-Date).Date
        if ($TimeSince.TotalDays -ge $NumberOfDaysToKeepFiles)
        {
            Remove-Item $FilePath
        }
    }
}

[ruflin]

@AnderssonPeter Thanks for sharing. Any chance you could also share one of the events that is created by the script above to see the fields inside?

[AnderssonPeter]

@ruflin im new to elastic search so i had no idea what document format would be best, i tried to decide between:

"Timestamp";"buffer cache hit ratio";"page life expectancy";"page splits/sec"
"2016-11-08 10:11:19";"100";"683";"0"

and

"Timestamp";"Type";"Value"
"2016-11-08 10:11:19";"buffer cache hit ratio";"100"
"2016-11-08 10:11:19";"page life expectancy";"683"
"2016-11-08 10:11:19";"page splits/sec";"0"

I picked the first one as it generated less overhead (so the powershell script generates the first one)
The first one has one drawback if you add a new field to the powershell script then we have to generate a new file as csv files only have one header line, the second file format would not have that problem.

[ruflin]

No worries about the structure at the moment. This is more for me to understand the data we are talking about. ES will be JSON, but that is something that would be done on the beats side.

Is this just an excerpt of the existing fields? How many different fields are there?

[AnderssonPeter]

Well that's a difficult question to answer.
Every server/pc has a list of counters you can query and every counter can contain multiple values.
Example:
If we query Processor time it contains one value for total and then one for each cpu core/hardware thread.

In my powershell script i only queried one value per counter as i had no need to get multiple, but when querying you can get multiple values as shown above.

[fenneh]

+1 for this feature.

The only other place I've seen this in the beat world is - https://github.com/eskibars/wmibeat

[AnderssonPeter]

It seems my approach was somewhat flawed if a value spiked between my scheduling's i missed it.
I fixed that using https://gist.github.com/AnderssonPeter/8261e255630d6f672ba5bc80f51b017f
Change -CollectionTime to the interval you are logging at (in seconds) so if you set it to 60 then run the powershell file every 60 seconds, and you wont miss any data.

[andrewkroh]

There is a windows perfmon module in Metricbeat now. It's being tracked in #3828. This will be released with 6.0. Please give it a try and leave any feedback (good or bad) in #3828. The nightly build is here.