[Elastic Agent] Send Agent logs to elasticsearch #19811
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
botelastic
bot
added
the
needs_team
botelastic
bot
removed
the
needs_team
|
Pinging @elastic/ingest-management (Team:Ingest Management) |
blakerouse
force-pushed
the
agent-logs-to-fleet
branch
from
d53e7d2 to
ae55a79
Compare
❕ Build Aborted
Expand to view the summary
Build stats
Test stats 🧪
Steps errorsExpand to view the steps failures
Log outputExpand to view the last 100 lines of log output
|
michalpristas
approved these changes
Jul 13, 2020
There was a problem hiding this comment.
LGTM maybe another look for logger part from @urso would be handy
urso
reviewed
Jul 13, 2020
| } | ||
|
|
||
| encoder := zapcore.NewJSONEncoder(ecszap.ECSCompatibleEncoderConfig(logp.JSONEncoderConfig())) | ||
| return ecszap.WrapCore(zapcore.NewCore(encoder, rotator, logp.DebugLevel.ZapLevel())), nil |
There was a problem hiding this comment.
I wonder if we could have reduced some of the plumbing by exporting logp.createLogOutput only and split configure into configure log output and configure logging with output. The zapcore.Core is basically the output here. The core that is produced here coulbe be build via CreateLogOutput(agentConfig).
Allowing us to pass a separate output (zapcore.Core) to logp.Configure might allow us in the future to change how logging will be configured in other products, but still work with existing code in Beats that requires logp to function as is (e.g. for agent or collector).
Splitting can also be interesting for testing, as we could allow tests to configure a Core that logs to the active *testing.T. This might be helpful when investigating test failures.
|
@urso I am trying to make as little change as possible, don't want to cause a big refactor on FF day. I have added a |
urso
reviewed
Jul 14, 2020
|
@urso Added comment. |
urso
reviewed
Jul 14, 2020
urso
approved these changes
Jul 14, 2020
4 tasks
blakerouse
added a commit
to blakerouse/beats
that referenced
this pull request
Jul 14, 2020
* Work on logging twice. * Work on agent logging to fleet. * Commit example index strategy. * More work on logging to ES. * Revert change to release/version.go * Fix indexes for metricbeat sidecars. * Add to changelog. * Fix fmt. * Don't expose zapLevel, add ConfigureWithOutputs. * Update comment. * Update comment. (cherry picked from commit 2297636)
blakerouse
added a commit
that referenced
this pull request
Jul 14, 2020
* Work on logging twice. * Work on agent logging to fleet. * Commit example index strategy. * More work on logging to ES. * Revert change to release/version.go * Fix indexes for metricbeat sidecars. * Add to changelog. * Fix fmt. * Don't expose zapLevel, add ConfigureWithOutputs. * Update comment. * Update comment. (cherry picked from commit 2297636)
melchiormoulin
pushed a commit
to melchiormoulin/beats
that referenced
this pull request
Oct 14, 2020
* Work on logging twice. * Work on agent logging to fleet. * Commit example index strategy. * More work on logging to ES. * Revert change to release/version.go * Fix indexes for metricbeat sidecars. * Add to changelog. * Fix fmt. * Don't expose zapLevel, add ConfigureWithOutputs. * Update comment. * Update comment.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Agent will now always log to
${path.data}/logs/elastic-agent-json.logno matter the settings fromelastic-agent.yml(those still apply, for the user specified choice). With monitoring always on Filebeat will always be started and will send the logs from${path.data}/logs/elastic-agent-json.logto elasticsearch.The spawned filebeat and metricbeat now log to
filebeat-json.logandmetricbeat-json.login JSON format and forward to elasticsearch.Indexes:
logs-elastic.agent-default{logs,metrics}-elastic.agent.filebeat-default{logs,metrics}-elastic.agent.metricbeat-defaultWhy is it important?
So all logs of the Elastic Agents on hosts are in elasticsearch so monitoring/troubleshooting of the Fleet is much easier.
Checklist
[ ] I have made corresponding changes to the documentation[ ] I have made corresponding change to the default configuration filesCHANGELOG.next.asciidocorCHANGELOG-developer.next.asciidoc.Related issues