feat: advanced logging #1539

jeromevdl · 2023-12-15T22:44:48Z

Issue #, if available: #965 #1508

Description of changes:

Removed the message logging in JSON, message will remain a simple String
If people want to add custom fields, they can use either
- MDC: I removed the LoggingUtils.appendKey to use the standard MDC
- StructuredArguments: I've introduced this as suggested in RFC: Switch to logback, SLF4J and logstash as logging framework #965 by @roamingthings. But I did not introduce the library. I've created something similar but lighter in Powertools. Structured arguments can be any kind of object, JSON, ... and will be logged as such.
Events and responses are now logged using StructuredArguments and not in the message anymore
Simplified the Json serialization in logback module
Breaking Change: updated the JsonConfig ObjectMapper to enable/disable some features:
- do not fail on unknown properties
- ignore null fields ==> this breaks idempotency compatibility (hash is different now)
- order fields alphabetically

Checklist

Meet tenets criteria
Update tests
Update docs ==> need to do that if we agree on the code
PR title follows conventional commit semantics

Breaking change checklist

Breaking change on the Jackson ObjectMapper that now comes with default config (ignore null fields)

RFC issue #:

Migration process documented
Implement warnings (if it can live side by side)

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

github-actions · 2023-12-15T22:45:44Z

💾 Artifacts Size Report

Module	Version	Size (KB)
powertools-common	2.0.0-SNAPSHOT	9.59
powertools-serialization	2.0.0-SNAPSHOT	18.22
powertools-logging	2.0.0-SNAPSHOT	33.07
powertools-logging-log4j	2.0.0-SNAPSHOT	20.60
powertools-logging-logback	2.0.0-SNAPSHOT	17.02
powertools-tracing	2.0.0-SNAPSHOT	14.00
powertools-metrics	2.0.0-SNAPSHOT	14.05
powertools-parameters	2.0.0-SNAPSHOT	17.46
powertools-validation	2.0.0-SNAPSHOT	20.80
powertools-cloudformation	2.0.0-SNAPSHOT	16.99
powertools-idempotency-core	2.0.0-SNAPSHOT	35.55
powertools-idempotency-dynamodb	2.0.0-SNAPSHOT	12.90
powertools-large-messages	2.0.0-SNAPSHOT	17.45
powertools-batch	2.0.0-SNAPSHOT	16.89
powertools-parameters-ssm	2.0.0-SNAPSHOT	10.69
powertools-parameters-secrets	2.0.0-SNAPSHOT	9.90
powertools-parameters-dynamodb	2.0.0-SNAPSHOT	11.94
powertools-parameters-appconfig	2.0.0-SNAPSHOT	11.98

codecov-commenter · 2023-12-15T22:51:00Z

Codecov Report

All modified and coverable lines are covered by tests ✅

❗ No coverage uploaded for pull request base (v2@f625b33). Click here to learn what that means.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@          Coverage Diff          @@
##             v2    #1539   +/-   ##
=====================================
  Coverage      ?   78.32%           
  Complexity    ?      593           
=====================================
  Files         ?       66           
  Lines         ?     2265           
  Branches      ?      319           
=====================================
  Hits          ?     1774           
  Misses        ?      373           
  Partials      ?      118

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

jeromevdl · 2023-12-18T07:27:23Z

...src/main/java/org/apache/logging/log4j/layout/template/json/resolver/PowertoolsResolver.java

            };

    private final EventResolver internalResolver;

-    private static final Map<String, EventResolver> eventResolverMap = Stream.of(new Object[][] {
+    private static final Map<String, EventResolver> eventResolverMap = Collections.unmodifiableMap(Stream.of(new Object[][] {


Can we move to Java 11 to declare this more easily... ? :(

powertools-logging/src/main/java/software/amazon/lambda/powertools/logging/LoggingUtils.java

...ng/src/main/java/software/amazon/lambda/powertools/logging/internal/LambdaLoggingAspect.java

scottgerring

Some initial feedback - I believe the approach makes sense and represents an improvement to the UX. I would love to review the whole thing once you've done the docs too.

powertools-logging/src/main/java/software/amazon/lambda/powertools/logging/LoggingUtils.java

...-logging/src/main/java/software/amazon/lambda/powertools/logging/argument/ArrayArgument.java

...s-logging/src/main/java/software/amazon/lambda/powertools/logging/argument/JsonArgument.java

...logging/src/main/java/software/amazon/lambda/powertools/logging/internal/JsonSerializer.java

...ng/src/main/java/software/amazon/lambda/powertools/logging/argument/StructuredArguments.java

...est/java/software/amazon/lambda/powertools/logging/internal/handler/PowertoolsArguments.java

sonarcloud · 2024-01-08T08:09:23Z

Quality Gate passed

The SonarCloud Quality Gate passed, but some issues were introduced.

5 New issues
0 Security Hotspots
No data about Coverage
0.6% Duplication on New Code

See analysis details on SonarCloud

scottgerring

Some doc feedback

scottgerring · 2024-01-24T08:30:05Z

docs/core/logging.md

@@ -563,35 +444,35 @@ we provide [built-in JMESPath expressions](#built-in-correlation-id-expressions)

 #### Custom keys

-???+ warning "Custom keys are persisted across warm invocations"
-    Always set additional keys as part of your handler method to ensure they have the latest value, or explicitly clear them with [`clearState=true`](#clearing-state).
+** Using StructuredArguments **


"Appending Additional Keys" ? So that the heading reflects when you can learn to do, not a detail of how to do it

scottgerring · 2024-01-24T08:32:02Z

docs/core/logging.md

+
+** Using MDC **
+
+Mapped Diagnostic Context (MDC) is essentially a Key-Value store. It is supported by the [SLF4J API](https://www.slf4j.org/manual.html#mdc){target="_blank"},


key-value, no need to capitalise

scottgerring · 2024-01-24T08:32:51Z

docs/core/logging.md

+
+### Removing additional keys
+
+You can remove additional keys added with the MDC using `MDC.remove("key")`. 



"Typically, you'd remove additional keys at the end of your request handler, so they do not apply to subsequent requests"

scottgerring · 2024-01-24T08:33:05Z

docs/core/logging.md

+
+### Removing additional keys
+
+You can remove additional keys added with the MDC using `MDC.remove("key")`. 

 #### Clearing state

 Logger is commonly initialized in the global scope. Due to [Lambda Execution Context reuse](https://docs.aws.amazon.com/lambda/latest/dg/runtimes-context.html){target="_blank"},


The logger is

scottgerring · 2024-01-24T08:33:22Z

docs/core/logging.md


 #### Clearing state

 Logger is commonly initialized in the global scope. Due to [Lambda Execution Context reuse](https://docs.aws.amazon.com/lambda/latest/dg/runtimes-context.html){target="_blank"},
-this means that custom keys can be persisted across invocations. If you want all custom keys to be deleted, you can use
+this means that custom keys, added with the MDC can be persisted across invocations. If you want all custom keys to be deleted, you can use


Capitalisation "This"

scottgerring · 2024-01-24T08:34:09Z

docs/core/logging.md

@@ -1071,33 +1005,6 @@ If you need to customize format and timezone, you can change use the following:
    </encoder>
 ```



No more custom object mapper?

sonarcloud · 2024-02-28T12:00:49Z

Quality Gate passed

Issues
5 New issues

Measures
0 Security Hotspots
No data about Coverage
0.6% Duplication on New Code

See analysis details on SonarCloud

jeromevdl · 2024-02-28T12:43:38Z

🎉

jeromevdl added 5 commits December 12, 2023 16:56

improve logging

42f7d94

adding arguments to log events

e7bd14a

!update ObjectMapper configuration

6cb5a11

remove LoggingUtils appendKey, use standard MDC

403f3e7

examples in the end

8ef1027

jeromevdl marked this pull request as draft December 15, 2023 22:44

pull-request-size bot added the size/XXL label Dec 15, 2023

Merge branch 'v2' into feat/advanced-logging

0a86dc8

clean sonar warnings

8d14422

clean spotbugs issues

94795af

jeromevdl commented Dec 18, 2023

View reviewed changes

powertools-logging/src/main/java/software/amazon/lambda/powertools/logging/LoggingUtils.java Outdated Show resolved Hide resolved

jeromevdl commented Dec 18, 2023

View reviewed changes

...ng/src/main/java/software/amazon/lambda/powertools/logging/internal/LambdaLoggingAspect.java Show resolved Hide resolved

jeromevdl commented Dec 18, 2023

View reviewed changes

...ng/src/main/java/software/amazon/lambda/powertools/logging/internal/LambdaLoggingAspect.java Show resolved Hide resolved

scottgerring self-requested a review December 18, 2023 09:08

scottgerring reviewed Dec 21, 2023

View reviewed changes

jeromevdl added 4 commits January 5, 2024 15:29

Merge branch 'v2' into feat/advanced-logging

56d3632

code review comments

fa948b0

update documentation

80a5d68

remove LoggingUtils

4e4cdfc

jeromevdl self-assigned this Jan 8, 2024

scottgerring marked this pull request as ready for review January 24, 2024 08:34

scottgerring suggested changes Jan 24, 2024

View reviewed changes

scottgerring added 2 commits February 28, 2024 11:01

Merge v2 down

37d9c3c

Merge remote-tracking branch 'origin/v2' into feat/advanced-logging

bba3fae

scottgerring approved these changes Feb 28, 2024

View reviewed changes

scottgerring merged commit 7dff30c into v2 Feb 28, 2024
18 checks passed

scottgerring deleted the feat/advanced-logging branch February 28, 2024 12:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: advanced logging #1539

feat: advanced logging #1539

jeromevdl commented Dec 15, 2023

github-actions bot commented Dec 15, 2023 •

edited

Loading

codecov-commenter commented Dec 15, 2023 •

edited

Loading

jeromevdl Dec 18, 2023

scottgerring left a comment

sonarcloud bot commented Jan 8, 2024

scottgerring left a comment

scottgerring Jan 24, 2024

scottgerring Jan 24, 2024

scottgerring Jan 24, 2024

scottgerring Jan 24, 2024

scottgerring Jan 24, 2024

scottgerring Jan 24, 2024

sonarcloud bot commented Feb 28, 2024

jeromevdl commented Feb 28, 2024


		Using MDC

		Mapped Diagnostic Context (MDC) is essentially a Key-Value store. It is supported by the [SLF4J API](https://www.slf4j.org/manual.html#mdc){target="_blank"},


		### Removing additional keys

		You can remove additional keys added with the MDC using `MDC.remove("key")`.

		@@ -1071,33 +1005,6 @@ If you need to customize format and timezone, you can change use the following:
		</encoder>
		```

feat: advanced logging #1539

feat: advanced logging #1539

Conversation

jeromevdl commented Dec 15, 2023

Description of changes:

Breaking change checklist

github-actions bot commented Dec 15, 2023 • edited Loading

💾 Artifacts Size Report

codecov-commenter commented Dec 15, 2023 • edited Loading

Codecov Report

jeromevdl Dec 18, 2023

Choose a reason for hiding this comment

scottgerring left a comment

Choose a reason for hiding this comment

sonarcloud bot commented Jan 8, 2024

Quality Gate passed

scottgerring left a comment

Choose a reason for hiding this comment

scottgerring Jan 24, 2024

Choose a reason for hiding this comment

scottgerring Jan 24, 2024

Choose a reason for hiding this comment

scottgerring Jan 24, 2024

Choose a reason for hiding this comment

scottgerring Jan 24, 2024

Choose a reason for hiding this comment

scottgerring Jan 24, 2024

Choose a reason for hiding this comment

scottgerring Jan 24, 2024

Choose a reason for hiding this comment

sonarcloud bot commented Feb 28, 2024

Quality Gate passed

jeromevdl commented Feb 28, 2024

github-actions bot commented Dec 15, 2023 •

edited

Loading

codecov-commenter commented Dec 15, 2023 •

edited

Loading