Alerting with email as a destination not working in version 13.1.0.1.

[adityaj1107]

Issue by Kanmanir
Wednesday May 26, 2021 at 09:44 GMT
Originally opened as opendistro-for-elasticsearch/alerting#382

---

Alerting with email as a destination not working in version 13.1.0.1.

I have added email as a destination , I followed procedure given on official page

1. Created a sender with smtp as outlook 365 with TLS port 587 and recipient email group for the destination.
2. Tried to add the credentials using the below command,
   ./bin/elasticsearch-keystore add opendistro.alerting.destination.email.xxxx.username
   ./bin/elasticsearch-keystore add opendistro.alerting.destination.email.xxxx.password

when tried to see what is the code used in elasticsearch-keystore, I found out it uses a class KeyStoreCli.

But KeyStoreCli class "org.elasticsearch.common.settings.KeyStoreCli" is not available in 7.10.2 in javadocs instead it has class BaseKeyStoreCommand

https://www.javadoc.io/doc/org.elasticsearch/elasticsearch/7.10.2/org/elasticsearch/common/settings/BaseKeyStoreCommand.html

keyStoreCli class is available upto 7.7.1 in javadocs
https://www.javadoc.io/doc/org.elasticsearch/elasticsearch/7.7.1/org/elasticsearch/common/settings/KeyStoreCli.html

But in opendistro version 1.13.0.1 has jar available for KeyStoreCli
/usr/share/elasticsearch/lib/tools/keystore-cli and this jar containes the KeyStoreCli.class

[root@3995dd7839e7 keystore-cli]# jar xvf keystore-cli-7.10.2.jar
created: META-INF/
inflated: META-INF/MANIFEST.MF
inflated: META-INF/LICENSE.txt
inflated: META-INF/NOTICE.txt
created: org/
created: org/elasticsearch/
created: org/elasticsearch/common/
created: org/elasticsearch/common/settings/
inflated: org/elasticsearch/common/settings/ AddFileKeyStoreCommand.class
inflated: org/elasticsearch/common/settings/KeyStoreCli.class
inflated: org/elasticsearch/common/settings/CreateKeyStoreCommand.class
inflated: org/elasticsearch/common/settings/ListKeyStoreCommand.class
inflated: org/elasticsearch/common/settings/RemoveSettingKeyStoreCommand.class
inflated: org/elasticsearch/common/settings/AddStringKeyStoreCommand.class

it is using this lib and class why email destination not working, and I am not able to get where it is storing credentials of sender configured from destination.

Can anybody help with configuring email destination and make it work for alerting.

[dishabodade]

Hi,
Any update on the above bug or any workaround to make it work?

[Kanmanir]

Hai,
Did you find any workaround to make it work? or any schedule when this bug will be fixed?

[skkosuri-amzn]

@qreshi Could you pls triage this.

[qreshi]

Hey @Kanmanir,

I can take a closer look at this. Would you able to answer a couple questions for me to get some more details on the issue?

1. Did you try on an older version of ODFE (I believe email was introduced in Alerting 1.10.1 so any version between that and this current one), are you getting the same problem there?
2. Is the problem with adding settings into the keystore itself, do you get any error when doing so? After adding the settings what do you see when you call ./bin/elasticsearch-keystore list?

[Kanmanir]

Hi Qreshi,

I didn't try the older version of ODFE. The docker image tag for both we are using is 1.13.2 and 1.13.0.0 is the alerting plugin version.

sh-4.2$ ./bin/elasticsearch-plugin list
opendistro-alerting
opendistro-anomaly-detection
opendistro-asynchronous-search
opendistro-index-management
opendistro-job-scheduler
opendistro-knn
opendistro-performance-analyzer
opendistro-reports-scheduler
opendistro-sql
opendistro_security

sh-4.2$ ./bin/kibana-plugin list
configurationAgent@1.0.0
opendistroAlertingKibana@1.13.0.0
opendistroAnomalyDetectionKibana@1.13.0.0
opendistroGanttChartKibana@1.13.0.0
opendistroIndexManagementKibana@1.13.0.1
opendistroNotebooksKibana@1.13.2.0
opendistroQueryWorkbenchKibana@1.13.0.0
opendistroReportsKibana@1.13.2.0
opendistroSecurityKibana@1.13.0.1
opendistroTraceAnalyticsKibana@1.13.2.0

Which version of alerting plugin you want me to try? I will install it and update you.

Ans 2 : We have elasticsearch cluster setup with 4 nodes. I executed add command in all the node as per opendistro documentation. I configured outlook365 as SMTP host and tried to send test mail. I am not getting any error in logs in kibana as well as in elasticsearch.

sh-4.2$ ./bin/elasticsearch-keystore list
keystore.seed
opendistro.alerting.destination.email.kanmani.password
opendistro.alerting.destination.email.kanmani.username

The keystore list command is listing the output as above. so, I think credentials are getting added properly.

[qreshi]

Hmm, assuming your trigger conditions are being met and the Action is being executed, are you seeing the following log messages in your elasticsearch.log:

logger.info("Message published for action name: $name, messageid: $responseContent, statuscode: $responseStatusCode")

It should give us some insight into whether or not the email action is failing.

Also, did you add the credentials to the keystore while ES was already running? If so, you might want to call the _reload API, if you started up ES afterwards or restarted it, you can disregard that.

[Kanmanir]

I have tried to configured the email alerting with latest version (1.13.1.0) alerting plugin. Now its working fine.
I am able to get the email as an alert.

Thanks for helping me.

[qreshi]

Hey @Kanmanir,

Before closing this out, to understand better, did you see any errors in the ES logs for the previous version you used when sending the message and/or did you add your keystore settings while ES was running and not call the _reload API?

I ask since I don't believe Alerting 1.13.1.0 made any changes that were related to the Email Destination so if it works there, it should have worked in 1.13.0.1.

[AndreyByhalenko]

Hi,

I think i facing the same issue on opensearch 1.2.1, the latest version.
I'm using email address(not gmail) with port 25 without encryption.

javax.mail.MessagingException: Invalid Addresses at org.opensearch.alerting.destination.client.DestinationEmailClient.execute(DestinationEmailClient.java:100) ~[alerting-notification-1.2.1.0.jar:?] at org.opensearch.alerting.destination.factory.EmailDestinationFactory.publish(EmailDestinationFactory.java:52) [alerting-notification-1.2.1.0.jar:?] at org.opensearch.alerting.destination.factory.EmailDestinationFactory.publish(EmailDestinationFactory.java:39) [alerting-notification-1.2.1.0.jar:?] at org.opensearch.alerting.destination.Notification.lambda$publish$0(Notification.java:56) [alerting-notification-1.2.1.0.jar:?] at java.security.AccessController.doPrivileged(AccessController.java:312) [?:?] at org.opensearch.alerting.destination.Notification.publish(Notification.java:54) [alerting-notification-1.2.1.0.jar:?] at org.opensearch.alerting.model.destination.Destination.publish(Destination.kt:319) [opensearch-alerting-1.2.1.0.jar:1.2.1.0] at org.opensearch.alerting.MonitorRunner$runAction$2.invokeSuspend(MonitorRunner.kt:680) [opensearch-alerting-1.2.1.0.jar:1.2.1.0] at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-1.3.72.jar:1.3.72-release-468 (1.3.72)] at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:233) [kotlinx-coroutines-core-1.1.1.jar:?] at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:594) [kotlinx-coroutines-core-1.1.1.jar:?] at kotlinx.coroutines.scheduling.CoroutineScheduler.access$runSafely(CoroutineScheduler.kt:60) [kotlinx-coroutines-core-1.1.1.jar:?] at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:742) [kotlinx-coroutines-core-1.1.1.jar:?] [2021-12-19T08:10:42,993][INFO ][o.o.a.m.MonitorRunResult ] [opensearch-node1] Internal error: javax.mail.MessagingException: Invalid Addresses. See the opensearch.log for details java.lang.IllegalStateException: javax.mail.MessagingException: Invalid Addresses

[lezzago]

@AndreyByhalenko Have you double checked the email address that is being used? Based on this error, the email address is not valid.

[AndreyByhalenko]

@lezzago hi, the issue was solved, thank you.

[brijos]

Closing

[Sandy381]

I'm using OSD 1.2.0
Opensearch Reporting scheduler not working with E-mails.
Don't want to add emails to my settings everytime a report has to be sent over a new email address.
Anyworkaround?

[qreshi]

@Sandy381 Looks like you've already found your way to opensearch-project/reporting#72, that would be the best place to continue discussions as it pertains to Reporting integrating with the Notifications plugin.