Bug: Service messages being sent with wrong phone numbers

[em-herrick]

Who discovered this?

Em + WA staff

What happened?

A bug happened! Messages being sent by WA's service are sometimes being delivered by the wrong sending number.

WA's assigned number: 1-855-531-7292

Two wrong numbers are displaying for messages sent with WAs service:

At least one recipient received a WA message from a phone number assigned to another service (State Dept): 1-888-904-6435. Agency staff found out via monitoring a Facebook Group where a recipient posted a screen shot.

To validate this: Em sent a one-off message to her phone using WA's service and received the message via a different sending number—the Notify.gov number: 1-844-795-2263

Environment

production

What pages did this appear on?

This is a database/AWS/ phone number issue.

In the UI, the phone number is displayed correctly when it appears within the send flow.

In the Settings Menu, the service has two phone numbers associated with it. WA's assigned number and The Notify.gov number (1-844-795-2263).

User Acceptance Criteria:

• A service sends messages only from it's assigned number
• Team members with Notify.gov accounts still receive messages from related to 2FA from Notify.gov number (1-844-795-2263)

What browsers are you seeing the problem on?

Microsoft Edge

Relevant log output

No response

Additional information

This is pulled from #2051

We have had a break in the cold case of messages coming from incorrect and more than one service numbers. A GSA team started using Notify to text staff and learned after sending out their first batch that the texts are coming from multiple numbers.

Here are some of the different numbers that people were getting the texts from:
844-734-2791
888-904-6435
888-840-2596
844-734-2791
844-795-2263
(833) 595-1552

Cliff traced through the code and added a bunch of debug, but was never able to repro and we stopped hearing about it from users. I added the experimental validation that should have blocked messages with the wrong sender number but apparently is not doing so. The issue is that this was happening a few months ago, but when we went to investigate, it stopped happening. And now it's started up again.

The check was put in based on thought experiment since we can't seem to reproduce ourselves.

[ccostino]

Thanks, @em-herrick! Adding just a bit more context/details:

• Reminder that phone numbers are all managed in AWS Pinpoint SMS
• Our current list of who is assigned to what is found in our documentation (this also looks like it needs updating!)
• All of the phone numbers in question are managed by us; we've verified in AWS, our documentation, and in the DB

Additionally, given that this is WA State who is seeing this issue, it's possible (though maybe just coincidence) that this is related to this API issue: GSA/notifications-api#1169 - it's worth taking a look just to see and verify that they're truly unrelated, or if there's any possible connection.

[ccostino]

@stvnrlly it'd be good to get your take on here, in case there's something I'm forgetting about or unaware of with the phone number set up for a partner and any system configuration.

[ccostino]

One other thought about this! Given that there seems to be some wires crossed with numbers here, could this also be affecting the report generation for the State Department?

[ccostino]

@xlorepdarkhelm and I noticed another interesting thing in production with the service settings for both Washington State and the State Department:

• As shown above in the screenshot, WA State has Notify.gov as the other number, which we expect and had been updated a while back to show as Notify.gov, not US Notify
• The State Department, however, shows US Notify as the other number, and there's a different GUID with it as well.

So the question here is, did something else not get fully updated/changed over when we updated the underlying Notify.gov number that we use internally in the app (and was used for the older MFA code generation)?

[xlorepdarkhelm]

@ccostino This is something which can be more easily looked at inside the prod database itself. In theory, my logging should provide which outgoing SMS numbers are available to be used whenever an outgoing message is sent.

[ccostino]

@xlorepdarkhelm with the logging now deployed in production, hopefully we can see or at least catch this happening again now. If we absolutely need to, we can peak in the prod database to see what's in there. According to the settings of each service though, they each only have two numbers assigned: their correct number, and the default Notify.gov number.

[ecayer]

Experimental fix + logging. But worth checking those logs!

[ecayer]

@em-herrick Just to check, why was this moved to "done"? Is it superseded by another issue? Thanks!

[em-herrick]

yes, when we heard about the more recent Federal Plaza wrong numbers, I actually moved this one out of 'done' and then back into 'done' and ended up creating #2051

We didn't solve this one, but did put some logging in. So I am not sure what exactly what to do with it...

[ccostino]

I've updated the description above with the info from #2051, which we'll close that issue as a duplicate.

Additional, @terrazoon added some additional debugging around this that just went out the door with the last production deploy yesterday, so 🤞 we start to see/collect some useful data about what is actually happening with this!