Successfull backups reporting a failure

[gudakrist]

Hi

We've been using your backup solution for may years and we love it.
We encountered a strange behavior after changing to a new storage solution (Cohesity) for the backups. We are getting "Job failed" on successful jobs at random, and we do suspect that this is due to a slower response from the backup media.

Here is a typical log from one of these:
message
Executed as user: MYDOMAIN\MyServerName_A. ...000) Server: MyServerName [SQLSTATE 01000] (Message 50000) Version: 12.0.6433.1 [SQLSTATE 01000] (Message 50000) Edition: Enterprise Edition (64-bit) [SQLSTATE 01000] (Message 50000) Platform: Windows [SQLSTATE 01000] (Message 50000) Procedure: [DBATools].[dbo].[DatabaseBackup] [SQLSTATE 01000] (Message 50000) Parameters: @databases = 'USER_DATABASES', @Directory = '\cohesitycluster.domain.org\SQLBackup_NAME\PROD', @BackupType = 'DIFF', @verify = 'Y', @Cleanuptime = 2232, @CleanupMode = 'AFTER_BACKUP', @compress = NULL, @copyonly = 'N', @ChangeBackupType = 'N', @BackupSoftware = NULL, @Checksum = 'Y', @Blocksize = NULL, @BufferCount = NULL, @MaxTransferSize = NULL, @NumberOfFiles = NULL, @MinBackupSizeForMultipleFiles = NULL, @MaxFileSize = NULL, @CompressionLevel = NULL, @description = NULL, @threads = NULL, @Throttle = NULL, @Encrypt = 'N', @EncryptionAlgorithm = NULL, @ServerCertificate = NULL, @ServerAsymmetricKey = NULL, @EncryptionKey = NULL, @ReadWriteFileGroups = 'N', @OverrideBackupPreference = 'N', @NoRecovery = 'N', @url = NULL, @credential = NULL, @MirrorDirectory = NULL, @MirrorCleanupTime = NULL, @MirrorCleanupMode = 'AFTER_BACKUP', @MirrorURL = NULL, @AvailabilityGroups = NULL, @Updateability = 'ALL', @AdaptiveCompression = NULL, @ModificationLevel = NULL, @LogSizeSinceLastLogBackup = NULL, @TimeSinceLastLogBackup = NULL, @DataDomainBoostHost = NULL, @DataDomainBoostUser = NULL, @DataDomainBoostDevicePath = NULL, @DataDomainBoostLockboxPath = NULL, @DirectoryStructure = '{ServerName}${InstanceName}{DirectorySeparator}{DatabaseName}{DirectorySeparator}{BackupType}{Partial}{CopyOnly}', @AvailabilityGroupDirectoryStructure = '{ClusterName}${AvailabilityGroupName}{DirectorySeparator}{DatabaseName}{DirectorySeparator}{BackupType}{Partial}{CopyOnly}', @filename = '{ServerName}${InstanceName}{DatabaseName}{BackupType}{Partial}{CopyOnly}{Year}{Month}{Day}{Hour}{Minute}{Second}{FileNumber}.{FileExtension}', @AvailabilityGroupFileName = '{ClusterName}${AvailabilityGroupName}{DatabaseName}{BackupType}{Partial}{CopyOnly}{Year}{Month}{Day}{Hour}{Minute}{Second}{FileNumber}.{FileExtension}', @FileExtensionFull = NULL, @FileExtensionDiff = NULL, @FileExtensionLog = NULL, @Init = 'N', @Format = 'N', @ObjectLevelRecoveryMap = 'N', @ExcludeLogShippedFromLogBackup = 'Y', @StringDelimiter =... Version: 2020-12-31 18:58:56 [SQLSTATE 01000] (Message 50000) Source: https://ola.hallengren.com [SQLSTATE 01000] (Message 50000) [SQLSTATE 01000] (Message 50000) Date and time: 2021-06-11 13:00:01 [SQLSTATE 01000] (Message 50000) Database: [Department] [SQLSTATE 01000] (Message 50000) State: ONLINE [SQLSTATE 01000] (Message 50000) Standby: No [SQLSTATE 01000] (Message 50000) Updateability: READ_WRITE [SQLSTATE 01000] (Message 50000) User access: MULTI_USER [SQLSTATE 01000] (Message 50000) Recovery model: FULL [SQLSTATE 01000] (Message 50000) Encrypted: No [SQLSTATE 01000] (Message 50000) Is accessible: Yes [SQLSTATE 01000] (Message 50000) Differential base LSN: 358000000351800037 [SQLSTATE 01000] (Message 50000) Differential base is snapshot: No [SQLSTATE 01000] (Message 50000) Last log backup LSN: 358000000468000001 [SQLSTATE 01000] (Message 50000) Allocated extent page count: 16296 (127.312500 MB) [SQLSTATE 01000] (Message 50000) Modified extent page count: 128 (1.000000 MB) [SQLSTATE 01000] (Message 50000) [SQLSTATE 01000] (Message 50000) Date and time: 2021-06-11 13:00:01 [SQLSTATE 01000] (Message 50000) Database context: [master] [SQLSTATE 01000] (Message 50000) Command: DECLARE @ReturnCode int EXECUTE @ReturnCode = dbo.xp_create_subdir N'\cohesitycluster.domain.org\SQLBackup_NAME\PROD\MyServerName\Department\DIFF' IF @ReturnCode <> 0 RAISERROR('Error creating directory.', 16, 1) [SQLSTATE 01000] (Message 50000) Outcome: Succeeded [SQLSTATE 01000] (Message 50000) Duration: 00:00:00 [SQLSTATE 0... The step failed.

Any Ideas on how to fix this?

Best regards
Gudmundur

[olahallengren]

Are you still having this issue?

Could you send me a complete output - file, with the issue?
https://ola.hallengren.com/contact.html

[gudakrist]

Hi Ola
Here is the output from DB backup history, I've replaced names for security reasons. Please let my know if you need anything else.
Best regards, Gudmundur

Date 1/5/2022 10:30:00 AM
Log Job History (DatabaseBackup - USER_DATABASES - LOG)

Step ID 1
Server MY_SERVERNAME
Job Name DatabaseBackup - USER_DATABASES - LOG
Step Name DatabaseBackup - USER_DATABASES - LOG
Duration 00:01:00
Sql Severity 16
Sql Message ID 50000
Operator Emailed
Operator Net sent
Operator Paged
Retries Attempted 0

Message
Executed as user: MYDOMAINNAME\MY_BACKUP_USER. ... 50000) Server: MY_SERVERNAME [SQLSTATE 01000] (Message 50000) Version: 13.0.6300.2 [SQLSTATE 01000] (Message 50000) Edition: Enterprise Edition: Core-based Licensing (64-bit) [SQLSTATE 01000] (Message 50000) Platform: Windows [SQLSTATE 01000] (Message 50000) Procedure: [DBATools].[dbo].[DatabaseBackup] [SQLSTATE 01000] (Message 50000) Parameters: @databases = 'USER_DATABASES', @Directory = '\cohesityserver.cohesity.domain.com\SQLBackup_COM\PROD', @BackupType = 'LOG', @verify = 'Y', @Cleanuptime = 2232, @CleanupMode = 'AFTER_BACKUP', @compress = NULL, @copyonly = 'N', @ChangeBackupType = 'N', @BackupSoftware = NULL, @Checksum = 'Y', @Blocksize = NULL, @BufferCount = NULL, @MaxTransferSize = NULL, @NumberOfFiles = NULL, @MinBackupSizeForMultipleFiles = NULL, @MaxFileSize = NULL, @CompressionLevel = NULL, @description = NULL, @threads = NULL, @Throttle = NULL, @Encrypt = 'N', @EncryptionAlgorithm = NULL, @ServerCertificate = NULL, @ServerAsymmetricKey = NULL, @EncryptionKey = NULL, @ReadWriteFileGroups = 'N', @OverrideBackupPreference = 'N', @NoRecovery = 'N', @url = NULL, @credential = NULL, @MirrorDirectory = NULL, @MirrorCleanupTime = NULL, @MirrorCleanupMode = 'AFTER_BACKUP', @MirrorURL = NULL, @AvailabilityGroups = NULL, @Updateability = 'ALL', @AdaptiveCompression = NULL, @ModificationLevel = NULL, @LogSizeSinceLastLogBackup = NULL, @TimeSinceLastLogBackup = NULL, @DataDomainBoostHost = NULL, @DataDomainBoostUser = NULL, @DataDomainBoostDevicePath = NULL, @DataDomainBoostLockboxPath = NULL, @DirectoryStructure = '{ServerName}${InstanceName}{DirectorySeparator}{DatabaseName}{DirectorySeparator}{BackupType}{Partial}{CopyOnly}', @AvailabilityGroupDirectoryStructure = '{ClusterName}${AvailabilityGroupName}{DirectorySeparator}{DatabaseName}{DirectorySeparator}{BackupType}{Partial}{CopyOnly}', @filename = '{ServerName}${InstanceName}{DatabaseName}{BackupType}{Partial}{CopyOnly}{Year}{Month}{Day}{Hour}{Minute}{Second}{FileNumber}.{FileExtension}', @AvailabilityGroupFileName = '{ClusterName}${AvailabilityGroupName}{DatabaseName}{BackupType}{Partial}{CopyOnly}{Year}{Month}{Day}{Hour}{Minute}{Second}{FileNumber}.{FileExtension}', @FileExtensionFull = NULL, @FileExtensionDiff = NULL, @FileExtensionLog = NULL, @Init = 'N', @Format = 'N', @ObjectLevelRecoveryMap = 'N', @ExcludeLogShippedFromLogBackup = 'Y', @StringDelimiter = ... Version: 2020-12-31 18:58:56 [SQLSTATE 01000] (Message 50000) Source: https://ola.hallengren.com [SQLSTATE 01000] (Message 50000) [SQLSTATE 01000] (Message 50000) Date and time: 2022-01-05 10:30:00 [SQLSTATE 01000] (Message 50000) Database: [APICommon] [SQLSTATE 01000] (Message 50000) State: ONLINE [SQLSTATE 01000] (Message 50000) Standby: No [SQLSTATE 01000] (Message 50000) Updateability: READ_WRITE [SQLSTATE 01000] (Message 50000) User access: MULTI_USER [SQLSTATE 01000] (Message 50000) Recovery model: FULL [SQLSTATE 01000] (Message 50000) Encrypted: No [SQLSTATE 01000] (Message 50000) Is accessible: Yes [SQLSTATE 01000] (Message 50000) Database mirroring role: PRINCIPAL [SQLSTATE 01000] (Message 50000) Differential base LSN: 163000001279600039 [SQLSTATE 01000] (Message 50000) Last log backup LSN: 163000001442300001 [SQLSTATE 01000] (Message 50000) Last log backup: 2022-01-05 10:15:01 [SQLSTATE 01000] (Message 50000) Log size since last log backup (MB): 0.060546 [SQLSTATE 01000] (Message 50000) [SQLSTATE 01000] (Message 50000) Date and time: 2022-01-05 10:30:01 [SQLSTATE 01000] (Message 50000) Database context: [master] [SQLSTATE 01000] (Message 50000) Command: DECLARE @ReturnCode int EXECUTE @ReturnCode = dbo.xp_create_subdir N'\cohesityserver.cohesity.domain.com\SQLBackup_COM\PROD\MY_SERVERNAME\APICommon\LOG' IF @ReturnCode <> 0 RAISERROR('Error creating directory.', 16, 1) [SQLSTATE 01000] (Message 50000) Outcome: Succeeded [SQLSTATE 01000] (Message 50000) Duration: 00:00:00 [SQLSTATE... The step failed.

[jacobgexigo]

We see this sometimes too for our NAS. It is when we have the @verify parameter set to 'Y'. For whatever reason the backup runs fast, but then the verify seems to stale out reading from the location.

[web1701]

We also have a similar issue as our backups are on a network share. When there is a "failure", it is during the verification phase.

[Firmbyte]

Was there a solution for this issue? I'm also encountering it. The Job reports an error "Error creating directory.', 16, 1) " but in most cases the directory already exists.

[reginaldosantana1]

a alguma solução ou apenas ignorar e da o backup feito com sucesso