Veeam Backup Failing (‘VSS_WS_FAILED_AT_PREPARE_SNAPSHOT’)
I had a Veeam backup job that was failing with: Retrying snapshot creation attempt (Writer ‘Microsoft Hyper-V VSS Writer’ is failed at ‘VSS_WS_FAILED_AT_PREPARE_SNAPSHOT’. The writer experienced a non-transient error. If the backup process is retried, the error is likely to reoccur. –tr:Failed to verify writers state. –tr:Failed to perform pre-backup tasks.)
Researching this error online was telling me the issue was on the host, but I wasn’t believing that as all of my other vm’s were backing up without issue daily.
To play it safe I checked the host by running: vssadmin list writers
I received the following error on the host:
microsoft hyper-v vss writer non-retryable error
Looking further on the host’s event logs for the error I saw this:
At this point I was still convinced the host wasn’t at fault due to the fact all other vm’s still backed up fine, so I logged onto the vm in question and ran: vssadmin list writers
I received the following on the vm:
sqlserverwriter non-retryable error
Looking into the event viewer I saw:
Researching these errors online I found several solution saying to delete the old backup software. This server used to use another backup solution prior to Veeam called Altaro, which I was pretty sure I had removed a long time ago. I checked add/remove programs and verified Altaro wasn’t listed. I even checked the vss writers for any other backup software listed and found nothing. Running out of ideas, I checked Windows backup to make sure it wasn’t running and no backup jobs were listed. I then looked into Task Scheduler and found a few manual backup jobs listed. I disabled and deleted these jobs. I then restarted the SQL VSS writer service, restarted SQL VSS service, verified it showed no errors after re-running vssasdmin list writers. I then retried the Veeam backup again and it failed out once again.
Re-running vss list writer I received the same error. I was now convinced this was tied to the old task scheduled backups I had removed.
Next, I tried: vssadmin delete shadows /all
After running that command, I received:
Error: Snapshots were found, but they were outside of your allowed context. Try
removing them with the backup application which created them.
After much more research, I found an outside the box way of deleting the snapshots from another site.
How to Fix “outside of your allowed context” Errors
In order to get rid of these kinds of shadows we need to apply a “trick”. Basically the VSS diff area storage is where VSS keeps these shadows “alive”.
By seriously cutting this limit to the bare minimum we invoke a mechanism in VSS itself that causes it to dump all shadows.
So we proceed by telling VSS to cut the limit down to 401 MB. For some reason the user interface will claim the bottom is 300MB but on several versions of Windows it refuses and reports:
Error: Specified number is invalid
The command that works uses 401MB and is (adapt it to your drive letter as needed):
vssadmin resize shadowstorage /for=D: /on=D: /maxsize=401MB
*****I ran this against the C: and D: drive of my VM*****
Then once you get “success” you can increase the limit once again to the recommended “unbounded” setting, or an actual limit value if you are using shadow copies for other purposes:
vssadmin resize shadowstorage /for=d: /on=D: /maxsize=unbounded
*****I ran this against the C: and D: drive of my VM*****
Then, vssadmin happily reports:
Successfully resized the shadow copy storage association
and a quick check using
vssadmin list shadows
reveals all VSS shadow copies are now gone!
I then re-ran Veeam the Veeam backup job against the VM and it ran successfully!