How to recover deleted backups from an Avamar node that is replicated to another Avamar node

Avamar replication is quite often a life saver! you replicate your data to an offsite location in the event of disaster and restore from there if needed. Normally, this is a one way process as in not needing to replicate back to the source node.

Recently however, I had a situation were a site had a weeks worth of backups deleted for a single server by mistake. We were fortunate enough to have this occur after garbage collection, so we were quickly able to re-backup the server to ensure 99% of the chunks were not deleted however, we were still missing that week worth of backups!!

We had a replicated copy of this data at our replica site and figured it shouldn't be too hard to get that data replicated back into that clients backup set for local restoration purposes.

Challenge accepted! this was our first attempt:

Run a manual replicate command with the DPN user from the replica node:

ssh-agent bash
ssh-add ~dpn/.ssh/dpnid

replicate --dstaddr=destination.mydomain.com --dstid=repluser --dstpassword=******** --hfsaddr=source.mydomain.com --id=repluser --password=******** --dstpath=/ORIGINALDOMAIN/client.mydomain.com --srcpath=/REPLICATE/destination.mydomain.com/ORIGINALDOMAIN/client.mydomain.com

It phrased correctly and immediately produced this lovely error:

ERROR: Source path cannot contain /REPLICATE for replicate!

Damn! back to the drawing board.. after a few hours and some hair pulling we were able to get a working command to run from the replica node and replicate that data back to the original location. The missing flag was "--restore" and below is the command that needs to be run from the replica node as the DPN user:

ssh-agent bash
ssh-add ~dpn/.ssh/dpnid

replicate --restore --dstaddr=destination.mydomain.com --dstid=repluser --dstpassword=******** --hfsaddr=source.mydomain.com --id=repluser --password=******** --dstpath=/ORIGINALDOMAIN/client.mydomain.com --srcpath=/REPLICATE/destination.mydomain.com/ORIGINALDOMAIN/client.mydomain.com

The replication will appear as a normal backup job in the GUI for you to monitor or you can watch the replicate output from the CLI on the replica node.

The data restoration time back to the original node will take minutes or hours depending on how any chunks have been deleted if a GC was run. Hopefully if you are reading this the GC hasn't run or you have a fast pipe between the replica and master!!