How to recover deleted backups from an Avamar node that is replicated to another Avamar node

Avamar replication is quite often a life saver! you replicate your data to an offsite location in the event of disaster and restore from there if needed. Normally, this is a one way process as in not needing to replicate back to the source node.

Recently however, I had a situation were a site had a weeks worth of backups deleted for a single server by mistake. We were fortunate enough to have this occur after garbage collection, so we were quickly able to re-backup the server to ensure 99% of the chunks were not deleted however, we were still missing that week worth of backups!!

We had a replicated copy of this data at our replica site and figured it shouldn't be too hard to get that data replicated back into that clients backup set for local restoration purposes.

Challenge accepted! this was our first attempt:

Run a manual replicate command with the DPN user from the replica node:

ssh-agent bash
ssh-add ~dpn/.ssh/dpnid

replicate --dstaddr=destination.mydomain.com --dstid=repluser --dstpassword=******** --hfsaddr=source.mydomain.com --id=repluser --password=******** --dstpath=/ORIGINALDOMAIN/client.mydomain.com --srcpath=/REPLICATE/destination.mydomain.com/ORIGINALDOMAIN/client.mydomain.com

It phrased correctly and immediately produced this lovely error:

ERROR: Source path cannot contain /REPLICATE for replicate!

Damn! back to the drawing board.. after a few hours and some hair pulling we were able to get a working command to run from the replica node and replicate that data back to the original location. The missing flag was "--restore" and below is the command that needs to be run from the replica node as the DPN user:

ssh-agent bash
ssh-add ~dpn/.ssh/dpnid

replicate --restore --dstaddr=destination.mydomain.com --dstid=repluser --dstpassword=******** --hfsaddr=source.mydomain.com --id=repluser --password=******** --dstpath=/ORIGINALDOMAIN/client.mydomain.com --srcpath=/REPLICATE/destination.mydomain.com/ORIGINALDOMAIN/client.mydomain.com

The replication will appear as a normal backup job in the GUI for you to monitor or you can watch the replicate output from the CLI on the replica node.

The data restoration time back to the original node will take minutes or hours depending on how any chunks have been deleted if a GC was run. Hopefully if you are reading this the GC hasn't run or you have a fast pipe between the replica and master!! 

Creating Avamar backup schedules from the CLI

Creating a schedule for your backup policies is a must.

Here we add a schedule called "DAILYBACKUP" that is assigned to the root / domain in Avamar (available to all policies) and it's set to run Monday thru Sunday from 6pm for a duration of 12hrs using the timezone of SYDNEY. 

Take note of the timezone, if you are setting up an Avamar node in a different timezone use the correct city, using your GUI interface from a different country will create the schedule based on YOUR timezone. 

This is why I use the CLI to create the schedules for nodes that are not located in my local timezone.

mccli schedule add --name=DAILYBACKUP --domain=/ --days=Mu,Tu,W,Th,F,Sa,Su --start=18:00 --duration=12:00 --tz=Asia/Sydney

Creating a default Avamar retention policy

With all backups you want to have a "default" retention period for the backup set. You can create this default retention set with the below mccli command from the CLI. Adjust the "+0D" to match your required retention timing. In my example below we have basic and adv-daily retention set to 60 days. This means any policy running this retention will have backups kept for 60 days before garbage collection will remove these backups from the grid.

mccli retention add --name=RETENTION_NAME --basic=+60D --adv-daily=+60D --adv-weekly=+0D --adv-monthly=+0D --adv-yearly=+0D

Creating and manipulating Avamar datasets

Datasets must be created in the Avamar GUI (or at least I cannot figure out how to create them in the CLI). 

Once created you can add mass exclusions via the CLI with the script below. The script adds exclusions for the WINDOWS FILE SYSTEM. You can change this by adjusting the plugin to match your required dataset.

Copy the mccli dataset..... line as much as you like changing the --exclude="**\" section to match the folders/files you wish to exclude from your data set.

---- SNIP HERE ----

#!/bin/bash

EXPECTED_ARGS=2
E_BADARGS=65

if [ $# -ne $EXPECTED_ARGS ]
then
  echo "Usage: `basename $0` <domain name> <dataset name>"
  exit $E_BADARGS
fi

mccli dataset add-exclude --domain=$1 --exclude="**\path\filename*.dat" --name="$2" --plugin=3001
mccli dataset add-exclude --domain=$1 --exclude="**\parth\subpath\*" --name="$2" --plugin=3001

---- SNIP HERE ----

Save above as whatever.sh and make the file executable via "chmod 755 whatever.sh" to make it executable. 

Run the command "./whatever.sh DOMAIN MYDATASET"

Sit back and relax as your exclusions are created! this is great when you are rolling out a default exclusion set across multiple Avamar nodes.

iDRAC and Avamar - How to reset your iDRAC password if you have forgotten it from the CLI

You cannot remember your iDRAC password? fear not! connect to the CLI and run this trusty command on the Avamar node that needs to reset the iDRAC password to factory default or whatever you choose.

racadm config -g cfgUserAdmin -o cfgUserAdminPassword -i 2 "calvin"

Viewing and acknowledging Avamar events

Avamar events can be quite annoying to acknowledge when you have a large number of them appearing in the GUI. After you have fixed all the issues that caused these events the quickest way to acknowledge them is from the CLI. You can archive this by use the trusty "mccli" command.

The first command using "event show --unack=true" option will shows all the unacknowledged events.

AVAMARNODE:~/#: mccli event show --unack=true
0,23000,CLI command completed successfully.
ID Date Type Code Category Severity Domain Summary
-- ---- ---- ---- -------- -------- ------ -------

The second command using "event ack --all=true" option will acknowledge all events.

AVAMARNODE:~/#: mccli event ack --all=true
0,23000,CLI command completed successfully.
Attribute    Value
------------ -----

events-acked 0

Monitoring activity on your Avamar node

Monitoring activity on your Avamar node is done via the "mccli" command. There are a few switches that can be used and here we use "activity show". This command will show all active jobs that are currently running on the system. They will be listed with their current status and progress. 

AVAMARNODE:~/#: mccli activity show --active=true

0,23000,CLI command completed successfully.

ID Status Error Code Start Time Elapsed End Time Type Progress Bytes New Bytes

-- ------ ---------- ---------- ------- -------- ---- -------------- ---------

In some scenarios you will be checking on the backup status of a particular server to see if backups have been performed successfully. You can do this with the following "mccli" command using the --name flag to show one particular clients backup and replication status for the last few days. Note that the domain is case sensitive as is the client name, if yours are uppercase it must be upper case in the command.

AVAMARNODE:~/#: mccli activity show --name=/DOMAIN/client.somedomain.com

0,23000,CLI command completed successfully.

ID               Status    Error Code Start Time           Elapsed     End Time             Type               Progress Bytes    New Bytes

---------------- --------- ---------- -------------------- ----------- -------------------- ------------------ ----------------- ---------

9140222880005509 Completed 0          2014-06-08 22:00 EST 03h:04m:30s 2014-06-09 01:04 EST Scheduled Backup   2,316,679,062,932 <0.05%

1402367398208228 Completed 0          2014-06-10 12:29 EST 00h:02m:33s 2014-06-10 12:32 EST Replication Source 1,600,559,188,768 <0.05%

9140231520004309 Completed 0          2014-06-09 22:00 EST 03h:21m:50s 2014-06-10 01:21 EST Scheduled Backup   2,316,608,844,685 <0.05%

1402195033679571 Completed 0          2014-06-08 12:37 EST 00h:02m:34s 2014-06-08 12:40 EST Replication Source 1,158,866,936,890 <0.05%

9140214240003409 Completed 0          2014-06-07 22:00 EST 02h:59m:33s 2014-06-08 00:59 EST Scheduled Backup   2,316,668,147,366 <0.05%

1402281253953797 Completed 0          2014-06-09 12:34 EST 00h:03m:05s 2014-06-09 12:37 EST Replication Source 1,348,096,880,224 <0.05%