How to recover deleted backups from an Avamar node that is replicated to another Avamar node

Avamar replication is quite often a life saver! you replicate your data to an offsite location in the event of disaster and restore from there if needed. Normally, this is a one way process as in not needing to replicate back to the source node.

Recently however, I had a situation were a site had a weeks worth of backups deleted for a single server by mistake. We were fortunate enough to have this occur after garbage collection, so we were quickly able to re-backup the server to ensure 99% of the chunks were not deleted however, we were still missing that week worth of backups!!

We had a replicated copy of this data at our replica site and figured it shouldn't be too hard to get that data replicated back into that clients backup set for local restoration purposes.

Challenge accepted! this was our first attempt:

Run a manual replicate command with the DPN user from the replica node:

ssh-agent bash
ssh-add ~dpn/.ssh/dpnid

replicate --dstaddr=destination.mydomain.com --dstid=repluser --dstpassword=******** --hfsaddr=source.mydomain.com --id=repluser --password=******** --dstpath=/ORIGINALDOMAIN/client.mydomain.com --srcpath=/REPLICATE/destination.mydomain.com/ORIGINALDOMAIN/client.mydomain.com

It phrased correctly and immediately produced this lovely error:

ERROR: Source path cannot contain /REPLICATE for replicate!

Damn! back to the drawing board.. after a few hours and some hair pulling we were able to get a working command to run from the replica node and replicate that data back to the original location. The missing flag was "--restore" and below is the command that needs to be run from the replica node as the DPN user:

ssh-agent bash
ssh-add ~dpn/.ssh/dpnid

replicate --restore --dstaddr=destination.mydomain.com --dstid=repluser --dstpassword=******** --hfsaddr=source.mydomain.com --id=repluser --password=******** --dstpath=/ORIGINALDOMAIN/client.mydomain.com --srcpath=/REPLICATE/destination.mydomain.com/ORIGINALDOMAIN/client.mydomain.com

The replication will appear as a normal backup job in the GUI for you to monitor or you can watch the replicate output from the CLI on the replica node.

The data restoration time back to the original node will take minutes or hours depending on how any chunks have been deleted if a GC was run. Hopefully if you are reading this the GC hasn't run or you have a fast pipe between the replica and master!! 

Creating Avamar backup schedules from the CLI

Creating a schedule for your backup policies is a must.

Here we add a schedule called "DAILYBACKUP" that is assigned to the root / domain in Avamar (available to all policies) and it's set to run Monday thru Sunday from 6pm for a duration of 12hrs using the timezone of SYDNEY. 

Take note of the timezone, if you are setting up an Avamar node in a different timezone use the correct city, using your GUI interface from a different country will create the schedule based on YOUR timezone. 

This is why I use the CLI to create the schedules for nodes that are not located in my local timezone.

mccli schedule add --name=DAILYBACKUP --domain=/ --days=Mu,Tu,W,Th,F,Sa,Su --start=18:00 --duration=12:00 --tz=Asia/Sydney

Creating a default Avamar retention policy

With all backups you want to have a "default" retention period for the backup set. You can create this default retention set with the below mccli command from the CLI. Adjust the "+0D" to match your required retention timing. In my example below we have basic and adv-daily retention set to 60 days. This means any policy running this retention will have backups kept for 60 days before garbage collection will remove these backups from the grid.

mccli retention add --name=RETENTION_NAME --basic=+60D --adv-daily=+60D --adv-weekly=+0D --adv-monthly=+0D --adv-yearly=+0D

Creating and manipulating Avamar datasets

Datasets must be created in the Avamar GUI (or at least I cannot figure out how to create them in the CLI). 

Once created you can add mass exclusions via the CLI with the script below. The script adds exclusions for the WINDOWS FILE SYSTEM. You can change this by adjusting the plugin to match your required dataset.

Copy the mccli dataset..... line as much as you like changing the --exclude="**\" section to match the folders/files you wish to exclude from your data set.

---- SNIP HERE ----

#!/bin/bash

EXPECTED_ARGS=2
E_BADARGS=65

if [ $# -ne $EXPECTED_ARGS ]
then
  echo "Usage: `basename $0` <domain name> <dataset name>"
  exit $E_BADARGS
fi

mccli dataset add-exclude --domain=$1 --exclude="**\path\filename*.dat" --name="$2" --plugin=3001
mccli dataset add-exclude --domain=$1 --exclude="**\parth\subpath\*" --name="$2" --plugin=3001

---- SNIP HERE ----

Save above as whatever.sh and make the file executable via "chmod 755 whatever.sh" to make it executable. 

Run the command "./whatever.sh DOMAIN MYDATASET"

Sit back and relax as your exclusions are created! this is great when you are rolling out a default exclusion set across multiple Avamar nodes.

iDRAC and Avamar - How to reset your iDRAC password if you have forgotten it from the CLI

You cannot remember your iDRAC password? fear not! connect to the CLI and run this trusty command on the Avamar node that needs to reset the iDRAC password to factory default or whatever you choose.

racadm config -g cfgUserAdmin -o cfgUserAdminPassword -i 2 "calvin"

Viewing and acknowledging Avamar events

Avamar events can be quite annoying to acknowledge when you have a large number of them appearing in the GUI. After you have fixed all the issues that caused these events the quickest way to acknowledge them is from the CLI. You can archive this by use the trusty "mccli" command.

The first command using "event show --unack=true" option will shows all the unacknowledged events.

AVAMARNODE:~/#: mccli event show --unack=true
0,23000,CLI command completed successfully.
ID Date Type Code Category Severity Domain Summary
-- ---- ---- ---- -------- -------- ------ -------

The second command using "event ack --all=true" option will acknowledge all events.

AVAMARNODE:~/#: mccli event ack --all=true
0,23000,CLI command completed successfully.
Attribute    Value
------------ -----

events-acked 0

Monitoring activity on your Avamar node

Monitoring activity on your Avamar node is done via the "mccli" command. There are a few switches that can be used and here we use "activity show". This command will show all active jobs that are currently running on the system. They will be listed with their current status and progress. 

AVAMARNODE:~/#: mccli activity show --active=true

0,23000,CLI command completed successfully.

ID Status Error Code Start Time Elapsed End Time Type Progress Bytes New Bytes

-- ------ ---------- ---------- ------- -------- ---- -------------- ---------

In some scenarios you will be checking on the backup status of a particular server to see if backups have been performed successfully. You can do this with the following "mccli" command using the --name flag to show one particular clients backup and replication status for the last few days. Note that the domain is case sensitive as is the client name, if yours are uppercase it must be upper case in the command.

AVAMARNODE:~/#: mccli activity show --name=/DOMAIN/client.somedomain.com

0,23000,CLI command completed successfully.

ID               Status    Error Code Start Time           Elapsed     End Time             Type               Progress Bytes    New Bytes

---------------- --------- ---------- -------------------- ----------- -------------------- ------------------ ----------------- ---------

9140222880005509 Completed 0          2014-06-08 22:00 EST 03h:04m:30s 2014-06-09 01:04 EST Scheduled Backup   2,316,679,062,932 <0.05%

1402367398208228 Completed 0          2014-06-10 12:29 EST 00h:02m:33s 2014-06-10 12:32 EST Replication Source 1,600,559,188,768 <0.05%

9140231520004309 Completed 0          2014-06-09 22:00 EST 03h:21m:50s 2014-06-10 01:21 EST Scheduled Backup   2,316,608,844,685 <0.05%

1402195033679571 Completed 0          2014-06-08 12:37 EST 00h:02m:34s 2014-06-08 12:40 EST Replication Source 1,158,866,936,890 <0.05%

9140214240003409 Completed 0          2014-06-07 22:00 EST 02h:59m:33s 2014-06-08 00:59 EST Scheduled Backup   2,316,668,147,366 <0.05%

1402281253953797 Completed 0          2014-06-09 12:34 EST 00h:03m:05s 2014-06-09 12:37 EST Replication Source 1,348,096,880,224 <0.05%

Checking the status of your Avamar node from the CLI

Checking the status of your Avamar node can be done with a few simple commands via a telnet client. To check the current status of the node run "dpnctl status".

AVAMARNODE:~/#: dpnctl status

Identity added: /home/dpn/.ssh/dpnid (/home/dpn/.ssh/dpnid)

dpnctl: INFO: gsan status: up

dpnctl: INFO: MCS status: up.

dpnctl: INFO: EMS status: up.

dpnctl: INFO: Backup scheduler status: up.

dpnctl: INFO: dtlt status: up.

dpnctl: INFO: Maintenance windows scheduler status: enabled.

dpnctl: INFO: Unattended startup status: enabled.

Everything should be "up" and the maintenance windows enabled unless your have deliberately disabled them to perform e.g. An out of schedule garbage collection.

The next command "status.dpn" will give us the system-status, access-status, checkpoint, garbage collection & hfscheck statuses. Use this command to confirm your maintenance activities have run correctly. If they haven't run you'll need to investigate the logs.

AVAMARNODE:~/#: status.dpn

Tue Jun 10 17:34:57 EST 2014  [AVAMARNODE] Tue Jun 10 07:34:57 2014 UTC (Initialized Thu Jun 27 02:17:56 2013 UTC)

Node   IP Address     Version   State   Runlevel  Srvr+Root+User Dis Suspend Load UsedMB Errlen  %Full   Percent Full and Stripe Status by Disk

0.0     xxx.xxx.xxx.xxx   x.x.x-xxx  ONLINE fullaccess mhpu+0hpu+0hpu   1 false   0.21 30737  3921868  20.4%  20%(onl:1777) 20%(onl:1772) 20%(onl:1779) 20%(onl:1769) 20%(onl:1772) 20%(onl:1770)

Srvr+Root+User Modes = migrate + hfswriteable + persistwriteable + useraccntwriteable

All reported states=(ONLINE), runlevels=(fullaccess), modes=(mhpu+0hpu+0hpu)

System-Status: ok

Access-Status: full

Last checkpoint: cp.20140609233540 finished Tue Jun 10 09:39:21 2014 after 03m 41s (OK)

Last GC: finished Tue Jun 10 17:06:24 2014 after 14m 52s >> recovered 10.69 GB (OK)

Last hfscheck: finished Tue Jun 10 09:35:09 2014 after 26m 20s >> checked 3635 of 3635 stripes (OK)

Maintenance windows scheduler capacity profile is active.

  The maintenance window is currently running.

  Next backup window start time: Tue Jun 10 18:00:00 2014 EST

  Next blackout window start time: Wed Jun 11 06:00:00 2014 EST

  Next maintenance window start time: Wed Jun 11 09:00:00 2014 EST

How to find the serial number of your Avamar node from the CLI

Note: All commands are run from the CLI using putty or some other telnet/ssh terminal.

For Avamar Datastore Gen 2 and Gen 3 nodes

/usr/bin/omeport chassis frontpanel | grep "LCD Line"

For Avamar Gen 2 and Gen 3 nodes setup in a RAIN array connect to the Utility node and run

mapall --all+ --quiet '/usr/bin/omreport chassis frontpanel | grep "LCD Line 1" | sed "s/^.*: *\(.*\)/\1/"'

For Avamar Datastore Generation 4 nodes

/usr/bin/ipmitool fru print 0 | grep "Product Asset Tag" | sed "s/^.*: *\(.*\)/\1/"

For Avamar Gen 4 nodes in RAIN array connect to the Utility node and run

mapall --noerror --user=root --nodes=all+ --quiet 'echo `hostname -i`": "`/usr/bin/ipmitool fru print 0 | grep "Product Asset Tag" | sed "s/^.*:\s\(.*$\)/\1/"`'

The serial number will be the first part of the result. You will also receive the IP address of the node.

Avamar default passwords

To access an Avamar node use a telnet/ssh client such as putty all the default passwords are below:



You should have changed your default passwords with the EMC engineer during the installation process. If not change them!

Introduction

I've been working with Avamar for over 5 years, It's now time to share some of the quirky commands and knowledge I've acquired over the years.

For those of you new to Avamar you'll fall in love with the Avamar Administrator while also loosing some hair ;) but after a while you'll get used to the quirks and realise the CLI is your friend and much faster than the GUI.

I hope the tips and commands I share help you in some way.

Enjoy.

Jason