Poor Performance Followup

August 7, 2014, 11:29 am

≪ Previous: Why Are You Not Using Google & YouTube?

As a follow-up to the previous poor performance post I thought I'd post what the outcome was. As it turns out we checked performance tuning settings in TSM and AIX and no performance increase was seen. We asked the DB2 admins to review any of their settings and they could not find any tunables that had not already been implemented. We sent in servermon.pl output and although they saw performance that was sub-par, they couldn't designate what was causing it. There are no server/adapter/switch/disk/tape errors so nothing emerged as the culprit for our poor throughput performance.

So we reviewed the backup time of each TSM storage agent server used to backup this 101 TB SAP database. At the time the storage agents that perform the backup consisted of 5 LPARS, 4 of those in a single frame each with their own assigned I/O drawer. The 5th was in a separate 740 frame with its own I/O drawer. The 5th storage agent was completing the backup in a fraction of the time of the other 4 so we concluded we must be overloading the CEC on the 740. We moved one of the four storage agents out of the frame to a secondary frame and the results were awesome. See below:

You'll notice that the backup time didn't change with the update of the tape drives from E06 to E07. Hardware layout matters more than the performance of the tape drives. When a vendor tells you just updating the hardware to newer iterations will increase performance take it will a grain of salt. In our case we did testing of the new tape drives and no performance gains were seen but the go ahead was given to upgrade to the newer hardware and as you'll see we didn't gain anything until we reworked the environment. Our task now is to identify how to increase TSM internal job performance (i.e. migration and storage pool backup) which has not seen significant performance gains from the tape upgrades.

↧

Learn From My Mistakes

February 22, 2017, 8:27 am

≫ Next: Spectrum Protect 8.1 New DB Functions Supported

≪ Previous: Poor Performance Followup

We recently did a DR test and ran into a serious issue with the recovery of the TSM instance. The DR test was to restore the TSM DB from data center A (DC-A) to data center B. The DC-A instance had been created the previous year so we only needed to remove the DB (DSMSERV REMOVEDB TSMDB1) and restore the DB. I followed the steps and restored the DB and after restoring the DB 99% the restore failed. The restore failure error pointed us to the size available being too small for the required. We were off by a couple GB in our DB directory sizes so we had to add another directory to the list. I tried to rerun the format of the space and quickly received and error that the DB was present. I then ran the DSMSERV REMOVEDB TSMDB1 command again then tried to rerun the format and received the same results. We tried multiple commands

db2 uncatalog db TSMDB1

db2idrop serverA

/opt/tivoli/tsm/db2/bin/db2greg -delinstrec service=db2,version=9.7.0.9,instancename=serverA,instancepath=/drtsmserver/serverA/sqllib

This last command allows you to manually drop the DB from the global registry. When we tried this a couple times the instance would be removed from the global registry and when we would try to rerun the restore we would get the error that the DB was present and the instance would reappear in the global registry. (Notice the instance path....that is going to come play into the final solution.)

So at this point we went round and round with various commands to drop/remove/delete the instance from DB2 and the commands would fail saying that the instance was not present and then an attempt to restore the DB would fail saying the DB was present. We went round and round and round until I finally restored the DB under a new ID and instance directory which was successful.....BUUUUUUTTTTTT.....there in lies another issue. Once the DB restore was successfully restored it failed to allow us to bring up the TSM instance because the new ID was not the original owner of the DB instance. Immediately I realized my mistake as this had happened to me years ago and I totally forgot this DB2 requirement. You see DB2 does not allow an ID other than those that have been granted permission to bring the DB up. I get it, it's a security thing, but the TSM instances almost all Admins create only has one active ID allowed ownership of the DB2 instance. You can add as many as you like but it's not something IBM or Tivoli openly suggest or recommend. So if the ID you typically use wont allow the restore or for some reason can't be used and you want to use a new ID for the DB your S.O.L. If it's after the fact, IBM cannot provide a way for you to assign an additional owner of the DB2 TSM instance. IBM's recommendation was go back to DC-A and grant the new ID rights to the instance, run a TSM DB backup, copy/replicate the instance to DC-B, then restore the DB and it would work under the new ID.

By this point in the DR process we were so behind in the recovery of the TSM server we scrapped our portion of the DR test (we were not a significant portion of test, which was to test Recovery Point capabilities and TSM was a fall back). I continued to work with IBM on the restore under the original ID and the looping issue and after a couple attempts, the IBM level 2 DB2 support person identified that the creation of the TSM instance had created DB files in

/drtsmserver/serverA/serverA/NODE0000

and a simple rename of that directory should the the DB present/not present issue. We had repeatedly cleared out the DB and Log file systems but had forgotten about the DB folder and files created in the instance directory. TSM had created that subdirectory with DB related files. We had renamed the /drtsmserver/serverA/sqllib directory more than once, but we had forgotten about the other instance directory as IBM states in this technote. When we renamed the second serverA directory and tried the restore and it ran to completion successfully. HEAD SLAP! This was one of the times where even with another set of eyes we were not seeing the bigger picture. IBM level one support was probably looking at the wrong issue and level two was stumped because we moved away from the initial issue we called about and were now asking them how bring up the TSM DB that we restored under the alternate ID.

OK at this point I was exhausted and frustrated with myself. So what did I come away with in this situation....

Triple verify your DB file space size is adequate. (DOH!)
From the instance home directory copy the critical files to an alternate location.

This is in case the restore fails and clears them or updates them incorrectly (trust me this happened more than once)

If the restore fails or a rerun of the restore is required rename both <inst home>/<inst name>/<inst name> and <inst home>/<inst name>/sqllib

IMHO I would also create another ID or two on the production server and grant them access to the TSM DB instance just in case you had an issue with the primary ID.

This is not required but cant hurt.

↧

Spectrum Protect 8.1 New DB Functions Supported

April 20, 2017, 3:12 pm

≫ Next: TSM Menu - Bash Shell Scripting

≪ Previous: Learn From My Mistakes

I upgraded some of my 6.3.4 servers to TSM 8.1 recently and have found the newer DB2 version (v11.1) offers some added functions for SQL queries. I have been trying to build a report to track an Exchange DAG servers backups. The problem is that the easiest way to track the backups of DB's is to view the last completed backup for each of nodes file spaces. That's the easy part. The hard part is that the file space name is huge.

TEST-AP-DAG\Microsoft Exchange Writer\{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}\TEST-MB-05\0e41e645-9acf-4b80-bc85-d606e04fe4d8

TEST-AP-DAG\Microsoft Exchange Writer\{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}\Mailbox Database 1223544393\2bc06db2-1966-4fd7-9545-f667102b0b7d

So how to extract the DB name, in this case TEST-MB-05 and Mailbox Database 1397248650 from the name when the length of the DB name changes? After some investigation and trial and error I found that the LOCATE_IN_STRING function works in v8.1. Whether it works in 7.x (DB2 v10.5) I can't say since I don't have a 7.x server to run it against. If any of you out there try this and it works let me know. So here's the script. I think you can read it and see what I am doing. If not let me know in the comments and I'll explain...

tsm: TSMSERV>q script DAG-EXCH-RPT f=raw

select -
VARCHAR(NODE_NAME,12) as NODE, -
varchar(substring(filespace_name, LOCATE_IN_STRING(filespace_name, '\', 1, 3), -
LOCATE_IN_STRING(filespace_name, '\', 1, 4) - LOCATE_IN_STRING(filespace_name, '\', 1, 3)),30) as Exch_DB, -
date(backup_end) AS LAST_GOOD_BACKUP -
from FILESPACES -
where -
node_name='TEST-AP-DAG' -
order by Exch_DB, backup_end asc

Here is sample output:

NODE EXCH_DB LAST_GOOD_BACKUP

------------- ------------------------------- -----------------

TEST-AP-DAG \TEST-MB-01 2017-04-18

TEST-AP-DAG \TEST-MB-02 2017-04-18

TEST-AP-DAG \TEST-MB-03 2017-04-18

TEST-AP-DAG \TEST-MB-04 2017-04-18

TEST-AP-DAG \TEST-MB-05 2017-04-18

TEST-AP-DAG \TEST-MB-TEMP 2017-04-18

TEST-AP-DAG \MISC-MB-TEMP 2017-04-18

TEST-AP-DAG \Mailbox Database 1128394465 2017-04-18

TEST-AP-DAG \Mailbox Database 1397248650 2017-04-18

If another function would work better feel free to correct the script and leave it for everyone in the comments. Now to go and find more functions I can use to manipulate my data!

↧

TSM Menu - Bash Shell Scripting

May 5, 2017, 11:08 am

≫ Next: TSM SQL TDP Better Backup Script

≪ Previous: Spectrum Protect 8.1 New DB Functions Supported

For years I have maintained a shell script on our jump server that was a menu for accessing all the TSM servers in our environment. It was a nice interface and great for noobies. It allowed for you to set your password and ID variables before the menu loaded so anyone could run the menu under their ID, then presented them with a nice menu to pick which TSM server they wished to access. It was nice and easy. The only problem was when servers were added or decommissioned. When that occurred I would then have to go out and revise the menu. This was a pain and soon to be more painful as we plan to add a lot more TSM servers in the near future. I remember seeing someone post that they had created a menu system based off of the dsm.sys file so I decided to write my own.

Before:

##################################################################
# #
# Infocrossing #
# UNIX ADMINISTRATION #
# TSM MENU #
# #
##################################################################
Omaha
0) ouax000 1) ouax001
2) ouax002 3) ouax003
4) OTSM01 5) OTSM02
------------------------------------------------------------------
Phoenix
6) puax001 7) puax002
8) puax008 9) PTSM01
10) PTSM02 11) PTSM20
------------------------------------------------------------------
Singapore
20) SIN01ZZ 99) STSM01
------------------------------------------------------------------
Germany
21) DTSM01 22) DTSM02
23) DTSM03 24) DTSM20
------------------------------------------------------------------
Kings Mtn
25) CTSM01 26) CTSM20
27) CTSM30 28) CTSM40
------------------------------------------------------------------

X/x) Cross Server

S/s) Cmd Prompt

DD/dd) Data Domain Admin

Q/q) Return to Main Menu


Selection:

After (Notice servers listed here that were not in above menu):

################################################################
# #
# Infocrossing #
# Recovery Services #
# TSM MENU #
# #
################################################################

1) C1IF 12) LABTSM 23) PTSM01
2) C1KDC2 13) LUAX001 24) PTSM02
3) CTSM01 14) LUAX002 25) PTSM20
4) CTSM20 15) LTSM01 26) PTSM25-Arch
5) CTSM30 16) NTSM01 27) SIN01ZZ
6) CTSM40 17) OUAX003 28) STSM01
7) CTSM46-Arch 18) OUAX000 29) Cross Server
8) DTSM01 19) OUAX001 30) Refresh Menu
9) DTSM02 20) OTSM01 31) Exit
10) DTSM03 21) OTSM02
11) DTSM20 22) PESTSM01

Please enter your choice:

Ok, so the Before is more organized by region but as new servers come in I wanted the menu to update dynamically. So this new menu is built around the BASH shell's select menu function. The script pulls all the TSM server names from the dsm.sys and populates an array that the select statement reads and generates this menu. Make sure the variables the script declares work for you and your dsm.sys. My dsm.sys looks like this if I grep servername:

SERVERNAME DTSM01 * Germany LIB-MAN
SERVERNAME DTSM02 * Germany
SERVERNAME DTSM03 * Germany
SERVERNAME DTSM20 * Germany LIB-MAN
SERVERNAME CTSM01 * Kings_Mnt LIB-MAN
SERVERNAME C1KDC2 * Kings_Mnt
SERVERNAME C1IF * Kings_Mnt
SERVERNAME CTSM20 * Kings_Mnt
SERVERNAME CTSM30 * Kings_Mnt
SERVERNAME CTSM40 * Kings_Mnt
SERVERNAME CTSM46-Arch * Kings_Mnt
SERVERNAME LUAX001 * Leonia LIB-MAN
SERVERNAME LUAX002 * Leonia
SERVERNAME LTSM01 * Leonia
SERVERNAME NTSM01 * Norcross LIB-MAN
SERVERNAME OUAX000 * Omaha LIB-MAN
SERVERNAME OUAX001 * Omaha
SERVERNAME OUAX003 * Omaha
SErvername OTSM01 * Omaha LIB-MAN
SErvername OTSM02 * Omaha
SERVERNAME PESTSM01 * PEST LIB-MAN
SERVERNAME PTSM01 * Tempe LIB-MAN
SERVERNAME PTSM02 * Tempe
SERVERNAME PTSM20 * Tempe
SERVERNAME PTSM25-Arch * Tempe
SERVERNAME STSM01 * Singapore
SERVERNAME SIN01ZZ * Singapore

SERVERNAME LABTSM

With the dsm.sys file formated this way I can use grep and awk to get specific data. This allows the script to build my menus according to my needs. Especially with the Cross Server sub menu that allows me to run a command across multiple servers at once. In the sub-menu it groups by distinct location which I placed in the servername line. Leave me a comment if you have any comments, questions, or suggestions.

See the script below:

#!/bin/bash
#set -x

### Set Color Variables ###

NORMAL=`echo "\033[0m"`
WHITE=`echo "\033[37m"` #Blue
YELLOW=`echo "\033[33m"` #yellow
GREEN=`echo "\033[32m"`
RED=`echo "\033[31m"`
PURPLE=`echo "\033[35m"`

##### Declare Variables & Arrays ############

prompt="
Please enter your choice: "

TSM_SYS="/opt/tivoli/tsm/client/ba/bin/dsm.sys"
all_servers=(`cat $TSM_SYS | grep -i servername | awk -v OFS='\t''{print $2}' | sort`)
xservers=(`cat $TSM_SYS | grep -i servername | awk -v OFS='\t''{print $4}' | sort -u`)
Germany=(`cat $TSM_SYS | grep -i servername | grep -i germany |awk -v OFS='\t''{print $2}'`)
Omaha=(`cat $TSM_SYS | grep -i servername | grep -i omaha |awk -v OFS='\t''{print $2}'`)
Norcross=(`cat $TSM_SYS | grep -i servername | grep -i norcross |awk -v OFS='\t''{print $2}'`)
Tempe=(`cat $TSM_SYS | grep -i servername | grep -i tempe |awk -v OFS='\t''{print $2}'`)
PEST=(`cat $TSM_SYS | grep -i servername | grep -i pest |awk -v OFS='\t''{print $2}'`)
Leonia=(`cat $TSM_SYS | grep -i servername | grep -i leonia |awk -v OFS='\t''{print $2}'`)
Kings_Mnt=(`cat $TSM_SYS | grep -i servername | grep -i kings_mnt |awk -v OFS='\t''{print $2}'| sort`)
Singapore=(`cat $TSM_SYS | grep -i servername | grep -i singapore |awk -v OFS='\t''{print $2}'`)
lib_man=(`cat $TSM_SYS | grep -i servername | grep -i lib-man | awk -v OFS='\t''{print $2}'`)

export COLUMNS=70
########################

######################
function clearlogin
######################
{
echo "Please wait while I clear all login information."|while read x;do for((i=0;i<${#x};i++));do echo -n "${x:$i:1}";sleep .03;done;echo;done;
unset TSMID
unset TSMPA
unset DDID
echo "."|while read x;do for((i=0;i<${#x};i++));do echo -n "${x:$i:1}";sleep .16;done;echo;done;
echo "."|while read x;do for((i=0;i<${#x};i++));do echo -n "${x:$i:1}";sleep .16;done;echo;done;
echo "Logins have been cleared! "|while read x;do for((i=0;i<${#x};i++));do echo -n "${x:$i:1}";sleep .03;done;echo;done;
sleep 1
}
######################
# end clearlogin #
######################

###############
#END MAIN-MENU#
###############

#########################
function tsmadmin
#########################
{
clear
if [ -z "${TSMID}" ]; then
echo ""
echo -e "$YELLOW Enter your TSM Admin ID: $GREEN \c"
read TSMID
echo ""
echo -e "$YELLOW Enter your TSM Admin Password: \c "
#stty -echo
read -s TSMPA
#stty $stty_orig
fi
clear
tsm-menu
exit
}
####################
# end tsmadmin #
####################

#####################
function tsm-menu
#####################
{
all_done=0
export COLUMNS=70
while ((!all_done )); do
clear
echo -e "$WHITE ################################################################$NORMAL "
echo -e "$WHITE # #$NORMAL "
echo -e "$WHITE # $YELLOW Infocrossing $WHITE #"
echo -e "$WHITE # Recovery Services #"
echo -e "$WHITE # $RED TSM MENU $WHITE #"
echo -e "$WHITE # #$NORMAL "
echo -e "$WHITE ################################################################$NORMAL "
echo -e ""
length=`echo "$((${#all_servers[@]} + 4))"`
PS3="$prompt "
select host in "${all_servers[@]}""Cross Server""Reset Login""Refresh Menu""Exit"; do
if [ "$host" = "Exit" ]; then
all_done=1; clear; exit
elif [ "$host" = "Refresh Menu" ]; then
all_done=1; tsm-menu
elif [ "$host" = "Reset Login" ]; then
all_done=1; clearlogin; tsmadmin
elif [ "$host" = "Cross Server" ]; then
all_done=1; xmenu
elif [ "$REPLY" -gt "$length" ]; then
echo ""
echo "You chose option $REPLY which is invalid. Try again! "
echo ""
else
clear
echo -e "$GREEN "
dsmadmc -id=$TSMID -pa=$TSMPA -servern="$host"
all_done=1; tsm-menu
fi
done
done
}
##########################
# end function tsm-menu #
##########################

#####################
function xmenu
#####################
{
all_done=0
export COLUMNS=50
while ((!all_done )); do
clear
echo -e "$WHITE ################################################################$NORMAL "
echo -e "$WHITE # #$NORMAL "
echo -e "$WHITE # $YELLOW Infocrossing $WHITE #"
echo -e "$WHITE # Recovery Services #"
echo -e "$WHITE # $RED TSM Cross Server Command MENU $WHITE # "
echo -e "$WHITE # #$NORMAL "
echo -e "$WHITE ################################################################$NORMAL "
echo -e ""
length=`echo "$((${#xservers[@]} + 4))"`
PS3="$prompt "
select host in "${xservers[@]}""All Servers""Library Managers""Refresh Menu""Exit"; do
if [ "$host" = "Exit" ]; then
all_done=1; tsm-menu
elif [ "$host" = "Refresh Menu" ]; then
all_done=1; xmenu
elif [ "$host" = "Library Managers" ]; then
clear
echo -e "$GREEN "
xserver $TSMID $TSMPA lib_man
all_done=1; xmenu
elif [ "$host" = "All Servers" ]; then
clear
echo -e "$GREEN "
xserver $TSMID $TSMPA all_servers
all_done=1; xmenu
elif [ "$REPLY" -gt "$length" ]; then
echo ""
echo "You chose option $REPLY which is invalid. Try again! "
echo ""
else
clear
echo -e "$GREEN "
xserver $TSMID $TSMPA "$host"
all_done=1; xmenu
fi
done
done
}
##########################
# end function xmenu #
##########################

###################
function xserver
###################
{
DC=$3
echo -e ""
echo -e " $YELLOW Enter the command to process: $NORMAL \c"
read CMD
echo ""
echo -e " $YELLOW Do you want to save the data to a commadelimited csv file? (Yes or No [N]) $NORMAL \c"
read ANSWER
case $ANSWER
in
y|Y|YES|Yes|yes)
echo ""
echo -e " $YELLOW Enter the file name to save the data: $NORMAL \c"
read FILE
echo ""
touch $FILE
cat /dev/null > $FILE
eval varAlias=\${$DC[@]}
for Server in ${varAlias[@]}
do
echo ""
echo -e "Processing TSM server $Server "
echo ""
dsmadmc -id=$1 -pa=$2 -dataonly=yes -commadelimited -servern=$Server $CMD | grep -v ANR | grep -v ANS >> $FILE
echo ""
done
xmenu
;;
n|N|*)
echo ""
eval varAlias=\${$DC[@]}
for Server in ${varAlias[@]}
do
echo ""
echo -e "Processing TSM server $Server "
echo ""
dsmadmc -id=$1 -pa=$2 -dataonly=yes -servern=$Server $CMD
echo ""
done
sleep 2
xmenu
;;
*)
echo ""
eval varAlias=\${$DC[@]}
for Server in ${varAlias[@]}
do
echo ""
echo -e "Processing TSM server $Server "
echo ""
dsmadmc -id=$1 -pa=$2 -dataonly=yes -servern=$Server $CMD
echo ""
done
sleep 2
xmenu
;;
esac
}
########################
# end xserver function #
########################
clear
echo -e "$GREEN"
echo "Please Wait Loading Menu....."|while read x;do for((i=0;i<${#x};i++));do echo -n "${x:$i:1}";sleep .03;done;echo;done;
sleep 2

tsmadmin

↧

TSM SQL TDP Better Backup Script

June 1, 2017, 10:57 am

≫ Next: TSM/Spectrum Protect 7.x Client Issue

≪ Previous: TSM Menu - Bash Shell Scripting

On a call today I had an admin complain that the TSM SQL TDP was issuing failed backup errors when the script encountered a database that was offline. So I asked if he had searched Google for an alternate script and he had not. I did a quick query and found this nice batch file that appears to do exactly what is needed. The reader on the site commented that the script did not work on his SQL cluster. Does anyone have a similar script that would work on a cluster? Or could you modify this script to work in a cluster environment? The link to the script is provided below.

http://www.applepie.se/tivoli-storage-manager-and-sql-backup

↧

TSM/Spectrum Protect 7.x Client Issue

August 7, 2017, 10:54 am

≫ Next: Spectrum Protect 8.1.0 vs. 8.1.2 Client Issue

≪ Previous: TSM SQL TDP Better Backup Script

I ran across an issue with the TSM/Spectrum Protect client recently that through investigation showed that the issue has not been patched and the supposed resolution works....sometimes. The issue is that when an AIX backup is running you can experience the following error:

08/02/17 03:24:52 ANS1512E Scheduled event 'CS-FS-U-02.00' failed. Return code = 12.
08/02/17 13:54:36 ANS2820E An interrupt has occurred. The current operation will end and the client will shut down.
08/03/17 03:39:20 calloc() failed: Size 31496 File ../mem/mempool.cpp Line 1090
08/03/17 03:39:20 ANS1999E Incremental processing of '/usr/ibm' stopped.

This is a memory issue and can occur if TSM cannot allocate enough memory during the scheduled backup. The interesting thing is that manual backups run without issue. It is only when a scheduled backup is run that we experience the error. The server in question has only 8GB so we do know that real memory is limited so I followed the troubleshooting tips I saw online and checked the file system for excessive files. Querying the file system determined that was not the issue. I could not exclude the file system and so I added the MEMORYEFFICIENTBACKUP YES option thinking that would resolve the issue. Unfortunately the addition of the MEMORYEFFICIENTBACKUP YES option did not work. Subsequent investigation showed some people had to downgrade their TSM client version to a 6.4 level to resolve the problem. Unwilling to do so I changed the option so it used the disk cache function. So far using disk cache has worked without issue, but it's concerning that a change to the TSM/SP client has created this memory issue. So the fix was to add the following two options to my dsm.sys:

memoryefficientbackup DISKCACHEM
diskcachelocation /tmp

Please note that my /tmp file system is over 2GB in size and only 1% used so make sure you have a sufficiently sized file system you can use if you use the DISKCACHEMETHOD option.

↧

Spectrum Protect 8.1.0 vs. 8.1.2 Client Issue

October 25, 2017, 7:04 am

≫ Next: Stay Away From The Spectrum Protect For Windows 8.x Client

≪ Previous: TSM/Spectrum Protect 7.x Client Issue

I just wanted to let everyone know that there is a compatibility issue with the Spectrum Protect 8.1.2 client with the 8.1.0 server. If you upgrade the client to SP 8.1.2 and your server is at 8.1.1 or 8.1.0 your server will be unable to start. The error you will get is

Spectrum Protect ANR0162W Supplemental database diagnostic information: -1:08001:-30082 ([IBM][CLI Driver] SQL30082N Security processing failed with reason "24" ("USERNAME AND/OR PASSWORD INVALID").

The issue has to do with the GSKit8.gskcrypt64.ppc.rte and GSKit8.gskssl64.ppc.rte in the 8.1.2.0 client. When we rolled back to the 8.1.0 client the SP server started without an issue.

↧

Stay Away From The Spectrum Protect For Windows 8.x Client

December 4, 2017, 2:48 pm

≫ Next: Audit Occupancy Issue

≪ Previous: Spectrum Protect 8.1.0 vs. 8.1.2 Client Issue

I wanted to make everyone aware that starting with the 8.1 Windows Client, IBM has deprecated the System State restore process so that any restore attempt will require the use of a WinPE environment. With the 8.1.x client you can no longer do a complete restore of a server within the normal windows environment and will have to make sure you have a compatible WinPE for every Windows Hardware type or environment. This, in my opinion, is a huge break of the software. Although IBM will swear it's because Microsoft does not support online system state restores it appears to be a weak argument. I say this knowing that if you are on a 7.x or earlier version of the client IBM will support you with the system state restore, but as of 8.1 you will be directed to build a Windows PE environment and follow a series of step that the support person I spoke with treated as if it should be second nature to any Windows Admin.

We recently were migrating a clients server and in the process the disks were corrupted. The client installed on the new machine was 8.1.2 and when the Windows Admins tried to restore the system state they experienced the following error.

After speaking with support we were directed to the following document by Andrew Raibeck that discussed creating the WinPE disk image with the TSM client code that would allow the offline restore of the System State. We attempted to build the ISO but when the Windows Admin tried to run the dsmc client we experienced the following error:

Who do we call when we experience an error like this in a WinPE environment? IBM or Microsoft? This is ridiculous and I don't think IBM fully grasps how big of an issue this will be with their customers. Am I now required to make sure that to be ready for a DR scenario or test that we have WinPE ISOs and that my admins are fully aware how to use and troubleshoot them? So basically make sure you add a handful of hours to any Windows full server restore situation. If you were fully aware of this and took precautions what were they? My thinking is that many admins have no idea this is going to bite them and its better you know now than find out the hard way.

↧

Audit Occupancy Issue

January 23, 2018, 7:41 am

≫ Next: Spectrum Protect Server 8.1.4 DB Issue

≪ Previous: Stay Away From The Spectrum Protect For Windows 8.x Client

1/24/2018Update
IBM Support contacted me and stated that the developers said the issue was actually a defect (APAR IT23153) and that you can fix the issue by logging on the SP server as the instance owner and running the following commands:

db2 connect to tsmdb1
db2 set schema tsmdb1
db2 "update tsmdb1.sd_pool_clusters set physoccupancy=1 where physoccupancy=0"
login as admin to Spectrum Protect instance
run AUDIT LICense command
run Q AUDITOCCupancy command

You should now see the AUDITOCC table is populated.

1/23/2018
We setup a new Spectrum Protect instance with a Directory Container Storage Pool and when we went to gather storage data for billing purposes we discovered that the AUDITOCC table in the database showed no storage data. The first troubleshooting step was to check to see if AUDITSTORAGE option was set to YES which it was. Then we reran the AUDIT LICENSE command to have Spectrum Protect audit the storage and after completing there was still no data in the AUDITOCC table. I contacted IBM and asked for support to tell me why I had no data in my AUDITOCC table and was met with level 1 support unable to provide an answer (which was probably due to them not knowing my servers full configuration). When the PMR was passed up to the chain to level 2 support and they reviewed the log details I sent in, they responded that DIRECTORY CONTAINER storage pools do not provide occupancy data to the AUDITOCC table. To gather the required data that the AUDITOCC table normally would provide you have to run the GENERATE DEDUPSTATS command.

GENERATE DEDUPSTATS <STORAGE POOL> <NODE NAME>

Please note that there is nothing in the Spectrum Protect manuals that will inform you that DIRECTORY CONTAINER storage pools do not provide AUDITOCC table data. IBM Level 2 support has opened a PMR requesting that the documentation be updated for the QUERY AUDITOCC command. I am not sure if you can use a * with the node name and have GENERATE DEDUPSTATS but will be trying it today and will let you all know the results in the comments below.

↧

Spectrum Protect Server 8.1.4 DB Issue

July 17, 2018, 1:14 pm

≫ Next: Personal Back Ups!

≪ Previous: Audit Occupancy Issue

Recently we ran into a situation where our Spectrum Protect 8.1.4 server started to crash. We reviewed the db2diag.log file and identified the error as ANR0131E Server DB space exhausted. Upon review of our DB file systems the OS showed 7GB available in the 5 file systems assigned for DB space.

/dev/tsm20log 111616.00 9784.93 92% 205 1% /tsmserver/tsm20log

/dev/tsm20arch 102400.00 66821.52 35% 100 1% /tsmserver/tsm20arch

/dev/tsm20fail 101376.00 100964.16 1% 21 1% /tsmserver/tsm20fail

/dev/tsm20dblv01 101376.00 7622.45 93% 104 1% /tsmserver/tsm20dblv01

/dev/tsm20dblv02 101376.00 7622.48 93% 104 1% /tsmserver/tsm20dblv02

/dev/tsm20dblv03 101376.00 7622.45 93% 104 1% /tsmserver/tsm20dblv03

/dev/tsm20dblv04 101376.00 7622.46 93% 105 1% /tsmserver/tsm20dblv04

/dev/tsm20dblv05 101376.00 7622.45 93% 105 1% /tsmserver/tsm20dblv05

The server had been crashing from the previous night and we had at first thought it was due to the active log filling and not writing out to the archive log fast enough. What was weird is we could bring TSM up and it would run for an hour or two then crash again. When we identified it was not the active log but the DB space (earlier the next morning) we were confused. Why would we get a space notification when the DB has 35GB of space still available. My co-worker created an additional file system of the same size as the other DB file systems and ran the dsmserv extend dbspace command and the file systems almost immediately jumped to 99% utilized. So it appears Spectrum Protect needed to write 35+GB of DB data and could not due to space available? We also noticed that our offshore people had added tsm20dblv05 a few months back but had used a different disk array (slow SATA) and had not informed us, which we think might have contributed to the performance issues we had been seeing on this server recently and maybe was related to the DB space issue??? Not sure, but be aware that this can occur. I will need to research how DB2 allocates space and in what chunk size it does so.

↧

Personal Back Ups!

July 19, 2018, 11:28 am

≫ Next: DB2 Select Example In TSM/SP

≪ Previous: Spectrum Protect Server 8.1.4 DB Issue

I have been doing this backup/archive work for almost 20 years and I can't go a week without some acquaintance asking me about backing up their personal data. It used to be large portable USB drives were the best option but as home internet connections have increased there throughput to the point of some offering giga-bit speeds, it makes more sense to use a cloud based backup provider. This article is based off of my experience and I only offer my opinion here so take it for what it is, but I recommend everyone look into a good backup solution or else you could loose all the important memories you have accumulated as life has become more digital.

So what do I use to protect my memories? I used to use Dropbox, then when Google Drive became available I used it, and then I started using Microsoft SkyDrive now known as OneDrive. I began to exclusively use OneDrive several years ago when I was able to get more than the paltry 5 - 10GB that the others allowed. At one point I freely acquired 100GB by using Bing and gathering Bing points. I am cheap and didn't want to pay for my cloud space. I had portable hard drives with my data but knew it wasn't the solution. So when my free OneDrive space was due to expire I was offered an Office365 family plan with 1TB of space for up to 5 people in the family. With my wife and I each having 128GB iphone that my wife has actually filled with video and pictures during trips, I knew it was the best solution for us. Not only did I get 1TB for each of us, it could cover our personal home computer and our daughter phone also. Oh, and did I mention it comes with access to download Office to our home and daughters laptop? All this is good but it would be useless if the OneDrive app was terrible. Truthfully, I find it quite easy to use. Microsoft has continued to fix bugs and add features it has become a great tool I use almost daily.

I write this article because it's important you think about how you will protect memories. Sure you can upload to Google Photo's or put all your items on Facebook, but each has it's own drawbacks. Google Photo's does not keep the photo at its original resolution, and Facebook is not designed to be a backup solution. I have lost personal photos and documents from when I was younger due to losing them during moves or becoming damaged by mishandling. I will never get them back, had I scanned/digitzed them I would have been able to keep them safe. I also understand that nothing is 100% protected or safe, but by putting my files in the cloud they are more safe than on a portable drive or sitting in a drawer.

↧

DB2 Select Example In TSM/SP

July 27, 2018, 2:23 pm

≫ Next: Spectrum Protect Server 8.1.x Paging Space Issue

≪ Previous: Personal Back Ups!

While trying to create a specific report that a manager requested I realized I needed something more powerful. Well now that TSM/Spectrum Protect runs on DB2 we have all the power we need. Here is an example of generating a report with an Inner and Outer Join.

====================================
Inner and Outer Join Example
====================================

Part 1 - Will return the columns from two tables into a third temporary table

Select -
varchar(o.node_name,45) ENTITY, -
cast(s.SUM_AFFECTED as dec(12,0)) AFFECTED,
cast(s.SUM_BYTES/1073741824 as decimal(12,2)) GB_BACKED_UP, -
cast(o.SUM_NUM_FILES as dec(15,0)) TOTAL_FILES_STORED, -
cast(o.SUM_LOGICAL_MB/1024 as dec(12,2)) GB_TSM_OCCUPANCY -
from -

Part 2 - Gather two columns from occupancy

select node_name, -
sum(NUM_FILES) SUM_NUM_FILES, -
sum(LOGICAL_MB) SUM_LOGICAL_MB -
from occupancy -
group by node_name

Part 3 - Gather three columns from summary for BACKUP and ARCHIVES for last 30 days

select varchar(entity,45) ENTITY, -
sum(bytes/1073741824) SUM_BYTES, -
sum(affected) SUM_AFFECTED -
from summary -
where -
end_time>=(current_timestamp - 30 days) -
and activity in ('BACKUP','ARCHIVE') -
group by entity

select -
varchar(o.node_name,45) ENTITY, -
cast(s.SUM_AFFECTED as dec(12,0)) AFFECTED, -
cast(s.SUM_BYTES as decimal(12,2)) GB_BACKED_UP, -
cast(o.SUM_NUM_FILES as dec(15,0)) TOTAL_FILES_STORED, -
cast(o.SUM_LOGICAL_MB/1024 as dec(12,2)) GB_TSM_OCCUPANCY -
from -
(select node_name, -
sum(NUM_FILES) SUM_NUM_FILES, -
sum(LOGICAL_MB) SUM_LOGICAL_MB -
from occupancy -
group by node_name) o, -
(select entity, -
sum(bytes/1073741824) SUM_BYTES, -
sum(affected) SUM_AFFECTED -
from summary -
where -
end_time>=(current_timestamp - 30 days) -
and activity in ('BACKUP','ARCHIVE') -
group by entity) s -
where ENTITY in (select node_name from nodes where domain_name<>'MN-STD-POL-DOM') -
and o.node_name=s.entity -
order by ENTITY

The generated report shows the total amount of files backed up, GB backed up for the last 30 days while also showing the total files stored in TSM and the nodes current total occupancy in GB.

ENTITY AFFECTED GB_BACKED_UP TOTAL_FILES_STORED GB_TSM_OCCUPANCY
---------- --------------- --------------- ------------------- -----------------
DORPORA05 920 47513.00 8354 73263.60
IXPFLDNAS 60792 27.00 15443947 9434.27
ILDNASJ 12680 5.00 1972680 715.60
ILDNASK 8 0.00 83004 197.48
ILDNASL 10739 4.00 1649434 574.51
ILDNASM 45448 25.00 1221367 1722.03
ILDNASM 4386 7.00 1122826 557.64
ILDNASN 5 0.00 99769 46.52
ILDNASO 1688 0.00 459683 138.16
ILDNASP 9369 11.00 1402332 1623.92
ILDNASR 28787 29.00 462479 457.15
ILDNASS 5 0.00 36771 17.51
ILDNAST 6329 21.00 982728 1038.08
ILDNASU 6848 19.00 1548369 671.68
ILDNASV 18197 48.00 999905 1333.92
ILDNASW 5 0.00 245366 116.89
RSD1W030S 59979 7.00 5207060 547.23
RSD1W031S 3444 2.00 5070450 473.15
RSD1W241S 6879 47.00 517358 1782.92
RSD1W242S 6478 51.00 995006 2109.45
RSD1W266S 4377 7.00 492810 184.69
RSD1W267S 5700 3.00 328391 100.95
RSD1W320C 40881 12.00 2692541 576.50
RSD1W341C 8194 55.00 456567 1120.11
RSD1W342C 17003 7.00 625487 281.84
RSD1W343C 4628 2.00 298699 79.63
RSD1W504SV 1941 12.00 42444 551.35
RSD1W505SV 1079 105.00 38524 4033.06
RSD1W5067S 683 64.00 22389 1343.82
RSD1W506SV 489 0.00 14349 4.53
RSD1W508SV 2662 31.00 79340 699.12
RSD1W509SV 4017 74.00 139073 1617.35
RSD1W510SV 698 544.00 18086 9161.23
RSD1W511SV 1365 295.00 65303 6588.06

↧

Spectrum Protect Server 8.1.x Paging Space Issue

August 6, 2018, 6:01 am

≫ Next: Win2K & Win2003 - Data Currently Unavailable On Server

≪ Previous: DB2 Select Example In TSM/SP

I thought I would pass this along...I have two Spectrum Protect 8.1.4 instances on the same AIX 7 LPAR that since their upgrade to this version has had frequent exhausting of paging space crashes. We reviewed and checked processes and figured it was an AIX issue. Upon deeper investigation I noticed there were some previous APARs for DB2 exhausting paging space due to a 64k block size. Investigating that further led me to the following APAR for Spectrum Protect

IT23731: SPECTRUM PROTECT SERVER ON AIX MAY RUN INTO PAGING AND OUT OF MEMORY CONDITION

So to check I ran the following shell command to list the top 10 processes using memory/paging

# svmon -Pt 10 -O summary=basic,unit=MB

Unit: MB

----------------------------------------------------------------

Pid Command Inuse Pin Pgsp Virtual

4391072 db2sysc 74805.10 51.4 15266.27 72775.38

7274744 db2sysc 71323.35 50.2 15579.61 71796.98

10486106 db2vend 71164.58 47.6 15084.41 71589.28

4915692 dsmserv 15230.22 49.5 6629.03 20345.98

2490670 dsmserv 1931.09 49.6 299.12 2197.74

5701694 db2sysc 632.43 47.6 42.1 732.73

8126714 db2sysc 569.93 47.6 29.1 665.60

6553630 db2vend 561.76 47.6 37.0 660.47

6422602 db2sysc 559.20 47.6 37.2 657.31

4849682 db2sysc 559.20 47.6 37.2 657.31

So the db2sysc and dsm2vend along with the dsmserv are using up large amounts of paging.

Here is my using in topas

PAGING SPACE
Size,MB 64512
% Used 60
% Free 40

So what's the solution? Restart the TSM instance(s) until you can upgrade to 8.1.5

↧

Win2K & Win2003 - Data Currently Unavailable On Server

August 29, 2018, 12:26 pm

≫ Next: Active Data Report

≪ Previous: Spectrum Protect Server 8.1.x Paging Space Issue

This past weekend we performed a DR test on quit a few servers and discovered a situation that was a wake up call for anyone still using Windows 2000 or Windows 2003. The issue is that when we ran a restore the IBM Spectrum Protect 8.1 server issued a warning on all the files for the particular node with the following message type…

ANE4035W (Session: 79, Node: ODC-WINCS-1-DR) File '\\od-wincs-1\e$\834\c-oh\files-20170823-123000' currently unavailable on server.

The tape was in a readonly state and accessible by the library so it was not a tape unavailable problem. I then querried the file and ran some tests suggested by IBM in the following link

http://www-01.ibm.com/support/docview.wss?uid=swg21249032

The data is present and on tape but TSM will not restore it. (See the queries below) IBM would not help us because the OS is unsupported as is the TSM client. We could not restore the System State, System Services, or any of the Data to the DR replica. What we could do is access the data from a supported client level on a supported OS level (i.e. TSM Client 6.4 from a Windows 2008 server). We were unable to recover the Win2K/Win2003 servers during the DR test and unfortunately I know many companies out there still have some of these servers in their environment. It's time to upgrade! Your data is at risk!

tsm: OMATSM02>select * from backups where node_name='ODC-WINCS-1-DR' and FILESPACE_ID=1 and hl_name=upper('\834\c-oh\') and LL_NAME=upper('files-20170823-123000')

NODE_NAME: ODC-WINCS-1-DR
FILESPACE_NAME: \\od-wincs-1\e$
FILESPACE_ID: 1
STATE: ACTIVE_VERSION
TYPE: DIR
HL_NAME: \834\C-OH\
LL_NAME: FILES-20170823-123000
OBJECT_ID: 11926208445
BACKUP_DATE: 2018-08-10 03:04:37.000000
DEACTIVATE_DATE:
OWNER:
CLASS_NAME: WIN-DR-MC
ACTUAL_SIZE:

tsm: OMATSM02>show bfo 11926208445

Bitfile Object: 11926208445
Active
**Sub-bitfile 11926208445 is stored in the following aggregate(s)
Super-bitfile: 11926199674, Offset: 26157087, Length 735, Deduped: F

tsm: OMATSM02>show bfo 11926199674

Bitfile Object: 11926199674
**Super-bitfile 11926199674 contains following aggregated bitfiles,
Bitfile Id, offset, length, active state or owner, link bfid
11926199674 0 8732 Active
11926199676 8732 1527 Active
.
.
.
11926208946 26573407 5510 Active
11926208947 26578917 49168 Active

**Sub-bitfile 11926199674 is stored in the following aggregate(s)
Super-bitfile: 11926199674, Offset: 0, Length 8732, Deduped: F

**Archival Bitfile Entry
Bitfile Type: PRIMARY Storage Format: 22
Bitfile Size: 26631381 Number of Segments: 1, flags: 0
Storage Pool ID: 33 Volume ID: 22903 Volume Name: L42803

tsm: OMATSM02>q vol L42803 f=d

Volume Name: L42803
Storage Pool Name: WIN-DR-TAPE
Device Class Name: IBM-LTO-4
Estimated Capacity: 4.2 T
Scaled Capacity Applied:
Pct Util: 100.0
Volume Status: Full
Access: Read-Only
Pct. Reclaimable Space: 0.0
Scratch Volume?: Yes
In Error State?: No
Number of Writable Sides: 1
Number of Times Mounted: 227
Write Pass Number: 1
Approx. Date Last Written: 08/12/18 16:44:25
Approx. Date Last Read: 08/25/18 22:06:34
Date Became Pending:
Number of Write Errors: 0
Number of Read Errors: 0
Volume Location:
Volume is MVS Lanfree Capable : No
Last Update by (administrator): CSMALL
Last Update Date/Time: 08/25/18 20:29:37
Begin Reclaim Period:
End Reclaim Period:
Drive Encryption Key Manager: IBM Spectrum Protect
Logical Block Protected: No

tsm: OMATSM02>q libvol dr-libr L42803

Library Name Volume Name Status Owner Last Use Home Device
Element Type
------------ ----------- ---------------- ---------- --------- ------- ------
DR-LIBR L42803 Private Data 1,089

↧

Active Data Report

December 13, 2018, 6:22 am

≫ Next: Q STATUS Question

≪ Previous: Win2K & Win2003 - Data Currently Unavailable On Server

So me an a coworker have been hounding IBM for a way to generate an active data report for management. For various reasons they want to be able to see how much data a server would require for rebuild/restore and also to guage any growth. As many of you might be aware this has been something the TSM/Spectrum Protect community has been requesting for some time. So IBM created a perl script that when analyzed issues the following SQL select:

(I've added continuation dashes to make the select more readable)

select -
(sum(bk.bfsize )/1048576) as front_end_size_mega_byte, -
count(bk.bfsize ) as number_of_objects -
from -
backups b, backup_objects bk -
where -
b.state='ACTIVE_VERSION' -
and -
b.object_id=bk.objid -
and -
b.filespace_id in -
( select f.filespace_id -
from filespaces f -
where -
b.node_name=f.node_name -
and -
f.filespace_id=b.filespace_id -
and -
f.filespace_type not like 'API:%' -
and -
f.filespace_type not like 'TDP%' -
) -
and -
b.node_name in -
( select node_name -
from nodes -
where repl_mode not in('RECEIVE','SYNCRECEIVE') -
)

You will notice, however, that the select excludes node replicas and any API or TDP data. This is purely for file system backups. Supposedly they have something for TDP/API backups but I have not worked with it yet. I will post it as soon as I have a chance to review it and make it readable.

NOTE: This command can take a somewhat considerable amount of time to run. I have seen it take upwards of 10+ minutes to complete when run during our non-backup window times. Be patient!

↧

Q STATUS Question

January 14, 2019, 8:26 am

≫ Next: Q STATUS UPDATE!

≪ Previous: Active Data Report

The company to which I am employed was recently acquired by another company that is a heavy Netbackup user. They bill not based on overall occupancy but on the "Front End Usage" of their servers. I don't know how that works for accurate billing but it's what they do and now we are being asked to provide "Front End Capacity" numbers so they can compare. The problem is ADSM/TSM/SP has always had issues with reporting "active only" data. Just recently I was working with an select IBM provided to pull the active for file systems but they also stated we had to go to the individual application servers to pull DB data. Not fun. So when I recently looked at all the detail of the Q STATUS I noticed the following near the bottom.

SUR Occupancy (TB): 703

SUR Occupancy Date/Time: 2019-01-07, 10:23:18

Front-End Capacity (MB): 148.21

Front-End Client Count: 408

Front-End Capacity Date: 2019-01-07, 10:23:18

Product Offering: IBM Spectrum Protect

According to IBM these last few statements are:

SUR Occupancy (TB)

If you have an IBM Spectrum Protect Suite (SUR) license, this field specifies the SUR occupancy on the server. The SUR occupancy is the amount of space that is used to store data that is managed by the IBM Spectrum Protect products that are included in the SUR bundle.

SUR Occupancy Date/Time

Specifies the date and time when SUR occupancy data was last collected.

Front-End Capacity (MB)

Specifies the amount of primary data that is reported as being backed up by clients. Clients include applications, virtual machines, and systems. This value is used for the front-end licensing model.

Front-End Client Count

Specifies the number of clients that reported capacity usage based on the front-end licensing model.

Front-End Capacity Date

Specifies the date and time when front-end capacity data was last collected.

Product Offering

Specifies a product offering.

So what exactly is the SUR license and does the EE license cover that? How is the data generated because my server reports front-end data at 148MB but if it is actually TB then that would fit more closely with overall capacity as front-end would be about 21% of overall occupancy. IBM obviously has some function to gather front-end numbers so why do they also have us going down the rabbit hold of gathering the data application data from the clients?

↧

Q STATUS UPDATE!

January 16, 2019, 1:57 pm

≫ Next: IBM Spectrum Protect 8.1.6 Tier to Cloud by State

≪ Previous: Q STATUS Question

So it turns out that TSM is calculating Front-End usage automatically when you install Ops Center or have SET STATUSMONITOR ON. I had installed Operations Center when I installed the latest SP and wasn't familiar witht he front-end licensing model until now. So lucky me it show's the data our management wants without me having to go to each application server to manually gather the data. Note that it is supposed to show the total in MB but from the command line it appears as if SP didn't calculate correctly. If you go into Ops Center and check licensing then it shows that the

Front-End Capacity (MB): 148.21

According to the Front-End licensing model this instance ACTIVE DATA is acutally 148TB. If you you want to know the overall ACTIVE DATA the SP instance is protecting then look at the Front-End numbers. When you do a Q STATUS, if it's not reporting in TB than it probably is expriencing the bug that needs correcting by IBM, but that value is probably correct but in TB not MB. Ops Center will also tell you how many nodes are not reporting and you can select that and view the list to see if any should be reporting.

↧

IBM Spectrum Protect 8.1.6 Tier to Cloud by State

February 20, 2019, 5:46 am

≫ Next: Error Upgrading to 8.1.8.0

≪ Previous: Q STATUS UPDATE!

A coworker recently did a presentation on Amazon S3 with Netbackup and I went searching for the capabilities available with TSM/Spectrum Protect. It appears as of 8.1.6 they have added some additional features that greatly expand the usage of cloud based storage. Check out this informative demo from IBM.

↧

Error Upgrading to 8.1.8.0

August 14, 2019, 10:40 am

≫ Next: End of the Road?

≪ Previous: IBM Spectrum Protect 8.1.6 Tier to Cloud by State

I performed an upgrade today on a server going from 8.1.0.0 to 8.1.8.0 and after the upgrade we encountered the following errors in the actlog as the system was starting.

08/14/2019 11:59:59 ANR9999D_2835836899 InstGetDB2stmtPass1(instr.c:550) Thread<404>: Error in fetching DB2 statement stats, rc=235908/14/2019 11:59:59 ANR9999D Thread<404> issued message 9999 from:08/14/2019 11:59:59 ANR9999D Thread<404> 0x00000001000365b4 StdPutText08/14/2019 11:59:59 ANR9999D Thread<404> 0x000000010003784c OutDiagToCons08/14/2019 11:59:59 ANR9999D Thread<404> 0x000000010000cf08 outDiagfExt08/14/2019 11:59:59 ANR9999D Thread<404> 0x000000010019d0d0 IPRA.$InstGetDB2stmtPass108/14/2019 11:59:59 ANR9999D Thread<404> 0x00000001001a15e0 instrumentStart08/14/2019 11:59:59 ANR9999D Thread<404> 0x00000001012d7b98 AdmInstrumentBegin08/14/2019 11:59:59 ANR9999D Thread<404> 0x00000001008da68c AdmCommandLocal08/14/2019 11:59:59 ANR9999D Thread<404> 0x00000001008d7af0 admCommand08/14/2019 11:59:59 ANR9999D Thread<404> 0x00000001013fbe68 RunCommand08/14/2019 11:59:59 ANR9999D Thread<404> 0x00000001013f7a48 ServerMonTwentyMinThread08/14/2019 11:59:59 ANR9999D Thread<404> 0x0000000100010648 StartThread

Although the server started, I wanted to make sure this would not cause issues with the instance. I searched the error ANR9999D_2835836899 and found the following IBM APAR notice. It appears that after the upgrade you may have to manually upgrade the DB2 version using the DB2 update tool. While it appears to be a minor issue, I would recommend you check your instances after upgrade to verify whether you need to run the manual steps to resolve the errors. I've included them here for your convenience.

Local fix
Even though the db2-level might be updated to the latest version
11.1.4.4, run the below set of commands

Stop TSM server and issue the following commands as the instance owner.
db2 force application all
db2 terminate
login as root user
run /opt/tivoli/tsm/db2/instance/db2iupdt -d <instance name>
login as the instance owner ID
issue the command: db2updv111 -d TSMDB1

This is supposedly fixed in version 8.1.9.0 and higher.

↧

End of the Road?

April 21, 2020, 9:06 am

≫ Next: HBA Info Script

≪ Previous: Error Upgrading to 8.1.8.0

For those of you who come here periodically and are wondering why content has been lacking, I wanted to let you all know that it is due to my current employment situation. Currently my employer is in the process of replacing Spectrum Protect throughout the company with Commvault. Once the conversion has completed I will be released and so am currently looking for new employment. Since the 8.1.8 upgrade of our servers I have been working steadily on reporting and prepping for the conversion. The initial thought was that I would stay employed and transition to learning Commvault, but that turned out to not be the case. I have enjoyed managing ADSM/TSM/Spectrum Protect environments for the last 22 years but have seen a drop off of its use as IBM has retreated from marketing their products and moved more towards other areas of the IT field. As I have looked for jobs in the Spectrum Protect realm I have noticed that they are rarer than ever, especially with the current situation the world finds itself. I plan to leave the site and up and will post anything you the readers feel will benefit the Spectrum Protect community. Feel free to send me your article for review and if I feel it fits the nature of this website it will be posted. Unfortunately, unless things change I see myself transitioning into a new role with a cloud emphasis and posts here will end unless you all take over the mantle and provide other admins with advice and examples from your own experiences. I have enjoyed posting and helping the community since founding this website and wish you all success going forward.

↧