Static Routing in Red Hat

Adding a route in Red Hat

It’s a wee bit different from AIX, where you can just add the route with:

# route add so.ur.ce.ip de.st.ip.ad

Under Red Hat you need to define static routing using route command. The configuration is stored under /etc/sysconfig/network-scripts/route-eth0 for eth0 interface.

Update route using route command

Type the following command:
# route add -net 10.0.0.0 netmask 255.0.0.0 gw 10.8.2.65 eth0
# route -n

 

 

Create static routing file

The drawback of abive ‘route’ command is that, when RHEL reboots it will forget static routes. So store them in configuration file:
echo '10.0.0.0/8 via 10.8.2.65' >> /etc/sysconfig/network-scripts/route-eth0
Restart networking:
# service network restart
Verify new changes:
# route -n
# ping 10.8.2.65
# ping 10.8.2.10
# ping google.com
# traceroute google.com
# traceroute 10.8.2.10

Further readings:

  • man pages ip, route command

Tracing AIX Disks To VIO Server Mappings

This post is a list of the steps taken to map virtual SCSI disks back to the VIO servers on an AIX system. I had to carry out this process recently and decided to write up the steps for future reference.  Hopefully this can be of some help.

Requirements:

All scripts are run from NIM server and with keys exchanged between lpars, VIO servers and HMCs. SAN disks are allocated to VIO servers and are mapped to LPARs from VIOS.

http://www.ibm.com/developerworks/aix/library/au-aix-vioserver-v2/vir_scsi.gif

I recently had to document the mappings for a number of LPARs, documenting which disks led to which SAN LUNs on the respective VIO servers and found there was still no easy way to do this from the LPAR itself, so had to come up with a process to do so.

Step 1 – gather information from LPAR

First step is to gather info from AIX server which we would like to map back to the VIO. this can be done in a number of ways, but we will use lsdev -Cc disk to check for Virtual SCSI disks, then the lspath command to trace slot numbers back to the VIO. Once we determine that hdisk0 is served by our vio server, we could just use lscfg -l hdisk0 to find the local slot number, however this is a dual vio environment and it is necessary to check that the slot numbers are the same on each vio server. Hence we will use lspath -F “name,status,parent,connection” -l hdisk0 to give full mapping info to both VIO servers.

# lspath -F "name,status,parent,connection" -l hdisk0
hdisk0,Enabled,vscsi0,810000000000
hdisk0,Enabled,vscsi1,810000000000

From this output, we can see that there are two paths to this disk, through vscsi0 and vscsi1. Each vscsi device maps to a vhost device on a VIO server. As this is a dual VIO server configuration, we can assume that each path is a vhost device on a separate VIO server. Also, each path has the same LUN number of “810000000000″, suggesting that both VIO server are in sync and have the same vhost mapping. As each vhost / vscsi device can map multiple LUNs, each client LUN has an id beginning at “810000000000″, then “820000000000″, 83xxx and so on.

So, now we need the  client slot number for each device. Again there are multiple ways of doing this, including lscfg -l vscsix, however it’s neater to use lsslot -c slot to find everything we need:

# lsslot -c slot
# Slot                    Description       Device(s)
...
U8205.E6C.0123ABC-V12-C4  Virtual I/O Slot  vscsi0
U8205.E6C.0123ABC-V12-C5  Virtual I/O Slot  vscsi1

From here, we can see that the LPAR ID is represented by V12 and the client slots are C4 and C5 repectively. Cool, now we need the info from the HMC to get the server slots. Unfortunately, this is a bit of a faff and we need to log into the HMC to trace this back. I will provide a script to automate this process, however it’s important that we understand the steps involved before we can do so.

This can be done from the GUI, but for speed, we will go to the command line. So, from the HMC command line you need to type the following:

-> lssyscfg -r sys -F "name" | grep 0123ABC 
U8205-E6C-0123ABC-> lshwres -r virtualio --rsubtype scsi -m U8205-E6C-0123ABC --filter lpar_ids=12 -F "slot_num,remote_slot_num,remote_lpar_name"
5,24,vio33b
4,24,vio33a

See here for more on command line basics.

However, this is easier done from the NIM server if you have your ssh keys set up. See here, for more info on setting or resetting your ssh keys on NIM.

nim01 # ssh -q hscroot@hmc1 lssyscfg -r sys -F "name" | grep 0123ABC
U8205-E6C-0123ABC

nim01 # ssh -q hscroot@hmc1 lshwres -r virtualio --rsubtype scsi -m U8205-E6C-0123ABC --filter lpar_ids=12 -F "slot_num,remote_slot_num,remote_lpar_name"
5,24,vio33b
4,24,vio33a

Obviously replace hscroot@hmc1 with your username@hmcname and -m <systemname> with your system name.

Now to gather the info from the VIO server. You can do this by logging into the VIOS as padmin and typing lsmap -all |more and search through till you find your slot number from before (24 in our case).

SVSA            Physloc                                      Client Partition ID
--------------- -------------------------------------------- ------------------
vhost14         U8205.E6C.0123ABC-V1-C24                     0x0000000c

VTD                   vtscsi01
Status                Available
LUN                   0x8100000000000000
Backing device        hdiskpower01
Physloc               U78AA.001.XDHJSJD-P1-C2-T1-L73
Mirrored              false

VTD                   vtscsi01
Status                Available
LUN                   0x8200000000000000
Backing device        hdiskpower02
Physloc               U78AA.001.XDHJSJD-P1-C2-T1-L73
Mirrored              false

From here we can see that the vhost device at slot 24 (vhost14) is holding a number of disks. We can identify the disk we looked up from earlier by refering to the LUN ID of 810000000000. We know this is hdisk0 from the LPAR information.

Now see that the backing device for this is hdiskpower01. This is because I’m using EMCPower software for my MPIO. This value may read hdiskX in your case. If so, you won’t need to follow the steps required for mapping the EMCPower disks.

So, from this information, we can run an lspv on the VIO server to find the physical disk

vio33a # lspv | grep hdiskpower1
hdiskpower1          00f7e02exxxx1111                    rootvg          active

Of course if the pvid was set up correctly (by installing with: chdev -l hdiskpower01 -a pvid=yes), then we could possibly just take the pvid of each disk from the LPAR and grep for that same pvid on the VIO server.

lpar1 # lspv | grep hdisk0
hdisk0          00f7e02exxxx1111                    rootvg          active
padmin@vio33a $ lspv |grep 00f7e02exxxx1111                    
hdiskpower1     00f7e02exxxx1111                    None

However, this is not always reliable if these steps were not followed correctly during setup, so we will continue with this, more robust method.

So now we have the hdiskpower device on the VIO server. If we want to find the actual SAN LUNs which were mapped via EMCPower Path, we need to execute the following command:

root@vio33a # powermt display dev=all |grep -p hdiskpower1$
Pseudo name=hdiskpower1
Symmetrix ID=000123456789
Logical device ID=1ABC
state=alive; policy=SymmOpt; priority=0; queued-IOs=0;
==============================================================================
--------------- Host ---------------   - Stor -   -- I/O Path --  -- Stats ---
###  HW Path               I/O Paths    Interf.   Mode    State   Q-IOs Errors
==============================================================================
   0 fscsi0                   hdisk3    FB  8fA   active  alive       0      0
   1 fscsi1                   hdisk7    FB  9fB   active  alive       0      0

So now we finsally have a mapping of the LUN ID, 1ABC and the low level disks used on the VIO server from the SAN software. Now we need to do that for the second VIO server and alll the other disks, while we’re at it!

Pain the neck?

Indeed, so we need a way of automating this process. I’ve adapted a small perl script to suite my needs. As previously stated, all command are run from my NIM server, which has passwordless ssh access to all lpars and the HMCs. From here, we can gather all the information we want and blat it out to a file.

Handy Sed One Liners

Remove all whitespaces from the beginning of a line:
:%s/^\s\+

Delete all trailing whitespace (at the end of each line) with:

:%s/\s\+$//

Tidy up ugly HTML files to be spaced like a normal person:

vim fileIn.html -c "set sw=2 | %s/>/>\r/ | execute 'normal gg=G' | set nohlsearch | g/^\\s*\$/d"

Set up SSH Tunnel

If you don’t have firewall access to a specific server on a subnet, you can use ssh tunnelling to forward requests from an existing host.

Find a server you do have access to and create the ssh tunnel as below, where destinationhost is the server you ultimately want to access and jumphost is the server you have access to:

destinationhost# ssh -L22033:localhost:22033 -R22033:10.220.112.173:670 -N jumphost

jumphost# ssh -p22033 userid@localhost

You may need to add a hostfile entry for jumphost

Creating a Subsystem in AIX

Want a deamon to be controlled by the system resource controller?

Easy:

Create fksubsys service:

mkssys -p /path/to/fkcmd \ # Absolute path to subsystem bin or script
-a “arg1 arg2″ \     # if Arguments to the command
-s fksubsys \        # Unique name for the subsystem (-t fkalias)
-G fkgrp \           # Subsys group
-u 202 \             # User id for the subsystem (0 for root)
-i /dev/console \    # Stdin got from (default)
-o /dev/console \    # Stdout sent to (default)
-e /dev/console \    # Stderr sent to (default)
-R \                 # Restart subsys if halted abnormally (or -O)
-Q \                 # One instance only # -q if multiple instances
-S \                 # Subsys supports signals
\                   # or -I msgqueue -m msgtype (or -K if sockets)
-f 9 \               # signal KILL for forced stopsrc -f
-n 15 \              # signal TERM for normal stopsrc
-w 10 \              # 10 seconds between a TERM and KILL & restart)
-E 24 \              # Subsys priority (20 default)
-d                   # lssrc -a lists subsys even if inactive else -D

lssrc -S -s fksubsys # Check the service’s configuration: OR…
odmget -q subsysname=fksubsys SRCsubsys # … via the odm.

startsrc -s fksubsys  # startsrc -g fkgrp  # two ways to launch it
stopsrc -s fksubsys # and to stop it

mkitab -i cron fkcmd:2:respawn:startsrc -s fksubsys # start subsys via /etc/inittab

DB2 for UNIX Admins

Want to stop a DB2 DB and there’s no DBAs on hand?

Here’s all you need:

# su – [db2instanceowner]

Try:

# db2stop

If that fails with:

 

31/08/2013 21:05:30 0 0 SQL1025N The database manager was not stopped because databases are still active.

SQL1025N The database manager was not stopped because databases are still active.

Then try:

# db2 list applications show detail

If that shows connections, run:

# ipclean

Application ipclean: Removing DB2 engine and client IPC resources for dbxxxx

 

# ipcs | grep dbxxx| awk ‘{print ” ipcrm -”$1″ “$2 }’ > ipcln

# cat ipcln

ipcrm -q 1055916063

ipcrm -q 904921121

ipcrm -m 6291458

# chmod +x ipcln

# ipcln

ipcrm: 0515-020 shmid(6291458) was not found.

# db2stop

# ps -ef |grep db2s

If that returns nothing, the database is down!

Now reboot the box and run db2start if necessary.

TC

Here are some more basic commands you might need:

  • Size of your database:
    • =>db2 ‘call get_dbsize_info(?,?,?,0)’
  • Version of DB2 you are running:
    • =>db2level
  • Stop the db2 instance
    • =>db2stop
  • Stop an instance that has current connections
    • =>db2 force applications all
    • =>db2 deactivate db <dbname>
    • =>db2 terminate
    • =>db2stop force
    • =>ipclean
  • Start the db2 instance
    • =>db2start
  • Kill a hung instance (last resort)
    • =>db2_kill -all
  • List the databases in an instance
    • =>db2 list db directory
  • List the cataloged nodes
    • =>db2 list node directory
  • Show the database manager configuration settings
    • =>db2 get dbm cfg
  • Show the database level configuration settings
    • =>db2 get db cfg for <dbname>
  • Activate a database
    • =>db2 activate db <dbname>
  • Deactivate a database
    • =>db2 deactivate db <dbname>
  • View the DB2 License
    • =>db2licm -l
  • To switch between partitions
    • =>db2 terminate; export DB2NODE=<new-node-num>

     

More:

 

DB2 System Commands
  • DB2LEVEL — checks version of DB2 installed.
  • DB2ILIST — lists all instances installed
  • DB2CMD — opens a command line processor
  • DB2CC — opens db2 control center
  • DB2LICM -l — gets db2 type.
Command Line Processor Commands
  • DB2 LIST NODE DIRECTORY — Lists all nodes
  • DB2 CATALOG TCPIP NODE DB2NODE REMOTE MACHINE215 SERVER 50000 – catalogs node.  In this case, node is db2Node on the machine with name machine215. Port is 50000.
  • DB2 LIST DATABASE DIRECTORY – list databases
  • DB2 GET DB CFG FOR SAMPLE – get configuration info for the SAMPLE db.
  • DB2 CONNECT TO alexDB USER myuser USING mypass – connect to db. In this case, database is alexdb, usern is myuser and password is mypass.
  • DB2 DISCONNECT alexdb  – disconnects
  • DB2 LIST APPLICATIONS SHOW DETAIL – shows all running db’s
  • DB2 GET DBM CFG – view authentication paramater (e.g. something like server_encrypt)
  • DB2 UPDATE DBM CFG USING AUTHENTICATION SERVER_ENCRYPT – alter the authentication mechanism to server_encrypt
  • DB2 GET AUTHORIZATIONS – get authorisation level.
Database commands via Command Line Processor (CLP)
  • DB2 GET DATABASE CONFIGURATION — gets current database configuration
  • DB2 VALUES CURRENT USER – - gets the current user
  • DB2 VALUES CURRENT SCHEMA – gets the current schema
  • DB2 VALUES CURRENT QUERY OPTIMIZATION — get query optimization level.
Schemas
  • DB2 SELECT SCHEMANAME FROM SYSCAT.SCHEMATA — list all schemas
  • DB2 VALUES CURRENT SCHEMA – gets the current schema
  • DB2 SET SCHEMA ALEXSCHEMA — set schema
Tables
  • DB2 LIST TABLES FOR schema_name — list all tables for particular schema
  • DB2 LIST TABLES SHOW DETAIL; — show detail about tables
  • DECLARE GLOBAL TEMPORARY TABLE — declares a temporary table
  • CREATE TABLE MQT AS (SELECT c.cust_name, c.cust_id, a.balance FROM customer c, account a WHERE c._cust_name IN (‘Alex’) AND a.customer_id – c.cust_id) DATA INITIALLY DEFERRED REFRESH DEFERRED — Creates a materialised query table. In this case the MQT is based on a join query from the customer and account table.
Tablespaces
  • DB2 LIST TABLESPACES SHOW DETAIL – show detail about table spaces
  • SELECT * FROM SYSCAT.TABLESPACES;  – show what syscat has about tablespaces
  • SELECT tbspace, bufferpoolid from syscat.tablespaces;  – get tablespace and bufferpoolid
  • SELECT TABNAME FROM SYSCAT.TABLES WHERE TBSPACE=2; – Check what TABLES are in tablespace where id = 2.
 
Constraints
  • SELECT * FROM SYSCAT.TABCONST;  – Table constraints
  • SELECT * FROM SYSCAT.CHECKS;  – Colum checks
  • SELECT * FROM SYSCAT.COLCHECKS; — Column constraints
  • SELECT * FROM SYSCAT.REFERENCES; —  Referential constraints
Sequences
  • CREATE SEQUENCE STESTRESULT AS INTEGER INCREMENT BY 1 START WITH 1 NO MINVALUE NO MAXVALUE NO CYCLE CACHE 10 ORDER;  – Create Sequence starting with 1 which cache 10 values
  • SELECT * FROM SYSCAT.SEQUENCES; — Gets systcat info on sequences
  • VALUES NEXT VALUE FOR MYSEQ; — Gets next value from sequence myseq
  • ALTER SEQUENCE MYSEQ RESTART WITH 11 INCREMENT BY 1 MAXVALUE 10000 CYCLE CACHE 12 ORDER — Changes MySeq sequence
 
Locksize
  • SELECT TABNAME, LOCKSIZE FROM SYSCAT.TABLES WHERE TABNAME = ‘ EMPLOYEES’;  – Check locksize which can be tablespace, table, partition, page, row – (usually row).
Bufferpools
  • SELECT bpname, npages, pagesize from syscat.bufferpools — get useful buffer pool info.
  • SELECT buffer.bufferpoolid, buffer.bpname, buffer.npages, buffer.pagesize, tablespace.tbspace, tablespace.tbspaceid from syscat.bufferpools buffer, syscat.tablespaces tablespace where tablespace.bufferpoolid = buffer.bufferpoolid;  – gets buffer pool and corresponding tablespace info.
Indexes
  • SELECT * FROM SYSCAT.INDEXES —  show all indexes
  • SELECT COLNAMES, TABNAME, INDEXTYPE, CLUSTERRATIO, CLUSTERFACTOR FROM SYSCAT.INDEXES WHERE TABNAME = ‘TPERSON’;  – some useful columns
Functions
  • SELECT * FROM SYSCAT.FUNCTIONS;  – check what functions DB has.
SYSDUMMY1 commands
  • SELECT CURRENT DATE FROM SYSIBM.SYSDUMMY1; — gets current date.
  • SELECT HEX(36) FROM SYSIBM.SYSDUMMY1;  – same as VALUES HEX(36)
  • SELECT XMLCOMMENT (‘This is an XML comment’) FROM SYSIBM.SYSDUMMY1;
Runstats
  • RUNSTATS ON TABLE TAUSER1.TOSUSER FOR INDEXES ALL;  – runstats for all indexes

Checking the last time runstats was run…

  • SELECT CARD, STATS_TIME FROM SYSCAT.TABLES WHERE TABNAME = ‘TOSUSER’;
  • SELECT NLEAF, NLEVELS, FULLKEYCARD, STATS_TIME, TABNAME, INDNAME FROM SYSCAT.INDEXES WHERE TABNAME = ‘TOSUSER’;
The following catalog columns can be queried to determine if RUNSTATS has been performed on the tables and indexes:

  • If the CARD column of the SYSCAT.TABLES view displays a value of -1, or the STATS_TIME column displays a NULL value for a table, then the RUNSTATS utility has not run for that table.
  • If the NLEAF, NLEVELS and FULLKEYCARD columns of the SYSCAT.INDEXES view display a value of -1, or the STATS_TIME column displays a NULL value for an index, then the RUNSTATS utility has not run for that index.

 

HMC Comand Line Basics

vtmenu – get list of systems and connect via console

lssysconn -r all – list connection information for all managed systems

lssyscfg -r sys – list all managed systems

 

 
HMC command line:

HMC related commands:

monhmc -r mem -n 0                                            shows total, used and free mamory of HMC
monhmc -r disk -n 0                                           shows filesystems and usage info (same as “df -k”)
monhmc -r proc -n 0                                           shows cpu usage of each processor
monhmc -r swap -n 0                                           shows paging space usage

Managed system and LPAR related commands:

 commands have –help option and man pages

lssyscfg -r sys -F name | grep aix21                          shows the full name of the managed system (what we can use in other commands)
lssyscfg -r lpar -m <managedsysname> -F name                  shows the full name of the lpars of the managed system
lssyscfg -r sys -m <managedsysname> -F name,state –header    shows the state of the managed system
lssyscfg -r lpar -m <managedsysname> -F name,state –header   shows the state of the lpars of the managed system
lssyscfg -r prof -m aix10-SN0603C6H –filter “lpar_names=aix10″ -F name,boot_mode   it will show the boot mode in the profile
lssyscfg -r prof -m <man.sys> -F lpar_name,virtual_eth_adapters                     shows LPARS with ethernet adaptesr and VLAN tags

lshwres -r mem -m <managedsysname> –level sys                shows memory information of the managed system
lshwres -r mem -m <managedsysname> –level lpar               shows memory information of lpars of the managed system
lshwres -r proc -m <managedsysname> –level sys               shows processor information of the managed system
lshwres -r proc -m <managedsysname> –level lpar              shows processor information of lpars of the manage system
lshwres -r io -m <managedsysname> –rsubtype slot             shows IO slot information
lshwres -r proc -m <managedsysname> –level sys -F installed_sys_proc_units:configurable_sys_proc_units

lshwres -r virtualio –rsubtype scsi -m <managedsysname> –level lpar               lists scsi devices by LPAR
lshwres -r virtualio –rsubtype eth -m <managedsysname> –level lpar                lists virt. eth. devices by LPAR
lshwres -r virtualio –rsubtype eth -m <managedsysname> –level lpar -F lpar_name,port_vlan_id

chsysstate -m <managedsysname> -o standby -r sys  
           power on a system to standby
chsysstate -r sys -m <managedsysname> -o off                  normal power off the managed system
chsysstate -r sys -m <managedsysname> -o off –immed          fast power off the managed system

chsysstate -m <managed system> -r lpar -n <lparname> -o shutdown –restart          it will reboot an lpar with dump
chsysstate -m <managed system> -r lpar -n <lparname> -o shutdown –immed –restart  it will reboot an lpar immediately (without dump)
chsysstate -m <managedsysname> -r lpar -n <lparname> -o shutdown –immed            it will shutdown the oprating system
chsysstate -m aix10-SN65158BE -o on -r lpar -n aix10 -f default                     it will activate an lpar

lspartition -dlpar                                            shows dlpar capable partitions
                                                              (it will show if RMC connection is OK between the HMC and LPAR)

lssysconn -r all                                              to see what IPs are assigned by the HMC
mksysconn -o auto                                             tells HMC to re-discover all servers’ IP address

lssvcevents -t console -d 60                                  lists console events in the past 60 days
lssvcevents -t hardware -d 0                                  list serviceable events which occured today
lssvcevents -t console -d 300 | grep DLPAR                    list DLPAR operations of last 300 days (you can grep more to mem or proc)
lssvcevents -t console -d 7 | grep Migration                  list last 7 days LPMs

lshmc -V                                                      hmc version
lshmc -v                                                      hmc model
lshmc -n                                                      lists the network settings of the hmc
hmcshutdown -t now -r                                         reboot the HMC (-t: timing in minutes, -r: reboot)

vtmenu                                                        hmc menu options (console session can be opened as well)
logout: ~~. (2 tilde and a dot)

lslparmigr -r sys -m <managed system> | sed “s/,/\n/g”        shows how many concurrent migr. is possible

ssh hscroot@hmcXX ‘for i in `lssyscfg -r sys -F name`;do echo $i;lssyscfg -r lpar -m $i -F name|grep <LPARNAME>;echo;done’ <–shows man. system of given LPAR

————————————

Default root password: passw0rd (try su -)

————————————

opening/closing virtual terminal from hmc:

lssyscfg -r sys -F name                         <–get managed system name
lssyscfg -r lpar -m <managed system> -F name    <–get lpar name

mkvterm -m <managed system> -p <lpar>           <–opens a terminal window
rmvterm -m <managed system> -p <lpar>           <–closes a terminal window

~~.                                             <–logout from te session (not necessary to leave AIX)

————————————

Adding/removing VLAN TAG dynamically:
(it will not add to profile)

adding:
chhwres -r virtualio –rsubtype eth -m <man.sys.> -o s -p <LPAR> -s <adapter id> -a “addl_vlan_ids+=<VLAN TAG>”

removing:
chhwres -r virtualio –rsubtype eth -m <man.sys.> -o s -p <LPAR> -s <adapter id> -a “addl_vlan_ids-=<VLAN TAG>”

————————————

 

Update hscroot authorized_keys2 file on HMC

After an HMC upgrade, sometimes you will lose entries that allow password-less logins to your HMC.

If this happens, run the following commands:

Log on to HMC manually:

ssh hscroot@hmc1

mkauthkeys -a “string”

where “string” is the whole ssh key from the remote hosts, e.g. “ssh-rsa asdfasdfasdfasdfasdfasdfasdfasdfaasdfasd= root@nim01″

Should be done!