Machines have been created to do hard work for us. We also engage them in doing things we cannot do fast enough. Most users have learned to take this technology-driven world for granted. Often, we do not realize that it is still the humans who maintain the systems working behind the curtain. If you happen to use Cisco devices to provide network services, you may find the tips featured in this article helpful while maintaining devices.
Here are another five tips to automate and simplify maintenance tasks:
- * Automated configuration backup using Command Scheduler utility (Kron).
- * Using Syslog facility to store important system messages.
- * Using Remote Monitoring (RMON) alarms.
- * Using Role Based Access Control (RBAC) allowing restricted access to devices.
- * Using Menus in order to help less experienced administrators accomplish simple goals.
All of the features above offer much more than this short article can describe. For more information, please refer to Cisco IOS documentation.
Automated configuration backup using Command Scheduler utility (Kron).
In my previous article on IOS Speed Tips, I wrote about a method of archiving the configuration using the ‘archive’ command. Most network administrators will also need to create a backup configuration of their devices at certain time intervals. For instance, the company policy may stipulate that configuration of all switches and routers must be backed up on a TFTP or FTP server weekly and upon any configuration change saved in NVRAM.
Once you know how to use ‘archive’ command, you can augment this solution making an extra copy of the configuration and store it on an external server. In order to illustrate how it can be automated, I am going to use the following policy on my switch named SW1:
– Backup Configuration: weekly (every 10080 minutes) or upon using ‘write memory’ command
– Destination: ftp://192.168.1.4/upload
– FTP Server Credentials:
- user: ftp,
- password: cisco
Since I use FTP server, I will need to provide user’s name and password to upload my configuration onto it.
Step 1 – Configure FTP credentials
SW1(config)#ip ftp username ftp
SW1(config)#ip ftp password cisco
Of course, you must make sure that your FTP credentials (user name and password), as well as the permissions to upload the files are configured properly on the server.
The next thing I am going to do is to configure the archive, but unlike using flash as the destination for our configurations, this time I am going to specify FTP server as a storage place. This means that every time an administrator issues ‘write memory’ command, the running-configuration will be sent out to our FTP server. Also, we are going to add an extra twist to this, by making sure that the file archive contains the host name and the time of creation rather than using plain ‘-1’, ‘-2’, etc. as the names of our respective configurations. In order for this to work, the time of our device must be properly set using either NTP service or a manually assigned ‘clock set’ command.
Step 2 – Configure archive for configuration backup
SW1(config-archive)# path ftp://192.168.1.4/upload/$h-$t
SW1(config-archive)# time-period 10080
Notice that in the path I have used in $h-$t parameters. If you think that they are variables that will be substituted for the actual device host name and the file creation time respectively, you are right. The command ‘write-memory’ instructs the archive utility to be triggered whenever the admin saves configuration in NVRAM (‘write memory’) and also every week as stipulated by the last option (time-period 10080 minutes = 1 week).
If you are configuring your device while reading this article, at this point you should check if the FTP server is receiving the file when you issue the ‘write memory command.
In the image above, the time shows as 22:10:42. Then, I have saved the configuration in NVRAM using ‘write memory’ command. As per our ‘archive’ configuration, the file should be stored on our FTP server using the credentials configured before (user: ftp, password: cisco).
Indeed, the file has been created on the FTP server with the host name SW1, the time stamp and ‘-1’ number identifying it as the first archived configuration.
Now, we need to make sure that a backup copy of our configuration is created on FTP server every week. We will engage IOS ‘kron’ utility to accomplish this goal.
In the step below, I want my kron to issue ‘write memory’ command which, as per our archive configuration in step 2, will also trigger configuration upload to our FTP server.
Step 3 – Configure Kron Policy-List
SW1(config)# kron policy-list CONFIG-BACKUP
SW1(config-kron-policy)# cli write-memory
In my example, I am going to start kron every Wednesday at 22:45 (you can use any time of the week here).
SW1(config)# kron occurrence BACKUP at 22:45 wed recurring
SW1(config-kron-occurrence)# policy-list CONFIG-BACKUP
Now, all we have to do is wait until kron is triggered (here: in 5 minutes and 34 seconds) as depicted in the last line of image below.
When the given time has elapsed, we can see that the new configuration file has been sent to our FTP server.
Kron has created another configuration backup on my FTP server (signified with ‘-2’ at the end of the file name).
Concluding this whole configuration, I made my kron utility run every Wednesday at 22:45. It will issue ‘write memory’ command. This will, in turn, trigger the ‘archive’ to upload the configuration to my FTP server.
More information on using kron: http://www.cisco.com/en/US/docs/ios-xml/ios/cns/configuration/xe-3s/asr1000/cns-cmd-sched.html
Using Syslog facility to store important system messages.
Syslog protocol is a transportation method allowing devices to send messages and alerts to a so-called syslog server (message collector). Those messages use UDP as the transport protocol and are directed to port 514 by default. Syslog messages typically use the following format:
Look at the brief explanation of the message fields:
- Time stamp – local time when the message was generated.
- Facility – category of the message (Cisco will describe a device, protocol or a module generating the message here and is not compliant with original syslog implementation described in RFC 3164).
- Severity – describes how serious event is.
- Mnemonic – device specific code identifying the message.
- Message-text – actual message the device is sending.
Severity of the message is very important. System uses eight levels of severity ranked with a single digit number. The lower the number, the more serious the situation has occurred on a device. Here are the severity numbers and their brief explanation:
0 – Emergency: System is unusable.
1 – Alert: Action must be taken immediately.
2 – Critical: Critical conditions.
3 – Error: Error conditions.
4 – Warning: Warning conditions.
5 – Notice: Normal but significant condition.
6 – Informational: Informational messages.
7 – Debug: Debug-level messages.
Syslog message analysis helps administrators find out as to what happened and may also speed up the process of finding what needs to be done to fix a problem. This is why collecting system messages are so important. The easiest way to set up a collector is either to use some commercial software or implement free syslog collectors such as Windows KIWI syslog server or use the one the Linux operating system comes with by default.
In case you want to use Linux, below is an example of how to set up an Ubuntu OS as a Cisco syslog collector.
Step 1 – Enable UDP (port 514) message reception
Edit (as the root user) the file /etc/rsyslog.conf and uncomment the two lines as presented below (there are no leading ‘#’ which is there by default):
# provides UDP syslog reception
Step 2 – Specify the path for Cisco logs
Create the file named cisco.conf under /etc/rsyslog.d/ (full path: /etc/rsyslog.d/cisco.conf). And in the file create the following content:
# Log from Cisco Clients
Step 3 – Enable log rotation
Create a file /etc/logrotate.d/cisco in which add the following lines:
reloadrsyslog>/dev/null 2>&1 || true
rotate 4 = keep 4 files with log history (can be changed)
weekly = rotate weekly (can be changed to daily, monthly etc.)
More detailed explanation of the syslog server configuration is beyond the scope of this article.
Step 4 – Restart the service for those changes to take effect (here Linux CLI is used).
$ sudo service rsyslog restart
Now, we need to configure Cisco devices to send all the messages to the syslog server. The configuration of my Cisco switch could look like the one shown below:
SW1(config)# logging 192.168.1.3
SW1(config)# logging facility local7
SW1(config)# logging source-interface loopback0
SW1(config)# service timestamps log datetime
A brief explanation of the configuration:
logging 192.168.1.3 = send log messages to this address
logging facility local7 = Cisco devices use this facility by default allowing to place messages into the appropriate file on the server (here: /var/log/cisco.log)
logging source-interface loopback0 = allows signed messages with the stable address (must have loopback0 interface configured with ip address to use it).
Service timestamps log datetime = nicely presented time stamp for each message.
I must mention that, by default, Cisco devices will log severity messages from 6 (informational) down to 0 (emergency). You can change it by using ‘logging trap number‘ command. The number will indicate which severity level to log (level 0 up to the number inclusive). For instance:
SW1(config)# logging trap 5
This will log messages with severity 5, 4, 3, 2, 1, and 0.
In the output above, you may notice that my Server time (09:18:24) is off by almost a minute compared to my switch (09:17:22). It is because it is manually set and I did not use NTP in this example.
Using Remote Monitoring (RMON) alarms
Router’s interfaces get congested at times, CPU can be heavily utilized, but the problem is that a network administrator does not get notified about it by default.
If you know the basics of SNMP protocol and have SNMP server which collects information from devices, it is relatively easy to make your device ‘speak’ when certain critical thresholds are reached.
Here’s a typical example. There is a router connected to 172.16.12.0/24 (Vlan 12) in my lab network. It gets congested now and then. I am the admin who wants to receive SNMP trap when that happens. A trap is an unsolicited SNMP message sent by a device when something of importance happens.
Assuming that I have an SNMP server running in my network (I can easily install free server on Linux box), I need to find out which Management Information Base (MIB) object ID collects information about packets arriving on an interface of a router. This is where Google search engine shines. A quick search for ‘cisco mib interface incoming packets‘ gives me few links. The one below seems informative enough:
It shows me that MIB object in question is called: ifInUcastPkts. I can use Google search again and look for Cisco MIB Object Navigator. This tool will verify if the object name is correct.
Now, I need to configure the router to send alarm messages. Here’ s my setup:
- – SNMP Server IP address: 10.0.1.1
- – SNMP Server Community: cisco123
- – Router’s IP address (R1, in my case): 172.16.12.1/24 (Vlan12 gateway)
As far as the scenario is concerned, let us assume that alarm should be sent to the SNMP server if the number of packets arriving on router’s interface has increased to 200 packets/min.. In such situation, an SNMP message should be sent informing an operator about the congestion of the interface. In case the number of packets drops to 50 packets/min, another message should be sent letting the admin know that there is no more congestion on the interface. Here’s how you can go about it:
Step 1 – Configure destination for SNMP messages
R1(config)# snmp-server host 10.0.1.1 cisco123
R1(config)# snmp-server ifindex persist
SNMP server IP address is 10.0.1.1, it will expect all messages signed with the community of ‘cisco123’ (community is a sort of like a password).
snmp-server ifindex persist = this ensures that the router is not going to change the MIB index of the interface upon re-boot.
Step 2 – Configure RMON events with the message descriptions.
Verify which interface will be monitored (Vlan 12 gateway).
Now configure two events with appropriate descriptions.
R1(config)# rmon event 1 log trap cisco123 description “Interface Fas0/0” owner test
R1(config)# rmon event 2 log trap cisco123 description “Interface Fas0/0 Not Congested” owner test
rmon event 1 = the reference event number that will be used by the rmon alarm (here event number = 1).
trap = unsolicited message sent by the device to SNMP server.
cisco123 = community (string cisco123) expected by the server in order to accept the message.
description = what message will be sent (message in inverted quotes).
owner test = optional command, the owner of the event is called test.
Step 3 – Create the alarm referencing the two trap messages created in step 2.
In order to create the alarm, you must know which interface it is going to be monitored. Here’s how you can check this:
The final command triggering messages:
R1(config)# rmon alarm 1 ifInUcastPkts.1 60 delta rising-threshold 200 1 falling-threshold 50 2 owner test
rmon alarm 1 = alarm number 1.
ifInUcastPkts.1 = the MIB object name I found on the internet. The .1 stands for the interface in question.
60 = number of seconds (1 min) over which the counter will measure the traffic rate.
delta = used to sample data that change frequently (others such as memory or CPU utilization should use ‘absolute‘ keyword instead.
200 = number of packets by which the alarm needs to be triggered.
1 – event no. 1 needs to be used (interface congested).
50 – the number by which the number of packets needs to fall in order to send message.
2 – event no. 2 needs to be used (interface not congested).
owner test – optional command specifying the owner of the event.
As always, let’s do few tests to see the configuration operational:
And now, let’s send some traffic (ping with high repeat count) trying to trigger one of those events. While pinging, I am going to enable ‘debug snmp packet’ to see the messages being sent by RMON alarm. After about a minute, the following shows in the debug:
I have stopped the traffic (sent 350 packets), and minute later the debug shows another event:
Using Role Based Access Control (RBAC) allowing restricted access to devices.
RBAC is a mechanism used to enhance the authorization model in IOS. It allows us to build user roles, referred to as VIEWS, with very specific privilege levels. Views can also be grouped in order to create a hierarchical structure in which one user can inherit the privileges assigned another user.
A simple example is the best way to describe the concept. Let’s create three views on the router and give each one of them a separate privilege level (what they are authorized to do).
Device: R1 (router)
Privileges: FastEthernet0/0 IP configuration
Device: R1 (router)
Privileges: FastEthernet0/0 IP configuration
Device: R2 (router)
Privileges: view Admin1 and view Admin2
Here’s our configuration to accomplish the goal:
Step 1 – Enable ‘aaa new-model’
For the next task, you must be in the privileged mode and not in the global config mode.
Step 2 – Enter the view context
*Mar 1 00:14:45.775: %PARSER-6-VIEW_SWITCH: successfully set to view ‘root’.
The ‘enable view’ command is going to ask you for the password set up for enabled mode (privilege level 15). The system is going to switch you to so-called ‘root’ view (which is the same privilege level as enabled mode).
Step 3 – Create VIEWS with the appropriate privileges
R1(config)#parser view Admin1
R1(config-view)#commands exec include configure terminal
R1(config-view)#commands configure include interface
R1(config-view)#commands configure include interface FastEthernet0/0
R1(config-view)#commands interface include all ip
‘exec’ – refers to privileged mode
‘configure’ – refers to global config mode
‘interface’ – refers to interface mode (config-if)
R1(config)#parser view Admin2
R1(config-view)#commands exec include debug ipospf events
R1(config-view)#commands exec include undebugipospf events
R1(config)#parser view Super superview
In order to verify what particular views can do, you must enter config mode using appropriate view name.
View Admin1 only has authority to configure IP on FastEthernet0/0. Accessing another interface mode (FastEthernet0/1), will show a syntax error.
Using Menus in order to help less experienced administrators accomplish simple goals.
If the administrators responsible for implementing/maintaining are slowly introduced into IOS CLI, you may give them access to a menu allowing them to issue command without knowing them initially.
Let’s create an admin account with full privileges (privilege level 15), but make sure that upon telnet/ssh connection to a device, the administrator (here we call her/him SYSOP, password is cisco123), will be presented with the menu rather than regular CLI.
Step 1 – Create username SYSOP, password cisco123 with access to the menu called SYSOP_MENU
Enter configuration commands, one per line. End with CNTL/Z.
R2(config)#username SYSOP autocommand menu SYSOP_MENU
R2(config)#username SYSOP password cisco123
R2(config)#username SYSOP privilege 15
R2(config)#line vty 0 4
Now, we need to create a menu called SYSOP_MENU (as per previous step).
Step 2 – Create Menu for SYSOP user
R2(config)#menu SYSOP_MENU text 1 Display Interfaces
R2(config)#menu SYSOP_MENU command 1 show ip interface brief
R2(config)#menu SYSOP_MENU text 2 Display Running Config
R2(config)#menu SYSOP_MENU command 2 show run
R2(config)#menu SYSOP_MENU text 3 Display Routing Table
R2(config)#menu SYSOP_MENU command 3 show ip route
R2(config)#menu SYSOP_MENU text 4 Exit
R2(config)#menu SYSOP_MENU command 4 exit
R2(config)#menu SYSOP_MENU clear-screen
It is not difficult to guess what is going to happen when SYSOP logs in to R2 via tty lines. Take a look at the image below:
Upon successful log onto the device, the operator SYSOP is presented with the menu of choices.
Now, R2 is waiting for the user’s choice. After selecting the number in the menu, the router will display the appropriate information and will clear the screen. The last option will terminate telnet/ssh session.