The management pack enables the monitoring of UPS’s components like battery, environmental monitoring card etc. and allows controlling UPS’s parameters such as input voltage, current frequency, output load, battery temperature etc.

The management pack includes availability and configuration monitoring, performance data collection, and default thresholds.

The gathered information is presented to the monitoring operators using all the visualizing abilities of the Microsoft System Center Operations Manager 2007 – e-mail notifications, diagrams, health explorer, performance graphs views etc.

In comparison with other products, the Metrex Engineering Management Pack operates solely using the functions of the Microsoft System Center Operations Manager 2007. This makes the installation and the configuration very similar to that of the standard Microsoft management packs. The management pack doesn’t require any dedicated databases, servers or processes to operate.

In most of the cases the end user will not be required to carry out any post-installation fine tuning – the threshold values are pre-configured based on long-term statistics, gathered from live power circuits.

Supported UPSs

Metrex APC Monitoring Management Pack utilizes only standard APC monitoring techniques using the SNMP and SNMP-traps. This enables the ability to monitor virtually any models UPSs by APC. Additionally, Ethernet connector in UPS is recommended but not required.

System Requirements

Microsoft System Center Operations Manager 2007 R2

• Switches
• Routers
• Firewalls
• Wireless Access Points

The management pack enables the monitoring of both hardware  components such as RAM, CPUs, Power Supplies as well as logical components such as processes and IPSLA monitors.

With the ability to monitor IPSLA, now the SCOM can monitor the quality of the links to the remote offices.

The management pack includes availability and configuration monitoring, performance data collection, and default thresholds.

The gathered information is presented to the monitoring operators using all the visualizing abilities of the Microsoft System Center Operations Manager 2007 – performance graphs views, health explorer, diagrams, e-mail notifications etc.

In order to simplify the troubleshooting, all the rules and monitors included into this management pack are linked to corresponding highly detailed articles in  the Knowledge Base.

In comparison with other products, the Metrex Engineering Management Pack operates solely using the functions of the Microsoft System Center Operations Manager 2007. This makes the installation and the configuration very similar to that of the standard Microsoft management packs. The management pack doesn’t require any dedicated databases, servers or processes to operate.

In most of the cases the end user will not be required to carry out any post-installation fine tuning – the threshold values are pre-configured based on long-term statistics, gathered from live networks.

Supported Equipment

Metrex Cisco Management Pack utilizes only standard Cisco monitoring techniques using the SNMP and Syslog. This enables the ability to monitor virtually any models of switches, routers, firewalls or wireless access points by Cisco. In very rare cases in order to get a complete readout the device need to be upgraded to the latest IOS version

System Requirements

Microsoft System Center Operations Manager 2007 R2

Warranty

Metrex Cisco Management Pack

• Successfully passed the beta-testing phase in several live networks
• Was submitted to the developers of the Microsoft System Center Operations Manager 2007 as was confirmed to be compliant
• Is implemented in several large networks

Annual Management Packs’ supporting contract costs 15% of total Management Packs’ purchase sum. This contract includes e-mail technical support and major updates for our Management Packs.

The minimum amount of purchases is $5,000.

Also our discounts for large purchase amount are still valid:

• 20% when purchase amount is greater than $100,000

Management Pack’s licensing model is “Per device”.

Let’s see the following example: your company plans to monitor about 30 switches, 4 firewalls, 10 routers and 20 UPSs with support for our Management Packs. So the monitoring solution will cost:

30*70+4*175+10*210+20*105 = $7,000

Supporting contract will cost: $7,000 * 15% = $1,050

Total sum is $8,050

In addition to the standard requirements for “SNMP Service”, “SNMP WMI provider” and  Powershell v1.0 the installer requires the hotfix 942589. This patch is responsible for fixing the issues with the File System Redirector in Windows x64

Let’s start with the description of what the File System Redirector is in the first place. This service is required for any 32-bit applications to operate normally in a 64-bin environment.

The reason for the redirector is interoperability. Most of the applications are using the Windows’s system components and libraries but in reality most of the 32 bit applications can intercommunicate correctly only with other 32 bit application. This is one of the reasons why Microsoft does include 32-bit version of the most important system components (IIS, Powershell, SNMP, etc…) into Windows x64

The redirector operates very straightforwardly. When a 32-bit application attempts to access a file in C:\Windows\System32, the OS routes the request to a sandbox witch lives at c:\windows\SysWOW64

This folder contains the 32-bit version of the system components that we’ve mentioned before. And the set of the components is amazing – IIS, Powershell and SNMP were mentioned before, msixec, mmc.exe, net.exe, cmd.exe… Even mspaint!

But what happens when a 32 bit application needs to interact with a 64bit application? While the list of the 32 bit components available is impressing, some components do exist only in 64-bit versions. Examples could be powershell comandlets, required to interact with SCOM 2007. For those cases there’s a special path c:\Windows\Sysnative. This allows to bypass the redirector and access files, located in System32 instead of SysWOW64.

Unfortunately this mechanism doesn’t work in Windows 2003 x64 unless you apply the hotfix 942589.

Why is it so important? Our licensing mechanisms were originaly designed for the x86 architecture. While they operate normally under x64, they extensively use the sysnative path to run the 64 bit powershell. So for those who did not upgrade to Windows 2008 but does prefer 64bit system we have bad news and good news:

The bad news is : you will have to install the hotfix and reboot the server after the installation.

The good news is : Our licensing module runs only on the SCOM serveк with the RMS role. You will have to apply the hotfix only to this particular server. SCOM servers with roles of MS, Gateway, Agent are used as SNMP proxies and do not require this hotfix.

In general, real-time services (RDP, VoIP, videoconferencing, etc) are not only affected by link or network failures – packet loss and jitter are the main contributors to the service degradation.

As the result the tasks of VPN monitoring can be defined as follows: notification of the support operator (using alerts) and fast preliminary diagnostics of the problem. After that the problem is forwarded to the network support group (with the symptoms and preliminary diagnostics) for further actions – double check the symptoms and run deeper diagnostics, then either contact the Network Provider or solve the problem locally. With backup channels at hand and proper reaction times this will minimize the probability of full disconnect of the remote office. This procedure also allows for QoS monitoring, providing the personnel on the remote locations with a comfortable level of service.

Let’s consider a real-life example. A company with 20 regional offices that are connected to the central location with 3 links each – about 60 channels altogether. We’ve started to monitor and log network failures with the duration of more than 30 minutes on the 17^th of March. During the 15 days of March we’ve logged 40 network faults. In 6 cases the Network Operators were contacted and in 3 cases the users noticed the performance degradation. In April we’ve logged 51 faults, in 22 cases the NOs were contacted and only in 2 cases the users did notice the faults. During 20 days of May 16 faults were logged, in 8 cases the NOs were contacted and no user-affecting cases. The next diagram shows the average faults per day for each month:

As you can see the overall amount of faults decreased since we’ve started the monitoring and the logging process. The difference between the overall faults and the faults that required the NOs to be contacted cannot be explained by the fact of the 24/7 monitoring process itself, but it looks like the NOs don’t want to lose their customers when they realize that the QoS is being monitored J On the other hand it’s just a theory.

How does I work? The Operator get the alerts as SMSes, E-mail and also observes it in the SCOM’s console:

The time between the failure and the generated alert is configurable, but in our case is no longer than 4 minutes. Such a long time between the event and the alerts is used in order to avoid the alarm “noise” as the amount of short-term network outages is quite high.

As the alert contains the name of the monitored channel,  the operator have to choose the site (in the SCOM console):
/p>

Then a window with the graphs of the most critical parameters is opened:

The selected parameters and the order of the grouping is the result of a long-term testing process. This particular grouping allows for the quickest possible “1-click” diagnosis of the problem.

Let’s consider the screenshot in more details:

• Graphs 1 and  2 shows the state of the router (CPU load, memory consumption etc…) and a quick glance will show whether the problem is affected by the router or something else.
• The graph 3  shows the RTT of the integral (i.e. the main and both backup channels together)  channel between the remote location and the central office, as “seen” by the end user.
• Graph 4’s positive values on the Y-axis shows the amount of lost packets over all of the channels for that region, the negative Y values show the current state. In this particular case it is “6”, which means “channel failure”. Different types за channels are color-coded differently – integral is red, MPLS is blue and the two backup channels are green and yellow.
• Graph 5 shows the throughput through the VPNs. You can see the traffic migrating from the main MPLS link to the backup public VPN
• And at last graph 6 shows the RTT of each of the channels to the site, except the integral one.

Based on the priority (failures of the main MPLS link and one of the backup VPNs obviously do have different priorities), the operator waits a predefined amount of time in case the problem will self-repair and then forwards the problem to the network department. Failures that were observed for more than 30 minutes are logged and are included into the monthly report together with the SCOM statistics:

And the RND’s case wasn’t a  long one – the case was forwarded to the network department, but the engineer didn’t have the time to log into the router – the problem disappeared and the case was closed. Fortunately this happens in most cases:

As far as the packet loss is concerned, the mechanism is principally the same  and of course everything can be tuned, tweaked, redefined  and customized.

P.S. And for the Grand Finale – most interesting screenshots from real-life situations. This illustrated how our ISPs sometimes do “provide”. No comments:

And here the somebody started to download a movie…

Why did we start writing this management pack? The power supply monitoring task becomes more urgent when you have to maintain distributed server rooms. In such environment it is hard to expect that monitoring a single UPS by the native APC application in one room you could predict the behavior of the others.

So let’s take a look at existing UPS monitoring solutions. As far as we have implemented SCOM in our company earlier, we had not been looking for monitoring solutions designed for other (non-SCOM) monitoring systems. Some of our UPSs have an imbedded network management card, some – not and these devices are connected to the servers via USB or a serial cable.  Of course we would like to have a universal monitoring tool working regardless to the connectivity type. These are the solutions we had been tested:

• Configure each UPS with its embedded software (if the UPS is intelligent enough) to send messages to a predefined list of the recipients. It’s a simplest solution having a lot of disadvantages:

1)      each UPS must have an access to the email server;

2)      UPS generates same notifications frequently in a short period of time unless you solve the issue. For example – if the temperature exceeds the predefined value for about half an hour you’ll get a bulk of identical messages. However if the temperature gets normal you definitely will NOT get the corresponding (closing) notification as there is no such feature in the embedded toolset. And it seems to be critical from the monitoring point of view as if something happened during the night in a telecom room, the operator getting back to the desk would like to pay attention to those events that still actual;

3)      It is necessary to configure notifications (i.e. types of events, list of the recipients and email server) on each UPS. You cannot configure notifications and this is another issue. For instance – if the temperature exceeds the threshold you’ll get the corresponding message notifying that it’s above the threshold but without the actual temperature value;

4)      the imbedded notification system is not flexible enough. You cannot configure it to send the calibration test outcome (which is meaningful only to the operator) to one list of recipients and the power failure notifications (system and network administrators would like to be aware) to another one;

5)      it works only for those UPSs that are directly connected to the network and have an embedded monitoring card;

• Use the native APC solution. This is about Powerchute Business Edition. The Basic version seems to be interesting one as it allows to monitor and control up to five UPSs simultaneously. However all disadvantages of the imbedded toolset (bulk of messages, no closing notifications, inflexible alerting) are still remain. The major advantage of the Powerchute is the ability to monitor those UPSs that are connected via USB and RS232.
• Power Management Pack for Operations  Manager by Quest Software. As a matter of fact this software works independently from SCOM having a connector to the Operations Manager. Due to this all settings must be done via the special utility and not via SCOM. In addition to need to be familiar with a third party software you lose the ability to use native SCOM tools like Powershell. And the installation process seems to be compecated.
• Miscellaneous free of charge third party management packs. All these MPs are using SNMP Traps as a way of getting the information. The trap being sent via UDP packet can be lost and there is no way to monitor the performance of the UPS.

This is why we decided to make our own UPS monitoring management pack. Currently our MP works with the APC UPSs however in a short while it will be able work with others which a RFC1628 compliant.

We had the following requirements to the management pack:

• Monitor the APC UPS via SCOM native features, without external connectors to third party products.
• It must be useful to the operator, i.e. typical interface, usual state models functionality;
• Rich customization capabilities – adjustable thresholds and polling intervals and etc. In a distributed enterprise environment it is really important because each server room might have its own power requirements and you might like to customize monitoring settings for several UPSs in different locations.
• Support for the different connection types – Ethernet, COM and USB.

To give a better understanding of the package features I’ll give my comments to the screenshots.

Overall list of devices

Looking into this list you can see the current state of all monitored devices. The table view allows to see a large number of objects, let’s say for 1024×768 resolution there are 30 to 40 UPSs on one screen:

List of UPSs monitoried by SCOM

Diagrams

Diagrams give you a good way to find the problem moving along the tree of objects. There is a diagram of all UPSs connected to the monitoring system:

All devices in one diagram

If you’d like to see detailed information about the item just open up the properties window.

Properties of the UPS

The model of a particular UPS, its location (this field must be filled in on the corresponding UPS), IP address and other useful information is available.

Getting back to the diagram you might mention that some devices experience problems. To better understand the issue we can drill down the object:

A UPS with the environment monitoring card installed

This is a Smart-UPS 5000 with the management/monitoring card. It is clear now that the environmental controls look fine however the battery has an alarm.

The diagram view might be considered as a main view for the monitoring system operator. It allows to open up all other views via the context menu (a performance graph or a list of alerts or whatever). The operator would see the only information available for this particular object. If you hit the “performance graphs” item for the UPS – you’ll see all available graphs for the UPS. However if you do the same thing for the battery – you’ll see only the battery-related graphs.

Health Explorer

Finding out a battery related problem we’d like to dig into the details. In order to do this we open a Health Explorer or a State Model. This is what we see for the battery:

Health Explorer gives a list of current problems on the UPS

It is obvious now that the UPS remaining time under current load is the issue. The current remaining time is lower that threshold. The actual value and the time when the threshold had been violated are available. The history of events is available as well and this helps to locate the problem.

Alerts

State model is great if you need to see the list of current events. However in order to see the list of events already closed it is more convenient to use Alerts view. This view lets you see the events within the period for a single UPS or for all UPSs at a glance. You can figure out that within the period we had two events for a single UPS, the first one is closed the second is still active:

This view is good to see the active and closed events

Graphs

Graph is another one nice troubleshooting tool. It is obvious that graphs exist only for those values that can be measured. You can see here the input voltage graph for the period of several hours:

Input Voltage Graph

There are two temperature graphs for a single UPS. The red one is for the internal temperature sensor, the orange one is for the external sensor attached to the front panel of the rack shelf near the UPS. Both graphs follow each other and the internal temperature is naturally higher:

Internal and External temperature graphs

And here is the interesting graph of the UPS remaining time  (the purple one) and the current load (the yellow-green one):

On this graph you can see the correlation between the UPS remaining time and the current UPS load

Events

For those events which become more important happening again and again – like “Incorrect password” – it is more convenient to use Event View:

Some one have been trying to guess the UPS password and it was detected by the monitoring system

It is reasonable to notify the operator if the number of repetitive events exceeds the predefined value within the particular period of time.

Notifications

It is practically impossible to track on-line all events and changes via SCOM console. This is why the email notification system is so convenient:

After the numerous wrong attempts to enter the password the operator would get the corresponding notification

Low UPS remaining runtime notification

Graphs, alerts and notifications

The following graphs are currently available:

• Input Voltage, V
• Input Frequency, Hz
• Output Voltage, V
• Output Frequency, Hz
• Output Current, A
• Output Load, %
• Battery Capacity, %
• Battery Current, A
• Battery Voltage, V
• Battery Time Remaining, m
• Battery Temperature, C
• Probe Temperature, C
• Probe Humidity, %

There is a list of alerts and notifications that currently available:

We continue working extending the management pack functionality and expect to make it available for non-APC UPSs which a RFC1628 compliant. However the main PROs will remain:

• Management pack installation process is same as for the others. No special spadework required. No external connectors and software needed. Being installed the package detects UPSs among the devices discovered by SCOM and detects the imbedded components.
• Management pack supports UPSs connected via Ethernet, USB and COM-ports.
• Alerting is quite customizable and the management pack can be easily adjusted to the particular needs.

I’d like to start from how did we get this way, i.e. why did we start writing our own Cisco management pack.

Some time ago we’ve been trying to use Zabbix, Nagios, OpManager and other systems for the monitoring issue however I’ll skip this part as far as we’re not using it anymore.

The issue is that it’s been a while since we installed SCOM and it had been working well for Windows – based servers. Then we’ve got a predictable idea to have a monitoring system for both servers, network equipment and UPSs. And we have started digging of how to monitor CISCOs with SCOM. In a catalogue on the Microsoft web site we found two third party packages (designed by MS partners and sold separately). I have installed both of them – Quest Software and Jalasoft and have shown to our company network dept… In any event we agreed that they are not suitable for our needs and rather than trying to adjust to the existing solution we’d try to write our own management pack.

I should describe in a few words the things the cons of the third party MPs we have been testing.

First of all they work separately from SCOM. As a matter of fact installer is setting up a program (executable) on a server (sometimes on a standalone server) and this program is polling CISCOs grabbing data and then exports collected data to SCOM for visualizing. It implies that you have to configure all settings out of SCOM. You’re using a third party software’s interface to make all configurations and I’d like to say that this interface is not actually intuitively obvious.

It seems like MPs are designed people who have not been working as network administrators. Packages collect a lot of data but they do not collect the data which is really needed.

The core and nicest feature of the SCOM is poorly supported by these MPs. I’m talking about notifications. If this feature is natively properly configured in SCOM there is no need to take a look into the console on regular basis as you get all noticeable notifications via email or via SMSs on a cellular phone (it is up to you: our network guys receive critical notifications via SMSs and noncritical – via emails).

Customizing. The way the network administrator is administering the network highly depends on network quality and demands placed on the network. Leaving the default MP settings untouched would lead to the unacceptably huge number of notifications or vice versa – one notification per month as far as network uses experience network quality issues. Of course every administrator working with MP might change some thresholds however in real environment it’s not always enough and it’s needed to make arrangements at lower, more generic level.

Distributing the MPs. Monitoring the networks in the real company with several branches is a complicated task (sometimes a real challenge) and idea of implementing the monitoring system pressing NEXT-NEXT-NEXT in the installation wizard looks unreal. The wise implementation approach should combine an MP with implementation methodology and experience, so is seems reasonable to distribute the MP as a part of the monitoring solution, i.e. implementation and customization project. However MPs are usually sold for the “out of box” use.

Well, let’s get back to our management pack overview.

First of all we introduced three not actually equal classes of CISCO: Routers, Switches and Firewalls. The most sophisticated from the monitoring point of view class is a class of Routers, the second one is Switches and the last one is a class of Firewalls.

It’s not necessary to dig deeply into the MP features so I’m going to place MP screenshots which seem interesting to me and add a few tables with the details for those who might need it.

1. State view.

To glance over the current state of the monitored Ciscos the following view seems to be mostly suitable.

2. Diagrams.

To see the monitored components of the particular Cisco and their current state the we defined this view. The following picture reveals the monitored components of the Cisco 1841 router:

The poor thing has one cooling fan and it’s currently damaged.

Routers of the higher classes have better monitoring capabilities. Here’s a diagram screenshot of a Cisco 2851 router. The entire diagram doesn’t suite well into the 1280×1024 screen however it seems quite suitable on my 19” monitor.

Diagram view for Cisco 2851

On this picture you can see the list of the most CPU consuming processes.

list of the most CPU consuming processes

As you might notice router is paying the bill for being monitored and SNMP – related processes are in the top-processes list.

And the brilliant feature set at last – IP SLA Monitoring. This set contains different monitors so SCOM will split instances into groups according to their current state.

As you can see – the data channel shown has some issues.

3. Graphs.

This is great if you can glance over the current state of the whole list of Ciscos however sometimes you might need details. Not everything can be shown on a graph. Let’s say I doubt whether the state of the power supply should be placed on the graph, however many other figures might (and must) be interested in a graph view.

The classical example – free memory %.

Internal temperature graphs of misc routers.

Interface traffic (IN, OUT, Total).

For any objects you can get many different graphs. As an example – there are 13 different graphs for UDP-Jitter IP SLA.

Lost packets %.

RTTMax andRTTMin

If you would like to see the complete list of available graphs – you could take a look at a table below, I’m not going to show all of them on pictures, just a few.

4. Alerts.

Every change of the current state of the object causes opening or closing of the alert. To see the alerts currently opened (or see the list of alerts happened within N days) open up the following view.

alerts currently opened.

5. Notifications.

For sure it’s a must to be able to get notifications about important events via email or SMS on a cellular phone. This feature is available and the information value seems to be good enough.

High CPU load notification.

It’s shown that the CPU load level was above the threshold (>75%) and the actual value was 81%. The top list of the processes is presented (yeah, SNMP Engine), the part consumed by the interrupts is shown as well (84%). It’s easy to make a conclusion on what is the source of the high CPU load. The calculation you might have done seem to be inexact and looks a little bit strange: 2%+84% = 81%. However it’s because Cisco is always giving only instant value for the “Load Due to Interrupts” request (i.e. I/O operations).

The data channel reveals a reduced performance.

Here we’ve got a significant packets lost on a data channel. For 10 samples used for link quality evaluation 13% were lost. The threshold value of 12% and the number of samples can be redefined.

The channel is down.

We consider channel as down if the estimation could not be done (due to data packets loss) 2 times repeatedly. Of cause thresholds and values can be redefined.

6. Health Explorer.

To find the reason Cisco got red (state list) and one of its modules is shining yellow on a diagram we can dig into email notifications sent earlier or take a look at the alerts list. However sometimes it’s more useful to open the health explorer.

State mode

As you might see Cisco is yellow because it’s fan is now in “shutdown” state (it’s a Cisco term). In addition this fan is not totally broken as far as it is getting back to “OK” state from time to time.

7. Syslog.

And of cause we couldn’t leave Syslog monitoring unattended. The most important events (from our point of view) are placed for alerting and email notification:

The user has logged in via SSH.

8. Customizing.

It’s not always interesting to track the data channel condition looking into the object list. Sometimes it’s good to make a customized view to comply with specific needs.

Regional site related graphs?

9. Reports.

If you would like to represent the inter-site data channel reliability and availability report (on a quarterly or hourly or whatever basis) – you might like the following one:

The Tables.

Routers

Switches

Firewalls

We have been working on management pack for about a year. I like it and it satisfies our network department. Essentially we could get rid of disadvantages we found in third party MPs. The benefits of the package we created could be boiled down to the following sentences:

Our CISCO management pack doesn’t install additional standalone software and doesn’t need it to work. In fact it’s a typical management pack like others, for example like MP for MS SQL written by Microsoft. It must be properly imported into SCOM and then you discover CISCOs like typical network devices. Management pack would recognize Cisco equipment would assign appropriate classes, discover the modules that are embedded and the monitoring show would kick off.

The management pack development was engaged by our network team and they are taking an active part in this project. This is why MP is monitoring those things they are interested in but not everything that theoretically can be monitored. I thought to implement misc things I saw in Jalasoft and Quest packs however decided not to go this way as it’s not needed for the end users – network guys.

Our management pack is oriented on unreliable, unstable (“dirty”) data channels which might cause a huge number of useless sequential alerts and this might lead to a real headache for the administrator. For our Scandinavian or UK clients it is typical to have stable and powerful channels within the countries. However if we step away from the MPLS networks and start using Internet VPN tunnels to join the company’s sites in different countries – then we enter the world we kept in mind creating the pack. Short time (just a few seconds) outages, a small percent of lost packets all the time – this is what I’m talking about. We have been testing the package on Russian networks and hence you might be sure that even being installed as is without fine tuning you won’t be overwhelmed with the message storm, quite the contrary – you’ll be notified with the meaningful notifications. However it is still possible to redefine thresholds to be notified about every little tiny event with dozens of event – related SMSs.

We keep working extending the management pack functionality. We expect to introduce the following features:

Package – specific reports. Currently we are using usual SCOM reports however sometimes this is not enough.

Network diagram visualization. The idea is quite simple – you’re drawing the network diagram in MS Visio and “connect” it to SCOM and the diagram start blinking with all this green, yellow and red.

If you are interested in implementing this MP in your company and would like to evaluate it please send the email requests to scom-team@metrex.ru to get the trial version.