346           

NT Infrastructure Management  -  Moving Closer to Control 

 

                               Mark J. Konwiczka,  MCSE

                                Hewlett-Packard Company

                                  1718 Indianwood Circle

                                    Maumee, Ohio  43537

                                         mark_konwiczka@hp.com

                                      fax 419-891-2680

 

                              Objectives of this Paper

 

1)      Identify the five major support levels within the infrastructure that all management solutions must address

2)      Look at management trends, highlight some of the functionality that customers

      are asking framework vendors to  supply and discuss how closely the vendors

      are delivering to these requests

3)      Propose a technique for self evaluation to help organizations determine where

      the management holes are

      4)Define “smart” technology and its characteristics

      5) Present some thoughts on what characteristics today’s management tools

      should possess and compare the merits of “home grown” and off-the-shelf”

      solutions

 

Background

 

As distributed NT environments become more complex and sophisticated, so too does the need for not just the tools, but the entire management methodology to tick it up a notch in the area of sophistication.  IT organizations are under tremendous pressure to weather downsizing, with the accompanying leaner budgets, while at the same time being asked to improve service levels.  The renewed emphasis on Service Level Objectives and Service Level Agreements has placed added pressure on an already strained staff struggling to cope with enormous change.   “The quality of information delivery is a critical factor in the ability of firms to compete in today’s dynamic environment.”  [i]

Today’s IT managers must possess the wisdom of Solomon, the patience of Job and the magical talents of Merlin (not to mention substantial pharmaceutical holdings) to approach being successful in managing the infrastructure.   Moving closer to control of the NT infrastructure will require support staff to have a working familiarity with the distributed compute model, it’s structure and composition, an appreciation of the importance of the three major solution components (people, possess and technology) and an understanding of the characteristics that new, “smart” management tools must possess in order to effectively manage this rapidly evolving NT infrastructure.   These “smart” tools are designed using Microsoft “defacto” industry standard technology and are generally more intuitive, lighter weight (network friendly), quicker to install and deploy and they leverage the common Microsoft features familiar to frequent NT or Windows 9x users.  We’ll discuss the objectives of the  NT infrastructure and what it looks like today, the industry trends that will effect the NT management space, how to take your own NT infrastructures pulse and we will examine tradeoffs between taking a home grown approach as opposed to implementing an off-the-shelf solution.

 

 

 

1)    The Challenge – What Do We Need to Manage?

    

 

To develop an effective management solution, it is always helpful to have a clear understanding of the challenge.    Lets start with a definition.  By  “NT Infrastructure” we mean “the systems and their related services needed to accomplish basic computing.”   Although this definition is fairly concise, it has a deeper meaning, the implications of which are much more complex.

 

Ultimately, we want this NT infrastructure to provide;

1)      Efficient and reliable transmission of network services (DNS, 

       DHCP, WINS, etc.)  in support of

2)       Efficient and reliable transmission of business services (Payroll,

       HR, E-mail, Microsoft BackOffice, etc.)

 

This is a pretty tall order considering NT’s relatively tender age and the fact that it is a rapidly evolving technology in terms of functionality and complexity.  In the case of NT, perhaps the correct approach is to take a step back and assess the state of the current environment, prior to any heavy-duty solution development.

In order to insure operational harmony in these two areas, it is appropriate, in many cases, to conduct an initial assessment of not only the network infrastructure and related services (1) but also the processes used by an organization in their support.  Ideally, this is done prior to significant work being done in the business services area (2), that is, prior to widespread deployment of NT.  The risk of bypassing an assessment and going directly to deployment (2) is that of putting a good business solution on top of a bad network infrastructure.   So, what might initially start out as a shortcut, turns into a costly, time-consuming detour.

Once we know what we are after (1 & 2) and what the existing infrastructure looks like, then we can start to dissect the infrastructure into it’s better understood hierarchical levels.  I relate better to pictures than words.  If  I can visualize it, I can better understand it.   I will use a simple pyramid structure to depict the areas I deem critical to effective management of an NT infrastructure.  So, lets look at the pyramid and build from there.

 

                                                                      Figure 1

 

In Figure 1, we have a simple pyramid which presents the infrastructure as a series of distinct management levels.  These levels have their own unique management characteristics and should have a management solution designed specifically to meet those characteristics.   Security, viewed from a management perspective, has different characteristics dependant on what is being secured.  It is depicted here as surrounding this model when, in fact, it actually permeates all levels.  Due to the complexity of the topic of security, it’s broad scope and growing importance, it is mentioned here as a critical component to be reckoned with, but is not discussed in any depth.

 

 

 

 

 

 

 

Figure 2, represents examples of the associated monitoring and status functionality  typically provided at each level.

 

 

 

 

 

 

                                                                 Figure 2

 

Level 1,  LAN/WAN,  addresses the physical topology of the network and all nodes, both system and network devices, which constitute this structure.

Solution functionality at this level should include; auto-discovery, mapping and node identification with color-coded status indications.  These are fundamental requirements that should be included in the standard solution set at his level.

More complex infrastructures might include sophisticated event correlation of network events and detailed real-time analysis of network traffic (throughput by network device, top talkers etc)

 

Level 2 focuses on the NT Servers, their health, their effective provision of respective business services and usage trend collection and analysis.  Microsoft provides detailed performance counters which can be used for real-time system monitoring.  This can be done via SNMP or through the use of “smart” tools designed to take the drudgery out of performance analysis.  This level should be manageable from three perspectives.  The first is “Operational”, the NT server should be accessible and controllable as though that server were right there in front of you.  The second and third perspectives deal with the ability to monitor and manage the “physical” hardware characteristics of the machine as well as the “processes” running within.   Monitoring of the physical components should be proactive so as to alert prior to performance degradation.  This is accomplished via disk or memory re-read counts, for example.  Monitoring of the system and application processes should include automatic failure detection and restart capability.

 

Level 3 deals with the user desktops running a variety of Microsoft code. 

Depending on the degree of involvement of the desktop on the network the degrees of management will vary dramatically.    Low intelligence, dumb terminal type devices will require only red/green monitoring at level 1 whereas high performance client/server workstations will require some degree of process monitoring (client side application running?) as well as critical resource monitoring (i.e. CPU, Memory, Disk & NIC).  Additional control might entail

“client usage” tracking to determine time spent in business applications or other diversions.

 

Level 4 addresses the database(s) running on the NT servers.  The contestants at this level are Microsoft SQL, Oracle, Informix and Sybase.  Some database characteristics of interest are; database availability and utilization, buffer utilization, disk and lock monitoring, etc.  The monitoring scripts should be reviewed for their thoroughness and the number of metrics tracked.

 

Level 5 pertains to a plethora of business applications to be supported on the NT servers, both store bought and proprietary.   Some management packages come pre-configure to monitor and manage the more popular store bought packages which provide NT functionality in areas such as virus protection, system backup or software distribution.

 

As I mentioned earlier, security is a very broad topic and beyond the scope of this paper.  Suffice it to say that new areas of concern are taking their place beside the traditional, and that the security topic is being addressed by the creation of a security admin position.

 

Many solutions will span multiple levels dependant on what the management objectives are.  If, for example, virus protection is the intent, then we will have a solution that spans levels 2 through 5.  It will run at levels 2 and 3 but also impact levels 4 and 5. 

 

2) Trends in NT Infrastructure Management

 

Customers are always more than willing to provide vendors with suggestions (solicited or not) on what the next hot set of functionality should be in the NT management space  The most common offering in the area of recommendations (not specifically limited to NT) is oft posed question, “Why can’t vendors give us a comprehensive framework that will manage what we have?”  I have heard customers say that the solutions are out there, but they have to buy a hundred of them to address a particular management need (such as security).  The answer, to a great extent, is that vendors are trying to hit a moving target.  As Professor Yechiam Yemini states “networks are organic systems – unlike any other large scale engineering systems.”  Vendors are trying to deliver to a definition, but the definition keeps changing.   Only a few short years ago we didn’t have; firewalls, fax servers, web servers, e-commerce servers, etc. to contend with, so we are indeed dealing with a dynamic the likes of which have not been seen before.

Having provided the framework vendors with the obligatory alibi, lets look at customer expectations and industry trends.  To fully appreciate the customer’s dilemma,  lets take a look at how we got to where we are, along with the respective ramifications.

 

This next slide shows that through the decades (60’s through 90’s) s compute architecture changed and evolved, the IT staff has steadily decreased in numbers while the number of systems needed to be maintained  and managed has steadily increased.

                                                          Figure 3

 

 

 

Trends:

 

Economic

 

According to a survey by IDC Research the e-commerce market will boom from $2.6 billion in 1996 to $220 billion in 2001.  Why am I mentioning this first, since this is an economic factor rather than a technological factor?  I present this survey information first due to what I will call the “black hole” effect.  Numbers of this magnitude cannot be ignored and numbers of this magnitude have a tendency to warp the adjacent landscape.  Anything that will be an enabling element that will allow one to be successful in this e-commerce space will become a critical element to business. 

 

Since networks, and their effective management, lie directly in the critical path of this success, the visibility of their effective deployment and management are fixed squarely in the cross hairs of upper level management.

 

Total Cost of Ownership (TCO) and Return on Investment (ROI) considerations  still loom large when planning large-scale infrastructure management solutions.

 

Technical

 

NT server deployment continues to grow

 

As NT server deployment continues to grow, it follows that these servers are being deployed in places where little or no system expertise exists.

 

NT based management tools continue to be developed, mature and improve.

 

 

Customer Expectations

 

Through my experiences in working directly with customers and in the process of reviewing “Requests for Proposal”, I have gathered some common, or re-occuring themes in the way of what customers would like to see the management solutions look like.   The following are the most common (not listed in any particular order).

  

Simplify the view for ease of operations and management.  Provide a simplified icon centric display with one icon for each critical business system being managed.  As an example, a management screen might display one icon representing end-to-end status of the payroll, one for inventory, one for payables etc.  One icon representing the status of the entire SAP, Peoplesoft or Baan environment.  If that one icon changed status then the operator could drill down to find the offending sub-component.  This is called a holistic view by HP and a “business process view” by some other framework suppliers.

 

Manage each of the aforementioned critical business system as a “service”

 

Provide integrated solutions that are as comprehensive as possible through the existing framework.  For the necessary corner or peripheral functionality, provide easy integration into the overall solution framework.

 

Lower the “total cost of ownership” (TCO)

 

Provide the “appropriate technology” to change the complexion of enterprise management from reactive to proactive mode.

 

Drive down the skill level needed to manage complex environments such that lesser trained operations staff can provide this function

 

Provide more sophistication in the management tools.  Instead of bombarding operations staff with large quantities of relevant but unfathomable data, preprocess this data and provide possible courses of action.

 

Provide “lights out” management of remote NT sites.

 

Provide easy and comprehensive “remote control” of systems (servers and desktops)

 

Customers want to do bite-size chunks (of a network implementation) at their own pace, yet with a fast implementation cycle”  according to Judith Hurwitz, president of Framingham, Mass-based Hurwitz Consulting Group.

 

 

How close are framework vendors to fulfilling these customer expectations?

 

Framework vendors are still scrambling to provide solutions that are comprehensive and integrate well.   Some are busily jettisoning the “framework” concept for a more “kinder and gentler” management image.  One reason for this paradigm shift is that the framework approach is being viewed as too inflexible.   Customers want solutions that are practical, pragmatic, address their specific need and drop in quickly.   This shift has accelerated the search for solutions better suited to the Microsoft product suite.  Solution components are being provided that have been designed using Microsoft technologies such as Distributed Component Object Module (DCOM), Microsoft Management Console (MMC) and ActiveX. As NT continues to be deployed as the predominant platform of choice, Microsoft has laid the groundwork for DCOM, MMC and ActiveX to become “defacto” industry standards for management technology in the NT space. 

 

3)  Techniques for Self Evaluation

 

When speaking to the “total solution” this means considering the three main solution components which are; people, process and technology.  Although this paper takes an admitted technological slant, it must be stated that technology is but one component of the solution as a whole. This section will pull together these three solution components and illustrate their individual importance as well as their collective contribution to the desired management solution.   IT technicians must understand the process and the technology and be involved in the development of both.  They must play an integral part in solution development.  Solutions developed exclusively in the “ivory tower”, then graciously handed down to the minions to expertly execute, will work much better in theory than in practice.

 

In this discussion of “evaluation”, I will expect the appropriate representation and involvement from representatives of all areas that the solution impacts, upper level management through front line operations. 

 

There are guidelines for the traditional IT Service Management areas.  One such set of guidelines that can be used to help improve how organizations use IT is the IT Infrastructure Library.  From a pure process approach, the Information Technology Infrastructure Library  (ITIL) model provides a comprehensive set of guidelines with which to both assess and improve IT service delivery.   ITIL was developed in collaboration with subject matter experts, practitioners, consultants and trainers, and it is linked to a user group, the IT Infrastructure Management Forum Ltd.   (see reference model diagram in appendix A)  It addresses traditional  IT Service Management process areas such as; Help Desk, Problem Management, Change Management, Service Level Management, etc. 

 

When I am involved in an evaluation (Assessment Phase), I try to touch on all three solution components through the use of the “Gap Analysis Worksheet”.  If the importance of process is not apparent in my discussions with the technicians in each area it is noted as an area that needs attention on the “Gap” sheet.  This helps me determine how involved each support area is in the solution process, how well they understand their technical and process responsibilities and how well they are leveraging technology to make their lives simpler.  On this Gap Analysis Worksheet, people, process and technology issues are recorded.

 

 

 

 

 

 

 

 

 

Gap Analysis Worksheet

 

Black – Functionality which already exists or can be provided by target date

Red -    Functionality  missing  - needed for effective management   (The Gap)

 

Support Area	Tools used Today	Action Needed	New Tools /Functionality Needed
Network Operations Center	Network Node Manager for NT     (NNM/NT) 5.02	Upgrade & enhance   NNM/NT 5.02
“		Integrate Third Party  Products
“			Configure  NNM Event Correlation
“		Share current NNM/NT Maps
“		Install & Deploy	Manage/X for Server management
“		Install & Deploy	Manage/X Smart Plug-In for Exchange
Operations  & Lan Admin	NNM 5.02/6.0	Expand View to network devices & servers
“		On-the-job training for new functionality
“			Print Monitoring & Management Capability
		On-the-job training for new functionality
Helpdesk	Provide MGMT View for Exchange Servers
		Enhance Remote Admin capability of client desktops
“	Remedy	Enhance current functionality
Database & Application Administration	Microsoft or Third Party tools for Exchange Management	Provide management view of SQL DB’s
“		Provide management view,  network & servers
“			Provide Access to a “Remote Control”  tool such as SMS or pcANYWHERE
Client Administration			Install Desktop Monitoring Tool  “Big Brother”
			Install ManageX Console  for Remote Admin
Security Administration			Install Node Sentry hacker intercept Tool
		Update to new Anti-virus pattern file
“

The first column lists the traditional support areas within IT.  The second column lists the tool or tools currently being used by the customer for management in that area.  The third column identifies action needed relating to existing tools or processes and the last column lists new tools or functionality that the organization would benefit from.  This “gap” worksheet allows me to develop a list of projects, with resource and time estimates, that would be needed to fill the gaps.   I also note whether the expertise exists in house to accomplish a specific task, and the estimated cost(s). 

 

We have looked at the “pyramid” which illustrates what technology exists where, and discussed the management characteristics.  We have mentioned the importance of “process” and how ITIL could be used as a yardstick in process evaluation.  The third solution component is people.   The following figure illustrates the relationship between the technology pyramid, the related processes and the technicians needed to make it all work.    This diagram shows people as the key ingredient.  They must understand the “big picture” solution and the important part that technology  and process play in it.  Here is the NT infrastructure “rocket”, it has to be fully understood before it can fly!

  4)   “Smart” Technology

 

Framework vendors are jockeying for position to be the solution provider of choice for the NT space.  The catch phrase of the management industry seems to be “Eat or be Eaten” with regards to software acquisitions.  Small firms are being purchased and others merging to fill the gaps in their management offerings.   Software vendors who take a proprietary approach to NT management might doom themselves to a “Ground Hog Day” syndrome of having to repeat the phrase, “I could have been a contender”.   Building solutions based on technology already integrated into Microsoft’s NT server architecture seems a prudent investment.  The Microsoft Management Console is, by Microsoft’s own definition the “ISV-extensible, common console framework for management applications”.   Management products which incorporate Microsoft technology and leverage Microsoft’s stated direction will have a “leg up” on the competition.

 

Who you gonna call?

 

Looking at the management landscape from a Microsoft perspective, one finds four vendors listed on their management web page.  These four vendors are:

            Boole  and Babbage  (recently merged with BMC 3/30/99)

            Computer Associates

            Hewlett-Packard

            Tivoli Systems

These are just some of the vendors, although dominant players, who provide solutions in the enterprise management space for NT.  There are a number of both privately held and public companies, which provide top notch products for managing certain aspects of the NT environment.  A collective review of the consultant group reports, trade reviews and appropriately surfed Web sites will provide volumes of data on who the point solution players are.

 

Technology with an Edge

 

Lets take a look at an example of a management tool that fully leverages Microsoft technology to manage the NT infrastructure and is a prime example of what I refer to as  “smart” technology!  One such product that leverages not only core Microsoft management technologies but also incorporates the user keystroke functionality common to Microsoft products is ManageX.    ManageX is a product initially developed by Nuview, Inc., and purchased by Hewlett-Packard in November of 1997, to provide a comprehensive management solution for the NT platform.  

ManageX functionality includes;

a)      Core Microsoft technologies such as MMC, DCOM and ActiveX

b)      A variety of  (150+) pre-packaged policies for immediate productivity

(Policies perform automated management tasks according to pre-scripted management criteria)

c)      Proactive management of systems and their respective applications

d)      Guaranteed Message Delivery

e)      Availability of Smart Plug-ins

 

 

Another dimension of this “smart” technology approach is the concept of smart plug-ins.  Smart plug-ins are add-on management modules providing much more in depth management than is provided in the base product.   Smart plug-ins are typically used in areas where a specific application is determined to be business or mission critical to an organization.  These “super scripts”  do a more sophisticated analysis of the performance metrics being tracked.  As an example, a number of these metrics might be looked at, in total, to produce a more comprehensive picture of overall performance of an application.

 

 

 

 

 

 

 

5)   SNMP If You Have To!

 

The traditional method for managing devices on a network is via the manager/agent concept.  In the simplest of scenarios a central  management station manages a variety of  network devices. These devices typically have a management agent installed with which they communicate to the management station.  A number of standard protocols exist for the management of network devices (network elements) on the network.  One such standard defined by the Internet Engineering Task Force is Simple Network Management Protocol (SNMP).     SNMP has dominated the management marketplace for over a decade.   Management frameworks are built on it and it is not likely to go away any time soon.  It works well and is firmly entrenched in the LAN/WAN support level of our pyramid.   This  technique won’t be too popular with the NT crowd, however.

A more sophisticated and reliable management protocol needs to be used, where status information to be reported is not left to chance, especially on heavily loaded networks.   Compelling reasons exist to evolve to a new management model based on “smart” technology.

 “Smart” technology means NT management tools with specialized and individually focused management policies for monitoring different aspects of the NT environment (System performance, System backups, SQL, Exchange, Anti-Virus, etc.).  This inevitable shift will take place for the following economic and pragmatic reasons.

 

Reason No. 1    Keeping pace with the “speed of business”

 

Once new compute infrastructures are in place or old ones replaced, the management solution must be dynamic enough to be installed or modified quickly  with minor modifications made expeditiously.  The “speed of business” today dictates that the management solutions compliment the business environment.  Management tools need to be “mature”, meaning that they drop in easily with minimal, if any, adjustment and provide strong “out-of-the-box” relevant functionality.  The “relevant functionality” part used to be accomplished by the tedious task of writing snmp traps or some other customization technique.  When you were done, you had a “customized” management solution that addressed your specific concerns, and if you were really lucky, the solution was still relevant when your project was complete.   However, the downside is, the more complex the environment the more costly the customization, and the more costly an environment to maintain.  The snmp trap skill set is a “niche” specialty which commands specialized talent to maintain which usually equates to more expensive and technically focused support personnel.

 

Reason No. 2   The Cost  (…of installation and customization)

 

When looking at the cost of managing the distributed NT environment, we must determine at which level we are managing in the pyramid structure, the number of elements to be managed and the degree to which we want to manage.   As an example, at  level one, LAN/WAN, we at least want to know what type of devices exist, where they are and how they are.  At this level and for these characteristics SNMP will suffice.  As we move up the pyramid and look at the variety of servers and their differing responsibility in the enterprise, more sophisticated functionality is required.  Microsoft provides some SNMP based functionality here through the implementation of the perf2mib (Performance Monitor MIB builder) tool found in the 3.51 Resource Kit.  Utilization of the perf2mib exposes the performance data available through the HKEY_PERFORMANCE_DATA registry key to SNMP.  This approach to management might be feasible for small NT environments, but for management of a large, complex enterprise,  it collapses under the weight of installation and customization.

 

 

Reason No. 3   Maintainability

 

Windows 2000 will be a more complex, feature rich product.  The in-house, custom scripts written for earlier versions will probably still work but will not address the most current OS and performance enhancements. 

The inevitable modifications to the performance data available on Microsoft NT systems would make the maintenance of volumes of static scripts a manager’s nightmare.  So the cost of maintaining both the scripts and the personnel to maintain the script would be a costly proposition.  The analogy that could be used here is of the large mainframe shops of the ‘60’s and 70’s that custom wrote large in house applications like payroll and accounts payables.   A programmer or group of programmers was responsible for the maintenance of this application with all the associated costs needed to support this design/test/implement environment.  The onus of keeping the code current with all government regulations fell squarely in the in-house staff’s lap.    In contrast, solution vendors providing policy rich, “smart technology” solutions will be much better positioned to keep abreast of the Microsoft product changes and to then make the respective changes to their management products.

 

 

Home Grown vs. Off-The-Shelf

 

The corner cases will still exist where a proprietary application will need a custom management script to properly manage it.  Overall, however, the Microsoft environment and the demands each component imposes continues to be better understood.  The sophistication of off-the-shelf technology will continue to be refined and drastically reduce the need to write custom management code.   So when looking beyond the first level support in the pyramid SNMP;

1)      Provide only basic management functionality

2)      Limited or no exception handling capabilities

3)      To fully implement requires in depth MIB expertise

4)      Questionable reliability over busy networks

“Smart” technology provides;

1)      Solutions based on Microsoft’s own management technology

2)      Drag and drop deployment mechanism

3)      Pre-scripted policies which focus on specific Microsoft tasks

4)      Exception based reporting which reduces network traffic

 

 

Summary

 

To be successful in NT infrastructure management you need to understand the challenge. 

The NT infrastructure must provide:

 

1)      Efficient and reliable transmission of network services (DNS, DHCP,

       WINS, etc.)  in support of

2)      Efficient and reliable transmission of business services (Payroll, HR, E-mail,

      BackOffice Applications, etc.)

 

Next, the solution needs to address the three key solution components; people, process and technology.   Each must have equal weight in the solution equation to be successful.   Infrastructure management will continue to be a hot topic on management’s front burner.

 

Since networks, and their effective management, lie directly in the critical path of this (e-commerce) success, the visibility of their effective deployment and management are fixed squarely in the cross hairs of upper level management.

 

The use of “smart” technology, developed specifically for management of NT environments, will be deployed to provide the necessary quick control and return-on-investment being demanded.

 

As businesses make the paradigm shift to e-commerce, tight control and management of these evolving environments will become increasingly important.

Quick implementation of focused solutions to effectively manage business and mission critical applications will continue to be key management objectives.

 

In the case of NT infrastructure management…. this race will be won by the swift!

 

 

Appendix A

 

 

 

 

 

Glossary

 

 

Enterprise -  vast expanse of interconnecting WANs and LANs geographically dispersed                             

 

Distributed Compute Model – where clients and servers may be geographically dispersed and exchange information in a peer-to-peer, two or three-tier client/server architecture.

 

Framework - encompasses the three integral enterprise management which are the network(s), systems and applications

 

 

SNMP      -  Simple Network Management Protocol  is a popular protocol in

                   TCP/IP  networks for managing network devices.

 

MIB   -       Management Information Base 

 

 

References

 

Choosing The Right Windows NT Management Solution For The Year 2000 and Beyond, Web White paper, by Karl Chen, Vice President NuView, Inc.

 

Jill Huntington-Lee Kornel Terplan & Jeff Gibson, HP OpenView  A Manager’s Guide, McGraw-Hill, 1996

 

Management Goes Modular, by Scott Tiazkun, Computer Reseller News

 

Raj Ananthanpillai, Managing Messaging Networks, A Systemic Approach, Artech House, 1995

 

Network Management A practical Perspective  , Allan Leinwand and Karen Fang ,  Addison Wesley1993

 

The Simple Book, Marshall T. Rose, Prentice Hall, 1990

 

Various HP white papers and product literature

 

Footnotes