Here is the abstract:

Windows machines running under virtualization are subject to memory management from two distinct sources. VMware manages the physical memory consumed by guest machines aggressively in order to minimize the memory footprint in order to increase the number of virtual machines that can be run on a single physical host. Significant memory savings are possible, for example, by sharing duplicate pages across similarly configured virtual machines. The second agent involved in memory management is the Windows Guest OS, which generally pursues a different goal; namely, minimizing page fault delays by aggressively caching older virtual memory pages in physical memory to avoid time-consuming paging operations to physical disk. How these two different memory management policies interact and potentially collide is the subject of this presentation. Results from several experiments where memory contention is introduced in the virtual environment are also presented, and the interpretation of the performance measurement data to monitor memory usage reported in both VMware and Windows are discussed.

You can download the presentation here.

Mark Friedman will be presenting a session titled “Memory Management in Windows Guest Machines”.  Mark will be discussing how the memory management policies of the VMware host and the memory management policies of the Windows operating system running in a virtual machine interact and potentially collide.

Additionally, Mark will be holding a Workshop on 15 May at 2pm titled “Web Application Performance and Tuning in Windows: A Performance Engineering Approach”.  Mark will  introduce a set of patterns and practices essential to developing responsive and highly scalable web-based applications.

Mark Friedman will be presenting a session titled “Memory Management in Windows Guest Machines”.  Mark will be discussing how the memory management policies of the VMware host and the memory management policies of the Windows operating system running in a virtual machine interact and potentially collide.

Additionally, Mark will be holding a Workshop on 15 May at 2pm titled “Web Application Performance and Tuning in Windows: A Performance Engineering Approach”.  Mark will  introduce a set of patterns and practices essential to developing responsive and highly scalable web-based applications.

OK, now that all the hoopla of introducing Windows 8 is over, we can now get down to business.  As I wrote in another entry, we at Demand Technology Software, have revised our software installation to work with both Windows 8 and Windows Server 2012.  As part of the revision, we have updated our Master Collection Set with new performance objects that Microsoft introduced with this new version of Windows.

I’ll simply mention the most interesting performance objects here.  Mark Friedman has already given his first impression of the new operating system, but he plans to do some further investigation into these new objects and counters and discuss them in his performance blog.

Private Cloud Performance Objects

Of course, with Server 2012 improving on virtualization and the notion of building your own private cloud, there are quite a few new performance counters in the existing Hyper-V objects as well as some new performance objects, like ‘Hyper-V Virtual Switch Processor’ and ‘Hyper-V VM Save, Snapshot, and Restore’.

New Memory Performance Object

Along with some new counters in the Memory Performance Object, Microsoft introduced a new performance object called ‘NUMA Node Memory’.  This object contains performance counters to report approximate memory utilization metrics by node in a NUMA system.

Network and File Sharing Performance Objects

There are quite a few new performance objects related to network and file sharing including the new ‘Network Adapter’ object which has the same counters as the ‘Network Interface’ object.  However, the Network Adapter object includes virtual adapter instances, as well as physical adapter instances.  It is nice that Microsoft kept the original object, while providing support for virtual instances in a new object.  Speaking of virtual adapters, there is also a new object called ‘Network Virtualization’ which according to the explain text for the object “monitors the network activity matching network virtualization policies.”  There is also a new ‘Physical Network Interface Card Activity’ performance object which “measures events on a physical network card”.  This object essentially measures the physical state of the card, whereas the ‘Network Interface’ and ‘Network Adapter’ objects measure the activity (network traffic) on the card.

In support of the SMB (Server Manage Block) file sharing system (Samba software uses this protocol), there are three new objects beginning with ‘SMB’ on the server side and one object on the client side.

Finally, there are objects beginning with ‘WinNAT’ and contain performance counters for Network Address Translated packets for the ICMP, TCP, and UDP protocols.

Collect the new Performance Objects with Performance Sentry

Of course, if you want to collect these Windows performance metrics in cost effective and resource efficient way, you’ll want to use Performance Sentry from Demand Technology Software!

One of our software line items was maintenance for a software product called ‘JARS’ which I think stood for ‘Johnson Accounting and Reporting System’.  We were paying thousands of dollars a year for this product.  We realized its only purpose was to provide daily, weekly, and monthly reports for the manager of operations.  The software processed the same data tapes that our MXG software was processing into a Performance Database (PDB) on a nightly basis.  I consider MXG one of the first ‘shareware’ products.  It is masterfully maintained and distributed (still at an incredibly low price) by Barry Merrill, his colleagues, and contributors.  Again, I digress.

I looked at the reports that were being produced and suggested to the manager receiving them that I could produce the same reports in parallel from the MXG PDB. I wrote a program, tested it and showed the results to the manager who approved it for production use.  The end result was that we eliminated a line item in our software budget costing us thousands of dollars a year and eliminated a batch job thereby reduducing CPU costs and extra tape mounts (even though tape mounts had been automated by this time, there was still a cost involved).

So what’s the point of this story?  Performance Sentry a.k.a. NTSMF (for mainframe performance geeks that stands for SMF for ‘Windows NT’ ) was created with MXG in mind as a way to efficiently and cost effectively do performance analysis and capacity planning for Windows Servers.  The same MXG data repository and reporting tools can be used for Windows performance data produced by Performance Sentry.  Performance Sentry uses approximately .3 percent CPU and 16MB to 20MB per day to monitor CPU, Disk, Memory, Network, and all running processes on Windows Servers.

Companies using MXG could replace that astonishingly large line item in their software budget for Windows performance monitoring, reporting, and capacity planning with the more cost effective (and more robust) Perfomance Sentry from Demand Technology Software.

Have I piqued your interest?  Give us a call at 239-261-8945 or send inquiries to sales@demandtech.com, or learn more about Performance Sentry by clicking here.

You can download the latest release of Performance Sentry here.

After you install, use the Windows key on the keyboard to go to the start screen where you will see shortcut icons for ‘Performance Sentry Administration’ and ‘Performance Sentry Portal’:

Figure 1 Showing the Windows Server 2012 Start Screen with Performance Sentry Installed

What about the other shortcuts to Performance Sentry documentation that appear in the ‘Start/Programs’ menu in earlier version of Windows?  Right click on the desktop and click on ‘All Apps’.  All shortcuts will then appear under categories, just like they would appear under the ‘Start’ menu tree in earlier Windows versions:

Figure 2 Showing Windows Server 2012 Start All Apps with Performance Sentry Installed

The Performance Sentry Performance Database (PDB) is a flexible and highly scalable repository designed specifically for optimal storage and reporting of detailed, daily performance data gathered using the performance monitoring facilities of the Windows operating system. This section introduces the data model that the PDB uses to store performance data, which is influenced by several key characteristics of the performance monitoring facilities of the Windows operating system.

Performance Objects and Counters

The Windows performance counters instrument the operating system, the hardware, its major subsystems, and many of its applications. All the performance data is available using a single, common collection mechanism and shares a common format. The performance data consists of named counter values that are time-stamped and organized by performance object. While Windows supports a large number of distinct counter types, most of the counters that are available either report an activity rate (counter type Perf_Counter_Counter) or an instantaneous value (counter type Perf_Counter_Rawcount).

Each counter value is uniquely identified in the Windows Perfmon application using a naming string with the following syntax:

 \\MachineName\ObjectName(ParentName/ObjectInstance#InstanceIndex)\CounterName

where the InstanceName, InstanceIndex, and ParentName can be necessary to disambiguate the counter reference.

Windows counters are grouped by performance object:

               object → List<counters>

but not all performance objects contain multiple instances For example, the Memory object contains a set of system-wide performance counters accounting for usage of both physical and virtual memory. For any one point in time for a given machine, there is only one instance of the Memory object reported. However, instances of the Processor object report processor utilization per processor. The unique instance name associated with the Processor object is a processor ID. (See Overview of Performance Data Collection for additional details regarding the performance data naming convention.)

The data model also supports one level of parent-child nesting, but only a few performance objects, such as the Thread, implement the parent-child relationship. (The parent of a Thread is its Process.) The Instance index is an index synthesized by the Windows Performance Monitor to distinguish multiple instances of the Process object with the same process name (such as separate iexplore processes).

Figure 1 represents this conceptual data model for performance objects and counters in Windows visually showing one or more performance counters associated with a unique instance of a performance object:

 Figure 1. A conceptual model showing the relationship between Performance Objects and Counters in Windows.

The Performance Sentry PDB adopts this conceptual schema, but with the following key differences:

•       All the counter values associated with a given unique instance of a performance object are stored in the PDB by timestamp in a single database    table data row (where the SQL column name corresponds to the performance counter name). This results in considerably more efficient use of disk storage, compared to storing each counter value separately in its own unique row in a PDB, as many other products do.
•      Instead of a synthetic index for unique instances of the Process object, the PDB differentiates among unique instances of the Process object using the Process ID, or PID. At any point in time, the Process ID (or ID Process, as the counter is labeled in the Process object) is guaranteed to be unique. Using the Process ID is the identification scheme used by the OS itself.

The Process ID value is also valuable because it can be used to correlate the process level measurements taken by the OS with internal application-level measurements whenever the PID is explicitly available. (For example, the performance counters at the application level provided by the .NET Framework report the Process name as the Instance name along with a separate Process ID field, for example.) In the terminology of data modeling, the process ID is a Key to the Process table. When the Process ID is stored in a table other than the Process table, it serves as a Foreign Key that can be used to JOIN data from that table to data in the Process table. Figure 1 above shows ProcessID as a Foreign Key in the PerformanceCounter data structure.

Machine-specific Data Tables

To promote scalability, the Performance Sentry PDB stores all the performance counter data in time-series in a set of machine-specific data tables. Machine-specific database tables, one per performance object per machine, are created dynamically by the PDB loader program when the data file is processed. A unique database table is generated with a table name in the form of MachineName@ObjectName for each set of performance counters found in the data file being processed. The number of machine-specific data tables created from a data file is a function of the input data.

The PDB does contain several statically defined database tables. These include a Machines table that contains a row for every machine which has performance data loaded in the PDB. The Machines table entry for a machine contains a pointer (another Foreign Key) to the name of the Data Collection Set (DCS) that is stored in the DCS Data Definition table, also allocated statically. Entries in the DCS Data Definition table define the performance objects and counters that are present in a data file from that specific machine.  

Figure 2 provides an overview of the PDB schema, indicating how the statically-defined Machines and DCS_DataDef tables are used to generate the machine-specific database tables dynamically during a database Load. (In the diagram, not all columns are represented to save space.) Note, that if the machine-specific database tables already exist for a machine, the Loader program validates them against the DCS data definition in the current data file and, if there are any schematic changes, updates the machine data tables accordingly.

 Figure 2. The relationship between the static database tables that are machine-oriented and the dynamic, machine-specific tables that hold the performance counter data. The Machines table entry contains a pointer to the DCS Data Definition, which is then used to construct the machine-specific data tables. Two machine-specific data tables are illustrated, a simple MachineName@Memory table and an instanced MachineName@Process table. For the sake of brevity, not all counters that are in the objects are shown.

During a database Load operation, the NtsmfDataAccess component is called, passing a data file name to process.  Each data file is then processed in three distinct stages:

1.     Processing the Configuration records embedded at the beginning of the data file that define the performance objects contained in the file, and their associated performance counters.
2.      Processing the data records themselves that occupy the bulk of the data file. These are formatted as a set of DataTables, one per performance object, and stored as a collection of DataTables contained in a DataSet object.
3.     One by one, Inserting new rows of data into SQL Server database tables from each of the DataTables formatted in step 2.

At the outset of stage 3 in the database load process, the set of SQL Server tables that are present are validated against the DataTables resident in memory that were built from the current data file, and any changes or updates that are detected are propagated to the disk-resident database versions of the data tables automatically at that time.

Design Trade-offs

In any complex system, design decisions inevitably involve trade-offs. In the case of the dynamic data model that is implemented in the Performance Sentry PDB, there are some distinct advantages of the approach used, but also some implications are not uniformly positive.

Among the advantages of this approach, the ability of the PDB to scale to accommodate updating and reporting on thousands of machines is particularly noteworthy. The data model supports a classic SIMD (single instruction, multiple data) approach to parallelizing the nightly database load. Database loads are accomplished using a simple command:

               ntdacmd –d folder-name –threads auto|n

where the loader program is pointed at a folder containing the latest set of daily data files. When loading data files in parallel, the loader program scans the input folder, instantiates a .NET ThreadPool object, and then assigns each worker thread in the thread pool a file to process, until all the files that were found in the folder are processed. Worker threads make a few accesses to the shared static data tables at the beginning of each file loading operation. Thereafter, Database load operations are accomplished in parallel. The speed-up is roughly a function of the number of update threads because it is virtually a shared nothing data model for distributing the data over multiple sets of database tables. In practice, however, the performance of Writes to the database disks, especially the database logging disk, usually serves as an upper bound on the throughput of the database load process.

In principle, the Performance Sentry PDB data model is also very effective in scaling the performance of queries that span multiple machines, which can take advantage of Microsoft SQL Server’s automatic parallel query optimization facilities. See the MSDN Help topics “Parallel Query Processing” and “Parallel LINQ” for details. When the focus is reporting data from a single machine, since data from different performance objects is stored in separate database tables, many machine-specific queries can also be parallelized.

In general, the Performance Sentry PDB data model is optimized for reporting. In the most common scenarios for reporting on Windows performance counter data, all the data needed for a single report is contained in a single data table, which are each indexed by time of day to facilitate logical JOIN operations within and across data tables. Figure 3 shows the results of a typical PDB query that Joins data from the Process table with the Processor table for a single machine. Because the machine@Process and the machine@Processor tables are distinct, SQL Server can create a query execution Plan that runs queries against each table in parallel.  

In the Query to build the chart illustrated in Figure 3, the Processor table contains 32 processor instances reported every minute, while the Process table is a sparse matrix that contains only instances of processes that exceed defined thresholds for CPU (or memory) usage. The SQL Query sums the processor instances by time interval. These are JOINed by time interval to a report of the Top 5 overall processes, based on CPU consumption over the query interval. Complex queries such as these can benefit from running in parallel on a multi-core processor.

Figure 3. A common reporting scenario Joins data from multiple machine-specific database tables in the Performance Sentry PDB. Here data from the machine@Process and machine@Processor tables are Joined by time of day.

This performance optimization is not free, however. There are serious trade-offs that customers encounter with this design. The data model is simple conceptually, but, in practice, navigating to the desired data is complex. With multiple data tables generated for each machine, the developer that builds a PDB query must understand the data model and know how to navigate to the proper set of machine-specific data tables.

One general approach to navigation in the PDB is to locate the tables of interest in the System Views that explicitly represent the Database Definition Language (DDL). Using LINQ to SQL, for example, a static method IsValidPDBTableName can be called to check for a valid SQL Table name. The method is a LINQ-to-SQL wrapper around a query of PDB.INFORMATION_SCHEMA.TABLES:

string metadataquery = @”Select TABLE_NAME From [PDB].[INFORMATION_SCHEMA].[TABLES]“ +

                                ” Where TABLE_NAME = ‘” + TableName + “‘”;

Using the Machines table to navigate to the DCS Data Definition table where the schema metadata is represented implicitly is another viable approach developers can use.

Another complication is the sheer number of data tables that are generated when the PDB is used to process performance data from a large number of machines. Simple SQL Server query tools, including Excel’s otherwise excellent support for querying SQL Server databases tend to choke when the number of data tables resident in the PDB exceeds 100,000. Similarly, the time it takes to run the basic SQL Server Management Studio function to enumerate the PDB data tables can be prohibitive with large numbers of tables.

Our new DTS Data Services Software Developers’ Kit (SDK) helps to mitigate these concerns for developers interested in building their own PDB queries. The SDK packages together the common queries that reporting programs run in our Web Portal. The SDK implements the common queries used to locate machines and their associated machine-specific data tables. It contains methods to find the objects and counters associated with specific machines in the PDB, and then gather the specific counter data specified. The SDK also includes a sample desktop graphical reporting application that demonstrates the use of all the important APIs for querying the PDB.

Click here to download the Data Access Services SDK

Mark Friedman will be presenting a session titled “Measuring Processor Utilisation in Windows and Windows Applications Software”.  Mark will be discussing legacy techniques for measuring processor utilization in Windows and introducing newer techniques that are event-driven.  Mark presented the same session at the CMG ’11 International Conference in December 2011.

Additionally, Mark will be holding a Workshop on 15 May at 2pm titled “Web Application Performance and Tuning in Windows: A Performance Engineering Approach”.  Mark will  introduce a set of patterns and practices essential to developing responsive and highly scalable web-based applications.

You can read additional information about Mark’s UKCMG sessions at his Performance By Design blog.