Balance Memory Guidelines for Intel Xeon Scalable Family Processors

Servers running Intel Xeon Scalable Family processors have six memory channels per processor and up to two DIMMs per channel. To maximize a server’s memory bandwidth and overall performance, it is important to have a balanced memory configuration installed. When incorrectly configured, available memory bandwidth to the server is limited and overall performance can be greatly reduced.

Memory Interleaving

The Intel Xeon Scalable Family processor optimizes memory accesses by creating interleave sets across the memory controllers and memory channels. For each processor, there are two memory controllers. Each memory controller is then attached to three memory channels with up to two DIMMs per channel.

 

 

 

An interleaving set is created when memory channels have the same total memory capacities. If four memory channels have the same total memory capacity and are using identical DIMMs on two memory channels on each memory controller, a 4-way interleave set is created.

 

 

 

In each memory channel, another level of interleaving called “Memory Rank Interleaving” is possible. A DIMM can contain multiple memory ranks, which improves memory performance as the total number of ranks on a memory channel increases. The Intel Scalable Family Processors are optimized for two to four memory ranks per memory channel.

Guidelines for Balanced Memory Configurations

For a balanced memory subsystem the following guidelines should be followed:

  1. All populated memory channels have the same total memory capacity and the same total number of ranks
  2. All memory controllers on a processor socket have the same configuration of DIMMs
  3. All processor sockets on the same physical server should have the same configuration of DIMMs.

Examples of balanced memory configurations

2-channel interleave set

 

 

 

 

This example consists of 2 DIMMs with both populated memory channels with the same memory capacity and the same configuration on each memory controller.

4-channel interleave set

 

 

 

 

This example consists of 4 DIMMs with four populated memory channels with the same memory capacity and the same configuration on each memory controller.

 

 

 

 

This example consists of 8 DIMMs with four populated memory channels with the same memory capacity and the same configuration on each memory controller.

6-channel interleave sets

 

 

 

 

This example consists of 6 DIMMs with six populated memory channels with the same memory capacity and the same configuration on each memory controller.

 

 

 

 

This example consists of 12 DIMMs with six populated memory channels with the same memory capacity and the same configuration on each memory controller.

1      Near Balanced Memory

Near Balanced Memory configuration that use two different DIMMs and follow all balanced memory guidelines. There are 3% memory loss in memory bandwidth as long as the even number of total memory ranks on each memory channel

2      How to maximize memory bandwidth

  1. The same config of DIMMs to be used for
    1. Balancing the memory across processor sockets.
    2. Balancing the memory across memory controller
    3. Balancing the memory across populated memory channels.
  2. Optimal Memory Bandwidth

Step1: Determine the needed memory capacity

Step 2: Divide by 12 to min. memory DIMM capacity needed

Step 3: Round the value and populate in 12 DIMMs capacity

  • For Example, if 256GB of total memory capacity is needed, each DIMM needs to be a bit more than 21GB. The closest memory available DIMM size is 32GB. So, populate the server with twelve 32GB DIMMs
  • Another alternative will be 32GB and 16GB DIMM on each memory channel which provides 6 x (32 GB + 16 GB) = 288 GB of total memory capacity. This is called as Near Balanced memory configuration.
  • If the system memory capacity requirement is small, divide by 6 in step 2 and above and use only six DIMMs
  • Finally, Balancing Memory across memory controllers and the memory channels produce memory configuration producing the highest possible memory bandwidth
  • An unbalanced memory configuration can reduce the total memory bandwidth to as low as 17% of balanced memory configuration

3      How to Achieve High Memory Performance

Each Intel Xeon SP CPU has 6 memory channels, each with up to 2 DIMMs per channel (DPC) supporting a maximum of 12 DIMMs per CPU, or 24 DIMMs per server.  While the maximum memory speeds available range from 2133, 2400 or 2666 (depending upon the CPU selected)

There are some general guidelines on how to optimize your memory performance:

  • Use a balanced configuration: In a balanced configuration, the memory is interleaved equally across all DIMMs; they uniformly have high bandwidth and they have a centralized memory region.
  • Use identical DIMM types in your server.
  • Use 1 DPC per integrated memory controller or duplicate the memory

Unbalanced Memory Configuration

In “unbalanced configuration“, the memory is broken up into multiple memory regions creating an environment leading to unpredictable performance.

A configuration of 512 GB on 14G Skylake is possible but suboptimal. Across CPU models (Platinum 8176 down to Bronze 3106), there is a 65% to 35% drop in memory bandwidth when using an unbalanced memory configuration when compared to a balanced memory configuration! The figure compares 512 GB to 384 GB, but the same conclusion holds for 512 GB vs 768 GB as Figure 2 has shown us that a balanced 384 GB configuration performs similarly to a balanced 768 GB configuration.

Conclusion

The memory subsystem performance differences with balanced vs. unbalanced configurations and with different types of memory modules are not new to Skylake or Dell EMC servers. Previous studies for previous generations of servers and CPUs are listed below and show similar conclusions.
  • Memory should ideally be populated in a balanced configuration with all memory channels populated and populated identically for best performance. The number of memory channels is determined by the CPU and system architecture.
  • DIMM rank, type, and memory speed have an impact on performance.
  • Unbalanced configurations are not recommended when optimizing for performance
  • Some near-balanced configurations are a reasonable alternative when the memory capacity requirements demand a compromise.

 

Server Rental provides the best Dell, IBM, Sun Oracle and HP Server Storage for rental at best price with High quality in Bangalore, Chennai, Hyderabad, and Pune, Delhi, Noida & Gurgaon. Lowest rental price Guaranteed!! To meet these growing demands of your business, Visit Our Web page: https://serverental.com/  sends an Enquiry to sales@serverental.com OR calls us +91 9886699666

Contact Us

ServeRental (www.serverental.com)

Address: St Johns Rd, DasappaColony, Hermit Colony, Sivanchetti Gardens, Bengaluru, Karnataka 560042

Email: sales@serverental.com

Phone: +91 9886699666