DescentBB

High-speed clustering on 10Gbit InfiniBand proves superior on AMD platforms

During IDF, we had the pleasant experience of being able to speak with PathScale, a company with products that are shaping the landscape of high-speed networking and communications for clustering environments. These applications include biotech research, space research, large-scale 3D rendering and any other high-level computational applications.

Greg Lindahl, PathScale's Distinguished Engineer, spoke to us about some of the company's controllers, specifically its InfiniPath controller, which is a 10Gbit/sec InfiniBand interconnect capable of scaling its performance depending on the number of CPU cores in a system. PathScale says that while other controllers are limited to 2 to 4 million messages (small packets of data), a single PathScale controller is capable of transmitting up to 10 million messages a second, providing the industry’s best InfiniBand performance.

Quick specifications:

* 1.29 Microseconds MPI Latency (0-byte, 1/2 round trip)
* 88 byte message size to achieve 1/2 of peak bandwidth (streaming, 8 CPU cores)
* 954 MB/s peak uni-directional bandwidth (streaming)
* 1884 MB/s peak bi-directional bandwidth (streaming)
* 583 MB/s TCP/IP throughput with 6.7 microsecond latency with a standard Linux stack
* 8.2 million messages/sec (four CPU cores per adapter)
* 3.3 million messages/sec (one CPU core per adapter)

Currently, PathScale delivers its controllers in two formats: a PCI Express InfiniBand adapter and another that interfaces with AMD's HyperTransport. With PCI Express, PathScale says that customers will get better performance than the competition but performance will still be capped to the interface limitations of PCI Express. PathScale tells us that if enterprise board developers use its InfiniPath controller on a HyperTransport interface, performance will at least double that of the best PCI Express systems. Even better yet, multi-core Opterons coupled with HyperTransport will give the best performance available.

Lloyd Dickman, PathScale’s Distinguished Architect, tells us that not only does InfiniBand performance receive benefits, but high-speed clustering applications will realize the best performance utilizing an AMD platform. This claim comes as no surprise since Google, and several well established universities are already utilizing AMD platforms for their most demanding applications.

Lindahl left us with a few details about how PathScale's product scale well with more cores and while he said that the technique was conceptualized more than a decade ago, utilizing AMD's HyperTransport allows PathScale to fully capitalize on multi-core systems. Lindahl says that even on an 8-way Xeon system, a PCI Express InfiniPath adapter is unable to tap the full potential of all 8 cores simply because it must communicate through a single Northbridge chip. Lindahl said that while Intel has an answer to AMD's HyperTransport, it is unavailable and there is no set announcement date.

More about PathScale's InfiniPath products can be found here. The company also links to independent benchmarks sources which utilize systems that contain up to 512 physical CPUs (Opterons).