SPARC64 VI

History

In the late 1990s, HAL Computer Systems, a subsidiary of Fujitsu, was designing a successor to the SPARC64 GP as the SPARC64 V. First announced at Microprocessor Forum 1999, the HAL SPARC64 V would have operated 1 GHz and had a wide superscalar organization with superspeculation, an L1 instruction trace cache, a small but very fast 8 KB L1 data cache, and separate L2 caches for instructions and data. It was designed in Fujitsu's CS85 process, a 0.17 μm CMOS process with six levels of copper interconnect; and would have consisted of 65 million transistors on a 380 mm2 die. Originally scheduled for a late 2001 release in Fujitsu GranPower servers, it was canceled in mid-2001 when HAL was closed by Fujitsu, and replaced by a Fujitsu design.

The first Fujitsu SPARC64 Vs were fabricated in December 2001. They operated at 1.1 to 1.35 GHz. Fujitsu's 2003 SPARC64 roadmap showed that the company planned a 1.62 GHz version for release in late 2003 or early 2004, but it was canceled in favor of the SPARC64 V+. The SPARC64 V was used by Fujitsu in their PRIMEPOWER servers.

The SPARC64 V was first presented at Microprocessor Forum 2002. At introduction, it had the highest clock frequency of both SPARC and 64-bit server processors in production; and the highest SPEC rating of any SPARC processor.

Description

The SPARC64 V is a four-issue superscalar microprocessor with out-of-order execution. It was based on the Fujitsu GS8900 mainframe microprocessor.

Pipeline

The SPARC64 V fetches up to eight instructions from the instruction cache during the first stage and places them into a 48-entry instruction buffer. In the next stage, four instructions are taken from this buffer, decoded and issued to the appropriate reserve stations. The SPARC64 V has six reserve stations, two that serve the integer units, one for the address generators, two for the floating-point units, and one for branch instructions. Each integer, address generator and floating-point unit has an eight-entry reserve station. Each reserve station can dispatch an instruction to its execution unit. Which instruction is dispatched firstly depends on operand availability and then its age. Older instructions are given higher priority than newer ones. The reserve stations can dispatch instructions speculatively (speculative dispatch). That is, instructions can be dispatched to the execution units even when their operands are not yet available but will be when execution begins. During stage six, up to six instructions are dispatched.

The register files are read during stage seven. The SPARC architecture has separate register files for integer and floating-point instructions. The integer register file has eight register windows. The JWR (Joint Work Register) contains 64 entries and has eight read ports and two write ports. The JWR contains a subset of the eight register windows, the previous, current and next register windows. Its purpose is reduce the size of register file so that the microprocessor can operate at higher clock frequencies. The floating-point register file contains 64 entries and has six read ports and two write ports.

Execution begins during stage nine. There are six execution units, two for integer, two for loads and stores, and two for floating-point. The two integer execution units are designated EXA and EXB. Both have an arithmetic logic unit (ALU) and a shift unit, but only EXA has multiply and divide units. Loads and stores are executed by two address generators (AGs) designated AGA and AGB. These are simple ALUs used to calculate virtual addresses.

The two floating-point units (FPUs) are designated FLA and FLB. Each FPU contains an adder and a multiplier, but only FLA has a graphics unit attached. They execute add, subtract, multiply, divide, square root and multiply–add instructions. Unlike its successor SPARC64 VI, the SPARC64 V performs the multiply–add with separate multiplication and addition operations, thus with up to two rounding errors. The graphics unit executes Visual Instruction Set (VIS) instructions, a set of single instruction, multiple data (SIMD) instructions. All instructions are pipelined except for divide and square root, which are executed using iterative algorithms. The FMA instruction is implemented by reading three operands from the operand register, multiplying two of the operands, forwarding the result and the third operand to the adder, and adding them to produce the final result.

Cache

The SPARC64 V has two-level cache hierarchy. The first level consists of two caches, an instruction cache and a data cache. The second level consists of an on-die unified cache.

The level 1 (L1) caches each have a capacity of 128 KB. They are both two-way set associative and have 64-byte line size. They are virtually indexed and physically tagged. The instruction cache is accessed via a 256-bit bus. The data cache is accessed with two 128-bit buses. The data cache consists of eight banks separated by 32-bit boundaries. It uses a write-back policy. The data cache writes to the L2 cache with its own 128-bit unidirectional bus.

The second level cache has a capacity of 1 or 2 MB and the set associativity depends on the capacity.

System bus

The microprocessor has a 128-bit system bus that operates at 260 MHz. The bus can operate in two modes, single-data rate (SDR) or double-data (DDR) rate, yielding a peak bandwidth of 4.16 or 8.32 GB/s, respectively.

Physical

The SPARC64 V consisted of 191 million transistors, of which 19 million are contained in logic circuits. It was fabricated in a 0.13 μm, eight-layer copper metallization, complementary metal–oxide–semiconductor (CMOS) silicon on insulator (SOI) process. The die measured 18.14 mm by 15.99 mm for a die area of 290 mm2.

Electrical

At 1.3 GHz, the SPARC64 V has a power dissipation of 34.7 W. Fujitsu PrimePower servers using the SPARC64 V supply a slightly higher voltage to the microprocessor to enable it to operate at 1.35 GHz. The increased voltage and operating frequency increased the power dissipation to ~45 W.

SPARC64 V+

The SPARC64 V+, code-named "Olympus-B", is a further development of the SPARC64 V. Improvements over the SPARC64 V included higher clock frequencies of 1.82–2.16 GHz and a larger 3 or 4 MB L2 cache.

The first SPARC64 V+, a 1.89 GHz version, was shipped in September 2004 in the Fujitsu PrimePower 650 and 850. In December 2004, a 1.82 GHz version was shipped in the PrimePower 2500. These versions have a 3 MB L2 cache. In February 2006, four versions were introduced: 1.65 and 1.98 GHz versions with 3 MB L2 caches shipped in the PrimePower 250 and 450; and 2.08 and 2.16 GHz versions with 4 MB L2 caches shipped in mid-range and high-end models.

It contained approximately 400 million transistors on an 18.46 mm by 15.94 mm die for an area of 294.25 mm2. It was fabricated in a 90 nm CMOS process with ten levels of copper interconnect.

SPARC64 VI

The SPARC64 VI, code-named Olympus-C, is a two-core processor (the first multi-core SPARC64 processor) which succeeded the SPARC64 V+. It is fabricated by Fujitsu in a 90 nm, 10-layer copper, CMOS silicon on insulator (SOI) process, which enabled two cores and an L2 cache to be integrated on a die. Each core is a modified SPARC64 V+ processor. One of the main improvements is the addition of two-way coarse-grained multi-threading (CMT), which Fujitsu called vertical multi-threading (VMT). In CMT, which thread is executed is determined by time-sharing, or if the thread is executing a long-latency operation, then execution is switched to the other thread. The addition of CMT required duplication of the program counter and the control, integer, and floating-point registers so there is one set of each for every thread. A floating-point fused multiply-add (FMA) instruction was also added, the first SPARC processor to do so.

The cores share a 6 MB on-die unified L2 cache. The L2 cache is 12-way set associative and has 256-byte lines. The cache is accessed via two unidirectional buses, a 256-bit read bus and a 128-bit write bus. The SPARC64 VI has a new system bus, the Jupiter Bus. The SPARC64 VI consisted of 540 million transistors. The die measures 20.38 mm by 20.67 mm (421.25 mm2).

The SPARC64 VI was originally to have been introduced in mid-2004 in Fujitsu's PrimePower servers. Development of the PrimerPowers were canceled after Fujitsu and Sun Microsystems announced in June 2004 that they would collaborate on new servers called the Advanced Product Line (APL). These servers were scheduled to be introduced in mid-2006, but were delayed until April 2007, when they were introduced as the SPARC Enterprise. The SPARC64 VI processors featured in the SPARC Enterprise at its announcement were a 2.15 GHz version with a 5 MB L2 cache, and 2.28 and 2.4 GHz versions with 6 MB L2 caches.

SPARC64 VII

The SPARC64 VII (previously called the SPARC64 VI+), code-named Jupiter, is a further development of the SPARC64 VI announced in July 2008. It is a quad-core microprocessor. Each core is capable of two-way simultaneous multithreading (SMT), which replaces two-way coarse-grained multithreading, termed vertical multithreading (VMT) by Fujitsu. Thus, it can execute eight threads simultaneously. Other changes include more RAS features; the integer register file is now protected by ECC, and the number of error checkers has been increased to around 3,400. It consists of 600 million transistors, is 21.31 mm × 20.86 mm (444.63 mm2) large, and is fabricated by Fujitsu in its 65 nm CMOS, copper interconnect process.

The SPARC64 VII was featured in the SPARC Enterprise. It is socket-compatible with its predecessor, the SPARC64 VI, and is field-upgradeable. SPARC64 VIIs could coexist, whilst operating at their native clock frequency, alongside SPARC64 VIs. The first versions of the SPARC64 VII were a 2.4 GHz version with a 5 MB L2 cache used in the SPARC Enterprise M4000 and M5000, and a 2.52 GHz version with a 6 MB L2 cache. On 28 October 2008, a 2.52 GHz version with a 5 MB L2 cache was introduced in the SPARC Enterprise M3000. On 13 October 2009, Fujitsu and Sun introduced new versions of the SPARC64 VII (code-named Jupiter+), a 2.53 GHz version with a 5.5 MB L2 cache for the M4000 and M5000, and a 2.88 GHz version with a 6 MB L2 cache for the M8000 and M9000. On 12 January 2010, a 2.75 GHz version with a 5 MB L2 cache was introduced in the M3000.

SPARC64 VII+

The SPARC64 VII+ (Jupiter-E), referred to as the M3 by Oracle, is a further development of the SPARC64 VII. The clock frequency was increased up to 3 GHz and the L2 cache size was doubled to 12 MB. This version was announced on 2 December 2010 for the high-end SPARC Enterprise M8000 and M9000 servers. These improvements resulted in an approximately 20% increase to overall performance. A 2.66 GHz version was for mid-range M4000 and M5000 models. On 12 April 2011, a 2.86 GHz version with two or four cores and a 5.5 MB L2 cache was announced for the low-end M3000. The VII+ is socket-compatible with its predecessor, the VII. Existing high-end SPARC Enterprise M-Series servers are able to upgrade to the VII+ processors in the field.

SPARC64 VIIIfx

The SPARC64 VIIIfx (Venus) is an eight-core processor based on the SPARC64 VII designed for high-performance computing (HPC). As a result, the VIIIfx did not succeed the VII, but existed concurrently with it. It consists of 760 million transistors, measures 22.7 mm by 22.6 (513.02 mm2;), is fabricated in Fujitu's 45 nm CMOS process with copper interconnects, and has 1,271 I/O pins. The VIIIfx has a peak performance at 2 GHz of 128 GFLOPS and a typical power consumption of 58 W at 30 °C for an efficiency of 2.2 GFLOPS/W. The VIIIfx has four integrated memory controllers for a total of eight memory channels. It connects to 64 GB of DDR3 SDRAM and has a peak memory bandwidth of 64 GB/s.

History

The VIIIfx was developed for the Next-Generation Supercomputer Project (also called Kei Soku Keisenki and Project Keisoku) initiated by Japan's Ministry of Education, Culture, Sports, Science and Technology in January 2006. The project aimed to produce the world's fastest supercomputer with performance of over 10 PFLOPS by March 2011. The companies contracted to develop the supercomputer were Fujitsu, Hitachi, and NEC. The supercomputer was originally envisioned to have a hybrid architecture containing scalar and vector processors. The Fujitsu-designed VIIIfx was to have been the scalar processor, with the vector processor to have been jointly designed by Hitachi and NEC. However, due to the 2008 financial crisis, Hitachi and NEC announced in May 2009 that they would leave the project because manufacturing the hardware they were responsible for would result in financial losses for them. Afterwards, Fujitsu redesigned the supercomputer to use the VIIIfx as its only processor type.

By 2010, the supercomputer that would be built by the project was named the K computer. Located at the RIKEN's Advanced Institute for Computational Science (AICS) in Kobe, Japan; it obtains its performance from 88,128 VIIIfx processors. In June 2011, the TOP500 Project Committee announced that the K computer (still incomplete with only 68,544 processors) topped the LINPACK benchmark at 8.162 PFLOPS, realizing 93% of its peak performance, making it the fastest supercomputer in the world at that time.

Description

The VIIIfx core is based on that of the SPARC64 VII with numerous modifications for HPC, namely High Performance Computing-Arithmetic Computational Extensions (HPC-ACE) a Fujitsu-designed extension to the SPARC V9 architecture. The front-end had coarse-grained multi-threading removed, the L1 instruction cache halved in size to 32 KB; and the number of branch target address cache (BTAC) entries reduced to 1,024 from 8,192, and its associativity reduced to two from eight; and an extra pipeline stage was inserted before the instruction decoder. This stage accommodated the greater number of integer and floating-point registers defined by HPC-ACE. The SPARC V9 architecture was designed to have only 32 integer and 32 floating-point number registers. The SPARC V9 instruction encoding limited the number of registers specifiable to 32. To specify the extra registers, HPC-ACE has a "prefix" instruction that would immediately follow one or two SPARC V9 instructions. The prefix instruction contained (primarily) the portions of the register numbers that could not fit within a SPARC V9 instruction. This extra pipeline stage was where up to four SPARC V9 instructions were combined with up to two prefix instructions in the preceding stage. The combined instructions were then decoded in the next pipeline stage.

The back-end was also heavily modified. The number of reservation station entries for branch and integer instructions were reduced to six and ten, respectively. Both the integer and floating-point register files had registers added to them: the integer register file gained 32, and there were a total of 256 floating-point registers. The extra integer registers are not part of the register windows defined by SPARC V9, but are always accessible via the prefix instruction; and the 256 floating-point registers could be used by both scalar floating-point instructions and by both integer and floating-point SIMD instructions. An extra pipeline stage was added to the beginning of the floating-point execution pipeline to access the larger floating-point register file. The 128-bit SIMD instructions from HPC-ACE were implemented by adding two extra floating-point units for a total of four. SIMD execution can perform up four single- or double-precision fused-multiply-add operations (eight FLOPs) per cycle. The number of load queue entries was increased to 20 from 16, and the L1 data cache was halved in size to 32 KB. The number of commit stack entries, which determined the number of instructions that could be in-flight in the back-end, was reduced to 48 from 64.

Miscellaneous specifications

Physical address range: 41 bits

Cache:

L1: 32 KB two-way set-associative data, 32 KB two-way set-associative instruction (128-byte cache line), sectored

L2: 6 MB 12-way set-associative (128-byte line), index-hashed, sectored

Translation lookaside buffer (TLB):

A 16-entry micro-TLB; and 256-entry, four-way set-associative TLB for instructions

A 512-entry, four-way set-associative TLB for data, no victim cache

Page sizes: 8 KB, 64 KB, 512 KB, 4 MB, 32 MB, 256 MB, 2 GB

SPARC64 IXfx

The SPARC64 IXfx is an improved version of the SPARC64 VIIIfx designed by Fujitsu and LSI first revealed in the announcement of the PRIMEHPC FX10 supercomputer on 7 November 2011. It, along with the PRIMEHPC FX10, is a commercialization of the technologies that first appeared in the VIIIfx and K computer. Compared to the VIIIfx, organizational improvements included doubling the number of cores was to 16, doubling the amount of shared L2 cache to 12 MB, and increasing peak DDR3 SDRAM memory bandwidth to 85 GB/s. The IXfx operates at 1.848 GHz, has a peak performance of 236.5 GFLOPS, and consumes 110 W for a power efficiency of more than 2 GFLOPS per watt. It consisted of 1 billion transistors and was implemented in a 40 nm CMOS process with copper interconnects.

SPARC64 X

The SPARC64 X is a 16-core server microprocessor announced in 2012 and used in Fujitsu's M10 servers (which are also marketed by Oracle). The SPARC64 X is based on the SPARC64 VII+ with significant enhancements to its core and chip organization. The cores were improved by the inclusion of a pattern history table for branch prediction, speculative execution of loads, more execution units, support for the HPC-ACE extension (originally from the SPARC64 VIIIfx), deeper pipeline for a 3.0 GHz clock frequency, and accelerators for cryptography, database, and decimal floating-point number arithmetic and conversion functions. The 16 cores share a unified, 24 MB, 24-way set-associative L2 cache. Chip organization improvements include four integrated DDR3 SDRAM memory controllers, glueless four-way symmetrical multiprocessing, ten SERDES channels for symmetrical multiprocessing scalability to 64 sockets, and two integrated PCI Express 3.0 controllers. The SPARC64 X contains 2.95 billion transistors, measures 23.5 mm by 25 mm (587.5 mm2), and is fabricated in a 28 nm CMOS process with copper interconnects.

SPARC64 X+

The SPARC64 X+ is an enhanced SPARC64 X processor announced in 2013. It features minor improvements to the core organization, and a higher 3.5 GHz clock frequency obtained through better circuit design and layout. It contained 2.99 billion transistors, measured 24 mm by 25 mm (600 mm2), and is fabricated in the same process as the SPARC64 X. On 8 April 2014, 3.7 GHz speed-binned parts became available in response to the introduction of new Xeon E5 and E7 models by Intel; and the impending introduction of the POWER8 by IBM.

SPARC64 XIfx

Fujitsu introduced the SPARC64 XIfx in August 2014 at the Hot Chips symposium. It is used in the Fujitsu PRIMEHPC FX100 supercomputer, which succeeded the PRIMEHPC FX10. The XIfx operates at 2.2 GHz and has a peak performance of 1.1 TFLOPS. It consists of 3.75 billion transistors and is fabricated by the Taiwan Semiconductor Manufacturing Company in its 20 nm high-κ metal gate (HKMG) process. The Microprocessor Report estimated the die to have an area of 500 mm2; and a typical power consumption of 200 W.

The XIfx has 34 cores, 32 of which are compute cores used to run user applications, and 2 assistant cores used to run the operating system and other system services. The delegation of user applications and operating system to dedicated cores improves performance by ensuring that the private caches of the compute cores are not shared with or disrupted by non-application instructions and data. The 34 cores are further organized into two Core Memory Groups (CMGs), each consisting of 16 compute cores and 1 assistant core sharing a 12 MB L2 unified cache. The division of the cores into CMGs enabled 34 cores to be integrated on a single die by easing the implementation of cache coherence and avoiding the need for the L2 cache to be shared between 34 cores. The two CMGs share the memory through a ccNUMA organization.

The XIfx core was based on the SPARC64 X+ with organizational improvements. The XIfx implements an improved version of the HPC-ACE extensions (HPC-ACE2), which doubled the width of the SIMD units to 256 bits and added new SIMD instructions. Compared to the SPARC64 IXfx, the XIfx has an improvement of a factor of 3.2 for double precision and 6.1 for single precision. To complement the increased width of the SIMD units, the L1 cache bandwidth was increased to 4.4 TB/s.

Improvements to the SoC organization were to the memory and interconnect interfaces. The integrated memory controllers were replaced with four Hybrid Memory Cube (HMC) interfaces for decreased memory latency and improved memory bandwidth. According to the Microprocessor Report, the IXfx was the first processor to use HMCs. The XIfx is connected to 32 GB of memory provided by eight 4 GB HMCs. The HMCs are 16-lane versions, with each lane operating at 15 Gbit/s. Each CMG has two HMC interfaces, and each HMC interface is connected to two HMCs via its own ports. Each CMG has 240 GB/s (120 GB/s in and 120 GB/s out) of memory bandwidth.

Future

Fujitsu announced at the International Supercomputing Conference in June 2016 that its future exascale supercomputer will feature processors of its own design that implement the ARMv8 architecture. The A64FX will implement extensions to the ARMv8 architecture, equivalent to HPC-ACE2, that Fujitsu is developing with ARM Holdings.

SPARC64 XII

SPARC64 XII was launched in 2017 with Fujitsu's SPARC M12 servers. It nominally features 12 cores, but similar to IBM's POWER9 that was launched the same year, each of the twelve cores consists of two separate pipelines. The only resources shared between SPARC64 XII core's pipelines are the TLB, L1 instruction cache and L2 cache, and as a result the single-threaded performance is almost unchanged from SPARC64 X. SPARC64 XII operates at up to 4.25 GHz base frequency and 4.35 GHz boost frequency. The size of the chip is 25.8mm × 30.8mm (795mm2), containing 5.45 billion transistors made on TSMC's 20 nm process. Each of the two pipelines of a core can fetch 8 instructions, decode 4 instructions and execute 6 instructions per cycle, and supports 4 SMT threads (for 96 threads per CPU). Each pipeline has a private 32 KB 4-way L1 data cache, and two pipelines share a 64 KB 4-way associative L1 instruction cache and a 512 KB 16-way associative L2 cache. SPARC64 XII is Fujitsu's first SPARC CPU with L3 cache (32 MB 16-way). The number of 8-lane PCIe 3.0 ports has been doubled to 4 per chip. Memory speed has been increased by 50% to 2400 MT/s, bringing the theoretical combined bandwidth of the eight DDR4 channels of the chip to 153 GB/s, and the capacity per CPU is up to 1.5 TB across 24 slots. Two CPUs are connected in a Building Block, and up to 16 Building Blocks can be connected to create a 32-CPU server with up to 48 TB of memory capacity.