K1.2 Overview Hardware Architectures

HPC computer architectures are parallel computer architectures. A parallel computer is built out of

• Compute units.
• Main memory.
• A high-speed network.

• elementary processing elements like CPUs, GPUs, many-core architectures
• vector systems, and FPGAs
• the NUMA architecture used for symmetric multiprocessing systems where the memory access time depends on the memory location relative to the processor
• network demands for HPC systems (e.g. high bandwidth and low latency)
• typical network architectures used for HPC systems, like fast Ethernet (1 or 10 Gbit) or InfiniBand

• Comprehend that in traditional CPUs - although CPU stands for Central Processing Unit - there is no central, i.e. single, processing unit any more because today all CPUs have multiple compute cores which all have the same functionality
• Comprehend that GPUs (Graphical Processing Units) or GPGPUs (General Purpose Graphical Processing Units) were originally used for image processing and displaying images on screens before people started to utilize the computing power of GPUs for other purposes
• Comprehend that FPGAs (Field-Programmable Gate Arrays) are devices that have configurable hardware and configurations are specified by hardware description languages
• Comprehend that FPGAs are interesting if one uses them to implement hardware features that are not available in CPUs or GPUs (e.g. low precision arithmetic that needs only a few bits)
• Comprehend that Vector units are successors of vector computers (i.e. the first generation of supercomputers) and that they are supposed to provide higher memory bandwidth than CPUs
• Comprehend that at an abstract level the high-speed network connects compute units and main memory which leads to three main parallel computer architectures
  • Shared Memory where all compute units can directly access the whole main memory
  • Distributed memory where individual computers are connected with a network
  • NUMA (Non-Uniform Memory Access) combines properties from shared and distributed memory systems, because at the hardware level a NUMA system resembles a distributed memory
• Comprehend that in general, the effort for programming parallel applications for distributed systems is higher than for shared memory systems
• parallelization techniques at the instruction level of a processing element (e.g. pipelining, SIMD processing)
• advanced instruction sets that improve parallelization (e.g., AVX-512)
• hybrid approaches, e.g. combining CPUs with GPUs or FPGAs
• typical network topologies and architectures used for HPC systems, like fat trees based on switched fabrics using e.g. fast Ethernet (1 or 10 Gbit) or InfiniBand
• special or application-specific hardware (e.g. TPUs)