from the city of Vienna: Notes on SPMD architecture A II

Local and Distributed memory processing.

I‘ll stick at building ships

George Steinbrenner

The object of this post is to describe the relationship between the resources of the nodes and the local and distributed memory processing.

--------------------

We will call algorithm input data unit (input data unit) to the data object on which the algorithm works. This data unit can not be split, it has to be processed as a whole. The algorithm’s result of processing an input data unit is an algorithm output data unit (output data unit). The data units characteristics include the shape (linear or rectangular), size (number of elements) and data type (integer, floating-point, etc.).

Latency and Throughput are parameters which characterize the performance of a processing machine or node and that were advanced in Introduction. In terms of data units, node Latency stands for the time delay between an input data unit and the correspondent output data unit -this time is the necessary time to process the input data unit and to produce the output data unit (processing time)- and node Throughput stands for the number of data units that the node is able to process per time unit (input and output Throughputs can be different).

Hereinafter, we will refer to the memory and Throughput of the node as the resources of the node.

If total resources of the set of processing nodes are not enough to perform the required processing in the required time, and the architecture is scalable, the number of processing nodes can be increased until these resources are enough. In real life, the increment of the number of processing nodes is limited by hardware constraints.

Regarding the resources of the node, we will consider two cases. In the first case, the node has enough memory but not enough Throughput to perform the required processing, so the processing is performed on the nodes’ local memory (local memory processing). In the second case, the node does not have enough memory or Throughput, so the processing is performed on the distributed memory in the processing nodes set (distributed memory processing).

In the writing of this article, Florence + The Machine - You've Got The Love (Live at the Rivolli Ballroom) have collaborated in an involuntary but decisive way.