Parallel and distributed

computing systems

Unit I
Topics to be discussed

● Eras of computing
● Parallel vs distributed computing
● Elements of parallel computing
● Elements of distributed computing
● Technologies for distributed computing
Eras of Computing
Eras of computing

The two fundamental and dominant models of computing are sequential and parallel.

The four key elements of computing developed during these eras are architectures,
compilers, applications, and problem-solving environments.

Every aspect of this era underwent a three-phase process: research and development
(R&D), commercialization, and commoditization
Sequential vs parallel era timeline

Figure 1: Shows the evolution of computing paradigms

Parallel vs Distributed computing
Parallel vs. distributed computing

The terms parallel computing and distributed computing are often used interchangeably.

The term parallel implies a tightly coupled system, whereas distributed refers to a wider class of
system, including those that are tightly coupled.

The term parallel computing refers to a model in which the computation is divided among several
processors sharing the same memory.

The architecture of a parallel computing system is often characterized by the homogeneity of

components: each processor is of the same type and it has the same capability as the others. The
shared memory has a single address space, which is accessible to all the processors. Parallel
programs are then broken down into several units of execution that can be allocated to different
processors and can communicate with each other by means of the shared memory.
Distributed computing

The term distributed computing encompasses any architecture or system that allows the
computation to be broken down into units and executed concurrently on different
computing elements, whether these are processors on different nodes, processors on the
same computer, or cores within the same processor.
Elements of Parallel Computing
 ● Reason for evolution of parallel
computing
● What is parallel processing?
Elements of Parallel ● Hardware architectures for parallel
computing ●
processing (Flynn’s classification)
Approaches to parallel programming
● Levels of parallelism
● Laws of caution
Elements of parallel computing

Reason for evolution of parallel computing

It is now clear that silicon-based processor chips are reaching their physical limits.
Processing speed is constrained by the speed of light, and the density of transistors
packaged in a processor is constrained by thermodynamic limitations. A viable
solution to overcome this limitation is to connect multiple processors working in
coordination with each other to solve “Grand Challenge” problems. The first steps in this
direction led to the development of parallel computing, which encompasses techniques,
architectures, and systems for performing multiple activities in parallel.
 Parallel computing

What is parallel processing?

Processing of multiple tasks simultaneously on multiple processors is called parallel processing.

The parallel program consists of multiple active processes (tasks) simultaneously solving a given
problem. A given task is divided into multiple subtasks using a divide-and-conquer technique, and
each subtask is processed on a different central processing unit (CPU). Programming on a
multiprocessor system using the divide-and-conquer technique is called parallel programming.

Computational requirements are ever increasing in the areas of both scientific and business
computing.

Sequential architectures are reaching physical limitations as they are constrained by the speed
of light and thermodynamics laws.

Hardware improvements in pipelining, superscalar, and the like are non-scalable and require
sophisticated compiler technology.
Hardware architectures for parallel processing (Flynn’s
classification)

1. Single-instruction, single-data (SISD) systems

2. Single-instruction, multiple-data (SIMD) systems

3. Multiple-instruction, single-data (MISD) systems

4. Multiple-instruction, multiple-data (MIMD) systems

Hardware architectures for parallel processing
Single-instruction, single-data (SISD) systems

An SISD computing system is a uniprocessor machine capable of executing a single instruction,

which operates on a single data stream

In SISD, machine instructions are processed sequentially; hence computers adopting this model are
popularly called sequential computers.

Example: IBM PC, Macintosh, and workstations

Hardware architectures for parallel processing

Single-instruction, multiple-data (SIMD) systems

An SIMD computing system is a multiprocessor machine capable of executing the same

instruction on all the CPUs but operating on different data streams. Machines based on an
SIMD model are well suited to scientific computing since they involve lots of vector and matrix
operations. For instance, statements such as C i = Ai + Bi can be passed to all the processing
elements (PEs); organized data elements of vectors A and B can be divided into multiple sets (N-
sets for N PE systems);

Dominant representative SIMD systems

are Cray’s vector processing machine

Hardware architectures for parallel processing
Multiple-instruction, single-data (MISD) systems

An MISD computing system is a multiprocessor machine capable of executing different

instructions on different PEs but all of them operating on the same data set For
instance, statements such as y = sin(x) +cos(x) + tan(x) perform different operations on the
same data set. Machines built using the MISD model are not useful in most of the
applications; Not manufactured commercially.
Hardware architectures for parallel processing
Multiple-instruction, multiple-data (MIMD) systems

An MIMD computing system is a multiprocessor machine capable of executing multiple instructions

on multiple data sets . Each PE in the MIMD model has separate instruction and data streams; hence
machines built using this model are well suited to any kind of application. Unlike SIMD and MISD
machines, PEs in MIMD machines work asynchronously.

MIMD machines are broadly categorized into shared-memory MIMD and distributed-memory MIMD
based on the way PEs are coupled to the main memory.
Approaches to parallel programming

A sequential program is one that runs on a single processor and has a single line of
control. To make many processors collectively work on a single program, the program
must be divided into smaller independent chunks so that each processor can work on
separate chunks of the problem. The program decomposed in this way is a parallel
program.

A wide variety of parallel programming approaches are available.

● Data parallelism
● Process parallelism
● Farmer-and-worker model
Parallel programming approaches

These three models are all suitable for task-level parallelism.

In the case of data parallelism, the divide-and-conquer technique is used to split
data into multiple sets, and each data set is processed on different PEs using the
same instruction.
In the case of process parallelism, a given operation has multiple (but distinct)
activities that can be processed on multiple processors.
In the case of the farmer-and-worker model, a job distribution approach is used:
one processor is configured as master and all other remaining PEs are
designated as slaves; the master assigns jobs to slave PEs and, on completion,
they inform the master, which in turn collects results.
Levels of parallelism

Levels of parallelism are decided based on the lumps of code (grain size) that can be a potential candidate for
parallelism. Parallelism within an application can be detected at several levels:

• Large grain (or task level)

• Medium grain (or control level)

• Fine grain (data level)

• Very fine grain (multiple-instruction issue)

Figure 2 : Demonstration of the different levels of parallelism

Laws of caution

Here are two important guidelines to take into

account:

• Speed of computation is proportional to the

square root of system cost; they never increase
linearly. Therefore, the faster a system
becomes, the more expensive it is to increase its
speed

• Speed by a parallel computer increases as the

logarithm of the number of processors (i.e., y
=klog(N)).
Figure 3a: speed vs cost curve
Figure 3b: speed vs #processor curve
Elements of Distributed
Computing
 ● Distributed computing and its components
● Architectural styles for distributed
computing
Elements of ○ Data-centred
Distributed ○ Data flow
○ Virtual machine
Computing ○ Call and return
○ Independent component
● System architectural styles
○ Client-Server model
○ Peer to Peer model
● Models for inter-process communication
● RPC- Remote Procedure Calling
● Distributed object frameworks
Distributed computing and its components

A distributed system is a collection of independent computers that appears to its users

as a single coherent system

OR

A distributed system is one in which components located at networked computers

communicate and coordinate their actions only by passing messages.

Components of a distributed system

A distributed system is the result of the interaction of several components that traverse
the entire computing stack from hardware to software. It emerges from the
collaboration of several elements that—by working together—give users the illusion of
a single coherent system.
 Elements of distributed computing

At the very bottom layer, computer and

network hardware constitute the physical
infrastructure; these components are directly
managed by the operating system, which
provides the basic services for interprocess
communication (IPC), process scheduling
and management, and resource management
in terms of file system and local devices.
Taken together these two layers become the
platform on top of which specialized software
is deployed to turn a set of networked
computers into a distributed system

Figure 4: Layered scheme of Distributed systems

Architectural styles for distributed computing
Data centred architectures

These architectures identify the data as the fundamental element of the software system, and
access to shared data is the core characteristic of the data- centered architectures.

The repository architectural style :It is characterized by two main components: the central
data structure, which represents the current state of the system, and a collection of
independent components, which operate on the central data. The ways in which the
independent components interact with the central data structure can be very heterogeneous.
The blackboard architectural style is characterized by three main components:
● Knowledge sources. These are the entities that update the knowledge base that is maintained
in the blackboard.
● Blackboard. This represents the data structure that is shared among the knowledge sources
and stores the knowledge base of the application.
● Control. The control is the collection of triggers and procedures that govern the interaction
with the blackboard and update the status of the knowledge base.
Data-flow architectures- Batch Sequential

In the case of data-flow architectures, it is the availability of data that controls the
computation. With respect to the data-centered styles, in which the access to data is the core
feature, data-flow styles explicitly incorporate the pattern of data flow, since their design is
determined by an orderly motion of data from component to component, which is the
form of communication between them.
Batch Sequential Style. The batch sequential style is characterized by an ordered sequence
of separate programs executing one after the other. These programs are chained together
by providing as input for the next program the output generated by the last program after its
completion, which is most likely in the form of a file.
Data-flow architectures- Pipe and filter

Pipe-and-Filter Style. The pipe-and-filter style is a variation of the previous style for
expressing the activity of a software system as sequence of data transformations. Each
component of the processing chain is called a filter, and the connection between one filter
and the next is represented by a data stream.

In comparison to the batch sequential style, data is processed incrementally and each filter
processes the data as soon as it is available on the input stream. As soon as one filter
produces a consumable amount of data, the next filter can start its processing. Filters
generally do not have state, know the identity of neither the previous nor the next filter,
and they are connected with in-memory data structures such as first-in/first-out (FIFO)
buffers or other structures.

Example a unix command pipe- cat <filename>| grep <pattern> | wc -l

Virtual machine architectures

The virtual machine class of architectural styles is characterized by the presence of an

abstract execution environment (generally referred as a virtual machine) that simulates
features that are not available in the hardware or software. Applications and systems are
implemented on top of this layer and become portable over different hardware and software
environments as long as there is an implementation of the virtual machine they interface with.
Very popular examples within this category are rule-based systems, interpreters, and
command-language processors.

Rule-Based Style. This architecture is characterized by representing the abstract execution

environment as an inference engine. Programs are expressed in the form of rules or
predicates that hold true. The input data for applications is generally represented by a set of
assertions or facts that the inference engine uses to activate rules or to apply predicates, thus
transforming data. The output can either be the product of the rule activation or a set of
assertions that holds true for the given input data. The set of rules or predicates identifies the
knowledge base that can be queried to infer properties about the system.
Virtual machine architectures

Interpreter Style. The core feature of the interpreter style is the presence of an engine that is
used to interpret a pseudo-program expressed in a format acceptable for the interpreter.
The interpretation of the pseudo-program constitutes the execution of the program
itself.

Systems modeled according to this style exhibit four main components: the interpretation
engine that executes the core activity of this style, an internal memory that contains the
pseudo-code to be interpreted, a representation of the current state of the engine, and a
representation of the current state of the program being executed. This model is quite
useful in designing virtual machines for high-level programming (Java, C#) and scripting
languages (Awk, PERL, and so on).
Call & return architectures- Top down style

All systems that are organised into components mostly connected together by method
calls( or a chain of method calls). We can differentiate these by the way the system is
structured and how methods are invoked: top-down style, object-oriented style, and layered
style.

Top-Down Style. : Systems developed according to this style are composed of one large
main program that accomplishes its tasks by invoking subprograms or procedures. The
components in this style are procedures and subprograms, and connections are method calls
or invocation. The calling program passes information with parameters and receives data
from return values or parameters. Method calls can also extend beyond the boundary of a
single process by leveraging techniques for remote method invocation RMI, such as
remote procedure call (RPC) and all its descendants.
Call & return architectures - Object Oriented style

Object-Oriented Style : This architectural style is designed and implemented for leveraging
the abstractions of object-oriented programming (OOP). Systems are specified in terms of
classes and implemented in terms of objects. Classes define the type of components by
specifying the data that represent their state and the operations that can be done over these
data. One of the main advantages over the top-down style is that there is a coupling
between data and operations used to manipulate them.

Layered Style : The layered system style allows the design and implementation of software
systems in terms of layers, which provide a different level of abstraction of the system. Each
layer generally operates with at most two layers: the one that provides a lower abstraction
layer and the one that provides a higher abstraction layer. Specific protocols and interfaces
define how adjacent layers interact.
Independent components

Communicating Processes. In this architectural style, components are represented by

independent processes that leverage IPC facilities for coordination management. This is
an abstraction that is quite suitable to modeling distributed systems that, being distributed
over a network of computing nodes, are necessarily composed of several concurrent
processes. Each of the processes provides other processes with services and can leverage the
services exposed by the other processes.

Event Systems. In this architectural style, the components of the system are loosely coupled
and connected. In addition to exposing operations for data and state manipulation, each
component also publishes (or announces) a collection of events with which other components
can register. In general, other components provide a call-back that will be executed when
the event is activated.
System architectural styles

This covers physical organization of components and processes over a distributed

infrastructure. They provide a set of reference models for the deployment of such systems
and help engineers not only have a common vocabulary in describing the physical layout
of systems but also quickly identify the major advantages and drawbacks of a given
deployment and whether it is applicable for a specific class of applications.
There are two fundamental reference styles:
1. Client/server
2. Peer-to-peer.
System architectural styles- Client/Server

1. Client/server : the client/server model features two major components: a server and a
client. These two components interact with each other through a network connection using
a given protocol. The communication is unidirectional: The client issues a request to the
server, and after processing the request the server returns a response. There could be
multiple client components issuing requests to a server that is passively waiting for them.
Hence, the important operations in the client-server paradigm are request, accept (client
side), and listen and response (server side).

Thin-client model. In this model, the load of data processing and transformation is put on
the server side, and the client has a light implementation that is mostly concerned with
retrieving and returning the data it is being asked for, with no considerable further
processing.
System architectural styles - Client/Server

Figure 5: Two tier, three tier and N-tier client server architectures
 System architectural styles- Peer to Peer

2. Peer-to-peer
Introduces a symmetric architecture in which all the components, called peers, play the same
role and incorporate both client and server capabilities of the client/server model. More
precisely, each peer acts as a server when it processes requests from other peers and as a client
when it issues requests to other peers. With respect to the client/ server model that partitions
the responsibilities of the IPC between server and clients, the peer-to-peer model attributes the
same responsibilities to each component.

Figure 6: Peer to Peer model

Models for Inter-process communication(IPC)
Distributed systems are composed of a collection of concurrent processes interacting
with each other by means of a network connection. Therefore, IPC is a fundamental aspect
of distributed systems design and implementation. IPC is used to either exchange data and
information or coordinate the activity of processes. IPC is what ties together the different
components of a distributed system, thus making them act as a single system.
1. Message-based communication : This involves many important topics such as
● Message passing- MPI ( Message passing interface MPI) and OpenMP
● Remote procedure call (RPC).,
● Distributed objects.- CORBA (Common Object Request Broker Architecture),
COM,DCOM, Java RMI, .NET Remoting
● Distributed agents and active objects,
● Web services. - ( WSDL, HTTP, SOAP, Representational State Transfer (REST))
 Models for interprocess communication( IPC)
2. Models for message-based communication

● Point-to-point message model This model organizes the communication among single components.
Each message is sent from one component to another, and there is a direct addressing to identify the
message receiver. the message is sent directly to the receiver and processed at the time of reception. In
the latter, the receiver maintains a message queue in which the messages received are placed for later
processing.
● Publish-and-subscribe message model . This is based on notification among components. There are
two major roles: the publisher and the subscriber. The former provides facilities for the latter to
register its interest in a specific topic or event. Specific conditions holding true on the publisher side
can trigger the creation of messages that are attached to a specific event.
○ Push strategy. In this case it is the responsibility of the publisher to notify all the subscribers—
for example, with a method invocation.
○ Pull strategy. In this case the publisher simply makes available the message for a specific
event, and it is responsibility of the subscribers to check whether there are messages on the
events that are registered,
● Request-reply message model - For each message sent by a process a reply message is received.
Technologies for Distributed
Computing
 RPC- Remote Procedure Calling

RPC is the fundamental abstraction enabling the execution of procedures on client’s request. The
called procedure and calling procedure may be on the same system or they may be on different
systems in a network.

The system is based on a client/server model. The server process maintains a registry of all the
available procedures that can be remotely invoked and listens for requests from clients that specify
which procedure to invoke, together with the values of the parameters required by the procedure. the
calling process thread remains blocked until the procedure on the server process has completed its
execution and the result (if any) is returned to the client.

Marshaling, identifies the process of converting parameter and return values into a form that is more
suitable to be transported over a network through a sequence of bytes. The term unmarshaling refers
to the opposite procedure. Marshaling and unmarshaling are performed by the RPC runtime
infrastructure, and the client and server user code does not necessarily have to perform these tasks.
 RPC- Remote Procedure Calling

The RPC runtime is not only responsible for parameter packing and unpacking but also for handling
the request-reply interaction that happens between the client and the server process in a completely
transparent manner.

System leveraging RPC for IPC consists of the following steps: • Design and implementation of the
server procedures that will be exposed for remote invocation.• Registration of remote procedures with
the RPC server on the node where they will be made available. • Design and implementation of the
client code that invokes the remote procedure(s).
RPC- Remote Procedure Calling

Figure 7: RPC reference model

Distributed object frameworks
Distributed object frameworks extend object-oriented programming systems by allowing
objects to be distributed across a heterogeneous network and provide facilities so that they can
coherently act as though they were in the same address space.
The common interaction pattern is the following:
1. The server process maintains a registry of active objects that are made available to other
processes. According to the specific implementation, active objects can be published using
interface definitions or class definitions.
2. The client process, by using a given addressing scheme, obtains a reference to the active
remote object. This reference is represented by a pointer to an instance that is of a shared type of
interface and class definition.
3. The client process invokes the methods on the active object by calling them through the
reference previously obtained. Parameters and return values are marshaled as happens in the
case of RPC.
Distributed object frameworks

Figure 8: Distributed object processing model

 Service oriented computing

1. What is a Service ?

A service encapsulates a software component that provides a set of coherent and related
functionalities that can be reused and integrated into bigger and more complex applications.
service is a general abstraction that encompasses several different implementations using
different technologies and protocols. Services may have the following characteristics like
boundaries of services are explicit, services are autonomous, services share schema and
contracts (not class or interface definitions) services compatibility is determined based on
policy.
Service oriented computing

2. Service oriented architecture

SOA is an architectural style supporting service orientation. SOA encompasses a set of design
principles that structure system development and provide means for integrating components into a
coherent and decentralized system. SOA-based computing packages functionalities into a set of
interoperable services, which can be integrated into different software systems belonging to separate
business domains. interoperability, standards, and service contracts play a fundamental role.
In particular, the following guiding principles, which characterize SOA platforms, are winning
features within an enterprise context: Standardized service contract, Loose coupling. Abstraction,
Reusability, Autonomy, Lack of state, Discoverability, Composability. The Service-Oriented
Modeling Framework (SOMF), the Organization for Advancement of Structured Information
Standards (OASIS) are the prime organizations of importance.
 Technologies for distributed computing - Web Service

Web services are the prominent technology for implementing SOA systems and applications.
1. They allow for interoperability across different platforms and programming languages.
2. They are based on well-known and vendor-independent standards such as HTTP, SOAP,
XML, and WSDL.
3. They provide an intuitive and simple way to connect heterogeneous software systems,
enabling the quick composition of services in a distributed environment.
4. They provide the features required by enterprise business applications to be used in an
industrial environment. They define facilities for enabling service discovery, which
allows system architects to more efficiently compose SOA applications, and service
metering to assess whether a specific service complies with the contract between the
service provider and the service consumer.
 Technologies for distributed computing - Web Service

Using as a basis the object-oriented abstraction, a Web service exposes a set of operations that
can be invoked by leveraging Internet-based protocols. Method operations support parameters
and return values in the form of complex and simple types. The semantics for invoking Web
service methods is expressed through interoperable standards such as XML and WSDL, which
also provide a complete framework for expressing simple and complex types in a platform-
independent manner. Web services are made accessible by being hosted in a Web server;
therefore, HTTP is the most popular transport protocol used for interacting with Web services.
Technologies for distributed computing - Web Service - WSDL

The service description document, expressed by means of Web Service Definition Language
(WSDL), can be either uploaded to a global registry or attached as a metadata to the
service itself.
Service consumers can look up and discover services in global catalogs using Universal
Description Discovery and Integration (UDDI)
The Web service description document allows service consumers to automatically
generate clients for the given service and embed them in their existing application.
Web services are now extremely popular, so bindings exist for any mainstream
programming language in the form of libraries or development support tools. This makes
the use of Web services seamless and straightforward with respect to technologies such as
CORBA, Java RMI, DCOM/COM, .NET remoting that require much more integration
efforts.
 Web Service

Simple Object Access Protocol (SOAP), an XML-based

language for exchanging structured information in a platform-
independent manner, constitutes the protocol used for Web
service method invocation.
SOAP is considered an application layer protocol that leverages
the transport level, most commonly HTTP, for IPC.
The envelope defines the boundaries of the SOAP message. The
header is optional and contains relevant information on how to
process the message.
In addition, it contains information such as routing and delivery
settings, authentication and authorization assertions, and
transaction contexts. The body contains the actual message to be
processed.
Figure 11: SOAP messages for web service invocation a SOAP message used to invoke a Web service method that retrieves the price of a given
stock and the corresponding reply.
 Technologies for distributed computing

Web Service

Figure 9: Web Service interaction reference scenario

Technologies for distributed computing - Web Service -
REST
In a RESTful system, a client sends a request over HTTP using the standard HTTP methods
(PUT, GET, POST, and DELETE), and the server issues a response that includes the
representation of the resource. By relying on this minimal support, it is possible to provide
whatever it needed to replace the basic and most important functionality provided by SOAP,
which is method invocation. The GET, PUT, POST, and DELETE methods constitute a
minimal set of operations for retrieving, adding, modifying, and deleting data.
Together with an appropriate URI organization to identify resources, all the atomic
operations required by a Web service are implemented. The content of data is still transmitted
using XML as part of the HTTP content, but the additional markup required by SOAP is
removed. For this reason, REST represents a lightweight alternative to SOAP, which
works effectively in contexts where additional aspects beyond those manageable through
HTTP are absent.
Web service

Figure 10: Web service technologies stack

Technologies for distributed computing - WSDL, Web
2.0, AJAX

Web Service Description Language (WSDL) is an XML-based language for the

description of Web services. It is used to define the interface of a Web service in terms of
methods to be called and types and structures of the required parameters and return values
Web 2.0 provide and contribute to enrich and empower Web applications and then SOA-
based systems.These fall under the names of Asynchronous JavaScript and XML (AJAX),
JavaScript Standard Object Notation (JSON), and others.
AJAX is a conceptual framework based on JavaScript and XML that enables asynchronous
behavior in Web applications by leveraging the computing capabilities of modern Web
browsers.
*****Parallel and Distributed systems*****