CS-735/835: Introduction to Parallel and Distributed Programming

(The syllabus can be downloaded as a pdf.)

Next offered: Spring 2021

Catalog description

Programming with multiple processes and threads on distributed and parallel computer systems. Introduces programming tools and techniques for building applications on such platforms. Course requirements consist primarily of programming assignments. Prereq: CS 520.

Overview

Parallel and distributed computing have become mainstream and demand to be covered in both the undergraduate and graduate curricula. Revised curricula from the NSF (2012)—“we must now prepare [students] for the very dynamic world of parallel and distributed computing”—, the ACM (2013)—“parallel and distributed computing has moved from a largely elective topic to become more of a core component of undergraduate computing curricula”—, and ABET (2018)—“the curriculum requirements specify […] exposure to […] parallel and distributed computing”—have given a more prominent place to these topics.

This course is an introduction to parallel and distributed computing from a programmer’s standpoint (see CS-745/845 for a more abstract/theoretical viewpoint). It covers fundamental concepts (e.g., race conditions, atomicity, locking) and a few more advanced ones (e.g., non-sequentially consistent memory models, non-blocking algorithms, functional concurrent programming) through examples. Students complete several programming assignments and a multi-threaded client-server project. The course uses Java, its java.util.concurrent library and its support for Remote Method Invocation, but similar concepts are found in most modern programming environments.

The course does not assume prior knowledge of Java threads. However, it assumes that students can read and write Java code, understand basic concepts of object-oriented programming and can implement, test and debug a medium-sized Java application. Students should also be at least familiar with the concept of thread (or process) and parallelism (through an operating systems course, for instance). Coursework is identical for graduate (CS-835) and undergraduate (CS-735) students, but graduate students need to achieve at least a B^– grade to pass the course.

Attributes

• This course is one of the CS electives designated as implementation intensive.

Evaluation

Five programming assignments (50%), one project (20%), quizzes (15%) and one exam (15%).

Minimum score for each grade: A: 90, A^–: 87, B⁺: 83, B: 80, B^–: 77, C⁺: 73, C: 70, C^–: 67, D⁺: 63, D: 60, D^–: 57.

ABET Outcomes

• Outcome 2: students write concurrent and distributed programs using modern programming constructs; these programs are thoroughly evaluated for correctness and performance.
• Outcome 6: students learn standard algorithms and patterns from concurrent and distributed programming.

ABET Curriculum

• Curriculum 1: programming techniques; testing and debugging skills; standard tools (version control, unit testing and IDE).
• Curriculum 4: a) blocking and non-blocking synchronization algorithms; c) programming language support for concurrent and distributed computing (threads, thread pools, futures, synchronizers, remote method invocation, …); d) design and performance evaluation of multi-threaded software.
• Curriculum 6: a) performance of multi-threaded applications on multi-core systems; e) introduction to concurrent and distributed programming.

Topics

• Basic concepts of concurrent and distributed programming:
  • parallelism, threads, scheduling, nondeterminism
  • blocking operations, deadlocks, livelocks, timeouts
  • atomicity, synchronization, data races
  • memory models, memory barriers, visibility
  • serialization, remote procedure call
• Client-server paradigm:
  • sockets, socket servers
  • Remote Method Invocation
• Synchronizers:
  • locks, semaphores, latches, barriers, conditions, futures
  • blocking and bounded queues
• Concurrent data structures:
  • synchronized vs concurrent, parallel operations
  • atomic operations, client-side locking
  • non-blocking algorithms
• Executor services:
  • tasks, futures, thread pools, failures, cancellations, shutdown
  • parallelizing recursive computations
  • timers and scheduled executors
• Engineering concurrent programs:
  • correctness, thread-safety
  • state, invariants, sharing, locality, mutability, immutability
  • decomposition, tasks, execution services
  • thread-safe and concurrent data structures
  • interrupts, poison pills, cancellation, termination, abortion, non-interruptible blocking
  • non-blocking functional concurrency (futures as monads)
  • actors
• Performance:
  • Amdahl’s law
  • contention, thread creation, parking/unparking, I/O
  • throughput, responsiveness
  • resource management
• Programming language support (Java):
  • synchronized blocks, final fields, volatile fields
  • wait, notify, notifyAll
  • java.lang.ThreadLocal
  • java.util.concurrent (usage and implementation)
  • java.net, java.io.Serializable, java.rmi
• Advanced topics:
  • compare-and-set, non-blocking algorithms
  • Java Memory Model
  • java.util.concurrent.lock
  • java.util.concurrent.atomic
  • building custom synchronizers

Textbooks

Required:

• Brian Goetz et al. Java Concurrency in practice, Addison-Wesley, 2006. ISBN: 978-0321349606.

Additional (for students new to Java):

• Joshua Bloch. Effective Java, 2nd edition, Addison-Wesley, 2008. ISBN 0-321-35668-3.