> For the complete documentation index, see [llms.txt](https://blog.ruipan.xyz/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://blog.ruipan.xyz/earlier-readings-and-notes/cs759-hpc-course-notes/lecture-21-openmp-work-sharing..md).

# Lecture 21: OpenMP Work Sharing.

## Lecture Summary

* Last time: OpenMP nested parallelism, work sharing (for loops, sections)
* Today
  * OpenMP: nested parallelism, work sharing (tasks)
  * OpenMP: variable scoping, synchronization, loose ends

## OpenMP Work Sharing

### omp sections

Ending example

![](/files/-MWSTn2X37MJRJRVI-tw)

![](/files/-MWSTr_VW11tfM0YFYDo)

### omp tasks

* Pros: Allows parallelization of irregular problems
  * Unbounded loops
  * Recursive algorithms
  * Producer/consumer
* Cons: Relatively tricky to deal with & introduce some overhead 
* Motivations
  * OpenMP started to be tailored for large array-based applications
  * For example, the parallelization of a dynamic list traversal cannot be done in OpenMP for a long time
  * Storing pointers to list elements in an array: High overhead for array construction (not easy to parallelize)
  * Using single nowait inside a parallel region: High cost of the single construct. Also, each thread needs to traverse the entire list to determine if another thread has already processed that element
* Who does what and when?
  * The developer
    * Uses a pragma to specify where & what the tasks are
    * Ensures that there are no dependencies (that is, tasks can be executed independently)
  * The OpenMP runtime system
    * Generates a new task whenever a thread encounters a task construct
    * Decide the moment of execution (can be immediate or delayed)
* Definition: A task is a specific instance/combo of executable code along w/ its data environment (the shared & private data manipulated by the task) and ICV (internal control variables: thread scheduling and environment variables, typically associated with OpenMP)
* Synchronization issues. Solution: use task barriers (`#pragma omp barrier`, `#pragma omp taskwait`) to ensure the completion of tasks.

![](/files/-MWSWArBj6G2eAqlx9TX)

![](/files/-MWSWfDdLK1Hrm_64t0x)

![](/files/-MWSXBEAWGJLJYw73vYk)

## OpenMP Variable Scoping Issues

* Threads have access to a pool of memory that is shared
* Threads can also have private data
* Basic rule: Any variable declared prior to a parallel region is shared in that parallel region
* The private clause reproduces for each thread variables declared private in the pragma
* There are also OpenMP variables treated as private by default
  * Stack (local) variables in functions called from within parallel regions
  * Loop iteration variables
  * Automatic variables within a statement block
* When in doubt, always explicitly indicate something to be private
* firstprivate: Specifies that each thread should have its own instance of a variable. Moreover, the variable is initializes using the value of the variable of the same name from the master thread
  * Usage: #pragma omp parallel num\_threads(4) firstprivate(i) 
* lastprivate: The enclosing context's version of the variable is set equal to the private version of whichever thread executes the final iteration of the work-sharing construct (for or section)
* Data scoping is a common source of errors in OpenMP. It is the programmer's responsibility to make sure data dependencies do not lead to race conditions

![](/files/-MWSZIRLN6GvZpV7Wtu8)

![Example of what's being shared and what's not](/files/-MWSf4FMi0EBydTdwIaK)

## OpenMP Synchronization

* Explicit barrier: #pragma omp barrier
* Implicit barriers: parallel, for, single, sections
* Unnecessary barriers hurt performance and can be removed with the nowait clause (applicable to for, single, sections)

![The nowait clause](/files/-MWScsVyj_zlTjnb_axv)

![The critical construct: prevents race conditions and protects access to shared, modifiable data](/files/-MWScy5Eon9-swpWx3bD)

![The critical construct in action. Note that naming the critical construct RES\_lock is optional but highly recommended](/files/-MWSdeBK4I5YQxxeN8OF)