通信原语 Collective communication primitive

Broadcast

Scatter

Gather

Reduce

All-to-All

Reduce-Scatter

All-Gather

All-Reduce

all-reduce = reduce-scatter + all-gather

Ring AllReduce

ring all-reduce 每一步每个节点都参加了，有点是充分利用带宽，但是缺点是小数据包时延迟突显。

2D-Torus

优点：步骤更少。缺点：每步节点都不相同，连接无法复用。

---

Reference

Collective operation - wikipedia