lvwerra HF staff commited on
Commit
894c434
·
verified ·
1 Parent(s): e7323a7
assets/images/5D_nutshell_tp_sp.svg CHANGED
assets/images/5d_nutshell_cp.svg CHANGED
assets/images/5d_nutshell_ep.svg CHANGED
dist/assets/images/5D_nutshell_tp_sp.svg CHANGED
dist/assets/images/5d_nutshell_cp.svg CHANGED
dist/assets/images/5d_nutshell_ep.svg CHANGED
dist/index.html CHANGED
@@ -1648,7 +1648,7 @@
1648
 
1649
  <p><strong>Tensor Parallelism</strong> (with Sequence Parallelism) is naturally complementary and can be combined with both Pipeline Parallelism and ZeRO-3 as it relies on the distributive property of matrix multiplications which allows weights and activations to be sharded and computed independently before being combined.</p>
1650
 
1651
- <img alt="TP & SP diagram" src="/assets/images/5D_nutshell_tp_sp.svg" style="width: 1000px; max-width: none;" />
1652
  <!-- <p><img alt="image.png" src="/assets/images/placeholder.png" /></p> -->
1653
 
1654
 
 
1648
 
1649
  <p><strong>Tensor Parallelism</strong> (with Sequence Parallelism) is naturally complementary and can be combined with both Pipeline Parallelism and ZeRO-3 as it relies on the distributive property of matrix multiplications which allows weights and activations to be sharded and computed independently before being combined.</p>
1650
 
1651
+ <img alt="TP & SP diagram" src="/assets/images/5d_nutshell_tp_sp.svg" style="width: 1000px; max-width: none;" />
1652
  <!-- <p><img alt="image.png" src="/assets/images/placeholder.png" /></p> -->
1653
 
1654
 
src/index.html CHANGED
@@ -1648,7 +1648,7 @@
1648
 
1649
  <p><strong>Tensor Parallelism</strong> (with Sequence Parallelism) is naturally complementary and can be combined with both Pipeline Parallelism and ZeRO-3 as it relies on the distributive property of matrix multiplications which allows weights and activations to be sharded and computed independently before being combined.</p>
1650
 
1651
- <img alt="TP & SP diagram" src="/assets/images/5D_nutshell_tp_sp.svg" style="width: 1000px; max-width: none;" />
1652
  <!-- <p><img alt="image.png" src="/assets/images/placeholder.png" /></p> -->
1653
 
1654
 
 
1648
 
1649
  <p><strong>Tensor Parallelism</strong> (with Sequence Parallelism) is naturally complementary and can be combined with both Pipeline Parallelism and ZeRO-3 as it relies on the distributive property of matrix multiplications which allows weights and activations to be sharded and computed independently before being combined.</p>
1650
 
1651
+ <img alt="TP & SP diagram" src="/assets/images/5d_nutshell_tp_sp.svg" style="width: 1000px; max-width: none;" />
1652
  <!-- <p><img alt="image.png" src="/assets/images/placeholder.png" /></p> -->
1653
 
1654