<SYSTEM_TASK:>
Solve the following problem using Python, implementing the functions described below, one line at a time
<END_TASK>
<USER_TASK:>
Description:
 def parse_all_reduce_spec(all_reduce_spec):
 """Parse all_reduce_spec. Args: all_reduce_spec: a string specifying a combination of all-reduce algorithms to apply for gradient reduction. Returns: a list of AllReduceSpecTuple. Raises: ValueError: all_reduce_spec is not well-formed. An all_reduce_spec has BNF form: int ::= positive whole number g_int ::= int[KkMGT]? alg_spec ::= alg | alg#int range_spec ::= alg_spec | alg_spec/alg_spec spec ::= range_spec | range_spec:g_int:range_spec Not all syntactically correct specifications are supported. Examples of supported all_reduce_spec strings, with semantics explained: "xring" == apply ring all-reduce to all tensors "xring#2" == apply ring all-reduce to all tensors, using two simultaneous transfer rings, each operating on 1/2 of each tensor. "nccl" == apply NCCL all-reduce to all tensors (only works within a single worker process where all devices are GPUs) "nccl/xring" == apply NCCL all-reduce to all tensors within each worker to produce at least one full-reduced (locally) value, then apply ring all-reduce to one such value from each worker, then apply NCCL broadcast to propagate those globally reduced values back to every device within each worker. "pscpu" == Shuffle reduce using worker CPUs as the gather devices: each distributed tensor is reduced by copying all instances to one of the worker CPUs, computing the reduction there, then copying back to each participating device. Tensor reductions are assigned to specific CPUs round-robin. "psgpu#4" == Arrange all GPUs across all workers into groups of 4. Each distributed tensor is shuffle reduced against one such group of 4 GPUs, selected round-robin. That is, each tensor is split across 4 shards for the reduction. "pscpu:2k:pscpu#2:64k:xring" == Apply single-shard pscpu to tensors of size <= 2048 elements, apply 2-shard pscpu to tensors up to size 64k elements, apply xring to larger tensors. "pscpu/pscpu#2" == Use shuffle gather to locally reduce each tensor on the worker's CPU, then use 2-shard shuffle to reduce those locally reduced tensors across workers (on the worker CPUs), then scatter the globally reduced values locally from each worker CPU. """

    range_parts = all_reduce_spec.split(":") + ["-1"]
    if len(range_parts) % 2:
        raise ValueError(
            "all_reduce_spec not well formed: %s" % all_reduce_spec)
    limit = 0
    spec = []
    alg = None
    shards = 1
    for i, range_part in enumerate(range_parts):
        if i % 2 == 1:
            try:
                limit = parse_general_int(range_part)
                spec.append(
                    AllReduceSpecTuple(alg=alg, shards=shards, limit=limit))
            except ValueError:
                raise ValueError(
                    "all_reduce_spec (%s) contains non-integer range %s" %
                    (all_reduce_spec, range_part))
        else:
            alg = range_part
            alg_parts = range_part.split("#")
            alg = alg_parts[0]
            if len(alg_parts) > 1:
                try:
                    shards = int(alg_parts[1])
                except ValueError:
                    raise ValueError(
                        "all_reduce_spec (%s) contains non-integer "
                        "shards %s" % all_reduce_spec, alg_parts[1])
            else:
                shards = 1
            if alg not in [
                    "nccl", "nccl/xring", "nccl/rechd", "nccl/pscpu", "xring",
                    "pscpu", "psgpu", "pscpu/pscpu"
            ]:
                raise ValueError("all_reduce_spec (%s) contains invalid alg %s"
                                 % (all_reduce_spec, alg))
    return spec