From a dict `docs` with document ID -> terms/tokens list mapping, generate an array of vocabulary (i.e.
    unique terms of the whole corpus `docs`), an array of document labels (i.e. document IDs), a dict
    with document ID -> document terms array mapping and a sum of the number of unique terms per document.

    The returned variable `sum_uniques_per_doc` tells us how many elements will be non-zero in a DTM which
    will be created later. Hence this is the allocation size for the sparse DTM.

    This function provides the input for create_sparse_dtm().

    Return a tuple with:
    - np.array of vocabulary
    - np.array of document names
    - dict with mapping: document name -> np.array of document terms
    - overall sum of unique terms per document (allocation size for the sparse DTM)

def get_vocab_and_terms(docs):
    """
    From a dict `docs` with document ID -> terms/tokens list mapping, generate an array of vocabulary (i.e.
    unique terms of the whole corpus `docs`), an array of document labels (i.e. document IDs), a dict
    with document ID -> document terms array mapping and a sum of the number of unique terms per document.

    The returned variable `sum_uniques_per_doc` tells us how many elements will be non-zero in a DTM which
    will be created later. Hence this is the allocation size for the sparse DTM.

    This function provides the input for create_sparse_dtm().

    Return a tuple with:
    - np.array of vocabulary
    - np.array of document names
    - dict with mapping: document name -> np.array of document terms
    - overall sum of unique terms per document (allocation size for the sparse DTM)
    """
    vocab = set()
    docs_terms = {}
    sum_uniques_per_doc = 0

    # go through the documents
    for i, (doc_label, terms) in enumerate(docs.items()):
        terms_arr = np.array(terms)
        docs_terms[doc_label] = terms_arr

        # update the vocab set
        terms_unique = set(terms)
        vocab |= terms_unique

        # update the sum of unique values per document
        sum_uniques_per_doc += len(terms_unique)

    doc_labels = docs_terms.keys()

    return np.fromiter(vocab, dtype='<U%d' % max(map(len, vocab)), count=len(vocab)), \
           np.fromiter(doc_labels, dtype='<U%d' % max(map(len, doc_labels)), count=len(doc_labels)),\
           docs_terms, sum_uniques_per_doc