"""
Time Series Pattern Extraction


This project is the implementation of finding the maneuver patterns of time-series data. Note that this framework can
be used not only for lane change sequences, but also for all kinds of time-series sequences.

A two-step framework is established to realize our extraction task. In the first step, the time series segmentation
algorithms were employed separately to segment the lane change series into blocks, when samples in each block remain
the common trend. In the second step, the segmented blocks were clustered using various modified LDAs. Note that the
original algorithm of LDA is not used here because the model is unable to handle floating input (the expected input of
LDA is an enumeration feature).

Since there already exists many time-series segmentation algorithms with elegant mathematical structure, we utilized
these models in our framework:

    1. BMASS: Bayesian model-based agglomerative sequence segmentation, created by <NAME>. In this model, 
              the greedy based agglomerate search method (like hierarchical clustering) was utilized to find the 
              locally optimal marginal likelihood of the conjugate distribution parameters of the given samples.

    2. BMOSS: Bayesian model-based online sequence segmentation, also created by <NAME>. It has the same 
              assumption of conjugate distribution as BMASS, but it adjusts the searching method to the forward-backward 
              algorithm (like HMM).

    3. HDP-HSMM: hierarchical Dirichlet process hidden semi-Markov model, created by @Mat<NAME>. When a HMM was used 
              to fit time series, we can regard adjacent substrings with the same hidden state as the inferred segments, 
              thus we can also use HMM to segment time-series sequences. Given that the original HMM often had frequent 
              state switches, the use of hidden semi-Markov model (HSMM) allows the hidden state to remain in the same 
              state for a longer period of time. In addition, as a Bayesian nonparametric prior, the hierarchical 
              Dirichlet process (HDP) can be added into the HMM-based models to automatically infer the number of hidden 
              states of HMM, thus reducing excessive manual intervention.

Note that BMASS is implemented using MATLAB, thus it requires users to install MATLAB. In addition, the implementation
of HDP-HSMM uses the pyhsmm package, which has a number of unsupported problems of underlying C++ code in Windows
environment. Thus, we strongly recommend to run this program in a Linux environment.

In the second step, we employed two extended LDA models to achieve segments clustering:

    1. GMM-LDA: Gaussian mixture model-latent Dirichlet allocation, which is our implementation in our framework. The
              clusters learning from input samples were generated by using GMM, and the cluster label of each sample
              is treated as the input of LDA. In this implementation, we improved the implementation of GMM created by
              @<NAME>, since the program is easy to produce a series of numerical calculation errors in
              large sample data.

    2. mLDA: Multimodal latent Dirichlet allocation, created by @Naka Tomo. The topic-word distribution of each feature 
              in the sample is modeled separately to obtain a juxtaposed distribution model. Some of the construction 
              and implementation details of the program have been tweaked to make the model more robust and retain 
              richer inferential information in the model output.

In this framework, we default that the feature dimensions of all data are the same, and use the same instance of a model
to learn characteristic of samples from different data sources. Readers can change the algorithms of each step according
to their own needs.

For any problem, please contact us on <EMAIL>

@<NAME>
"""

import os
import argparse
import sys 
sys.path.append("utils/")
sys.path.append("segmentation/bmass/")
sys.path.append("segmentation/bmoss/")
sys.path.append("segmentation/hdp_hsmm/")
sys.path.append("clustering/gmm_lda/")
sys.path.append("clustering/gmm_lda/gmm/")
sys.path.append("clustering/gmm_lda/lda/")
sys.path.append("clustering/mlda/")


from segmentation.bmass.bmass_main import bmass_infer
from segmentation.bmoss.bmoss_main import bmoss_infer
from segmentation.hdp_hsmm.hdp_hsmm_main import hdp_hsmm_infer
from clustering.gmm_lda.gmm_lda_main import gmmlda_infer
from clustering.mlda.mlda_main import mlda_infer



def check_dirs(input_dir, seg_output_dir, lda_output_dir):
    for path in (input_dir, seg_output_dir, lda_output_dir):
        try: os.makedirs(path)
        except: pass
    
    
def pattern_detection():
    parser = argparse.ArgumentParser()
    parser.add_argument('--input_dir', type=str, default='data/')
    parser.add_argument('--seg_output_dir', type=str, default='output/seg/')
    parser.add_argument('--lda_output_dir', type=str, default='output/lda/')
    parser.add_argument('--seg_alg', type=str, default='bmoss')
    parser.add_argument('--lda_alg', type=str, default='gmm-lda')
    parser.add_argument('--cluster_num', type=int, default=5)
    args = parser.parse_args()
    check_dirs(args.input_dir, args.seg_output_dir, args.lda_output_dir)
    
    
    # segmentation
    print('data segmentation...')
    if args.seg_alg == 'bmass':
        bmass_infer(args.input_dir, args.seg_output_dir)
    elif args.seg_alg == 'bmoss':
        bmoss_infer(args.input_dir, args.seg_output_dir)
    elif args.seg_alg == 'hdp-hsmm':
        hdp_hsmm_infer(args.input_dir, args.seg_output_dir)
    else:
        Exception('ERROR: The specified segmentation model was not found')
    print('done')
    
    
    # clustering
    print('segments clustering...')
    if args.lda_alg == 'gmm-lda':
        gmmlda_infer(args.input_dir, args.seg_output_dir, args.lda_output_dir)
    elif args.lda_alg == 'mlda':
        mlda_infer(args.input_dir, args.seg_output_dir, args.lda_output_dir)
    else:
        Exception('ERROR: The specified clustering model was not found')
    print('done')



if __name__ == '__main__':
    pattern_detection()

"""
Time Series Pattern Extraction


This project is the implementation of finding the maneuver patterns of time-series data. Note that this framework can
be used not only for lane change sequences, but also for all kinds of time-series sequences.

A two-step framework is established to realize our extraction task. In the first step, the time series segmentation
algorithms were employed separately to segment the lane change series into blocks, when samples in each block remain
the common trend. In the second step, the segmented blocks were clustered using various modified LDAs. Note that the
original algorithm of LDA is not used here because the model is unable to handle floating input (the expected input of
LDA is an enumeration feature).

Since there already exists many time-series segmentation algorithms with elegant mathematical structure, we utilized
these models in our framework:

    1. BMASS: Bayesian model-based agglomerative sequence segmentation, created by <NAME>. In this model, 
              the greedy based agglomerate search method (like hierarchical clustering) was utilized to find the 
              locally optimal marginal likelihood of the conjugate distribution parameters of the given samples.

    2. BMOSS: Bayesian model-based online sequence segmentation, also created by <NAME>. It has the same 
              assumption of conjugate distribution as BMASS, but it adjusts the searching method to the forward-backward 
              algorithm (like HMM).

    3. HDP-HSMM: hierarchical Dirichlet process hidden semi-Markov model, created by @Mat<NAME>. When a HMM was used 
              to fit time series, we can regard adjacent substrings with the same hidden state as the inferred segments, 
              thus we can also use HMM to segment time-series sequences. Given that the original HMM often had frequent 
              state switches, the use of hidden semi-Markov model (HSMM) allows the hidden state to remain in the same 
              state for a longer period of time. In addition, as a Bayesian nonparametric prior, the hierarchical 
              Dirichlet process (HDP) can be added into the HMM-based models to automatically infer the number of hidden 
              states of HMM, thus reducing excessive manual intervention.

Note that BMASS is implemented using MATLAB, thus it requires users to install MATLAB. In addition, the implementation
of HDP-HSMM uses the pyhsmm package, which has a number of unsupported problems of underlying C++ code in Windows
environment. Thus, we strongly recommend to run this program in a Linux environment.

In the second step, we employed two extended LDA models to achieve segments clustering:

    1. GMM-LDA: Gaussian mixture model-latent Dirichlet allocation, which is our implementation in our framework. The
              clusters learning from input samples were generated by using GMM, and the cluster label of each sample
              is treated as the input of LDA. In this implementation, we improved the implementation of GMM created by
              @<NAME>, since the program is easy to produce a series of numerical calculation errors in
              large sample data.

    2. mLDA: Multimodal latent Dirichlet allocation, created by @Naka Tomo. The topic-word distribution of each feature 
              in the sample is modeled separately to obtain a juxtaposed distribution model. Some of the construction 
              and implementation details of the program have been tweaked to make the model more robust and retain 
              richer inferential information in the model output.

In this framework, we default that the feature dimensions of all data are the same, and use the same instance of a model
to learn characteristic of samples from different data sources. Readers can change the algorithms of each step according
to their own needs.

For any problem, please contact us on <EMAIL>

@<NAME>
"""

import os
import argparse
import sys 
sys.path.append("utils/")
sys.path.append("segmentation/bmass/")
sys.path.append("segmentation/bmoss/")
sys.path.append("segmentation/hdp_hsmm/")
sys.path.append("clustering/gmm_lda/")
sys.path.append("clustering/gmm_lda/gmm/")
sys.path.append("clustering/gmm_lda/lda/")
sys.path.append("clustering/mlda/")


from segmentation.bmass.bmass_main import bmass_infer
from segmentation.bmoss.bmoss_main import bmoss_infer
from segmentation.hdp_hsmm.hdp_hsmm_main import hdp_hsmm_infer
from clustering.gmm_lda.gmm_lda_main import gmmlda_infer
from clustering.mlda.mlda_main import mlda_infer



def check_dirs(input_dir, seg_output_dir, lda_output_dir):
    for path in (input_dir, seg_output_dir, lda_output_dir):
        try: os.makedirs(path)
        except: pass
    
    
def pattern_detection():
    parser = argparse.ArgumentParser()
    parser.add_argument('--input_dir', type=str, default='data/')
    parser.add_argument('--seg_output_dir', type=str, default='output/seg/')
    parser.add_argument('--lda_output_dir', type=str, default='output/lda/')
    parser.add_argument('--seg_alg', type=str, default='bmoss')
    parser.add_argument('--lda_alg', type=str, default='gmm-lda')
    parser.add_argument('--cluster_num', type=int, default=5)
    args = parser.parse_args()
    check_dirs(args.input_dir, args.seg_output_dir, args.lda_output_dir)
    
    
    # segmentation
    print('data segmentation...')
    if args.seg_alg == 'bmass':
        bmass_infer(args.input_dir, args.seg_output_dir)
    elif args.seg_alg == 'bmoss':
        bmoss_infer(args.input_dir, args.seg_output_dir)
    elif args.seg_alg == 'hdp-hsmm':
        hdp_hsmm_infer(args.input_dir, args.seg_output_dir)
    else:
        Exception('ERROR: The specified segmentation model was not found')
    print('done')
    
    
    # clustering
    print('segments clustering...')
    if args.lda_alg == 'gmm-lda':
        gmmlda_infer(args.input_dir, args.seg_output_dir, args.lda_output_dir)
    elif args.lda_alg == 'mlda':
        mlda_infer(args.input_dir, args.seg_output_dir, args.lda_output_dir)
    else:
        Exception('ERROR: The specified clustering model was not found')
    print('done')



if __name__ == '__main__':
    pattern_detection()

Time Series Pattern Extraction
 
 
 This project is the implementation of finding the maneuver patterns of time-series data. Note that this framework can
 be used not only for lane change sequences, but also for all kinds of time-series sequences.
 
 A two-step framework is established to realize our extraction task. In the first step, the time series segmentation
 algorithms were employed separately to segment the lane change series into blocks, when samples in each block remain
 the common trend. In the second step, the segmented blocks were clustered using various modified LDAs. Note that the
 original algorithm of LDA is not used here because the model is unable to handle floating input (the expected input of
 LDA is an enumeration feature).
 
 Since there already exists many time-series segmentation algorithms with elegant mathematical structure, we utilized
 these models in our framework:
 
     1. BMASS: Bayesian model-based agglomerative sequence segmentation, created by <NAME>. In this model, 
               the greedy based agglomerate search method (like hierarchical clustering) was utilized to find the 
               locally optimal marginal likelihood of the conjugate distribution parameters of the given samples.
 
     2. BMOSS: Bayesian model-based online sequence segmentation, also created by <NAME>. It has the same 
               assumption of conjugate distribution as BMASS, but it adjusts the searching method to the forward-backward 
               algorithm (like HMM).
 
     3. HDP-HSMM: hierarchical Dirichlet process hidden semi-Markov model, created by @Mat<NAME>. When a HMM was used 
               to fit time series, we can regard adjacent substrings with the same hidden state as the inferred segments, 
               thus we can also use HMM to segment time-series sequences. Given that the original HMM often had frequent 
               state switches, the use of hidden semi-Markov model (HSMM) allows the hidden state to remain in the same 
               state for a longer period of time. In addition, as a Bayesian nonparametric prior, the hierarchical 
               Dirichlet process (HDP) can be added into the HMM-based models to automatically infer the number of hidden 
               states of HMM, thus reducing excessive manual intervention.
 
 Note that BMASS is implemented using MATLAB, thus it requires users to install MATLAB. In addition, the implementation
 of HDP-HSMM uses the pyhsmm package, which has a number of unsupported problems of underlying C++ code in Windows
 environment. Thus, we strongly recommend to run this program in a Linux environment.
 
 In the second step, we employed two extended LDA models to achieve segments clustering:
 
     1. GMM-LDA: Gaussian mixture model-latent Dirichlet allocation, which is our implementation in our framework. The
               clusters learning from input samples were generated by using GMM, and the cluster label of each sample
               is treated as the input of LDA. In this implementation, we improved the implementation of GMM created by
               @<NAME>, since the program is easy to produce a series of numerical calculation errors in
               large sample data.
 
     2. mLDA: Multimodal latent Dirichlet allocation, created by @Naka Tomo. The topic-word distribution of each feature 
               in the sample is modeled separately to obtain a juxtaposed distribution model. Some of the construction 
               and implementation details of the program have been tweaked to make the model more robust and retain 
               richer inferential information in the model output.
 
 In this framework, we default that the feature dimensions of all data are the same, and use the same instance of a model
 to learn characteristic of samples from different data sources. Readers can change the algorithms of each step according
 to their own needs.
 
 For any problem, please contact us on <EMAIL>
 
 @<NAME> # segmentation # clustering