# Expectation Maximisation

$
\newcommand{\x}{\mathbf{x}}
\newcommand{\y}{\mathbf{y}}
\newcommand{\z}{\mathbf{z}}
\newcommand{\class}{\mathcal{C}}
\newcommand{\mean}{\boldsymbol{\mu}}
\newcommand{\cov}{\boldsymbol{\Sigma}}
\newcommand{\icov}{\boldsymbol{\Sigma}^{-1}}
\newcommand{\normdist}{\mathcal{N}}
\newcommand{\tp}{\top}
\newcommand{\params}{\mathbf{\theta}}
$

| | |
|--|--|
| **Names** | *Anna Berger, Ashwin Sadananda Bhat* |
| **Group** | *AML 25* |

In this lab we will build a probabilistic classifier and estimate its parameters. In order to illustrate how the complexity of learning changes when we do not observe all the variables, we will start with the model depicted in figure 1. We will then switch to the graphical model depicted in figure 2 and compare the two
models’ performance.

Read through the complete exercise handout before starting your implementation: the hints at the end, and the instructions for the subsequent exercises will help you for the current one

## Gaussian Distribution

  
**Figure 1**: Bayesian classifier with Normally distributed observations. The class labels C are discrete variables in one-of-K format. The observations x are multivariate and continuous, and their distribution is described with a Gaussian probability density function. 


In this model, we want to find the posterior probability
$p(\class_k|\x_n)$ for all data points $\x_n$. We get this from Bayes' rule

$$
  p(\class_k|\x_n) = \frac{ p(\x_n|\class_k) p(\class_k) }{
    \sum_j p(\x_n|\class_j) p(\class_j) },
$$

where we can see from the graph that the data elements are
conditionally independent given the class and we model the
class-conditional probability with a normal distribution

$$
  p(\x_n|\class_k) = \normdist(\x_n;\mean_k,\cov_k)\\
  = \frac{1}{(2\pi)^{d/2}|\cov|^{1/2}} \exp -\frac{1}{2}
    (\x_n-\mean_k)^\tp \icov_k (\x_n-\mean_k)
$$



Load the data in `target.mat`, which consists
of two classes `A` and `B`. Split the data in train and test
sets and plot the training data, in order to get a feel for it. Just by looking at the data, do you think the model will perform well?

**Question 1 [10 marks]**
Do the training, which consists of finding the prior probabilities for each class, $p(\class_k)$, and the parameters of the class-conditional distributions, the means $\mean_k$ and covariance matrices $\cov_k$. Now use Bayes' rule to compute the most likely class label for each data point in the test set.  Include the resulting confusion matrix and error rate in
you report.




```python
%matplotlib inline
import numpy as np
import scipy.stats as st
import scipy.ndimage as ndim
import matplotlib.pyplot as plt
import operator
import pylab
import time
from IPython import display

from matplotlib.patches import Ellipse

pylab.rcParams['figure.figsize'] = (10.0, 8.0)


def normPDF(data,mu,sigma):
    """My implementation of the Gaussian distribution. Returns the density, given mean mu and covariance sigma,
    for every data point (row) of data"""
    N,nDim = data.shape
    res = np.zeros(N)
    
    icov = np.linalg.inv(sigma)
    covdet = np.linalg.det(sigma)
    denom = np.sqrt((2*np.pi)**nDim * covdet)
    
    for i in range(N):
        diff = data[i,:]-mu
        res[i] = np.exp(-.5 * np.dot(diff, np.dot(icov,diff.T))) / denom
    
    return res


# Load in the data
D = np.load("target.npz")
data = D['data']
labels = D['labels']
A=data[labels==0,:]
B=data[labels==1,:]

nA = A.shape[0]
nB = B.shape[0]
N = nA + nB

data = np.vstack((A,B))
plt.scatter(A[:,0],A[:,1])
plt.scatter(B[:,0],B[:,1],20,'red')

labels = np.concatenate((np.zeros(nA),np.ones(nB)))

perm = np.random.permutation(N)
datap = data[perm,:]
labelp = labels[perm]

trainData = datap[:4*N/5,:]
trainLab  = labelp[:4*N/5]
testData  = datap[4*N/5:,:]
testLab   = labelp[4*N/5:]
```


```python
# [ANSQ1] Do not modify this line
# Answer Q1 is below
```

SHORT DISCUSSION FOR Q1

From the picture we can see assume that this model will not perform well as data points have the shape of concetric circles and the data points of one class aren't placed together in the space but separated by the data points of another class and it's hard to imagine such Gaussian distributions which can handle this problem.


```python
from sklearn.metrics import accuracy_score, confusion_matrix
```


```python
def norm_param(arr):
    return np.mean(arr, axis = 0), np.cov(np.transpose(arr))


def posterior(data, mu, sigma):
    p_x_0 = normPDF(data, mu[0], sigma[0])
    p_x_1 = normPDF(data, mu[1], sigma[1])
    return p_x_0 * priors[0] / (p_x_0 * priors[0] + p_x_1 * priors[1]) 


def classify(data, mu, sigma):
    posterior_c_0 = posterior(data, mu, sigma)
    predictions = [0 if prob > 0.5 else 1 for prob in posterior_c_0]
    return predictions
```


```python
classes = 2
priors = np.zeros(classes)
mu = np.zeros((classes, classes))
sigma = np.zeros((2, classes, classes))

for i in range(classes):
    class_data = trainData[trainLab == i]
    priors[i] = float(len(class_data)) / len(trainLab)
    mu[i], sigma[i] = norm_param(class_data)
    
y_pred = classify(testData, mu, sigma)
```


```python
# visualisation of trained models

fig = plt.gcf()
plt.clf()        
ax = fig.add_subplot(111, aspect='equal')
plt.scatter(data[:,0], data[:,1])
for m in range(classes):
    [val,vect] = np.linalg.eig(sigma[m])
    e = Ellipse(xy=mu[m], width=2*np.sqrt(val[0]), height=2*np.sqrt(val[1]), angle=180*np.arctan2(vect[0,0],vect[1,0])/np.pi)
    e.set_facecolor([1,0,0])
    ax.add_artist(e)
plt.show()
```


```python
acc = accuracy_score(testLab, y_pred)
conf_matrix = confusion_matrix(testLab, y_pred)
print "error_rate = ", 1 - acc
print "conf_matrix = \n", conf_matrix
```

    error_rate =  0.44625
    conf_matrix = 
    [[244 157]
     [200 199]]


SHORT DISCUSSION

As we assumed, the accuracy of the proposed model is low due to the fact that one Gaussian distribution per class can't represent the actual distribution of the datapoints of the class well.

## Mixture of Gaussians

 
**Figure 2**: Graphical model of a Bayesian classifier with observations described by a Mixture of Gaussians
(MOG). Each class has a number of mixture elements, which are each modelling part of the data. Which
datapoints belong to which mixture element is controlled by the latent variable $\z$.


In this model, the class-conditional distributions $p(\x|\class_k)$ are
modelled by a **mixture** of Gaussians rather than a single Gaussian, so that the total data-likelihood
\begin{align}
  p(\x_n,\class_k) &= p(\class_k) \sum_j p(\x_n|z_j,\class_k)p(z_j|\class_k),
\end{align}
where
\begin{align}
  p(\x_n|z_j,\class_k) = \normdist(\x_n;\mean_{jk},\cov_{jk}).
\end{align}

Since the latent variables controlling the mixture components are not observed, we need to optimise the log-likelihood with the EM
algorithm. You are provided with some code to help you pull this off:


```python
# [ANSQ2] [ANSQ3] Do not modify this line

def invalid(sigma):
    """Is this a valid covariance matrix?"""
    if np.isfinite(sigma).all():
        return np.linalg.cond(sigma)>1e3
    return True

class MoG:
    def __init__(self,data,nMix=2):
        self.pi = np.ones(nMix) / nMix
        self.nPoints, self.nDim = data.shape
        self.nMix = nMix
        self.data = data
        mu = np.mean(data)
        self.mu = []
        self.sigma = []
        self.resp = np.zeros((self.pi.shape[0], self.data.shape[0]))
        for i in range(nMix):
            self.mu.append(mu + .1 * i * np.random.randn(self.nDim))
            self.sigma.append(np.eye(self.nDim))
            
    def em(self,maxIter=100,minInc=1e-5):
        llhist = np.zeros(maxIter)
        for iteration in range(maxIter):
            resp, ll = self.expectation()
            llhist[iteration]=ll
            if iteration>0:
                delta = llhist[iteration] - llhist[iteration-1]
                print "Iteration",iteration,":",delta
#                 print llhist[:iteration+1],llhist[iteration], llhist[iteration-1],delta
                if delta < minInc:
                    if delta < 0.:
                        print "WARNING: log-likelihood decreased in iteration",iteration
                    llhist = llhist[:iteration]
                    break
            self.maximisation(resp)
            self.plot()
        return llhist
            
    def expectation(self):
        """Compute p(z|x,\theta)"""
        # Answer to Q2
        for i in range(self.nMix):
            self.resp[i] = self.pi[i] * normPDF(self.data, self.mu[i], self.sigma[i])

        self.resp = self.resp / sum(self.resp)  
        return self.resp, sum(np.log(self.likelihood(self.data)))
    
    def maximisation(self,resp):
        # Answer to Q3
        count = np.sum(resp, axis = 1)
        self.pi = count / self.nPoints
            
        for i in range(self.nMix):
            self.mu[i] = np.dot(self.data.T, resp[i]) / count[i]
    
        for i in range(self.nMix):
            sigma_new = np.dot((self.data - self.mu[i]).T * resp[i], self.data - self.mu[i]) / count[i]
            
        # if values are too big then reinitialising
            if invalid(sigma_new):
                print "INVALID"
                mean = np.mean(self.data) + .1*np.random.randn(self.nDim)
                self.sigma[i] = np.eye(self.nDim)
            else:
                self.sigma[i] = sigma_new
                
            
    def plot(self):
        fig = plt.gcf()
        display.display(fig)
        display.clear_output(wait=True)
        plt.clf()        
        ax = fig.add_subplot(111, aspect='equal')
        plt.scatter(self.data[:,0],self.data[:,1])
        for m in range(self.nMix):
            [val,vect] = np.linalg.eig(self.sigma[m])
            e = Ellipse(xy=self.mu[m], width=2*np.sqrt(val[0]), height=2*np.sqrt(val[1]), angle=180*np.arctan2(vect[0,0],vect[1,0])/np.pi)
                    
            e.set_facecolor([1,0,0])
            ax.add_artist(e)
        plt.show()
        time.sleep(.1)
        
    def likelihood(self,data):
        """Compute the marginal likelihood of the data given the mixture model, p(x|\theta), for
        all elements in `data`"""
        p = np.zeros(data.shape[0])
        for m in range(self.nMix):
            p += self.pi[m] * normPDF(data,self.mu[m],self.sigma[m])
        return p
```

SHORT DISCUSSION 

We noticed that the final position of the ellipses really depends on the initialisation.  Multiple runs may result in different position: rotation approximately 90 degrees. It can be explained by the symmetry of training data points or by the fact that this data can't be approximated well by this number of components in Gaussian mixture.

However, the relative position of three (there actually three of them, one is just covered by another) ellipses stays approximately the same from run to run.


```python
mog = MoG(A,3)
llhist = mog.em()
plt.figure()
plt.plot(llhist)   
```

The code above requires a number of methods in order to work,
implementing the intialisation, E-step and M-step of the EM
algorithm. Those methods are:
 
* `(resp, ll) = expectation()` which implements the E-step
  of the algorithm. It uses the data, and the current parameter values. The function does not change
  the parameters but computes `resp`, a matrix that contains the
  probability of each mixture component given the data
  $p(\z|\x,\params)$, and `ll`, the log-likelihood of the data set
  under the mixture model. This is **Question 2 [10 marks]**
* `maximisation(resp)` wich uses the data and takes as inputs the corresponding responsibilities. The method updates the class by storing the updated parameter values. This is **Question 3 [10 marks]**

Include comments in your answer, explaining
how you compute the responsibilities $p(\z|\x,\params)$ and what the
updates are that maximise the complete expected log-likelihood.


**Question 4 [10 marks]**
Use the same data in `target.npz`, and optimise the model parameters for the classifier with
mixture-of-Gaussians distributions. Use the same train and test sets
as for exercise1 and compare the performance of both
classifiers. Discuss the differences you found.

*Optional Extra:* The contour plots of the distributions do not
reflect the priors ($p(\z|\class_l)$) and therefore do not give you an
exact idea of the probability density under the model. In order to
visualise that, you could plot the probability density under both
models in three dimensions.


### Hints
1. Include a check in your M-step to avoid singularities. You can
  check this by computing `c = np.linalg.cond(SIGMA)`: if c becomes larger
  than a given threshold (say, $10^{10}$), do not perform the
  parameter update and reset the mean and covariance. The provided function,
  `invalid(SIGMA)`, does that check for you.
1. The responsibilities $p(\z|\x,\params)$ are stored in the matrix
  `Q` in the E-step of the algorithm. You know the size of this
  matrix in advance: if you create a matrix of zeros of that size and
  fill in the values, your code will be much more efficient than if
  your grow the matrix as you compute the values.


```python
# [ANSQ4] Do not modify this line
# Answer to Q4

def training_mixture(data, k):
    mog = MoG(data, k)
    llhist = mog.em()
    return mog

k0 = 9
mog_0 = training_mixture(trainData[trainLab == 0], k0)
```


```python
k1 = 9
mog_1 = training_mixture(trainData[trainLab == 1], k1)
```


```python
def posterior(data):
    p_x_0 = mog_0.likelihood(data)
    p_x_1 = mog_1.likelihood(data)
    return p_x_0 * priors[0] / (p_x_0 * priors[0] + p_x_1 * priors[1]) 


def classify(data):
    posterior_c_0 = posterior(data)
    predictions = [0 if prob > 0.5 else 1 for prob in posterior_c_0]
    return predictions
```


```python
y_pred = classify(testData)
acc = accuracy_score(testLab, y_pred)
conf_matrix = confusion_matrix(testLab, y_pred)
print "error_rate on the test set = ", 1 - acc
print "conf_matrix = \n", conf_matrix
```

    error_rate on the test set =  0.02875
    conf_matrix = 
    [[393   8]
     [ 15 384]]


The mixture model allows us to represent the distribution of our data points much more accurate than a single Gaussian distribution per class. However, the training step now takes much longer to compute.

## Discussion of the Implementation 

**Question 5 [8 marks]**

Discuss the following with your teammate and include your thoughts in
your report:
1. What is a sensible way to initialise the parameters of your
  model?
1. How many components should you try? Having plotted the data, can
  you guess what number of components should be optimal? How does your
  classification accuracy evolve as the number of components change?
1. Do the results depend on the initialisation?
1. Why is hint 1 important?


[ANSQ5] Do not modify this line

Answer to Q5

1) As the EM algorithm usually takes more iterations to converge than the K-means algorithm and each cycle requires significantly more computation, it is common to find initialization for a Gaussian mixture model throught running K-means algorithm. The covariance matrices can be initialized to the sample covariances of the clusters, the same goes for the means - the means of the clusters. The mixing coefficients can be set to the fractions of data points assigned to respective clusters.

2) 
* The optimal choice of k should strike a balance between maximum compression of the data using a single cluster, and maximum accuracy by assigning each data point to its own cluster. 
* Yes, the plot may give us some hints on the number components: we can see how many relatively close groups of points there are and therefore, choose that number of groups as number of components. But we need to limit ourselves: as we grow the number of components, we need more data to estimate the parameters well and avoid overfitting. 
* When we increase the number of components, the classification accuracy increases as the number of components grows up. But at some point, increasing the number of components doesn't result in any sufficient benefit in accuracy on the test set and we start overfitting on the train set: the accuracy on train set continues growing (we can come to the cricial case of number of components = number of points) but accuracy on test set starts decreasing.


3) Yes, the results depend on the intitialisation: in general, there multiple local maxima of log likelihood function and the EM algorithm is not guaranteed to find the largest of them.

4) It is important because we want our algorithm to converge as fast as possible, and EM algorithm can take multiple iterations to converge, that's why we try to choose sensible parameters not very far from actual values.

## Log-Probabilities

As we have seen in the class, one problem that occurs often when doing probabilistic modelling, is that since probabilities lay in the range $[0,\dots,1]$, the joint probability of multiple variables becomes very small very fast. This is a problem, because computers can only represent numbers with a finite precision and too small a probability will result in an
underflow, hence introducing numerical errors in an otherwise mathematically correct algorithm.

The solution that is typically used is then to use the logarithm of
probabilities, rather than the probabilities themselves. The products
of probabilities then become sums of log-probabilities:

$$
  \ln[p(a)p(b)] = \ln p(a) + \ln p(b)
$$

However a solution must be found when sums are involved. How do we
compute

$$
  \ln[p(a) + p(b)]
$$

when $\ln p(a)$ and $\ln p(b)$ are known? The solution is not to
do the following:

$$
  \ln[p(a) + p(b)] = \ln[ \exp (\ln p(a)) + \exp(\ln p(b)) ].
$$

As an example to illustrate this, try to compute $\ln[p(a)+p(b)]$ when $\ln
p(a) = -1000$ and $\ln p(b) = -1001$. (Really, do try the above
approach before reading on. The correct solution is $-999.69$.) We've seen that we can solve this problem as follows:

$$
  \ln[p(a)+p(b)] = \ln[ \exp( \ln p(a)) + \exp( \ln p(b))] \\
  = \ln\left[ \exp(\ln p(a)) \left( 1 + \frac{\exp( \ln p(b))}{\exp( \ln
      p(a))}\right)\right] \\
  = \ln p(a) + \ln[1+\exp( \ln p(b) - \ln p(a) ) ],
$$

which is now easy to compute, because either the difference between
$\ln p(a)$ and $\ln p(b)$ is small (and then the exponent can be
computed without numerical problems) or it is large (and then the
exponent of the difference goes to zero, so that the $\ln(1+0)
\rightarrow \ln(1) = 0$. You do still need to pay attention that you
take the largest value out of the $\ln$, so that the difference goes
to large negative values rather than large positive values. 

The above discussion is pretty important: as we start using larger
data sets, the numerical issues become dominant.  
In effect, you
should never use probabilities in your code and always work with
log-probabilities. The products then become sums, and the sums become
the function explained above. This has the added advantage that $\ln$
and $\exp$ are pretty expensive operations. If you use distributions
from the exponential family, such as the Gaussian, computing the
log-probability is computationally slightly cheaper than computing the
probability. A final benefit is that sometimes the code is easier to
debug, because it is clear what errors are due to implementation bugs
and what errors are due to the limited precision of the machine. Have a look at the lecture notes for more information on this topic.

** Question 6 [10 marks] **

You may have noticed such numerical problems in the previous exercise,
especially when using larger numbers of mixture components. In this
exercise, we will change the E-step of the previous exercise and
compute everything with log-probabilities.

1. First implement a function called  `logsumexp` that computes
  the sum of two log-probabilities as described above. Make sure that
  this function takes the largest value out of the logarithm. Write the
  function so that both scalars and matrices can be used as arguments.
1. Discuss how we can use this function to compute sums of more
  than two probabilities using log-probabilities.
1. Check your implementation by computing $\ln[p(a)+p(b)]$ when
  $\ln p(a)=-1000$ and $\ln p(b)=-1001$.
1. Now convert the E-step of the previous exercise. Implement the
  function `logNormPDF` which is functionally equivalent
  to `normPDF` but never computes the logarithm or exponent and is therefore numerically stable:
$$
    \normdist(\x_n;\mean,\cov)
    = \frac{1}{(2\pi)^{d/2}|\cov|^{1/2}} \exp -\frac{1}{2} 
    (\x-\mean)^\tp \icov (\x-\mean)\\
    \ln \normdist(\x_n;\mean,\cov)
    = -\frac{1}{2} \left( d \ln 2\pi + \ln |\cov| + (\x-\mean)^\tp
    \icov (\x-\mean) \right)
$$

Notice, however, that only the function `expectation` is changed,
and that it must still return $p(\z|\x,\params)$, not $\ln
p(\z|\x,\params)$. In the adapted function, you compute $\ln
p(\z|\x,\params)$, but you need to exponentiate it before returning
the value. However now the computation does not suffer from the
underflows and is therefore more precise.





```python
# [ANSQ6] Do not modify this line
# Your answer to Question 6

def logsumexp(pa_log, pb_log):
    """Implementation of the logarithm of sum of two log-probabilities. 
    Returns the sum of two log-probabilities"""
    if pb_log > pa_log:
        temp = pb_log
        pb_log = pa_log
        pa_log = temp
    return pa_log + np.log(1 + np.exp(pb_log - pa_log))

vect_logsumexp = np.vectorize(logsumexp)
```

If one wants to apply logsumexp function to a non-scalar, then it's possible to use vectorized version of this function which is available under the name "vect_logsumexp".


```python
vect_logsumexp([-1000, -2], [-1001, -3])
```




    array([-999.68673831,   -1.68673831])



2) To compute the sum of more than two probabilites using log-probabilities and the function defined above, we can apply the function logsumexp recursively: first compute the sum of the first two log-probabilities, and after that, add all the rest elements, one by one, to the sum.

Given the list $p = [p_0, p_1, p_2, ..., p_{n-1}]$ of $n$ log-probabiltites:
1. s = p_0
2. for i in range(1, n):
    * s = logsumexp(s, p_i)
    
3) We check the correctness of implementation of `logsumexp` function:


```python
logsumexp(-1000, -1001)
```




    -999.68673831248179



4) To implement the more efficient and stable version of the E-step of the EM algorithm, we make use of the following formula:

$$ \gamma(z_{nk}) = \frac{\pi_k \normdist(\x_n;\mean,\cov)}{\sum_{j=1}^K \pi_j \normdist(\x_j;\mean,\cov)} $$
$$ \ln \gamma(z_{nk}) = \ln\pi_k + \ln\normdist(\x_n;\mean,\cov) - \ln\sum_{j=1}^K \pi_j \normdist(\x_j;\mean,\cov) $$


First, we implement lognormPDF:

$$
    \ln \normdist(\x_n;\mean,\cov)
    = -\frac{1}{2} \left( d \ln 2\pi + \ln |\cov| + (\x-\mean)^\tp
    \icov (\x-\mean) \right)
$$

After that, we inherit the class MoG_log from class MoG and change the implementation of functions `expectation` and `likelihood` using new function `lognormPDF`.


```python
def lognormPDF(data,mu,sigma):
    """Implementation of the logarithm of Gaussian distribution. Returns the logarithm of density, 
    given mean mu and covariance sigma, for every data point (row) of data"""
    N,nDim = data.shape
    res = np.zeros(N)
    
    icov = np.linalg.inv(sigma)
    covdet = np.linalg.det(sigma)
    denom = np.sqrt((2*np.pi)**nDim * covdet)
    
    for i in range(N):
        diff = data[i,:]-mu
        res[i] = -.5 * (nDim * np.log(2 * np.pi) + np.log(covdet)+ np.dot(diff, np.dot(icov,diff.T)))
    
    return res
```

#### Test correctness of the implementation


```python
np.exp(lognormPDF(data, mu[0], sigma[0]))
```




    array([ 0.00028789,  0.00122773,  0.00083388, ...,  0.00380406,
            0.00250394,  0.00331732])




```python
normPDF(data, mu[0], sigma[0])
```




    array([ 0.00028789,  0.00122773,  0.00083388, ...,  0.00380406,
            0.00250394,  0.00331732])




```python
class MoG_log(MoG):
    def expectation(self):
        """Compute p(z|x,\theta)"""
        # Answer to Q6.4
        for i in range(self.nMix):
            self.resp[i] = np.log(self.pi[i]) + lognormPDF(self.data, self.mu[i], self.sigma[i])

        sumexp = self.resp[0]
        for i in range(1, self.nMix): 
            sumexp = vect_logsumexp(sumexp, self.resp[i])
        
        self.resp -= sumexp
        return np.exp(self.resp), sum(self.likelihood(self.data))
        
    def likelihood(self,data):
        """Compute the marginal likelihood of the data given the mixture model, p(x|\theta), for
        all elements in `data`"""
        p = np.log(self.pi[0]) + lognormPDF(self.data, self.mu[0], self.sigma[0])
        for m in range(1, self.nMix):
            p = vect_logsumexp(p, np.log(self.pi[m]) + lognormPDF(self.data, self.mu[m], self.sigma[m]))
        return p
```


```python
mog = MoG_log(A, 3)
llhist = mog.em()
```

6317553a83f34ae9730c85aa5ad7f9290cf27725
ipynb
Jupyter Notebook
labs/lab2/lab2_em.ipynb
melaanya/twente-aml
3c483ca4c570c67370759bddc8e372e2c100e257
MIT
labs/lab2/lab2_em.ipynb
melaanya/twente-aml
3c483ca4c570c67370759bddc8e372e2c100e257
MIT
labs/lab2/lab2_em.ipynb
melaanya/twente-aml
3c483ca4c570c67370759bddc8e372e2c100e257
MIT
Qwen/Qwen-72B
1. YES
2. YES