PE1

PE2

Fig. 1. Basic structure of HMM-based recognition hardware.

evaluated in Section 5, and conclusions are presented in Section 6.

can be found in (Nakamura et al., 2010; Yoshizawa et al., 2002; 2004; 2006).

Feature vectors (stored in RAM)

PE1s

PE2s

**2. HMM-based recognition systems**

**2.1 HMM-based recognition hardware**

...

Let **O**1, **O**2, ..., and **O***T* be a sequence of *P*-dimensional input feature vectors to HMMs, where **O***t* = (*ot*1, *ot*2, ..., *otP*), 1 ≤ *t* ≤ *T*. *T* is the number of input feature vectors, and *P* is the dimension of the input feature vector. For any input feature vector **O***t*, the output probability of *N*-state continuous HMMs at the *j*-th state is given by

$$\log b\_j(\mathbf{O}\_t) = \omega\_j + \sum\_{p=1}^{P} \sigma\_{jp} (o\_{tp} - \mu\_{jp})^2 \tag{1}$$

where *ωj*, *σjp* and *μjp* are the parameters of the Gaussian probability density function which are precomputed and stored in ROM. The OPC circuit computes log *bj*(**O***t*) based on Eq. (1), where 1 ≤ *j* ≤ *N* and 1 ≤ *t* ≤ *T*. All HMM parameters *ωj*, *σjp*, and *μjp* are stored in ROM and the input feature vectors are stored in RAM. The values of *T*, *N*, *P*, and the number of HMMs *V* differ for each recognition system. For isolated word recognition systems, *T*, *N*, *P*, and *V* are 86, 32, 38, and 800, respectively (Yoshizawa et al., 2004; 2006), and for another word recognition system, *T*, *N*, *P*, and *V* are 89, 12, 16 and 100 (Yoshizawa et al., 2002).

For output probabilitie log *bj*(**O***t*), where 1 ≤ *j* ≤ *N* and 1 ≤ *t* ≤ *T*, the log-likelihood score log *S*∗ is given by

$$\log \delta\_1(j) = \log \pi\_j + |\log b\_j(\mathbf{O}\_1)| \,\tag{2}$$

$$\log \delta\_l(j) = \min[\log \delta\_{l-1}(j-1) + |\log a\_{j-1,j}|\_{\nu}]$$

$$\log \delta\_{t-1}(j) + |\log a\_{j,j}| + |\log b\_j(\mathbf{O}\_t)|\_\prime \tag{3}$$

$$\log S^\* = \min\_{1 \le j \le N} [\log \delta\_T(j)].\tag{4}$$

All HMM parameters log*πj*, log *aj*−1,*j*, and log *aj*,*j* are stored in ROM, and the Viterbi scorer computes log *δt*(*j*) based on Eqs. (2) and (3). A flowchart of OPCs and LSCs is shown in Fig. 2 (Yoshizawa et al., 2004; 2006). All HMM output probabilities are obtained by *P* · *N* · *T* · *V* times the partial computation of log *bj*(**O***t*) calls. Partial computation of log *bj*(**O***t*) performs four arithmetic operations, an addition, a subtraction and two multiplications in Eq. (1). All likelihood scores are obtained by *N* · *T* · *V* times the partial computation of log *δt*(*j*) calls. Partial computation of log *δt*(*j*) performs three additions in Eq. (3). The OPC circuit and the Viterbi scorer accelerate these computations. More details can be found in (Yoshizawa et al., 2002; 2004; 2006).

#### **2.3 Block parallel processing for OPCs and LSCs**

BPP for OPCs and LSCs was proposed as an efficient high-speed parallel processing method for HMM-based isolated word speech recognition (Yoshizawa et al., 2002). In BPP, the set of input feature vectors is called a *block*, and HMM parameters are effectively shared between the different input feature vectors for OPC. In recent years, two types of BPP are classified according to input data flow: *StreamBPP* and *StoreBPP* (Nakamura et al., 2010).

A block can be considered as a *M* · *P* matrix whose elements are *ot*� *p*, where 1 ≤ *t*� ≤ *M* (≤ *T*) and 1 ≤ *p* ≤ *P*. *StreamBPP* performs arithmetic operations on the input stream *o*11, *o*12, ..., *o*1*P*, *o*21, *o*22, ..., *o*2*P*, ..., *oM*1, *oM*2, ..., *oMP* (Yoshizawa et al., 2006). StreamBPP performs *N* OPCs in parallel with *N PE1*s (Fig. 1) and obtains *N* output probabilities log *b*1(**O***t*), log *b*2(**O***t*), ..., and log *bN*(**O***t*) simultaneously. These *N* output probabilities are obtained every *P* clock cycles

computations which are performed in parallel and a ROM access are performed simultaneously in StoreBPP, where two HMM parameters −*μj*,*p*+1 and *σj*,*p*+1 are required for next OPC. Because it takes two cycles to read two HMM parameters from ROM, it was

159 A VLSI Architecture for Output Probability

StoreBPP performs �*M*/2� OPCs in parallel by using a register array of size *M*, where *M* is the size of block (Nakamura et al., 2010). We improve StoreBPP by reducing the required register size for performing *M* OPCs in parallel. We modify the parallel computation of �*M*/2� OPCs by doubling the bit length of the input to OPC. By this bit length extension, two HMM parameters can be read simultaneously. We call the modified parallel processing *fast store-based block parallel processing* (*FastStoreBPP*), and we show a pipelined Viterbi scorer that supports FastStoreBPP. It performs *M* OPCs in parallel by using a register array of size

A flowchart of our FastStoreBPP is shown in Fig. 3. The flowchart consists of two tasks, *M-parallel OPC* and �*M*/*P*�*-stage pipelined LSC*. In Fig. 3, the *M*-parallel OPC and �*M*/*P*�-stage pipelined LSC are framed by dashed and double dashed lines, respectively. The *M*-parallel OPC performs *M* OPCs in parallel with *M PE1*s. The �*M*/*P*�-stage pipelined LSC performs

Loops A, B, and D (Fig. 2), where *M* HMM output probabilities and *M* intermediate scores are computed with *M PE1*s and �*M*/*P*� *PE2*s. Loop C1 is based on StoreBPP, where Loop C

*PE*11, *PE*12, ..., and *PE*1*M* (Fig. 3) represent *M* processing elements which perform partial computations log *bj*(**O***t*�+1), log *bj*(**O***t*�+2), ..., and log *bj*(**O***t*�+*M*) using Eq. (1). *M* HMM output

read from ROM simultaneously. These values are needed for next computation in Loop A�

Next, the *M* HMM output probabilities log *bj*(**O***t*�+1), log *bj*(**O***t*�+2), ..., and log *bj*(**O***t*�

are fed to �*M*/*P*�-stage pipelined LSC (Fig. 3) which is a new Viterbi scorer introduced for FastStoreBPP. *PE*2s represent �*M*/*P*� processing elements for computing *M* intermediate

Loops E1, E2, ..., and E�*M*/*P*� (Fig. 3) perform LSC, where the *i*-th intermediate score log *δt*�+*i*(*j*)

and computes *P* intermediate scores sequentially, where *P* clock cycles are required for the computation. Loop E�*M*/*P*� has a *PE*2 and computes *M* − *P* · (�*M*/*P*� − 1) intermediate scores sequentially by using *M* − *P* · (�*M*/*P*� − 1) clock cycles. All Loop Es are pipelined in

proceed simultaneously, where each Loop E*i*� finishes its computation before the next *M*

Our FastStoreBPP architecture that supports FastStoreBPP is shown in Fig. 4, where we assume *M* ≤ *P* and hence �*M*/*P*� = 1. The FastStoreBPP architecture consists of an OPC circuit and a Viterbi scorer. The architecture has two register arrays (Reg**O** and Reg*ω*),

.

+2), ..., and log *bj*(**O***t*�

. The modified *M* OPCs are performed in parallel and a ROM access

, B�

+*M*(*j*) using Eqs. (2) and (3). In our FastStoreBPP,

+1. Loop A� and Loop E*i*� s, 1 ≤ *i*

, and D� correspond to

+*M*) are simultaneously obtained

. The two HMM parameters are

� ≤ �*M*/*P*� − 1, has a *PE*2

·*P*(*j*), obtained by Loop E*i*� , 1 ≤ *i*

.

+*M*)

� ≤

� ≤ �*M*/*P*�,

LSC in serial with �*M*/*P*� *PE2*s and �*M*/*P*� registers. Loops A�

−*μj*,*p*+1 and *σj*,*p*+1 are performed simultaneously in Loop A�

+2(*j*), ..., and log *δt*�

is computed by Loop E�*i*/*P*�. Each Loop E*i*� , where 1 ≤ *i*

FastStoreBPP, where the last intermediate score log *δt*�+*i*�

�*M*/*P*� − 1, is fed to the next stage Loop E*i*�

output probabilities are obtained by Loop A�

**3.2 FastStoreBPP architecture for OPCs and LSCs**

(Fig. 2) is partially expanded in Fig. 3.

probabilities log *bj*(**O***t*�+1), log *bj*(**O***t*�

every *P* cycles by Loop A�

scores log *δt*�+1(*j*), log *δt*�

appropriate to use a two-step process in StoreBPP.

and Likelihood Score Computations of HMM-Based Recognition Systems

*M*.

Fig. 2. Flowchart of OPCs and LSCs.

and they are fed to the Viterbi scorer for LSCs. An Viterbi scorer that supports the OPCs and LSCs in StreamBPP was presented in (Yoshizawa et al., 2006), where *N* LSCs are performed with *N PE2*s (Fig. 1). In the Viterbi scorer, *N* intermediate scores log *δt*(1), log *δt*(2), ..., and log *δt*(*N*) are computed simultaneously.

A block can be considered as a set of *M* input feature vectors whose elements are **O***t*� where 1 ≤ *t*� ≤ *M* (≤ *T*). *StoreBPP* performs arithmetic operations to locally stored input feature vectors **O**1, **O**2, ..., and **O***M* (Nakamura et al., 2010). StoreBPP performs �*M*/2� OPCs in parallel with �*M*/2� *PE1*s (Fig. 1) and obtains *M* HMM output probabilities log *bj*(**O***t*� +1), log *bj*(**O***t*�+2), ..., and log *bj*(**O***t*�+*M*). These *M* HMM output probabilities are obtained every 2 · *P* clock cycles and they are fed to the Viterbi scorer for LSCs. Different BPPs require different Viterbi scorer architectures. In StoreBPP, a Viterbi scorer that supports the OPCs, where *M* HMM output probabilities are computed simultaneously, is required, but the Viterbi scorer for the StoreBPP was not addressed in (Nakamura et al., 2010). An easy introduction of the Viterbi scorer to StoreBPP requires many registers, which reduces the advantage of the StoreBPP architecture.

#### **3. VLSI architecture for OPCs and LSCs with fast store-based block parallel processing**

#### **3.1 Fast store-based block parallel processing**

A two-step process was adopted in StoreBPP to compute *M* HMM output probabilities log *bj*(**O***t*�+1), log *bj*(**O***t*�+2), ..., and log *bj*(**O***t*�+*M*), where half of the output probabilities are computed simultaneously with �*M*/2� *PE1*s (Nakamura et al., 2010). The �*M*/2� 4 Will-be-set-by-IN-TECH

*v = v +* 1

*t = t +* 1

*j = j +* 1

*p = p +* 1

*Partial computation of logb j* (*Ot*)

*Partial computation of log*δ *t*( *j*)

Fig. 2. Flowchart of OPCs and LSCs.

log *δt*(*N*) are computed simultaneously.

**3.1 Fast store-based block parallel processing**

StoreBPP architecture.

**processing**

log *bj*(**O***t*�

*j >N*

NO

*Loop B*

*Loop A*

*Loop C*

*Loop D*

NO

+1),

*t >T*

YES

*p >P* NO YES

YES

and they are fed to the Viterbi scorer for LSCs. An Viterbi scorer that supports the OPCs and LSCs in StreamBPP was presented in (Yoshizawa et al., 2006), where *N* LSCs are performed with *N PE2*s (Fig. 1). In the Viterbi scorer, *N* intermediate scores log *δt*(1), log *δt*(2), ..., and

A block can be considered as a set of *M* input feature vectors whose elements are **O***t*� where 1 ≤ *t*� ≤ *M* (≤ *T*). *StoreBPP* performs arithmetic operations to locally stored input feature vectors **O**1, **O**2, ..., and **O***M* (Nakamura et al., 2010). StoreBPP performs �*M*/2� OPCs in parallel with �*M*/2� *PE1*s (Fig. 1) and obtains *M* HMM output probabilities log *bj*(**O***t*�

log *bj*(**O***t*�+2), ..., and log *bj*(**O***t*�+*M*). These *M* HMM output probabilities are obtained every 2 · *P* clock cycles and they are fed to the Viterbi scorer for LSCs. Different BPPs require different Viterbi scorer architectures. In StoreBPP, a Viterbi scorer that supports the OPCs, where *M* HMM output probabilities are computed simultaneously, is required, but the Viterbi scorer for the StoreBPP was not addressed in (Nakamura et al., 2010). An easy introduction of the Viterbi scorer to StoreBPP requires many registers, which reduces the advantage of the

**3. VLSI architecture for OPCs and LSCs with fast store-based block parallel**

A two-step process was adopted in StoreBPP to compute *M* HMM output probabilities

are computed simultaneously with �*M*/2� *PE1*s (Nakamura et al., 2010). The �*M*/2�

+1), log *bj*(**O***t*�+2), ..., and log *bj*(**O***t*�+*M*), where half of the output probabilities

*v >V* NO YES

*v =* 0

*t* = 0

*j* = 0

*p* = 0

computations which are performed in parallel and a ROM access are performed simultaneously in StoreBPP, where two HMM parameters −*μj*,*p*+1 and *σj*,*p*+1 are required for next OPC. Because it takes two cycles to read two HMM parameters from ROM, it was appropriate to use a two-step process in StoreBPP.

StoreBPP performs �*M*/2� OPCs in parallel by using a register array of size *M*, where *M* is the size of block (Nakamura et al., 2010). We improve StoreBPP by reducing the required register size for performing *M* OPCs in parallel. We modify the parallel computation of �*M*/2� OPCs by doubling the bit length of the input to OPC. By this bit length extension, two HMM parameters can be read simultaneously. We call the modified parallel processing *fast store-based block parallel processing* (*FastStoreBPP*), and we show a pipelined Viterbi scorer that supports FastStoreBPP. It performs *M* OPCs in parallel by using a register array of size *M*.

A flowchart of our FastStoreBPP is shown in Fig. 3. The flowchart consists of two tasks, *M-parallel OPC* and �*M*/*P*�*-stage pipelined LSC*. In Fig. 3, the *M*-parallel OPC and �*M*/*P*�-stage pipelined LSC are framed by dashed and double dashed lines, respectively. The *M*-parallel OPC performs *M* OPCs in parallel with *M PE1*s. The �*M*/*P*�-stage pipelined LSC performs LSC in serial with �*M*/*P*� *PE2*s and �*M*/*P*� registers. Loops A� , B� , and D� correspond to Loops A, B, and D (Fig. 2), where *M* HMM output probabilities and *M* intermediate scores are computed with *M PE1*s and �*M*/*P*� *PE2*s. Loop C1 is based on StoreBPP, where Loop C (Fig. 2) is partially expanded in Fig. 3.

*PE*11, *PE*12, ..., and *PE*1*M* (Fig. 3) represent *M* processing elements which perform partial computations log *bj*(**O***t*�+1), log *bj*(**O***t*�+2), ..., and log *bj*(**O***t*�+*M*) using Eq. (1). *M* HMM output probabilities log *bj*(**O***t*� +1), log *bj*(**O***t*�+2), ..., and log *bj*(**O***t*� +*M*) are simultaneously obtained every *P* cycles by Loop A� . The modified *M* OPCs are performed in parallel and a ROM access −*μj*,*p*+1 and *σj*,*p*+1 are performed simultaneously in Loop A� . The two HMM parameters are read from ROM simultaneously. These values are needed for next computation in Loop A� .

Next, the *M* HMM output probabilities log *bj*(**O***t*�+1), log *bj*(**O***t*�+2), ..., and log *bj*(**O***t*�+*M*) are fed to �*M*/*P*�-stage pipelined LSC (Fig. 3) which is a new Viterbi scorer introduced for FastStoreBPP. *PE*2s represent �*M*/*P*� processing elements for computing *M* intermediate scores log *δt*�+1(*j*), log *δt*�+2(*j*), ..., and log *δt*�+*M*(*j*) using Eqs. (2) and (3). In our FastStoreBPP, Loops E1, E2, ..., and E�*M*/*P*� (Fig. 3) perform LSC, where the *i*-th intermediate score log *δt*�+*i*(*j*) is computed by Loop E�*i*/*P*�. Each Loop E*i*� , where 1 ≤ *i* � ≤ �*M*/*P*� − 1, has a *PE*2 and computes *P* intermediate scores sequentially, where *P* clock cycles are required for the computation. Loop E�*M*/*P*� has a *PE*2 and computes *M* − *P* · (�*M*/*P*� − 1) intermediate scores sequentially by using *M* − *P* · (�*M*/*P*� − 1) clock cycles. All Loop Es are pipelined in FastStoreBPP, where the last intermediate score log *δt*�+*i*� ·*P*(*j*), obtained by Loop E*i*� , 1 ≤ *i* � ≤ �*M*/*P*� − 1, is fed to the next stage Loop E*i*� +1. Loop A� and Loop E*i*� s, 1 ≤ *i* � ≤ �*M*/*P*�, proceed simultaneously, where each Loop E*i*� finishes its computation before the next *M* output probabilities are obtained by Loop A� .

#### **3.2 FastStoreBPP architecture for OPCs and LSCs**

Our FastStoreBPP architecture that supports FastStoreBPP is shown in Fig. 4, where we assume *M* ≤ *P* and hence �*M*/*P*� = 1. The FastStoreBPP architecture consists of an OPC circuit and a Viterbi scorer.

*PE*11

*PE*12

*PE*1*M*

RAM **O**

OPC circuit

Regμ Regσ Regω Reg**O**

. . .

+ × +

×

OPC starts by reading *M* input feature vectors **O***t*�

parallel with *M PE*1s based on Loop A� (Fig. 3).

are log *bj*(**O***t*�+1), log *bj*(**O***t*�+2), ..., and log *bj*(**O***t*�

parameters by using *M PE*1s (Fig. 4) based on Loop A� (Fig. 3).

+ +

×

×

+ +

. . .

×

Fig. 4. FastStoreBPP architecture for OPCs and LSCs (*M* ≤ *P*).

and two multipliers, which are used for computing *ωj* + ∑*P*

×

and Likelihood Score Computations of HMM-Based Recognition Systems

. . .

 Each *PE*1 consists of two adders

the FastStoreBPP architecture, the StreamBPP architecture (Yoshizawa et al., 2006), and the StoreBPP architecture (Nakamura et al., 2010) are identical but differ in number. In addition, the architecture has three register arrays (RegIn*δ*1, RegLast*δ*, and RegTmp*δj*−1 1), three registers (Reg*aj*,*j* 1, Reg*aj*−1,*j* 1, and RegTmp*δj* 1), and a *PE*2 for LSCs. *PE*2 consists of three adders, two selectors and two comparators, which are used for LSC based on Eqs. (2) and (3). *PE*2 in the FastStoreBPP architecture and the StreamBPP architecture (Yoshizawa et al., 2006)

storing them in Reg**O** in OPC circuit (Fig. 4) based on Loop C1 (Fig. 3). *M* · *P*/2 cycles are required for reading *M* input feature vectors. Then, the HMM parameters of *v*-th HMM are read from ROM, which are −*μ*11, *σ*11, and *ω*1, and stored in Reg*μ*, Reg*σ*, and Reg*ω*, respectively, based on Loop C1 and Loop B� (Fig. 3). For the stored input feature vectors, *M* intermediate results of *M* OPCs are simultaneously computed with the stored HMM

The stored HMM parameters are shared by all *PE*1s, and the obtained *M* intermediate results are stored in Reg*ω*. At the same time, two HMM parameters −*μjp*+1 and *σjp*+1 of *v*-th HMM are read from ROM and stored in Reg*μ* and Reg*σ*, respectively, where the values are overwritten. The HMM parameters are used in next *M* computations which are performed in

*M* HMM output probabilities are simultaneously obtained every *P* cycles by *M PE*1s, which

M M N

. . .

*PE*2

CMP (*t* == 1)

+

+1, **O***t*�

. . .

. . .

RegTmpδ*j* 1

. . .

+

. . .

*p*=1 *σjp*(*otp* − *μjp*)2. *PE*1*i* in

+*M* from RAM and

RegInδ1

+2, ..., and **O***t*�

+*M*) of *v*-th HMM based on Loop A� (Fig. 3).

M

SEL SEL +

CMP <

. . .

RegTmpδ*j*−1 1

RegLastδ

. . .

Viterbi scorer

Reg*aj*,*j* 1 Reg*aj*−1,*j* 1

M

. . .

Fig. 3. Flowchart of OPCs and LSCs with FastStoreBPP.

6 Will-be-set-by-IN-TECH

v = 0

*j* = 1, *p* = 1

v = v + 1

� + 1, *t*

*t* � *max* = 0

� + 2, ..., *t*

Load −μ1,1 and σ1,1 to Regμ and Regσ, respectively (1 cycle) *j* = 0

*j* = *j* + 1

(and Load log *aj*−1,*j* to Reg*aj*−1,*j* when 2 ≤ *j*) (1 cycle).

*p* = *p* + 1

*PE*11, *t* = *t*

*p* = 0

*t* � = *t* � *max* , *t* � *max* = *t* � *max* + *M*

Load ω*j* to Regω (and load log π*j* to RegTmpδ*j* when *t* = *t*

Load log *aj*,*j* to Reg*aj*,*j*

Load −μ*j*,*p*+1 and σ*j*,*p*+1 to Regμ and Regσ (1 cycle)

> *t* �� 1 *last* = *t*

� + *P t* �� 1 = *t* ��

*t* �� 1 = *t* �

> log δ*t* ��

> > *t* �� 1 ≥ *t* �� 1 *last*

YES

*j* ≥ *N*

Fig. 3. Flowchart of OPCs and LSCs with FastStoreBPP.

Load **O***t* to Reg**O** (*t* = *t*

*Loop D*�

*Loop C*1

*Loop B*�

� + *M*), (*M* · *P*/2 cycles)

� + 1 == 1) (1 cycle)

*Loop A*�

*M*-parallel OPC

� + *M*

 *. . .*

 *. . .*

NO

 *. . .*

 *. . .*

Copy Regω to RegInδ

*p* ≥ *P*

YES

 *. . .*

log *bj*(**O***t*�+1) log *bj*(**O***t*�+*M*)

NO

� + 1 *PE*1*M*, *t* = *t*

*Loop E*1 *Loop E*�*M*/*P*�

�� �*M*/*P*� (*j*)

≥ *t* �� �*M*/*P*� *last*

NO NO

YES

v ≥ *V*

�*M*/*P*� *last* = *t*

*t* �� �*M*/*P*�

NO

� + *M*

YES

� + *P* · (�*M*/*P*� − 1)

�*M*/*P*�-stage pipelined LSC

*t* ��

�� �*M*/*P*� = *t* �� �*M*/*P*� + 1

*t* �� �*M*/*P*� = *t*

 *. . .*

 *. . .*

1 + 1 *t*

*PE*2 *PE*2

NO

YES

YES

*t* � *max* ≥ *T*  *. . .*

1 (*j*) log δ*t*

Fig. 4. FastStoreBPP architecture for OPCs and LSCs (*M* ≤ *P*).

two registers (Reg*μ* and Reg*σ*), and *M PE*1s for OPCs. Each *PE*1 consists of two adders and two multipliers, which are used for computing *ωj* + ∑*P p*=1 *σjp*(*otp* − *μjp*)2. *PE*1*i* in the FastStoreBPP architecture, the StreamBPP architecture (Yoshizawa et al., 2006), and the StoreBPP architecture (Nakamura et al., 2010) are identical but differ in number. In addition, the architecture has three register arrays (RegIn*δ*1, RegLast*δ*, and RegTmp*δj*−1 1), three registers (Reg*aj*,*j* 1, Reg*aj*−1,*j* 1, and RegTmp*δj* 1), and a *PE*2 for LSCs. *PE*2 consists of three adders, two selectors and two comparators, which are used for LSC based on Eqs. (2) and (3). *PE*2 in the FastStoreBPP architecture and the StreamBPP architecture (Yoshizawa et al., 2006) are identical but differ in number.

OPC starts by reading *M* input feature vectors **O***t*�+1, **O***t*�+2, ..., and **O***t*�+*M* from RAM and storing them in Reg**O** in OPC circuit (Fig. 4) based on Loop C1 (Fig. 3). *M* · *P*/2 cycles are required for reading *M* input feature vectors. Then, the HMM parameters of *v*-th HMM are read from ROM, which are −*μ*11, *σ*11, and *ω*1, and stored in Reg*μ*, Reg*σ*, and Reg*ω*, respectively, based on Loop C1 and Loop B� (Fig. 3). For the stored input feature vectors, *M* intermediate results of *M* OPCs are simultaneously computed with the stored HMM parameters by using *M PE*1s (Fig. 4) based on Loop A� (Fig. 3).

The stored HMM parameters are shared by all *PE*1s, and the obtained *M* intermediate results are stored in Reg*ω*. At the same time, two HMM parameters −*μjp*+1 and *σjp*+1 of *v*-th HMM are read from ROM and stored in Reg*μ* and Reg*σ*, respectively, where the values are overwritten. The HMM parameters are used in next *M* computations which are performed in parallel with *M PE*1s based on Loop A� (Fig. 3).

*M* HMM output probabilities are simultaneously obtained every *P* cycles by *M PE*1s, which are log *bj*(**O***t*�+1), log *bj*(**O***t*�+2), ..., and log *bj*(**O***t*�+*M*) of *v*-th HMM based on Loop A� (Fig. 3).

. . .

+

. . .

. . .

**processing**

extension, *L*� · *M*�

Each *M*�

MultipleStoreBPP performs *M*�

The flowchart consists of *L*� *M*�

shown in Fig. 6 for performing *L*� · *M*�

/2-parallel OPC and �*M*�

to *L*� HMMs and *L*� · *M*�

*M*

. . .

. . .

+

CMP (*t* == 1) CMP (*t* == 1)

. . .

Fig. 5.

RegTmpδ*j*−1 1 RegTmpδ*j*−1 �*M*/*P*�

. . .

*M* mod *P*

RegInδ�*M*/*P*�

*M* mod *P*

CMP

*M* mod *P*

. . .

/2 OPCs are performed in parallel,

/2-parallel OPCs.

/(2 · *P*)�-stage pipelined LSCs.

, B��, and A�� in Fig. 6

, C1�

/2 OPCs in parallel in Loop A��.

**4. VLSI architecture for OPCs and LSCs with multiple store-based block parallel**

FastStoreBPP is obtained from StoreBPP by doubling the bit length of the input to OPC for

We further extend the bit length of the input to OPC and we obtain *multiple store-based block*

StoreBPP performs *M*/2 OPCs in parallel by using a register array of size *M* and *M*/2 *PE*1s, where *M*/2 OPCs are performed by a single HMM (Nakamura et al., 2010). We improve StoreBPP and further reduce the required register size for performing OPCs in parallel. We modify *M*/2*-parallel OPC*, where *M*/2 OPCs are performed in parallel, to deal with multiple HMMs by further extending the bit length of the input to OPC. By this bit-length

from ROM simultaneously. We call the modified parallel processing *MultipleStoreBPP*. Our

are based on StoreBPP. In our MultipleStoreBPP, Loop D' (Fig. 3) is partially expanded as

/2-parallel OPCs and *L*� �*M*�

/2*-parallel OPC*, where *M*�

A flowchart of our MultipleStoreBPP is shown in Fig. 6. Loops D2�

input feature vectors are effectively shared between different *M*�

/2 OPCs are performed in parallel, where *L*� HMM parameters can be read

/2 OPCs are performed in parallel by using a register array of size *M*�

The results are copied from Reg*ω* to RegIn*δ*1 to start LSCs log *δt*��+1(*j*), log *δt*��+1(*j*), ..., and log *δt*��+*M*(*j*) and the next *M* OPCs for the (*j* + 1)-th state of *v*-th HMM log *bj*+1(**O***t*� +1), log *bj*+1(**O***t*� +2), ..., and log *bj*+1(**O***t*�+*M*) based on Loop B� (Fig. 3). *M* · *N* HMM output probabilities of *v*-th HMM are obtained by Loop B� (Fig. 3). *M* · *N* · *T* HMM output probabilities of *v*-th HMM are obtained by Loop C1 (Fig. 3). *M* · *N* · *T* · *V* HMM output probabilities of all HMM are obtained by Loop D� (Fig. 3).

Viterbi scorer, denoted by double -dashed lines in (Fig. 4), performes �*M*/*P*�-stage pipelined LSC, denoted by double -dashed lines in Fig. 3. LSC starts by reading HMM parameters of *v*-th HMM, log *π*1 and log *a*1,1, from ROM and storing them in RegTmp*δj* 1 and Reg*aj*,*j* 1, respectively based on Loop B� (Fig. 3). Then, an intermediate score log *δ*1(1) is computed by *PE*2 with the HMM parameter, log *π*1, and the HMM output probability log *b*1(**O**1) obtained using Eq. (1). The obtained intermediate score is stored in both RegTmp*δj* 1 and RegTmp*δj*−1 1 (Fig. 4). RegTmp*δj* 1 stores an intermediate score that is needed in the next computation in Loop E1 (Fig. 3). RegTmp*δj*−1 1 stores *M* intermediate scores log *δt*�+1(*j*), log *δt*�+2(*j*), ..., and log *δt*�+*M*(*j*), which is needed in the next LSC for the (*j* + 1)-th state of the HMM in Loop B� . After the computation of log *δ*1(1), *M* − 1 intermediate scores log *δ*2(1), log *δ*3(1), ..., and log *δM*(1), are sequentially computed by *PE*2 using Eq. (3). In sequential computation, the last obtained intermediate score log *δM*(1) is stored in RegTmp*δj*−1 1 and RegLast*δ* (Fig. 4). RegLast*δ* stores *N* intermediate scores that are the last obtained intermediate scores by Loop E1 during Loop B� (Fig. 3). These intermediate scores are log *δt*�+*M*(1), log *δt*�+*M*(2), ..., and log *δt*� +*M*(*N*) of *v*-th HMM, which are required when starting LSC with new *M* HMM output probabilities log *bj*(**O***t*�+*M*+1), log *bj*(**O***t*�+*M*+2), ..., and log *bj*(**O***t*�+2·*M*) at the first computation in Loop E1 (Fig. 3), given as log *δt*�+*M*+1(1), log *δt*�+*M*+1(2), ..., and log *δt*�+*M*+1(*N*). A required intermediate score is read from RegLast*δ* and is stored in RegTmp*δj* 1 before computation. Reg*aj*−1,*j* 1 (Fig. 4) stores an HMM parameter log *aj*−1,*j* of *v*-th HMM, which is used for computing log *δt*(*j*) based on Eq. (3), when 2 ≤ *t* and 2 ≤ *j*.

Our Viterbi scorer, which support FastStoreBPP, was presented in Fig. 4, where *M* ≤ *P* and �*M*/*P*� = 1. Our �*M*/*P*�-stage pipelined Viterbi scorer, which supports *P < M* and 1 *<* �*M*/*P*�, is shown in Fig. 5. The Viterbi scorer in Fig. 4 is an instance of the generalized �*M*/*P*�-stage pipelined Viterbi scorer where �*M*/*P*� = 1. The �*M*/*P*�-stage pipelined Viterbi scorer consists of RegLast*δ* and �*M*/*P*� sub Viterbi scorers. The *i*-th stage, i.e., *i*-th sub Viterbi scorer consists of two register arrays (RegIn*δi* and RegTmp*δj*−1 *i*), three registers (RegTmp*δj i*, Reg*aj*,*j i* and Reg*aj*−1,*j i*) and a *PE*2. Each RegIn*δi* consists of *i* · *P* registers, where *i* = 1, *i* = 2, ..., �*M*/*P*� − 1. RegIn*δ*�*M*/*P*� consists of �*M*/*P*� · (*M* mod *P*) registers. In each RegIn*δi*, rows are shifted upward every *P* clock cycles. Each RegTmp*δj*−1 *i* consists of *P* registers, where *i* = 1, *i* = 2, ..., and �*M*/*P*� − 1. RegTmp*δj*−1 �*M*/*P*� consists of *M* mod *P* registers. HMM parameters in Reg*aj*,*j i* and Reg*aj*−1,*j i* are copied every *P* clock cycles to Reg*aj*,*j i*+1 and Reg*aj*−1,*j i*+1, respectively, where *i* = 1, *i* = 2, ..., and �*M*/*P*� − 1. The last obtained intermediate score by *PE*2 based on Loop E*i* (Fig. 3), where *i* = 1, *i* = 2, ..., and �*M*/*P*� − 1, is stored in RegTmp*δj*−1 *i* and RegTmp*δj i*+1 every *P* clock cycles based on �*M*/*P*�-state pipelined LSC (Fig. 3). The last obtained intermediate score by *PE*2 based on Loop E�*M*/*P*� (Fig. 3) is stored in RegTmp*δj*−1 �*M*/*P*� and RegLast*δ* every *P* clock cycles during Loop B� (Fig. 3). The stored intermediate scores are required when starting LSC with new *M* output probabilities at the first computation by Loop E1 (Fig. 3). The required intermediate score is read from RegLast*δ* and stored in RegTmp*δj* 1 before computation.

8 Will-be-set-by-IN-TECH

The results are copied from Reg*ω* to RegIn*δ*1 to start LSCs log *δt*��+1(*j*), log *δt*��+1(*j*), ..., and log *δt*��+*M*(*j*) and the next *M* OPCs for the (*j* + 1)-th state of *v*-th HMM log *bj*+1(**O***t*�+1), log *bj*+1(**O***t*�+2), ..., and log *bj*+1(**O***t*�+*M*) based on Loop B� (Fig. 3). *M* · *N* HMM output probabilities of *v*-th HMM are obtained by Loop B� (Fig. 3). *M* · *N* · *T* HMM output probabilities of *v*-th HMM are obtained by Loop C1 (Fig. 3). *M* · *N* · *T* · *V* HMM output

Viterbi scorer, denoted by double -dashed lines in (Fig. 4), performes �*M*/*P*�-stage pipelined LSC, denoted by double -dashed lines in Fig. 3. LSC starts by reading HMM parameters of *v*-th HMM, log *π*1 and log *a*1,1, from ROM and storing them in RegTmp*δj* 1 and Reg*aj*,*j* 1, respectively based on Loop B� (Fig. 3). Then, an intermediate score log *δ*1(1) is computed by *PE*2 with the HMM parameter, log *π*1, and the HMM output probability log *b*1(**O**1) obtained using Eq. (1). The obtained intermediate score is stored in both RegTmp*δj* 1 and RegTmp*δj*−1 1 (Fig. 4). RegTmp*δj* 1 stores an intermediate score that is needed in the next computation in

log *δt*�+*M*(*j*), which is needed in the next LSC for the (*j* + 1)-th state of the HMM in Loop

. After the computation of log *δ*1(1), *M* − 1 intermediate scores log *δ*2(1), log *δ*3(1), ..., and log *δM*(1), are sequentially computed by *PE*2 using Eq. (3). In sequential computation, the last obtained intermediate score log *δM*(1) is stored in RegTmp*δj*−1 1 and RegLast*δ* (Fig. 4). RegLast*δ* stores *N* intermediate scores that are the last obtained intermediate scores by Loop E1 during Loop B� (Fig. 3). These intermediate scores are log *δt*�+*M*(1), log *δt*�+*M*(2), ..., and

+*M*(*N*) of *v*-th HMM, which are required when starting LSC with new *M* HMM output probabilities log *bj*(**O***t*�+*M*+1), log *bj*(**O***t*�+*M*+2), ..., and log *bj*(**O***t*�+2·*M*) at the first computation in Loop E1 (Fig. 3), given as log *δt*�+*M*+1(1), log *δt*�+*M*+1(2), ..., and log *δt*�+*M*+1(*N*). A required intermediate score is read from RegLast*δ* and is stored in RegTmp*δj* 1 before computation. Reg*aj*−1,*j* 1 (Fig. 4) stores an HMM parameter log *aj*−1,*j* of *v*-th HMM, which is used for

Our Viterbi scorer, which support FastStoreBPP, was presented in Fig. 4, where *M* ≤ *P* and �*M*/*P*� = 1. Our �*M*/*P*�-stage pipelined Viterbi scorer, which supports *P < M* and 1 *<* �*M*/*P*�, is shown in Fig. 5. The Viterbi scorer in Fig. 4 is an instance of the generalized �*M*/*P*�-stage pipelined Viterbi scorer where �*M*/*P*� = 1. The �*M*/*P*�-stage pipelined Viterbi scorer consists of RegLast*δ* and �*M*/*P*� sub Viterbi scorers. The *i*-th stage, i.e., *i*-th sub Viterbi scorer consists of two register arrays (RegIn*δi* and RegTmp*δj*−1 *i*), three registers (RegTmp*δj i*, Reg*aj*,*j i* and Reg*aj*−1,*j i*) and a *PE*2. Each RegIn*δi* consists of *i* · *P* registers, where *i* = 1, *i* = 2, ..., �*M*/*P*� − 1. RegIn*δ*�*M*/*P*� consists of �*M*/*P*� · (*M* mod *P*) registers. In each RegIn*δi*, rows are shifted upward every *P* clock cycles. Each RegTmp*δj*−1 *i* consists of *P* registers, where *i* = 1, *i* = 2, ..., and �*M*/*P*� − 1. RegTmp*δj*−1 �*M*/*P*� consists of *M* mod *P* registers. HMM parameters in Reg*aj*,*j i* and Reg*aj*−1,*j i* are copied every *P* clock cycles to Reg*aj*,*j i*+1 and Reg*aj*−1,*j i*+1, respectively, where *i* = 1, *i* = 2, ..., and �*M*/*P*� − 1. The last obtained intermediate score by *PE*2 based on Loop E*i* (Fig. 3), where *i* = 1, *i* = 2, ..., and �*M*/*P*� − 1, is stored in RegTmp*δj*−1 *i* and RegTmp*δj i*+1 every *P* clock cycles based on �*M*/*P*�-state pipelined LSC (Fig. 3). The last obtained intermediate score by *PE*2 based on Loop E�*M*/*P*� (Fig. 3) is stored in RegTmp*δj*−1 �*M*/*P*� and RegLast*δ* every *P* clock cycles during Loop B� (Fig. 3). The stored intermediate scores are required when starting LSC with new *M* output probabilities at the first computation by Loop E1 (Fig. 3). The required intermediate

+1(*j*), log *δt*�+2(*j*), ..., and

probabilities of all HMM are obtained by Loop D� (Fig. 3).

Loop E1 (Fig. 3). RegTmp*δj*−1 1 stores *M* intermediate scores log *δt*�

computing log *δt*(*j*) based on Eq. (3), when 2 ≤ *t* and 2 ≤ *j*.

score is read from RegLast*δ* and stored in RegTmp*δj* 1 before computation.

B�

log *δt*�

Fig. 5. Pipelined Viterbi scorer for the FastStoreBPP architecture.

#### **4. VLSI architecture for OPCs and LSCs with multiple store-based block parallel processing**

FastStoreBPP is obtained from StoreBPP by doubling the bit length of the input to OPC for reading two HMM parameters simultaneously.

We further extend the bit length of the input to OPC and we obtain *multiple store-based block parallel processing* (*MultipleStoreBPP*).

#### **4.1 Multiple store-based block parallel processing**

StoreBPP performs *M*/2 OPCs in parallel by using a register array of size *M* and *M*/2 *PE*1s, where *M*/2 OPCs are performed by a single HMM (Nakamura et al., 2010). We improve StoreBPP and further reduce the required register size for performing OPCs in parallel. We modify *M*/2*-parallel OPC*, where *M*/2 OPCs are performed in parallel, to deal with multiple HMMs by further extending the bit length of the input to OPC. By this bit-length extension, *L*� · *M*� /2 OPCs are performed in parallel, where *L*� HMM parameters can be read from ROM simultaneously. We call the modified parallel processing *MultipleStoreBPP*. Our MultipleStoreBPP performs *M*� /2*-parallel OPC*, where *M*� /2 OPCs are performed in parallel, to *L*� HMMs and *L*� · *M*� /2 OPCs are performed in parallel by using a register array of size *M*� .

A flowchart of our MultipleStoreBPP is shown in Fig. 6. Loops D2� , C1� , B��, and A�� in Fig. 6 are based on StoreBPP. In our MultipleStoreBPP, Loop D' (Fig. 3) is partially expanded as shown in Fig. 6 for performing *L*� · *M*� /2 OPCs in parallel in Loop A��. By the expansion, input feature vectors are effectively shared between different *M*� /2-parallel OPCs.

The flowchart consists of *L*� *M*� /2-parallel OPCs and *L*� �*M*� /(2 · *P*)�-stage pipelined LSCs. Each *M*� /2-parallel OPC and �*M*� /(2 · *P*)�-stage pipelined LSC are denoted by dashed and

double-dashed lines, respectively, in Fig. 6. Each *M*�

and Likelihood Score Computations of HMM-Based Recognition Systems

next computation in Loop A��. Then, the obtained *L*� · *M*�

**4.2 MultipleStoreBPP architecture for OPCs and LSCs**

in Fig. 3. These *L*� LSCs support LSC of (*v*� + 1)-th to (*v*� + *L*�

obtained every *P* clock cycles by Loop A��. In Loop A��, firstly, *L*� *M*�

ROM access −*μj*,*p*+1 of *L*� HMMs are performed simultaneously. Secondly, *L*� *M*�

HMMs, are computed with *L*� · *M*�

LSCs proceed simultaneously.

Reg*ω*), *L*� registers (Reg*μ*), and *L*� · *M*�

input feature vectors, **O***t*�+1, ..., and **O***t*�+*M*�

are shared by all *PE*1s, and the obtained first *M*�

/2+1, ..., and **O***t*�+2·*M*� , *L*� · *M*�

and hence �*M*�

but differ in number.

**O***t*�*M*�

Each �*M*�

in parallel with the same *PE*1s used in StoreBPP and FastStoreBPP, but differ in number.

165 A VLSI Architecture for Output Probability

pipelined LSC denoted by double-dashed lines in Fig. 3 with the same *PE*2s used in FastStoreBPP, but differ in number. Loops A� and B� (Fig. 3) correspond to Loops A�� and B��, respectively. By Loop A��, the output probabilities of *L*� HMMs, (*v*� + 1)-th to (*v*� + *L*�

OPCs and ROM access *σj*,*p*+1 of *L*� HMMs are performed simultaneously. 2 · *L*� HMM parameters are read from ROM using two cycles. These HMM parameters are needed for

LSCs, where each LSC is the same Viterbi scorer as that introduced for FastStoreBPP, as shown

Our MultipleStoreBPP VLSI architecture is shown in Fig. 7, where we assume *M*� ≤ 2 · *P*

and *L*� Viterbi scorers. The architecture has 2 · *L*� + 1 register arrays (Reg**O**, Reg*σ* and

*PE*1s in our MultipleStoreBPP and FastStoreBPP architectures, Yoshizawa et al. (2006), and Nakamura et al. (2010) are identical but differ in number. In addition, the architecture has 3 · *L*� register arrays (RegIn*δ*1, RegLast*δ*, and RegTmp*δj*−1 1), 3 · *L*� registers (Reg*aj*,*j* 1, Reg*aj*−1,*j* 1, and RegTmp*δj* 1), and *L*� *PE*2s for LSCs of *L*� HMMs. *PE*2 consists of three adders, two selectors and two comparators, which are used for LSC on the basis of Eqs. (2) and (3). *PE*2s in our MultipleStoreBPP and FastStoreBPP architectures and Yoshizawa et al. (2006) are identical

them in Reg**O** in OPC circuit1 (Fig. 7) based on Loop C1� (Fig. 6). *M*� · *P*/*L*� cycles are required for reading *M*� feature vectors. Then, the HMM parameters of *L*� HMMs, i.e., (*v*� + 1)-th to (*v*� + *L*)-th HMMs, are read from ROM, which are −*μ*11, *σ*11 and *ω*1, and stored in Reg*μ*, Reg*σ*, and Reg*ω*, respectively, based on Loop *C*1� and Loop *B*�� (Fig. 6). For half of the stored

(Fig. 7) based on Loop A�� (Fig. 6). Then, for the other half of the stored input feature vectors,

In each OPC circuit, denoted by dashed line in Fig. 7, the two HMM parameters, −*μjp* and *σjp*

intermediate results are stored in Reg*ω* using two cycles. At the same time, the two HMM parameters −*μjp*+1 and *σjp*+1 are read from ROM and stored in Reg*μ* and Reg*σ*, respectively,

A�� (Fig. 6). The stored *M*� input feature vectors are effectively shared by all OPC circuits.

are simultaneously computed with the stored HMM parameters by using *L*� · *M*�

/2, *L*� · *M*�

of two adders and two multipliers, which are used for computing *ωj* + ∑*P*

OPC starts by reading *M*� input feature vectors **O***t*�+1, ..., and **O***t*�

computed with the stored HMM parameters by using *L*� · *M*�

/(2 · *P*)� = 1. The MultipleStoreBPP architecture consists of *L*� OPC circuits

/(2 · *P*)�-stage pipelined LSC computes likelihood scores based on �*M*/*P*�-stage

/2-parallel OPC performs *M*�

/2 *PE*1s. These output probabilities are simultaneously

/2 *PE*1s for OPCs of *L*� HMMs. Each *PE*1 consists

/2 intermediate results of *L*� OPCs are simultaneously

/2 intermediate results and the second *M*�

/2 OPCs

)-th

/2-parallel

/2-parallel OPCs and

*p*=1 *σjp*(*otp* − *μjp*)2.

+*M*� from RAM and storing

/2 *PE*1s

/2

/2 intermediate results of *L*� OPCs

/2 *PE*1s (Fig. 7) based on Loop

/2 output probabilities are fed to *L*�

)-th HMMs. Loop A�� and *L*�

Fig. 6. Flowchart of OPCs and LSCs with MultipleStoreBPP.

10 Will-be-set-by-IN-TECH

*max* + *L*�

v� *max* = 0

*max* − *L*�

� + *M*�

v = v� + 1, *j* = 1, *p* = 1

), (*M*� · *P*/*L*� cycles)

� + 1 == 1) (2 cycles)

, v� = v�

For *L*� HMMs, load −μ1,1 and σ1,1 to Regμ and Regσ, respectively (2 cycles) *j* = 0 *j* = *j* + 1

� + 2, ..., *t*

For *L*� HMMs, load *aj*,*j* to Reg*aj*,*j* (and load *aj*−1,*j* to Reg*aj*−1,*j* when 2 ≤ *j*) (2 cycles)

> *p* = 0 *p* = *p* + 1

> > *M*�

*M*�

*t* � *max* = 0

v� *max* = v�

� + 1, *t*

For *L*� HMMs, load ω*j* to Regω (and load log π*j* to RegTmpδ*j* when *t* = *t*

*t* � = *t* � *max*, *t* � *max* = *t* � *max* + *M*�

Load **O***t* to Reg**O** (*t* = *t*

*M*�

For *L*� HMMs, load −μ*j*,*p*+1 to Regμ (1 cycle)

For *L*� HMMs, load σ*j*,*p*+1 to Regσ (1 cycle)

> /(2 · *P*)�-stage pipelined LSC

v = v� + 1

�*M*�

*M*�

�*M*�

/2 OPCs (*t* = 1, ..., *M*�

v = v� + 1

/2 OPCs (*t* = *M*�

v = v� + 1

/(2 · *P*)�-stage pipelined LSC

Fig. 6. Flowchart of OPCs and LSCs with MultipleStoreBPP.

v = v� + 2

/2)

/2+1, ..., *M*�

)

*p* ≥ *P*

For *L*� HMMs, copy Regω to RegInδ

YES

*j* ≥ *N*

 *. . .*

.........

 *. . .*

.........

 *. . .*

YES

double-dashed lines, respectively, in Fig. 6. Each *M*� /2-parallel OPC performs *M*� /2 OPCs in parallel with the same *PE*1s used in StoreBPP and FastStoreBPP, but differ in number. Each �*M*� /(2 · *P*)�-stage pipelined LSC computes likelihood scores based on �*M*/*P*�-stage pipelined LSC denoted by double-dashed lines in Fig. 3 with the same *PE*2s used in FastStoreBPP, but differ in number. Loops A� and B� (Fig. 3) correspond to Loops A�� and B��, respectively. By Loop A��, the output probabilities of *L*� HMMs, (*v*� + 1)-th to (*v*� + *L*� )-th HMMs, are computed with *L*� · *M*� /2 *PE*1s. These output probabilities are simultaneously obtained every *P* clock cycles by Loop A��. In Loop A��, firstly, *L*� *M*� /2-parallel OPCs and ROM access −*μj*,*p*+1 of *L*� HMMs are performed simultaneously. Secondly, *L*� *M*� /2-parallel OPCs and ROM access *σj*,*p*+1 of *L*� HMMs are performed simultaneously. 2 · *L*� HMM parameters are read from ROM using two cycles. These HMM parameters are needed for next computation in Loop A��. Then, the obtained *L*� · *M*� /2 output probabilities are fed to *L*� LSCs, where each LSC is the same Viterbi scorer as that introduced for FastStoreBPP, as shown in Fig. 3. These *L*� LSCs support LSC of (*v*� + 1)-th to (*v*� + *L*� )-th HMMs. Loop A�� and *L*� LSCs proceed simultaneously.

#### **4.2 MultipleStoreBPP architecture for OPCs and LSCs**

Our MultipleStoreBPP VLSI architecture is shown in Fig. 7, where we assume *M*� ≤ 2 · *P* and hence �*M*� /(2 · *P*)� = 1. The MultipleStoreBPP architecture consists of *L*� OPC circuits and *L*� Viterbi scorers. The architecture has 2 · *L*� + 1 register arrays (Reg**O**, Reg*σ* and Reg*ω*), *L*� registers (Reg*μ*), and *L*� · *M*� /2 *PE*1s for OPCs of *L*� HMMs. Each *PE*1 consists of two adders and two multipliers, which are used for computing *ωj* + ∑*P p*=1 *σjp*(*otp* − *μjp*)2. *PE*1s in our MultipleStoreBPP and FastStoreBPP architectures, Yoshizawa et al. (2006), and Nakamura et al. (2010) are identical but differ in number. In addition, the architecture has 3 · *L*� register arrays (RegIn*δ*1, RegLast*δ*, and RegTmp*δj*−1 1), 3 · *L*� registers (Reg*aj*,*j* 1, Reg*aj*−1,*j* 1, and RegTmp*δj* 1), and *L*� *PE*2s for LSCs of *L*� HMMs. *PE*2 consists of three adders, two selectors and two comparators, which are used for LSC on the basis of Eqs. (2) and (3). *PE*2s in our MultipleStoreBPP and FastStoreBPP architectures and Yoshizawa et al. (2006) are identical but differ in number.

OPC starts by reading *M*� input feature vectors **O***t*�+1, ..., and **O***t*� +*M*� from RAM and storing them in Reg**O** in OPC circuit1 (Fig. 7) based on Loop C1� (Fig. 6). *M*� · *P*/*L*� cycles are required for reading *M*� feature vectors. Then, the HMM parameters of *L*� HMMs, i.e., (*v*� + 1)-th to (*v*� + *L*)-th HMMs, are read from ROM, which are −*μ*11, *σ*11 and *ω*1, and stored in Reg*μ*, Reg*σ*, and Reg*ω*, respectively, based on Loop *C*1� and Loop *B*�� (Fig. 6). For half of the stored input feature vectors, **O***t*�+1, ..., and **O***t*�+*M*� /2, *L*� · *M*� /2 intermediate results of *L*� OPCs are simultaneously computed with the stored HMM parameters by using *L*� · *M*� /2 *PE*1s (Fig. 7) based on Loop A�� (Fig. 6). Then, for the other half of the stored input feature vectors, **O***t*�*M*�/2+1, ..., and **O***t*�+2·*M*� , *L*� · *M*� /2 intermediate results of *L*� OPCs are simultaneously computed with the stored HMM parameters by using *L*� · *M*� /2 *PE*1s (Fig. 7) based on Loop A�� (Fig. 6). The stored *M*� input feature vectors are effectively shared by all OPC circuits.

In each OPC circuit, denoted by dashed line in Fig. 7, the two HMM parameters, −*μjp* and *σjp* are shared by all *PE*1s, and the obtained first *M*� /2 intermediate results and the second *M*� /2 intermediate results are stored in Reg*ω* using two cycles. At the same time, the two HMM parameters −*μjp*+1 and *σjp*+1 are read from ROM and stored in Reg*μ* and Reg*σ*, respectively,

using two clock cycles. The HMM parameters are used in the next *M*�

and Likelihood Score Computations of HMM-Based Recognition Systems

Each Viterbi scorer, denoted by double-dashed lines in Fig. 7, performs �*M*�

*M*� input feature vectors **O***t*�+1, ..., and **O***t*�+*M*� .

+1(1), ..., and log *δt*�+*M*�

scorers shown in Fig. 7 are instances of the generalized �*M*�

/(2 · *P*)� = 1. The �*M*�

*L*� · *M*� output probabilities are simultaneously obtained every 2 · *P* clock cycles by *L*� OPC circuits, which are log *bj*(**O***t*�+1), ..., and log *bj*(**O***t*�+*M*�) of *L*� HMMs based on Loop A�� (Fig. 6). The results are copied from Reg*ω* to RegIn*δ*1 for starting LSCs of *L*� HMMs and next OPCs for (*j* + 1)-th states of *L*� HMMs log *bj*+1(**O***t*�+1), ..., and log *bj*+1(**O***t*�+*M*�) based on Loop B�� (Fig. 6). *L*� · *M*� · *N* output probabilities of *L*� HMMs are obtained by Loop B�� with the same

167 A VLSI Architecture for Output Probability

pipelined LSC, denoted by double-dashed lines in Fig. 3. LSC starts by reading HMM parameters of *L*� HMMs, log*π*1 and log *a*1,1, from ROM and storing them in RegTmp*δj* 1 and RegTmp*aj*,*j* 1 based on Loop B�� (Fig. 6). Then, in each Viterbi scorer, an intermediate score log *δ*1(1) is computed by *PE*2 with the HMM parameter and the output probability obtained using Eq. (1). The obtained intermediate score is stored in both RegTmp*δj* 1 and RegTmp*δj*−1 1 (Fig. 7). RegTmp*δj* 1 stores an intermediate score that is needed in the next computation in Loop E1 (Fig. 3). RegTmp*δj*−1 1 stores *M*� intermediate scores log *δt*�+1(*j*), ..., and log *δt*�+*M*�(*j*), which is needed in the next LSC for the (*j* + 1)-th state of the HMM in Loop B��. After the computation of log *δ*1(1), *M*� − 1 intermediate scores log *δ*2(1), ..., and log *δM*�(1) are sequentially computed by *PE*2 on the basis of Eq. (3). In sequential computation, the last obtained intermediate score log *δM*�(1), is stored in RegTmp*δj*−1 1 and RegLast*δ* (Fig. 7). RegLast*δ* stores *N* intermediate scores that are the last obtained intermediate scores by Loop E1 during Loop B��. These intermediate scores are log *δt*�+*M*�(1), ..., and log *δt*�+*M*�(*N*) of *v*-th HMM, which are required when starting LSC with new *M*� output

RegLast*δ* and is stored in RegTmp*δj* 1 before computation. Reg*aj*−1,*j* 1 (Fig. 7) stores an HMM parameter log *aj*−1,*j*, which is used for computing log *δt*�(*j*) on the basis of Eq. (3) when 2 ≤ *t*�

Our Viterbi scorers, which support MultipleStoreBPP, were presented in Fig. 7 as Viterbi

scorer consists of two register arrays (RegIn*δi* and RegTmp*δj*−1 *i*), three registers (RegTmp*δj i*, Reg*aj*,*j i*, and Reg*aj*−1,*j i*), and a *PE*2. Each RegIn*δi* consists of *i* · *P* registers, where *i* = 1, ...,

In each RegIn*δi*, rows are shifted upward every *P* clock cycles. Each RegTmp*δj*−1 *i* consists

*M*� mod (2 · *P*) registers. HMM parameters in Reg*aj*,*j i* and Reg*aj*−1,*j i* are copied every

last obtained intermediate score by *PE*2 based on Loop E*i* (Fig. 3), where *i* = 1, ..., and

/(2 · *P*)� − 1, is stored in RegTmp*δj*−1 *i* and RegTmp*δj i*+1 every *P* cycles based on

/(2 · *P*)�-stage pipelined LSC (Fig. 3). The last obtained intermediate score by *PE*2 based

/(2·*P*)� consists of �*M*�

*P* cycles to Reg*aj*,*j i*+1 and Reg*aj*−1,*j i*+1, respectively, where *i* = 1, ..., �*M*�

/(2 · *P*)� = 1. Our �*M*�

+1), ..., and log *bj*(**O***t*�+2·*M*�) at the first computations in Loop E1,

+1(*N*). A required intermediate score is read from

/(2 · *P*)�-stage pipelined Viterbi scorer consists of a

/(2 · *P*)� sub Viterbi scorers. The *i*-th stage, i.e., *i*-th sub Viterbi

/(2 · *P*)� − 1. RegTmp*δj*−1 �*M*�

/(2 · *P*)�-stage pipelined Viterbi

/(2 · *P*)�-stage pipelined Viterbi

/(2·*P*)� consists of

/*P*� − 1. The

/(2 · *P*)�, is shown in Fig. 8. The Viterbi

/(2 · *P*)� · (*M*� mod (2 · *P*)) registers.

in the OPC circuit.

probabilities log *bj*(**O***t*�+*M*�

scorer*i*, where *M*� ≤ 2 · *P* and �*M*�

register array RegLast*δ* and �*M*�

/(2 · *P*)� − 1. RegIn*δ*�*M*�

of *P* registers, where *i* = 1, ..., and �*M*�

scorer, which supports 2 · *P < M*� and 1 *<* �*M*�

given as log *δt*�+*M*�

scorer where �*M*�

and 2 ≤ *j*.

and �*M*�

�*M*�

�*M*�

/(2 · *P*)�-stage

Fig. 7. MultipleStoreBPP architecture for OPCs and LSCs.

12 Will-be-set-by-IN-TECH

. . .

. . .

M�

Regω

. . .

M�

SEL SEL

CMP <

*PE*2

CMP (*t* == 1)

+

. . .

M�

RegTmpδ*j* 1

Reg*aj*,*j* 1

Viterbi scorer1

. . .

Viterbi scorer2

. . .

M�

RegTmpδ*j* 1

Reg*aj*,*j* 1

. . .

M�

RegTmpδ*j* 1

Reg*aj*,*j* 1

Viterbi scorer*L*�

. . .

N

Reg*aj*−1,*j* 1

RegTmpδ*j*−1 1

Reg*aj*−1,*j* 1

RegLastδ

RegTmpδ*j*−1 1

RegInδ1

+ +

. . .

N

RegLastδ

RegInδ1

+ +

. . .

OPC circuit1

ROM μ, σ, ω, *a*, *a*, π

. . .

*PE*1*M*�/2

. . .

Regμ

Reg**O** Regω

*PE*11

RAM **O**

. . .

*PE*1*M*�/2

Regμ

*PE*11

. . .

*PE*1*M*�/2

Fig. 7. MultipleStoreBPP architecture for OPCs and LSCs.

 The HMM parameters are used in the next *M*� -parallel computation in the OPC circuit.

*L*� · *M*� output probabilities are simultaneously obtained every 2 · *P* clock cycles by *L*� OPC circuits, which are log *bj*(**O***t*�+1), ..., and log *bj*(**O***t*�+*M*�) of *L*� HMMs based on Loop A�� (Fig. 6). The results are copied from Reg*ω* to RegIn*δ*1 for starting LSCs of *L*� HMMs and next OPCs for (*j* + 1)-th states of *L*� HMMs log *bj*+1(**O***t*�+1), ..., and log *bj*+1(**O***t*�+*M*�) based on Loop B�� (Fig. 6). *L*� · *M*� · *N* output probabilities of *L*� HMMs are obtained by Loop B�� with the same *M*� input feature vectors **O***t*�+1, ..., and **O***t*�+*M*� .

Each Viterbi scorer, denoted by double-dashed lines in Fig. 7, performs �*M*� /(2 · *P*)�-stage pipelined LSC, denoted by double-dashed lines in Fig. 3. LSC starts by reading HMM parameters of *L*� HMMs, log*π*1 and log *a*1,1, from ROM and storing them in RegTmp*δj* 1 and RegTmp*aj*,*j* 1 based on Loop B�� (Fig. 6). Then, in each Viterbi scorer, an intermediate score log *δ*1(1) is computed by *PE*2 with the HMM parameter and the output probability obtained using Eq. (1). The obtained intermediate score is stored in both RegTmp*δj* 1 and RegTmp*δj*−1 1 (Fig. 7). RegTmp*δj* 1 stores an intermediate score that is needed in the next computation in Loop E1 (Fig. 3). RegTmp*δj*−1 1 stores *M*� intermediate scores log *δt*�+1(*j*), ..., and log *δt*�+*M*�(*j*), which is needed in the next LSC for the (*j* + 1)-th state of the HMM in Loop B��. After the computation of log *δ*1(1), *M*� − 1 intermediate scores log *δ*2(1), ..., and log *δM*�(1) are sequentially computed by *PE*2 on the basis of Eq. (3). In sequential computation, the last obtained intermediate score log *δM*�(1), is stored in RegTmp*δj*−1 1 and RegLast*δ* (Fig. 7). RegLast*δ* stores *N* intermediate scores that are the last obtained intermediate scores by Loop E1 during Loop B��. These intermediate scores are log *δt*�+*M*�(1), ..., and log *δt*� +*M*�(*N*) of *v*-th HMM, which are required when starting LSC with new *M*� output probabilities log *bj*(**O***t*�+*M*� +1), ..., and log *bj*(**O***t*�+2·*M*�) at the first computations in Loop E1, given as log *δt*�+*M*� +1(1), ..., and log *δt*�+*M*� +1(*N*). A required intermediate score is read from RegLast*δ* and is stored in RegTmp*δj* 1 before computation. Reg*aj*−1,*j* 1 (Fig. 7) stores an HMM parameter log *aj*−1,*j*, which is used for computing log *δt*�(*j*) on the basis of Eq. (3) when 2 ≤ *t*� and 2 ≤ *j*.

Our Viterbi scorers, which support MultipleStoreBPP, were presented in Fig. 7 as Viterbi scorer*i*, where *M*� ≤ 2 · *P* and �*M*� /(2 · *P*)� = 1. Our �*M*� /(2 · *P*)�-stage pipelined Viterbi scorer, which supports 2 · *P < M*� and 1 *<* �*M*� /(2 · *P*)�, is shown in Fig. 8. The Viterbi scorers shown in Fig. 7 are instances of the generalized �*M*� /(2 · *P*)�-stage pipelined Viterbi scorer where �*M*� /(2 · *P*)� = 1. The �*M*� /(2 · *P*)�-stage pipelined Viterbi scorer consists of a register array RegLast*δ* and �*M*� /(2 · *P*)� sub Viterbi scorers. The *i*-th stage, i.e., *i*-th sub Viterbi scorer consists of two register arrays (RegIn*δi* and RegTmp*δj*−1 *i*), three registers (RegTmp*δj i*, Reg*aj*,*j i*, and Reg*aj*−1,*j i*), and a *PE*2. Each RegIn*δi* consists of *i* · *P* registers, where *i* = 1, ..., and �*M*� /(2 · *P*)� − 1. RegIn*δ*�*M*� /(2·*P*)� consists of �*M*� /(2 · *P*)� · (*M*� mod (2 · *P*)) registers.

In each RegIn*δi*, rows are shifted upward every *P* clock cycles. Each RegTmp*δj*−1 *i* consists of *P* registers, where *i* = 1, ..., and �*M*� /(2 · *P*)� − 1. RegTmp*δj*−1 �*M*� /(2·*P*)� consists of *M*� mod (2 · *P*) registers. HMM parameters in Reg*aj*,*j i* and Reg*aj*−1,*j i* are copied every *P* cycles to Reg*aj*,*j i*+1 and Reg*aj*−1,*j i*+1, respectively, where *i* = 1, ..., �*M*� /*P*� − 1. The last obtained intermediate score by *PE*2 based on Loop E*i* (Fig. 3), where *i* = 1, ..., and �*M*� /(2 · *P*)� − 1, is stored in RegTmp*δj*−1 *i* and RegTmp*δj i*+1 every *P* cycles based on �*M*� /(2 · *P*)�-stage pipelined LSC (Fig. 3). The last obtained intermediate score by *PE*2 based

MultipleStoreBPP [bit]

169 A VLSI Architecture for Output Probability

+ (*M* mod 2 · *P*) + 1 + (*M* mod 2 · *P*) · �*M*/(2 · *P*)�} · *x f*

FastStoreBPP [bit]

StreamBPP Yoshizawa et al. (2006) [bit]

StoreBPP Nakamura et al. (2010) [bit]

MultipleStoreBPP [cycle]

FastStoreBPP [cycle] OPC *V* · {*P* · �*M*/2� + (1 + *P*) · *N*}�*T*/*M*�

StreamBPP Yoshizawa et al. (2006)[cycle]

StoreBPP Nakamura et al. (2010) [cycle] OPC �*V*/*L*�{*P* · *M* + (1 + 2 · *P*) · *L* · *N*}�*T*/*M*�

Table 3 shows the register size, processing time, and the number of *PEs* for computing output probabilities of 800 HMMs and likelihood scores, where it is assumed that *N* = 32, *P* = 38, *T* = 86, *xμ* = 8, *xσ* = 8, *x f* = 24, *xo* = 8, *xa* = 8, and *V* = 800. These values are the same as those used in a recent circuit design for isolated word recognition Nakamura et al. (2010); Yoshizawa et al. (2004; 2006). In addition, we assume that *M*� = 12, *L*� = 4 for the MultipleStoreBPP architecture. Futhermore, ratios compared with StreamBPP are shown in Table 3. Compared with the StreamBPP architectures, the MultipleStoreBPP architecture has fewer registers (48% = 10,432/21,752) and requires less processing time (91% = 4,233,600/4,661,600). The number of *PE*2s in MultipleStoreBPP and FastStoreBPP are less

Figure 9 shows the processing time, and the number of *PE*s in MultipleStoreBPP, FastStoreBPP,

�{2 · *N* · �*T*/*M*�

/*L*�

� + (1 + 2 · *P*) · *N*}�*T*/*M*�

� + *N*}

�{�*P* · *M*�

*L*� · [{*N* + (2 · *P* + 1)(�(*M* + 1)/(2 · *P*)� − 1) + 2 · *P* · ∑�(*M*+1)/(2·*P*)�−1

*i*=0 *i*

*i*=0 *i* + (*M* mod *P*)

�

/2 for MultipleStoreBPP.

 Register size.

Table 2. Processing times.

than that in StreamBPP.

Fig. 8. Pipelined Viterbi scorer for the MultipleStoreBPP architecture (Viterbi scorer*i*, *i* = 1, ..., *L*� ).

on Loop E�*M*� /(2·*P*)� (Fig. 3) is stored in RegTmp*δj*−1 �*M*� /(2·*P*)� and RegLast*δ* every *P* cycles during Loop B�� (Fig. 6). The stored intermediate scores are required when starting LSC with new *M*� output probabilities for first computation in Loop E1 (Fig. 3). The required intermediate score is read from RegLast*δ* and stored in RegTmp*δj* 1 before computations.

#### **5. Evaluation**

We compared StreamBPP Yoshizawa et al. (2004; 2006), StoreBPP Nakamura et al. (2010), FastStoreBPP (Figs. 3, 4, and 5), and MultipleStoreBPP (Figs. 6, 7, and 8) VLSI architectures.

Table 1 shows the register size of MultipleStoreBPP, FastStoreBPP, StoreBPP, and StreamBPP architectures, where *xμ*, *xσ*, *xω*, *xo* , *xa*, and *x f* represent the bit length of *μjp*, *σjp*, *ωj*, *otp*, *ajj*, and the output of *PE*1, respectively. *N*, *P*, and *M* are the number of HMM states, the dimension of the input feature vector, and the number of input feature vectors in a block, respectively. *M*� and *L*� are the number of input feature vectors in a block with MultipleStoreBPP and the number of HMMs whose output probabilities are simultaneously computed by Loop A�� (Fig. 6) with MultipleStoreBPP, respectively. OPC and Viterbi scorer represent the register size of the OPC circuit and Viterbi scorer, respectively.

Table 2 shows the processing time for computing output probabilities of *V* HMMs and likelihood scores with MultipletStoreBPP, FastStoreBPP, StoreBPP, and StreamBPP architectures, where *L* is the number of HMMs whose output probabilities are computed using the same input feature vectors with StoreBPP Nakamura et al. (2010). OPC and the Viterbi scorer represent the number of clock cycles for OPC and additional cycles for LSC, respectively.

Table 1. Register size.

14 Will-be-set-by-IN-TECH

. . .

RegTmpδ*j* 1 RegTmpδ*j* �*M*� /(2·*P*)�

Reg*aj*,*j* �*M*� /(2·*P*)� Reg*aj*−1,*j* 1

. . .

SEL SEL SEL SEL +

< CMP

+ +

. . .

�*M*�

. . .

during Loop B�� (Fig. 6). The stored intermediate scores are required when starting LSC with new *M*� output probabilities for first computation in Loop E1 (Fig. 3). The required intermediate score is read from RegLast*δ* and stored in RegTmp*δj* 1 before computations.

We compared StreamBPP Yoshizawa et al. (2004; 2006), StoreBPP Nakamura et al. (2010), FastStoreBPP (Figs. 3, 4, and 5), and MultipleStoreBPP (Figs. 6, 7, and 8) VLSI architectures. Table 1 shows the register size of MultipleStoreBPP, FastStoreBPP, StoreBPP, and StreamBPP architectures, where *xμ*, *xσ*, *xω*, *xo* , *xa*, and *x f* represent the bit length of *μjp*, *σjp*, *ωj*, *otp*, *ajj*, and the output of *PE*1, respectively. *N*, *P*, and *M* are the number of HMM states, the dimension of the input feature vector, and the number of input feature vectors in a block, respectively. *M*� and *L*� are the number of input feature vectors in a block with MultipleStoreBPP and the number of HMMs whose output probabilities are simultaneously computed by Loop A�� (Fig. 6) with MultipleStoreBPP, respectively. OPC and Viterbi scorer

Table 2 shows the processing time for computing output probabilities of *V* HMMs and likelihood scores with MultipletStoreBPP, FastStoreBPP, StoreBPP, and StreamBPP architectures, where *L* is the number of HMMs whose output probabilities are computed using the same input feature vectors with StoreBPP Nakamura et al. (2010). OPC and the Viterbi scorer represent the number of clock cycles for OPC and additional cycles for LSC,

Fig. 8.

*N*

RegLastδ

Reg*aj*−1,*j* �*M*� /(2·*P*)�

. . .

+

. . . RegInδ1

*P*

<

. . .

*M*�

*i* = 1, ..., *L*�

on Loop E�*M*�

**5. Evaluation**

respectively.

).

. . .

*PE*2 *PE*2

CMP (*t* == 1) CMP (*t* == 1)

+

*P*

Table 2. Processing times.

Table 3 shows the register size, processing time, and the number of *PEs* for computing output probabilities of 800 HMMs and likelihood scores, where it is assumed that *N* = 32, *P* = 38, *T* = 86, *xμ* = 8, *xσ* = 8, *x f* = 24, *xo* = 8, *xa* = 8, and *V* = 800. These values are the same as those used in a recent circuit design for isolated word recognition Nakamura et al. (2010); Yoshizawa et al. (2004; 2006). In addition, we assume that *M*� = 12, *L*� = 4 for the MultipleStoreBPP architecture. Futhermore, ratios compared with StreamBPP are shown in Table 3. Compared with the StreamBPP architectures, the MultipleStoreBPP architecture has fewer registers (48% = 10,432/21,752) and requires less processing time (91% = 4,233,600/4,661,600). The number of *PE*2s in MultipleStoreBPP and FastStoreBPP are less than that in StreamBPP.

Figure 9 shows the processing time, and the number of *PE*s in MultipleStoreBPP, FastStoreBPP, and StreamBPP architectures, and the value of *M*, where *M* = *M*� · *L*� /2 for MultipleStoreBPP.

0

number of *PE*1s is 44.

MultipleStoreBPP architecture.

**6. Conclusions**

Fig. 10. Register size and the value of *M*.

10 20 30 40 50 60 70 80 90

StreamBPP FastStoreBPP MultipleStoreBPP

and Likelihood Score Computations of HMM-Based Recognition Systems

171 A VLSI Architecture for Output Probability

The value of M(block size of FastStoreBPP)

MultipleStoreBPP (#*PE*1 = 44, *M* = 22, *L* = 4) 1,042,492 2.8 4.5 FastStoreBPP (#*PE*1 = 32, *M* = 32) 849,955 2.7 3.5 FastStoreBPP (#*PE*1 = 44, *M* = 44) 1,155,595 2.7 4.8

minimum clock period and area of the circuit, respectively. Power represents the power dissipation of the circuit whose operating clock frequency is 11 MHz, and for an 800-word real-time isolated word recognition, recognition in 0.2 s is achieved by the MultipleStoreBPP architecture for *T* = 86, a 1-s speech, *V* = 800, *N* = 32, *P* = 38, *M* = 22 and *L* = 4. Compared with the FastStoreBPP architecture, the MultipleStoreBPP architecture has lower power and less area, because the MultipleStoreBPP architecture has fewer registers when the

We presented MultipleStoreBPP for OPCs and LSCs and presented a new VLSI architecture. MultipleStoreBPP performs parallel-OPCs and pipelined-LSCs for multiple HMMs. Compared with the conventional StoreBPP architecture, the MultipleStoreBPP architecture supports LSC. Furthermore, compared with StreamBPP and FastStoreBPP architectures, the MultipleStoreBPP architecture requires fewer registers and less processing time. In terms of the VLSI architecture the comparison shows the efficiency of the

area [*μ*m2]

delay [ns]

power [mW]

Arch.

Table 4. Area, delay and power of OPC and LSC circuits.

5

10

15

20

Register Size [Kbit]

25

30

35

40

Table 3. Evaluation of the MultipleStoreBPP, FastStoreBPP, and StreamBPP performance.

Fig. 9. Processing time, the number of *PE*s and value of *M*.

This graph shows that the processing time of MultipleStoreBPP is less than that of FastStoreBPP architecture. It is also less than that of the StreamBPP architecture when *M* is greater than 22.

Figure 10 shows the register size of MultipleStoreBPP, FastStoreBPP, and StreamBPP architectures as well as the value of *M* (block size). This graph shows that the register size of MultipleStoreBPP is less than those of FastStoreBPP, and StreamBPP architectures when *M* is greater than 36 and less than 74.

Table 4 shows the circuit area, clock period, and power dissipation of the OPC and LSC circuits based on the MultipleStoreBPP and FastStoreBPP architectures, which are derived from the report of the Synopsys Design Compiler (Ver. B-2008.09-SP5), where the target technology is the 90nm technology (STARC 90nm) and the report on power dissipation is obtained with report\_power command after logic synthesis. In the table, the delay and area represent the

Fig. 10. Register size and the value of *M*.

Table 4. Area, delay and power of OPC and LSC circuits.

minimum clock period and area of the circuit, respectively. Power represents the power dissipation of the circuit whose operating clock frequency is 11 MHz, and for an 800-word real-time isolated word recognition, recognition in 0.2 s is achieved by the MultipleStoreBPP architecture for *T* = 86, a 1-s speech, *V* = 800, *N* = 32, *P* = 38, *M* = 22 and *L* = 4. Compared with the FastStoreBPP architecture, the MultipleStoreBPP architecture has lower power and less area, because the MultipleStoreBPP architecture has fewer registers when the number of *PE*1s is 44.

#### **6. Conclusions**

16 Will-be-set-by-IN-TECH

)=(12, 4) 10,432

Table 3. Evaluation of the MultipleStoreBPP, FastStoreBPP, and StreamBPP performance.

MultipleStoreBPP (*M*

#PEs (FastStoreBPP) 12+1

*12+4*

*16+4*

0

is greater than 36 and less than 74.

is greater than 22.

2

4

6

Processing Time [Mcycle]

8

10

, *L*

16+1

20+1

*#PEs(MultipleStoreBPP)*

Fig. 9. Processing time, the number of *PE*s and value of *M*.

*32+4, ...*

*24+4*

FastStoreBPP (*M* = 24) 9,848

StreamBPP 21,752

Reg. Proc. #*PEs*

4,233,600 (91%)

5,580,800 (120%)

4,661,600 (100%)

#PEs = #PE1s + #PE2s #PEs(StreamBPP) = all 32 + 32

StreamBPP FastStoreBPP MultipleStoreBPP

44+2, ..., ..., 84+3

[cycle] PE1 PE2

24

24

32 (100%)

(75%) 4 (13%)

(75%) 1 (3%)

*..., 84+4*

32 (100%)

size [bit] time

(48%)

(45%)

(100%)

0 10 20 30 40 50 60 70 80 90

The value of M(block size of FastStoreBPP)

This graph shows that the processing time of MultipleStoreBPP is less than that of FastStoreBPP architecture. It is also less than that of the StreamBPP architecture when *M*

Figure 10 shows the register size of MultipleStoreBPP, FastStoreBPP, and StreamBPP architectures as well as the value of *M* (block size). This graph shows that the register size of MultipleStoreBPP is less than those of FastStoreBPP, and StreamBPP architectures when *M*

Table 4 shows the circuit area, clock period, and power dissipation of the OPC and LSC circuits based on the MultipleStoreBPP and FastStoreBPP architectures, which are derived from the report of the Synopsys Design Compiler (Ver. B-2008.09-SP5), where the target technology is the 90nm technology (STARC 90nm) and the report on power dissipation is obtained with report\_power command after logic synthesis. In the table, the delay and area represent the We presented MultipleStoreBPP for OPCs and LSCs and presented a new VLSI architecture. MultipleStoreBPP performs parallel-OPCs and pipelined-LSCs for multiple HMMs. Compared with the conventional StoreBPP architecture, the MultipleStoreBPP architecture supports LSC. Furthermore, compared with StreamBPP and FastStoreBPP architectures, the MultipleStoreBPP architecture requires fewer registers and less processing time. In terms of the VLSI architecture the comparison shows the efficiency of the MultipleStoreBPP architecture.

**9**

*Taiwan*

**for Speech Processing**

*1National Cheng Kung University 2Tung Fang Design University*

Jhing-Fa Wang1, Po-Chun Lin2 and Bo-Wei Chen1

**Design and Applications of Embedded Systems**

This chapter focuses on speech processing techniques, which involve speech feature extraction, sound localization, speaker identification/verification, and interactive retrieval of spoken documents. Several hardware design issues are discussed in each section. Speech processing applications frequently involve extensive mathematical computation, making resource and power consumption management important. Therefore, this chapter presents not only algorithms but also their corresponding improved solutions to embedded systems, such as fixed-point arithmetic design, field-programmable gate array (FPGA) verification, ARM-based system-on-a-programmable-chip (SoPC) architecture, and other single-chip

The rest of this chapter is organized as follows. Section 2 introduces the feature extraction method that is used in speech processing. Section 3 then describes details of the sound localization technique. Next, Sections 4 and 5 elucidate speaker identification/verification and interactive retrieval of spoken documents. Conclusions are finally drawn in Section 6,

Speech feature extraction is critical in speech processing applications. This section describes in detail frequently used speech features and the design of chips for extracting them. The computational complexity and memory requirement of the associated algorithms are also analyzed in detail to ensure favorable performance. Furthermore, a hybrid approach for fixed-point arithmetic and hardware design is developed to ensure low computational complexity. Finally, a single FPGA development board is considered as a case study to

The complete step-by-step process for calculating coefficients is described as follows (Vergin

**1. Introduction**

designs.

realize the design.

**2.1 Methodology**

et al., 1996; Wang et al*.*, 2003).

along with recommendations for future research.

**2. Embedded system design for speech feature extraction**

**2.1.1 Algorithm for calculating mel-frequency cepstral coefficients**

#### **7. Acknowledgements**

This work is supported by the VLSI Design and Education Center (VDEC), the University of Tokyo in collaboration with Synopsys, Inc. and the Semiconductor Technology Academic Research Center (STARC).

#### **8. References**

### **Design and Applications of Embedded Systems for Speech Processing**

Jhing-Fa Wang1, Po-Chun Lin2 and Bo-Wei Chen1 *1National Cheng Kung University 2Tung Fang Design University Taiwan*

#### **1. Introduction**

18 Will-be-set-by-IN-TECH

172 Embedded Systems – High Performance Systems, Applications and Projects

This work is supported by the VLSI Design and Education Center (VDEC), the University of Tokyo in collaboration with Synopsys, Inc. and the Semiconductor Technology Academic

B. Mathew; A. Davis & A. Ibrahim. (2003). Perception Coprocessors for Embedded Systems.

B. Mathew; A. Davis & Z. Fang. (2003). A Low-Power Accelerator for the SPHINX 3 Speech

K. Nakamura; M. Yamamoto; K. Takagi & N. Takagi. (2010). A VLSI Architecture for Output

S. Yoshizawa; N. Wada; N. Hayasaka & Y. Miyanaga. (2004). Scalable Architecture for Word

S. Yoshizawa; N. Wada; N. Hayakawa & Y. Miyanaga. (2006). Scalable Architecture for Word

Y. Kim & H. Jeong. (2007). A Systolic FPGA Architecture of Two-Level Dynamic Programming

HMM-Based Speech Recognition. *Proc. of ISCAS'04*, 417 – 420

*IEEE TRANS. ON CIRC. & SYST.*, Vol. 53, No. 1, 70 – 77

Recognition System. *Proc. of Int'l Conf. on Compilers, Architecture and Synthesis for*

Probability Computations of HMM-Based Recognition Systems with Store-Based Block Parallel Processing, *IEICE TRANS. INF. & SYST.*, Vol. E93-D, No. 2, 300 – 305 S. Yoshizawa; Y. Miyanaga & N. Yoshida. (2002). On a High-Speed HMM VLSI Module with

Block Parallel Processing. *IEICE TRANS. Fundamentals*, Vol. J85-A, No. 12, 1440 – 1450

HMM-based Speech Recognition and VLSI Implementation in Complete System.

for Connected Speech Recognition. *IEICE TRANS. INF. & SYST.*, Vol. E90-D, No. 2,

**7. Acknowledgements**

Research Center (STARC).

562 – 568

*Proc. of ESTIMedia*, 109 – 116

*Embedded Systems*, 210 – 219

**8. References**

This chapter focuses on speech processing techniques, which involve speech feature extraction, sound localization, speaker identification/verification, and interactive retrieval of spoken documents. Several hardware design issues are discussed in each section. Speech processing applications frequently involve extensive mathematical computation, making resource and power consumption management important. Therefore, this chapter presents not only algorithms but also their corresponding improved solutions to embedded systems, such as fixed-point arithmetic design, field-programmable gate array (FPGA) verification, ARM-based system-on-a-programmable-chip (SoPC) architecture, and other single-chip designs.

The rest of this chapter is organized as follows. Section 2 introduces the feature extraction method that is used in speech processing. Section 3 then describes details of the sound localization technique. Next, Sections 4 and 5 elucidate speaker identification/verification and interactive retrieval of spoken documents. Conclusions are finally drawn in Section 6, along with recommendations for future research.

#### **2. Embedded system design for speech feature extraction**

Speech feature extraction is critical in speech processing applications. This section describes in detail frequently used speech features and the design of chips for extracting them. The computational complexity and memory requirement of the associated algorithms are also analyzed in detail to ensure favorable performance. Furthermore, a hybrid approach for fixed-point arithmetic and hardware design is developed to ensure low computational complexity. Finally, a single FPGA development board is considered as a case study to realize the design.

#### **2.1 Methodology**

#### **2.1.1 Algorithm for calculating mel-frequency cepstral coefficients**

The complete step-by-step process for calculating coefficients is described as follows (Vergin et al., 1996; Wang et al*.*, 2003).

**Step 1.** Short-time fast Fourier transform (FFT)

$$\begin{cases} Y(m) = \frac{1}{F} \sum\_{n=0}^{F-1} z(n) w(n) \mathcal{W}\_F^{nm} \\ w(n) = \beta(0.5 - 0.5 \cos \frac{2\pi n}{F-1}) \end{cases} \tag{1}$$

Design and Applications of Embedded Systems for Speech Processing 175

Addition/subtraction *L*(*M-1*) *M*(*F*/2*-*1) 12x19=228 20x127=2540 2768 Multiplication *LM MF*/2 12x20=240 20x128=2560 2800 Logarithm *M* 0 20 0 20 ROM size (words) *LM MF*/2 12x20=240 20x128=2560 2800 Table 1. Number of operations and the required memory estimated using the original MFCC

Improvements

subtraction *L*(M/2+M/2-1) 0 12x19=228 113 341 87.6 Multiplication *LM*/2 50 12x10=120 128 248 91.1 Logarithm *M* 0 20 0 20 0

(words) *LM*/2 50 12x10=120 128 228 91.8

Table 2. Improvement of *C*[*n*] calculation by rescheduling the original MFCC algorithm and

The modified procedure in the MFCC algorithm is based mainly on the improved *S k*[ ] calculation, as discussed below and shown in Fig. 1 (Wang et al., 2000). Every Mel-window is divided into two blocks with equal bandwidth on the Mel-scale. Because the Melwindows overlap each other, every block except for the first and last belongs to two Mel-

**required memory** *K***=256,** *M***=20,** *L***=12**  Computational power Actual computational power *C*[*n*] *S*[*k*] *C*[*n*] *S*[*k*] Total

*C***[***n***] calculation Total improvement**

(%) *C*[*n*] *S*[*k*] Total Improvement

(%)

**Number of operations and**

**Improvement of**

algorithm.

Addition/

ROM size

windows.

Operation Computational

power

the total improvement provided by the proposed method

Fig. 1. Modified procedure for calculating *S*[*k*].

**Step 2.** Find the energy spectrum, 2 *Xm Ym* ( ) .

**Step 3.** Calculate the energy in each channel:

$$\begin{cases} S[k] = \sum\_{j=0}^{F/2-1} \mathcal{W}\_k(j)X(j) \\ \sum\_{j=0}^{F/2-1} \mathcal{W}\_k(j) = 1 \end{cases} \tag{2}$$

**Step 4.** Take the logarithm and perform the cosine transform to obtain the Mel-frequency cepstral coefficients (MFCCs),

$$\mathbb{C}\left[n\right] = \sum\_{k=0}^{M-1} \log(\mathbb{S}[k]) \cos[n(k+0.5)\frac{\pi}{M}] \tag{3}$$

#### **2.1.2 Improved algorithm for calculating mel-frequency cepstral coefficients**

Generally, the required computational power and ROM in each frame can be determined clearly according to Table 1 (Wang et al*.*, 2000; Wang et al., 2003). As shown in the table, the total required computational power is quite high due to the redundant operations and memory that stores the required constants. Accordingly, some modifications must be made to reduce the computational load.

The weighted energy spectrum in the Mel-window, 1( ) *E j k* , can be obtained by subtracting the weighted energy spectrum ( ) *E j k* from energy spectrum *X j* ( ) . All of the multiplications in (4) can be replaced by subtraction operations. Therefore, the memory required to store the weight constants for (4) becomes redundant and can be eliminated.

$$E\_{k+1}\left(j\right) = X(j)\frac{d - \left(L + D\right)}{\left(L + 2D\right) - \left(L + D\right)}\tag{4}$$

Additionally, applying the symmetric property of the cosine function to (3) flattens all of the operations and enables related items to be combined in a new formula, given below.

$$\mathbb{C}\left[n\right] = \sum\_{k=0}^{M/2-1} \{ \log \left( S \left[ k \right] \right) + (-1)^n \log \left( S \left[ M - k - 1 \right] \right) \} \cos[n(k + 0.5) \frac{\pi}{M}] \tag{5}$$

Therefore, the computational complexity of *C*[*n*] operations can be re-estimated, and the result is given in Table 2 (Wang et al., 2000; Wang et al., 2003).

*n Y m z n wnW F*

*w n cos*

1

0 1 ( )

*Sk W jXj*

**Step 4.** Take the logarithm and perform the cosine transform to obtain the Mel-frequency

Generally, the required computational power and ROM in each frame can be determined clearly according to Table 1 (Wang et al*.*, 2000; Wang et al., 2003). As shown in the table, the total required computational power is quite high due to the redundant operations and memory that stores the required constants. Accordingly, some modifications must be made

The weighted energy spectrum in the Mel-window, 1( ) *E j k* , can be obtained by subtracting the weighted energy spectrum ( ) *E j k* from energy spectrum *X j* ( ) . All of the multiplications in (4) can be replaced by subtraction operations. Therefore, the memory required to store the

> 1 ( ) ( ) 2( ) *k d LD E j Xj L D LD*

operations and enables related items to be combined in a new formula, given below.

Additionally, applying the symmetric property of the cosine function to (3) flattens all of the

*C n log S k log S M k cos n k M*

Therefore, the computational complexity of *C*[*n*] operations can be re-estimated, and the

{ ( 1) 1 } [ 0.5 ] *n*

(5)

0

*j*

/2 1

*F*

*k*

*W j*

1

log( [ ])cos[ ( 0.5) ]

*M*

(3)

*k*

2 (0.5 0.5 ) 1

*F nm*

*F*

*n*

(1)

(2)

(4)

*F*

( )

/2 1

*F*

0

*j*

1

*Cn Sk nk*

**2.1.2 Improved algorithm for calculating mel-frequency cepstral coefficients**

*M*

*k*

weight constants for (4) becomes redundant and can be eliminated.

0

**Step 1.** Short-time fast Fourier transform (FFT)

**Step 2.** Find the energy spectrum, 2 *Xm Ym* ( ) .

**Step 3.** Calculate the energy in each channel:

cepstral coefficients (MFCCs),

M/2 1

0

result is given in Table 2 (Wang et al., 2000; Wang et al., 2003).

*k*

to reduce the computational load.

Table 1. Number of operations and the required memory estimated using the original MFCC algorithm.

Table 2. Improvement of *C*[*n*] calculation by rescheduling the original MFCC algorithm and the total improvement provided by the proposed method

The modified procedure in the MFCC algorithm is based mainly on the improved *S k*[ ] calculation, as discussed below and shown in Fig. 1 (Wang et al., 2000). Every Mel-window is divided into two blocks with equal bandwidth on the Mel-scale. Because the Melwindows overlap each other, every block except for the first and last belongs to two Melwindows.

Fig. 1. Modified procedure for calculating *S*[*k*].

Design and Applications of Embedded Systems for Speech Processing 177

The use of improved partitioned look-up tables is another commonly used method to perform such elementary functions as logarithm, square root, and trigonometric functions,

(a)

(b) Fig. 2. Circuit design of an embedded system. (a) Architecture of the proposed MFCC chip.

(b) Architecture of the look-up table (Wang et al., 2003).

for example. Figure *2*(a) shows the proposed four-stage pipeline architecture.

**2.2.2 Circuit design**

#### **2.2 Hardware implementation**

#### **2.2.1 Fixed-point arithmetic design**

The word recognition system is based on the hidden Markov model (HMM). To achieve area-efficiency, MFCC chips are designed using xed-point arithmetic. The procedure for implementing the xed-point program is as follows.

The format of xed-point variables is determined based on the dynamic range of the input variables in the first module. Once this module has been analyzed, the output is fed into the next module and analysis continued until all modules fit the xed-point data format*,* as presented in Table 3 (Wang et al., 2003).

Table 3. Analysis of dynamic range and determined xed-point data format.

**Step 3.** Error measurement for each module

This step evaluates the quantization error by comparing their outputs with the output of the corresponding oating-point routines as shown in Table 4 (Wang et al., 2003).

Table 4. Average error with the determined xed-point data format.

#### **Step 4.** Performance measurement

The impact of the recognition rate of the xed-point MFCC algorithm is evaluated at this stage, as shown in Table 5 (Wang et al., 2003).

Table 5. Comparison of recognition rates achieved using oating-point and xed-point structure.

#### **2.2.2 Circuit design**

176 Embedded Systems – High Performance Systems, Applications and Projects

The word recognition system is based on the hidden Markov model (HMM). To achieve area-efficiency, MFCC chips are designed using xed-point arithmetic. The procedure for

**Step 1.** Partition the algorithm into *n* modules; this involves calculations of the energy

The format of xed-point variables is determined based on the dynamic range of the input variables in the first module. Once this module has been analyzed, the output is fed into the next module and analysis continued until all modules fit the xed-point data format*,* as

This step evaluates the quantization error by comparing their outputs with the output of the

Word with vowel phonemes 0.020563623 0.022464977 0.474503142 Word with nasal phonemes 0.028652758 0.035190628 0.481810443 Word with fricative phonemes 0.031959564 0.039397264 0.698307547 Word with stop phonemes 0.052471004 0.057502398 0.454653425 Word with affricate phonemes 0.041492785 0.05067771 0.454107475

The impact of the recognition rate of the xed-point MFCC algorithm is evaluated at this

Floating-point 92.0 90.0 93.0 91.0 93.0 91.8 Fixed-point 91.0 88.0 91.0 88.0 90.0 89.6 Table 5. Comparison of recognition rates achieved using oating-point and xed-point

**User 1(%) User 2(%) User 3(%) User 4(%) User 5(%) Average (%)**

Energy spectrum 3121190.0012 0.000311 0.000311 24.8 Energy in each channel 2172253.6092 0.124351 0.124351 24.8 MFCC 214.006766 -75.082199 1.567230 9.7 Logarithm value 5.336967 -0.905350 0.905350 5.11

Table 3. Analysis of dynamic range and determined xed-point data format.

corresponding oating-point routines as shown in Table 4 (Wang et al., 2003).

Table 4. Average error with the determined xed-point data format.

**Energy spectrum (%)**

**Maximum Minimum Abs. minimum Fixed-point format**

**Energy in each channel (%)**

**MFCC (%)**

**2.2 Hardware implementation**

**2.2.1 Fixed-point arithmetic design**

presented in Table 3 (Wang et al., 2003).

**Step 3.** Error measurement for each module

**Step 4.** Performance measurement

structure.

stage, as shown in Table 5 (Wang et al., 2003).

implementing the xed-point program is as follows.

spectrum, channel energy, and MFCCs.

**Step 2.** Determine the lower bound and upper bound on each module.

The use of improved partitioned look-up tables is another commonly used method to perform such elementary functions as logarithm, square root, and trigonometric functions, for example. Figure *2*(a) shows the proposed four-stage pipeline architecture.

Fig. 2. Circuit design of an embedded system. (a) Architecture of the proposed MFCC chip. (b) Architecture of the look-up table (Wang et al., 2003).

Design and Applications of Embedded Systems for Speech Processing 179

When the pair of microphones receives the sound signals, the system begins to determine whether the input signal needs to be handled. The detection of sound activity comprises

 Threshold Value Detection: Whether the amplitude of the input signal exceeds a threshold is determined by this step. If the amplitude exceeds the threshold, then the

 Zero-Crossing Rate Estimation: In acoustics, a sound wave has positive and negative values of displacement around the zero amplitude. Zero-crossing rates are calculated by counting the crossings of the baseline over time. The presence of an active ZCR signal

End-Point Detection: An end-point beacon is generated when an ongoing input signal

Figure 5 displays a microphone array, where *x*1(*t*) and *x*2(*t*) represent the acoustic signals that are received by microphones 1 and 2 respectively; d denotes the distance between these two microphones; *θ* is the direction between the array and an unknown source, whose signal is represented as *s*(*t*). The source is assumed to be far enough from the microphone array so that the acoustic wave-front that impinges upon the microphone array can be approximated as a plane wave. Let microphone 1 be the reference point; the relationship between the

received signals and the source signal in the time domain is given by the equation,

Fig. 4. Workflow of the sound localization system.

can improve threshold value detection.

**3.1.2 Direction-of-arrival estimation**

Fig. 5. Direction-of-arrival illustration.

falls below the threshold for a preset period.

system begins to store the input signal data in memory.

**3.1.1 Sound activity detection**

three steps, which are as follows.

Based on such architecture, only one processing unit is used and all data are processed in pipeline fashion. Figure 2(b) displays the architecture of the improved partitioned table. The shifter logic is used to find the Q value of the minimum left shift and to output the least significant 16 bits, which are the addresses of the two subtables. Only one two-stage pipelined multiplier and adder, which is shared by both the main data path and the look-up table, is used.

At the verification stage, an FPGA board is utilized to implement the MFCC system prototype. First, synthesizable Verilog-HDL descriptions are coded. Synopsys FPGA Express (www.synopsys.com) generates the corresponding netlist files. The Xilinx Flow-Engine completes generating placement, routing, and bit-stream files. The design is implemented successfully in the XC4062XL FPGA chip.

### **3. Embedded system design for sound localization**

This section introduces a sound localization system, which exploits the average magnitude difference function (AMDF), for finding the directions of environmental signal sources. To verify the accuracy of the algorithm, the entire system is implemented on a single FPGA development board using the Quartus II software tool. Then, the System-on-Chip (SoC) design, based on the FPGA code with the 0.18μm CMOS process, is implemented. The experimental results indicate that the proposed system can achieve higher accuracy with reduced complexity and area of the hardware.

#### **3.1 Methodology**

Figure 3 presents the overall architecture of the sound localization system, including a sound signal amplifier, an analogue-to-digital (AD) converter, a sound activity detector, and an AMDF module. External acoustic signals are received by a pair of microphones and magnified by an amplifier. The AD converter transforms analogue data to digital data. The sound activity detection block consists of threshold value detection, zero-crossing rate (ZCR), and end-point detection modules. Three methods are utilized to distinguish desired segments from silent periods. Finally, the AMDF module (Wang et al., 2008a; Wang et al., 2009) estimates the delay based on the desired signal segments, and converts the delay into angles. A brief workflow of the system is shown in Fig. 4

Fig. 3. Overview of the sound localization system (Wang et al., 2011).

Fig. 4. Workflow of the sound localization system.

#### **3.1.1 Sound activity detection**

178 Embedded Systems – High Performance Systems, Applications and Projects

Based on such architecture, only one processing unit is used and all data are processed in pipeline fashion. Figure 2(b) displays the architecture of the improved partitioned table. The shifter logic is used to find the Q value of the minimum left shift and to output the least significant 16 bits, which are the addresses of the two subtables. Only one two-stage pipelined multiplier and adder, which is shared by both the main data path and the look-up

At the verification stage, an FPGA board is utilized to implement the MFCC system prototype. First, synthesizable Verilog-HDL descriptions are coded. Synopsys FPGA Express (www.synopsys.com) generates the corresponding netlist files. The Xilinx Flow-Engine completes generating placement, routing, and bit-stream files. The design is

This section introduces a sound localization system, which exploits the average magnitude difference function (AMDF), for finding the directions of environmental signal sources. To verify the accuracy of the algorithm, the entire system is implemented on a single FPGA development board using the Quartus II software tool. Then, the System-on-Chip (SoC) design, based on the FPGA code with the 0.18μm CMOS process, is implemented. The experimental results indicate that the proposed system can achieve higher accuracy with

Figure 3 presents the overall architecture of the sound localization system, including a sound signal amplifier, an analogue-to-digital (AD) converter, a sound activity detector, and an AMDF module. External acoustic signals are received by a pair of microphones and magnified by an amplifier. The AD converter transforms analogue data to digital data. The sound activity detection block consists of threshold value detection, zero-crossing rate (ZCR), and end-point detection modules. Three methods are utilized to distinguish desired segments from silent periods. Finally, the AMDF module (Wang et al., 2008a; Wang et al., 2009) estimates the delay based on the desired signal segments, and converts the delay into

implemented successfully in the XC4062XL FPGA chip.

reduced complexity and area of the hardware.

angles. A brief workflow of the system is shown in Fig. 4

Fig. 3. Overview of the sound localization system (Wang et al., 2011).

**3. Embedded system design for sound localization**

table, is used.

**3.1 Methodology**

When the pair of microphones receives the sound signals, the system begins to determine whether the input signal needs to be handled. The detection of sound activity comprises three steps, which are as follows.

#### **3.1.2 Direction-of-arrival estimation**

Figure 5 displays a microphone array, where *x*1(*t*) and *x*2(*t*) represent the acoustic signals that are received by microphones 1 and 2 respectively; d denotes the distance between these two microphones; *θ* is the direction between the array and an unknown source, whose signal is represented as *s*(*t*). The source is assumed to be far enough from the microphone array so that the acoustic wave-front that impinges upon the microphone array can be approximated as a plane wave. Let microphone 1 be the reference point; the relationship between the received signals and the source signal in the time domain is given by the equation,

Fig. 5. Direction-of-arrival illustration.

$$
\begin{bmatrix}
\mathbf{x}\_1(t) \\
\mathbf{x}\_2(t)
\end{bmatrix} = \begin{bmatrix}
s(t) \\
s(t-\tau)
\end{bmatrix} \tag{6}
$$

Design and Applications of Embedded Systems for Speech Processing 181

However, while running the AMDF, the system must perform the correlation analysis,

*i x x* , *N* times. The variable *N* is set herein to 64 for convenience of chip implementation. To reach a favorable trade-off between the chip area and performance, the system used a folding technique to realize the SAA architecture (Fig. 6 and (9)). A comparison with the unfolded SAA architecture revealed that the number of adders had been reduced from 127 to eight, and the number of units that performed the absolute value operation had been decreased from 64 to four. The length of the critical paths was effectively

*xt xt t j*

*xt xt t j*

The sound localization system was tested with sources in different directions, ranging from 15° to 90° in steps of 15°, at five distances (1–5m). The experimental results indicated that the average accuracy was 80%–90%. The estimation error could be maintained in the range ±5°– ±10°. With respect to chip performance, the number of logic gates was reduced to 32,616. Also, the core size and power consumption were minimized (see (Wang et al., 2008a; Wang

The field of speaker recognition has existed for five decades (Furui, 2004). Recently, speaker recognition systems have found many applications in the real world. It is highly flexible and convenient for a wide range of daily-life applications. Various approaches, involving neural networks (Clarkson et al., 2001), Gaussian mixture models (GMMs) (Burget et al., 2007), and

**4. Embedded system design for speaker identification/verification**

when 16 .

when 16 1,2,...,15.

(9)

1 2 1 *N i i*

minimized, enhancing the clock rate.

4

1 4

*j*

*j*

1

Fig. 6. Folded SAA architecture (Wang et al., 2011).

et al., 2009; Wang et al., 2011) for details).

**3.4 Experimental results**

1 2 accumulative

1 2

*j j*

*j j*

where *τ* is the propagation delay from the source to the microphone. As shown in the figure, after the wave-front impinges on microphone 2, the wave-front takes time "*τ"* to reach microphone 1. The distance between the wave-front and microphone 1 is *d*cos*θ* (Chen et al., 2010). Therefore,

$$
\tau = \frac{d\cos\theta}{c} \tag{7}
$$

where *c* is the sound velocity. In real-life applications, noise and reverberation may distort wave shapes, potentially affecting the propagation delay. A feasible way to estimate the accurate time delay involves using the AMDF. The AMDF firstly fixes the signal at microphone 1, and then shifts the signal at microphone 2 to calculate the time delay. When both signals are the most similar, the difference between the waves will be minimized. In other words, the *τ* value is obtained when the correlation between the waveforms of the both microphone signals is maximal. Let *N* be the total number of windows and *i* represent the sliding window index. The AMDF can be expressed as

$$\hat{\sigma} = \operatorname\*{arg\,min}\_{\tau} \frac{1}{N} \sum\_{i=1}^{N} \left| \mathbf{x}\_1^i(t) - \mathbf{x}\_2^i(t-\tau) \right|. \tag{8}$$

#### **3.2 FPGA implementation**

The entire sound localization system (except for the microphone signal amplifiers) was implemented on a single Altera DE2-70 FPGA board. Software design was developed by using the Quartus II software tool. Firstly, on the FPGA board, the AD converter controller used the I2C protocol to control serial input and serial output data. The sound activity detection block was divided into three modules and implemented separately. At the timedelay estimation stage, the AMDF block used conventional basic operational logic elements, such as shift registers, subtraction, absolute value operands, and accumulation, to facilitate the entire design. All blocks implemented the pipeline technique to further accelerate computation. Finally, the output result is displayed on the DE2-70 board using a sevensegment display and LEDs. The system used a total of around 15,600 logic elements (around 188,000 logic gates).

#### **3.3 SoC implementation**

After the FPGA simulation and validation were complete, the sound localization system was ported to the chip level. In this system, after an input signal passed through the sound activity detection module, it was stored in the left and right SRAMs respectively. Next, the subtraction/absolute/accumulation (SAA) (Wang et al., 2011) module performed the major operations in the AMDF, including subtraction, taking absolute values, and accumulation. Hence, the AMDF block was able to estimate the time delay using the SAA module and convert it into a direction by accessing a predefined table in the ROM.

*x t st x t st*

where *τ* is the propagation delay from the source to the microphone. As shown in the figure, after the wave-front impinges on microphone 2, the wave-front takes time "*τ"* to reach microphone 1. The distance between the wave-front and microphone 1 is *d*cos*θ* (Chen et al.,

> *d* cos *c*

where *c* is the sound velocity. In real-life applications, noise and reverberation may distort wave shapes, potentially affecting the propagation delay. A feasible way to estimate the accurate time delay involves using the AMDF. The AMDF firstly fixes the signal at microphone 1, and then shifts the signal at microphone 2 to calculate the time delay. When both signals are the most similar, the difference between the waves will be minimized. In other words, the *τ* value is obtained when the correlation between the waveforms of the both microphone signals is maximal. Let *N* be the total number of windows and *i* represent

1

The entire sound localization system (except for the microphone signal amplifiers) was implemented on a single Altera DE2-70 FPGA board. Software design was developed by using the Quartus II software tool. Firstly, on the FPGA board, the AD converter controller used the I2C protocol to control serial input and serial output data. The sound activity detection block was divided into three modules and implemented separately. At the timedelay estimation stage, the AMDF block used conventional basic operational logic elements, such as shift registers, subtraction, absolute value operands, and accumulation, to facilitate the entire design. All blocks implemented the pipeline technique to further accelerate computation. Finally, the output result is displayed on the DE2-70 board using a sevensegment display and LEDs. The system used a total of around 15,600 logic elements (around

After the FPGA simulation and validation were complete, the sound localization system was ported to the chip level. In this system, after an input signal passed through the sound activity detection module, it was stored in the left and right SRAMs respectively. Next, the subtraction/absolute/accumulation (SAA) (Wang et al., 2011) module performed the major operations in the AMDF, including subtraction, taking absolute values, and accumulation. Hence, the AMDF block was able to estimate the time delay using the SAA module and

convert it into a direction by accessing a predefined table in the ROM.

*i*

*N*

1 2

. (8)

*i i*

*xt xt*

(6)

(7)

the sliding window index. The AMDF can be expressed as

1 ˆ arg min

*N*

1 2

2010). Therefore,

**3.2 FPGA implementation**

188,000 logic gates).

**3.3 SoC implementation**

However, while running the AMDF, the system must perform the correlation analysis, 1 2 1 *N i i i x x* , *N* times. The variable *N* is set herein to 64 for convenience of chip implementation. To reach a favorable trade-off between the chip area and performance, the system used a folding technique to realize the SAA architecture (Fig. 6 and (9)). A comparison with the unfolded SAA architecture revealed that the number of adders had been reduced from 127 to eight, and the number of units that performed the absolute value operation had been decreased from 64 to four. The length of the critical paths was effectively minimized, enhancing the clock rate.

$$\Gamma = \begin{bmatrix} \sum\_{i=1}^{4} \left| x\_1^i(t) - x\_2^i(t) \right| + \Gamma\_{\text{accumulate}} & \text{when } t = 16j. \\ \sum\_{i=1}^{4} \left| x\_1^i(t) - x\_2^i(t) \right| & \text{when } t = 16j + 1, 2, \dots, 15. \end{bmatrix} \tag{9}$$

$$\begin{array}{c} \mathbf{x}\_2^\* - \bigoplus\_{\mathbf{x}\_1^\* \mathbf{1}} \mathbf{\neg} \left| \bigoplus\_{\mathbf{x}\_1^\* \mathbf{1}} \mathbf{\neg} \left( \mathbf{\neg} \mathbf{\neg} \right) \right| \\\\ \mathbf{x}\_3^\* - \bigoplus\_{\mathbf{x}\_1^\* \mathbf{1}} \mathbf{\neg} \left( \mathbf{\neg} \mathbf{\neg} \right) \\\\ \mathbf{x}\_2^\* - \bigoplus\_{\mathbf{x}\_1^\* \mathbf{1}} \mathbf{\neg} \left( \mathbf{\neg} \mathbf{\neg} \right) \end{array} \tag{10}$$

$$\begin{array}{c} \mathbf{x}\_2^\* - \bigoplus\_{\mathbf{x}\_1^\* \mathbf{1}} \mathbf{\neg} \left( \mathbf{\neg} \mathbf{\neg} \right) \\\\ \mathbf{x}\_1^\* - \bigoplus\_{\mathbf{x}\_1^\* \mathbf{1}} \mathbf{\neg} \left( \mathbf{\neg} \mathbf{\neg} \right) \end{array}$$

Fig. 6. Folded SAA architecture (Wang et al., 2011).

#### **3.4 Experimental results**

The sound localization system was tested with sources in different directions, ranging from 15° to 90° in steps of 15°, at five distances (1–5m). The experimental results indicated that the average accuracy was 80%–90%. The estimation error could be maintained in the range ±5°– ±10°. With respect to chip performance, the number of logic gates was reduced to 32,616. Also, the core size and power consumption were minimized (see (Wang et al., 2008a; Wang et al., 2009; Wang et al., 2011) for details).

#### **4. Embedded system design for speaker identification/verification**

The field of speaker recognition has existed for five decades (Furui, 2004). Recently, speaker recognition systems have found many applications in the real world. It is highly flexible and convenient for a wide range of daily-life applications. Various approaches, involving neural networks (Clarkson et al., 2001), Gaussian mixture models (GMMs) (Burget et al., 2007), and

Design and Applications of Embedded Systems for Speech Processing 183

is called a nonlinear SVM when *φ* maps *xi* into a higher-dimensional space. Equation (12) is

1 11

*i ij*

The basic problem of the SMO algorithm is the need to find hyperplane parameters, *w* and *b*, by updating Lagrange parameter *α*. The SMO algorithm searches through the feasible region of the dual problem and maximizes the objective function by choosing two *α* terms and jointly optimizes them (with the values of the other *α* terms fixed) in each iteration. Then, the

> 2 2 1 11 11 2 2 2 22 1 2 1 212

*TT T*

*y yx xa yyx xa yyx xaa*

old 22 1

12 11 22 2 2 1

> , if > , if

*H H*

*xx xx xx*

12 11 22

Let the minimum and maximum feasible values of *α*2 be *L* and *H*, respectively. The

new,clipped new new 2 2 2

2-- 2 *TTT KKK*

*x x*

a clipping function, (16), is used to generate the new constrained new,clipped

2( )( ) Constant ]

*yx yx y x*

111 22 2 2

old old

2 must be checked to determine whether it is in the feasible range. Then,

new 2

, if <

*L L*

new 2

*L H*

2 .

*yE E* ( )

*Di i j j i j*

*i*

*T*

(12)

(13)

(14)

(15)

(16)

0, 0 , 1

*y C i N*

 1

*L y y x x*

*N NN*

the Lagrange function for imperfect separation.

1

*i*

**4.1.2 Sequential Minimal Optimization**

objective function can be written as (13).

Let

2 0 *LD*

unconstrained new

, yielding (14).

*D*

*L*

*N*

arg max 2

*i i i*

Basically, (12) is a QP problem and can be solved using the SMO algorithm.

12 1

Constant

1 [2 2

*T iii*

new old 22 2

2

3

where old *E*1 and old *E*2 are prediction errors, and *η* is given by (15).

*i*

*N*

support vector machines (SVMs) (Cortes et al., 1995), have been adopted for recognizing speakers. Among them, SVM-based speaker recognition has recently attracted much attention.

Based on the idea of the working set, Platt et al. (1998) proposed the use of the sequential minimal optimization (SMO) algorithm, which is a widely used learning algorithm that involves decomposition, to solve the quadratic programming (QP) problem. Basically, the SMO algorithm performs the following two processes repeatedly: 1) selecting a fixed number of Lagrange multipliers, and 2) solving the QP problem of the multipliers until an optimal solution is found. Although the SMO algorithm makes SVM learning feasible when the number of training samples is very large, the number of required computational iterations still results in a heavy computational burden, which makes it unsuitable for use with stand-alone embedded devices.

The operation of the proposed system based on SMO involves a training phase and an identification phase. Since the SMO training algorithm has huge computational load, it is realized as a dedicated, very large-scale integration (VLSI) module, which is a hardware component. The rest processes of the system, such as speech preprocessing, speech feature extraction, and SVM-based voting, are implemented in software. The proposed system has 90% less training time than the embedded C-based ARM processor, and achieves an 89.9% accuracy with the 2010 speaker recognition database of the National Institute of Standards and Technology (NIST). The proposed system was tested and found to be fully functional on a Socle CDK prototype development board (www.socle-tech.com.tw) with an AMBAbased Xilinx FPGA board and an ARM926EJ processor.

#### **4.1 Methodology**

#### **4.1.1 Support vector machine**

Support vector classification (Cortes et al., 1995) is a computationally efficient means of finding hyperplanes in a high-dimensional feature space. Training an SVM is the equivalent to finding a hyperplane with the maximum margin.

The canonical representation of a decision hyperplane is (10),

$$\{y\_i(w^T\phi(\mathbf{x}\_i) + b) \ge 1, \quad i = 1, \ldots, N\} \tag{10}$$

where *w* is the weights of training instances; *b* is a constant; *y*i is the label of *xi*. The optimization problem involves minimizing 2 *w* . In imperfect separation, the optimal hyperplane is obtained by solving the following constrained optimization problem (11),

$$\begin{cases} \min\_{w,b,\xi} \frac{1}{2} w^T w + \mathcal{C} \left( \sum\_{i=1}^N \xi\_i \right) \\ y\_i (w\phi(x\_i) + b) + \xi\_i - 1 \ge 0, \quad \xi\_i \ge 0, \ 1 \le i \le N \end{cases} \tag{11}$$

where *C* is a real-valued cost parameter, and *ξi* is a penalty parameter (slack variable). If ( )*i i x x* , the SVM finds a linear separating hyperplane with the maximal margin. An SVM is called a nonlinear SVM when *φ* maps *xi* into a higher-dimensional space. Equation (12) is the Lagrange function for imperfect separation.

$$\begin{cases} \underset{\alpha}{\arg\max} \ L\_{\mathcal{D}} = \sum\_{i=1}^{N} \alpha\_{i} - \frac{1}{2} \sum\_{i=1}^{N} \sum\_{j=1}^{N} y\_{i} y\_{j} \mathbf{x}\_{i}^{T} \mathbf{x}\_{j} \alpha\_{i} \alpha\_{j} \\\\ \sum\_{i=1}^{N} y\_{i} \alpha\_{i} = 0, \quad 0 \le \alpha\_{i} \le \mathbf{C}, \mathbf{1} \le i \le N \end{cases} \tag{12}$$

Basically, (12) is a QP problem and can be solved using the SMO algorithm.

#### **4.1.2 Sequential Minimal Optimization**

182 Embedded Systems – High Performance Systems, Applications and Projects

support vector machines (SVMs) (Cortes et al., 1995), have been adopted for recognizing speakers. Among them, SVM-based speaker recognition has recently attracted much

Based on the idea of the working set, Platt et al. (1998) proposed the use of the sequential minimal optimization (SMO) algorithm, which is a widely used learning algorithm that involves decomposition, to solve the quadratic programming (QP) problem. Basically, the SMO algorithm performs the following two processes repeatedly: 1) selecting a fixed number of Lagrange multipliers, and 2) solving the QP problem of the multipliers until an optimal solution is found. Although the SMO algorithm makes SVM learning feasible when the number of training samples is very large, the number of required computational iterations still results in a heavy computational burden, which makes it unsuitable for use

The operation of the proposed system based on SMO involves a training phase and an identification phase. Since the SMO training algorithm has huge computational load, it is realized as a dedicated, very large-scale integration (VLSI) module, which is a hardware component. The rest processes of the system, such as speech preprocessing, speech feature extraction, and SVM-based voting, are implemented in software. The proposed system has 90% less training time than the embedded C-based ARM processor, and achieves an 89.9% accuracy with the 2010 speaker recognition database of the National Institute of Standards and Technology (NIST). The proposed system was tested and found to be fully functional on a Socle CDK prototype development board (www.socle-tech.com.tw) with an AMBA-

Support vector classification (Cortes et al., 1995) is a computationally efficient means of finding hyperplanes in a high-dimensional feature space. Training an SVM is the equivalent

> 1, 1,..., *T i i y wxb i N*

where *w* is the weights of training instances; *b* is a constant; *y*i is the label of *xi*. The

hyperplane is obtained by solving the following constrained optimization problem (11),

*N*

*ii i i*

( ( ) ) 1 0, 0, 1

*y wx b i N*

where *C* is a real-valued cost parameter, and *ξi* is a penalty parameter (slack variable). If

*x x* , the SVM finds a linear separating hyperplane with the maximal margin. An SVM

(10)

*w* . In imperfect separation, the optimal

(11)

attention.

with stand-alone embedded devices.

**4.1 Methodology**

( )*i i*

**4.1.1 Support vector machine**

based Xilinx FPGA board and an ARM926EJ processor.

to finding a hyperplane with the maximum margin.

optimization problem involves minimizing 2

1 min 2

The canonical representation of a decision hyperplane is (10),

, , 1

*ww C*

*T i w b i* The basic problem of the SMO algorithm is the need to find hyperplane parameters, *w* and *b*, by updating Lagrange parameter *α*. The SMO algorithm searches through the feasible region of the dual problem and maximizes the objective function by choosing two *α* terms and jointly optimizes them (with the values of the other *α* terms fixed) in each iteration. Then, the objective function can be written as (13).

$$\begin{aligned} L\_D &= a\_1 + a\_2 + \text{Constant}\_1\\ &- \frac{1}{2} [y\_1 y\_1 \mathbf{x}\_1^T \mathbf{x}\_1 a\_1^2 + y\_2 y\_2 \mathbf{x}\_2^T \mathbf{x}\_2 a\_2^2 + 2 y\_1 y\_2 \mathbf{x}\_1^T \mathbf{x}\_2 a\_1 a\_2 \\ &+ 2(\sum\_{i=3}^N a\_i y\_i \mathbf{x}\_i^T)(y\_1 \mathbf{x}\_1 a\_1 + y\_2 \mathbf{x}\_2 a\_2) + \text{Constant}\_2 \end{aligned} \tag{13}$$

Let 2 0 *LD* , yielding (14).

$$\begin{split} \alpha\_2^{\text{new}} &= \alpha\_2^{\text{old}} + \Delta \alpha\_2 \\ &= \alpha\_2^{\text{old}} + \frac{y\_2(E\_2^{\text{old}} - E\_1^{\text{old}})}{\eta} \end{split} \tag{14}$$

where old *E*1 and old *E*2 are prediction errors, and *η* is given by (15).

$$\begin{aligned} \eta &= 2K\_{12} \text{--} K\_{11} \text{--} K\_{22} \\ &= 2\mathbf{x}\_1^T \mathbf{x}\_2 - \mathbf{x}\_1^T \mathbf{x}\_1 - \mathbf{x}\_2^T \mathbf{x}\_2 \\ &= -\left\| \mathbf{x}\_2 - \mathbf{x}\_1 \right\|^2 \end{aligned} \tag{15}$$

Let the minimum and maximum feasible values of *α*2 be *L* and *H*, respectively. The unconstrained new 2 must be checked to determine whether it is in the feasible range. Then, a clipping function, (16), is used to generate the new constrained new,clipped 2 .

$$a\_2^{\text{new,clipped}} = \begin{cases} H, & \text{if } H \ge a\_2^{\text{new}} \\ a\_2^{\text{new}}, & \text{if } L \le a\_2^{\text{new}} \le H \\ & L, \quad \text{if } a\_2^{\text{new}} \le L \end{cases} \tag{16}$$

Eventually, new 1 can be obtained from (17).

$$\begin{aligned} \alpha\_1^{\text{new}} &= \alpha\_1^{\text{old}} + \Lambda a\_1 \\ &= \alpha\_1^{\text{old}} - y\_1 y\_2 \Delta a\_2 \end{aligned} \tag{17}$$

Design and Applications of Embedded Systems for Speech Processing 185

Fig. 8. Proposed hardware/software co-design system for speaker identification

The NIST 2010 speaker recognition evaluation (SRE10) speech corpus (by nine speakers) was adopted to evaluate the proposed hardware/software co-design framework. Six datasets, including nine speakers' files in SRE10, were used to evaluate a speaker identification system for an entrance security application. The training utterance of each speaker was 10s long. The duration of the testing utterances was 2–6s. The order of the

Figure 9 presents a time-cost comparison between the proposed hardware/software system and the embedded C code system (ARM-ported system). The proposed design had a 90% lower time-cost than the embedded C code one in the case of interest. Details of the

**4.3 Experimental results**

linear predictive cepstral coefficients (LPCCs) was 18.

evaluation can be found in (Wang et al., 2008b).

Fig. 9. Performance evaluation based on time cost.

The terms Δ*α*1 and Δ*α*2 are used to update the hyperplane parameters *w* and *b* according to (18) and (19).

$$
\Delta w = \Delta a\_1 y\_1 \mathbf{x}\_1 + \Delta a\_2 y\_2 \mathbf{x}\_2 \tag{18}
$$

$$\begin{aligned} \Delta b &= \frac{1}{2} \left( b\_1 + b\_2 \right) \\ &= \frac{1}{2} \left[ \left( E\_1 + y\_1 \Delta a\_1 \mathbf{x}\_1^T \mathbf{x}\_1 \right) + \left( E\_2 + y\_2 \Delta a\_2 \mathbf{x}\_2^T \mathbf{x}\_2 \right) \right] \end{aligned} \tag{19}$$

#### **4.2 Hardware implementation**

The proposed system can perform both speaker training and identification. Based on the complexity analysis in Fig. 7, the SMO training, which takes 90.89% of the training time, is the computational bottleneck. Hence, the SMO is realized in hardware and the rest processes, including preprocessing, feature extraction and voting analysis, are implemented in software. As shown in Fig. 8, the proposed design comprises four blocks, which are the software-based extraction block (SEB), hardware-based training block (HTB), and softwarebased voting block (SVB). The SEB mainly performs speech preprocessing and speech feature extraction. The HTB executes the SMO algorithm, and the SVB is designed to find the target speaker based on a multiclass SVM. This design can be applied to a fast-trainable system in a stand-alone embedded environment (see (Kuan et al*.,* 2010) for details).

Fig. 7. Complexity analysis for speaker identification.

Fig. 8. Proposed hardware/software co-design system for speaker identification

#### **4.3 Experimental results**

184 Embedded Systems – High Performance Systems, Applications and Projects

1 12 2 *y y*

*E y xx E y xx*

The proposed system can perform both speaker training and identification. Based on the complexity analysis in Fig. 7, the SMO training, which takes 90.89% of the training time, is the computational bottleneck. Hence, the SMO is realized in hardware and the rest processes, including preprocessing, feature extraction and voting analysis, are implemented in software. As shown in Fig. 8, the proposed design comprises four blocks, which are the software-based extraction block (SEB), hardware-based training block (HTB), and softwarebased voting block (SVB). The SEB mainly performs speech preprocessing and speech feature extraction. The HTB executes the SMO algorithm, and the SVB is designed to find the target speaker based on a multiclass SVM. This design can be applied to a fast-trainable

1 1 11 1 2 2 22 2

*T T*

(17)

(19)

111 222 (18)

new old 11 1 old

system in a stand-alone embedded environment (see (Kuan et al*.,* 2010) for details).

The terms Δ*α*1 and Δ*α*2 are used to update the hyperplane parameters *w* and *b* according to

 *w yx yx*

Fig. 7. Complexity analysis for speaker identification.

*b bb*

1 2

1 can be obtained from (17).

Eventually, new

(18) and (19).

**4.2 Hardware implementation**

The NIST 2010 speaker recognition evaluation (SRE10) speech corpus (by nine speakers) was adopted to evaluate the proposed hardware/software co-design framework. Six datasets, including nine speakers' files in SRE10, were used to evaluate a speaker identification system for an entrance security application. The training utterance of each speaker was 10s long. The duration of the testing utterances was 2–6s. The order of the linear predictive cepstral coefficients (LPCCs) was 18.

Figure 9 presents a time-cost comparison between the proposed hardware/software system and the embedded C code system (ARM-ported system). The proposed design had a 90% lower time-cost than the embedded C code one in the case of interest. Details of the evaluation can be found in (Wang et al., 2008b).

Fig. 9. Performance evaluation based on time cost.

Design and Applications of Embedded Systems for Speech Processing 187

, ... , ...

*DQ d d d q q q*

*FF F F F F F F*

[,]

*subj subi*

*subj subi d q* is the local similarity score, which quantifies the similarity between

*subj subi d q* depends on the feature distances between every pair of FPUs, *F*

distance threshold is required (Itoh, 2001; Itoh & Tanaka 2002). Further, this threshold is difficult to define owing to variation in speech. Without a threshold comparison, an attempt is made herein to find a better similarity score function based on only the feature distances.

To test which weighting function performs well, experiments on inverse exponential weighting (IEW(*X*) = 1/*eX*) and inverse distance weighting (IDW(*X*) = 1 *p X* , where *p* is an integer weighting power) techniques for summing local similarity scores were conducted (see the previous work (Lin & Wang, 2007) for details). Based on this experiment, the IDW function outperformed the IEW function; therefore, IDW was used to evaluate the similarity

*subj subi d q* can also be implemented using [ , ] *S S*

1 1

*i j*

where [ , ] *F F*

database sentences. Clearly, [ , ] *F F*

*subi q* and *F*

*subj d* . Although [ , ] *F F*

Fig. 10. Proposed feature-level partial matching algorithm.

score. The global similarity score function Ψ is defined as,

FPUs *F*

and *F*

*l k*

12 12

*sub sub subk sub sub subl*

*FF F*

*subj d* ; ( ) 1 / *M F D k* represents the normalization factor for different

*d q MD*

(20)

*subi q*

*subj subi d q* , however, a

#### **5. Embedded system design for interactive retrieval of spoken documents**

Owing to the increasingly widespread use of personal portable devices, an efficient method for retrieving spoken data with limited resources is required. This section proposes an efficient feature-based sentence-matching algorithm for speaker-dependent personal spoken sentence retrieval. Such a system can efficiently retrieve database sentences only partially matched to query sentence inputs. A whole matching plane-based accumulation (WMPB) scheme is then designed to determine the global similarity score. The proposed algorithms are based on the feature-level comparison and do not require acoustical and language models.

#### **5.1 Methodology**

#### **5.1.1 Sentence matching for retrieving spoken sentences**

Sentence matching is performed to determine the similarity between two sentences. Consider two spoken sentences *A* and *B*: Assume that *A aa a* 1 2 ... *m* is an *m*-word spoken sentence and *B bb b* 1 2 ... *n* is an *n*-word spoken sentence. The similarity between *A* and *B* can then be directly determined from the number of matched words (common words) in these two sentences. For example, if spoken sentences *A* and *B* are "I have a meeting in London tomorrow" and "Where is my meeting tomorrow?" respectively, then "meeting" and "tomorrow" are the matched words. Since only subsets of words in sentences are matched, sentence matching is a form of partial matching. This partial sentence-matching concept can be applied to spoken sentence retrieval.

Because this similarity is defined semantically, using a speech recognition system with acoustical and language models to transcribe spoken sentences into semantic texts is intuitive. To develop a language-independent retrieval system with a small required memory and favorable performance for a medium-sized sentence database, feature-level partial matching algorithms that do not use acoustic and language models are proposed herein.

#### **5.1.2 Spoken sentence retrieval based on feature-level partial matching**

This subsection presents a new partial matching system that is applied to the feature level. Figure 10 shows the proposed feature-level partial matching. First, the features of the spoken sentence are extracted frame by frame. The feature sequence is then segmented into equally sized matching units that are called feature pattern units (FPUs). Given a query sentence *Q* with *l* FPUs and a database sentence *D* with *k* FPUs, the sentences *Q* and *D* are denoted by *Q* = 1 2... *FF F sub sub subl qq q* and *D* = 1 2... *FF F sub sub subk dd d* . These equally sized FPUs of the query and database sentences form a matching plane, shown in Fig. 11. Each matching block in the matching plane is associated with an FPU in the query sentence and the database sentence.

Let be the feature-level similarity function. The global similarity score for *Q* and *D* in the feature-level is calculated

Owing to the increasingly widespread use of personal portable devices, an efficient method for retrieving spoken data with limited resources is required. This section proposes an efficient feature-based sentence-matching algorithm for speaker-dependent personal spoken sentence retrieval. Such a system can efficiently retrieve database sentences only partially matched to query sentence inputs. A whole matching plane-based accumulation (WMPB) scheme is then designed to determine the global similarity score. The proposed algorithms are based on the feature-level comparison and do not require acoustical and language

Sentence matching is performed to determine the similarity between two sentences. Consider two spoken sentences *A* and *B*: Assume that *A aa a* 1 2 ... *m* is an *m*-word spoken sentence and *B bb b* 1 2 ... *n* is an *n*-word spoken sentence. The similarity between *A* and *B* can then be directly determined from the number of matched words (common words) in these two sentences. For example, if spoken sentences *A* and *B* are "I have a meeting in London tomorrow" and "Where is my meeting tomorrow?" respectively, then "meeting" and "tomorrow" are the matched words. Since only subsets of words in sentences are matched, sentence matching is a form of partial matching. This partial sentence-matching

Because this similarity is defined semantically, using a speech recognition system with acoustical and language models to transcribe spoken sentences into semantic texts is intuitive. To develop a language-independent retrieval system with a small required memory and favorable performance for a medium-sized sentence database, feature-level partial matching algorithms that do not use acoustic and language models are proposed

This subsection presents a new partial matching system that is applied to the feature level. Figure 10 shows the proposed feature-level partial matching. First, the features of the spoken sentence are extracted frame by frame. The feature sequence is then segmented into equally sized matching units that are called feature pattern units (FPUs). Given a query sentence *Q* with *l* FPUs and a database sentence *D* with *k* FPUs, the sentences *Q* and *D* are

the query and database sentences form a matching plane, shown in Fig. 11. Each matching block in the matching plane is associated with an FPU in the query sentence and the

Let be the feature-level similarity function. The global similarity score for *Q* and *D* in the

*sub sub subk dd d* . These equally sized FPUs of

**5.1.2 Spoken sentence retrieval based on feature-level partial matching**

*sub sub subl qq q* and *D* = 1 2... *FF F*