11institutetext: Media Informatics Special Interest Group, Universiti
Teknologi Brunei, Brunei 22institutetext: School of Computing and Information
Technology, Universiti Teknologi Brunei 33institutetext: National Institute of
Information and Communications Technology, Tokyo, Japan 44institutetext:
National Electronics and Computer Technology Center (NECTEC), Thailand
55institutetext: CLEALINKTECHNOLOGY Co., Ltd., Kyoto, Japan
55email: somnuk.phonamnuaisuk<EMAIL_ADDRESS>ken.murata;
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
# Visual-based Positioning and Pose Estimation

Somnuk Phon-Amnuaisuk 1122 0000-0003-2130-185X Ken T. Murata 33 La-Or
Kovavisaruch 44 Tiong-Hoo Lim 11 Praphan Pavarangkoon 33 Takamichi Mizuhara 55

###### Abstract

Recent advances in deep learning and computer vision offer an excellent
opportunity to investigate high-level visual analysis tasks such as human
localization and human pose estimation. Although the performance of human
localization and human pose estimation have significantly improved in recent
reports, they are not perfect and erroneous localization and pose estimation
can be expected among video frames. Studies on the integration of these
techniques into a generic pipeline that is robust to noise introduced from
those errors are still lacking. This paper fills the missing study. We
explored and developed two working pipelines that suited the visual-based
positioning and pose estimation tasks. Analyses of the proposed pipelines were
conducted on a badminton game. We showed that the concept of tracking by
detection could work well, and errors in position and pose could be
effectively handled by a linear interpolation technique using information from
nearby frames. The results showed that the _Visual-based Positioning and Pose
Estimation_ could deliver position and pose estimations with good spatial and
temporal resolutions.

###### Keywords:

Object detection Visual-based positioning Pose estimation

##  1 Introduction

Recent progress in object detection and tracking has opened up various
potential areas of applications such as visual-based indoor position analyses,
visitors’ hot spot analyses, and visitors’ activity analyses from pose
information. In this paper, a study of a badminton game analyses is discussed.
The players’ positions on the badminton court and their skeleton
poses111Action analysis based on a skeleton figure, i.e., a stick man figure.
were extracted from a sequence of images. This work proposed a pipeline of
deep learning and computer vision tasks that extracted players’ movements and
actions from an image sequence. In summary, players’ positions were detected
and tracked using Mask R-CNN. The positions read from a 2D camera view
coordinate were transformed into positions in 2D top-view in the world
coordinate. The players’ positions across a frame sequence would reveal the
players’ movement over time; analyzing each player’s pose would then reveal
their actions. These two sources of information could be employed in further
high-level visual analytic tasks.

We coined the following two terms to facilitate the presentation and
discussion in this paper: Outside-in Vision-based Positioning (OV-Positioning)
workflow and Skeleton-based 3D Pose (S3D-Pose) estimation workflow. Outside-in
referred to the fact that an external camera was used to track the agents in
the scene. The term _outside-in_ was borrowed from the augmented reality and
virtual reality (AR/VR) community222AR/VR applications could either track the
position of a head mounted display unit in a 3D world space using external
sensors (outside-in) or using internal sensors (inside-out) equipped on the
head-mounted display device..

This paper is the expanded version of our paper titled _Visual-based
Positioning and Pose Estimation_ [1]. The contributions of this application
paper are from: (i) the Outside-in Vision-based Positioning (OV-Positioning)
workflow, and (ii) the Skeleton-based 3D Pose (S3D-Pose) estimation workflow.
The rest of the materials in this paper are organized into the following
sections; Section 2 describes related works; Section 3 describes the OV-
Positioning, and the S3D-Pose workflows; Section 4 presents the evaluations,
and Section 5 provides the conclusions and future direction.

##  2 Related Works

Fast R-CNN[2] and Faster R-CNN [3] extend Region-based CNN (R-CNN) with an
improved region proposal process that reduces the number of proposed regions
while improving the relevancy of the proposals. Another augmentation over
R-CNN is the introduction of the RoI pooling process. The RoI pooling layer
takes feature maps from the previous CNN layer and transforms output feature
maps to a fixed length vector then feed it to two fully connected layers which
output bounding boxes and classes. A significant improvement of the Faster
R-CNN over the Fast R-CNN is that the Region Proposal Network (RPN) in the
Faster R-CNN is trained as part of the Faster R-CNN.

Mask R-CNN [4] further extends Faster R-CNN by adding a new segmentation masks
output. The mask branch is a small Fully Convolutional Network (FCN) applied
to each RoI, predicting a segmentation mask at a pixel level. RoIAlign is
another important enhancement introduced in Mask R-CNN, and it shows a
substantial performance improvement over the Faster R-CNN, which uses the RoI
pooling process. The RoI pooling process commits to some information loss when
the bounding box size is rounded up to integer values during the RoI pooling
process. This drawback has been mitigated using the RoIAlign process, where
information from four neighbour points is employed to compute the value of
each sampling point without quantization.

###  2.1 Human Activity Recognition

Human Activity Recognition (HAR) is a popular open research problem.
Recognizing human activities from a 2D image is a challenging problem since
further detailed distinctions of objects in the same class must be identified,
i.e., to identify various humans’ actions from the same human class. Since
image data does not explicitly encode activity information, relevant
discriminative features must be extracted from image data first. This makes
vision-based HAR (VHAR) a complex problem in comparison to formulating and
solving HAR using other types of sensors, for example, using accelerometer
data to track body or limbs movements, or using radio frequency tag (RFID) to
track the place where activities are taking place. Although VHAR requires
extra processing efforts to extract relevant discriminative features, it has
many attractive points when its practicality is considered. Since humans
recognize activities mainly from vision, the VHAR deployment often requires a
minimum or no extra environmental preparation.

####  2.1.1 Knowledge Representation

VHAR problems are commonly formulated from the image recognition task
perspective. Representing an image as a feature vector derived from the whole
image is a common option. The original pixels information may also be
preprocessed and transformed to enhance salient information: grayscale, binary
silhouette for displaying the shape of an object, Histogram of Gradient (HOG),
Scale Invariant Feature Transformation (SIFT); or be transformed to other
representation schemes: space-time interest points (STIPs) [5], sparse
representations and, representation extracted from CNN feature maps [6].

A computer representation of an image can truthfully capture the intensity
information of pixels but fails to capture the extra-meaning, which is the
real semantic content of the image. On the contrary, humans can recognize
information which is implicitly and explicitly encoded in an image, i.e.,
objects, their positions and interactions among objects. This fact is well
appreciated, and contemporary image analysis research commonly focuses on low-
level vision analysis. High-level activity analysis, such as VHAR is still
developing where the recognition of activities from the skeleton is one of the
major directions. Recent works in this direction capture activities using
techniques such as concept hierarchical of actions [7], an action graph model
and a scene graph generation model [8].

Activities in a sport game can be described based on the agents’ actions
observed in each image, e.g., when the player jumps and returns the
shuttlecock using a forceful stroke (in a badminton game). It seems intuitive
that recognizing activities from skeleton pose should be more effective than
modelling the recognition model from features extracted from the whole 2D
image.

###  2.2 Human Skeleton Pose Estimation

Motion Capture (MOCAP) is an example of a 3D pose estimation task. Optical
based MOCAP systems determine 3D pose based on multiple cameras detecting
markers attached to actors’ key points, e.g., wrists, elbows, hands, knees,
and head.

In [9], the authors demonstrated a real-time human pose estimation from a
single depth image. The approach enabled accurate recognition of a limited set
of activities which could be expressive enough for some game-based
applications. Pose estimation from RGBD image has stirred up interest in
performing 2D pose estimation using RGB image.

Object detection technique can be employed to detect human body key points
such as head, shoulders, elbows, knees, without markers. With advances of deep
learning and computer vision techniques in the past decade, human pose
estimation techniques have improved significantly. CNNs were employed to
identify 2D key-points on the human body [10, 11, 12, 13] from unconstrained
images. There are always many possible 3D interpretations of a 2D pose due to
an extra dimension. Fortunately, physical constraints of human anatomy and
intraframe information (in case of an image sequence) can be applied to filter
out improbable 3D pose interpretations. In [14, 15], the authors demonstrated
approaches to transform human pose in 2D space to 3D space using deep learning
techniques.

##  3 A Study of a Badminton Game Analyses

Badminton is a popular indoor sport played on a 20 ft$\times$44 ft court. Two
teams play against each other, and each team has one person (or two persons
for a doubles match). Players use a racquet to hit a shuttlecock across the
net. A team scores a point when the other side cannot return the shuttlecock,
and it lands inside the court.

Traditional analysis of a badminton game is carried out manually by a human
expert analyzing the movement of players, the opportunity to score and the
errors made. Statistics of players’ performance can reveal useful insights.
Due to lacking sophisticated video analysis techniques, this information is
manually entered, and it is the bottleneck of the manual approach. Leverage on
deep learning and computer vision techniques, players’ movements and actions
can be automatically extracted from video frames.

A more in-depth insight of the badminton game may be revealed by associating
the position and action of players to their scoring performance. In this
section, we are going to discuss the proposed pipeline of tasks that automate
the position and pose analysis from an image sequence: OV-Positioning and
S3D-Pose.

Figure 1: Top pane: Image frames from a camera are processed by Mask R-CNN,
players’ positions and bounding boxes are determined. The three images on the
right show example output of the detection from Mask R-CNN. Bottom pane:
Details the processing pipeline from 2D positions in camera view to a 2D top-
view in the world space. Detection: Mask R-CNN detects multiple objects which
could be noisy (top row). Tracking-by-detection: various heuristics are
applied to filter out undesired boxes (row 2 & 3). At the final stage,
players’ positions are translated from the camera viewport into top-view in
world space (bottom row).

###  3.1 Outside-in Vision-based Positioning Workflow (OV-Positioning)

The OV-Positioning workflow consisted of the following tasks: object
detection, tracking-by-detection, and translation to the positions in the
world coordinates. Figure 1 shows a graphical summary of the OV-Positioning
workflow.

####  3.1.1 Object Detection

A sequence of image frames, each with 852 $\times$ 472 (width $\times$ height
in pixels), passed through the object detection block. Here, Mask R-CNN with
Resnet-101 backbone333matterport was employed for human detection. ResNet is a
convolutional neural network with skip connections. This seemingly simple
modification improves learning and performance of the deep network
significantly. Mask R-CNN is the current state-of-the-art semantic
segmentation technique. It provides high-quality bounding box output of
objects. This is achieved from the combination of Feature Pyramidal Network
(FPN), Region Proposal Network (RPN) and RoIAlign which effectively exploit
information in the feature maps at different scales and with precise pixel
alignment (than older techniques in Faster R-CNN).

####  3.1.2 Tracking-by-detection

Tracking-by-detection operates on the assumption that object tracking can be
simply derived from object detection, provided that an effective object
detector can detect objects with good spatial accuracy and high resolution in
time. In real life applications, detection errors should be expected and dealt
with. In this badminton game analysis task, the Mask R-CNN returned many
detected objects. Here, the following heuristics were implemented: (i) a
spatial constraint was applied to filter out all detected objects which were
not in the badminton court. This heuristic ensured that only two players were
detected; (ii) a continuity in spatial space was applied across frames. This
heuristic filtered out erroneously detected boxes that resulted in a big jump
in spatial space, and finally (iii) there were only two detected players in
each frame. This heuristic dealt with extra detected boxes (false positive) or
missing detected boxes (false negative). The missing detection or extra
detections could be dealt with based on the information of the boxes from
other frames.

####  3.1.3 Perspective Transformation Between Planes

The position of a human in the scene on the floor (x,y) coordinate (i.e., a
virtual top-view camera) can be transformed from the position obtained from
the camera view. This process is known as homography transformation. In the
first stage, the players’ positions ${\bf p}(x_{c},y_{c})$ were detected on
the camera viewport. We apply the perspective transformation to transformed
the camera viewport plane to its corresponding 2D top-view in the world
coordinate ${\bf p}(x_{w},y_{w})$ using the homography matrix $H$;
$P_{w}=HP_{c}$.

Figure 2: With at least four known coordinates on the badminton court from the
camera view and their corresponding coordinates in the world view, the
homography matrix can be estimated and employed to compute the players’
positions as if they were viewed from a virtual top-view camera.

The homography matrix was estimated based on at least four known coordinates
of the badminton court in the camera view and the corresponding coordinates in
the world view. Figure 2 illustrates the transformation of the camera view to
the world view.

Figure 3: The processing pipeline for 3D pose estimation: Mask R-CNN detects
players location in the scene. The player bounding box is expanded, and key
points in the 2D view are extracted. The 3D pose is estimated based on the 2D
pose.

###  3.2 Skeleton-based 3D Pose (S3D-Pose) estimation workflow

A graphical summary of the S3D-Pose estimation workflow is shown in Figure 3.
The S3D-Pose estimation workflow consisted of the following tasks: object
detection, tracking-by-detection, 2D pose estimation and 3D pose estimation
from a 2D pose. It should be noted that the processing of Mask R-CNN, object
detection, and tracking by detection processes were shared with the OV-
Positioning discussed in the previous section.

The image patch of a player was then extracted with an extra 30-pixels
expansion in all four directions, i.e., up, down, left and right. This was to
ensure that the player’s full body was captured since the bounding box from
the object detection process is commonly designed to fit the object. We
leverage on the implementation from [14], _Lifting from the Deep (LD)_ , for
the 2D and 3D pose estimation. LD fused probabilistic knowledge of 3D human
pose with a multi-stage CNN architecture and used the knowledge of plausible
3D landmark locations to refine the search for better 2D locations.

The 3D poses extracted from 2D poses might not be consistent. It was possible
for 3D poses to go missing or be misinterpreted in some frames. We applies (i)
linear interpolation to fill in the 3D poses from any given two keyframes; and
(ii) inter-frames continuity constraint to fix the abnormal jerk of positions
between consecutive frames.

##  4 Evaluation of Position and Pose Estimation

We evaluated the output from the OV-Positioning workflow by comparing the
projected position obtained from the homography transformation against the
footage obtained from a top-view camera. Fortunately, some duels between the
two players were also presented from a top-view camera. We manually marked the
positions of the two players obtained from the top-view camera as a ground
truth. The comparison between the projected positions and the ground truth
positions are shown in Figures 4 and 5. We discovered that the estimated
positions of the players nearer to the camera had the average error of 13
pixels (equivalent to 32 cm. in our setup), and for the player further from
the camera, the average error was 25 pixels (62 cm). Erroneous results were
correlated with fast movements which could be easily mitigated with a higher
frame sampling rate.

Figure 4: The estimated positions obtained from the OV-Positioining process
are compared with the ground truth positions. From the current setup, the
discrepancy of one pixel is approximately 2.5 cm. The estimated positions of
the player further from the camera are less accurate (the player on the right
part of the frame).

Figure 5: Comparisons between estimated positions and ground truth positions,
$x$ axis is frame ID, and $y$ axis is a discrepancy in pixel units. Top row:
horizontal and vertical positions of the player on the left (near to the
camera). Bottom row: horizontal and vertical positions of the player on the
right (further from the camera). Figure 6: The 3D skeleton lifted from the 2D
skeleton which is interpreted from the 2D image on the rightmost column. Four
skeletons are displayed at 30, 120, 210, and 300 azimuth degrees,
respectively.

Figure 7: Top pane: plots of $(x,y,z)$ coordinates from different key points
from a short video segment. Bottom pane: the mapping of 51 features (from 17
key points) to 2D coordinate using t-SNE method, clusters of activities
emerge.

The results obtained from this study are very encouraging. We found that OV-
Positioning could provide estimated positions with good temporal resolution
and good spatial resolution. The drawbacks from the visual-based approach are
from occlusion or when the object is far away from the camera. These issues
can be approached using a multi-camera system.

Figure 6 shows the skeletons in 3D positions (at 30, 120, 210, and 300 azimuth
degrees, respectively). Even though OV-positioning provided accurate positions
of players such that appropriate human images were fed to the S3D-Pose for
skeleton pose estimations, erroneous 3D poses were common and need to be
cleaned up. We resorted to the _inbetweening_ technique to fill in-between
frames between any two keyframes. Firstly, outlier frames were identified as
erroneous frames using handcrafted constraints. The outlier frame was the
frame with the improbable movement of keypoints from the previous frame to the
next frame. After good keyframes and erroneous keyframes were identified, two
suitable frames were taken as the two keyframes and the keypoints of the
sandwiched outlier frame were then interpolated from the good keyframes. If
there were many sandwiched frames, the interpolation was performed recursively
by filling a frame in the middle point first then repeating the process with
the newly added frame, now a new keyframe.

Figure 7 (top pane) shows plots from the $(x,y,z)$ keypoints. These positions
can be employed to train a classification model for activities recognition. We
applied the _t-distributed stochastic neighbor embedding_ method to map 51
features (from 17 keypoints) in each frame to coordinates in a 2D space and
plot corresponding activities, see Figure 7 (bottom pane). Clusters of various
activities are clearly seen. One of the salient features of S3D-Pose is that
the estimated 3D positions are invariant to different points of view. Hence,
S3D-Pose is ideal for human activity analysis tasks that describe physical
activities.

##  5 Conclusion & Future Directions

Activities in a badminton game can be described based on the players’ actions
in each image, e.g., the player serves the shuttlecock. In this context,
analyzing physical activities from the skeleton poses is more effective than
confabulating an image caption using the similarity of image labels [16].

We present two workflows: OV-Positioning and S3D-Pose for the position and
pose analysis, respectively. The OV-Positioning estimates a position using
visual information. It offers many salient features compared to other
positioning approaches such as traditional indoor positioning systems (IPS)
using WiFi or Bluetooth beacon technology. As discussed above, the OV-
Positioning performs visual analyses to extract the agents’ positions in the
scene. Therefore, an extra WiFi system is not required to estimate their
positions. The estimated positions from OV-Positioning have a better spatial
resolution and with less latency than the estimation obtained from traditional
wireless indoor position systems. S3D-Pose estimates human pose in 3D. This
offers many salient features since human pose in 3D are invariant to
viewpoints. The approach is effective for physical activity identification
task and could enable multi-disciplinary studies in various areas such as
sport science and enhance broadcasting experience.

## Acknowledgments

This publication is the output of the ASEAN IVO http://www.
nict.go.jp/en/asean_ ivo/index.html project titled _Event Analysis:
Applications of computer vision and AI in smart tourism industry_ and
financially supported by NICT (http://www.nict.go.jp). We would also like to
thank anonymous reviewers for their constructive comments and suggestions.

## References

  * [1] S. Phon-Amnuaisuk, K.T. Murata, L. Kovavisaruch, T.H. Lim, P. Pavarangkoon, and T. Mizuhara. Visual-based positioning and pose estimation. In: Proceedings of International Conference on Neural Information Processing. ICONIP (4) 2020:410-417 (2020)
  * [2] Girshick, R.: Fast R-CNN. In: Proceedings of the IEEE Conference on Computer Vision (ICCV), pp. 1140-1148 CoRR arXiv:1504.08083v2 (2015)
  * [3] Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: Towards real-time object detection with region proposal networks. In: Proceedings of the International Conference on Advances in Neural Information Processing Systems (NIPS), pp. 91-99 (2015)
  * [4] He, K., Gkioxari, G., Dollár, P. and Girshick, R.: Mask R-CNN. arXiv:1703.06870v3 (2018)
  * [5] Laptev, I., Lindeberg, T.: Space-time interest points. In: Proceedings of the 9th IEEE International Conference on Computer Vision, Vol. 1, pp. 432–439, Nice, France (2003)
  * [6] Mo, L., Li, F., Zhu, Y., Huang, A.: Human physical activity recognition based on computer vision with deep learning model. In: Proceedings of the IEEE International Conference on Instrumentation and Measurement Technology. pp. 1–6, Taipei, Taiwan, (2016)
  * [7] Kojima, A., Tamura, T., Fukunaga, K.: Natural language description of human activities from video images based on concept hierarchical of actions. International Journal of Computer Vision 50(2):171-184 (2002)
  * [8] Xu, D., Zhu, Y., Choy, C.B. Li, F.F.: Scene graph generation by iterative message passing In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. (CVPR’17), pp. 5410-5419 (2017)
  * [9] Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman A. and Blake, A.: Real-time human pose recognition in parts from single depth images. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’11) pp. 1297-1304, (2011)
  * [10] Toshev, A., Szegedy, C.: Deeppose: Human pose estimation via deep neural networks. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’14) pp. 1297-1304, (2011)
  * [11] Tompson, J., Goroshin, R., Jain, A., LeCun, Y. and Bregler, C.: Efficient Object Localization Using Convolutional Networks. arXiv:1411.4280v3 (2015)
  * [12] Wei, S.E., Ramakrishna, V., Kanade, T., and Sheikh, Y.: Convolutional pose machine. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’16) (2016)
  * [13] Newell, A., Yang, K., Deng, J.: Stacked hourglass networks for human pose estimation. In: Proceedings of the 14th European Conference on Computer Vision (ECCV’16) (2016)
  * [14] Tome, D., Russell, C., Agapito, L.: Lifting from the deep: Convolutional 3D pose estimation from a single image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17) (2017)
  * [15] Kudo, Y., Ogaki, K., Matsui, Y. and Odagiri Y.: Unsupervised adversarial learning of 3D human pose from 2D joint locations. arXiv:1803.08244v1 (2018)
  * [16] Phon-Amnuaisuk, S., Murata, K.T., Pavarangkoon, P., Mizuhara, T., Hadi, S.: Children activity descriptions from visual and textual associations. In: Proceedings of the 13th International Conference on Multi-disciplinary Trends in Artificial Intelligence (MIWAI’19), pp. 121-132, Malaysia (2019)


\end{aligned}
\label{eq:twopoint1}
\end{equation}

In the two-axis model we assume that in the stator voltage equations, the terms $\dot{\lambda}_d$
and $\dot{\lambda}_q$ are negligible compared to the speed voltage terms $\omega\lambda_q$ and
$\omega\lambda_d$, and that $\omega \cong \omega _R$ = 1 p.u. Thus, neglecting $\frac{d\lambda_d}{dt}$ and $\frac{d\lambda_q}{dt}$
in <ref> and substituting $\omega \cong \omega _R$ we get

\begin{equation}
\begin{aligned}
v_d &= -ri_d-\omega _R\lambda_q\\
v_q &= -ri_q+\omega _R\lambda_d\\
\end{aligned}
\label{eq:two3}
\end{equation}

Substituting $\lambda _d$ and $\lambda _q$ from <ref> in <ref> we get

\begin{equation}
\begin{aligned}
v_d &= -ri_d-\omega _R(\lambda' _q+L'_qi_q)\\
&= -ri_d-\omega _R\lambda' _q-\omega _RL'_qi_q\\
v_q &= -ri_q+\omega _R(\lambda' _d+L'_di_d)\\
&= -ri_q+\omega _R\lambda' _d+\omega _RL'_di_d\\
\end{aligned}
\label{eq:two3point1}
\end{equation}

Also substituting $e'_d \triangleq -\omega \lambda '_q=-\omega _R\lambda '_q$, and
$e'_q \triangleq \omega \lambda '_d=\omega _R\lambda '_d$ as given in <ref> into <ref>
we get

\begin{equation}
\begin{aligned}
v_d &= -ri_d-\omega _RL'_qi_q+e'_d\\
v_q &= -ri_q+\omega _RL'_di_d+e'_q\\
\end{aligned}
\label{eq:two4}
\end{equation}

substituting $\omega _R$ = 1 p.u. in <ref> and rearranging we get

\begin{equation}
\begin{aligned}
e'_d &= v_d+ri_d+L'_qi_q\\
&= v_d+ri_d+L'_di_q+(L'_q-L'_d)i_q\\
e'_q &= v_q+ri_q-L'_di_d\\
\end{aligned}
\label{eq:two5}
\end{equation}

The above equation is in the per unit system. Since the term $(L'_q-L'_d)i_q$ is usually small,
it can be neglected in <ref>.
Thus we can write, approximately,

\begin{equation}
e'_d \cong  v_d+ri_d+L'_di_q
\label{eq:two6}
\end{equation}

The voltages $e'_q$ and $e'_d$ are the $q$ and $d$ components of a voltage $e'$ behind the transient reactance.
It is interesting to note that since $e'_d$ and $e'_q$ are $d$ and $q$ axis stator voltages, they represent
$\sqrt {3}$ times the equivalent stator rms voltages. For example, $e'_d=\sqrt{3}E'_d$, and $e'_q=\sqrt{3}E'_q$.
Also, in this model the voltage $e'$, which corresponds to the transient flux linkages in the machine, is not a constant.
Rather it will change due to the changes in the flux linkage of the $d$ and $q$ axis rotor circuits.

We now develop the differential equations for the voltages $e'_d$ and $e'_q$ for the two axis model.
Note, all the equations given below are in per unit. The $d$ axis flux linkage equations for the truth model of
a synchronous generator as given in <ref> are

\begin{equation}
\bbm \lambda_d \\ \lambda_F \\ \lambda_D \ebm =
\bbm L_d & kM_F & kM_D \\ kM_F & L_F & M_R \\ kM_D & M_R & L_D\\ \ebm
\bbm i_d \\ i_F \\ i_D \ebm
\label{eq:dqaxis}
\end{equation}

\begin{equation}
\begin{aligned}
\lambda _d &= L_di_d+kM_Fi_F+kM_Di_D\\
\lambda _F &= kM_Fi_d+L_Fi_F+M_Ri_D\\
\end{aligned}
\label{eq:dqaxis1}
\end{equation}

In the two axis model the damper winding effects are neglected. Thus, the expression for $\lambda_D$ and the terms $kM_Di_D$,
and $M_Ri_D$ in <ref> and <ref>, which are related to the direct axis damper winding are neglected.
Neglecting the above terms, the $d$ axis flux linkage equations for the two-axis model of a synchronous generator are

\begin{equation}
\begin{aligned}
\lambda _d &= L_di_d+kM_Fi_F\\
\lambda _F &= kM_Fi_d+L_Fi_F\\
\end{aligned}
\label{eq:two7}
\end{equation}

The previous two equations can be solved simultaneously to compute $i_F$

\begin{equation}
i_F=\frac{\lambda _F}{L_F}-\frac{kM_F}{L_F}i_d
\label{eq:two7point1}
\end{equation}

Substituting $i_F$ from <ref> in the $\lambda _d$ expression of <ref> we have

\begin{equation}
\begin{aligned}
\lambda _d &= L_di_d+kM_F\bigg{(}\frac{\lambda _F}{L_F}-\frac{kM_F}{L_F}i_d\bigg{)}\\
&= L_di_d-\frac{(kM_F)^2}{L_F}i_d+kM_F\frac{\lambda _F}{L_F}\\
&= \bigg{(}L_d-\frac{(kM_F)^2}{L_F}\bigg{)}i_d+kM_F\frac{\lambda _F}{L_F}\\
\end{aligned}
\label{eq:two7point2}
\end{equation}

From <ref> and <ref> we have

\begin{equation}
L'_d= \bigg{(}L_d-\frac{(kM_F)^2}{L_F} \bigg{)}
\label{eq:two7point3}
\end{equation}

\begin{equation}
\lambda _F\frac{\omega _RkM_F}{L_F}=\sqrt{3}E'_q
\label{eq:two7point4}
\end{equation}

Since all the equations are in per unit we substitute $\omega _R$ = 1 p.u. in the above expression.

\begin{equation}
kM_F\frac{\lambda _F}{L_F}=\sqrt{3}E'_q
\label{eq:two7point5}
\end{equation}

Substituting <ref> and <ref> in <ref> we get

\begin{equation}
\lambda _d=L'_di_d+\sqrt{3}E'_q
\label{eq:two7point6}
\end{equation}

\begin{equation}
\lambda _d-L'_di_d=\sqrt{3}E'_q
\label{eq:two7point6.5}
\end{equation}

Using <ref> and <ref> we can verify that

\begin{equation}
\lambda '_d \triangleq \lambda _d-L'_di_d=\omega _R\lambda '_d=e'_q=\sqrt{3}E'_q
\label{eq:two7point7}
\end{equation}

The $q$ axis flux linkage equations of the synchronous generator as given in <ref> are

\begin{equation}
\bbm \lambda_q \\ \lambda_Q \ebm = \bbm L_q & kM_Q \\ kM_Q & L_Q \ebm
\bbm i_q \\ i_Q \ebm
\label{eq:two7point8}
\end{equation}

\begin{equation}
\begin{aligned}
\lambda_q &= L_qi_q+kM_Qi_Q\\
\lambda_Q &= kM_Qi_q+L_Qi_Q\\
\end{aligned}
\label{eq:two7point9}
\end{equation}

The previous two equations can be solved simultaneously to compute $i_Q$

\begin{equation}
i_Q=\frac{\lambda _Q}{L_Q}-\frac{kM_Q}{L_Q}i_q
\label{eq:two7point10}
\end{equation}

Substituting $i_Q$ from <ref> in the $\lambda_q$ expression of <ref> we have

\begin{equation}
\begin{aligned}
\lambda_q &= L_qi_q+kM_Q\bigg{(}\frac{\lambda _Q}{L_Q}-\frac{kM_Q}{L_Q}i_q\bigg{)}\\
&= L_qi_q-\frac{(kM_Q)^2}{L_Q}i_q+kM_Q\frac{\lambda _Q}{L_Q}\\
&= \bigg{(}L_q-\frac{(kM_Q)^2}{L_Q}\bigg{)}i_q+kM_Q\frac{\lambda _Q}{L_Q}\\
\end{aligned}
\label{eq:two7point11}
\end{equation}

From <ref> we have

\begin{equation}
\label{eq:two7point12}
\end{equation}

Substituting <ref> in <ref> we get

\begin{equation}
\lambda _q= L'_qi_q+kM_Q\frac{\lambda _Q}{L_Q}
\label{eq:two7point13}
\end{equation}

\begin{equation}
\lambda _q-L'_qi_q=kM_Q\frac{\lambda _Q}{L_Q}
\label{eq:two7point14}
\end{equation}

Substituting $\lambda '_q \triangleq \lambda _q-L'_qi_q$ as given in <ref>
and $e'_d \triangleq -\lambda '_q=-\omega _R\lambda '_q$ as given in <ref> in <ref>
we can verify that

\begin{equation}
\lambda '_q \triangleq \lambda _q-L'_qi_q=kM_Q\frac{\lambda _Q}{L_Q}=\omega _R\lambda '_q=-e'_d
\label{eq:two7point15}
\end{equation}

\begin{equation}
e'_d \triangleq \sqrt{3}E'_d= -\frac{kM_Q}{L_Q}\lambda _Q
\label{eq:two7point16}
\end{equation}

We define

\begin{equation}
\begin{aligned}
\sqrt{3}E &=e_q \triangleq kM_Fi_F \\
\sqrt{3}E_d &=e_d \triangleq -kM_Qi_Q \\
\end{aligned}
\label{eq:two15}
\end{equation}

From <ref> we have

\begin{equation}
\lambda _q-L'_qi_q=kM_Q\frac{\lambda _Q}{L_Q}=-e'_d
\label{eq:two15point1}
\end{equation}

Substituting $\lambda_q = L_qi_q+kM_Qi_Q$ from <ref> in <ref> we get

\begin{equation}
\label{eq:two15point2}
\end{equation}

Substituting  $\sqrt{3}E_d =e_d \triangleq -kM_Qi_Q $ from <ref> in <ref> we get

\begin{equation}
\label{eq:two15point3}
\end{equation}

Dividing <ref> by $\sqrt{3}$ we get

\begin{equation}
\label{eq:two15point4}
\end{equation}

Converting all quantities to rms values, i.e. by substituting $\frac{i_q}{\sqrt{3}}=I_q$, and $\frac{e'_d}{\sqrt{3}}=E'_d$
where $I_q$ and $E'_d$ are the rms quantities, <ref> can be written as

\begin{equation}
\label{eq:two15point5}
\end{equation}

\begin{equation}
\label{eq:two15point6}
\end{equation}

Since the damper windings are short circuited, from the $Q$ circuit voltage equation as given in <ref> we have

\begin{equation}
v_Q=r_Qi_Q+\frac{d\lambda _Q}{dt}=0
\label{eq:two15point7}
\end{equation}

Substituting $\lambda_Q = kM_Qi_q+L_Qi_Q$ from <ref> in <ref>

\begin{equation}
\label{eq:two15point8}
\end{equation}

Dividing <ref> by $\sqrt{3}$ we have

\begin{equation}
\label{eq:two15point9}
\end{equation}

Writing $I_Q=\frac{i_Q}{\sqrt{3}}$, where $I_Q$ is the rms value of $i_Q$

\begin{equation}
\label{eq:two15point10}
\end{equation}

Rearranging <ref> we get

\begin{equation}
\label{eq:two15point11}
\end{equation}

From <ref> we have
$\sqrt{3}E_d =e_d \triangleq -kM_Qi_Q$. Expressing all terms as rms quantities we write

\begin{equation}
E_d =-kM_QI_Q
\label{eq:two15point12}
\end{equation}

Differentiating <ref> we get

\begin{equation}
\dot{E}_d =-kM_Q\dot{I}_Q
\label{eq:two15point13}
\end{equation}

Substituting $-\dot{I}_Q$ from <ref> in <ref>

\begin{equation}
\dot{E}_d=kM_Q\frac{r_Q}{L_Q}I_Q+\frac{(kM_Q)^2}{L_Q}\dot{I}_q
\label{eq:two15point14}
\end{equation}

Differentiating <ref> we get

\begin{equation}
\dot{E}'_d=\dot{E}_d-(L_q-L'_q)\dot{I}_q
\label{eq:two16}
\end{equation}

Substituting $\dot{E}_d$ from <ref> in <ref>

\begin{equation}
\begin{aligned}
\dot{E}'_d &= kM_Q\frac{r_Q}{L_Q}I_Q+\frac{(kM_Q)^2}{L_Q}\dot{I}_q-(L_q-L'_q)\dot{I}_q\\
&= \frac{r_Q}{L_Q}kM_QI_Q+\bigg{(}\frac{(kM_Q)^2}{L_Q}-L_q+L'_q\bigg{)}\dot{I}_q\\
\end{aligned}
\label{eq:two17}
\end{equation}

Substituting $\frac{L_Q}{r_Q}=\tau '_{q0}$, $kM_QI_Q=-E_d$ from <ref>, and
$\frac{(kM_Q)^2}{L_Q}-L_q=-L'_q$ from <ref> in <ref> we get

\begin{equation}
\dot{E}'_d = -\frac{1}{\tau '_{q0}}E_d
\label{eq:two18}
\end{equation}

From <ref> we have $E'_d=E_d-(L_q-L'_q)I_q$. Thus, substituting $-E_d=-E'_d-(L_q-L'_q)I_q$
in <ref>

\begin{equation}
\dot{E}'_d = \frac{1}{\tau '_{q0}}[-E'_d-(L_q-L'_q)I_q]
\label{eq:two19}
\end{equation}

The field voltage equation for the truth model of the synchronous generator as given in <ref> is

\begin{equation}
v_F = r_Fi_F + \frac{d\lambda_F}{dt}
\label{eq:eqprime}
\end{equation}

$\lambda _F = kM_Fi_d+L_Fi_F$ from <ref> in <ref>

\begin{equation}
v_F = r_Fi_F + kM_F\dot{i}_d+L_F\dot{i}_F
\label{eq:eqprime1}
\end{equation}

From <ref>, <ref>, and <ref> we have

\begin{equation}
\begin{aligned}
&i_F\omega _RkM_F = \sqrt {3}E\\
&\lambda _F\frac{\omega _RkM_F}{L_F}=\sqrt{3}E'_q\\
&\bigg{(}\frac{v_F}{r_F}\bigg{)}\omega _RkM_F=\sqrt {3}E_{FD}\\
\end{aligned}
\label{eq:eqprime2}
\end{equation}

Since all equations are in per unit, $\omega _R$ = $1$.
$\lambda _F = kM_Fi_d+L_Fi_F$ from <ref> in the $\sqrt{3}E'_q$ expression of <ref>

\begin{equation}
\begin{aligned}
& (kM_Fi_d+L_Fi_F)\frac{\omega _RkM_F}{L_F}=\sqrt{3}E'_q\\
& \frac{\omega _R(kM_F)^2}{L_F}i_d+i_F\omega _RkM_F=\sqrt{3}E'_q\\
\end{aligned}
\label{eq:eqprime3}
\end{equation}

Substituting $i_F\omega _RkM_F = \sqrt {3}E$ from <ref> and $\frac{(kM_F)^2}{L_F}=L_d-L'_d$ from
<ref> in <ref> we get

\begin{equation}
(L_d-L'_d)i_d+\sqrt {3}E=\sqrt{3}E'_q
\label{eq:eqprime4}
\end{equation}

\begin{equation}
\sqrt {3}E+L_di_d=\sqrt{3}E'_q+L'_di_d
\label{eq:eqprime5}
\end{equation}

Multiplying <ref> by $\frac{kM_F}{r_F}$ we get

\begin{equation}
\frac{v_F}{r_F}kM_F = i_FkM_F + \frac{(kM_F)^2}{r_F}\dot{i}_d+\frac{L_F}{r_F}kM_F\dot{i}_F
\label{eq:eqprime6}
\end{equation}

Now substituting $\frac{v_F}{r_F}kM_F=\sqrt {3}E_{FD}$, $i_FkM_F=\sqrt {3}E$, and $kM_F\dot{i}_F=\sqrt{3}\dot{E}$
from <ref> in <ref> we get

\begin{equation}
\sqrt {3}E_{FD}= \sqrt {3}E + \frac{(kM_F)^2}{r_F}\dot{i}_d+\frac{L_F}{r_F}\sqrt{3}\dot{E}
\label{eq:eqprime7}
\end{equation}

On differentiating <ref>

\begin{equation}
\sqrt {3}\dot{E}+L_d\dot{i}_d=\sqrt{3}\dot{E}'_q+L'_d\dot{i}_d
\label{eq:eqprime8}
\end{equation}

Substituting $\sqrt {3}\dot{E}$ from <ref> in <ref> we get

\begin{equation}
\begin{aligned}
\sqrt {3}E_{FD} &= \sqrt {3}E + \frac{(kM_F)^2}{r_F}\dot{i}_d+\frac{L_F}{r_F}(\sqrt{3}\dot{E}'_q-L_d\dot{i}_d+L'_d\dot{i}_d)\\
&= \sqrt {3}E+\frac{L_F}{r_F}\sqrt{3}\dot{E}'_q+
\bigg{(}\frac{(kM_F)^2}{r_F}+\frac{L_F}{r_F}(L'_d-L_d)\bigg{)}\dot{i}_d\\
\end{aligned}
\label{eq:eqprime9}
\end{equation}

Substituting $\frac{(kM_F)^2}{L_F}=L_d-L'_d$ from <ref> in <ref>

\begin{equation}
\begin{aligned}
\sqrt {3}E_{FD} &= \sqrt {3}E+\frac{L_F}{r_F}\sqrt{3}\dot{E}'_q+
\bigg{(}\frac{L_F}{r_F}(L_d-L'_d)+\frac{L_F}{r_F}(L'_d-L_d)\bigg{)}\dot{i}_d\\
&= \sqrt {3}E+\frac{L_F}{r_F}\sqrt{3}\dot{E}'_q+0\\
\end{aligned}
\label{eq:eqprime10}
\end{equation}

Dividing <ref> by $\sqrt {3}$ and rearranging we get

\begin{equation}
\dot{E}'_q = \frac{1}{\tau '_{d0}}(E_{FD}-E)
\label{eq:eqprime11}
\end{equation}

where $\tau '_{d0}=\frac{L_F}{r_F}$, and $\sqrt {3}E+L_di_d=\sqrt{3}E'_q+L'_di_d$ i.e

\begin{equation}
\label{eq:eqprime12}
\end{equation}

here $I_d$ is the rms quantity which is given by $I_d=\frac{i_d}{\sqrt{3}}$.

To complete the description of the system, the electrical torque is given by
$T_{e\phi }=\lambda _di_q-\lambda _qi_d$. Substituting $\lambda _d=\lambda '_d+L'_di_d$, and $\lambda _q=\lambda '_q+L'_qi_q$
from <ref> we compute

\begin{equation}
T_{e\phi }=\lambda '_di_q+L'_di_di_q-\lambda '_qi_d-L'_qi_qi_d
\label{eq:te1}
\end{equation}

Substituting $e'_d=-\omega _R\lambda '_q$, and  $e'_q=\omega _R\lambda '_d$ from <ref>
in <ref>

\begin{equation}
T_{e\phi }=e'_di_d+e'_qi_q+L'_di_di_q-L'_qi_qi_d
\label{eq:te2}
\end{equation}

Converting all terms to rms quantities by dividing <ref> by $3$

\begin{equation}
\frac{T_{e\phi }}{3}=\frac{e'_d}{\sqrt{3}}\frac{i_d}{\sqrt{3}}+\frac{e'_q}{\sqrt{3}}\frac{i_q}{\sqrt{3}}
\label{eq:te3}
\end{equation}

By using $I_q=\frac{i_q}{\sqrt{3}}$, $I_d=\frac{i_d}{\sqrt{3}}$, $E'_d=\frac{e'_d}{\sqrt{3}}$, $E'_q=\frac{e'_q}{\sqrt{3}}$,
and $T_e=\frac{T_{e\phi }}{\sqrt{3}}$ where $I_q$, $I_d$, $E'_d$, $E'_q$ are the rms quantities, $T_{e\phi }$ is the
per unit generator electromagnetic torque defined on a per phase VA base, and $T_e$ is
the per unit generator electromagnetic torque defined on a three-phase $(3\phi )$ VA base.
Thus, the electromagnetic torque equation of the synchronous generator is

\begin{equation}
\label{eq:te4}
\end{equation}

From <ref>, <ref>, and <ref>
we can write the mechanical dynamics for the two-axis model of the synchronous generator in the p.u. system as

\begin{equation}
\begin{aligned}
\tau _j\dot{\omega } &= T_m-D\omega -(E'_dI_d+E'_qI_q-(L'_q-L'_d)I_dI_q)\\
\dot{\delta } &= \omega -1\\
\end{aligned}
\label{eq:two24}
\end{equation}

where $\tau _j$ is the time constant in per unit and is given by $\tau _j=2H\omega _R=2H\omega _B$, where $\omega _B=\omega _R$ = rated or base
angular velocity of the synchronous generator.
Thus, the system equations for the two-axis model of the synchronous generator consist of four differential equations
which are summarized below,

\begin{equation}
\begin{aligned}
\dot{E}'_d &= \frac{1}{\tau '_{q0}}[-E'_d-(L_q-L'_q)I_q]\\
\dot{E}'_q &= \frac{1}{\tau '_{d0}}(E_{FD}-E)\\
\dot{\omega } &= \frac{1}{\tau _j}[T_m-D\omega -(E'_dI_d+E'_qI_q-(L'_q-L'_d)I_dI_q)]\\
\dot{\delta } &= \omega -1\\
\end{aligned}
\label{eq:two25}
\end{equation}

§.§.§ The one-axis or the third-order simplified model

To obtain the third order simplified model which is also referred to in literature
as the one-axis model the following assumptions are made:

* Amortisseur or damper winding effects are neglected.

* The $\dot{\lambda}_d$ and $\dot{\lambda}_q$ terms in the stator and load voltage equations are neglected compared to the speed
voltage terms $\omega\lambda_q$ and $\omega\lambda_d$.

* The terms $\omega\lambda$ in the stator and load voltage equations are assumed to be approximately equal to $\omega_R\lambda$.

* The effect of the $Q$ circuit i.e. the differential equation for $E'_d$ or $e'_d$ which is a function of the current $i_Q$ is neglected.

It is similar to the two-axis model presented in the previous section except that the absence of the $Q$
circuit eliminates the differential equation for $E'_d$ or $e'_d$ which is a function of the current $i_Q$.
The voltage behind the transient reactance $e'$ has only the component $e'_q$ changing by the field effects
according to <ref>. The component $e'_d$ is completely determined from the currents and $v_d$.
Thus, the system equations in per unit are

\begin{equation}
\dot{E}'_q = \frac{1}{\tau '_{d0}}(E_{FD}-E)
\label{eq:two26}
\end{equation}

\begin{equation}
\label{eq:two27}
\end{equation}

as given in <ref>.
Since the damper winding effects are neglected substituting
$\dot{\lambda }_d=0$ in <ref>, and using <ref> and
<ref> we obtain the equation for $E'_d$ as

\begin{equation}
\label{eq:two28}
\end{equation}

From <ref> the $\dot{E}'_d$ expression for the two-axis model is

\begin{equation}
\dot{E}'_d = \frac{1}{\tau '_{q0}}[-E'_d-(L_q-L'_q)I_q]
\label{eq:two29}
\end{equation}

However, in the one-axis or the third order model an additional assumption is that the differential equation for
$E'_d$ or $e'_d$ which is a function of the current $i_Q$ is neglected. Thus, substituting
$\dot{E}'_d$ = $0$ in <ref> we get

\begin{equation}
\label{eq:two30}
\end{equation}

<ref> gives the expression for $E'_d$ for the one-axis model, and it is treated as an algebraic constraint.
Thus, the basic difference between the two-axis and the one-axis model is that in the two-axis model we have a differential
equation for $E'_d$ whereas in the one-axis model we have an algebraic expression for $E'_d$.
The electromagnetic torque equation for the two-axis model as given in <ref> is

\begin{equation}
\label{eq:two31}
\end{equation}

Now substituting the algebraic constraint $E'_d=-(L_q-L'_q)I_q$ from <ref> in
<ref> we obtain the electromagnetic torque equation for the one-axis model

\begin{equation}
\begin{aligned}
T_e &= -(L_q-L'_q)I_qI_d+E'_qI_q-(L'_q-L'_d)I_dI_q\\
&= E'_qI_q+(-L_q+L'_q-L'_q+L'_d)I_dI_q\\
&= E'_qI_q-(L_q-L'_d)I_dI_q\\
\end{aligned}
\label{eq:two32}
\end{equation}

The mechanical dynamics for the one-axis model of the synchronous generator in the per unit system are

\begin{equation}
\begin{aligned}
\tau _j\dot{\omega } &= T_m-D\omega -T_e\\
\tau _j\dot{\omega } &= T_m-D\omega -[E'_qI_q-(L_q-L'_d)I_dI_q] \\
\dot{\delta } &= \omega -1\\
\end{aligned}
\label{eq:two33}
\end{equation}

The system equations for the one-axis or the third order simplified model of the synchronous generator
are summarized below,

\begin{equation}
\begin{aligned}
\dot{E}'_q &= \frac{1}{\tau '_{d0}}(E_{FD}-E) \\
\dot{\omega } &= \frac{1}{\tau _j}[T_m-D\omega -(E'_qI_q-(L_q-L'_d)I_dI_q)]\\
\dot{\delta } &= \omega -1\\
\end{aligned}
\label{eq:two34}
\end{equation}