# Home Action Genome: Cooperative Compositional Action Understanding

Nishant Rai1 Haofeng Chen1 Jingwei Ji1 Rishi Desai1 Kazuki Kozuka2 Shun
Ishizaka2
Ehsan Adeli1 Juan Carlos Niebles1
1Stanford University 2Panasonic Corporation
http://www.homeactiongenome.org

###### Abstract

Existing research on action recognition treats activities as monolithic events
occurring in videos. Recently, the benefits of formulating actions as a
combination of atomic-actions have shown promise in improving action
understanding with the emergence of datasets containing such annotations,
allowing us to learn representations capturing this information. However,
there remains a lack of studies that extend action composition and leverage
multiple viewpoints and multiple modalities of data for representation
learning. To promote research in this direction, we introduce Home Action
Genome (HOMAGE): a multi-view action dataset with multiple modalities and
view-points supplemented with hierarchical activity and atomic action labels
together with dense scene composition labels. Leveraging rich multi-modal and
multi-view settings, we propose Cooperative Compositional Action Understanding
(CCAU), a cooperative learning framework for hierarchical action recognition
that is aware of compositional action elements. CCAU shows consistent
performance improvements across all modalities. Furthermore, we demonstrate
the utility of co-learning compositions in few-shot action recognition by
achieving 28.6% mAP with just a single sample.

Figure 1: Given an activity instance (e.g., ‘do laundry’) and corresponding
multiple views, we compute features using modality-specific deep encoders ($f$
modules). Different modalities may capture different semantic information
regarding the action. Cooperatively training all modalities together allows us
to see improved performance. We utilize training using both video-level and
atomic action labels to allow both the videos and atomic actions to benefit
from the compositional interactions between the two. As discussed in the
results, we see significantly improved performance when using the above
components together.

##  1 Introduction

Action understanding in videos is a critical task with various use-cases and
real-world applications, from robotics [1, 2] and human-computer interaction
[3] to healthcare [4, 5] and elderly behavior monitoring [6, 7]. Despite the
recent success of deep learning methods for image classification, complex and
holistic action or event understanding remains an elusive task.

There are several challenges associated with the task of action understanding.
The inherent variability in executing complex activities poses one of the most
critical difficulties in building action understating models. To understand
these challenges, it is essential to understand what actions are composed of.
As opposed to bounding boxes in the object detection task, actions are
composed of various parts spanned in space and time. For instance, the action
of “laundry” involves multiple entities, e.g., humans, objects, and their
relationships, and is composed of a number of atomic actions. Such partonomy
of actions [8, 9, 10] both in space and time defines a hierarchical structure.
Furthermore, to capture the variability in executing complex activities,
understanding each part (e.g., body limbs, objects, or atomic actions) becomes
crucial. Since actions happen in the 3D world, a holistic understanding of the
world requires capturing the subtle movements or parts using multiple
modalities (e.g., RGB and audio) and from multiple viewpoints.

Each of these challenges has previously been separately investigated using
different datasets and advanced methods. For instance, numerous datasets were
put together for generic action recognition and spatio-temporal localization
in YouTube or broadcasting third-person videos, such as Kinetics [11],
Charades [12], ActivityNet [13], UCF101 [14]. Other datasets such as EPIC
Kitchens [15] were used for ego-centric action recognition. Action Genome [10]
focused on using scene information in action recognition, while others [16]
focused on hierarchical action modeling from events to low-level atomic
actions. Several studies target learning from long instructional videos and
release datasets [17, 18, 19, 20] for the same, exploring the partonomy of
actions in long sequences. Others also focused on observing and recognizing
actions from multiple views, such as LEMMA [21] and HumanEva [22]. In
parallel, there have been numerous recent advances in contrastive and
cooperative learning [23, 24] applied to multi-modal and multi-view datasets
as a self-supervised pre-training strategy to improve downstream recognition
results. Despite all these advances, action understanding and generalizability
of such models remains a challenging problem due to complexities brought by
their complicated nature and numerous object interactions. Multi-modal
approaches [25, 26, 27] have shown superior performance in tackling such
issues. However, there is still a need for a benchmark that unifies all these
challenges and tasks. In this paper, we release a dataset along with a novel
method for hierarchical action recognition to tackle these problems.

We introduce a new benchmark for action recognition, Home Action Genome
(HOMAGE), that includes multi-modal synchronized videos from multiple
viewpoints along with hierarchical action and atomic-action labels. Actions in
homes are challenging as we deal with long-term actions, interactions with
objects, and frequent occlusions. Having multiple views and sensors to handle
occlusions and scene graph information to capture object interaction allows us
to tackle these complexities. In addition, synchronous videos provide implicit
alignment that facilitates multi-modal training. Additionally, access to
sensor information enables future research in privacy-aware recognition where
we avoid audio-visual modalities. HOMAGE also provides temporal annotations of
high-level activity and low-level atomic action supplemented with spatio-
temporal scene-graphs. Annotations regarding interaction of objects within
actions and atomic actions within high-level actions enable research in
explainable video understanding, early action prediction, and long-range
action recognition.

For this new benchmark, we introduce a novel method to perform simultaneous
co-training with multiple modalities (RGB, audio, and annotations of scene
composition) and viewpoints that enable the learning of rich video
representations. Training involves a co-training strategy that leverages
information from all views and modalities to build the representation space.
During inference, we set up different experiments and observe improved action
recognition performance even when only a single modality is used, which
suggests training on HOMAGE improves performance with no need for other
modalities during inference. In this paper, we explore audio-visual data (of
interest to the vision community). Future sensor-fusion work can further
exploit other modalities we release (e.g., for privacy-preserving studies).

HOMAGE aims to unify various aspects and challenges of action recognition,
specifically targeting multi-modal and compositional perception for home
actions. Moreover, the presence of a large number of modalities in our dataset
encourages research in areas such as privacy-aware recognition and sensor-
fusion. To summarize, our contributions are as follows:
(1) We introduce a new dataset, Home Action Genome (HOMAGE) with multiple
views and modalities densely annotated with scene graphs and hierarchical
activity labels (overall activity and atomic actions).
(2) We propose a novel learning framework (CCAU) that leverages multiple
modalities and hierarchical action labels and improves the performance of the
baselines trained on each individual modality. We demonstrate the benefits of
our approach with an improvement of +6.4% using only ego-view during
inference.

##  2 Related Work

Action Recognition in Videos. Action recognition has continuously been an
important direction for the computer vision research community. The success of
2D convolutions in image classification allowed frame-level action recognition
to become a viable approach. Subsequently, two-stream networks for action
recognition [25] have led to many competitive approaches, which demonstrates
using multiple modalities such as optical flow helps improve performance
considerably. Their work motivated other approaches that model temporal motion
features together with spatial image features from videos. [28, 29]
demonstrated that replacing 2D convolutions with 3D convolutions leads to
further performance improvements. Recent approaches such as I3D [30] inflate a
2D convolutional network into 3D to benefit from the use of pre-trained
models. 3D-ResNet [31] adds residual connections building a very deep 3D
network leading to improved performance.

Related Datasets. MSR-Action3D [32] provides depth map sequences containing 20
actions of interactions with game consoles. [33, 34, 35, 36] use the Microsoft
Kinect sensor to collect multi-modal action data with RGB and depth map
sequences. NTU RGB+D [35] consists of RGB, depth map, infrared frames with 3D
human joints annotations with 40 human subjects, and 80 distinct camera
viewpoints. However, for action labels, each video in these datasets has a
single video-level label and thus tough to use for action localization
applications.

Other datasets [13, 33, 37, 21, 10] provide annotations for temporally
localized actions. MMAct [37] is a large-scale action recognition benchmark
multimodal data including RGB videos, keypoints, acceleration, gyroscope, and
orientation. It provides an ego-view and 4 third-person views and temporally
localized actions. However, MMAct does not provide bounding box annotations
for spatial localization and relationships between objects. LEMMA [21] is a
recent multi-view and multi-agent human activity recognition dataset,
providing bounding box annotations on third-person views and compositional
action labels annotated with predefined action templates and verbs/nouns.
However, they do not provide bounding boxes of objects the subjects (human)
interact with. Action Genome [10] is built upon the videos from Charades [38],
with the additional annotation of spatio-temporal scene graph labels. However,
it only provides videos from a single camera view. HOMAGE aims to provide 1)
multiple modalities to promote multi-modal video representation learning, 2)
high-level activity labels and temporally localized atomic action labels, and
3) scene graphs that provide spatial localization cues for both the subject
and the object and their relationship.

Multi-Modal Learning. Multiple modalities of videos are rich sources of
information for both supervised [25] and self-supervised learning [26, 27,
39]. [40, 27] introduce a contrastive learning framework to maximize the
mutual information between modalities in a self-supervised manner. The method
achieves state-of-the-art results on unsupervised learning benchmarks while
being modality-agnostic and scalable to any number of modalities. Two stream
networks for action recognition [25] have led to many competitive approaches,
which demonstrate using even derivable modalities such as optical flow helps
improve performance considerably. There have been approaches [41, 39, 27, 26]
utilizing diverse modalities, sometimes derivable from one other, to learn
better representations.

##  3 Home Action Genome (HOMAGE)

Home Action Genome (HOMAGE) is a new benchmark for action recognition that
includes multi-modal synchronized video data from multiple viewpoints (ego-
view, third-person) with both high-level activity and low-level action
definitions. HOMAGE focuses on actions in residential settings due to the
challenges involved i.e. complexity and long duration of actions, object
interactions, and frequent occlusions. HOMAGE provides multiple views and
sensors to tackle these challenges. We describe the design, data collection,
and data annotation process of the HOMAGE dataset in this section.

Figure 2: Multiple Views of Home Action Genome (HOMAGE) Dataset. Each sequence has one ego-view video as well as at least one or more synchronized third person views.  Dataset | Seq | hrs | Modalities | Views | HL | HL Classes | TL | TL Classes | TL Ins | SG
---|---|---|---|---|---|---|---|---|---|---
RGBD-HuDaAct [34] | 1.19K | 46 | 2 | 1 | ✓ | 12 | - | - | - | -
UCF101 [14] | 13K | 27 | 1 | 1 | ✓ | 101 | - | - | - | -
ActivityNet [13] | 28K | 648 | 1 | 1 | ✓ | 200 | - | - | - | -
Kinetics-700 [11] | 650K | 1.79K | 1 | 1 | ✓ | 700 | - | - | - | -
AVA [42] | 430 | 108 | 1 | 1 | - | - | ✓ | 80 | 1.58M | -
PKU-MMD [33] | 1.08K | 50 | 3 | 3 | - | - | ✓ | 51 | 20K | -
EPIC-Kitchens [15] | - | 55 | 1 | 1 | - | - | ✓ | 125 | 39.6K | -
MMAct [37] | 36K | - | 6 | 5 | - | - | ✓ | 37 | 36.8K | -
Action Genome [10] | 10K | 82 | 1 | 1 | - | - | ✓ | 157 | 66.5K | ✓
Breakfast [43] | - | 77 | 1 | 1 | ✓ | 10 | ✓ | 48 | - | -
LEMMA [21] | 324 | 10.1 | 2 | 4 | ✓ |  15111We here refer to the “task classes” in [21] | ✓ | 863 | 11.8K | -
Ours | 1.75K | 25.4 | 12 |  2$\sim$5 | ✓ | 75 | ✓ | 453 | 24.6K | ✓
Table 1: Comparison between related datasets and HOMAGE. (Seq: number of
synchronized sequences, Modalities: sensor modalities not including annotation
data or derived data like optical flow, Views: number of synchronized
viewpoints for a given sample, HL: high-level activity label (often assigned
one per video), TL: temporally localized atomic action label, SG: scene
graph). HOMAGE provides rich multi-modal action data, including dense
annotations such as scene graphs, along with hierarchical action labels.

Activities and Scenarios. Our goal is to build an activity recognition dataset
that depicts behaviors observed in living spaces. To cover daily activities,
we employed the activity taxonomy in the American Time Use Survey (ATUS) [44].
The ATUS taxonomy organizes activities according to two key dimensions: 1)
social interactions and 2) the locations of the activities. The ATUS coding
lexicon contains a large variety of daily human activities organized under 18
top-level categories such as Personal Care, Work-Related, Education, and
Household activities.

Each participant was asked to perform tasks according to the instructions
assigned. To make sure the behaviors are as natural as possible, we did not
specify detailed procedures and time limits within the activities, and let the
individual participants perform the activity freely.

Data Collection. We recorded 27 participants in kitchens, bathrooms, bedrooms,
living rooms, and laundry rooms in two different houses. We used 12 sensor
types: cameras (RGB), infrared (IR), microphone, RGB light, light,
acceleration, gyro, human presence, magnet, air pressure, humidity,
temperature. We refer to the set of data collected from a given activity with
different modalities as one synchronized action sequence. Sensors were
attached to several locations in the room for third-person views and to the
participants’ heads for ego-view. On average, there are more than 3 views per
action sequence. We synchronized the sensor recordings of all views giving us
synced videos which allowed for ease of use without requiring any additional
post-processing.

Ground-truth Annotation. Home Action Genome is a dataset with (1) video-level
activity labels, (2) temporally localized atomic activity labels, and (3)
spatio-temporal scene-graph labels. Figure 3 visualizes our annotation
pipeline. For the atomic actions, we annotated all atomic actions performed
during the activities. Note that while each video can only have a single
activity label, a given frame can be assigned with multiple atomic action
labels when atomic actions overlap with each other. For the action graph, we
annotated the person performing the action and the objects they interact with
on videos from third-person views. We uniformly sampled 3 or 5 to annotate
scene graphs across the range of each atomic action interval (3 for intervals
less than 3 seconds and 5 otherwise). This action-oriented dynamic sampling
provides more labels where more actions occur which is very valuable for
describing complex primitive actions. [45] also shows this sampling scheme
performs remarkably well.

Figure 3: HOMAGE annotation pipeline: For every action, we uniformly sample 3
or 5 frames across the action and annotate the bounding boxes of the person
performing the action along with the objects they interact with. We also
annotate the pairwise relationships between the subject and the objects.

Dataset Statistics. We annotated 75 activities and 453 atomic actions in 1,752
synchronized sequences and 5,700 videos in total. We split the dataset into
1,388 train sequences and two test splits containing 198 and 166 sequences
each. Each sequence has a high-level activity category. We annotated atomic
actions in each of these videos by providing the start and end frames and the
category of the atomic action. There are 20,039 training, 2,062, and 2,468
atomic action sequences in the three splits mentioned above respectively. For
scene graphs, we annotate one third-person view video in each synchronized
sequence by providing bounding boxes of the subject and the object along with
the relationship between them. There are 86 object classes (excluding
“person”), and 29 relationship classes in the dataset. Overall, there are
annotations of 497,534 bounding boxes and 583,481 relationships. .

The duration of atomic actions in HOMAGE is often short in time: there are
about 60% of the atomic actions under 2 seconds and 80% under 5 seconds. For
scene graphs, some of the most common objects are “countertop,” “clothes,” and
“table”; and the most common relationships include “in front of,” “looking
at,” and “holding.” More details on the statistics are available in the
supplement.

Relevance of Modalities. In this paper, we only study the effect of modalities
of interest to the vision community; however, HOMAGE provides rich sensor
information which could be useful for privacy-aware recognition. Modalities
such as angular velocity, acceleration, and geomagnetic sensors can be used to
extract motion information in ego-view, and environmental sensors, e.g.,
temperature and humidity can capture changes in the scene before and after an
activity. Thermal sensors can extract people or heat sources (e.g., extracting
heat sources can be useful for recognition in places such as kitchens), and
human presence and light sensors can determine the presence of people without
using visual cues. Although not explored in detail in this paper, future
sensor-fusion work can exploit these other modalities as well.

##  4 Cooperative Compositional Action Understanding (CCAU)

We discuss the benefits of HOMAGE and propose our approach Cooperative
Compositional Action Understanding (CCAU) allowing us to exploit the rich
annotations present in the dataset for improved action understanding. We
discuss how CCAU employs simultaneous cooperative training with multiple
modalities to improve the model’s understanding of actions and the associated
atomic-actions. We start by discussing a few preliminaries and proceed to
discuss different components of our model. Note that “modalities” refer to
both different camera views, as well as, modes such as images, audio, and
scene graphs.

###  4.1 Preliminaries

A video $V$ is a sequence of $T$ frames with resolution $H\times W$ and $C$
channels, $\\{\mathbf{i}_{1},\mathbf{i}_{2},\ldots,\mathbf{i}_{T}\\}$, where
$\mathbf{i}_{t}\in\mathbb{R}^{H\times W\times C}$. Assume $T=N*K$, where $N$
is the number of blocks and $K$ denotes the number of frames per block. We
partition a video clip $V$ into $N$ disjoint blocks
$V=\\{\mathbf{x}_{1},\mathbf{x}_{2},\ldots,\mathbf{x}_{N}\\}$, where
$\mathbf{x}_{j}\in\mathbb{R}^{K\times H\times W\times C}$ and a deep encoder
$f(\cdot)$ transforms each input block $x_{j}$ into its latent representation
$z_{j}=f(x_{j})$. An aggregation function, $g(\cdot)$ takes a sequence
$\\{z_{1},z_{2},\ldots,z_{j}\\}$ as input and generates a context
representation $c_{j}=g(z_{1},z_{2},\ldots,z_{j})$. In our setup,
$z_{j}\in\mathbb{R}^{H^{\prime}\times W^{\prime}\times D}$ and
$c_{j}\in\mathbb{R}^{D}$. $D$ represents the embedding size and $H^{\prime}$,
$W^{\prime}$ represent down-sampled resolutions as different regions in
$z_{j}$ represent features for different spatial locations. We define
$c=F(V)$, where $F(\cdot)=g(f(\cdot))$. In our experiments,
$H^{\prime}=4,W^{\prime}=4,D=256$. The computed representations are then
utilized in order to perform per-block classification to generate the
necessary predictions, e.g., activity label or atomic-action label. For
multiple modalities, we define $c_{m}=F_{m}(V_{m})$, where $V_{m}$, $c_{m}$
and $F_{m}$ represent the video input, context feature and composite encoder
for modality $m$, respectively.

RGB Videos with Multiple Viewpoints. An interesting aspect of HOMAGE is the
presence of multiple viewpoints, specifically, a single ego-centric viewpoint
and numerous third-person views. For simplicity, we treat these multiple
viewpoints as two separate modalities, i.e., ego-view and third-person view.
Each of these modalities has a dedicated encoder to generate clip-level
features.

Audio. Along with having multiple camera viewpoints, we also have associated
audio clips for each viewpoint. For simplicity, we only use the audio
associated with the ego-centric view. For each audio clip, we generate the
associated log-mel spectrogram [46] and treat it as an image input. Following
numerous other works [47, 40], we utilize a VGG19 backbone to generate a
representation for the passed-in spectrogram.

Scene Graph. A scene graph in a given frame $G$ contains a set of objects
$O=\\{o_{1},o_{2},...\\}$ and a set of relationships
$R=\\{r_{1},r_{2},...\\}$. Each object $o_{j}$ contains an object ID, bounding
box coordinates of the object, and object category. Each relationship $r_{j}$
contains the object IDs for both the subject and the object of the
relationship, as well as the category of the relationship.

###  4.2 Multi-Modal Cooperative Learning

As discussed earlier, we define $c_{m}=F_{m}(V_{m})$, where $V_{m}$, $c_{m}$
and $F_{m}$ represent the input, context feature, and composite encoder for
modality $m$, respectively. We simultaneously train encoders for each modality
while ensuring that the views improve with cooperation. Such a training regime
allows us to observe improved performance during inference even when using a
single modality.

Intuitively, we expect different modalities to impart complementary
information to other modalities during training. This can be similar to
existing approaches such as student-teacher frameworks or knowledge
distillation [48, 37]. However, as we demonstrate in the experiments section,
CCAU manages to learn better representations. We argue this is because the
unidirectional formulation of student/teacher does not suit such setups as
different modalities serve as a collective cohort of students as opposed to
one of them being significantly dominant compared to others. CCAU utilizes
contrastive multi-modal losses to promote cooperation between the learners.

Noise Contrastive Estimation (NCE) [49, 50, 51] constructs a binary
classification task where a classifier is fed with real and noisy samples with
the training objective to distinguish them. We utilize a simple task of
performing alignment between different modes $m,m^{\prime}$. The task becomes
choosing the correct in-sync instance amongst multiple noisy instances.
Similar to [27], we use an NCE loss over our feature embeddings $c$ described
in Eq. (1). $c^{m}_{i}$ represents the feature embedding for the $m^{th}$
modality’s $i^{th}$ temporal block. This effectively becomes a cross-entropy
loss distinguishing one positive pair from all the negative pairs present in a
video. In a batch setting with multiple video clips, it is possible to have
more inter-clip negative pairs. The objective function for a single pair of
modalities will hence be:

$\mathcal{L}^{m,m^{\prime}}_{align}=-\sum_{i}\bigg{(}\text{log}\frac{\text{exp}({c}^{m}_{i}\cdot
c^{m^{\prime}}_{i})}{\sum_{j}\text{exp}({c}^{m}_{i}\cdot
c^{m^{\prime}}_{j})}\bigg{)}.$ (1)

To extend this to multiple views, we utilize the same objective for all pairs
and simultaneously optimize:

$\mathcal{L}_{align}=\sum_{m,m^{\prime}}\mathcal{L}^{m,m^{\prime}}_{align}.$

Self-supervised attention [52] has been shown to be useful to auto-learn
associations between different modalities. We model attention by predicting
importance weights over the grid. We predict $H^{\prime}\times W^{\prime}$
values $\alpha_{i,j}$ representing weights of each feature corresponding to
spatial location $(i,j)$. Given feature $c$ of shape $D\times H^{\prime}\times
W^{\prime}$, we extract $c_{agg}$ from it as given in Eq. (2). Where $\tau$
refers to the temperature. Further details are provided in the appendix.

$c_{agg}=\sum_{i,j}p_{i,j}\cdot
c_{i,j}\;\;\text{,}\;\;p_{i,j}=\frac{\text{exp}(\alpha_{i,j}\mathbin{/}\tau)}{\sum_{a,b}\text{exp}(\alpha_{a,b}\mathbin{/}\tau)}$
(2)

###  4.3 Compositional Action Recognition

In addition to the multi-modal nature of HOMAGE, another one of its
differentiating factors is having fine-grained atomic-action labels along with
video-level action labels. The compositional nature of atomic-actions is
useful in determining both the overall activity as well as learning
relationships between atomic-actions and high-level actions.

We leverage the compositionality of atomic-actions and activities in CCAU by
simultaneously utilizing both activity and atomic action level labels in our
learning task. The intuition being our model will be able to learn the
composition and relationships between atomic-actions and activities improving
its understanding. We utilize the contextual features $c$ in order to predict
class labels for both video and atomic-action classes. The video action
prediction task is a standard one-hot classification task, while we formulate
the atomic-action prediction task as multi-target classification. We represent
their corresponding losses as $\mathcal{L}_{video}=\mathcal{L}_{v}$ and
$\mathcal{L}_{atomic}=\mathcal{L}_{a}$. The overall compositional loss is
represented by $\mathcal{L}_{composition}=\mathcal{L}_{c}$.

We explore two variants to define $\mathcal{L}_{c}$. The first involves
manually chosen hyper-parameters modulating each component, i.e.,
$\mathcal{L}_{c}=\mathcal{L}_{v}+\lambda\mathcal{L}_{a}$. The second
automatically learns the appropriate multi-task weights [53]. The numbers
reported in the paper represent use the first approach with $\lambda=10$. For
details refer to the appendix.

##  5 Experiments

We discussed the rich annotations in Home Action Genome (HOMAGE) that allows
us to explore multiple aspects previously not possible due to the lack of such
datasets. CCAU utilizes cooperative and compositional learning to learn
improved representations for action understanding. Co-training with other
modalities such as audio imparts additional structure and knowledge to
individual modalities, also leading to improved single-view performance. We
design and discuss multiple quantitative experiments to verify the validity of
our claims. We also conduct qualitative experiments to gain deeper insights
into our approach. In this section, we briefly go over our experiment
framework. Additional details are provided in the appendix.

###  5.1 Implementation Details

Following our design discussed earlier to allow inference using individual
modalities, we use separate encoders for each. We use different designs as
mentioned in Section 4.1.

Images. In all of our experiments, we treat ego-view as one modality and all
third-person view videos as another. We resize each input frame to the size of
128x128. We employ a 3D-ResNet similar to [31] as the encoder $f(\cdot)$.
Following [54], we only expand the convolutional kernels present in the last
two residual blocks to be 3D ones and use 3D-ResNet18 for our experiments,
denoted as ResNet18. A weak aggregation function $g(\cdot)$ is used to learn a
strong encoder $f(\cdot)$. Specifically, we use a one-layer Convolutional
Gated Recurrent Unit (ConvGRU) with kernel size (1, 1) as $g(\cdot)$. The
weights are shared amongst all spatial positions in the feature map. This
design allows the aggregation function to propagate features in the temporal
axis.

We use a dropout [55] with $p=0.1$ to compute the hidden state at each time
step. A shallow two-layer perceptron is used as the predictive function
$\phi(\cdot)$. Recall $z_{j}^{\prime}=Pool(z_{j})$ where $z^{\prime}_{j}\in
R_{D}$. We utilize stacked max pool layers as $Pool(\cdot)$. To construct
blocks to pass to the network, we uniformly choose one out of every 3 frames.
Then, they are grouped into 8 blocks containing 5 frames each. Since the
videos are usually 30fps, each block roughly covers 0.5 seconds and 8 blocks
sums to about 4 seconds worth of action. Given the 256D final representations,
we pass this through fully connected layers to compute the final
classification where we use a dropout of $p=0.5$.

Audio. To process audio clips, we convert audio to MP3 format, compute log-mel
spectrograms [46], and pass it through a VGG19-like convolutional
architecture. We sample fixed intervals of the spectrogram image to represent
the action clip. Similar to the image encoder, we have fully connected layers
to perform classification.

###  5.2 Quantitative Results

In this section, we analyze various aspects of our proposed model. To
objectively evaluate model performance, classification accuracy is utilized as
a proxy for learned representation quality. Evaluation is performed on two
different splits of HOMAGE. Although models have access to other modalities
during training, this is not the case during inference. Therefore, evaluation
only involves inference using individual modalities. However, we see an
improvement despite this constraint due to co-training. We also study the
improvement imparted through compositional learning with both high-level
action and atomic-action labels.

####  5.2.1 Comparisons with Baselines

Method | Audio | Ego |  $3rd$ Person
---|---|---|---
Single Modality | 28.5 | 31.3 | 21.8
Cooperative Ours | 33.3 | 37.7 | 24.7
Static KD | 28.5 | 32.3 | 21.8
Cooperative KD | 32.1 | 32.1 | 23.5
Table 2: Video classification accuracy. Cooperative Ours outperforms the
baselines. Cooperative KD performs better than its counterparts, further
validating benefits of cooperative learning.

In this section, we investigate the effectiveness of cooperative multi-modal
learning for action understanding. We study the impact of cooperative learning
and compare the performance to knowledge distillation approaches.

Impact of Cooperation. Our co-operative training approach hinges on the
assumption that multi-modal information helps in improving overall
representation quality. To verify our hypothesis, we study the performance of
CCAU compared along with a few other comparable approaches. (1) Single
Modality Training (SM) \- Training of modalities independently (2) Cooperative
Ours Training (CT) \- Co-Training of all modalities and individual inference.
Table 2 summarizes our results demonstrating a consistent improvement in
performance across modalities.

Comparison with Knowledge Distillation. Given the potential applicability of
student-teacher approaches in this setting, we also study their performance
compared to our approach. We study two variants. (1) Static Knowledge
Distillation (SKD) \- We transfer knowledge from other trained modalities into
the ego-view encoder. (2) Cooperative Knowledge Distillation (CKD) \- To
isolate the effect of cooperation leading to improved performance, we also
propose a cooperative version of knowledge distillation that allows all
modalities to simultaneously improve (details in the appendix). Table 2
summarizes our results demonstrating the performance difference between these
approaches. We notice a performance improvement when utilizing cooperative KD
compared to the static variant. CT outperforms CKD even though both allow
cooperation, due to the incorrect student-teacher hierarchy even with a
symmetric knowledge distillation setup. CT allows cooperation in a softer
manner without an implicit assumption of hierarchy.

####  5.2.2 Impact of Additional Modalities

We saw the benefits of Cooperative Training in the previous section and
established the performance improvements accompanying training with multiple
modalities. In this section, we look at the implications modalities have on
performance by studying the impact of training with multiple modalities. We
consider 1) Training each modality separately; 2) Joint training of multi-
camera views, i.e., Ego and $3rd$ Person RGB video clips, and 3) Joint
training of multi-camera views with ego-centric audio clips.

Activity Classification. Table 3 summarizes the results of our approach
trained with different modalities. Compared with training with single views
individually, co-training with the two video views and video + audio
consistently improves the performance together with more modalities.

Atomic Action Classification. We also investigate the impact of cooperative
training on multi-target classification for atomic actions. Table 4 summarizes
our results. The Mean Average Precision scores for each modality are reported.

Method | Audio | Ego | $3rd$
---|---|---|---
Single Modality | 28.5 | 31.3 | 21.8
Coop - Ego + $3rd$ | - | 35.1 | 23.5
Coop - Ego + $3rd$ \+ Aud | 33.3 | 37.7 | 24.7
Table 3:  Co-training encoders with different modalities on activity classification. We see a distinct performance improvement across modalities as we co-train with increasing number of modes, possibly due to the presence of rich complementary information. Method | Audio | Ego |  $3rd$ Person
---|---|---|---
Single Modality | 7.0 | 20.5 | 11.7
Cooperative | 13.2 | 28.5 | 15.3
Table 4: Effect of co-training encoders with different modalities on atomic action classification. The numbers reported are support weighted mAP scores. Method | Ego |  $3rd$ Person
---|---|---
Cooperative | 32.5 | 19.1
Cooperative with Attention | 34.8 | 20.8
Table 5: Effect of co-training encoders using the proposed attention module. We see a consistent performance improvement across both modalities. The $3rd$ person mode benefits as attention allows potential localization of the region of interest - despite the lack of dense associations between the ego and $3rd$ person view. Method | Acc | mAP
---|---|---
Audio | Ego |  $3rd$ Person | Audio | Ego |  $3rd$ Person
Cooperative - Activity | 28.3 | 31.1 | 17.0 | - | - | -
Cooperative - Atomic Actions | - | - | - | 5.9 | 18.5 | 9.5
Compositional | 23.5 | 32.1 | 16.2 | 16.4 | 26.3 | 12.2
Cooperative Compositional | 29.3 | 34.9 | 19.2 | 21.7 | 29.3 | 13.8
Table 6: Effect of co-training encoders with images and audio on activity classification. We see a distinct performance improvement compared to the Ego, $3rd$ Person Co-Training case; due to the rich complementary information present in audio encoders. Missing numbers denote the model was not trained for the associated subtask. Results are averaged over the two test splits. Method - Ego | Atomic Action - mAP
---|---
1 shot | 5 shot | 10 shot | 20 shot
Single Modality | 22.4 | 35.3 | 38.6 | 40.6
CCAU | 28.6 | 36.9 | 39.4 | 49.4
Table 7: Compositional learning with few shot learning. With compositional
action understanding, CCAU demonstrates much better generalizability than
other baseline, showing the potential of co-learning with compositional labels
in improving action understanding. Results are averaged over the two testing
splits.

####  5.2.3 Cooperative Compositional Learning

We analyze the role of both our proposed soft attention module and CCAU’s
compositional learning framework.

Impact of Co-training with Attention. Table 5 summarizes the results of the
cross-modality co-training experiment with and without attention module. With
attention, the model yields better accuracy on the video modalities compared
with its counterpart. The model can implicitly learn localization and
correspondence between views to form representations with view-invariant
information.

Impact of Compositional Learning. Our compositional learning framework hinges
on the assumption that simultaneously learning both activity labels and atomic
action labels leads to improved performance. To verify this hypothesis, we
compare different variants such as (1) train with activity labels, (2) train
with atomic-action labels, (3) train with both activity and atomic actions
without cooperation and (4) CCAU - cooperatively train with both video and
atomic actions. In Table 6, we see a consistent improvement across both
activity and atomic-action performance.

####  5.2.4 Few-Shot Compositional Action Learning

We have discussed the benefits of our cooperative and compositional approach.
Intuitively, predicting activities should be easier if we have an idea of the
atomic-actions composing the higher-order action. We now showcase the ability
and potential of CCAU to generalize to rare actions.

Setup. In our few-shot action recognition experiments, we split the 75 action
classes into a base set of 60 classes and a novel set of 15 classes. We use
CCAU as our feature extractor. Note that we do not finetune the backbone.
Next, we train each model with only k examples from each novel class, where k
= 1, 5, 10, 20. Finally, we evaluate the trained models on all examples of
novel classes in the validation set.

Results. We report few-shot experiment performance in Table 7. CCAU improves
the single modality baseline on all 1, 5, 10, 20-shot experiments.
Furthermore, CCAU shows a +6.2% 1-shot and +8.8% 20-shot mAP improvement.

###  5.3 Qualitative Results

One of the motivating factors behind CCAU was the benefits of co-training
different encoders together to gain higher-order perspectives provided through
different modalities. We observe the learned structure across modalities
results in the emergence of higher-order semantics without additional
supervision, e.g., sensible class relationships and good feature
representations. Jointly training with modalities gives rise to better
representations and byproducts such as localization of visual regions of
interest.

t-SNE Visualization. We explore t-SNE visualizations of our learned
representations. For clarity, only a few action classes are displayed. We
loosely order the action classes according to their relationships; classes
having similar colors are semantically similar. Fig. 4 summarizes our results.

Figure 4: t-SNE visualization of Ego-View features from CCAU trained with ego,
$3rd$ and audio modalities. The color mapping represents the relationships
between the action classes, e.g., Red: Clothes; Green: Grooming; Blue:
Kitchen. CCAU is able to learn meaningful clusters by utilizing compositional
information.

Multi-Modal Localization. A by-product of learning attention using contrastive
losses is the ability to localize potential points of interest in images
(details in the appendix).

##  6 Conclusion

We introduced Home Action Genome (HOMAGE), a human action recognition
benchmark with multiple modalities and viewpoints with hierarchical activity
and atomic action labels. We also proposed CCAU, a cooperative and
compositional learning method to leverage information across multiple
modalities along with action compositions in HOMAGE for better representation
learning. Due to the nature of cooperative learning, CCAU allows inference on
individual modalities where no privileged information and other modalities are
available. We demonstrated the benefits of learning atomic-actions
compositions leading to significantly improved results in a few-shot learning
setting.

With rich multi-modal data and compositional annotations, HOMAGE facilitates
research in subfields such as multi–modal action recognition and localization,
explainable action understanding, and reasoning with spatio-temporal scene
graphs. We hope HOMAGE promotes research in multi-modal cooperative learning
and action understanding using compositions for richer feature representations
in human action recognition as well as raises interest in generalizable video
understanding.

Acknowledgement. This work has been supported by Panasonic. This article
solely reflects the opinions and conclusions of its authors and not Panasonic
or any entity associated with Panasonic.

## References

  * [1] F. Rezazadegan, S. Shirazi, B. Upcrofit, and M. Milford, “Action recognition: From static datasets to moving robots,” in 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 3185–3191, IEEE, 2017.
  * [2] B. Liu, E. Adeli, Z. Cao, K.-H. Lee, A. Shenoi, A. Gaidon, and J. C. Niebles, “Spatiotemporal relationship reasoning for pedestrian intent prediction,” IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 3485–3492, 2020\.
  * [3] T. Santini, W. Fuhl, and E. Kasneci, “Calibme: Fast and unsupervised eye tracker calibration for gaze-based pervasive human-computer interaction,” in Proceedings of the 2017 chi conference on human factors in computing systems, pp. 2594–2605, 2017.
  * [4] A. Haque, A. Milstein, and L. Fei-Fei, “Illuminating the dark spaces of healthcare with ambient intelligence,” Nature, vol. 585, no. 7824, pp. 193–202, 2020.
  * [5] M. Lu, K. Poston, A. Pfefferbaum, E. V. Sullivan, L. Fei-Fei, K. M. Pohl, J. C. Niebles, and E. Adeli, “Vision-based estimation of mds-updrs gait scores for assessing parkinson’s disease motor severity,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 637–647, Springer, 2020.
  * [6] A. Jalal, S. Kamal, and D. Kim, “A depth video-based human detection and activity recognition using multi-features and embedded hidden markov models for health care monitoring systems.,” International Journal of Interactive Multimedia & Artificial Intelligence, vol. 4, no. 4, 2017.
  * [7] Z. Luo, J.-T. Hsieh, N. Balachandar, S. Yeung, G. Pusiol, J. Luxenberg, G. Li, L.-J. Li, N. L. Downing, A. Milstein, et al., “Computer vision-based descriptive analytics of seniors’ daily activities for long-term health monitoring,” Machine Learning for Healthcare (MLHC), vol. 2, 2018.
  * [8] U. Blanke and B. Schiele, “Remember and transfer what you have learned-recognizing composite activities based on activity spotting,” in International Symposium on Wearable Computers (ISWC) 2010, pp. 1–8, IEEE, 2010.
  * [9] J. M. Zacks and B. Tversky, “Event structure in perception and conception.,” Psychological bulletin, vol. 127, no. 1, p. 3, 2001.
  * [10] J. Ji, R. Krishna, L. Fei-Fei, and J. C. Niebles, “Action genome: Actions as compositions of spatio-temporal scene graphs,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10236–10247, 2020.
  * [11] J. Carreira, E. Noland, C. Hillier, and A. Zisserman, “A short note on the kinetics-700 human action dataset,” arXiv preprint arXiv:1907.06987, 2019\.
  * [12] G. A. Sigurdsson, G. Varol, X. Wang, A. Farhadi, I. Laptev, and A. Gupta, “Hollywood in homes: Crowdsourcing data collection for activity understanding,” in Proceedings of the European Conference on Computer Vision (ECCV), pp. 510–526, Springer, 2016.
  * [13] B. G. Fabian Caba Heilbron, Victor Escorcia and J. C. Niebles, “Activitynet: A large-scale video benchmark for human activity understanding,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 961–970, 2015.
  * [14] K. Soomro, A. R. Zamir, and M. Shah, “Ucf101: A dataset of 101 human actions classes from videos in the wild,” arXiv preprint arXiv:1212.0402, 2012\.
  * [15] D. Damen, H. Doughty, G. Maria Farinella, S. Fidler, A. Furnari, E. Kazakos, D. Moltisanti, J. Munro, T. Perrett, W. Price, et al., “Scaling egocentric vision: The epic-kitchens dataset,” in Proceedings of the European Conference on Computer Vision (ECCV), pp. 720–736, 2018.
  * [16] S. Malla, B. Dariush, and C. Choi, “Titan: Future forecast using action priors,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11186–11196, 2020.
  * [17] C.-Y. Chang, D.-A. Huang, D. Xu, E. Adeli, L. Fei-Fei, and J. C. Niebles, “Procedure planning in instructional videos,” in European Conference on Computer Vision, pp. 334–350, Springer, 2020.
  * [18] A. Miech, D. Zhukov, J.-B. Alayrac, M. Tapaswi, I. Laptev, and J. Sivic, “Howto100m: Learning a text-video embedding by watching hundred million narrated video clips,” in Proceedings of the IEEE international conference on computer vision, pp. 2630–2640, 2019.
  * [19] Y. Tang, D. Ding, Y. Rao, Y. Zheng, D. Zhang, L. Zhao, J. Lu, and J. Zhou, “Coin: A large-scale dataset for comprehensive instructional video analysis,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1207–1216, 2019.
  * [20] L. Zhou, C. Xu, and J. J. Corso, “Towards automatic learning of procedures from web instructional videos,” in AAAI Conference on Artificial Intelligence, pp. 7590–7598, 2018.
  * [21] B. Jia, Y. Chen, S. Huang, Y. Zhu, and S.-C. Zhu, “Lemma: A multi-view dataset for learning multi-agent multi-task activities,” in Proceedings of the European Conference on Computer Vision (ECCV), pp. 767–786, 2020.
  * [22] L. Sigal, A. O. Balan, and M. J. Black, “Humaneva: Synchronized video and motion capture dataset and baseline algorithm for evaluation of articulated human motion.,” International Journal of Computer Vision, vol. 87, no. 1-2, pp. 4–27, 2010.
  * [23] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in International conference on machine learning, pp. 1597–1607, PMLR, 2020.
  * [24] T. Han, W. Xie, and A. Zisserman, “Self-supervised co-training for video representation learning,” Advances in Neural Information Processing Systems, vol. 33, 2020.
  * [25] K. Simonyan and A. Zisserman, “Two-stream convolutional networks for action recognition in videos,” in Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 1, NIPS’14, (Cambridge, MA, USA), p. 568–576, MIT Press, 2014.
  * [26] N. Sayed, B. Brattoli, and B. Ommer, “Cross and learn: Cross-modal self-supervision,” CoRR, vol. abs/1811.03879, 2018.
  * [27] Y. Tian, D. Krishnan, and P. Isola, “Contrastive multiview coding,” CoRR, vol. abs/1906.05849, 2019.
  * [28] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, “Learning spatiotemporal features with 3d convolutional networks,” in Proceedings of the IEEE international conference on computer vision, pp. 4489–4497, 2015\.
  * [29] G. Varol, I. Laptev, and C. Schmid, “Long-term temporal convolutions for action recognition,” IEEE transactions on pattern analysis and machine intelligence, vol. 40, no. 6, pp. 1510–1517, 2017.
  * [30] J. Carreira, A. Zisserman, and Q. Vadis, “Action recognition? a new model and the kinetics dataset,” in IEEE Conference on Computer Vision and Pattern Recognition, pp. 4724–4733, 2018.
  * [31] K. Hara, H. Kataoka, and Y. Satoh, “Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet?,” in Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 6546–6555, 2018.
  * [32] W. Li, Z. Zhang, and Z. Liu, “Action recognition based on a bag of 3d points,” in 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, pp. 9–14, IEEE, 2010.
  * [33] C. Liu, Y. Hu, Y. Li, S. Song, and J. Liu, “Pku-mmd: A large scale benchmark for continuous multi-modal human action understanding,” arXiv preprint arXiv:1703.07475, 2017.
  * [34] B. Ni, G. Wang, and P. Moulin, “Rgbd-hudaact: A color-depth video database for human daily activity recognition,” in 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), pp. 1147–1153, IEEE, 2011.
  * [35] A. Shahroudy, J. Liu, T.-T. Ng, and G. Wang, “Ntu rgb+ d: A large scale dataset for 3d human activity analysis,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1010–1019, 2016.
  * [36] J. Sung, C. Ponce, B. Selman, and A. Saxena, “Human activity detection from rgbd images.,” plan, activity, and intent recognition, vol. 64, 2011.
  * [37] Q. Kong, Z. Wu, Z. Deng, M. Klinkigt, B. Tong, and T. Murakami, “Mmact: A large-scale dataset for cross modal human action understanding,” in Proceedings of the IEEE International Conference on Computer Vision, pp. 8658–8667, 2019.
  * [38] G. A. Sigurdsson, G. Varol, X. Wang, A. Farhadi, I. Laptev, and A. Gupta, “Hollywood in homes: Crowdsourcing data collection for activity understanding,” in European Conference on Computer Vision, pp. 510–526, Springer, 2016.
  * [39] C. Vondrick, H. Pirsiavash, and A. Torralba, “Generating videos with scene dynamics,” CoRR, vol. abs/1609.02612, 2016.
  * [40] B. Korbar, D. Tran, and L. Torresani, “Cooperative learning of audio and video models from self-supervised synchronization,” arXiv preprint arXiv:1807.00230, 2018.
  * [41] Z. Luo, B. Peng, D. Huang, A. Alahi, and L. Fei-Fei, “Unsupervised learning of long-term motion dynamics for videos,” CoRR, vol. abs/1701.01821, 2017\.
  * [42] C. Gu, C. Sun, D. A. Ross, C. Vondrick, C. Pantofaru, Y. Li, S. Vijayanarasimhan, G. Toderici, S. Ricco, R. Sukthankar, et al., “Ava: A video dataset of spatio-temporally localized atomic visual actions,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6047–6056, 2018.
  * [43] H. Kuehne, A. Arslan, and T. Serre, “The language of actions: Recovering the syntax and semantics of goal-directed human activities,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 780–787, 2014.
  * [44] D. S. Hamermesh, H. Frazis, and J. Stewart, “Data watch: The american time use survey,” Journal of Economic Perspectives, vol. 19, no. 1, pp. 221–232, 2005.
  * [45] J. Ji, R. Krishna, L. Fei-Fei, and J. C. Niebles, “Action genome: Actions as compositions of spatio-temporal scene graphs,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10236–10247, 2020.
  * [46] “Librosa mel spectrogram.” https://librosa.org/doc/main/generated/librosa.feature.melspectrogram.html.
  * [47] T. Afouras, A. Owens, J. S. Chung, and A. Zisserman, “Self-supervised learning of audio-visual objects from video,” arXiv preprint arXiv:2008.04237, 2020\.
  * [48] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” arXiv preprint arXiv:1503.02531, 2015.
  * [49] M. Gutmann and A. Hyvärinen, “Noise-contrastive estimation: A new estimation principle for unnormalized statistical models,” in Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 297–304, 2010.
  * [50] A. Mnih and K. Kavukcuoglu, “Learning word embeddings efficiently with noise-contrastive estimation,” in Advances in neural information processing systems, pp. 2265–2273, 2013.
  * [51] A. v. d. Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” arXiv preprint arXiv:1807.03748, 2018.
  * [52] A. Owens and A. A. Efros, “Audio-visual scene analysis with self-supervised multisensory features,” in Proceedings of the European Conference on Computer Vision (ECCV), pp. 631–648, 2018.
  * [53] A. Kendall, Y. Gal, and R. Cipolla, “Multi-task learning using uncertainty to weigh losses for scene geometry and semantics,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7482–7491, 2018\.
  * [54] T. Han, W. Xie, and A. Zisserman, “Video representation learning by dense predictive coding,” in Workshop on Large Scale Holistic Video Understanding, ICCV, 2019.
  * [55] G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov, “Improving neural networks by preventing co-adaptation of feature detectors (2012),” arXiv preprint arXiv:1207.0580, 2012.

## Appendix A - Dataset

#### Sensors and Modalities

We build multi-modal sensor kits for data collection as shown in Figure A.1.
This kit assists the creation of the multi-modal dataset by dramatically
simplifying the data collection process through simple recording and timing
synchronization. The data from all viewpoints are collected by these sensor-
kits. Figure A.2 shows the photo of the multi-modal sensor mounted on the head
of a subject participant.

The audio and video data from the sensor is saved to a video file, and the
sensor data is saved in the same file as additional tracks. By using lossless
codecs like the Free Lossless Audio Codec (FLAC) or WavPack, we can save the
sensor data with high fidelity. Both codecs support multi-channel audio in
8-32 bit integer format at frequencies as low as 1Hz. Sensor data is acquired
over I2C with constant timing adjustments to maintain synchronization with
audio and video.

HOMAGE contains 12 modalities with multiple viewpoints. Specifically, the
infrared data is obtained by the Grid-EYE 8x8 pixel infrared array sensor. The
RGB light data is obtained by a photodiode array sensor that provides an RGB
spectral response with IR blocking filter. The sensor kit also includes an
ambient light sensor that combines a broadband photodiode and an infrared-
responding photodiode on a single CMOS-integrated circuit to provide ambient
light data. The human presence sensor is a 4-channel nondispersive infrared
(NDIR) sensor. The magnetic field data is acquired from a magnetometer in the
sensor kit.

Figure A.1: Multi-modal sensor kit used in data collection. Figure A.2: The
sensor, mounted on the participant’s head.

#### Data Collection

We collect human action data from different viewpoints using our multimodal
sensor kits. We provide additional details in Table A.1. Specifically, we
synchronize the data from different modalities by using the following scheme.
(1) The participants were instructed to start the activity displayed on the
screen after they heard the start tone. (2) The content of the participants
was specified by activity unit (e.g. make bed). We do not specify a detailed
sequence of atomic actions. (3) We sounded the end tone when the participant’s
activity is finished. We synchronized the data of multiple sensor-kits using
the signal of start/end tone.

It is worth noting that in order to measure natural activities in a situation
where we are in control of the collecting location and the objects, we did not
give instructions to the actors as much as possible. We do not provide any
sequence of actions or objects to touch. Furthermore, to match the activity
labels like “make bed”, the activity instructions were presented in text by
display, and the actors did what they could imagine with the activity.

Table A.1: List of sensors in our multi-modal sensor Sensor | Sensor information
---|---
Model no. | rate
Video | OmniVision OV5647 | 30fps
I2S Digital Microphones | SPH0645LM4H | 48KHz
GridEYE Thermal Imager | AMG8833 | 10Hz
Human Presence (PIR) | AK9753AE | 2Hz
Ambient Light Intensity | TSL2591 | 2Hz
Ambient Color | ISL29125 | 10Hz
CO2/Humidity/Pressure/Temp. | BME680 | 5Hz
Magnetometer | MLX90393 | 10 Hz

#### Data Synchronization

When storing video, audio, and sensor data together, each data stream is
stored in a container by multiplexing the streams. We use H264 for the video
stream, and FLAC (Free Lossless Audio Codec) for audio and sensor data.

To synchronize the sensor data, A 60Hz, fixed-length time-division
multiplexing scheduler is used to query the sensors over the inter-integrated
circuit (I2C) bus. The scheduler monitors the drift between expected and
actual query times and adjusts its timing on the fly to achieve sub-
millisecond accuracy on average. Sensor data are timestamped and passed to the
main thread and encoded into its respective track immediately to guarantee
synchronization.

Figure A.3: The flow chart of data collection.

#### Data Statistics

In this section, we include further details about the HOMAGE dataset. For the
spatio-temporal scene graph, Figure A.4 shows the most frequent object classes
and Figure A.5 shows the most frequent object relationships. Figure A.6 shows
the joint distribution of object classes and relationships. Figure A.7 shows
the distribution of the durations of atomic actions.

To encompass activities in the living space, the types of activities in this
dataset were determined by referring to the American Time Use Survey (ATUS),
which is a survey of time at home allowing researchers to look at how much
time people spend doing different activities. As there are several existing
references defining atomic action for daily activities, we borrow definitions
from datasets such as Charades [12], EPIC-KITCHEN [15] and Action Genome [10].

Figure A.4: Distribution of object classes (top 25).  Figure A.5: Distribution
of relationship classes (top 10).  Figure A.6: The co-occurrence statistics
for objects and relationships in Home Action Genome.  Figure A.7: Distribution
of duration of atomic actions.

## Appendix B - Additional Experiments

#### Self-Supervised Pre-Training

Approach Our base backbone remains similar to the one we discuss in the main
paper and the overall approach is inspired by [54]. To summarize, an
aggregation function, $g(\cdot)$ takes a sequence
$\\{z_{1},z_{2},\ldots,z_{j}\\}$ as input and generates a context
representation $c_{j}=g(z_{1},z_{2},\ldots,z_{j})$. In our setup,
$z_{j}\in\mathbb{R}^{H^{\prime}\times W^{\prime}\times D}$ and
$c_{j}\in\mathbb{R}^{D}$. $D$ represents the embedding size and $H^{\prime}$,
$W^{\prime}$ represent down-sampled resolutions as different regions in
$z_{j}$ represent features for different spatial locations. We define
$z^{\prime}_{j}=Pool(z_{j})$ where $z^{\prime}_{j}\in\mathbb{R}^{D}$ and
$c=F(V)$ where $F(\cdot)=g(f(\cdot))$. In our experiments,
$H^{\prime}=4,W^{\prime}=4,D=256$.

To learn effective representations, we create a prediction task involving
predicting $z$ of future blocks similar to [54]. In the ideal scenario, the
task should force our model to capture all the necessary contextual semantics
in $c_{t}$ and all frame-level semantics in $z_{t}$. We define $\phi(\cdot)$
which takes as input $c_{t}$ and predicts the latent state of the future
frames. The formulation is given in Eq. (3). Fig. B.1 provides a compact
visual representation of the learning framework.

$\begin{split}\widetilde{z}_{t+1}=\phi(c_{t}),\\\
\widetilde{z}_{t+1}=\phi(g(z_{1},z_{2},\ldots,z_{t})),\\\
\widetilde{z}_{t+2}=\phi(g(z_{1},z_{2},\ldots,z_{t},\widetilde{z}_{t+1})),\end{split}$
(3)

where $\phi(\cdot)$ takes $c_{t}$ as input and predicts the latent state of
the future frames. We then utilize the predicted $\widetilde{z}_{t+1}$ to
compute $\widetilde{c}_{t+1}$. We can repeat this for as many steps as we
want, in our experiments we restrict ourselves to predict till 3 steps in to
the future.

Note that we use the predicted $\widetilde{z}_{t+1}$ while predicting
$\widetilde{z}_{t+2}$ to force the model to capture long-range semantics. We
can repeat this for a varying number of steps, although the difficulty
increases tremendously as the number of steps increases as seen in [54]. In
our experiments, we predict the next three blocks using the first five blocks.

Figure B.1: A diagram of the learning framework utilized. We look at features
in a sequential manner while simultaneously trying to predict representations
for future states.

##### Results

To study the value of multiple viewpoints of the video data, we perform pre-
training with the above learning framework weights to get a self-supervised
initialization for our experiment. We first train our model in the self-
supervised setting for 500 epochs. We use the pre-trained weights to
initialize the ego-view and third-person view encoders and train with
supervision loss to the same number of epochs as the randomly initialized
baseline. Note that in the supervision phase, each modality is trained
separately and no cross-modality loss is used. Table B.1 shows that
cooperative learning with different modalities results in distinctively
improved performance compared to random initialization as we are able to
utilize structural information naturally present in the examples. We also
observe that the model with self-supervised pre-training converges faster than
the baseline. This demonstrates the additional possibility of utilizing Home
Action Genome to evaluate multi-modal self-supervision approaches.

Method | Ego-View | 3rd Person
---|---|---
SV | 31.8 | 21.8
SS + SV | 33.1 | 24.8
Table B.1: Effect of self-supervised pre-training on atomic action
classification. We see considerable performance improvements when initializing
our model with pre-training using multi-modal self supervision.

### Baseline with Oracle Scene Graphs

We provide a baseline for human action classification using oracle scene
graphs. This experiment gives a rough reference of the upper bound of action
inference using spatio-temporal information.

We represent the ground-truth scene graph input as a matrix $M$ of size
$n_{obj}\times n_{rel}$, with $n_{obj}$ and $n_{rel}$ be the number of object
and relationship categories, respectively, initialized to be filled with 0. We
encode a relationship with object category $s$, and relationship category $r$
by setting $M[s,r]$ to be 1. The input representation is then flattened and
fed into an MLP-based encoder.

Table B.2 shows the performance of activity classification using ground-truth
scene graphs, with the encoding scheme described above. We observe that the
modality of the ground-truth scene graph is very informative compared with the
other modalities, highlighting the potential for scene graph prediction on
human action understanding.

Acc1 | Acc3
---|---
76.0 | 91.7
Table B.2: Classification of activities using ground-truth scene graphs.
Results are averaged over the two test splits.

#### Multi-Task Loss

As discussed in Section 4.3, we utilize two variants for our multi-task
losses. The first is an equally weighed variant where both $\mathcal{L}_{a}$
and $\mathcal{L}_{v}$ have the same weights, while the other is similar to the
one proposed in [53] utilizing task-dependent uncertainty to automatically
weigh losses. The loss is defined as:

$\mathcal{L}_{c}=\mathcal{L}_{v}\mathbin{/}\sigma_{v}^{2}+\mathcal{L}_{a}\mathbin{/}\sigma_{a}^{2}+\log(\sigma_{v}.\sigma_{a})$
(4)

Where $\sigma_{i}$ refers to the task dependent uncertainty (aleatoric
homoscedastic). Although the latter has shown improved results in numerous
settings, we noticed that it led to slower convergence and the performance
improvements were not consistent across modalities. For this reason, all
results reported utilize the simple equally weighted multi-task loss.

#### Learning Attention

As mentioned in the main text, we also explore the usage of an attention
module that allows auto-learning of associations between different modalities
similar to [52] which do it for audio and visual modalities. We setup
attention in a slightly different manner by predicting weights over the grid.
Recall that our features are arranged in a grid of shape $H^{\prime}\times
W^{\prime}$. We predict $H^{\prime}\times W^{\prime}$ values $\alpha_{i,j}$
representing the weight of each feature corresponding to spatial location
$(i,j)$. Given an original context $c$ of shape $D\times H^{\prime}\times
W^{\prime}$, we extract $c_{agg}$ from it as given in Eq. (2). Note that we
generate attention weights for each pair of modalities to capture the
associations between them.

In our experiments, we did not notice any differences between choosing various
values of temperature as it seems the network modulated the learned $\alpha$’s
accordingly. $p$’s are utilized to infer regions of interest, as cells with
higher $p$ correspond to relevant portions of the modalities. Another thing
worth noting is that this attention module is only used in conjunction with
image modalities, as we found attention over an audio spectrogram was not
directly interpretable in the traditional sense.

Figure B.2: Visual results for multi-modal attention between ego-centric and
third person view. We show four instances where the left image refers to the
third person view, while the right shows the predicted attention weights
(White represents higher importance for attention). As we can see, CCAU is
loosely able to predict areas of interest using our proposed self-supervised
losses.

#### Knowledge Distillation

We discuss Knowledge Distillation briefly in the main text as one of the
important baselines in Section 5.3.1. The framework we used is similar to the
famously used one proposed in [48]. Without going into details, the overall
loss is given in Eq. (5).

$\mathcal{L}_{kd}=\alpha\cdot\mathcal{H}(y,\sigma(zs))+\beta\cdot\mathcal{H}(\sigma(zt,\tau),\sigma(zs,\tau))$
(5)

Eq. (5) is an instance of matching logit distributions leading to the
distillation of knowledge from the teacher to the student. Where $H$
represents the cross-entropy loss, $\tau$ represents the temperature. $zs$ and
$zt$ are outputs for the student and teacher, respectively.

For multiple modalities, the loss is just repeated multiple times for each
modality. For our experiments we use $\alpha=1$ and $\beta=0.1$. We choose
$\tau=2.5$ as the models are similar in capacity. We also experiment with two
variants i.e. Static and Cooperative Knowledge Distillation. The difference
being Static KD involves static teachers while the cooperative variants allow
all modalities to serve as both students and teachers.


# Signs behaviour of sums of weighted numbers of compositions

Filip Gawron and Maciej Ulas

###### Abstract.

Let $A$ be a subset of positive integers. For a given positive integer $n$ and
$0\leq i\leq n$ let $c_{A}(i,n)$ denotes the number of $A$-compositions of $n$
with exactly $i$ parts. In this note we investigate the sign behaviour of the
sequence $(S_{A,k}(n))_{n\in\mathbb{N}}$, where
$S_{A,k}(n)=\sum_{i=0}^{n}(-1)^{k}i^{k}c_{A}(i,n)$. We prove that for a broad
class of subsets $A$, the number $(-1)^{n}S_{A,k}(n)$ is non-negative for all
sufficiently large $n$. Moreover, we show that there is
$A\subset\mathbb{N}_{+}$ such that the sign behaviour of $S_{A,k}(n)$ is not
periodic.

###### Key words and phrases:

composition; sum; sign; sums of compositions

Research of the authors was supported by a grant of the National Science
Centre (NCN), Poland, no. UMO-2019/34/E/ST1/00094

##  1\. Introduction

Let $\mathbb{N}$ be the set of non-negative integers and $\mathbb{N}_{+}$
denotes the set of positive integers. Let $A\subset\mathbb{N}_{+}$ be given.
With the set $A$ and a non-negative integer $n$, one can connect two
combinatorial objects: an $A$-partition of $n$ and an $A$-composition of $n$.
By $A$-partition of $n$, we understand a finite non-increasing sequence
$(\lambda_{1},\ldots,\lambda_{k})$, where $\lambda_{i}\in A$ and
$n=\sum_{i=1}^{k}\lambda_{i}$. By $A$-composition of $n$, we understand a
finite sequence $(\lambda_{1},\ldots,\lambda_{k})$, where $\lambda_{i}\in A$
and $n=\sum_{i=1}^{k}\lambda_{i}$. The difference is clear. In
$A$-compositions the order of summands in $n=\sum_{i=1}^{k}\lambda_{i}$ gives
different $A$-compositions. To see an example, let us consider
$A=\\{1,2,3\\}$. We have four $A$-partitions of 4 given by

$(3,1),(2,2),(2,1,1),(1,1,1,1).$

On the other hand, we have seven $A$-compositions of 4 given by

$(3,1),(1,3),(2,2),(2,1,1),(1,2,1),(1,1,2),(1,1,1,1).$

In the sequel, by $p_{A}(n)$ and $c_{A}(n)$ we denote the number of
$A$-partitions and $A$-compositions of $n$, respectively. It is clear that
$c_{A}(n)>p_{A}(n)$ for each $A\subset\mathbb{N}_{+}$, where $\\#A\geq 2$ and
$n>\operatorname{min}(A)$. From the general theory we know that

$P_{A}(x)=\prod_{a\in A}\frac{1}{1-x^{a}}=\sum_{n=0}^{\infty}p_{A}(n)x^{n}$

is an ordinary generating function for the sequence
$(p_{A}(n))_{n\in\mathbb{N}}$. One can also consider a polynomial
$p_{A,n}(t)$, called $n$-th $A$-partition polynomial, which can be seen as a
polynomial analogue of the number $p_{A}(n)$. More precisely, we have

$P_{A}(t,x)=\prod_{a\in
A}\frac{1}{1-tx^{a}}=\sum_{n=0}^{\infty}p_{A,n}(t)x^{n}.$

We write $p_{A,n}(t)=\sum_{i=0}^{n}p_{A}(i,n)t^{n}$ and observe that
$p_{A}(i,n)$ is the number of $A$-partitions of $n$ with exactly $i$ parts. In
particular, $p_{A,n}(1)=\sum_{i=0}^{n}p_{A}(i,n)$ is the number of
$A$-partitions of $n$. In a recent paper [2], we investigated the sign
behaviour of the sequence
$\left(\sum_{i=0}^{n}(-1)^{i}i^{k}p_{A}(i,n)\right)_{n\in\mathbb{N}}$, where
$k\in\mathbb{N}$ is fixed and $n\in\mathbb{N}$. We proved that for a broad
class of sets $A$ (including all sets containing only odd numbers) we have
$(-1)^{n}\sum_{i=0}^{n}(-1)^{i}i^{k}p_{A}(i,n)\geq 0$ for all
$k,n\in\mathbb{N}$.

In this paper we consider related problem and instead of $A$-partitions we
consider $A$-compositions. To formulate our basic problem we need some
notation. Let $A\subset\mathbb{N}_{+}$ and consider the ordinary generating
function $f_{A}(x)$ of the characteristic function of the set $A$, i.e., we
write

$f_{A}(x)=\sum_{a\in A}x^{a}.$

Then

$C_{A}(x)=\frac{1}{1-f_{A}(x)}=\sum_{n=0}^{\infty}c_{A}(n)x^{n}$

is the ordinary generating function of the sequence
$(c_{A}(n))_{n\in\mathbb{N}}$.

Then, the sequence of coefficients $(C_{A,n}(t))_{n\in\mathbb{N}}$ in the
power series expansion

$C_{A}(t,x)=\frac{1}{1-tf_{A}(x)}=\sum_{n=0}^{\infty}f_{A}(x)^{n}t^{n}=\sum_{n=0}^{\infty}f_{A,n}(t)x^{n}$

have a natural combinatorial interpretation, similar to the interpretation of
the coefficients of the $n$-th $A$-partition polynomial $p_{A,n}(t)$. More
precisely, let us write

$f_{A,n}(t)=\sum_{i=0}^{n}c_{A}(i,n)t^{n}.$

Then, the $i$th coefficient $c_{A}(i,n)$ of $f_{A,n}(t)$ is just the number of
compositions of an integer $n$ into exactly $i$ elements of the set $A$. In
particular, we have that $f_{A,n}(1)=\sum_{i=0}^{n}c_{A}(i,n)=c_{A}(n)$ \- the
number of an $A$-compositions of $n$. Similarly as in the case of
$A$-partitions we investigate the family of sums

$S_{A,k}(n)=\sum_{i=0}^{n}(-1)^{i}i^{k}c_{A}(i,n),$

where $k,n\in\mathbb{N}$, and formulate the following general question.

######  Question 1.1.

Let $A\subset\mathbb{N}_{+}$ and $k\in\mathbb{N}$ be fixed. What can be said
about the behaviour of the sequence
$(\operatorname{sign}(S_{A,k}(n)))_{n\in\mathbb{N}}$? In particular, under
which conditions on $A$, we have that $(-1)^{n}S_{A,k}(n)\geq 0$ for a given
$k$ and all but finitely many $n$?

As we will see, there are some similarities in behaviour of signs of the
sequence $(S_{A,k}(n))_{n\in\mathbb{N}}$ in case of $A$-partitions and
$A$-compositions as well as striking differences between them.

Let us describe the content of the paper in some details. In Section 2 we
collect basic results concerning the sequence of $A$-partition polynomials and
deduce certain recurrence relations for the double sequence
$(S_{A,k}(n))_{k,n\in\mathbb{N}}$. In particular, we obtain convolution-like
identity connecting $S_{A,k+1}(n)$ with $S_{A,j}(n-i)S_{A,k-j}(i)$ for
$j\in\\{0,\ldots,k\\}$ and $i\in\\{0,\ldots,n-1\\}$. In Section 3 we prove
that if $A$ contains only odd numbers, then $(-1)^{n}S_{A,k}(n)\geq 0$ for all
$k,n\in\mathbb{N}$. Moreover, the same statement is true for sets of the form
$A=\mathbb{N}_{+}\setminus E$, where $E\subset\mathbb{N}_{+}$ contains only
even integers. Finally, in the last section, we formulate some problems and
questions which may stimulate further research.

##  2\. Basic observations

In order to investigate the sign behaviour of the sequence
$(S_{A,k}(n))_{n\in\mathbb{N}}$, where $k\in\mathbb{N}$ is fixed, we consider
the differential operator $\delta:\;\mathbb{R}[t]\rightarrow\mathbb{R}[t]$
defined in the following way: for $f\in\mathbb{R}[t]$ we put

$\delta(f(t))=t\frac{df(t)}{dt}.$

For $k=0$ we put $\delta^{(0)}(f)=f$, and for $k\in\mathbb{N}_{+}$ we define
by induction a $k$-th power of the operator $\delta$ as $\delta^{(k)}(f)$,
where

$\delta^{(k)}(f)=\delta(\delta^{(k-1)}(f)).$

The basic properties of the operator $\delta$ can be summarized in the
following well known result (which can be also found in [2, Lemma 2.1]).

######  Lemma 2.1.

  1. (1)

$\delta$ is linear operator, i.e., for all
$u,v\in\mathbb{R},f,g\in\mathbb{R}[t]$ we have
$\delta(uf+vg)=u\delta(f)+v\delta(g)$;

  2. (2)

if $f\in\mathbb{R}[t]\setminus\mathbb{R}$ then
$\operatorname{deg}\delta(f)=\operatorname{deg}f$;

  3. (3)

for $k\in\mathbb{N}$ we have

$\delta^{(k)}(fg)=\sum_{i=0}^{k}\binom{k}{i}\delta^{(i)}(f)\delta^{(k-i)}(g).$

  4. (4)

if $f(t)=\sum_{i=0}^{n}a_{i}t^{i}\in\mathbb{R}[t]$ and $k\in\mathbb{N}$ is
given then

$\delta^{(n)}(f(t))=\sum_{i=0}^{n}a_{i}i^{n}t^{i}$

###### Proof.

The first two properties are easy consequences of the definition of the
operator $\delta$. The proof of the third property is just the application of
the Leibnitz rule together with induction on $k$. The last property follows
from the linearity of the operator $\delta$ and the equality
$\delta^{(k)}(t^{i})=i^{k}t^{i}$. ∎

As an immediate consequence of the fourth property of the operator $\delta$ we
get the equality

$S_{A,k}(n)=\sum_{i=0}^{n}(-1)^{i}i^{k}c_{A}(i,n)=\delta^{(k)}(f_{A,n}(t))\mid_{t=-1}.$

For a given set $V\in\mathbb{N}_{+}$ we put

$V(n):=V\cap\\{1,2,\ldots,n\\},$

i.e., $V(n)$ contains those elements of $V$ which are $\leq n$.

We start with the following general result.

######  Theorem 2.2.

Let $A\in\mathbb{N}_{+}$ and consider the sequence
$(f_{A,n}(t))_{n\in\mathbb{N}}$.

  1. (1)

We have $f_{A,0}(t)=1$ and for $n\in\mathbb{N}_{+}$ the following recurrence
relation holds:

$f_{A,n}(t)=t\sum_{a\in A(n)}f_{A,n-a}(t).$

  2. (2)

We have the following identity

$\sum_{a\in A(n)}a\delta(f_{A,n-a})(t)=\sum_{a\in A(n)}(n-a)f_{A,n-a}(t).$

  3. (3)

We have the following identity

$f_{A,n}(t)+f_{A,n}(-t)=2\sum_{i=0}^{n}f_{A,i}(-t)f_{A,n-i}(t).$

  4. (4)

Let $A=O_{A}\cup E_{A}$, where $O_{A}=\\{a\in A:\;a\equiv 1\mkern
4.0mu({\operator@font mod}\mkern 6.0mu2)\\}$ and $E_{A}=A\setminus
O_{A}=\\{a\in A:\;a\equiv 0\mkern 4.0mu({\operator@font mod}\mkern
6.0mu2)\\}$. Then, we have the following identity

$\frac{1}{2}\sum_{i=0}^{n}f_{O_{A},i}(-t)\left(f_{A,n-i}(-t)+(-1)^{n-i}f_{A,n-i}(t)\right)=\sum_{i=0}^{n}(-1)^{i}f_{A,i}(t)f_{A,n-i}(-t).$

In particular, if $E_{A}=\emptyset$, then for $n\in\mathbb{N}$ we have
$f_{A,n}(-t)=(-1)^{n}f_{A,n}(t)$.

###### Proof.

The first part of our theorem is an immediate consequence of the functional
relation $(1-tf_{A}(x))C_{A}(t,x)=1$. Indeed, for $n\in\mathbb{N}_{+}$, the
$n$-th term in the power series expansion of the left hand side is exactly the
difference of $C_{A,n}(t)$ and $t\sum_{a\in A(n)}C_{A,n-a}(t)$, and hence the
result.

In order to get the second identity, let us observe the following equalities

(1) $\frac{\partial C_{A}(t,x)}{\partial
t}=\frac{f_{A}(x)}{(1-tf_{A}(x))^{2}},\quad\quad\frac{\partial
C_{A}(t,x)}{\partial x}=\frac{tf^{\prime}_{A}(x)}{(1-tf_{A}(x))^{2}}.$

In consequence, we get the identity

$tf_{A}^{\prime}(x)\frac{\partial C_{A}(t,x)}{\partial
t}=f_{A}(x)\frac{\partial C_{A}(t,x)}{\partial x}.$

Comparison of the terms near $x^{n}$ in the expansions

$\displaystyle xtf_{A}^{\prime}(x)\frac{\partial C_{A}(t,x)}{\partial t}$
$\displaystyle=\left(\sum_{a\in
A}ax^{a}\right)\left(\sum_{n=0}^{\infty}tC_{A,n}^{\prime}(t)x^{n}\right)=\sum_{n=0}^{\infty}\left(t\sum_{a\in
A(n)}aC_{a,n-a}^{\prime}(t)\right)x^{n},$ $\displaystyle
xf_{A}(x)\frac{\partial C_{A}(t,x)}{\partial x}$
$\displaystyle=\left(\sum_{a\in
A}x^{a}\right)\left(\sum_{n=0}^{\infty}(n)C_{A,n}(t)x^{n}\right)$
$\displaystyle=\sum_{n=0}^{\infty}\left(\sum_{a\in
A(n)}(n-a)C_{A,n-a}(t)\right)x^{n},$

gives the required identity.

In order to prove (3) it is enough to note the functional identity

$C_{A}(t,x)+C_{A}(-t,x)=\frac{2}{1-t^{2}f_{A}^{2}(x)}=2C_{A}(t,x)C_{A}(-t,x).$

Comparing the terms on both sides of the expansions of
$C_{A}(t,x)+C_{A}(-t,x)$ and $2C_{A}(t,x)C_{A}(-t,x)$ we get the result.

Finally, in order to get the last identity we note the relation
$f_{A}(x)=f_{O_{A}}(x)+f_{E_{A}}(x)$ and $f_{A}(-x)=-2f_{O_{A}}(x)+f_{A}(x)$.
From the shape of $C_{A}(t,x)$, we easily get the identity

$\frac{1}{C_{A}(-t,x)}+\frac{1}{C_{A}(t,-x)}=2(1+tf_{O_{A}}(x))$

and thus

$\frac{1}{2}\frac{1}{1+tf_{O_{A}}(x)}(C_{A}(-t,x)+C_{A}(t,-x))=C_{A}(-t,x)C_{A}(t,-x).$

Noting that $1/(1+tf_{O_{A}}(x))=C_{O_{A}}(-t,x)$ and comparing the terms near
$x^{n}$ on both sides of the above identity, we get:

$\frac{1}{2}\sum_{i=0}^{n}f_{O_{A},i}(-t)(f_{A,n-i}(-t)+(-1)^{n-i}f_{A,n-i}(t))=\sum_{i=0}^{n}(-1)^{i}f_{A,i}(t)f_{A,n-i}(-t),$

which is exactly the identity we were looking for. If $E_{A}=\emptyset$, then
$f_{E_{A}}(x)=0$ and $f_{A}(-x)=f_{O_{A}}(-x)=-f_{O_{A}}(x)=-f_{A}(x)$. In
consequence $C_{A}(-t,x)=C_{A}(t,-x)$ and we get the result.

∎

In the second result we collect identities which will be useful in studying
signs behaviour of the sequence $S_{A,k}(n)$ for given $k,n\in\mathbb{N}$.

######  Theorem 2.3.

Let $A\subset\mathbb{N}_{+}$ and consider the power series expansion

$\frac{f_{A}(x)}{xf^{\prime}_{A}(x)}=\sum_{n=0}^{\infty}q_{A}(n)x^{n}.$

We have the following identities

$\delta(f_{A,n})(t)=\sum_{i=0}^{n}iq_{A}(n-i)f_{A,i}(t)$

and

$\delta(f_{A,n})(t)=\sum_{i=0}^{n-1}f_{A,n-i}(t)f_{A,i}(t).$

###### Proof.

To get the first equality let us note that from the identities (1) we get that

$\displaystyle\sum_{n=0}^{\infty}$
$\displaystyle\delta(f_{A,n})(t)x^{n}=t\frac{\partial C_{A}(t,x)}{\partial
t}=\frac{f_{A}(x)}{xf^{\prime}_{A}(x)}\cdot x\frac{\partial
C_{A}(t,x)}{\partial x}$
$\displaystyle=\left(\sum_{n=0}^{\infty}q_{A}(n)x^{n}\right)\left(\sum_{n=0}^{\infty}nf_{A,n}(t)x^{n}\right)=\sum_{n=0}^{\infty}\left(\sum_{i=0}^{n}iq_{A}(n-i)f_{A,i}(t)\right)x^{n}.$

Comparing the coefficients on penultimate sides of the above chain of
equalities we get the result.

To get the second identity we recall that $f_{A,0}(t)=1$ and note the
following

$\displaystyle\sum_{n=0}^{\infty}$
$\displaystyle\delta(f_{A,n})(t)x^{n}=t\frac{\partial C_{A}(t,x)}{\partial
t}=\frac{tf_{A}(x)}{(1-tf_{A}(x))^{2}}$
$\displaystyle=\frac{1}{(1-tf_{A}(x))^{2}}-\frac{1}{1-tf_{A}(x)}=\sum_{n=0}^{\infty}\left(\sum_{i=0}^{n}f_{A,i}(t)f_{A,n-i}(t)-f_{A,n}(t)\right)x^{n}$
$\displaystyle=\sum_{n=0}^{\infty}\left(\sum_{i=0}^{n-1}f_{A,i}(t)f_{A,n-i}(t)\right)x^{n}.$

Again, comparing coefficients on penultimate sides, we get the result. ∎

Having proved Theorem 2.3 as an immediate consequence we get the following two
recurrence relations satisfied by the double sequence $(S_{A,k}(n))_{k,n}$.

######  Corollary 2.4.

Let $k,n\in\mathbb{N}$. We have the following identities:

$S_{A,k+1}(n)=\sum_{i=0}^{n}iq_{A}(n-i)S_{A,k}(i),$

and

$S_{A,k+1}(n)=\sum_{i=0}^{n-1}\sum_{j=0}^{k}\binom{k}{j}S_{A,j}(n-i)S_{A,k-j}(i).$

In particular, if $(-1)^{n}S_{A,0}(n)\geq 0$ then for each $k\in\mathbb{N}$ we
have $(-1)^{n}S_{A,k}(n)\geq 0$.

###### Proof.

The first identity is a simple induction on $k$. Indeed, we note that the
statement is true for $k=0$. Next, recall that
$S_{A,k}(n)=\delta^{(k)}f_{A,n}(t)|_{t=-1}$. Applying the operator
$\delta^{(k)}$ to the first identity from Theorem 2.3 and taking $t=-1$ we get
the result.

To get the second identity we use the same approach together with Lemma 2.1
(3) and by taking $t=-1$ we get the result.

Finally, the ”in particular” part follows from a simple induction on $k$. The
case $k=0$ is our assumption. Suppose that our statement is true for each
integer $m<k$, i.e., for each $n$ we have $(-1)^{n}S_{A,m}(n)\geq 0$. Then we
have

$(-1)^{n}S_{A,k}(n)=\sum_{i=0}^{n-1}\sum_{j=0}^{k-1}\binom{k-1}{j}[(-1)^{n-i}S_{A,j}(n-i)][(-1)^{i}S_{A,k-j}(i)]\geq
0$

and our result follows. ∎

######  Remark 2.5.

The second recurrence relation given in Corollary 2.4 will be of great use in
the sequel. On the other side, it seems that the first one, although
interesting from a theoretical point of view, is of little use. The problem is
that we are not able to control the sign behaviour of the sequence
$(q_{A}(n))_{n\in\mathbb{N}}$. Indeed, consider the set $A=\\{1,2,3\\}$. We
have $f_{A}(x)=x+x^{2}+x^{3}$ and

$\frac{f_{A}(x)}{xf_{A}^{\prime}(x)}=\frac{x^{2}+x+1}{3x^{2}+2x+1}=\sum_{n=0}^{\infty}q_{A}(n)x^{n}.$

From Theorem 3.9 we easily get that the sequence
$(\operatorname{sign}(q_{A}(n)))_{n\in\mathbb{N}}$ is not periodic. On the
other hand, from Proposition 3.3 below, applied to $m=3$, we get that
$(-1)^{n}S_{A,k}(n)\geq 0$.

##  3\. Results

The aim of this section is to present several classes of sets $A$ having the
property $(-1)^{n}S_{A,k}(n)\geq 0$ for each $k,n\in\mathbb{N}$. We start with
the following general observation.

######  Corollary 3.1.

Let $A\subset 2\mathbb{N}+1$. Then for each $k,n\in\mathbb{N}$ we have
$(-1)^{n}S_{A,k}(n)\geq 0$.

###### Proof.

Let $f_{A}(x)=\sum_{a\in A}x^{a}$. Because $A\subset 2\mathbb{N}+1$ we have
$f_{A}(-x)=-f_{A}(x)$. Let us recall that $S_{A,0}(n)=f_{A,n}(-1)$. We thus
have

$\sum_{n=0}^{\infty}(-1)^{n}S_{A,0}(n)x^{n}=\sum_{n=0}^{\infty}S_{A,0}(n)(-x)^{n}=\frac{1}{1+f_{A}(-x)}=\frac{1}{1-f_{A}(x)}=\sum_{n=0}^{\infty}c_{A}(n)x^{n}.$

We thus have $(-1)^{n}S_{A,0}(n)=c_{A}(n)\geq 0$ and hence the result for
$k=0$. Applying second part of Corollary 2.4 we get that
$(-1)^{n}S_{A,k}(n)\geq 0$ for each $k,n\in\mathbb{N}$. ∎

######  Remark 3.2.

It is interesting fact that exactly the same result is true if we consider
partitions with parts in the set $A$, instead of compositions with parts in
the set $A$ (see [2, Lemma 3.3]).

In the next proposition we write $a_{m}$ for the sequence $(a,\ldots,a)$
containing exactly $m$ occurrences of the symbol $a$.

######  Proposition 3.3.

Let $m\in\mathbb{N}_{+}$ be fixed and put $A=\\{1,\ldots,m\\}$.

  1. (1)

If $m$ is odd then the sequence $((-1)^{n}S_{A,0}(n))_{n\in\mathbb{N}}$ is
periodic of period $m+1$. More precisely,

$(\operatorname{sing}((-1)^{n}S_{A,0}(n)))_{n\in\mathbb{N}}=\overline{(1,1,0_{m-1})}$

and we have $(-1)^{n}S_{A,k}(n)\geq 0$ for each $k,n\in\mathbb{N}$.

  2. (2)

If $m$ is even then the sequence $((-1)^{n}S_{A,0}(n))_{n\in\mathbb{N}}$ is
periodic of period $2(m+1)$. More precisely,

$(\operatorname{sign}((-1)^{n}S_{A,0}(n)))_{n\in\mathbb{N}}=\overline{(1,1,0_{m-1},-1,-1,0_{m-1})}.$

###### Proof.

We have $f_{A}(x)=\sum_{i=1}^{m}x^{i}=x\frac{1-x^{m}}{1-x}$.

If $m$ is odd, then we have

$\displaystyle\sum_{n=0}^{\infty}(-1)^{n}S_{A,0}(n)x^{n}$
$\displaystyle=F(-1,-x)=\frac{1}{1+f_{A}(-x)}=\frac{1+x}{1-x^{m+1}}$
$\displaystyle=1+\sum_{n=1}^{\infty}(I_{m}(n)+I_{m}(n-1))x^{n},$

where $I_{m}(n)=1$ if $m+1|n$ and 0 otherwise. It is clear that the expression
$I_{m}(n)+I_{m}(n-1)$ is non-negative and thus $(-1)^{n}S_{A,0}(n)\geq 0$.
Moreover, $(-1)^{n}S_{A,0}(n)=I_{m}(n)+I_{m}(n-1)=1$ if and only if $n\equiv
0,1\mkern 4.0mu({\operator@font mod}\mkern 6.0mum)$. Thus, we get the first
part of our statement. Applying Corollary 2.4 we get that
$(-1)^{n}S_{A,k}(n)\geq 0$ for each $k,n\in\mathbb{N}$.

If $m$ is even, then we have

$\displaystyle\sum_{n=0}^{\infty}$
$\displaystyle(-1)^{n}S_{A,0}(n)x^{n}=F(-1,-x)=\frac{1}{1+f_{A}(-x)}=\frac{1+x}{1+x^{m+1}}$
$\displaystyle=\frac{(1+x)(1-x^{m+1})}{1-x^{2(m+1)}}=\sum_{n=0}^{\infty}c_{m}(n)x^{n},$

where $c_{m}(n)=J_{m}(n)+J_{m}(n-1)-J_{m}(n-m-1)-J_{m}(n-m-2)$ and
$J_{m}(n)=1$ if $2(m+1)|n$ and 0 otherwise. A careful analysis of the
expression for $c_{m}(n)$ for
$n=2(m+1)p+q,q\in\\{0,1,\ldots,2(m+1)-1\\},p\in\mathbb{N}$, reveals that

$c_{m}(n)=\begin{cases}\begin{array}[]{lll}1&\mbox{for}&n\equiv 0,1\mkern
4.0mu({\operator@font mod}\mkern 6.0mu2(m+1)),\\\ -1&\mbox{for}&n\equiv
m+1,m+2\mkern 4.0mu({\operator@font mod}\mkern 6.0mu2(m+1)),\\\
0&&\mbox{otherwise},\end{array}\end{cases}$

which is exactly the form from the second part of our theorem. ∎

######  Remark 3.4.

Let $A=\\{1,\ldots,m\\}$, where $m$ is even. We proved that for $k=0$, the
sequence $(\operatorname{sign}((-1)^{n}S_{A,0}(n)))_{n\in\mathbb{N}}$ is
periodic with a pure period $2(m+1)$. Our numerical calculations suggest that
this behaviour persist also for $k>0$. We formulate the following.

######  Conjecture 3.5.

Let $A=\\{1,\ldots,m\\}$, where $m$ is even. The sequence
$((-1)^{n}S_{A,k}(n))_{n\in\mathbb{N}}$ is eventually periodic of period
length $2(m+1)$. If $k\geq 2(m+1)$, then the period $T_{m}$ has the form
$T_{m}=(1_{m+1},(-1)_{m+1})$.

######  Remark 3.6.

An infinite version of Proposition 3.3, i.e., the case $m=+\infty$, or
equivalently $A=\mathbb{N}_{+}$ is trivial. Indeed, in this case
$f_{A}(x)=x/(1-x)$ and

$C_{A}(t,x)=\frac{1-x}{1-(1+t)x}.$

In particular, $C_{A}(-1,x)=1-x$ and thus $S_{A,0}(n)=0$ for $n\geq 2$. As an
immediate consequence of this property and Theorem 2.4, we get that
$S_{A,k}(n)=0$ for $n>k+2$. This shows a striking difference between the case
of $A$-compositions and $A$-partitions. Indeed, it is conjectured in [2] that
the sequence sign behaviour of the sequence
$\left(\sum_{i=0}^{n}(-1)^{i}p_{A}(i,n)\right)_{n\in\mathbb{N}}$ is not
eventually periodic.

In Corollary 3.1 we proved that if $A\subset 2\mathbb{N}+1$, then
$(-1)^{n}S_{A,k}(n)\geq 0$ for each $k,n\in\mathbb{N}$. In the next theorem,
we present a broad class of infinite sets $A\subset\mathbb{N}_{+}$ containing
even integers such that $(-1)^{n}S_{A,k}(n)\geq 0$ for each
$k,n\in\mathbb{N}$. More precisely, we have the following

######  Theorem 3.7.

Let $E\subset 2\mathbb{N}_{+}$ and put $A=\mathbb{N}_{+}\setminus E$. Then,
for each $k,n\in\mathbb{N}$ we have $(-1)^{n}S_{A,k}(n)\geq 0$.

###### Proof.

From Corollary 2.4 we know that to get the result it is enough to prove it for
$k=0$. From our assumption we can write

$f_{A}(x)=\sum_{a\in A}x^{a}=\frac{x}{1-x}-f_{E}(x),$

where $f_{E}(x)=\sum_{e\in E}x^{e}$. Moreover, from the assumption on the set
$E$ we have $f_{E}(x)=f_{E}(-x)$. Now, we have the following chain of
equalities

$\displaystyle\sum_{n=0}^{\infty}(-1)^{n}S_{A,0}(n)x^{n}=$
$\displaystyle\frac{1}{1+f_{A}(-x)}=\frac{1}{1-\frac{x}{1+x}-f_{E}(x)}$
$\displaystyle=$
$\displaystyle\frac{1+x}{1-(1+x)f_{E}(x)}=(1+x)\cdot\frac{1}{1-f_{E^{\prime}}(x)}$
$\displaystyle=$
$\displaystyle(1+x)\sum_{n=0}^{\infty}c_{E^{\prime}}(n)x^{n}=1+\sum_{n=1}^{\infty}(c_{E^{\prime}}(n)+c_{E^{\prime}}(n-1))x^{n},$

where $E^{\prime}=E\cup(E+1)$ (note that from the assumption $E\subset
2\mathbb{N}_{+}$ we have $E\cap(E+1)=\emptyset$). We thus have the equality
$(-1)^{n}S_{A,0}(n)=c_{E^{\prime}}(n)+c_{E^{\prime}}(n-1)$ for
$n\in\mathbb{N}_{+}$. It is clear that this expression is non-negative. ∎

In the above theorem we constructed a broad class of infinite sets $A$
containing even (and odd) elements such that $(-1)^{n}S_{A,k}(n)\geq 0$. In
the next result we present another method which allow us to construct finite
sets $A$ containing even and odd elements with required property.

######  Theorem 3.8.

Let $B\subset 2\mathbb{N}+1$ ba a finite set and $m=\\#B$. Assume that the set
of all sums of at most $m$ distinct elements of the set $B$ has exactly
$2^{m}-1$ elements. Then, there is a finite set $A\subset\mathbb{N}$ such that
$B\subset A,A\cap 2\mathbb{N}\neq\emptyset$ and for each $k,n\in\mathbb{N}$ we
have $(-1)^{n}S_{A,k}(n)\geq 0$.

###### Proof.

From Corollary 2.4 we know that to get the result it is enough to prove it for
$k=0$. Without loss of generality we can assume that $\\#B\geq 2$. For
$B=\\{b_{1},\ldots b_{m}\\}$, where $m\in\mathbb{N}_{\geq 2}$ and
$b_{1},\ldots,b_{m}$ are odd, we consider the set $A$ containing all possible
sums of the form $\lambda_{1}+\ldots+\lambda_{j}$, where
$\lambda_{1}<\lambda_{2}<\ldots<\lambda_{i}$ for $i\in\\{1,\ldots,m\\}$ and
$\lambda_{j}\in B$. The elements of the set $A$ are coded by the exponent in
the (polynomial) expansion $f_{A}(x)=\prod_{i=1}^{m}(1+x^{b_{i}})-1$. We then
have the equality

$\displaystyle F_{A}(-1,-x)$
$\displaystyle=\sum_{n=0}^{\infty}(-1)^{n}S_{A,0}(n)x^{n}=\frac{1}{1+f_{A}(-x)}$
$\displaystyle=\prod_{i=1}^{m}\frac{1}{1-x^{b_{i}}}.$

By comparing the coefficients near $x^{n}$ on both sides, we get the equality
$(-1)^{n}S_{A,0}(n)=p_{B}(n)$, where $p_{B}(n)$ is the number of partitions
with parts in the set $B$. It is clear that $p_{B}(n)\geq 0$ for
$n\in\mathbb{N}$ and we get the result. ∎

In the above set of results we described several families of sets $A$ having
the property $(-1)^{n}S_{A,k}(n)\geq 0$. However, as we will see, there are
sets $A$ such that the sequence of signs of $S_{A,0}(n)$ is not eventually
periodic.

######  Theorem 3.9.

Let $p(x)\in\mathbb{R}[x]$ be a nonzero polynomial of the form

$p(x)=c\prod_{i=0}^{\rho}(1-\omega_{i}x)^{\alpha_{i}},$

where $\omega_{i}$ are pairwise different and $\alpha_{i}\in\mathbb{N}_{+}$
for $i=0,\ldots,\rho$. Assume that the number $\omega_{0}$ is non-real with
$\operatorname{Arg}(\omega_{0})\notin\mathbb{Q}\pi$ and such that its modulus
is maximal, i.e., $|\omega_{0}|>|\omega_{i}|$ for
$\omega_{i}\neq\omega_{0},\overline{\omega_{0}}$. Then, if
$1/p(x)=\sum_{k=0}^{\infty}a_{k}x^{k}$, then the sequence
$(\operatorname{sign}(a_{k}))_{k\in\mathbb{N}}$ is not eventually periodic.

###### Proof.

For convenience assume that $\omega_{1}=\overline{\omega_{0}}$. Set
$\omega=\omega_{0}$. We have

$a_{k}=P(k)\omega^{k}+Q(k)\overline{\omega}^{k}+\sum_{i>1}P_{i}(k)\omega_{i}^{k},$

where $P_{i},P,Q$ are polynomials with coefficients in $\mathbb{C}$. The
conjugation on $\mathbb{C}$ extends to the automorphism of the ring
$\mathbb{C}[x]$ in the following way:

$\mathbb{C}[x]\ni
f=\sum_{i=0}^{k}a_{i}x^{i}\mapsto\overline{f}:=\sum_{i=0}^{k}\overline{a_{i}}x^{i}\in\mathbb{C}[x].$

Claim: $Q=\overline{P}$
We will postpone the proof of the claim to the end. Let us write
$P(z)=bz^{n}+R(z)$, where $\operatorname{deg}R<n.$ We have the following chain
of equalites

$\displaystyle\frac{a_{k}}{|b\omega^{k}|k^{n}}$
$\displaystyle=\frac{R(k)}{|b|k^{n}}\zeta^{k}+\frac{\overline{R}(k)}{|b|k^{n}}\overline{\zeta}^{k}+\frac{b}{|b|}\zeta^{k}+\frac{\overline{b}}{|b|}\overline{\zeta}^{k}+\sum_{i>1}\frac{P_{i}(k)}{|b|k^{n}}\left(\frac{\omega_{i}}{|\omega|}\right)^{k}$
$\displaystyle=2\text{Re}(c\zeta^{k})+o(1),$

where $\zeta=\omega/|\omega|$ and $c=b/|b|.$ Finally, we get that

$\operatorname{sign}(a_{k})=\operatorname{sign}\left(\frac{a_{k}}{|bz^{k}|k^{n}}\right)=\operatorname{sign}(2\operatorname{Re}(c\zeta^{n})+o(1)).$

Set $c=e^{2\pi i\gamma}$ and $\zeta=e^{2\pi i\delta}$. The assumption
$\operatorname{Arg}(w)\notin\mathbb{Q}\pi$ says that $\delta$ is irrational.
Then $2\operatorname{Re}(c\zeta^{n})$ depends only on the fractional part
$\\{\gamma+n\delta\\}$. Also looking at the sign of
$2\text{Re}(c\zeta^{n})+o(1)$ we can ignore error as long as
$2\operatorname{Re}(c\zeta^{n})$ is big enough, i.e., the corresponding
fractional part is far from $\frac{1}{2}\mathbb{Z}.$ Now, if the sequence
$(\operatorname{sign}(a_{k}))_{k\in\mathbb{N}}$ is eventually periodic then we
can find $d$ such that $\operatorname{sign}(a_{kd})$ is constant for
sufficiently large $k$. This means that $\\{\gamma+k(d\delta)\\}$ always omits
the interval $(\epsilon,1/2-\epsilon)$ or the interval
$(1/2+\epsilon,1-\epsilon)$. However, the number $\delta$ (and so $d\delta$)
is irrational, and hence the sequence $(\gamma+k(d\delta))_{k\in\mathbb{N}}$
is equidistributed. This is a contradiction, and we are only left with proving
the claim.
Proof of the claim: Let

$S(x)=c(x-b_{1})^{s_{1}}\ldots(x-b_{m})^{s_{m}}(x^{2}+q_{1}x+r_{1})^{t_{1}}\ldots(x^{2}+q_{l}x+r_{l})^{t_{n}}$

be a decomposition of the polynomial $S$ into irreducible factors in
$\mathbb{R}[x].$ Then

$1/p(x)=\sum_{i=1}^{m}\sum_{s=1}^{s_{i}}\frac{A_{i,s}}{(x-b_{i})^{s}}+\sum_{j=1}^{l}\sum_{t=1}^{t_{j}}\frac{B_{j,t}x+C_{j,t}}{(x^{2}+q_{j}x+r_{j})^{t}},$

where $A_{i,s},B_{j,t},C_{j,t}\in\mathbb{R}$. Notice that it is enough to
prove the claim for $\frac{1}{(x^{2}+qx+r)^{t}}.$ Write
$x^{2}+qx+r=(x-z)(x-\overline{z})$ and observe that

$\frac{1}{(x^{2}+qx+r)^{t}}=\frac{f_{1}(x)}{(x-z)^{t}}+\frac{f_{2}(x)}{(x-\overline{z})^{t}},$

for $f_{1},f_{2}\in\mathbb{C}[x]$ satisfying
$\operatorname{deg}f_{1},\operatorname{deg}f_{2}<t.$ Now, if we prove that
$f_{2}=\overline{f_{1}}$, then we are done. However, we have

$1=f_{1}(x)(x-z)^{t}+f_{2}(x)(x-\overline{z})^{t}=\overline{f_{1}}(x)(x-\overline{z})^{t}+\overline{f_{2}}(x)(x-z)^{t},$

and hence $(x-z)^{t}\mid f_{2}(x)-\overline{f_{1}}(x)$. Because the degree of
$f_{2}-\overline{f_{1}}$ is $<t$, we get that $f_{2}=\overline{f_{1}}$. This
ends the proof of the claim and of the whole theorem. ∎

######  Corollary 3.10.

If $A=\\{2,3\\}$ or $A=\\{1,4\\}$ then the sequence
$(\operatorname{sign}(S_{A,0}(n)))_{n\in\mathbb{N}}$ is not eventually
periodic.

###### Proof.

We just need to check that assumptions of previous theorems hold in both
cases. Let us take first $A=\\{2,3\\}$ which gives $p(x)=1+x^{2}+x^{3}.$ Let
$t$ be the unique real root of $p$ and $z$ be its complex root. We have
$x\approx-1.46$ and $z\approx 0.23-0.79i$, and thus indeed $|z|<|x|.$ The last
thing to check is that $\zeta=z/|z|$ is not a root of unity. We have that
$[\mathbb{Q}(\zeta):\mathbb{Q}]$ is at most 12 so it is enough to check that
$\zeta^{12}\neq 1$ but $\zeta^{12}\approx-0.95-0.28i$ so all the conditions of
Theorem 3.9 are met. Similarly for the second case. ∎

##  4\. Some problems and a question

In Theorem 3.8 we constructed a broad class of finite sets $A$ containing even
(and odd) elements such that $(-1)^{n}S_{A,k}(n)\geq 0$. However, one can go
further and state the following general problem.

######  Problem 4.1.

Let $A\subset\mathbb{N}_{+}$ and suppose that there exist $k\in\mathbb{N}$
such that for infinitely many $n$ the inequality $(-1)^{n}S_{A,k}(n)<0$ holds.
Construct an optimal set $B\supset A$, i.e., with the minimal possible values
of $\\#(B\setminus A)$ and $\operatorname{min}(B)$, such that
$(-1)^{n}S_{B,k}(n)\geq 0$ for every $k$.

From Proposition 3.3 we know that if $A=\\{1,\ldots,m\\}$ and $m$ is even, the
optimal set for $A$ is $B=A\cup\\{m+1\\}$. However, in general, it is not
clear whether, for a given set $A$, an optimal set $B$ is uniquely determined.

######  Question 4.2.

Let $N\in\mathbb{N}_{+}$. What can be said about the quantity

$F(N):=\\#\\{A\subset\\{1,\ldots,N\\}:\;\forall
k\in\mathbb{N}\;\forall!n\in\mathbb{N}:\;(-1)^{n}S_{A,k}(n)\geq 0\\}?$

From Corollary 3.1 we know that $F(n)\geq 2^{\lfloor\frac{n+1}{2}\rfloor}$.

######  Problem 4.3.

For which disjoint sets $A,B\subset\mathbb{N}_{+}$ the conditions
$(-1)^{n}S_{A,0}(n)\geq 0,(-1)^{n}S_{B,0}(n)\geq 0$ implies $(-1)^{n}S_{A\cup
B,0}(n)\geq 0$?

One can formulate the following simpler problem.

######  Problem 4.4.

Let $B\subset 2\mathbb{N}+1$. Does there exist an even number $b$ such that
for $A=B\cup\\{b\\}$ we have $(-1)^{n}S_{A,0}(n)\geq 0$?

It seems that if $m>3$ is even and we take
$B=\\{\frac{m^{i}-1}{m-1}:\;i\in\mathbb{N}_{+}\\}$ then one can take $b=m$. In
general let us note that if $A\cap B=\emptyset$, then $f_{A\cup
B}(x)=f_{A}(x)+f_{B}(x)$ and thus, using the equalities

$C_{A}(t,x)=\frac{1}{1-tf_{A}(x)},\quad C_{B}(t,x)=\frac{1}{1-tf_{B}(x)},\quad
C_{A\cup B}(t,x)=\frac{1}{1-tf_{A\cup B}(x)},$

we can eliminate $f_{A},f_{B},f_{A\cup B}$ and get the functional relation

$\frac{1}{C_{A}(t,x)}+\frac{1}{C_{B}(t,x)}-\frac{1}{C_{A\cup B}(t,x)}=1,$

and natural question arises whether is it possible to use it to deduce any
sufficient conditions on $A,B$ which will guarantee that $(-1)^{n}S_{A\cup
B,0}(n)\geq 0$ for all $n\in\mathbb{N}$.

## References

  * [1] G. E. Andrews, The Theory of Partitions, Cambridge University Press, Cambridge, 1998.
  * [2] F. Gawron, M. Ulas, Sign behaviour of sums of weighted numbers of partitions, Ramanujan J. 60 (2023), 571–584.
  * [3] M. Merca, On the sum of parts in the partitions of $n$ into distinct parts, Bull. Aust. Math. Soc. 104, No. 2 (2021), 228–237.
  * [4] M. Ulas, B. Żmija, On arithmetic properties of binary partition polynomials, Adv. Appl. Math. 110 (2019), 153–179.

Filip Gawron, Jagiellonian University, Faculty of Mathematics and Computer
Science, Institute of Mathematics, Łojasiewicza 6, 30-348 Kraków, Poland;
email<EMAIL_ADDRESS>

Maciej Ulas, Jagiellonian University, Faculty of Mathematics and Computer
Science, Institute of Mathematics, Łojasiewicza 6, 30-348 Kraków, Poland;
email<EMAIL_ADDRESS>