# embedding_metadata

key: chroma:document
string_value:
Increasing the Inference and Learning Speed of
Tsetlin Machines with Clause Indexing
Saeed Rahimi Gorji1[0000−0002−2699−9903], Ole-Christoﬀer
Granmo1[0000−0002−7287−030X], Sondre Glimsdal1, Jonathan Edwards2, and
Morten Goodwin1[0000−0002−7287−030X]
1Centre for Artiﬁcial Intelligence Research, University of Agder, Grimstad, Norway
2Temporal Computing, Newcastle, United Kingdom
saeed.r.gorji@uia.no, ole.granmo@uia.no, sondre.glimsdal@uia.no,
jonny@temporalcomputing.com, morten.goodwin@uia.no


key: chroma:document
string_value:
Abstract. The Tsetlin Machine (TM) is a machine learning algorithm
founded on the classical Tsetlin Automaton (TA) and game theory. It fur-
ther leverages frequent pattern mining and resource allocation principles
to extract common patterns in the data, rather than relying on minimiz-
ing output error, which is prone to overﬁtting. Unlike the intertwined
nature of pattern representation in neural networks, a TM decomposes


key: chroma:document
string_value:
problems into self-contained patterns, represented as conjunctive clauses.
The clause outputs, in turn, are combined into a classiﬁcation decision
through summation and thresholding, akin to a logistic regression func-
tion, however, with binary weights and a unit step output function. In
this paper, we exploit this hierarchical structure by introducing a novel
algorithm that avoids evaluating the clauses exhaustively. Instead we


key: chroma:document
string_value:
use a simple look-up table that indexes the clauses on the features that
falsify them. In this manner, we can quickly evaluate a large number of
clauses through falsiﬁcation, simply by iterating through the features and
using the look-up table to eliminate those clauses that are falsiﬁed. The
look-up table is further structured so that it facilitates constant time
updating, thus supporting use also during learning. We report up to 15


key: chroma:document
string_value:
times faster classiﬁcation and three times faster learning on MNIST and
Fashion-MNIST image classiﬁcation, and IMDb sentiment analysis.
Keywords: Tsetlin Machines ·Pattern Recognition ·Propositional Logic
·Frequent Pattern Mining ·Learning Automata ·Pattern Indexing
1 Introduction
The Tsetlin Machine (TM) is a recent machine learning approach that is based on
the Tsetlin Automaton (TA) by M. L. Tsetlin [ 1], one of the pioneering solutions


key: chroma:document
string_value:
to the well-known multi-armed bandit problem [ 2,3] and the ﬁrst Learning
Automaton [ 4]. It further leverages frequent pattern mining [ 5] and resource
allocation [ 6] to capture frequent patterns in the data, using a limited number
of conjunctive clauses in propositional logic. The clause outputs, in turn, are
combined into a classiﬁcation decision through summation and thresholding,arXiv:2004.03188v1  [cs.LG]  7 Apr 2020


key: chroma:document
string_value:
2 S. R. Gorji et al.
akin to a logistic regression function, however, with binary weights and a unit
step output function [ 7]. Being based on disjunctive normal form (DNF), like
Karnaugh maps, the TM can map an exponential number of input feature value
combinations to an appropriate output [8].
Recent Progress on TMs. Recent research reports several distinct TM prop-
erties. The clauses that a TM produces have an interpretable form (e.g., ifX


key: chroma:document
string_value:
satisﬁes condition A and not condition B thenY = 1), similar to the branches
in a decision tree [ 7]. For small-scale pattern recognition problems, up to three
orders of magnitude lower energy consumption and inference time has been
reported, compared to neural networks alike [ 9]. The TM structure captures re-
gression problems, comparing favourably with Regression Trees, Random Forests
and Support Vector Regression [ 10]. Like neural networks, the TM can be used


key: chroma:document
string_value:
in convolution, providing competitive memory usage, computation speed, and
accuracy on MNIST, F-MNIST and K-MNIST, in comparison with simple 4-
layer CNNs, K-Nereast Neighbors, SVMs, Random Forests, Gradient Boosting,
BinaryConnect, Logistic Circuits, and ResNet [ 11]. By introducing clause weights
that allow one clause to represent multiple, it has been demonstrated that the
number of clauses can be reduced up to 50×, without loss of accuracy, leading to


key: chroma:document
string_value:
more compact clause sets [ 8]. Finally, hyper-parameter search can be simpliﬁed
with multi-granular clauses, eliminating the pattern speciﬁcity parameter [12].
Paper Contributions and Organization. The expression power of a TM is
governed by the number of clauses employed. However, adding clauses to increase
accuracy comes at the cost of linearly increasing computation time. In this paper,
we address this problem by exploiting the hierarchical structure of the TM. We


key: chroma:document
string_value:
ﬁrst cover the basics of TMs in Sect. 2. Then, in Sect. 3, we introduce a novel
indexing scheme that speeds up clause evaluation. The scheme is based on a
simple look-up tables that indexes the clauses on the features that falsify them.
We can then quickly evaluate a large number of clauses through falsiﬁcation,
simply by iterating through the features and using the look-up table to eliminate
those clauses that are falsiﬁed. The look-up table is further structured so that


key: chroma:document
string_value:
it facilitates constant time updating, thus supporting indexing also during TM
learning. In Sect. 4, we report up to 15 times faster classiﬁcation and three times
faster learning on three diﬀerent datasets: IMDb sentiment analysis, and MNIST
and Fashion-MNIST image classiﬁcation. We conclude the paper in Sect. 5.
2 Tsetlin Machines
The Tsetlin Machine (TM)[ 13] is a novel machine learning approach that is
particularly suited to low power and explainable applications [ 9,7]. The structure


key: chroma:document
string_value:
of a TM is similar to the majority of learning classiﬁcation algorithms, with
forward pass inference and backward pass parameter adjustment (Fig. 1). A
key property of the approach is that the whole system is Boolean, requiring an


key: chroma:document
string_value:
Tsetlin Machines with Clause Indexing 3
encoder and decoder for any other form of data. Input data is deﬁned as a set of
feature vectors: D={x1,x2, . . .}, and the output Yis formulated as a function
parameterized by the actions Aof a team of TAs: Y=f(x;A).
TA Team. The function fis parameterized by a team of TAs, each deciding the
value of a binary parameter ak∈A(labelling the full collection of parameters A).
Each TA has an integer state tk∈{1, . . . , 2N}, with 2Nbeing the size of the


key: chroma:document
string_value:
state space. The action akof a TA is decided by its state: ak= 1iftk> N else0.
Classiﬁcation Function. The function fis formulated as an ensemble of
nconjunctive clauses, each clause controlling the inclusion of a feature and
(separately) its negation. A clause Cjis deﬁned as:
Cj(x) =o⋀
k=1(xk∨¬aj
k)∧(¬xk∨¬aj
o+k). (1)
Here, xkis the kth feature of input vector x∈Do,ois the total number of
features and aj
kis the action of the TA that has been assigned to literal kfor


key: chroma:document
string_value:
clause j. A variety of ensembling methods can be used – most commonly a unit
step function, with half the clauses aimed at inhibition:
ˆy= 1 if
n/2∑
j=1C+
j(x)−n/2∑
j=1C−
j(x)
≥0else 0. (2)
When the problem is multi-classed (one-hot encoded [15]) thresholding is removed
and the maximum vote is used to estimate the class:
ˆy=argmaxi
n/2∑
j=1Ci+
j(x)−n/2∑
j=1Ci−
j(x)
, (3)
with ireferring to the class that the clauses belong to.


key: chroma:document
string_value:
Learning. Learning controls the inclusion of features or their negation. A TA k
is trained using rewards and penalties. If tk> N(action 1), a reward increases
the state, tk+= 1, up to 2N, whilst a penalty decreases the state, tk−= 1. If
tk<=N(action 0), a reward decreases the state, tk−= 1, down to 1, whilst a
penalty increases the state, tk+= 1. The learning rules are described in detail in
[13], the basic premise being that individual TA can be controlled by examining


key: chroma:document
string_value:
their eﬀect on the clause output relative to the desired output, the clause output,
the polarity (an inhibitory or excitory clause), and the action. For example, if (i)
a feature is 1, (ii) the TA state governing the inclusion of the feature is resolved
to action 1, and (iii) the overall clause outputs 1when a 1output is desired, then
the TA is rewarded, because it is trying to do the right action.
The learning approach has the following added randomness:


key: chroma:document
string_value:
4 S. R. Gorji et al.
Fig. 1.The backwards pass requires no gradient methods (just manipulation of all TA
states ( tk)). Due to the negation of features, clauses can singularly encode non-linearly
separable boundaries. There are no sigmoid style functions, all functions are compatible
with low level hardware implementations.
–An annealing style cooling parameter Tregulates how many clauses are
activated for updating. In general, increasing Ttogether with the number of


key: chroma:document
string_value:
clauses leads to an increase in accuracy.
–A sensitivity swhich controls the ratio of reward to penalty, based on a split
of the total probability, with values 1/sand1−1/s. This parameter governs
the ﬁne-grainedness of the clauses, with a higher sproducing clauses with
more features included.
When comparing the TMs with a classical neural network [ 14] (Fig. 1), noteworthy
points include: Boolean/integer rather than ﬂoating point internal representation;


key: chroma:document
string_value:
logic rather than linear algebra; and binary inputs/outputs.
3 Fast Clause Falsiﬁcation Through Indexing
In this section we describe the details of our clause indexing scheme, including
index based inference and learning.
Overall Clause Indexing Strategy. Recall that each TM clause is a conjunc-
tion of literals. The way that the TM determines whether to include a literal in a
clause is by considering the action of the corresponding TA. Thus, to successfully


key: chroma:document
string_value:
Tsetlin Machines with Clause Indexing 5
Class 1: x 1
Class 1:  x2
Class 1: ¬x 1
Class 1: ¬x 2
Class 2: x 1
Class 2:  x2
Class 2: ¬x1
Class 2: ¬x2n: 3 C1C1C2+ + -
n: 2 C1C2- -
n: 2 C2C1- -
n: 1 C2+
n: 3 C2C1C2- + +
n: 2 C1C2+ +
n: 2 C2C1+ -
n: 3 C2C1C1- + -C1C2C1C2+ + - -
1 3 2
2 11 2
1
2 3 1
1 2
1 2
3 2 1Class 1: x 1
Class 1:  x2
Class 1: ¬x 1
Class 1: ¬x 2
Class 2: x 1
Class 2:  x2
Class 2: ¬x1
Class 2: ¬x2
Fig. 2.Indexing structure.


key: chroma:document
string_value:
Class 2: ¬x2
Fig. 2.Indexing structure.
evaluate a clause, the TM must scan through all the actions of the team of
TAs responsible for the clause, and perform the necessary logical operations to
calculate its truth value. However, being a conjunction of literals, it is actually
suﬃcient to merely identify a single literal that is false to determine that the
clause itself is false. Conversely, if no false literals are present, we can conclude


key: chroma:document
string_value:
that the clause is true. Our approach is thus to maintain a list for each literal
that contains the clauses that include that literal, and to use these lists to speed
up inference and learning.
Index Based Inference. We ﬁrst consider the clause lists themselves (Fig. 2,
left), before we propose a position matrix that allows constant time updating of
the clause lists (Fig. 2, right). For each class-literal pair (i, k), we maintain an
inclusion list Li


key: chroma:document
string_value:
inclusion list Li
kthat contains all the clauses in class ithat include the literal
lk. Let the cardinality of Li
kbe|Li
k|=ni
k. Evaluating the complete set of class
iclauses then proceeds as follows. Given a feature vector x= (x1, x2, . . . , x o),
we deﬁne the index set LFfor the false literals as LF={k|lk= 0}. Then
Ci
F=⋃
k∈LFLi
kis the clauses of class ithat have been falsiﬁed by x. We
partitionCi
Finto clauses with positive polarity and clauses with negative polarity:
Ci
F=Ci+
F∪Ci−


key: chroma:document
string_value:
Ci
F=Ci+
F∪Ci−
F. The predicted class can then simply be determined based on
the cardinality of the sets of falsiﬁed clauses:
ˆy=argmaxi(
|Ci−
F|−|Ci+
F|)
, (4)
Alongside the inclusion list, an m×n×2omatrixM, with entries Mij
k, keeps
track of the position of each clause jin the inclusion list for literal k(Fig. 2,
right). With the help of M, searching and removing a clause from a list can take
place in constant time.


key: chroma:document
string_value:
6 S. R. Gorji et al.
Index Construction and Maintenance. Constructing the inclusion lists and
the accompanying position matrix is rather straightforward since one can initialize
all of the TAs to select the exclude action. Then the inclusion lists are empty,
and, accordingly, the position matrix is empty as well.
Insertion can be done in constant time. When the TA responsible for e.g.
literal lkin clause Ci
jchanges its action from exclude to include, the corresponding
inclusion list Li


key: chroma:document
string_value:
inclusion list Li
kmust be updated accordingly. Adding a clause to the list is fast
because it can be appended to the end of the list using the size of the list ni
k:
ni
k←ni
k+ 1
Li
k[ni
k]←Ci
j
Mij
k←ni
k.
Deletion, however, requires that we know the position of the clause that we
intend to delete. This happens when one of the TAs changes from including to
excluding its literal, say literal lkin clause Ci
j. To avoid searching for Ci
jinLi
k,
we use Mij
kfor direct lookup:
Li
k[Mij
k]←Li
k[ni


key: chroma:document
string_value:
kfor direct lookup:
Li
k[Mij
k]←Li
k[ni
k]
ni
k←ni
k−1
Mi,Li
k[Mij
k]
k←Mij
k
Mij
k←NA.
As seen, the clause is deleted by overwriting it with the last clause in the list,
and then updating Maccordingly.
Memory Footprint. To get a better sense of the impact of indexing on memory
usage, assume that we have a classiﬁcation problem with mclasses and feature
vectors of size o, to be solved with a TM. Thus, the machine has mclasses


key: chroma:document
string_value:
where each class contains nclauses. The memory usage for the whole machine is
about 2×m×n×obytes, considering a bitwise implementation and TAs with
8-bit memory. The indexing data structure then requires two tables with m×o
rows and ncolumns. Consequently, assuming 2-byte entries, each of these tables
require 2×m×n×obytes of memory, which is roughly the same as the memory
required by the TM itself. As a result, using the indexing data structure roughly
triples memory usage.


key: chroma:document
string_value:
triples memory usage.
Step-by-Step Example. Consider the indexing conﬁguration in Fig. 2 and the
feature vector x= (1,0). To evaluate the clauses of class 1, i.e., C+
1,C−
1,C+
2,
andC−
2(omitting the class index for simplicity), we initially assume that they
all are true. This provides two votes for and two votes against the class, which
gives a class score of 0. We then look up the literals ¬x1andx2, which both are
false, in the corresponding inclusion lists. We ﬁrst obtain C−
1andC−


key: chroma:document
string_value:
Tsetlin Machines with Clause Indexing 7
which falsiﬁes C−
1andC−
2. This changes the class score from 0to2because two
negative votes have been removed from the sum. We then obtain C−
1andC−
2
from x2, however, these have already been falsiﬁed, so are ignored. We have now
derived a ﬁnal class score of 2simply by obtaining a total of four clause index
values, rather than fully evaluating four clauses over four possible literals.
Let us now assume that the clause C+


key: chroma:document
string_value:
Let us now assume that the clause C+
1needs to be deleted from the inclusion
list of x1. The position matrix Mis then used to update the inclusion list in
constant time. By looking up x1andC+
1for class 1 inM, we obtain the position
ofC+
1in the inclusion list L1
1ofx1, which is 1. Then the last element in the list,
C+
2, is moved to position 1, and the number of elements in the list is decremented.
Finally, the entry 1at row Class 1 :x1and column C+
1is erased, while the entry


key: chroma:document
string_value:
1is erased, while the entry
3at row Class 1 :x1and column C+
2is set to 1to reﬂect the new position of
C+
2in the list. Conversely, if C+
1needs to be added to e.g. the inclusion list of
x2, it is simply added to the end of the Class 1 :x2list, in position 3. ThenM
is updated by writing 3into row Class 1 : x2and column C+
1.
Remarks. As seen above, the maintenance of the inclusion lists and the position
matrix takes constant time. The performance increase, however, is signiﬁcant.


key: chroma:document
string_value:
Instead of going through all the clauses, evaluating each and every one of them,
considering all of the literals, the TM only goes through the relevant inclusion
lists. Thus, the amount of work becomes proportional to the size of the clauses.
For instance, for the MNIST dataset, where the average length of clauses is
about 58, the algorithm goes through the inclusion list for half of the literals
(the falsifying ones), each containing about 740clauses on average. Thus, instead


key: chroma:document
string_value:
of evaluating 20 000clauses (the total number of clauses), by considering 1568
literals for each (in the worst case), the algorithm just considers 784falsifying
literals with an average of 740clauses in their inclusion lists. In other words,
the amount of work done to evaluate all the clauses with indexing is roughly
0.02compared to the case without indexing. Similarly, for the IMDB dataset,
the average clause length is about 116, and each inclusion list contains roughly


key: chroma:document
string_value:
232clauses on average. Without indexing, evaluating a clause requires assessing
10 000literals, while indexing only requires assessing the inclusion lists for
5 000falsifying literals. Using indexing reduces the amount of work to roughly
0.006of the work performed without indexing. Thus, in both examples, using
indexing reduces computation signiﬁcantly. We now proceed to investigate how
the indexing aﬀect inference and learning speed overall.
4 Empirical Results


key: chroma:document
string_value:
4 Empirical Results
To measure the eﬀect of indexing we use three well-known datasets, namely
MNIST, Fashion-MNIST, and IMDb. We explore the eﬀect of diﬀerent number of
clauses and features, to determine the eﬀect indexing has on performance under
varying conditions.
Our ﬁrst set of experiments covers the MNIST dataset. MNIST consists of
28×28-pixel grayscale images of handwritten digits. We ﬁrst binarize the images,


key: chroma:document
string_value:
8 S. R. Gorji et al.
converting each image into a 784-bit feature vector. We refer to this version of
binarized MNIST as M1. Furthermore, to investigate the eﬀect of increasing
the number of features on indexing, we repeat the same experiment using 2, 3
and 4 threshold-based grey tone levels. This leads to feature vectors of size 1568
(= 2×784), 2352 ( = 3×784) and 3136 ( = 4×784), respectively. We refer to
these datasets as M2, M3 and M4.


key: chroma:document
string_value:
these datasets as M2, M3 and M4.
Table 1 summarizes the resulting speedups. Diﬀerent rows correspond to
diﬀerent number of clauses. The columns refer to diﬀerent numbers of features,
coveringtrainingandinference.Asseen,increasingthenumberofclausesincreases
the eﬀect of indexing. In particular, using more than a few thousands clauses
results in about three times faster learning and seven times faster inference.
Table 1. Indexing speedup for diﬀerent number of clauses and features on MNIST


key: chroma:document
string_value:
Features 784 1568 2352 3136
Clauses TrainTestTrainTestTrainTestTrainTest
10001.782.751.542.791.763.741.914.15
20001.672.832.635.742.465.952.876.06
50002.625.953.238.023.266.883.347.17
10000 2.826.313.498.313.477.603.436.78
20000 2.695.223.588.023.346.443.356.39
Fig. 3 and Fig. 4 plots the average time of each epoch across our experiments,
as a function of the number of clauses. From the ﬁgures, we see that after an
initial phase of irregular behavior, both indexed and unindexed versions of the


key: chroma:document
string_value:
TM suggest a linear growth with roughly the same slopes. Thus, the general eﬀect
of using indexing on MNIST is a several-fold speedup, while the computational
complexity of the TM algorithm remains unchanged, being linearly related to
the number of clauses.
In our second set of experiments, we used the IMDb sentiment analysis dataset,
consisting of 50,000 highly polar movie reviews, divided in positive and negative


key: chroma:document
string_value:
reviews. We varied the problem size by using binary feature vectors of size 5000,
10000, 15000 and 20000, which we respectively refer to as I1, I2, I3 and I4.
Table 2 summarizes the resulting speedup for IMDb, employing diﬀerent
number of clauses and features. Interestingly, in the training phase, indexing
seems to slow down training by about 10%. On the ﬂip side, when it comes to
inference, indexing provided a speedup of up to 15 times.


key: chroma:document
string_value:
Fig. 5 and Fig. 6 compare the average time of each epoch across our ex-
periments, as a function of the number of clauses. Again, we observe some
irregularities for smaller clause numbers. However, as the number of clauses
increases, the computation time of both the indexed and unindexed versions of
the TM grows linearly with the number of clauses.
In our third and ﬁnal set of experiments, we consider the Fashion-MNIST


key: chroma:document
string_value:
Tsetlin Machines with Clause Indexing 9
1,000 2,000 5,000 10,000 20,0001101001,000
ClausesTime [s]Unindexed M4
Unindexed M3
Unindexed M2
Unindexed M1
Indexed M4
Indexed M3
Indexed M2
Indexed M1
Fig. 3.Average training time on MNIST
1,000 2,000 5,000 10,000 20,0001101001,000
ClausesTime [s]Unindexed M4
Unindexed M3
Unindexed M2
Unindexed M1
Indexed M4
Indexed M3
Indexed M2
Indexed M1
Fig. 4.Average inference time on MNIST


key: chroma:document
string_value:
10 S. R. Gorji et al.
Table 2. Indexing speedup for diﬀerent number of clauses and features on IMDb
Features 5000 10000 15000 20000
Clauses TrainTestTrainTestTrainTestTrainTest
10000.761.950.841.590.811.780.812.03
20001.004.600.873.170.844.470.834.81
50000.998.580.938.660.859.410.879.49
10000 1.0615.400.9211.920.9013.170.8713.03
20000 1.0515.190.9315.870.8613.840.8813.28
1,000 2,000 5,000 10,000 20,000101001,00010,000
ClausesTime [s]Unindexed I4
Unindexed I3
Unindexed I2
Unindexed I1
Indexed I4


key: chroma:document
string_value:
Unindexed I3
Unindexed I2
Unindexed I1
Indexed I4
Indexed I3
Indexed I2
Indexed I1
Fig. 5.Average training time on IMDb


key: chroma:document
string_value:
Tsetlin Machines with Clause Indexing 11
1,000 2,000 5,000 10,000 20,0001101001,000
ClausesTime [s]Unindexed I4
Unindexed I3
Unindexed I2
Unindexed I1
Indexed I4
Indexed I3
Indexed I2
Indexed I1
Fig. 6.Average inference time on IMDb
with a label from 10 classes of diﬀerent kinds of clothing items. Similar to MNIST,
we binarized and ﬂattened the data using diﬀerent numbers of bits for each pixel
(ranging from one to four bits by means of thresholding). This process produced


key: chroma:document
string_value:
four diﬀerent binary datasets with feature vectors of size 784, 1568, 2352 and
3136. We refer to these datasets as F1, F2, F3 and F4, respectively. The results
are summarized in Table 3 for diﬀerent number of clauses and features. Similar
to MNIST, in the training phase, we see a slight speedup due to indexing, except
when the number of clauses or the number of features are small. In that case,
indexing slows down training to some extent (by about 10%). On the other hand,


key: chroma:document
string_value:
for inference, we see a several-fold speedup (up to ﬁve-fold) due to indexing,
especially with more clauses.
Table 3. Indexing speedup for diﬀerent number of clauses and features on Fashion-
MNIST
Features 784 1568 2352 3136
Clauses TrainTestTrainTestTrainTestTrainTest
10000.860.820.971.331.041.681.041.77
20000.921.171.232.551.242.931.253.13
50001.232.211.322.851.332.841.332.95
10000 1.553.811.584.361.674.921.563.95
20000 1.553.541.654.541.704.741.675.05


key: chroma:document
string_value:
12 S. R. Gorji et al.
Fig. 7 and Fig. 8 compare the average time of each epoch as a function
of the number of clauses. Again, we observe approximately linearly increasing
computation time with respect to the number of clauses.
1,000 2,000 5,000 10,000 20,000101001,000
ClausesTime [s]Unindexed F4
Unindexed F3
Unindexed F2
Unindexed F1
Indexed F4
Indexed F3
Indexed F2
Indexed F1
Fig. 7.Average training time on Fashion-MNIST
5 Conclusion and Further Work


key: chroma:document
string_value:
5 Conclusion and Further Work
In this work, we introduced indexing-based evaluation of Tsetlin Machine clauses,
which demonstrated a promising increase in computation speed. Using simple
look-up tables and lists, we were able to leverage the typical sparsity of clauses.
We evaluated the indexing scheme on MNIST, IMDb and Fashion-MNIST. For
training, on average, the indexed TM produced a three-fold speedup on MNIST,
a roughly 50% speedup on Fashion-MNIST, while it slowed down IMDb training.


key: chroma:document
string_value:
Thus, the eﬀect on training speed seems to be highly dependent on the data. The
overhead from maintaining the indexing structure could counteract any speed
gains.
During inference, on the other hand, we saw a more consistent increase in
performance. We achieved a three- to eight-fold speedup on MNIST and Fashion-
MNIST, and a 13-fold speedup on IMDb, which is reasonable considering there
is no index maintenance overhead involved in classiﬁcation.


key: chroma:document
string_value:
In our further work, we intend to investigate how clause indexing can speed up
Monte Carlo tree search for board games, by exploiting the incremental changes
of the board position from parent to child node.


key: chroma:document
string_value:
Tsetlin Machines with Clause Indexing 13
1,000 2,000 5,000 10,000 20,0001101001,000
ClausesTime [s]Unindexed F4
Unindexed F3
Unindexed F2
Unindexed F1
Indexed F4
Indexed F3
Indexed F2
Indexed F1
Fig. 8.Average inference time on Fashion-MNIST
References
1.Michael Lvovitch Tsetlin. On behaviour of ﬁnite automata in random medium.
Avtomat. i Telemekh , 22(10):1345–1354, 1961.
2.Herbert Robbins. Some aspects of the sequential design of experiments. Bulletin of


key: chroma:document
string_value:
the American Mathematical Society , 1952.
3.JC Gittins. Bandit processes and dynamic allocation indices. Journal of the Royal
Statistical Society, Series B (Methodological) , 41(2):148–177, 1979.
4.K S Narendra and M A L Thathachar. Learning Automata: An Introduction .
Prentice-Hall, Inc., 1989.
5.Vegard Haugland et al. A two-armed bandit collective for hierarchical examplar
based mining of frequent itemsets with applications to intrusion detection. TCCI
XIV, 8615:1–19, 2014.


key: chroma:document
string_value:
XIV, 8615:1–19, 2014.
6.O.-C. Granmo et al. Learning Automata-based Solutions to the Nonlinear Fractional
Knapsack Problem with Applications to Optimal Resource Allocation. IEEE Trans.
on Systems, Man, and Cybernetics, Part B , 37(1):166–175, 2007.
7.Geir Thore Berge, Ole-Christoﬀer Granmo, Tor Oddbjørn Tveit, Morten Goodwin,
Lei Jiao, and Bernt Viggo Matheussen. Using the Tsetlin Machine to Learn
Human-Interpretable Rules for High-Accuracy Text Categorization with Medical


key: chroma:document
string_value:
Applications. IEEE Access , 7:115134–115146, 2019.
8.Adrian Phoulady, Ole-Christoﬀer Granmo, Saeed Rahimi Gorji, and Hady Ahmady
Phoulady. The Weighted Tsetlin Machine: Compressed Representations with Clause
Weighting. In Ninth International Workshop on Statistical Relational AI (StarAI
2020), 2020.
9.Adrian Wheeldon, Rishad Shaﬁk, Alex Yakovlev, Jonathan Edwards, Ibrahim
Haddadi, and Ole-Christoﬀer Granmo. Tsetlin Machine: A New Paradigm for


key: chroma:document
string_value:
14 S. R. Gorji et al.
Pervasive AI. In Proceedings of the SCONA Workshop at Design, Automation and
Test in Europe (DATE) , 2020.
10.K. Darshana Abeyrathna, Ole-Christoﬀer Granmo, Xuan Zhang, Lei Jiao, and
Morten Goodwin. The Regression Tsetlin Machine - A Novel Approach to Inter-
pretable Non-Linear Regression. Philosophical Transactions of the Royal Society A ,
378, 2019.
11.Ole-Christoﬀer Granmo, Sondre Glimsdal, Lei Jiao, Morten Goodwin, Christian W.


key: chroma:document
string_value:
Omlin, and Geir Thore Berge. The Convolutional Tsetlin Machine. arXiv preprint,
arXiv:1905.09688 , 2019.
12.Saeed Rahimi Gorji, Ole-Christoﬀer Granmo, Adrian Phoulady, and Morten Good-
win. A Tsetlin Machine with Multigranular Clauses. In Lecture Notes in Computer
Science: Proceedings of the Thirty-ninth International Conference on Innovative
Techniques and Applications of Artiﬁcial Intelligence (SGAI-2019) , volume 11927.
Springer International Publishing, 2019.


key: chroma:document
string_value:
Springer International Publishing, 2019.
13.Ole-Christoﬀer Granmo. The Tsetlin Machine - A Game Theoretic Bandit Driven
Approach to Optimal Pattern Recognition with Propositional Logic. arXiv preprint
arXiv:1804.01508 , 2018.
14.D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning representations by
back-propagating errors. Nature, 323:533–536, October 1986.
15.Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: a Novel Image


key: chroma:document
string_value:
THECONVOLUTIONAL TSETLIN MACHINE˚
A P REPRINT
Ole-Christoffer Granmo
CAIR, University of AgderSondre Glimsdal
CAIR, University of AgderLei Jiao
CAIR, University of Agder
Morten Goodwin
CAIR, University of AgderChristian W. Omlin
CAIR, University of AgderGeir Thore Berge
SSHF and CAIR
December 30, 2019
ABSTRACT
Convolutional neural networks (CNNs) have obtained astounding successes for important pattern


key: chroma:document
string_value:
recognition tasks, but they suffer from high computational complexity and the lack of interpretability.
The recent Tsetlin Machine (TM) attempts to address this lack by using easy-to-interpret conjunctive
clauses in propositional logic to solve complex pattern recognition problems. The TM provides
competitive accuracy in several benchmarks, while keeping the important property of interpretability.


key: chroma:document
string_value:
It further facilitates hardware-near implementation since inputs, patterns, and outputs are expressed as
bits, while recognition and learning rely on straightforward bit manipulation. In this paper, we exploit
the TM paradigm by introducing the Convolutional Tsetlin Machine (CTM), as an interpretable
alternative to CNNs. Whereas the TM categorizes an image by employing each clause once to the
whole image, the CTM uses each clause as a convolution ﬁlter. That is, a clause is evaluated multiple


key: chroma:document
string_value:
times, once per image patch taking part in the convolution. To make the clauses location-aware,
each patch is further augmented with its coordinates within the image. The output of a convolution
clause is obtained simply by ORing the outcome of evaluating the clause on each patch. In the
learning phase of the TM, clauses that evaluate to 1are contrasted against the input. For the CTM,
we instead contrast against one of the patches, randomly selected among the patches that made the


key: chroma:document
string_value:
clause evaluate to 1. Accordingly, the standard Type I and Type II feedback of the classic TM can be
employed directly, without further modiﬁcation. The CTM obtains a peak test accuracy of 99.4%
on MNIST, 96.31% on Kuzushiji-MNIST, 91.5% on Fashion-MNIST, and 100.0% on the 2D Noisy
XOR Problem, which is competitive with results reported for simple 4-layer CNNs, BinaryConnect,
Logistic Circuits, and a recent FPGA-accelerated Binary CNN.
1 Introduction


key: chroma:document
string_value:
1 Introduction
The Tsetlin Machine (TM) [ 12] is a novel machine learning paradigm introduced in 2018. It is based on the Tsetlin
Automaton (TA) [ 36], one of the pioneering solutions to the well-known multi-armed bandit problem [ 33,11] and the
ﬁrst Finite State Learning Automaton (FSLA) [ 25]. The TM has the following main properties that make it attractive
as a building block for machine learning [ 12]: (a) it solves complex pattern recognition problems with interpretable


key: chroma:document
string_value:
propositional formulae. This is crucial for high stakes decisions [ 34]. (b) It learns on-line, persistently increasing both
training- and test accuracy, before converging to a Nash equilibrium. The Nash equilibrium balances false positive
against false negative classiﬁcations, while combating overﬁtting with frequent pattern mining principles. (c) Resource
allocation dynamics are leveraged to optimize usage of limited pattern representation resources. By allocating resources


key: chroma:document
string_value:
uniformly across sub-patterns, local optima are avoided. (d) Inputs, patterns, and outputs are expressed as bits, while
recognition and learning rely on straightforward bit manipulation. This facilitates low-energy consuming hardware
design [ 40]. (e) Finally, the TM has provided competitive accuracy in comparison with classical and neural network
based techniques, while keeping the important property of interpretability [3, 1].


key: chroma:document
string_value:
˚Source code and demos for this paper can be found at https://github.com/cair/pyTsetlinMachineParallel.arXiv:1905.09688v5  [cs.LG]  27 Dec 2019


key: chroma:document
string_value:
APREPRINT - DECEMBER 30, 2019
The TM currently is state-of-the-art in FSLA-based pattern recognition. However, when compared with CNNs, it
struggles with attaining competitive accuracy, providing e.g. 98.5% mean accuracy on MNIST (without augmenting the
data) [ 12]. To address this deﬁciency, we here introduce the Convolutional Tsetlin Machine (CTM), a new kind of TM
designed for image classiﬁcation.


key: chroma:document
string_value:
designed for image classiﬁcation.
FSLA. The simple Tsetlin Automaton approach has formed the core of more advanced FSLA designs that solve a
wide range of problems. This includes resource allocation [ 15], decentralized control [ 37], knapsack problems [ 16],
searching on the line [ 27], meta-learning [ 28], the satisﬁability problem [ 13], graph colouring [ 4], preference learning


key: chroma:document
string_value:
[42], frequent itemset mining [ 17], adaptive sampling [ 14], spatio-temporal event detection [ 43], equi-partitioning [ 29],
streaming sampling for social activity networks [ 10], routing bandwidth-guaranteed paths [ 26], faulty dichotomous
search [ 44], and learning in deceptive environments [ 46], to list a few examples. The unique strength of all of these
FSLA designs is that they provide state-of-the-art performance when problem properties are unknown and stochastic,


key: chroma:document
string_value:
and the problem must be solved as quickly as possible through trial and error.
Rule-based Machine Learning. While the present paper focuses on extending the ﬁeld of FSLA, we acknowledge
the extensive work on rule-based interpretable pattern recognition from other ﬁelds of machine learning. Learning
propositional formulae to represent patterns in data has a long history. One prominent example is frequent itemset


key: chroma:document
string_value:
mining for association rule learning [ 2], for instance applied to predicting sequential events [ 35,24]. Other examples
include the work of Feldman who investigated the hardness of learning formulae in disjunctive normal form (DNF)
[9]. Furthermore, Probably Approximately Correct (PAC) learning has provided fundamental insight into machine
learning, as well as a framework for learning formulae in DNF [ 38]. Approximate Bayesian approaches have recently


key: chroma:document
string_value:
been introduced to provide more robust learning of rules [ 39,18]. However, in general, rule-based machine learning
scales poorly and is prone to noise. Indeed, for data-rich problems, in particular those involving natural language and
sensory inputs, state-of-the-art rule-based machine learning is inferior to deep learning. The recent hybrid Logistic
Circuits [ 22], however, have had success in image classiﬁcation. This approach uses local search to build Bayesian


key: chroma:document
string_value:
models that capture logical expressions, and learns to classify by employing stochastic gradient descent. The CTM, on
the other hand, attempts to bridge the gap between the interpretability of pure rule-based machine learning and the
accuracy of deep learning, by allowing the TM to more effectively deal with images.
CNNs. A myriad of image recognition techniques have been reported. However, after AlexNet won the ImageNet


key: chroma:document
string_value:
recognition challenge by a signiﬁcant margin in 2012, the entire ﬁeld of computer vision has been dominated by
CNNs. The AlexNet architecture was built upon earlier work by LeCun et al. [ 21] who introduced CNNs in 1998.
Countless CNN architectures, all following the same basic principles, have since been published, including the now
state-of-the-art Squeeze-and-Excitation networks [ 19]. In this paper, we introduce convolution to the TM paradigm


key: chroma:document
string_value:
of machine learning. By doing so, we simultaneously propose a new kind of convolution ﬁlter: an interpretable ﬁlter
expressed as a propositional formula. Our intent is to address a well-known disadvantage of CNNs, namely, that the
CNN models in general are complex and non-transparent, making them hard to interpret. Consequently, the knowledge
on why CNNs perform so well and what steps are need to improve the models is limited [45].


key: chroma:document
string_value:
Binary CNNs. Relying on a large number of multiply-accumulate operations, training CNNs is computationally
intensive. To mitigate this, there is an increasing interest in binarizing the CNNs. With only two possible values for the
synapse weights, e.g., ´1and1, many of the multiplication operations can be replaced with simple accumulations.
This could potentially open up for more specialized hardware and more compressed and efﬁcient models. One recent


key: chroma:document
string_value:
approach is BinaryConnect [ 6], which reached a near state-of-the-art accuracy of 98.99% on MNIST, and 99.13% in
combination with an SVM. Binary CNNs have been further improved with the introduction of XNOR-Nets, which
replace the standard CNN ﬁlters with binary equivalents [ 32]. BNN+ [ 7] is the most recent binary CNN, extending the
XNOR-Nets with two additional regularization functions and an adaptive scaling factor. The CTM can be seen as an


key: chroma:document
string_value:
extreme Binary CNN in the sense that it is entirely based on fast summation and logical operators. Additionally, learning
in the CTM is bandit based (online learning from reinforcement), while Binary CNNs are based on backpropagation.
Contributions and Paper Outline. Our contributions can be summarized as follows. First, in Sect. 2, we provide a
brief introduction to the TM and a succinct deﬁnition of both recognition and learning. Then, in Sect. 3, we introduce


key: chroma:document
string_value:
the concept of convolution to TMs. Whereas the classic TM categorizes an image by employing each clause once
on the whole image, the CTM uses each clause as a convolution ﬁlter. In all brevity, we propose a recognition and
learning approach for clause-based convolution that produces interpretable ﬁlters. In Sect. 4, we evaluate the CTM
on MNIST, Kuzushiji-MNIST, Fashion-MNIST, and the 2D Noisy XOR Problem, and discuss the merits of the new


key: chroma:document
string_value:
APREPRINT - DECEMBER 30, 2019
IncludeExcludex2x1¬x2¬x1⌵⌵⌵x2x1¬x2¬x1⌵⌵⌵Clause +Clause —
X = (x1, x2)Majority votingFeedback Type I or IIClause outputClause output
Input feature vectorDecided classFeedback Type I or II+—
123456
123456
123456
123456
123456
123456
123456
123456
Figure 1: The basic TM structure.
2 The Tsetlin Machine
The TM solves complex pattern recognition problems using conjunctive clauses in propositional logic, composed by a


key: chroma:document
string_value:
collective of TAs. Its roots in game theory, the bandit problem, resource allocation, and frequent pattern mining are
explored in depth in the original paper on the TM, which also includes pseudo code [ 12]. We here provide a more
succinct overview, together with a running example to clarify core concepts.
2.1 Structure
The structure of the TM is shown in Figure 1. As seen, the TM takes a vector Xofopropositional variables as input:


key: chroma:document
string_value:
X“pxkqPt0,1uo, with obeing the size of the vector and t0,1uoits domain. The input vector in the ﬁgure, for
instance, is of size 2and consists of the variables x1andx2with domaintp0,0q,p0,1q,p1,0q,p1,1qu.
The TM formulates patterns using conjunctive clauses . A conjunctive clause is built from literals , that is, the input
variables and their negations. With oinput variables, we have 2oliterals, L“plkqPt0,1u2o:
lk“"xk if1ďkďo,
␣xk´ootherwisepi.e.,o`1ďkď2oq.(1)


key: chroma:document
string_value:
lk“"xk if1ďkďo,
␣xk´ootherwisepi.e.,o`1ďkď2oq.(1)
Above, the ﬁrst oliterals are the unnegated variables, while the ofollowing are the negated ones. Thus, in our example,
there are four literals: x1, x2,␣x1, and␣x2.
The number of clauses is a user set parameter that decides the expression power of the TM. Assume we have mclauses
in our TM structure. Each clause, denoted by Cjand indexed by jPt1, . . . , mu, is simply a conjunction of a subset
IjĎt1, . . . , 2ouof the literals:
CjpXq“ľ
kPIjlk (2)


key: chroma:document
string_value:
APREPRINT - DECEMBER 30, 2019
or equivalently:
CjpXq“ź
kPIjlk. (3)
The subset Ijcontains the indexes of the literals that have been Included in the clause j. For the special case of Ij“H ,
i.e., an empty clause, we have:
CjpXq“"
1during learning
0otherwise .(4)
That is, during learning, empty clauses output 1and during classiﬁcation they output 0. Accordingly, overall, the
output cj“CjpXqPt0,1uof a clause is fully speciﬁed by the input Xand the selection of literals Ij. Continuing


key: chroma:document
string_value:
our example, in the ﬁgure we have two clauses ( m“2). The ﬁrst consists of the literals with indexes I1“t1,4u:
C1pXq“x1^␣x2. The second consists of the literals with indexes I2“t1,2u:C2pXq“x1^x2.
Each clause is further assigned a ﬁxed polarity pPt´1,`1u, which decides whether the clause output is negative or
positive. In the ﬁgure, the ﬁrst clause has positive polarity, whereas the second has negative. Positive clauses are used to


key: chroma:document
string_value:
recognize class y“1, while negative clauses are used to recognize class y“0. By default, half of the clauses are
positive and half are negative.
In the last step, the outputs of the clauses decide the class ˆyPt0,1uassigned to the input X. That is, a summation
operator aggregates the output of the clauses to form a majority vote.
2.2 Recognition
We now look at recognition in more detail. First of all, the output cp
jof each clause is organized as vectors C`“pc`
jqP
t0,1um
2andC´“pc´
jqPt0,1um


key: chroma:document
string_value:
jqP
t0,1um
2andC´“pc´
jqPt0,1um
2, grouped by polarity. Given an input X, these outputs can then be calculated as
follows (equivalent to Eqn. 2):
cp
j“"1ifIp
jĎI1
L,
0otherwise .(5)
Here, the subset I1
LĎt1, . . . , 2oucontains the indexes of the literals that take the value 1for the current input X:
I1
L“tk|lk“1,1ďkď2ou. To express this, the lower index Lstands for the literals and the upper index 1constrains


key: chroma:document
string_value:
these to those of value 1. Thus, Eqn. 5 compactly states that a clause outputs 1if and only if all of its literals are of
value 1. In Figure 1, for instance, the input px1, x2q“p1,0qproduces the literal values l1“1,l2“0,l3“0, and
l4“1. Since the ﬁrst clause only consists of l1andl4, it outputs 1. The second clause, on the other hand, consists of l1
andl2, hence it outputs 0.
The ﬁnal classiﬁcation decision is made as follows. A majority vote is ﬁrst organized by summing the output C`of


key: chroma:document
string_value:
the positive clauses, and subtracting the output C´of the negative clauses: v“ř
jc`
j´ř
jc´
j. This majority vote
decides the prediction of the TM: ˆy“p0ďv), with a tie resolved in favor of y“1.2So, in our example, the input
px1, x2q“p1,0qgathers a single positive vote, and no negative votes. Consequently, the classiﬁcation becomes ˆy“1.
2.3 Tsetlin Automata Collective
We now introduce the collective of TAs, which decides the composition of each clause. There are 2oTAs per clause j,


key: chroma:document
string_value:
one for each literal lkPtx1, . . . , x o,␣x1, . . . ,␣xou. Each individual TA has a state a´
j,k, a`
j,kPt1, . . . , 2Nu, and the
states of all of the TAs are organized in twom
2ˆomatrices. The ﬁrst matrix, A`“pa`
j,kqPt1, . . . , 2Num
2ˆ2o, contains
the states of the TAs belonging to the positive polarity clauses, whereas the matrix A´“pa´
j,kqPt1, . . . , 2Num
2ˆ2o
covers the negative polarity clauses. In Figure 1, there are eight TAs, one per literal for clause 1(positive polarity) and


key: chroma:document
string_value:
one per literal for clause 2(negative polarity). Each of these TAs has 6states: ap
j,kPt1, . . . , 6u.
The states decide the actions taken by the TAs. A TA decides to exclude its literal if in states ap
j,kďN, otherwise,
it includes the literal. That is, for each clause j, some literals are Included :Ip
j“!
k|ap
j,kąN,1ďkď2o)
, while
the remaining are Excluded :Ip
jc“t1, . . . , 2ouzIp
j(the complement of the included ones). So, in the ﬁgure, the TA


key: chroma:document
string_value:
associated with literal x1in the ﬁrst clause is in state 4and thus x1is included in that clause. The second TA, on the
other hand, is in state 3and, accordingly, literal x2is excluded. By checking the state of each TA in this manner, the
composition of all of the clauses is decided.
2Multi-class pattern recognition problems are modelled by employing multiple instances of this structure, replacing the threshold
operator with an argmax operator [12].
4


key: chroma:document
string_value:
APREPRINT - DECEMBER 30, 2019
2.4 Learning
Learning in the TM is based on coordinating the collective of TAs using a novel FSLA game. The game leverages
resource-allocation [ 16] and frequent pattern mining [ 17] principles, indicated by the feedback loop in Figure 1. The
feedback is handed out based on training examples pX, yq, consisting of an input Xand an output y.
As explored below, the TM employs two kinds of feedback: Type I and Type II. Type I feedback jointly combats


key: chroma:document
string_value:
false negatives and overﬁtting by stimulating recognition of frequent patterns. Type II feedback, on the other hand,
suppresses false positives by increasing the discrimination power of the patterns learnt.
Feedback is further regulated by the sum of clause outputs v(the majority vote) and a target value TPZąset for vby
the user. A larger T(with a corresponding increase in the number of clauses) makes the learning more robust. This is


key: chroma:document
string_value:
because more clauses are involved in learning each speciﬁc pattern, introducing an ensemble effect. However, note that
the resulting gain in accuracy comes at the cost of increased computational cost (cf. Table 3).
Figure 2: Clause feedback probability for T“4.
In the following, we ﬁrst deﬁne the Type I and Type II feedback operations for clauses with positive polarity. As
opposed to the feedback tables in [12], the representation here is in a more compact matrix form.


key: chroma:document
string_value:
Type I: For training output y“1, TAs belonging to positive clauses are given Type I feedback to make vapproach T.
A vector R`
1“pr`
j,1qPt0,1um
2picks out the positive clauses selected for feedback (the lower index of R`
1
refers to the output y“1):
r`
j,1“#
1with probabilityT´clamppv,´T,Tq
2T,
0otherwise .(6)
Here, the clamp operation restricts vto lie between ´TandT. Further, the lower index of r`
j,1refers to the


key: chroma:document
string_value:
j,1refers to the
clause jand the output y“1. As illustrated by plot y“1in Figure 2, clauses are randomly selected for
feedback, with a larger chance of being selected for lower values of v. In effect, clauses are updated more
aggressively the farther away the voting sum is from T, up to´T.
We now divide the Type I feedback into two parts, Type Ia and Type Ib. Type Ia reinforces Include actions
to make the patterns ﬁner. Only TAs taking part in clauses that output 1and whose associated literal


key: chroma:document
string_value:
takes the value 1are select for Type Ia feedback. We collect the indexes of these TAs in the set IIa
C`“␣
pj, kq|lk“1^c`
j“1^r`
j,1“1(
.
Type Ib feedback is designed to reinforce Exclude actions to combat over-ﬁtting. Type Ib feedback is handed
out to the TAs stochastically, using a user set parameter sě1.0(a larger sprovides ﬁner patterns). The
5


key: chroma:document
string_value:
APREPRINT - DECEMBER 30, 2019
stochastic part of the calculation is organized in a matrix Q`“pq`
j,kqPt0,1um
2ˆ2owith entries:
q`
j,k“"1with probability1
s,
0otherwise .(7)
Here, the lower index of q`
j,krefers to the clause jand the TA k, considering the positive clauses (up-
per index). The indexes of the TAs selected for Type Ib feedback are collected in the set IIb
C`“!
pj, kq|plk“0_c`
j“0q^r`
j,1“1^q`
j,k“1)
. That is, TAs taking part in clauses that output 0, or


key: chroma:document
string_value:
whose associated literal takes the value 0, are selected for Type Ib feedback, however, only if stochastically
pinpointed by r`
j,1“1andq`
j,k“1.
After the TAs has been selected for feedback, their states are updated by two state update operators ‘anda:
A`Ð`
A`‘IIa
C`˘
aIIb
C`These operators add/subtract 1from the states of the singled out TAs, however,
not beyond the given state space. In our example, if the TA associated with literal x1in the ﬁrst clause is


key: chroma:document
string_value:
singled out for Type Ia feedback, it would move from state 4to state 5, reinforcing the Include action. If it
instead receives Type Ib feedback, it would move to state 3, reinforcing Exclude .
Type II: For training output y“0, TAs belonging to positive clauses are given Type II feedback to suppress clause
output 1. This, together with the negative clauses, makes vapproach´T. A matrix R`
0“pr`
j,0qPt0,1um
2
picks out the positive clauses selected for feedback:
r`
j,0“#


key: chroma:document
string_value:
r`
j,0“#
1with probabilityT`clamppv,´T,Tq
2T,
0otherwise .(8)
The lower index of r`
j,0refers to the clause jand the output y“0, respectively. Here too, clauses are randomly
selected for feedback, but now with a larger chance of being selected for higher v(cf. plot y“0in Figure
2). Next, the TAs selected are the ones that will turn clauses that output 1into clauses that output 0, that is,
those that have excluded literals that take the value 0:III
C`“␣
pj, kq|lk“0^c`
j“1^r`
j,0“1(
. Again,


key: chroma:document
string_value:
C`“␣
pj, kq|lk“0^c`
j“1^r`
j,0“1(
. Again,
the states of the selected TAs are updated using the dedicated operators: A`ÐA`‘III
C`. Thus, if the TA
associated with literal x2in the ﬁrst clause of our example is selected for Type II feedback, it would change
state from 3to4, reinforcing the Include action.
All of the above operations are for positive clauses. For negative clauses, Type I feedback is simply replaced with Type
II feedback and vice versa!


key: chroma:document
string_value:
II feedback and vice versa!
1010100113 x 3 input image4 positive clauses4 negative clauses1
∑+1
xy+10000 < x10y <= 0010 < x11y <= 0000 < x00y <= 0110 < x01y <= 0100 < x10y <= 0010 < x10y <= 001000 < x11y <= 0000 < x00y <= 01100
(a)
1010100113 x 3 input image4 positive clauses4 negative clauses∑P(Feedback) = (T-1)/(2T) = 0.25
xyType I Feedback0 < x10y <= 0010 < x11y <= 0000 < x00y <= 0110 < x01y <= 0100 < x10y <= 0010 < x10y <= 0010 < x11y <= 0000 < x00y <= 011Type II Feedback+10000000+1 (b)


key: chroma:document
string_value:
APREPRINT - DECEMBER 30, 2019
0 < x01y <= 0101010100113 x 3 input image4 positive clauses4 negative clauses1
∑+2
xy+10000 < x10y <= 0010 < x11y <= 0000 < x00y <= 0110 < x01y <= 0100 < x10y <= 001000 < x11y <= 0000 < x00y <= 0110+1
(a)
5/20/2019 m.html
1/13
3 (b)
Figure 4: (a) Goal state for the Noisy 2D XOR Problem. (b) Illustration of image, ﬁlter and patches.
2.5 Integer Clause Weighting


key: chroma:document
string_value:
2.5 Integer Clause Weighting
It turns out that introducing clause weighting improves TM performance, both accuracy- and computation-wise [ 31].
By weighting the output of the clauses, one clause can replace multiple and the impact of a clause can be ﬁne-tuned. In
all brevity, classiﬁcation is then performed as a weighted majority vote:
s1pxq“m{2ÿ
j“1w`
jC`
jpxq´m{2ÿ
j“1w´
jC´
jpxq. (9)


key: chroma:document
string_value:
j“1w`
jC`
jpxq´m{2ÿ
j“1w´
jC´
jpxq. (9)
Whereas real-valued weighting was introduced in [ 31], we here address learning of integer weights, based on Stochastic
Searching on the Line [27]:
Initialization. First of all, the weights are initialized to 1:
w`
jÐ1, (10)
w´
jÐ1. (11)
In other words, the behaviour of the clauses are identical to those of a standard TM at the start of learning.
Weight Updating. The TA state updating is unmodiﬁed with weights. The weights, in turn, are updated based on


key: chroma:document
string_value:
Type I and Type II feedback.
Type I: For Type I feedback, a clause weight is increased by 1, however, only if the clause outputted 1:
w`
jÐw`
j`1,ifC`
jpxq“1, (12)
w´
jÐw´
j`1,ifC´
jpxq“1. (13)
Type II: For Type II feedback, we instead update the weights by subtracting 1, with 1being the minimum weight value:
w`
jÐw`
j´1,ifC`
jpxq“1^w`
ją1, (14)
w´
jÐw´
j´1,ifC´
jpxq“1^w´
ją1. (15)
When a clause evaluates to 0, its weight remains unchanged:
w`
jÐw`
j,ifC`
jpxq“0, (16)
w´
jÐw´
j,ifC´


key: chroma:document
string_value:
APREPRINT - DECEMBER 30, 2019
As seen, for Type I feedback, the weights are increased to strengthen the impact of the associated clauses, thus
reinforcing true positive frequent patterns. For Type II feedback, on the other hand, the weights are decreased instead.
This is to diminish the impact of the associated clauses, thus combating false positives, increasing discrimination power.
3 The Convolutional Tsetlin Machine


key: chroma:document
string_value:
3 The Convolutional Tsetlin Machine
Consider a set of images X“tXe|1ďeďEu, where eis the index of the images. Each image is of size XˆY, and
consists of Zbinary layers (which together encode the pixel colors using thresholding [ 8], one-hot encoding, or any
other appropriate binary encoding). A classic TM models such an image with an input vector X“pxkqPt0,1uXˆYˆZ
that contains XˆYˆZpropositional variables. Further, each clause is composed from XˆYˆZˆ2literals.


key: chroma:document
string_value:
Inspired by the impact convolution has had on deep neural networks, we here introduce the Convolutional Tsetlin
Machine (CTM).
Interpretable Rule-based Filters. The CTM uses ﬁlters with spatial dimensions WˆW, again with Zbinary layers.
Further, the clauses of the CTM take the role of ﬁlters. Each clause is accordingly composed from WˆWˆZˆ2
literals. Additionally, to make the clauses location-aware [ 23], we augment each clause with binary encoded coordinates.


key: chroma:document
string_value:
Location awareness may prove useful in applications where both patterns and their location are distinguishing features,
e.g. recognition of facial features such as eyes, eyebrows, nose, mouth, etc. in facial expression recognition. In
all brevity, when applying a ﬁlter of size WˆWon an imagepxkqPt0,1uXˆYˆZ, the ﬁlter will be evaluated on
B“BXˆBYimage patches. Here, BX“PX´W
dT
`1andBY“PY´W
dT
`1, with dbeing the step size of the


key: chroma:document
string_value:
dT
`1, with dbeing the step size of the
convolution. Each image patch thus has a certain location within the image, and we augment the input vector with the
coordinates of this location. We denote the resulting augmented input vector Xb:Xb“pxkqPt0,1uWˆWˆZ`BX`BY.
As seen, the input vector is extended with one propositional variable per position along each dimension, with the
position being encoded using thresholding [ 8] or one-hot encoding. Figure 4b illustrates an example of the image,


key: chroma:document
string_value:
patches, and a ﬁlter for X“Y“6,W“3, andd“2. In this example, the 3 ˆ3 ﬁlter moves from left to right, from
top to bottom, 2pixels per step.
Recognition. The CTM uses the classic TM procedure for recognition (see Sect. 2). However, for the CTM each clause
outputs Bvalues per image (one value per patch), as opposed to a single output for the TM (Eq. 5). We denote the
output of a positive clause jon patch bbycb,`
j. To turn the multiple outputs c1,`
j, . . . , cB,`
jof clause jinto a single


key: chroma:document
string_value:
j, . . . , cB,`
jof clause jinto a single
output denoted by c`
j, we simply OR the individual outputs:
c`
j“Bł
b“1cb,`
j. (18)
Learning. Learning in the CTM leverages the TM learning procedure. As seen in Sect. 2, Type Ia, Type Ib, and Type II
feedback are inﬂuencing each clause jbased on the literals of the input vector X. For the CTM, the input vector is an
image patch, and there are Bpatches in an image. There is thus Bliteral inputs Lb,1ďbďB, per clause. Therefore,


key: chroma:document
string_value:
to decide which patch to use when updating a clause, the CTM randomly selects a single patch among the patches that
made the clause evaluate to 1. The clause is then updated according to this patch. That is, the input Xbis drawn from
the set:tXb|cb,`
j“1,1ďbďBu. Observe that if the set is empty, only Type Ib feedback is applicable, and then
the input vector is not needed. For non-empty sets, the TAs to be updated are ﬁnally singled out using the randomly
selected patch:
IIa
C`“␣
pj, kq|lb


key: chroma:document
string_value:
selected patch:
IIa
C`“␣
pj, kq|lb
k“1^c`
j“1^r`
j,1“1(
, (19)
IIb
C`“!
pj, kq|plb
k“0_c`
j“0q^r`
j,1“1^q`
j,k“1)
, (20)
III
C`“␣
pj, kq|lb
k“0^c`
j“1^r`
j,0“1(
. (21)
The reason for randomly selecting a patch is to have each clause extract a certain sub-pattern, and the randomness of
the uniform distribution statistically spreads the clauses for different sub-patterns in the target image. Finally, observe


key: chroma:document
string_value:
that the computational complexity of the CTM grows linearly with the number of clauses m, and with the number of
patches B. However, the computations can be easily parallelized due to their decentralized nature.
Step-by-step Walk-through of Inference on Noisy 2D XOR. Rather than providing hand-crafted features which can
be used for image classiﬁcation, the CTM learns feature detectors. We will explain the workings of the CTM by an


key: chroma:document
string_value:
illustrative example of noisy 2D XOR recognition and learning (see Figure 4 and Sect. 4). Consider the CTM depicted
in Figure 3a. It consists of four positive clauses which represent XOR patterns that must be present in a positive
example image (positive features) and four negative clauses which represent patterns that will not trigger a positive
8


key: chroma:document
string_value:
APREPRINT - DECEMBER 30, 2019
image classiﬁcation (negative features). The number of positive and negative clauses is a user-deﬁned parameter. The
bit patterns inside each clause are represented by the output of eight TAs, two for each bit in a 2ˆ2ﬁlter.
Consider the 3ˆ3image shown in Figure 3b. The ﬁlter represented by the second positive clause matches the patch in
the top-right corner of the image and it is the only clause with output 1; similarly, none of the negative clauses respond


key: chroma:document
string_value:
since their patterns do not match the pattern found in the current patch (Figure 3b). Thus, the TM’s combined output is
v“1. Learning of feature detectors proceeds as follows: with the CTM’s voting target set to T“2, the probability of
feedback isT´v
2T“1
4, and thus learning takes place, which pushes the CTM’s output vtowards T“2. Note that Type
I feedback reinforces true positive output and reduces false negative output whereas Type II feedback reduces false
positive output.


key: chroma:document
string_value:
positive output.
A subsequent state of the CTM is shown in Figure 4a. There are now two positive clauses which detect their pattern
in the top-right corner patch. The combined output of all clauses is 2; thus, no further learning is necessary for the
detection of the XOR pattern in this patch. Also, the location of the occurrence of each pattern is included. The location
information uses a bit representation as follows: Suppose an XOR pattern occurs at the three X-coordinates 1,4, and6.


key: chroma:document
string_value:
For the corresponding binary location representation, these coordinates are considered as thresholds: If a coordinate is
greater than a threshold, then the corresponding bit in the binary representation will be 0; otherwise, it is set to 1. Thus,
the representation of the X-coordinates 1,4, and6will be “111", “011" and “001", respectively. These representations
of the location of 2ˆ2patterns are also learned by TAs.
Search Range 2D Noisy XOR MNIST K-MNIST Fashion-MNIST


key: chroma:document
string_value:
#Class Clauses 1´8 000 40 8 000 8 000 8 000
T 1´20 000 60 10 000 10 000 10 000
s 1.0´20.0 3.9 5.0 10.0 10.0
W 2´20 2 10 10 10ˇˇZˇˇ 1 1 1 1 1
Clause Weighting Yes/No No Yes Yes Yes
Table 1: CTM conﬁgurations.
4 Empirical Results
In this section, we evaluate the CTM on four different datasets.
2D Noisy XOR. The 2D Noisy XOR dataset contains 4ˆ4binary images, 2500 training examples and 10 000 test


key: chroma:document
string_value:
examples. The image bits have been set randomly, except for the 2ˆ2patch in the upper right corner, which reveals
the class of the image. A diagonal line is associated with class 1, while a horizontal or vertical line is associated with
class0. Thus the dataset models a 2D version of the XOR-relation. Furthermore, the dataset contains a large number of
random non-informative features to measure susceptibility towards the curse of dimensionality. To examine robustness


key: chroma:document
string_value:
towards noise we have further randomly inverted 40% of the outputs in the training data.
MNIST. The MNIST dataset has been used extensively to benchmark machine learning algorithms, consisting of
28ˆ28grey scale images of handwritten digits [21].
Kuzushiji-MNIST. This dataset contains 28ˆ28grayscale images of Kuzushiji characters, cursive Japanese. Kuzushiji-
MNIST is more challenging than MNIST because there are multiple distinct ways to write some of the characters
[5].


key: chroma:document
string_value:
[5].
Fashion-MNIST. This dataset contains 28ˆ28grayscale images of articles from the Zalando catalogue, such as
t-shirts, sandals, and pullovers [41]. This dataset is quite challenging, with a human accuracy of 83.50%.
The latter three datasets contain 60 000 training examples and 10 000 test examples. We binarize these datasets using
an adaptive Gaussian thresholding procedure with window size 11and threshold value 2. Accordingly, the CTM


key: chroma:document
string_value:
operates on images with merely 1 bit per pixel. Table 4 reports test accuracy for the CTM, while Table 1 contains the
corresponding conﬁgurations. The results are based on single-run averages, obtained from the last 100epochs of 250.
All of the experiments were run on a NVIDIA DGX-2 server. Note that these datasets are rather simple to solve for
deep CNNs, and have been selected to facilitate our study of interpretability.


key: chroma:document
string_value:
The hyperparameters were set using a light manual binary search. One particularly critical hyperparameter is the
number of clauses employed per class. Table 3 reports execution time per epoch as well as mean test accuracy for an
increasing number of class clauses. As seen, test accuracy increases steadily with the number of clauses employed,
while execution time increases sub-linearly due to parallelization.
9


key: chroma:document
string_value:
APREPRINT - DECEMBER 30, 2019
0: U L 0: U R 9: U L 9: U R 4: L L
00*000*000
*000*0*000
**00*00000
00000000**
00*000****
000000***1
000**0*111
0*00**111*
*****111*0
00**11**00
0ăXď6
5ăYď80*********
0******0**
********00
*111111**0
*111*11**0
*1*1*11***
111**11**0
1****111**
11***111**
*****111**
8ăXď17
0ăYď60*0*****1*
00000**1**
000*******
00***1**00
000**1**00
00**1*00*0
00****00**
0*******11
0*********
000*******
2ăXď7
5ăYď100**0*****0
*00*0*****
**0*******
*******0**
*11****0*0
****0**000


key: chroma:document
string_value:
**0*******
*******0**
*11****0*0
****0**000
****1**00*
****1***0*
****11**00
********0*
12ăXď18
2ăYď8******1***
*11*1*1***
***111****
*****11***
*000**1***
000**1****
0000*1*0**
0*00*1**0*
0**0***0**
0000****0*
7ăXď11
12ăYď17
0: L L 0: L R 9: L L 9: L R 3: U R
*000**1***
**********
**0******0
*00****110
*********0
*0*0***1*0
******11**
***0******
**000**1*1
0*0****11*
0ăXď5
10ăYď16*******1**
******1***
******1***
*****1**1*
****1*1***
****11****
*1*11*1***
1**11*****
*11*11****
*1*1******


key: chroma:document
string_value:
*1*11*1***
1**11*****
*11*11****
*1*1******
4ăXď18
13ăYď180000000***
00000001**
0000000110
0000000*1*
000000*0*1
00000000**
0000000000
0000*00000
0000000000
0000000000
0ăXď3
8ăYď14******00**
*****000*0
**11*0000*
**11*00000
1*11000*00
11**0000*0
111****00*
1**0000000
1**00*00**
1*0**00*00
12ăXď16
9ăYď15**0000*000
00000*0000
***0000000
*11**00000
*1*1**0000
0**1*00*00
0*1**0***0
**1*0000*0
*1***0*000
***0000000
12ăXď18
0ăYď3


key: chroma:document
string_value:
**1*0000*0
*1***0*000
***0000000
12ăXď18
0ăYď3
Table 2: Example 10 ˆ10 bit patterns produced by CTM for MNIST, including valid convolution positions. Here “0:
UL" means “upper left" of the image for digit “0". We can see clearly that “0: UL", “0: UR", “0: LL" and “0: LR"
jointly construct the shape “0”. The clauses for the other digits behave similarly, and we thus just illustrate selected
patch positions for each digit.
#Clauses per class 250 500 1000 2000 4000 8000


key: chroma:document
string_value:
#Clauses per class 250 500 1000 2000 4000 8000
MNIST accuracy (%) 98.82 98.98 99.14 99.22 99.28 99.33
MNIST execution time (s) 15.6 15.8 18.9 20.4 32.4 39.0
K-MNIST accuracy (%) 92.75 93.86 94.89 95.40 95.85 96.08
K-MNIST execution time (s) 15.5 17.2 19.7 21.1 33.5 40.5
F-MNIST accuracy (%) 88.25 88.79 89.42 89.89 90.65 91.18
F-MNIST execution time (s) 16.9 17.2 17.8 25.0 27.8 52.2
Table 3: CTM mean test accuracy and execution time per epoch for an increasing number of clauses.


key: chroma:document
string_value:
The CTM performs rather robustly, providing tight 95% conﬁdence intervals for the mean performance. As seen in
Table 4, we have also included results for selected popular algorithms, as points of reference. Note that the CTM is
an interpretable approach that does not employ multiple layers at this stage. Therefore, we believe a fair comparison
should target traditional/simple CNNs. Results listed in italic are reported in the corresponding papers. Results for


key: chroma:document
string_value:
BinaryConnect and FPGA-accelerated BNNs on K-MNIST and Fashion-MNIST were not available, so are not reported.
Notice that the CTM outperforms the binary CNNs for all the datasets, as well several simple baseline conﬁgurations,
including 4-layer CNNs, Random Forests, Gradient Boosting, SVMs, and a 4-nearest neighbour classiﬁer. Further
observe that both the TM and the CTM obtain higher accuracy than Logistic Circuits [ 22] with binary data. Thus,


key: chroma:document
string_value:
in our experiments, the TM learning paradigm outperforms the probabilistic circuits paradigm [ 22,30] at extracting
information from binary representations. However, only the CTM performs competitively with Logistic Circuits
when the Logistic Circuits are given the advantage of real-valued inputs. Note that the CTM supports using multiple
bits to encode the color of each pixel, however, in the present paper, we only investigate single bit representations.


key: chroma:document
string_value:
Finally, the CTM is outperformed by the more advanced deep learning network architectures PreActResNet-18 and
ResNET18+VGG.
Figure 5 depicts test accuracy for the CTM on MNIST, K-MNIST, and F-MNIST, epoch-by-epoch. As seen, test
accuracy climbs quickly in the ﬁrst epochs, e.g., passing 99% already in epoch 8for MNIST. MNIST test accuracy
10


key: chroma:document
string_value:
APREPRINT - DECEMBER 30, 2019
Model 2D N-XOR MNIST K-MNIST F-MNIST
4-Nearest Neighbour [5, 41] 61.62 97.14 91.56 85.40
SVM [5] 94.63 98.57 92.82 89.7
Random Forest [22] 70.73 97.3 - 81.6
Gradient Boosting Classiﬁer [41] 87.15 96.9 - 88.0
Simple CNN [5, 22] 91.05 99.06 95.12 90.7
BinaryConnect [6] - 98.99 - -
FPGA-accelerated BNN [20] - 98.70 - -
Logistic Circuit (binary) [22] - 97.4 - 87.6
Logistic Circuit (real-valued) [22] - 99.4 - 91.3
PreActResNet-18 [5] - 99.56 97.82 92.00


key: chroma:document
string_value:
PreActResNet-18 [5] - 99.56 97.82 92.00
ResNet18 + VGG Ensemble [5] - 99.60 98.90 -
TM 99.12 98.57 92.03 90.09
CTM (Mean) 99.99˘0.099.33˘0.096.08˘0.01 91.18˘0.01
CTM (95 %ile) 100.0 99.38 96.25 91.39
CTM (Peak) 100.0 99.40 96.31 91.50
Table 4: Empirical results - test accuracy in percent.
Model MNIST F-MNIST
Logistic Circuit (binary) [22] 1 072 kB 2 456 kB
Logistic Circuit (real-valued) [22] 728kB 1 768 kB
CNN w/3 conv. layers [22] 8 784 kB 8 784 kB
ResNet [22] 19 352 kB 19 352 kB


key: chroma:document
string_value:
ResNet [22] 19 352 kB 19 352 kB
TM w/ 4 000 class clauses 7 657 kB 7 657 kB
CTM w/ 250class clauses 127kB 127kB
CTM w/ 8 000 class clauses 2 109 kB 2 109 kB
Table 5: Model size in kilobytes (kB), assuming 32-bit real-valued weights.
peaks at 99.4% after 234epochs, while K-MNIST peaks at 96.31% after 169epochs. For F-MNIST, test accuracy
surpasses 90.0% in epoch 9and peaks at 91.5% in epoch 170.


key: chroma:document
string_value:
In Table 5, the size of the trained models are listed for MNIST and F-MNIST. Size is measured in kilobytes of memory
used. The memory usage of CTM is similar to Logistic Circuits, which both use signiﬁcantly less memory than the
neural network architectures. A potential advantage of CTM is the ability to trade off accuracy against computation
time and memory usage, by reducing the number of clauses employed. E.g., a CTM with 250 clauses per class reduces


key: chroma:document
string_value:
memory usage by more than an order of magnitude at a relatively small loss in test accuracy (cf. Table 3).
Table 2 contains example patterns produced by CTM for MNIST (with one bit per pixel using 8000 clauses). The “*"
symbol can either take the value “0" or “1". The remaining bit values require strict matching. The examples have been
selected to illustrate how several patterns together form complete numbers. As seen, the patterns are relatively easy to


key: chroma:document
string_value:
interpret for humans compared to, e.g., a neural network. They are also efﬁcient to evaluate for computers, involving
only logical operators, followed by summation.
5 Conclusion and Further Work
This paper introduced the Convolutional Tsetlin Machine (CTM), leveraging the learning mechanism of the Tsetlin
Machine (TM). Whereas the TM categorizes images by employing each clause once per XˆYinput, the CTM


key: chroma:document
string_value:
uses each clause as a WˆWconvolution ﬁlter. The ﬁlters learned by the CTM are interpretable, being formulated
using propositional formulae. To make the clauses location-aware, each patch is further enhanced with its coordinates
within the image. Location awareness may prove useful in applications where both patterns and their location are
distinguishing features. By randomly selecting which patch to learn from, the standard Type I and Type II feedback


key: chroma:document
string_value:
of the classic TM can be employed directly. In this manner, the CTM obtains results on MNIST, Kuzushiji-MNIST,
Fashion-MNIST, and the 2D Noisy XOR Problem that compare favorably with simple 4-layer CNNs, Logistic Circuits,
as well as two binary neural network architectures.
11


key: chroma:document
string_value:
APREPRINT - DECEMBER 30, 2019
0 20 40 60 80 100 120 140 160 180 200 220 240767880828486889092949698100
EpochAccuracy (%)
MNIST
K-MNIST
F-MNIST
Figure 5: Single-run test accuracy per epoch for CTM on MNIST, K-MNIST, and F-MNIST.
In our further work, we intend to investigate more advanced binary encoding schemes, to go beyond grey-scale images
(e.g., addressing CIFAR-10 and ImageNet). We further intend to develop schemes for deeper CTMs, with the ﬁrst step


key: chroma:document
string_value:
being a two-layer CTM, to introduce more compact and expressive patterns with nested propositional formulae.
References
[1]K. D. Abeyrathna, O.-C. Granmo, X. Zhang, L. Jiao, and M. Goodwin. The Regression Tsetlin Machine - A Novel
Approach to Interpretable Non-Linear Regression. Philosophical Transactions of the Royal Society A , 378, 2019.
[2]R. Agrawal, T. Imieli ´nski, and A. Swami. Mining association rules between sets of items in large databases.
SIGMOD Rec. , 22(2):207–216, 1993.


key: chroma:document
string_value:
SIGMOD Rec. , 22(2):207–216, 1993.
[3]G. T. Berge, O.-C. Granmo, T. O. Tveit, M. Goodwin, L. Jiao, and B. V . Matheussen. Using the Tsetlin Machine
to Learn Human-Interpretable Rules for High-Accuracy Text Categorization with Medical Applications. IEEE
Access , 7:115134–115146, 2019.
[4]N. Bouhmala and O.-C. Granmo. Stochastic Learning for SAT-Encoded Graph Coloring Problems. International
Journal of Applied Metaheuristic Computing , 1(3):1–19, 2010.


key: chroma:document
string_value:
[5]T. Clanuwat, M. Bober-Irizar, A. Kitamoto, A. Lamb, K. Yamamoto, and D. Ha. Deep Learning for Classical
Japanese Literature. arXiv:1812.01718 , Dec 2018.
[6]M. Courbariaux, Y . Bengio, and J.-P. David. Binaryconnect: Training deep neural networks with binary weights
during propagations. In Advances in neural information processing systems , pages 3123–3131, 2015.
[7]S. Darabi, M. Belbahri, M. Courbariaux, and V . Partovi Nia. BNN+: Improved Binary Network Training.


key: chroma:document
string_value:
arXiv:1812.11800 , Dec 2018.
[8]K. Darshana Abeyrathna, O.-C. Granmo, X. Zhang, and M. Goodwin. A Scheme for Continuous Input to the
Tsetlin Machine with Applications to Forecasting Disease Outbreaks. arXiv:1905.04199 , May 2019.
[9]V . Feldman. Hardness of Approximate Two-Level Logic Minimization and PAC Learning with Membership
Queries. Journal of Computer and System Sciences , 75(1):13–26, 2009.


key: chroma:document
string_value:
[10] M. Ghavipour and M. R. Meybodi. A streaming sampling algorithm for social activity networks using ﬁxed
structure learning automata. Applied Intelligence , 2018.
12


key: chroma:document
string_value:
APREPRINT - DECEMBER 30, 2019
[11] J. Gittins. Bandit processes and dynamic allocation indices. Journal of the Royal Statistical Society, Series B
(Methodological) , 41(2):148–177, 1979.
[12] O.-C. Granmo. The Tsetlin Machine - A Game Theoretic Bandit Driven Approach to Optimal Pattern Recognition
with Propositional Logic. arXiv:1804.01508 , Apr 2018.
[13] O.-C. Granmo and N. Bouhmala. Solving the Satisﬁability Problem Using Finite Learning Automata. International


key: chroma:document
string_value:
Journal of Computer Science and Applications , 4(3):15–29, 2007.
[14] O. C. Granmo and B. J. Oommen. Optimal sampling for estimation with constrained resources using a learning
automaton-based solution for the nonlinear fractional knapsack problem. Applied Intelligence , 33(1):3–20, 2010.
[15] O.-C. Granmo and B. J. Oommen. Solving Stochastic Nonlinear Resource Allocation Problems Using a Hierarchy
of Twofold Resource Allocation Automata. IEEE Transactions on Computers , 59(4):545–560, 2010.


key: chroma:document
string_value:
[16] O.-C. Granmo, B. J. Oommen, S. A. Myrer, and M. G. Olsen. Learning Automata-based Solutions to the Nonlinear
Fractional Knapsack Problem with Applications to Optimal Resource Allocation. IEEE Transactions on Systems,
Man, and Cybernetics, Part B , 37(1):166–175, 2007.
[17] V . Haugland, M. Kjølleberg, S.-E. Larsen, and O.-C. Granmo. A two-armed bandit collective for hierarchical ex-


key: chroma:document
string_value:
amplar based mining of frequent itemsets with applications to intrusion detection. Transactions on Computational
Collective Intelligence XIV , 8615:1–19, 2014.
[18] J. R. Hauser, O. Toubia, T. Evgeniou, R. Befurt, and D. Dzyabura. Disjunctions of Conjunctions, Cognitive
Simplicity, and Consideration Sets. Journal of Marketing Research , 47(3):485–496, 2010.
[19] J. Hu, L. Shen, and G. Sun. Squeeze-and-excitation networks. In Proceedings of the IEEE conference on computer


key: chroma:document
string_value:
vision and pattern recognition , pages 7132–7141, 2018.
[20] C. Lammie, W. Xiang, and M. Rahimi Azghadi. Accelerating Deterministic and Stochastic Binarized Neural
Networks on FPGAs Using OpenCL. arXiv:1905.06105 , May 2019.
[21] Y . LeCun, L. Bottou, Y . Bengio, P. Haffner, et al. Gradient-based learning applied to document recognition.
Proceedings of the IEEE , 86(11):2278–2324, 1998.


key: chroma:document
string_value:
Proceedings of the IEEE , 86(11):2278–2324, 1998.
[22] Y . Liang and G. Van den Broeck. Learning logistic circuits. In Proceedings of the 33rd Conference on Artiﬁcial
Intelligence (AAAI) , jan 2019.
[23] R. Liu, J. Lehman, P. Molino, F. P. Such, E. Frank, A. Sergeev, and J. Yosinski. An intriguing failing of
convolutional neural networks and the coordconv solution. In Proceedings of the 32Nd International Conference
on Neural Information Processing Systems , NIPS’18, pages 9628–9639, 2018.


key: chroma:document
string_value:
[24] T. McCormick, C. Rudin, and D. Madigan. A Hierarchical Model for Association Rule Mining of Sequential
Events: An Approach to Automated Medical Symptom Prediction. Annals of Applied Statistics , 2011.
[25] K. S. Narendra and M. A. L. Thathachar. Learning Automata: An Introduction . Prentice-Hall, Inc., 1989.
[26] B. Oommen, S. Misra, and O.-C. Granmo. Routing bandwidth-guaranteed paths in MPLS trafﬁc engineering: A


key: chroma:document
string_value:
multiple race track learning approach. IEEE Transactions on Computers , 56(7), 2007.
[27] B. J. Oommen. Stochastic Searching on the Line and its Applications to Parameter Learning in Nonlinear
Optimization. IEEE Transactions on Systems, Man, and Cybernetics, Part B , 27(4):733–739, 1997.
[28] B. J. Oommen, S.-W. Kim, M. T. Samuel, and O.-C. Granmo. A solution to the stochastic point location problem


key: chroma:document
string_value:
in metalevel nonstationary environments. Systems, Man, and Cybernetics, Part B: Cybernetics, IEEE Transactions
on, 38(2):466–476, 2008.
[29] B. J. Oommen and D. C. Ma. Deterministic Learning Automata Solutions to The Equipartitioning Problem. IEEE
Transactions on Computers , 37(1):2–13, 1988.
[30] R. Peharz, A. Vergari, K. Stelzner, A. Molina, M. Trapp, K. Kersting, and Z. Ghahramani. Probabilistic deep
learning using random sum-product networks. 2018.


key: chroma:document
string_value:
learning using random sum-product networks. 2018.
[31] A. Phoulady, O.-C. Granmo, S. R. Gorji, and H. A. Phoulady. The Weighted Tsetlin Machine: Compressed
Representations with Clause Weighting. In Proceedings of the Ninth International Workshop on Statistical
Relational AI (StarAI 2020) , 2020.
[32] M. Rastegari, V . Ordonez, J. Redmon, and A. Farhadi. Xnor-net: Imagenet classiﬁcation using binary convolutional


key: chroma:document
string_value:
neural networks. In European Conference on Computer Vision , pages 525–542. Springer, 2016.
[33] H. Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society ,
1952.
13


key: chroma:document
string_value:
APREPRINT - DECEMBER 30, 2019
[34] C. Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable
models instead. Nature Machine Intelligence , 1(5):206–215, 2019.
[35] C. Rudin, B. Letham, and D. Madigan. Learning theory analysis for association rules and sequential event
prediction. Journal of Machine Learning Research , 14:3441–3492, 2013.


key: chroma:document
string_value:
[36] M. L. Tsetlin. On behaviour of ﬁnite automata in random medium. Avtomat. i Telemekh , 22(10):1345–1354, 1961.
[37] B. Tung and L. Kleinrock. Using Finite State Automata to Produce Self-Optimization and Self-Control. IEEE
Transactions on Parallel and Distributed Systems , 7(4):47–61, 1996.
[38] L. G. Valiant. A Theory of the Learnable. Communications of the ACM , 27(11):1134–1142, 1984.


key: chroma:document
string_value:
[39] T. Wang, C. Rudin, F. Doshi-Velez, Y . Liu, E. Klampﬂ, and P. MacNeille. A Bayesian Framework for Learning
Rule Sets for Interpretable Classiﬁcation. The Journal of Machine Learning Research , 18(1):2357–2393, 2017.
[40] A. Wheeldon, R. Shaﬁk, A. Yakovlev, J. Edwards, I. Haddadi, and O.-C. Granmo. Tsetlin Machine: A New
Paradigm for Pervasive AI. In Proceedings of the SCONA Workshop at Design, Automation and Test in Europe
(DATE) , 2020.


key: chroma:document
string_value:
(DATE) , 2020.
[41] H. Xiao, K. Rasul, and R. V ollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning
algorithms. arXiv:1708.07747 , 2017.
[42] A. Yazidi, O.-C. Granmo, and B. Oommen. Service selection in stochastic environments: A learning-automaton
based solution. Applied Intelligence , 36(3):617–637, 2012.
[43] A. Yazidi, O.-C. Granmo, and B. Oommen. Learning-Automaton-Based Online Discovery and Tracking of


key: chroma:document
string_value:
Spatiotemporal Event Patterns. IEEE Transactions on Cybernetics , 43(3):1118 – 1130, 2013.
[44] A. Yazidi and B. John Oommen. On the analysis of a random walk-jump chain with tree-based transitions and its
applications to faulty dichotomous search. Sequential Analysis , 37:31–46, Jan 2018.
[45] M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. In European conference on
computer vision , pages 818–833. Springer, 2014.


key: chroma:document
string_value:
computer vision , pages 818–833. Springer, 2014.
[46] J. Zhang, Y . Wang, C. Wang, and M. Zhou. Symmetrical Hierarchical Stochastic Searching on the Line in
Informative and Deceptive Environments. IEEE Transactions on Cybernetics , 47(3):626 – 635, Jul 2016.
14


key: chroma:document
string_value:
Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022) , pages 3761–3770
Marseille, 20-25 June 2022
© European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0
3761ConvTextTM: An Explainable Convolutional Tsetlin Machine Framework
for Text Classification
Bimal Bhattarai, Ole-Christoffer Granmo, Lei Jiao
University of Agder
Norway
{bimal.bhattarai, ole.granmo, lei.jiao }@uia.no
Abstract


key: chroma:document
string_value:
Abstract
Recent advancements in natural language processing (NLP) have reshaped the industry, with powerful language models such
as GPT-3 achieving superhuman performance on various tasks. However, the increasing complexity of such models turns
them into “black boxes”, creating uncertainty about their internal operation and decision-making. Tsetlin Machine (TM)
employs human-interpretable conjunctive clauses in propositional logic to solve complex pattern recognition problems and has


key: chroma:document
string_value:
demonstrated competitive performance in various NLP tasks. In this paper, we propose ConvTextTM, a novel convolutional
TM architecture for text classification. While legacy TM solutions treat the whole text as a corpus-specific set-of-words
(SOW), ConvTextTM breaks down the text into a sequence of text fragments. The convolution over the text fragments
opens up for local position-aware analysis. Further, ConvTextTM eliminates the dependency on a corpus-specific vocabulary.


key: chroma:document
string_value:
Instead, it employs a generic SOW formed by the tokenization scheme of the Bidirectional Encoder Representations from
Transformers (BERT) (Devlin et al., 2019a). The convolution binds together the tokens, allowing ConvTextTM to address
the out-of-vocabulary problem as well as spelling errors. We investigate the local explainability of our proposed method
using clause-based features. Extensive experiments are conducted on seven datasets, to demonstrate that the accuracy of


key: chroma:document
string_value:
ConvTextTM is either superior or comparable to state-of-the-art baselines.
Keywords: Set-Of-Word (SOW), Convolutional Tsetlin Machine (CTM), Explainable, Human-Interpretable, Language
Model, Text Classification, Tsetlin Machine (TM).
1. Introduction
Recent advances in Artificial Intelligence (AI) have
brought increasingly accurate learning algorithms and
powerful computation platforms. However, the ac-
curacy gains come with escalating computation costs,


key: chroma:document
string_value:
and models are getting too complicated for humans
to comprehend. Prohibitive computation costs hinder
the training of state-of-the-art models, while a lack of
model transparency reduces trust. When end-users can-
not understand model decisions and researchers report
on how susceptible the models are to data bias, confi-
dence in machine learning models declines.
Mounting computation costs are problematic because
of the long-term environmental impact. The obscurity


key: chroma:document
string_value:
of AI-driven decisions, however, raises immediate eth-
ical concerns. Indeed, many decision errors can be fa-
tal in high-risk application domains, including bank-
ing (Le and Viviani, 2018), medicine (Esteva et al.,
2019), bioinformatics (Freitas et al., 2008), and self-
driving cars (Badue et al., 2020). Transparency is re-
quired to ensure fair and safe decisions. AI model inter-
pretability is thus a research area of increasing interest,
crucial for further advancement of the AI field.


key: chroma:document
string_value:
crucial for further advancement of the AI field.
The field of Natural Language Processing (NLP) man-
ifests all of the above concerns. With the develop-
ment of powerful language models such as Transform-
ers (Vaswani et al., 2017), the prediction accuracy al-
ready peaks for several NLP tasks, such as sentiment
analysis and text classification. However, model com-
plexity is immense, and only approximate interpreta-tion techniques are available. On the other side of


key: chroma:document
string_value:
the spectrum, traditional models like Linear Regres-
sion, Logistic Regression, and Decision Trees can be
highly interpretable but are far from achieving state-of-
art accuracy. As such, they demonstrate the challeng-
ing accuracy-interpretability trade-off researchers must
resolve.
The Tsetlin Machine (TM) (Granmo, 2018) is a rule-
based pattern recognition approach that aims to bridge
the gap between accuracy and interpretability. It uses
a majority vote among multiple rules for decision-


key: chroma:document
string_value:
making. As such, TMs unify summation-based (cf.
logistic regression) and rule-based (cf. decision trees)
approaches. TMs further employ three powerful strate-
gies for learning patterns: (i) Frequent pattern mining
with so-called Type I Feedback ; (ii) Pattern discrimi-
nation with so-called Type II Feedback ; and (iii) Data
dissection utilizing a vote margin .
Recent papers report TM accuracy results that are com-
parable to or surpass the accuracy of contemporary


key: chroma:document
string_value:
interpretable methods. These results cover sentiment
analysis (Yadav et al., 2021a), novelty detection (Bhat-
tarai et al., 2021a)(Bhattarai et al., 2022), fake news
detection (Bhattarai et al., 2021b), and semantic re-
lation analysis (Saha et al., 2020). Indeed, in some
cases, the TM outperforms state-of-the-art deep learn-
ing baselines (Bhattarai et al., 2021b). Based on propo-
sitional logic and bitwise operations for learning, TMs
offer an alternative approach to deep learning models.


key: chroma:document
string_value:
3762complexity, and comparable or better prediction accu-
racy in an increasing number of cases.
TMs achieve human-level interpretability through
AND rules with negation that uses set-of-words (SOW)
based features. In SOW, each word is one-hot encoded
and then ORed into a document bit vector. One can also
view each AND-rule as a conjunctive clause in propo-
sitional logic. A propositional variable represents each
word. As reported by Blakely et al., clauses learned


key: chroma:document
string_value:
by a TM are highly discriminative, while being easy to
interpret (Blakely and Granmo, 2020). Because each
clause is self-contained, one can interpret the TM glob-
ally simply by inspecting each clause.
With increasing vocabulary size, however, the SOW
representation becomes increasingly sparse. The
sparseness affects both the preprocessing time, train-
ing time, and accuracy. Yadav et al. were able
to reduce sparsity by boosting the SOW with GloVe
synonyms (Yadav et al., 2021b). However, reduc-


key: chroma:document
string_value:
synonyms (Yadav et al., 2021b). However, reduc-
ing pre-trained word embeddings to synonyms re-
moves information from the embeddings. A main
problem is that pre-trained word embeddings such as
word2vec (Mikolov et al., 2013), GloVe (Pennington
et al., 2014), and FastText (Bojanowski et al., 2017),
are real-valued. TMs, however, require binary input.
How to produce binary embeddings that are suitable
for TMs is an open problem. Additionally, using non-


key: chroma:document
string_value:
interpretable word embeddings would impact TM in-
terpretability.
Contributions: In this paper, we propose Conv-
TextTM, a novel convolutional TM (CTM) (Granmo
et al., 2019) architecture for text classification. While
legacy TM solutions treat the whole text as a corpus-
specific SOW, ConvTextTM breaks down the text into
a sequence of text fragments. The convolution over
the text fragments opens up for local position-aware
analysis. Further, ConvTextTM eliminates the depen-


key: chroma:document
string_value:
dency on a corpus-specific vocabulary. Instead, it
employs a generic SOW formed by the tokenization
scheme of the Bidirectional Encoder Representations
from Transformers (BERT). The convolution binds to-
gether the tokens, allowing ConvTextTM to address the
out-of-vocabulary problem as well as spelling errors.
We investigate the local explainability of our proposed
method using clause-based features. To our knowledge,
this is the first time CTM has been used in a text-based
application.


key: chroma:document
string_value:
application.
2. Related Work
Human interpretable machine learning models have a
long history, with Breiman’s research on decision trees
and random forests being an early example (Breiman,
2001)(Li et al., 1984). Since then, several approaches
for explaining the prediction of models have appeared.
However, only a few studies focus on model inter-
pretability (Lipton, 2018)(Lage et al., 2019). Some of
these address local interpretability, i.e., they study how


key: chroma:document
string_value:
to understand single outputs (Ribeiro et al., 2016). Oth-ers try to interpret the entire model, providing global
interpretability (Ustun and Rudin, 2016).
Gilpin et al. define interpretability of deep learning in
terms of three categories (Gilpin et al., 2018): (1) mod-
els that map an input to output, focusing on connec-
tivity, (2) models that explain latent variables for an
explanation, and (3) models that explain themselves.
Based on these categories, Agarwal et al. (Agarwal et


key: chroma:document
string_value:
al., 2020) proposed a deep learning-based interpretable
model inspired by the family of Neural Additive Mod-
els (NAMs). However, their model was not able to
surpass regular deep learning models accuracy-wise.
As recent advances have demonstrated, interpretable
rule-based approaches may provide a better trade-off
between interpretability and accuracy. Numerous ap-
proaches, such as frequent itemset mining for associ-
ation rule learning (Agrawal et al., 1993), and Prob-


key: chroma:document
string_value:
ably Approximately Correct (PAC) (Feldman, 2009)
have provided insights into how data patterns can be
represented in Disjunctive Normal Form (DNF) using
propositional formula. These techniques, however, are
generally not scalable and robust to noise, and cannot
compete with deep learning models.
TMs are a recent approach to rule-based machine learn-
ing that extracts human-interpretable patterns from data
using propositional reasoning. Berge et al. (Berge


key: chroma:document
string_value:
et al., 2019) employed TM for categorizing medical
records, with an emphasis on the interpretability of the
rules generated by TM. They demonstrated that TMs
are interpretable, yet provide competitive accuracy in
comparison with other machine learning techniques,
including Convolutional Neural Networks (CNN) and
Long Short-Term Memory (LSTM) Neural Networks.
Other researchers report similar findings, for example
within regression (Darshana Abeyrathna et al., 2020),


key: chroma:document
string_value:
natural language understanding (Yadav et al., 2021a;
Bhattarai et al., 2021b; Saha et al., 2020; Yadav et al.,
2021c), and speech understanding (Lei et al., 2021).
Additionally, the Convolutional TM performs com-
petitively on MNIST, Fashion-MNIST, and Kuzushiji-
MNIST, compared with several other contemporary
methods including CNNs (Granmo et al., 2019).
Methods for global and local model interpretation are
gaining increasing attention in the NLP research com-


key: chroma:document
string_value:
munity. (Jacovi et al., 2018) examined how CNN net-
works process and categorize text. They demonstrate,
by employing various alternative activation patterns,
how filters may capture several semantic classes of
n-grams. They compute word-level scores in two
ways: locally and globally. Both techniques, however,
need a specific input and employ leave-one-out evalu-
ations. The authors of (Xiong et al., 2018) conclude
that Layer-wise Relevance Propagation (LRP) extracts


key: chroma:document
string_value:
more relevant features and argue that a single word
does not adequately explain any given outcome. For
instance, a single word can be a component of a signif-
icant n-gram. However, their method requires knowl-
edge of the model parameters, which cannot be consid-


key: chroma:document
string_value:
3763
Sentence
window window 
window BER T Vocabulary
List
window 
window SOW features Figure 1: Preprocessing text for Convolutional TM.
ered an explanation for the black-box nature of the neu-
ral network. (Chen and Ji, 2020) use masks over word
embedding named variational word mask (VMASK)
and provide an interpretable architecture for text clas-
sification to learn task-specific important words. They
demonstrate the model’s interpretability by compar-


key: chroma:document
string_value:
ing it to LIME and SampleShapley (Strumbelj and
Kononenko, 2010). Similarly, (Nguyen, 2018) pre-
sented a robust technique for text categorization that
relies on local explanation. They demonstrate through
crowdsourcing experiments that automatic evaluation
metrics based on word elimination, such as LIME, are
reasonably consistent with human evaluations.
Despite these advances in NLP interpretability, strong
transformer-based language models such as BERT (De-


key: chroma:document
string_value:
vlin et al., 2019a) and XLNet (Yang et al., 2019a) out-
perform alternative approaches that rely on pre-trained
embeddings. In summary, previous work report that
as the complexity of the model increases, the interpre-
tation becomes more ambiguous (Lipton, 2018). For
example, it is possible that the small neural network
models may be more interpretable than large decision
trees. One can argue that some models employ an at-
tention mechanism that can be interpreted based on the


key: chroma:document
string_value:
weighting of the input features. Still, other researchers
report that attention weights do not provide a meaning-
ful explanation (Bai et al., 2020)(Serrano and Smith,
2019). To overcome these shortcomings, the interest
in rule-based interpretable techniques is rising, with
TM being a promising approach. However, as dis-
cussed, the SOW sparsity introduced by large vocab-
ularies is significantly limiting TMs. Yadav et al. were
able to partially mitigate increasing sparsity by boost-


key: chroma:document
string_value:
ing TM SOWs with GloVe synonyms, significantly in-
creasing accuracy (Yadav et al., 2021b). The work we
present here, however, attempts to address this issue by
employing a fixed-sized pre-trained BERT embedding
vocabulary. Accordingly, we establish a general TM
SOW of manageable size.
3. ConvTextTM Architecture
3.1. Tsetlin Machine Basics
A basic TM consists of two-action Tsetlin Au-
tomata (TAs) with 2Nstates. Each TA performs anaction associated with its current state, which is either


key: chroma:document
string_value:
an “Include” action (in State 1toN) or an “Exclude”
action ( N+ 1to2N). The state updates are based on
iterative feedback, i.e., rewards and penalties. Rewards
reinforce the action performed by the TA, while penal-
ties suppress the action. In this way, the TA progres-
sively shifts toward the optimal action (Tsetlin, 1961).
To address the pattern recognition problem, the collec-
tion of TAs construct conjunctive clauses in proposi-
tional logic. The input to a TM consists of proposi-


key: chroma:document
string_value:
tional values, i.e., the input is Boolean. A TM uses
both the non-negated and negated version of each input
when forming the clauses, each of which is handled by
an individual TA. Accordingly, each TM clause cap-
tures a specific sub-pattern. The output, in turn, is de-
termined based on votes for the available classes. It
is the class with the largest number of matching sub-
patterns that wins the round of voting and is output by
the TM.
The TM takes a Boolean input vector, X=


key: chroma:document
string_value:
the TM.
The TM takes a Boolean input vector, X=
(x1, . . . , x o),xk∈ {0,1},k∈ {1, . . . , o}to be clas-
sified into one of the Clclasses, Y= (y1, . . . , y Cl),
where Clis total number of classes. From the in-
put vector, we obtain 2oliterals L= (l1, l2, . . . , l 2o).
The literals consist of the inputs xkand their negated
counterparts ¯xk=¬xk= 1−xk, i.e., L=
(x1, . . . , x o,¬x1, . . . ,¬xo). The TM patterns are
formed using mconjunctive clauses. Subscript j=


key: chroma:document
string_value:
formed using mconjunctive clauses. Subscript j=
1, . . . , m/ 2denotes the clause index, while the super-
script describes the polarity of a clause. In brief, half
of the clauses are assigned positive polarity, i.e., C+
j,
and the other half are assigned negative polarity, i.e.,
C−
j. A clause Cξ
j,ξ∈ {− ,+},is formed by ANDing
a subset of the literals, i.e., the ones that are “included”
by the corresponding TAs, Lξ
j⊆L, as:
Cξ
j(X) =^
lk∈Lξ
jlk=Y
lk∈Lξ
jlk. (1)


key: chroma:document
string_value:
j⊆L, as:
Cξ
j(X) =^
lk∈Lξ
jlk=Y
lk∈Lξ
jlk. (1)
The final classification output is obtained by subtract-
ing the negative votes from the positive votes, and then
thresholding the resulting sum via the unit step function
u:
ˆy=u
m/2X
j=1C+
j(X)−m/2X
j=1C−
j(X)
. (2)
The TM learning process carefully guides the TAs to
make optimal decisions. The reinforcement is chan-
neled directly to the conjunctive clauses. Each clause,
in turn, forwards the feedback to its individual TAs ran-


key: chroma:document
string_value:
domly. To this end, the TM employs Type I and Type
II feedback. These feedback types control how rewards
and penalties are distributed to the TAs. The distribu-
tion depends on six factors: (1) target output ( y= 0or
y= 1), (2) clause polarity, (3) clause output ( Cj= 0or
1), (4) literals value ( x= 1, or¬x= 1), (5) vote sum,
and (6) the current state of the TA. Type I feedback is
intended to generate frequent patterns, eliminate false


key: chroma:document
string_value:
3764
window 
window Sentence
ArgmaxFeedbackPreprocessed text
Divide into
windowsFigure 2: ConvTextTM Architecture.
negatives, and make clauses evaluate to 1. Positive po-
larity clauses receive Type I input when y= 1, while
negative polarity clauses receive it when y= 0. Type II
feedback, on the other hand, improves pattern discrim-
ination, suppresses false positives, and spurs clauses to
evaluate to 0. Type II feedback is given to positive
polarity clauses when y= 0 and to negative polarity


key: chroma:document
string_value:
clauses when y= 1.
The feedback is further regulated by the sum of votes
vfor each output class. That is, the voting sum is
compared against a voting margin T, which is em-
ployed to guide distinct clauses to learn different sub-
patterns. The details of the learning process can be
found in (Granmo, 2018).
3.2. Convolutional Tsetlin Machine and
Pre-processing for Text
The CTM is a variation of TM developed for image
classification. The CTM takes an image X= (xk)∈


key: chroma:document
string_value:
classification. The CTM takes an image X= (xk)∈
{0,1}A×B×Cas input. Here, A×B specifies the image
dimension and Cdenotes the depth of the binary layers.
Because CTM is designed for fixed-sized images, we
must preprocess our text data prior to feeding it to the
CTM. To begin with, each input instance is segmented
into a number of equal-sized windows. To make the
window size equal, we append “PAD” tokens at the
end of the instances. Because the average lengths of


key: chroma:document
string_value:
instances in different datasets can be quite different,
the number of windows generated for different datasets
may vary.
To integrate the information from BERT, we create a
dictionary that indexes all of the tokens in the BERT
vocabulary. The words in each window are then one-
hot encoded using their indices in the dictionary and
the window number in which they appear. That is, each
word is denoted by a 2DSOW representation. For ex-
ample, suppose a word present in the second window of


key: chroma:document
string_value:
the first input instance is assigned the 500thindex in the
dictionary of BERT. Then, position (2, 500) of the input
is set to 1. After preprocessing, each input text is repre-
sented as a Boolean 2DSOW. Now, the complete input
dataset can be represented by (xk)∈ {0,1}A×B×C.
HereAdenotes the total number of input instances, B
denotes the total number of windows, and Crepresents
the total number of tokens in the BERT vocabulary list.
Figure 1 depicts the full preprocessing pipeline.


key: chroma:document
string_value:
Figure 1 depicts the full preprocessing pipeline.
The CTM for text uses clauses with spatial dimension
1×1as filters to perform convolution over the prepro-Dataset name Train set size Test set size Label
PolitiFact 716 238 2
GossipCop 15,175 5,058 2
BBCSports 517 220 5
Twitter 2,176 932 3
Query 17,500 3,850 2
R8 5,478 2,189 8
WOS-5736 4,588 1,148 11
Table 1: Dataset Statistics.
cessed input text. The filter evaluates P=B−1
q+ 1
patches of dimensionality C, with qbeing the step


key: chroma:document
string_value:
patches of dimensionality C, with qbeing the step
size of the convolution. Additionally, each clause can
specify binary encoded coordinates for location aware-
ness (visualized as red boxes in Figure 2). The coor-
dinates are encoded by augmenting each patch input
vector xp= [xp
k]∈ {0,1}C,p∈ {1,2, . . . , P }, with a
coordinate vector. The coordinate vector lp= [lp
k]∈
{0,1}P×2of each patch is threshold-encoded (Dar-
shana Abeyrathna et al., 2020). That is, the input vec-


key: chroma:document
string_value:
tor is extended with one propositional variable per po-
sition. The resulting augmented vector is denoted by
xp:xp= [xp
k]∈ {0,1}C+P. The overall ConvTextTM
architecture is depicted in Figure 2.
Classification in CTM is similar to that of vanilla TM,
as shown in Eq. 2. However, because we have Pinput
vectors xpfor CTM, each clause produces Poutput
values. The final output is obtained by OR ingthe indi-
vidual outputs from each patch, specified as follows:
ˆy=u
m/2X
j=1P_
p=1
^
lk∈L+
jlp


key: chroma:document
string_value:
ˆy=u
m/2X
j=1P_
p=1
^
lk∈L+
jlp
k
−m/2X
j=1P_
p=1
^
lk∈L−
jlp
k

.
(3)
CTM learning leverages the TM learning, with clauses
receiving the feedback based on the input vector and
label. However, for CTM, there are Pinput patches in
an input. Therefore, when a clause is updated, CTM
randomly selects one of the patch input vector xpthat
made the clause evaluate to 1:
xp
j=Random


xp
^
lk∈Lξ
jlp
k
= 1,1≤p≤P


.
(4)


key: chroma:document
string_value:
^
lk∈Lξ
jlp
k
= 1,1≤p≤P


.
(4)
The randomly chosen patch input xpenables each
clause to extract a specific sub-pattern, and the unpre-
dictability of the uniform distribution statistically scat-
ters the clauses for various sub-patterns in the target
text.
4. Experiments
4.1. Datasets
Table 1 provides an overview of the publicly available
datasets used in our experiment. The goal of this study
is to assess our framework across a variety of datasets


key: chroma:document
string_value:
3765Datasets Epochs #clauses Threshold (T) Sensitivity (s) Window
PolitiFact 200 10,000 100*100 10.0 3
GossipCop 200 15,000 100*100 20.0 3
BBCSports 100 10,000 150*150 10.0 2
Twitter 100 10,000 50*50 10.0 2
Query 100 5,000 150*150 10.0 2
R8 150 10,000 100*100 10.0 2
WOS-5736 150 10,000 150*150 10.0 2
Table 2: Hyperparameter configurations.
linguistic patterns. To this end, we employ datasets
related to fact-checking, document classification, aca-


key: chroma:document
string_value:
demic text classification, and short text classification.
We organize the datasets as follows:
•Fact Checking : We here utilize a publicly avail-
able data repository for fake news detection,
FakeNewsNet (Shu et al., 2020a). The labels
are provided by professional journalists from the
fact-checking websites PolitiFact andGossipCop .
PolitiFact focuses on U.S. political news, whereas
GossipCop focuses on entertainment news from
various media.
•Topic Classification : For topic classification, we


key: chroma:document
string_value:
use BBC Sports and a Twitter dataset. BBC Sports
consists of 737documents from the BBC Sports
website, collected between 2004 and 2005, la-
beled according to five sports categories. Further-
more, the Twitter dataset consists of a collection
of5,513 tweets that have been hand-classified
into one of the four topics: Apple, Google, Twit-
ter, and Microsoft. The tweets are further labeled
with sentiments positive, negative, or neutral.
•Short text classification : These datasets in-


key: chroma:document
string_value:
•Short text classification : These datasets in-
volve short text sentences. We explore perfor-
mance using the Query (Faruqui and Das, 2018)
and R8 (Debole and Sebastiani, 2005) datasets.
Google’s query dataset was created by crowd-
sourcing well-formedness annotations for 25,100
queries from the Paralex corpus. Each query
was binary annotated by five raters regarding
whether the query was well-formed. R8 is a subset
of the Reuters-21578 corpus that contains news


key: chroma:document
string_value:
of the Reuters-21578 corpus that contains news
documents classified into the eight most popular
classes. Note that the document distribution in R8
is severely skewed, with the smallest class holding
just 51 documents.
•Academic text classification : We utilize the Web
of Science (WoS)-5736 datset (Kowsari et al.,
2017a), which comprises 5,736published papers
with eleven categories organized under three main
categories.
4.2. Baselines
We begin by summarizing the baseline methods.


key: chroma:document
string_value:
We begin by summarizing the baseline methods.
• FastText (Joulin et al., 2017) uses a linear classi-
fier to input a document embedding, which is cre-
ated by the average of word/n-grams embeddings.• Rhetorical Structure Theory (RST) (Rubin et al.,
2015) uses a tree structure to depict the relation-
ship between words in a document. It extracts the
news style features from a bag of words by map-
ping them into a latent feature representation.
• Linguistic Inquiry and Word Count (LIWC) (Pen-


key: chroma:document
string_value:
• Linguistic Inquiry and Word Count (LIWC) (Pen-
nebaker et al., 2015) is used to extract and learn
features from psycholinguistic and deception cat-
egories.
• Hierarchical attention neural net-
work (HAN) (Yang et al., 2016) is employed for
embedding word-level attention on each sentence
and sentence-level attention on news content for
fake news detection.
• CNN-text (Kim, 2014) utilizes a convolutional
neural network (CNN) with pre-trained word vec-


key: chroma:document
string_value:
neural network (CNN) with pre-trained word vec-
tors to perform sentence-level classification. Mul-
tiple convolutional filters enable the model to col-
lect various granularities of text features from
news articles.
• LSTM-ATT (Lin et al., 2019a) employs long short
term memory (LSTM) in conjunction with an at-
tention mechanism. The model feeds a two-layer
LSTM with a 300-dimensional vector representa-
tion of news items as input for fake news detec-
tion.


key: chroma:document
string_value:
tion.
• Hierarchical Deep Learning for Text Classifica-
tion (HDLTex) (Kowsari et al., 2017b) performs
hierarchical classification by stacking deep learn-
ing architectures to give specialized comprehen-
sion at each level of the document hierarchy.
• RoBERTa-MWSS was proposed in (Shu et al.,
2020b), which employs the Multiple Sources of
Weak Social Supervision (MWSS) approach built
upon RoBERTa (Liu et al., 2019).
• Bidirectional Encoder Representations from


key: chroma:document
string_value:
• Bidirectional Encoder Representations from
Transformers (BERT) (Devlin et al., 2019b) is
a Transformer-based model which contains an
encoder with 12 transformer blocks, self-attention
heads, and a hidden shape size of 768.
• XLNet (Yang et al., 2019b) is a generalized auto-
regressive pretraining model that integrates au-
toencoding and a segment recurrence mechanism
from transformers.
Other approaches include Word Mover’s Dis-
tance (WMD), which defines the distance between


key: chroma:document
string_value:
tance (WMD), which defines the distance between
two documents as the minimum cost of converting
the words of one document into words of the other.
Deepsets adopt objective functions defined on sets
that are invariant to permutations, allowing for the
building of a deep network architecture that operates
on sets. Similarly, NNattention and Set-Transformer
are variants of Deepsets that substitutes an attention
mechanism for the sum operator.


key: chroma:document
string_value:
3766Datasets RST LIWC HAN CNN-text LSTM-ATT RoBERTa-MWSS BERT XLNet TM TMconv TMconv(max)
PolitiFact 60.7 76.9 83.7 65.3 83.3 82.5 88 89.5 87.1±0.24 90.27±0.33 91.21
GossipCop 53.1 73.6 74.2 73.9 79.3 80.3 85 85.5 84.2±0.03 85.82±0.27 86.28
Table 3: Performance comparison of our model with other baseline models for fact checking.
Datasets WMD Deepsets NNattention Transformer LSTM BERT XLNet TM TMconv TMconv(max)


key: chroma:document
string_value:
BBCSports 95.40±0.70 74.55±20.1 95.28±0.97 95.82±1.23 95.52 99 98 96.91 96.78 ±0.32 97.97
Twitter 71.3±0.70 70±1.62 70.91±0.62 72.21±0.47 72.1 74.71 78 71.13 70.67±0.27 71.91
Table 4: Performance comparison of our model with other baseline models for topic classification.
0 2 4 6 8 10
Window86878889909192Accuracy (%)
Figure 3: Accuracy vsWindow.
4.3. Training and Testing
We use a random train-test split of 70% /30% for all
datasets except R8 and query as they have predefined


key: chroma:document
string_value:
train and test sets. The process is repeated five times,
and the average accuracy and F1 scores are reported.
To provide robust results, we calculated an ensemble
average by first taking the average of 50stable epochs,
followed by taking the average of the resulting five av-
erages. The experiments were conducted on the server
- NVIDIA DGX-2 with dual Intel Xeon Platinum 8168,
2.7 GHz, 16 ×NVIDIA Tesla V100 (32 GB), and
Ubuntu 18.04 LTS x64. The hyperparameter config-


key: chroma:document
string_value:
Ubuntu 18.04 LTS x64. The hyperparameter config-
urations of the experiment for each dataset are listed in
Table 2.
5. Results and Discussions
In this section, we evaluate the performance of our
framework with the aforementioned baselines for each
classification task, including seven datasets.
For fact-checking (i.e., PolitiFact and GossipCop), we
see that HAN outperforms LIWC, CNN-text, and RST
on both datasets. This is arguable because HAN can


key: chroma:document
string_value:
capture the syntactic and semantic rules required to de-
tect fake news via hierarchical attention. Similarly, the
LIWC performs better than RST. One possible reason
for this is that LIWC is able to extract the linguistic
features from news articles based on words that de-
note psycholinguistic characteristics. The LSTM-ATT,
which incorporates substantial preprocessing utilizing
count features and sentiment features, as well as hyper-
parameter tweaking (Lin et al., 2019b), performs sim-


key: chroma:document
string_value:
ilarly to HAN in PolitiFact but exceeds it on Gossip-Cop. One possible explanation for this can be that the
attention mechanism is able to capture the relevant rep-
resentation of the input. Our model outperforms all the
baselines, including transformer-based models such as
RoBERTa-MWSS, BERT, and XLNet. Indeed, Con-
vTextTM achieves maximum accuracy of 91.21% for
PolitiFact and 86.28% for GossipCop. Besides, our
model is significantly simpler than the deep learning


key: chroma:document
string_value:
models since we do not use any pre-trained embed-
dings in preprocessing. This also contributes to the
transparency, interpretability, and explainability of our
model. Table 3 illustrates the comparison outcomes.
For topic classification (i.e., BBC Sports and Twit-
ter), Table 4 shows the classification accuracy of our
proposed model and those of the baselines. The
set-transformer outperforms other deep learning algo-
rithms including Deepsets, NNattention, and LSTM,


key: chroma:document
string_value:
rithms including Deepsets, NNattention, and LSTM,
marginally. The disparate performance of LSTM might
be explained by the fact that bidirectional LSTMs are
ineffective in modeling ungrammatical text. However,
BERT and XLNet surpass all other approaches, with
BERT achieving the accuracy of 99% in BBC Sports.
As the prior knowledge in the pre-trained BERT and
XLNet is not particular to any domain, fine-tuning is
required to unleash the actual potential of BERT and


key: chroma:document
string_value:
XLNet. Our proposed CTM method outperforms the
vanilla TM and all other techniques except BERT and
XLNet, and obtains an accuracy close to those of BERT
and XLNet. The low accuracy of our model in the Twit-
ter dataset is most probably because our model, by rely-
ing entirely on the SOW approach, overlooks important
word orders that are useful in sentiment classification.
For short text classification (i.e., Query and R8), Ta-
ble 5 compares the accuracy of ConvTextTM against


key: chroma:document
string_value:
ble 5 compares the accuracy of ConvTextTM against
state-of-the-art methods. As previously stated, with
large-scale pretraining and fine-tuning on small-data
tasks, BERT achieves 80% on Query and XLNet ob-
tains98% on R8, surpassing other approaches signifi-
cantly. Human evaluation was conducted on the query
dataset by submitting 1,000queries to a human profi-
cient in English, who were able to match the label with
88.4%accuracy, indicating an estimated upper bound.


key: chroma:document
string_value:
For academic text classification (i.e., WOS-5736), Ta-
ble 6 compares the accuracy of our model with other
techniques. While the RNN outperforms DNN and
CNN, HDLTex with stacked deep learning CNN ar-
chitecture performed surprisingly well, outperforming


key: chroma:document
string_value:
3767Datasets CNN Fasttext LSTM HAN BERT XLNet Human TM TMconv TMconv(max)
Query 67.38 62.1 65.8 64.64 80 77 88.4 53.87 67.43±0.22 67.94
R8 95.71 96.13 96.09 - 97.8 98 - 96.16 96.32±0.11 96.43
Table 5: Performance comparison of our model with other baseline models for short text classification.
Dataset DNN CNN RNN Stacking SVM HDLTex-CNN BERT XLNet TM TMconv TMconv(max)
WOS-5736 86.15 88.68 89.46 85.68 90.93 90.24 90 89.47 90.73±0.27 91.28


key: chroma:document
string_value:
Table 6: Performance comparison of our model with other baseline models for academic classification.
BERT, XLNet, and TM. However, the maximum accu-
racy from our proposed model outperformed HDLTex-
CNN by a slight margin.
5.1. Accuracy vs Window Size
We now use the PolitiFact dataset to investigate the
effect of increasing the window size Won accuracy.
As PolitiFact contains long sentences per example, our
model is able to achieve very high accuracy by utiliz-


key: chroma:document
string_value:
ing clauses to capture linguistic patterns in each win-
dow. However, for datasets with short sentences such
as Query and R8, it is not feasible to use more than two
windows since each window would contain only a few
words, rendering it useless for classification. Figure 3
illustrates the trade-off between window size and accu-
racy. We notice that as the window size increases, the
accuracy decreases. The highest accuracy is attained
whenW= 1, which means that the input text sentence


key: chroma:document
string_value:
has not been divided into any windows. The maximum
accuracy obtained for each dataset using W= 1 is
shown in Table 8. Because we are investigating the
local interpretability using parts of a text sentence, we
have to break the text sentence into windows. As a re-
sult,W= 2 andW= 3 are most suitable for us, as
their accuracy is comparable to that of W= 1, with
the added benefit of local interpretation. From this ex-
periment, we can conclude that the length of the text


key: chroma:document
string_value:
sentence determines the window size. That is, if the
dataset contains lengthy sentences, we may increase
the window size without compromising accuracy.
6. Interpretability Analysis
We examine the interpretability of our framework using
the R8 dataset. Here, we consider both global and lo-
cal interpretability. We define global interpretability as
identifying global features from the trained model that
contribute significantly to classification. In contrast, lo-


key: chroma:document
string_value:
cal interpretability refers to identifying significant fea-
tures inside a particular window. For example, consider
the following input sentence from the R8 dataset, “Tai-
wan shipbuilder looks for Japanese ventures Taiwan’s
state-owned China shipbuilding corp csbc plans to seek
joint production agreements with Japan and further di-
versify into ship repairing. ” with the classification label
“ship”. We divide the sentence into windows. Using a


key: chroma:document
string_value:
window size of two, the sentence can be separated into
two phrases: Window 1 :[Taiwan shipbuilder looks forJapanese ventures Taiwan’s state-owned China ship-
building corp] andWindow 2 :[csbc plans to seek joint
production agreements with Japan and further diversify
into ship repairing] . As a result, we can observe that
the first window is sufficient to classify the text. There-
fore, through local interpretability, our proposed model
determines which section of the sentence is relevant for


key: chroma:document
string_value:
classification. Additionally, global interpretability can
be used to highlight the relevant features inside the win-
dow as seen in Table 7.
We split the R8 dataset into two windows for our exper-
iment, given the dataset contains short sentences. That
is, as seen in Figure 1, each example sentence is split
into two local parts. The model is trained on the pre-
processed dataset, and clause-specific features are ex-
tracted. We note that certain clauses capture features


key: chroma:document
string_value:
of all the windows, whereas others capture features of
a particular window. This suggests that we can distin-
guish between features that have a global impact and
the ones that are responsible for a particular classifi-
cation on a local location-aware level. To demonstrate
the local interpretability, we visualize the features from
two windows for “ acq” class in Figure 4. The fea-
tures in each window signify the classification decision
based on the corresponding part of the sentence. That


key: chroma:document
string_value:
is, we can know which part of the sentence is essential
for the classification.
7. Conclusion
In this paper, we introduce an explainable Convolu-
tion Tsetlin Machine (CTM) architecture for text clas-
sification. Due to the fact that CTM is optimized for
image-based tasks, we preprocess the text document
into various windows. Thus, the clauses from the CTM
can capture the semantic features associated with each
particular window’s word patterns. We propose us-


key: chroma:document
string_value:
particular window’s word patterns. We propose us-
ing predefined fixed-length vocabulary features derived
from the standard Bidirectional Encoder Representa-
tions from Transformers (BERT) tokens. This allows
us to mitigate the TM dependency on vocabulary for
each dataset and the out-of-vocabulary problem. We
illustrate the effectiveness of our model on four clas-
sification tasks using seven different datasets. Our ex-
perimental findings demonstrate that our approach is


key: chroma:document
string_value:
superior to or competitive with more sophisticated and
non-transparent approaches, including BERT and XL-
Net. We then investigate the effect of different window


key: chroma:document
string_value:
3768acq crude earn grain interest money-fx ship trade
pointing ##connect accomplish island tesla kruger farmhouse ##ities
doping thorns ##sby treating ##oko ashton road harlow
##rified notably looting surface linux tonight agency pianos
evergreen ##enter ##osed facebook bed ##sitor ##voking sanskrit
phone dramatic endelle trail slams becker ##met plunged
premiere chloride confrontation ##emia photo bed rockies ##cured
demonstrated racers temporarily summer mural burning trafficking script


key: chroma:document
string_value:
collections fountain werewolf road vaccines handicap likeness theories
unwilling ##lon segregation garbage families lightning flaming ineffective
##liest safety presiding ##nity bronze patent gesellschaft extensive
Table 7: Significant features captured by our model for each class in R8 dataset.
(a) Window 0.
 (b) Window 1.
Figure 4: Wordcloud visualization of local interpretability for “ acq”.
Dataset Accuracy (max)
PolitiFact 92.46
GossipCop 85.94
BBCSports 99.54
Twitter 73.67
Query 68.25


key: chroma:document
string_value:
BBCSports 99.54
Twitter 73.67
Query 68.25
R8 96.93
WOS-5736 92.42
Table 8: Maximum accuracy obtained using window
size 1 (i.e., W= 1).
sizes on our model’s performance. Additionally, we
illustrate the global and local interpretability provided
by the CTM depending on the significance of features
captured by clauses in separate windows.
8. Bibliographical References
Agarwal, R., Frosst, N., Zhang, X., Caruana, R., and
Hinton, G. E. (2020). Neural additive models: In-


key: chroma:document
string_value:
Hinton, G. E. (2020). Neural additive models: In-
terpretable machine learning with neural nets. arXiv
preprint:2004.13912 .
Agrawal, R., Imieli ´nski, T., and Swami, A. (1993).
Mining association rules between sets of items in
large databases. In Proceedings of ACM SIGMOD
international conference on Management of data ,
pages 207–216.
Badue, C., Guidolini, R., Carneiro, R. V ., Azevedo, P.,
Cardoso, V . B., Forechi, A., Jesus, L., Berriel, R.,


key: chroma:document
string_value:
Paixao, T. M., Mutz, F., et al. (2020). Self-driving
cars: A survey. Expert Systems with Applications ,
page 113816.
Bai, B., Liang, J., Zhang, G., Li, H., Bai, K., and Wang,F. (2020). Why is attention not so attentive? arXiv
preprint:2006.05656 .
Berge, G. T., Granmo, O.-C., Tveit, T. O., Goodwin,
M., Jiao, L., and Matheussen, B. V . (2019). Us-
ing the tsetlin machine to learn human-interpretable
rules for high-accuracy text categorization with med-


key: chroma:document
string_value:
ical applications. IEEE Access , 7:115134–115146.
Bhattarai, B., Granmo, O.-C., and Jiao, L. (2021a).
Measuring the novelty of natural language text us-
ing the conjunctive clauses of a Tsetlin machine text
classifier. In Proceedings of ICAART .
Bhattarai, B., Granmo, O.-C., and Jiao, L. (2021b).
Explainable tsetlin machine framework for fake
news detection with credibility score assessment.
arXiv preprint:2105.09114 .
Bhattarai, B., Granmo, O.-C., and Jiao, L. (2022).


key: chroma:document
string_value:
Word-level human interpretable scoring mechanism
for novel text detection using tsetlin machines. Ap-
plied Intelligence .
Blakely, C. D. and Granmo, O.-C. (2020). Closed-
form expressions for global and local interpretation
of tsetlin machines with applications to explaining
high-dimensional data. arXiv preprint:2007.13885 .
Bojanowski, P., Grave, E., Joulin, A., and Mikolov, T.
(2017). Enriching word vectors with subword infor-
mation. Transactions of the ACL , 5:135–146.


key: chroma:document
string_value:
mation. Transactions of the ACL , 5:135–146.
Breiman, L. (2001). Random forests. Machine learn-
ing, 45(1):5–32.
Chen, H. and Ji, Y . (2020). Learning variational word
masks to improve the interpretability of neural text
classifiers. In Proceedings of EMNLP .
Darshana Abeyrathna, K., Granmo, O.-C., Zhang, X.,
Jiao, L., and Goodwin, M. (2020). The regression
tsetlin machine: a novel approach to interpretable


key: chroma:document
string_value:
3769nonlinear regression. Philosophical Transactions of
the Royal Society A , 378(2164):20190165.
Debole, F. and Sebastiani, F. (2005). An analysis of the
relative hardness of reuters-21578 subsets. Journal
of the American Society for Information Science and
technology , 56(6):584–596.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova,
K. (2019a). Bert: Pre-training of deep bidirectional
transformers for language understanding. In Pro-
ceedings of NAACL-HLT .


key: chroma:document
string_value:
ceedings of NAACL-HLT .
Devlin, J., Chang, M.-W., Lee, K., and Toutanova,
K. (2019b). Bert: Pre-training of deep bidirectional
transformers for language understanding. In Pro-
ceedings of NAACL-HLT .
Esteva, A., Robicquet, A., Ramsundar, B., Kuleshov,
V ., DePristo, M., Chou, K., Cui, C., Corrado, G.,
Thrun, S., and Dean, J. (2019). A guide to deep
learning in healthcare. Nature medicine , 25(1):24–
29.
Faruqui, M. and Das, D. (2018). Identifying well-


key: chroma:document
string_value:
Faruqui, M. and Das, D. (2018). Identifying well-
formed natural language questions. In Proceedings
of EMNLP , pages 798–803.
Feldman, V . (2009). Hardness of approximate two-
level logic minimization and pac learning with mem-
bership queries. Journal of Computer and System
Sciences , 75(1):13–26.
Freitas, A. A., Wieser, D. C., and Apweiler, R. (2008).
On the importance of comprehensible classification
models for protein function prediction. IEEE/ACM


key: chroma:document
string_value:
models for protein function prediction. IEEE/ACM
Transactions on Computational Biology and Bioin-
formatics , 7(1):172–182.
Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter,
M., and Kagal, L. (2018). Explaining explanations:
An overview of interpretability of machine learning.
InProceedings of IEEE DSAA , pages 80–89. IEEE.
Granmo, O.-C., Glimsdal, S., Jiao, L., Goodwin, M.,
Omlin, C. W., and Berge, G. T. (2019). The convo-
lutional tsetlin machine. arXiv preprint:1905.09688 .


key: chroma:document
string_value:
Granmo, O.-C. (2018). The Tsetlin machine - a
game theoretic bandit driven approach to optimal
pattern recognition with propositional logic. arXiv
preprint:1804.01508 .
Jacovi, A., Shalom, O. S., and Goldberg, Y . (2018).
Understanding convolutional neural networks for
text classification. In Proceedings of EMNLP Work-
shop BlackboxNLP .
Joulin, A., Grave, ´E., Bojanowski, P., and Mikolov, T.
(2017). Bag of tricks for efficient text classification.
InProceedings of EACL , pages 427–431.


key: chroma:document
string_value:
InProceedings of EACL , pages 427–431.
Kim, Y . (2014). Convolutional neural networks for
sentence classification. In Proceedings of EMNLP .
Kowsari, K., Brown, D. E., Heidarysafa, M.,
Meimandi, K. J., Gerber, M. S., and Barnes, L. E.
(2017a). Hdltex: Hierarchical deep learning for
text classification. In Proceedings of IEEE ICMLA ,
pages 364–371.
Kowsari, K., Brown, D. E., Heidarysafa, M.,
Meimandi, K. J., Gerber, M. S., and Barnes, L. E.(2017b). Hdltex: Hierarchical deep learning for


key: chroma:document
string_value:
text classification. In Proceedings of IEEE ICMLA ,
pages 364–371.
Lage, I., Chen, E., He, J., Narayanan, M., Kim, B.,
Gershman, S., and Doshi-Velez, F. (2019). An eval-
uation of the human-interpretability of explanation.
CoRR , abs/1902.00006.
Le, H. H. and Viviani, J.-L. (2018). Predicting
bank failure: An improvement by implementing a
machine-learning approach to classical financial ra-
tios. Research in International Business and Fi-
nance , 44:16–25.


key: chroma:document
string_value:
nance , 44:16–25.
Lei, J., Rahman, T., Shafik, R., Wheeldon, A.,
Yakovlev, A., Granmo, O.-C., Kawsar, F., and
Mathur, A. (2021). Low-power audio keyword spot-
ting using tsetlin machines. Journal of Low Power
Electronics and Applications , 11(2):18.
Li, B., Friedman, J., Olshen, R., and Stone, C. (1984).
Classification and regression trees (cart). Biomet-
rics, 40(3):358–361.
Lin, J., Tremblay-Taylor, G., Mou, G., You, D., and
Lee, K. (2019a). Detecting fake news articles. In


key: chroma:document
string_value:
Lee, K. (2019a). Detecting fake news articles. In
Proceedings of 2019 IEEE International Conference
on Big Data .
Lin, J., Tremblay-Taylor, G., Mou, G., You, D., and
Lee, K. (2019b). Detecting fake news articles. In
Proceedings of 2019 IEEE International Conference
on Big Data .
Lipton, Z. C. (2018). The mythos of model inter-
pretability: In machine learning, the concept of in-
terpretability is both important and slippery. Queue ,
16(3):31–57.


key: chroma:document
string_value:
16(3):31–57.
Liu, Y ., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D.,
Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov,
V . (2019). Roberta: A robustly optimized BERT
pretraining approach. CoRR , abs/1907.11692.
Mikolov, T., Sutskever, I., Chen, K., Corrado, G.,
and Dean, J. (2013). Distributed representations of
words and phrases and their compositionality. In
Proceedings of NIPS .
Nguyen, D. (2018). Comparing automatic and human
evaluation of local explanations for text classifica-


key: chroma:document
string_value:
tion. In Proceedings of NAACL-HLT (Long Papers) ,
pages 1069–1078.
Pennebaker, J., Boyd, R. L., Jordan, K., and Blackburn,
K. G. (2015). The development and psychometric
properties of liwc2015. Technical report.
Pennington, J., Socher, R., and Manning, C. D. (2014).
Glove: Global vectors for word representation. In
Proceedings of EMNLP , pages 1532–1543.
Ribeiro, M. T., Singh, S., and Guestrin, C. (2016). ”
why should i trust you?” explaining the predictions


key: chroma:document
string_value:
of any classifier. In Proceedings of ACM SIGKDD .
Rubin, V . L., Conroy, N., and Chen, Y . (2015). To-
wards news verification: Deception detection meth-
ods for news discourse. In Proceedings of the Rapid
Screening Technologies, Deception Detection and
Credibility Assessment Symposium, at the 48th An-


key: chroma:document
string_value:
3770nual Hawaii International Conference on System
Sciences .
Saha, R., Granmo, O.-C., and Goodwin, M. (2020).
Mining interpretable rules for sentiment and seman-
tic relation analysis using tsetlin machines. In Pro-
ceedings of International Conference on Innova-
tive Techniques and Applications of Artificial Intel-
ligence .
Serrano, S. and Smith, N. A. (2019). Is attention inter-
pretable? In Proceedings of ACL , pages 2931–2951.
Shu, K., Mahudeswaran, D., Wang, S., Lee, D., and


key: chroma:document
string_value:
Shu, K., Mahudeswaran, D., Wang, S., Lee, D., and
Liu, H. (2020a). Fakenewsnet: A data repository
with news content, social context, and spatiotempo-
ral information for studying fake news on social me-
dia.Big data , 83:171–188.
Shu, K., Zheng, G., Li, Y ., Mukherjee, S., Awadallah,
A. H., Ruston, S. W., and Liu, H. (2020b). Lever-
aging multi-source weak social supervision for early
detection of fake news. arXiv preprint:2004.01732 .
Strumbelj, E. and Kononenko, I. (2010). An efficient


key: chroma:document
string_value:
explanation of individual classifications using game
theory. The Journal of Machine Learning Research ,
11:1–18.
Tsetlin, M. L. (1961). On behaviour of finite au-
tomata in random medium. Avtomat. i Telemekh ,
22(10):1345–1354.
Ustun, B. and Rudin, C. (2016). Supersparse linear in-
teger models for optimized medical scoring systems.
Machine Learning , 102(3):349–391.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J.,
Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin,


key: chroma:document
string_value:
I. (2017). Attention is all you need. In Proceedings
of NIPS , pages 6000–6010.
Xiong, W., Ni’mah, I., Huesca, J. M., van Ipenburg, W.,
Veldsink, J., and Pechenizkiy, M. (2018). Looking
deeper into deep learning model: attribution-based
explanations of textcnn. In Proceedings of NIPS
2018 Workshop on Challenges and Opportunities for
AI in Financial Services: the Impact of Fairness, Ex-
plainability, Accuracy, and Privacy .
Yadav, R., Jiao, L., Granmo, O.-C., and Goodwin,


key: chroma:document
string_value:
Yadav, R., Jiao, L., Granmo, O.-C., and Goodwin,
M. (2021a). Human-level interpretable learning for
aspect-based sentiment analysis. In Proceedings of
AAAI .
Yadav, R. K., Jiao, L., Granmo, O.-C., and Good-
win, M. (2021b). Distributed word representation
in tsetlin machine. arXiv preprint:2104.06901 .
Yadav, R. K., Jiao, L., Granmo, O.-C., and Goodwin,
M. (2021c). Interpretability in word sense disam-
biguation using tsetlin machine. In Proceedings of
ICAART , pages 402–409.


key: chroma:document
string_value:
ICAART , pages 402–409.
Yang, Z., Yang, D., Dyer, C., He, X., Smola, A., and
Hovy, E. (2016). Hierarchical attention networks
for document classification. In Proceedings of HLT-
NAACL .
Yang, Z., Dai, Z., Yang, Y ., Carbonell, J., Salakhutdi-
nov, R., and Le, Q. V . (2019a). Xlnet: Generalizedautoregressive pretraining for language understand-
ing. In Proceedings of NIPS .
Yang, Z., Dai, Z., Yang, Y ., Carbonell, J., Salakhutdi-
nov, R., and Le, Q. V . (2019b). Xlnet: Generalized


key: chroma:document
string_value:
The Regression Tsetlin Machine: A Tsetlin
Machine for Continuous Output Problems
K. Darshana Abeyrathna, Ole-Christoﬀer Granmo, Lei Jiao, and
Morten Goodwin
Centre for Artiﬁcial Intelligence Research, University of Agder, Grimstad, Norway
{darshana.abeyrathna, ole.granmo, lei.jiao, morten.goodwin }@uia.no
Abstract. The recently introduced Tsetlin Machine (TM) has provided
competitive pattern classiﬁcation accuracy in several benchmarks, com-


key: chroma:document
string_value:
posing patterns with easy-to-interpret conjunctive clauses in proposi-
tional logic. In this paper, we go beyond pattern classiﬁcation by in-
troducing a new type of TMs, namely, the Regression Tsetlin Machine
(RTM). In all brevity, we modify the inner inference mechanism of the
TM so that input patterns are transformed into a single continuous out-
put, rather than to distinct categories. We achieve this by: (1) using the
conjunctive clauses of the TM to capture arbitrarily complex patterns;


key: chroma:document
string_value:
(2) mapping these patterns to a continuous output through a novel vot-
ing and normalization mechanism; and (3) employing a feedback scheme
that updates the TM clauses to minimize the regression error. The feed-
back scheme uses a new activation probability function that stabilizes
the updating of clauses, while the overall system converges towards an
accurate input-output mapping. The performance of the RTM is eval-
uated using six diﬀerent artiﬁcial datasets with and without noise, in


key: chroma:document
string_value:
comparison with the Classic Tsetlin Machine (CTM) and the Multiclass
Tsetlin Machine (MTM). Our empirical results indicate that the RTM
obtains the best training and testing results for both noisy and noise-
free datasets, with a smaller number of clauses. This, in turn, translates
to higher regression accuracy, using signiﬁcantly less computational re-
sources.
Keywords: Tsetlin Machine, Regression Tsetlin Machine, Tsetlin Automata,
Regression, Pattern Recognition, Propositional Logic.


key: chroma:document
string_value:
1 Introduction
Computational simplicity, ease of interpretation, along with competitive pattern
recognition accuracy, make the recently introduced Tsetlin Machine (TM) [1] a
promising new paradigm for machine learning. Indeed, the TM has outperformed
well-known machine learning algorithms such as Logistic Regression, Neural Net-
works, and Support Vector Machine (SVM) in several benchmarks, including Iris
Data Classiﬁcation, Handwritten Digits Classiﬁcation (MNIST), Predicting Op-


key: chroma:document
string_value:
timum Moves in the Axis and Allies Board Game, and Classiﬁcation of Noisy
XOR Data with Non-Informative Features [1].arXiv:1905.04206v2  [cs.LG]  21 Jun 2019


key: chroma:document
string_value:
Tsetlin Automata and the Tsetlin Machine. The core of the TM is built
on Tsetlin Automata (TAs), developed by M. L. Tsetlin in the early 1960s [2].
This powerful, yet simple, leaning mechanism has been used to solve a number
of machine learning and stochastic optimization problems, such as resource allo-
cation [3], stochastic searching on the line [4], distributed coordination [5], graph
coloring [6], and forecasting disease outbreaks [7]. In the TM, TAs represent lit-


key: chroma:document
string_value:
erals – input features and their negations. The literals, in turn, form conjunctive
clauses in propositional logic, as decided by the TAs. The ﬁnal TM output is
a disjunction of all the speciﬁed clauses. In this manner, the pattern composi-
tion and learning procedure of the TM is fully transparent and understandable,
facilitating human interpretation. In addition, the TM has an inherent compu-
tational advantage. That is, the inputs and outputs of the TM can naturally be


key: chroma:document
string_value:
represented as bits, and recognition and learning is performed by manipulating
those bits. The operation of the TM thus demands relatively small computa-
tional resources, and supports hardware-near and parallel computation e.g. on
GPUs.
Lately, the TM has provided state-of-the-art performance in several real-life
applications. Berge et al. have for instance successfully used the TM for medi-
cal text categorization [8]. They used the TM to provide interpretable pattern


key: chroma:document
string_value:
recognition for the analysis of electronic health records. The authors demon-
strated that the TM can outperform established machine learning algorithms
such as k-nearest neighbors (kNN), SVM, Random Forest, Decision Trees, Mul-
tilayer Perceptron (MLP), Long Short-Term Memory (LSTM) Neural Networks,
and Convolutional Neural Networks (CNNs), in terms of precision, recall, and
F-measure. Furthermore, Darshana et al. have shown that the TM can outper-


key: chroma:document
string_value:
form MLPs, Decision Trees, and SVMs in dengue fever outbreak prediction. The
latter result was achieved by making the TM capable of expressing thresholds
and intervals that capture patterns formed by continuous features. By carefully
selecting thresholds and intervals, the TM avoided losing information due to
binarization [9].
Research Question and Paper Contributions. The TM has been de-
signed for classiﬁcation, not for producing continuous output. How to best pro-


key: chroma:document
string_value:
duce continuous output is unclear, with the existing binarization schemes being
incapable of fully leveraging the natural ranking of numbers. In this paper, we
introduce the Regression Tsetlin Machine (RTM) to overcome above limitation
of the TM. The RTM is a novel variant of the Classic Tsetlin Machine (CTM),
speciﬁcally addressing the unique properties of regression. The novel modiﬁca-
tions that we introduce are subtle, but crucial. First of all, the clause polarities


key: chroma:document
string_value:
the CTM uses to discriminate patterns, using positive and negative examples,
are eliminated. Instead, the objective of the RTM is to use the clauses to map
the sum of the clause outputs into one single continuous output. The discrep-
ancy between predicted and target output is minimized with a new feedback
scheme tailored for regression, including a modiﬁed stochastic activation proba-
bility function.


key: chroma:document
string_value:
Paper Organization. The remainder of the paper is organized as follows. In
Section 2, we present the main contribution of this paper, which is the RTM, and
how we build it upon the CTM. We then investigate the behavior of the RTM
using six diﬀerent artiﬁcial datasets in Section 3. We demonstrate empirically
that the RTM is superior both to the CTM as well as its multiclass version when
it comes to predicting continuous output. We conclude our work in Section 4.


key: chroma:document
string_value:
2 The Regression Tsetlin Machine (RTM)
The RTM is a novel variant of the CTM. To highlight the unique properties of
the RTM, we start this section with ﬁrst reviewing the TM in more detail, and
then discuss how it can be modiﬁed to support continuous output.
2.1 The Classic Tsetlin Machine (CTM)
At the heart of the TM, we ﬁnd multiple teams of TAs that build conjunctive
clauses in propositional logic. The purpose is to capture hidden patterns in the
data.


key: chroma:document
string_value:
data.
Learning with TAs. Each Tsetlin Automaton (TA) learns the optimal
action in an environment by sequentially performing the actions that the envi-
ronment oﬀers. To identify the optimal actions, the TAs adjust their states based
on the feedback they receive from the environment, which can be penalties or
rewards. Asymptotically, a TA identiﬁes the action that provides the highest
probability of reward [10, 11]. These simple learning devices are capable of on-


key: chroma:document
string_value:
line learning, have a simple structure, and require modest computational power.
Yet, they are able to learn accurately with relatively few interactions with the
environment [12,13].
Clause Formation and the TA Team. The TM bases its operations on
the simplest form of TAs, namely, the two action one, with ﬁnite memory depth.
As illustrated in Table 1, a team of TAs cooperates to form a clause. The table
depicts the steps leading to a clause being formed. Consider an input feature


key: chroma:document
string_value:
Table 1. The steps used to form a clause based on the input features and the actions
of the TAs.
Phase  Operations  Comment s 
1 𝑥1 𝑥2 ……………  𝑥𝑜 Input Features, 𝑿 
2 𝑥1 ¬𝑥1 𝑥2  ¬𝑥2  𝑥𝑜 ¬𝑥𝑜 Literals  
3 TA11 TA12 TA21 TA22 ……… …….  TA 𝑜1 TA 𝑜2 TA 
4 in ex ex in ……………….  ex in Actions  = {in, ex} 
5 𝑥1 Ʌ ¬𝑥2 Ʌ………Ʌ  ¬𝑥𝑜 c1 = ⋀ 𝑥𝑘𝑜
𝑘=1 ,    ∀ TA𝑘1 = in  
 
c2 = ⋀ ¬𝑥𝑘𝑜
𝑘=1 ,   ∀ TA𝑘2 = in 
C = c1 Ʌ c2 
 
Phase  Operations  Comments  
1 x1 x2 ……………  Xn X = [x1, x2, x3, ……. Xn] 
2 x1 ¬ x1 x2  ¬ x2


key: chroma:document
string_value:
2 x1 ¬ x1 x2  ¬ x2  
xn ¬ xn Literals  
3 TA11 TA12 TA21 TA22 …………….  TA 𝑛1 TA 𝑛2 TA 
4 in ex ex in ……………….  ex in Actions = { in, ex} 
5 x1 Ʌ ¬ x2 Ʌ………Ʌ  ¬ xo c1 = ⋀ 𝑥𝑘𝑛
𝑘=1,    ∀ TA𝑘1 = in  
 
c2 = ⋀ ¬𝑥𝑘𝑛
𝑘=1 ,   ∀ TA𝑘2 = in 
 
C = c1 Ʌ c2 
 
 
 
∀ 
1 0 0  0 1 0  
0 0 1  0 1 1  1 0 1 1 1 0  
1 1 1  0 0 0


key: chroma:document
string_value:
vectorX= [x1, x2, . . . , x o]. Each TA represents either an input feature xkor
its negation¬xk(jointly referred to as literals). Further, each TA in the team
decides whether to include or exclude its assigned literal in the clause that the
team is forming. Accordingly, when there are oinput features, 2 ×oTAs are
needed to form the clause. The two actions available to each TA are {in,ex}.
Here, inrefers to including the literal controlled by the TA and exrefers to


key: chroma:document
string_value:
excluding it. As seen in the ﬁnal step in the table, the included literals form a
conjunctive clause, while the excluded ones are ignored.
Clauses and Voting. The number of clauses, m, needed for a particular
problem depends on the complexity of the dataset. It should at least be suﬃ-
cient to cover the full range of sub-patterns associated with each output {0, 1}.
However, with hidden and unknown sub-patterns, a grid search is required to
ﬁnd the best m.


key: chroma:document
string_value:
ﬁnd the best m.
Themclauses are assigned either a positive or negative polarity, and they
vote separately to decide the ﬁnal output of the TM. Clauses with odd index
are assigned positive polarity ( C+) and they vote for the ﬁnal output 1. Clauses
with even index are assigned negative polarity ( C−) and they vote for the ﬁnal
output 0. For both categories, a vote is submitted when the clause recognizes
a sub-pattern. If the clause is unable to ﬁnd a sub-pattern, it declines to vote.


key: chroma:document
string_value:
Finally, the output, y, is decided based on the number of votes gained by each
category{0, 1}as given in the Eq. (1):
y=

1, if∑
j=1,3,m−1C+
j>∑
j=2,4,mC−
j
0, if∑
j=1,3,m−1C+
j<∑
j=2,4,mC−
j.(1)
Learning Procedure. Learning in the TM is based on reinforcement learn-
ing. The reward, penalty, and inaction probabilities that guide the TAs in all
of the clauses depend on several factors, namely, the actual output, the clause


key: chroma:document
string_value:
output, the literal value, and the current state of the TA. The basic idea is to
alter the number of votes belong to each output category when the output is a
false negative or a false positive. In the TM, this is done by two types of feed-
back – Type I and Type II. Type I feedback eliminates false negative output and
reinforces true positive output, while Type II feedback eliminates false positive
output. Both of these kinds of feedback are summarized in Table 2.


key: chroma:document
string_value:
Type I feedback is given to clauses with positive polarity when the actual out-
put, ˆy, is 1 and clauses with negative polarity when the actual output, ˆ y, is 0. The
probability of activation of Type I feedback is [ T−max(−T,min(T,∑m
j=1Cj))]/2T.
Type II feedback is given to clauses with positive polarity when the actual out-
put, ˆy, is 0 and clauses with negative polarity when the actual output, ˆ y, is 1. The
probability of activation of Type II feedback is [ T+max(−T,min(T,∑m
j=1Cj))]/2T.


key: chroma:document
string_value:
j=1Cj))]/2T.
TAs remain unchanged if the vote diﬀerence,∑m
j=1Cj, is higher than or equal
toTwhen ˆ y= 1 and lower than or equal to - Twhen ˆ y= 0, according to the
activation probabilities of each type of feedback.
In all brevity, when the target output for a training instance ˆXis ˆy= 1, the
votes from the clauses with negative polarity must not outnumber the votes from


key: chroma:document
string_value:
the clauses with positive polarity (in order to correctly classify the instance).
Therefore, clauses with positive polarity receive Type I feedback (the activation
probability increases with the number of voting clauses with negative polarity)
since this reinforces clauses which output 1. Similarly, clauses with negative
polarity receive Type II feedback (the activation probability increases with the
number of voting clauses with positive polarity) since this suppresses voting


key: chroma:document
string_value:
activity by making clauses of negative polarity evaluate to 0. The procedure
is similar when the target output is ˆ y= 0. The TM then needs to make sure
that more clauses with negative polarity provide votes compared to those with
positive polarity. Eventually, the above feedback reduces the number of false
positives and false negatives to make the TM learn the propositional formulae
that provide high accuracy output.
2.2 The Multiclass Tsetlin Machine (MTM)


key: chroma:document
string_value:
2.2 The Multiclass Tsetlin Machine (MTM)
For the CTM, the ﬁnal summation operator aggregates all of the clause outputs
into one of the two available outputs: 0 or 1. However, for categorization tasks
with more classes than two, another design is needed. In the Multiclass Tsetlin
Machine (MTM), clauses are partitioned equally among the classes. The clauses
of each individual class then act separately, similarly to a single TM. However,


key: chroma:document
string_value:
the votes output for each class then form the basis for classiﬁcation. That is, an
argmax operator arbitrates the ﬁnal class, based on the votes collected for each
class. When there are nclasses, the output ycan thus be expressed as:
y= argmaxi=1,...,n{(∑
j=1,3,...(m
n)−1Ci
j−∑
j=2,4,...(m
n)Ci
j)}
. (2)
The training procedure is similar to the CTM training procedure. However,
in the MTM, the clauses of the class being the target of the current training


key: chroma:document
string_value:
sample are treated as if ˆ y= 1, while the clauses of a randomly selected class
from the remaining classes is treated as if ˆ y= 0. In each class, clauses with
positive polarity vote to say that the output belongs to the considered class.
Table 2. Type I and Type II feedback designed to eliminate false negative and false
positive output.
Feedback Type I II
Clause Output 1 0 1 0
Literal Value 1 0 1 0 1 0 1 0Current StateIncludeReward Probability (s-1)/s NA 0 0 0NA 0 0


key: chroma:document
string_value:
Inaction Probability 1/s NA (s-1)/s (s-1)/s 1NA 1 1
Penalty Probability 0 NA 1/s 1/s 0NA 0 0
ExcludeReward Probability 0 1/s 1/s 1/s 0 0 0 0
Inaction Probability 1/s (s-1)/s (s-1)/s (s-1)/s 1 0 1 1
Penalty Probability (s-1)/s 0 0 0 0 1 0 0
†s is the precision and controls the granularity of the sub-patterns captured [1]


key: chroma:document
string_value:
Similarly, the clauses with negative polarity vote to indicate that the output
does not belong to the considered class.
2.3 The Regression Tsetlin Machine (RTM)
When the output is continuous, neither the CTM or the MTM above are ideal.
However, we will now show that the CTM can be modiﬁed to produce continuous
output by means of three pertinent modiﬁcations.
In CTM and MTM, the polarity of clauses is used to classify data into dif-


key: chroma:document
string_value:
ferent classes. We now remove the polarity of clauses, since we intend to use
the clauses as additive building blocks that can be used to calculate continuous
output. That is, we intend to map the total vote count into a single continuous
output. As a result, the complexity of the RTM is actually reduced.
With merely one type of clauses, the summation operator outputs a value
between 0 and T, which is simply the number of clauses that evaluates to 1. This


key: chroma:document
string_value:
value is then normalized to produce the regression output. Thus, through this
simple modiﬁcation, the TM can now produce continuous output, with precision
that increases with higher T.
Let ˆymaxdenote the maximum output value ˆ yamong the Ntraining samples
Y= [ˆy1,ˆy2,ˆy3, . . . , ˆyN]. Then the sum of the votes from the clauses∑m
j=1Cj
of the TM is normalized to achieve the regression output by dividing by Tand
multiplying with ˆ ymax. So, for the othtraining sample, ( ˆXo,ˆyo), the TM output,


key: chroma:document
string_value:
yo, is calculated from the input ˆXoas follows:
yo=∑m
j=1Cj(ˆXo)×ˆymax
T. (3)
Feedback, then, is based on comparing the output, yoof the TM with the
target output ˆ yo. The target value ˆ yocan be higher or lower than the output
value yo. This is our basis for our new feedback scheme. That is, similarly to other
machine learning methods, certain internal operations are needed to minimize
the error between the predicted output, yo, and target output, ˆ yo. In the RTM,


key: chroma:document
string_value:
this is quite simply achieved by providing Type I and Type II feedbacks according
to the following criteria:
Feedback =

Type I , ifyo<ˆyo,
Type II , ifyo>ˆyo.(4)
As with the CTM, the idea here is to increase the number of clauses that
output 1 when the predicted output is less than the target output ( yo<ˆyo). To
achieve this, we then provide Type I feedback. Conversely, Type II feedback is
applied to decrease the number of clauses that evaluate to 1 when the predicted


key: chroma:document
string_value:
output is higher than the target output ( yo>ˆyo).
To stabilize learning, we use an activation probability function that makes
the probability of giving a clause feedback proportional to the diﬀerence between


key: chroma:document
string_value:
the predicted and target output (the error). That is, in the RTM, feedback to
clauses is determined stochastically using the following activation probability
function, Pact:
Pact=K×|yo−ˆyo|
ˆymax. (5)
As seen, the magnitude of the function is adjusted with the constant K. The
resulting activation function reduces the oscillation of the the predicted value
during the training process, stabilizing it around the target value.


key: chroma:document
string_value:
The behavior of the RTM is studied in the following sections, in comparison
with the CTM and MTM.
3 Empirical Results
3.1 Experiment Setup
We study the behavior of the RTM using six diﬀerent datasets. These datasets
have been constructed to facilitate empirical analysis of the optimality of RTM
learning, with the underlying input-output mapping being known. Dataset I
contains 2-bit feature input. The output is 100 times larger than the decimal


key: chroma:document
string_value:
value of the binary input (e.g., when the input is [1, 0], the output is 200).
The training set consists of 8000 samples while the testing set consists of 2000
samples, both without noise. Dataset II contains the same data as Dataset I,
except that the output of the training data is perturbed to introduce noise. For
Dataset III we introduce 3-bit input, without noise, and for Dataset IV we have
3-bit input with noisy output. Finally, Dataset V has 4-bit input without noise,


key: chroma:document
string_value:
and Dataset VI has 4-bit input with noisy.
Each input feature have been generated independently with equal probability
of 0 and 1 values, leading to a more or less uniform distribution of bit values.
In order to increase our understanding of the RTM, we investigate the eﬀect
the hyper-parameters Tandshave on learning.
Experiment I : We ﬁrst study the eﬀect varying Thas on performance for the
diﬀerent datasets.
Experiment II : The eﬀect of diﬀerent svalues (controlling the number of


key: chroma:document
string_value:
sub-patterns) is further investigated for all of the datasets.
Experiment III : We ﬁnally compare the RTM results with what can be
achieved with CTM and MTM.
3.2 Results and Discussion
We use Mean Absolute Error ( MAE ) to measure performance. Fig. 1 plots error
across 200 epochs, with learning inﬂuenced by diﬀerent Tvalues. Fig. 1(a) shows


key: chroma:document
string_value:
0102030
1
11
21
31
41
51
61
71
81
91
101
111
121
131
141
151
161
171
181
191MAEa: Dataset -I 3-T(0.00) 10-T(7.80)
30-T(0.00) 100-T(0.78)
500-T(0.50) 1000-T(0.20)
4000-T(0.33)
050100
1
11
21
31
41
51
61
71
81
91
101
111
121
131
141
151
161
171
181
191MAEb: Dataset -II3-T(7.15) 10-T(10.96)
30-T(8.78) 100-T(5.36)
500-T(5.53) 1000-T(5.23)
4000-T(5.36)
050100150
1
11
21
31
41
51
61
71
81
91
101
111
121
131
141
151
161
171
181
191MAEc: Dataset -III 7-T(0.00) 20-T(14.63)
70-T(0.00) 300-T(1.86)


key: chroma:document
string_value:
70-T(0.00) 300-T(1.86)
700-T(0.96) 2000-T(1.04)
4000-T(0.90)
050100150
1
11
21
31
41
51
61
71
81
91
101
111
121
131
141
151
161
171
181
191MAEd: Dataset -IV 7-T(7.34) 20-T(13.82)
70-T(6.60) 300-T(5.83)
700-T(5.87) 2000-T(5.59)
4000-T(5.54)
0102030
1
11
21
31
41
51
61
71
81
91
101
111
121
131
141
151
161
171
181
191MAE
Epoche: Dataset -V 7-T(9.84) 15-T(0.00)
70-T(1.69) 150-T(0.00)
700-T(0.24) 1500-T(0.15)
4000-T(0.16)
0100200300
1
11
21
31
41
51
61
71
81
91
101
111
121
131
141
151
161
171
181


key: chroma:document
string_value:
61
71
81
91
101
111
121
131
141
151
161
171
181
191MAE
Epochf: Dataset -VI 7-T(79.78) 15-T(51.43)
50-T(13.11) 150-T(10.26)
700-T(5.53) 1500-T(5.23)
4000-T(5.39)Fig. 1. Training error over training epochs. Each dataset is processed with diﬀerent T.
the results for Dataset I, Fig. 1(b) reports results for Dataset II, and so on. MAE
after 200 epochs is also given in brackets for each threshold in the legend.
From Fig. 1, we can observe that just 3 clauses ( T= 3) are enough to reduce


key: chroma:document
string_value:
error to zero for Dataset I, which can be explained by the noise-free data. Because
the output value is decided by the number of clauses that output 1, we require
twoclauses with TA1
1={in}, TA2
1={ex}, TA1
2={ex}, and TA2
2={ex}
to capture the pattern (1 .); see Phase 4 in Table 1. Further, we need one
clause with TA1
1={ex}, TA2
1={ex}, TA1
2={in}, and TA2
2={ex}to capture
the pattern ( .1). Here, .means an input feature that can take an arbitrary


key: chroma:document
string_value:
value, either 0 or 1. These three clauses can collectively form any outputs for the
Dataset I as shown in Table 3. For instance, input (0 1) only activates the clause
with TA1
1={ex}, TA2
1={ex}, TA1
2={in}, and TA2
2={ex}, which represents
the pattern ( .1). Accordingly, the RTM correctly computes the output, 100.
Likewise, input (1 0) only activates the twoclauses with TA1
1={in}, TA2
1=
{ex}, TA1
2={ex}, and TA2
2={ex}, which represent the pattern (1 .). Thus,


key: chroma:document
string_value:
2={ex}, which represent the pattern (1 .). Thus,
the output 200 is correctly computed. All the clauses are activated when the
input is (1 1) and therefore the output 300 is computed correctly as well.


key: chroma:document
string_value:
Table 3. Computing output for diﬀerent datasets by activating diﬀerent clauses.
Dataset Output Required number of clauses to represent diﬀerent patterns††
I0 None
100 1×(.1)
200 2×(1.)
300 2×(1.) + 1 ×(.1)
III0 None
100 1×(. . 1)
200 2×(.1.)
300 2×(.1.) + 1 ×(. . 1)
400 4×(1. .)
500 4×(1. .) + 1 ×(. . 1)
600 4×(1. .) + 2 ×(.1.)
700 4×(1. .) + 2 ×(.1.) + 1 ×(. . 1)
V0 None
100 1×(. . . 1)
200 2×(. . 1.)
300 2×(. . 1.) + 1 ×(. . . 1)
400 4×(.1. .)
500 4×(.1. .) + 1 ×(. . . 1)


key: chroma:document
string_value:
400 4×(.1. .)
500 4×(.1. .) + 1 ×(. . . 1)
600 4×(.1. .) + 2 ×(. . 1.)
700 4×(.1. .) + 2 ×(. . 1.) + 1 ×(. . . 1)
800 8×(1. . . )
900 8×(1. . . ) + 1 ×(. . . 1)
1000 8×(1. . . ) + 2 ×(. . 1.)
1100 8×(1. . . ) + 2 ×(. . 1.) + 1 ×(. . . 1)
1200 8×(1. . . ) + 4 ×(.1. .)
1300 8×(1. . . ) + 4 ×(.1. .) + 1 ×(. . . 1)
1400 8×(1. . . ) + 4 ×(.1. .) + 2 ×(. . 1.)
1500 8×(1. . . ) + 4 ×(.1. .) + 2 ×(. . 1.) + 1 ×(. . . 1)
††for example, “ twoclauses to represent the pattern (1 .)” is written as “2 ×(1.)”


key: chroma:document
string_value:
We observe similar behaviour for Dataset III and Dataset V. More speciﬁ-
cally, Dataset III requires seven clauses to represent the three diﬀerent patterns
it contains, namely, (4 ×(1. .), 2×(.1.), 1×(. .1))1. Further, Dataset
V requires ﬁfteen clauses to represent four diﬀerent patterns it contains (8 ×
(1. . . ), 4×(.1. .), 2×(. . 1.), 1×(. . . 1)). As we can see
from these 3 datasets, RTM can reach 0.00 for the training MAE when Tis a


key: chroma:document
string_value:
multiplier of the minimum required clauses. For example, Dataset I can also be
perfectly learned when there are 30 clauses.
However, when Tis not a multiplier of the minimum required clauses, RTM
cannot align its output yoto the target output ˆ yoduring the training phase.
For instance, by assigning four clauses for Dataset I, the training will end up
with e.g. allocating three clauses to represent the pattern (1 .) or twoclauses


key: chroma:document
string_value:
to represent the pattern ( .1). As a result, one or more output values cannot
be computed correctly. For example, if there are three clauses for the pattern (1
.) and oneclause for the pattern ( .1) after training, input (1 0) activates the
1In this expression, “ four clauses to represent the pattern (1 . .)” is written as “4
×(1. .)”


key: chroma:document
string_value:
0100200300400500600700
1 2 3 4 5 6 7 8 9 10MAE
sDataset-II (30-T)
Dataset-IV (70-T)
Dataset-VI (150-T)Fig. 2. Variation of MAE over diﬀerent sfor ﬁxed T.
Phase  Operations  Comment s 
1 x1 x2 ……………  xr X = [x1, x2, x3, ……. x r] 
2 x1 ¬ x1 x2  ¬ x2  
xr ¬ xr Literals  
3 TA11 TA12 TA21 TA22 ……… …….  TA𝑟1 TA𝑟2 TA 
4 in ex ex in ……………….  ex in Actions = { in, ex} 
5 x1 Ʌ ¬ x2 Ʌ………Ʌ  ¬ xr c1 = ⋀𝑥𝑖𝑟
𝑖=1,    ∀ TA𝑖1 = in  
 
c2 = ⋀¬𝑥𝑖𝑟
𝑖=1 ,   ∀ TA𝑖2 = in 
 
C = c1 Ʌ c2 
 
 
∀ 
1 0 0  0 1 0


key: chroma:document
string_value:
C = c1 Ʌ c2 
 
 
∀ 
1 0 0  0 1 0  
0 0 1  0 1 1  1 0 1 1 1 0  
1 1 1  0 0 0  
Fig. 3. Pattern distribution for the 3-bits input datasets.
clauses that represent the pattern (1 .), producing an incorrect output that is
300. Likewise, input (1 1) activates all four clauses to incorrectly compute the
output 400.
As a strategy for problems where the number of clauses is unknown, and
for real-world applications where noise plays a signiﬁcant role, the RTM can be


key: chroma:document
string_value:
initialized with a much larger T. Then, since the output, yo, is a fraction of the
threshold, T, the error decreases. This behaviour is veriﬁed empirically in Fig. 1,
showing how increasing Tleads to reduced error.
The eﬀect of sis studied by increasing it from 1 .0 to 10 .0 for Dataset II,
Dataset IV, and Dataset VI, with ﬁxed T. Fig. 2 shows the variation of MAE
over various svalues for noisy data. The MAE decreases when sincreases from


key: chroma:document
string_value:
1.0 to 2 .0. After 2 .0,MAE increases, and then stabilizes after a while.
For all of the datasets considered here, the optimum s, where the RTM learns
the datasets with minimum MAE , is equal to 2 .0. The reason can be explained
with the aid of Fig. 3, where one sees the distribution of patterns when the
dataset has 3 input bits.
The occurrence probability of any of the 3-bit patterns is1
8since there are
overall 8 unique patterns. However, to capture the pattern (1 . .) (shaded


key: chroma:document
string_value:
area), according to the TM dynamics [1],1
sshould be equal to the probability
of the considered pattern, which is4
8(=1
2). Hence, sshould be 2. For instance,
if someone assigns s= 4, clauses will start to learn much ﬁner patterns, such


key: chroma:document
string_value:
as (1 0 .), (1 1 .), and (0 1 .). This signiﬁcantly increases the number of
clauses needed to capture the sub-patterns. This is also the case for Dataset
II and Dataset VI. Then, the probability that (1 .) occurs is2
4(=1
2) and the
probability that (1 . . . ) occurs is7
14(=1
2).
To compare the performance of the RTM with CTM and MTM, each model
is tested with diﬀerent Tvalues. The training and testing MAE for all the cases
are summarized in Table 4 and 5, respectively.


key: chroma:document
string_value:
are summarized in Table 4 and 5, respectively.
Table 4. Training MAE after 200 training epochs with diﬀerent Ton various methods.
RTM CTM MTM
T 3 10 30 100 500 1000 4000 68000 1000 10000 16000Dataset1MAE 0.0 7.8 0.0 0.8 0.5 0.2 0.3 7.7 0.0 0.0 0.0 0.0
2MAE 7.2 11.0 8.8 5.4 5.5 5.2 5.4 11.1 24.1 8.4 7.1 7.9
T 7 20 70 300 700 2000 5000 14 8000 2000 10000 16000
3MAE 0.0 14.6 0.0 1.9 1.00 1.0 0.9 0.0 0.0 18.3 0.0 0.0
4MAE 7.4 13.8 6.6 5.8 5.9 5.6 5.5 111.3 13.3 14.2 8.8 8.4


key: chroma:document
string_value:
T 7 15 70 150 700 1500 4000 30 8000 4000 10000 16000
5MAE 9.8 0.0 1.7 0.0 0.2 0.2 0.2 149.7 158.7 373.1 0.0 0.0
6MAE 79.8 51.4 13.1 10.3 5.5 5.3 5.4 181.5 96.4 449.9 8.0 7.8
Table 5. Testing MAE for diﬀerent Ton various methods.
RTM CTM MTM
T 3 10 30 100 500 1000 4000 68000 1000 10000 16000Dataset1MAE 0.0 7.6 0.0 0.8 0.5 0.2 0.3 9.0 0.0 0.0 0.0 0.0
2MAE 5.0 10.6 7.1 1.2 2.7 1.6 1.8 9.4 25.3 7.5 5.4 7.0
T 7 20 70 300 700 2000 5000 14 8000 2000 10000 16000


key: chroma:document
string_value:
3MAE 0.0 14.2 0.0 2.1 1.0 1.2 1.0 0.0 0.0 22.0 0.0 0.0
4MAE 5.0 14.5 4.2 3.3 3.4 1.9 2.7 98.5 12.5 16.0 8.7 8.3
T 7 15 70 150 700 1500 4000 30 8000 4000 10000 16000
5MAE 9.9 0.0 1.8 0.0 0.3 0.2 0.2 154.6 155.5 372.9 0.0 0.0
6MAE 78.0 50.1 12.5 8.5 3.5 2.7 2.8 191.3 102.4 431.3 6.9 6.7
The training and testing MAE reach zero when the RTM operates with noise
free data and when Tequals the optimum clauses required. When the optimum


key: chroma:document
string_value:
Tis unknown, and when data is noisy, applying a higher Tis beneﬁcial. As an
example, Dataset III, which has 3 bits as inputs, can be perfectly learned with T
equal to 7 and 70. For the same dataset, RTM acquires training MAE 1.0 with
Tequaling 700, which is better than the MAE of 14.2 obtained when Tequals
20.
For CTM, the outputs are converted to bits and each bit position is then
trained and predicted separately. According to the training and testing MAE in


key: chroma:document
string_value:
I and Dataset III. However, with a higher number of inputs and with noisy
training data, performance decreases.
MTM requires a large number of clauses by nature when it works with con-
tinuous outputs since it has to consider all possible values from 0 to ˆ ymaxas
distinct classes (e.g. 300 classes for Dataset I and Dataset II, and 700 classes for
Dataset III and Dataset IV). According to the training and testing MAE in the


key: chroma:document
string_value:
Tables 4 and 5, MTM requires roughly 3 clauses or more per class. For instance,
the features in Dataset I can be learned with 1000 clauses, yet that amount is
insuﬃcient for Dataset III and Dataset V. Note that the noise free datasets can
be learned perfectly with 10000 or more clauses. However, this accuracy gain is
accompanied with a larger computational cost.
Overall, RTM obtains the best training and testing MAE for both noisy


key: chroma:document
string_value:
and noise free data with a smaller number of clauses compared with the CTM
and MTM. Dataset II, Dataset IV, and Dataset VI are more similar to real-
world datasets by being noisy. The minimum MAE values obtained by RTM
for these three Datasets are 1.6, 1.9, and 2.7, respectively. The average of these
minimum MAE values (2.07) is approximately 20 and 3.5 times lower than the
averages obtained with CTM and MTM, respectively. In terms of the number of


key: chroma:document
string_value:
clauses required to achieve the above results, RTM utilizes 1000 clauses, while
CTM and MTM utilize 8 and 16 times more clauses than that. This diﬀerence is
characteristic for RTM – it provides better MAE with less computational power.
4 Conclusion
In this paper we proposed the Regression Tsetlin Machine (RTM), a novel variant
of the Classic Tsetlin Machine that supports continuous output in regression
problems. In RTM, the polarities in clauses were removed and the total clause


key: chroma:document
string_value:
output was normalized to produce continuous output predictions. The number of
clauses to receive the feedback in RTM was decided stochastically using a linear
activation probability function. The prediction power of this novel approach was
studied using six diﬀerent datasets, with noise free and noisy training data. Our
empirical results showed signiﬁcantly better performance of RTM compared with
CTM and MTM, both in terms of training and the testing error, as well as the


key: chroma:document
string_value:
computational power required.
Potential applications for RTM can be weather prediction, sales forecasting,
stock predictions, energy forecasting, and outbreak forecasting, to name a few.
In our future work, we will evaluate RTM on the aforementioned applications
and performance will be compared with conventional machine learning methods.
References
1. O.-C. Granmo, “The Tsetlin Machine - A Game Theoretic Bandit Driven Ap-


key: chroma:document
string_value:
proach to Optimal Pattern Recognition with Propositional Logic,” arXiv e-prints,
p. arXiv:1804.01508 , Apr 2018
2. M. L. Tsetlin, ”On Behaviour of Finite Automata in Random Medium,” Avtom I
Telemekhanika , vol. 22, pp. 1345-1354, 1961.


key: chroma:document
string_value:
3. O.-C. Granmo and B. J. Oommen, ”Solving Stochastic Nonlinear Resource Alloca-
tion Problems Using a Hierarchy of Twofold Resource Allocation Automata,” IEEE
Transaction on Computers , vol. 59, no. 4, pp. 545-560, 2010.
4. B. J. Oommen, S.-W. Kim, M. T. Samuel, and O.-C. Granmo, ”A Solution to the
Stochastic Point Location Problem in Metalevel Nonstationary Environments,” IEEE
Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) , vol. 38, no.
2, pp. 466-476, 2008.


key: chroma:document
string_value:
2, pp. 466-476, 2008.
5. B. Tung and L. Kleinrock, ”Using Finite State Automata to Produce Self-
Optimization and Self-Control,” IEEE transactions on parallel and distributed sys-
tems, vol. 7, no. 4, pp. 439-448, 1996.
6. N. Bouhmala and O.-C. Granmo, ”Stochastic Learning for SAT-Encoded Graph
Coloring Problems,” International Journal of Applied Metaheuristic Computing
(IJAMC) , vol. 1, no. 3, pp. 1-19, 2010.
7. K. Abeyrathna, O.-C. Granmo, and M. Goodwin, ”A Novel Tsetlin Automata


key: chroma:document
string_value:
Scheme to Forecast Dengue Outbreaks in the Philippines,” 2018 IEEE 30th Interna-
tional Conference on Tools with Artiﬁcial Intelligence (ICTAI) , pp. 680- 685, IEEE,
2018.
8. G. T. Berge, O.-C. Granmo, T. Oddbjørn Tveit, M. Goodwin, L. Jiao, and B.
Viggo Matheussen, ”Using the Tsetlin Machine to Learn Human-Interpretable Rules
for High-Accuracy Text Categorization with Medical Applications,” arXiv e-prints,
p. arXiv:1809.04547 , Sep 2018.


key: chroma:document
string_value:
p. arXiv:1809.04547 , Sep 2018.
9. K. Abeyrathna, O.-C. Granmo, X. Zhang, and M. Goodwin, ”A Scheme for Con-
tinuous Input to the Tsetlin Machine With Applications to Forecasting Disease Out-
breaks,” arXiv e-prints, p. arXiv:1905.04199 , May 2019.
10. S. Misra, P. V. Krishna, and K. I. Abraham, ”A Simple Learning Automata-Based
Solution for Intrusion Detection in Wireless Sensor Networks,” Wireless Communi-
cations and Mobile Computing , vol. 11, no. 3, pp. 426-441, 2011.


key: chroma:document
string_value:
11. T. A. Tuan, L. C. Tong, and A. Premkumar, ”An Adaptive Learning Automata
Algorithm for Channel Selection in Cognitive Radio Network,” 2010 International
Conference on Communications and Mobile Computing , pp. 159-163, IEEE, 2010.
12. K. S. Narendra and M. A. Thathachar, Learning Automata: An Introduction.
Courier Corporation , 2012.
13. K. Narendra and M. Thathachar, ”Learning Automata-A Survey,” IEEE Transac-
tions on systems, man, and cybernetics , no. 4, pp. 323-334, 1974.


key: source
string_value:
source_documents\darshana-abeyrathna-et-al-2019-the-regression-tsetlin-machine-a-novel-approach-to-interpretable-nonlinear-regression.pdf


key: chroma:document
string_value:
royalsocietypublishing.org/journal/rsta
Research
Citethisarticle: DarshanaAbeyrathnaK,
GranmoO-C,ZhangX,JiaoL,GoodwinM.2019
TheregressionTsetlinmachine:anovelapproachtointerpretablenonlinearregression. P h i l .T r a n s .R .S o c .A 378:20190165.
http://dx.doi.org/10.1098/rsta.2019.0165
Accepted:30September2019Onecontributionof13toathemeissue
‘Harmonizingenergy-autonomouscomputingandintelligence’.
SubjectAreas:
artificialintelligence
Keywords:


key: chroma:document
string_value:
SubjectAreas:
artificialintelligence
Keywords:
Tsetlinmachine,regressionTsetlinmachine,patternrecognitionwithpropositionallogic,
InterpretableAI,interpretablemachine
learning,nonlinearregression
Authorforcorrespondence:
K.DarshanaAbeyrathna
e-mail:darshana.abeyrathna@uia.noTheregressionTsetlin
machine:anovelapproachtointerpretablenonlinearregression
K.DarshanaAbeyrathna,Ole-ChristofferGranmo,
XuanZhang,LeiJiaoandMortenGoodwin
CentreforArtificialIntelligenceResearch,UniversityofAgder,


key: chroma:document
string_value:
Grimstad,Norway
KDA,0000-0003-4816-2597
Relying simply on bitwise operators, the recently
introduced Tsetlin machine (TM) has provided
competitive pattern classiﬁcation accuracy in several
benchmarks, including text understanding. In this
paper, we introduce the regression Tsetlin machine
(RTM), a new class of TMs designed for continuous
input and output, targeting nonlinear regression
problems. In all brevity, we convert continuous inputinto a binary representation based on thresholding,


key: chroma:document
string_value:
and transform the propositional formula formed by
the TM into an aggregated continuous output. Ourempirical comparison of the RTM with state-of-the-art
regression techniques reveals either superior or on
par performance on ﬁve datasets.
This article is part of the theme issue ‘Harmonizing
energy-autonomous computing and intelligence’.
1. Introduction
Over the last few years, there has been tremendous
progress in research on predictive models, resulting in


key: chroma:document
string_value:
increasingly higher predictive accuracy. However, highpredictive accuracy is not always sufﬁcient to trust
a predictive model. For applications where humans
are making the ﬁnal decision, the rationale behind a
prediction can sometimes be essential to the decision-
making process. Therefore, human interpretability of
results is of great importance [ 1]. This importance has
been emphasized in many application domains such as
credit scoring [ 2,3], medicine [ 4,5], bioinformatics [ 6,7]


key: chroma:document
string_value:
and churn prediction [ 8,9].
2019TheAuthor(s)PublishedbytheRoyalSociety.Allrightsreserved.
 Downloaded from https://royalsocietypublishing.org/ on 05 December 2023


key: chroma:document
string_value:
2royalsocietypublishing.org/journal/rsta P h i l .T r a n s .R .S o c .A 378:20190165................................................................The Tsetlin machine (TM) is a recent pattern recognition approach that attempts to bridge the
gap between the interpretability of rule-based techniques and the high predictive accuracy of
deep learning [ 10].
The TM is relatively simple computationally, being based on propositional logic to form the


key: chroma:document
string_value:
classiﬁer, and straightforward bitwise operations for recognition and learning. This structuremakes the TM interpretable, yet it achieves competitive accuracy for many pattern recognition
problems.
The inputs and output of the TM are propositional variables, represented as bits. In the ﬁrst
TM layer, the inputs and their negations, referred to as literals, are connected by conjunctions
to form so-called conjunctive clauses . In the second layer, the output of the TM is decided by a


key: chroma:document
string_value:
majority vote among the clauses that have been formed in layer one. For learning patterns, eachclause is assigned one Tsetlin automaton (TA) [ 11] per literal. A literal takes part in a clause if the
designated TA decides to include it. To make this decision, the TA interacts with its environment
(the rest of the TM) in an iterative manner. It decides the next action based on its current state,which, in turn, is inﬂuenced by the feedback it receives. Feedback in the TM is governed by a


key: chroma:document
string_value:
novel game, with each TA trying to learn the optimal action after a series of interactions with the
environment [ 10].
The use of TAs to solve complex problems has a long history. Illustrative work includes
forecasting disease outbreaks [ 12], graph colouring [ 13], distributed coordination [ 14], stochastic
searching on the line [ 15], and resource allocation [ 16]. Despite its simplicity, the TM has provided


key: chroma:document
string_value:
competitive results in comparison with traditional machine learning techniques, including
multilayer perceptrons, support vector machines, logistic regression and Naıive Bayes, in well-known benchmarks such as handwritten digits classiﬁcation (MNIST), Iris data classiﬁcation,
and classiﬁcation of Noisy XOR data with non-informative features [ 10]. Recently, Berge et al.


key: chroma:document
string_value:
also studied the ability of the TM to categorize natural language text simply based on a documentword presence vector [ 17]. They further investigated the interpretability of the produced clauses,
for analysing electronic health records. This work demonstrated that the TM in some cases can
outperform vanilla deep learning techniques, including convolutional neural networks (CNN)and long short-term memory (LSTM) neural networks, while keeping the important property of
interpretability.


key: chroma:document
string_value:
interpretability.
Paper Contributions: The classic TM has been designed for binary inputs and output.
Although continuous input and output can be encoded in bit form, the natural ordering of
numbers is lost. We address this limitation in the present paper by introducing the regression
Tsetlin machine (RTM), which consists of three parts:
(i) For continuous input, we propose a data preprocessing procedure that transforms the


key: chroma:document
string_value:
input losslessly into a binary representation that maintains semantic relationship betweennumbers. In brief, the preprocessing procedure considers each unique data sample in each
continuous feature as a potential threshold. The original data samples are then compared
with the complete set of thresholds to create a new feature matrix that only contains bits.
(ii) To produce continuous output that leverages the natural ordering of numbers, we modify


key: chroma:document
string_value:
the inner inference mechanism of the TM. That is, the inputs are transformed into a single
continuous output, rather than to distinct categories. We achieve this by eliminating the
polarities of the clauses, and by summing all the non-polarized votes, mapping the sum
into a continuous output.
(iii) Finally, we propose a new feedback scheme for guiding the TA of the RTM, to support
regression. This scheme minimizes the discrepancy between the predicted and the target


key: chroma:document
string_value:
outputs, with the aid of a modiﬁed stochastic clause activation function.
Paper Organization: The remainder of the paper is organized as follows. In §2, we discuss
the structure and inference mechanisms of the TM, with a particular focus on the parts of thearchitecture that we build upon. Then, in §3 and 4, we present the main contribution of this
paper, which is the data preprocessing and regression procedures, leading to the RTM. The RTM


key: chroma:document
string_value:
3royalsocietypublishing.org/journal/rsta P h i l .T r a n s .R .S o c .A 378:20190165................................................................is analysed empirically in §5 by the help of both artiﬁcial and real-world datasets, in comparison
with selected state-of-the-art regression techniques. We conclude our work in §6.
2. TheTsetlinmachine
In classiﬁcation problems, a class can be represented by its constituting sub-patterns. The TM has


key: chroma:document
string_value:
been designed to capture these sub-patterns explicitly by operating upon a series of conjunctiveclauses, how many decided by the user. The structure of the TM as well as the procedures for
recognition and learning are described in the following.
TM structure: Consider an input feature vector X=(x
k)∈{0, 1}oconsisting of opropositional
variables xkwith domain {0, 1}. The TM considers both the features xkt h e m s e l v e sa sw e l la s


key: chroma:document
string_value:
their negations ¬xk(the literals) when forming the clauses. In all brevity, each clause jtakes the
following form:
cj=1∧⎛
⎜⎝⋀
k∈II
jxk⎞
⎟⎠∧⎛
⎜⎝⋀
k∈¯II
j¬xk⎞
⎟⎠. (2.1)
Above, II
jand¯II
jare non-overlapping subsets of the input variable indexes, II
j,¯II
j⊆{1,...,o},II
j∩
¯II
j=∅, that specify which of the literals take part in the clause j. The indexes of the included
original features are found in II
jwhile the indexes of the included negated features are found in ¯II
j.


key: chroma:document
string_value:
j.
The upper index Isignals that the literals in the subsets are included . Conversely, IE
jand¯IE
jspecify
the literals that are excluded .
To simplify notation, we now introduce an augmented feature vector X′, which can be written
asX′=[x1,x2,x3,...,xo,¬x1,¬x2,¬x3,...,¬xo], after concatenating the original features with the
negated features. The elements of the augmented feature vector x′
kare now the literals, and we
then only need to consider II
jand IE


key: chroma:document
string_value:
then only need to consider II
jand IE
j, however, for the expanded feature index set {1,...,2o}.
The TM employs two-action TAs to decide which feature indexes 1, ...,2ogo into II
j, with one
team of TAs per clause j. That is, with ofeatures, we need 2 ×oTAs per clause. Half of them
represent the original features and the remaining represent the negated features.
The two actions available to each TA are include and exclude . Here, include refers to including


key: chroma:document
string_value:
the literal assigned to the TA and exclude means excluding it. The action that the TA performs is
decided by its current state, aj,k∈{1,...,2N}. If the state is less than or equal to N, the literal is
excluded from the clause. Therefore, the subset of indexes of the excluded literals can be stated
as,IE
j={k|aj,k≤N,1≤k≤2o}. Conversely, if the state is higher than N, the corresponding literal
is inserted in the clause. The subset of the indexes of the included literals can be then written asI
I


key: chroma:document
string_value:
I
j={k|aj,k>N,1≤k≤2o}. All these states aj,kof all the clauses are organized in an m×2omatrix
A:A=(aj,k)∈{1,...,2N}m×2o.
Figure 1 illustrates the above described TM structure. In the ﬁgure, the upper index tof TAt
k
denotes the type of the feature (1 for the original and 2 for the negated) while the lower index k
denotes the original feature index.
Clause output: Since the TM clause is a conjunction of literals, including any literal of value


key: chroma:document
string_value:
0, even a single one, will make the clause output 0. Let the index set I1
X′be the indexes kof all of
the literals of value x′
k=1 in the input X′. Accordingly, a clause evaluates to 1 if and only if the
indexes of the included literals II
jis a subset of the indexes I1
X′of literals of value 1:
cj={
1i f II
j⊆I1
X′,
0 otherwise.(2.2)
Clause voting: The above clause outputs are then organized in a vector C=(cj)∈{0, 1}m.


key: chroma:document
string_value:
To increase the expressiveness of the TM, clauses are assigned positive or negative polarity.
Clauses with positive polarity ( C+) vote for the class y=1 and clauses with negative polarity
(C−) vote for the class y=0. To avoid bias, the clauses are equally divided among class 0
 Downloaded from https://royalsocietypublishing.org/ on 05 December 2023


key: chroma:document
string_value:
4royalsocietypublishing.org/journal/rsta P h i l .T r a n s .R .S o c .A 378:20190165................................................................
x1
x2
xox1
x2
.
.
xo
¬x1
.
.
¬xo–1
¬xoTA2
o−1inputs
literals
TA
TA 
decisions  
clause
conjunction
operatorTA1
2
.
.
TA1
o
.
.
TA2
oTA1
1
TA21
include/exclude ¬xo–1
include/exclude ¬xoinclude/exclude x2include/exclude x1
include/exclude ¬x1include/exclude xo Cj
Figure1.FormingaclauseusinginputfeaturesandtheactionsoftheTAs.
TA team 1
TA team 2


key: chroma:document
string_value:
TA team 1
TA team 2
TA team m+
_
_
output
yC1
C2
CmCjm
j=1TM 1
TM qTM 2youtput
argmax
operatorCj1
Cj2
Cjq
j=1j=1j=1m—q
m—q
m—q(a)( b)
Figure2.(a)ThebasicTMfortwoclassproblems.( b)Multi-classversionoftheTM.
and class 1. E.g. clauses with odd indexes can be assigned positive polarity and clauses with
even indexes can be assigned negative polarity. The TA state matrix Athen has two separate
sections: A+andA−. The section A+maintains the states of TAs of positive clauses: A+=
(a+


key: chroma:document
string_value:
(a+
j,k)∈{1,...,2N}(m/2)×2o, representing the odd rows of matrix A. The section A−maintains the
states of TAs of negative clauses: A−=(a−
j,k)∈{1,...,2N}(m/2)×2o, representing the even rows of
matrix A.
The resulting two-class architecture is illustrated in ﬁgure 2 a. The summation operator at the
end sums the votes for each class separately and considers the difference: v=∑
jc+
j−∑
jc−
j.A t
last, the ﬁnal output is decided as in (2.3).
y={
1i f v≥0
0i f v<0.(2.3)


key: chroma:document
string_value:
y={
1i f v≥0
0i f v<0.(2.3)
Note that for categorization tasks with more classes than two, separate TMs are needed for
each class as shown in ﬁgure 2 b. In such situations, clauses are partitioned equally among the
classes. Clauses with positive polarity vote in favour of the considered class, and clauses withnegative polarity vote against the considered class. In the end, an argmax operator arbitrates the


key: chroma:document
string_value:
class predicted for the input, based on the votes collected for each class. When there are qclasses
andv
iis the difference between positive and negative votes for class i, the output, y,c a nb e
 Downloaded from https://royalsocietypublishing.org/ on 05 December 2023


key: chroma:document
string_value:
5royalsocietypublishing.org/journal/rsta P h i l .T r a n s .R .S o c .A 378:20190165................................................................expressed as:
y=argmaxi=1,...,q{
vi}
. (2.4)
Learning procedure: The conjunctive clauses of the TM are formed by the decisions made by
the collective of TAs associated with all of the clauses. Hence, training in the TM entails carefulguiding of the TAs so that they as a collective make the optimal include and exclude decisions. We


key: chroma:document
string_value:
achieve this by reinforcement learning, organized as a game between the TAs. In all brevity, the
TAs adjust their states based on feedback from the game (environment), encompassing rewards,penalties, and inaction feedback. To achieve the learning goal, the probabilities of receiving these
different types of feedback have been designed to account for critical factors, namely, the actual
output, the clause outputs, the literal values, and current state of the TAs.


key: chroma:document
string_value:
In contrast to gradient-based learning, learning in TMs naturally combats false positives and
false negatives. As a consequence, it eludes the issues attached to gradient-based algorithms such
as vanishing/exploding gradients. In the case of categorization problems with two classes, thebasic idea is to penalize voters when they vote to procure a false positive or false negative and to
reward voters when they vote to procure a true positive or true negative. In the TM, this is done


key: chroma:document
string_value:
by two types of feedback—Type I and Type II.
Type I feedback: The clauses that receive Type I feedback is decided stochastically using the
following conditions:
p
j,y=⎧
⎨
⎩1 with probabilityT−max(−T,m i n ( T,v))
2T,
0 otherwise.(2.5)
Here, the lower indexes jand yofprepresent the clause and target class, respectively. A user
deﬁned target Tdecides the robustness of the learning. Higher Tmakes more clauses get


key: chroma:document
string_value:
involved in learning each sub-pattern, diversifying the representation. However, increasing T
also demands more clauses, increasing computational complexity.
Type I feedback is given to clauses with positive polarity when the actual output ˆyis 1 and
to clauses with negative polarity when the actual output ˆyis 0. Therefore, the complete matrix,
Py, that picks the clauses for feedback Type I, is a combination of P+
1andP−
0where P+
1=(p+
j,1)∈
{0, 1}(m/2)andP−
0=(p−


key: chroma:document
string_value:
0where P+
1=(p+
j,1)∈
{0, 1}(m/2)andP−
0=(p−
j,0)∈{0, 1}(m/2). Type I feedback does two jobs. Type Ia reinforces include
actions of TAs whose corresponding literal value is 1 when the clause output is 1. Type Ib combats
over-ﬁtting by reinforcing exclude actions of TAs when the corresponding literal is 0 or when the
clause output is 0. Both Type Ia and Type Ib feedback are provided to TAs stochastically using a
user set parameter s(s≥1).


key: chroma:document
string_value:
user set parameter s(s≥1).
Type Ia feedback: The matrix R,i . e .R=(rj,k)∈{0, 1}m×2ostores the decisions whether the kth
TA of the jthclause has been selected as candidate for feedback Type Ia. The decisions are made
stochastically as follows:
rj,k=⎧
⎨
⎩1 with probabilitys−1
s,
0 otherwise.(2.6)
Now, using the complete set of conditions, the indexes IIa
Cof the TAs which receive Type Ia
feedback can be identiﬁed as IIa
C={(j,k)|x′
k=1∧cj=1∧pj,y=1∧rj,k=1}.


key: chroma:document
string_value:
C={(j,k)|x′
k=1∧cj=1∧pj,y=1∧rj,k=1}.
Type Ib feedback: Similarly, candidates for receiving Type Ib feedback are decided
stochastically and decisions are stored in matrix Q,i . e . Q=(qj,k)∈{0, 1}m×2o. In this situation,
the decision for the kthTA of the jthclause is
qj,k=⎧
⎨
⎩1 with probability1
s,
0 otherwise.(2.7)
Again, considering all other conditions, the indexes IIb
Cof the TAs that receive feedback Type Ib
are expressed as IIb
C={(j,k)|(x′
k=0∨cj=0)∧pj,y=1∧qj,k=1}.


key: chroma:document
string_value:
C={(j,k)|(x′
k=0∨cj=0)∧pj,y=1∧qj,k=1}.
 Downloaded from https://royalsocietypublishing.org/ on 05 December 2023


key: chroma:document
string_value:
6royalsocietypublishing.org/journal/rsta P h i l .T r a n s .R .S o c .A 378:20190165................................................................The identiﬁed TAs are updated by changing their internal state. The available update
operations are ⊕and⊖. The operator ⊕adds 1 to the current state while operator ⊖subtracts
1 from the current state within the given state space: A←(A⊕IIa
C)⊖IIb
C.
In conclusion, Type I feedback stimulates positive clauses to output 1 when the actual output


key: chroma:document
string_value:
ˆyis 1, while it stimulates negative clauses to output 1 when the actual output ˆyis 0. Hence, Type
I feedback reinforces true positive output.
Type II feedback: Type II feedback is given to clauses with positive polarity when the actual
output ˆyis 0 and clauses with negative polarity when the actual output ˆyis 1. Whether the jth
clause for target output ˆyreceives Type II feedback is decided as follows:
pj,y=⎧
⎨
⎩1 with probabilityT+max(−T,m i n ( T,v))
2T,
0 otherwise.(2.8)


key: chroma:document
string_value:
2T,
0 otherwise.(2.8)
Similar to Type I feedback, Tis a user decided voting target, and decisions are stored in the matrix
Py. The idea is to alter the clause output of the clauses which output 1. This can be done by
simply including literals of value 0. The TAs that match the above conditions can be identiﬁedasI
II
C={(j,k)|x′
k=0∧cj=1∧pj,y=1}. The identiﬁed TAs are updated by changing their internal
state: A←A⊕III
C.


key: chroma:document
string_value:
state: A←A⊕III
C.
In this way, Type II feedback stimulates clauses to output 0. Type II feedback is applied to
clauses of positive polarity when the actual output ˆyis 0 and to clauses of negative polarity when
the actual output ˆyis 1. In this manner, it combats false positives.
The training procedure of the multi-class version of the TM is similar to the above training
procedure. Clauses of the class being the target of the current training sample are treated as if


key: chroma:document
string_value:
ˆy=1, while the clauses of a randomly selected class from the remaining classes are treated as
ifˆy=0. In each class, clauses with positive polarity vote to say that the output belongs to the
considered class. Similarly, the clauses with negative polarity vote to indicate that the output
does not belong to the considered class.
3. AnencodingschemeforcontinuousinputtotheTsetlinmachine
The input feature vector of the TM that we presented in the previous section only had binaryfeatures, i.e. X=[x


key: chroma:document
string_value:
1,x2,x3,...,xo] with xk,k=1, 2,...,o, being 0 or 1. Clauses are directly
composed by these features or their negations, as decided by the TAs. However, in many real-
world applications, one cannot necessarily expect binary features. Instead, the input features are
often continuous-valued. In order to cope with such applications, we introduce a preprocessing
procedure that transforms continuous features into binary ones, while maintaining ranking


key: chroma:document
string_value:
relationships among the continuous feature values.
By way of example, table 1 shows how two continuous features are binarized. As seen, the
two features are respectively listed in table column 1 and column 2. The preprocessing procedure
follows the following steps to convert them into binary form, one feature at a time.
(i) Identify the unique values {v1,v2,...,vu}of the feature.
(ii) Sort the identiﬁed unique values from smallest to largest.


key: chroma:document
string_value:
(iii) Consider each unique value vi,i=1, 2,...,u, as a threshold ‘ ≤vi’, as shown in sorted
order in the ‘Thresholds’ row in the table.
(iv) Compare each original continuous value in the feature with each of the sorted thresholds.
If the feature value is greater than the threshold, set the corresponding binary variable to0, otherwise, set the bit to 1.
(v) Repeat steps (i) to (iv) until all the continuous values are converted into binary form.


key: chroma:document
string_value:
According to the above procedure, the ﬁrst feature in the given example has three unique
values, i.e. 5.779, 10.008 and 3.834 (step (i)). These unique values are sorted and considered as
thresholds, ≤3.834, ≤5.779 and ≤10.008 (step (ii) and (iii)). Note that the number of binary feature
 Downloaded from https://royalsocietypublishing.org/ on 05 December 2023


key: chroma:document
string_value:
7royalsocietypublishing.org/journal/rsta P h i l .T r a n s .R .S o c .A 378:20190165................................................................Table1.Preprocessingoftwocontinuousfeatures.
rawfeature thresholds
12 ≤3.834 ≤5.779 ≤10.008 ≤11.6 ≤25.7 ≤32.4 ≤56.1
5.779 25.7 0 1 1 0 1 1 1
......................................... ............................................ .......................................... ..................................... ......................................


key: chroma:document
string_value:
10.008 56.1 0 0 1 0 0 0 1
......................................... ............................................ .......................................... ..................................... ......................................
5.779 11.6 0 1 1 1 1 1 1
......................................... ............................................ .......................................... ..................................... ......................................
3.834 32.4 1 1 1 0 0 1 1


key: chroma:document
string_value:
3.834 32.4 1 1 1 0 0 1 1
......................................... ............................................ .......................................... ..................................... ......................................
bits we are going to have after converting the continuous inputs equals the number of thresholds.
In this case, for Feature 1 in the table (ﬁrst column), three new binary feature bits are introduced,


key: chroma:document
string_value:
each of which corresponds to its respective threshold. If we compare the ﬁrst raw continuous
value of Feature 1, namely, 5.779, with the identiﬁed three thresholds, we get a 011 as the rawvalue is greater than 3.834 (resulting in 0), equal to 5.779 (resulting in 1), and less than 10.008
(resulting in 1) (Step iv). Similarly, the remaining raw continuous values, 10.008 and 3.834, are
converted into 001 and 111, respectively.


key: chroma:document
string_value:
converted into 001 and 111, respectively.
Once all the raw continuous values of the ﬁrst feature are converted, conversion of the second
feature can start. This procedure is iterated until all the continuous values in all the continuous
features have been converted to binary form (step (v)).
Note that the thresholds found in the above procedure are open-ended at the extreme ends.
Thus, arbitrarily low or high feature values can be processed, independent of the actual values in


key: chroma:document
string_value:
the training data.
The new binary representation of continuous features becomes particularly powerful because
it allows the TM to reason about the ordering of the values, forming conjunctive clauses that
specify rules based on thresholds, and with negated features, also rules based on intervals.
4. TheregressionTsetlinmachine
For many real-world prediction problems, the target yis continuous valued, referred to as


key: chroma:document
string_value:
regression problems. Although continuous output can be encoded in bit form, the inherent
properties of continuous values are lost, such as the ordering of values. Thus, the binary output
of the classic TM shown in ﬁgure 2 a,bis not ideal for representing continuous output. In order to
address this problem, we here propose a new type of TM—the RTM .
The structure of the RTM is illustrated in ﬁgure 3 . As the ﬁgure shows, we remove the polarity


key: chroma:document
string_value:
of the clauses in the vanilla TM structure. Instead, we use clauses as additive building blocksto calculate the continuous outputs. That is, the summation operator in ﬁgure 3 counts the total
number of clauses that evaluate to 1, and the resulting sum is further mapped into a continuous
value according to equation (4.1):
y=∑
m
j=1Cj(ˆX)
T×ˆymax. (4.1)
Above, Tis a user speciﬁed output resolution and ˆymaxis the maximum value of the target output


key: chroma:document
string_value:
found in the training data, assuming positive output.1This maximum value is merely a reference
point, to establish a scale, and should not be considered as an upper bound for the output.
Once a continuous output value yhas been calculated, it is compared with the target output
ˆy. Following the convention of regression analysis, we refer to the difference between the output
yandˆyas the error of the prediction. The output itself and the error are, in turn, fed back to the


key: chroma:document
string_value:
RTM, and inﬂuence the decision made by the TAs so that the error can be minimized. The goal is
for the RTM to learn patterns appropriate for regression from the training dataset.
1Our scheme also supports negative output values by employing clauses of negative polarity.
 Downloaded from https://royalsocietypublishing.org/ on 05 December 2023


key: chroma:document
string_value:
8royalsocietypublishing.org/journal/rsta P h i l .T r a n s .R .S o c .A 378:20190165................................................................TA team 1
TA team myTA team 2
outputmappingC1
C2
CmCj
j=1myˆmax
yˆmin
Figure3.TheRTMstructure.
Feedback for the RTM is derived slightly differently compared to the vanilla TM. That is,
whether it is given Type I or Type II feedback is based on the difference between yandˆy,a s
determined by equation (4.2):
Feedback ={
Type I, if y<ˆy,


key: chroma:document
string_value:
Feedback ={
Type I, if y<ˆy,
Type II, if y>ˆy.(4.2)
The rationale behind equation (4.2) is as follows. We want to increase the number of clauses that
output 1 when the predicted output is less than the target output ( y<ˆy). To achieve this, Type I
feedback is introduced. Conversely, Type II feedback is applied to decrease the number of clausesthat evaluate to 1 when the predicted output is higher than the target output ( y>ˆy).


key: chroma:document
string_value:
If the feedback determined by (4.2) is offered to all the clauses, the predicted value will keep
oscillating around the target output throughout the training process. Hence, we randomly selecta subset of the clauses to receive feedback, with the prediction error controlling the cardinality of
the subset. We achieve this by introducing the following feedback activation probability function,
P
act, by which the RTM decides stochastically which clauses to be updated:
Pact=K×|y−ˆy|
ˆymax. (4.3)


key: chroma:document
string_value:
Pact=K×|y−ˆy|
ˆymax. (4.3)
Note that in equation (4.3), the parameter Kis added to adjust the activation probability according
to the total number of clauses. In other words, the probability varies proportionally to the error,
and is reasonably scaled to ﬁt the total number of clauses.
The ﬂowchart given in ﬁgure 4 summarizes the complete training procedure of the RTM,
including the data preprocessing step introduced in §3.
5. Empiricalresultsandanalysis


key: chroma:document
string_value:
5. Empiricalresultsandanalysis
We now study the properties of the RTM empirically, by means of two artiﬁcial and ﬁve real-world
datasets.2
(a) Artificialdatasets
The two artiﬁcial datasets have been designed to demonstrate how the RTM form nonlinear
clauses, which in turn are used as additive components in the regression. The ﬁrst dataset, Dataset


key: chroma:document
string_value:
I, has 8 distinct outputs, each of which is triggered by a speciﬁc 3-bit input. Each input bit has anequal probability of being set to either 0 or 1, leading to a uniform distribution of bit values. The
continuous output is quite simply calculated by multiplying 100 with the decimal representation
of the input. For example, the continuous output for input (011) becomes 100 ×3=300. The


key: chroma:document
string_value:
2The artiﬁcial datasets and RTM code are available to download at https://github.com/cair/regression-tsetlin-machine .
 Downloaded from https://royalsocietypublishing.org/ on 05 December 2023


key: chroma:document
string_value:
9royalsocietypublishing.org/journal/rsta P h i l .T r a n s .R .S o c .A 378:20190165................................................................
data
preprocessing
(section 3): start regression Tsetlin machine
set: clauses m, precision s, threshold T
random initialization of TAs:
2 × o TA per clause
evaluate clauses: eqn (2.2)
feedbacks to
the clausescompute output y: eqn (4.1)
type II feedbacktype I feedback
stopping
criteria
ready to predict unseen datayesnotraining
sample Xˆ, yˆ
  data


key: chroma:document
string_value:
sample Xˆ, yˆ
  data
y<yˆ
satisfy eqn (4.3)
y>yˆ
satisfy eqn (4.3)
Figure4.Thetrainingwork-flowoftheRTM.
complete dataset consists of 10 000 samples, split into 80% training data and 20% test data. The
second dataset, Dataset II, is built in the same way, except that the output of the training data is
perturbed by adding a random value in the range [ −10, 10] to introduce noise.
We train and test the RTM separately on the above two datasets, with mean absolute error


key: chroma:document
string_value:
(MAE) used as evaluation criterion. For all of our experiments on artiﬁcial data, we use a
speciﬁcity value sof 2. Figures 5and 6depict training MAE across 1000 epochs, examining the
effect of resolution T. The ﬁnal training and test MAEs, i.e. the ones obtained after 1000 epochs,
are provided in the legends. As seen in ﬁgure 5 , noise-free data can be perfectly learnt by an
RTM with merely resolution 7, using 7 clauses. However, in general, as seen in ﬁgure 6 , higher


key: chroma:document
string_value:
resolution progressively reduces MAE under noise.
Let us now study the case with 7 clauses and noise-free data more carefully to cast further
light on how the RTM learns. With 3 uniformly distributed bits, there are 8 unique sub-patterns,
illustrated in ﬁgure 7 . Each of these occurs with probability 1 /8. The shaded area in the ﬁgure
contains the pattern (1 ✳✳), whose probability of occurrence is 4 /8. Here, ✳means either 0 or 1.


key: chroma:document
string_value:
Similarly, the probabilities of occurrence for both the pattern ( ✳1✳) and the pattern ( ✳✳ 1) is 4 /8,
too. According to the TM dynamics explained in [ 10], in order to capture these patterns, speciﬁcity
should be set to s=2.
With only 7 clauses available, the clauses need to encode the input in a very compact way,
shown in table 2 , to produce the correct output. As seen, we need four clauses for the pattern


key: chroma:document
string_value:
(1✳✳), two clauses for the pattern ( ✳1✳), and one clause for the pattern ( ✳✳ 1). Now, consider
 Downloaded from https://royalsocietypublishing.org/ on 05 December 2023


key: chroma:document
string_value:
10royalsocietypublishing.org/journal/rsta P h i l .T r a n s .R .S o c .A 378:20190165................................................................
020406080100120
1
6
11
16
21
26
31
36
41
46
51
56
61
667176
81
86
91
96
101
106
111
116121
126
131
136
141
146
151
156
161
166
171
176
181
186
191
196MAE
iterationDataset I 7-T (train MAE - 0.00, test MAE - 0.00) 20-T (train MAE - 18.60, test MAE - 18.50)
70-T (train MAE - 0.00, test MAE - 0.00) 500-T (train MAE - 0.40, test MAE - 0.41)


key: chroma:document
string_value:
5000-T (train MAE - 0.07, test MAE - 0.07)
Figure5.TrainingerrorperepochforDatasetI.Thedatasetisprocessedwithdifferent T.(Onlineversionincolour.)
020406080100120
1
6
11
16
21
26
31
36
41
46
51
56
61
66
71
76
81
86
91
96
101
106
111
116
121
126
131
136
141
146
151
156
161
166
171
176
181
186
191
196MAE
iterationDataset II 7-T (train MAE - 7.33, test MAE - 5.64) 20-T (train MAE - 15.49, test MAE - 15.23)
70-T (train MAE - 7.13, test MAE - 4.81) 500-T (train MAE - 5.40, test MAE - 1.51)


key: chroma:document
string_value:
5000-T (train MAE - 5.26, test MAE - 0.26)
Figure6.TrainingerrorperepochforDatasetII.Thedatasetisprocessedwithdifferent T.(Onlineversionincolour.)
1 0 0 0 1 0
0 0 10 1 1 1 0 11 1 0
1 1 10 0 0
Figure7.Patterndistributionforthe3-bitsinputdatasets.
input (0 1 1) as an example. This input matches both pattern ( ✳1✳) and pattern ( ✳✳ 1).
Therefore, it activates three clauses from the table. Consequently, after scaling, output 300 is


key: chroma:document
string_value:
correctly computed from (0 1 1). In this manner, the correct output can be calculated for all 3-bit
inputs from table 2 . Returning to ﬁgure 5 , we note that the MAE of the RTM is exactly 0 for 7
 Downloaded from https://royalsocietypublishing.org/ on 05 December 2023


key: chroma:document
string_value:
11royalsocietypublishing.org/journal/rsta P h i l .T r a n s .R .S o c .A 378:20190165................................................................Table2.Computingoutputsfordifferent3bitinputs[ 18].
input representedpatterns outputcomputation output
000 none 4 ×(1✳✳)×0=00 ×100=0
2×(✳1✳)×0=0
1×(✳✳1)×0=0


key: chroma:document
string_value:
2×(✳1✳)×0=0
1×(✳✳1)×0=0
......................................... ............................................ .......................................... ..................................... ......................................
001 ( ✳✳1) 4 ×(1✳✳)×0=01 ×100=100
2×(✳1✳)×0=0
1×(✳✳1)×1=1
......................................... ............................................ .......................................... ..................................... ......................................


key: chroma:document
string_value:
010 ( ✳1✳)4 ×(1✳✳)×0=02 ×100=200
2×(✳1✳)×1=2
1×(✳✳1)×0=0
......................................... ............................................ .......................................... ..................................... ......................................
011 ( ✳1✳)4 ×(1✳✳)×0=03 ×100=300
(✳✳1) 2 ×(✳1✳)×1=2
1×(✳✳1)×1=1


key: chroma:document
string_value:
(✳✳1) 2 ×(✳1✳)×1=2
1×(✳✳1)×1=1
......................................... ............................................ .......................................... ..................................... ......................................
100 (1 ✳✳)4 ×(1✳✳)×1=44 ×100=400
2×(✳1✳)×0=0
1×(✳✳1)×0=0


key: chroma:document
string_value:
2×(✳1✳)×0=0
1×(✳✳1)×0=0
......................................... ............................................ .......................................... ..................................... ......................................
101 (1 ✳✳)4 ×(1✳✳)×1=45 ×100=500
(✳✳1) 2 ×(✳1✳)×0=0
1×(✳✳1)×1=1


key: chroma:document
string_value:
(✳✳1) 2 ×(✳1✳)×0=0
1×(✳✳1)×1=1
......................................... ............................................ .......................................... ..................................... ......................................
110 (1 ✳✳)4 ×(1✳✳)×1=46 ×100=600
(✳1✳)2 ×(✳1✳)×1=2
1×(✳✳1)×0=0


key: chroma:document
string_value:
(✳1✳)2 ×(✳1✳)×1=2
1×(✳✳1)×0=0
......................................... ............................................ .......................................... ..................................... ......................................
111 (1 ✳✳)4 ×(1✳✳)×1=47 ×100=700
(✳1✳)2 ×(✳1✳)×1=2
(✳✳1) 1 ×(✳✳1)×1=1


key: chroma:document
string_value:
(✳1✳)2 ×(✳1✳)×1=2
(✳✳1) 1 ×(✳✳1)×1=1
......................................... ............................................ .......................................... ..................................... ......................................
clauses/resolution 7, meaning that the RTM has found the structure of table 2 by itself. Hence,
the power of the scheme!
For real-world cases, however, the exact number of clauses required is generally unknown.


key: chroma:document
string_value:
Then, selecting an inappropriate low resolution may lead to a high MAE. This is seen in ﬁgure 5
forT=20. Then one must instead resort to increasing the resolution to a sufﬁcient level, which
also helps with dealing with noisy data. That is, as seen, both training and testing MAEs drops
with T=500 and T=5000 for Dataset I. The latter reasoning also applies to the results for Dataset
II (ﬁgure 6 ), i.e. the training and testing MAEs decrease with increasing T. Indeed, it turns out


key: chroma:document
string_value:
that the testing MAE can be made arbitrarily close to 0 for the present task simply by increasingthe resolution, the number of training examples, and the number of training epochs.
(b) Real-worlddatasets
We now study the behaviour of the RTM on ﬁve different real-world datasets:
Dengue Incidences:3This dataset consists of monthly dengue incidences in the Philippines
per 100 000 population. The Philippines has 17 administrative regions. The department of health


key: chroma:document
string_value:
3Available from https://www.kaggle.com/grosvenpaul/dengue-cases-in-the-philippines .
 Downloaded from https://royalsocietypublishing.org/ on 05 December 2023


key: chroma:document
string_value:
12royalsocietypublishing.org/journal/rsta P h i l .T r a n s .R .S o c .A 378:20190165................................................................in the Philippines collected the data from all of these regions separately from 2008 to 2016. In
this experiment, the RTM is trained separately for each region using the data from 2008 to 2015.
Dengue incidences from the neighbouring regions are used as input features. More details about


key: chroma:document
string_value:
data preprocessing and feature selection can be found in [ 19]. Dengue incidences in 2016 are used
as testing data.
Energy Performance:4In this application, heating load of residential buildings has to be
estimated using eight input features: glazing area distribution, glazing area, orientation, relative
compactness, wall area, surface area, overall height, roof area, and orientation [ 20]. The complete


key: chroma:document
string_value:
dataset comprises 768 samples. We use 80% of the data samples to train the model and the rest to
evaluate it.
Stock Selection:5The US historical stock market data was used to simulate the weights of
the stock-picking concepts in [ 21]. These weights are employed to build a stock selection decision
support system. The dataset consists of six output variables. Only two of them, i.e. Annual Return


key: chroma:document
string_value:
and Real Win Rate is used here. Again, 80% of the data samples are used to train the model andthe rest is used for testing.
Real Estate Valuation:
6This is a time-series dataset that has been used to estimate the house
price per unit area in Taiwan [ 22]. The house price per unit area (Dollars per Ping, 1 Ping = 3.3
metre squared) is estimated using six features, namely the transaction date, the house age, the


key: chroma:document
string_value:
distance to the nearest MRT station, the number of convenience stores in the living circle on foot,
and the geographical coordinates (latitude and longitude). Again we use (80%) of the dataset for
training and (20%) for testing.
Aerofoil Noise:7The dataset comprises different sizes of NACA 0012 aerofoils (an aerofoil
shape for aircraft wings) at various wind tunnel speeds and angles of attack. The goal is to


key: chroma:document
string_value:
predict the self-generated noise of an aerofoil blade in decibels using ﬁve attributes, namely,
Frequency (in Hertzs), Angle of attack (in degrees), Chord length (in metres), Free-stream velocity(in metres per second), and Suction side displacement thickness (in metres). Out of the total 1503
data samples, 625 are randomly selected for doing the experiment. The RTM is trained on 80% of
the selected data and tested on the rest of them.


key: chroma:document
string_value:
the selected data and tested on the rest of them.
Apart from Dengue Incidences, we used the same number of clauses and the same resolution
for all of the datasets, respectively, 2 000 000 and 1 000 000. This setting was found by increasing
the number of clauses and resolution until the accuracy gain became insigniﬁcant. The speciﬁcitysfor each dataset was found by a manual binary search (4 for Energy Performance, 3 for Annual


key: chroma:document
string_value:
Return, 2.5 for Real Win Rate, 3.8 for Real Estate Valuation, and 2.7 for Aerofoil Noise). For
forecasting dengue incidences across all the regions in the Philippines, we used 200 000 clauses, aresolution of 100 000, and an s-value of 6.
Table 3 reports average MAE obtained by replicating each experiment 20 times, together
with 95% conﬁdence intervals. To evaluate the performance of the RTM, we used three other


key: chroma:document
string_value:
state-of-the-art machine learning models as a baseline. These models are regression tree (RT),
random forest (RF) and support vector regression (SVR). Each of these were conﬁgured basedon a thorough parameter search in order to facilitate a fair comparison. Each model predicts six
different data series using their relevant input features. In conclusion, RTM obtained the best


key: chroma:document
string_value:
MAE for four of the prediction tasks (Dengue Incidences, Energy Performance, Real Win Rate, andReal Estate Valuation). For Annual Return, SVR produced the lowest MAE, while RTM achieved
the second best. Finally, RF provided the best MAE for Aerofoil Noise. Here, RTM obtained the
third-best result.
In order to investigate the ability of the RTM to extrapolate, we cross-validated the model
using the Energy Performance dataset. To produce test data with extreme values not found in


key: chroma:document
string_value:
the training data, ten different datasets were created by randomly shufﬂing the original dataset.
4Available from https://archive.ics.uci.edu/ml/datasets/Energy+efﬁciency .
5Available from https://archive.ics.uci.edu/ml/datasets/Stock+portfolio+performance .
6Available from https://archive.ics.uci.edu/ml/datasets/Real+estate+valuation+data+set .
7Available from https://archive.ics.uci.edu/ml/datasets/Airfoil+Self-Noise# .
 Downloaded from https://royalsocietypublishing.org/ on 05 December 2023


key: chroma:document
string_value:
13royalsocietypublishing.org/journal/rsta P h i l .T r a n s .R .S o c .A 378:20190165................................................................Table3.MeanMAEwith95%confidenceintervalsforselectedmodelsonfivedatasets.
stockselectiondengue energy realestate
model incidences performance annualreturn realwinrate valuation aerofoilnoise
RTM 5.184 0.503 ±0.00 0.077 ±0.00 0.089 ±0.00 5.218 ±0.00 2.244 ±0.00


key: chroma:document
string_value:
......................................... ............................................ .......................................... ..................................... ......................................
RT 7.769 0.598 ±0.00 0.091 ±0.00 0.096 ±0.00 6.033 ±0.11 2.280 ±0.07
......................................... ............................................ .......................................... ..................................... ......................................


key: chroma:document
string_value:
RF 9.122 0.563 ±0.00 0.088 ±0.00 0.097 ±0.00 5.360 ±0.00 1.921 ±0.00
......................................... ............................................ .......................................... ..................................... ......................................
SVR 5.305 1.155 ±0.00 0.073 ±0.00 0.092 ±0.00 5.697 ±0.00 2.156 ±0.00


key: chroma:document
string_value:
......................................... ............................................ .......................................... ..................................... ......................................
Collectively, this procedure created 22 test outcomes outside the minimum and maximum output
values in the training data. The MAE of those 22 testing outputs was 0.516, which is only slightlyhigher than the overall MAE in table 3 , meaning that the RTM extrapolates reasonably well. This


key: chroma:document
string_value:
can be explained by the additive nature of RTM regression. By summing the binary output of
nonlinear clauses, the RTM avoids extreme nonlinear distortions when operating outside thevalue ranges found in the training data.
6. Conclusion
In this paper, we proposed the RTM, a novel variant of the TM that supports continuousoutput in nonlinear regression problems. We also introduced a data preprocessing procedure


key: chroma:document
string_value:
that converts continuous inputs into a lossless binary feature matrix. In RTM, the polarities in
clauses are removed and the inner inference mechanism is modiﬁed. That is, the input patterns are
transformed into individual continuous outputs, rather than to distinct categories. The activation
of clauses during RTM learning is based on a linear activation probability function that isgoverned by the magnitude of the regression error. The behaviour of the new algorithm was


key: chroma:document
string_value:
studied by applying it to both artiﬁcial and real-world datasets. In brief, RTM demonstrated
superior performance in comparison with Regression Trees, Random Forests, and Support VectorRegression, when predicting Dengue incidences in the Philippines, heating load in the Energy
Performance dataset, Real Win Rate in the Stock Selection dataset, and house price per unit


key: chroma:document
string_value:
area in the Real Estate Valuation dataset. The RTM also extrapolates reasonably well outside theminimum and maximum output values found in the training data.
Dataaccessibility.
This article does not contain any additional data.
Competinginterests. We declare we have no competing interests.
Funding. We received no funding for this study.
References
1. Freitas AA. 2014 Comprehensible classiﬁcation models: a position paper. ACM SIGKDD
Explor. Newsl. 15, 1–10. ( doi:10.1145/2594473.2594475 )


key: chroma:document
string_value:
2. Baesens B, Mues C, De Backer M, Vanthienen J, Setiono R. 2004 Building intelligent credit
scoring systems using decision tables. In Enterprise Information Systems V , pp. 131–137. Berlin,
Germany: Springer.
3. Huysmans J, Dejaeger K, Mues C, Vanthienen J, Baesens B. 2011 An empirical evaluation of
the comprehensibility of decision table, tree and rule based predictive models. Decis. Support
Syst. 51, 141–154. ( doi:10.1016/j.dss.2010.12.003 )


key: chroma:document
string_value:
4. Bellazzi R, Zupan B. 2008 Predictive data mining in clinical medicine: current issues and
guidelines. Int. J. Med. Inform. 77, 81–97. ( doi:10.1016/j.ijmedinf.2006.11.006 )
5. Pazzani MJ, Mani S, Shankle WR. 2001 Acceptance of rules generated by machine learning
among medical experts. Methods Inf. Med. 40, 380–385. ( doi:10.1055/s-0038-1634196 )
6. Freitas AA, Wieser DC, Apweiler R. 2010 On the importance of comprehensible classiﬁcation


key: chroma:document
string_value:
models for protein function prediction. IEEE/ACM Trans. Comput. Biol. Bioinform. (TCBB) 7,
172–182. ( doi:10.1109/TCBB.2008.47 )
 Downloaded from https://royalsocietypublishing.org/ on 05 December 2023


key: chroma:document
string_value:
14royalsocietypublishing.org/journal/rsta P h i l .T r a n s .R .S o c .A 378:20190165................................................................7. Szafron D et al. 2004 Proteome analyst: custom predictions with explanations in a web-
based tool for high-throughput proteome annotations. Nucleic Acids Res. 32, W365–W371.
(doi:10.1093/nar/gkh485 )
8. Lima E, Mues C, Baesens B. 2009 Domain knowledge integration in data mining using


key: chroma:document
string_value:
decision tables: case studies in churn prediction. J. Oper. Res. Soc. 60, 1096–1106. ( doi:10.1057/
jors.2008.161 )
9. Verbeke W, Martens D, Mues C, Baesens B. 2011 Building comprehensible customer churn
prediction models with advanced rule induction techniques. Expert Syst. Appl. 38, 2354–2364.
(doi:10.1016/j.eswa.2010.08.023 )
10. Granmo O-C. 2019 The Tsetlin machine - a game theoretic bandit driven approach to optimal


key: chroma:document
string_value:
pattern recognition with propositional logic. ( https://arxiv.org/abs/1804.01508 )
11. Tsetlin ML. 1961 On behaviour of ﬁnite automata in random medium. Avtom I Telemekhanika
22, 1345–1354.
12. Abeyrathna. KD, Granmo O-C, Goodwin M. 2018 A novel Tsetlin automata scheme to forecast
dengue outbreaks in the philippines. In 2018 IEEE 30th Int. Conf. on Tools with Artiﬁcial
Intelligence (ICTAI) , pp. 680–685. IEEE.
13. Bouhmala N, Granmo O-C. 2010 Stochastic learning for SAT-encoded graph coloring


key: chroma:document
string_value:
problems. Int. J. Appl. Metaheuristic Comput. (IJAMC) 1, 1–19. ( doi:10.4018/jamc.2010070101 )
14. Tung B, Kleinrock L. 1996 Using ﬁnite state automata to produce self-optimization and self-
control. IEEE Trans. Parallel Distrib. Syst. 7, 439–448. ( doi:10.1109/71.494637 )
15. Oommen BJ, Kim S-W, Samuel MT, Granmo O-C. 2008 A solution to the stochastic point
location problem in metalevel nonstationary environments. IEEE Trans. Syst. Man, Cybern.,


key: chroma:document
string_value:
Part B (Cybernetics) 38, 466–476. ( doi:10.1109/TSMCB.2007.913602 )
16. Granmo O-C, Oommen BJ. 2010 Solving stochastic nonlinear resource allocation problems
using a hierarchy of twofold resource allocation automata. IEEE Trans. Comput. 59, 545–560.
(doi:/10.1109/tc.2009.189 )
17. Berge GT, Granmo O-C, Oddbjørn Tveit T, Goodwin M, Jiao L, Viggo Matheussen B.
2019 Using the Tsetlin machine to learn human-interpretable rules for high-accuracy text


key: chroma:document
string_value:
categorization with medical applications. IEEE Access 7, 115 134–115 146. ( doi:10.1109/
ACCESS.2019.2935416 )
18. Abeyrathna KD, Granmo O-C, Jiao L, Goodwin M. 2019 The regression Tsetlin machine:
a Tsetlin machine for continuous output problems. In EPIA2019: EPIA Conf. on Artiﬁcial
Intelligence , pp. 268–280. Berlin, Germany: Springer.
19. Abeyrathna KD, Granmo O-C, Zhang X, Goodwin M. 2019 A scheme for continuous input


key: chroma:document
string_value:
to the Tsetlin machine with applications to forecasting disease outbreaks. In Int. Conf. on
Industrial, Engineering and Other Applications of Applied Intelligent Systems , pp. 564–578. Berlin,
Germany: Springer.
20. Tsanas A, Xifara A. 2012 Accurate quantitative estimation of energy performance of residential
buildings using statistical machine learning tools. Energy Build. 49, 560–567. ( doi:10.1016/
j.enbuild.2012.03.003 )


key: chroma:document
string_value:
j.enbuild.2012.03.003 )
21. Liu Y-C, Yeh I-C. 2017 Using mixture design and neural networks to build stock selection
decision support systems. Neural Comput. Appl. 28, 521–535. ( doi:10.1007/s00521-015-2090-x )
22. Yeh I-C, Hsu T-K. 2018 Building real estate valuation models with comparative approach
through case-based reasoning. Appl. Soft Comput. 65, 260–271. ( doi:10.1016/j.asoc.2018.01.029 )
 Downloaded from https://royalsocietypublishing.org/ on 05 December 2023


key: chroma:document
string_value:
Massively Parallel and Asynchronous Tsetlin Machine Architecture Supporting
Almost Constant-Time Scaling
Kuruge Darshana Abeyrathna* 1Bimal Bhattarai* 1Morten Goodwin* 1Saeed Rahimi Gorji* 1
Ole-Christoffer Granmo* 1Lei Jiao* 1Rupsa Saha* 1Rohan Kumar Yadav* 1
Abstract
Using logical clauses to represent patterns, Tsetlin
machines ( TMs) have recently obtained compet-
itive performance in terms of accuracy, memory
footprint, energy, and learning speed on several


key: chroma:document
string_value:
footprint, energy, and learning speed on several
benchmarks. Each TMclause votes for or against
a particular class, with classiﬁcation resolved us-
ing a majority vote. While the evaluation of
clauses is fast, being based on binary operators,
the voting makes it necessary to synchronize the
clause evaluation, impeding parallelization. In
this paper, we propose a novel scheme for desyn-
chronizing the evaluation of clauses, eliminating
the voting bottleneck. In brief, every clause runs


key: chroma:document
string_value:
in its own thread for massive native parallelism.
For each training example, we keep track of the
class votes obtained from the clauses in local vot-
ing tallies. The local voting tallies allow us to
detach the processing of each clause from the rest
of the clauses, supporting decentralized learning.
This means that the TMmost of the time will op-
erate on outdated voting tallies. We evaluated the
proposed parallelization across diverse learning
tasks and it turns out that our decentralized TM


key: chroma:document
string_value:
tasks and it turns out that our decentralized TM
learning algorithm copes well with working on
outdated data, resulting in no signiﬁcant loss in
learning accuracy. Furthermore, we show that the
proposed approach provides up to 50times faster
learning. Finally, learning time is almost constant
for reasonable clause amounts (employing from
20to7,000clauses on a Tesla V100 GPU). For
sufﬁciently large clause numbers, computation
time increases approximately proportionally. Our


key: chroma:document
string_value:
time increases approximately proportionally. Our
parallel and asynchronous architecture thus allows
processing of massive datasets and operating with
more clauses for higher accuracy.
*Equal contribution (The authors are ordered alphabetically by
last name.)1Department of Information and Communication Tech-
nology, Unviersity of Agder, Grimstad, Norway. Correspondence
to: Ole-Christoffer Granmo <ole.granmo@uia.no >.
Proceedings of the 38thInternational Conference on Machine


key: chroma:document
string_value:
Learning , PMLR 139, 2021. Copyright 2021 by the author(s).1. Introduction
Tsetlin machines ( TMs) (Granmo, 2018) have recently
demonstrated competitive results in terms of accuracy, mem-
ory footprint, energy, and learning speed on diverse bench-
marks (image classiﬁcation, regression, natural language
understanding, and speech processing) (Berge et al., 2019;
Yadav et al., 2021a; Abeyrathna et al., 2020; Granmo et al.,
2019; Wheeldon et al., 2020; Abeyrathna et al., 2021; Lei


key: chroma:document
string_value:
et al., 2021). They use frequent pattern mining and resource
allocation principles to extract common patterns in the data,
rather than relying on minimizing output error, which is
prone to overﬁtting. Unlike the intertwined nature of pat-
tern representation in neural networks, a TMdecomposes
problems into self-contained patterns, expressed as conjunc-
tive clauses in propositional logic (i.e., in the form ifinput
Xsatisﬁes condition Aand not condition Bthen output


key: chroma:document
string_value:
y= 1). The clause outputs, in turn, are combined into a
classiﬁcation decision through summation and thresholding,
akin to a logistic regression function, however, with binary
weights and a unit step output function. Being based on
the human-interpretable disjunctive normal form (Valiant,
1984), like Karnaugh maps (Karnaugh, 1953), a TMcan
map an exponential number of input feature value combina-
tions to an appropriate output (Granmo, 2018).


key: chroma:document
string_value:
tions to an appropriate output (Granmo, 2018).
Recent progress on TMs Recent research reports several
distinct TM properties. The TM can be used in convolution,
providing competitive performance on MNIST, Fashion-
MNIST, and Kuzushiji-MNIST, in comparison with CNNs,
K-Nearest Neighbor, Support Vector Machines, Random
Forests, Gradient Boosting, BinaryConnect, Logistic Cir-
cuits and ResNet (Granmo et al., 2019). The TMhas also
achieved promising results in text classiﬁcation (Berge et al.,


key: chroma:document
string_value:
2019), word sense disambiguation (Yadav et al., 2021b),
novelty detection (Bhattarai et al., 2021c;b), fake news de-
tection (Bhattarai et al., 2021a), semantic relation analysis
(Saha et al., 2020), and aspect-based sentiment analysis (Ya-
dav et al., 2021a) using the conjunctive clauses to capture
textual patterns. Recently, regression TMs compared favor-
ably with Regression Trees, Random Forest Regression, and
Support Vector Regression (Abeyrathna et al., 2020). The


key: chroma:document
string_value:
Massively Parallel and Asynchronous Tsetlin Machine Architecture
ous techniques. By introducing real-valued clause weights,
it turns out that the number of clauses can be reduced by
up to 50×without loss of accuracy (Phoulady et al., 2020).
Also, the logical inference structure of TMs makes it pos-
sible to index the clauses on the features that falsify them,
increasing inference- and learning speed by up to an order of
magnitude (Gorji et al., 2020). Multi-granular clauses sim-


key: chroma:document
string_value:
plify the hyper-parameter search by eliminating the pattern
speciﬁcity parameter (Gorji et al., 2019). In (Abeyrathna
et al., 2021), stochastic searching on the line automata (Oom-
men, 1997) learn integer clause weights, performing on-par
or better than Random Forest, Gradient Boosting, Neural
Additive Models, StructureBoost and Explainable Boosting
Machines. Closed form formulas for both local and global
TMinterpretation, akin to SHAP, was proposed by Blakely


key: chroma:document
string_value:
& Granmo (2020). From a hardware perspective, energy
usage can be traded off against accuracy by making infer-
ence deterministic (Abeyrathna et al., 2020). Additionally,
Shaﬁk et al. (2020) show that TMs can be fault-tolerant,
completely masking stuck-at faults. Recent theoretical work
proves convergence to the correct operator for “identity” and
“not”. It is further shown that arbitrarily rare patterns can
be recognized, using a quasi-stationary Markov chain-based


key: chroma:document
string_value:
analysis. The work ﬁnally proves that when two patterns are
incompatible, the most accurate pattern is selected (Zhang
et al., 2021). Convergence for the “XOR” operator has also
recently been proven by Jiao et al. (2021).
Paper Contributions In all of the above mentioned TM
schemes, the clauses are learnt using Tsetlin automaton
(TA)-teams (Tsetlin, 1961) that interact to build and in-
tegrate conjunctive clauses for decision-making. While


key: chroma:document
string_value:
producing accurate learning, this interaction creates a bottle-
neck that hinders parallelization. That is, the clauses must
be evaluated and compared before feedback can be provided
to the TAs.
In this paper, we ﬁrst cover the basics of TMs in Section 2.
Then, we propose a novel parallel and asynchronous archi-
tecture in Section 3, where every clause runs in its own
thread for massive parallelism. We eliminate the above
interaction bottleneck by introducing local voting tallies


key: chroma:document
string_value:
that keep track of the clause outputs, per training exam-
ple. The local voting tallies detach the processing of each
clause from the rest of the clauses, supporting decentralized
learning. Thus, rather than processing training examples
one-by-one as in the original TM, the clauses access the
training examples simultaneously, updating themselves and
the local voting tallies in parallel. In Section 4, we investi-
gate the properties of the new architecture empirically on


key: chroma:document
string_value:
regression, novelty detection, semantic relation analysis and
word sense disambiguation. We show that our decentralized
TMarchitecture copes well with working on outdated data,
with no measurable loss in learning accuracy. We furtherinvestigate how processing time scales with the number of
clauses, uncovering almost constant-time processing over
reasonable clause amounts. Finally, in Section 5, we con-
clude with pointers to future work, including architectures


key: chroma:document
string_value:
for grid-computing and heterogeneous systems spanning the
cloud and the edge.
The main contributions of the proposed architecture can be
summarized as follows:
• Learning time is made almost constant for reasonable
clause amounts (employing from 20to7,000clauses
on a Tesla V100 GPU).
•For sufﬁciently large clause numbers, computation
time increases approximately proportionally to the in-
crease in number of clauses.
•The architecture copes remarkably with working on


key: chroma:document
string_value:
outdated data, resulting in no signiﬁcant loss in learn-
ing accuracy across diverse learning tasks (regression,
novelty detection, semantic relation analysis, and word
sense disambiguation).
Our parallel and asynchronous architecture thus allows pro-
cessing of more massive data sets and operating with more
clauses for higher accuracy, signiﬁcantly increasing the im-
pact of logic-based machine learning.
2. Tsetlin Machine Basics
2.1. Classiﬁcation


key: chroma:document
string_value:
2. Tsetlin Machine Basics
2.1. Classiﬁcation
ATMtakes a vector X= [x1, . . . , x o]ofoBoolean features
as input, to be classiﬁed into one of two classes, y= 0or
y= 1. These features are then converted into a set of literals
that consists of the features themselves as well as their
negated counterparts: L={x1, . . . , x o,¬x1, . . . ,¬xo}.
If there are mclasses and nsub-patterns per class, a TM
employs m×nconjunctive clauses to represent the sub-


key: chroma:document
string_value:
patterns. For a given class1, we index its clauses by j,
1≤j≤n, each clause being a conjunction of literals:
Cj(X) =⋀
lk∈Ljlk. (1)
Here, lk,1≤k≤2o,is a feature or its negation. Further,
Ljis a subset of the literal set L. For example, the particular
clause Cj(X) =x1∧¬x2consists of the literals Lj=
{x1,¬x2}and outputs 1ifx1= 1andx2= 0.
The number of clauses nassigned to each class is user-
conﬁgurable. The clauses with odd indexes are assigned pos-


key: chroma:document
string_value:
itive polarity and the clauses with even indexes are assigned
1Without loss of generality, we consider only one of the classes,
thus simplifying notation. Any TMclass is modelled and processed
in the same way.


key: chroma:document
string_value:
Massively Parallel and Asynchronous Tsetlin Machine Architecture
negative polarity. The clause outputs are combined into a
classiﬁcation decision through summation and thresholding
using the unit step function u(v) = 1 ifv≥0else0:
ˆy=u(
Σn−1
j=1,3,...Cj(X)−Σn
i=2,4,...Cj(X))
.(2)
Namely, classiﬁcation is performed based on a majority
vote, with the positive clauses voting for y= 1 and the
negative for y= 0.
Input  
Type I feedback Type I feedbackinclude exclude include exclude


key: chroma:document
string_value:
1 N N+1 2 N1 N N+1 2 N
TAstransition
Step 1Step t
Figure 1. TMlearning dynamics for an XOR-gate training sample,
with input ( x1= 0, x2= 1) and output target y= 1.
2.2. Learning
TMlearning is illustrated in Fig. 1. As shown, a clause
Cj(X)is composed by a team of TAs. Each TAhas2N
states and decides to Include (from state 1toN) orExclude
(from state N+1to2N) a speciﬁc literal lkin the clause. In
the ﬁgure, TArefers to the TAs that control the original form


key: chroma:document
string_value:
of a feature ( x1andx2) while TA’ refers to those controlling
negated features (¬x1and¬x2). A TAupdates its state
based on the feedback it receives in the form of Reward,
Inaction, and Penalty (illustrated by the features moving in
a given direction in the TA-part of the ﬁgure). There are
two types of feedback associated with TMlearning: Type I
feedback and Type II feedback, which are shown in Table 1
and Table 2, respectively.
Type I feedback is given stochastically to clauses with odd


key: chroma:document
string_value:
indexes when y= 1and to clauses with even indexes when
y= 0. Each clause, in turn, reinforces its TAs based on:
(1) its output Cj(X); (2) the action of the TA–Include or
Exclude ; and (3) the value of the literal lkassigned to the
TA. As shown in Table 1, two rules govern Type I feedback:
•Include is rewarded and Exclude is penalized with prob-
abilitys−1
sifCj(X) = 1 andlk= 1. This reinforce-
ment is strong2(triggered with high probability) and
2Note that the probabilitys−1


key: chroma:document
string_value:
2Note that the probabilitys−1
sis replaced by 1when boosting
true positives.INPUTCLAUSE 1 0
LITERAL 1 0 1 0
INCLUDE LITERALP(R EWARD )s−1
sNA 0 0
P(INACTION )1
sNAs−1
ss−1
s
P(P ENALTY ) 0 NA1
s1
s
EXCLUDE LITERALP(R EWARD ) 01
s1
s1
s
P(INACTION )1
ss−1
ss−1
ss−1
s
P(P ENALTY )s−1
s0 0 0
Table 1. Type I Feedback
INPUTCLAUSE 1 0
LITERAL 1 0 1 0
INCLUDE LITERALP(R EWARD ) 0 NA 0 0
P(INACTION )1.0 NA 1.0 1.0
P(P ENALTY ) 0 NA 0 0
EXCLUDE LITERALP(R EWARD ) 0 0 0 0
P(INACTION )1.0 0 1.0 1.0


key: chroma:document
string_value:
P(INACTION )1.0 0 1.0 1.0
P(P ENALTY ) 0 1.0 0 0
Table 2. Type II Feedback
makes the clause remember and reﬁne the pattern it
recognizes in X.
•Include is penalized and Exclude is rewarded with prob-
ability1
sifCj(X) = 0 orlk= 0. This reinforcement
is weak (triggered with low probability) and coarsens
infrequent patterns, making them frequent.
Above, parameter scontrols pattern frequency.
Type II feedback is given stochastically to clauses with


key: chroma:document
string_value:
odd indexes when y= 0and to clauses with even indexes
when y= 1. As captured by Table 2, it penalizes Exclude
with probability 1ifCj(X) = 1 andlk= 0. Thus, this
feedback produces literals for discriminating between y= 0
andy= 1.
The “state” is realized as a simple counter per Tsetlin
Automaton. In practice, reinforcing Include (penalizing
Exclude or rewarding Include) is done by increasing the
counter, while reinforcing exclude (penalizing Include or


key: chroma:document
string_value:
rewarding Exclude) is performed by decreasing the counter.
As an example of learning, let us consider a dataset with
XOR-gate sub-patterns. In particular, consider the input
(x1= 0, x2= 1) and target output y= 1to visualize the
learning process (Fig. 1). We further assume that we have
n= 4clauses per class. Among the 4clauses, the clauses
C1andC3vote for y= 1and the clauses C0andC2vote for
y= 0. For clarity, let us only consider how C1andC3learn


key: chroma:document
string_value:
a sub-pattern from the given sample of XOR-gate input and
output. At Step 1 in the ﬁgure, the clauses have not yet learnt
the pattern for the given sample. This leads to the wrong
class prediction (ˆy= 0) , thereby triggering Type I feedback


key: chroma:document
string_value:
Massively Parallel and Asynchronous Tsetlin Machine Architecture
for the corresponding literals. Looking up clause output
C1(X) = 0 and literal value x1= 0 in Table 1, we note
that the TAcontrolling x1receives either Inaction or Penalty
feedback for including x1inC1, with probabilitys−1
sand
1
s, respectively. After receiving several penalties, with high
probability, the TAchanges its state to selecting Exclude .
Accordingly, literal x1gets removed from the clause C1.


key: chroma:document
string_value:
After that, the TAthat has excluded literal ¬x1fromC1also
obtains penalties, and eventually switches to the Include
side of its state space. The combined outcome of these
updates are shown in Step tforC1. Similarly, the TAthat
has included literal ¬x2in clause C1receives Inaction or
Penalty feedback with probabilitys−1
sand1
s, respectively.
After obtaining multiple penalties, with high probability,
¬x2becomes excluded from C1. Simultaneously, the TA


key: chroma:document
string_value:
that controls x2ends up in the Include state, as also shown
in Step t. At this point, both clause C1andC3outputs 1for
the given input, correctly predicting the output ˆy= 1.
Resource allocation dynamics ensure that clauses dis-
tribute themselves across the frequent patterns, rather than
missing some and over-concentrating on others. That is,
for any input X, the probability of reinforcing a clause
gradually drops to zero as the clause output sum
v= Σn−1
j=1,3,...Cj(X)−Σn
i=2,4,...Cj(X) (3)


key: chroma:document
string_value:
v= Σn−1
j=1,3,...Cj(X)−Σn
i=2,4,...Cj(X) (3)
approaches a user-set margin Tfory= 1 (and−Tfor
y= 0). If a clause is not reinforced, it does not give feed-
back to its TAs, and these are thus left unchanged. In the
extreme, when the voting sum vequals or exceeds the mar-
ginT(the TM has successfully recognized the input X),
no clauses are reinforced. They are then free to learn new
patterns, naturally balancing the pattern representation re-
sources (Granmo, 2018).


key: chroma:document
string_value:
sources (Granmo, 2018).
3. Parallel and Asynchronous Architecture
Even though CPUs have been traditionally geared to handle
high workloads, they are more suited for sequential process-
ing and their performance is still dependant on the limited
number of cores available. In contrast, since GPUs are pri-
marily designed for graphical applications by employing
many small processing elements, they offer a large degree
of parallelism (Owens et al., 2007). As a result, a growing


key: chroma:document
string_value:
body of research has been focused on performing general
purpose GPU computation or GPGPU. For efﬁcient use of
GPU power, it is critical for the algorithm to expose a large
amount of ﬁne-grained parallelism (Jiang & Snir, 2005;
Satish et al., 2009).
While the voting step of the TM(Eq. 3) hinders paral-
lelization, the remainder of the TMarchitecture is natively
parallel. In this section, we introduce our decentralized
inference scheme and the accompanying architecture that


key: chroma:document
string_value:
makes it possible to have parallel asynchronous learningand classiﬁcation, resolving the voting bottleneck by using
local voting tallies.
3.1. Voting Tally
A voting tally that tracks the aggregated output of the clauses
for each training example is central to our scheme. In a
standard TM, each training example (Xi, yi),1≤i≤q,
is processed by ﬁrst evaluating the clauses on Xiand then
obtaining the majority vote vfrom Eq. (3). Here, qis the
total number of examples. The majority vote vis then


key: chroma:document
string_value:
compared with the summation margin Twhen y= 1and
−Twhen y= 0, to produce the feedback to the TAs of each
clause, explained in the previous section.
Figure 2. Parallel Tsetlin machine architecture.
As illustrated in Fig. 2, to decouple the clauses, we now
assume that the particular majority vote of example Xihas
been pre-calculated, meaning that each training example
becomes a triple (Xi, yi, vi), where viis the pre-calculated
majority vote. With viin place, the calculation performed


key: chroma:document
string_value:
in Eq. (3) can be skipped, and we can go directly to give
Type I or Type II feedback to any clause Cj, without con-
sidering the other clauses. This opens up for decentralized
learning of the clauses, facilitating native parallelization
at all inference and learning steps. The drawback of this
scheme, however, is that any time the composition of a
clause changes after receiving feedback, all voting aggre-
gates vi,1≤i≤q,become outdated. Accordingly, the


key: chroma:document
string_value:
gates vi,1≤i≤q,become outdated. Accordingly, the
standard learning scheme for updating clauses must be re-
placed.
3.2. Decentralized Clause Learning
Our decentralized learning scheme is captured by Algo-
rithm 1. As shown, each clause is trained independently of
the other clauses. That is, each clause proceeds with training
without taking other clauses into consideration. Algorithm 1
thus supports native parallelization because each clause now
can run independently in its own thread.


key: chroma:document
string_value:
Massively Parallel and Asynchronous Tsetlin Machine Architecture
Algorithm 1 Decentralized updating of clause
Input: Example pool P, clause Cj, positive polarity
indicator pj∈ {0,1}, batch size b∈[1,∞), voting
margin T∈[1,∞), pattern speciﬁcity s∈[1,∞).
Procedure: UpdateClause : Cj, pj, P, b, T, s .
fori= 1tobdo
(Xi, yi, vi)←ObtainTrainingExample( P)
vc
i←clip(vi,−T, T)
e=T−vc
iifyi= 1elseT+vc
i
ifrand()≤e
2Tthen
ifyixorpjthen
Cj←TypeIIFeedback( Xi, Cj)
else
Cj←TypeIFeedback( Xi, Cj, s)
end if


key: chroma:document
string_value:
else
Cj←TypeIFeedback( Xi, Cj, s)
end if
oij←Cj(Xi)
o∗
ij←ObtainPreviousClauseOutput( i, j)
ifoij̸=o∗
ijthen
AtomicAdd( vi, oij−o∗
ij)
StorePreviousClauseOutput( i, j, o ij)
end if
end if
end for
Notice further how the clause in focus ﬁrst obtains a ref-
erence to the next training example (Xi, yi, vi)to process,
including the pre-recorded voting sum vi(Line 3). This
example is retrieved from an example pool P, which is the
storage of the training examples (centralized or decentral-
ized).


key: chroma:document
string_value:
ized).
The error of the pre-recorded voting sum viis then calcu-
lated based on the voting margin T(Line 5). The error, in
turn, decides the probability of updating the clause. The up-
dating rule is the standard Type I and Type II TMfeedback,
governed by the polarity pjof the clause and the speciﬁcity
hyper-parameter s(Lines 6-11).
The moment clause Cjis updated, all recorded voting sums
in the example pool Pare potentially outdated. This is


key: chroma:document
string_value:
because Cjnow captures a different pattern. Thus, to keep
all of the voting sums viinPconsistent with Cj,Cjshould
ideally have been re-evaluated on all of the examples in P.
To partially remedy for outdated voting aggregates, the
clause only updates the current voting sum vi. This happens
when the calculated clause output oijis different from the
previously calculated clause output o∗
ij(Lines 12-17). Note
that the previously recorded output o∗
ijis a single bit that


key: chroma:document
string_value:
ijis a single bit that
is stored locally together with the clause. In this manner,
the algorithm provides eventual consistency . That is, if
the clauses stop changing, all the voting sums eventually
become correct.A point to note here is that there is no guarantee that the
clause outputs in the parallel version will sum up to the
same number as in the sequential version. The reason is that
the updating of clauses is asynchronous, and the vote sums


key: chroma:document
string_value:
are not updated immediately when a clause changes. We
use the term eventual consistency to refer to the fact that
if the clauses stop changing, eventually, the tallied voting
sums become the exact sum of the clause outputs. Although
not analytically proven, experimental results show that the
two versions provide consistent ﬁnal accuracy results after
clause summation and thresholding.
Employing the above algorithm, the clauses access the train-


key: chroma:document
string_value:
ing examples simultaneously, updating themselves and the
local voting tallies in parallel. There is no synchronization
among the clause threads, apart from atomic adds to the
local voting tallies (Line 15).
4. Empirical Results
In this section, we investigate how our new approach to
TMlearning scales, including effects on training time and
accuracy. We employ seven different data sets that repre-
sent diverse learning tasks, including regression, novelty


key: chroma:document
string_value:
detection, sentiment analysis, semantic relation analysis,
and word sense disambiguation. The data sets are of var-
ious sizes, spanning from 300to100,000examples, 2to
20classes, and 6to102,176features. We have striven to
recreate TMexperiments reported by various researchers,
including their hyper-parameter settings. For comparison of
performance, we contrast with fast single-core TMimple-
mentations3both with and without clause indexing (Gorji


key: chroma:document
string_value:
et al., 2020). Our proposed architecture is implemented in
CUDA and runs on a Tesla V100 GPU (grid size 208and
block size 128). The standard implementations run on an
Intel Xeon Platinum 8168 CPU at 2.70GHz. Code is avail-
able at https://github.com/cair/ICML-Massively-Parallel-
and-Asynchronous-Tsetlin-Machine-Architecture.
We summarize the obtained performance metrics in Table 3.
For better reproducibility, each experiment is repeated ﬁve


key: chroma:document
string_value:
times and the average accuracy and standard deviation are
reported. We also report how much faster the CUDA TM
executes compared with the indexed version.
The focus in this section is to highlight the speed-ups
achieved via parallelization of the TM architecture. How-
ever, for the sake of completeness, we also include corre-
sponding results for SVM using ThunderSVM, which is
an established GPU implementation of SVM (Wen et al.,
2018). The performance results are compiled in Fig. 44.


key: chroma:document
string_value:
As seen, our TM parallelization is signiﬁcantly faster than
3Retrieved from https://github.com/cair/pyTsetlinMachine.
4Google’s Colab is used to run the two CUDA models in the
table for fair comparison.


key: chroma:document
string_value:
Massively Parallel and Asynchronous Tsetlin Machine Architecture
DATASET TM INDEXED TM NON -INDEXED TM CUDA SPEED UP
ACC F1 ACC F1 ACC F1
BBC S PORTS 85.08 ±1.75 85.58 ±1.69 87.36 ±1.91 88.02 ±1.47 84.64 ±2.20 86.13 ±2.24 38.9×
20 N EWSGROUP 79.37 ±0.25 80.38 ±0.92 82.33 ±0.28 82.89 ±0.34 79.00 ±0.46 78.93 ±0.44 49.3×
SEMEVAL 91.9 ±0.16 75.29 ±0.25 92.51 ±0.03 77.48 ±1.46 92.02 ±0.54 76.27 ±0.53 1.7×
IMDB 88.42 ±2.05 88.39 ±2.16 88.2 ±3.14 88.13 ±3.44 89.92 ±0.23 88.90 ±0.24 34.6×


key: chroma:document
string_value:
JAVA 97.03 ±0.02 96.93 ±0.02 97.50 ±0.02 97.40 ±0.02 97.53 ±0.02 97.42 ±0.01 6.0×
APPLE 92.65 ±0.02 92.20 ±0.02 92.46 ±0.01 91.82 ±0.02 95.01 ±0.01 94.68 ±0.01 9.7×
Table 3. Performance on multiple data sets. Mean and standard deviation are calculated over 5 independent runs. Speed up is calculated as
how many times faster is average execution time on CUDA implementation than on Indexed implementation.
ThunderSVM for all the investigated datasets. Note that


key: chroma:document
string_value:
our parallelization solution leverages novel TM properties,
allowing each TM clause to learn independently in its own
thread.
4.1. Regression
We ﬁrst investigate performance with regression Tsetlin
machines ( RTM s) using two datasets: Bike Sharing and
BlogFeedback.
The Bike Sharing dataset contains a 2-year usage log of the
bike sharing system Captial Bike Sharing (CBS) at Wash-
ington, D.C., USA. The total number of rental bikes (casual


key: chroma:document
string_value:
and registered), is predicted based on other relevant features
such as time of day, day of week, season, and weather. In
total, 16independent features are employed to predict the
number of rental bikes. Overall, the dataset contains 17,389
examples. More details of the Bike Sharing dataset can be
found in (Fanaee-T & Gama, 2014).
The BlogFeedback dataset considers predicting the number
of blog posts comments for the upcoming 24hours. The
data has been gathered from January, 2010 to March, 2012.


key: chroma:document
string_value:
In total, BlogFeedback contains 60,021data samples, each
with281features. In this study, we employ the ﬁrst 20,000
data samples. For both of the datasets, we use 80% of the
samples for training and the rest for evaluation.
We ﬁrst study the impact of the number of clauses on pre-
diction error, measured by Mean Absolute Error (MAE). As
illustrated in Fig. 3 for Bike Sharing, increasing the num-
ber of clauses (#clauses in x-axis) decreases the error by


key: chroma:document
string_value:
allowing the RTM to capture more detailed sub-patterns.
A larger number of clauses results in increased computation
time. Fig. 4 captures how execution time increases with the
number of clauses for the three different implementations.
For the non-indexed RTM , each doubling of the number of
clauses also doubles execution time. However, the indexed
RTM is less affected, and is slightly faster than the CUDA
implementation when less than 300 clauses are utilized.


key: chroma:document
string_value:
With more than 300 clauses, the CUDA implementation is
superior, with no signiﬁcant increase in execution time as
the number of clauses increases. For instance, using 1280
10 20 40 80 160 320 640 1280
Clauses20304050607080Error (MAE)Figure 3. MAE vs. #clauses on Bike Sharing.
10 20 40 80 160 320 640 1280
Clauses0200040006000800010000Execution time [sec.]Non-Indexed
Indexed
CUDA
Figure 4. Execution time vs. #clauses on Bike Sharing.
clauses, the CUDA implementation is roughly 3.64times


key: chroma:document
string_value:
faster than the indexed version. This can be explained by
the large number of threads available to the GPU and the
asynchronous operation of the new architecture.
Looking at how MAE and execution time vary over the
training epochs for Bike Sharing (Fig. 5 and Fig. 6, respec-
tively), we observe that MAE falls systematically across the


key: chroma:document
string_value:
Massively Parallel and Asynchronous Tsetlin Machine Architecture
CUDA M ODEL BBC S PORTS 20 NG SEMEVAL IMDB JAVA(WSD) APPLE (WSD) REGR 1 REGR 2
THUNDER SVM 0.23 S 6.95 S 7.42 S 3.89 S 1.31 S 0.39 S 6.24 S 59.94 S
TM 0.13 S 3.61 S 4.27 S 3.36 S 0.74 S 0.28 S 0.52 S 4.92 S
Table 4. Comparison of training time between ThunderSVM and TM. The run time is an average of 5 experiments.
0 20 40 60 80 100
Epochs203040506070Error (MAE)Non-Indexed
Indexed
CUDA
Figure 5. MAE over epochs on Bike Sharing.


key: chroma:document
string_value:
CUDA
Figure 5. MAE over epochs on Bike Sharing.
0 20 40 60 80 100
Epochs020406080100Execution Time [sec.]Non-Indexed
Indexed
CUDA
Figure 6. Per Epoch Execution time over epochs on Bike Sharing.
epochs, while the execution time remains stable (employing
T= 1280 ,s= 1.5,n= 1280 ). Execution on BlogFeed-
back exhibits similar behavior. Finally, the mean value of
the MAEs for each method are similar (Table 5) across 5
independent runs, indicating no signiﬁcant difference in
learning accuracy.


key: chroma:document
string_value:
learning accuracy.
Comparing MAE with comparable contemporary methods
on Bike Sharing (Regr1), we observe that the TM is com-
petitive – TM: 23.9; ANN: 27.4; and SVR: 22.6. Similarly,
for BlockFeedback (Regr2) we have – TM: 3.88; ANN: 7.2;
and SVR: 6.0.4.2. Novelty Detection
Novelty detection is another important machine learning
task. Most supervised classiﬁcation approaches assume a
closed world, counting on all classes being present in the


key: chroma:document
string_value:
data at training time. This assumption can lead to unpre-
dictable behaviour during operation, whenever novel, previ-
ously unseen, classes appear. We here investigate TM-based
novelty detection, as proposed by Bhattarai et al. (2021c;b).
To this end, we use the two datasets: 20 Newsgroup and
BBC Sports. In brief, we use the class voting sums (Section
2) as features measuring novelty. We then employ a Mul-
tilayer perceptron (MLP) for novelty detection that adopts


key: chroma:document
string_value:
the class voting sums as input.
The BBC sports dataset contains 737documents from the
BBC sport website, organized in ﬁve sports article categories
and collected from 2004 to 2005. Overall, the dataset en-
compasses a total of 4,613terms. For novelty classiﬁcation,
we designate the classes “Cricket” and “Football” as known
and “Rugby” as novel. We train on the known classes, which
runs for 100epochs with 5,000clauses, margin Tof100,
and speciﬁcity sof15.0. The training times for both in-


key: chroma:document
string_value:
dexed and non-indexed TMs are high compared with that of
CUDA TM, which is around 39times faster. The 20 News-
group dataset contains 18,828documents with 20classes.
The classes “comp.graphics” and “talk.politics.guns” are
designated as known, and “rec.sport.baseball” is considered
novel. We train the TMfor100epochs with a margin Tof
500,10,000clauses and speciﬁcity s= 25.0. The CUDA
TMimplementation is here about 49times faster than the
other versions.


key: chroma:document
string_value:
other versions.
To assess scalability, we record the execution time of both
the indexed and the CUDA TMwhile increasing the number
of clauses (Fig. 7). For the indexed TM, the execution time
increases almost proportionally with the number of clauses,
but no such effect is noticeable for the CUDA TM.
The novelty scores generated by the TMare passed to a
Multilayer Perceptron with hidden layer sizes ( 100,30) and
RELU activation functions, trained using stochastic gradient


key: chroma:document
string_value:
descent. As shown in Table 3, for both datasets, the non-
indexed TMslightly outperforms the other TMversions,
while the indexed and CUDA TMs have similar accuracy.
These differences can be explained by the random variation
ofTMlearning (i.e., the high standard deviations reported
in Table 3).


key: chroma:document
string_value:
Massively Parallel and Asynchronous Tsetlin Machine Architecture
DATASETSTM
INDEXEDTM
NON -INDEXEDTM
CUDASPEED UP
BIKESHARING 23.5±0.04 22.5±0.12 23.9±0.08 156.4×
BLOGFEEDBACK .3.91±0.00 3.74±0.03 3.88±0.02 211.6×
Table 5. MAE with conﬁdence interval, and Speed up on two regression datasets, calculated over 5 independent runs
Figure 7. Execution time vs. #clauses on BBC Sports.
Bhattarai et al. (2021c) investigate the competitiveness
of TMs on novelty detection. For BBC Sports they re-


key: chroma:document
string_value:
port the following accuracies – TM: 89.47%; One-class
SVM: 83.53%; and Average KNN: 55.54%. Likewise, for
20Newsgroup they report – TM: 82.51%; One-class SVM:
83.70%; and Average KNN: 81%.
4.3. Sentiment and Semantic Analysis
We adopt the SemEval 2010 Semantic Relations (Hendrickx
et al., 2009) and the ACL Internet Movie Database (IMDb)
(Maas et al., 2011) datasets to explore the performance of the
TMimplementations for a large number of sparse features,
as proposed by Saha et al. (2020).


key: chroma:document
string_value:
as proposed by Saha et al. (2020).
The SEMEV AL dataset focuses on identifying semantic
relations in text. The dataset has 10,717examples, and we
consider each to be annotated to contain either the relation
Cause-Effect or not. The presence of an unambiguous causal
connective is indicative of a sentence being a causal sentence
(Xuelan & Kennedy, 1992). For each TM, we use 40clauses
per class to identify this characteristic of causal texts. The


key: chroma:document
string_value:
IMDb dataset contains 50,000highly polar movie reviews,
which are either positive or negative. Due to the large variety
and combination of possibly distinguishing features, we
assign 7,000clauses to each class. For both datasets we use
unigrams and bigrams as features.
As noted in Table 3, the accuracy obtained by the CPU (non-
indexed) and the CUDA implementations are comparable
on the SEMEV AL dataset, while the indexed TMperforms
Figure 8. Per Epoch Execution time over epochs on SEMEV AL.


key: chroma:document
string_value:
slightly poorer. However, the execution time is much lower
for the CUDA version than the other two (Fig. 8). This is
further shown in Fig. 9. Clearly, the CPU-based TMwith
indexing takes an increasing amount of time to execute as
the number of clauses grows, but no such effect is observed
with CUDA TM. More speciﬁcally, the execution time
increases only by 40% when the number of clauses goes
from 20to2,560.
With the IMDB dataset, the CUDA version performs better


key: chroma:document
string_value:
in terms of accuracy, with less variance compared to the
CPU versions (Table 3). It exhibits similar behaviour as in
the SEMEV AL dataset with respect to execution time over
increasing number of epochs.
The following provides a comparison of accuracy, compiled
from related literature. For Semantic Relation Classiﬁca-
tion (SemEval) TM accuracy is again competitive (TM:
92.6%; RandomForest: 88.76%; NaiveBayes: 87.57%;
SVM: 92.3%; and CNN-LSTM: 90.73%). The same


key: chroma:document
string_value:
SVM: 92.3%; and CNN-LSTM: 90.73%). The same
goes for IMDB Sentiment Analysis (TM: 90.5%; NBOW:
83.62%; CNN-BiLSTM: 89.54%; Tree-LSTM: 90.1%; Self-
AT-LSTM: 90.34%; and DistilBERT: 92.82%).
4.4. Word Sense Disambiguation
Word Sense Disambiguation (WSD) is a vital task in
NLP (Navigli, 2009) that consists of distinguishing the
meaning of homonyms – identically spelled words whose
sense depends on the surrounding context words. We here
perform a quantitative evaluation of the three TMimplemen-


key: chroma:document
string_value:
Massively Parallel and Asynchronous Tsetlin Machine Architecture
Figure 9. Execution time vs. #clauses on SEMEV AL.
tations using a recent WSD evaluation framework (Loureiro
et al., 2020) based on WordNet. We use a balanced dataset
for coarse-grained classiﬁcation, focusing on two speciﬁc
domains. The ﬁrst dataset concerns the meaning of the word
“Apple”, which here has two senses: “apple inc.” (company)
and “apple apple” (fruit). The other dataset covers the word


key: chroma:document
string_value:
“JA V A”, which has the two senses: “java java” (geographi-
cal location) and “java comp.” (computer language). The
Apple dataset has 2,984samples split into training and test-
ing samples of 1,784and1,200, respectively. The JA V A
dataset has 5,655samples split into 3,726and1,929sam-
ples, for training and testing. For preprocessing, we ﬁlter
the stop words and stem the words using the Porter Stemmer
to reduce the effect of spelling mistakes or non-important


key: chroma:document
string_value:
variations of the same word. To build a vocabulary (the fea-
ture space), we select the 3,000most frequent terms. The
number of clauses n, margin T, and speciﬁcity sused are
300,50,5.0respectively, for both datasets.
The accuracy and F1 scores of the non-indexed and indexed
TMs are quite similar for the Apple dataset (Table 3). How-
ever, the CUDA TM outperforms both of them by a signiﬁ-
cant margin. In the case of JA V A dataset, the performance


key: chroma:document
string_value:
is comparable for all three, and CUDA TMis slightly better.
The reader is requested to refer to (Yadav et al., 2021b) for
further details on TM-based WSD.
The following is a comparison of accuracy with respect
to related literature (Loureiro et al., 2020). For Apple we
have TM: 95.1%; ThunderSVM: 80.32%; FastTextBase-
1NN: 96.3%; FastTextCommonCrawl-1NN: 97.8%; and
BERTbase: 99.0%. Correspondingly, for JA V A we get
TM:97.53%; ThunderSVM: 93.62%; FastTextBase-1NN:


key: chroma:document
string_value:
TM:97.53%; ThunderSVM: 93.62%; FastTextBase-1NN:
98.7%; FastTextCommonCrawl-1NN: 99.5%; and BERT-
base: 99.6%. Note that the latter TM results are from
our current paper. Another recent paper, however, shows
that better hyperparameters can make the TMoutperformFastTextBase-1NN, with the TM reaching 97.58% and
99.38% for Apple and JA V A, respectively (Yadav et al.,
2021b).
4.5. Analysis of Experiment Results
The above results support our initial claims that the pro-


key: chroma:document
string_value:
posed TMarchitecture provides learning speeds that are
nearly constant for a reasonable number of clauses. Beyond
a sufﬁciently large clause number, the computation time
increases at approximately the same rate as the increase in
number of clauses. The number of clauses to use highly de-
pends on the dataset. In our case, using up to 7,000clauses
provides high accuracy for all the datasets, simultaneously
demonstrating almost constant training time overall. A fur-


key: chroma:document
string_value:
ther increase does not signiﬁcantly improve accuracy. I.e.,
the number of clauses was decided empirically to provide a
general benchmark. Employing 7,000clauses also allows
us evaluate the capability of leveraging the 5,120cores on
the GPU. Hence, considering both datasets and available
cores, 7,000clauses are enough to demonstrate the almost
constant time scaling, until exhausting the cores. However,
we also extended the clauses to 15,000as shown in Fig. 7,


key: chroma:document
string_value:
and the execution time remained almost the same, thereby
validating excellent exploitation of the available cores. To
achieve this, our scheme adds 1bit per example per clause
for tallying, enabling desynchronization. TMs are further
reported to use less memory than ANNs (Lei et al., 2020).
5. Conclusions and Future Work
In this paper, we proposed a new approach to TMlearning,
to open up for massively parallel processing. Rather than
processing training examples one-by-one as in the original


key: chroma:document
string_value:
TM, the clauses access the training examples simultane-
ously, updating themselves and local voting tallies in paral-
lel. The local voting tallies allow us to detach the processing
of each clause from the rest of the clauses, supporting de-
centralized learning. There is no synchronization among the
clause threads, apart from atomic adds to the local voting
tallies. Operating asynchronously, each team of TAs most of
the time executes on partially calculated or outdated voting
tallies.


key: chroma:document
string_value:
tallies.
Based on the numerical results, our main conclusion is
thatTMlearning is very robust towards relatively severe
distortions of communication and coordination among the
clauses. Our results are thus compatible with the ﬁndings
of Shaﬁk et al. (2020), who shows that TM learning is
inherently fault tolerant, completely masking stuck-at faults.
In our future work, we will investigate the robustness of TM
learning further, which includes developing mechanisms


key: chroma:document
string_value:
Massively Parallel and Asynchronous Tsetlin Machine Architecture
References
Abeyrathna, K. D., Granmo, O.-C., Shaﬁk, R., Yakovlev, A.,
Wheeldon, A., Lei, J., and Goodwin, M. A Novel Multi-
Step Finite-State Automaton for Arbitrarily Deterministic
Tsetlin Machine Learning. In Proceedings of the 40th
International Conference on Innovative Techniques and
Applications of Artiﬁcial Intelligence (SGAI), Cambridge,
UK. Springer International Publishing, 2020.


key: chroma:document
string_value:
UK. Springer International Publishing, 2020.
Abeyrathna, K. D., Granmo, O.-C., Zhang, X., Jiao, L.,
and Goodwin, M. The Regression Tsetlin Machine - A
Novel Approach to Interpretable Non-Linear Regression.
Philosophical Transactions of the Royal Society A , 378,
2020.
Abeyrathna, K. D., Granmo, O.-C., and Goodwin, M.
Extending the Tsetlin Machine With Integer-Weighted
Clauses for Increased Interpretability. IEEE Access , 9:
8233 – 8248, 2021.
Berge, G. T., Granmo, O.-C., Tveit, T., Goodwin, M.,


key: chroma:document
string_value:
Jiao, L., and Matheussen, B. Using the Tsetlin Machine
to Learn Human-Interpretable Rules for High-Accuracy
Text Categorization With Medical Applications. IEEE
Access , 7:115134–115146, 2019.
Bhattarai, B., Granmo, O.-C., and Jiao, L. Explainable
Tsetlin Machine framework for fake news detection with
credibility score assessment, 2021a.
Bhattarai, B., Granmo, O.-C., and Jiao, L. Word-level
Human Interpretable Scoring Mechanism for Novel Text
Detection Using Tsetlin Machines, 2021b.


key: chroma:document
string_value:
Detection Using Tsetlin Machines, 2021b.
Bhattarai, B., Jiao, L., and Granmo, O.-C. Measuring the
Novelty of Natural Language Text Using the Conjunctive
Clauses of a Tsetlin Machine Text Classiﬁer. In 13th Inter-
national Conference on Agents and Artiﬁcial Intelligence
(ICAART), Vienna , Austria . INSTICC, 2021c.
Blakely, C. D. and Granmo, O.-C. Closed-Form Expressions
for Global and Local Interpretation of Tsetlin Machines
with Applications to Explaining High-Dimensional Data.


key: chroma:document
string_value:
arXiv preprint arXiv:2007.13885 , 2020.
Fanaee-T, H. and Gama, J. Event labeling combining en-
semble detectors and background knowledge. Progress
in Artiﬁcial Intelligence , 2(2-3):113–127, 2014.
Gorji, S., Granmo, O. C., Glimsdal, S., Edwards, J., and
Goodwin, M. Increasing the Inference and Learning
Speed of Tsetlin Machines with Clause Indexing. In
International Conference on Industrial, Engineering
and Other Applications of Applied Intelligent Systems


key: chroma:document
string_value:
(IEA/AIE), kitakyushu, Japan . Springer, 2020.Gorji, S. R., Granmo, O.-C., Phoulady, A., and Goodwin, M.
A Tsetlin Machine with Multigranular Clauses. In Pro-
ceedings of the Thirty-ninth International Conference on
Innovative Techniques and Applications of Artiﬁcial Intel-
ligence (SGAI), Cambridge, UK , volume 11927. Springer
International Publishing, 2019.
Granmo, O.-C. The Tsetlin Machine - A Game The-
oretic Bandit Driven Approach to Optimal Pattern


key: chroma:document
string_value:
oretic Bandit Driven Approach to Optimal Pattern
Recognition with Propositional Logic. arXiv preprint
arXiv:1804.01508 , 2018.
Granmo, O.-C., Glimsdal, S., Jiao, L., Goodwin, M., Om-
lin, C. W., and Berge, G. T. The Convolutional Tsetlin
Machine. arXiv preprint arXiv:1905.09688 , 2019.
Hendrickx, I., Kim, S. N., Kozareva, Z., Nakov, P.,
´O S´eaghdha, D., Pad ´o, S., Pennacchiotti, M., Romano,
L., and Szpakowicz, S. Semeval-2010 task 8: Multi-


key: chroma:document
string_value:
way classiﬁcation of semantic relations between pairs of
nominals. In Proceedings of the Workshop on Semantic
Evaluations: Recent Achievements and Future Directions,
Boulder, Colorado , pp. 94–99. ACL, 2009.
Jiang, C. and Snir, M. Automatic tuning matrix multi-
plication performance on graphics hardware. In 14th
International Conference on Parallel Architectures and
Compilation Techniques (PACT’05), Krasnoyarsk, Russia ,
pp. 185–194. IEEE, 2005.


key: chroma:document
string_value:
pp. 185–194. IEEE, 2005.
Jiao, L., Zhang, X., Granmo, O.-C., and Abeyrathna, K. D.
On the Convergence of Tsetlin Machines for the XOR
Operator. arXiv preprint arXiv:2101.02547 , 2021.
Karnaugh, M. The map method for synthesis of combi-
national logic circuits. Transactions of the American
Institute of Electrical Engineers, Part I: Communication
and Electronics , 72(5):593–599, 1953.
Lei, J., Wheeldon, A., Shaﬁk, R., Yakovlev, A., and Granmo,


key: chroma:document
string_value:
O.-C. From Arithmetic to Logic based AI: A Comparative
Analysis of Neural Networks and Tsetlin Machine. In
2020 27th IEEE International Conference on Electronics,
Circuits and Systems (ICECS) , pp. 1–4. IEEE, 2020.
Lei, J., Rahman, T., Shaﬁk, R., Wheeldon, A., Yakovlev, A.,
Granmo, O.-C., Kawsar, F., and Mathur, A. Low-Power
Audio Keyword Spotting Using Tsetlin Machines. Jour-
nal of Low Power Electronics and Applications , 11, 2021.
URLhttps://www.mdpi.com/2079-9268/11/
2/18 .


key: chroma:document
string_value:
URLhttps://www.mdpi.com/2079-9268/11/
2/18 .
Loureiro, D., Rezaee, K., Pilehvar, M. T., and Camacho-
Collados, J. Language models and word sense disam-
biguation: An overview and analysis, 2020.


key: chroma:document
string_value:
Massively Parallel and Asynchronous Tsetlin Machine Architecture
Maas, A., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y ., and
Potts, C. Learning word vectors for sentiment analysis.
InProceedings of the 49th annual meeting of the asso-
ciation for computational linguistics: Human language
technologies, Portland, Oregon, USA , pp. 142–150. ACL,
2011.
Navigli, R. Word sense disambiguation: A survey. ACM
Computing Surveys , 41:10:1–10:69, 2009.


key: chroma:document
string_value:
Computing Surveys , 41:10:1–10:69, 2009.
Oommen, B. J. Stochastic searching on the line and its ap-
plications to parameter learning in nonlinear optimization.
IEEE Transactions on Systems, Man, and Cybernetics,
Part B (Cybernetics) , 27(4):733–739, 1997.
Owens, J. D., Luebke, D., Govindaraju, N., Harris, M.,
Kr¨uger, J., Lefohn, A. E., and Purcell, T. J. A survey of
general-purpose computation on graphics hardware. In
Computer graphics forum , volume 26, pp. 80–113. Wiley
Online Library, 2007.


key: chroma:document
string_value:
Online Library, 2007.
Phoulady, A., Granmo, O.-C., Gorji, S. R., and Phoulady,
H. A. The Weighted Tsetlin Machine: Compressed Repre-
sentations with Clause Weighting. In Proceedings of the
Ninth International Workshop on Statistical Relational
AI (StarAI), New York, USA , 2020.
Saha, R., Granmo, O.-C., and Goodwin, M. Mining In-
terpretable Rules for Sentiment and Semantic Relation
Analysis using Tsetlin Machines. In Proceedings of the
40th International Conference on Innovative Techniques


key: chroma:document
string_value:
and Applications of Artiﬁcial Intelligence (SGAI), Cam-
bridge, UK . Springer International Publishing, 2020.
Satish, N., Harris, M., and Garland, M. Designing efﬁ-
cient sorting algorithms for manycore GPUs. In 2009
IEEE International Symposium on Parallel & Distributed
Processing, Rome, Italy , pp. 1–10. IEEE, 2009.
Shaﬁk, R., Wheeldon, A., and Yakovlev, A. Explainability
and Dependability Analysis of Learning Automata based
AI Hardware. In IEEE 26th International Symposium


key: chroma:document
string_value:
AI Hardware. In IEEE 26th International Symposium
on On-Line Testing and Robust System Design (IOLTS),
Naples, Italy . IEEE, 2020.
Tsetlin, M. L. On behaviour of ﬁnite automata in random
medium. Avtomat. i Telemekh , 22(10):1345–1354, 1961.
Valiant, L. G. A Theory of the Learnable. Communications
of the ACM , 27(11):1134–1142, 1984.
Wen, Z., Shi, J., Li, Q., He, B., and Chen, J. ThunderSVM:
A fast SVM library on GPUs and CPUs. The Journal of
Machine Learning Research , 19(1):797–801, 2018.


key: chroma:document
string_value:
Machine Learning Research , 19(1):797–801, 2018.
Wheeldon, A., Shaﬁk, R., Rahman, T., Lei, J., Yakovlev, A.,
and Granmo, O.-C. Learning Automata based Energy-
efﬁcient AI Hardware Design for IoT. Philosophical
Transactions of the Royal Society A , 2020.Xuelan, F. and Kennedy, G. Expressing causation in written
english. RELC Journal , 23(1):62–80, 1992.
Yadav, R. K., Jiao, L., Granmo, O.-C., and Goodwin, M.
Human-Level Interpretable Learning for Aspect-Based


key: chroma:document
string_value:
Sentiment Analysis. In Proceedings of the AAAI Confer-
ence on Artiﬁcial Intelligence, Vancouver, Canada . AAAI,
2021a.
Yadav, R. K., Jiao, L., Granmo, O.-C., and Goodwin, M.
Interpretability in Word Sense Disambiguation using
Tsetlin Machine. In 13th International Conference on
Agents and Artiﬁcial Intelligence (ICAART), Vienna, Aus-
tria. INSTICC, 2021b.
Zhang, X., Jiao, L., Granmo, O.-C., and Goodwin, M. On
the Convergence of Tsetlin Machines for the IDENTITY-


key: chroma:document
string_value:
Drop Clause: Enhancing Performance, Robustness and Pattern Recognition
Capabilities of the Tsetlin Machine
Jivitesh Sharma, Rohan Yadav, Ole-Christoffer Granmo, Lei Jiao
Center for Artificial Intelligence Research (CAIR)
University of Agder, Norway
{jivitesh.sharma, rohan.k.yadav, ole.granmo, lei.jiao}@uia.no
Abstract
Logic-based machine learning has the crucial advantage of
transparency. However, despite significant recent progress,
further research is needed to close the accuracy gap between


key: chroma:document
string_value:
logic-based architectures and deep neural network ones. This
paper introduces a novel variant of the Tsetlin machine (TM)
that randomly drops clauses, the logical learning element of
TMs. In effect, TM with Drop Clause ignores a random se-
lection of the clauses in each epoch, selected according to a
predefined probability. In this way, the TM learning phase be-
comes more diverse. To explore the effects that Drop Clause
has on accuracy, training time and robustness, we conduct


key: chroma:document
string_value:
extensive experiments on nine benchmark datasets in natural
language processing (IMDb, R8, R52, MR, and TREC) and
image classification (MNIST, Fashion MNIST, CIFAR-10,
and CIFAR-100). Our proposed model outperforms baseline
machine learning algorithms by a wide margin and achieves
competitive performance compared with recent deep learn-
ing models, such as BERT-Large and AlexNet-DFA. In brief,
we observe up to 10% increase in accuracy and 2×to4×


key: chroma:document
string_value:
faster in learning than those of the standard TM. We visu-
alize the patterns learnt by Drop Clause TM in the form of
heatmaps and show evidence of the ability of drop clause
to learn more unique and discriminative patterns. We finally
evaluate how Drop Clause affects learning robustness by in-
troducing corruptions and alterations in the image/language
test data, which exposes increased learning robustness.
Introduction
Researchers across various fields are increasingly paying at-


key: chroma:document
string_value:
tention to the interpretability of AI techniques. While inter-
pretability previously was inherent in most machine learn-
ing approaches, the state-of-the-art methods now increas-
ingly rely on black-box deep neural network-based models.
Natively, these can neither be interpreted during the learn-
ing stage nor while producing outputs. For this reason, a
surge of techniques attempts to open the black box by visual
explanations and gradient-based interpretability (Simonyan,


key: chroma:document
string_value:
Vedaldi, and Zisserman 2014; Zhang, Wu, and Zhu 2018;
Radhakrishnan et al. 2018; Bau et al. 2017; Selvaraju et al.
2017; Ribeiro, Singh, and Guestrin 2016).
The Tsetlin Machine (TM) is an interpretable rule-
based machine learning algorithm that produces logical
Copyright © 2023, Association for the Advancement of Artificial
Intelligence (www.aaai.org). All rights reserved.rules (Granmo 2018). In brief, it employs an ensemble of
Tsetlin Automata (TA) that learns propositional logic ex-


key: chroma:document
string_value:
pressions from Boolean input features, and TM achieves in-
terpretability by leveraging sparse disjunctive normal form.
Indeed, humans are particularly good at understanding flat
and short logical AND-rules, reflecting human reason-
ing (Noveck et al. 1991). In addition, due to its Boolean
representations and finite-state learning mechanisms, it pos-
sesses a minimalistic memory footprint.
Similar to neural networks, the vanilla TM suffers overfit-


key: chroma:document
string_value:
ting in the sense that its learning elements, i.e., clauses,
are prone to capture noisy and redundant patterns. Initially,
we believed that this could be simply because of the large
number of clauses. However, by decreasing the number of
clauses, there was a proportionate decrease in performance.
In order to mitigate overfitting, inspired by Dropout (Srivas-
tava et al. 2014), we propose a method named Drop Clause
(DC) in this paper1. More specifically, DC method randomly


key: chroma:document
string_value:
drops or switches off a set of clauses during training, which
is similar to what dropout does in neural networks. Unlike
dropout, DC induces stochasticity in TM learning, boosting
performance in terms of accuracy and learning speed. It fur-
ther improves the patterns captured by the clauses, making
them more robust towards input perturbations. As a direct
consequence of DC during training, the training time is re-
duced proportionally and the heatmaps produced becomes
more distinct.


key: chroma:document
string_value:
more distinct.
Paper Contributions:
• We propose to drop clauses randomly during each train-
ing iteration, which introduces additional stochasticity.
• We demonstrate that DC makes the TM capture more
unique patterns, improving its generalization perfor-
mance.
• DC TM leads to competitive results to large deep neural
network models on nine benchmark datasets for NLP and
image classification. Since TM is more akin to a stan-
dard machine learning algorithm, like a decision tree,


key: chroma:document
string_value:
than a deep learning model, we compare DC TM with
well-known machine learning algorithms, documenting
superior performance across all the datasets.
1The code is available online on: https://github.com/
Anonymous-2491/Drop-Clause-Interpretable-TM.
The Thirty-Seventh AAAI Conference on Artificial Intelligence (AAAI-23)
13547


key: chroma:document
string_value:
• Via heatmaps, we visualize that DC TM becomes more
clear-cut and robust on image classification and language
sentiment analysis.
Drop Clause for Tsetlin Machines
Modern society needs reliable, unbiased, and trustworthy
AI systems. Researchers have shown that neural networks
are not fully mature for integration into society. For exam-
ple, neural network interpretability is fragile towards adver-
sarial examples (Ghorbani, Abid, and Zou 2019). Gener-


key: chroma:document
string_value:
ating adversarial perturbations that produce visually indis-
tinguishable images for humans leads to dramatically dif-
ferent neural network interpretations, without changing the
label. While methods such as GradCAM (Selvaraju et al.
2017) and LIME (Ribeiro, Singh, and Guestrin 2016) can
take a peek inside black-box models, the explanations of-
fered can be arbitrarily inaccurate, as argued by Rudin et al.
They pinpoint several reasons for choosing inherently inter-


key: chroma:document
string_value:
pretable models over attempting to explain black-box ones
(Rudin 2019). They further raise three algorithmic chal-
lenges that the machine learning community faces to suc-
ceed with human-level interpretability. Although neural net-
works are inherently inadequate for these challenges, the
TM addresses them natively:
•Challenge 1: Rule-based logical models. The TM learns
a rule-based model using logical operations. Learning
is game-theoretic with Nash equilibria that correspond


key: chroma:document
string_value:
to the optimal propositional logic expressions (Granmo
2018; Zhang et al. 2022; Jiao et al. 2022).
•Challenge 2: Linear models with sparse scoring systems.
The TM is a linear model that produces sparse clauses
and integer weighted scores (Abeyrathna, Granmo, and
Goodwin 2021).
•Challenge 3: Domain-specific interpretable AI. The TM
is inherently interpretable, used for producing inter-
pretable models across several domains (Abeyrathna
et al. 2020; Lei et al. 2021; Yadav et al. 2021, 2022).


key: chroma:document
string_value:
In what follows, we first briefly present the basic TM and
its convolutional version. Thereafter, we introduce the DC
technique, which enhances the stochasticity of TM learning.
TM and Convolutional TM
A TM in its simplest form takes a feature vector x=
[x1, x2, . . . , x o]∈ {0,1}oofoBoolean values as in-
put, producing a Boolean output ˆy∈ {0, 1}. Pat-
terns are expressed as conjunctive clauses Cj(AND-
rules), built from literals L={l1, l2, . . . , l 2o}=


key: chroma:document
string_value:
{x1, x2, . . . , x o,¬x1,¬x2, . . . ,¬xo}:
Cj=2o^
k=1h
g(aj
k)⇒lki
. (1)
In Eq. (1), aj
kis the TA state that controls inclusion of literal
lkin clause jandg(·)maps the TA state to action 0or1. The
imply operator, in turn, implements the action in the clause.
That is, if g(aj
k)is1, then the imply operator requires that
the literal is 1for the whole expression to be 1. Ifg(aj
k)is0,on the other hand, the expression becomes 1regardless the
true value of lk.
ˆy=(
1,Pn−1
j=1,3,...Cj−Pn


key: chroma:document
string_value:
true value of lk.
ˆy=(
1,Pn−1
j=1,3,...Cj−Pn
j=2,4,...Cj≥0,
0,Otherwise.(2)
Clauses are divided into even-indexed/odd-indexed clauses
that vote for output ˆy={0,1}. Odd-indexed clauses vote
in favour of the class whereas even-indexed clauses vote
against the class, as shown in Eq. (2). A positive vote count
means majority of the clauses predict the input to belong to
the class and a negative vote count implies majority of the
clauses believe the input does not belong to the class. A de-


key: chroma:document
string_value:
tailed explanation can be found in (Granmo 2018).
The Convolutional TM (CTM) is as an interpretable alter-
native to CNNs. Whereas the TM categorizes an image by
employing each clause once to the whole image, the CTM
uses each clause as a convolution filter. That is, a clause is
evaluated multiple times, once per image patch taking part
in the convolution. The output of a convolution clause is ob-
tained simply by ORing the outcome of evaluating the clause
on each patch:
ˆy=(
1,Pn−1
j=1,3,...WB


key: chroma:document
string_value:
on each patch:
ˆy=(
1,Pn−1
j=1,3,...WB
b=1Cj−Pn
j=2,4,...WB
b=1Cj≥0,
0,Otherwise.
Here, brefers to one out of Bavailable image patches. A
detailed explanation can be found in (Granmo et al. 2019).
Please see the supplementary material for further details on
TM and CTM.
Drop Clause
Here we propose a novel regularization method for the TM,
i.e., DC. This technique is inspired by the dropout method
for neural networks. In DC, clauses are removed with a prob-
ability pin each training epoch:
ˆy=(


key: chroma:document
string_value:
ability pin each training epoch:
ˆy=(
1,Pn−1
j=1,3,...πjCj−Pn
j=2,4,...πjCj≥0,
0,Otherwise.(3)
Above, πj∈ {0, 1}for clause jis zero with probability p
for each complete epoch. Clearly, the vanilla TM and its DC
variant are equivalent if p= 0.
DC TM is similar to dropout in neural networks, and the pur-
pose is to reduce the chance of learning redundant patterns.
A vanilla TM seeks to minimize prediction error on the train-
ing data. Achieving this, there may still be unused patterns


key: chroma:document
string_value:
available in the training data. These patterns can potentially
be useful when facing new data, such as test data. By ran-
domly dropping clauses for complete epochs, we mobilize
other clauses to take over the role of the dropped clauses.
DC can be thought of as having a pool of clauses and se-
lecting a set of clauses with 1−pprobability for training
every epoch. However, due to this stochastic learning, the
mobilized clauses may solve the task in a different way each


key: chroma:document
string_value:
time. Hence, robustness increases overall when the complete
set of clauses are turned on again. The resulting effects are
evaluated experimentally in the next section.
The stochasticity induced by DC in the TM learning process
13548


key: chroma:document
string_value:
is similar to the stochasticity induced by stochastic gradi-
ent descent (SGD) (Robbins 2007). In SGD, a mini-batch of
training samples is selected randomly from the training data.
While, in DC, various sets of clauses are selected for train-
ing with probability 1−p. The major difference is that, SGD
is stochastic in terms of data sampling during the learning
stage, whereas DC is stochastic in terms of sampling the el-
ements of the model structure (clauses are the elements that


key: chroma:document
string_value:
form the TM model). Another difference is that the stochas-
ticity of DC can be adjusted with the DC probability hyper-
parameter p. Note that the Drop Clause TM still retains its
convergence properties. Please see supplementary material
for details.
Enhanced Performance
We here investigate the effects that DC has on the perfor-
mance of TMs. To assess the generality of our approach,
we test DC both on NLP and image classification. For NLP,


key: chroma:document
string_value:
we use TM with weighted clauses, and for image classifi-
cation, we use the CTM. For evaluation, we compare our
approach with the comparable state-of-the-art deep learning
models. Since TM is a rule-based machine learning method,
we also cover more standard machine learning algorithms.
The DC implementation is written in PyCUDA for paral-
lel GPU computation. We train our models on 16 NVIDIA
Tesla V100 TensorCore GPUs for the image classification


key: chroma:document
string_value:
task and on one NVIDIA RTX 3070 8GB GPU for the NLP
tasks2.
Natural Language Processing
We explore the performance of DC on NLP tasks first.
The first experiments assess how varying the DC proba-
bility affects classification accuracy and then compare the
best found configuration with similar machine learning al-
gorithms and state-of-the-art techniques. We adopt 5popular
standard datasets, summarized below:
•IMDb consists of 50,000 movie reviews for sentiment


key: chroma:document
string_value:
analysis, split in half for training and testing. Here, we
use10,000clauses and set TM threshold Tto8,000and
specificity sto2.0.
•Reuters-21578 contains the text categorization datasets
R8andR52. R8 is divided into 8categories, with 5,485
training and 2,189testing samples, whereas R52 consists
of52categories, divided into 6,532training and 2,568
testing samples. We here employ 3,000clauses with T=
2,000ands= 7.0.
•MRis another movie review dataset for binary sentiment


key: chroma:document
string_value:
classification, consisting of 10,662samples. We use the
train-test split as in (Tang, Qu, and Mei 2015) and use
5,000clauses with T= 4,000ands= 6.0.
•TREC is a question classification dataset that encom-
passes 6categories. There are a total of 6,000samples,
5,500for training and 500for testing. Here, we employ
5,000clauses with T= 4,000ands= 2.0.
2Note that these are not hardware requirements. We use these
GPUs just to train faster.TM p=
0p=.1p=.25p=.5p=.75
IM
DB 87.2 88.3 89.6 90.4 91.27


key: chroma:document
string_value:
IM
DB 87.2 88.3 89.6 90.4 91.27
R8 96.16 97.6 98.1 98.5 98.94
R52 89.14 89.5 90.8 91.5 92.75
MR 75.14 77.25 77.9 78.2 78.67
TREC 88.05 89.8 90.1 90.5 89.9
Table 1: Effect of DC on NLP Datasets.
CT
M p= 0 p=.1p=.25p=.5p=.75
MN
IST 99.3 99.3 99.45 99.35 98.2
F-MNIST 91.5 91.75 92.5 92.25 91.25
CIFAR-10 69.3 70.5 73.2 75.1 72.6
CIFAR-100 35.5 39.5 42.6 45.2 40.8
Table 2: Effect of DC on Image Classification.
Our first step is to compare the difference in perfor-


key: chroma:document
string_value:
mance with respect to changing DC probability p, as
shown in Table 1. The selected DC probabilities are p∈
{0.1,0.25,0.5,0.75}. For IMDb, R8, R52 and MR, the best
performance is achieved with p= 0.75, which is equivalent
of dropping 75% of the total clauses per class, per epoch.
As for TREC, p= 0.5works best. As seen, DC has sig-
nificant effect on accuracy, with average accuracy going up
by4.07% for IMDb, 2.78% for R8, 3.61% for R52, 3.53%
for MR and 2.45% for TREC. Additionally, we observe a


key: chroma:document
string_value:
substantial reduction in training time proportional to the DC
ratio. Inference times on these datasets are less than 2.3sper
1000 samples on average on an NVIDIA GTX 1080 GPU.
The TM is a rule-based machine learning algorithm, so we
also compare our proposed TM model with a few traditional
machine learning techniques. Specifically, we compare our
proposed TM with DC against Support Vector Machine
(SVM), Random Forests (RF), K-Nearest Neighbours (K-


key: chroma:document
string_value:
NN) and XGBoost (XGB). Table 2 displays the comparison
results3. From Table 2, it is clear that DC TM outperforms
other machine learning algorithms by a wide margin.
We also compare our model with deep learning methods,
some of which representing the state of the art. Tables 4
and 5 show the results of the comparisons. Enhanced with
DC, the accuracy of TM is comparable not only to CNNs
and LSTMs but also to computationally complex and para-


key: chroma:document
string_value:
metrically large state-of-the-art models like BERT-Large.
On all datasets in Table 4, inclusion of DC propels the
performance of the TM to outperform CNN (Kim 2014)
and bidirectional LSTM with pretrained word embeddings.
In fact, on R8 and R52, DC TM is able to achieve better
accuracy than BERT (Devlin et al. 2019) and comes close
to its performance on MR. Also, comparing our model with
the state-of-the-art graph convolutional neural network,
S2GC (Zhu and Koniusz 2020), DC TM achieves better


key: chroma:document
string_value:
performance on R8 and MR, and comparable performance
3Note that the machine learning methods used here for compar-
ison have default parameters from sklearn.
13549


key: chroma:document
string_value:
TM
TM SVM RF K-NN XGB
(DC)
IM
DB 87.2 91.27 83.2 78.5 72.5 84.4
R8 96.16 98.94 84.7 82.4 77.3 85.1
R52 89.14 92.75 71.0 72.5 58.2 75.4
MR 75.14 78.67 52.2 51.7 39.1 56.3
TREC 88.05 90.5 67.5 65.4 56.1 77.6
Table 3: Comparison with ML methods.
TM
CNN B ILSTM BERT S2GC
(DC) LARGE
IM
DB 91.27 87.9 88.9 95.4 -
R8 98.94 95.71 96.31 96.02 97.4
R52 92.75 87.59 90.54 89.66 94.5
MR 78.67 77.75 77.68 79.24 76.7
Table 4: Comparison on IMDb, R8, R52 and MR.


key: chroma:document
string_value:
Table 4: Comparison on IMDb, R8, R52 and MR.
on R52. Similar results are obtained on the TREC dataset
shown in Table 5. DC TM outperforms a baseline LSTM
model as well as a vanilla Transformer (TF) and Trans-
former with feature projection (FP) (Qin, Hu, and Liu
2020).
Even though BERT-Large achieves better performance on
TREC dataset, TM has major advantages over these models
in terms of computational complexity and transparency,
which are further enhanced by DC (See Section “Enhanced


key: chroma:document
string_value:
Robustness”). While it is disputed whether attention is
explainable (Jain and Wallace 2019), attention is much
more complex (quadratic complexity with floating-point op-
erations) than our proposed model (linear complexity with
Boolean operations) that achieves comparable performance.
Note that our model only relies on simple bag-of-words
tokens in the target datasets, without considering any
additional pretrained world knowledge (Yadav et al. 2021),


key: chroma:document
string_value:
like word2vec, Glove or BERT features. Yet, it achieves
competitive performance compared to deep learning and
some state-of-the-art models.
Image Classification
We now turn to image classification, again focusing on how
DC affects performance of TM. To this end, we evaluate DC
on four benchmark image classification datasets: MNIST,
Fashion-MNIST, CIFAR-10 and CIFAR-100. We binarize
the datasets using an adaptive Gaussian thresholding pro-


key: chroma:document
string_value:
cedure as proposed in (Granmo et al. 2019). This binariza-
tion results in images with only 1bit per pixel per channel,
considerably reducing the memory overhead. For the task of
image classification, we employ the CTM.
•MNIST encompasses 70,000 28 ×28gray-scale im-
ages of hand written single digits, 60,000 for training
and10,000for testing. Here, we use 8,000clauses with
T= 6,400ands= 5.0.TM
(DC) LSTM TF FP+TF BERT-L ARGE
TR
EC-6 90.5 87.19 87.33 89.5 95.6
Table 5: Comparison on TREC-6.


key: chroma:document
string_value:
Table 5: Comparison on TREC-6.
•Fashion-MNIST contains 28×28gray-scale images
from the Zalando catalogue. For this dataset, we use the
same parameters as the MNIST dataset.
•CIFAR-10 and CIFAR-100 consist of 50,000 32 ×32
color images of objects divided into 10categories for
CIFAR-10 and 100for CIFAR-100. Another 10,000im-
ages are provided for testing. Here, we make use of
60,000clauses with T= 48,000ands= 10.0.
Again, we explore the effects of four DC probability set-


key: chroma:document
string_value:
tings, p∈ {0. 1,0.25,0.5,0.75}, shown in Table 2. The best
performance is achieved with p= 0.25for the MNIST
and Fashion-MNIST, while p= 0.5gives the best result
for the CIFAR datasets, which is equivalent to dropping a
quarter (or half) of the clauses per training iteration. No-
tice the considerable performance increase, especially on the
CIFAR datasets, with DC of p= 0.5. Peak accuracy on
MNIST is 99.45% (±0.25%) , and on Fashion-MNIST it is


key: chroma:document
string_value:
92.5% (±0.25%) . CIFAR-10 and CIFAR-100 peak at 75.1%
(±0.4%) and45.2% (±0.2%) respectively, averaged over
100runs. Apart from the accuracy gain, DC with p= 0.5
reduces training time by approximately 50%, for the CI-
FAR datasets. For instance, training time per epoch drops
from 61.83sto32.58sfor CIFAR-10. The inference time
for the DC CTM on CIFAR-10 is 1.5sper1000 images on
one NVIDIA Tesla V100 GPU, leveraging the computation
benefits of binary operations for TM inference.


key: chroma:document
string_value:
benefits of binary operations for TM inference.
Table 6 shows the comparison of the DC CTM with other
popular machine learning techniques on the image classifi-
cation. As seen, CTM outperforms the traditional machine
learning techniques by a wide margin, further widened by
the introduction of DC4.
Table 7 contains a comparison between the DC CTM with
certain related and state-of-the-art techniques. Binary neural
networks can be represented exactly as a propositional logic


key: chroma:document
string_value:
expression (Narodytska et al. 2018). As such, they are partic-
ularly comparable to CTM and we therefore also include re-
sults for EEV-BNN (Jia and Rinard 2020). EEV-BNN is a bi-
nary neural network that has been verified with Boolean sat-
isfiability. Employing EEV-BNN, we use the MNIST-MLP
and Conv-Large BNN architectures from (Jia and Rinard
2020) for comparisons. The DC CTM significantly outper-
forms EEV-BNN on MNIST and CIFAR-10, as shown in


key: chroma:document
string_value:
forms EEV-BNN on MNIST and CIFAR-10, as shown in
Table 7. We also compare our method with a transformer
based model of reduced complexity to make it more compa-
rable to our model. The Vision Nystromformer (ViN) (Jee-
van and Sethi 2021) is a vision transformer (ViT) based
model that reduces the quadratic computational complex-
ity of ViT by using the Nystrom methods for approximat-
4Note that the accuracy on CIFAR-100 is not displayed in Ta-


key: chroma:document
string_value:
CT
M CTM SVM RF K-NN XGB
(DC)
MN
IST 99.3 99.45 93.4 93.8 95.3 96.2
F-MNIST 91.5 92.5 84.6 87.5 85.4 88.4
CIFAR-10 69.3 75.1 37.5 48.7 33.9 47.8
CIFAR-100 35.5 45.2 - - - -
Table 6: Image Classification with ML methods.
CT
M EEV H YBRID ALEXNET MAXOUT
(DC) -BNN -V IN -DFA
MN
IST 99.45 98.25 96.4 98.2 99.06
F-MNIST 92.5 - - 91.66 -
CIFAR-10 75.1 63.45 75.26 64.7 86.8
CIFAR-100 45.2 - - 52.62 59.52
Table 7: Image Classification with SOTA models.


key: chroma:document
string_value:
Table 7: Image Classification with SOTA models.
ing self-attention. It achieves 65.06% accuracy on CIFAR-
10 whereas the Hybrid-ViN, which incorporates rotary posi-
tional embedding, obtains slightly better accuracy as shown
in Table 7, comparable to our model.
We also contrast our model against AlexNet with Direct
Feedback Alignment (DFA) (Webster, Choi, and changwook
Ahn 2021) for parallel backpropagation training, which
makes it comparable to the fast training of TM. The results


key: chroma:document
string_value:
in Table 7 show that AlexNet-DFA achieves similar perfor-
mance on MNIST and Fashion-MNIST, whereas it is out-
performed by DC CTM on CIFAR-10. However, there is a
considerable difference on the CIFAR-100 dataset due to the
larger label space. Finally, as DC is inspired from dropout,
we compare our model with the Maxout network (Goodfel-
low et al. 2013), which was a natural companion to dropout.
Maxout achieves significantly better performance on the CI-


key: chroma:document
string_value:
FAR datasets as shown in Table 7. Note that on color images,
binarization can lead to loss of important information espe-
cially when we use 1-bit per channel, compared with 8-bits
per channel for lossless color information. Also, as previ-
ously stated, TM possesses computational advantages over
these methods along with transparency.
Another point to note is that, originally, dropout can increase
the performance of basic CNNs by about 3%and6%on


key: chroma:document
string_value:
the performance of basic CNNs by about 3%and6%on
CIFAR-10 and CIFAR-100, respectively (Srivastava et al.
2014). The DC method shows promising results by improv-
ing the performance of vanilla CTM by about 6%on CIFAR-
10 and 10% on CIFAR-100.
Enhanced Pattern Recognition
In this section, we observe more distinct heatmaps produced
by TM and CTM for NLP and image classification tasks as a
result of DC, thereby visually exemplifying its superior pat-


key: chroma:document
string_value:
tern recognition capabilities. We show enhancements in the
quality of heatmaps on the MR dataset at the word-level and
on the CIFAR-10 dataset at the pixel-level. The enhance-
ments encompass richer and more distinct word representa-
tions for NLP tasks and more clear-cut pixel representations
for objects in images. More specifically, these enhancementsindicate that DC is able to capture more unique, relevant and
discriminative patterns. These enhancements further mani-


key: chroma:document
string_value:
fest the stronger pattern recognition capabilities of DC5.
Natural Language Sentiment Analysis
We investigate local word-level patterns captured by TM and
DC TM in NLP using a randomly selected test example from
MR: “A waste of fearless purity in the acting craft”. The ex-
ample was selected among the ones that correctly classified
with DC but incorrectly classified without it. The purpose is
to contrast and exemplify how DC captures better patterns


key: chroma:document
string_value:
and enhances the heatmaps. Note that for NLP, the major-
ity of the features appear in negated form. In order to have
word-level understanding of the model, we use a frequency-
based interpretation, i.e., we highlight the features based on
how frequently they appear in clauses (Yadav et al. 2021).
An arguably unique property of TMs is that the patterns they
produce are both descriptive (frequent) and discriminative.
In other words, each clause captures a full description of the


key: chroma:document
string_value:
target concept, not merely the discrimination boundary.
Figure 1a highlights the top 100most frequent features in
negated form (darker color means high frequency), present-
ing in the clauses triggered by the given sample using vanilla
TM without DC. Three example clauses C0,C2, and C4
are shown below for the color-coded features. In this case,
the prediction is wrong. More importantly, the literals in the
conjunctions do not make sense.
The corresponding result for DC TM is depicted in Fig-


key: chroma:document
string_value:
ure 1b. Here, the 100most frequent features in the negated
form seem intuitive for predicting negative sentiment. Fea-
tures such as “NOT witti (witty)”, “NOT grace (graceful)”,
“NOT terrif (terrific)”, “NOT honest”, “NOT cool”, or “NOT
intellig (intelligent)” generally mean absence of positive
sentiment, which in this case makes the model draw the cor-
rect conclusion (negative sentiment). Some of the specific
patterns that are responsible for predicting the correct out-


key: chroma:document
string_value:
put are C0, C2,andC4, as shown in the figure. Although
randomly chosen, this example is representative for how TM
with DC is able to capture a larger variety of correct patterns
than what the vanilla TM is capable of.
Image Classification
In image classification, TM clauses form self-contained pat-
terns by joining pixels into multi-pixel structures. That is,
the image pixels are sent directly to the clauses, following
propositional AND-rules. Therefore, Boolean propositional


key: chroma:document
string_value:
expressions (clauses) capture patterns in an image that con-
tains pixels as literals in the clauses, which are compara-
tively easy for humans to comprehend (Valiant 1984).
The CTM forms clauses from the pixels of the square image
patches obtained in the convolution. Accordingly, we extract
the top kweighted clauses per class in patch form. For vi-
sualization, we represent the non-negated pixels of a clause
as1, negated pixels as −1, and excluded pixels as 0. In ad-


key: chroma:document
string_value:
ditional to the image content, each clause also encodes po-
sitions in the image where it is valid. If a certain position is
5More examples of enhanced pattern recognition and inter-
pretability can be found in the supplementary material.
13551


key: chroma:document
string_value:
A waste of fearless purity in
the acting craft
positive sentiment(a) Without drop clause.
A waste of fearless purity in
the acting craft
negative sentiment
(b) With drop clause (p= 0.75).
Figure 1: Word-level patterns of a sample captured: (a) without and (b) with DC.
invalid, the clause literals corresponding to the pixel values
are treated as 0to indicate no activation at that location. The
resulting clause masks are applied to the image and their ac-


key: chroma:document
string_value:
tivation maps are added up to produce a heatmap.
In Figure 2, we show the advantage of DC TM via an exam-
ple. As can be seen from the figure, DC is able to capture
the object with more precision by capturing more discrimi-
native patterns and including pixels relevant to classification.
We believe this is because the remaining clauses are forced
to substitute the dropped clauses, learning to perform their
tasks more independently. Due to the stochastic nature of


key: chroma:document
string_value:
learning, they will, however, learn to perform the tasks dif-
ferently than the dropped clauses. As a result, DC reduces
redundancy and induces diversity in the learning of the pat-
terns. An example of how these heatmaps can be interpreted
is shown in the supplementary material.
(a) Image
 (b) TM (p= 0)
 (c) TM (p= 0.5)
Figure 2: CIFAR-10 patterns captured: (b) without and (c)
with DC.
Enhanced Robustness
In this section, we show empirically that introduction of DC


key: chroma:document
string_value:
makes TM more robust towards input perturbations during
testing. We evaluate robustness on both sentiment analysis
13552


key: chroma:document
string_value:
TM TM (DC) BiLSTM
MR 75.14 78.67 77.68
MR (perturb) 73.12 77.12 75.85
Table 8: Robustness Test on MR.
CTM CTM (DC) BNN
MNIST 99.3 99.45 98.25
MNIST-C 88.12 91.75 85.5
Table 9: Robustness Test on MNIST-C.
and image classification6.
Natural Language Sentiment Analysis
We select the MR dataset to test the robustness of TM on
sentiment analysis. We train our models on the standard
training data and perturb the testing data to evaluate ro-
bustness towards out of distribution changes. We monitor


key: chroma:document
string_value:
the decrease in testing accuracy with and without perturba-
tions. During testing, an input instance (sentence) is selected
for perturbation with probability 0.5. Then, a word is cho-
sen from this sentence randomly with uniform probability.
The selected word is swapped with the most similar word
according to the cosine similarity between their respective
Glove embedding vectors. Swapping with the most similar
word results in changing the binary feature representation


key: chroma:document
string_value:
by a hamming distance of 2, while keeping the meaning
and sentiment of the sentence unchanged. The following is
an MR example demonstrating this strategy: “Magnificent
drama well worth watching” →“Splendid drama well worth
watching”. The word Magnificent is changed to its synonym
Splendid. Table 8 shows the comparison between TM and
DC TM on MR with and without test data perturbations. The
difference between the test accuracy on the dataset with and


key: chroma:document
string_value:
without perturbations is 2.02% and1.45% for the standard
TM and DC TM respectively. In addition, DC makes TM
more robust compared with BiLSTM, where the accuracy
drops by 1.83%. The increase in TM robustness could be
due to the fact that since clauses are stochastically dropped,
they seem to be less prone to pick up irrelevant or noisy pat-
terns.
Image Classification
We compare the robustness of CTM and DC CTM using the
MNIST-C dataset (Mu and Gilmer 2019), which is a cor-


key: chroma:document
string_value:
rupted version of MNIST. It consists of 15corruptions such
as Gaussian blur, scale, rotate, shear, impulse noise, canny
edges, fog etc. We train our models on the standard MNIST
training data without any augmentations. During testing, we
select whether to use non-corrupted or corrupted test image
with probability 0.5. We then select one of the 15corrup-
tions to apply on the test image with uniform probability.
DC enhances the robustness of CTM, as can be seen from


key: chroma:document
string_value:
6Note that we do not test for adversarial robustness. We only
test the robustness of our models with corrupted input.Table 9. There is a drop of 11.18% for the standard CTM
whereas the drop in accuracy is reduced to 7.7%with DC.
Also, from Table 9, we observe that both the TM versions are
more robust than binarized neural networks (BNN)7, whose
accuracy drops by 12.75%.
Discussion
To summarize our results, DC improves the performance
of the TM, enhancing the advantages that TMs have over


key: chroma:document
string_value:
neural networks and traditional machine learning techniques
when it comes to computational complexity, memory and
energy consumption, training and inference time and robust-
ness (Abeyrathna et al. 2021; Lei et al. 2020). The compu-
tational advantages make TMs suitable for federated learn-
ing and deployment on edge devices. On the other hand,
the TM still achieves lower accuracy on the datasets we
use here when compared with certain deep neural network


key: chroma:document
string_value:
models. However, TM is a machine learning method that
achieves overall good performance while being inherently
interpretable unlike deep neural networks that are black-
box models. For certain NLP tasks, pretrained models, such
as word2vec, GloVe, BERT, and GPT, have been dominat-
ing, leveraging unlabelled data. TMs require Boolean in-
put, making it difficult to leverage existing high dimensional
embeddings, impeding performance, with the exception of


key: chroma:document
string_value:
enhancing the input with GloVe-derived synonyms (Yadav
et al. 2021). However, there is huge difference in model
sizes, where SOTA deep neural networks have parameters
upwards of 10million floating-point parameters, easily go-
ing beyond 100 million (BERT-Large has 340 million pa-
rameters). On the other hand, TM employs a few thousand
Boolean clauses. Also, the pixel-level interpretability on im-
age classification can be difficult for humans to interpret


key: chroma:document
string_value:
as the Boolean expressions for clauses capturing patterns
can be quite long, containing numerous pixels, especially
for large images. Nevertheless, there are several O(n)tech-
niques that can easily simplify and reduce Boolean expres-
sions (Meurer et al. 2017).
Conclusions
In this paper, we propose drop clause as a technique to im-
prove the generalization ability of the TM. Drop clause en-
hances stochasticity during training, to capture more diverse


key: chroma:document
string_value:
patterns. As a result, the performance of TM is significantly
boosted, which is empirically shown on a varied collection
of datasets, both in terms of accuracy and training time. We
further show how drop clause improves the pattern recogni-
tion capabilities of TM, simultaneously improving the clar-
ity of the produced patterns. When comparing our model
with the state-of-the-art deep learning models, we observe
competitive accuracy levels. Since TM is more akin to tradi-


key: chroma:document
string_value:
tional machine learning techniques, we also compare it with
a selection of those, reporting superior performance of the
TM. We finally establish that drop clause improves the ro-
bustness of the TM towards data corruptions and perturba-
tions during testing.
7The MNIST-MLP architecture from (Jia and Rinard 2020) is
used here.
13553


key: chroma:document
string_value:
References
Abeyrathna, K. D.; Bhattarai, B.; Goodwin, M.; Gorji, S. R.;
Granmo, O.-C.; Jiao, L.; Saha, R.; and Yadav, R. K. 2021.
Massively Parallel and Asynchronous Tsetlin Machine Ar-
chitecture Supporting Almost Constant-Time Scaling. In
Meila, M.; and Zhang, T., eds., Proceedings of the 38th
International Conference on Machine Learning, volume
139 of Proceedings of Machine Learning Research, 10–20.
PMLR.
Abeyrathna, K. D.; Granmo, O.-C.; and Goodwin, M.


key: chroma:document
string_value:
Abeyrathna, K. D.; Granmo, O.-C.; and Goodwin, M.
2021. Extending the Tsetlin Machine With Integer-Weighted
Clauses for Increased Interpretability. IEEE Access, 9.
Abeyrathna, K. D.; Pussewalage, H. S. G.; Ranasinghea,
S. N.; Oleshchuk, V . A.; and Granmo, O.-C. 2020. Intru-
sion Detection with Interpretable Rules Generated Using the
Tsetlin Machine. In 2020 IEEE Symposium Series on Com-
putational Intelligence (SSCI). IEEE.
Bau, D.; Zhou, B.; Khosla, A.; Oliva, A.; and Torralba, A.


key: chroma:document
string_value:
2017. Network Dissection: Quantifying Interpretability of
Deep Visual Representations. CoRR, abs/1704.05796.
Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019.
BERT: Pre-training of Deep Bidirectional Transformers for
Language Understanding. In Proceedings of the 2019 Con-
ference of the North American Chapter of the Association
for Computational Linguistics: Human Language Technolo-
gies, Volume 1 (Long and Short Papers), 4171–4186. Min-


key: chroma:document
string_value:
neapolis, Minnesota: Association for Computational Lin-
guistics.
Ghorbani, A.; Abid, A.; and Zou, J. 2019. Interpretation
of Neural Networks Is Fragile. Proceedings of the AAAI
Conference on Artificial Intelligence, 33(01): 3681–3688.
Goodfellow, I.; Warde-Farley, D.; Mirza, M.; Courville, A.;
and Bengio, Y . 2013. Maxout Networks. In Dasgupta, S.;
and McAllester, D., eds., Proceedings of the 30th Interna-
tional Conference on Machine Learning, volume 28 of Pro-


key: chroma:document
string_value:
ceedings of Machine Learning Research, 1319–1327. At-
lanta, Georgia, USA: PMLR.
Granmo, O.-C. 2018. The Tsetlin Machine - A Game
Theoretic Bandit Driven Approach to Optimal Pattern
Recognition with Propositional Logic. arXiv preprint
arXiv:1804.01508.
Granmo, O.-C.; Glimsdal, S.; Jiao, L.; Goodwin, M.; Omlin,
C. W.; and Berge, G. T. 2019. The Convolutional Tsetlin
Machine. arXiv preprint arXiv:1905.09688.
Jain, S.; and Wallace, B. C. 2019. Attention is not Ex-


key: chroma:document
string_value:
planation. In Proceedings of the 2019 Conference of the
North American Chapter of the Association for Computa-
tional Linguistics: Human Language Technologies, Volume
1 (Long and Short Papers), 3543–3556. Minneapolis, Min-
nesota: Association for Computational Linguistics.
Jeevan, P.; and Sethi, A. 2021. Vision Xformers: Efficient
Attention for Image Classification. arXiv:2107.02239.
Jia, K.; and Rinard, M. 2020. Efficient Exact Verification of


key: chroma:document
string_value:
Binarized Neural Networks. In Larochelle, H.; Ranzato, M.;
Hadsell, R.; Balcan, M. F.; and Lin, H., eds., Advances in
Neural Information Processing Systems, volume 33, 1782–
1795. Curran Associates, Inc.Jiao, L.; Zhang, X.; Granmo, O.-C.; and Abeyrathna, K. D.
2022. On the Convergence of Tsetlin Machines for the XOR
Operator. IEEE Trans. Pattern Anal. Mach. Intell.
Kim, Y . 2014. Convolutional Neural Networks for Sen-
tence Classification. In Proceedings of the 2014 Confer-


key: chroma:document
string_value:
ence on Empirical Methods in Natural Language Processing
(EMNLP), 1746–1751. Doha, Qatar: Association for Com-
putational Linguistics.
Lei, J.; Rahman, T.; Shafik, R.; Wheeldon, A.; Yakovlev, A.;
Granmo, O.-C.; Kawsar, F.; and Mathur, A. 2021. Low-
Power Audio Keyword Spotting Using Tsetlin Machines.
Journal of Low Power Electronics and Applications, 11.
Lei, J.; Wheeldon, A.; Shafik, R.; Yakovlev, A.; and
Granmo, O.-C. 2020. From Arithmetic to Logic based AI: A


key: chroma:document
string_value:
Comparative Analysis of Neural Networks and Tsetlin Ma-
chine. In 2020 27th IEEE International Conference on Elec-
tronics, Circuits and Systems (ICECS), 1–4.
Meurer, A.; Smith, C. P.; Paprocki, M.; ˇCert´ık, O.; Kir-
pichev, S. B.; Rocklin, M.; Kumar, A.; Ivanov, S.; Moore,
J. K.; Singh, S.; Rathnayake, T.; Vig, S.; Granger, B. E.;
Muller, R. P.; Bonazzi, F.; Gupta, H.; Vats, S.; Johansson,
F.; Pedregosa, F.; Curry, M. J.; Terrel, A. R.; Rou ˇcka, v.; Sa-


key: chroma:document
string_value:
boo, A.; Fernando, I.; Kulal, S.; Cimrman, R.; and Scopatz,
A. 2017. SymPy: symbolic computing in Python. PeerJ
Computer Science, 3: e103.
Mu, N.; and Gilmer, J. 2019. MNIST-C: A Robustness
Benchmark for Computer Vision. CoRR, abs/1906.02337.
Narodytska, N.; Kasiviswanathan, S.; Ryzhyk, L.; Sagiv, M.;
and Walsh, T. 2018. Verifying properties of binarized deep
neural networks. In Proceedings of the AAAI Conference on
Artificial Intelligence, volume 32.


key: chroma:document
string_value:
Artificial Intelligence, volume 32.
Noveck, I.; Lea, R. B.; Davidson, G. M.; and O’brien, D.
1991. HUMAN REASONING IS BOTH LOGICAL AND
PRAGMATIC. Intellectica, 11: 81–109.
Qin, Q.; Hu, W.; and Liu, B. 2020. Feature Projection for
Improved Text Classification. In Proceedings of the 58th
Annual Meeting of the Association for Computational Lin-
guistics, 8161–8171. Online: Association for Computational
Linguistics.
Radhakrishnan, A.; Durham, C.; Soylemezoglu, A.; and Uh-


key: chroma:document
string_value:
ler, C. 2018. Patchnet: Interpretable Neural Networks for
Image Classification. arXiv:1705.08078.
Ribeiro, M. T.; Singh, S.; and Guestrin, C. 2016. ”Why
Should I Trust You?”: Explaining the Predictions of Any
Classifier. In Proceedings of the 22nd ACM SIGKDD In-
ternational Conference on Knowledge Discovery and Data
Mining, KDD ’16, 1135–1144. New York, NY , USA: Asso-
ciation for Computing Machinery. ISBN 9781450342322.
Robbins, H. 2007. A Stochastic Approximation Method.


key: chroma:document
string_value:
Annals of Mathematical Statistics, 22: 400–407.
Rudin, C. 2019. Stop Explaining Black Box Machine Learn-
ing Models for High Stakes Decisions and Use Interpretable
Models Instead. Nature Machine Intelligence, 1.
Selvaraju, R. R.; Cogswell, M.; Das, A.; Vedantam, R.;
Parikh, D.; and Batra, D. 2017. Grad-CAM: Visual Explana-
tions from Deep Networks via Gradient-Based Localization.
13554


key: chroma:document
string_value:
In2017 IEEE International Conference on Computer Vision
(ICCV), 618–626.
Simonyan, K.; Vedaldi, A.; and Zisserman, A. 2014. Deep
Inside Convolutional Networks: Visualising Image Classifi-
cation Models and Saliency Maps. arXiv:1312.6034.
Srivastava, N.; Hinton, G.; Krizhevsky, A.; Sutskever, I.; and
Salakhutdinov, R. 2014. Dropout: A Simple Way to Pre-
vent Neural Networks from Overfitting. Journal of Machine
Learning Research, 15(56): 1929–1958.


key: chroma:document
string_value:
Learning Research, 15(56): 1929–1958.
Tang, J.; Qu, M.; and Mei, Q. 2015. PTE: Predictive Text
Embedding through Large-scale Heterogeneous Text Net-
works. CoRR, abs/1508.00200.
Valiant, L. G. 1984. A Theory of the Learnable. Communi-
cations of the ACM, 27(11): 1134–1142.
Webster, M. B.; Choi, J.; and changwook Ahn. 2021. Learn-
ing the Connections in Direct Feedback Alignment. In In-
ternational Conference on Learning Representations.
Yadav, R. K.; Jiao, L.; Granmo, O.-C.; and Goodwin, M.


key: chroma:document
string_value:
2021. Enhancing Interpretable Clauses Semantically using
Pretrained Word Representation. In BLACKBOXNLP.
Yadav, R. K.; Jiao, L.; Granmo, O.-C.; and Goodwin, M.
2021. Human-Level Interpretable Learning for Aspect-
Based Sentiment Analysis. In The Thirty-Fifth AAAI Con-
ference on Artificial Intelligence (AAAI-21). AAAI.
Yadav, R. K.; Jiao, L.; Granmo, O.-C.; and Goodwin, M.
2022. Robust Interpretable Text Classification against Spu-
rious Correlations Using AND-rules with Negation. In Inter-


key: chroma:document
string_value:
national Joint Conference on Artificial Intelligence (IJCAI) .
Zhang, Q.; Wu, Y . N.; and Zhu, S.-C. 2018. Interpretable
convolutional neural networks. In Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition ,
8827–8836.
Zhang, X.; Jiao, L.; Granmo, O.-C.; and Goodwin, M.
2022. On the Convergence of Tsetlin Machines for the
IDENTITY- and NOT Operators. IEEE Trans. Pattern Anal.
Mach. Intell., 44(10): 6345–6359.
Zhu, H.; and Koniusz, P. 2020. Simple spectral graph con-


key: chroma:document
string_value:
electronics
Article
Adaptive Sparse Representation of Continuous Input for
Tsetlin Machines Based on Stochastic Searching on the Line
Kuruge Darshana Abeyrathna *, Ole-Christoffer Granmo and Morten Goodwin
/gid00030/gid00035/gid00032/gid00030/gid00038/gid00001/gid00033/gid00042/gid00045 /gid00001
/gid00048/gid00043/gid00031/gid00028/gid00047/gid00032/gid00046
Citation: Abeyrathna, K.D.; Granmo,
O.-C.; Goodwin, M. Adaptive Sparse
Representation of Continuous Input
for Tsetlin Machines Based on


key: chroma:document
string_value:
for Tsetlin Machines Based on
Stochastic Searching on the Line.
Electronics 2021 ,10, 2107. https://
doi.org/10.3390/electronics10172107
Academic Editor: Dimitris Apostolou
Received: 1 July 2021
Accepted: 25 August 2021
Published: 30 August 2021
Publisher’s Note: MDPI stays neutral
with regard to jurisdictional claims in
published maps and institutional afﬁl-
iations.
Copyright: © 2021 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article


key: chroma:document
string_value:
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).Centre for Artiﬁcial Intelligence Research, University of Agder, 4870 Grimstad, Norway;
ole.granmo@uia.no (O.-C.G.); morten.goodwin@uia.no (M.G.)
*Correspondence: darshana.abeyrathna@uia.no
Abstract: This paper introduces a novel approach to representing continuous inputs in Tsetlin


key: chroma:document
string_value:
Machines (TMs). Instead of using one Tsetlin Automaton (TA) for every unique threshold found
when Booleanizing continuous input, we employ two Stochastic Searching on the Line (SSL) automata
to learn discriminative lower and upper bounds. The two resulting Boolean features are adapted to
the rest of the clause by equipping each clause with its own team of SSLs, which update the bounds
during the learning process. Two standard TAs ﬁnally decide whether to include the resulting


key: chroma:document
string_value:
features as part of the clause. In this way, only four automata altogether represent one continuous
feature (instead of potentially hundreds of them). We evaluate the performance of the new scheme
empirically using ﬁve datasets, along with a study of interpretability. On average, TMs with SSL
feature representation use 4.3times fewer literals than the TM with static threshold-based features.
Furthermore, in terms of average memory usage and F1-Score, our approach outperforms simple


key: chroma:document
string_value:
Multi-Layered Artiﬁcial Neural Networks, Decision Trees, Support Vector Machines, K-Nearest
Neighbor, Random Forest, Gradient Boosted Trees (XGBoost), and Explainable Boosting Machines
(EBMs), as well as the standard and real-value weighted TMs. Our approach further outperforms
Neural Additive Models on Fraud Detection and StructureBoost on CA-58 in terms of the Area Under
Curve while performing competitively on COMPAS.


key: chroma:document
string_value:
Curve while performing competitively on COMPAS.
Keywords: Tsetlin Machine; Tsetlin automata; Stochastic Searching on the Line automaton; inter-
pretable AI; interpretable machine learning; XAI; rule-based learning; decision support system
1. Introduction
Deep learning (DL) has signiﬁcantly advanced state-of-the-art models in machine
learning (ML) over the last decade, attaining remarkable accuracy in many ML application


key: chroma:document
string_value:
domains. One of the issues with DL, however, is that DL inference cannot easily be inter-
preted [ 1]. This limits the applicability of DL in high-stakes domains such as medicine [ 2,3],
credit-scoring [ 4,5], churn prediction [ 6,7], bioinformatics [ 8,9], crisis analysis [ 10], and
criminal justice [ 11]. In this regard, the simpler and more interpretable ML algorithms,
such as Decision Trees, Logistic Regression, Linear Regression, and Decision Rules, can


key: chroma:document
string_value:
be particularly suitable. However, they are all hampered by low accuracy when facing
complex problems [ 12]. This limitation has urged researchers to develop machine learning
algorithms that are capable of achieving a better trade-off between interpretability and
accuracy.
While some researchers focus on developing entirely new machine learning algo-
rithms, as discussed above, other researchers try to render DL interpretable. A recent


key: chroma:document
string_value:
attempt to make DL interpretable is the work of Agarwal et al. [ 11]. They introduce Neural
Additive Models (NAMs), which treat each feature independently. The assumption of
independence makes NAMs interpretable but impedes accuracy compared with regular
DL [ 11]. Another approach is to try to explain DL inference with surrogate models. Here,
one strives to attain local interpretability; i.e., explaining individual predictions [ 13]. Nev-


key: chroma:document
string_value:
Electronics 2021 ,10, 2107. https://doi.org/10.3390/electronics10172107 https://www.mdpi.com/journal/electronics


key: chroma:document
string_value:
Electronics 2021 ,10, 2107 2 of 25
ertheless, these explanations are only approximate and cannot explain the complete DL
model (global interpretability) [14].
Many prominent interpretable ML approaches are based on natively interpretable
rules, tracing back to some of the well-known learning models such as association rule
learning [ 15]. These have for instance been used to predict sequential events [ 16]. Other


key: chroma:document
string_value:
examples include the work of Feldman on the difﬁculty of learning formulae in Disjunctive
Normal Form (DNF) [ 17] and Probably Approximately Correct (PAC) learning, which has
provided fundamental insights into machine learning as well as a framework for learning
formulae in DNF [ 18]. Approximate Bayesian techniques are another set of approaches
for the robust learning of rules [ 19,20]. Hybrid Logistic Circuits (HLC), introduced in [ 21],


key: chroma:document
string_value:
are yet another novel approach to interpretable machine learning. Here, layered logical
operators translate into a logistic regression function. HLC has demonstrated promising
accuracy in image classiﬁcation. However, in general, rule-based machine learning scales
poorly and is prone to noise. Indeed, for data-rich problems, in particular those involving
natural language and sensory inputs, rule-based machine learning is inferior to DL.


key: chroma:document
string_value:
Another recent interpretable approach to machine learning is Explainable Boosting
Machines (EBMs) [ 22]. EBMs are highly intelligible and explainable, while their accuracy
is comparable to state-of-the-art machine learning methods such as Random Forest and
Boosted Trees [ 23]. Indeed, EBMs are recognized as state-of-the-art within Generalized
Additive Models (GAMs) [ 22,23]. The EBMs learn feature functions independently, using


key: chroma:document
string_value:
methods such as gradient boosting or bagging. This allows the user to determine how
much each feature contributes to the model’s prediction and is hence directly interpretable.
Despite being rule-based, the recently introduced Tsetlin Machines (TMs) [24] have
obtained competitive accuracy in a wide range of domains [ 25–29], while producing
human-interpretable outputs. At the same time, TMs utilize comparably low computa-


key: chroma:document
string_value:
tional resources [ 30]. Employing a team of TAs [ 31], a TM learns a linear combination of
conjunctive clauses in propositional logic, producing decision rules similar to the branches
in a decision tree (e.g., ifXsatisﬁes condition A and not condition B then Y = 1) [ 26]. In
other words, the TM can be said to unify logistic regression and rule-based learning in a
way that boosts accuracy while maintaining interpretability.


key: chroma:document
string_value:
Regarding recent progress on TMs, they have recently been adopted for various
application domains such as natural language understanding [ 32,33], image classiﬁca-
tion [ 34], and speech processing [ 35]. Simultaneously, the TM architecture and learning
mechanism has been improved in terms of accuracy, computation speed, and energy usage.
The convolutional TM provides competitive performance on MNIST, Fashion-MNIST,


key: chroma:document
string_value:
and Kuzushiji-MNIST, in comparison with CNNs, K-Nearest Neighbor, Support Vector
Machines, Random Forests, Gradient Boosting, Binary Connect, Logistic Circuits, and
ResNet [ 36]. The regression TM [ 27] opens up for continuous output, achieving on par or
better performance compared to Random Forest, Regression Trees, and Support Vector
Regression. With the proposed Booleanization scheme in [ 37], the TM is also able to operate
with continuous features.


key: chroma:document
string_value:
with continuous features.
Furthermore, clauses are enhanced with weights in [ 25]. The weights reduce the
number of clauses required without any loss of accuracy. Later, integer weights replaced
real-valued weights to both reduce the number of clauses and to increase the interpretability
of the TM [ 38]. On several benchmarks, the integer-weighted TM version outperformed
simple Multi-Layered Artiﬁcial Neural Networks, Support Vector Machines, Decision Trees,


key: chroma:document
string_value:
Random Forest, K-Nearest Neighbor, Random Forest, Explainable Boosting Machines
(EBMs), and Gradient Boosted Trees (XGBoost), as well as the standard TM. Further, the
introduced multi-granular clauses for TM in [ 28] eliminate the pattern speciﬁcity parameter
from the TM, consequently simplifying the hyper-parameter search. By indexing the
clauses on the features that falsify them, up to an order of magnitude faster inference and


key: chroma:document
string_value:
learning has been reported [ 29]. Several researchers have further introduced techniques and
architectures that reduce memory footprint and energy usage [ 39], while other techniques
improve learning speed [39,40] and support explainability [41,42].


key: chroma:document
string_value:
Electronics 2021 ,10, 2107 3 of 25
Recent theoretical work proves the convergence to the correct operator for “identity”
and “not”. It is further shown that arbitrarily rare patterns can be recognized using a
quasi-stationary Markov chain-based analysis. The work ﬁnally proves that when two
patterns are incompatible, the most accurate pattern is selected [ 43]. Convergence for the
“XOR” operator has also recently been proven by Jiao et al. [44].


key: chroma:document
string_value:
The approach proposed in [ 37] is currently the most effective way of representing
continuous features through Booleanization. However, the approach requires a large
number of TAs to represent the Booleanized continuous features. Indeed, one TA is needed
per unique continuous value. Consequently, this increases the training time of the TM
as it needs to update all the TAs in all of clauses for each training iteration. Further, this


key: chroma:document
string_value:
adds more post-processing work for generating interpretable rules out of TM outputs. To
overcome this challenge in TMs, we propose a novel approach to represent continuous
features in the TM, encompassing the following contributions:
• Instead of representing each unique threshold found in the Booleanization process by
a TA, we use a Stochastic Searching on the Line (SSL) automaton [ 45] to learn the lower
and upper limits of the continuous feature values. These limits decide the Boolean


key: chroma:document
string_value:
representation of the continuous value inside the clause. Only two TAs then decide
whether to include these bounds in the clause or to exclude them from the clause. In
this way, one continuous feature can be represented by only four automata, instead of
representing it by hundreds of TAs (decided by the number of unique feature values
within the feature).
• A new approach to calculating the clause output is introduced to match with the
above Booleanization scheme.


key: chroma:document
string_value:
above Booleanization scheme.
• We update the learning procedure of the TM accordingly, building upon Type I and
Type II feedback to learn the lower and upper bounds of the continuous input.
• Empirically, we evaluate our new scheme using eight data sets: Bankruptcy, Balance
Scale, Breast Cancer, Liver Disorders, Heart Disease, Fraud Detection, COMPAS,
and CA-58. With the ﬁrst ﬁve datasets, we show how our novel approach affects


key: chroma:document
string_value:
memory consumption, training time, and the number of literals included in clauses, in
comparison with the threshold-based scheme [ 46]. Furthermore, performances on all
these datasets are compared against recent state-of-the-art machine learning models.
This paper is organized as follows. In Section 2, we present the learning automata
foundation we build upon and discuss the SSL automaton in more detail. Then, in Section 3 ,


key: chroma:document
string_value:
we introduce the TM and how it traditionally has dealt with continuous features. We then
propose our new SSL-based scheme. We evaluate the performance of our new scheme
empirically using ﬁve datasets in Section 5. In this section, we use the Bankruptcy dataset
to demonstrate how rules are extracted from TM clause outputs. The prediction accuracy
of the TM SSL-based continuous feature representation is then compared against several


key: chroma:document
string_value:
competing techniques, including ANNs, SVMs, DTs, RF, KNN, EBMs (the current state-of-
the-art of Generalized Additive Models (GAMs) [ 22,23]), Gradient Boosted Trees (XGBoost),
and the TM with regular continuous feature representation. Further, we contrast the
performance of the TM against reported results on recent state-of-the-art machine learning
models, namely NAMs [ 11] and StructureBoost [ 47]. Finally, we conclude our paper in
Section 6.


key: chroma:document
string_value:
Section 6.
2. Learning Automata and the Stochastic Searching on the Line Automaton
The origins of Learning Automata (LA) [ 48] can be traced back to the work of M. L.
Tsetlin in the early 1960s [ 31]. In a stochastic environment, an automaton is capable of
learning the optimum action that has the lowest penalty probability through trial and error.
There are different types of automata; the choice of a speciﬁc type is decided by the nature
of the application [49].


key: chroma:document
string_value:
of the application [49].
Initially, the LA randomly perform an action from an available set of actions. This
action is then evaluated by its attached environment. The environment randomly produces
feedback; i.e., a reward or a penalty as a response to the action selected by the LA. De-


key: chroma:document
string_value:
Electronics 2021 ,10, 2107 4 of 25
pending on the feedback, the state of the LA is adjusted. If the feedback is a reward, the
state changes towards the end state of the selected action, reinforcing the action. When
the feedback is a penalty, the state changes towards the center state of the selected action,
weakening the action and eventually switching the action. The next action of the automaton
is then decided by the new state. In this manner, an LA interacts with its environment


key: chroma:document
string_value:
iteratively. With a sufﬁciently large number of states and a reasonably large number of
interactions with the environment, an LA learns to choose the optimum action with a
probability arbitrarily close to 1.0 [48].
During LA learning, the automaton can make deterministic or stochastic jumps as a
response to the environment feedback. LA make stochastic jumps by randomly changing
states according to a given probability. If this probability is 1.0, the state jumps are deter-


key: chroma:document
string_value:
ministic. Automata of this kind are called deterministic automata. If the transition graph of
the automaton is kept static, we refer to it as a ﬁxed-structure automaton. The TM employs
TAs to decide which literals to include in the clauses. A TA is deterministic and has a ﬁxed
structure, formulated as a ﬁnite-state automaton [ 31]. A TA with 2Nstates is depicted in
Figure 1. States 1 to Nmap to Action 1 and states N+1 to 2 Nmap to Action 2.


key: chroma:document
string_value:
1         2    .....…   N-1      N                  N+1    N+2  …….  2N -1    2N  Action 1          Action 2         
Reward  
Penalty  
Figure 1. Transition graph of a two-action Tsetlin Automaton with 2N memory states.
The Stochastic Searching on the Line (SSL) automaton pioneered by Oommen [ 45] is
somewhat different from regular automata. The SSL automaton is an optimization scheme


key: chroma:document
string_value:
designed to ﬁnd an unknown optimum location λ∗on a line, seeking a value between 0to
1,[0, 1].
In SSL learning, λ∗can be one of the Npoints. In other words, the search space is
divided into Npoints, { 0, 1/N, 2/N,. . .,(N−1)/N, 1}, with Nbeing the discretization
resolution. Depending on the possibly faulty feedback from the attached environment ( E),
λmoves towards the left or right from its current state on the created discrete search space.


key: chroma:document
string_value:
We consider the environment feedback 1, E=1, as an indication to move towards right (or
to increase the value of λ) by one step. The environment feedback 0, E=0, on the other
hand, is considered as an indication to move towards the left (or to decrease the value of λ)
by one step. The next location of λ,λ(n+1)can thus be expressed as follows:
λ(n+1) ={
λ(n) +1/N, if E(n) =1 and 0 ⩽λ(n)<1 ,
λ(n)−1/N, if E(n) =0 and 0 <λ(n)⩽1 .(1)
λ(n+1) ={
λ(n), if λ(n) =1 and E(n) =1 ,


key: chroma:document
string_value:
λ(n+1) ={
λ(n), if λ(n) =1 and E(n) =1 ,
λ(n), if λ(n) =0 and E(n) =0 .(2)
Asymptotically, the learning mechanism is able to ﬁnd a value arbitrarily close to λ∗
when N→∞and n→∞.
3. Tsetlin Machine (TM) for Continuous Features
As seen in Figure 2, conceptually, the TM decomposes into ﬁve layers to recognize
sub-patterns in the data and categorize them into classes. In this section, we explain the
job of each of these layers in the pattern recognition and learning phases of the TM. The


key: chroma:document
string_value:
Electronics 2021 ,10, 2107 5 of 25
 
 
 
 
 
 
 
 
 
 
      
 
 
 
 
 
 
 
 
 
  
 
 
 
 𝑳=[𝑥1,𝑥2,𝑥3…𝑥𝑜,¬𝑥1,¬𝑥2,¬𝑥3…¬𝑥𝑜] ……….  Clause -1            
           
           
           
                                                                               (𝑚×2𝑜) 𝑘 
𝑗 𝑎𝑗,𝑘 𝑨= 𝑎𝑗,𝑘   
𝑎1,𝑘∈{1,…2𝑁}   Clause -2 
𝑎2,𝑘∈{1,…2𝑁}   Clause -m 
𝑎𝑚,𝑘∈{1,…2𝑁}   1/0 1/0 1/0 ……….  𝑣= ∑𝑐𝑗+− ∑𝑐𝑗−
𝑗 𝑗    
- - + 
Layer 1  Layer 2 Layer 3 Layer 4 Layer 5 
Figure 2. The TM structure.


key: chroma:document
string_value:
Figure 2. The TM structure.
Table 1. Parameters and symbols used in Section 3.
Parameter/Symbol Description Parameter/Symbol Description
XInput vector containing
opropositional variablesLThe augmented feature set containing
both original and negated features
xkkthpropositional
variablelk kthliteral
m The number of clauses cj jthclause and stored in vector C
IjThe index set of the included
literals in clause jNThe number of states per action
in the TA
aj,kState of the kthliteral


key: chroma:document
string_value:
in the TA
aj,kState of the kthliteral
in the jthclause, stored
in matrix AI1
XThe indexes of the literals of
value 1
vThe difference between positive
and negative clause outputspjDecision on receiving Type I or Type II
feedback, stored in vector P
T Feedback threshold s Learning sensitivity
rj,kThe decision whether the kth
TA of the jthclause is to receive
Type Ia feedback, stored in
matrix Rqj,kThe decision whether the kth
TA of the jthclause is to receive
Type Ib feedback, stored in
matrix Q


key: chroma:document
string_value:
Type Ib feedback, stored in
matrix Q
IIa Stores TA indexes selected for
Type Ia feedbackIIb Stores TA indexes selected for
Type Ib feedback
⊕Denotes adding 1 to the current
state value of the TA⊖Denotes substracting 1 from the current
state value of the TA
III Stores TA indexes selected for
Type II feedback


key: chroma:document
string_value:
Electronics 2021 ,10, 2107 6 of 25
Layer 1: the input. In the input layer, the TM receives a vector of opropositional
variables: X,xk∈{0, 1}o. The objective here of the TM is to classify this feature vector
into one of the two classes, y∈{0, 1}. However, as shown in Figure 2, the input layer
also includes negations of the original features, ¬xk, in the feature set to capture more
sophisticated patterns. Collectively, the elements in the augmented feature set are called


key: chroma:document
string_value:
literals: L= [x1,x2, . . . , xo,¬x1,¬x2, . . . ,¬xo] = [ l1,l2, . . . , l2o].
Layer 2: clause construction. The sub-patterns associated with class 1and class 0are
captured by mconjunctive clauses. The value mis set by the user where more complex
problems might demand a large m. All clauses receive the same augmented feature set
formulated at the input layer, L. However, to perform the conjunction, only a fraction of the


key: chroma:document
string_value:
literals is utilized. The TM employs two-action TAs in Figure 1 to decide which literals are
included in which clauses. Since we found a number 2×oof literals in L, the same number
of TAs—one per literal k—is needed by a clause to decide the included literals in the clause.
When the index set of the included literals in clause jis given in Ij, the conjunction of the
clause can be performed as follows:
cj=⋀
k∈Ijlk. (3)


key: chroma:document
string_value:
cj=⋀
k∈Ijlk. (3)
Notice how the composition of a clause varies from another clause depending on
the indexes of the included literals in the set Ij⊆{1,. . ., 2o}. For the special case of
Ij=∅—i.e., an empty clause—we have
cj={1during learning
0otherwise .(4)
That is, during learning, empty clauses output 1, and during classiﬁcation, they output 0.
Layer 3: storing states of TAs of clauses in the memory. The TA states on the left-


key: chroma:document
string_value:
hand side of the automaton (states from 1toN) ask to exclude the corresponding literal
from the clause while the states on the right-hand side of the automaton (states from N+1
to2N) ask to include the literal in the clause. The systematic storage of states of TAs in the
matrix, A:A= (aj,k)∈{1,. . ., 2N}m×2o, with jreferring to the clause and kto the literal,
allows us to ﬁnd the index set of the included literals in clause j,IjasIj={k|aj,k>N,
1≤k≤2o}.


key: chroma:document
string_value:
1≤k≤2o}.
Layer 4: clause output. Once the TA decisions are available, the clause output can be
easily computed. Since the clauses are conjunctive, a single literal of value 0is enough to
turn the clause output to 0if its corresponding TA has decided to include it in the clause.
To make the understanding easier, we introduce set I1
X, which contains the indexes of the
literals of value 1. Then, the output of clause jcan be expressed as
cj={
1 if Ij⊆I1
X,
0 otherwise.(5)


key: chroma:document
string_value:
cj={
1 if Ij⊆I1
X,
0 otherwise.(5)
The clause outputs, computed as above, are now stored in vector C, i.e., C= (cj)∈
{0, 1}m.
Layer 5: classiﬁcation: The TM structure given in Figure 2 is used to classify data into
two classes. Hence, sub-patterns associated with each class have to be separately learned.
For this purpose, the clauses are divided into two groups, where one group learns the
sub-pattern of class 1while the other learns the sub-patterns of class 0. For simplicity,


key: chroma:document
string_value:
clauses with an odd index are assigned a positive polarity ( c+
j), and they are used to capture
sub-patterns of output y=1. Clauses with an even index, on the other hand, are assigned
negative polarity ( c−
j) and they seek the sub-patterns of output y=0.
The clauses that recognize sub-patterns output 1. This makes the classiﬁcation process
easier as we simply need to sum the clause outputs of each class and assign the sample


key: chroma:document
string_value:
Electronics 2021 ,10, 2107 7 of 25
identiﬁed from the designated class and that there is a higher chance of the sample being
in that class. Hence, with vbeing the difference in clause output, v=∑jc+
j−∑jc−
j, the
output of the TM is decided as follows:
ˆy={
1 if v≥0
0 if v<0 .(6)
A TM learns online, updating its internal parameters according to one training sample
(X,y)at a time. As we discussed, a TA team decides the clause output, and collectively, the


key: chroma:document
string_value:
output of all the clauses decides the TM’s output. Hence, to maximize the accuracy of the
TM’s output, it is important to sensibly guide individual TAs in clauses. We achieve this
with two kinds of reinforcement: Type I and Type II feedback. Type I and Type II feedback
decide if the TAs in clauses receive a reward, a penalty, or inaction feedback, depending
on the context of their actions. How the type of feedback is decided and how the TAs are


key: chroma:document
string_value:
updated according to the selected feedback type is discussed below in more details.
Type I feedback: Type I feedback has been designed to reinforce the true positive
outputs of the clauses and to combat against the false negative outputs of the clauses. To
reinforce the true positive output of a clause (the clause output is 1 when it has to be 1),
include actions of TAs whose corresponding literal value is 1 are strengthened. However,


key: chroma:document
string_value:
more ﬁne-tuned patterns can be identiﬁed by strengthening the exclude actions of TAs in the
same clause whose corresponding literal value is 0. To combat the false negative outputs
of the clauses (the clause output is 0 when it has to be 1), we erase the identiﬁed pattern
by the clause and make it available for a new pattern. To do so, the exclude actions of TAs,
regardless of their corresponding literal values, are strengthened. We now sub-divide the


key: chroma:document
string_value:
Type I feedback into Type Ia and Ib, where Type Ia handles the reinforcing of exclude actions
while Type Ib works on reinforcing exclude actions of TAs. Together, Type Ia and Type
Ib feedback force clauses to output 1. Hence, clauses with positive polarity need Type I
feedback when y=1and clauses with negative polarity need Type I feedback when y=0.
To diversify the clauses, they are targeted for Type I feedback stochastically as follows:
pj={
1 with probabilityT−max(−T,min(T,v))
2T,


key: chroma:document
string_value:
pj={
1 with probabilityT−max(−T,min(T,v))
2T,
0 otherwise.(7)
All clauses in each class should not learn the same sub-pattern, nor only a few subpatterns.
Hence, clauses should be smartly allocated among the sub-patterns. The user set target T
in(7)does this while deciding the probability of receiving Type I feedback; i.e., Tnumber
of clauses are available to learn each sub-pattern in each class. A higher Tincreases the


key: chroma:document
string_value:
robustness of learning by allocating more clauses to learn each sub-pattern. Now, Ttogether
with vdecides the probability of clause jreceiving Type I feedback, and accordingly, the
decision pjis made. The decisions for the complete set of clauses to receive Type I feedback
are organized in the vector P= (pj)∈{0, 1}m.
Once the clauses to receive Type I feedback are singled out as per (7), the probability
of updating individual TAs in selected clauses is calculated using the user-set parameter


key: chroma:document
string_value:
s(s≥1), separately for Type Ia and Type Ib. According to the above probabilities, the
decision whether the kthTA of the jthclause is to receive Type Ia feedback, rj,kor Type Ib
feedback, qj,kis made stochastically as follows:
rj,k={
1 with probabilitys−1
s,
0 otherwise.(8)
qj,k={
1 with probability1
s,
0 otherwise.(9)


key: chroma:document
string_value:
Electronics 2021 ,10, 2107 8 of 25
The above decisions are stored in the two matrices Rand Q, respectively; i.e., R=
(rj,k)∈{0, 1}m×2oand Q= (qj,k)∈{0, 1}m×2o. Using the complete set of conditions, TA
indexes selected for Type Ia are IIa={(j,k)|lk=1∧cj=1∧pj=1∧rj,k=1}.Similarly,
TA indexes selected for Type Ib are IIb={
(j,k)|(lk=0∨cj=0)∧pj,y=1∧qj,k=1}
.
The states of the identiﬁed TAs are now ready to be updated. Since Type Ia strengthens


key: chroma:document
string_value:
theinclude action of TAs, the current state should move more towards the include action
direction. We denote this as ⊕, and here⊕adds 1 to the current state value of the TA. The
Type Ib feedback, on the other hand, moves the state of the selected TA towards the exclude
action direction to strengthen the exclude action of TAs. We denote this by ⊖, and here,⊖
subtracts 1 from the current state value of the TA. Accordingly, the states of TAs in Aare
updated as A←(
A⊕IIa)⊖IIb.


key: chroma:document
string_value:
updated as A←(
A⊕IIa)⊖IIb.
Type II feedback: Type II feedback has been designed to combat the false positive
output of clauses (the clause output is 1 when it has to be 0). To turn this clause output
from 1to0, a literal value of 0can simply be included in the clause. Clauses with a positive
polarity need Type II feedback when y=0and clauses with negative polarity need this
when y=1as they do not want to vote for the opposite class. Again, using the user-set


key: chroma:document
string_value:
target T, the decision for the jthclause is made as follows:
pj={
1 with probabilityT+max(−T,min(T,v))
2T,
0 otherwise.(10)
The states of the TAs whose corresponding literal of value 0in selected clauses
according to (10) are now moved towards the include action direction with a probability of 1.
Hence, the index set of this kind can be identiﬁed as III={(j,k)|lk=0∧cj=1∧pj=1}.
Accordingly, the states of TAs in Aare updated as A←A⊕III.


key: chroma:document
string_value:
When training has been completed, the ﬁnal decisions of the TAs are recorded, and
the resulting clauses can be deployed for operation.
Booleanization of Continuous Features: In the TM discussed so far, the input layer
accepted only Boolean features; i.e., X= [x1,x2,x3,. . .,xo]with xk,k=1, 2, ..., o, being 0 or
1. These features and their negations were directly fed into the clauses without any further
modiﬁcations. However, continuous features in machine learning applications are more


key: chroma:document
string_value:
common than values of simply 1or0. In one of our previous papers [ 37], we presented
a systematic procedure of transforming continuous features into Boolean features while
maintaining ranking relationships among the continuous feature values.
We here summarize the previous Booleanization scheme using the example presented
in [27]. As seen in Table 2, we Booleanize the two continuous features listed in table
column 1 and column 2.
1. First, for each feature, the unique values are identiﬁed;


key: chroma:document
string_value:
2. The unique values are then sorted from smallest to largest;
3. The sorted unique values are considered as thresholds. In the table, these values can
be seen in the “Thresholds” row;
4. The original feature values are then compared with identiﬁed thresholds, only from
their own feature value set. If the feature value is greater than the threshold, set the
corresponding Boolean variable to 0; otherwise, set it to 1;


key: chroma:document
string_value:
5. The above steps are repeated until all the features are converted into Boolean form.
The ﬁrst feature in the ﬁrst column of the table contains three unique values: 5.779,
10.008, and 3.834 (step (i)). Once they are sorted as required in step ii, we obtain 3.834,
5.779, and 10.008. Now, we consider them as thresholds: ≤3.834,≤5.779, and≤10.008 (step
(iii)). Here, we ﬁnd that each original feature in column 1 is going to represent 3 bit values.


key: chroma:document
string_value:
According to step iv, we now compare the original values in the ﬁrst feature against its
thresholds. The ﬁrst feature value of 5.779 is greater than 3.834 (resulting in 0), equal to
5.779 (resulting in 1), and less than 10.008 (resulting in 1). Hence, we replace 5.779 with 011.
Similarly, 10.008 and 3.834 can be replaced with 001 and 111, respectively.


key: chroma:document
string_value:
Electronics 2021 ,10, 2107 9 of 25
The conversion of the feature values for the second feature starts once all the feature
values in the ﬁrst feature are completed. This procedure is iterated until all the continuous
values of all the continuous features have been converted to Boolean form (step (v)).
Table 2. Preprocessing of two continuous features.
Raw Feature Thresholds
1 2 ≤3.834 ≤5.779 ≤10.008 ≤11.6 ≤25.7 ≤32.4 ≤56.1
5.779 25.7 0 1 1 0 1 1 1
10.008 56.1 0 0 1 0 0 0 1


key: chroma:document
string_value:
10.008 56.1 0 0 1 0 0 0 1
5.779 11.6 0 1 1 1 1 1 1
3.834 32.4 1 1 1 0 0 1 1
This Boolean representation of continuous features is particularly powerful as it allows
the TM to reason about the ordering of the values, forming conjunctive clauses that specify
rules based on thresholds, and with negated features, also rules based on intervals. This
can be explained again with the following example.
The threshold≤3.834 in the “Threshold” row stands for the continuous value 3.834 of


key: chroma:document
string_value:
the ﬁrst feature. Similarly, threshold ≤5.779 and≤10.008 represent the continuous values
5.779 and 10.008 , respectively. Consider a clause with threshold ≤5.779 included in the
clause, which is the only threshold included in the clause. Then, for any input value less
than or equal to≤5.779 from that feature, the clause outputs 1. Now, consider the case
of having two thresholds, ≤5.779 and≤10.008, included in the clause. The threshold


key: chroma:document
string_value:
≤5.779 still decides the clause output due to the fact that the AND of≤5.779 and≤10.008
threshold columns in Table 2 yields the threshold column ≤5.779. When a clause includes a
negated threshold it reverses the original threshold. Consider a clause that only includeds
the negation of the threshold ≤3.834. Now, the clause outputs 1 for all the values greater
than≤3.834 from that feature, as the NOT of≤3.834 is equivalent to 3.834 <.


key: chroma:document
string_value:
The above explanation of the threshold selection reveals that the lowest original
threshold included in the clause and the highest negated threshold included in the clause
decide the upper and lower boundary of the feature values, and these thresholds are the
only important thresholds for calculating the clause output. Hence, this motivates us to
represent the continuous features in clauses in a new way and train the clauses accordingly
as follows.


key: chroma:document
string_value:
as follows.
4. Sparse Representation of Continuous Features
However, the above representation of continuous values in clauses is costly as it needs
two times the total number of unique values of TAs per clause. This is more severe when the
dataset is large and when there is a large number of input features to be considered. Hence,
we introduce SSL automata to represent the upper and lower limits of the continuous


key: chroma:document
string_value:
features. With the new representation, a continuous feature can be then represented by only
two automata instead of having two times the number of unique values in the considered
continuous feature. The new parameters and symbols used in this section are explained
and summarized in Table 3.


key: chroma:document
string_value:
Electronics 2021 ,10, 2107 10 of 25
Table 3. Parameters and symbols used in Section 4.
Parameter/Symbol Description Parameter/Symbol Description
Xc Input vector containing
ocontinuous featuresxc
kkthcontinuous feature
SSLl
j,kLower limit of the continuous
feature kin clause jSSLu
j,kUpper limit of the continuous
feature kin clause j
El
j,kFeedback to the SSL automata
which represent the lower limit
of the feature kin clause jEu
j,kFeedback to the SSL automata
which represent the upper limit


key: chroma:document
string_value:
which represent the upper limit
of the feature kin clause j
TAl
j,kTA which decides whether to
include or exclude lower limit
of the kthfeature in clause jTAu
j,kTA which decides whether to
include or exclude upper limit
of the kthfeature in clause j
rl
j,kThe decision whether the TA of the
lower limit of kthfeature in
thejthclause is to receive
Type Ia feedbackru
j,kThe decision whether the TA of the
upper limit of kthfeature in
thejthclause is to receive
Type Ia feedback
ql


key: chroma:document
string_value:
thejthclause is to receive
Type Ia feedback
ql
j,kThe decision whether the TA of the
lower limit of kthfeature in
thejthclause is to receive
Type Ib feedbackqu
j,kThe decision whether the TA of the
upper limit of kthfeature in
thejthclause is to receive
Type Ib feedback
lj,kComputed literal value for the
kthfeature in clause j
Input Features. As discussed earlier, the TM takes opropositional variables as input,


key: chroma:document
string_value:
X= [x1,x2,x3,. . .,xo]. In this section, we discuss how the TM maps Xcwhich contains o
continuous features, Xc= [xc
1,xc
2,xc
3, . . . , xc
o]into one of two classes y=1 or y=0.
Feature Representation. Each continuous feature is assigned two SSLs to represent
the upper and lower limits of the continuous value in a clause; i.e., SSLl
1,SSLu
1,. . .,SSLl
k,
SSLu
k,. . .,SSLl
o,SSLu
o. Here, SSLl
kand SSLu
kare lower and upper limit values of the kth


key: chroma:document
string_value:
kare lower and upper limit values of the kth
continuous feature, respectively. However, step size Nwithin an SSL in this case is not
constant. When Ein (1) and (2) is 1, the SSL state moves to the higher neighboring unique
value of the attached continuous feature of the SSL. Similarly, when Eis 0, SSL state moves
to the lower neighboring unique value of the considered continuous feature.
Clauses. Each conjunctive clause in the TM receives Xcas an input. The inclusion


key: chroma:document
string_value:
and exclusion decisions of the corresponding upper and lower bounds of xc
kin the clause
are made by TAs. Hence, each clause now needs 2oTAs, where half of them make the
decision related to the lower bound of the continuous features while the other half make
the decision related to the upper bound of the continuous features. The matrix Atherefore
still contains m×2oelements: A= (aj,k)∈{1, . . . , 2 N}m×2o.
In the phase of calculating clause outputs, both limit values given by SSLs and the


key: chroma:document
string_value:
decisions of TAs on their corresponding SSLs are considered. The value of the kthliteral, lj,k,
which represents the kthcontinuous feature inside the clause, j, to perform the conjunction
is evaluated as follows:
• Condition 1: Both TAl
j,kand TAu
j,kwhich respectively make the decision on SSLl
j,kand
SSLu
j,kdecide to include them in the clause. Then,
lj,k={
1, if SSLl
j,k<xc
k≤SSLu
j,k,
0, if xc
k≤SSLl
j,k∨SSLu
j,k<xc
k.(11)


key: chroma:document
string_value:
Electronics 2021 ,10, 2107 11 of 25
• Condition 2: The TAl
j,kdecides to include SSLl
j,kin the clause and TAu
j,kdecides to
exclude SSLu
j,kfrom the clause. Then,
lj,k={
1, if SSLl
j,k<xc
k,
0, if xc
k≤SSLl
j,k.(12)
• Condition 3: The TAu
j,kdecides to include SSLu
j,kin the clause and TAl
j,kdecides to
exclude SSLl
j,kfrom the clause. Then,
lj,k={
1, if xc
k≤SSLu
j,k,
0, if SSLu
j,k<xc
k.(13)
• Condition 4: Both TAl
j,kand TAu
j,kdecide to exclude their corresponding SSLs from


key: chroma:document
string_value:
the clause, which consequently takes the lower limit to the lowest possible and the
upper limit to the highest possible values. Hence, lj,kalways becomes 1 or can be
excluded when conjunction is performed.
Hence, when at least one of the TAs that represent the lower and upper limits decides
to include its corresponding limit in the jthclause, the index of the feature is included in Ij,
Ij⊆{1,. . .,o}. Then, depending on the literal value according to the above conditions, the


key: chroma:document
string_value:
clause output is computed, cj=⋀
k∈Ijlk,j=1, . . . , m.
Classiﬁcation. Similar to the standard TM, the vote difference vis computed as
v=∑jc+
j(Xc)−∑jc−
j(Xc). Once the vote difference is known, the output class is decided
using (6).
Learning. In this new setup, the clauses still receive Type I and Type II feedback.
However, both TAs and SSLs have to be updated as feedback is received. In other words,
Type I and Type II feedback should be able to guide SSLs to learn the optimal lower and


key: chroma:document
string_value:
upper limits of the continuous features in each clause and lead TAs to correctly decide
which limits should be included or excluded in individual clauses.
As discussed earlier, Type Ia feedback reinforces the true positive outputs of clauses
by rewarding the include action of TAs when the literal value is 1. In the new setting, Type
Ia feedback updates both SSLs and TAs when xc
kis within the lower and upper boundaries,
SSLl
j,k<xc
k≤SSLu


key: chroma:document
string_value:
SSLl
j,k<xc
k≤SSLu
j,kand when the clause output is 1 when it has to be 1 (positive clauses
when y=1and negative clauses when y=0). Under these conditions, the decision
regarding whether both the TAs of upper and lower bounds of kthfeature in the jthclause
are to receive Type Ia feedback, rl
j,kand ru
j,k, is stochastically made as follows:
rl
j,k={
1 with probabilitys−1
s,
0 otherwise.(14)
ru
j,k={
1 with probabilitys−1
s,
0 otherwise.(15)
The environment feedbacks El
j,kto update SSLl


key: chroma:document
string_value:
The environment feedbacks El
j,kto update SSLl
j,kand Eu
j,kto update SSLu
j,kare0and
1, respectively. By doing so, we force SSLs to tighten up the boundary of the continuous
feature kand include them in the clause jby reinforcing the include action of TAs. Notice
that the above updates are made only if the condition in (7) is satisﬁed.
Type Ib feedback activates if xc
kis outside any of the upper or lower boundaries,
xc
k≤SSLl
j,k∨SSLu
j,k<xc


key: chroma:document
string_value:
xc
k≤SSLl
j,k∨SSLu
j,k<xc
kor if the clause output is 0. For the case where xc
k≤SSLl
j,kor the
clause output is 0 when it has to be 1, the decision on whether the TA of the lower bound of
thekthfeature in the jthclause should receive Type Ib feedback, ql
j,k, is stochastically made
as follows:


key: chroma:document
string_value:
Electronics 2021 ,10, 2107 12 of 25
ql
j,k={
1 with probability1
s,
0 otherwise.(16)
Similarly, when it violates the upper bound requirement—i.e., SSLu
j,k<xc
kor if the
clause output is 0 when it has to be 1—the decision to receive Type Ib feedback on the TA
which represents upper bound is made as follows:
qu
j,k={
1 with probability1
s,
0 otherwise.(17)
The environment feedbacks for the SSLs when the Type Ib feedback is applied are 0
and 1 for El
j,kand Eu


key: chroma:document
string_value:
and 1 for El
j,kand Eu
j,k, respectively. In this way, SSLs are forced to expand the boundary
and TAs are discouraged from the inclusion of their respective SSLs in the clause.
Once the eligible clauses (positive clauses when y=0and negative clauses when
y=1) to receive Type II feedback are stochastically selected using (10), the states of the
individual SSLs and TAs in them are updated. The original idea of Type II feedback is


key: chroma:document
string_value:
to combat the false positive output of the clause. In the new updating scheme, this is
achieved by expanding the boundaries of the kthfeature if xc
kis outside of the boundary
and including them in the clause, which then turns the clause output to 0 eventually. Hence,
ifxc
k≤SSLl
j,k, the environment feedback on SSLl
j,k,El
j,kbecomes 0 and the state of the TA
that appears for SSLl
j,kincreases by one step with probability 1. Likewise, if SSLu
j,k<xc
k,
the environment feedback on SSLu
j,k,Eu


key: chroma:document
string_value:
j,k<xc
k,
the environment feedback on SSLu
j,k,Eu
j,kbecomes 1 and the state of the TA that appears for
SSLu
j,kincreases by one step with probability 1.
The above decisions on receiving Type Ia, Type Ib, and Type II are stored in IIa,IIb,
and III, respectively. The processing of the training example in the new scheme ends with
the state matrix Aof TAs being updated as A←((
A⊕IIa)⊖IIb)⊕III, and the states
of SSLs are updated according to (1) and (2) with the identiﬁed environment feedback of


key: chroma:document
string_value:
individual SSLs, E.
5. Empirical Evaluation
In this section, the impact of the new continuous input feature representation to the
TM is empirically evaluated using ﬁve real-world datasets (the implementation of Tsetlin
Machine with versions of relevant software libraries and frameworks can be found at:
https://github.com/cair/TsetlinMachine , accessed on 24 August 2021). The datasets Liver
Disorder dataset ,Breast Cancer dataset , and Heart Disease dataset are from the health sector. The


key: chroma:document
string_value:
Balance Scale and Corporate Bankruptcy datasets are the two other remaining datasets. The
Liver Disorder dataset ,Breast Cancer dataset ,Heart Disease dataset , and Corporate Bankruptcy
datasets were selected as these applications in the health and ﬁnance sectors demand
both interpretability and accuracy in predictions. The Balance Scale dataset was added
to diversify the selected applications. As an example, we use the Corporate Bankruptcy


key: chroma:document
string_value:
dataset to examine the interpretability of the TM using both the previous continuous feature
representation and the proposed representation. A summary of these datasets is presented
in Table 4.
Table 4. Binarizing categorical features in the Bankruptcy dataset.
Dataset Number of Instances Number of Attributes Interpretability Needed
Corporate Bankruptcy 250 7 Yes
Balance Scale 625 4 Not necessarily
Breast Cancer 286 9 Yes
Liver Disorders 345 7 Yes
Heart Disease 270 13 Yes


key: chroma:document
string_value:
Electronics 2021 ,10, 2107 13 of 25
The performance of the TM is also contrasted against several other standard machine
learning algorithms, namely Artiﬁcial Neural Networks (ANNs), Decision Trees (DTs),
Support Vector Machines (SVMs), Random Forest (RF), K-Nearest Neighbor (KNN), Ex-
plainable Boosting Machines (EBMs), [ 22], and Gradient Boosted Trees (XGBoost) [ 50]
along with two recent state-of-the-art machine learning approaches: StructureBoost [ 47]


key: chroma:document
string_value:
and Neural Additive Models [ 11]. For comprehensiveness, three ANN architectures are
used: ANN-1—with one hidden layer of 5 neurons, ANN-2—with two hidden layers
of 20 and 50 neurons each, and ANN-3—with three hidden layers and 20, 150, and 100
neurons. The other hyperparameters of each of these models are decided using trial and
error. The reported results in this sections are the average measure over 50 independent


key: chroma:document
string_value:
experiment trials. The data are randomly divided into training (80%) and testing (20%) sets
for each experiment.
5.1. Bankruptcy
In ﬁnance, interpretable machine learning algorithms are preferred over black-box
methods to predict bankruptcy as bankruptcy-related decisions are sensitive. However,
at the same time, the accuracy of the predictions is also important to mitigate ﬁnancial
losses [51].
The Bankruptcy dataset (available from: https://archive.ics.uci.edu/ml/datasets/


key: chroma:document
string_value:
qualitative_bankruptcy, accessed on 24 August 2021) that we consider in this experi-
ment contains historical records of 250companies. The output—i.e., bankruptcy or non-
bankruptcy—is determined by six pertinent features: (1) industrial risk, (2) management
risk, (3) ﬁnancial ﬂexibility, (4) credibility, (5) competitiveness, and (6) operation Risk.
These are categorical features where each feature can be in one of three states: negative (N),
average (A), or positive (P).


key: chroma:document
string_value:
average (A), or positive (P).
The output “bankruptcy” is considered as class 0 while “non-bankruptcy” is class
1. The features are, however, ternary. Thus, the TM has to be used with the proposed
SSL scheme to represent categorical features directly in clauses or features should be
Booleanized using the Booleanization scheme before feeding them into the TM. If the
features are Booleanized beforehand, each feature value can be represented in three


key: chroma:document
string_value:
Boolean features as shown in Table 5. Thus, the complete Booleanized dataset contains 18
Boolean features.
Table 5. Binarizing categorical features in the Bankruptcy dataset.
Category Integer CodeThresholds
≤0 ≤1 ≤2
A 0 1 1 1
N 1 0 1 1
P 2 0 0 1
First, the behavior of the TM with 10 clauses is studied. The included literals in
all these 10 clauses at the end of training are summarized in Table 6. In the TM with


key: chroma:document
string_value:
Booleanized features, the TAs in clause 1 decide to include only the negation of feature 11,
¬x11. Feature 11is the negative credibility , which we can ﬁnd after binarizing all features.
The TAs in clauses 2, 4, 6, 8, and 10 decide to include the negation of average competitiveness
and negative competitiveness , which are non-negated in clauses. The TAs in clauses 3, 5, and
9, on the other hand, decide to include negated negative competitiveness . Clause 7 is “empty”


key: chroma:document
string_value:
Electronics 2021 ,10, 2107 14 of 25
Table 6. Clauses produced by TM with Booleanization and SSL schemes for m=10.
Clause ClassTM with
Booleanized SSLs
1 1 ¬x11 -
2 0 ¬x13∧x14 1<xc
5≤2
3 1 ¬x14 2<xc
5
4 0 ¬x13∧x14 1<xc
5≤2
5 1 ¬x14 -
6 0 ¬x13∧x14 1<xc
5≤2
7 1 - -
8 0 ¬x13∧x14 1<xc
5≤2
9 1 ¬x14 -
10 0 ¬x13∧x14 1<xc
5≤2
Accuracy (Training/Testing) 0.98/1.00 0.99/0.96
Table 6 also contains the clauses learnt by the TM when the SSL continuous feature


key: chroma:document
string_value:
approach is used. The clauses 2, 4, 6, 8, and 10 which vote for bankruptcy activate negative
competitiveness . On the other hand, clause 3 recognizes the sub-patterns of class 1 outputs 1
forpositive competitiveness . There are four free votes for class 1 from the “empty” clauses 1,
5, 7, and 9, which are again ignored during classiﬁcation. Note also that, without loss of
accuracy, the TM with the SSL approach simpliﬁes the set of rules by not including negative


key: chroma:document
string_value:
credibility in any of the clauses. With the identiﬁed thresholds for the continuous values
(categorical in this application), the TM with the SSL approach ends up with the simple
classiﬁcation rule:
Outcome ={
Bankruptcy ifNegative Competitiveness
Non-bankruptcy otherwise .(18)
By asking the TMs to utilize only two clauses, we can obtain the above rule more
directly, as shown in Table 7. As seen, again, the TM with both feature representations
achieves similar accuracy.


key: chroma:document
string_value:
achieves similar accuracy.
Table 7. Clauses produced by TM with Booleanization and SSL schemes for m=2.
Clause ClassTM with
Booleanized SSLs
1 1 ¬x14 2<xc
5
2 0 ¬x13∧x14 1<xc
5≤2
Accuracy (Training/Testing) 0.99/0.96 0.96/0.98
The previous accuracy results represent the majority of experiment trials. Some experi-
ments fail to obtain this optimum accuracy. Instead of conducting the experiments multiple
times to ﬁnd the optimum clause conﬁguration in the TM, the number of clauses can be


key: chroma:document
string_value:
increased to ﬁnd more robust conﬁgurations of clauses. Even though this provides stable
higher accuracy for almost all the trials, a large number of clauses affects the interpretability.
This is where we have to consider achieving a balance between accuracy and interpretabil-
ity. For the Bankruptcy dataset, how robustness increases with clauses can be seen in
Tables 8 and 9 . The average performance (precision, recall, F1-Score, accuracy, speciﬁcity)


key: chroma:document
string_value:
Electronics 2021 ,10, 2107 15 of 25
Table 8. Performance of TM with Booleanized continuous features on Bankruptcy dataset.
m 2 10 100 500 2000 8000
Precision 0.754 0.903 0.997 0.994 0.996 0.994
Recall 1.000 1.000 1.000 0.998 1.000 1.000
F1-Score 0.859 0.948 0.984 0.996 0.998 0.997
Accuracy 0.807 0.939 0.998 0.996 0.998 0.996
Speciﬁcity 0.533 0.860 0.995 0.993 0.996 0.990
No. of Lit. 19 88 222 832 3622 15,201
Table 9. Performance of TM with SSL continuous feature scheme on Bankruptcy dataset.


key: chroma:document
string_value:
m 2 10 100 500 2000 8000
Precision 0.622 0.777 0.975 0.995 0.994 0.996
Recall 0.978 0.944 0.994 1.000 0.997 0.995
F1-Score 0.756 0.843 0.984 0.997 0.995 0.995
Accuracy 0.640 0.787 0.982 0.997 0.995 0.994
Speciﬁcity 0.191 0.568 0.967 0.994 0.993 0.993
No. of Lit. 8 40 94 398 1534 7286
Table 8 reports the results of the TM with a regular Booleanization scheme. The goal
here is to maximize the F1-Score, since accuracy can be misleading for imbalanced datasets.


key: chroma:document
string_value:
As can be seen, the F1-Score increases with clauses and peaks at m=2000 . To obtain this
performance with the Booleanized features, the TM classiﬁer uses 3622 (rounded to nearest
integer) literals (include actions).
Despite the slight reduction in the F1-Score, the TM with the proposed continuous
feature representation reaches its best F1-Score with only 398literals in the TM classiﬁer.
This represents a greater than nine-fold reduction of literals, which is more signiﬁcant


key: chroma:document
string_value:
compared to the reduction of the F1-Score ( 0.001 ). At this point, the number of clauses, m,
equals 500.
Figure 3 outlines how the number of literals varies with the increase of the number
of clauses. The TM with the new continuous feature representation scheme consistently
uses fewer literals in its classiﬁer than the TM with regular feature representation. The
difference between the number of literals in both approaches increases with the number of
clauses.
2 10 100 500 2000 8000


key: chroma:document
string_value:
clauses.
2 10 100 500 2000 8000
No. of Clauses0200040006000800010,00012,00014,000No. of LiteralsBinarized features
SSLs scheme16,000
Figure 3. The number of literals included in TM clauses to work with Bankruptcy dataset.
The performance of the TM with both continuous feature arrangements is compared
against multiple standard machine learning models: ANN, KNN, XGBoost, DT, RF, SVM,
and EBM. The performances of these techniques along with the best performance of the


key: chroma:document
string_value:
Electronics 2021 ,10, 2107 16 of 25
TM setups are summarized in Table 10. The best F1-Score is obtained by the TM with
regular Booleanized features. The second best F1-Score belongs to ANN-3 and the TM with
the SSL scheme. Memory wise, the TM with both input feature representations together
with DT needs close to zero memory at both training and testing, while ANN-3 requires a
training memory of 28,862.65 KB and a testing memory of 1297.12 KB. More importantly,


key: chroma:document
string_value:
the training time per epoch and the number of literals in clauses are reduced with the SSL
scheme for the TM compared to the Booleanization approach.
Table 10. Performance comparison for Bankruptcy dataset.
Prec. Reca. F1 Acc. Spec. No. of Lit.Memory RequiredTraining Time(Training/Testing)
ANN-1 0.990 1.000 0.995 0.994 0.985 - ≈942.538 KB/≈26.64 KB 0.227 s.
ANN-2 0.995 0.997 0.996 0.995 0.993 - ≈3476.76 KB/≈590.76 KB 0.226 s.


key: chroma:document
string_value:
ANN-3 0.997 0.998 0.997 0.997 0.995 - ≈28,862.65 KB/≈1297.12 KB 0.266 s.
DT 0.988 1.000 0.993 0.993 0.985 - ≈0.00 KB/≈0.00 KB 0.003 s.
SVM 1.000 0.989 0.994 0.994 1.000 - ≈90.11 KB/≈0.00 KB 0.001 s.
KNN 0.998 0.991 0.995 0.994 0.998 - ≈0.00 KB/≈286.71 KB 0.001 s.
RF 0.979 0.923 0.949 0.942 0.970 - ≈180.22 KB/≈0.00 KB 0.020 s.
XGBoost 0.996 0.977 0.983 0.983 0.992 - ≈4964.35 KB/≈0.00 KB 0.009 s.
EBM 0.987 1.000 0.993 0.992 0.980 - ≈1425.40 KB/≈0.00 KB 13.822 s.


key: chroma:document
string_value:
TM (Booleanized) 0.996 1.000 0.998 0.998 0.996 3622 ≈0.00 KB/≈0.00 KB 0.148 s.
TM (SSLs) 0.995 1.000 0.997 0.997 0.994 398 ≈0.00 KB/≈0.00 KB 0.119 s.
5.2. Balance Scale
We then move to the Balance Scale dataset (available from http://archive.ics.uci.edu/
ml/datasets/balance+scale, accessed on 24 August 2021). The Balance Scale dataset consists
of three classes: a balance scale that tips to the left, tips to the right, or that is in balance.


key: chroma:document
string_value:
The above class is decided collectively by four features: (1) the size of the weight on the
left-hand side, (2) distance from the center to the weight on the left, (3) size of the weight on
the right-hand side, and (4) distance from the center to the weight on the right. However,
we remove the third class—i.e., “balanced”—and contract the output to a Boolean form.
The resulting dataset ends up with 576 data samples.


key: chroma:document
string_value:
Tables 11 and 12 contain the results of the TM with two continuous feature representa-
tions, with varying m. The F1-Score reaches its maximum of 0.945 atm=100for the TM
with the Boolean feature arrangement. The average number of literals used to achieve the
above performance is 790. The TM with the SSL scheme reaches its maximum performance
when m=500. The number of literals used in the classiﬁer to achieve this performance
is 668.


key: chroma:document
string_value:
is 668.
Table 11. Performance of TM with Booleanized continuous features on Balance Scale dataset.
m 2 10 100 500 2000 8000
Precision 0.647 0.820 0.966 0.949 0.926 0.871
Recall 0.986 0.965 0.930 0.934 0.884 0.746
F1-Score 0.781 0.886 0.945 0.933 0.880 0.749
Accuracy 0.728 0.875 0.948 0.936 0.889 0.780
Speciﬁcity 0.476 0.782 0.966 0.935 0.905 0.819
No. of Lit. 17 77 790 3406 15,454 60,310
Table 12. Performance of TM with SSL continuous feature scheme on Balance Scale dataset.


key: chroma:document
string_value:
m 2 10 100 500 2000 8000
Precision 0.579 0.717 0.919 0.961 0.877 0.851
Recall 0.957 0.947 0.972 0.938 0.867 0.794
F1-Score 0.717 0.812 0.944 0.946 0.854 0.781
Accuracy 0.612 0.777 0.943 0.948 0.854 0.795
Speciﬁcity 0.254 0.598 0.916 0.959 0.840 0.795
No. of Lit. 4 17 140 668 2469 9563


key: chroma:document
string_value:
Electronics 2021 ,10, 2107 17 of 25
The variation of the number of literals over different numbers of clauses in the TM
with these two continuous feature arrangements is graphed in Figure 4. The TM with the
SSL scheme uses a smaller number of literals for all the considered number of clauses, with
the difference increasing with number of clauses.
2 10 100 500 2000 8000
No. of Clauses010,00020,00030,00040,00050,00060,000No. of LiteralsBinarized features
SSLs scheme


key: chroma:document
string_value:
SSLs scheme
Figure 4. The number of literals included in TM clauses to work with the Balance Scale dataset.
For the Balance Scale dataset, the performances of the other machine learning al-
gorithms are also obtained. Along with the TM performance, the prediction accuracies
of other models are presented in Table 13. The highest F1-Score from all the considered
models is procured by EBM. Out of the two TM approaches, the TM with the SSL scheme


key: chroma:document
string_value:
shows the best performance in terms of the F1-Score, while using less training time and
training memory.
Table 13. Performance comparison for Balance Scale dataset.
Prec. Reca. F1 Acc. Spec. No. of Lit.Memory RequiredTraining Time(Training/Testing)
ANN-1 0.993 0.987 0.990 0.990 0.993 - ≈966.57 KB/≈24.56 KB 0.614 s.
ANN-2 0.995 0.995 0.995 0.995 0.994 - ≈3612.65 KB/≈589.82 KB 0.588 s.
ANN-3 0.995 0.995 0.995 0.995 0.995 - ≈33,712.82 KB/≈1478.64 KB 0.678 s.


key: chroma:document
string_value:
DT 0.984 0.988 0.986 0.986 0.985 - ≈131.07 KB/≈0.00 KB 0.007 s.
SVM 0.887 0.889 0.887 0.887 0.884 - ≈65.53 KB/≈241.59 KB 0.001 s.
KNN 0.968 0.939 0.953 0.953 0.969 - ≈249.77 KB/≈126.87 KB 0.001 s.
RF 0.872 0.851 0.859 0.860 0.871 - ≈0.00 KB/≈0.00 KB 0.021 s.
XGBoost 0.942 0.921 0.931 0.931 0.942 - ≈1126.39 KB/≈0.00 KB 0.030 s.
EBM 1.000 1.000 1.000 1.000 1.000 - ≈1642.49 KB/≈0.00 KB 15.658 s.
TM (Booleanized) 0.966 0.930 0.945 0.948 0.966 790 ≈16.37 KB/≈0.00 KB 0.011 s.


key: chroma:document
string_value:
TM (SSLs) 0.961 0.938 0.946 0.948 0.959 668 ≈9.43 KB/≈0.00 KB 0.004 s.
5.3. Breast Cancer
The nine features in the Breast Cancer dataset (available from: https://archive.ics.uci.
edu/ml/datasets/Breast+Cancer, accessed on 24 August 2021) predict the recurrence of
breast cancer. The nine features in the dataset are age, menopause, tumor size, inverse
nodes, node caps, degree of malignancy, side (left or right), the position of the breast,


key: chroma:document
string_value:
and irradiation status. The numbers of samples in the “non-recurrence” and “recurrence”
classes are 201 and 85, respectively. However, some of these samples are removed as they
contain missing values in their features.
The performances of the TMs with two feature arrangements and the number of
literals they included in their clauses to achieve this performance are summarized in
Tables 14 and 15 , respectively, for the Booleanized scheme and the SSL scheme. In contrast


key: chroma:document
string_value:
Electronics 2021 ,10, 2107 18 of 25
peaks at m=2. The performance then decreases with the increase of m. The numbers of
literals at this phase in TMs with Booleanized and SSL feature arrangements are 21 and 4,
respectively. Overall, the TM with the SSL scheme requires the smallest amount of literals,
as can be seen in Figure 5.
Table 14. Performance of TM with Booleanized continuous features on Breast Cancer dataset.
m 2 10 100 500 2000 8000
Precision 0.518 0.485 0.295 0.101 0.058 0.054


key: chroma:document
string_value:
Precision 0.518 0.485 0.295 0.101 0.058 0.054
Recall 0.583 0.380 0.416 0.205 0.200 0.250
F1-Score 0.531 0.389 0.283 0.089 0.090 0.088
Accuracy 0.703 0.737 0.644 0.633 0.649 0.581
Speciﬁcity 0.742 0.864 0.731 0.800 0.800 0.750
No. of Lit. 21 73 70 407 1637 6674
Table 15. Performance of TM with SSL continuous feature scheme on Breast Cancer dataset.
m 2 10 100 500 2000 8000
Precision 0.465 0.468 0.071 0.126 0.090 0.070
Recall 0.759 0.575 0.233 0.467 0.333 0.233


key: chroma:document
string_value:
Recall 0.759 0.575 0.233 0.467 0.333 0.233
F1-Score 0.555 0.494 0.109 0.195 0.141 0.107
Accuracy 0.645 0.701 0.630 0.525 0.589 0.628
Speciﬁcity 0.599 0.753 0.778 0.551 0.682 0.775
No. of Lit. 4 16 101 321 997 4276
2 10 100 500 2000 8000
No. of Clauses01000200030004000500060007000No. of LiteralsBinarized features
SSLs scheme
Figure 5. The number of literals included in TM clauses to work with Breast Cancer dataset.


key: chroma:document
string_value:
All the other algorithms obtain an F1-Score of less than 0.5. The performances of DT,
RF, SVM, XGBoost, and EBM can be identiﬁed as the worst of all models as summarized
inTable 16 . The best F1-Score is obtained by the TM with the SSL feature representation
procedure, while the TM with Booleanized features obtain the second best F1-Score. The
increase of the F1-Score from 0.531 to 0.555 comes also with the advantage of having 19


key: chroma:document
string_value:
less literals in clauses for the SSL approach. Both the training and testing memory usage
of the TM with these two feature arrangements is negligible. The TM also has the lowest
training time of all algorithms, amounting to 0.001 s.


key: chroma:document
string_value:
Electronics 2021 ,10, 2107 19 of 25
Table 16. Performance comparison for Breast Cancer dataset.
Prec. Reca. F1 Acc. Spec. No. of Lit.Memory RequiredTraining Time(Training/Testing)
ANN-1 0.489 0.455 0.458 0.719 0.822 - ≈1001.97 KB/≈35.74 KB 0.249 s.
ANN-2 0.430 0.398 0.403 0.683 0.792 - ≈3498.47 KB/≈608.71 KB 0.248 s.
ANN-3 0.469 0.406 0.422 0.685 0.808 - ≈38,645.07 KB/≈1837.76 KB 0.288 s.
DT 0.415 0.222 0.276 0.706 0.915 - ≈102.39 KB/≈0.00 KB 0.005 s.


key: chroma:document
string_value:
SVM 0.428 0.364 0.384 0.678 0.805 - ≈241.66 KB/≈299.00 KB 0.001 s.
KNN 0.535 0.423 0.458 0.755 0.871 - ≈249.85 KB/≈61.43 KB 0.001 s.
RF 0.718 0.267 0.370 0.747 0.947 - ≈139.26 KB/≈0.00 KB 0.020 s.
XGBoost 0.428 0.344 0.367 0.719 0.857 - ≈1327.10 KB/≈0.00 KB 0.026 s.
EBM 0.713 0.281 0.389 0.745 0.944 - ≈1724.41 KB/≈0.00 KB 6. 007 s.
TM (Booleanized) 0.518 0.583 0.531 0.703 0.742 21 ≈0.00 KB/≈0.00 KB 0.001 s.
TM (SSLs) 0.465 0.759 0.555 0.645 0.599 4 ≈0.00 KB/≈0.00 KB 0.001 s.


key: chroma:document
string_value:
5.4. Liver Disorders
The Liver Disorders dataset (available from: https://archive.ics.uci.edu/ml/datasets/
Liver+Disorders, accessed on 24 August 2021) was created in the 1980s by BUPA Medical
Research and Development Ltd. (hereafter “BMRDL”) as a part of a larger health-screening
database. The dataset contains data in seven columns: mean corpuscular volume, alkaline
phosphotase, alamine aminotransferase, aspartate aminotransferase, gamma-glutamyl


key: chroma:document
string_value:
transpeptidase, number of half-pint equivalents of alcoholic beverages (drunk per day),
and selector. By taking the selector attribute as a class label, some researchers have used this
dataset incorrectly [ 52]. However, in our experiments, the “number of half-pint equivalents
of alcoholic beverages” is used as the dependent variable, Booleanized using the threshold
≥3. Further, only results of various blood tests are used as feature attributes; i.e., the
selector attribute is discarded.


key: chroma:document
string_value:
selector attribute is discarded.
Tables 17 and 18 summarize the performance of the TM with two feature arrangements.
As can be seen, the F1-Scores of the TM with Booleanized continuous features peak at
m=2, while this value for the TM with the SSL scheme is m=10. With 10 clauses, the
method of representing continuous features for the TM with theSSL scheme considers
merely 9 literals in clauses to acquire a better F1-Score. The increase of the number of


key: chroma:document
string_value:
literals included in TM clauses with the increase of the number of clauses can be seen in
Figure 6. Again, this conﬁrms that the TM with the SSL scheme uses a considerably smaller
number of literals overall.
Table 17. Performance of TM with Booleanized continuous features on Liver Disorders dataset.
m 2 10 100 500 2000 8000
Precision 0.566 0.540 0.506 0.455 0.442 0.417
Recall 0.799 0.597 0.508 0.595 0.500 0.593
F1-Score 0.648 0.550 0.389 0.450 0.375 0.437


key: chroma:document
string_value:
F1-Score 0.648 0.550 0.389 0.450 0.375 0.437
Accuracy 0.533 0.540 0.516 0.522 0.526 0.504
Speciﬁcity 0.204 0.436 0.497 0.395 0.500 0.396
No. of Lit. 27 51 117 509 2315 8771
Table 18. Performance of TM with SSL continuous feature scheme on Liver Disorders dataset.
m 2 10 100 500 2000 8000
Precision 0.619 0.591 0.546 0.420 0.414 0.522
Recall 0.905 0.924 0.605 0.700 0.700 0.407
F1-Score 0.705 0.709 0.447 0.525 0.520 0.298
Accuracy 0.587 0.574 0.526 0.546 0.543 0.461


key: chroma:document
string_value:
Accuracy 0.587 0.574 0.526 0.546 0.543 0.461
Speciﬁcity 0.101 0.098 0.400 0.300 0.300 0.600
No. of Lit. 2 9 89 452 1806 7229


key: chroma:document
string_value:
Electronics 2021 ,10, 2107 20 of 25
2 10 100 500 2000 8000
No. of Clauses02000400060008000No. of LiteralsBinarized features
SSLs scheme
Figure 6. The number of literals included in TM clauses to work with the Liver Disorders dataset.
From the performance of the other machine learning models, summarized in Table 19,
we can observe that the highest F1-Score ( 0.729 ) is produced by RF. The performance of DT
in terms of the F1-Score is comparable to the performance of RF. However, DT requires a


key: chroma:document
string_value:
training memory of 49.15 KB, while RF uses a negligibly small memory both for training
and testing to work with the Liver Disorders dataset. The TM, on the other hand, performs
better with SSL continuous features representation than with the Booleanized continuous
features. This performance is the fourth best among all the other models. For training and
testing, the TMs with both feature representation approaches require an insigniﬁcantly


key: chroma:document
string_value:
small amount of memory. However, the TM with SSL feature representation requires less
time for training.
Table 19. Performance comparison for Liver Disorders dataset.
Prec. Reca. F1 Acc. Spec. No. of Lit.Memory RequiredTraining Time(Training/Testing)
ANN-1 0.651 0.702 0.671 0.612 0.490 - ≈985.13 KB/≈18.53 KB 0.305 s.
ANN-2 0.648 0.664 0.652 0.594 0.505 - ≈3689.39 KB/≈598.26 KB 0.305 s.
ANN-3 0.650 0.670 0.656 0.602 0.508 - ≈38,365.46 KB/≈1758.23 KB 0.356 s.


key: chroma:document
string_value:
DT 0.591 0.957 0.728 0.596 0.135 - ≈49.15 KB/≈0.00 KB 0.025 s.
SVM 0.630 0.624 0.622 0.571 0.500 - ≈1597.43 KB/≈0.00 KB 0.005 s.
KNN 0.629 0.651 0.638 0.566 0.440 - ≈0.00 KB/≈434.17 KB 0.001 s.
RF 0.618 0.901 0.729 0.607 0.192 - ≈0.00 KB/≈0.00 KB 0.017 s.
XGBoost 0.641 0.677 0.656 0.635 0.568 - ≈3219.45 KB/≈0.00 KB 0.081 s.
EBM 0.641 0.804 0.710 0.629 0.406 - ≈7790.59 KB/≈0.00 KB 10.772 s.
TM (Booleanized) 0.566 0.799 0.648 0.533 0.204 27 ≈0.00 KB/≈0.00 KB 0.003 s.


key: chroma:document
string_value:
TM (SSLs) 0.591 0.924 0.709 0.574 0.098 9 ≈0.00 KB/≈0.00 KB 0.001 s.
5.5. Heart Disease
The last dataset we use is the Heart Disease dataset (Available from: https://archive.ics.
uci.edu/ml/datasets/Statlog+%28Heart%29, accessed on 24 August 2021). The goal of
this dataset is to predict future heart disease risk based on historical data. The complete
dataset consists of 75 features. However, in this experiment, the updated version of the


key: chroma:document
string_value:
dataset, containing 13 features, is used: one ordered, six real-valued, three nominal, and
three Boolean features.
Tables 20 and 21 summarize the performance of the TM with two feature arrangement
schemes. For the TM with Boolean features, the best F1-Score occurs with m=10, achieved
by using 346literals on average. The F1-Score of the TM with SSL continuous features
peaks again at m=10with only 42literals. Even though the TM with Boolean features


key: chroma:document
string_value:
performs better in terms of accuracy, the TM with SSL feature representation outperforms
the Boolean representation of continuous features by obtaining a higher F1-Score.


key: chroma:document
string_value:
Electronics 2021 ,10, 2107 21 of 25
Table 20. Performance of TM with Booleanized continuous features on Heart Disease dataset.
m 2 10 100 500 2000 8000
Precision 0.547 0.607 0.835 0.507 0.351 0.360
Recall 0.938 0.815 0.626 0.408 0.646 0.486
F1-Score 0.682 0.687 0.665 0.383 0.446 0.392
Accuracy 0.593 0.672 0.749 0.619 0.533 0.584
Speciﬁcity 0.306 0.566 0.848 0.803 0.460 0.665
No. of Lit. 118 346 810 1425 11,399 52,071


key: chroma:document
string_value:
No. of Lit. 118 346 810 1425 11,399 52,071
Table 21. Performance of TM with SSL continuous feature scheme on Heart Disease dataset.
m 2 10 100 500 2000 8000
Precision 0.529 0.588 0.562 0.305 0.674 0.687
Recall 0.971 0.915 0.504 0.431 0.660 0.667
F1-Score 0.680 0.714 0.510 0.343 0.571 0.555
Accuracy 0.591 0.674 0.709 0.630 0.633 0.581
Speciﬁcity 0.272 0.471 0.853 0.701 0.582 0.512
No. of Lit. 10 42 151 783 3152 12,365


key: chroma:document
string_value:
No. of Lit. 10 42 151 783 3152 12,365
Considering the number of literals used with an increasing number of clauses ( Figure 7 ),
both approaches behave almost similarly until m=500, and then the TM with Booleanized
features includes more literals in clauses than the proposed approach.
2 10 100 500 2000 8000
No. of Clauses010,00020,00030,00040,00050,000No. of LiteralsBinarized features
SSLs scheme
Figure 7. The number of literals included in TM clauses to work with Heart Disease dataset.


key: chroma:document
string_value:
Out of the considered machine learning models, as summarized in Table 22, EBM
obtains the best F1-Score. However, EBM needs the highest training time and uses the
second largest amount of training memory, while both TMs use negligible memory during
both training and testing and consume much less training time than EBM.
Table 22. Performance comparison for Heart Disease dataset.
Prec. Reca. F1 Acc. Spec. No. of Lit.Memory RequiredTraining Time(Training/Testing)


key: chroma:document
string_value:
ANN-1 0.764 0.724 0.738 0.772 0.811 - ≈973.64 KB/≈16.46 KB 0.297 s.
ANN-2 0.755 0.736 0.742 0.769 0.791 - ≈3659.59 KB/≈578.11 KB 0.266 s.
ANN-3 0.661 0.662 0.650 0.734 0.784 - ≈33,952.49 KB/≈1513.41 KB 0.308 s.
DT 0.827 0.664 0.729 0.781 0.884 - ≈0.00 KB/≈266.23 KB 0.016 s.
SVM 0.693 0.674 0.679 0.710 0.740 - ≈1363.96 KB/≈262.14 KB 0.004 s.
KNN 0.682 0.615 0.641 0.714 0.791 - ≈0.00 KB/≈319.48 KB 0.001 s.
RF 0.810 0.648 0.713 0.774 0.879 - ≈413.69 KB/≈0.00 KB 0.017 s.


key: chroma:document
string_value:
XGBoost 0.712 0.696 0.701 0.788 0.863 - ≈3694.58 KB/≈0.00 KB 0.057 s.
EBM 0.827 0.747 0.783 0.824 0.885 - ≈4763.64 KB/≈0.00 KB 11.657 s.
TM (Booleanized) 0.607 0.815 0.687 0.672 0.566 346 ≈0.00 KB/≈0.00 KB 0.014 s.
TM (SSLs) 0.588 0.915 0.714 0.674 0.471 42 ≈0.00 KB/≈0.00 KB 0.001 s.


key: chroma:document
string_value:
Electronics 2021 ,10, 2107 22 of 25
5.6. Summary of Empirical Evaluation
To compare the overall performance of the various techniques, we calculate the
average F1-Scores across the datasets. Furthermore, to evaluate the overall interpretability
of TMs, we also report the average number of literals used overall.
In brief, the average F1-Scores of ANN-1, ANN-2, ANN-3, DT, SVM, KNN, RF, XG-
Boost, EBM, TM (Booleanized), and TM (SSL) are 0.770 ,0.757 ,0.744 ,0.742 ,0.713 ,0.737 ,


key: chroma:document
string_value:
0.724 ,0.728 ,0.775 ,0.762 , and 0.782 , respectively. Out of all the considered models, the
TM with SSL continuous feature representation obtains the best average F1-Score, which
is0.782 . It should also be noted that increasing ANN model complexity (from ANN-1 to
ANN-3) reduces the overall F1-Score, which can potentially be explained by the small size
of the datasets.
Indeed, the average numbers of literals employed are 961for the TM with Booleanized


key: chroma:document
string_value:
continuous features and 224for the TM with the SSL feature scheme. That is, the TM
with SSL feature representation uses 4.3times fewer literals than the TM with Booleanized
continuous features.
The average combined memory requirements (training and testing) for the TM ap-
proaches are 3.27 KB and 1.89 KB for Booleanized features and SSL features, respec-
tively. The combined memory usage of the TM with SSL feature representation is signif-


key: chroma:document
string_value:
icantly less compared to the other models—ANN-1: ≈528 times, ANN-2: ≈2211 times,
ANN-3:≈19,197 times, DT:≈59times, SVM:≈441times, KNN:≈1836 times, RF:≈78
times, XGBoost:≈1517 times, and EBM: ≈2121 times.
It should also be noted that increasing the number of clauses stabilizes the precision,
recall, F1-score, accuracy, and speciﬁcity measures, rendering variance insigniﬁcant. That
is, variance becomes negligible for all the datasets and feature representations.


key: chroma:document
string_value:
5.7. Comparison against Recent State-of-the-Art Machine Learning Models
In this section, we compare the accuracy of the TM with reported results on recent state-
of-the-art machine learning models. First, we perform experiments on Fraud Detection
and COMPAS: Risk Prediction in Criminal Justice datasets to study the performance of
the TM in comparison with Neural Additive Models [ 11]. A Neural Additive Model is
a novel member of the so-called General Adaptive Models. In Neural Additive Models,


key: chroma:document
string_value:
the signiﬁcance of each input feature towards the output is learned by a dedicated neural
network. During the training phase, the complete set of neural networks is jointly trained
to learn complex interactions between inputs and outputs.
To compare the performance against StructureBoost [ 47], we use the CA weather
dataset [ 53]. For simplicity, we use only the CA-58 subset of the dataset in this study.
StructureBoost is based on gradient boosting and is capable of exploiting the structure of


key: chroma:document
string_value:
categorical variables. StructureBoost outperforms established models such as CatBoost
and LightBoost on multiple classiﬁcation tasks [47].
Since the performance of both of the above techniques has been measured in terms of
the Area Under the ROC Curve (AUC), here, we use a soft TM output layer [ 54] to calculate
the AUC. The performance characteristics are summarized in Table 23.
Table 23 shows that for the Fraud Detection dataset, the TM with the SSL continuous


key: chroma:document
string_value:
feature representation approach performs on par with XGBoost and outperforms NAMs
and all the other techniques mentioned in [ 11]. For the COMPAS dataset, the TM with
the SSL feature arrangement exhibits competitive performance compared to NAMs, EBM,
XGBoost, and DNNs. The TM with SSL feature representation shows, however, superior
performance compared to Logistic Regression and DT on COMPAS. The performance of
the TM on CA-20 is better in comparison to StructureBoost, LightBoost, and CatBoost


key: chroma:document
string_value:
Electronics 2021 ,10, 2107 23 of 25
Table 23. Performance (in AUC) comparison against recent state-of-the-art machine learning models.
Model Fraud Detection COMPAS CA-58
Logistic Regression 0.975 0.730 -
DT 0.956 0.723 -
NAMs 0.980 0.741 -
EBM 0.976 0.740 -
XGBoost 0.981 0.742 -
DNNs 0.978 0.735 -
LightBoost - - ≈0.760 †
CatBoost - - ≈0.760 †
StructureBoost - - ≈0.764 †
TM (SSLs) 0.981 0.732 0.770
† These results were extracted from graphs in [47].
6. Conclusions


