# The Theory of Intrinsic Time

A Primer

James B. Glattfelder  Lykke Corp  Richard B. Olsen  Lykke Corp

###### Abstract

The concept of time mostly plays a subordinate role in finance and economics.
The assumption is that time flows continuously and that time series data
should be analyzed at regular, equidistant intervals. Nonetheless, already
nearly 60 years ago, the concept of an event-based measure of time was first
introduced. This paper expands on this theme by discussing the paradigm of
intrinsic time, its origins, history, and modern applications. Departing from
traditional, continuous measures of time, intrinsic time proposes an event-
based, algorithmic framework that captures the dynamic and fluctuating nature
of real-world phenomena more accurately. Unsuspected implications arise in
general for complex systems and specifically for financial markets. For
instance, novel structures and regularities are revealed, otherwise obscured
by any analysis utilizing equidistant time intervals. Of particular interest
is the emergence of a multiplicity of scaling laws, a hallmark signature of an
underlying organizational principle in complex systems. Moreover, a central
insight from this novel paradigm is the realization that universal time does
not exist; instead, time is observer-dependent, shaped by the intrinsic
activity unfolding within complex systems. This research opens up new avenues
for economic modeling and forecasting, paving the way for a deeper
understanding of the invisible forces that guide the evolution and emergence
of market dynamics and financial systems. An exciting and rich landscape of
possibilities emerges within the paradigm of intrinsic time.

##  1 Introduction

Two main paradigms exist for decoding the workings of nature. One approach
views its evolution as a smooth and continuous process. This thinking was
instrumental in the inception of classical physics. Following the Pythagorean-
Aristotelian tradition, mathematics was understood as the fundamental
principle underlying the structure and function of the universe (Glattfelder,,
2019, Ch. 2). Specifically, a new brand of mathematics was divined that
mirrored the smooth and continuous behavior thought to be the essential
characteristic of the cosmos’s dynamics (Glattfelder,, 2019, Sec. 5.3.1).

This analytical toolbox unleashed an astounding understanding of natural
processes, conquering ever-new domains of application. To this day, the
mathematical engine driving the most accurate physical theories—for instance,
the standard model of particle physics and general relativity—depends on this
framework. Physics Nobel Laureate Eugene Wigner mused about “the unreasonable
effectiveness of mathematics in the natural sciences” (Wigner,, 1960). To this
day, equations reign supreme in our fundamental understanding of reality,
burrowing ever deeper into the fabric of existence.

In contrast, foundational to the second paradigm is the view that nature’s
evolution is rule-based, progressing in discrete steps. In other words,
reality is computational. The conceptual focus shifts from an analytical
perspective to an algorithmic one. This approach has been very successful in
decoding complexity, otherwise a seemingly insurmoutable challenge in the
equation-based paradigm. Echoing Wigner’s epistemological surprise, the
computer scientist, theoretical physicist, and entrepreneur Stephen Wolfram
commented on a remarkable feature of complex systems. In essence, what appears
as complex behavior from afar emerges from simple rules of local interaction
(Wolfram,, 2002, 2020)—a second “miracle” rendering the cosmos amenable to the
cognitive and information-processing faculties of humans. Thus, complex
systems are best understood by analyzing the structure of the interactions
contained within them, a feat an equation-based approach falls short of.

Central to this paradigm is the multifaceted, intricate, and elusive concept
of information. What was initially understood as a human concept has turned
out to play a fundamental role in the fabric of reality. By design,
information lies at the heart of computation. However, information processing
is not exclusively related to human affairs. The enigmas of life and
consciousness can be reframed in this context, offering new insights. Perhaps
most remarkable, we find the notion of information at the foundations of
reality. Information is a candidate for being an ontological primitive; that
is to say, it could be one of the fundamental building blocks of reality
itself, if not the only one (Glattfelder,, 2019, 2025).

In summary, the human mind has two main methods of comprehending the patterns
appearing in the tapestry of reality. The analytical approach utilizes
equations to uncover the knowledge comprising the edifice of physical science.
Then, an algorithmic understanding of complexity gives us access to the simple
structuring mechanism underlying it (Glattfelder et al.,, 2010).

##  2 The Rise of Intrinsic Time

Within this latter paradigm, we find the concept of intrinsic time. While the
flow of physical time is traditionally understood as a continuous process,
intrinsic time is an example of an event-based conception of time. In other
words, it is algorithmically defined. In general, it should be noted that
(Glattfelder and Golub,, 2022, p. 1):

> The flow of time is a central tenet in the subjective perception of reality.
> Human consciousness is eternally locked in the continuous transition between
> the past and the future, experienced as the moment of “now.” In stark
> contrast to the experiential familiarity of time, its ontological structure
> is obscure. From philosophy (McTaggart,, 1908) to physics (Glattfelder,,
> 2019, Ch. 10), the nature of time has been debated for centuries; sometimes,
> its reality even wholly rejected (Connes and Rovelli,, 1994; Barbour,,
> 2001). The discovery of time’s malleability (Einstein,, 1905) and its
> resistance to being quantized (DeWitt,, 1967) only add to the enigma.

Historically, the understanding of time has always been linked to the
unfolding of physical events: from lunar calendars, sundials, hourglasses, and
quartz clocks to atomic clocks, the passage of time is made visible by
measuring physical changes. In essence, time can be viewed as a property
arising from complex networks of events and their interactions governed by
quantum mechanics and thermodynamics (Rovelli,, 2019).

In economics, the notion of time usually plays a marginal role. Nonetheless,
the practice of using an alternatively defined conception of time has a long
tradition. Specifically (Glattfelder and Golub,, 2022, p. 2):

> The idea of modeling financial time series in a new temporal paradigm goes
> back to (Mandelbrot and Taylor,, 1967) and has been a reoccurring theme
> since (Clark,, 1973; Ané and Geman,, 2000; Easley et al.,, 2012). In
> essence, physical time is substituted with an event-based notion of time.
> This is to say that these novel measures of time are operationally defined
> using certain intrinsic features of the data being analyzed—as an example,
> driven by transaction numbers or trading volumes.

Then, thirty years after the original inception of event-based time, a seminal
paper from a team at Olsen & Associates proposed a simple algorithm focusing
on a key characteristic of financial time series: changes in the direction of
the price evolution (Guillaume et al.,, 1997). The contours of what is today
understood as intrinsic time became visible. Historically, researchers at
Olsen & Associates had coined the term one year earlier (Dacorogna et al.,,
1996, p. 213):

> The only information needed to define the [time] scale are the values of the
> time series themselves. Thus we have chosen to call this time scale
> intrinsic time. The consequence of using such a scale is to expand periods
> of high volatility and concentrate those of low volatility, thus better
> capturing the relative importance of events in the market.

However, although the methodology did allow for a decoupling from physical
time by focusing on the intrinsic behavior of time series, it was defined
analytically. The power of simple rules could only be fully harnessed within
the directional change methodology unveiled in 1997.

The algorithmic formulation of intrinsic time functions similarly to its
analytical counterpart: In periods of low market activity, the clocks tick
slower and, conversely, speed up during phases of high market activity. The
endogenous atoms of intrinsic time are given by the directional changes. In
the simplest terms, they are reversals of price moves measured from local
extrema at different scales (the details are given below). Directional changes
are more sophisticated versions of what are known as drawups and drawdowns
(Pospisil et al.,, 2009). The concept of directional changes has also entered
computational finance, especially in the context of high-frequency data (Aloud
et al.,, 2012; Li et al.,, 2022; Tsang et al.,, 2024).

From a conceptual perspective, the development of intrinsic time reflects the
acceptance that an abstract universal time does not exist and that the concept
of time is always observer-dependent. In other words, the intuitions from
special relativity are applied in a broader context. Generically, intrinsic
time can be understood as a process of how time emerges within a system of
interacting agents (Olsen,, 1983). This idea marks a pronounced metaphysical
shift, moving from the concept of time as an ontological category of the
universe to understanding it as a secondary, derived property dependent on
intrinsic interactions. It is interesting to note that a modern understanding
of quantum gravity supports the idea that time (specifically spacetime) is an
emergent property, perhaps derived from entangled quantum information,
succinctly encapsulated by the mnemonic term “ER=EPR” (Maldacena and
Susskind,, 2013).

In 2011, the concept of intrinsic time was further refined. Next to the event-
based triggers described by the directional changes, a deeper layer of
intrinsic activity was unveiled, called overshoots. Overshoots are robust
statistical properties hidden in time series (see the discussion in the next
section). In general, after every directional change, the price evolution
continues, on average, one overshoot length before the following directional
change disrupts it.

Intrinsic time is algorithmically derived from a selected threshold $\delta$,
describing a price move in percent. The oscillatory nature of directional
changes requires two modes to be specified: up and down. Without loss of
generality, one can begin in an up mode at a price of $x_{0}$ and set the
extremum price to this value, $x^{ext}=x_{0}$. The directional change
algorithm evolves as follows:

  * •

if the price moves up, $x^{ext}$ is updated to this new value;

  * •

if the price moves down, the percentage difference between $x^{ext}$ and the
current price is evaluated.

The inevitable outcome is that there will come a price move for which the
percentage difference between $x^{ext}$ and the current price $x_{t}$ exceeds
$\delta$. A directional change tick is registered, the mode changes to down,
and $x^{ext}=x_{t}$. Now, the algorithm continues correspondingly, with
switched directions (i.e., $x^{ext}$ is updated on lower prices, and $\delta$
measures its difference to higher prices). Also, note that two intrinsic time
clocks will be synchronized after two directional changes even if they were
initialized in different modes. The pseudocode for the algorithm can be found
in (Glattfelder et al.,, 2011).

If the price continues to move by $\delta$ in the same direction after a
directional change, the first overshoot is registered, representing the next
tick of intrinsic time. Now, if the price evolution continues in the same
direction by $\delta$, a second overshoot is detected, and so forth. It is
thus possible to define the coastline of a time-series as the sum of all price
moves of $\delta$ belonging to directional changes or overshoots. Crucially,
intrinsic time is a multi-scale concept, defined by a set of thresholds
$[\delta_{1},\dots,\delta_{n}]$. As a result, for every smaller choice of
$\delta$, a more granular and longer coastline emerges. In essence, every
directional change threshold allows the tick-by-tick price curve to be
transformed into a representation filtered by intrinsic market activity. See
Fig. 1 for a visual representation.

Figure 1: Intrinsic time transformations: The raw time series is decomposed
into directional change (DC) and overshoot (OS) events according to the
procedure defined in the main text utilizing two thresholds for measuring the
intrinsic time events. Specifically, $\bar{\delta}$ and $\hat{\delta}$ define
the directional changes and $\bar{\omega}$ and $\hat{\omega}$ the overshoots,
with $\bar{\delta}=\bar{\omega}$ and $\hat{\delta}=\hat{\omega}$,
respectively. The equality of the variables is derived in Eq. (2). In effect,
intrinsic time defined by different thresholds of resolution yields
transformations from a tick-by-tick representation of time series to event-
based price curves, evolving in intrinsic time increments $\bar{\tau}_{i}$ and
$\hat{\tau}_{j}$, respectively. The dependency on physical time is thus
substituted by a dependency on intrinsic market activity, defined at chosen
levels of granularity, revealing the underlying coastlines.

The rigid application of physical time—i.e., the traditional approach to
modeling financial time series assuming time intervals to be uniformly spaced,
for instance, in seconds or days—is limiting and may obscure essential
characteristics of financial time series. In contrast, the adaptive quality of
an event-based conception of time offers a more flexible method for exploring
the complexity and heterogeneity of market data. Within the new dynamic
paradigm of intrinsic time, novel structures and regularities are uncovered,
otherwise invisible to the inflexible cadence given by equidistant time
intervals.

##  3 The Power of Scaling Laws

Viewed through the lens of the philosophy of science, scaling laws can be
understood as laws of nature applicable to complex systems. Specifically
(Glattfelder,, 2025, p. 86):

> Laws of nature are regularities and structures in a complex universe that
> critically only depend on a small set of conditions and are independent of
> many other factors which could potentially have an effect. Universal scaling
> laws can be understood as such laws relating to nature’s complexity. A
> scaling law is a simple polynomial relationship between a function’s input
> and output. Specifically, a relative change in the input quantity results in
> a proportional change in the output quantity, independent of the initial
> size of the input. Scaling laws are scale-invariant, meaning there is no
> preferred or defining scale, reminiscent of the self-similarity of fractals.
> Historically, the first study of scaling laws and scaling effects can be
> traced back to Galileo, who investigated how different attributes of ships
> and animals scale with size. Over 250 years later, the linguist George
> Kingsley Zipf studied rank- frequency distributions in the 1950s. He
> discovered that in English, the most common words “the,” “of,” and “and” are
> used disproportionally. In contrast, all other words follow a scaling-law
> relation of lesser and lesser importance. In 1964, the economist and
> sociologist Vilfredo Pareto analyzed the distribution of wealth and coined
> the term 80-20 rule. Also known as the Pareto principle, this approximate
> rule says that 20% of the population controls 80% of the wealth. Formally,
> the distribution Pareto identified was a scaling law.

The hallmark of a scaling law is its linear behavior in a log-log plot.
However, stringent statistical tests should be performed to rule out
misidentification (Clauset et al.,, 2009). Scaling-law relations can be
observed in a spectacular diversity of complex systems. Generally, there are
four basic types of scaling laws to be distinguished: (1) allometric scaling
laws, (2) scaling-law distributions, (3) scale-free networks, and (4)
cumulative scaling-law relations (Glattfelder,, 2019, Sec. 6.4.2).

The first category bridges biology, ecology, and physics, describing how
different biological quantities scale in relation to each other as the size of
an organism changes. For example, a relationship exists between a mammal’s
body mass and either the number of heartbeats or the lifespan. The larger the
animal, the slower its heart beats, but the longer it lives. As a remarkable
consequence, a fundamental invariant of life emerges: the number of heartbeats
of any mammal is constant over its lifetime and is approximately $1.5\cdot
10^{9}$ (West,, 1999).

The second category spans a vast array of phenomena, as the dynamics of many
complex systems are not captured by a Gaussian distribution characterized by a
mean scale. In a nutshell, scaling-law distributions reveal how a small
fraction of a population has a dominant influence while the remaining
population is mostly inconsequential. They emerge in such diverse phenomena as
(Newman,, 2005):

  * •

the size of cities, earthquakes, moon craters, solar flares, computer files,
sand particles, wars, and price moves in financial markets;

  * •

the number of scientific papers written, citations received by publications,
hits on web pages, and species in biological taxa;

  * •

the sales of music, books, and other commodities;

  * •

the population of cities;

  * •

the income of people;

  * •

the frequency of words used in human languages and of occurrences of personal
names;

  * •

the areas burnt in forest fires.

The third category describes the organizing principles seen in real-world
complex networks. The turn of the millennium brought about a revolution in the
fundamental understanding of their structure and dynamics. Seminal discoveries
were the small-world (Watts and Strogatz,, 1998) and scale-free (Barabási and
Albert,, 1999) nature of these networks, bringing about a new science of
networks (Dorogovtsev and Mendes,, 2002). In detail, scale-free networks have
a degree distribution following a scaling law. Again, the scaling relation
dictates that a few super-hubs are of high relevance while most nodes in the
network play a subordinate role.

Finally, the last type of scaling-law relation manifests in collections of
random variables known as stochastic processes. Notable examples are financial
time series, where empirical scaling laws govern the relationship between
various measurable quantities. Interest in the scaling relations inherent in
market data was sparked in 1990 by a seminal paper by researchers at Olsen &
Associates relating the mean absolute change of the logarithmic mid-prices,
sampled at different time intervals over a sample of size, to the size of the
time interval (Müller et al.,, 1990). Later, in 1997, the group discovered a
second scaling law relating the number of directional changes $N$ in a sample
to the directional-change threshold $\delta$ (Guillaume et al.,, 1997):

$N(\delta)=a\delta^{b},$ (1)

where $a$ is a proportionality constant and $b$ is the scaling law exponent.
Then, in 2011, 12 independent new empirical scaling laws were discovered in
foreign exchange data, holding for close to three orders of magnitude and
across 13 currency exchange rates (Glattfelder et al.,, 2011). Similarly to
the scaling relation of 1997, this new set of laws emerged within the
directional change framework. One such scaling relation proved to be
remarkably insightful. The size of the average overshoot length $\omega$ after
a directional change is approximately the size of the threshold $\delta$:

$\left<\omega(\delta)\right>\approx\delta,$ (2)

where the angle brackets denote the sample average. Thus, the motivation arose
for defining an overshoot increment to be the size of the directional change
threshold, $\omega=\delta$, expanding the scope of the intrinsic time
framework. It should be noted that Eq.(2) only holds for the average behavior.
The individual instances of actually measured overshoot lengths reveal that
the average value is never actually manifested. There is a large variability
in the distribution where most occurring overshoots are small, with some
disproportionately large ones conspiring to yield the average behavior seen in
Eq.(2).

Herein lies the power of scaling laws. They uncover an underlying organizing
principle coordinating over many orders of magnitude. It is surprising to find
such regularity in the structure and evolution of complex systems. Once
detected, these patterns make them amenable to systematic analysis and
predictive modeling. This allows researchers to discern novel behaviors and
relationships that provide insights into the fundamental dynamics of complex
systems across various scales and multiple domains.

##  4 Decoding Complexity

The insights from the intrinsic time framework inspired further research. For
instance, the concept of multi-scale liquidity was introduced (Golub et al.,,
2016), systematic trading strategies devised (Golub et al.,, 2018), a
variation of the notion of volatility defined (Petrov et al., 2019a, ), and an
agent-based framework formulated (Petrov et al.,, 2020). The notion of
intrinsic time can be extended to a multi-dimensional methodology,
incorporating more than one financial time series (Petrov et al., 2019b, ).
Other research groups have also adopted the directional-change framework
(Tsang et al.,, 2017; Tsang,, 2022; Tsang et al.,, 2024).

Recent work uncovered an additional scaling law relating to the behavior of
overshoots. A central result was a recipe for the decomposition of time series
into their liquidity and volatility components, both only visible through the
lens of intrinsic time (Glattfelder and Golub,, 2022):

$\langle r(\Delta
t)\rangle_{2}\propto\langle\omega(\delta)-\delta\rangle_{2}N(\delta),$ (3)

where $r(\Delta t)$ denotes the price changes or returns sampled in physical
time, and $\langle x\rangle_{2}=\frac{1}{n}\sum_{i=1}^{n}x_{i}^{2}$ is the
sample average of the squared values. Note that
$\langle\omega(\delta)-\delta\rangle_{2}$ represents the variability of
overshoots (and scales with $\delta^{2}$). Then, by observing that the
overshoots function as a proxy for liquidity (Golub et al.,, 2016) and the
number of directional changes measures volatility (Petrov et al., 2019a, ),
financial time series can be non-trivially decomposed into these two defining
characteristics, yielding novel understanding.

Taking a step back, the utility of intrinsic time can also be assessed in more
general terms. Humanity’s technological prowess is truly breathtaking,
witnessed by unprecedented leaps in progression, overshadowing former
advancements thought to be unsurpassable. Fuelling this cognitive revolution
is the knowledge generated in the analytical paradigm, epitomized by physical
science. In contrast, our understanding of complexity is still in its infancy.
Although complexity science emerged as a fusion of various intellectual
traditions—including cybernetics, systems theory, early artificial
intelligence, cellular automata, non-equilibrium thermodynamics, agent-based
modeling, non-linear dynamics, fractal geometry, chaos theory, network
science, metacybernetics, and complex systems theory—we are still waiting for
deeper insights into emergent phenomena like the enigmas of life and
consciousness (Glattfelder,, 2025).

Perhaps the most glaring failure regarding our understanding of complexity
relates to finance and economics. It is bitterly ironic that the very systems
humans collectively engineered, representing the backbone of global stability
and progress, are also among the least understood in terms of their intricate
workings, long-term sustainability, and susceptibility to systemic risk
(Glattfelder,, 2019, Ch. 7). It is within this domain that we have an
excellent opportunity to apply some general insights gained from intrinsic
time. Specifically, the dissemination of information triggering the responses
of economic agents, mythologized by Adam Smith’s “invisible hand” (Smith,,
1776), appears in a new light. By accounting for the observer-dependent
reference frames defined by an event-based framework, it is naive to assume
the efficient collective allocation of resources in response to signals, even
by discounting for human irrationality (Kahneman,, 2003). Accepting the high
degree of idiosyncrasy, variability, and heterogeneity in the individual
perception and processing of information—exemplified by the ticking of local
intrinsic clocks—collective behaviors like herding and cascading effects, so
detrimental to stability and sustainability, can be reevaluated in a framework
more attuned to the nuances of complex systems of interacting agents.

The exploration of the intrinsic time framework, galvanized by the emergence
of scaling laws, represents a paradigm shift in the overall understanding of
financial markets and complex systems. It underscores a profound departure
from static, linear models of analysis towards dynamic, event-driven
frameworks that more accurately reflect the complexities of real-world
phenomena. This shift fundamentally challenges the traditional perceptions of
time and uncovers its event-based emergence in the context of an interaction-
based, algorithmic worldview. A rich landscape for innovation in both
theoretical and practical terms is discovered. Essentially, a more subtle
grasp of nature’s underlying dynamics becomes possible within this paradigm.
Thus, intrinsic time seems like an indispensable item in an intellectual
toolbox designed to decode the complexity arising from humanity’s expanding
quest for knowledge.

## References

  * Aloud et al., (2012) Aloud, M., Tsang, E., Olsen, R., and Dupuis, A. (2012).  A directional-change event approach for studying financial time series.  Economics, 6(1):20120036.
  * Ané and Geman, (2000) Ané, T. and Geman, H. (2000).  Order flow, transaction clock, and normality of asset returns.  The Journal of Finance, 55(5):2259–2284.
  * Barabási and Albert, (1999) Barabási, A.-L. and Albert, R. (1999).  Emergence of scaling in random networks.  Science, 286(5439):509–512.
  * Barbour, (2001) Barbour, J. (2001).  The end of time: The next revolution in physics.  Oxford University Press.
  * Clark, (1973) Clark, P. K. (1973).  A subordinated stochastic process model with finite variance for speculative prices.  Econometrica: journal of the Econometric Society, pages 135–155.
  * Clauset et al., (2009) Clauset, A., Shalizi, C. R., and Newman, M. E. (2009).  Power-law distributions in empirical data.  SIAM review, 51(4):661–703.
  * Connes and Rovelli, (1994) Connes, A. and Rovelli, C. (1994).  Von Neumann algebra automorphisms and time-thermodynamics relation in generally covariant quantum theories.  Classical and Quantum Gravity, 11(12):2899.
  * Dacorogna et al., (1996) Dacorogna, M. M., Gauvreau, C. L., Müller, U. A., Olsen, R. B., and Pictet, O. V. (1996).  Changing time scale for short-term forecasting in financial markets.  Journal of Forecasting, 15(3):203–227.
  * DeWitt, (1967) DeWitt, B. S. (1967).  Quantum theory of gravity. I. The canonical theory.  Physical Review, 160(5):1113.
  * Dorogovtsev and Mendes, (2002) Dorogovtsev, S. N. and Mendes, J. F. (2002).  Evolution of networks.  Advances in physics, 51(4):1079–1187.
  * Easley et al., (2012) Easley, D., de Prado, M. M. L., and O’Hara, M. (2012).  The volume clock: Insights into the high-frequency paradigm.  The Journal of Portfolio Management, 39(1):19–29.
  * Einstein, (1905) Einstein, A. (1905).  Zur Elektrodynamik bewegter Körper.  Annalen der Physik, 4.
  * Glattfelder, (2019) Glattfelder, J. B. (2019).  Information—Consciousness—–Reality: How a new understanding of the universe can help answer age-old questions of existence.  Springer, Cham.
  * Glattfelder, (2025) Glattfelder, J. B. (2025).  The Sapient Cosmos: What a modern-day synthesis of science and philosophy teaches usaAbout the emergence of information, consciousness, and meaning.  Essentia Books, London.
  * Glattfelder et al., (2010) Glattfelder, J. B., Bisig, T., and Olsen, R. B. (2010).  R&D strategy document.  arXiv preprint arXiv:1405.6027.
  * Glattfelder et al., (2011) Glattfelder, J. B., Dupuis, A., and Olsen, R. B. (2011).  Patterns in high-frequency FX data: Discovery of 12 empirical scaling laws.  Quantitative Finance, 11(4):599–614.
  * Glattfelder and Golub, (2022) Glattfelder, J. B. and Golub, A. (2022).  Bridging the gap: Decoding the intrinsic nature of time in market data.  arXiv preprint arXiv:2204.02682.
  * Golub et al., (2016) Golub, A., Chliamovitch, G., Dupuis, A., and Chopard, B. (2016).  Multi-scale representation of high frequency market liquidity.  Algorithmic Finance, 5(1-2):3–19.
  * Golub et al., (2018) Golub, A., Glattfelder, J. B., and Olsen, R. B. (2018).  The Alpha Engine: Designing an automated trading algorithm.  In Dempster, M. A. H., Kanniainen, J., Keane, J., and Vynckier, E., editors, High-Performance Computing in Finance: Problems, Methods, and Solutions. Chapman & Hall, London.
  * Guillaume et al., (1997) Guillaume, D. M., Dacorogna, M. M., Davé, R. R., Müller, U. A., Olsen, R. B., and Pictet, O. V. (1997).  From the bird’s eye to the microscope: A survey of new stylized facts of the intra-daily foreign exchange markets.  Finance and Stochastics, 1(2):95–129.
  * Kahneman, (2003) Kahneman, D. (2003).  Maps of bounded rationality: Psychology for behavioral economics.  American economic review, 93(5):1449–1475.
  * Li et al., (2022) Li, S., Tsang, E. P., and O’Hara, J. (2022).  Measuring relative volatility in high-frequency data under the directional change approach.  Intelligent Systems in Accounting, Finance and Management, 29(2):86–102.
  * Maldacena and Susskind, (2013) Maldacena, J. and Susskind, L. (2013).  Cool horizons for entangled black holes.  Fortschritte der Physik, 61(9):781–811.
  * Mandelbrot and Taylor, (1967) Mandelbrot, B. and Taylor, H. M. (1967).  On the distribution of stock price differences.  Operations Research, 15(6):1057–1062.
  * McTaggart, (1908) McTaggart, J. E. (1908).  The unreality of time.  Mind, pages 457–474.
  * Müller et al., (1990) Müller, U. A., Dacorogna, M. M., Olsen, R. B., Pictet, O. V., Schwarz, M., and Morgenegg, C. (1990).  Statistical study of foreign exchange rates, empirical evidence of a price change scaling law, and intraday analysis.  Journal of Banking & Finance, 14(6):1189–1208.
  * Newman, (2005) Newman, M. E. (2005).  Power laws, pareto distributions and zipf’s law.  Contemporary physics, 46(5):323–351.
  * Olsen, (1983) Olsen, Richard, B. (1983).  Interaction between law and society: An interpretation of law in action and the function of property rights.  PhD thesis, University of Zurich.
  * (29) Petrov, V., Golub, A., and Olsen, R. (2019a).  Instantaneous volatility seasonality of high-frequency markets in directional-change intrinsic time.  Journal of Risk and Financial Management, 12(2):54.
  * Petrov et al., (2020) Petrov, V., Golub, A., and Olsen, R. (2020).  Agent-based modelling in directional-change intrinsic time.  Quantitative Finance, 20(3):463–482.
  * (31) Petrov, V., Golub, A., and Olsen, R. B. (2019b).  Intrinsic time directional-change methodology in higher dimensions.  Available at SSRN 3440628.
  * Pospisil et al., (2009) Pospisil, L., Vecer, J., and Hadjiliadis, O. (2009).  Formulas for stopped diffusion processes with stopping times based on drawdowns and drawups.  Stochastic Processes and their Applications, 119(8):2563–2578.
  * Rovelli, (2019) Rovelli, C. (2019).  The order of time.  Penguin.
  * Smith, (1776) Smith, A. (1776).  An inquiry into the nature and causes of the wealth of nations.  Cadell.
  * Tsang, (2022) Tsang, E. P. (2022).  Directional change for handling tick-to-tick data.  Journal of Chinese Economic and Business Studies, 20(2):171–182.
  * Tsang et al., (2024) Tsang, E. P., Ma, S., and Chinthalapati, V. R. (2024).  Nowcasting directional change in high frequency FX markets.  Intelligent Systems in Accounting, Finance and Management, 31(1):e1552.
  * Tsang et al., (2017) Tsang, E. P., Tao, R., Serguieva, A., and Ma, S. (2017).  Profiling high-frequency equity price movements in directional changes.  Quantitative Finance, 17(2).
  * Watts and Strogatz, (1998) Watts, D. J. and Strogatz, S. H. (1998).  Collective dynamics of “small-world” networks.  nature, 393(6684):440–442.
  * West, (1999) West, G. B. (1999).  The origin of universal scaling laws in biology.  Physica A: Statistical Mechanics and its Applications, 263(1-4):104–113.
  * Wigner, (1960) Wigner, E. (1960).  The unreasonable effectiveness of mathematics in the natural Sciences.  Communications in Pure And Applied Mathematics, 13(1):222–237.
  * Wolfram, (2002) Wolfram, S. (2002).  A new kind of science.  Wolfram Media Inc, Champaign.
  * Wolfram, (2020) Wolfram, S. (2020).  A project to find the fundamental theory of physics.  Wolfram Media Inc, Champaign.


# Uncertainty-aware Self-supervised Target-mass Grasping
of Granular Foods

Kuniyuki Takahashi†, Wilson Ko†, Avinash Ummadisingu†, Shin-ichi Maeda† † K.
Takahashi, W, Ko, A. Ummadisingu, and S. Maeda are associated with Preferred
Networks, Inc<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Food packing industry workers typically pick a target amount of food by hand
from a food tray and place them in containers. Since menus are diverse and
change frequently, robots must adapt and learn to handle new foods in a short
time-span. Learning to grasp a specific amount of granular food requires a
large training dataset, which is challenging to collect reasonably quickly. In
this study, we propose ways to reduce the necessary amount of training data by
augmenting a deep neural network with models that estimate its uncertainty
through self-supervised learning. To further reduce human effort, we devise a
data collection system that automatically generates labels. We build on the
idea that we can grasp sufficiently well if there is at least one low-
uncertainty (high-confidence) grasp point among the various grasp point
candidates. We evaluate the methods we propose in this work on a variety of
granular foods- coffee beans, rice, oatmeal and peanuts, each of which has a
different size, shape and material properties such as volumetric mass density
or friction. For these foods, we show significantly improved grasp accuracy of
user-specified target masses using smaller datasets by incorporating
uncertainty.111An accompanying video is available at the following link:
https://youtu.be/5pLkg7SpmiE

##  I Introduction

Within the food packing industry, the ability to manipulate multiple, highly
variable food and place them in their assigned positions is essential. This
requirement is exemplified by the Japanese lunch boxes known as “Bentos”.
Bentos often combine numerous types of food and change in contents every few
weeks to make use of seasonal ingredients. Not only is there immense inter-
category variation in the types of food that need to be manipulated, but there
is also significant intra-category irregularity in shape, size. The
continually changing menus and variation make automation of the process a
significant challenge and has so far only been applied to a small portion of
mostly fixed foodstuffs. In times of labor shortages, it has become crucial to
develop intelligent systems that can learn to adapt to this industry’s
variety, pace and scale.

Figure 1: Concept: The robot estimates grasp masses at several grasp points.
The uncertainty of the prediction is evaluated at the grasp points which are
close to the target mass. Then, the grasp point sufficiently close to the
target mass with the lowest uncertainty (highest confidence) is selected.

The amount of food to be placed in a bento box is specified either by the
number of pieces for _solid food_ (e.g. fried chicken and fish) or by mass for
_granular food_ (e.g. beans and shredded vegetables). Prior work in robotics
has been successfully demonstrated for the manipulation of _solid foods_ in
the food industry [1, 2]. However, despite a substantial number of used
_granular foods_ , their manipulation remains relatively unexplored and they
are difficult to deal with due to their particulate, deformable nature. So
far, specialized tools or measured scoops are often deployed to measure the
required quantity of food, but this often requires significant engineering
effort to adapt to different foods and amounts.

In our work, we focus on learning to pick _granular food_ by identifying the
best pick points to grasp a user-specific target mass from an RGB-D image of
the entire food tray (See Fig. 1). We use feed-forward planning based on
vision in order to avoid the cost of additional sensors, such as wrist-based
load cells. In designing this system, there are three requirements from the
food packing industry. R1) The ability to handle a variety of deformable and
irregularly shaped food. R2) The ability to grasp any user-defined target mass
of food without the need for additional training. R3) The ability to adapt to
new food in a short period of time. We propose a grasping method that
satisfies these three requirements.

Since granular foods are deformable and the individual particles heavily
interact with each other, it is difficult to simulate accurately or model. A
learning-based approach from real-world interaction was hence the most
feasible choice for our work. Our proposed system leverages the
representational power of deep learning to learn from real-world RGB-D images
directly (R1). The aim of the neural network is to predict the mass of grasped
food for several candidates within the tray. The most likely grasp to result
in the user-requested mass is chosen and executed (R2). Although deep learning
is powerful, it usually needs a large training dataset to be optimized
correctly. Unfortunately, data-collection on the real robot takes an enormous
amount of time and working with real food is challenging due to spoilage,
damage due to handling, moisture, etc, and thus limits the amount of data we
can collect and learn from. This limitation inevitably makes the estimate of
the trained deep neural networks unreliable. To cope with the incompleteness,
we model the uncertainty of the estimate with self-supervised learning and
utilize the most reliable estimate (R3).

##  II Related Work

###  II-A Granular Food Manipulation

For granular food manipulation, prior work is typically limited to a specific
food or task, e.g. in [3], the authors created a candy-scooping robot. In [4],
the robot learned a scooping motion of pinto beans from one tray to another to
arrange them into the desired shape. A study by [5] is highly relevant to our
research. The system proposed in this work scoops a user-specified target mass
of food using a box-shaped end-effector, given the depth information of the
food tray, where it tries to pick up food with a single grasp position with
fixed orientation. In scooping, the insertion position, angle and depth of the
gripper are required at minimum (such as in [5]) in order to control the
trajectory. On the other hand, gripping only requires the position and
insertion depth of the gripper. Therefore, grasping requires fewer parameters
to be executed. We chose grasping with a gripper over scooping to simplify the
picking process so that we can focus on reducing the required training data.
Additionally, we improve inference performance and further reduce training
data by taking uncertainty into account. Note that our methods of uncertainty
estimation can also be applied for scooping.

###  II-B Estimation of Uncertainties

Literature on uncertainty makes a distinction between two sources of
uncertainty- _epistemic_ and _aleatoric_ [6]. Epistemic (or systematic)
uncertainty stems from the lack of knowledge or from not knowing or accounting
for results outside of what is encountered so far. Epistemic uncertainty is
considered reducible, as the acquisition of more data or knowledge of the data
generating process would reduce it. Aleatoric (or statistical) uncertainty
represents the inherent randomness or variability in the outcome of an
experiment. It would typically cover uncertainty in sensing, the robot
dynamics, and the future state of the environment after interaction.

A number of recent works aim at introducing uncertainty into neural networks.
From the Bayesian perspective, aleatoric uncertainty may be modelled by
placing a distribution over the output of the network [7] and epistemic
uncertainty is modelled by placing a prior distribution over a model’s
weights. The latter’s formulation is often referred to as Bayesian Neural
Networks to [8, 9, 10, 11, 12, 13] and their related extensions [14, 12, 7,
15]. However, their adoption, as well as methods that employ ensembles [16]
that capture both kinds of uncertainty, are computationally expensive.

As we operate in the small-data regime, we adopt methods from ML literature to
explicitly capture uncertainty and use it in a two-stage grasp selection
criterion that chooses the least uncertain grasp that leads to our target
mass.

###  II-C Grasping

Grasp synthesis, or the problem of choosing a grasp configuration among an
infinite set of candidates has been addressed frequently in the robotics
community [17, 18, 19]. Prior work is broadly classified into _analytical
methods_ that explicitly model and/or deal with one or more sources of
(aleatoric) uncertainty [20, 21, 22, 23] or _data-driven methods_ [24, 18]
that grew with the advent of Graspit! [25] which allow for a less strict
parametrization of the grasp and therefore accommodate better for
uncertainties in perception and execution [24, 26, 27, 28, 29]. Applications
of neural networks to grasp synthesis tend to use adapted versions of popular
CNN architectures to output the grasp quality metric for either a single
grasp, a collection of grasp candidates, or directly output the best grasp
candidate. They are typically trained on synthetic data [30, 31, 32], using
datasets like the Cornell object dataset [33] in [33, 34, 35, 36] or learn
from sizable collections of real-world grasp attempts [37, 38].

The methods described above see limited applicability in our setting for the
following reasons. 1\. The use of synthetic data/simulators is limited by the
variety and the difficulty in accurately and efficiently simulating granular,
deformable foodstuffs at scale. 2\. Large-scale data collection is difficult
due to concerns of spoilage, damage and hygiene so the training procedure is
required to be short. 3\. Existing methods focus on grasping a _single solid
target object_ with fixed shape. Focus on grasping a single object allows
their problem to be cast as a binary success/failure task. If we were to model
our task as a binary classification task, it would be necessary to retrain the
network to capture success/failure for grasping every required target weight
and would violate one of the key requirements we put forth in Section I.

##  III Contributions

In addressing the requirements mentioned in Section I, we believe that we have
developed a novel system that can reliably grasp a user-specified amount of
granular food with a focus on minimizing the data to be collected. The
contributions of this work are as follows: 1) Proposal of a learning-based
system for grasping a user-specified mass of various granular food with
automatic data labeling. 2) A self-supervised method for modeling its
predictive uncertainties. 3) A system that chooses grasps by leveraging model
uncertainty to reduce the required amount of training data and to improve
accuracy.

Figure 2: The network models for uncertainties evaluators

##  IV Method

Our proposed method to grasp a user specified target mass of granular food
consists of the following steps (see also Fig. 1): 1) Grasp mass predictions
are obtained for RGB-D patches across the food tray. 2) The uncertainty in the
predicted grasp mass is obtained. 3) Select the point with the lowest
uncertainty for the estimated grasp mass among the points whose estimated
grasp mass are within $\pm$0.5 g from the target mass.

###  IV-A Grasp Mass Estimation from RGB-D

Our grasp mass estimation network predicts the mass of food to be grasped for
cropped patches of the RGB-D image. Therefore, if we command the gripper to
grasp at a certain patch’s location, it is expected that the gripper will
grasp the mass that was predicted for that patch.

When the gripper inserts itself into the food during grasping, we find that
disturbance is typically localized to the region around the grasp point.
Therefore, we find it sufficient to crop these regions centered around the
grasp point. Additionally, we process the cropped depth image such that the
center of the patch has a value of 0, and the heights of the other points
within the patch are relative to it. This step aims to reduce the diversity of
the input data.

The expected grasp mass for a patch $i$ is estimated by the neural network
$F(x_{img,i},x_{depth,i};\xi)=y^{\prime}_{mass,i}$ where the input $x_{img,i}$
and $x_{depth,i}$ are the $i$-th cropped RGB image and depth image,
respectively. $\xi$ is a set of parameters to be optimized. The loss function
$L$ is minimized w.r.t $\xi$:

$\vspace{-2mm}\min_{\xi}\frac{1}{n}\sum_{i=1}^{n}L(y_{mass,i},F(x_{img,i},x_{depth,i};\xi))$
(1)

where $n$ is the number of cropped images used for the training, and
$y_{mass,i}{\in}{R}^{d}$ is the teaching signal for the $i$-th image, i.e.,
the actual grasp amount.

Possible choices of the loss function $L$, include ones used for regression or
classification tasks. Although the nature of the task is regression because we
need to estimate a grasp amount expressed as a continuous value, a
classification loss may be used and works well for certain cases (see related
studies in Section II-C). This is probably due to the high degree of freedom
of multinomial distributions which is optimized by solving the classification
task. In compensation for the high degree of freedom, however, we usually need
a large training dataset. Since we want to work with small training datasets
and the output of the network should represent the entire range of possible
grasp amounts with a resolution as fine as the used scale (see Section VI-A),
we chose to optimize a regression loss, MSE in equation (1).

###  IV-B Estimation of Uncertainties

In this section, we propose several methods to explicitly capture the
uncertainty of the model. Uncertainty in this context is the likelihood that
the actual grasp mass will be different from the predicted mass for the RGB-D
input described in Section IV-A. Predictions with low uncertainty indicate the
model’s high confidence that a real grasp’s outcome will match its prediction.

As we describe in Section II-B, the uncertainty we estimate here includes both
_aleatoric_ and _epistemic_ uncertainty. Aleatoric uncertainty is the
uncertainty intrinsic to the system such as sensor noise, motor noise, or
unpredictable stochastic phenomena. On the other hand, epistemic uncertainty
is uncertainty due to the incompleteness of the estimation method. In our
setting, the shortage of the training data is the primary source of
-_epistemic_ \- uncertainty.

The uncertainty estimation methods we present below work in tandem with the
mass estimation model described in the previous section and can be either
stand-alone models or combined with the mass estimation model (see Fig. 2).

####  IV-B1 Uncertainties by Error Estimation

Here, we describe a method inspired by literature on improved exploration in
Reinforcement Learning [39], which we adapt to estimate the uncertainty of the
model. The original method proposes the use of absolute temporal difference
errors to identify states in which effects of actions are harder to predict,
and use that to encourage/discourage exploration. This metric is reportedly
more robust to noise than the standard deviation.

Given the $i$-th input image, the error estimation (_EE_) model learns to
predict the absolute difference, $y^{\prime}_{error,i}$, between the actual
grasp amount $y_{mass,i}$ and the estimated grasped amount
$y^{\prime}_{mass,i}$ which is estimated by the mass estimation model (see
Fig. 2 (1)). The EE model is obtained as the one that minimizes the following
loss function:

$\vspace{-2mm}\min_{\xi}\frac{1}{n}\sum_{i=1}^{n}L(|y_{mass,i}-y^{\prime}_{mass,i}|,y^{\prime}_{error,i})$
(2)

Note here that we do not need any external supervised signal other than
$y_{mass,i}$ which we assume available for the training of the mass estimation
model. Thus the training of the EE model is a kind of self-supervised
learning. Since the mass estimation model is trained such that the error is
small for the training data, the closer the input data is to the training
data, the closer the output of the EE model is to zero. Furthermore, for the
error estimation of the training data, EE model output will reflect the
difficulty of the mass estimation because the easier the mass estimation is,
the closer the estimated grasp mass will be to the actual grasp amount. We
interpret the estimated error as a proxy measure of model uncertainty.

####  IV-B2 Uncertainty estimation inspired by RND

Here we propose to use the idea of Random Network Distillation (_RND_) [40] to
estimate the uncertainty of the estimated grasp mass (See Fig. 2 (2)). RND was
originally proposed as a technique to explore novel state-action spaces for
better exploration in reinforcement learning. When exploring, it is necessary
to determine whether something has already been experienced, or has not yet
been experienced, and RND helps determine this. In RND, two networks
$(f,f^{\prime})$ are used. One is the _target network_ $f$ which maps an
observation to an embedding $f:O\mapsto{R}^{k}$, and the other is the
_predictor network_ $f^{\prime}$ for $f^{\prime}:O\mapsto{R}^{k}$. The target
network produces teaching signals for the training of the predictor network.
In practice, it is represented by a random neural network whose weights are
randomly initialized and kept fixed. Then, the predictor network is trained by
gradient descent to minimize the average
MSE$\frac{1}{n}\sum_{i=1}^{n}||f^{\prime}(x_{img,i},x_{depth,i};\theta)-f(x_{img,i},x_{depth,i})||^{2}$
with respect to its parameter $\theta$ in our case. Note here that
$\\{x_{img,i},x_{depth,i}\\}$ are exactly the same images used for the
training of the mass estimation model, but the teaching signal is given by the
target network. Therefore, when $f$ and $f^{\prime}$ are given inputs outside
of the training set, this difference is expected to be large. We adopt RND’s
property to model uncertainty, i.e., the closer the input is to the training
data, the lower the uncertainty is for the output of the mass estimation
model. This method requires two networks $f$ and $f^{\prime}$ in addition to
the mass estimation model but doesn’t affect its learning.

###  IV-C Grasp Point Selection

In this section, we describe how we select the grasp point from the food tray.
We start with the case of a model using ordinary mass estimation without
uncertainty. Patches centered around each candidate grasp point are cropped
from the RGB-D data and fed into the mass estimation model. The patch with the
predicted mass closest to the target grasp mass is selected for execution.
This method works well if there is enough data to train the mass estimation
model, as the difference between the estimated and actual grasped masses will
be small. If the training data is not sufficient, the difference between the
actual grasp and the estimate is sometimes large because the grasp estimation
model is not sufficiently accurate. As our work deals with granular foods, a
large amount of data is needed to capture the immense variety of their
possible configurations. This, coupled with the intrinsic randomness in the
grasp process makes it unlikely to ever obtain a perfect model. Therefore, we
propose to augment the method with uncertainty models as described in the
previous section.

We use a two-stage grasp selection criterion where the method first filters
the points for which the estimated masses are within an acceptable region,
which is specified to be within $\pm$0.5 g from the target mass, which
corresponds to the resolution of the scale we used for creating the teaching
signal. Then, we select the ones with the lowest uncertainty amongst them to
maximize the probability to grasp the food of the estimated grasp amount. We
expect that this two stage process with the inclusion of uncertainty in the
selection process achieves significant improvements over the ones that didn’t,
particularly when the size of the training samples is small. Using only the
mass estimation model to pick a grasp point is equivalent to choosing amongst
the qualifying candidate uncertainties uniformly at random. Choosing the
candidate with least uncertainty is expected to be as good as, or better than
uniformly sampling them.

##  V Experiment Setup for Food Picking

The goal of the experiments is to verify and compare several models that can
grasp a target amount of granular food, which make use of uncertainties to
improve performance.

Figure 3: Experiment setup: (a) robot setup used for the experiment, (b)
target object images; coffee beans with 1 g and 22 g (left), and rice with 1 g
and 60 g (right), (c) examples of RGB and depth images of coffee beans (left)
and rice (right), (d) oatmeal (top) and peanuts (bottom)

###  V-A Robot Setup for Data Collection System

Our robotic system, shown in Figure 3 (a), consists of a Sawyer 7-DOF robotic
arm equipped with a self-designed two-fingered parallel gripper driven by a
servo motor (Dynamixel XM430-W350-R). A sponge is attached to the edge of the
end-effector since a gap occurs when the food is pinched, and this sponge
prevents the food from spilling out (See black lining around the end-effector
in Fig. 3 (a)). Furthermore, we use an Ensenso N35 stereo camera in
combination with an IDS uEye RGB camera to overlook the workspace of the robot
arm and use them to retrieve registered point clouds of the scene. We use two
food trays and the robot continuously picks from the one to the other and vice
versa until a user-specified number of iterations. The inner dimensions of the
food tray are $603mm\times 377mm\times 145mm$ which is a commonly used tray
size in the food industry. Two EW-12Ki scales are placed under each food tray,
and the amount grasped by the robot is calculated from the masses of the trays
after the robot picks from the one and after it places in the other. The
Sawyer, gripper, RGB-D sensor, and scales are connected to a PC running Ubuntu
16.04 with ROS Kinetic. We target coffee beans and uncooked rice which have
different particle sizes, surface friction and densities (See Fig. 3 (b)).
Additionally we show that our methods work well for two other foods, namely
oatmeal and peanuts (See Fig. 3 (d)) We primarily focus on the grasping of
generic dry, cohesion-less granular foodstuffs. Non-dry viscous foods or ones
that are prone to tangling are outside the scope of our work and would likely
require hardware modifications and additional manipulation such as pre-
grasping as seen in [41]. We find our system works reasonably well on most
granular materials that satisfy the above criterion and have enough mass to
measure well on our scale which has a resolution of 1 g.

###  V-B Data Collection Process

We built a data collection system to collect data indefinitely, without the
need for human supervision or labeling effort. Algorithm 1 details the data
collection process.

We choose the home position such that the robot is out of view of the RGB-D
camera. To decide which tray to pick from and which to place in, we weigh the
trays and choose the heaviest one to pick from first. Once the mass of the
pick tray becomes lower than a certain threshold, the robot switches
direction, so the current pick tray will become the place tray and vice versa.
After deciding the pick/place trays, the system captures an RGB-D image and
fills in all NaN values, as seen in lines 6-7 of Algorithm 1.

In line 8 of Algorithm 1, we process the RGB-D images in the following ways.
From the entire RGB-D image, we obtain cropped patches of 150x150 pixels with
the grasp point as the center of the patch. 1 pixel corresponds to roughly 1
mm of the tray. The depth information is used to compute the relative heights
from the food surface’s height at the grasping point w.r.t. the height at the
center of the cropped patch. Note that this is an arbitrary choice to
normalize input height. Then it was converted to RGB by OpenCV’s pseudo-
coloring API, applyColorMap() [42].

Note that in line 10, the pick and place poses are randomly selected within
the tray areas, which are located through Aruco markers at line 9 [43, 44].
Only the $x$ and $y$ coordinates are randomized though, while the pick height
is always set to be 2 cm below the surface of the food. By fixing the grasp
height, we ensure a more consistent and diverse dataset due to the reduction
in exploration of the height parameter, and the constantly decreasing height
of the food as the bin empties, leading to a variety of grasped volumes and
surface conditions. Though outside the scope of this work, when volume of food
becomes too small to grasp well, simple motions can be executed to gather the
remaining food into piles.

This picking system uses two food trays, and food grasped from one tray is
placed in the other. Therefore, the total amount of food in the tray being
picked from continues to decrease as the other increases. The surface of the
food in both trays continue to evolve significantly and constantly (See Fig. 3
(c)). This allows us to capture diverse surface configurations that may arise
in a real factory setting.

The whole pick-and-place process, including data storage, takes about 12.5 sec
per cycle (See attached video). During inference, it takes an additional 1.5
sec, leading to a total of 14.0 sec per cycle. For the training data, we
collected 1300 data points for rice and coffee, respectively. Out of 1300
samples collected, 1000 were used for training, and 300 were used to evaluate
the network model. During training of the network(s), we improve
generalization by exploiting the gripper symmetry and apply vertical and
horizontal flipping of input RGB-D independently with a probability of 0.5.

Algorithm 1 Data Collection Process

1:for $\rm{\textsc{iter}}=1$ to $N$ do

2: $q_{angles}\leftarrow\textsc{home}$ $\triangleright$ Move out of view.

3: $tray_{pick}\leftarrow\textsc{SelectTray()}$ $\triangleright$ Select the
pick tray.

4: if $mass_{tray}<\epsilon$ then SwitchDirection()

5: $RGBD\leftarrow\textsc{CaptureImage\&Depth()}$

6: while Depth.ContainsNaN() do

7: RunningAverage()

8: process(RGBD)

9: $P_{tray_{pick}},P_{tray_{place}}\leftarrow\textsc{FindTrayLocations()}$

10: $X,Y\leftarrow\textsc{GenerateRandomXYCoordinates()}$

11: Pick(X, Y, Z = 0.02$m$) $\triangleright$ Pick at fixed height from the
food surface

12: Place() $\triangleright$ Drop in $tray_{place}$

13: MeasureScales() $\triangleright$ Measure picked amount

###  V-C Deep Learning & Training & Inference

We trained the networks on a machine equipped with 256 GB RAM, an Intel Xeon
E5-2667v4 CPU, and Tesla P100-PCIE with 12GB. Training for each model
(initialized randomly for each food) varied from 10 to 70 minutes for
difference dataset size. Our experiments for inference with trained models
were performed on a machine equipped with 31.3 GB RAM, an Intel Core i7-7700K
CPU, and GeForce GTX 1080 Ti. Inference time was about 1.5 sec. 222Details of
architectures for mass estimation and uncertainty prediction are shown in the
video, but note that other architectures can also be used.

Figure 4: Histogram of (a) coffee beans and (b) rice for 50, 100, 200, 500, and 1000 times grasps randomly for training data. TABLE I: Success rates for coffee beans and rice. Numbers in bold represent better success rates than _Baseline_. Dataset sizes for training | Model | Coffee beans | Rice
---|---|---|---
17 g | 22 g | 27 g | 45 g | 60 g | 75 g
$5\%$ $(1g)$ |  $10\%$ $(2g)$ |  $5\%$ $(2g)$ |  $10\%$ $(3g)$ |  $5\%$ $(2g)$ |  $10\%$ $(3g)$ |  $5\%$ $(3g)$ |  $10\%$ $(5g)$ |  $5\%$ $(3g)$ |  $10\%$ $(6g)$ |  $5\%$ $(4g)$ |  $10\%$ $(8g)$
50 | Random | 0.22 | 0.32 | 0.22 | 0.38 | 0.32 | 0.36 | 0.16 | 0.24 | 0.24 | 0.40 | 0.16 | 0.28
Baseline | 0.34 | 0.52 | 0.56 | 0.78 | 0.42 | 0.52 | 0.24 | 0.32 | 0.32 | 0.58 | 0.34 | 0.67
EE | 0.52 | 0.82 | 0.76 | 0.94 | 0.74 | 0.82 | 0.32 | 0.49 | 0.54 | 0.79 | 0.52 | 0.73
RND | 0.46 | 0.70 | 0.64 | 0.82 | 0.52 | 0.64 | 0.24 | 0.40 | 0.33 | 0.65 | 0.45 | 0.77
100 | Random | 0.17 | 0.25 | 0.29 | 0.43 | 0.31 | 0.36 | 0.15 | 0.20 | 0.22 | 0.33 | 0.15 | 0.25
Baseline | 0.48 | 0.64 | 0.60 | 0.76 | 0.60 | 0.84 | 0.53 | 0.81 | 0.58 | 0.83 | 0.36 | 0.75
EE | 0.54 | 0.84 | 0.80 | 0.92 | 0.80 | 0.88 | 0.64 | 0.82 | 0.66 | 0.96 | 0.38 | 0.76
RND | 0.56 | 0.76 | 0.64 | 0.86 | 0.68 | 0.86 | 0.62 | 0.82 | 0.59 | 0.82 | 0.55 | 0.73
200 | Random | 0.14 | 0.255 | 0.315 | 0.465 | 0.25 | 0.32 | 0.14 | 0.22 | 0.19 | 0.295 | 0.135 | 0.255
Baseline | 0.42 | 0.68 | 0.56 | 0.82 | 0.68 | 0.82 | 0.60 | 0.78 | 0.60 | 0.88 | 0.46 | 0.76
EE | 0.62 | 0.80 | 0.80 | 0.94 | 0.76 | 0.84 | 0.64 | 0.80 | 0.56 | 0.90 | 0.64 | 0.90
RND | 0.66 | 0.84 | 0.74 | 0.82 | 0.76 | 0.90 | 0.64 | 0.88 | 0.70 | 0.92 | 0.62 | 0.94
500 | Random | 0.154 | 0.25 | 0.34 | 0.466 | 0.236 | 0.34 | 0.138 | 0.214 | 0.19 | 0.308 | 0.136 | 0.26
Baseline | 0.50 | 0.86 | 0.74 | 0.84 | 0.84 | 0.92 | 0.56 | 0.78 | 0.54 | 0.84 | 0.70 | 0.94
EE | 0.50 | 0.84 | 0.80 | 0.94 | 0.92 | 0.96 | 0.68 | 0.92 | 0.58 | 0.92 | 0.54 | 0.94
RND | 0.58 | 0.80 | 0.90 | 0.96 | 0.80 | 0.92 | 0.58 | 0.80 | 0.54 | 0.86 | 0.70 | 0.96
1000 | Random | 0.151 | 0.259 | 0.367 | 0.493 | 0.208 | 0.311 | 0.144 | 0.216 | 0.17 | 0.307 | 0.133 | 0.26
Baseline | 0.56 | 0.70 | 0.86 | 0.98 | 0.86 | 0.92 | 0.66 | 0.92 | 0.60 | 0.98 | 0.62 | 1.00
EE | 0.64 | 0.88 | 0.80 | 0.90 | 0.72 | 0.88 | 0.66 | 0.88 | 0.56 | 0.86 | 0.54 | 0.92
RND | 0.66 | 0.80 | 0.90 | 0.98 | 0.86 | 0.96 | 0.64 | 0.92 | 0.72 | 1.00 | 0.68 | 0.94
TABLE II: Success rates for oatmeal and peanuts with 50 dataset. Model | 20 g (Oatmeal) | 30 g (Peanuts)
---|---|---
$5\%$ $(1g)$ |  $10\%$ $(2g)$ |  $5\%$ $(2g)$ |  $10\%$ $(3g)$
Random | 0.28 | 0.34 | 0.20 | 0.30
Baseline | 0.36 | 0.54 | 0.31 | 0.43
EE | 0.37 | 0.62 | 0.44 | 0.61
RND | 0.46 | 0.65 | 0.36 | 0.48

##  VI Experiment Results

In our experiments, we compare the following methods: A) _Random:_ Uniformly
sample grasp points within the tray (this is the data collection method) B)
_Baseline:_ Evaluate 900 equidistant grasp points from a 45$\times$20 grid
with about 1 cm grid size and select the point where mass estimation model’s
output is closest to the target mass. C) _EE:_ _Baseline_ with uncertainty
estimation by the proposed error estimation model and, D) _RND:_ same as C but
using the RND model.

###  VI-A Random Grasping for Data Collection

Figure 4 shows the frequency with which a particular mass of coffee beans or
rice was grasped. We show frequencies for training datasets of sizes 50, 100,
200, 500, and 1000. These datasets were also used to train our neural
networks.

For both coffee beans and rice, the data increasingly looks normally
distributed for sizes 200 onwards. We also note that the variance of coffee
beans is smaller than that of rice due to rice’s higher density leading to
more varied outcomes.

For the smaller dataset sizes of 50 and 100, we note the data frequency is a
lot more discontinuous and sparse. Some values like 60 g of rice are notably
absent in the dataset of size 50, despite being one of the evaluated targets
in the next section. Methods trained with classification would not interpolate
well and compensate for these missing values as they have no concept of their
labels’ continuity.

###  VI-B Target-mass Grasping with Uncertainties

We collect datasets as described in Section VI-A and partition them into
datasets of size 50, 100, 200, 500, and 1000. We train the methods described
in Section IV for each of these datasets and evaluate them by attempting to
grasp three target masses each. Using the training set of size 1000, we
calculate the _mean_ and standard deviation (_std_) of the grasped masses and
use them to set the grasp targets _(mean-std, mean, and mean+std)_. Coffee
beans were grasped at targets of (17 g, 22 g, and 27 g), and rice at (45 g, 60
g, and 75 g), respectively. We define a _successful_ grasp as one which is
within an acceptable tolerance range of the required target (expressed in %).
More specifically, we count a grasp as a success when
$\frac{|predicted\\_mass-target\\_mass|}{target\\_mass}\leq tol$. We present
success rates in Table I for two tolerance values- 5% and 10%, which are the
accepted food industry standards for human picks and robot picks,
respectively.

For each method, we attempt to grasp every target mass 50 times (with rice
trained with datasets of size 50 and 100 being the exceptions with 100
attempts required to stabilize results). To select the grasping point, 900
points are selected via grid search over the food tray, with a resolution of
roughly 1cm. Among these 900 points evaluated by the model, there are
typically more than 10 points within the acceptable range of $\pm$0.5 g (See
Fig. 4).

From Table I, we see that methods that incorporate uncertainty in the grasp
selection process, namely _EE_ and _RND_ , tend to outperform the _baseline_ ,
especially for the small dataset sizes. We highlight these results in bold.
For datasets of size 50-200, we see both _EE_ and _RND_ outperform the
baseline in almost every instance, with the difference being particularly
stark for size 50. Table I also shows that for coffee beans, the performances
of _EE_ and _RND_ trained on a dataset of size 50 is similar to that of the
baseline trained on datasets of size 200 or larger. Although the experimental
results for rice do not show as remarkable a difference as coffee beans, the
use of uncertainty still shows a marked performance improvement. For rice we
observe that _EE_ and _RND_ reach performances similar to the baseline, but
consistently requiring 50-100 less training samples.

The _random_ method, which we present as the null hypothesis, is always
outperformed by other methods. We note that as the size of the dataset
increases, _baseline_ catches up to or sometimes exceeds the performance of
the methods we present. This behavior is as expected, as the increasing
training dataset sizes decrease the epistemic uncertainty in the model’s
prediction as it is better able to capture the data generating process.
Incorporating these uncertainty predictions in the grasp selection process no
longer provides pertinent information that isn’t already captured in the
baseline’s maximum likelihood estimate.

The lack of significant improvement for large datasets suggests methods like
_EE_ and _RND_ predominantly capture epistemic uncertainty, which is
considered reducible by the gathering of additional data. However, since data
comes at a premium for applications like ours, it is essential to minimize it
as much as possible. Incorporating uncertainty into the decision process
provides a practical, easy-to-implement method to learn with significantly
fewer data samples.

We study the effect of varying data-size and target masses on coffee beans and
rice and confirm the possibility of small-dataset training. Furthermore, we
test its veracity via additional experiments conducted on specific target
weights for oatmeal and peanuts. Oatmeal and peanuts have yet again different
dynamics compared to coffee beans and rice, but Table II shows that our
methods still work robustly while using the same gripper. Oatmeal is light and
planar and gets compressed when grasped, while peanuts break into halves when
grasped too tightly. These differences heavily influence the nature of the
collected data, but despite of this our methods were able to capture the
dynamics of the different granular foods well using few training data.

##  VII Conclusion

In this paper, we introduce methods to capture a model’s uncertainty through
self-supervised learning and use it in grasp selection to obtain competitive
performance using remarkably smaller data sizes. Additionally, we have
developed an autonomous data collection system. To capture the uncertainty, we
adopt _EE_ and _RND_ models, and apply them to our grasp mass estimation
models for granular foods. We show empirically that a simple modification to a
baseline grasp selection heuristic that chooses the least uncertain prediction
among the grasp candidates results in significant performance improvements.
This modification allows for success rates similar to the baseline, but
requires less training data compared to the baseline.

## ACKNOWLEDGMENT

The authors would like to thank Dr. Naoki Fukaya for creating the grippers
used in our experiments.

## References

  * [1] J. Matsuoka, Y. Tsurumine, Y. Kwon, T. Matsubara, T. Shimmura, and S. Kawamura, “Learning food-arrangement policies from raw images with generative adversarial imitation learning,” in _2020 17th International Conference on Ubiquitous Robots (UR)_. IEEE, 2020, pp. 93–98.
  * [2] E. Misimi, A. Olofsson, A. Eilertsen, E. R. Øye, and J. R. Mathiassen, “Robotic handling of compliant food objects by robust learning from demonstration,” in _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2018, pp. 6972–6979.
  * [3] M. Hoerger, J. Song, H. Kurniawati, and A. Elfes, “Pomdp-based candy server: Lessons learned from a seven day demo,” in _Proceedings of the International Conference on Automated Planning and Scheduling_ , vol. 29, no. 1, 2019, pp. 698–706.
  * [4] C. Schenck, J. Tompson, S. Levine, and D. Fox, “Learning robotic manipulation of granular media,” in _Conference on Robot Learning_ , 2017, pp. 239–248.
  * [5] S. Clarke, T. Rhodes, C. G. Atkeson, and O. Kroemer, “Learning audio feedback for estimating amount and flow of granular material,” _Proceedings of Machine Learning Research_ , vol. 87, 2018.
  * [6] S. C. Hora, “Aleatory and epistemic uncertainty in probability elicitation with an example from hazardous waste management,” _Reliability Engineering & System Safety_, vol. 54, no. 2-3, pp. 217–223, 1996.
  * [7] A. Kendall and Y. Gal, “What uncertainties do we need in bayesian deep learning for computer vision?” in _Advances in neural information processing systems_ , 2017, pp. 5574–5584.
  * [8] D. J. MacKay, “Bayesian methods for adaptive models,” Ph.D. dissertation, California Institute of Technology, 1992.
  * [9] W. L. Buntine and A. S. Weigend, “Bayesian back-propagation,” _Complex Systems_ , vol. 5, pp. 603–643, 1991.
  * [10] R. M. Neal, “Bayesian learning for neural networks,” Ph.D. dissertation, University of Toronto, 1995.
  * [11] J. F. G. De Freitas, “Bayesian methods for neural networks,” Ph.D. dissertation, University of Cambridge, 2003.
  * [12] Y. Gal, “Uncertainty in deep learning,” _University of Cambridge_ , vol. 1, no. 3, 2016.
  * [13] J. S. Denker and Y. LeCun, “Transforming neural-net output levels to probability distributions,” in _Advances in neural information processing systems_ , 1991, pp. 853–859.
  * [14] Y. Gal and Z. Ghahramani, “Bayesian convolutional neural networks with bernoulli approximate variational inference,” _arXiv preprint arXiv:1506.02158_ , 2015.
  * [15] ——, “Dropout as a bayesian approximation,” _arXiv preprint arXiv:1506.02157_ , 2015.
  * [16] B. Lakshminarayanan, A. Pritzel, and C. Blundell, “Simple and scalable predictive uncertainty estimation using deep ensembles,” in _Advances in neural information processing systems_ , 2017, pp. 6402–6413.
  * [17] A. Bicchi and V. Kumar, “Robotic grasping and contact: A review,” in _Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No. 00CH37065)_ , vol. 1. IEEE, 2000, pp. 348–353.
  * [18] J. Bohg, A. Morales, T. Asfour, and D. Kragic, “Data-driven grasp synthesis—a survey,” _IEEE Transactions on Robotics_ , vol. 30, no. 2, pp. 289–309, 2013.
  * [19] A. Sahbani, S. El-Khoury, and P. Bidaud, “An overview of 3d object grasp synthesis algorithms,” _Robotics and Autonomous Systems_ , vol. 60, no. 3, pp. 326–336, 2012.
  * [20] R. C. Brost, “Automatic grasp planning in the presence of uncertainty,” _The International Journal of Robotics Research_ , vol. 7, no. 1, pp. 3–17, 1988.
  * [21] K. Y. Goldberg and M. T. Mason, “Bayesian grasping,” in _Proceedings., IEEE International Conference on Robotics and Automation_. IEEE, 1990, pp. 1264–1269.
  * [22] Y. Zheng and W.-H. Qian, “Coping with the grasping uncertainties in force-closure analysis,” _The International Journal of Robotics Research_ , vol. 24, no. 4, pp. 311–327, 2005.
  * [23] V. N. Christopoulos and P. Schrater, “Handling shape and contact location uncertainty in grasping two-dimensional planar objects,” in _2007 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2007, pp. 1557–1563.
  * [24] C. Goldfeder and P. K. Allen, “Data-driven grasping,” _Autonomous Robots_ , vol. 31, no. 1, pp. 1–20, 2011.
  * [25] A. T. Miller and P. K. Allen, “Graspit! a versatile simulator for robotic grasping,” _IEEE Robotics & Automation Magazine_, vol. 11, no. 4, pp. 110–122, 2004.
  * [26] K. Hsiao, P. Nangeroni, M. Huber, A. Saxena, and A. Y. Ng, “Reactive grasping using optical proximity sensors,” in _2009 IEEE International Conference on Robotics and Automation_. IEEE, 2009, pp. 2098–2105.
  * [27] P. Pastor, L. Righetti, M. Kalakrishnan, and S. Schaal, “Online movement adaptation based on previous sensor experiences,” in _2011 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2011, pp. 365–371.
  * [28] K. Hsiao, S. Chitta, M. Ciocarlie, and E. G. Jones, “Contact-reactive grasping of objects with partial shape information,” in _2010 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2010, pp. 1228–1235.
  * [29] J. Kim, K. Iwamoto, J. J. Kuffner, Y. Ota, and N. S. Pollard, “Physically based grasp quality evaluation under pose uncertainty,” _IEEE Transactions on Robotics_ , vol. 29, no. 6, pp. 1424–1439, 2013.
  * [30] E. Johns, S. Leutenegger, and A. J. Davison, “Deep learning a grasp function for grasping under gripper pose uncertainty,” in _2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2016, pp. 4461–4468.
  * [31] J. Mahler, J. Liang, S. Niyaz, M. Laskey, R. Doan, X. Liu, J. A. Ojea, and K. Goldberg, “Dex-net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics,” _arXiv preprint arXiv:1703.09312_ , 2017.
  * [32] A. T. Miller, S. Knoop, H. I. Christensen, and P. K. Allen, “Automatic grasp planning using shape primitives,” in _2003 IEEE International Conference on Robotics and Automation (Cat. No. 03CH37422)_ , vol. 2. IEEE, 2003, pp. 1824–1829.
  * [33] I. Lenz, H. Lee, and A. Saxena, “Deep learning for detecting robotic grasps,” _The International Journal of Robotics Research_ , vol. 34, no. 4-5, pp. 705–724, 2015.
  * [34] J. Redmon and A. Angelova, “Real-time grasp detection using convolutional neural networks,” in _2015 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2015, pp. 1316–1322.
  * [35] S. Kumra and C. Kanan, “Robotic grasp detection using deep convolutional neural networks,” in _2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2017, pp. 769–776.
  * [36] D. Morrison, J. Leitner, and P. Corke, “Closing the loop for robotic grasping: A real-time, generative grasp synthesis approach,” 06 2018.
  * [37] L. Pinto and A. Gupta, “Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours,” in _2016 IEEE international conference on robotics and automation (ICRA)_. IEEE, 2016, pp. 3406–3413.
  * [38] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, and S. Levine, “Scalable deep reinforcement learning for vision-based robotic manipulation,” ser. Proceedings of Machine Learning Research, A. Billard, A. Dragan, J. Peters, and J. Morimoto, Eds., vol. 87. PMLR, 2018, pp. 651–673. [Online]. Available: http://proceedings.mlr.press/v87/kalashnikov18a.html
  * [39] C. Gehring and D. Precup, “Smart exploration in reinforcement learning using absolute temporal difference errors,” in _Proceedings of the 2013 international conference on Autonomous agents and multi-agent systems_ , 2013, pp. 1037–1044.
  * [40] Y. Burda, H. Edwards, A. Storkey, and O. Klimov, “Exploration by random network distillation,” _arXiv preprint arXiv:1810.12894_ , 2018.
  * [41] P. Ray and M. J. Howard, “Robotic untangling of herbs and salads with parallel grippers.”
  * [42] G. Bradski and A. Kaehler, _Learning OpenCV: Computer vision with the OpenCV library_. ” O’Reilly Media, Inc.”, 2008.
  * [43] S. Garrido-Jurado, R. Munoz-Salinas, F. J. Madrid-Cuevas, and R. Medina-Carnicer, “Generation of fiducial marker dictionaries using mixed integer linear programming,” _Pattern Recognition_ , vol. 51, pp. 481–491, 2016.
  * [44] F. J. Romero-Ramirez, R. Muñoz-Salinas, and R. Medina-Carnicer, “Speeded up detection of squared fiducial markers,” _Image and vision Computing_ , vol. 76, pp. 38–47, 2018.