# Aptly: Making Mobile Apps from Natural Language

Evan W. Patton1§, David Y.J. Kim1§, Ashley Granquist2§, Robin Liu2, Arianna
Scott2, Jennet Zamanova2 Harold Abelson1,2 Email: {ewpatton<EMAIL_ADDRESS>1
MIT Computer Science and Artificial Intelligence Laboratory  2 MIT Electrical
Engineering and Computer Science

###### Abstract

We present Aptly, an extension of the MIT App Inventor platform enabling
mobile app development via natural language powered by code-generating large
language models (LLMs). Aptly complements App Inventor’s block language with a
text language designed to allow visual code generation via text-based LLMs. We
detail the technical aspects of how the Aptly server integrates LLMs with a
realtime collaboration function to facilitate the automated creation and
editing of mobile apps given user instructions. The paper concludes with
insights from a study of a pilot implementation involving high school
students, which examines Aptly’s practicality and user experience. The
findings underscore Aptly’s potential as a tool that democratizes app
development and fosters technological creativity.

###### Index Terms:

MIT App Inventor, Large Language Model, Block programming

§§footnotetext: Equal contribution

##  I Introduction

The rise of technology in daily life highlights the need for everyone,
particularly youth, to engage with digital tools to solve real-world
challenges. Mobile applications serve as a powerful medium for such engagement
given the widespread adoption of smartphones and tablets across diverse
demographics, including under-represented communities [1]. However, the
traditional app development process, which often requires a solid foundation
in computer science, has historically excluded many aspiring creators from
participating. To democratize this process, the MIT App Inventor initiative
has revolutionized the creation of mobile apps by introducing a user-friendly,
visual programming environment that empowers anyone, regardless of their
technical background, to develop functional applications [2]. App Inventor
utilizes a block-based programming approach, which allows users to visually
assemble their applications using geometric shapes, making the development
process more intuitive and accessible [3]. The tool not only fosters
creativity among novice programmers but also encourages a more inclusive
approach to technological education and development [4]. Tissenbaum et al. [5]
call the process by which people leverage technology like App Inventor to make
societal impact computational action.

The advent of large language models (LLMs) and their ability to generate code
[6, 7] has opened new avenues for mobile app development. Capitalizing on this
technological advancement, our research introduces Aptly, a groundbreaking
platform that allows users to create mobile applications using natural
language to lower barriers to computational action. For example, a user could
simply state, “Make me an app that translates English to Spanish,” and Aptly
would automatically generate a functional app based on this input. This paper
details the framework of Aptly, beginning with an overarching view of the
system’s architecture. We will discuss the development of the Aptly Language
and our method of integrating it with an LLM to produce the desired syntax.
Additionally, we explore the mechanisms that enable users to refine their apps
through natural language commands. We conclude by examining Aptly’s current
functionalities and presenting preliminary findings from its deployment among
high school students.

##  II Aptly Architecture

Figure 1: An interaction diagram between the different components of the MIT
App Inventor with Aptly system.

Aptly augments MIT App Inventor with generative AI coding capabilities. We
accomplish this by building on a version of App Inventor supporting real-time
collaboration (RTC) [8]. The Aptly agent acts as a collaborator with users,
taking their instructions in natural language and manipulating the project
just as another user might. The overall system involves the following
components:

  * •

MIT App Inventor: The base version of the App
Inventor111https://github.com/mit-cml/appinventor-sources is extended to
integrate with the Aptly system and the RTC server to facilitate enhanced
functionality.

  * •

MIT App Inventor RTC: The real-time collaboration (RTC) server mediates
between agents in the system architecture.

  * •

Aptly Server: Detailed in the subsequent section, the Aptly server invokes the
LLM and converts its responses into events understood by App Inventor RTC.

  * •

OpenAI GPT-3.5/4: Utilizing OpenAI’s LLM models, this component provides
advanced code completion services [9, 10].

Figure 1 depicts the relationship between the various components as they are
used by the app creator.

###  II-A The Aptly Language

⬇

Screen1 = Screen()

HA1 = HorizontalArrangement(Screen1)

Label1 = Label(HA1, Text = "Weight in lbs: ")

EarthWeight = TextBox(HA1, NumbersOnly = True)

Label2 = Label(Screen1, Text = "Select Planet:")

PlanetList = ListView(Screen1, ElementsFromString = "Mercury, Venus, Mars,
Jupiter, Saturn, Uranus, Neptune")

Calculate = Button(Screen1, Text = ’Calculate’)

PlanetaryWeight = Label(Screen1)

initialize gravities = {"Mercury": 0.38, "Venus": 0.91, "Mars": 0.38,
"Jupiter": 2.34, "Saturn": 0.93, "Uranus": 0.92, "Neptune": 1.12}

to compute_weight(earth_lbs, planet):

return earth_lbs * dictionaries_lookup(planet, global gravities, ’not found’)

when Calculate.Click():

set PlanetaryWeight.Text = call compute_weight(EarthWeight.Text,
PlanetList.Selection)

Listing 1:  An app to calculate weights on different planets expressed in
Aptly. Figure 2: An example of the Aptly code in Listing 1 and its
correspondence to the App Inventor interface.

The Aptly language is designed to be used with LLMs. It is a textual
representation of App Inventor programs inspired by Python. The decision to
base Aptly on Python was made for two reasons:

  1. 1.

Python’s pseudocode-like syntax closely aligns with the readability and
structure of App Inventor blocks, facilitating intuitive understanding and
use.

  2. 2.

LLMs have been extensively trained on Python code, enhancing their ability to
generate accurate and functional code in a similar syntax.

Given these attributes, we hypothesize that generative models can be
effectively fine-tuned to interface with Aptly, leveraging its Python-like
structure. However, it is important to note that while Aptly draws inspiration
from Python, it is distinct. Aptly has been specifically constrained to ensure
a one-to-one correspondence with App Inventor: every valid App Inventor
program is also a valid Aptly program, and vice versa. See Listing 1 for an
example Aptly program and Figure 2 for showcasing how the code is represented
as blocks.

Figure 3: When the user requests an application with its textual description,
we automatically synthesize the prompt by adding several example pairs along
with the desired application’s textual description. This constructed prompt is
fed into one of OpenAI’s GPT-X models as an input, which outputs code that can
be converted into a fully functional mobile application

To ensure the fidelity and accuracy of the one-to-one correspondence between
the Aptly and App Inventor languages, we employ a comprehensive suite of
integration tests. These tests validate the conversion cycle by processing
Aptly code through a round-trip transformation to an App Inventor (.aia) file
and back to Aptly. Furthermore, any new additions to the Aptly language
introduced through pull requests are required to include corresponding tests
that affirm the functionality and behavior of the new features.

For maintaining the precise correspondence of the Aptly language, we define
the following four components:

  * •

Parser: Parses the Aptly code into its respective representation in the
Abstract Syntax Tree (AST). Each method adheres to the extended Backus-Naur
Form (eBNF) of the Aptly language.

  * •

ProjectReader: Transforms the App Inventor representations into the
corresponding Aptly AST.

  * •

ProjectWriter: Converts Aptly AST into the App Inventor representation (either
in the designer or blocks format).

  * •

Serializer: Outputs the AST into the Aptly code format.

In tandem, these four components provide the core functionality to enable app
creation with Aptly by allowing us to convert between the text and block
representations via the Aptly AST.

Figure 4: An app created from a prompt given in Dutch.

###  II-B Few-shot Prompt Engineering

Aptly leverages machine learning research on generative models, which are
probabilistic frameworks that learn how datasets are generated to facilitate
new data creation. We specifically employ a transformer-based large language
model to develop mobile apps [11]. This approach capitalizes on the recent
significant advancements in large language models, an area that has garnered
substantial interest for its impressive research outcomes and substantial
commercial opportunities. Generative models operate based on “prompts” which
are crucial inputs that significantly influence their performance. The
effectiveness of these models hinges on the quality of the prompts, a practice
known as prompt engineering—an active field within machine learning research
[12]. A common method of prompt engineering involves “few-shot” prompts, where
a model is given a limited number of solved task examples as part of its
input. This approach is often interpreted as the model “learning” the task in
real-time from these few examples [9]. Aptly uses Open AI’s LLMs, which are
capable of generating computer code from natural language descriptions in
several programming languages including Java, JavaScript, Python, C, C#, and
Swift.

When the user requests an app with their natural description, we synthesize a
prompt which is a natural language description (denoted as $D$) of the desired
app to be created, together with a set of example pairs, such as the following
$<<d\\_1,c\\_1>><<d\\_2,c\\_2>>...<<d\\_k,c\\_k>>$, where $d\\_i$ is the
description of application $i$, and $c\\_i$ is the corresponding Aptly code of
the description for application $i$. The example pairs come from a database of
unique Aptly examples compiled by the team from apps created on the App
Inventor platform. The example pairs are not expected to include the literal
description $D$ to be processed nor the actual Aptly Code to be generated
which would make the generation task trivial. Instead, generative models use
the provided examples to guide their processing in generating new original
output.

Aptly’s method for providing relevant prompts to accompany a description $D$
is to provide a set of example application descriptions paired with
appropriate Aptly Code. We use semantic similarity to highlight the
“appropriateness” of code and description. The method relies on having a way
to automatically measure the similarity of text descriptions and the program.
For computing the similarity of the text description and program, we use
embeddings of data elements, which are numerical representations of concepts
converted to number sequences [13]. In our scenario, an embedding represents
the semantic meaning of a natural language description or code. To measure the
similarity of two items, one embeds them in the same vector space and takes
the distance between the two embedding vectors as a measure of similarity
(Small distance implies high similarity while large distance implies low
similarity). We use cosine distance, which reflects the angle between vectors,
to compute the similarity between two vectors.

We employ OpenAI’s Text/Code Embedding model to generate embeddings from
descriptions and associated code. These neural network models utilize
Contrastive Pre-Training—a training method that functions by clustering in
vector space [14]. Specifically, it draws predefined positive examples (i.e.,
matching text and code) closer together, while pushing negative examples
(i.e., contrasting text and code) apart. This technique enhances our model’s
ability to identify and select the example pair most relevant to a user’s
specified description. We feed the synthesized prompt into GPT, which outputs
the Aptly Code corresponding to the user’s requested app description. We can
then convert the generated code into App Inventor blocks to generate a fully
functional application (Figure 3).

An interesting feature of the use of LLMs for this process is that the user
input need not be expressed in English as might be the case with traditional
natural language processing techniques. As an example, we provided this prompt
in Dutch: “Maak een app met een tekstbox, een lijst met zes talen en een knop
met de tekst ‘vertalen’. Wanneer er op de knop wordt gedrukt dan vertaal je de
tekst in de geselecteerde taal en laat de vertaling zien.” This produced an
app that gave the user the option to translate into English, Spanish, French,
German, Italian, and Japanese, with all of the app content expressed in Dutch
(Figure 4). Importantly, all of the example pairs we provide the LLM are
written in English–there are no examples expressed in Dutch. Ultimately, these
abilities of LLMs will reduce the barrier to computational action by removing
the need to learn English before or while learning to code.

Figure 5: An overview of the information flow during an edit operation using
Aptly. Arrows represent the flow of information and should be read as “tail
label head,” e.g., “App Inventor sends AIA file” or “RTC Event Sequence
consumed by App Inventor RTC.”

###  II-C The Editing Capability

Often it is unreasonable to assume one can create a full app in a single shot
request. We aim to empower programmers not only to generate apps using natural
language, but also to refine, iterate, and evolve their designs continuously
through natural language commands. This approach allows for a more dynamic
development process, enabling creators to adapt and modify their applications
easily as new ideas or requirements emerge. This section details how editing
of apps is achieved in Aptly.

Figure 6: The update operations (blue) to transform the AST of the original
program into the AST of the modified program. Some unmodified nodes are
omitted for brevity.

####  II-C1 Editing Projects using Descriptions

Editing a project with Aptly is done by emitting a sequence of edit operations
over the MIT App Inventor RTC system. When a user requests an edit by
providing a description of their desired change, App Inventor sends the AIA
for the current project plus the user-provided edit description to the
endpoint of the Aptly server.

The editing process uses the following steps (see also Figure 5). The Aptly
frontend sends the project state and user request to the server (1), which
utilizes the ProjectReader and Serializer to obtain the corresponding Aptly
code. The server then constructs an edit prompt–a top-ranked selection of
example edits plus the project (converted to Aptly code) and the
description–to be sent (2) to the LLM. We use the same technique for ranking
examples as discussed previously.

When the LLM returns updated Aptly code, the Aptly server constructs the AST
representation of the original code (via the ProjectReader) and the modified
code (via the Parser), and computes the minimum-cost sequence of edit
operations to transform the original AST into the modified one. The Aptly
server utilizes the tree edit distance algorithm proposed by Zhang and Shasha
[15] to compute the minimum-cost sequence of edit operations (Fig. 5 note 3).
Aptly specifically utilizes the Python zss module.The edit operations returned
are INSERT, REMOVE, UPDATE, and MATCH. The update_cost function is customized
for App Inventor by taking into account the type of two nodes; for example, if
there are two component declarations (e.g., Button1 = Button(...) and Label1 =
Label(...)), there should be infinite update cost to transform one of those
component declarations into the other because Button and Label are two
different types and App Inventor does not allow a way to change a button into
a label or vise versa. Similarly, in App Inventor, if there are two bound
component events (e.g., when Button1.Click() and when Button2.Click()), it
should only be possible to update one to be the other if the type of the
component is the same because this is the only type of change App Inventor
allows in the Blocks editor for a bound component event. If the user wants to
change which method they’re using (e.g., Click vs. TouchUp) or change the
event to a different component type (e.g., Button1.GotFocus vs
Switch1.GotFocus), the user would have to grab a new block from the blocks
drawer and delete the old block, so Aptly cannot make this change via an
UPDATE operation either.

Given the sequence of insertions, removals, updates, and matches recommended
by zss, the TreeDiff class emits a sequence of edit events compatible with the
App Inventor RTC system (Fig. 5 note 4). These edit events include creating,
updating, moving, and deleting components and blocks. Some changes require
multiple edit events; for example, since block stacks are represented as
linked lists, inserting a block in the middle of a block stack requires
creating the block, moving it to the correct location (i.e., setting its
parent correctly), and moving its descendent (if applicable) to now point to
it rather than its previous parent. As another example, deleting an item in a
list or deleting a pair in a dictionary requires issuing a move event for all
items after the deleted item in the list, as even though these structures are
not represented as linked lists, each subsequent item’s input name (e.g.,
‘ITEM1’ vs. ‘ITEM0’ for list items) will change.

To emit the correct edit events via the RTC system, we process the operations
returned from zss in several stages:

  * •

Preinsertion: To handle moving components, we need to perform a preprocessing
step that aggregates all of the new components into a map based on their name
and type. We process all of the insert operations to build this map, which
will be consumed as part of the remove stage to determine whether a component
was renamed, moved, or removed. If we did not include such a stage, we might
end up deleting and then inserting the same node rather than moving a node,
incurring unnecessary cost.

  * •

Remove: We emit removal events for deleted components first. When a component
is removed, we log this information so that any block removals involving that
component are ignored since they are covered by the component’s removal.
Likewise, we need to log all of the block IDs that are removed so that we do
not also emit removal events for the subtree starting at that node. Note that
there is a special case in regards to moved components; for subtrees starting
at nodes that are marked as insertions, a removal does not take place.

  * •

Match: The Match operation in the tree edit sequence is a no-op. However, when
translating the operations into App Inventor events, we use this step to
perform some bookkeeping necessary to support insertion operations later.

  * •

Update: The Update stage converts the value of a node in the source tree to a
new value at the same position in the destination tree. Common reasons for
update operations involve renaming components, changing property values, or
renaming variables or procedures.

  * •

Insert Stage: The Insert operation adds a new subtree to the AST. Because we
enumerate the changes in breadth-first order, an insertion in the tree covers
the whole subtree at that node. When subtrees are inserted, we must return to
looking at the parent node (which would be a match/update operation) in order
to issue the necessary move events needed based on the inserted node (such as
moving the new node to the right place and adjusting the parent and input name
of successors if needed).

Once all of the edit events have been created, the Aptly server joins the
user’s project in the RTC system and emits the stream of changes (Fig. 5 notes
5, 6).

####  II-C2 Applying Updates via Real-Time Collaboration System

Aptly leverages the App Inventor Real-Time Collaboration (RTC) [8] system to
edit the project based on the event sequence produced in the previous step.
Aptly is conceptualized as a collaborative user within the project
environment. Once the event sequence is generated by the Aptly server, it
joins the user’s project in the RTC server. It then sends the event sequence
to the server, which relays the changes to the user’s App Inventor session to
be applied to the project. Leveraging the RTC system, Aptly allows for
modifications to be reversed if they do not meet the user’s expectations. This
functionality is facilitated by the RTC’s capability to execute events in
reverse, relying solely on a stack of events that can be undone, thereby
eliminating the need for maintaining additional state information. This design
enhances user control and flexibility, making the editing process more user-
friendly and adaptable to changes.

####  II-C3 Worked Example

Consider the following Aptly code representing the app “HelloPurr,” App
Inventor’s equivalent to a Hello World program.

⬇

Screen1 = Screen(AppName = "HelloPurr")

Cat = Button(Screen1, Image = "kitty.png")

Meow = Sound(Screen1, Source = "meow.mp3")

when Cat.Click():

call Meow.Play()

One might offer in natural language “add a label below the kitty that says
‘pet the kitty’.” The LLM may generate the following Aptly code in response:

⬇

Screen1 = Screen(AppName = "HelloPurr")

Cat = Button(Screen1, Image = "kitty.png")

Label1 = Label(Screen1, Text = "pet the kitty")

Meow = Sound(Screen1, Source = "meow.mp3")

when Cat.Click():

call Meow.Play()

An illustration of the above two codes AST, respectively noted as $a$ and $b$,
is shown in figure 6. The two ASTs are then processed into a sequence of edits
needed to transform the AST of the first snippet into the AST of the second
snippet. In this particular scenario, the Zhang-Shasha algorithm will result
in MATCH operations except for the INSERT operations that introduce the AST
nodes corresponding to the insertion on line 3:

⬇

INSERT Label1 (Identifier)

INSERT Label (Reference)

INSERT Screen1 (Reference)

INSERT Text (Reference)

INSERT "pet the kitty" (StringValue)

INSERT Text = "pet the kitty" (DesignerAssignment)

INSERT ComponentTypeInvocation(Label1, Screen1, {Text: "pet the kitty"})
(ComponentTypeInvocation)

INSERT ComponentDecl(Label1, ComponentTypeInvocation(Label1, Screen1, {Text:
"pet the kitty"})) (ComponentDecl)

For the processing of INSERT operations, Aptly considers the operations in a
breadth-first fashion, which leads to the ComponentDecl being the first node,
n, encountered during traversal. We know that an insertion of a node a depth d
subsumes all of the children below depth d in the subtree rooted at n. The
TreeDiff class checks to see if it has a method to handle the insertion of the
AST type of n (i.e., ComponentDecl). If such a method exists, it is called.
Otherwise, the subtree starting at n will continue to be traversed breadth-
first. In the case of ComponentDecl, such a method does exist and it
interprets the insertion of a ComponentDecl node as the RTC event sequence of
ComponentAdd, ComponentMove, and one or more ComponentProperty events. For
example, if the UUID of the label component is 12345:

⬇

ComponentAdd(12345, "Label", [])

ComponentMove(12345, 0, 1)

ComponentProperty(12345, "$Name", "Label1")

ComponentProperty(12345, "Text", "pet the kitty")

This sequence of events is then streamed to the user’s workspace via the RTC
server.

##  III Results

###  III-A Coverage of App Inventor

Table I summarizes the current coverage Aptly provides for App Inventor
functionality. We achieved our target of having one to one correspondence of
the block language, with a few exceptions for new language features added by
the App Inventor team since the Aptly project began.

TABLE I: Summary of Aptly’s block language coverage Category | Coverage %
---|---
Controls | 100
Logic | 100
Math | 100
Text | 94
Lists | 100
Dictionaries | 94
Colors | 100
Procedures | 75
Variables | 80
Components | 80
Total | 96
TABLE II: Summary of Aptly Examples’ coverage of App Inventor functionality. Feature | Coverage %
---|---
Components | 40.0
Events | 13.6
Methods | 6.4
Properties | 13.7

Coverage of App Inventor’s component library is more limited as shown in Table
II. When building examples, we have aimed to cover the most common
functionality observed in projects as studied by the App Inventor team [16].
Importantly, there is a very long tail of component usage that suggests that
our initial set of example projects likely covers most use cases encountered
by App Inventor users.

###  III-B Pilot User Study

To evaluate the effectiveness and user experience of our tool, Aptly, we
conducted a study involving 10 high school students with the following
demographics:

  * •

Age: 17 (5), 18 (5)

  * •

Gender: Female (8), Male (2)

  * •

General programming experience: None (3), $<$ 1 year (2), 3-4 years (3), 5+
years (2)

  * •

Familiarity with block-based programming: None (3), Beginner (2), Familiar (5)

This diverse group was selected to yield insights into how students with
different expertise levels and app creation familiarity engage with our tool.
The study duration ranged from 50 to 80 minutes per participant. Initially, we
introduced the students to Aptly, demonstrating how to create a basic app with
interactive features such as a button and a label, and how to modify
properties like color and font using the tool’s editing capabilities. We then
assigned two distinct tasks to assess the tool’s utility and the participants’
ability to leverage it.

Figure 7: App Inventor Basic Tutorial: Digital Doodle

####  III-B1 Recreation Task

Students were shown “Digital Doodle” (Figure 7), a beginner-level app from App
Inventor, and asked to recreate it using Aptly. This task was designed to
observe whether participants could effectively translate a clear visual
concept into a functional app through verbal descriptions, reflecting their
understanding and communication of app design concepts. The task demonstrated
the platform’s flexibility in understanding and executing a range of
instructions, from detailed descriptions specifying colors and functionalities
of buttons to succinct commands like “Draws with RGB.” This versatility
highlights Aptly’s capacity to interpret and implement user input effectively,
allowing users with no expert programming skills to successfully create
functional apps.

Participants employed different strategies in app creation, some opting for a
more incremental approach using Aptly’s editing features. For instance, one
participant began with a basic layout and progressively added functionalities
and aesthetic adjustments, such as alignment and color changes, through
subsequent commands. This iterative process was particularly favored by
participants with programming backgrounds, reflecting a workflow similar to
traditional software development where features are built and refined over
time. This suggests that Aptly can accommodate various user preferences and
workflows, enhancing its usability across different skill levels and
development styles. However, the study also uncovered challenges related to
the precision of language used in commands. Some participants struggled to
articulate their desired app functionalities clearly, leading to discrepancies
between their intentions and the app’s behavior. For example, a participant’s
command to “draw a line when I click a button” was too vague without
specifying the color interaction, illustrating the current limitations of AI
in interpreting user intent without explicit instruction.

####  III-B2 Scenario-Based Task

Participants were given a scenario. Specifically, they were asked the
following scenario: “In your class you have a friend that is visually impaired
and is having trouble with their math class. You want to help that student by
creating a calculator app that will help them with their addition and
subtraction. Try using Aptly to create that app. Feel free to use the editing
and also manually drag and drop the blocks.” This task aimed to evaluate
participants’ ability to ideate and articulate solutions to real-world
problems using Aptly. It also encouraged them to consider accessibility
features, such as voice commands and easy-to-navigate interfaces, enhancing
their problem-solving and design-thinking skills.

Participants provided initial descriptions that ranged from simple one-liners
like “Make an app that is a calculator” to detailed paragraphs specifying the
layout and functionality, including voice recognition and auditory feedback.
This task demonstrated the ability of participants to conceptualize and
articulate diverse solutions using Aptly, catering to a specific accessibility
need. Participants’ approaches varied; some chose to incrementally build and
refine their app’s user interface and functionalities using Aptly’s editing
tools, while others outlined their complete vision from the onset. For
example, one participant started with a basic calculator design and
progressively enhanced its features and interface to be more accessible for
visually impaired users. This approach highlights Aptly’s flexibility to
accommodate different development styles and the iterative nature of designing
user-centric applications.

Moreover, the participants’ creative solutions for accessibility features
showcased the potential of integrating advanced technologies such as text-to-
speech and tactile feedback into apps. One innovative approach involved using
screen taps and phone vibrations to communicate numbers and results,
demonstrating how users can leverage Aptly not only as a development tool but
also as a platform for ideation. The diversity in usage and the degree of
reliance on Aptly’s capabilities varied, indicating that while some users
fully embraced the AI-driven development process, others preferred a more
hands-on approach in shaping their applications. This underscores the
importance of versatile tools like Aptly in modern app development, especially
when addressing the specific needs of diverse users.

####  III-B3 General Observation

Observations revealed valuable insights into user behavior and platform
design. Participants exhibited a general impatience with the system’s response
times, often proceeding to plan subsequent changes if immediate implementation
of edits was not observed. Challenges arose when Aptly misunderstood user
commands; while some participants attempted to rephrase their inputs multiple
times using Aptly editing, others opted to manually make the changes. Notably,
more experienced programmers frequently preferred manual adjustments and
issued a higher number of commands compared to their less experienced
counterparts. These seasoned programmers expressed a desire for greater
control, including access to error messages to better understand the system’s
handling of their commands. This feedback highlights a tension between
designing intuitive interfaces suitable for beginners and providing detailed
control and feedback that experienced users seek. This dichotomy suggests that
while simplifying interfaces can benefit novices, it may frustrate advanced
users who expect deeper interaction and transparency from the development
tools.

####  III-B4 Participant responses

To capture the changes in participants’ perceptions before and after using
Aptly, we asked the participants two questions before and after the
experiment.

  * •

“Programming is accessible for everyone”: Prior to the workshop, students’
views on programming accessibility varied, with some expressing skepticism
about universal accessibility due to factors such as technological
availability and socioeconomic barriers. Common responses ranged from
“somewhat disagree” to acknowledging partial agreement, contingent on access
to technology like computers and the internet. After using Aptly, participants
generally showed a nuanced view, with some noting improvements in their
perceptions of accessibility. Notably, though some recognized improvements,
they still highlighted limitations, particularly for individuals who might
lack basic digital literacy or access to necessary hardware.

  * •

“Artificial Intelligence can be a powerful tool to help people in
programming”: Initially, responses indicated a positive sentiment towards AI
as a powerful aid in programming, with a slight variety hinting at potential
job displacement concerns. After the workshop, reflections were more detailed,
emphasizing the practical benefits and challenges observed during the use of
AI-driven tools like Aptly. Participants appreciated AI’s potential to
simplify the programming process, especially for beginners, yet also discussed
the importance of understanding underlying logic beyond AI assistance. This
shift suggests a more informed appreciation of AI’s capabilities and
limitations in programming contexts.

We also asked each participant the question “Did Aptly change your interest in
learning how to program?” on a scale of 1 to 5, where higher values signify
increased interest. The range of responses spanned from a low of 2 to a high
of 4, with no extreme sentiments expressed, as the ratings did not include the
lowest (1) or highest (5) possible marks. This distribution implies that while
Aptly was generally effective in increasing interest among most participants,
it had a varied impact. The mean response was 3.7 (SD = 0.67). The mixed
response highlights the potential of Aptly to engage users, albeit not
uniformly, suggesting avenues for further investigation into individual
differences and tool optimization to broaden its appeal and effectiveness.

####  III-B5 Limitation of Aptly based on responses

In our study, participants identified several limitations of the Aptly
platform, providing valuable insights into its operational shortcomings and
areas ripe for development. Notably, Aptly struggled with complex command
executions such as playing specific audio files, drawing predefined shapes,
and making API calls. Furthermore, it demonstrated limitations in processing
abstract or intricate commands, which underscores deficiencies in its natural
language processing capabilities.

Feedback from users on improving Aptly emphasized enhancing both functionality
and user interface design. Participants suggested increasing the accuracy and
range of command interpretation to reduce errors, thereby making the platform
more robust for complex programming tasks. Improvements in the user interface
were recommended to make the tool more intuitive and accessible, with clearer
design elements that could facilitate easier navigation and usage. Enhanced
error messaging and user feedback, particularly for users with programming
experience, were noted as critical for allowing more effective debugging and
application refinement.

These findings indicate a significant need for advancements in natural
language understanding within Aptly to accommodate a broader array of user
instructions and technical demands. The study highlights the essential balance
between developing a tool that is accessible for beginners while being
sufficiently advanced to support more complex user requirements. Expanding
functionalities to include sophisticated web operations and comprehensive
multimedia management, along with providing educational examples or templates,
could help users better understand and utilize Aptly’s full range of
capabilities. This comprehensive feedback not only reflects the current
usability challenges faced by Aptly but also provides a strategic roadmap for
future enhancements to better serve a diverse user base for tool developers.

##  IV Related Work

Automatic programming [17, 18], with its rich historical tapestry, has always
been about bridging the communication gap between human intent and machine
execution. At its core, it embodies the principle of transforming higher-level
instructions into forms that a computer can readily understand and act upon.
Today’s automatic programming aims to streamline the software development
process, enabling developers to emphasize problem definition over intricate
implementation details. This evolution promises a future of enhanced
collaboration between humans and machines in software creation, making the
process more accessible, efficient, and resilient against errors.

Modern automatic programming systems leverage machine learning models to
generate code from a higher-level specification or even from natural language
descriptions [19]. This has broadened the scope of automatic programming from
simple code translation to complex code synthesis, opening up new
possibilities for software development [20, 21]. Take, for instance, GitHub
Copilot, a coding assistant powered by OpenAI Codex [22]. GitHub Copilot
offers contextually relevant code suggestions as you type, effectively
automating part of the coding process. Whether you’re learning a new language
or building a complex system, Copilot serves as a knowledgeable companion,
aiding in writing new code, navigating existing code, and even generating unit
tests [23, 24, 25, 26, 27, 28, 29, 30].

Aptly is an attempt to push such boundaries to automatically create block code
based on the user’s utterance. Text2App is a similar system based on natural
language processing to convert user input into a representation eventually
compatible with MIT App Inventor [31]. Such tools are moving toward a
direction where anyone can create functional software without any need for
traditional coding [32].

##  V Discussion & Conclusion

Overall, the design and implementation of tools like Aptly should inform how
LLMs may apply to other visual programming environments, whether they are
block based, flow based, or some other format. Our findings are most
applicable to other block based programming environments built on Google
Blockly, but the concepts should be translatable to other visual paradigms as
well. By designing Aptly to be like Python while maintaining a correspondence
to the original visual environment, we are able to leverage the depth of
knowledge of LLMs to speed system development. We were also surprised by how
well the LLMs were able to transform non-English instructions into working
code examples as none of our example prompts were expressed in any language
other than English. This suggests that we may see a much lower barriers to
entry in the future with regard to computer science education as LLMs gain
adoption.

In this work, we described the Aptly language, architecture, and server, and
how they form a single system alongside the MIT App Inventor visual
programming environment. We demonstrated how large language models are able to
synthesize programs in this new language without any training on a new corpus.
We further showed how edits to Aptly programs can be translated into
corresponding edits in App Inventor projects, enabling Aptly to act as a
collaborative programming agent. A small pilot study showed that students were
better able to articulate how they would leverage AI systems like Aptly after
working with the system, but that there was not a significant shift in their
overall interest in learning to program.

The advent of no-code platforms such as Aptly poses both opportunities and
challenges for computer science education, which merit careful exploration and
discussion. As this trend continues to revolutionize the software industry,
it’s becoming increasingly clear that our approach to computer science
education may need to evolve to keep pace with these changes [33]. Traditional
computer science education has long been centered on manual coding and
algorithm design. It emphasizes the need for students to understand the
intricacies of programming languages, the efficiency of algorithms, and the
design of complex software systems. AI-based code-generation techniques may
incentivize students to bypass such productive thinking. Future research on
students’ usage of such tools, what they learn while using them [34], and how
they feel about using these tools [35] will be active research.

## Acknowledgment

We thank the whole MIT App Inventor team who helped throughout the development
process, especially our staff and students including Evie Cai, Jacky Chen,
Mark Friedman, Maura Kelleher, Jasmine Shone, Joyce Yuan. Thank you also to
OpenAI and Drew Houston for their support and resources.

The second listing of Section II-C was generated using Aptly’s generative AI
capabilities.

## References

  * [1] P. M. Napoli and J. A. Obar, “The emerging mobile internet underclass: A critique of mobile internet access,” _The Information Society_ , vol. 30, no. 5, p. 323–334, oct 2014. [Online]. Available: https://doi.org/10.1080/01972243.2014.944726
  * [2] D. Wolber, H. Abelson, and M. Friedman, “Democratizing computing with app inventor,” _GetMobile: Mobile Comp. and Comm._ , vol. 18, no. 4, p. 53–58, jan 2015. [Online]. Available: https://doi.org/10.1145/2721914.2721935
  * [3] S. N. H. Mohamad, A. Patel, R. Latih, Q. Qassim, L. Na, and Y. Tew, “Block-based programming approach: challenges and benefits,” in _Proceedings of the 2011 International Conference on Electrical Engineering and Informatics_ , 2011, pp. 1–5.
  * [4] R. Bottino, A. Chioccariello, and L. Freina, _Computational Thinking in Primary School Through Block-Based Programming_. Singapore: Springer Singapore, 2020, pp. 147–166. [Online]. Available: https://doi.org/10.1007/978-981-15-4952-6_10
  * [5] M. Tissenbaum, J. Sheldon, and H. Abelson, “From computational thinking to computational action,” _Communications of the ACM_ , vol. 62, no. 3, pp. 34–36, 2019.
  * [6] S. Sarsa, P. Denny, A. Hellas, and J. Leinonen, “Automatic generation of programming exercises and code explanations using large language models,” in _Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1_ , ser. ICER ’22. New York, NY, USA: Association for Computing Machinery, 2022, p. 27–43. [Online]. Available: https://doi.org/10.1145/3501385.3543957
  * [7] H. Su, J. Ai, D. Yu, and H. Zhang, “An evaluation method for large language models’ code generation capability,” in _2023 10th International Conference on Dependable Systems and Their Applications (DSA)_ , 2023, pp. 831–838.
  * [8] X. Deng, “Group collaboration with App Inventor,” M. Eng. thesis, Massachusetts Institute of Technology, 2017.
  * [9] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, “Language models are few-shot learners,” in _Proceedings of the 34th International Conference on Neural Information Processing Systems_ , ser. NIPS ’20. Red Hook, NY, USA: Curran Associates Inc., 2020.
  * [10] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat _et al._ , “Gpt-4 technical report,” _arXiv preprint arXiv:2303.08774_ , 2023.
  * [11] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , ser. NIPS’17. Red Hook, NY, USA: Curran Associates Inc., 2017, p. 6000–6010.
  * [12] L. Reynolds and K. McDonell, “Prompt programming for large language models: Beyond the few-shot paradigm,” in _Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems_ , ser. CHI EA ’21. New York, NY, USA: Association for Computing Machinery, 2021. [Online]. Available: https://doi.org/10.1145/3411763.3451760
  * [13] J. Pennington, R. Socher, and C. Manning, “GloVe: Global vectors for word representation,” in _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , A. Moschitti, B. Pang, and W. Daelemans, Eds. Doha, Qatar: Association for Computational Linguistics, Oct. 2014, pp. 1532–1543. [Online]. Available: https://aclanthology.org/D14-1162
  * [14] A. Neelakantan, T. Xu, R. Puri, A. Radford, J. M. Han, J. Tworek, Q. Yuan, N. Tezak, J. W. Kim, C. Hallacy _et al._ , “Text and code embeddings by contrastive pre-training,” _arXiv preprint arXiv:2201.10005_ , 2022.
  * [15] K. Zhang and D. Shasha, “Simple fast algorithms for the editing distance between trees and related problems,” _SIAM journal on computing_ , vol. 18, no. 6, pp. 1245–1262, 1989.
  * [16] E. Patton, “A look at component usage in MIT App Inventor,” 2020. [Online]. Available: https://appinventor.mit.edu/blogs/evan/2020/12/20/component-usage-mit-app-inventor
  * [17] A. Solar-Lezama, “Program synthesis by sketching,” Ph.D. dissertation, University of California at Berkeley, USA, 2008, aAI3353225.
  * [18] D. Basin, Y. Deville, P. Flener, A. Hamfelt, and J. Fischer Nilsson, _Synthesis of Programs in Computational Logic_. Berlin, Heidelberg: Springer Berlin Heidelberg, 2004, pp. 30–65. [Online]. Available: https://doi.org/10.1007/978-3-540-25951-0_2
  * [19] A. Kumar and P. Sharma, “Open ai codex: An inevitable future?” _International Journal for Research in Applied Science and Engineering Technology_ , 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID:256889504
  * [20] D. M. Yellin, “The premature obituary of programming,” _Commun. ACM_ , vol. 66, no. 2, p. 41–44, jan 2023. [Online]. Available: https://doi.org/10.1145/3555367
  * [21] C. Bull and A. Kharrufa, “Generative artificial intelligence assistants in software development education: A vision for integrating generative artificial intelligence into educational practice, not instinctively defending against it,” _IEEE Software_ , vol. 41, no. 2, pp. 52–59, 2024.
  * [22] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba, “Evaluating large language models trained on code,” 2021.
  * [23] N. Nguyen and S. Nadi, “An empirical evaluation of github copilot’s code suggestions,” in _Proceedings of the 19th International Conference on Mining Software Repositories_ , ser. MSR ’22. New York, NY, USA: Association for Computing Machinery, 2022, p. 1–5. [Online]. Available: https://doi.org/10.1145/3524842.3528470
  * [24] P. Denny, V. Kumar, and N. Giacaman, “Conversing with copilot: Exploring prompt engineering for solving cs1 problems using natural language,” _Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1_ , 2022. [Online]. Available: https://api.semanticscholar.org/CorpusID:253157479
  * [25] A. Moradi Dakhel, V. Majdinasab, A. Nikanjam, F. Khomh, M. C. Desmarais, and Z. M. J. Jiang, “Github copilot ai pair programmer: Asset or liability?” _J. Syst. Softw._ , vol. 203, no. C, sep 2023. [Online]. Available: https://doi.org/10.1016/j.jss.2023.111734
  * [26] S. Imai, “Is github copilot a substitute for human pair-programming? an empirical study,” in _2022 IEEE/ACM 44th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion)_ , 2022, pp. 319–321.
  * [27] B. Yetistiren, I. Ozsoy, and E. Tuzun, “Assessing the quality of github copilot’s code generation,” in _Proceedings of the 18th International Conference on Predictive Models and Data Analytics in Software Engineering_ , ser. PROMISE 2022. New York, NY, USA: Association for Computing Machinery, 2022, p. 62–71. [Online]. Available: https://doi.org/10.1145/3558489.3559072
  * [28] S. Lunn, M. Marques Samary, and A. Peterfreund, “Where is computer science education research happening?” in _Proceedings of the 52nd ACM Technical Symposium on Computer Science Education_ , 2021, pp. 288–294.
  * [29] T. Rietz and A. Maedche, “Cody: An ai-based system to semi-automate coding for qualitative research,” _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ , 2021.
  * [30] R. Puri, D. S. Kung, G. Janssen, W. Zhang, G. Domeniconi, V. Zolotov, J. Dolby, J. Chen, M. Choudhury, L. Decker _et al._ , “Codenet: A large-scale ai for code dataset for learning a diversity of coding tasks,” _arXiv preprint arXiv:2105.12655_ , 2021.
  * [31] M. Hasan, K. S. Mehrab, W. U. Ahmad, and R. Shahriyar, “Text2app: A framework for creating android apps from text descriptions,” _arXiv preprint arXiv:2104.08301_ , 2021.
  * [32] M. Welsh, “The end of programming,” _Communications of the ACM_ , vol. 66, pp. 34 – 35, 2022.
  * [33] E. Klopfer, J. Reich, H. Abelson, and C. Breazeal, “Generative AI and K-12 Education: An MIT Perspective,” _An MIT Exploration of Generative AI_ , mar 27 2024, https://mit-genai.pubpub.org/pub/4k9msp17.
  * [34] M. Johnson, “Generative ai and cs education,” _Commun. ACM_ , vol. 67, no. 4, p. 23–24, mar 2024. [Online]. Available: https://doi.org/10.1145/3632523
  * [35] J. K. Eshraghian, “Human ownership of artificial creativity,” _Nature Machine Intelligence_ , vol. 2, no. 3, pp. 157–160, 2020.


different repetitions of the experiments, in terms of coefficient of
determination for regression and accuracy for classification. The _Finance_
dataset has been retrieved from Kaggle
(https://www.kaggle.com/datasets/dgawlik/nyse). The _Bankruptcy_ and
_Parkinson_ datasets have been retrieved from the UCI ML repository
(https://archive.ics.uci.edu/ml/datasets/Parkinson%27s+Disease+Classification,
https://archive.ics.uci.edu/ml/datasets/Polish+companies+bankruptcy+data).

The climate datasets is composed of continuous climatological features and a
scalar target which represents the state of vegetation of a basin of Po river.
The first one (_Climate, Climate(Class.)_) also considers the state of the
vegetation of neighbouring basins as inputs, while the second one is a more
difficult problem since it tries to predict it only from temperature and
precipitation features. This datasets have been composed by the authors
merging different sources for the vegetation index, temperature and
precipitation over different basins (see [8, 6, 33]), and they are available
in the repository of this work.

The two tables show the performances of the NonLinCFA considering five
different hyperparameters
($\epsilon\in\\{0.01,0.001,0.0001,0.00001,0.000001\\}$). The same holds for
the GenLinCFA algorithm, where the hyperparameters have been selected in order
to show some aggregations that are not too small (everything is aggregated) or
is too large (more than $50$ reduced features). The state of the art methods
applied have all been considered identifying the best validation performance
between $1$ and $50$ reduced features, when possible. The results, discussed
also in the main paper, show better or competitive results with respect to the
baselines.

Table 7: Experiments on climate datasets. The total number of samples $n$ has
been divided into train (66% of data) and test (33% of data) sets.

Quantity | Climate | Climate (Class.) | Climate II | Climate II (class.)
---|---|---|---|---
# samples $n$ | $981$ | $981$ | $867$ | $867$
# features $D$ | $1991$ | $1991$ | $2408$ | $2408$
Reduced Dimension |  |  |  |
NonLinCFA ($\epsilon=0.01$) | $7.4\pm 1.3$ | NA | $7.0\pm 0.9$ | NA
NonLinCFA ($\epsilon=0.001$) | $\mathbf{16.0\pm 0.8}$ | NA | $11.4\pm 0.9$ | NA
NonLinCFA ($\epsilon=0.0001$) | $19.4\pm 1.4$ | NA | $12.2\pm 0.6$ | NA
NonLinCFA ($\epsilon=0.00001$) | $19.6\pm 0.9$ | NA | $12.4\pm 0.7$ | NA
NonLinCFA ($\epsilon=0.000001$) | $19.6\pm 1.3$ | NA | $12.4\pm 0.7$ | NA
GenLinCFA ($\epsilon=\epsilon_{1}$) | $31.6\pm 10.9$ | $33.75\pm 11.5$ | $13.6\pm 4.8$ | $34.0\pm 6.1$
GenLinCFA ($\epsilon=\epsilon_{2}$) | $26.0\pm 9.1$ | $27.0\pm 8.2$ | $7.2\pm 1.1$ | $26.0\pm 6.1$
GenLinCFA ($\epsilon=\epsilon_{3}$) | $18.6\pm 5.4$ | $22.3\pm 5.7$ | $4.6\pm 0.9$ | $18.7\pm 5.4$
GenLinCFA ($\epsilon=\epsilon_{4}$) | $15.0\pm 4.3$ | $21.3\pm 5.2$ | $2.4\pm 0.4$ | $11.0\pm 4.5$
GenLinCFA ($\epsilon=\epsilon_{5}$) | $13.8\pm 3.0$ | $\mathbf{17.5\pm 3.3}$ | $2\pm 0$ | $2.5\pm 0.4$
LinCFA | $38.2\pm 1.6$ | NA | $222.0\pm 2.7$ | NA
PCA | $18.2\pm 0.7$ | $18.2\pm 0.7$ | $29.4\pm 0.1$ | $29.4\pm 0.4$
LDA | NA | $1$ | NA | 1
Kernel PCA | $35.6\pm 7.9$ | $27.0\pm 4.8$ | $\mathbf{21.8\pm 9.5}$ | $\mathbf{11.2\pm 2.3}$
Isomap | $2.6\pm 0.7$ | $12.6\pm 10.8$ | $42.4\pm 13.3$ | $16.6\pm 9.2$
LLE | $15.4\pm 13.8$ | $28.2\pm 12.9$ | $35.0\pm 10.7$ | $25.8\pm 13.1$
Supervised PCA | $41.8\pm 2.5$ | $37.0\pm 6.6$ | $7.4\pm 3.1$ | $13.8\pm 4.9$
NCA | NA | $24.2\pm 2.9$ | NA | $7.0\pm 0.6$
Test performance |  $\mathbf{R^{2}}$ score | Accuracy score |  $\mathbf{R^{2}}$ score | Accuracy score
NonLinCFA ($\epsilon=0.01$) | $0.8524\pm 0.0407$ | NA | $0.2949\pm 0.0156$ | NA
NonLinCFA ($\epsilon=0.001$) | $\mathbf{0.9395\pm 0.0125}$ | NA | $0.2547\pm 0.0182$ | NA
NonLinCFA ($\epsilon=0.0001$) | $0.9124\pm 0.0055$ | NA | $0.2541\pm 0.0121$ | NA
NonLinCFA ($\epsilon=0.00001$) | $0.9113\pm 0.0056$ | NA | $0.2529\pm 0.0148$ | NA
NonLinCFA ($\epsilon=0.000001$) | $0.9121\pm 0.0047$ | NA | $0.2530\pm 0.0146$ | NA
GenLinCFA ($\epsilon=\epsilon_{1}$) | $0.9194\pm 0.0039$ | $0.9068\pm 0.0037$ | $0.2439\pm 0.0246$ | $0.6820\pm 0.0199$
GenLinCFA ($\epsilon=\epsilon_{2}$) | $0.9226\pm 0.0026$ | $0.9075\pm 0.0029$ | $0.2841\pm 0.0051$ | $0.7127\pm 0.0159$
GenLinCFA ($\epsilon=\epsilon_{3}$) | $0.9207\pm 0.0052$ | $0.9056\pm 0.0039$ | $0.2764\pm 0.0069$ | $0.7094\pm 0.0181$
GenLinCFA ($\epsilon=\epsilon_{4}$) | $0.9269\pm 0.0012$ | $0.9062\pm 0.0036$ | $0.2493\pm 0.0125$ | $0.7061\pm 0.0105$
GenLinCFA ($\epsilon=\epsilon_{5}$) | $0.9275\pm 0.0004$ | $\mathbf{0.9107\pm 0.0022}$ | $0.2269\pm 0.0157$ | $0.6776\pm 0.0159$
LinCFA | $0.9007\pm 0.031$ | NA | $-1.2861\pm 0.2322$ | NA
PCA | $0.7536\pm 0.019$ | $0.8515\pm 0.0054$ | $0.1917\pm 0.0395$ | $0.6868\pm 0.0147$
LDA | NA | $0.7357\pm 0.0188$ | NA | $0.5526\pm 0.0224$
Kernel PCA | $0.7990\pm 0.0061$ | $0.8698\pm 0.0081$ | $\mathbf{0.3889\pm 0.0199}$ | $\mathbf{0.7640\pm 0.0062}$
Isomap | $0.1354\pm 0.0118$ | $0.6443\pm 0.0041$ | $0.3216\pm 0.0146$ | $0.7360\pm 0.0145$
LLE | $0.1149\pm 0.0139$ | $0.6444\pm 0.0052$ | $0.3102\pm 0.0367$ | $0.7456\pm 0.0140$
Supervised PCA | $0.8454\pm 0.0049$ | $0.8827\pm 0.0098$ | $0.3835\pm 0.0230$ | $0.7482\pm 0.0067$
NCA | NA | $0.8776\pm 0.0086$ | NA | $0.7638\pm 0.0051$

Table 8: Experiments on real world datasets. The total number of samples $n$
has been divided into train (66% of data) and test (33% of data) sets.

Quantity | Finance | Bankruptcy (class.) | Parkinson (class.)
---|---|---|---
# samples $n$ | $1299$ | $1084$ | $384$
# features $D$ | $75$ | $65$ | $753$
Reduced Dimension |  |  |
NonLinCFA ($\epsilon=0.01$) | $5.6\pm 0.4$ | NA | NA
NonLinCFA ($\epsilon=0.001$) | $7.2\pm 0.7$ | NA | NA
NonLinCFA ($\epsilon=0.0001$) | $7.4\pm 0.4$ | NA | NA
NonLinCFA ($\epsilon=0.00001$) | $7.4\pm 0.4$ | NA | NA
NonLinCFA ($\epsilon=0.000001$) | $\mathbf{7.4\pm 0.4}$ | NA | NA
GenLinCFA ($\epsilon=\epsilon_{1}$) | $15.2\pm 1.2$ | $27.6\pm 5.6$ | $53.4\pm 5.9$
GenLinCFA ($\epsilon=\epsilon_{2}$) | $8.0\pm 1.4$ | $16.4\pm 3.4$ | $26.4\pm 1.2$
GenLinCFA ($\epsilon=\epsilon_{3}$) | $4.8\pm 0.4$ | $9.8\pm 1.5$ | $23.4\pm 1.1$
GenLinCFA ($\epsilon=\epsilon_{4}$) | $4.2\pm 0.4$ | $8.2\pm 0.4$ | $20.2\pm 1.5$
GenLinCFA ($\epsilon=\epsilon_{5}$) | $3.8\pm 0.3$ | $7.0\pm 0.6$ | $14.2\pm 1.3$
LinCFA | $11.4\pm 0.7$ | NA | NA
PCA | $26.6\pm 0.4$ | $4.8\pm 0.9$ | $77.6\pm 3.8$
LDA | NA | $1$ | $1$
Kernel PCA | $36.0\pm 10.8$ | $13.8\pm 5.1$ | $32.4\pm 7.7$
Isomap | $27.2\pm 6.8$ | $21.2\pm 9.8$ | $14.8\pm 9.3$
LLE | $44.0\pm 7.3$ | $1$ | $39.6\pm 4.6$
Supervised PCA | $31.0\pm 13.8$ | $16.2\pm 7.5$ | $21.7\pm 3.4$
NCA | NA | $\mathbf{12.0\pm 9.5}$ | $28.2\pm 15.3$
Test performance |  $\mathbf{R^{2}}$ score | Accuracy score | Accuracy score
NonLinCFA ($\epsilon=0.01$) | $0.8131\pm 0.0032$ | NA | NA
NonLinCFA ($\epsilon=0.001$) | $0.8061\pm 0.0076$ | NA | NA
NonLinCFA ($\epsilon=0.0001$) | $0.8133\pm 0.0037$ | NA | NA
NonLinCFA ($\epsilon=0.00001$) | $0.8133\pm 0.0037$ | NA | NA
NonLinCFA ($\epsilon=0.000001$) | $\mathbf{0.8136\pm 0.0036}$ | NA | NA
GenLinCFA ($\epsilon=\epsilon_{1}$) | $0.8104\pm 0.0015$ | $0.7503\pm 0.0012$ | $0.7984\pm 0.0112$
GenLinCFA ($\epsilon=\epsilon_{2}$) | $0.8119\pm 0.0010$ | $0.7480\pm 0.0018$ | $0.7647\pm 0.0130$
GenLinCFA ($\epsilon=\epsilon_{3}$) | $0.8101\pm 0.0003$ | $0.7430\pm 0.0044$ | $0.8016\pm 0.0069$
GenLinCFA ($\epsilon=\epsilon_{4}$) | $0.8114\pm 0.0003$ | $0.7413\pm 0.0025$ | $0.7808\pm 0.0071$
GenLinCFA ($\epsilon=\epsilon_{5}$) | $0.8107\pm 0.0009$ | $0.7408\pm 0.0024$ | $0.7712\pm 0.0095$
LinCFA | $0.8010\pm 0.0128$ | NA | NA
PCA | $0.7559\pm 0.0027$ | $0.7413\pm 0.0037$ | $0.7840\pm 0.0117$
LDA | NA | $0.7587\pm 0.0065$ | $0.7632\pm 0.0489$
Kernel PCA | $0.7764\pm 0.0118$ | $0.7592\pm 0.0061$ | $0.7920\pm 0.0177$
Isomap | $0.2610\pm.0458$ | $0.7463\pm 0.0039$ | $0.7856\pm 0.0112$
LLE | $0.7281\pm 0.0267$ | $0.7402\pm 0$ | $0.7696\pm 0.0525$
Supervised PCA | $0.7731\pm 0.0126$ | $0.7508\pm 0.0018$ | $0.7913\pm 0.0069$
NCA | NA | $\mathbf{0.7637\pm 0.0079}$ | $0.7952\pm 0.0181$