Automatic melody generation for pop music has been a long-time aspiration for
 both AI researchers and musicians. However, learning to generate euphonious
 melody has turned out to be highly challenging due to a number of factors. Representation
 of multivariate property of notes has been one of the primary challenges.
 It is also difficult to remain in the permissible spectrum of musical variety, outside
 of which would be perceived as a plain random play without auditory pleasantness.
 Observing the conventional structure of pop music poses further challenges.
 In this paper, we propose to represent each note and its properties as a unique
 ‘word,’ thus lessening the prospect of misalignments between the properties, as
 well as reducing the complexity of learning. We also enforce regularization policies
 on the range of notes, thus encouraging the generated melody to stay close
 to what humans would find easy to follow. Furthermore, we generate melody
 conditioned on song part information, thus replicating the overall structure of a
 full song. Experimental results demonstrate that our model can generate auditorily
 pleasant songs that are more indistinguishable from human-written ones than
 previous models. Recent explosion of deep learning techniques has opened up new potentials for various fields of multimedia. Vision and language have been its primary beneficiary, particularly with rising interest in generation task. Considerable amount of recent works on vision and language have hinged beyond mere generation onto artistic aspects, often producing works that are indistinguishable from human works BID2 ; BID14 ; BID13 ). On the other hand, it is only recently that deep learning techniques began to be applied to music, and the quality of the results are yet far behind those in other domains, as there are few works that demonstrate both euphonious sound and structural integrity that characterize the human-made musical contents. This unfortunate status holds true for both music in its physical audio format and its abstraction as notes or MIDI (Musical Instrument Digital Interface).Such lagging of deep learning-enabled music generation, particularly in music as abstraction, can be attributed to a number of factors. First , a note in a musical work contains various properties, such as its position, pitch, length, and intensity. The overall tendency of each property and the correlation among them can significantly vary depending on the type of music, which makes it difficult to model. Second , the boundary between musical creativity and plain clumsiness is highly indefinite and difficult to quantify, yet exists. As much as musical creativity cannot be limited, there is yet a certain aspect about it that makes it sound like (or not sound like) human-written music. Finally , music is not merely a series of notes, but entails an overall structure of its own. Classical music pieces are well-known for their high structural complexity, and much of pop music follows the general convention of verse -pre-chorus -chorus structure. This structure inevitably necessitates different modeling of musical components; for example, notes in the chorus part generally tend to be more high-pitched. It goes without saying that these structure-oriented variations further complicate the modeling of music generation.In this paper, we propose a new model for music generation, specifically symbolic generation of melodies for pop music in MIDI format. The term "pop music " can have different meanings depending on the context, but we use the term in this paper to refer to its musical characteristics as conventionally accepted. Specifically, it refers to the songs of relatively short lengths, mostly around 3 minutes, with simple and memorable melodies that have relatively low structural complexity, especially in comparison to classical music. Music in MIDI format (or , equivalently, in notes) can be considered a discrete abstraction of musical sound, analogous to the relationship between text and speech. Just as understanding text is not only essential in its own merit, but provides critical clues to speech and language in general, understanding music at its abstraction can provide an ample amount of insights as to music and sound as a physical format, while being fun and significant per se.We address each of the challenges described above in our proposed model. First, we propose to treat a note and its varying properties as a unique 'word,' as opposed to many previous approaches that took each property into consideration separately, by implementing different layers for generation. In our model, it suffices to train only one model for generation, as each 'word' is an incarnation of all of its properties, thus forming a melody as a 'sentence' consisting of those notes and the properties. This approach was inspired by recent successes in image captioning task BID9 ; BID19 ; BID20 ), in which a descriptive sentence is generated with one word at a time in a recurrent manner, while being conditioned on the image features. Likewise, we generate the melody with one note at a time in a recurrent manner. The difference is that, instead of image features obtained via convolutional neural networks (CNN), we condition the generation process on simple two-hot vectors that contain information on chords sequences and the part within the song. Chord sequences and part annotations are automatically generated using multinomial hidden markov model (HMM) whose state transition probabilities are computed from our own dataset. Combining Bayesian graphical models with deep neural netweorks (DNN) has become a recent research interest BID1 ), but our model differs in that HMM is purely used for feature input generation that is processed by neural networks.Second, we enforce regularization policy on the range of notes. Training with a large amount of data can lead to learning of excessively wide range of pitches, which may lead to generation of melodies that are not easy to sing along. We alleviate these problem by assigning a loss function for the range of notes. Finally, we train our system with part annotation, so that more appropriate melody for the corresponding part can be generated, even when the given chord sequences are identical with other parts of the song. Apart from the main model proposed, we also perform additional experiments with generative adversarial networks BID2 ) and with multi-track songs.Our main contributions can be summarized as following:• proposal of a model to generate euphonious melody for pop music by treating each note and its properties as single unique "word", which alleviates the complexity of learning • implementation of supplementary models, such as chord sequence generation and regularization, that refine the melody generation • construction of dataset with chord and part annotation that enables efficient learning and is publicly available. Being able to automatically describe the content of an image using properly formed English sentences is a very challenging task, but it could have great impact, for instance by helping visually impaired people better understand the content of images on the web. This task is significantly harder, for example, than the well-studied image classification or object recognition tasks, which have been a main focus in the computer vision community [27] . Indeed, a description must capture not only the objects contained in an image, but it also must express how these objects relate to each other as well as their attributes and the activities they are involved in. Moreover, the above semantic knowledge has to be expressed in a natural language like English, which means that a language model is needed in addition to visual understanding.Most previous attempts have proposed to stitch together FIG0 . NIC, our model, is based end-to-end on a neural network consisting of a vision CNN followed by a language generating RNN. It generates complete sentences in natural language from an input image, as shown on the example above.existing solutions of the above sub-problems, in order to go from an image to its description [6, 16] . In contrast, we would like to present in this work a single joint model that takes an image I as input, and is trained to maximize the likelihood p(S|I) of producing a target sequence of words S = {S1, S2, . . .} where each word St comes from a given dictionary, that describes the image adequately.The main inspiration of our work comes from recent advances in machine translation, where the task is to transform a sentence S written in a source language, into its translation T in the target language, by maximizing p(T |S). For many years, machine translation was also achieved by a series of separate tasks (translating words individually, aligning words, reordering, etc), but recent work has shown that translation can be done in a much simpler way using Recurrent Neural Networks (RNNs) [3, 2, 30] and still reach state-of-the-art performance. An "encoder" RNN reads the source sentence and transforms it into a rich fixed-length vector representation, which in turn in used as the initial hidden state of a "decoder" RNN that generates the target sentence.Here, we propose to follow this elegant recipe, replacing the encoder RNN by a deep convolution neural network (CNN). Over the last few years it has been convincingly shown that CNNs can produce a rich representation of the input image by embedding it to a fixed-length vector, such that this representation can be used for a variety of vision Section 4.3, our model can generate simultaneous notes for a single instrument. BID5 also take a similar approach of applying Gibbs sampling to generate Bach-like chorale music, but mostly share the same drawbacks that make a contrast to our model. BID8 proposed RL Tuner to supplement recurrent neural networks with reinforcement learning by imposing cross-entropy reward function along with off-policy methods from KL control. Note RNN trained on MIDI files is implemented to assign rewards based on the log probability of a note given a melody. They defined a number of music-theory based rules to set up the reward function. Our model, on the other hand, does not require any pre-set rules, and the outcome can be easily controlled with simple regularizations. BID0 proposed a hierarchical recurrent neural network model to produce multi-track songs, where the bottom layers generate the melody and the higher levels generate the drums and chords. They built separate layers for pitch and duration that generate an output at each time step, whereas our model needs only one layer for pitch and duration and does not have to be aware of time step. They also conditioned their model on scale types, whereas we condition our model on chord sequence and part information.While generating music as physical audio format is out of scope of this paper, we briefly discuss one of the recent works that demonstrated promising results. Originally designed for text-to-speech conversion, WaveNet (van den Oord et al. (2016) ) models waveform as a series of audio sample x t conditioned on all previous timesteps, whose dependence is regulated by causal convolutional layers that prevent the violations in ordering. When applied to music, it was able to reconstruct the overall characteristics of corresponding music datasets. While only for a few seconds with frequent inconsistency, it was able to generate samples that often sound harmonic and pleasant.3 GENERATION MODEL Although our model was inspired by the model used in image captioning task, its task objective has a fundamental difference from that of image captioning. In image captioning task, more resemblance to human-written descriptions reflects better performance. In fact, matching human-written descriptions is usually the evaluation scheme for the task. However, in melody generation, resembling human-written melody beyond certain extent becomes plagiarism. Thus, while we need sufficient amount of training to learn the patterns, we also want to avoid overfitting to training data at the same time. This poses questions about how long to train, or essentially how to design the loss function. We examined generations with parameters learned at different epochs. Generated songs started to stay in tune roughly after 5 epochs. However, after 20 epochs and on, we could frequently observe the same melodies as in the training data, implying overfitting (check our demo). So there seems to exist a 'safe zone' in which it learns enough from the data but not exceedingly to copy it. Previous approaches like BID8 have dealt with this dilemma by rewarding for following the pre-determined rules, but encouraging off-policy at the same time. Since we aim for learning without pre-determined rules, alternative would be to design a loss function where matching the melody in training data over n consecutive notes of threshold is given penalty. Designing a more appropriate loss function remains as our future work. On the other hand, generating songs with parameters obtained at different stages within the 'safe zone' of training leads to diversity of melodies, even when the input vectors are identical. This property nicely complements our relatively low-dimensional input representation. In this paper, we proposed a novel model to generate melody for pop music. We generate melody with word representation of notes and their properties, instead of training multiple layers for each property, thereby reducing the complexity of learning. We also proposed a regularization model to control the outcome. Finally, we implemented part-dependent melody generation which helps the generated song preserve the overall structure, along with a publicly available dataset. Experimental results demonstrate that our model can generate songs whose melody sounds more like human-written ones, and is more well-structured than previous models. Moreover, people found it more difficult to distinguish the songs from our model from human-written songs than songs from previous models. On the other hand, examining other styles such as music of minor scale, or incorporating further properties of notes, such as intensity or vibrato, has not been examined yet, and remains as future work. As discussed in Section 4, learning to model the correlations among different instruments also remains to be done, and designing an appropriate loss function for the task is one of the most critical tasks to be done. We plan to constantly update our dataset and repository, addressing the future works.
We propose a novel model to represent notes and their properties, which can enhance the automatic melody generation. This paper proposes a generative model of symbolic (MIDI) melody in western popular music which jointly encodes note symbols with timing and duration information to form musical "words". The paper proposes to facilitate generation of melody by representing notes as "words", representing all of the note's properties and thus allowing the generation of musical "sentences".