Unsupervised text style transfer is the task of re-writing text of a given style into a target style without using a parallel corpus of source style and target style sentences for training. Style transfer systems are evaluated on their ability to generate sentences that 1) possess the target style, 2) are fluent and natural sounding, and 3) preserve the non-stylistic parts (content) of the source sentence. We train a reinforcement learning (RL) based unsupervised style transfer system that incorporates rewards for the above measures, and describe novel rewards shaping methods for the same. Our approach does not attempt to disentangle style and content, and leverages the power of massively pre-trained language models as well as the Transformer. Our system significantly outperforms existing state-of-art systems based on human as well as automatic evaluations on target style, fluency and content preservation as well as on overall success of style transfer, on a variety of datasets. Text style transfer is an important natural language generation problem, since it has wide applications across different domains. It has been used to adapt texts to specific artistic writing styles (Jhamtani et al., 2017) , make texts formal or informal (Rao & Tetreault, 2018) , alter sentiment 1 (Hu et al., 2017; Shen et al., 2017; Fu et al., 2018; , rewrite factual sentences into romantic or humorous ones , generate poetry (Yang et al., 2018a) , personalize dialogue systems (Zhou et al., 2017) and obfuscate gender in social media posts (Reddy & Knight, 2016) . Most recent works perform unsupervised style transfer due to the unavailability of parallel style corpora. Most previous works on unsupervised style transfer attempt to disentangle the stylistic parts (hereby, 'attributes') and non-stylistic parts (hereby, 'content') of texts, and then modify the attributes while preserving the content. Some of these works encode style and content in separate latent representations, and decode the style-dependent output from these representations (Fu et al., 2018; Shen et al., 2017; Hu et al., 2017; Yang et al., 2018b) . A few others explicitly identify attribute and content words from input texts and then train models to generate target style sentences from the content Wu et al., 2019c ). More recently, Lample et al. (2018b) showed that many previous works that attempt to disentangle content and style in latent representation spaces are unsuccessful in doing so in practice. Further, these approaches are prone to instability of training, low sample efficiency and consequently poor quality outputs. Approaches where attribute words are explicitly removed from the sentence require heuristics and thresholds to decide attribute words, which makes them sensitive to and require manual setting of thresholds. This causes core content words to be incorrectly deleted in some cases, and source attribute words to be incorrectly preserved in others. Moreover, in some of these works Wu et al., 2019b) , the final output generators are provided with only the content information of the input sentence and not the attributes. This leads to awkward outputs where the model inserts target attribute words that are either not suitable to the content, or are wrongly positioned. However, it has been observed that these approaches are more controllable, easier to train and produce better quality outputs than approaches based on latent representations. A few recent works (Lample et al., 2018b; Dai et al., 2019; Luo et al., 2019) avoid style-content disentanglement altogether. While most works use recurrent networks for encoding and decoding, transformers (Vaswani et al., 2017) have been shown to be significantly better for the task (Dai et al., 2019; Wu et al., 2019b) . show significant gains over previous state of the art by leveraging the combined power of transformers and massively pre-trained language models, by using a decoder-only GPT (Radford et al.) . In doing so, they do away with the traditional encoderdecoder mechanism. Finally, RL has been used in previous works to leverage the use of non-differentiable training objectives and overcome the lack of parallel corpora. Xu et al. (2018) use a cycled RL approach where a neutralization model first disentangles the content and attributes, and then passes the content to an emotionalization model. However, cascading errors propagated from the neutralization to the emotionalization due to a discretization of embeddings to words in between the two, leads to poor quality outputs. Gong et al. (2019) use adversarially trained discriminators whose feedbacks are used as rewards by a generator, and Luo et al. (2019) train a dual RL system wherein separate models exist for source-to-target prediction and target-to-source prediction. Style and content rewards are built into this dual structure. However their model tends to be majority-biased towards certain attributes (such as 'happy' and 'loved' on the task of sentiment transfer), which are abruptly inserted in the output sentences without being meaningfully transferred versions of the source attributes. For instance, one would not find it meaningful for the source sentence 'so i asked for the card to be refunded' to be mapped to the target sentence 'so i loved the credit card to be happy'. Taking into consideration the drawbacks and strengths of previous works, our contributions are as follows: we introduce a novel RL based model for style transfer that 1) uses the decoder-only GPT (Radford et al.) in order to leverage the power of transformers and massively pre-trained language models, 2) directly learns mappings from source to target sentences without any disentanglement, 3) does not require any parallel corpus but is instead warm-started by using a synthetic parallel corpus generated by the trained GST and 4) provides for controllable generation by allowing trade-offs between style, content retention and fluency. Our approach significantly outperforms current state-of-art systems based on human evaluation as well as on evaluations using automatic metrics. In the interest of reproducibility, we publish all our code, data and results for this work on our Github repository, the link to which will be added here in the camera-ready version if accepted. As has been observed by most previous works, the automatic evaluations in Table 1 show that many previous works trade-off target style match and content retention. It is easy to achieve very high numbers on either of AC or BL R . A model that simply copies the input sentence will achieve high BL R and a model that simply chooses a random target training sentence will achieve high AC score. SE has very low AC but considerably high BL R while BT has a high AC but considerably low BL R . However, RL-ST achieves very high target style accuracy but not at the cost of content retention -it achieves considerably good BL R scores too. HM and GM scores indicate how well the models perform on both style and content. Our model (RL-ST) ranks highest on both these scores across datasets (except on HM for CAPTIONS, where it ranks the second highest) as well as achieves low PL, outperforming even the average scores of all the human references on HM, GM and PL on YELP. However, automatic metrics do not capture nuances that human evaluation does. For instance, on the Yelp dataset, DRL is biased towards frequently using the attributes 'loved' and 'happy' in its positive outputs, and PTO over-uses the word 'delighted' even in sentences where it is not meaningful to. The classifier still awards these outputs high AC scores. Further, model-based metrics such as AC and PL are also sensitive to the quality of their training data available. From human evaluations in Table 2 , we see our model outperforms previous state-of-art models by a good margin on all metrics across all datasets. On the Overall scores (All), we outperform previous state-of-the art models by 19.8%, 24.5% and 19% on YELP, GYAFC and CAPTIONS respectively, averaging across the top performing models considered for human evaluation in Table 2 for each of these datasets. From manual inspection we observe that RL-ST performs better than previous state-of-art models in the following ways: 1) generates sentences that are more natural sounding, We present an RL-based, sample efficient style transfer model that outperforms current state-of-art systems on human as well as automatic evaluations. The approach is generalizable across a variety of style transfer tasks, as we show with diverse datasets. We show the merits of directly learning to map source to target sentences without disentanglement, shaping RL rewards efficiently, and leveraging the power of massively pre-trained transformer-based language models. in each block. All internal states (keys, queries, values, word embeddings, positional embeddings) are 768-dimensional. Input text is tokenized using Byte-Pair Encoding (BPE). In equation 9, λ S = 1, λ C = 0.3 and λ F = 1 for all 3 datasets.
A reinforcement learning approach to text style transfer Introduces an RL-based method which leverages a pre-trained language model to transfer text style, without a disentanglement objective, while using style-transfer generations from another model. The authors propose a combination reward composed of fluency, content, and style for text style transfer.