DeepWriting: Making Digital Ink Editable via Deep Generative Modeling

authors: E. Aksan, F. Pece, O. Hilliges
publication: ACM SIGCHI, Montréal, Canada, April 2018
Teaser-Picture

We propose a novel generative neural network architecture that is capable of disentangling style from content and thus makes digital ink editable. Our model can synthesize handwriting from typed text while giving users control over the visual appearance (A), transfer style across handwriting samples (B, solid line box synthesized stroke, dotted line box reference style), and even edit handwritten samples at the word level (C).




Abstract

Digital ink promises to combine the flexibility and aesthetics of handwriting and the ability to process, search and edit digital text. Character recognition converts handwritten text into a digital representation, albeit at the cost of losing personalized appearance due to the technical difficulties of separating the interwoven components of content and style. In this paper, we propose a novel generative neural network architecture that is capable of disentangling style from content and thus making digital ink editable. Our model can synthesize arbitrary text, while giving users control over the visual appearance (style). For example, allowing for style transfer without changing the content, editing of digital ink at the word level and other application scenarios such as spell-checking and correction of handwritten text. We furthermore contribute a new dataset of handwritten text with fine-grained annotations at the character level and report results from an initial user evaluation.




Video



Dataset

We collected data from 94 authors by using IAMOnDB corpus. After discarding noisy samples of IAMOnDB, we compiled a dataset of 294 authors, fully segmented. You can find the link to our training and validation splits below. If you use our data, we kindly ask you to cite our work, and also fill IAMOnDB's registration form and follow their citation requirements since our dataset extends IAMOnDB.

Citation

@inproceedings{Aksan:2018:DeepWriting,
	author = {Aksan, Emre and Pece, Fabrizio and Hilliges, Otmar},
	title = {{DeepWriting: Making Digital Ink Editable via Deep Generative Modeling}},
	booktitle = {SIGCHI Conference on Human Factors in Computing Systems},
	series = {CHI '18},
	year = {2018},
	location = {Montr{\'}eal, Canada},
	publisher = {ACM},
	address = {New York, NY, USA},
}