Learning Blind Video Temporal Consistency (ECCV 2018)

Wei-Sheng Lai, Jia-Bin Huang, Oliver Wang, Eli Shechtman, Ersin Yumer, and Ming-Hsuan Yang

European Conference on Computer Vision (ECCV), 2018

[Project page][Paper]

Table of Contents

1. Introduction
2. Citation
3. Requirements and Dependencies
4. Installation
5. Dataset
6. Apply Pre-trained Models
7. Training and Testing
8. Image Processing Algorithms

Introduction

Our method takes the original unprocessed and per-frame processed videos as inputs to produce a temporally consistent video. Our approach is agnostic to specific image processing algorithms applied on the original video.

Citation

If you find the code and datasets useful in your research, please cite:

@inproceedings{Lai-ECCV-2018,
    author    = {Lai, Wei-Sheng and Huang, Jia-Bin and Wang, Oliver and Shechtman, Eli and Yumer, Ersin and Yang, Ming-Hsuan}, 
    title     = {Learning Blind Video Temporal Consistency}, 
    booktitle = {European Conference on Computer Vision},
    year      = {2018}
}

Requirements and dependencies

• Pytorch 0.4
• TensorboardX
• FlowNet2-Pytorch (Code and model are already included in this repository)
• LPIPS (for evaluation)

Our code is tested on Ubuntu 16.04 with cuda 9.0 and cudnn 7.0.

Installation

Download repository:

git clone https://github.com/phoenix104104/fast_blind_video_consistency.git

Compile FlowNet2 dependencies (correlation, resample, and channel norm layers):

./install.sh

Dataset

Download our training and testing datasets:

cd data
./download_data.sh [train | test | all]

For example, download training data only:

./download_data.sh train

Download both training and testing data:

./download_data.sh all

You can also download the results of [Bonneel et al. 2015] and our approach:

./download_data.sh results

Apply pre-trained models

Download pretrained models (including FlowNet2 and our model):

cd pretrained_models
./download_models.sh

Training and testing

Image Processing Algorithms

We use the following algorithms to obtain per-frame processed results:

Style transfer

• WCT: Universal Style Transfer via Feature Transforms, NIPS 2017
• Fast Neural Style Transfer: Perceptual Losses for Real-Time Style Transfer and Super-Resolution, ECCV 2016

Image Enhancement

• DBL: Deep Bilateral Learning for Real-Time Image Enhancement, Siggraph 2017

Intrinsic Image Decomposition

• Intrinsic Images in the Wild, Siggraph 2014

Image-to-Image Translation

• CycleGAN: Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks, ICCV 2017

Colorization

• Colorful Image Colorization, ECCV 2016
• Let there be Color!: Joint End-to-end Learning of Global and Local Image Priors for Automatic Image Colorization with Simultaneous Classification, Siggraph 2016