Skip to content

Unsupervised learning of action classes with continuous temporal embedding (CVPR'19)

License

Notifications You must be signed in to change notification settings

Annusha/unsup_temp_embed

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

42 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Unsupervised learning of action classes with continuous temporal embedding

Official implementation in python. https://arxiv.org/abs/1904.04189

If you use the code, please cite

@inproceedings{kukleva2019unsupervised,
  title={Unsupervised learning of action classes with continuous temporal embedding},
  author={Kukleva, Anna and Kuehne, Hilde and Sener, Fadime and Gall, Jurgen},
  booktitle={IEEE Conference on Computer Vision and Pattern Recognition (CVPR'19)},
  year={2019}
}

To train or reproduce numbers: HowTo

Pipeline for one activity class. Figure 1 in the paper.

alt text

Proposed pipeline for unsupervised learning with unknown activity classes. Figure 2 in the paper.

alt text

Visualization of embeddings via t-SNE on the 50Salads dataset

Each frame is color coded a) with the corresponding ground truth subaction label, b) with K assigned subaction labels after clustering as the second step in Fig.1 in our main paper, c)with the predicted labels after the decoding stage. The optimization of our network is performed with respect to relative timestep of each frame. In d) we show the respective relative time label in the continuous temporal embedding assigned to each frame feature. The color bar depicts that bright blue corresponds to 0 (startof the video) and pink to 1 (end of the video).

The number of subactions K

Breakfast dataset
Activity class name # subactions (K)
Coffe 7
Cereals 5
Tea 7
Milk 5
Juice 8
Sandwich 9
Scrambledegg 12
Friedegg 9
Salat 8
Pancake 14
YouTube Instractions dataset
Activity class name # subactions (K)
Changing tire 11
Making cofee 10
CPR 7
Jump car 12
Repot plant 8

Qualitative results

Breakfast dataset. The order of subactions: SIL, take bowl, pour cereals, pour milk, stir cereals, SIL

Breakfast dataset. The order of subactions: SIL, take cup, add teabag, pour water, SIL

Breakfast dataset. The order of subactions: SIL, spoon powder, pour milk, stir milk, SIL

Breakfast dataset. The order of subactions: SIL, take knife, cut orange, squeeze orange, pour juice, squeeze orange, pour juice, squeeze orange, pour juice, squeeze orange, pour juice, SIL

Breakfast dataset. The order of subactions: SIL, cut bun, smear butter, put toppingOnTop, SIL

50Salads dataset. The order of subactions: start, cut, place, cut, place, cut, place, cut, place, null, null, add oil, add pepper, mix dressing, end

50Salads dataset. The order of subactions: start, cut, place, cut, place, cut, place, peel cucumber, cut, place, mix ingredients, add oil, null, add pepper, null, mix dressing, serve salad onto plate, add dressing, end

About

Unsupervised learning of action classes with continuous temporal embedding (CVPR'19)

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages