Continual Self Supervised Learning through Distillation and Replay

Self-supervised learning aims to learn useful representations of input data without relying on human annotations. When trained offline with enormous amounts of unlabeled data, self-supervised models have been found to provide visual representations that are equivalent to or better than supervised models. However, in continual learning (CL) circumstances, where data is fed to the model sequentially, their efficacy is drastically diminished.

Project Overview

This study tackles the challenging problem of continually learning usable representations when input data arrives sequentially. By utilizing distillation and proofreading, the project aims to prevent severe forgetfulness and maintain model performance in continual learning scenarios.

Key Features

• Distillation: Adds a prediction layer that forces current representation vectors to match frozen learned representations.
• Proofreading with Memory: Implements an effective selection memory to revisit and refine previous data.
• Continual Learning: Enables the model to learn sequentially without significant performance degradation.

Applications

• Computer Vision: Improves representation learning for vision tasks in dynamic environments.
• Unlabeled Data: Leverages self-supervised learning to handle large-scale, unlabeled datasets effectively.

Skills and Technologies

• Self-Supervised Learning: Core methodology for representation learning.
• Continual Learning (CL): Focused on sequential data processing.
• Distillation: Ensures alignment of current and past representations.
• Machine Learning: Powers the overall learning and optimization process.

This project demonstrates the potential of combining distillation and replay techniques to address the challenges of continual self-supervised learning, paving the way for more robust and adaptive models.