Self-Supervised Learning

A training technique where a model learns from unlabeled data by generating its own supervisory signals. Imagine teaching yourself a new language by figuring out grammar rules from countless books without ever needing a teacher. The model predicts missing parts or future states of the data, forming a rich understanding before any human-provided labels are introduced. This method is crucial for pre-training large foundation models.