Don't Watch the Loss. Watch the Samples. #ai #tech

What I'd do differently, in one clip. For a 28-clip dataset at rank 32: stop somewhere between 1750 and 2500 steps. And don't watch the loss. Watch samples - and watch them for the right things. Can it still move? Does it still do what the prompt says? Those are the two things that break first, and neither of them shows up in a loss curve. Mine fell for 164 hours while the model got steadily worse. The full experiment, 32 checkpoints deep: https://youtu.be/w_7EpghF8fw #LoRA #AITraining #MachineLearning #StableDiffusion #DeepLearning