Can I just replace brake pads and not rotors?

Yes, but it depends on the condition of the brake discs. Unless it is damaged beyond the thickness of the scrap or weakened, you can certainly change out worn bandages.