Brainstorm does not include any iterative dipole fitting algorithm. The dipole modeling option is not an iterative method, it is documented here:
https://neuroimage.usc.edu/brainstorm/Tutorials/TutDipScan
For iterative dipole fitting, we rely on FieldTrip's function ft_dipolefitting.m. For understanding what this function does, I recommend you read directly the code of the function:
https://neuroimage.usc.edu/brainstorm/Tutorials/DipoleFitting
fieldtrip/ft_dipolefitting.m at master · fieldtrip/fieldtrip · GitHub