If you are already working outside of the interface, you can call the function bst_scout_value from your script.
The parameter you are interested in is “XyzFunction”, which defines how the values for the 3 orientations are combined into one value per vertex. Only three options are available: ‘norm’, ‘pca’, ‘none’.
Norm takes (x^2+y^2+z^2) at each time point, PCA takes the first mode of an SVD decomposition.
Do you mean spatial or temporal smoothing? It can be interesting to smooth a bit the individual source maps on the cortex surface before doing any group analysis, so that there is a higher chance of overlap between subjects (process Sources > Spatial smoothing).
There is no function to estimate the smoothness of data.