There are no set in stone steps to repro this - it varies by mapframe settings and source files.
I can give you very rough steps to repro this - but again like trying to figure anything out in TB's mapframe properties - it is pot luck.
Generate a hieghtfield that is suitable to be imported into TB
Create satellite & Surface masks at 10/5/2/1m/pixel resolutions relative to your terrain dimensions.
Try and get TB to Generate Layers for each of the above resolutions - compile the succesful layer generations into a working map
Load them up in-game to check for layer mask errors.
This is such an annoying bug due to the unhelpful nature of TB (vague error messages), the sheer amount of time involved in generating the data required for import only for TB to break itat the layer generation stage, when it should not generate tiles like this in the first place - it should either refuse to generate or at least point to where the error is occurring.