When you terraform a tile, the dig down action also alters the height of the 8 tiles around the one you are stood on.
Imagine a 3x3 grid with you stood on the center tile, when you dig down the 8 tiles around you are affected.
So if a ghost wall is on one of those tiles, then because the dirt has lowered , the wall is no longer supported.
Is this what you are seeing ?