I'm surprised by this after all this time, but I've run through a number of scenarios, and can recreate the problem.
If FastMode is set to True AFTER shapefile data is loaded into memory, AND the shapefile does not have M or Z data, then the individual Shape data is inadvertently cleared. This occurs in the IShapeWrapper::CopyTo method, called by CShape::put_FastMode.
The problem is that IShapeWrapper::CopyTo attempts to put parts and points, rather than insert parts and points, into the Shape copy. The part and point vectors are empty, and thus the put calls return False (but the return values are not checked).
As an example, if a Shapefile is simply Opened before adding it to the map, the data is not immediately loaded, and therefore the FastMode property can be set without any problem. However, if it is opened via AddLayerFromFilename, or by any other means is added to the map prior to setting FastMode, the data will be cleared. This also applies to Ogr layers, if the FastMode property is set after GetBuffer is called (whether implicitly or explicitly). There may be scenarios I have not observed (whether positive or negative), but the bug exists none-the-less.
I will submit a change to the CopyTo method, using InsertPart and InsertPointXY to add data to the shape copy.