According to Microsoft, this error occurs when your workbook gets close to exceeding one of the built-in limitations of Excel. A workbook can contain only about 64,000 “custom formats.”
Custom formats are unique combinations of formatting attributes, applied to cells. For instance, let’s say that you have two cells, both formatted for 10-pt. Arial type. Both use the same number format, and neither of them has borders applied. If you change the font size of one of the cells to 11-pt., then you now have two unique custom formats. If you copy the format of the 11-pt. cell to a third cell, you still have only two custom formats. But if you add a border to one of the 11-pt. cells, you now have three custom formats.
A custom format is any unique combination of things like typefaces, font sizes, colours, alignment, borders, or other cell attributes. If you have a workbook that has, say, 100 complex worksheets, then it is easy to get close to the 64,000 custom-format limit.
There is only one way to correct the situation: Reduce the number of custom formats. Select a large block of cells in the workbook, display the Home tab of the ribbon, and then click Clear | Clear Formats (in the Editing group). If Excel doesn’t let you do this even (perhaps you have put the workbook into an unstable state by having way too many custom formats), then you should copy the contents of your worksheets, one by one, to a new workbook. Make sure you use Paste Special to paste only the formulas and not the formatting.