Actually, we do know this by irrefutable scientific observation. From recorded information in both the ice, sea-floor, and rock strata. For example, there are tell-tale signs of glaciation. One of those signs is called glacial erratic and is typified by pieces of rock that differs from the size and type from rock native to the area in which it rests. They could only have gotten there by being transported by ice.
You can choose to ignore the evidence, but that doesn't make it speculative. It just makes you ignorant.
What has not been established is the cause for either the ice-ages, the prior warming periods, or the current warming period. Since this current warming began around 1850 it couldn't have anything do with humans. The fascist left want to blame humans so they can impose more government.
This current warming period is also not unprecedented. The last warming period was called the Medieval Warming and it occurred between 950 AD and 1250 AD, and it was warmer than today. Prior to that was the Roman Warming which occurred between 200 BC and 400 AD, and it was warmer than either the Modern or Medieval warming periods.
Sources:
Benchley, Peter J & DAT Harper (1998), Palaeoecology: Ecosystems, Environments and Evolution. Chapman & Hall, ISBN-10: 0412434504, ISBN-13: 978-0412434501.
Ward, Peter D (2002), Rivers in Time, Columbia University Press, ISBN-10: 0231118635, ISBN-13: 978-0231118637