Since annelids are soft-bodied, their fossils are rare. Polychaetes' fossil record consists mainly of the jaws that some species had and the mineralized tubes that some secreted. Some Ediacaran fossils such as Dickinsonia in some ways resemble polychaetes, but the similarities are too vague for these fossils to be classified with confidence. The small shelly fossil Cloudina, from 549 to 542 million years ago, has been classified by some authors as an annelid, but by others as a cnidarian (i.e. in the phylum to which jellyfish and sea anemones belong). Until 2008 the earliest fossils widely accepted as annelids were the polychaetes Canadia and Burgessochaeta, both from Canada's Burgess Shale, formed about 505 million years ago in the early Cambrian. Myoscolex, found in Australia and a little older than the Burgess Shale, was possibly an annelid. However, it lacks some typical annelid features and has features which are not usually found in annelids and some of which are associated with other phyla. Then Simon Conway Morris and John Peel reported Phragmochaeta from Sirius Passet, about 518 million years old, and concluded that it was the oldest annelid known to date. There has been vigorous debate about whether the Burgess Shale fossil Wiwaxia was a mollusc or an annelid. Polychaetes diversified in the early Ordovician, about 488 to 474 million years ago. It is not until the early Ordovician that the first annelid jaws are found, thus the crown-group cannot have appeared before this date and probably appeared somewhat later. By the end of the Carboniferous, about 299 million years ago, fossils of most of the modern mobile polychaete groups had appeared. Many fossil tubes look like those made by modern sessile polychaetes  , but the first tubes clearly produced by polychaetes date from the Jurassic, less than 199 million years ago.

When did Cloudina exist?