One should always be aware of one's shortcomings. I, for example, use the word "actually" too much and am utterly incapable of caring for a houseplant. I also have a forehead-slappingly stupid habit of seeing an interesting sight, commenting - to myself or even to someone else - that I should find out what it is or at least snap a picture, and then continuing on my way, doing nothing, and wondering for hours/days/weeks what on Earth that thing was, anyway. As a result I have a growing list of mystery places/structures/objects, invariably located somewhere I'll have no cause to go back to for quite some time, awaiting definition in my mind.
The answer is that this is (or was) the Woodruff house, and it's neither a part of Guilfords early settlement history nor a meaningless condemned property. In fact, it's a Thing all on its own. (I use "thing" too much, too.) The Woodruff in question was Rollin Woodruff, Governor of Connecticut in the early 1900s. The house was an Adirondak-style summer home, with a formal sunken garden. Later, the property became a dairy farm. The house was burned in 2000, and the town bought the land in 2003. There are committees and plans and proposals for the site, but as far as I can tell, the actual (see?) activity, or lack thereof, has been surprisingly New London-ish.
So for now it stands, not only the chimney as I'd remembered it but the stone foundation of the house as well. It looks a bit vulnerable, yet permanent, like a natural part of this landscape where downtown business district bleeds into farmland. It looks like it belongs.