Charles Townley was born on 1 October 1737 at Towneley Hall, the eldest in the family of three sons and a younger daughter of William Towneley (1714–1742) and his wife, Cecilia (1714–1777). The spelling Towneley is now usual for both the family and the place, but Charles signed himself Townley and this name is used for his entry in the Oxford Dictionary of National Biography.
He lived much of his life in London but regularly came back to Towneley. A leaflet The Charles Townley Trail was written in 2005 to complement an exhibition celebrating his life. It highlights his contributions to Towneley that can still be seen there. These include the Barcroft table, the Hayward fireplace and the Todmorden Bed.
Charles Townley rescued the Foldys Cross from Burnley churchyard, where it had been partly destroyed in 1789, and re-erected it at the back of the Hall. It was removed to its present position at the top of the Lime Avenue in 1911. He built a brew house and laundry in 1790 that still stands behind the new wing and today contains a local history museum.
However Charles Townley's main concern was that the whole landscape should form a picturesque view and he planted thousands of trees. One of the most notable planting still standing is the large cedar of Lebanon directly to the east of the house, near to the war memorial. Perhaps his greatest contribution was the creation of the woods at Thanet Lee, which today is the location of a sculpture trail where the sculptures have been carved from the wood of trees some of which he planted.