Materiality as narrative: from cultural narrative to the production of thought

Working on architectural projects generally means starting with a preliminary analysis of the site or project. But why not start from the other end and begin with the materials used for the building? Does producing materiality, like a project or a site, mean constructing a cultural narrative that determines our perception of reality? Given that writing is itself a material that invents narratives, is there an underlying similarity between determining the materiality of architecture and writing a literary text?