For that matter, many ceramics could be characterized as crystalline (not all, though; glass is an amorphous ceramic). The catch, however, is that by this measure, most if not all metals could qualify as well. I suspect, somehow, that polycrystalline structures were very much not what was intended when the premise was given.
The problem, thus, should be from whence they got their start. A completely barren world, without wood or common metals, is the premise. From here, we are asked the likelihood of a "crystalline age" civilization emerging (by analogy to a stone, iron, or bronze age civilization). The absence of wood creates another interesting immediate premise. First, I assume by analogy to this and logically following the statement of a "barren" world, we should disregard any plant that may be strong enough to serve by itself as a construction material, of which woody plants are but a subset (bamboo is an example that springs immediately to mind, but I believe others exist, or like banana trees, may be so utilized). As such, we can expect very few shrubs or trees akin to those on Earth; even large herbaceous plants such as banana trees could be used for construction if they existed. However, the issue raised is very simple - how did these people ever harness fire? Grasses don't burn very well, nor do ferns. Ceramics and glass both require fire, or more precisely, the ability to harness and control significant amounts of thermal energy, of which fire is one of the easiest methods.
That's not to say it's impossible. Perhaps this world has some alternate source of fuel (maybe it was once highly-seeded in plant life, and is thus very rich in fossil fuels; the use of lignite as fuel is separated from the birth of Christ by almost a thousand years more than separates Christ from us). Perhaps the problem can be bypassed: again ignoring the idea of intelligent protozoa, the growth of these crystals may be a consequence of some biological process, either on the part of the sapient species itself or on the part of some animal they have domesticated. For instance, humans make milk, but the vast majority of our milk production comes from domesticated animals. Bone was and is a material used today for crafts, art, and small goods. If, for instance, you had some massive animal that grew an equally-massive crystalline, semi-ceramic shell (perhaps the barren world is your archetypal "desert world," more to the point with a large amounts of free silicates that an animal can easily absorb/consume from its surroundings and utilize in its growth), this shell could be used as a building material. If it also regularly moults as it grows, you wouldn't even need to kill the animal to obtain the shell; you just take care of it and keep feeding it until it sheds its shell, then use the detritus to build your own house.
But, I suspect what we're talking about is not the early days of this species' paleolithic-equivalent. From the talk and statements on what has been ruled out, I have this vague feeling we're talking Crystal Spires-inspired architecture, something like the
Emerald City of Oz fame. That is to say, it's not that the materials used are necessarily actually crystalline in terms of molecular structure (as iron, for instance, would be), but rather that it rather has to look crystalline. Is this an accurate feeling?