Well, now we get into the realm of why money has value at all. Labor gives money value. This is clear if you think what would happen if everybody stopped working. Goods would become scarce, therefore prices would skyrocket, which is the same thing as money losing most of it's value.
That's what Adam Smith thought, so you're in good company with that logic. Adam Smith's theories of economics weren't hindered by this perspective, though he did have to split the idea of price into "natural price" (the price you would expect from the labor put into the goods) and "market price" (the price we actually see in practice). Despite this, the whole labor theory of value seemed forced to some later economists, and I agree. I'll give a couple of examples to make the case that money derives its value directly from goods and services, and labor is merely one way we produce them.
Imagine that nobody is working, but each person has a vending machine in their room that dispenses, on a regular schedule, some kind of unique good that only that person has. Some people get pumpkin pies, some people get window cleaner, some people get iPhones. Each person in this community is given a fixed amount of money. They must trade amongst themselves to get the resources they need to survive. In this economy, there is no labor, but goods and money still have value. How can the absence of labor devalue anything if an economy still functions more or less normally when it is entirely absent?
If that seems too abstract and unrealistic, then consider two people. One is a brilliant artist who sits down once a day to spend fifteen minutes sketching out a beautiful drawing that she can take to the market and sell for a hundred dollars. The other is an aspiring artist who labors twelve hours a day, but produces a large volume of laboriously crafted work, but nobody wants to buy it, because the aesthetic quality is lower. You can see in this example that how hard they work is nothing but a footnote -- what matters is the goods themselves.*
The purpose of these examples is to show that the intuitive conclusion of your example -- that the absence of labor creates scarcity which drives up prices -- isn't necessarily tied to the absence of labor. The value of money is only related to goods and services, and is independent of how much labor went into those goods and services. If nobody works, prices don't go up because the absence of labor is depriving money of value, prices go up because the number of goods in the system is decreasing while the amount of money is held constant, so the amount of goods each dollar or euro or yen corresponds to is decreasing.
If you want to devalue money, all you have to do is print lots of it. If money isn't scarce, its value goes down, because the amount of money is disproportionate to the amount of goods in the economy. If you want to increase the value of money, then slowly drain it out of the system. The goods in the system are constant or increasing, but the amount of money is decreasing, so each unit of increasingly scarce currency corresponds to a larger and larger amount of goods.
* If the thought comes up that the first artist gets more money because she works less and therefore her works are more scarce, then what if she spends the same twelve hours painting a single picture instead of five minutes sketching one; intuitively, would she not then make substantially more money from her daily paintings than from her sketches? She can have the same labor as the other artist or much less, and it doesn't really matter -- only the quality of the product matters.