Your fever is dangerously high. A fever of 104 degrees is where the risk of brain damage and organ failure begins for adults, and though increases in fever tend to become exponentially rare anything above 101 is cause for concern.
If your fever holds at 102 or reaches 103 at all you should seek immediate medical attention. If it even spikes to 104 you are at high risk of losing consciousness, at which point preventing damage will obviously be difficult.
Take the OTC meds. I used to be against unnecessary use as well, and it got me pneumonia and the worst sickness of my life, which lasted over a month. Regulating illness is one of the benefits of modern society, and cracking down on small things as a matter of course makes large things less likely to develop.
As was said, you feel cold despite being warm due to temperature difference between your surface and core, as well as the fever playing hell with your thermoregulation sense. It's not unusual, and you aren't really at risk of raising your fever if you stay in blankets like a healthy person would. Don't wrap yourself into a heat sink, obviously, but there's nothing wrong with staying in bed. Remember, the fever is not being caused by the illness, it is "intentionally" triggered by your immune system in order to burn out the pathogen. It is still a thermostatic process, and so you don't "add" to it anymore than wearing a blanket normally adds to it. The issue is just that there's no upper bound for the fever response to run into before it goes so hard that it starts damaging important things. Like your liver.