Edit: FYI if you dig into the mythical origins of dwarves, there was no indication that they were short. The term dwarf was a rough translation of "minor god" (of crafting) which somehow turned "minor" into "small". Most of the modern tropes were already in place at the time that Tolkein and Lewis wrote their fantasy novels (well before Gygax), though their differences indicate that the issue of female dwarves was not yet settled.
^Yes, actually almost all pagan critters and characters were made smaller and uglier during the reign of Christianity to try to stop people from thinking that Paganism was cool. Thus mythology became fair tale became children's stories. I've heard that Leprechauns were supposedly once human sized or even ogre sized and represented god or angel like creatures that were shrunken down and forced into hiding by the power of Christianity.
In fact the Norse Dvergar are surprisingly metal. The Norse likened them to corpses, living underground, they had deathly pale skin and eyes like glowing coals and could stand motionless for days until something grabbed their attention. To a modern reader, they would resemble vampires more than dwarves. But the Dvergar were also skilled craftsdwarves, forging Thor's hammer Mjollnir and other artifacts for the Aesir. When the gods couldn't build a chain strong enough to bind Fenrir, they barted with the Dvergar for the unbreakable ribbon Gleipnir, made of the sound of a cat's feet, the beard of a woman, the roots of a mountain, the sinews of a bear, the breath of fish and the spittle of a bird.
Britain was home to another sinister species of dwarf, known as the Redcap, Powrie or Dunter. Redcaps murder travels and die their hats red with blood. They have to keep this up, for if their hats dry out, the Redcap dies. Redcaps are fleet of foot, and are famous for wearing iron shoes and wielding pikes as their weapon of choice.
The mythical roots of Goblins, Orcs and Ogres are less specific. The word for goblin or gobelin is derived from the Greek creature Kobalos, which is also the root for kobolds. I've never read an authentic tale of Kobaloi, but they're described as mischief makers, rogues and jokers, but potentially vicious. I think one hangs out with Dionysus, the party god. One account I've heard describes the Kobaloi as insubstantial spirits rather than physical entities. Possibly similar to European trickster spirits like the Brownie, Puck, Pixie, Bogart and yes, the Kobold. Speaking of Kobolds, their fairy tales make them sound like a cross between dwarf and goblin. I used to think of Kobolds as bad dwarves, rather than rats or lizards or whatever, but that's probably Castle of the Winds' doing.
"Orc" was an Old English abbreviation of Orcus, an underworld god similar to Hades or Hel. The word became unspecific as the god's cult died out, like saying "monster" or "demon", until Tolkien used the word in his books. "Ogre" comes from medieval French, and is also possibly related to Orcus, or to the "Giants" "Jotuns" "Iotuns" "Ettins" or to "Orpheus" or "Og" or any number of things. Ogres are usually associated with cannibalism.
The Norse version of Ogres, the Troll, was a lot more interesting before Gygax and Tolkien touched them. Trolls were usually largely built, but not necessary ugly or stupid. Trolls were also associated with magic, in fact the modern Swedish word for wizard or magician is trollkarl. Troll-women were also prevalent in stories and kennings. Kormak uses the kenning "troll-woman's fair breeze" to mean "mind". I've never understood the relation between Trolls and Jotnar, Jotuns. The Jotnar are more like demons or even gods in their own right. Some take the form of hideous monsters, while others (Gerd) are wildly attractive and intermarry with the gods. They are called "giants" though their size is never described nor is it particularly important to any of the myths.
This may not resonate with everyone, but there a lot of translations for Eastern mythology describe miscellaneous creatures as "goblins" and "ogres". Japanese Oni have had a strong impact on the appearance of "ogres" and some types of Tengu are goblin-like. In China, "ogres" work as part of the underworld bureaucracy, using their fearful presence to keep misbehaving ghosts or goblins in line. You see this pop up in anime occasionally, like Dragon Ball Z and Yu Yu Hakusho.
Hinduism is widely believed to be related to European mythology in general, especially Norse mythology, with the Hindu demons "Asura" representing the Norse "Aesir" and the Norse rival gods "Vanir" representing the Hindu gods, "Devas" or "Vedic" gods. Hindu epics are filled with a lot of cool creatures that would fit in well with Gygaxian fantasy gaming, the flying poisonous cannibals, Rakshasa, the forest dwelling monkey men, Vanara, the sky-dwelling musicians, Gandharva, the snake bodied Naga, and the nature spirit "fairies", Yaksha. In Thailand, the Yaksha has a demon/ogre/goblin like appearance and are placed around temples as guardians. In India, the king of the Yaksha, Kubera, was a dwarf who lived in a mountain and guarded treasure (I think, it's been a while since I've read the Ramayana and there's more than one translation), which might be linked to the various earth spirits in Europe.
I'm rambling now. I guess what I'm trying to say is that mythology is cool.
If you're interested in some role reversal, you could cast dwarves as the bad guys, as Dvergar, Powries and Kobolds(no, that's a thing, dangit!), while the Goblins are Robin Hood trickster rogues getting the best of them.