Bay 12 Games Forum

Please login or register.

Login with username, password and session length
Advanced search  
Pages: 1 ... 318 319 [320] 321 322 ... 350

Author Topic: The Generic Computer Advice Thread  (Read 550815 times)

wierd

  • Bay Watcher
  • I like to eat small children.
    • View Profile
Re: The Generic Computer Advice Thread
« Reply #4785 on: December 09, 2021, 10:36:13 am »

If you are using an sdcard to host a linux deployment, please, for the love of gawd, format it with stripe and stride width settings!!

While there is some voodoo involved, the card will suffer significantly less wear, and even performs moderately faster.


https://thelastmaimou.wordpress.com/2013/05/04/magic-soup-ext4-with-ssd-stripes-and-strides/

For SDCards, the same logic applies as for SSDs. Just the erase block sizes will be much larger. 
Logged

delphonso

  • Bay Watcher
  • menaces with spikes of pine
    • View Profile
Re: The Generic Computer Advice Thread
« Reply #4786 on: December 09, 2021, 07:40:20 pm »

You'd think this would have changed in the last 8 years.

LordBaal

  • Bay Watcher
  • System Lord and Hanslanda lees evil twin.
    • View Profile
Re: The Generic Computer Advice Thread
« Reply #4787 on: December 09, 2021, 08:23:09 pm »

Well for me is like ssd were just invented. Perhaps pi disk handling is somehow interlinked with Venezuelan average access to technology?
Logged
I'm curious as to how a tank would evolve. Would it climb out of the primordial ooze wiggling it's track-nubs, feeding on smaller jeeps before crawling onto the shore having evolved proper treds?
My ship exploded midflight, but all the shrapnel totally landed on Alpha Centauri before anyone else did.  Bow before me world leaders!

delphonso

  • Bay Watcher
  • menaces with spikes of pine
    • View Profile
Re: The Generic Computer Advice Thread
« Reply #4788 on: December 09, 2021, 08:38:06 pm »

Yeah, this new hard drive is actually the first SSD I've had (and now there's M.2 for desktop computers...christ...). I was surprised by how much of an impact it had on boot-up times, though regular operation is still bottlenecked by my CPU. I thought it was all just hype.

wierd

  • Bay Watcher
  • I like to eat small children.
    • View Profile
Re: The Generic Computer Advice Thread
« Reply #4789 on: December 10, 2021, 12:15:09 am »

You'd think this would have changed in the last 8 years.

Expecting media makers to produce compatible but costly microcontrollers for the flash array, instead of "compatible" but cheap ones!?

What planet are you from!?

No, it has gotten WORSE in the past 8 years, not better.

The flash itself has gotten more durable, but the controller issue (that causes the write amplification) has only gotten worse, as erase block sizes seem to only grow.

This is especially true of very cheap flash, like SDCard.

Please, for the love of gawd, use stripe and stride-width settings, and make use of zram swap and tmpfs mounts for browser cache, when using an sdcard to host a linux system!
Logged

wierd

  • Bay Watcher
  • I like to eat small children.
    • View Profile
Re: The Generic Computer Advice Thread
« Reply #4790 on: December 11, 2021, 01:04:12 am »

I haven't tried it with Raspbian/SD-Card, but you could take a note from the Live CD Linuxes (Linii?) that let you place whole or 'differential' mountpoints on the 'host' machine's HDD[1] to overcome the default state of recurring amnesia.

You could even try to make the whole startup work from that USB-platterdrive (not sure if the Pi firmware does (or can be asked to) use that in leiu/advance of a valid SD card), but a minimal loader[2] that makes all RW mounts reside elsewhere. (Perhaps put /swap and other latent amd transient high-volume data areas upon an USB Ram-based dongle if that gives better IO performance than a more mechanical or even remote read/write location, and easily replaced if/when it starts to 'wear out'.)

I haven't done so much fiddling with this sort of obsessive tweaking myself, but if someone hasn't perfected this setup already and made a handy guide (or even setup-wizard) for this already then I'm very much surprised.


[1] Possibly across a networked drive, whether Cloud or Fog or NAS variety, but takes a bit more effort than a 'keystone' .cfg file on the host's drive to explain to the receptive hard-booting nonRW startup what options had previously been decided in a previously booted-up session.

[2] That you can keep an image/copy of in the event of the first finely-tuned SD going bad from too many reads or other loss of integrity.

The /etc/fstab file can define any volume as the / volume. The initial ramdisk and bootloader will have to live on the SDCard, but the USB 3.0 spinny disk can totally be /.

The traditional way of handling this, is having the boot partition mounted as /boot. 

Personally though, the way I would deal with it is to have the SDCard remain /, but use stride and stripe-width settings appropriate for the card,  turn on zram backed swap, then set a 100mb or so tmpfs mount in the appropriate part of the user's home directory to catch the browser's cache writes. Similar tmpfs mounts for /var/log and pals, but with a custom start/stop script that copies the logs to the sdcard on shutdown. (This preserves the logs, while enforcing atomic writes instead of shittons of incremental ones, and saves the card that stress. We dont give 2 shits about the browser cache being volatile; you get wiped cache and cookies as an added bonus, no extra work required. Suck it NSA.)

Browser devs want very much to pretend the system disk is an unlimited resource with no write life limits, and can actually get quite cross when you suggest tmpfs mount on the cache folder location, stating it defeats the purpose of the cache.  I crossly point out that gigabytes of old web assets laying around is pointlessly wasteful, aggressive small writes are deadly to flash based disks, and spinny disks are also getting sensitive to such writes, with SMR based tech taking over. We tend to agree to disagree.  With an SDCard as the volume, just accept the ram backed cache. It will save lots of headache, trust me.

I learned this lesson the hard way with a hacked chromebook. To get around the obscenely small eMMC baked in storage issue, i have a 256gb microSD living permanently in the slot, and hosting the /home directory. (eMMC is hosting /) The aggressive use of disk by the browser quickly nuked my card the first time I set it up that way, within about 6 months. It was then that I went on the crusade to minimize writes, and found that blog post.

After setting it up RIGHT, i have had the same microSD in that slot for 3+ years.

Raspbian does not turn on stride and stripe-width options on the ext4 images they peddle, so I am being a hardline fucker on this.  You can set the options after the fact with tune2fs.

https://linux.die.net/man/8/tune2fs

Raspbian does not know the magic parameters needed for an arbitrary SDCard, so they just go without this protection. (But sadly dont warn you!)

The easiest way to get the erase block size is to check the SDCard before you format it, and see what clustersize the ExFAT filesystem is using.  However, if you have already blown it away, you can do the black magic voodoo approach with flashbench, and interrogate it empirically.


Logged

delphonso

  • Bay Watcher
  • menaces with spikes of pine
    • View Profile
Re: The Generic Computer Advice Thread
« Reply #4791 on: December 11, 2021, 01:21:31 am »

LordBaal and anyone else making a PC: this site is real convenient (I know I'm very late to the party...)
https://www.pcpartpicker.com/

They don't have everything, but it's hard to find a better consolidation of all the key components to a PC and what's currently on the market. I'm starting the preliminary parts search now - including a GPU marketed towards /women/ that is about 5% cheaper than its non-gender equivalent. It comes with anime stickers and I'll be buying it as soon as I move.

King Zultan

  • Bay Watcher
    • View Profile
Re: The Generic Computer Advice Thread
« Reply #4792 on: December 11, 2021, 02:05:52 am »

a GPU marketed towards /women/ that is about 5% cheaper than its non-gender equivalent. It comes with anime stickers and I'll be buying it as soon as I move.
Why would you market that towards a specific gender it's not like you'll see it once the computer is put together.


Also my curiosity needs to know what kind of stickers come with it.
Logged
The Lawyer opens a briefcase. It's full of lemons, the justice fruit only lawyers may touch.
Make sure not to step on any errant blood stains before we find our LIFE EXTINGUSHER.
but anyway, if you'll excuse me, I need to commit sebbaku.
Quote from: Leodanny
Can I have the sword when you’re done?

delphonso

  • Bay Watcher
  • menaces with spikes of pine
    • View Profile
Re: The Generic Computer Advice Thread
« Reply #4793 on: December 11, 2021, 02:17:12 am »

Why would you market that towards a specific gender it's not like you'll see it once the computer is put together.


Also my curiosity needs to know what kind of stickers come with it.

Nowadays most cases have glass sides - I was also surprised that motherboards and ram are now all stylish with built in lights. Kids these days...

(I'll post a picture of the stickers when/if they arrive.)

wierd

  • Bay Watcher
  • I like to eat small children.
    • View Profile
Re: The Generic Computer Advice Thread
« Reply #4794 on: December 11, 2021, 02:21:24 am »

Caveat emptor!!

The "waifu" gpus tend to be poor performers with bad cooling.

RESEARCH EXHAUSTIVELY FIRST!
Logged

delphonso

  • Bay Watcher
  • menaces with spikes of pine
    • View Profile
Re: The Generic Computer Advice Thread
« Reply #4795 on: December 11, 2021, 02:53:03 am »

It's good for the price - from all the reviews I've seen. Cooling is a nightmare anyway, and I'm already overshooting cooling in the "case" (currently considering a pure mesh box if any box at all...)

In university, I always had a dream of building a PC out of 3 box fans - taking the fan bits out of the middle one and mounting the board inside of it - extra points if I could turn it on by turning the fan toggle.
« Last Edit: December 11, 2021, 02:55:00 am by delphonso »
Logged

Starver

  • Bay Watcher
    • View Profile
Re: The Generic Computer Advice Thread
« Reply #4796 on: December 11, 2021, 08:31:12 am »

[...] (not sure if the Pi firmware does (or can be asked to) use that in leiu/advance of a valid SD card), but a minimal loader[2] that makes all RW mounts reside elsewhere. [...])
The /etc/fstab file can define any volume as the / volume. The initial ramdisk and bootloader will have to live on the SDCard, but the USB 3.0 spinny disk can totally be /.
Yeah, I was anticipating that, but wondering if it could have been 'BIOS set' (as it were). From what I know, the (pseudo-)RTC on later Pis now at least remembers what time it was when it was last working[1], but I'm not sure if that's picking up off the (continually updated/written-on-shutdown) SD or there's actually some small-volume EPROM/similar that the bare metal checks and can be entirely re-instructed to prebootload elsewhere. The self-contained drive/partition drivers needed may be the sticking point. Design it exactly for SD-Card spec and you don't need to worry about FAT/ext/whatever variations, and more hardware-related divergences.

And there's not obviously any battery(/supercapacitor?)-sustained CMOS as per the older, bigger brothers in the motherboard world. BICBW.

(Which reminds me, I need a new 2032 cell for one of my older PCs. But by now I'm so used to telling it F1 to continue, whenever I do need to boot back up. Even while the mains power seems to keep the RTC going (unplug/powercut sends me back to the '80s, or whenever!), it doesn't ever trust itself to know that the only drive it can find is the one it should reboot up upon again and keeps on asking me and I keep on telling it "Yes!", as it lacks the "Try to boot anyway" option of other later Mobo configs.. ;) )

I really ought to RTFM on all the RP's features, but this was never something I needed to look at. Maybe a jumper, whatever they might use for firmware re-flashing. I'm sure must be possible because I remember vague warnings about how not to brick the thing.

Anyway, just thinking aloud. Maybe I'll actually check at some point. This was intended as food for thought, for others but now it's just me speaking with my mouth full.  8)


[1] It syncs to world-time when connected (as per SD-stored bootup connectivity arameters), perhaps polling the ITS,  but my main Pi use is disconnected and standalone, so while they used to reset to their personal firmware epoch+constant, now it just acts as an offline-ptime()-like thing. They may have changed it again since my last basic Pi board purchase!
Logged

wierd

  • Bay Watcher
  • I like to eat small children.
    • View Profile
Re: The Generic Computer Advice Thread
« Reply #4797 on: December 11, 2021, 01:26:00 pm »

The problem is that sdcard spec specifically mandates exfat, abd exfat lacks essential features. (For a modern OS)

What it DOES do, is allow absurdly huge cluster sizes. (In the multiple megabyte ranges) which is what allows a factory formatted sdcard to not die extremely quickly from write amplification.

The SDCard Assn pointedly and purposefully does not standardize erase block sizes. This is why Raspbian does not have "industry defaults!" to fall back on, and why they dont set the extended ext4fs options to get erase-block size atomic writes.

Ext4fs is tailored for modern spinny disks, and uses a 4k cluster size, fixed.

The filesystem driver will queue up sectors to write in a software buffer until the appropriate atomic write size is reached, then commit them in batches-- when the raid options are enabled.  This is intended for efficient writes across an array, but a flash disk is best viewed as an interleaved raid-0 LUN. This is why the raid feature works so well to maximize life and IOP thruput to a flash based single disk.

SDCards especially, can change radically from one production run to the next, based entirely on commodity component pricing and availability, which is precisely why the assn does not standardize erase block configurations, and instead focuses on the filesystem used:  as long as the chunks are consistent, the manufacturer can use whatever arrangement they want, format the device accordingly, and spec compliant devices will commit chunks of that size.  Medium stays happy, healthy, and speedy.

This is why the easiest way to learn the eraseblock size is just to get the exfat cluster size used by the factory.

Otherwise, you have to use flashbench, and voodoo.

The volatility between production runs means that yku cannot just say "its a 256gb Transend", and have known block sizes. The actual flash chip and the microcontroller that services it  can be any combination of dozens of offerings, and still be "256gb Transend".

SSDs tend to be vastly more consistent, and manufacturers are less guarded about what is actually in there driving it.

Rules of thumb for SDCards:

Look at the size of the gap at the start of the disk, before the start of the first partition, on the factory format.

Look at the cluster size of the ExFAT filesystem of the factory format.

These will tell you the erase block and page sizes, respectively.

(The card manufacturer wants to avoid having the partition table and primary FAT get compromised by excessive writes, so they pad THE ENTIRE ERASE UNIT that these structures live in, by setting the start of the partition that size apart from the start of the volume, on the very beginning of the NEXT block. Tjis is why the gap reveals the page size.

The micro controller that controls the flash array itself, has a finite amount of RAM it can use to do operations on that array with. This is the page size. The manufacturer optimizes the exfat filesystem to work in chunks of this size, so the controller does not waste iop cycles.  This is especially true for "high speed" SDCards.)

What the "Stride" and "Stripe-Width" parameters do, is define how much data to write across an array to optimally write on all disks equally, and what size increments to send to the array controller to maximize card cache use efficiency.  These are directly analogous to the erase block, and page size. Thats why it works.


In terms of a real  RTC inside a Pi, NO. It does not have one.  The SoC it leverages, must be configured on each and every boot. This is one of the many reasons for the Pi-Specific pi config file.

It is also why the Pi must initially boot from the SDCard.  That boot process COULD be chainloaded, in a manner similar to other embedded devices with uboot and pals, but it would not give much utility, and would still need to be protected/managed by the OS.





« Last Edit: December 11, 2021, 01:33:58 pm by wierd »
Logged

LordBaal

  • Bay Watcher
  • System Lord and Hanslanda lees evil twin.
    • View Profile
Re: The Generic Computer Advice Thread
« Reply #4798 on: December 11, 2021, 01:51:04 pm »

Thanks for the site del! Will try it latter
Logged
I'm curious as to how a tank would evolve. Would it climb out of the primordial ooze wiggling it's track-nubs, feeding on smaller jeeps before crawling onto the shore having evolved proper treds?
My ship exploded midflight, but all the shrapnel totally landed on Alpha Centauri before anyone else did.  Bow before me world leaders!

Mephisto

  • Bay Watcher
    • View Profile
Re: The Generic Computer Advice Thread
« Reply #4799 on: December 17, 2021, 05:31:13 pm »

Bought my first new PC since '13 and hoo boy this is going to be game changing.

For context, I'm coming from a 3rd gen i5, GTX 960, and 12 gigs of DDR3 something-or-other.



Not really computer advice, other than "go through a system integrator nowadays if you can."
« Last Edit: December 17, 2021, 06:10:58 pm by Mephisto »
Logged
Pages: 1 ... 318 319 [320] 321 322 ... 350