I’m just wondering whether anyone else can reproduce this bizarre corruption on Fedora 44 Desktop.
When I burn an ISO image to an SD card hanging off of USB (HS or SS speed, hub or not, other devices or not, all irrelevant), such as the Fedora 44 image, it reads back with corruption most of the time. The corruption, if present, is the same on each read of the same media, even if ejected and reinserted, suggesting that this is exclusively introduced on the write path.
The location of the corruption is always on a 64KiB (but not necessarily 128KiB) boundary and its location relative to the start of the media is constant unless (rarely) there is no corruption at all. The location is the same even if I swap SD cards to a different size and brand. Strangely, it’s either just 32 bits of random garbage or 10KiB of other random garbage. In the latter case, it’s the very last 10KiB of the file, which suggests a media flush failure, but I’ve done everything imaginable to force synchronization before removing it each time.
The same thing happens under Ubuntu on the same machine so of course my primary suspicion is a defective memory module; the Ubuntu address of the corruption is different from the Fedora address but is otherwise constant and a multiple of 64KiB.
I can use dd or cat to write the ISO directly to /dev/something but it makes no difference. Options on dd that control cache bypassing or media synchronization also don’t matter. Using non-power-of-2 block sizes in an attempt to force read-modify-write updates to the media also have no effect.
It’s possible that this is a broader issue affecting all systems with hardware which is, in some as yet unidentified way, similar to my own. I’d rather not dump my entire hardware and software fingerprint here in public but I’m willing to do some selective disclosure if necessary.
Small chance that I just discovered a serious bug somewhere in the stack, like at the USB driver level. Big chance that I’m just wasting your time with a bad memory cell being deterministically mapped to the same address all the time (because address space randomization is disturbingly not enabled for the transfer buffer). If that tradeoff doesn’t scare you, then I’d be interested to see if you can reproduce this. It happens about 80% of the time.
Using different media and different sdcards sounds like you eliminated a lot of possible issue.
I’ve not heard of writting an ISO to an sdcard.
When I use an sdcard for my RPi I use the arm-image-installer.
When I write ISO it’s usually to a USB stick.
Have you checked the journal for errors?
Have you tested the media with the f3 tool?
f3.x86_64 Utility to test for fake flash drives and cards
This sounds a lot like an incomplete write …
IF you are doing this in Fedora/Linux … before you unmount/remove/unplug the USB device, do a sync at the command line and wait for it to complete AND if using the file manager, click on the eject media and wait for it to report it is safe to remove the media.
Thanks for the input. It can’t possibly be a media error because the errors are in the exact same position on 2 radically different SD cards. But now that you mention it, there might be an error recorded in the journal. Which journal and how might I check that?
Hi Einer, yes it sounds like an incomplete write. But it’s definitely not. I used sync, even twice, after the write. I also waited about an hour at one point after the dd terminated, before physically ejecting.
Also, it’s only in one manifestation that the garbage appears at the end (the last 10KiB) of the written area (which would be consistent with an incomplete write to media that previously had garbage on it). In the other manifestation, the first 32 bits of this last 10KiB is corrupt but the rest of it is OK. In either case, I’m positive that the media was properly synced before ejection.
I’m guessing that the second manifestation is due to a pointer getting corrupted, whereas the first manifestation is simply hitting the data itself. So the OS goes and fetches some cached encrypted data which happened to be hanging around from my encrypted drive, and throws it onto USB.
I’m not using a GUI at all, just CLI, in order to keep things as simple as possible.
The system journal most likely for a low level I/O error.
This will will show warning and errors since your booted.
sudo journalctl -b 0 -p warning
You can also check how much dirty pages the system has that may contain the data that seems not to be written with:
grep Dirty /proc/meminfo
What seems odd to me is that the missing/corrupt data is near the start of the data you are writing. When I’ve seen this in the past it was the end of the data what was missing.
Good thinking but I checked and there are no dirty pages or salient journal warnings (except that I ejected some media, which is the same regardless of corruption or not).
I now think I know what’s going on here, which would explain why the corruption address is constrained.
I’m writing an ISO (usually but not necessarily just the Fedora 44 image) to an initially empty (undefined) storage medium which happens to reside on USB. Then I sync. At some point during this process, Fedora discovers a brand new volume hanging off of USB (which is the one that I just wrote or, worse, am in the process of writing). Instead of doing the right thing, which is to ignore it until I mount it, Fedora assumes that it’s already mounted, so it proceeds to “help” me by updating some file access time or the like, which was prompted by its uninvited browsing of that media.
The question is: who is at fault here? Is it me, because I changed the media from undefined to a valid volume (and sometimes back and forth a few times, separated by syncs)? Or is it Fedora for messing with a volume that it was never instructed to touch?
I need to do more research on this but that’s my current thinking, inspired by a new form of corruption that I observed, which appears to involve the alteration of file system metadata or file metadata.
Also, I know for a fact that actual file data has been modified in the past. (This was how I initially discovered it, doing a file compare just to be paranoid. Like you spend your whole life never seeing a compare failure, and then suddenly, there you go! But at the time, I dismissed it as definitely a bad block on the SD card because it was so consistent: the same file at the same offset with roughly the same fail-or-not frequency.) Now that I think about it, this is smelling a lot like Fedora remembering the file system that used to be there, then continuing to update that file system’s metadata long after the actual resident file system has been swapped out under its nose. sync should have fixed all that but it didn’t. Like Fedora remembers, say, an Ext4 file system that used to be on that SD card, but has long since been replaced by NTFS or just another Ext4 file system with totally different files on it.
You first have to know wheter or not automount is enabled or not. When you insert a usb device which contains some files, is it automatically mounted or not?
By the way, what desktop environment are you using?
Are you trying to create a bootable USB installer or just transferring an ISO file? By “empty (undefined) storage medium” do mean previous contents of the USB drive were deleted leaving a filesystem, drive overwritten with zeros or random data, etc.?
I think my latest test (GNOME, to answer your question) is pretty revealing. I rebooted the machine and inserted my SD card. It had previously been trashed with random data so no partition showed up, only the usual “/dev/sda” in lsblk.
I then did:
dd if=image.iso of=/dev/sda bs=16M; sync
which is simply an efficient way to write the image to the SD card and then flush the media before returning to the command prompt.
Once it had completed, I went into Fedora’s native file explorer and manually ejected the SD’s newly minted filesystem (which was actually redundant, or should have been, on account of the sync command). I then physically ejected the SD, and then reinserted it.
Before doing anything else, I read it back:
dd if=/dev/sda of=image2.iso bs=16M; sync
and then compared the hashes of image.iso and image2.iso. They were found to be unequal, meaning that the image was once again corrupted. Considering how polite I had been about ensuring file system synchronization, there’s simply no way this should have occurred, absent a physical error on the SD, which for the reasons stated above is extremely unlikely.
I do believe automount is happening but in this case, there was nothing to mount, so it shouldn’t matter.
Be aware that copying from /dev/sda will read beyond the end of the date that was originally written. I would expect that the size of image2.iso would be different from the size of the original iso file, thus the hashes alone for that reason would be different.
You can check the internal checksum of a Fedora iso file using checkisomd5, and you can do that on the original iso file and on the /dev/sda device.
Gnome does not want you to know how to do that, but Google knows the secret. For my own system (xfce) I make sure automount is off and I can mount the external device when and if I need to.
Yes normally my command above would read beyond the end of the data but it doesn’t in this case because image.iso is the exact size of the SD itself. I have verified that image.iso and image2.iso are the same size.
You might be right but in that case this is a severe bug. How can I disable automount? (I mean what’s the proper way as opposed to the LLM smartypants way?)
On second thought, that shouldn’t be relevant because the SD had been trashed before I even powered on, so there was no automounting possible even if had been attempted.
Type in “how do I disable automount in gnome” in a google search, for example.
The following is probably AI generated, but it seems to be correct.
Using dconf Editor
Open dconf Editor: If you don't have it installed, you can install it via your package manager.
Navigate to the Setting:
Go to org > gnome > desktop > media-handling.
Change the Automount Option:
Find the automount setting.
Set it to False to disable automount.
I will try that and report back, probably tomorrow.
Is anyone willing to try this, even without modifying automount? You can use truncate to cut down the file size after breaking the readback process if your image is less than the SD capacity. (I never did manage to reproduce this with a USB stick, for whatever random reason, so I recommend using an actual SD card, on USB, if possible.)
The only possible trend I’ve found in all of this is that the error is more likely to happen when the system has been busy for a while. So maybe sit around copying files and watching videos first. I have nothing else to offer by way of reproducible conditions, if this is even accurate. It’s just that flakey. At least I can see it 80% of the time on my particular setup.