The rpm-ostree reset command will remove any mutations of the base image that you have installed (layered packages) and return your system to the current base image as released from Fedora’s repo, no layered packages. It will not affect any user data or environment settings, aside from what removal of the layered packages would require. AFAIK this also means anything that the layered packages place in the user home dir. So when you re-install those layered packages usually preferences are still setup as you had them. I didn’t want to reset my system because of a few codecs, so I removed the ref’s (with ostree), deleted the metadata for the repos in /etc/yum.repos.d, did a cleanup on my image with rpm-ostree cleanup -m then uninstalled the offending packages. I reboot at this point before I do the rpm-ostree update, then follow that with another reboot and finally the rebase. I know there is a lot of talk around about the need to reboot or not but I apply this simple approach, if I make a change to rpm-ostree in some way, other than meta-data cleanup, I reboot.
As for the recommended approach, rpm-ostree is a variant of ostree, and ostree is the sysadmin command for Silverblue, when you need more control over your install than rpm-ostree gives. There is a lack of a clear documentation trail that I can point you to other than read the fedora system administrators guide for the release you are using, as this technology is becoming more relevant to the sysadmins so there is an increase of info there with each release.
In keeping with this topic’s thread, this link is to the administrators guide for rpm-ostree and this here is the link to the FAQ for Silverblue about upgrades/rollbacks/pinning commits. etc…
rpm-ostree ex is for the experimental commands and options, such as making your immutable OS mutable. I would strongly recommend against doing such unless you have specific need to, and understand the ramifications. You can just use rpm-ostree reset <option> with the option of resetting layered, overrides, iniramfs, all. Or you can specify exactly what to remove with --uninstall, and you can even install overlayed packages with the same command.