A friend of mine has F43 workstation KDE and suspects he was hacked. How do I properly make a forensic investigation of his computer?
Start with why your friend thinks they where hacked.
As @barryascott notes, determine what the scene is - what makes them believe there was any intrusion?
Do not reboot, but I suspect that is already too late.
Look for weird IP addresses or unexpected open ports.
ss -tupna > network_connections
Disconnect the machine from Wi-Fi and unplug Ethernet
Look for hidden processes, weird paths, or processes running as root that shouldn’t be.
ps auxef --forest > ps_tree
See who is currently on the machine and who logged in recently.
who -a > current_users && last -a > login_history && lastb > failed_logins
Take a full image of the drive
Shut down the machine, boot into a live Linux USB and ddrescue or dd to clone the impacted drive(s) to external storage.
All further analysis takes place on the cloned copy.
1. Check Authentication and System Logs
Query the system journal for recent login events, graphical sessions, and sudo usage or ssh attempts
Verify System Binaries
rpm -Va > rpm_verification - you want to be looking for changes to usr/bin/, /usr/sbin and so on.
If you see an 5 (MD5/SHA hash mismatch) or S (File size mismatch) next to a standard utility, itr’s been fiddled with.
Check crontab -l (for the user) and sudo crontab -l (for root). Also check /etc/cron*. Don’t forget to check the systemd timers for anything out of the ordinary.
Look for newly created user or system services in /etc/systemd/system/.
Check ~/.ssh/authorized_keys to see if any unknown public key(s) were added.
If you find modified binaries, unrecognized listening ports, or weird cron jobs, the system is (probably) compromised.
If the system is rooted, nuke it; wipe the drive completely, do a clean install and restore your personal documents from your backups.
He probably hasn’t been compromised, but if you find anything suspicious or have good reason to believe he’s not being overly paranoid, twat it into oblivion anyway and restore from a known good backup.
@barryascott
Stalker(s) indirectly revealed spying on his computer, probably for their own fun or intimidation. I looked at the messaging and believe him. On the technical side there were no indications, just random crashes and 2 or 3 times local package database was corrupted and had to be fixed with rpm --rebuilddb.
We both agree that there may be no hard evidence left as intrusion was seemingly highly skilled.
@anothermindbomb
You are right, PC was rebooted already.
He checked connections with netstat -antup on the PC and also used wireshark supplied by tcpdump data from his router ssh root@192.168.0.1 tcpdump -i br-lan -U -s0 -w - 'not port 22' | sudo wireshark -k -i - some days prior to disconnecting the PC, nothing unusual appeared, just tor traffic.
rkhunter revealed no known rootkits.
For most of the time he used solely tor browser from that PC and considers it to be prime candidate as an attack vector for browser-related malware. Also occasionally downloaded torrents. Apart from using widely known repos he ran mubeng binary downloaded from Release v0.23.0 · mubeng/mubeng · GitHub and bitcoin core.
There’s not much what could be used as initial point for intrusion to begin with.
Process tree shows no weird paths or processes, however there are a lot of kworker instances running as root.
sudo ps auxef --forest output: ps auxef --forest - JustPaste.it
Past failed logins contain no suspicious entries.
sudo rpm -Va gives many mismatched or missing files and there are mismatched binaries among them: rpm -Va | grep "/usr/bin" - JustPaste.it
I decided to check one of them:
rpm -qf /usr/bin/x0vncserver
tigervnc-x11-server-1.16.2-1.fc43.x86_64
tigervnc-x11-server-1.16.2-2.fc43.x86_64
rpm -q --dump tigervnc-x11-server-1.16.2-2.fc43.x86_64 | grep x0vncserver
/usr/bin/x0vncserver 570296 1776297600 352346dd02c211bc4ca76dfbdc5007241b9453ad7db7d05c775c1e566696907e 0100755 root root 0 0 0 X
/usr/lib/.build-id/26/bae75c1aa5065e44902be11e808581b87b6cc2 31 1776297600 0000000000000000000000000000000000000000000000000000000000000000 0120777 root root 0 0 0 ../../../../usr/bin/x0vncserver
Manual checksum comparison of actual binary /usr/bin/x0vncserver and downloaded from fedora repository showed no differences, sha256 is exactly as above. However table “Packages” from rpm database /usr/lib/sysimage/rpm/rpmdb.sqlite had sha256sum from tigervnc-x11-server-1.16.2-1.fc43.x86_64, while actual binary is from tigervnc-x11-server-1.16.2-2.fc43.x86_64.
Is there way to print sha256 values stored in the database with rpm command? I had to use hex editor to find checksum in a database blob.
Crontab, /etc/cron*, systemd timers do not contain anything strange.
No new systemd services nearly for the last 4 months and nothing strange-looking there either.
~/.ssh/authorized_keys does not have keys.
aureport -n output: aureport -n - JustPaste.it
There are KDE crash logs for 98-106 and 108-110 entries. No coredumps left. Cannot post them now because of link limit per post for new users. If those logs may be of any help, I will post links in a separate post.
How do I approach audit logs for further investigation? Amount of data is overwhelming for my first attempts. What may be considered an indicator of abnormal process behavior?
Before wiping the drive clean, I want to be sure that I tried everything to find evidence in the OS.
Quite normal.
┌─🎩 lurcher ~
├─
└─➜ ps aux | grep kworker | wc -l 21:11 Thu 18-Jun
202
That’s quite the list, especially given that it’s just /usr/bin. Has he had a failed upgrade at some point or somewthing like that? For reference, here’s my entire F44 list of weirdos, and most of these I can explain as I’ve changed them myself.
└─➜ sudo rpm -Va ⏱️ 17s 20:42 Thu 18-Jun
S.5....T. /usr/share/texlive/texmf-dist/tex/generic/config/language.dat
S.5....T. /usr/share/texlive/texmf-dist/tex/generic/config/language.def
S.5....T. c /etc/texlive/web2c/updmap.cfg
.M....... g /usr/share/texlive/texmf-dist/web2c/fmtutil.cnf
..5....T. c /etc/sudoers
S.5....T. c /etc/sysconfig/lm_sensors
.M....... g /var/lib/boltd
.....UG.. g /var/run/pcscd
S.5....T. c /etc/tuned/tuned-main.conf
.M....... g /var/lib/selinux/targeted/active/modules/200/vncsession
.M....... c /etc/audit/rules.d/audit.rules
S.5....T. c /etc/dnf/dnf.conf
S.5....T. c /etc/sddm.conf
......G.. /run/sddm
.M....... /var/lib/sddm
S.5....T. c /etc/tuned/ppd.conf
.......T. c /etc/yum.repos.d/rpmfusion-nonfree-nvidia-driver.repo
..5....T. c /etc/yum.repos.d/rpmfusion-nonfree-steam.repo
.M....... /boot/efi/System
.M....... /boot/efi/System/Library
.M....... /boot/efi/System/Library/CoreServices
.M....... /boot/efi/System/Library/CoreServices/SystemVersion.plist
.M....... /boot/efi/mach_kernel
.......T. /usr/share/nvim/runtime/doc/tags
.M....... c /etc/sysconfig/irqbalance
S.5....T. c /etc/sysconfig/livesys
SM5....T. c /etc/opensnitchd/system-fw.json
.M....... g /var/lib/snapd/seccomp/bpf/global.bin
.M....... g /var/lib/snapd/snap/README
.M....... g /var/lib/snapd/state.json
.M....... /var/cache/tailscale
.M....... /var/lib/tailscale
.M....... /var/lib/fwupd
S.5....T. c /etc/selinux/targeted/contexts/files/file_contexts.local
rpm package metadata: rpm -qa --qf '%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH} %{SHA256HEADER}\n'
I’d rotate all passwords, which I assume you’ve already done and ensure that you recognise all the users in /etc/passwd.
Pay attention to any users which have an actual login shell (i.e not /usr/bin/false or /sbin/nologin) are valid, and you know what they are. I’d expect root and the owners actual user account and nothing else.
Assuming the user does not remote SSH into this box, I’d either firewall port 22 off or just stop the sshd daemon. Alternatively stick fail2ban on there.
Then I’d monitor it for a few weeks, just to see if there’s any actual evidence of anyone either trying to get into the machine or actually gaining access.
If someone tells you they have access to your machine they almost certainly do not. Rule number one of accessing a remote machine is not actively going out of your way to alert the owner.
I see spam emails claiming this all the time… never true.
Has he had a failed upgrade at some point or somewthing like that?
According to ~/.bash_history database corruption happened before upgrade to F43 as he ran rpm --rebuilddb prior to it.
sudo dnf upgrade --refresh
sudo dnf system-upgrade download --releasever=43
sudo dnf clean all
sudo dnf system-upgrade download --releasever=43
sudo dnf upgrade --refresh
sudo dnf autoremove
sudo dnf reinstall bind-libs-32:9.18.43-1
sudo rpm --rebuilddb
sudo dnf upgrade --refresh
sudo dnf system-upgrade download --releasever=43
sudo dnf5 offline reboot
He does not recall why and how it became corrupted in the first place, seemingly happened for no big reason. After said upgrade there were two times database required rebuilding. 3 times in total.
rpm package metadata:
I meant checksums of individual files from DB. Found how to get them:
rpm -qa --qf '%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH}\n[%{FILENAMES} %{FILEDIGESTS}\n]\n'
Have not finished comparing all listed binaries yet. For those I did there are existing entries of two package versions in the database. It seems like rpm -Va reads file digests from the older package version while installed version is newer. Will update on this if I find binary not from any legit package.
Pay attention to any users which have an actual login shell (i.e not
/usr/bin/falseor/sbin/nologin) are valid, and you know what they are. I’d expectrootand the owners actual user account and nothing else.
All users have valid nologin shells except root, current user and:
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
Assuming the user does not remote SSH into this box, I’d either firewall port 22 off or just stop the sshd daemon.
sshd was never running, PC was used locally.
Then I’d monitor it for a few weeks, just to see if there’s any actual evidence of anyone either trying to get into the machine or actually gaining access.
Good idea. Gonna do it when there’s nothing left to do on that PC.
If someone tells you they have access to your machine they almost certainly do not. Rule number one of accessing a remote machine is not actively going out of your way to alert the owner.
True if continued surveillance is the primary target. But if the primary target is a person himself, rule may be broken in specific circumstances. Be it an act of demonstration of power, intimidation, discrediting a person as mentally ill or to instill feeling of being permanently watched. Stalking isn’t exactly about staying invisible.
My friend was not alerted by one single act of disclosure, but by long series of incremental steps meant to gradually increase psychological tension with no single obvious reason for an outside observer.
I see spam emails claiming this all the time… never true.
Never said it was simple as receiving email with unsubstantiated claims.
It was like stuff he looked up on the web or did on his PC surfacing as talks of anonymous users on places he visits. Like surfacing systematically with time delay, at first - months or weeks, then slowly shortening to hours. When unknown posters started commenting his actions basically in real time, he disconnected. There was no way for that kind of information to leak except spying on a PC. I can post slightly rephrased (as friend asks not to disclose exact places where he was) example of messaging when he disconnected, just don’t think it would be relevant for this forum.
Also stories from years of my friend’s life have indications of an ongoing stalking by proxies.
Mean like to look at a tooth brush on one web site and the see adds for tooth brushes all over the web?
This happens as you browse the web if you do not take care to limit use of cookies and turn on other fingerprinting techniques that the data brokers use.
For now all mismatches were results of corrupted database.
Can’t find these older packages on rpmfind.net to complete verifiying binaries:
transmission-common-4.1.1-2.fc43.x86_64
exfatprogs-1.3.2-1.fc43.x86_64
kf6-kconfig-6.25.0-1.fc43.x86_64
kf6-kguiaddons-6.25.0-1.fc43.x86_64
kf6-kdbusaddons-6.25.0-1.fc43.x86_64
kf6-kservice-6.25.0-1.fc43.x86_64
kf6-solid-6.25.0-1.fc43.x86_64
kf6-sonnet-6.25.0-1.fc43.x86_64
kf6-kfilemetadata-6.25.0-1.fc43.x86_64
kf6-kpackage-6.25.0-1.fc43.x86_64
kf6-kded-6.25.0-1.fc43.x86_64
kf6-kuserfeedback-6.25.0-1.fc43.x86_64
plasma-activities-6.6.4-1.fc43.x86_64
libkscreen-6.6.4-1.fc43.x86_64
crun-1.27.1-1.fc43.x86_64
vim-minimal-9.2.390-1.fc43.x86_64
converseen-0.15.2.3-1.fc43.x86_64
plasma-systemmonitor-6.6.4-3.fc43.x86_64
cups-client-2.4.18-1.fc43.x86_64
cups-ipptool-2.4.18-1.fc43.x86_64
If anyone knows how to get them, post please. Probably they will be fine too, but I’d prefer to check.
Also posting crash logs of tor browser. Can such segfaulting be an indication of attempted malicious actions?
see adds for tooth brushes all over the web?
No. Read my post please. It’s about actions of actual people.
cookies and turn on other fingerprinting techniques
Again, read the posts please. My friend used tor browser, what cookies and fingerprinting are you talking about?
If anyone knows how to get them, post please.
[/quote]
Install the fedora-repos-archive. Example:
dnf -q list transmission-common-4.1.1-2.fc43.x86_64 \
--releasever=43 --enablerepo=updates-archive
Available packages
transmission-common.x86_64 4.1.1-2.fc43 updates-archive
You can get older builds from https://koji.fedoraproject.org/koji/packageinfo?packageID=4013
Or perhaps from update-archive. To enable this repository install fedora-repos-archive.
Enabling fedora-repos-archive allowed to download and verify almost all packages except few from third party repositories that were on computer in question. All files from those packages are fine and checksums in database are from legitimate files.
How often does unintentional database corruption happens? What in your opinion are chances of unintentional corruption vs intentional just to generate false alarms?
Primary version now is attack with browser exploit that allowed remote access to display server data. Do such events leave any traces in system logs except segfaulting when exploit fails?
@francismontagnac @vekruse
Thanks.
Rarely - if it did we’d all have issues with checksums failing, installs and uninstalls failing and so on. I’ve never seen it, but it could happen in the same way that you could win a lottery.
As for people doing it intentionally to sow panic, again it’s possible but if you can do this you already have enough access to do whatever you like so it’s somewhat far-fetched.
That very much depends on the sophistication of the exploit and its ability to remove its own evidence trail. In order to do so it would require a high level of access. If you were able to do so, you would help yourself to everything you wanted, leave and then not announce the fact to anyone that you’d exfiltrated the data unless you were extorting.
If your friend is not being extorted, is not a nation state target and you can find no actual evidence of any kind of intrusion other than someone saying they did, I’d imagine this incident is not really genuine and there’s nothing to be found because nothing ever happened.
If you suspect you have a compromised system the gold-plate solution is to wipe the disk and reinstall. At my previous work place someone triggered the security people to think there was the potential for a system to be compromised. That system was wiped and the user was not allowed to do a backup first.
As for people doing it intentionally to sow panic, again it’s possible but if you can do this you already have enough access to do whatever you like so it’s somewhat far-fetched.
Could be leaving a false trail as concealing an exact method always makes sense. Since it was an attack against individual person, thourough investigation was not expected. Or, as it happened just before an upgrade, to discourage upgrading out of fear of losing remote access. Anyway, no sense in guessing.
That very much depends on the sophistication of the exploit and its ability to remove its own evidence trail.
My question is about how log system works. Let’s assume exploit had a bare minimum of functions - break out of browser container, access the memory of display server process, relay it to the attacker. No additional stuff. Would such behavior show up in logs? Reviewing system journal and audit logs made me wonder if event of one process successfully accessing memory of another is directly or indirectly logged at all.
If your friend is not being extorted, is not a nation state target and you can find no actual evidence of any kind of intrusion other than someone saying they did, I’d imagine this incident is not really genuine and there’s nothing to be found because nothing ever happened.
Well, as said earlier, there are indications of an ongoing stalking by proxies, there actually is some unnatural interest in him.
Imagine this:
You’re on the phone with someone, you walk up to the window of your bedroom and he’s telling you “what a nice yellow curtains” (you got yellow). Next you go to your balcony to have a cigarette and he tells you that smoking helps when stressed. Finally you go to the garage, get in your car while listening to his “you should fill up at the gas station”. This all happening in a single evening.
That is a kind of situation my friend was in. Just instead of walking around he was doing his stuff on desktop, instead of phone - text comments at a place he’s reading. Very synchronised in meaning and in time.
This topic is about help with figuring out how that was possible and to help him avoid being spied upon in the future.
@barryascott
Having no idea how system was compromised how do you prevent repeating same mistake and getting compromised again by that actor in a same way?
- perform a clean install with totally wiping the drive
- Avoid any risky sites.
Web browsing is a very common attack vector when the user is not cautious about clicking links and/or visits off color sites. - do not click on links received by email unless 100% certain the sender is known. In fact do not even open suspicious emails.
- change the email address and password to break the spam email chain.
- Do not reuse passwords between sites and use strong & secure passwords everywhere.
Social engineering is common and the most prevalent cause of any intrusions by malicious actors. Your friend may need to change his habits or experience repeat issues of having his equipment compromised.
A keyboard logger is simple, easy to hide, and relays exactly what was entered to the intruder.
As far as phone issues – NEVER answer calls from unkown numbers. If they leave a reasonable voice mail then you MIGHT consider calling back.
I have had as many as 15+ spam/bot calls in a single day.
As for how it would be logged or show up in the logs - no idea - I’ve never seen it happen.
As for how to cover ones tracks; in “the good old days” when the syslog was just a file in /var/log/ it was trivial to remove lines from it - sed or even vi and just delete what you want.
systemd’s journal format though is binary, which in itself makes it harder to snip lines out of, but in addition to that it’s encrypted and assuming Forward Secure Sealing (FSS) is activated then there isn’t any way to alter the file without systemd knowing it’s been altered, and you’ll get a message(s) to that effect.
You can check all of your journals for tampering with systemctl --verify. Be prepared for a small wait (and this will also flag up corrupted logs from hard crashes so a failure to verify a specific journal does not necessarily mean it has been maliciously altered)
As for one memory process accessing the memory of another process, that is possible, but to do so you’d have to be effectively running as a kernel process which itself would leave footprints. This itself can be worked around by having encrypted code on disk and running entirely in memory, much as nation state malware would tend to do (think Stuxnet code and the like).
Possible - yes.
Likely - probably not.
Unless, of course, your mate is on some kind of watchlist and would attract the attention of nation state actors… but then why are they effectively letting him know his machine is compromised so that he can simply ditch it and use another?
Well, as said earlier, there are indications of an ongoing stalking by proxies, there actually is some unnatural interest in him.
That’s certainly possible, but targeted stalking/harassment seem less likely – what might be the end goal of such a campaign?
If you apply Occam’s Razor, it seems more likely that it’s apophenia and not a targeted (by dozens of advertisers, ad platforms, Internet fora, etc. – A cast of thousands, maybe?) effort to (do what, exactly?) to your compatriot.
I’m not saying there is nothing there, just that as humans, we are really good at pattern recognition, even if no pattern actually exists.
That said, as suggested by others, it certainly wouldn’t hurt to wipe the system and re-install from known-to-be-clean media and be careful in what’s installed moving forward.
@anothermindbomb
journalctl --verify yielded one strange result:
4a7fe0: Invalid hash (dcd11167a8e3b422 vs. 23fc12c66d2e0ac9)
4a7fe0: Invalid object contents: Bad message
File corruption detected at log/journal/119c410b68c94c1ea370fb8789d724ca/system@000652df78ff452c-fdece66e0334d8db.journal~:4882400 (of 8388608 bytes, 58%).
FAIL: log/journal/119c410b68c94c1ea370fb8789d724ca/system@000652df78ff452c-fdece66e0334d8db.journal~ (Bad message)
File system@000652df78ff452c-fdece66e0334d8db.journal does not exist and its naming scheme system@000652df78ff452c-fdece66e0334d8db is different from all other *.journal files that are present.
Regarding access to display server data, applications within Xwayland server are not isolated and accessible to each other without privilege escalation.
Sources:
Tor browser and second application that was commented upon are running in Xwayland.
But another one was running in native wayland, so there must have been some way to bypass its window isolation. Did not find signs of privilege escalation in logs yet.
This itself can be worked around by having encrypted code on disk and running entirely in memory, much as nation state malware would tend to do (think Stuxnet code and the like).
Wouldn’t be the same possible with remotely delivered code if there was a way to run it? Stuxnet happened to have an airgapped target.
why are they effectively letting him know his machine is compromised so that he can simply ditch it and use another?
Passive surveillance gives nothing. Any possibly valuable info was already collected and my friend is not getting himself involved in anything fishy. So they initiate active measures to achieve something. Not in a position to guess what is the actual endgoal. My friend can describe it as use of weaponized psychology to destabilize a person. Probably no good intentions there.
That’s certainly possible, but targeted stalking/harassment seem less likely – what might be the end goal of such a campaign?
Kinda asking the wrong person. Just think anyone in business of stalking is not keen on openly stating true intents.
If you apply Occam’s Razor, it seems more likely that it’s apophenia
I think people can have different life stories. Here is just an example that there’s enough things deliberately made to be dismissed by medieval philosophical principle and common everyday psychiatry Zersetzung - Wikipedia
I’d like to stick to original discussion about determining how something is possible or finding that it is technically impossible.
That said, as suggested by others, it certainly wouldn’t hurt to wipe the system and re-install from known-to-be-clean media and be careful in what’s installed moving forward.
Will agree, but when deliberately hacked there is sense in determining possible extent of compromise and circumstances because of risk of repeated attempts. It’s not like you accidentally download a trojan.
I have several similarly named files. I note you left off the ~ that was at the end of the file name in that message, which could explain why the file does not seem to exist.
Those appear to be a temporary journal that may have been interrupted before being stored to the full name by something such as a power loss or crash.
When using vi/vim to edit a file, a temporary copy of the file is created with the ~ at the end of the file name and it remains until the file is properly closed in vim. If vim is closed without saving the file that temporary copy remains on the system. I suspect the journald system does similarly.
I ran journalctl --verify and found 2 similarly named files that gave similar results to yours, and I know for a fact that my system has not had an intrusion. I can relate those 2 files to a time when I had a power loss and unclean shutdown. In fact both those files were on the same date. One was the user journal and the other was the system journal for that date.