filesystem-inodes-and-recovery.md (13428B)
1 --- 2 title: "Filesystem, Inodes and Recovery" 3 section: "Linux" 4 sectionSlug: "linux-hardening" 5 sourcePath: "src/linux-hardening/main-system-information/filesystem-inodes-and-recovery.md" 6 sourceUrl: "https://github.com/HackTricks-wiki/hacktricks/blob/188de82beb54e70956b2952367a0af91d26758b8/src/linux-hardening/main-system-information/filesystem-inodes-and-recovery.md" 7 sha: "188de82beb54e70956b2952367a0af91d26758b8" 8 isIndex: false 9 modified: true 10 license: "CC-BY-NC-4.0" 11 --- 12 13 # Filesystem, Inodes and Recovery 14 15 Filesystem abuse is often about confusing the relationship between a visible path and the object behind it. 16 17 Disk images may hide another filesystem.<sup>[[1]](#references)</sup> Writable mounts may be consumed by privileged jobs. 18 19 Hardlinks may expose the same inode through a different name.<sup>[[3]](#references)</sup> Deleted files may still be readable through an open file descriptor.<sup>[[5]](#references)[[6]](#references)</sup> 20 21 This page focuses on the technique, not on one specific lab or target. 22 23 ## Disk Images and Loop Mounts 24 25 A regular file can contain a complete filesystem, so a disk image can expose a second filesystem tree when mounted.<sup>[[1]](#references)</sup> 26 27 Backup images, copied block devices, VM artifacts, or renamed blobs can therefore contain credentials, scripts, SSH keys, configuration files, or flags even when they do not look useful from the outside. 28 29 Identify likely images with `file` to classify a candidate, `blkid` to probe recognized filesystem metadata, and `strings -a` to scan the whole file for printable sequences.<sup>[[10]](#references)[[11]](#references)[[12]](#references)</sup> 30 31 ```bash 32 file ./candidate 33 ls -lh ./candidate 34 blkid ./candidate 2>/dev/null 35 strings -a ./candidate | head -n 50 36 ``` 37 38 When mounting is allowed, use a loop mount with `ro` so the image is attached read-only; the `find` command below limits the inspection depth and file type.<sup>[[1]](#references)[[4]](#references)</sup> 39 40 ```bash 41 mkdir -p /tmp/imgmnt 42 sudo mount -o loop,ro ./candidate /tmp/imgmnt 43 find /tmp/imgmnt -maxdepth 3 -type f -ls 2>/dev/null 44 sudo umount /tmp/imgmnt 45 ``` 46 47 If mounting is not available and the image is ext2/ext3/ext4, inspect its metadata directly with `debugfs`.<sup>[[2]](#references)</sup> 48 49 ```bash 50 debugfs -R 'ls -l /' ./candidate 2>/dev/null 51 debugfs -R 'stat /' ./candidate 2>/dev/null 52 ``` 53 54 The technique is useful because it turns a normal-looking file into a second filesystem tree.<sup>[[1]](#references)</sup> Treat it as a way to recover hidden data, not as a privilege escalation by itself. 55 56 ## Writable Mount Abuse 57 58 A writable mount becomes dangerous when a more privileged context later trusts something inside it. The important question is not only "can I write here?", but "who later reads, executes, imports, or loads from here?". 59 60 Use `findmnt` to inspect mounted filesystems and their options.<sup>[[9]](#references)</sup> 61 62 Find writable mounts and suspicious consumers with the documented `find` permission, type, and filesystem-boundary predicates, then use recursive `grep` to search likely consumer configuration.<sup>[[4]](#references)[[20]](#references)</sup> 63 64 ```bash 65 findmnt -o TARGET,SOURCE,FSTYPE,OPTIONS 66 find /mnt /media /srv /opt -xdev -type d -writable -ls 2>/dev/null 67 find /mnt /media /srv /opt -xdev -type f -writable -ls 2>/dev/null | head -n 50 68 grep -RniE 'cron|systemd|ExecStart|backup|hook|plugin|sh |bash |python' /mnt /media /srv /opt 2>/dev/null | head -n 50 69 ``` 70 71 Common abuse patterns: 72 73 - A cron job or systemd service runs a writable script from the mount.<sup>[[13]](#references)[[14]](#references)</sup> 74 - A privileged service loads plugins, config, templates, or helper binaries from the mount. 75 - A mount contains SUID files and allows modification, replacement, or path manipulation. 76 - A container or chroot exposes a host-backed path that is writable from the restricted environment. Mount namespaces provide distinct mount hierarchies, while `chroot()` only changes pathname resolution and is not a full sandbox.<sup>[[15]](#references)[[16]](#references)</sup> 77 78 Generic validation pattern using the same `find` predicates.<sup>[[4]](#references)</sup> 79 80 ```bash 81 find /mnt /media /srv /opt -xdev -perm -4000 -type f -ls 2>/dev/null 82 find /mnt /media /srv /opt -xdev -type f -writable -ls 2>/dev/null | head -n 50 83 ``` 84 85 When proving impact in an authorized lab, keep the payload observable and minimal, for example writing `id` output to a temporary file.<sup>[[23]](#references)</sup> The core technique is delayed execution through a trusted writable location. 86 87 ## Inodes and Path Confusion 88 89 An inode is the filesystem object; a path is only a name pointing to it. Device and inode metadata let you distinguish objects across filesystems, while link counts expose multiple hard links.<sup>[[3]](#references)</sup> A deleted pathname does not always mean the data is gone while a process still has the file open.<sup>[[5]](#references)</sup> 90 91 The `find` predicates below compare inode identity, link counts, device boundaries, and timestamps.<sup>[[4]](#references)</sup> 92 93 Compare files by inode and device with `ls -i` and `stat` metadata formats.<sup>[[17]](#references)[[18]](#references)</sup> 94 95 ```bash 96 ls -li /path/a /path/b 97 stat -c 'dev=%d inode=%i links=%h mode=%A owner=%U:%G path=%n' /path/a /path/b 98 ``` 99 100 Find every visible pathname for the same inode with `find -samefile`.<sup>[[4]](#references)</sup> 101 102 ```bash 103 find / -xdev -samefile /path/to/file -ls 2>/dev/null 104 ``` 105 106 Search directly by inode number with `find -inum` when you only have metadata.<sup>[[4]](#references)</sup> 107 108 ```bash 109 find / -xdev -inum <inode_number> -ls 2>/dev/null 110 ``` 111 112 This technique is useful when a file appears under an unexpected name, when an application validates one path but uses another, or when a privileged wrapper interacts with an inode that is also reachable somewhere else. 113 114 ## Hardlink Abuse 115 116 Hardlinks create multiple names for the same inode. They do not point to a target path like symlinks do; they are equal names for the same file object.<sup>[[3]](#references)</sup> 117 118 Find SUID files with multiple hardlinks using `find`'s permission and link-count predicates.<sup>[[4]](#references)</sup> 119 120 ```bash 121 find / -xdev -perm -4000 -type f -links +1 -ls 2>/dev/null 122 ``` 123 124 Inspect one suspicious file with `stat` and `find -samefile`.<sup>[[4]](#references)[[17]](#references)</sup> 125 126 ```bash 127 stat /path/to/suspicious 128 find / -xdev -samefile /path/to/suspicious -ls 2>/dev/null 129 ``` 130 131 Why it matters: 132 133 - A sensitive file may be reachable through a less obvious path. 134 - A SUID wrapper may be hidden behind a name that does not look privileged. 135 - Cleanup that removes one pathname may leave another hardlink alive. 136 137 Linux's `fs.protected_hardlinks` sysctl can restrict hardlink creation across privilege boundaries.<sup>[[7]](#references)</sup> Existing hardlinks still merit review. 138 139 ## Deleted File Recovery Through Open FDs 140 141 When a process keeps a file open, unlinking its last pathname leaves the file alive until the last descriptor closes; Linux exposes those descriptors under `/proc/<pid>/fd/`.<sup>[[5]](#references)[[6]](#references)</sup> 142 143 Find deleted open files by listing `/proc` descriptors and filtering open-file output.<sup>[[5]](#references)[[6]](#references)[[18]](#references)[[19]](#references)[[20]](#references)</sup> 144 145 ```bash 146 ls -l /proc/*/fd/* 2>/dev/null | grep ' (deleted)' | head -n 50 147 lsof 2>/dev/null | grep deleted | head -n 50 148 ``` 149 150 Recovering through these links is permission-dependent because dereferencing `/proc/<pid>/fd` is subject to ptrace access checks and file permissions.<sup>[[6]](#references)</sup> 151 152 When permitted, `readlink` shows the descriptor target and `cp` copies its contents.<sup>[[21]](#references)[[22]](#references)</sup> 153 154 ```bash 155 readlink /proc/<pid>/fd/<fd> 156 cp /proc/<pid>/fd/<fd> /tmp/recovered-file 157 file /tmp/recovered-file 158 ``` 159 160 This is a practical technique for recovering deleted logs, temporary secrets, dropped binaries, rotated files, or scripts removed after execution. 161 162 ## ext Recovery With debugfs 163 164 On ext2/ext3/ext4 filesystems, `debugfs` can inspect inode metadata and dump inode contents from a block device or image; without `-w`, it opens the filesystem read-only.<sup>[[2]](#references)</sup> Work on a copy or a read-only image whenever possible. 165 166 List entries and inspect inodes with `debugfs` requests for directory listings, inode status, and inode-to-path checks.<sup>[[2]](#references)</sup> 167 168 ```bash 169 debugfs -R 'ls -l /' ./disk.img 170 debugfs -R 'stat <inode_number>' ./disk.img 171 debugfs -R 'ncheck <inode_number>' ./disk.img 172 ``` 173 174 Dump a known inode with the `debugfs dump` command, then classify the recovered output with `file`.<sup>[[2]](#references)[[10]](#references)</sup> 175 176 ```bash 177 debugfs -R 'dump <inode_number> /tmp/recovered.bin' ./disk.img 178 file /tmp/recovered.bin 179 ``` 180 181 This is not guaranteed recovery. It depends on filesystem state, whether blocks were reused, and whether the metadata still exists. For ext3/ext4, the `debugfs` manual notes that deleted-inode recovery may fail because released inode data blocks are no longer available.<sup>[[2]](#references)</sup> The technique is still valuable because it lets you inspect inode-level state without relying on normal path traversal. 182 183 ## Inode Exhaustion and Ordering 184 185 Inode exhaustion happens when a filesystem runs out of file nodes even if free disk space remains.<sup>[[8]](#references)[[17]](#references)</sup> It usually causes reliability failures, but it can also explain strange behavior during incident response or lab triage. 186 187 Use `df -i` to report inode information instead of block usage.<sup>[[8]](#references)</sup> 188 189 Check inode pressure with `df` and a `find` count of directory parents.<sup>[[4]](#references)[[8]](#references)</sup> 190 191 ```bash 192 df -h 193 df -i 194 find /var /tmp /home -xdev -printf '%h\n' 2>/dev/null | sort | uniq -c | sort -n | tail 195 ``` 196 197 Inode numbers and timestamps can also help reconstruct activity in simple lab environments. 198 199 The `find` format directives below expose those fields.<sup>[[4]](#references)</sup> 200 201 ```bash 202 find /path -xdev -printf '%i %TY-%Tm-%Td %TH:%TM %p\n' 2>/dev/null | sort -n | tail -n 50 203 find /path -xdev -newermt '2026-01-01' -ls 2>/dev/null 204 ``` 205 206 Treat ordering as a clue, not proof. Copy operations, archive extraction, filesystem type, restores, and concurrent writes can all change allocation patterns. 207 208 ## Defensive Notes 209 210 - Mount unknown images read-only during analysis.<sup>[[1]](#references)</sup> 211 - Keep privileged scripts, service units, plugins, and helper paths outside user-writable mounts. 212 - Use `nosuid`, `nodev`, and `noexec` where operationally appropriate; these options disable set-ID/capability execution, device interpretation, or direct binary execution on the mount.<sup>[[1]](#references)</sup> Do not treat them as a complete boundary. 213 - Restrict access to `/proc/<pid>/fd`; dereferencing those links is controlled by ptrace access checks and file permissions.<sup>[[6]](#references)</sup> Restrict broader process metadata and cross-user inspection where possible. 214 - Monitor writable mount points, unexpected hardlinks to privileged files, and deleted-but-open sensitive files. 215 216 ## References 217 218 - [1] [mount(8) — Linux manual page](https://man7.org/linux/man-pages/man8/mount.8.html) 219 - [2] [debugfs(8) — Linux manual page](https://man7.org/linux/man-pages/man8/debugfs.8.html) 220 - [3] [inode(7) — Linux manual page](https://man7.org/linux/man-pages/man7/inode.7.html) 221 - [4] [find(1) — Linux manual page](https://man7.org/linux/man-pages/man1/find.1.html) 222 - [5] [unlink(2) — Linux manual page](https://man7.org/linux/man-pages/man2/unlink.2.html) 223 - [6] [proc_pid_fd(5) — Linux manual page](https://man7.org/linux/man-pages/man5/proc_pid_fd.5.html) 224 - [7] [Documentation for /proc/sys/fs/ — The Linux Kernel documentation](https://www.kernel.org/doc/html/latest/admin-guide/sysctl/fs.html) 225 - [8] [df(1) — Linux manual page](https://man7.org/linux/man-pages/man1/df.1.html) 226 - [9] [findmnt(8) — Linux manual page](https://man7.org/linux/man-pages/man8/findmnt.8.html) 227 - [10] [file(1) — Linux manual page](https://man7.org/linux/man-pages/man1/file.1.html) 228 - [11] [blkid(8) — Linux manual page](https://man7.org/linux/man-pages/man8/blkid.8.html) 229 - [12] [strings(1) — Linux manual page](https://man7.org/linux/man-pages/man1/strings.1.html) 230 - [13] [crontab(5) — Linux manual page](https://man7.org/linux/man-pages/man5/crontab.5.html) 231 - [14] [systemd.service(5) — Linux manual page](https://man7.org/linux/man-pages/man5/systemd.service.5.html) 232 - [15] [mount_namespaces(7) — Linux manual page](https://man7.org/linux/man-pages/man7/mount_namespaces.7.html) 233 - [16] [chroot(2) — Linux manual page](https://man7.org/linux/man-pages/man2/chroot.2.html) 234 - [17] [stat(1) — Linux manual page](https://man7.org/linux/man-pages/man1/stat.1.html) 235 - [18] [ls(1) — Linux manual page](https://man7.org/linux/man-pages/man1/ls.1.html) 236 - [19] [lsof(8) — Linux manual page](https://man7.org/linux/man-pages/man8/lsof.8.html) 237 - [20] [grep(1) — Linux manual page](https://man7.org/linux/man-pages/man1/grep.1.html) 238 - [21] [readlink(1) — Linux manual page](https://man7.org/linux/man-pages/man1/readlink.1.html) 239 - [22] [cp(1) — Linux manual page](https://man7.org/linux/man-pages/man1/cp.1.html) 240 - [23] [id(1) — Linux manual page](https://man7.org/linux/man-pages/man1/id.1.html)