sponge.md (10812B)
1 --- 2 title: "sponge" 3 description: "sponge is a command-line utility from the moreutils package that reads all input from stdin before writing to a file. Unlike standard shell redirects (>)โฆ" 4 category: tools 5 tags: ["tools", "adcs"] 6 tools: [] 7 difficulty: intermediate 8 updated: "2026-08-10" 9 source: "vault:Misc/sponge-cheatsheet.md" 10 --- 11 # ๐งฝ Sponge Command Cheat Sheet 12 13 ## ๐ Introduction 14 15 `sponge` is a command-line utility from the **moreutils** package that reads all input from stdin before writing to a file. Unlike standard shell redirects (`>`), which open the output file immediately (potentially truncating it before reading completes), `sponge` buffers the entire input in memory first, then writes it out. 16 17 This "read-all-then-write" behavior makes `sponge` essential for **safely modifying files in-place** within pipelines. 18 19 ### ๐ง Installation 20 21 ```bash 22 # Debian/Ubuntu 23 sudo apt install moreutils 24 25 # Fedora/RHEL 26 sudo dnf install moreutils 27 28 # Arch Linux 29 sudo pacman -S moreutils 30 31 # macOS (Homebrew) 32 brew install moreutils 33 ``` 34 35 ### โ๏ธ Basic Syntax 36 37 ```bash 38 command | sponge [OPTIONS] filename 39 ``` 40 41 ### ๐๏ธ Options 42 43 | Option | Description | 44 |--------|-------------| 45 | `-a` | Append to file instead of overwriting | 46 | No options | Overwrite the file with buffered content | 47 48 --- 49 50 ## โ ๏ธ The Problem Sponge Solves 51 52 Without `sponge`, this command **destroys your data**: 53 54 ```bash 55 # โ DANGEROUS - file gets truncated before reading completes 56 grep 'pattern' file.txt > file.txt 57 # Result: empty file! 58 ``` 59 60 The shell opens `file.txt` for writing (truncating it) before `grep` starts reading. 61 62 With `sponge`, the operation is **safe**: 63 64 ```bash 65 # โ SAFE - sponge buffers all input before writing 66 grep 'pattern' file.txt | sponge file.txt 67 ``` 68 69 --- 70 71 ## ๐ Using Sponge with grep 72 73 `grep` searches for patterns in files. Combined with `sponge`, you can filter files in-place. 74 75 ### Keep Only Matching Lines 76 77 ```bash 78 # Keep only lines containing 'error' 79 grep 'error' logfile.txt | sponge logfile.txt 80 ``` 81 82 ### Remove Matching Lines 83 84 ```bash 85 # Remove all lines containing 'DEBUG' 86 grep -v 'DEBUG' application.log | sponge application.log 87 ``` 88 89 ### Remove Comments from Config Files 90 91 ```bash 92 # Remove lines starting with # (comments) 93 grep -v '^#' config.conf | sponge config.conf 94 ``` 95 96 ### Remove Empty Lines 97 98 ```bash 99 # Remove blank lines from a file 100 grep -v '^$' data.txt | sponge data.txt 101 ``` 102 103 ### Case-Insensitive Filtering 104 105 ```bash 106 # Keep lines containing 'warning' (case-insensitive) 107 grep -i 'warning' alerts.log | sponge alerts.log 108 ``` 109 110 ### Chain Multiple grep Commands 111 112 ```bash 113 # Remove comments AND empty lines 114 grep -v '^#' config.conf | grep -v '^$' | sponge config.conf 115 ``` 116 117 --- 118 119 ## โ๏ธ Using Sponge with sed 120 121 `sed` (stream editor) performs text transformations. With `sponge`, you can apply complex edits in-place safely. 122 123 ### Simple Find and Replace 124 125 ```bash 126 # Replace 'old' with 'new' (first occurrence per line) 127 sed 's/old/new/' file.txt | sponge file.txt 128 ``` 129 130 ### Global Replacement 131 132 ```bash 133 # Replace ALL occurrences of 'foo' with 'bar' 134 sed 's/foo/bar/g' file.txt | sponge file.txt 135 ``` 136 137 ### Case-Insensitive Replacement 138 139 ```bash 140 # Replace 'error' regardless of case 141 sed 's/error/warning/gi' logfile.txt | sponge logfile.txt 142 ``` 143 144 ### Delete Specific Lines 145 146 ```bash 147 # Delete line 5 148 sed '5d' file.txt | sponge file.txt 149 150 # Delete lines 10 through 20 151 sed '10,20d' file.txt | sponge file.txt 152 153 # Delete the last line 154 sed '$d' file.txt | sponge file.txt 155 ``` 156 157 ### Delete Lines Matching Pattern 158 159 ```bash 160 # Delete all lines containing 'obsolete' 161 sed '/obsolete/d' file.txt | sponge file.txt 162 ``` 163 164 ### Remove Leading/Trailing Whitespace 165 166 ```bash 167 # Remove leading whitespace 168 sed 's/^:space:*//' file.txt | sponge file.txt 169 170 # Remove trailing whitespace 171 sed 's/:space:*$//' file.txt | sponge file.txt 172 173 # Remove both 174 sed 's/^:space:*//;s/:space:*$//' file.txt | sponge file.txt 175 ``` 176 177 ### Convert Tabs to Spaces 178 179 ```bash 180 # Replace tabs with 4 spaces 181 sed 's/\t/ /g' code.py | sponge code.py 182 ``` 183 184 ### Multiple Substitutions 185 186 ```bash 187 # Chain multiple replacements 188 sed -e 's/apple/orange/g' -e 's/banana/grape/g' fruits.txt | sponge fruits.txt 189 ``` 190 191 ### Remove Carriage Returns (Windows โ Unix) 192 193 ```bash 194 # Convert Windows line endings to Unix 195 sed 's/\r$//' file.txt | sponge file.txt 196 ``` 197 198 --- 199 200 ## โ๏ธ Using Sponge with cut 201 202 `cut` extracts sections from each line of input. With `sponge`, you can trim files down to specific columns. 203 204 ### Extract Specific Field (Delimiter-Based) 205 206 ```bash 207 # Extract usernames (first field) from /etc/passwd format 208 cut -d':' -f1 users.txt | sponge users.txt 209 ``` 210 211 ### Extract Multiple Fields 212 213 ```bash 214 # Keep fields 1 and 3 (colon-delimited) 215 cut -d':' -f1,3 data.txt | sponge data.txt 216 ``` 217 218 ### Extract Range of Fields 219 220 ```bash 221 # Keep fields 2 through 5 222 cut -d',' -f2-5 data.csv | sponge data.csv 223 ``` 224 225 ### Extract by Character Position 226 227 ```bash 228 # Keep only first 20 characters of each line 229 cut -c1-20 file.txt | sponge file.txt 230 ``` 231 232 ### Extract with Tab Delimiter (Default) 233 234 ```bash 235 # Extract second column (tab-separated) 236 cut -f2 data.tsv | sponge data.tsv 237 ``` 238 239 ### Remove a Specific Column 240 241 ```bash 242 # Keep all fields EXCEPT field 3 243 cut -d',' -f1,2,4- data.csv | sponge data.csv 244 ``` 245 246 ### Extract from Specific Position to End 247 248 ```bash 249 # Keep characters 10 to end of each line 250 cut -c10- file.txt | sponge file.txt 251 ``` 252 253 --- 254 255 ## ๐ Using Sponge with paste 256 257 `paste` merges lines from multiple files side by side. With `sponge`, you can combine and overwrite efficiently. 258 259 ### Merge Two Files Side by Side 260 261 ```bash 262 # Combine file1 and file2 with tabs between them 263 paste file1.txt file2.txt | sponge combined.txt 264 ``` 265 266 ### Merge with Custom Delimiter 267 268 ```bash 269 # Combine files with comma separator 270 paste -d',' file1.txt file2.txt | sponge combined.csv 271 ``` 272 273 ### Convert Column to Row (Serial Mode) 274 275 ```bash 276 # Join all lines into a single line with commas 277 paste -sd',' file.txt | sponge file.txt 278 ``` 279 280 ### Create Tab-Separated Output 281 282 ```bash 283 # Merge three files with tabs 284 paste file1.txt file2.txt file3.txt | sponge merged.tsv 285 ``` 286 287 ### Transpose Single File (Column โ Row) 288 289 ```bash 290 # Convert newline-separated values to comma-separated 291 cat values.txt | paste -sd',' | sponge values.txt 292 # Input: apple Output: apple,banana,cherry 293 # banana 294 # cherry 295 ``` 296 297 ### Merge with Multiple Delimiters 298 299 ```bash 300 # Alternate between colon and newline 301 paste -d':\n' file1.txt file2.txt | sponge result.txt 302 ``` 303 304 ### Combine paste with cut 305 306 ```bash 307 # Extract field 1 from both files and merge 308 paste <(cut -d',' -f1 a.csv) <(cut -d',' -f1 b.csv) | sponge ids.txt 309 ``` 310 311 --- 312 313 ## ๐ข Using Sponge with sort 314 315 `sort` orders lines alphabetically or numerically. With `sponge`, you can sort files in-place. 316 317 ### Basic Alphabetical Sort 318 319 ```bash 320 # Sort lines alphabetically 321 sort file.txt | sponge file.txt 322 ``` 323 324 ### Reverse Sort 325 326 ```bash 327 # Sort in descending order 328 sort -r file.txt | sponge file.txt 329 ``` 330 331 ### Numeric Sort 332 333 ```bash 334 # Sort numerically (not lexicographically) 335 sort -n numbers.txt | sponge numbers.txt 336 ``` 337 338 ### Sort and Remove Duplicates 339 340 ```bash 341 # Sort and keep only unique lines 342 sort -u file.txt | sponge file.txt 343 ``` 344 345 ### Sort by Specific Column 346 347 ```bash 348 # Sort by second column (comma-delimited) 349 sort -t',' -k2 data.csv | sponge data.csv 350 351 # Sort by third column numerically 352 sort -t',' -k3n data.csv | sponge data.csv 353 ``` 354 355 ### Case-Insensitive Sort 356 357 ```bash 358 # Ignore case when sorting 359 sort -f names.txt | sponge names.txt 360 ``` 361 362 ### Human-Readable Numeric Sort 363 364 ```bash 365 # Sort file sizes (1K, 2M, 3G, etc.) 366 sort -h sizes.txt | sponge sizes.txt 367 ``` 368 369 ### Sort IP Addresses 370 371 ```bash 372 # Sort IP addresses correctly 373 sort -t'.' -k1,1n -k2,2n -k3,3n -k4,4n ips.txt | sponge ips.txt 374 ``` 375 376 ### Stable Sort 377 378 ```bash 379 # Preserve original order for equal elements 380 sort -s -k2 data.txt | sponge data.txt 381 ``` 382 383 --- 384 385 ## ๐ Combining Multiple Tools 386 387 The real power comes from chaining tools together. 388 389 ### Clean and Sort Config File 390 391 ```bash 392 # Remove comments, empty lines, and sort 393 grep -v '^#' config.conf | grep -v '^$' | sort -u | sponge config.conf 394 ``` 395 396 ### Extract, Transform, and Sort 397 398 ```bash 399 # Get usernames, make lowercase, sort uniquely 400 cut -d':' -f1 passwd.txt | sed 's/.*/\L&/' | sort -u | sponge users.txt 401 ``` 402 403 ### Filter and Format Log Entries 404 405 ```bash 406 # Extract error lines, keep timestamp and message, sort by time 407 grep 'ERROR' app.log | cut -d' ' -f1,4- | sort | sponge errors.txt 408 ``` 409 410 ### Deduplicate and Clean Data 411 412 ```bash 413 # Remove duplicates, trim whitespace, sort 414 sort data.txt | uniq | sed 's/^:space:*//;s/:space:*$//' | sponge data.txt 415 ``` 416 417 ### CSV Processing Pipeline 418 419 ```bash 420 # Remove header, sort by column 2, add header back 421 tail -n +2 data.csv | sort -t',' -k2 | (head -1 data.csv && cat) | sponge data.csv 422 ``` 423 424 --- 425 426 ## ๐ Appending with Sponge 427 428 Use the `-a` flag to append instead of overwrite. 429 430 ```bash 431 # Append filtered results to existing file 432 grep 'WARN' new_logs.txt | sponge -a all_warnings.txt 433 ``` 434 435 --- 436 437 ## ๐ก Pro Tips 438 439 1. **Memory Considerations**: `sponge` buffers all input in memory. For very large files (larger than available RAM), consider alternatives or ensure sufficient memory. 440 441 2. **Atomic Operations**: When possible, `sponge` updates files atomically by writing to a temp file first, then renaming. 442 443 3. **Preserve Permissions**: `sponge` attempts to preserve file permissions when overwriting. 444 445 4. **Test First**: Always test your pipeline without `sponge` first to verify output: 446 ```bash 447 # Test output first 448 grep -v 'pattern' file.txt | head 449 450 # Then apply with sponge 451 grep -v 'pattern' file.txt | sponge file.txt 452 ``` 453 454 5. **Backup Important Files**: For critical files, make a backup first: 455 ```bash 456 cp important.txt important.txt.bak 457 sed 's/old/new/g' important.txt | sponge important.txt 458 ``` 459 460 --- 461 462 ## ๐ Quick Reference Table 463 464 | Tool | Common Use with Sponge | Example | 465 |------|------------------------|---------| 466 | `grep` | Filter lines in-place | `grep 'keep' f.txt \| sponge f.txt` | 467 | `grep -v` | Remove lines in-place | `grep -v 'remove' f.txt \| sponge f.txt` | 468 | `sed` | Find/replace in-place | `sed 's/a/b/g' f.txt \| sponge f.txt` | 469 | `cut` | Extract columns in-place | `cut -d',' -f1 f.csv \| sponge f.csv` | 470 | `paste` | Merge files | `paste a.txt b.txt \| sponge c.txt` | 471 | `sort` | Sort in-place | `sort f.txt \| sponge f.txt` | 472 | `sort -u` | Sort + dedupe in-place | `sort -u f.txt \| sponge f.txt` | 473 474 --- 475 476 ## ๐ See Also 477 478 - `tee` - Write to file while also passing to stdout (but not safe for in-place) 479 - `moreutils` - The package containing sponge and other useful utilities 480 - `sed -i` - Built-in in-place editing (but requires temp file internally) 481 - `sort -o` - Sort's built-in in-place output option 482 483 --- 484 485 *Created for efficient command-line text processing workflows* ๐ง