How to Remove Line Breaks in Linux & Bash (tr, sed, awk & perl)

Why Line Break Removal in Linux Is More Complex Than It Looks

In the Unix philosophy, text streams are organized as lines separated by an ASCII 0x0A Line Feed (\n). But when you try to remove them, standard expectations quickly collide with how core utilities buffer data.

Nearly every programmer who opens a terminal has tried typing sed 's/\n//g' filename, only to watch the tool spit out the exact same text completely untouched. Why does that happen? Because standard Unix line processors — including sed and grep — read input into memory one line at a time, strip the newline delimiter before processing, run their rules, and then re-append a newline character when writing to standard output. The newline character is literally invisible to the pattern matcher during normal execution.

Furthermore, text arriving from external environments introduces encoding and carriage return complications. Text copied from a Windows server, an email client, or an uploaded CSV often carries CRLF (\r\n, byte sequence 0x0D 0x0A). If you strip the newline (\n) but leave the carriage return (\r), terminal outputs overwrite themselves, JSON parsers fail, and database imports crash with mysterious control character errors.

To clean your data cleanly, you need to understand four distinct dimensions:

• Character Deletion vs. Replacement: Do you want words glued together (word1word2) or separated by a clean single space (word1 word2)?
• Paragraph Boundary Preservation: Are you flattening an entire document into a single line, or joining soft wraps inside paragraphs while retaining double blank lines?
• Memory Footprint & Stream Processing: Can the utility process a 20 GB log file in a stream, or will it attempt to slurp the entire file into RAM and trigger the Linux kernel Out-Of-Memory (OOM) killer?
• Cross-Platform Carriage Returns: Does the file contain hidden Windows \r bytes that require sanitization alongside Unix \n?

1. The tr Command: Maximum Speed Stream Translation

When your goal is purely to strip or substitute newline characters across a stream without needing conditional regex logic, tr (translate) is your best friend. Written in optimized C and part of GNU Coreutils, tr does not build complex syntax trees or parse grammar. It reads raw bytes from stdin and writes transformed bytes directly to stdout.

Deleting Line Breaks Completely

If you have a base64 string, an encrypted key, or an unbroken hash that was accidentally split across 80-character lines, you want to remove the newlines entirely without inserting spaces:

# Remove all Unix LF line breaks (leaves no spaces between words)
tr -d '\n' < broken_key.txt > clean_key.txt

# Remove both Windows CRLF (\r and \n) safely
tr -d '\r\n' < windows_dump.txt > clean_single_line.txt

Replacing Line Breaks with Spaces

When dealing with sentences, paragraphs, or lists of names, deleting newlines directly causes the last word of line 1 to collide with the first word of line 2 (e.g., "hellothere" instead of "hello there"). Use tr to translate every newline into a standard space:

# Replace every newline with a single space
tr '\n' ' ' < multiline_paragraph.txt > single_line.txt

# Pipe from curl or cat directly
cat server_logs.txt | tr '\n' ' ' | grep "ERROR"

2. The sed Command: In-Place Editing & Multi-Line Pattern Buffers

The stream editor sed is installed on virtually every Linux distribution and macOS installation. While its default behavior prevents single-line matching of newlines, its branching instructions allow you to build an accumulator that loads the entire file into its pattern space before applying a global substitution.

The Classic sed Accumulator Loop Explained

Here is the standard POSIX-compliant sed command used by system administrators to replace all newlines with spaces:

sed ':a;N;$!ba;s/\n/ /g' input.txt > output.txt

Let us dissect why this specific command works when s/\n/ /g fails on its own:

1. :a creates a label named a in the script execution flow.
2. N appends the next line of input into the pattern space, preserving the embedded newline character between the current line and the next.
3. $!ba checks if the parser has reached the last line ($). If not at the last line ($!), it branches (b) back to label a. This loops continuously until the entire document resides in a single unified pattern space.
4. s/\n/ /g runs once the loop completes, replacing every embedded newline character with a single space.

In-Place File Replacement

One major advantage of sed over tr is its ability to edit files in-place without manual temporary file creation:

# In-place edit on Linux (GNU sed)
sed -i ':a;N;$!ba;s/\n/ /g' config_list.txt

# In-place edit on macOS / BSD sed (requires empty extension string)
sed -i '' ':a;N;$!ba;s/\n/ /g' config_list.txt

3. The awk Command: Record Separators & Paragraph Awareness

awk is designed around the concept of records and fields. By default, records are separated by newlines, but you can override both the Input Record Separator (RS) and the Output Record Separator (ORS) to achieve elegant newline transformations with minimal code.

Method A: Modifying the Output Record Separator (ORS)

To join all lines with a space while streaming line-by-line (which uses virtually zero memory):

# Replaces every newline with a space
awk -v ORS=' ' '1' input.txt > output.txt

# Replaces every newline with a comma and space (ideal for CSV lists)
awk -v ORS=', ' '1' ids.txt > csv_row.txt

In this command, -v ORS=' ' changes the string printed after every record from a newline to a space, and the trailing '1' is idiomatic awk for "print the current record without modification."

Method B: Preserving Paragraphs (The Paragraph Mode)

What if you copied text from a PDF, an academic thesis, or an email, where soft wraps need to be joined, but genuine blank lines between paragraphs must remain untouched? Setting RS="" puts awk into special paragraph mode:

# Preserves double blank lines between paragraphs while joining single line breaks
awk -v RS="" -v ORS="\n\n" '{gsub(/\n/, " "); print}' article.txt > cleaned_article.txt

In this mode, any sequence of two or more blank lines is treated as a record boundary. Within each record (paragraph), gsub(/\n/, " ") replaces internal single line breaks with spaces. The output is then followed by \n\n, restoring the visual paragraph structure. This is equivalent to our web-based Preserve Paragraphs tool, executed natively inside your Linux shell.

4. paste & xargs: The Cleanest One-Liners

If you have a file containing a list of server hostnames, user IDs, or Git commit hashes, you usually want to serialize them into a single horizontal line for an argument list or shell variable. Writing loops or complex regex is overkill when paste or xargs is available.

Using paste for Serial Delimiting

The paste utility merges lines of files. With the -s (serial) and -d (delimiter) flags, it collapses all lines into a single row using whatever delimiter character you specify:

# Join lines with a single space
paste -sd " " servers.txt

# Join lines with a comma (perfect for SQL IN clauses: 'id1,id2,id3')
paste -sd "," ids.txt

# Join lines with no delimiter at all (glues characters together)
paste -sd "" chunks.txt

Using xargs for Command Argument Flattening

Because xargs natively splits on whitespace and newlines, feeding a file to xargs automatically outputs all tokens separated by single spaces on a single line:

# Flattens all lines into space-separated items
xargs < wordlist.txt

# Capture into an environment variable
HOSTS=$(xargs < hosts.txt)
echo "Deploying to: $HOSTS"

5. Perl One-Liners: Full Regex Power with Slurp Mode

When standard POSIX utilities hit their limit — for instance, when dealing with complex lookaround assertions, Unicode normalization, or variable CRLF sequences — Perl one-liners provide battle-tested reliability without compiling external binaries.

# Replace all newlines with spaces across a file in-place
perl -i -pe 's/\r?\n/ /g' document.txt

# Slurp mode (-0777): loads the whole file and replaces single newlines while keeping \n\n
perl -0777 -pe 's/(?<!\n)\n(?!\n)/ /g' report.txt > cleaned_report.txt

The -0777 switch instructs Perl to read the entire file as a single scalar string (slurp mode). The negative lookbehind (?<!\n) and negative lookahead (?!\n) ensure that only lonely single newlines are matched. If two or more newlines exist together, the regex skips them, preserving intentional paragraph separation.

Linux Tool Performance & Feature Matrix

To help you select the exact command for your script or automated pipeline, the following table summarizes memory behavior, maximum speed, and feature capabilities:

Real-World Automation Recipes

Cleaning Multiline JSON or Base64 in Bash Pipelines

When fetching OAuth tokens or SSH keys via API or curl, responses often include unwanted breaks:

# Strip formatting newlines from a public SSH key
TOKEN=$(curl -s https://api.example.com/v1/token | tr -d '\r\n')
echo "Clean Bearer: $TOKEN"

Batch Cleaning Multiple Files in a Directory

If you have a directory of raw text files exported from a document management system, loop through them using a safe Bash subshell:

# Clean every .txt file in the current directory and save to clean/
mkdir -p clean_output
for file in *.txt; do
  [ -f "$file" ] || continue
  awk -v RS="" -v ORS="\n\n" '{gsub(/\n/, " "); print}' "$file" > "clean_output/$file"
done
echo "Batch processing finished successfully."

Pre-Execution Checklist

• Character Encoding Verified: Check if your file uses Windows CRLF line endings with file input.txt. If it returns with CRLF line terminators, use tr -d '\r' or dos2unix first.
• File Size Checked: If the file exceeds 500 MB, avoid sed ':a;N;...' and perl -0777 to prevent memory thrashing. Stick to tr or streaming awk.
• Spacing Intent Confirmed: Determine whether you want words merged with no space (tr -d '\n') or joined with spaces (tr '\n' ' ').
• Paragraph Structure Evaluated: If the document contains multi-paragraph prose, use awk -v RS="" or our Preserve Paragraphs tool rather than blind global replacement.
• Original File Backed Up: When using in-place flags (-i), always make a copy first (e.g. sed -i.bak ...).

Frequently Asked Questions

Explore Related Tools & Tutorials