Skip to main content

Pipes and Composability

If you're new to programming

Pipes (|) connect programs: one program's output becomes another program's input. Your SmartNotes CLI can participate in this chain. This lesson teaches two things: how to read data that another program sends to you, and how to write output that another program can consume.

If you've coded before

sys.stdin iteration, stdout as the universal interface. You saw composable tools in Ch 20 when Claude Code built sum.py reading from stdin. Now you write them yourself, with isatty() detection for dual-mode commands.

In Lesson 4, James added exit codes and stderr to SmartNotes. The CLI can now report errors properly. But it still lives in isolation: every command reads from its own database and prints to the terminal. It cannot participate in a chain of tools.

James remembers Chapter 20. Claude Code built a sum.py that read numbers from stdin and printed the total. He piped cat numbers.txt | python sum.py and watched the output appear. He understood the philosophy then. Now he needs to write it himself.

"SmartNotes export writes JSON to the terminal," he tells Emma. "Could I pipe that into another tool? Like smartnotes export --format json | python analyze.py?"

"That is exactly what pipes are for," Emma says. "And it works in the other direction too. Someone could pipe data into SmartNotes: cat backup.json | smartnotes import."

Reading from stdin​

sys.stdin is a file-like object. You can iterate over it line by line, the same way you iterate over an open file. When another program pipes data to your script, that data arrives through sys.stdin.

Create a file called import_notes.py:

import sys
import json


def import_from_stdin() -> int:
    """Read JSON notes from stdin and save them."""
    raw = sys.stdin.read()
    notes = json.loads(raw)
    # For now, just count what we received
    return len(notes)


if __name__ == "__main__":
    count = import_from_stdin()
    print(f"Imported {count} notes from stdin")

Test it by piping a JSON file:

cat notes.json | python import_notes.py

Output:

Imported 3 notes from stdin

sys.stdin.read() consumes all piped input as a single string. json.loads() parses it. The script has no idea where the data came from: a file, another program, or a curl command. It only knows that data arrived on stdin.

For large inputs, you can also read line by line:

import sys

for line in sys.stdin:
    stripped = line.strip()
    if stripped:
        print(f"Got: {stripped}")

This processes one line at a time without loading the entire input into memory.

Writing to stdout for Downstream Tools​

Your SmartNotes export command already prints JSON to the terminal. That is stdout. Any program that prints to stdout can be the left side of a pipe.

Three ways to use the same command:

# 1. Print to terminal (human reads it)
smartnotes export --format json

# 2. Redirect to file (save for later)
smartnotes export --format json > backup.json

# 3. Pipe to another tool (program reads it)
smartnotes export --format json | python count_words.py

The export command does not change between these three uses. The shell handles the routing. When you write > backup.json, the shell redirects stdout to a file. When you write | python count_words.py, the shell connects stdout to the next program's stdin.

Here is a simple downstream tool that counts total words across all exported notes:

import sys
import json


def count_words() -> None:
    """Read JSON notes from stdin and count total words."""
    data = json.loads(sys.stdin.read())
    total = 0
    for note in data:
        total += note.get("word_count", 0)
    print(f"Total words: {total}")


if __name__ == "__main__":
    count_words()
smartnotes export --format json | python count_words.py

Output:

Total words: 250

The export command writes JSON to stdout. The counting script reads JSON from stdin. Neither knows about the other. They communicate through the pipe.

Detecting Piped Input​

Sometimes a command should work both ways: read from a pipe when piped, or read from a file argument when run directly. sys.stdin.isatty() tells you which mode you are in. (The name tty stands for "teletypewriter," a historical name for a text terminal. isatty() returns True when stdin is connected to an interactive terminal, False when input is piped from another program.)

import sys

if sys.stdin.isatty():
    print("Running interactively (no pipe)")
else:
    print("Receiving piped input")
# Interactive mode
python detect.py

Output:

Running interactively (no pipe)
# Piped mode
echo "hello" | python detect.py

Output:

Receiving piped input

isatty() returns True when stdin is connected to a terminal (the user is typing). It returns False when stdin is connected to a pipe or a file redirect.

Use this to build a dual-mode import command:

import sys
import json
import argparse
from pathlib import Path


def import_notes(args: argparse.Namespace) -> None:
    """Import notes from stdin (if piped) or from a file argument."""
    if not sys.stdin.isatty():
        # Data is being piped in
        raw = sys.stdin.read()
        source = "stdin"
    elif args.file:
        # File argument provided
        raw = Path(args.file).read_text()
        source = args.file
    else:
        print("Error: provide a filename or pipe data to stdin", file=sys.stderr)
        sys.exit(1)

    notes = json.loads(raw)
    print(f"Imported {len(notes)} notes from {source}")


def main() -> None:
    parser = argparse.ArgumentParser(description="Import notes")
    parser.add_argument("file", nargs="?", help="JSON file to import")
    args = parser.parse_args()
    import_notes(args)


if __name__ == "__main__":
    main()

Three ways to call it:

# From a file argument
python import_cmd.py backup.json

# From a pipe
cat backup.json | python import_cmd.py

# Neither (error)
python import_cmd.py

Output (file argument):

Imported 3 notes from backup.json

Output (pipe):

Imported 3 notes from stdin

Output (neither):

Error: provide a filename or pipe data to stdin

The same command handles both cases. The isatty() check is invisible to the user. They pipe data or pass a filename and it works.

Testing CLI Commands with subprocess.run​

Composable tools are only trustworthy if you can verify they work. You have been testing Python code with pytest throughout this book. But how do you test a CLI tool? You run it as a separate process and check the output. Python's subprocess.run does exactly that:

import subprocess

result: subprocess.CompletedProcess[str] = subprocess.run(
    ["smartnotes", "list"],
    capture_output=True,
    text=True,
)

print(result.stdout)       # what the command printed to stdout
print(result.stderr)       # what it printed to stderr
print(result.returncode)   # exit code: 0 = success, non-zero = failure

capture_output=True captures stdout and stderr instead of printing them. text=True returns strings instead of bytes. The returned CompletedProcess object gives you everything you need to assert in a test:

def test_list_command() -> None:
    result = subprocess.run(
        ["smartnotes", "list"],
        capture_output=True,
        text=True,
    )
    assert result.returncode == 0
    assert "notes" in result.stdout.lower()

You will use this pattern in the capstone (Lesson 6) to test every subcommand of your SmartNotes CLI.

PRIMM-AI+ Practice: Predict the Pipe​

Predict [AI-FREE]​

Press Shift+Tab to enter Plan Mode.

Given this command:

echo '[{"title": "A", "word_count": 10}, {"title": "B", "word_count": 20}]' | python count_words.py
  1. What does sys.stdin.read() return inside count_words.py?
  2. What does json.loads() produce from that string?
  3. What is the final output?

Write your answers before running.

Check your prediction
  1. sys.stdin.read() returns the JSON string: '[{"title": "A", "word_count": 10}, {"title": "B", "word_count": 20}]'
  2. json.loads() produces a Python list of two dictionaries.
  3. Output: Total words: 30

Run​

Press Shift+Tab to exit Plan Mode.

Create the count_words.py file and run the echo/pipe command. Verify the output matches your prediction.

Investigate​

Run the dual-mode import command three ways: with a file, with a pipe, and with no input. Compare the outputs. Then ask Claude Code:

What happens if I pipe malformed JSON into import_cmd.py?
Where should I add error handling?

The AI identifies the missing try/except around json.loads() and shows how to report the parse error to stderr with a non-zero exit code.

Modify​

Add an --output flag to count_words.py. When provided, write the result to a file instead of stdout. When not provided, default to stdout (preserving pipe compatibility). Use argparse for the flag.

Make [Mastery Gate]​

Write a dual-mode import subcommand for SmartNotes that reads JSON notes from stdin or from a file argument, validates each note has title and body fields, and adds them to the SmartNotes database. In Claude Code, type /tdg to guide you through the cycle:

  1. Write the stub with typed signature: def import_notes(source: str) -> list[Note] that accepts a JSON string and returns parsed notes
  2. Write tests by calling the function directly with JSON strings (test valid JSON, empty list, invalid JSON, missing fields). You will learn to test the CLI command itself via subprocess.run in the capstone.
  3. Prompt AI to implement
  4. Run uv run ruff check, uv run pyright, uv run pytest

Try With AI​

Opening Claude Code

If Claude Code is not already running, open your terminal, navigate to your SmartNotes project folder, and type claude. If you need a refresher, Chapter 44 covers the setup.

Prompt 1: Review My Pipe Handling​

Here is my dual-mode import command:

[paste your import_cmd.py]

Review the stdin handling. What happens if stdin is
piped but empty? What if the JSON is valid but contains
no notes (empty array)? What edge cases am I missing?

What you're learning: Pipe handling has subtle edge cases. Empty pipes, partial writes, and encoding issues are real problems. The AI identifies failure modes beyond the happy path.

Prompt 2: Build a Filter Tool​

I want to write a SmartNotes filter tool that reads JSON
notes from stdin, filters by a tag given as a command-line
argument, and writes matching notes to stdout as JSON.

Usage: smartnotes export --format json | python filter_notes.py python

Use argparse for the tag argument and sys.stdin for input.
Write it with type hints and error handling.

What you're learning: You are building a composable tool from scratch. The filter reads JSON in, writes JSON out. It can be placed anywhere in a pipe chain: export | filter python | count_words. This is the Unix philosophy in practice.

Prompt 3: Explore Real Pipe Chains​

In Claude Code, type:

Show me 5 examples of useful pipe chains I could build
with SmartNotes tools:
1. export | filter | count
2. export | jq (if installed)
3. external data | import
4. export | grep
5. export | sort

For each chain, show the exact command and explain what
each stage does.

What you're learning: Composability multiplies the value of each tool. Five simple tools create dozens of useful combinations. You do not need to build every feature into SmartNotes; you build the connectors and let the ecosystem fill the gaps.

James watches the pipe chain run: export, filter, count. Three separate programs, each doing one thing.

"In the warehouse," he says, "every station had the same input/output connector. Packages arrived on the same size pallet, left on the same size pallet. The station inside was different: one sorted by weight, another by destination, another by priority. But the interface was universal. That is stdin and stdout."

Emma nods. "And when a station broke, you replaced it without redesigning the whole line."

"We swapped the sorting station once. Took an hour. The packing station never knew."

"Same here. If you rewrite filter_notes.py to be faster, count_words.py does not change. They communicate through the pipe, not through each other's code."

James looks at the SmartNotes project. Add, search, list, export, import. Five subcommands. Error handling. Pipes. "It is starting to feel like a real tool."

"It is a real tool," Emma says. "One thing remains: making it work as a system command. Right now you type python -m smartnotes add. Real tools are just smartnotes add. That is the capstone."