17 Working with Files in Python

Learning Objectives

Be able to read and write files in Python
Understand how to use the with method and a context manager to work with files

17.1 Introduction

Working with files is essential in coding because it allows programs to store, read, and process data beyond a single run of the program. Files make it possible to handle real-world data such as user input, logs, and datasets, enabling programs to be more useful, persistent, and scalable. Imagine if you could not store information with files?

17.2 Working Directory

We have already introduced the idea of a working directory. To briefly recap:

A working directory is the directory within the file system where your program will use as a default to look for and read files or output files. It easier than always defining absolute file paths and can allow code to be transferred.

When running python scripts from the console, the working directory is the one where you launched the script from.

The os module is a library - a collection of functions - that can be used to work with files. It is a standard python library - so you do not need to install anything extra to use it, however, you need to import it to allow your current script to access the functions.

The os module can be used to check the working directory and change it.

Get working Directory:

import os
print(os.getcwd())  #current working directory

Change working Directory:

os.chdir('/path/to/directory')

Like in this course setup, it is recommended to have scripts and data in the same directory, or a subdirectory of the project directory. My common setup might be:

/home/user/project/
    ├── data/
    │   └── dataset.csv
    ├── scripts/
    |   └── analysis.py
    ├── logs/
    │   └── error-log.log
    │   └── output-log.log
    └── output/
        └── output.csv

17.3 Opening and closing files in python

You can open files in different modes using the open() function. The common modes include:

'r': Read (default)
'w': Write (overwrites the file if it exists)
'a': Append (adds content to the end of the file)
'b': Binary mode (for non-text files)
'x': Exclusive creation (fails if the file already exists)

If we make a file named my_notes.txt with the sentences:

These are my notes about python.
You can open files with the open() function.

# Open the file in read mode
file = open('my_notes.txt', 'r')

# Read the contents of the file into a string
my_notes = file.read()

# Close the file
file.close()

# The my_notes variable now holds the content of my_notes.txt
print(my_notes)  # This will display the content of the file

These are my notes about python.
You can open files with the open() function.

file.readlines() will return a list with each line:

file = open('my_notes.txt', 'r')
my_notes_lines = file.readlines()
file.close()
print(my_notes_lines)

['These are my notes about python.\n', 'You can open files with the open() function.\n']

Using for loops to iterate line by line:

file = open('my_notes.txt', 'r')
for n, line in enumerate(file, start=1):
    # Strip trailing whitespace or newline characters from the end of each line
    note_line = line.strip()
    print(f"line number: {n}: {note_line}")  # Process or print the current line
file.close()

line number: 1: These are my notes about python.
line number: 2: You can open files with the open() function.

This method is memory-efficient, especially for large files, since it doesn’t load the entire file into memory at once.

Consider the problems with working with files this way:

Discussion: Exceptions after open()

Level:

If an exception occurs after open() what will happen? What have you already learnt as a method to stop this?

Answer

The file will never close properly - you could use try, except, finally. The with statement presented in the next section is a better way to deal with this.

try:
    file = open('my_notes.txt', 'r')
    my_notes = file.read()
    print(my_notes)
except FileNotFoundError:
    print("The file could not be found.")
finally:
    file.close()

These are my notes about python.
You can open files with the open() function.

The finally block runs whether or not an exception occurred, so the file is guaranteed to close properly.

17.4 Using `with` when working with files:

Using the with statement is a better way to work with files in Python as it ensures that files are properly opened and closed.

Example:

with open('my_notes.txt', 'r') as file:
    for line in file:
        # Strip the newline character from each line and print it
        note_line = line.strip()
        print(note_line)

These are my notes about python.
You can open files with the open() function.

Note it uses the same kind of syntax as for loops and conditionals

with does this by working with a context manager. The open() function returns a fileobject which has the .__enter___() and .__exit__() methods. You can read more about this in your own time.

17.5 Writing to a file

The same methods are used to write to a file:

Write to a new file or replace an existing file

Writing to a new file or overwriting an existing file is simple:

with open('names.txt', 'w') as file:
    file.write("John\n")
    file.write("Martin\n")

file.writelines(): Takes an iterable (like a list) and writes each item to the file. You can also use for loops.

the \n is needed when manually adding lines.

Appending to an existing file

This is exactly the same however you need to replace 'w' with 'a' for append mode.

binary files Use the 'wb' for writing binaries for example.

Reading and writing files

Level:

Use the above in python to first write to a file your expenses in the correct format that our expense_calculator expects it. Then read and append to the file some more expenses. Finally test your file with the expense_calculator script.

17.6 Moving files

The shutil module can be used to move files in python.

import shutil
import os

# Define paths
source_file = 'source_folder/file.txt' 
destination_file = 'destination_folder/file.txt' 

# Create the destination folder if it doesn't exist
os.makedirs(os.path.dirname(destination_file), exist_ok=True)

# Move the file
shutil.move(source_file, destination_file)

The same can be used to move directories. If the main task is moving files around - I would generally avoid using python to move around files, if you can use a bash script instead. Generally python scripts will open, read and create files during for example data analysis. Attend the training facilities unix course to learn more

17.7 Exercises

Exercise 1 - Exercise 1: Finding and changing your working directory

Level:

The os module lets you check and change your current working directory.

Import the os module and use it to print your current working directory.
Create a new folder called practice_files in your current directory (hint: look up os.mkdir()).
Change your working directory into practice_files using os.chdir().
Print your working directory again to confirm it has changed.

Answer

import os

# 1. Print the current working directory
print(os.getcwd())

# 2. Create a new folder
os.mkdir('practice_files')

# 3. Change into the new folder
os.chdir('practice_files')

# 4. Confirm the change
print(os.getcwd())

Exercise 2 - Writing, reading and appending to a file

Level:

Using with, create a file called shopping_list.txt and write three items to it, one per line.
Open the file again and read its contents, printing each item on its own line.
Open the file in append mode and add two more items to it.
Read the file one final time to check all five items are present, this time printing each line with its line number (e.g. 1: eggs).

Answer

# 1. Write the initial list
with open('shopping_list.txt', 'w') as file:
    file.write("eggs\n")
    file.write("milk\n")
    file.write("bread\n")

# 2. Read and print the contents
with open('shopping_list.txt', 'r') as file:
    for line in file:
        print(line.strip())

# 3. Append two more items
with open('shopping_list.txt', 'a') as file:
    file.write("butter\n")
    file.write("tea\n")

# 4. Read again, with line numbers
with open('shopping_list.txt', 'r') as file:
    for n, line in enumerate(file, start=1):
        print(f"{n}: {line.strip()}")

Exercise 3 - Organising files with shutil and os

Level:

Create two folders called raw_data and processed_data.
Create a file called results.txt inside raw_data containing a line of your choice.
Use shutil.move() to move results.txt from raw_data into processed_data.
Use os.listdir() to print the contents of both folders and confirm the file has moved.
Discussion: for a task like this, would a bash script or a Python script be more appropriate, and why?

Answer

import os
import shutil

# 1. Create the folders
os.mkdir('raw_data')
os.mkdir('processed_data')

# 2. Create a file inside raw_data
with open('raw_data/results.txt', 'w') as file:
    file.write("Experiment results go here.\n")

# 3. Move the file
shutil.move('raw_data/results.txt', 'processed_data/results.txt')

# 4. Confirm the move
print("raw_data contains:", os.listdir('raw_data'))
print("processed_data contains:", os.listdir('processed_data'))

Either could work for a simple move like this, but a bash script (e.g. using mv) is generally preferable for straightforward file organisation tasks, since it is quicker to write and does not require starting a Python interpreter. Python is usually a better choice when the moving is only one step within a larger task, such as reading, processing or renaming files based on their contents.

Exercise 4 - Reopening and appending to a file

Level:

Using with, create a file called log.txt and write a single line to it: "Log started."
Close the file (the with block will do this automatically), then reopen log.txt in append mode and add two more lines: "Step 1 complete." and "Step 2 complete."
Reopen the file once more, this time in read mode, and print out every line in the file to check all three lines are present.
Discussion: what would happen to the contents of log.txt if you had accidentally used 'w' instead of 'a' in step 2?

Answer

# 1. Create the file with an initial line
with open('log.txt', 'w') as file:
    file.write("Log started.\n")

# 2. Reopen in append mode and add more lines
with open('log.txt', 'a') as file:
    file.write("Step 1 complete.\n")
    file.write("Step 2 complete.\n")

# 3. Reopen in read mode and print every line
with open('log.txt', 'r') as file:
    for line in file:
        print(line.strip())

Using 'w' instead of 'a' would overwrite the file completely, deleting the "Log started." line before writing the new ones — you would end up with only "Step 1 complete." and "Step 2 complete." in the file, having lost the original content.

Exercise 5 - Discussion — why use both with and try/except?

Level:

Below is a function from file_reader.py that loads expenses from a file. Read it carefully, then discuss why it uses both with and try/except, given that with already handles closing the file safely.

def load_expenses(filename):
    """Uses try-except and a for loop to load data."""
    data = []
    try:
        with open(filename, "r") as file:
            for line in file:
                parts = line.strip().split(",")
                if len(parts) == 2:
                    data.append([parts[0].lower(), float(parts[1])])
    except FileNotFoundError:
        print("No existing records found. Starting fresh.")
    return data

Answer

with and try/except are solving two different problems here, which is why both are needed:

with guarantees that the file is closed properly once the block finishes, whether it finishes normally or an exception occurs partway through. This handles the resource management problem discussed earlier in this section.
try/except handles the situation where open() itself fails, before there is even a file object for with to manage. If filename does not exist, open() raises a FileNotFoundError immediately — with alone cannot catch this or recover from it, so without the surrounding try/except, the program would crash instead of printing a friendly message and returning an empty list.

In short: with looks after closing the file once it is open; try/except looks after what happens if it cannot be opened at all. Using with on its own would still crash on a missing file, and using try/except on its own (with plain open()/file.close()) would reintroduce the risk of the file never closing if an error occurred while reading it.

This pattern — wrapping a with block in try/except — is very common when working with files that may or may not exist yet, such as loading data that is created the first time a program runs successfully.

17.8 Summary

Files can be opened using the open() function, with different modes for reading ('r'), writing ('w'), appending ('a') and binary data ('b')
Always close a file once you have finished with it, otherwise it may not save properly or could remain locked by your program
The with statement is the preferred way to work with files, as it automatically closes the file for you, even if an error occurs
file.read() returns the whole file as a single string, file.readlines() returns a list of lines, and iterating over a file object with a for loop reads it line by line — the most memory-efficient option for large files
Writing to a file uses the same open() function with 'w' (overwrite) or 'a' (append) modes, and remember to add \n manually when writing lines yourself
The shutil module can be used to move files and directories in Python, though a bash script is often a better tool for the job if file-moving is the main task
Many packages, such as pandas, come with their own built-in functions for reading and writing specific file types, which are often more convenient than the built-in open() function

17.1 Introduction

17.2 Working Directory

17.3 Opening and closing files in python

17.4 Using with when working with files:

17.5 Writing to a file

17.6 Moving files

17.7 Exercises

17.8 Summary

17.4 Using `with` when working with files: