DEV Community

Cover image for Dealing with files in Python

Dealing with files in Python

Addicted to writing books and teaching. Likes fantasy books, comics and anime
Originally published at ・5 min read

This chapter will discuss the open() built-in function and some of the built-in modules for file processing.

open and close

The open() built-in function is one of the ways to read and write files. The first argument to this function is the filename to be processed. The filename is a relative/absolute path to the location of the file. Rest are keyword arguments that you can configure. The output is a TextIOWrapper object (i.e. a filehandle), which you can use as an iterator. Here's an example:

# default mode is rt i.e. read text
>>> fh = open('ip.txt')
>>> fh
<_io.TextIOWrapper name='ip.txt' mode='r' encoding='UTF-8'>
>>> next(fh)
'hi there\n'
>>> next(fh)
'today is sunny\n'
>>> next(fh)
'have a nice day\n'
>>> next(fh)
Traceback (most recent call last):

# check if the filehandle is active or closed
>>> fh.closed
# close the filehandle
>>> fh.close()
>>> fh.closed
Enter fullscreen mode Exit fullscreen mode

The mode argument specifies what kind of processing you want. Only text mode will be covered in this chapter, which is the default. You can combine options, for example, rb means read in binary mode. Here's the relevant details from the documentation:

  • 'r' open for reading (default)
  • 'w' open for writing, truncating the file first
  • 'x' open for exclusive creation, failing if the file already exists
  • 'a' open for writing, appending to the end of the file if it exists
  • 'b' binary mode
  • 't' text mode (default)
  • '+' open for updating (reading and writing)

The encoding argument is meaningful only in the text mode. You can check the default encoding for your environment using the locale module as shown below. See docs.python: standard encodings and docs.python: Unicode for more details.

>>> import locale
>>> locale.getpreferredencoding()
Enter fullscreen mode Exit fullscreen mode

Here's how Python handles line separation by default, see documentation for more details.



If the given filename doesn't exist, you'll get a FileNotFoundError exception.

>>> open('xyz.txt', 'r', encoding='ascii')
Traceback (most recent call last):
FileNotFoundError: [Errno 2] No such file or directory: 'xyz.txt'
Enter fullscreen mode Exit fullscreen mode

Context manager

Quoting from docs.python: Reading and Writing Files:


with open('ip.txt', 'r', encoding='ascii') as f:
    for ip_line in f:
        op_line = ip_line.rstrip('\n').capitalize() + '.'
Enter fullscreen mode Exit fullscreen mode

Recall that as keyword was introduced in the try-except section. Here's the output of the above program:

$ python3.9
Hi there.
Today is sunny.
Have a nice day.
Enter fullscreen mode Exit fullscreen mode

See The Magic of Python Context Managers for more details.

read, readline and readlines

The read() method gives you entire remaining contents of the file as a single string. The readline() method gives next line of text and readlines() gives all the remaining lines as a list of strings.

>>> open('ip.txt').read()
'hi there\ntoday is sunny\nhave a nice day\n'

>>> fh = open('ip.txt')
# 'readline' is similar to 'next'
# but returns empty string instead of StopIteration exception
>>> fh.readline()
'hi there\n'
>>> fh.readlines()
['today is sunny\n', 'have a nice day\n']
>>> fh.readline()
Enter fullscreen mode Exit fullscreen mode


with open('op.txt', 'w', encoding='ascii') as f:
    f.write('this is a sample line of text\n')
    f.write('yet another line\n')
Enter fullscreen mode Exit fullscreen mode

You can call the write() method on a filehandle to add contents to that file (provided the mode you have set supports writing). Unlike print(), the write() method doesn't automatically add newline characters.

$ python3.9 

$ cat op.txt
this is a sample line of text
yet another line

$ file op.txt
op.txt: ASCII text
Enter fullscreen mode Exit fullscreen mode

If the file already exists, the w mode will overwrite the contents (i.e. existing content will be lost).

You can also use the print() function for writing by passing the filehandle to the file argument. The fileinput module supports in-place editing and other features (examples will be discussed later).

File processing modules

This section gives introductory examples for some of the built-in modules that are handy for file processing. Quoting from docs.python: os:


>>> import os

# current working directory
>>> os.getcwd()

# value of an environment variable
>>> os.getenv('SHELL')

# file size
>>> os.stat('ip.txt').st_size

# check if given path is a file
>>> os.path.isfile('ip.txt')
Enter fullscreen mode Exit fullscreen mode

Quoting from docs.python: glob:


>>> import glob

# list of files (including directories) containing '_file' in their name
>>> glob.glob('*_file*')
['', '']
Enter fullscreen mode Exit fullscreen mode

Quoting from docs.python: shutil:


>>> import shutil

>>> shutil.copy('ip.txt', 'ip_file.txt')
>>> glob.glob('*_file*')
['', 'ip_file.txt', '']
Enter fullscreen mode Exit fullscreen mode

Quoting from docs.python: pathlib:


>>> from pathlib import Path

# use 'rglob' instead of 'glob' if you want to match names recursively
>>> list(Path('programs').glob('*'))
[PosixPath('programs/'), PosixPath('programs/')]
Enter fullscreen mode Exit fullscreen mode

See pathlib module: taming the file system and stackoverflow: How can I iterate over files in a given directory? for more details and examples.

There are specialized modules for structured data processing as well, for example:


  • Write a program that reads a known filename f1.txt which contains a single column of numbers in Python syntax. Your task is to display the sum of these numbers, which is 10485.14 for the given example.

    $ cat f1.txt 
  • Read the documentation for glob.glob() and write a program to list all files ending with .txt in the current directory as well as sub-directories, recursively.

Discussion (0)