In this tutorial, we will explore how to write a simplified, cross-platform version of the grep command line utility.

If you don’t know, grep is a posix utility present on nearly every version of Linux. grep takes a regular expression and an optional list of files, then it outputs every line of the files that match the regular expression.

If you don’t know what regular expressions are, please stay tuned for a regular expression tutorial that I will have coming soon.

Step 1: Setting up the basic structure

First, we are going to set up the basic structure of the program.

Take a look at the code, and then we will explore what it is doing:

import sys

def main(argv):
    return 0

if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))

We import the sys module on the first line, which provides functions and variables for interacting with the operating system and the Python interpreter.

The main function is defined next. It is empty now save for the return 0, but we will soon add our logic here.

The if __name__ == "__main__": block checks if the script is being run directly (i.e., not being imported as a module by another script). If so, it calls the main function with the command-line arguments.

The sys.argv[1:] expression returns the command-line arguments passed to the script, excluding the script name itself.

Step 2: Adding argument parsing

Next, we will add argument parsing to the program.

Take a look at the code, and then we will explore what it is doing:

import sys
import argparse

def main(argv):
    parser = argparse.ArgumentParser(
        prog="grepy",
        description="A grep-inspired tool with different trade-offs",
    )
    parser.add_argument(
        "expression",
        type=str,
        help="The regular expression to search for",
    )
    parser.add_argument(
        "input_files",
        type=argparse.FileType("r"),
        nargs="*",
        default=tuple("-"),
        help="The file(s) to search"
    )
    args = parser.parse_args(argv)
    return 0

if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))

Here, we add argparse to the list of imports at the top, which provides a way to define the command-line interface of the program.

The main function is then updated to create an ArgumentParser object, which defines the available arguments.

In this case, we define two arguments: expression and input_files.

The expression argument is a string that specifies the regular expression to search for, and the input_files argument is a list of files to search in.

The nargs="*" argument specifies that input_files should be a list of files, and the default=tuple("-") argument specifies that if no files are provided, the program should search stdin (i.e., the - character is replaced stdin).

The value "-" is being passed to the tuple factory, I’m doing it like this because the nargs="*" will cause any arguments passed to be in a list, but if no arguments are passed the default value will be used. I chose tuple instead of list because lists are mutable and would be attached to the ArgumentParser instance which creates an oportunity for the value to change unexpectedly.

Step 3: Adding loops to go through each line of the files

Next, we will add in the loops to go through each line of each of the files.

Take a look at the code, and then we will explore what it is doing:

import sys
import argparse

def main(argv):
    parser = argparse.ArgumentParser(
        prog="grepy",
        description="A grep-inspired tool with different trade-offs",
    )
    parser.add_argument(
        "expression",
        type=str,
        help="The regular expression to search for",
    )
    parser.add_argument(
        "input_files",
        type=argparse.FileType("r"),
        nargs="*",
        default=tuple("-"),
        help="The file(s) to search"
    )
    args = parser.parse_args(argv)
    for input_file in args.input_files:
        for line in input_file:
            ...
    return 0

if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))

Here, we add loops to go through each line of the files in turn.

We import the re module, which provides regular expression matching operations.

The main function is then updated to iterate over the files and then to iterate through each line of the file.

Step 4: Adding support for regular expression search

Next, we are going to add the magic of our script with the Python re module

Take a look at the code, and then we will explore what it is doing:

# Add support for regular expression search
import re
import sys
import argparse

def main(argv):
    parser = argparse.ArgumentParser(
        prog="grepy",
        description="A grep-inspired tool with different trade-offs",
    )
    parser.add_argument(
        "expression",
        type=str,
        help="The regular expression to search for",
    )
    parser.add_argument(
        "input_files",
        type=argparse.FileType("r"),
        nargs="*",
        default=tuple("-"),
        help="The file(s) to search"
    )
    args = parser.parse_args(argv)
    expression = re.compile(args.expression)
    for input_file in args.input_files:
        for line in input_file:
            if expression.search(line):
                print(line.strip())
    return 0

if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))

The re module was added to the imports at the top.

Then expression.search(line) is used to see if the line matches the regular expression, and if it matches, the print(line.strip()) statement prints the matched line.

Step 5: Adding support for filenames and line numbers

There are two common options that I almost always end up passing to grep, so let’s add them now.

Take a look at the code, and then we will explore what it is doing:

import re
import sys
import argparse

def main(argv):
    parser = argparse.ArgumentParser(
        prog="grepy",
        description="A grep-inspired tool with different trade-offs",
    )
    parser.add_argument(
        "-n", "--line-number",
        action="store_true",
        help="Print the line number with the match",
    )
    parser.add_argument(
        "-H", "--with-filename",
        action="store_true",
        help="Print the filename with the match",
    )
    parser.add_argument(
        "expression",
        type=str,
        help="The regular expression to search for",
    )
    parser.add_argument(
        "input_files",
        type=argparse.FileType("r"),
        nargs="*",
        default=tuple("-"),
        help="The file(s) to search"
    )
    args = parser.parse_args(argv)
    expression = re.compile(args.expression)
    for input_file in args.input_files:
        for line_number, line in enumerate(input_file):
            if expression.search(line):
                message = line.strip()
                if args.line_number:
                    message = f"{line_number}: {message}"
                if args.with_filename:
                    message = f"{input_file.name}: {message}"
                print(message)
    return 0

if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))

Here, we add support for adding the filenames and line numbers to the output.

The first change you will probably notice are the updates the main function to define two additional arguments: --line-number and --with-filename. I tried to add these options with the same names as are used in GNU grep.

The --line-number argument specifies whether to print the line number with the match, and the --with-filename argument specifies whether to print the filename with the match.

The enumerate(input_file) function returns an iterator over the lines of the input file, along with their line numbers so we can add them to the output.

The if expression.search(line) call checks if the line matches the regular expression, and the print(message) statement prints the matched line with the specified options.

Conclusion

Thanks for taking the time to read this, I tried to make it as accessible and sensible as I could, but if you have any questions or concerns, please feel free to open an issue in the Github repo for this blog.

A Flexible CLI Template using just the standard library

I often need to create little CLI scripts, and I usually reach for Python for this task. Now there are many easy ways to make sensible CLIs with Python such as click, cliff and others.

However, reaching for third party libraries, while frequently necessary, can lead to problems later on as you continue to support your utility. Sometimes libraries become unmaintained or introduce incompatibilities and leave you with extra work updating all of your scripts.

Luckily, Python’s standard library contains everything you need to have a sensible, featureful CLI and has a pretty strong history of maintaining backwards compatibility.

In this article, we will explore the template I made to kickstart simple CLI scripts using the Python standard library.

The code

The template (at of the time of writing) is shown below, I am pasting it here in its entirety and I will refer to pieces of the code throughout this article.

NOTE: You can find the latest version of this template in the repository here, please feel free to open issues or submit pull requests.

"""This script is meant to serve as an example and a
template for writing programs with a Command Line
Interface.

Features Include:
    * -vvv style logging verbosity arguments
    * Option to output logs to a file in addition to stderr
    * Sensible Return Codes
    * Intuitive code organization
    * Input file that defaults to stdin
    * Output file that defaults to stdout 

This code is released under the GPL v3.

Feel free to copy this file `command.py` for the scaffolding
of the script and the file `test_command.py` for the tests.
"""
import sys
import pathlib
import logging
import argparse

LOG_LEVELS = [
    'CRITICAL',
    'FATAL',
    'ERROR',
    'WARNING',
    'INFO',
    'DEBUG',
]

RETURN_CODES = {
    "SUCCESS": 0,
    "UNHANDLED_EXCEPTION": 1,
}

def parse_args(argv):
    """This function is responsible for parsing the command line
    arguments as well as any required post-processing.
    """
    parser = argparse.ArgumentParser()
    parser.add_argument(
        "source",
        type=argparse.FileType("r"),
        default="-",
        nargs="?",
        help="The input file. Defaults to stdin",
    )
    parser.add_argument(
        "destination",
        type=argparse.FileType("w"),
        default="-",
        nargs="?",
        help="The output file. Defaults to stdout.",
    )
    parser.add_argument(
        "-v", "--verbose",
        action="count",
        default=0,
        help="If specified, increases logging verbosity. "
             "Can be specified multiple times (ie. -vvv)",
    )
    parser.add_argument(
        "-l", "--log-file",
        type=pathlib.Path,
        help="If provided, it must be the path to a file. The "
             "given file will be used for logging. NOTE: This file"
             "will be overwritten."
    )
    args = parser.parse_args(argv)
    args.verbose = LOG_LEVELS[min(args.verbose, len(LOG_LEVELS)-1)]
    return args

def configure_logging(level, filename=None):
    """This function configures the Python logging system
    based on the level and filename provided.
    """
    handlers = [
        logging.StreamHandler(
            stream=sys.stderr,
        ),
    ]
    if filename is not None:
        handlers.append(
            logging.FileHandler(
                filename,
                mode="w",
            )
        )

    logging.basicConfig(
        level=level,
        handlers=handlers,
    )

def _main(args):
    """This function is responsible for executing the application logic.
    """
    log = logging.getLogger(__name__)
    for line in args.source:
        log.debug(f"Found line: {line}")
        bytes_written = args.destination.write(line)
        log.debug(f"Wrote {bytes_written} bytes to {args.destination}")
    log.info(f"Exiting successfully")
    return RETURN_CODES["SUCCESS"]


def main(argv):
    """This method is executed when this script is called
    from the command line.

    This function is responsible for the following:

    * parse command line arguments
    * Configure logging based on arguments
    * Call the function `_main` which should hold the application logic
    * return the return code as determined by function `_main`
    """
    log = logging.getLogger(__name__)
    try:
        args = parse_args(argv)
    except Exception as exception:
        log.critical(f"An unhandled exception has occurred: {exception}")
        return RETURN_CODES["UNHANDLED_EXCEPTION"]
    configure_logging(args.verbose, args.log_file)
    log = logging.getLogger(__name__)
    log.debug(f"Received args: {args}")
    try:
        return_value = _main(args)
    except Exception as exception:
        log.critical(f"An unhandled exception has occurred: {exception}")
        return_value = RETURN_CODES["UNHANDLED_EXCEPTION"]
    return return_value

if __name__ == "__main__":
    try:
        rc = main(sys.argv[1:])
    except:
        rc = RETURN_CODES["UNHANDLED_EXCEPTION"]
    finally:
        logging.shutdown()
    sys.exit(rc)

What the Script does

As a template, there was little point in writing complex application logic, but I still wanted to show a use-case for context, so I decided to give the script the task to take a source file and writes its contents to a destination file. A simple echo program.

The Execution

When executed, the interpreter will run from top-to-bottom defining the global variables and functions until it encounters the line if __name__ == "__main__":. That’s where our application logic begins.

Inside the if __name__ == "__main__": block, we execute the main function in a try…except block. Any exceptions that are caught cause a generic return code. Finally, we shutdown the logging and exit with our return code which was either returned by main or assigned the generic error code UNHANDLED_EXCEPTION.

The main function handles the following tasks:

• parse the command line arguments
• Configure logging based on user-provided arguments
• Call the function _main
• return the return code as returned by function _main

The _main function is where the bulk of our application logic will live. In our case, the _main function logs some information and copies the contents of source to destination.

That’s the basic execution flow, let’s look next at some specifics.

Input and Output Files

One of the things that makes command.py so flexible is its input and output options. By default, the input is stdin and the output is stdout. However, users can also specify a file path to read or write to files on disk.

Let’s look at where those arguments are defined:

...
parser.add_argument(
    "source",
    type=argparse.FileType("r"),
    default="-",
    nargs="?",
    help="The input file. Defaults to stdin",
)
parser.add_argument(
    "destination",
    type=argparse.FileType("w"),
    default="-",
    nargs="?",
    help="The output file. Defaults to stdout.",
)
...

As you can see in the code above, we’re creating two arguments:

• source: This argument represents the input file(s) for our program. We’re using argparse.FileType(“r”) to specify that this argument should accept a file path that opens for reading (“r”). The default=”-“ means that if no input file is provided, it will default to the standard input (stdin).
• destination: This argument represents the output file(s) for our program. We’re using argparse.FileType(“w”) to specify that this argument should accept a file path that opens for writing (“w”). The default=”-“ means that if no output file is provided, it will default to the standard output (stdout).

Logging Made Easy (for the user)

One of the key features that sets command.py apart is how flexible the logging is on the user’s end.

With a simple -v flag, you can control the verbosity of your logs. What’s clever about this is that you can specify -v multiple times to increase the logging level with each occurrence. For example, the default is CRITICAL and -v would change the log level to FATAL, -vvvv would output info messages, -vvvvv would output debug messages. This pattern makes it easy to fine-tune the logging at runtime to suit your needs.

Let’s look at how that works:

LOG_LEVELS = [
    'CRITICAL',
    'FATAL',
    'ERROR',
    'WARNING',
    'INFO',
    'DEBUG',
]

def parse_args(argv):
    parser = argparse.ArgumentParser()
    ...
    parser.add_argument(
        "-v", "--verbose",
        action="count",
        default=0,
        help="If specified, increases logging verbosity. "
             "Can be specified multiple times (ie. -vvv)",
    )
    ...
    args = parser.parse_args(argv)
    args.verbose = LOG_LEVELS[min(args.verbose, len(LOG_LEVELS)-1)]

The first part of the verbosity option is the action="count"" which causes the ArgumentParser to count the number of times the option was specified at the command line.

Next, let’s examine where the real magic happens. At the top of the script, in the global scope, we define a constant called LOG_LEVELS.

LOG_LEVELSis is used in the semi-confusing line args.verbose = LOG_LEVELS[min(args.verbose, len(LOG_LEVELS)-1)]. Here we are pulling out an item from LOG_LEVELS based on the count of -vs present in the arguments. The weird part is the call to min inside of the square brackets (ie. []), this is needed in case someone specifies too many -vs in which case we will select the most verbose option available (ie. DEBUG)

The next piece of logging configuration exposed to the user is the ability to specify a file to write logs to using the --log-file option. Logs default to going to stderr, but if a user provides a log file, logs will be written to the log file in addition to stderr. This flexibility makes it easy for your users to customize their logging to suit their own needs.

Error Codes, Simplified

When it comes to error handling, command.py uses a simple global dictionary called RETURN_CODES to keep track of all possible return codes. This makes it easy to add new error codes or modify existing ones without having to modify the underlying code.

In order to use the RETURN_CODES dict, simply return the relevant return code from the dict, like we do in _main in the line return RETURN_CODES["SUCCESS"].

Free and Open-Source

Finally, we want to emphasize that command.py is free to use and is released under the GPL v3. This means that you can modify and distribute the code as you see fit, without worrying about licensing restrictions.

We hope you find command.py to be a valuable tool in your development workflow. Whether you’re building a new CLI tool or just need a template to get started, this script is designed to make your life easier. So go ahead, give it a try, and see what you can create!

Python is a popular and versatile programming language that has become an essential tool for various industries, including web development, data science, and artificial intelligence. If you’re new to Python, installing the latest version can seem daunting, but don’t worry, we’ve got you covered. In this article, we’ll walk you through the process of installing Python on Windows, Linux, and MacOS.

Prerequisites

Before you begin, make sure you have:

• A computer with a compatible operating system (Windows, Linux, or MacOS)
• A reliable internet connection

Installing Python on Windows

1. Download the latest version of Python: Go to the official Python website’s Windows download page (www.python.org), Click on the link that says “Latest Python 3 Release - Python 3.x.x” where x.x is the latest minor and patch version of Python.
2. Run the installer: Once the download is complete, run the installer file (usually named “python-3.x.x.exe”). Follow the prompts to select the installation location and other options.
3. Choose the installation type: You’ll be presented with two installation options:
   • Add Python 3.x to PATH: This option adds Python to your system’s PATH environment variable, allowing you to run Python from anywhere.
   • Install for me: This option installs Python in a specific location, such as “C:\Python39” (for Python 3.9.5). Choose the option that suits your needs.

Installing Python on Linux

1. Update your package list: Open a terminal or command prompt and run the command sudo apt-get update (for Ubuntu-based distributions) or sudo yum update (for Red Hat-based distributions).
2. Install Python: Run the command sudo apt-get install python3 (for Ubuntu-based distributions) or sudo yum install python3 (for Red Hat-based distributions).
3. Verify the installation: Run python3 --version to verify that Python is installed correctly.

Installing Python on MacOS

1. Install Python using Homebrew: Open the Terminal app and run the command brew install python.
2. Verify the installation: Run python3 --version to verify that Python is installed correctly.

Verifying the Installation

After installing Python, you can verify that it’s working correctly by running the following commands:

• Windows: python --version
• Linux: python3 --version
• MacOS: python3 --version

Conclusion

Congratulations! You’ve successfully installed the latest version of Python on your computer. Start exploring the world of Python programming and enjoy the journey!

I will keep it short for this first post so I can start in on my plans for this blog.

Stay tuned to this blog or follow our RSS feed because we have some big plans coming soon.