Intermediate Python for Astronomical Software Development

This course aims to teach a core set of established, intermediate-level software development skills and best practices for working as part of a team in a research environment using Python as an example programming language (see detailed learning objectives below). The core set of skills we teach is not a comprehensive set of all-encompassing skills, but a selective set of tried-and-tested collaborative development skills that forms a firm foundation for continuing on your learning journey.

A typical learner for this course may be someone who is working in a research environment, needing to write some code, has gained basic software development skills either by self-learning or attending, e.g., a novice Software Carpentry Python course. They have been applying those skills in their domain of work by writing code for some time, e.g. half a year or more. However, their software development-related projects are now becoming larger and are involving more researchers and other stakeholders (e.g. users), for example:

Software is becoming more complex and more collaborative development effort is needed to keep the software running
Software is going further than just the small group developing and/or using the code - there are more users and an increasing need to add new features
‘Technical debt’ is increasing with demands to add new functionality while ensuring previous development efforts remain functional and maintainable

They now need intermediate software engineering skills to help them design more robust software code that goes beyond a few thrown-together proof-of-concept scripts, taking into consideration the lifecycle of software, writing software for stakeholders, team ethic and applying a process to understanding, designing, building, releasing, and maintaining software.

Target Audience

This course is for you if:

You have been writing software for a while, which may be used by people other than yourself, but it is currently undocumented or unstructured
You want to learn:
- more intermediate software engineering techniques and tools
- how to collaborate with others to develop software
- how to prepare software for others to use
You are currently comfortable with:
- basic Python programming (though this may not be the main language you use) and applying it to your work on a regular basis
- basic version control using Git
- command line interface (shell)

This course is not for you if:

You have not yet started writing software (in which case have a look at the Software Carpentry course or some other Python course for novices first)
You have learned the basics of writing software but have not applied that knowledge yet (or are unsure how to apply it) to your work. In this case, we suggest you revisit the course after you have been programming for at least 6 months
You are well familiar with the learning objectives of the course and those of individual episodes
The software you write is fully documented and well architected

Prerequisites

To attend this course you should meet the following criteria. You can also test your prerequisite knowledge by taking this short quiz.

Git

You are familiar with the concept of version control

You have experience configuring Git for the first time and creating a local repository

You have experience using Git to create and clone a repository and add/commit changes to it and to push to/pull from a remote repository

Optionally, you have experience comparing various versions of tracked files or ignoring specific files

Python

You have a basic knowledge of programming in Python (using variables, lists, conditional statements, functions and importing external libraries)

You have previously written Python scripts or iPython/Jupyter notebooks to accomplish tasks in your domain of work

Shell

You have experience using a command line interface, such as Bash, to navigate a UNIX-style file system and run commands with arguments

Optionally, you have experience redirecting inputs and outputs from a command

Learning Objectives for the Workshop

Set up and use a suitable development environment together with popular source code management infrastructure to develop software collaboratively

Use a test framework to automate the verification of correct behaviour of code, and employ parameterisation and continuous integration to scale and further automate your testing

Design robust, extensible software through the application of suitable programming paradigms and design techniques

Understand the code review process and employ it to improve the quality of code

Prepare and release your software for reuse by others

Manage software improvement from feedback through agile techniques

Setup

Please make sure that you have all the necessary software and accounts setup ahead of the workshop as described in the Setup section. Also check the list of common issues, fixes & tips if you experience any problems running any of the tools you installed - your issue may be solved there.

Schedule

		Setup	Download and setup software required for the course
	00:00	Setting the Scene	What are we teaching in this course? What motivated the selection of topics covered in the course?
Section 1	00:00	Section 1: Setting Up Environment For Collaborative Code Development	What tools are needed to collaborate on code development effectively?
	00:10	Introduction to Our Software Project	What is the design architecture of our example software project? Why is splitting code into smaller functional units (modules) good when designing software?
	00:40	Virtual Environments For Software Development	What are virtual environments in software development and why you should use them? How can we manage Python virtual environments and external (third-party) libraries?
	01:10	Integrated Software Development Environments	What are Integrated Development Environments (IDEs)? What are the advantages of using IDEs for software development? How does Jupyter Lab interact with virtual environments?
	01:50	Best practices for Jupyter	How to avoid chaos when writing code in Notebooks? What is the workflow when using Jupyter Lab for software development?
	02:15	Collaborative Software Development Using Git and GitHub	What are Git branches and why are they useful for code development? What are some best practices when developing software collaboratively using Git?
	02:50	Python Code Style Conventions	Why should you follow software code style conventions? Who is setting code style conventions? What code style conventions exist for Python?
	03:25	Verifying Code Style Using Linters	What tools can help with maintaining a consistent code style? How can we automate code style checking?
	03:50	Section end
Section 2	00:00	Section 2: Ensuring Correctness of Software at Scale	What should we do to ensure our code is correct?
	00:05	Automatically Testing Software	Does the code we develop work the way it should do? Can we (and others) verify these assertions for themselves? To what extent are we confident of the accuracy of results that appear in publications?
	00:55	Scaling Up Unit Testing	How can we make it easier to write lots of tests? How can we know how much of our code is being tested?
	01:10	Github Issues and Collaborative Development	How can we keep track of identified issues and the list of tasks the team has to do? How can we communicate within a team on code-related issues and share responsibilities? How can we plan, prioritise and manage tasks for future development?
	01:55	Continuous Integration for Automated Testing	How can I automate the testing of my repository’s code in a way that scales well? What can I do to make testing across multiple platforms easier?
	02:40	Issue Diagnostics with a Debugger and Other Tools	Once we know our program has errors, how can we locate them in the code? How can we make our programs more resilient to failure?
	03:30	Section end
Section 3	00:00	Section 3: Software Development as a Process	How can we design and write ‘good’ software that meets its goals and requirements?
	00:05	Programming Paradigms	How does the structure of a problem affect the structure of our code? How can we use common software paradigms to improve the quality of our software?
	00:15	Functional Programming	What is functional programming? Which situations/problems is functional programming well suited for?
	01:15	Object Oriented Programming	How can we use code to describe the structure of data? How should the relationships between structures be described?
	02:25	Software Requirements	Where do we start when beginning a new software project? How can we capture and organise what is required for software to function as intended?
	03:00	Software Architecture and Design	What should we consider when designing software? How to determine which components our software will have?
	03:35	Architecture Revisited: Adding a .py Controller	How can we extend our software within the constraints of the MVC architecture?
	04:40	Section end
Section 4	00:00	Section 4: Collaborative Software Development for Reuse	What practices help us develop software collaboratively that will make it easier for us and others to further develop and reuse it?
	00:05	Developing Software In a Team: Code Review	How do we develop software in a team? What is code review and how it can improve the quality of code?
	00:45	Preparing Software for Reuse and Release	What can we do to make our programs reusable by others? How should we document and license our code?
	01:40	Packaging Code for Release and Distribution	How do we prepare our code for sharing as a Python package? How do we release our project for other people to install and reuse?
	02:00	Section end
	00:00	Wrap-up	Looking back at what was covered and how different pieces fit together Where are some advanced topics and further reading available?
	00:15	Finish

The actual schedule may vary slightly depending on the topics and exercises chosen by the instructor.