The Luxembourg SuperComputing Competence Center will host a half-day online course introducing environment management and containers for HPC systems.
Users of a programming language like Python, or any other software tool like R, quite often must install packages and tools to extend the functionality of their base system. Linux/GNU distributions offer various packages; however, packages are often missing, or the available versions are old as administrators try to reduce the size of distributions and the associated maintenance.
To efficiently distribute application software distribution systems that specialize on various categories of applications have been developed. Systems such as Conda distribute generic types of software, whereas systems such as PyPI (Python), Packrat (R), and Pkg (Julia) specialize in a single kind of software. All software distribution system however offers a uniform interface that includes:
• the ability to create and reproduce software environments,
• isolation between environments, and between an environment and the system, and
• easy sourcing of packages from a variety of package sources.
On the other hand, containers allow the creation of a completely isolated system where the user can install any application they require through any available method. This additional flexibility adds a significant overhead, but correctly used containers can deploy software reliably across multiple systems.
Who should attend?
This workshop is designed for individuals who are often called to deploy software to HPC systems. This includes administrators, but also researchers who perform tasks such as performance comparisons between various applications implementing the same functionality and require a method to reliably and reproducibly install specific software versions.
What will you learn and how?
The objective of this course is to cover the basics of software and package management.
Learning outcomes
By the end of the course, participants will be able to:
• create Conda environments and use them to manage the software and package dependencies;
• create containers, install software in containers, and launch containers in HPC systems;
• natively use underlying system resource in containers, such as network (InfiniBand interconnect) drivers;
• document and reproduce software environments and containers;
• combine Conda and containers with application-specific package management tools;
• choose effectively between the available methods for deploying software.
Short presentations of the various components will be followed by a practical session where the participants can setup and try their environment.
Prerequisites
This course focuses on generic aspects of package management and containers. It is assumed that the participants have some basic knowledge of how to use packages in Python, R, or some other application. The techniques covered are applicable both in personal machines and on HPC clusters.
Computing Resources
Participants will have access to the MeluXina supercomputer CPU and GPU partitions during the training. For more information about MeluXina, please refer to the Meluxina overview and the MeluXina – Getting Started Guide.
Agenda
This half-day course will be hosted online in Central European Time (CET) on February 28th, 2025 (9:00 AM – 1:00 PM CET).
Schedule
9:00 AM – 9:20 AM: Lecture Part 1: Linux environment and the Conda environment management system
9:20 AM – 9:20 AM: Practical session 1
9:50 AM – 10:10 AM: Lecture Part 2: Containers in HPC systems
10:10 AM – 10:40 AM: Practical session 2
10:40 AM – 11:00 AM: Break
11:00 AM – 11:20 AM: Lecture Part 3: Reproducibility with Conda environments and Apptainer type containers
11:20 AM – 11:50 AM: Practical session 3
11:50 AM – 12:10 PM: Lecture Part 4: Interoperability with Module systems
12:10 PM – 12:40 PM: Practical session 4
12:40 PM – 1:00 PM: Q&A
Important: Limited spots available!
Contact person for more information:
Georgios KAFANAS - georgios.kafanas[at]uni.lu
