This post is the first in a series of posts about Systems Administration at CS++.

The Problem

CS++ has modern computing and networking hardware in a dedicated server room in TU Dublin.

These servers run services for ourselves and multiple other societies in the university, as well as being involved in the Backup Intersocs Nationally initiative.

CS++ is run solely by student volunteers, with five students officially being declared 'Sysadmins' who maintain and grow the infrastructure.

Sysadmins may come from any one of SIX Computer Science courses in any THREE of TU Dublin's campuses, or may not even be a Computer Science Student. Of these six courses, there is no module that is universal between them all.

Furthermore, the Sysadmins can be students at any period in their education, from a First Year in an Undergraduate Course to a Masters' or PhD student.

Currently, the five sysadmins represent four different undergraduate courses, with two in fourth year, two in third year and one in second year.

CS++ needs to maintain critical services with a team that is fully replaced every four years, where that team has no guaranteed knowledge of the same technologies.

Sysadmin Challenges

I have been involved in CS++ since I started at TU Dublin, and architected the technical stack we are currently using. Now that I am in my final year, my goals have shifted from hands-on involvement into handing over the topics, skills and knowledge that I have learned from my experience in CS++.

The largest problem I aimed to tackle was having a common set of skills and understanding around the technologies we use as a society. To do this, I created a flight of tasks called the Sysadmin Challenges.

These 11 challenges build upon each other with the goal of covering all aspects of the hardware and software that we use. Starting from creating a Virtual Machine from a template, to networking, security and ending with building a complete cluster from scratch.

The Sysadmin role is advertised as 'experience preferred, but not needed', as the relevant skills are only taught in one of the six courses. I was the only Sysadmin with prior knowledge / experience in systems administration, architecture, networking and security. These skills needed to be passed on to the other Sysadmins to allow them to perform their duties, and to start new projects of their own.

These skills are transferable to personal projects and help improve their understanding of what they learn in class, and most importantly, are beneficial in their careers after graduation.

In the Open

Open Source software has been one of my core beliefs since I started my degree, and I wanted this training to be the same. Like all other CS++ projects, the Sysadmin Challenges is GNU GPL V3.0 licensed. While highly-specialised for our technical stack, other societies and sysadmin teams can theoretically use the approach to practical challenges as a template for their own environments.

Don't Just Tell Them

The Sysadmin Challenges were highly inspired by the CTF Challenges from Zerodays and the AWS GameDays. By providing high-level instructions and leaving out common pitfalls, the Sysadmins are forced to troubleshoot and research on their own to achieve a goal.

Task instructions follow a simple format:

1. A series of milestones to be achieved
2. Any relevant notes
3. Relevant services (To avoid misdirection)
4. Milestone Headers
5. Completion Criteria

This approach helps inform the Sysadmin on what they must use for that challenge, and provides relevant hints but avoids hand-holding to actively stimulate learning.

Doing this also allows the Sysadmins to attempt challenges whenever they wish, instead of in an environment where I am available.

Version Controlled

All things change, and the current technical stack should be no different. As software reaches EOL, and CS++'s procedures and infrastructure changes, the challenges should be amended to be accurate to the new infra.

By being a public repository on CS++'s GitHub organisation, this allows all Sysadmins to amend the contents, as well as anyone with a GitHub account to open issues and pull requests to suggest changes.

Sysadmin Docs

Practical experience without understanding is useless. It is best to know why a task needs to be done than just know how to do it.

Outside of building the Sysadmin Challenges, I have also been working on our docs. As described in our Docs Philosophy, these docs are written with a strong consideration for the 'Bus Factor' and the chance that there may be no handover at some point.

As a solution, the docs are written as a series of layers (an approach pioneered by Seán in the University of Galway CompSoc).

You can't automate tasks when you don't have tasks to do. You have no tasks without having services. You have no services without an Operating System. You have no Operating System without Hardware.

CS++'s Approach to Docs

The aim of this approach to documentation is to prepare for a possible full Sysadmin reset, and empower them to learn about our systems, what we did, why we did it and how it was done.

The docs and the challenges give Sysadmins the why and how, but this does not empower them to choose their own actions and to feel like they can contribute without being told what to do.

Empower by Owning

In February of 2025, I changed how Sysadmins were delegated tasks and implemented the concept of 'Ownership'. This meant that there was someone responsible for every service, topic or area.

Being an owner does not mean that they must do all of the work in their area. But instead they are the expert in that area and decisions on their area are usually deferred to them, empowering them to get involved and giving them agency.

As students, we automatically assume a hierarchical power dynamic, which student-led societies should not be. Solutions to this should unconsciously empower people and make them feel committed to what they are working on. Ownership does just that and makes it easy to redistribute workload as academic workload and personal commitments change over time. A flat hierarchy lets students be equal to eachother.

Ownership also naturally supports handover. Last year I was the owner for more than half of the ownership topics. Now I am just responsible for the password manager, VPN, systems architecture and security, and with sysadmins shadowing and docs for all of these. Once I graduate, these can be seamlessly handed over and the team can continue to grow.