Taking the TU Delft Carpentries Workshops Online

Due to the COVID - 19 pandemic, Delft University of Technology in the Netherlands (TU Delft) stopped its activities on campus until autumn 2021 and moved all teaching activities to an online setting. This article describes the challenges and lessons learned from successfully moving basic programming workshops, Software Carpentry workshops, online. The article details the local TU Delft context, the online workshop tools that were employed, and the roles that the organising team played to organise and run these online workshops. To successfully adapt to the online context, it was important to adjust the original planning and programme for the Carpentry workshops. General challenges of online workshops and solutions that worked for the TU Delft team are also shared. Through iteratively developing the online workshops over the past year, the team has enhanced both learners ’ and organisers ’ experience. The lessons learned will continue to be valuable when the workshops are transitioned back to a physical setting when COVID - 19 protective measures are lifted.


Introduction
Approximately half of the scientists and research software engineers are selfteaching programming skills (Hettrick et al. 2014;Lee 2018).Improving computing skills saves researchers time and resources as it prevents them from spending hours on doing things manually (Wilson 2016).Moreover, automated workflows can lead to more reproducible results and better documentation leads to code that is easier to reuse (Lee 2018;Wilson et al. 2017).Researchers from the Delft University of Technology (TU Delft, the Netherlands) have the opportunity to follow workshops that teach researchers basic programming skills.These workshops use the Carpentries materials and have been part of the TU Delft training programme since 2018.In this article we introduce the Carpentries pedagogy and how the Carpentry workshops are implemented in the programming training at the TU Delft.As these workshops could no longer take place in person in 2020 due to the COVID-19 pandemic the TU Delft team faced some challenges in transitioning the Carpentries workshops online.We provide our lessons learned and list some of the solutions that have improved the efficacy of the TU Delft Carpentries workshops.

Carpentries: the pedagogical approach
The Carpentries (https://carpentries.org), a non-profit organisation, provide accessible research computing training for scientists.Carpentries workshops have proven to be an effective means of providing programming training for researchers, with its current format of two intensive days of programming widely used since 2012 (Wilson 2016).Because of the intensive format, the workshops primarily aimed to teach the most fundamental and required skills that demonstrate high-level concepts, without the learner actively noticing this (Wilson 2016).The program increases participants' computational understanding through live coding and leads them to adopt tools and techniques that can aid them in their research (Wilson 2016).The Carpentries workshops have a Code of Conduct 1 that ensures the participants can learn in a safe environment.The teaching materials used in the workshops are openly shared under the Creative Commons Attribution License (CC-BY), so the lessons can be adjusted if required.Anyone can run a Carpentries workshop with the Carpentries' teaching materials, provided that workshop information is shared with the Carpentries, at least one Carpentries-certified Instructor is present during the sessions and the workshop should cover specific core topics of the workshop curriculum.
The Carpentries workshops, differing from academic teaching, are tailored for gaining hands-on computational skills through peer learning.At least two Instructors teach at the workshops.For the hands-on approach to work smoothly, a Helper for every eight learners is present (Wilson 2016).These Helpers do any troubleshooting necessary.The workshops also use a collaborative note-taking document, in which computational commands used can be captured and questions 1 https://docs.carpentries.org/topic_folders/policies/code-of-conduct.htmlTaking the TU Delft Carpentries workshops online JeSLIB 2022; 11(1): e1221 https://doi.org/10.7191/jeslib.2022.1221can be asked.Another collaborative aspect of the workshop are the exercises that can be conducted in pairs.The participants are encouraged to ask each other questions and help fellow participants.
The Carpentries workshops stimulate feedback and input by the learners.Each learner is unique, and by frequently and actively soliciting and incorporating learners' feedback, Instructors, Helpers and organisers can best tailor the workshop to suit learners' needs.Before the workshop the participants have to fill in a pre-workshop survey, which asks about their previous programming experiences and their expectations for the workshop.Real-time feedback during the workshop is provided by the use of two sticky notes of different colours, allowing the Instructors to do quick true/false questions and checking if the participants are following the programme as they are teaching.Before the breaks (or at the end of the workshop day), learners can also use these stickies to write down one thing they learned or liked from the workshop and one thing they found difficult or didn't like.This feedback allows the Instructors to adjust the rest of the programme accordingly.At the end of the day, there is another round of feedback where the learners are asked to write down one positive and one negative point about the day.After the workshop, a post-workshop assessment questionnaire is filled in to evaluate the progress of the learners and the effect of the workshop on their progress.This feedback can be used to improve future workshops.
The pandemic meant that these types of basic programming workshops had to be delivered online.While the content of the workshops seemed relatively easy to transfer to an online setting, the Carpentries workshops focus a lot on interactions between Instructors, Helpers and learners.The community reacted quickly and provided guidance and recommendations.We have built our online workshops on the recommendations of, in particular, the Carpentries, MetaDocencia, the Software Sustainability Institute and the Netherlands eScience Center (Chen 2020;The Carpentries 2021b;2021a;Lescak et al. 2020;MetaDocencia 2020;Nenadic and Antonioletti 2021;Sufi et al. 2020;Tippin, Kalbach, and Chin 2018;Vanichkina 2020).

Carpentry workshops at TU Delft
The Delft University of Technology (TU Delft) is the largest technical university in the Netherlands, with ~5000 employees including PhD candidates, ~23,500 students and eight faculties.TU Delft's first Software Carpentry workshop took place at the end of 2018.In 2019, TU Delft became members of The Carpentries (sponsored by TU Delft Library and 4TU.ResearchData repository (Martinez Lavanchy 2019)).Since then, the team runs three to four Software Carpentry workshops and multiple Data Carpentry workshops per year (Figure 1).The TU Delft Carpentries workshops are free to attend for any TU Delft student or staff.Researchers from other institutions are allowed to take part (at no cost) when there are unfilled places.PhD candidates attending the workshops will receive Graduate School credits, which they need to complete their doctoral education programme.For the physical workshops that took place between 2018-2019, the workshop programme took place in two working days and provided space for 30 learners.The workshop content was taught by two Instructors, supported by 4-6 Helpers per day.The Instructor was live programming or presenting using a single screen projected by a beamer in the front of the room.The Coordinator, in charge of organising and coordinating the workshop, introduced the day, made sure the instructors, helpers and learners had everything they need, ensured that food and drinks arrive in time for the breaks, and took care of any physical materials (sticky notes, extra laptops, etc.) that were needed.During in person workshops notetaking was done using an Etherpad 2 .
The last physical workshop at TU Delft was the Data Carpentry workshop for Social Sciences in the beginning of 2020.After the cancellation of a Software Carpentry workshop right before the Netherlands went into a lockdown in March 2020, the workshops were moved to an online setting in July 2020 (Figure 1).Whether taking place physically or online, the workshops are organised by a large team from 4TU.ResearchData (the data repository of the technical universities of the Netherlands), TU Delft support staff (Data Stewards, ICT support and the Digital 2 https://etherpad.orgCompetence Centre), as well as volunteers (Data Champions, PhD candidates, and researchers interested in providing workshops) (see Appendix A).

Teaching Platform
Zoom was the online conferencing tool that was available to TU Delft staff that met our requirements for real time feedback and possibilities to use breakout rooms for troubleshooting and exercises.
• Breakout rooms allow the host to split up the meeting into separate sessions that can be joined by the participants (either automatically, manually or allowing the participants to select and enter breakout rooms themselves).A timer can be set so that a countdown is displayed in the breakout rooms to ensure that everyone knows how much time is left.
• Real time feedback can be obtained through the reactions list, with options such as 'Yes', 'No', 'Go Slower', 'Go Faster' that are helpful to determine the pace of the workshop.The Yes/No reactions can be used in place of the red and green stickies that are normally used to assess the progress of learners in physical Carpentry workshops.
• There is a waiting room that can be enabled, so that Instructors, Helpers and the Coordinator can use the same room to prepare the workshop before the participants enter.
• The chat is the easiest way for learners to get in touch for help and ask their questions, as learners already have the Zoom window open to follow the instructor.By using the same tool, the learner's attention does not have to be divided over multiple tools or windows simultaneously.When a question is relevant to all participants, the Instructor is interrupted by the Coordinator so that the question can be asked by either the participant or the Coordinator.If the question is minor, or very specific to the individual learner, the question is answered by the Helpers in the chat.Using @name is very helpful in this case, as the chat cannot be ordered other than chronologically.

Backchannel communication
The team uses Slack for any backchannel communication.A separate channel is set up for all the Helpers/Instructors of each workshop.The Instructor is not on Slack during the lessons that they teach, as they (and the participants) would otherwise get distracted by the notifications and messages.If the Instructor's input is needed for anything, the Coordinator should interrupt the workshop and ask for the Instructor's input directly.The Slack channel does provide a good opportunity for the Helpers to brainstorm about issues that they do not have a

Collaborative document
At TU Delft we are no longer using the Etherpad that we used to communicate information and document the code in physical workshops.Instead, a Google Document is used as it allows for the sharing of pictures.This can be very helpful when participants have to share a screenshot of their issue, or if the code documenter wants to take screenshots of the lessons rather than typing everything out (this works particularly well for Jupyter Notebooks).The collaborative document and the materials used by the Instructors (presentations and notebooks) are hosted in a workshop specific Google Drive folder.
Participants valued the code documenting during the workshop.Participants were also positive about the additional information, materials and reference materials linked in the document.

Recordings
We have discussed providing the learners with pre-recorded instructional videos but have so far not done this.Live coding and interaction during the workshop are the most important aspects of the workshop, and we have referred the learners to additional learning materials where applicable (see for example the recordings of the Software Sustainability Institute workshops3 ).Furthermore, not everyone will be comfortable with workshops being recorded and it might deter them from asking for help when they need it.

Registration
Registration for the TU Delft workshops took place via EventBrite until the start of 2021.We started with opening registrations to 20 learners for the first online workshop and gradually expanded to the original number of 30 learners.Currently the university's graduate school system (CoachView) is used.This platform allows PhD candidates to enrol directly, it keeps track of a waiting list, and allows for easy participation registration which is needed for the PhD candidates to obtain the credits that are offered through the workshops.While online workshops are easier to attend by externals we have generally not encouraged this due to our long internal waiting list (with a steady 90 individuals) for future workshops.

Online workshop roles
During the workshop we have the following roles assigned to the organising team: Taking the TU Delft Carpentries workshops online JeSLIB 2022; 11(1): e1221 https://doi.org/10.7191/jeslib.2022.1221 • Instructor: Focuses on teaching the content of the lesson.
• Helper (Troubleshooter): Keeps track of the Zoom chat and collaborative document for questions, is responsible for any trouble shooting in breakout rooms and takes participants through exercises in breakout rooms.
• Helper (Code documenter): Responsible for keeping track of the command log in the collaborative notes document.
• Coordinator: Ensures that the Code of Conduct is followed, keeps track of the timing of the programme, regulates the breaks, takes care of technical difficulties, mutes people when needed, creates and assigns breakout rooms, and ensures that all the questions in the chat or the collaborative document are answered by the Helpers.
Each of these roles is briefly explained in the introduction of the first session by the Coordinator and are visible to the participants through the name tags in Zoom (for example, Helper -name).Both Instructors and Helpers (Troubleshooters) are assigned as co-hosts to ensure that they have all the assigned privileges needed to perform their tasks.

Instructor
The role of the Instructor did not change much in online workshops.Nevertheless, the lessons did require more preparation from the Instructor's side.To avoid rushing through the content, it is important to carefully consider the core topics of a session and ensure that there is sufficient time for the teaching and selected exercises.Several of the TU Delft Instructors have set up detailed notes of the Carpentries material to structure their sessions.This structure had a positive impact on the clarity of the content delivery and made it easier to keep to time.
The Instructor notes by Manuel Garcia Alvarez and Niket Agrawal are publicly available (Agrawal 2021; Garcia Alvarez 2021).The minimum amount of time spent on a workshop as an Instructor includes one session of teaching (~2-4 hours, depending on whether the session is taught by one or multiple Instructors) and any preparation required, which varies with experience (4 to 8 hours).Each workshop had at least two Instructors, with some workshops having up to four Instructors.
Next to having to prepare the lesson in more detail, another major difference for the Instructors in an online workshop is the lack of live feedback from the learners.There is currently almost no direct interaction between the Instructor and the learners, with the exception of questions that are asked directly to the Instructor.In an online workshop, it is difficult for the Instructor to teach, check all the time whether the learners are following along and keep track of the chat.Therefore, we encourage Instructors to solely focus on the materials and live coding, and only pay attention to any vocal interruptions during the lessons.
Instructors that tried to keep up with teaching, keeping track and the chat were not able to cover all the materials as this multitasking slows them down, or they indicated that they felt overwhelmed by all the tasks.

Helpers
The role of the Helpers at TU Delft Carpentry workshops has been described in more practical detail in an information sheet (Plomp et al. 2021).The Helpers' role is even more crucial in the online setting: having Helpers allows for more interaction in breakout rooms and the trouble shooting allows the workshop to run according to schedule.The TU Delft workshops generally have one Helper per 5-7 participants (4-7 Helpers per session).The Carpentries recommend a high Helper-to-learner ratio (1:5 or better4 ), which makes these workshops very intensive to host.Having fewer Helpers requires a very well organised workshop, otherwise the workshop will be slowed down through more difficult parts of the lessons.Limiting the use of breakout rooms for exercises also reduces the need for Helpers.The time commitment for Helpers varied greatly: some Helpers were present for a couple of hours and others attended all workshop sessions (typically between 4-18 hours, excluding any preparation).

Coordinator
For the TU Delft workshops, the Coordinator is responsible for the organisation of the workshop, next to facilitation during the workshop.Please see the TU Delft Coordinator information sheet for more detailed information on this role (Plomp, Tsang, and Lavanchy 2021).Based on our previous experiences, it takes ~12 hours of the Coordinators time to organise a single workshop, next to the ~20 hours of workshop attendance.Some administration and management support are provided by a student assistant (Lauren Besselaar, Femke van Giessen), who sends out the pre-workshop email and post-workshop survey reminder for each workshop.For some of the workshops the student assistant keeps track of the registrations and waiting list.The Coordinator's role has been expanded in online workshops.In physical workshops, the Instructors and Helpers do not need a lot of guidance.During an online workshop, however, it is important that someone keeps track of the time and is in charge of the technicalities (breakout rooms, waiting room, keeping track of the learners).Especially in online meetings there should be one chair that facilitates who is speaking and when.In short, the Coordinator allows the Helpers and Instructors to focus on their roles.To ensure that learners can effectively follow the programme, our online workshops make more extensive use of a collaborative note document which the Coordinator prepares (Plomp, Tsang, and Lavanchy 2021).For the TU Delft workshops we had one Coordinator per workshop, but the role could be divided between several people if the responsibilities are clearly outlined.

Planning and Programme
Careful planning of an online workshop is very important to ensure effective learning and teaching: Instructors, Helpers and learners new to online learning will need more guidance; coordinating remote teams requires more attention and frequent communication via email (before the start of the workshop to ensure that everyone is prepared and in the background on Slack when issues are occurring that cannot be resolved in the Zoom chat).In person, last-minute communications and any issues that arise during the workshop are easier and faster addressed compared to the online setting.To compensate for longer communication times in the online setting, we deliberately leave out parts of the Carpentry workshops materials to not overwhelm the learners or rush through the programme.This is done for physical workshops as well, as the Carpentries teaching materials are generally too much to be able to cover during one workshop, however, this has been increasingly important in the online setting (see also Chiewphasa and Moeller 2021 for similar experiences with time constraints).In both physical and online workshops the learners are directed to the Carpentries materials to learn more.
TU Delft Carpentry workshops are hosted in four separate sessions that last 4-4,5 hours, with two 15-minute breaks each day (instead of two full days, which was the case for our physical workshops).The breaks are scheduled in advance and announced to the participants.We noticed that the learners became more tired as the workshop progressed, so whenever possible, more complicated topics were covered earlier in the workshop.The concentration of the organising team also declines with time: The Instructor, who has to teach to a screen without live feedback from participants, can experience more exhaustion from teaching online than in physical workshops.The scheduled breaks and shorter sessions over more days help avoid over-exhausting learners and Instructors.
It is important to set up a schedule to avoid spending too little or too much time on each of the topics that are taught during the session.Planning the exercises that will be used before the workshop starts is important for time management, but also for the Helper preparation needed and technical organisation (setting up breakout rooms) during the workshop.Some exercises are too short to run in breakout rooms and should be run in plenum.Exercises in plenum can be prepared by the Instructor and learners can be given some time to finish the exercises and write responses in the chat, use the reaction options, or unmute to provide their answers.Exercises that are done in breakout rooms are 15 minutes or longer and may need more guidance from Helpers.
We have shared a more detailed example of the programme (see Appendix A in Plomp, Tsang, and Lavanchy 2021) and focus on the main lessons that we learned below.The Instructors, Helpers and Coordinator sign in on Zoom before the start of the workshop (10-15 minutes) to ensure everyone's set up is working and to go over any introductions if needed.During the workshop itself, the Coordinator can briefly introduce the Instructors/Helpers or they can introduce themselves.The role of the Helper is explained by the Coordinator and the participants are notified Taking the TU Delft Carpentries workshops online JeSLIB 2022; 11(1): e1221 https://doi.org/10.7191/jeslib.2022.1221that anyone with the Helper-tag in front of their display names can be contacted for any support.Our workshops have 20-30 learners, which does not allow for any meaningful individual introductions from the learners in plenum.Learner introductions can instead be done in breakout rooms, so that every learner gets to know at least a couple of other attendees and is comfortable with speaking.An icebreaker is done to teach the participants how to work with the collaborative document (see Appendix B in Plomp, Tsang, and Lavanchy 2021), or how to use the 'Yes/No/Go Faster/Go Slower' interaction buttons.Sometimes the icebreaker information is used to set up breakout rooms with people that have common interests to facilitate interaction.
Breakout rooms provide a possibility for more interaction between the participants as they will be subdivided in smaller groups.At the same time, using breakout rooms for this purpose, as recommended by the Carpentries, is quite intensive.The breakout rooms need to be set up and have in our cases generally been facilitated by a Helper (Troubleshooter).Depending on the number of participants, this means that several Helpers are needed during the workshop (~5-7).We found that breakout rooms can be disruptive for the learning experience if the exercise is not explained beforehand, or if there are too many exercises per breakout session.Ideally, one (extended) exercise should be done during a breakout session which should last for at least 10 minutes (ideally between 15-20 minutes) to allow for enough time to go through the exercise and address any questions.Alternatively, for one of the workshops (Data Carpentry for Social Sciences), we stopped using breakout rooms and did the exercises in the main room after negative feedback about breakout rooms from the participants (for example: "Instead of breakout rooms we can get more information or do practical activities…" and "Using break-out rooms for activities is ok if there's plenty of time to spare, but it is really frustrating while there's limited time" from the feedback on session 1 and "Not having breakout rooms was great!" from session 2 of the same workshop.See Appendix B for more details).In some cases, the exercises were done right before the break and the participants were more in control of when they would do the exercise (either before the break or first take the break to then do the exercise).Afterwards the Instructor briefly went over the answers of the exercise and answered any questions.Learner feedback from this workshop was positive, indicating that breakout rooms are not necessarily needed to go over the exercises.On the downside, there was a lot less time for interaction/questions and the workshop felt longer as everything took place in the same setting (in the main room).

Challenges and Solutions
Below we list some challenges that are inherent to Carpentries online workshops, followed by some solutions the TU Delft team tested: Software installations are already a problem at physical workshops but can be detrimental during online workshops, particularly if found out during the workshop itself.In our pre-workshop email (see Appendix D in Plomp, Tsang, and Lavanchy Taking the TU Delft Carpentries workshops online JeSLIB 2022; 11(1): e1221 https://doi.org/10.7191/jeslib.2022.12212021), we have added more explicit installation instructions to the existing Carpentries email templates5 , as well as verification instructions for the participants to check the installations, and direct links to the datasets that they need to download6 .
Computer screens are limited in their space (participants have a Zoom window, as well as a coding window and a collaborative document).If Zoom is run on a separate machine, learners cannot easily share their coding windows with the Helpers and they have to either work around using screenshots or describe their issues which makes troubleshooting a long and complicated process.We now recommend learners to use one machine and to connect any additional screens if they have access to this.
Social interaction is limited.Social interactions between Instructors, learners and Helpers are important for establishing trust and creating a friendly environment where learners feel more comfortable to ask for help.Whereas previously this happened spontaneously in physical workshops, it is now hampered and difficult to facilitate.Most interaction takes place through the chat or in breakout rooms.At the same time, the chat should not get too busy as questions of the participants will be lost and the chat flow can be a distraction for those that need all their time to follow the steps of the instructor.To counter this, we set up icebreaker exercises in the beginning of the workshop session to help participants warm up.Ideally, information collected during icebreakers can be used during teaching as well as to help better connect with learners, individually or as a group.
Real time feedback is still provided through the reaction options, but it is not the same as being able to see the faces of the learners.We try to mitigate this by keeping some of the cameras of the Helpers and Coordinator on so that the Instructor has some continuous live feedback from the virtual room.To evaluate how learners are handling the materials they are regularly asked to check-in using the Yes/No reactions in Zoom.In general, if the chat starts to get busier it is a sign that the Instructor needs to slow down or repeat some steps so that the learners can get back on track.
Troubleshooting takes longer because there is no direct path to the screen of the learner.Often the problem is first discussed in the chat and if it cannot be solved there a breakout room is set up where the participant can share their screen.We have only been using breakout rooms as a last resort, as the participant will miss part of the main session if they are in a breakout room (otherwise the remaining majority of the participants have to wait for the issues to be resolved).We have tried to mitigate the loss of the experience in the main workshop with the note documenting and keeping track when exactly they went into the breakout room and came back into the main room (see Appendix C).In general, participants were able to catch up by themselves and did not need additional help afterwards.Whenever more than four participants are in separate breakout rooms to solve various issues, the lesson is slowed down by repeating some concepts or by encouraging the participants in the main room to ask questions.This ensures that there are also enough Helpers available in the main room in case there are further problems, and prevents a pile up of breakout rooms.
Breaks are an important part of the programme, allowing everyone to recharge.We have also used the breaks to address more difficult or more specific issues, or to allow learners to catch up after they have had an issue in a breakout room during the workshop.It is important to balance the need to address all the issues while at the same time ensure that learners take time away from their screens.
Internet access/bandwidth or technical difficulties can be an enormous issue.The code documenting in the collaborative notes serves as a backup solution for this, although it cannot substitute the real live coding lessons.Furthermore, the document may become long and difficult to load for those with low bandwidth.The added value of exchanging screenshots has so far outweighed any connectivity problems.Nevertheless, text-based documents may provide a solution in case connectivity issues are more stringent.While we have not yet recorded any of the workshop lessons, recordings could be used as a backup solution.We have not had major problems with internet connections and when learners disconnected they could usually catch up thanks to the notes of the note documenter.Alternatively, the part that learners missed was repeated for them by a Helper in a breakout room at the very end of the session.

Conclusion
The transition from physical to online Carpentries workshops was challenging, but at the same time a rewarding experience.The continuation of the workshops was much appreciated by the learners, as many other workshops were cancelled instead of continued online.The online workshops also provided an opportunity for the team to work together on shared goals to continuously improve the workshops.The online workshops also forced us to think more carefully about the materials that we use during our lessons, allowing for a more targeted programme that is easier to follow for learners.Our improved communications around the preparations for the workshop ensure a smooth start for the learners.The reduced cognitive load and start-up time for the learners will ultimately also increase the effectiveness of the physical workshops, whenever the COVID-19 protective measures are lifted.

Figure 1 :
Figure 1: Overview of the Carpentry workshops that TU Delft (co)organised from 2018 onwards.From 2020 onwards, all workshops were hosted online, with the exception of the last physical workshop (Social Science Data Carpentry).The Social Science Data Carpentries in 2021 were co-organised together with the Centre for Digital Scholarship of Leiden University Library (LEI).
/doi.org/10.7191/jeslib.2022.1221quickanswer for, ask for additional help, report technical issues or mention that they are taking a break or experiencing other interruptions.