Education | 2i2c

2i2c communities at AGU 2024

Mon, 09 Dec 2024 00:00:00 +0000

We are proud to share that several of 2i2c’s community partners are presenting their work at AGU 2024! In each case, 2i2c’s infrastructure plays a part in helping communities create and share knowledge, and grow their community. As an organization rooted in community-centric practices, we are particularly excited to see 2i2c represented “indirectly” at this conference, and to see ourselves as a supporting role enabling the impact of others.

Here’s a summary and links to all of the sessions. See below for a brief overview of seach one.

ED31G-2272 Breaking down the barriers to Open Science with Project Pythia #

Link to session

Hall B-C (Poster Hall) (Convention Center)

Abstract #

Project Pythia is an open access educational initiative established with funding from the U.S. National Science Foundation. Its mission is to help students and scientists enhance their skills and adopt best practices using the tools and technologies of open science. As part of the Pangeo community, Project Pythia primarily focuses on the Pangeo stack, which includes cloud computing, Jupyter technologies, GitHub, and various software packages in the Scientific Python ecosystem, centered around Xarray. Project Pythia offers a wide range of open access content, such as datasets, software, tutorials, and annotated real-world workflows presented in the form of Jupyter Books.

Project Pythia serves as a resource for scientists, promoting and fostering open science. Although it is not a scientific research artifact itself, the development of Project Pythia adheres to many best practices advocated by open science proponents. The Pythia team actively encourages community engagement and collaborates openly with scientists and technologists to create new content. All Pythia resources are freely accessible, and the project follows the FAIR principles (Findable, Accessible, Interoperable, and Reusable) for managing research outputs, including publications, data, and other materials. We support and facilitate open evaluation and peer reviews of content to ensure verifiability and trust. Lastly, we endeavor to openly discuss ideas, designs, and methods before implementation.

This presentation will provide an overview of Project Pythia’s extensive educational resources and share our experiences in applying many open science principles to develop this flagship training resource for the geoscience community.

Authors #

John Clyne - NSF National Center for Atmospheric Research (first author)
Drew Camron - University Corporation for Atmospheric Research
Orhan Eroglu - NSF National Center for Atmospheric Research
Robert Ford - University at Albany State University of New York
Julia Kent - NSF National Center for Atmospheric Research
Ryan May - University Corporation for Atmospheric Research
James Munroe - 2i2c / Code for Science and Society
Brian E J Rose - SUNY at Albany

ED31G-2277 PACE Hackweek: An open community keeping up with PACE #

Link to session

Abstract #

The NASA Plankton, Aerosol, Cloud, ocean Ecosystem (PACE) mission, while bringing NASA’s Earth System Observatory up to speed with aquatic, atmospheric, and terrestrial science capabilities, is also providing data records of the Earth System for the next generation of scientists to grow into. The goal of the PACE Hackweek, supported by the Ocean Carbon & Biogeochemistry program and hosted at the University of Maryland Baltimore County in August 2024, was to enrich and support the practice of open science by both emerging and established researchers. Cloud-compute resources for the event were provided by CryoCloud, a NASA funded collaboration between ICESat-2 and the International Interactive Computing Collaboration (2i2c) to provide cryosphere researchers with a shared JupyterHub. We, the hackweek mentors, were buoyed by the NASA Openscapes program and adopted its mantra of striving toward “a kinder science for future us.” Participants faced two novelties: the “firehose” of data from the PACE instrument array (a hyper-spectral imaging spectrometer, a wide-swath hyper-angular polarimeter, and a narrow-swath spectro-polarimeter), and the distribution of PACE collections through the NASA Earthdata Cloud (a first for the Ocean Biology Distributed Active Archive Center). We present our approach and the challenges undertaken to hold an in-person, social coding event with 45 participants that provided a collaborative, supportive launchpad for doing open science with PACE. All lectures and tutorials produced for the event are freely available for examination and reuse. Our results additionally include highlights from the demonstration projects pursued by event participants and results from two post-event, qualitative surveys. One anonymous survey gathered participant feedback that will inform plans for growing these 45 participants into a lasting, open community. A separate, anonymous survey recorded participant demographics in order to evaluate our efforts at increasing diversity within the community of PACE data users. Key points of discussion include participant views, informed by our event, on whether and how the NASA Earthdata Cloud is a significant resource for the practice of open science with PACE, and how a shared JupyterHub can further the practice of open science by the community it serves.

Authors #

Ian Carroll - NASA Goddard Space Flight Center; University of Maryland Baltimore County (first author)
Kelsey Bisson - NASA Headquarters
Sean Foley - NASA Goddard Space Flight Center; Morgan State University
Patrick Clifton Gray - University of Maine
Elizabeth E Holmes - NOAA Fisheries
Carina Poulin - Science Systems and Applications, Inc.; NASA Goddard Space Flight Center
Tasha Snow - NASA Goddard Space Flight Center; University of Maryland College Park
Guoqing Wang - NASA Goddard Space Flight Center; Science Systems and Applications, Inc.
Jeremy Werdell - NASA Goddard Space Flight Center
Anna Windle - NASA Goddard Space Flight Center; Science Systems and Applications, Inc.
Pengwang Zhai - Department of Physics, University of Maryland Baltimore County

IN13A-2147 Including more solutions and more solvers via actionable open science #

Link to session

Monday, 9 December 2024 13:40 - 17:30
Hall B-C (Poster Hall) (Convention Center)

Abstract #

If we’re asking people to change for open science, we must be willing to change ourselves. Internalizing this as individuals and institutions is critical - “to address our climate emergency, we must rapidly, radically reshape society. We need every solution and every solver” (Johnson & Wilkinson, All We Can Save).

Radically reshaping our society and including more solvers requires Earth scientists of all disciplines, across AGU, to work together in new ways. Many of these shifts can be considered Open science. They change how we work daily, not just the open products we produce. And for that, people need to consider themselves part of a team, let go of perfection and embrace a growth mindset to continually reflect and improve skills – no matter their job title. Further, open science requires all of us to see ourselves as leaders making small changes that collectively add up to a movement.

Openscapes is an open source approach to cultivating leaders and change makers. Openscapes’ flywheel approach intervenes and builds momentum through identifying mentors within organizations and mentoring teams curious about shifting to open science (Robinson & Lowndes 2022). Collectively, the Openscapes flywheel iterations have had a significant impact over the past five years across institutions like the federal government and academia that seem impossible to change. Through stories working with professional scientists over the past 5 years including at NASA, NOAA, Black Women in Ecology, Evolution, and Marine Science, and many universities, and open source software communities like 2i2c, Posit, Pangeo, and RLadies, we will share actionable insights for flourishing in the open science commons, and are interested in learning with and growing flywheel momentum further at AGU.

Authors #

Julia S. Stewart Lowndes - Openscapes, LLC (first author)
Ileana Faye Fenwick - Hampton University
Elizabeth E Holmes - NOAA Fisheries
Luis Alberto Lopez - National Snow and Ice Data Center (NSIDC), CIRES, University of Colorado Boulder
Erin Robinson - Self Employed

IN34A-01 Beyond Open Data: Ensuring True Accessibility for All (Invited) #

Link to session

Abstract #

The Earth Observation (EO) industry has seen rapid technological advancements alongside a massive increase in the number of private and public missions, leading to exponentially growing data archives. For publicly funded entities, this data is typically required to be freely available. However, open data does not always guarantee accessibility, and significant barriers remain for even the most advanced users. The stagnation in the use and adoption of open data can be attributed to several factors, including 1) challenges in unifying and maintaining metadata standards, 2) inefficiencies associated with legacy data formats, 3) a lack of training and resources for transitioning to cloud-based infrastructure, and 4) systemic social inequalities.

This talk will explore real-life examples of these barriers and highlight success stories that have emerged from partnerships largely originating within open-source communities which foster diverse connections between private and public entities including efforts like GeoZarr, pangeo-forge, Openscapes and 2i2c. While many advancements in improving the usability and accessibility of EO data have come from private efforts (i.e. Google Earth Engine), the shutdown of the Planetary Computer is a reminder of the need for publicly funded alternatives. The sustainability of open source projects will be addressed, with questions posed around reliable funding mechanisms as a means to ensure equitable development to address barriers and ensure accessibility for all. While this talk will be presented by one individual, it is the review and reflection of the work done by dozens of people across various organizations.

Authors #

Brianna Rita Pagán - NASA Goddard Space Flight Center; ADNET Systems Inc. Greenbelt (first author)

Introducing GeoLab - An EarthScope JupyterHub for Enabling Collaborative Cloud-Native Geophysical Data Analysis and Skill Development Workshops #

Link to session

Abstract #

The EarthScope Consortium manages NSF’s GAGE and SAGE facilities and makes all of its geophysical data available in a commercial cloud system. This enables EarthScope and the communities it supports to leverage the abundant computational resources and cost-effective benefits of adopting data-proximate workflows with direct access to large, analysis-ready geophysical data sets.

In recent years, JupyterHub environments have gained popularity with data enthusiasts for their ability to provide open access to powerful compute resources. As part of a broad effort to support communities with intuitive resources to quickly adapt their workflows to the cloud, EarthScope has partnered with 2i2c to operate a scalable JupyterHub environment in AWS that will provide equitable access to cloud compute resources for researchers, educators, and the general public. GeoLab, the EarthScope hub, is aligned with related open science initiatives to establish rigorous and transparent standards for reproducible, data-intensive workflows. In addition to promoting interdisciplinary and inter-institutional collaborative work between researchers in GeoLab, EarthScope is developing and hosting workshops that can support both in-person and asynchronous learning modules that will train users how to utilize these new resources and transition their work to the cloud. We are excited to invite all geophysical data users to participate in the vigorous growth of this new platform and collaborate with adjacent open-science compute hub initiatives.

Authors #

Robert T Weekly - EarthScope Consortium (first author)
Tammy K Bravo - EarthScope Consortium
Jerry A Carter - EarthScope Consortium
Tim Dittmann - EarthScope Consortium
Alex Hamilton - EarthScope Consortium
David Mencin - EarthScope Consortium
Chad Trabant - EarthScope Consortium
Sarah Wilson - EarthScope Consortium

Link to session

Abstract #

Project Pythia is the flagship education and training initiative of the Pangeo community. Pangeo has advanced transformative platforms and paradigms for “Big Data” geoscience in the cloud; Pythia is creating on-ramps for new users with open, interactive learning resources centered on Python in the geosciences. Pythia is now building a vibrant community-owned clearinghouse of accessible, reusable, and reproducible tutorials and exemplar workflows in the cloud known as Pythia Cookbooks.

“Cookbooks” imply collections of recipes for transforming raw ingredients (publicly available data) into scientifically useful results. Based on Jupyter notebooks, Cookbooks are explicitly tied to reproducible computational environments and supported by a rich cloud-based infrastructure enabling collaborative authoring and automated health-checking – essential tools in the struggle against the widespread notebook obsolescence problem. Cookbooks are hosted on Pythia’s searchable gallery and nurtured by a growing community of open science enthusiasts from across the geosciences. The Pythia Cookbook gallery is essentially a crowd-sourced, community-curated collection of best practices for data analysis and visualization.

Here we will outline the stack of technologies and infrastructure enabling cookbook creation, collaboration, testing, publication, and interactive deployment, and how these are used in service of building an inclusive participatory community. We will discuss existing technical and social hurdles for contributors, as well as new infrastructure developments in collaboration with the Executable Books Project that are reducing these hurdles.

Authors #

Brian E J Rose - University at Albany State University of New York (first author)
Drew Camron - University Corporation for Atmospheric Research
John Clyne - NCAR
Orhan Eroglu - NSF National Center for Atmospheric Research
Robert Ford - University at Albany State University of New York
Julia Kent - NSF National Center for Atmospheric Research
Ryan May - University Corporation for Atmospheric Research
James Munroe - 2i2c / Code for Science and Society
Kevin Tyle - SUNY at Albany

U13A-2350 Supporting NASA Earthdata users in the Cloud: NASA Openscapes JupyterHub and User Onboarding & Fledging #

Link to session

Abstract #

In this talk we will highlight our NASA Openscapes community teaching approach to using the 2i2c-managed JupyterHub – how we’ve collaboratively developed it to meet user needs, and how it continues to enable researchers and users of NASA Earthdata in the new Cloud paradigm.

NASA Openscapes is an open source mentor community across NASA Earth science data centers ( DAACs) that helps users explore and use the Cloud for their science and applications. Earthdata Cloud Cookbook is a learner-focused open source tutorial collection that we update openly as we learn together.

A critical piece of the NASA Openscapes effort is our NASA Openscapes 2i2c JupyterHub, a managed cloud computing space. By working with cloud early adopters and science Champions, responding and co-developing solutions, the JupyterHub has evolved since its early days in 2021. We support cloud computing for several languages (python, R, Matlab, QGIS) and common science libraries with corn; we streamlined how to bulk-add workshop participants via GitHub Teams; we established policy and technology for a special authentication mechanism for large scale workshops; we are developing earthaccess as an community-developed python library for NASA Earthdata search and access, whether locally or in the cloud.

Based on the last 3 years of engaging with the user community and the Hub, we have evolved how we onboard (first experience in the cloud) and fledge (set up for Cloud that includes a plan, how to do it, how to pay for it; leaving the nest and perhaps building your own). Fledging is an important part of adoption and initiating users to the Cloud - where do researchers go when they decide to do their science in the Cloud? We’ve been developing practices that aim to be equitable and consider policy (cost), technical (where do people go, what admin setup is needed, what tech like base images etc are needed), and social (how do I learn, get support) aspects, and look forward to discussing further at AGU.

Authors #

Julia S. Stewart Lowndes - Openscapes, LLC (first author)
Andrew Barrett - National Snow and Ice Data Center (NSIDC)
Carl Boettiger - University of California Santa Cruz
Aaron M Friesz - Organization Not Listed
Elizabeth E Holmes - NOAA Fisheries
Alexis Hunzinger - NASA Goddard Space Flight Center
Daniel Kaufman - NASA Langley Research Center
Luis Alberto Lopez - National Snow and Ice Data Center (NSIDC), CIRES, University of Colorado Boulder
Catalina M Oaida Taglialatela - NASA Jet Propulsion Laboratory, California Institute of Technology
Yuvi Panda - University of California, Berkeley
Michele Thornton - Oak Ridge National Laboratory

V31A-08 VICTOR – A new Cyber-infrastructure for Volcanology #

Link to session

Abstract #

Numerical models are essential for forecasting volcanic hazards for both short-term responses and long-term hazard assessment. While many models of volcanic processes already exist, challenges in finding, installing, and evaluating these models, coupled with limited computational resources, hinder their widespread use. To address this, we introduce VICTOR, the Volcanology Infrastructure for Computational Tools and Resources.

VICTOR is a cutting-edge cyber-infrastructure platform offering an open-source, cloud-based environment tailored for the volcanology community. It features Jupyter notebooks that integrate existing volcano models, such as the lava flow codes MOLASSES and IMEX_lava, the tephra and ash dispersal codes Tephra2 and HYSPLIT, and the mass flow code TITAN2D. The backend of VICTOR is managed as a JupyterHub, operated by the non-profit 2i2c under the Code for Science and Society.

VICTOR not only provides access to individual modeling tools, but also hosts workflows that use them in data inversion, model benchmarking, and uncertainty quantification. For example, we developed a workflow to validate mass flow models using multiple metrics and Bayesian statistics. VICTOR provides built-in access to external databases such as OpenTopography, Copernicus, and NASA’s remote sensing products to streamline obtaining and using data in workflows.

VICTOR also serves as an educational resource. In Spring 2023 and 2024 we taught graduate level, multi-institutional courses in Computational Volcanology using VICTOR, and we are creating multilingual tutorials for the workflows. We are developing teaching modules on topics such as lava flows and remote sensing to be shared with instructors. Lastly, VICTOR collaborates with national efforts including CONVERSE and SZ4D.

In summary, VICTOR addresses the critical need for accessible, effective volcanic hazard modeling tools and resources, fostering advancements in both research and education within the volcanology community.

Plain-language Summary #

VICTOR is a new online platform designed to help scientists predict volcanic hazards more easily. Traditional models can be difficult to find, use, and combine with other tools. VICTOR solves these problems by offering a cloud-based, open-source environment with user-friendly tools.

VICTOR includes tools like Jupyter notebooks that combine various volcano models for lava flows, ash dispersal, and mass movements. It operates through JupyterHub, managed by the non-profit 2i2c. The platform not only provides access to these models but also offers workflows for tasks like data analysis and model validation. For example, it has a workflow for testing mass flow models using multiple evaluation methods.

VICTOR simplifies data access by connecting directly to databases like OpenTopography and NASA’s remote sensing products. It’s also an educational tool, used in graduate courses and offering multilingual tutorials. Additionally, VICTOR is developing teaching materials on topics like lava flows and remote sensing and collaborates with national projects like CONVERSE and SZ4D.

In essence, VICTOR makes volcanic hazard modeling more accessible and effective, benefiting both research and education in volcanology.

Authors #

Einat Lev - Columbia University of New York (first author)
Sylvain J Charbonnier - University of South Florida
Charles Connor - University of South Florida
Samuel Krasnoff - Lamont -Doherty Earth Observatory
Abani K Patra - University at Buffalo

NeuroHackademy Summer School Reflections

Mon, 19 Aug 2024 00:00:00 +0000

Thank you to Ariel Rokem and Noah Benson for guest writing this blog post!

Group photo from NeuroHackademy 2024

What is NeuroHackademy? #

Part summer school, part free-wheeling hackathon, all focused on the use of data science methods in neuroscience, NeuroHackademy is an event that was recently hosted by the University of Washington eScience Institute in Seattle, WA, USA. This event, that has been running annually since 2016, aims to provide early-career researchers in Psychology, Medicine, Neuroscience, and other related fields with the skills and knowledge that they need to effectively and rigorously work with open source tools and workflows for analyzing human neuroscience data. This supports the effort to make scientific analysis and results shareable, reproducible, and accessible.

Global and inclusive #

In 2020, the event had to rapidly pivot to an online format, and this format was also used in 2021. Through this experience, the organizers ( Ariel Rokem and Noah Benson) realized that many participants preferred the online format. For example, participants who could not afford to travel to Seattle, or participants who had care-taking responsibilities that precluded them from participating in a two-week event away from their homes. In 2022, the event pioneered a hybrid format, where half of the participants are present in-person and half join the event via zoom, slack, GitHub, and of course through a dedicated 2i2c JupyterHub. Taken together, this format allows the participation of students from a larger range of backgrounds and locations. This aspect plays an important part in building a global and inclusive community of practice. See the paper Hands-On Neuroinformatics Education at the Crossroads of Online and In-Person: Lessons Learned from NeuroHackademy to read more on this subject.

Collaboration with 2i2c #

Previous years #

NeuroHackademy has been an early adopter of the cloud-based JupyterHub model, setting up its first hub using the zero-to-jupyterhub guide in 2018. NeuroHackademy partnered with 2i2c as soon as it was founded, and 2i2c has operated a JupyterHub for the last 3 years. The hub provides an interactive computing platform for learners, and implements the “digital watering hole” for practical and immediate access to a range of cloud-based datasets in human neuroscience (see blog post announcing support for this year’s event).

In terms of the software environment, the following tools and features that have benefited the event over the years include

nbgitpuller allows students to synchronise lesson content with an organizational GitHub repository that is collaboratively maintained by the lesson instructors.
Shared data file storage with read-only access for learners and read-write access for instructors
Access to an abundance of neuroimaging data hosted in cloud object storage
- The Human Connectome Project
- The Natural Scenes Dataset
- OpenNeuro
- The Healthy Brain Network
- And more.

This year #

This year 2i2c supported the following tools and features for NeuroHackademy

A “Bring your own image” option where users can pull any image hosted on a container registry into the hub. See our Integrating BinderHub with JupyterHub: Empowering users to manage their own environments blog post for more details.
repo2docker and GitHub actions to build and prototype images from a repository.
The support services provided by 2i2c and the ability for instructors to open pull requests on 2i2c infrastructure for speedy resolution.
GPU instances to support more compute intensive workloads for machine learning.

Next year #

One thing we have learned is that 2i2c automatically shuts down a user server after one hour of inactivity by default to ensure efficient resource usage and limit runaway cloud costs. Naturally, we are seeing increasing demand from learners for longer and more complex analyses. In response to this, we are keen to explore how the jupyter-keepalive extension can keep the server alive for long-running processes.

We are pleased that learners have made great progress in making use of cloud-native, open-source workflows for analyzing human neuroscience data. We are keen to benefit from lessons learned this year and are looking forward to collaborating with 2i2c once again to deliver the NeuroHackademy Summer School in 2025.

Watch this space next year!

Acknowledgements #

Funded by grant R25MH112480 from the US National Institute of Mental Health awarded to Ariel Rokem and Noah Benson.

The NeuroHackademy Summer School is sponsored by

References #

Keeping PACE with GPU enabled compute to detect global cloud cover using satellite data

Mon, 12 Aug 2024 00:00:00 +0000

(left, b&w) Model inputs and (right, color) model outputs of a simple multi-layer perceptron for detecting cloud cover.

PACE is the NASA Plankton, Aerosol, Cloud, ocean Ecosystem mission that focuses on understanding ocean health and its impact on the atmosphere. Together with the Ocean Carbon and Biochemistry (OCB) program, a one-week hackathon ran from Aug 4 to Aug 8 on the 2i2c-hosted CryoCloud hub. The goal of the hackathon was to explore new Earth science data streams provided by the OCI, SPEXone and HARP2 instruments using Python.

Machine Learning with GPUs #

One of the most advanced tutorials delivered during the hackathon was the Machine Learning Tutorial. The tutorial focused on creating a machine learning pipeline to detect cloud cover from satellite imagery. This was done by training a convolutional neural network (CNN) to assign each pixel a binary value to indicate whether the location was covered by cloud or not. To improve the spatial context beyond a single pixel value, as the likelihood of a pixel containing cloud cover increases if its neighbours also contain cloud cover, the CNN needs to be trained on the entire image at once rather than at a single pixel level. This massively increases the training time, but also allows the CNN to learn more complex relationships between pixels.

GPUs have a far greater number of cores than CPUs that are well-suited for accelerating the massive parallel processing needed to train a neural network on the large amounts of image data in the above scenario. PyTorch is a popular Python library for training CNNs, available for both CPUs and GPUs, and is an ideal tool for performing this kind of work. In terms of the accelerator hardware available on the CryoCloud hub, 2i2c provisions an instance with an NVIDIA Tesla T4 GPU with 4 CPUS, 16GB of RAM and 2,560 CUDA cores.

Managing shared memory on 2i2c hubs #

While developing the above tutorial, tutorial lead Sean Foley (NASA/GSFC/SED & Morgan State University & GESTAR II) noticed that training neural networks was way slower than it should be given the GPUs available to them. They investigated the issue, and with help from the 2i2c engineering team, it was determined that shared memory was the issue. PyTorch uses shared memory via /dev/shm for faster parallel processing, and maximizing use of GPU. However in containerized environments, this is limited to a maximum of 64MB by default.

You can check the amount of shared memory available on your hub in a terminal with the command

df -h | grep /dev/shm

As you might expect, 64 MB of shared memory is not enough for training over 160,000 images in the tutorial. 2i2c was able to remove the limit, making /dev/shm share the memory the user has selected via their profile list, rather than be artificially limited to any particular size. This was done for all users on the CryoCloud hub within an hour of the issue being reported and we upstreamed the change for all 2i2c hubs (see GitHub pull requests for CryoCloud and all 2i2c hubs).

Conclusion #

This event demonstrates the economy of how running shared and open infrastructure dynamically solves problems for the benefit of many users, not just for one occasion. Learning experiences such as the above are transferred and embedded upstream into transparent and flexible open source software that impacts not only all users of 2i2c operated hubs, but also generalized for the wider research community at large (case in point, see the Slack thread below from Eli Holmes, operator of the NOAA Fisheries hubs)! We are grateful for the strong partnerships with our communities who help us to co-design impactful solutions that are specific for their needs and accessible to all.

The power of open infrastructure beyond 2i2c-operated hubs

References and Acknowledgments #

Sean Foley (NASA/GSFC/SED & Morgan State University & GESTAR II)
Tasha Snow (ESSIC UMD & NASA GSFC & CryoCloud)
PACE Hackweek Jupyter Book

Ephemeral Interactive Computing for the AmeriGEO Workshop

Mon, 05 Aug 2024 00:00:00 +0000

AmeriGEO provides a framework for cooperation in the Americas for the use of Earth data to benefit science and society with data driven decision-making. As part of a virtual workshop held on 1st August 2024, 2i2c provided an interactive computing environment to support the delivery of a NASA Open Science / ScienceCore water module.

The workshop was facilitated by Kytt MacManus and Juan F. Martinez (both of CIESIN, Columbia University, New York). Juan presented interactive R code, with explanatory content written in Quarto, for hotspot vulnerability analysis for floods and landslides focused on Ecuador using earth observation data and socioeconomic data to develop an index of vulnerability. Check out their GitHub repo and explore interactively yourself on our BinderHub (see our blog post for more details of how to access the BinderHub deployments to provide Ephemeral Interactive Computing for NASA Communities).

Over 100 participants were able to access the interactive workshop on our infrastructure, with 8 GB of RAM per user to facilitate the processing of large amounts of earth data. We are pleased that the workshop was successful and the platform was able to provide a great experience for participants. After the workshop, Kytt reported that

The technology worked as expected and we didn’t run into 1 major technical problem. Thank you so much for that!".

Acknowledgements #

Funding from Science Mission Directorate’s Open Source Science Initiative, Research Opportunities in Space and Earth Science (ROSES-2022), F.14 Transform to Open Science
NASA NSPIRES F.15 High Priority Open-Source Science Award NNH22ZDA001N-HPOSS
NASA Socioeconomic Data and Applications Center (SEDAC)
Kytt MacManus
Juan F. Martinez
James Munroe for providing support and assistance for setting up the cyberinfrastructure for this workshop.

Openscapes Host a Surface Biology and Geology Workshop with Shared Password Feature

Tue, 09 Jul 2024 00:00:00 +0000

Thanks to Brianna Lind, Julia Lowndes and Andy Teucher for contributing to this blog post!

Surface Biology and Geology: VITALS Workshop

Openscapes is a value-based initiative that supports kinder, better science based on open source community. NASA Openscapes is in its fourth year as a project supporting NASA Earth science in the Cloud, co-developed by Julia Lowndes (Openscapes) and Erin Robinson (Metadata Game Changers).

The initiative recently supported the Surface Biology and Geology: VITALS Workshop hosted by NASA Land Processes Distributed Activate Archive Center (LP DAAC) and NASA Jet Propulsion Laboratory (JPL).

Instructors used the 2i2c Openscapes Hub to lead hands-on exercises teaching learners how to manipulate data collected from the ECOSTRESS and EMIT instruments onboard the International Space Station. They used Jupyter notebooks in the Hub to demonstrate how open source tools together with cloud data and compute resources could effectively analyse the the Canopy Water Content and the Land Surface Temperature over the Jack and Laura Dangermond Preserve, Santa Barbara, CA.

Plot of the Canopy Water Content over the Jack and Laura Dangermond Preserve, Santa Barbara, CA from a VITALS Workshop Jupyter notebook.

This event was attended by around 250 participants. An event of this size therefore requires a frictionless login flow so that organizers could focus on the essential complexity of teaching data analysis rather than the accidental complexity of managing Hub authorization. GitHub authentication is the default option for most 2i2c Hubs for research use cases, but for an educational event of this size this option was not fit for purpose since organizers had to

Retrieve the GitHub usernames of each participant (assuming everyone was familiar with GitHub!)
Manually invite GitHub users to a GitHub organization to authorize access to the Hub (invitations would expire within seven days)
Repeat the above two steps last-minute for participants who showed up on the day without preparing
Manually remove GitHub users from the GitHub organization if they wanted to revoke access to the Hub after the event.

In response to this need, we developed a shared password feature so that workshop organizers can simply hand the share password out to learners for access to the Hub. This bypassed the manual labour of managing GitHub accounts while not adding to the learner’s high cognitive load and improving the participant’s learning experience overall.

One of the elements that enabled us to recognize and solve this issue effectively is our close partnership with the Openscapes team. We engage in regular 6-weekly catch-ups where we can learn about user requirements and how we can develop our infrastructure to co-create optimal solutions. Together with our Product Delivery Flow, we were quickly able to architect the shared password solution in time for the workshop.

Feedback from Brianna Lind (LP DAAC)

We have documented the technical infrastructure changes required to enable a shared password for the Hub in our Infrastructure Guide and hope to support many future events with this mechanism!

Acknowledgements #

NASA Openscapes
NASA LP DAAC
NASA JPL
NASA ROSES funding
NASA Open Science / ScienceCore for supporting some of our work on JupyterHub.

Determining Climate Risks with NASA Earthdata Cloud at Scipy 2024

Mon, 08 Jul 2024 00:00:00 +0000

Determining Climate Risks with NASA Earthdata Cloud is a ScienceCore curriculum module that comprises part of NASA’s Open Science and Transform to Open Science (TOPS) initiatives. The aim of this module is to deliver a hands-on experience with “data-proximate computing” in the cloud with NASA Earthdata products with content co-developed with MetaDocencia.

This module was delivered as a SciPy tutorial at this year’s conference. 2i2c have been working closely with the organizers to provide the hub infrastructure for the tutorial, including enabling a shared password for easy authentication (see our Openscapes post for more about this feature) and operating a small binder service for participants to view content after the event.

You can take a look at the tutorial on the NASA ephemeral hub!

The event was well-attended, with 40 learners taking part. Special thanks go to the organizers Dhavide Aruliah, Karthik Venkataramani and Patricia A. Loto for leading the tutorial.

Acknowledgements #

NASA F.14 Transform to Open Science Training award NNH22ZDA001N-TOPST (80NSSC23K0861), which seeded this contribution to NASA ScienceCore
MetaDocencia
Dhavide Aruliah
Karthik Venkataramani
Patricia A. Loto

Ephemeral Interactive Computing for NASA Communities

Thu, 27 Jun 2024 00:00:00 +0000

We are pleased to announce that we have deployed two ephemeral hubs for NASA communities!

What did we do? #

As part of the deliverables for our NASA High Priority Open-Source Science (HPOSS) award, we deployed two new ephemeral hubs:

a public small BinderHub that offers a “reader” experience where learners can interactively view GitHub repositories that deliver light scientific content with small compute and no barriers to authentication
a big BinderHub that offers an “explorer” experience where learners can log in to access more substantial compute resources to interactively investigate large datasets and run large workflows from any GitHub repository.

These services enrich the interactive computing ecosystem for NASA communities by

improving the shareability and reproducibility of scientific information
broadening participation for historically excluded and under-resourced science communities
enabling activities such as hackathons, demonstrations or training, during workshops and conferences.

How did we do it? #

Ephemeral interactive computing services benefited from some of our previous development work in collaboration with GESIS (see our detailed blog post for more information). The research and development of this project achieved wide-reaching impact across many NASA communities we currently serve, including ScienceCore, Openscapes, US Greenhouse Gas Center, VEDA and CryoCloud; as well as networks beyond the NASA scope, such as the NSF-funded Project Pythia and HHMI-funded Spyglass projects.

What next? #

We will focus on bolstering the community- and knowledge-building needed for making the best use of these binder services in the next phase of our HPOSS work to accelerate broader participation in science. This includes opportunities such as running workshops and tutorials, as well as disseminating best practices for collaborative research. Further engineering developments will proceed in collaboration with the NASA VEDA project to set up a binder service, improve the sharing of custom environments, and refine QGIS integrations for analysing geospatial data.

Can I use this ephemeral hub service? #

The answer is yes!

For the public small BinderHub anyone can view GitHub repositories that deliver light scientific content with small compute and no barriers to authentication
For the big BinderHub you will need to be member of a NASA community. This requires a GitHub account for membership of the GitHub Team 2i2c-nasa-binder-access:big-binder-team for authorization. Please send us an email at binder-requests@2i2c.org to be added to the GitHub Team.

Acknowledgements #

NASA NSPIRES F.15 High Priority Open-Source Science Award NNH22ZDA001N-HPOSS
The NASA ScienceCore community (originally seeded by TOPS-T award 80NSSC23K0861), an anchor user community whose training delivery requirements helped shape the design of these ephemeral binder services

Neurohackademy Summer School in Neuroimaging and Data Science 2024

Fri, 21 Jun 2024 00:00:00 +0000

Neurohackademy Summer School

2i2c are pleased to support the Neurohackademy Summer School in neuroimaging and data science again!

Following the success of our collaboration in previous years (see the event page for 2023), this year’s course will be held July 29th – August 10th, 2024 and will be hosted by the University of Washington eScience Institute.

We provide an interactive computing platform for participants to get hands on experience in data pipelining, machine learning and data visualization techniques. Take a look at the following links to learn more about the neurohackathon:

Acknowledgements #

Funded by grant R25MH112480 from the US National Institute of Mental Health awarded to Ariel Rokem and Noah Benson.

The Neurohackademy Summer School is sponsored by

US Greenhouse Gas Center supports summer school at CIRA

Thu, 20 Jun 2024 00:00:00 +0000

Summer school for inverse modeling of greenhouse gases 2024

The Cooperative Institute for Research in the Atmosphere ( CIRA) is an interdisciplinary cooperation between NOAA research scientists and Colorado State University. CIRA is hosting a summer school for inverse modeling of greenhouse gases using atmospheric data assimilation techniques. The US Greenhouse Gas Center is supporting the workshop by providing 40+ attendees access to their interactive computing hub operated by 2i2c (see our blog post about their launch).

Hub administrators have built a customized software environment with container technology for use at the workshop. In doing so, this bypasses the need for participants to individually install software on their own machines and the online hub provides a consistent and reproducible interactive computing environment that is easily accessible and scalable.

Acknowledgements #

2i2c would like to credit the following individuals for their great efforts supporting this workshop:

Sanjay Bhangar ( Development seed)
Slesa Adhikari (NASA IMPACT)

Low storage alerting for the UToronto cluster

Fri, 14 Jun 2024 00:00:00 +0000

The UToronto hub landing page

2i2c has operated The University of Toronto hub since 2021 and this hub supports over 6000 educators and learners in a day! With a community of this size, file storage can quickly grow out of control and cause issues.

The 2i2c engineering team have implemented a low storage alerting system for Microsoft Azure, so that they can pre-emptively take remedial action before the filesystem is about to run out of diskspace.

Great job team 🚀

UToronto hub usage

UNITEFA forms the first community of the Catalyst Project at UNC, through the CCAD (in Spanish)

Mon, 23 Oct 2023 00:00:00 +0000

2i2c partners with UC Berkeley and CloudBank to provide data science education hubs for community colleges in California

Fri, 01 Jan 2021 00:00:00 +0000

Data8. MIT licensed.

Data8 began as a large introductory data science class at UC Berkeley. It uses a Jupyter Book for all course materials, and uses JupyterHub magic links to distribute course content from the textbook.

2i2c is working with the Data 8 team to deploy JupyterHubs for community colleges in California that run the Data8 course, to make the infrastructure and content broadly accessible.

Giving people curricula, content, and infrastructure goes a long way to adoption. Communities want to remix content for their specific needs.

Acknowledgements #

This effort is funded by 2i2c together with UC Berkeley and CloudBank.

Education | 2i2c

2i2c communities at AGU 2024

ED31G-2272 Breaking down the barriers to Open Science with Project Pythia #

Abstract #

Authors #

ED31G-2277 PACE Hackweek: An open community keeping up with PACE #

Abstract #

Authors #

IN13A-2147 Including more solutions and more solvers via actionable open science #

Abstract #

Authors #

IN34A-01 Beyond Open Data: Ensuring True Accessibility for All (Invited) #

Abstract #

Authors #

Introducing GeoLab - An EarthScope JupyterHub for Enabling Collaborative Cloud-Native Geophysical Data Analysis and Skill Development Workshops #

Abstract #

Authors #

U13A-2349 Sharing recipes for cloud computing: the Project Pythia Cookbook Initiative #

Abstract #

Authors #

U13A-2350 Supporting NASA Earthdata users in the Cloud: NASA Openscapes JupyterHub and User Onboarding & Fledging #

Abstract #

Authors #

V31A-08 VICTOR – A new Cyber-infrastructure for Volcanology #

Abstract #

Plain-language Summary #

Authors #

NeuroHackademy Summer School Reflections

What is NeuroHackademy? #

Global and inclusive #

Collaboration with 2i2c #

Previous years #

This year #

Next year #

Acknowledgements #

References #

Keeping PACE with GPU enabled compute to detect global cloud cover using satellite data

Machine Learning with GPUs #

Managing shared memory on 2i2c hubs #

Conclusion #

References and Acknowledgments #

Ephemeral Interactive Computing for the AmeriGEO Workshop

Acknowledgements #

Openscapes Host a Surface Biology and Geology Workshop with Shared Password Feature

Acknowledgements #

Determining Climate Risks with NASA Earthdata Cloud at Scipy 2024

Acknowledgements #

Ephemeral Interactive Computing for NASA Communities

What did we do? #

How did we do it? #

What next? #

Can I use this ephemeral hub service? #

Acknowledgements #

Neurohackademy Summer School in Neuroimaging and Data Science 2024

Acknowledgements #

US Greenhouse Gas Center supports summer school at CIRA

Acknowledgements #

Low storage alerting for the UToronto cluster

UNITEFA forms the first community of the Catalyst Project at UNC, through the CCAD (in Spanish)

2i2c partners with UC Berkeley and CloudBank to provide data science education hubs for community colleges in California

Acknowledgements #