Damián Avila | 2i2c

Open Source Infrastructure Engineer

Tue, 10 Oct 2023 00:00:00 +0000

We are looking for an experienced Open Source Infrastructure Engineer who will help shape the future of data-intensive scientific research and make a big impact on democratizing the design and access to cloud-based resources for research and education purposes. This engineer will be part of an awesome engineering team pushing forward the development and reliable operations of our cloud-based infrastructure.

For more information and to apply, see our Greenhouse application page for this job.

❌❌❌
This posting is closed to new applications.
See the jobs page for our open positions.
❌❌❌

About the position

Salary: $121,600 - $130,500
Location: Remote, required overlapping for US Pacific timezones
Deadline: We will begin reviewing applications around October 16th, and will accept them on a rolling basis until the position is filled.

Major duties

Cloud infrastructure management and operations
Site Reliability Engineering
Development of open source infrastructure for hosted JupyterHub service

For more general information about 2i2c, see the links below:

Click here to Apply

GESIS - 2i2c collaborate to build a persistent BinderHub experience

Mon, 28 Nov 2022 00:00:00 +0000

Introduction #

Mybinder.org enables researchers across the world to replicate computational environments in the cloud. It allows researchers to turn static code into interactive literate coding environments with a click of a button within seconds. The mybinder.org service is powered by BinderHub, an open-source tool developed by the Jupyter Project that many organizations have deployed for their own communities. It does this by _dynamically building _the software environment needed to reproduce a computation (using a tool called repo2docker), and making this environment available to users.

BinderHub was developed for use-cases that are temporary and fully open by design. BinderHub sessions are destroyed after a fixed amount of time and there is no persistent storage or authentication. However, many research institutions also need more “standard” service features like authentication and persistent storage.

Over the past several years, the GESIS Notebooks team made the first steps towards bridging this gap through their Persistent BinderHub implementation. This was a modified and authenticated BinderHub that included persistent storage across sessions. The Persistent BinderHub service was very successful at GESIS and with its partner communities, and the team wishes to build this functionality into the JupyterHub community’s core technology so that these features can be enjoyed for more use-cases and by many communities.

To enable this vision, we have partnered with GESIS in cooperation with NFDI4DS (GAN: 460234259), CESSDA, and members of the JMTE project. This collaboration has three primary goals:

Generalize the Persistent BinderHub functionality/experience to run on cloud-agnostic infrastructure, so that other stakeholders in NFDI, CESSDA, and the broader scientific community may benefit from this functionality and experience.
Upstream this functionality by making contributions into Jupyter community projects, so that it will be maintained and improved by a community moving forward, thus improving its reliability and sustainability.
Improve the implementation and user experience around Persistent BinderHub, in order to make it more reliable, scalable, productive, and enjoyable to use.

We began this collaboration several months ago, and have focused our efforts on exploring potential implementation pathways for this functionality. We believe that we now have a path forward for this functionality, and this blog post is a brief report of our efforts and future plans as we undertake this effort. See this GitHub Projects Board for issues that implement this effort.

Exploration 1: Adding persistent storage directly into BinderHub #

Our initial intention was to incorporate persistent storage and authentication from the GESIS Persistent BinderHub into the BinderHub codebase. We began by holding a series of meetings to discuss technical requirements from our experience in the JupyterHub/BinderHub ecosystem, and also conducted an audit of the Persistent BinderHub codebase. The Persistent BinderHub implementation is a modified Helm Chart that configures a JupyterHub to expose its authentication and persistent storage functionality, overriding the BinderHub default behavior. We were concerned that building this functionality natively into BinderHub would be challenging given that the BinderHub codebase was designed for ephemeral user sessions.

So, we decided to take another approach:

Exploration 2: Add dynamic image building to JupyterHub #

We realized that there is a way to make this functionality more broadly useful and more maintainable, while still achieving the end-user experience that the GESIS team needed. Instead of modifying BinderHub to incorporate JupyterHub’s storage and authentication features, we would give JupyterHub the ability do dynamically generate user environments using repo2docker.

This would give JupyterHub users more flexibility over the environments served by their hub, and expose Binder-style workflows to the “typical” JupyterHub workflow. BinderHub could then be simplified to re-use JupyterHub’s image building functionality as a part of its own service. We also identified a prototype of this functionality in the tljh-repo2docker project that QuantStack had built for the PlasmaBio project. This implementation was seen as successful, and something others in the community had wanted to generalize for some time.

Our implementation plan #

Two phases of implementation #

With this alternative implementation route in place, we identified two major steps to accomplish this project:

Build a back-end for dynamic environment building. JupyterHub needs to understand how to call repo2docker’s image generation from a Docker-based environment. It needs to expose this ability via APIs that others can build interfaces on top of.
**Build a front-end that is user-friendly and accessible. **Once the back-end is functional, we must build a front-end experience that feels familiar to BinderHub users and is easy and intuitive to use.

Here are a few tasks that we’re carrying out next to make progress on the above two items.

Build a working prototype for image generation via a JupyterHub Service (see below for current status)
Research the tljh-repo2docker code base to understand how we could build upon its UX and functionality.
Understand the typical process that GESIS and NFDI users follow in their BinderHub workflows, to ensure that it can be replicated via this new implementation.
Perform UI/UX research validation to inform the implementation from a user’s perspective.

As a follow-up, we’ll likely re-work the BinderHub codebase to utilize JupyterHub’s new repo2docker service, rather than defining its own custom repo2docker functionality.

Back-end implementation as a JupyterHub Service #

We’re planning to use JupyterHub Services to add the back-end functionality for dynamic image building to a JupyterHub. Services are a way to expose functionality via a JupyterHub REST API, and would allow us to expose basic image generation on-the-fly with repo2docker. We aim for basic functionality to be as close as possible to repo2docker’s default behavior, but to make this functionality _composable and customizable _if a JupyterHub administrator wants to provide different out-of-the-box functionality.

While details may still change, we believe that the major architectural plan has now been settled. We have an experimental codebase with a basic implementation of the repo2docker service described above. As part of this effort, this team has also made a number of improvements to repo2docker’s codebase and project infrastructure. We hope that this effort will continue to push resources and attention to repo2docker by growing the number of users and stakeholders that rely on the project for their success.

Front-end implementation that uses this service #

Once the back-end setup exists, we can use it to begin prototyping user interactions that can trigger and use repo2docker’s image generation. Bringing dynamic environment image building into JupyterHub is a significant improvement in its functionality, and may introduce new kinds of workflows that we hadn’t initially imagined. Understanding, interpreting, and extending the original “Persistent BinderHub” workflow will require a more thorough understanding of user stories and needs in order to identify new workflows that feel natural not only in a JupyterHub scenario but also in a Binder-like scenario.

For example, here are a few major UI/UX questions we must answer:

When a user builds an image, should it also become available to other users as well?
How can a user store, find, and delete old images that they’ve built?
What about new versions of the same image?
Should we simply mimic the mybinder.org UX, or should this be improved as well?

We must answer these and several other questions next. With that vision in place, we’d like to expand our efforts in UI/UX and user research. This will help inform the technical implementation of this work as we expand on our prototype, and help us choose the right way to expose this functionality to users. We welcome collaboration around this work - if you know of an organization that is interested in collaboration, please reach out.

Collaborate and follow along #

Below you’ll find a rough project plan to give an idea for the major actions needed and a timeline for when we hope they’ll be completed. We’ll track further updates and progress on this project in this dedicated GitHub issue and this dedicated GitHub project board.

After a few months working on this project, we are even more excited about the potential for dynamically building environments in a JupyterHub. We believe that it adds a new class of workflows to JupyterHubs that were not possible before, and will be immediately useful to the hundreds of communities that deploy JupyterHub for their communities.

Open Source Infrastructure Engineer: Site Reliability Engineering and Cloud Infrastructure

Fri, 14 Oct 2022 00:00:00 +0000

2i2c manages, supports, and builds community-centric infrastructure for interactive computing in the cloud with partner communities in research and education.

We’re looking for an Open Source Infrastructure Engineer that will join our Site Reliability Engineering team and make our cloud infrastructure more reliable, scalable, and efficient. It will help build a future of data-intensive scientific research and democratize the design and access to cloud-based resources for research and education purposes.

If you’re interested, learn more about 2i2c at the links below, and more about this job posting on the rest of this page.

❌❌❌
This posting is closed to new applications.
See the jobs page for our open positions.
❌❌❌

About the position

Salary: $120,000 - $135,000
Location: preferred at the US/Pacific time zone
Deadline: We will begin reviewing applications around November 17th, and will accept them on a rolling basis until the position is filled.

Major duties

Site Reliability Engineering
Cloud infrastructure management, operations, and support.
Development of open source infrastructure for hosted JupyterHub service

For more general information about 2i2c, see the links below:

Click here to Apply

Who we are #

2i2c is a non-profit organization with a mission to make interactive computing more accessible, scalable, and powerful for research and education: 2i2c.org/organization/.

We accomplish this mission by providing managed cloud services for interactive computing, by providing development and technical leadership to researchers and educators that utilize this infrastructure for specific communities, and by providing support for open source tools and communities in this ecosystem.

We have deep ties to the open source community, and have been leaders and core contributors across dozens of projects - in particular in the Jupyter ecosystem. We also have deep ties to research and education - our team has spent years deploying infrastructure for universities, community colleges, and research teams, and now we’re bringing this experience to a wider audience with 2i2c.

We believe strongly in communities that are inclusive, transparent, equitable, effective, and diverse, especially 2i2c itself. We believe that our values should permeate everything about 2i2c, including the work we do, the communities we serve, and our own organizational culture.

2i2c is a fiscally sponsored project of Code for Science and Society, a registered US 501(c)(3) nonprofit corporation.

What you’ll do #

Ensure the reliable operation of the 2i2c infrastructure (leveraging production-ready cloud-based tools such as JupyterHub, BinderHub and Dask).
Address support issues
Explore emerging technologies in the Cloud / DevOps space, design and implement cloud computing architecture in partnership with our team.
Participate in upstream open source communities we rely on (such as JupyterHub, BinderHub, Dask, etc) in partnership with the established leaders of those communities and collaborate with the Community Lead in the education and outreach around cloud computing.
Work with a distributed and global team - team members are given a lot of autonomy, and expected to be proactive at communicating with one another and working with others to allocate effort that will maximize our impact.

Essential requirements #

Experience deploying applications on cloud infrastructure.
Experience deploying and developing with Linux container-based technologies, such as Docker and Kubernetes.
Experience with continuous integration services (e.g. Circle CI, GitHub workflows).
Experience developing tools in a general purpose programming language (eg. Python).
Experience collaborating and coordinating work via online platforms, such as GitHub, GitLab, or BitBucket, and distributed revision control.
Experience working with distributed service teams that use asynchronous methods of communication

Desirable requirements #

Experience with major cloud providers.
Experience in programming and software engineering with a track record of leadership in open, collaborative projects with broad community adoption.
Experience working on geographically distributed open-source projects.
Experience with the Jupyter ecosystem and other tools for interactive computing.
Evidence of existing connections and relationships in the worldwide ecosystem of open source software for data-intensive research and ability to establish new ones.
Experience with common data science methods, platforms, workflows, and infrastructures; with data management systems, practices, and standards; and the capacity to gain familiarity with new related topics.
Experience engaging with highly technical researchers across a variety of methodological fields, research domains, and computational platforms.
Experience building and maintaining continuous deployment pipelines.
Interpersonal skills to work with researchers and students, including skills to communicate complex information in a clear and concise manner both verbally and in writing.

What it is like to work at 2i2c and why you’ll like working with us #

2i2c is a remote-first team that believes strongly in our organizational values and mission. We recommend that you visit our jobs page, to learn about what it’s like to work at 2i2c and why you should work with us. You will also find relevant information about our compensation philosophy and our commitment to equal opportunity. For an even deeper look into how 2i2c works, check out our Team Compass.

How to apply #

Please fill out the following application form. We will begin reviewing applications on November 17th, and will accept them on a rolling basis until the position is filled.

We are eager to receive your application!