Monitoring | 2i2c

Improving our community hub reliability and stability in Q4 2025

Tue, 16 Dec 2025 00:00:00 +0000

This year we’ve prioritized making the cloud safe to try for our member communities. This has driven work in monitoring, alerting, and automating infrastructure so that we resolve small problems before they become big problems. In the last quarter of 2025, we wrapped up this effort by testing the following hypothesis:

We can reduce P1 incidents if we shorten the time to act on current alerts and learnings from prior incidents.

Here’s what we accomplished and what we learned.

What we accomplished #

In short: we’re now much more confident in the stability of community infrastructure. Here’s a snapshot of our new incident dashboard, which shows high-level trends for the stability of our infrastructure:

See the real-time status of our community hubs at status.2i2c.org

We improved infrastructure reliability for our communities #

We made several technology and team process improvements that led to these benefits for our communities:

We are now more likely to catch outages before a community reports them to us.
We are now less likely to have an outage happen more than once, or affect more than one community, because we consistently fix the issues that cause outages.

We saw a consistent drop in critical alerts that required immediate response:

For August and September we had an average of 7 outages/month (6 from alerts, 1 from community)
In October, November, and December we had an average of 3 outages/month (9 in October, 0 in November, 1 in December, with only one of these being reported by a community)

We became more efficient, responsive, and focused #

We also got several team benefits from this work:

We get fewer interruptions and distractions from deeper work.
We have clear assignment policies to make it clear who is responsible for acting in response to alerts.
We avoid invisible work from falling down rabbit-holes when responding to outages.
We decreased the stress and pressure of doing upgrades, making them easier to split into sprint items and more likely to get done consistently.

The improvements we made #

Infrastructure improvements #

Created a status page for all 2i2c community hubs, giving our team and communities visibility into the status of our infrastructure.
Created an alert that triggers when two servers fail to start consecutively in a 30-minute time window.
Improved deployment infrastructure so that we can roll out sub-chart upgrades to individual clusters, allowing us to roll out major changes in batches.
Removed our “configurator” application from community hubs, because it was causing more confusion than it was resolving.
Allowed servers to start even when users hit their storage quotas.
Provided a number of upgrades to Kubernetes and the support services that we run alongside each community hub.

Process improvements #

Made a team commitment to prioritize issues from incident reports and other stability-related problems.
Defined incident escalation policies using the status page to calibrate the urgency of our response to the severity of incidents.
Defined “on-call” procedures so our team knows when and how to be more responsive to outages.
Time-boxed our alert response process to avoid accidentally falling down rabbit holes for non-urgent problems.
Created a more reliable process for responding to incidents and writing incident reports.

Looking forward #

After this push around infrastructure reliability, we’re significantly more confident in the stability and transparency of our community hub infrastructure. This will deliver better service for our member communities and free up more of our time to engage with them instead of fighting infrastructure fires.

We will continue to improve our infrastructure, and have a better foundation to do so incrementally in the coming quarters. Here are a few things we’d still like to improve:

We still need to improve how reliably we complete follow-up actions from incidents (e.g., writing incident reports). When a process doesn’t fit into planning & scoping ceremonies, we struggle to follow it consistently.
We’d like to improve our testing framework for major upgrades across all hubs (e.g., Kubernetes version upgrades) to catch bugs before communities do.

Learn More #

Faster reporting of user home directory sizes

Tue, 09 Dec 2025 00:00:00 +0000

Storage quotas help users avoid running out of space unexpectedly and give administrators visibility into capacity planning. However, storage usage can change rapidly, and it’s important to have quick information so that administrators know whether they are close to hitting limits.

We’ve improved how quickly hub administrators can see user home directory sizes across our JupyterHubs. This makes monitoring more responsive and adds quota limit visibility that wasn’t possible before.

Using `jupyterhub-home-nfs` for near-instant disk usage metrics #

Our existing storage monitoring tool, prometheus-dirsize-exporter, deliberately runs slowly to avoid excessive disk I/O. This meant home directory metrics could be hours out of date on systems with many users or large directories. Plus, there was no way to report user quota limits at all.

Our home directory storage is managed by jupyterhub-home-nfs, which enforces per-user quotas. It could also expose usage and limit information as Prometheus metrics using data from the underlying filesystem quota system. Because this information is already tracked by the filesystem, it’s available immediately without scanning individual files.

We made two key improvements:

Make disk usage reporting almost instantaneous. We made jupyterhub-home-nfs export total_size_bytes and hard_limit_bytes metrics to Prometheus for near-instant reporting. We used the same metric names and namespace as prometheus-dirsize-exporter for compatibility. See 2i2c-org/jupyterhub-home-nfs#76
Allow this to be used upstream in JupyterHub Grafana Dashboards so that it can support both types of disk usage reporting. This means users of the upstream JupyterHub Grafana dashboards get the same useful view about home directory usage, regardless of whether the metric comes from prometheus-dirsize-exporter or jupyterhub-home-nfs. See 2i2c-org/prometheus-dirsize-exporter#29

These changes were deployed across all our communities, so administrators can now access current home directory information within minutes regardless of directory size.

Home Directory Usage dashboard showing total size metrics from jupyterhub-home-nfs and other data from prometheus-dirsize-exporter

Try it out #

2i2c member organizations can try this out now. If you have access to your hub’s Grafana instance, you can see these new metrics in the Home Directory Usage dashboard:

Open your hub’s Grafana dashboard.
Go to Dashboards -> JupyterHub Default Dashboards -> Home Directory Usage.
Check the table for up-to-date total size and quota limit values.

For more details, see our docs on filesystem and disk dashboards.

Coming next #

We’d like to build on this work to enable alerting when individual users near their disk quotas. This will make it easier to more reliably track user disk usage across a community. See this issue for tracking: 2i2c-org/infrastructure#7166

Acknowledgements #

This was a directed contribution supported by NASA VEDA to enable more proactive monitoring and alerting for hub administrators.

Adding User Group Insights to Cloud Cost Dashboards with Grafana

Mon, 24 Nov 2025 00:00:00 +0000

We are excited to announce that we have extended our cloud cost dashboards to support display costs filtered by user groups using Grafana! This new feature allows administrators to monitor and manage cloud expenses based on user group memberships in JupyterHub.

Available for dedicated AWS clusters only (and excluding CloudBank managed accounts). Other deployments on GCP will be supported in the future.

Learn more #

Take a look at the Community Hub Guide to see what’s new
Check out the documentation of the 2i2c-org/jupyterhub-cost-monitoring project to see how it all works
Jenny recently presented her work on the cost monitoring system at JupyterCon 2025 earlier this month. Watch a video or look at the slides.

Give us feedback! Click here to provide feedback that will help us make this more impactful.

Acknowledgements #

Tarashish @ Development Seed for collaborating on this project with us.
NASA VEDA and the DSE Team at NASA MSFC ODSI for funding much of this work.
Kyle Lesinger from the NASA MSFC Office of Data Science and Informatics for providing valuable feedback and bug reports during development.

Announcing `jupyterhub-groups-exporter`: monitor usage based on JupyterHub group membership with Prometheus and Grafana

Wed, 11 Jun 2025 00:00:00 +0000

Managing user groups in JupyterHub can be a challenging task, especially in environments with dynamic user bases and complex group structures. This post describes how we can leverage the latest group management features in JupyterHub, along with Prometheus and Grafana, to monitor group-level resource usage effectively.

⭐ Members of 2i2c’s community network can use this feature in their hubs by following our cost attribution documentation.

Motivation #

Hub admins have a strong impetus to monitor usage and costs by user groups because it allows them to advocate for better funding and cost recovery models based on data-driven insights. Group-level resource monitoring can help them to answer questions like:

How many people participated in our workshop group?
How much GPU compute is our power user group using?
Is our resource usage cost-effective for X group persona or Y group persona?

Current methods and workarounds include:

ring-fencing resources for specific user groups personas, e.g. creating a separate hub for a workshop group, or creating a separate Dask cluster for a power user group, which increases the admin burden of managing multiple hub instances
writing custom scripts to aggregate per user metrics, that are already available, into groups – which can be time-consuming and error-prone

JupyterHub and user groups #

Recent key developments upstream in JupyterHub for groups management, such as Authenticator managed group membership, makes this piece of work a prime and timely opportunity to be tackled. For more technical details of these upstream contributions, see GitHub PRs jupyterhub/oauthenticator#735 and jupyterhub/oauthenticator#498.

Users can access JupyterHub using a variety of authentication methods. Authentication providers like GitHub have built-in user management features that allow admins to create and manage user groups. These groups can then be configured in JupyterHub to authorize access to the hub, as well as control access to certain hardware profiles.

Following the key upstream contributions above, we can leverage Authenticator-managed group membership to automatically pass user group memberships from the authentication layer to JupyterHub itself. This allows us to capitalize on JupyterHub’s REST API to retrieve user group memberships from other services, such as exporting them as Prometheus metrics.

Exporting user group memberships to Prometheus #

The jupyterhub-groups-exporter project provides a service that integrates with JupyterHub to export user group memberships as Prometheus metrics. This component is readily deployable as part of any JupyterHub instance, such as a standalone deployment or a Zero to JupyterHub deployment on Kubernetes.

The exporter provides a Gauge metric called jupyterhub_user_group_info, which contain the following labels:

namespace – the Kubernetes namespace where the JupyterHub is deployed
usergroup – the name of the user group
username – the unescaped username of the user
username_escape – the escaped username

Escaped usernames are useful because Kubernetes pods have characterset limits for valid pod label names (this limit does not apply to pod annotations). Storing both types of usernames allows us to join escaped versions with their more human-readable unescaped usernames.

Exposing this metric as an endpoint for Prometheus to scrape allows us to query and join groups data with a range of usage metrics to gain powerful group-level insights. Here is an example PromQL query that retrieves the memory usage by user group:

sum(
 container_memory_working_set_bytes{name!="", pod=~"jupyter-.*", namespace=~"$hub_name"}
 * on (namespace, pod) group_left(annotation_hub_jupyter_org_username, usergroup)
 group(
 kube_pod_annotations{namespace=~"$hub_name", annotation_hub_jupyter_org_username=~".*", pod=~"jupyter-.*"}
 ) by (pod, namespace, annotation_hub_jupyter_org_username)
 * on (namespace, annotation_hub_jupyter_org_username) group_left(usergroup)
 group(
 label_replace(jupyterhub_user_group_info{namespace=~"$hub_name", username=~".*", usergroup=~"$user_group"},
 "annotation_hub_jupyter_org_username", "$1", "username", "(.+)")
 ) by (annotation_hub_jupyter_org_username, usergroup, namespace)
) by (usergroup, namespace)

Visualizing user group resource usage with Grafana #

The PromQL query above is rather long and complex to construct! However, you can benefit from an upstream contribution to the jupyterhub/grafana-dashboards project where we have encapsulated the PromQL queries as Jsonnet code and represented them as Grafana Dashboard visualizations (also known as Grafonnet). If you have a Kubernetes cluster running JupyterHub, try deploying these Grafana Dashboards and let us know what you think!

Our particular PromQL query above is visualized in the Grafana Dashboard User Groups Diagnostics under the Memory Usage panel (see also the corresponding screenshot at the top of this post). This is equivalent to its counterpart User Diagnostics dashboard, but with resource usage visualized on a per-group level rather than a per-user level 🎉

Future work #

We have laid the foundation for joining user group data to usage metrics with Prometheus by extracting memberships from JupyterHub’s database. This unlocks potent ways in which observability systems can be extended to group-level reporting and monitoring.

Future directions for this work include:

visualising cloud cost by user group in Grafana
developing more group-level reporting and monitoring dashboards
introducing group-level resource quotas.

What do you think? How would you like to see JupyterHub’s group management features evolve? Have you tried deploying this yourself? We welcome your feedback and feel free to open GitHub issues or make contributions to the repositories mentioned in this post.

Acknowledgements #

Thanks to the JupyterHub project for their collaboration and review of this work.

Track and manage cloud costs using Grafana

Fri, 15 Nov 2024 00:00:00 +0000

Grafana dashboard showing cloud costs broken down by compute, storage and other components for the Openscapes hub.

We are pleased to unveil a new feature to track cloud costs within our Grafana dashboards! Community Champions now have the ability to monitor the cost and usage of their 2i2c-managed hubs that displays up to date aggregated costs as well as detailed breakdowns for more granular reports.

Note that this feature is currently available to AWS hosted hubs only and will be rolled out to other cloud providers in the future.

Accessing the cloud cost dashboard #

Community Champions can view the Cloud Cost dashboard from their Grafana instance (please see the Service Guide for how to gain access).

From the main menu of Grafana, navigate to Dashboards > Cloud cost dashboards > Cloud cost attribution to view the dashboard.

Understanding the cloud cost dashboard #

A typical 2i2c-managed deployment comprises of a staging hub and a production hub, although some other communities may have extra hubs such as a workshop hub. By default, costs are not broken down on a per hub basis unless the community has opted in to this feature.

The dashboard is made of several panels:

Daily costs
Daily costs per hub (opt-in only)
Total daily costs per component
Daily costs per component per hub (opt-in only).

For more detailed information on the data that each panel displays, please consult our Service Guide for reference.

The dashboard can be shared with other community members and stakeholders so they can understand usage and cost patterns. Community Champions can export data to a CSV file, or they can generate a snapshot of the Grafana dashboard and share a public link.

For instructions on how to export data from the dashboard, please see our Service Guide for reference.

Next steps #

We would love to know whether this feature is useful and how it can be improved. We will be contacting individual communities to share their feedback with us – please share your thoughts with us!

We will work on rolling out this service to GCP hosted clusters in future. Stay tuned to know when this feature is available to your community.

Acknowledgements #

Thank you to Erik for spearheading the rollout effort and to the rest of the 2i2c team for their support.
Thanks to Openscapes and Cryocloud communities for providing valuable insights during the prototyping and testing phase, and for funding part of this work.

Monitoring | 2i2c

Improving our community hub reliability and stability in Q4 2025

What we accomplished #

We improved infrastructure reliability for our communities #

We became more efficient, responsive, and focused #

The improvements we made #

Infrastructure improvements #

Process improvements #

Looking forward #

Learn More #

Faster reporting of user home directory sizes

Using jupyterhub-home-nfs for near-instant disk usage metrics #

Try it out #

Coming next #

Acknowledgements #

Adding User Group Insights to Cloud Cost Dashboards with Grafana

Learn more #

Acknowledgements #

Announcing `jupyterhub-groups-exporter`: monitor usage based on JupyterHub group membership with Prometheus and Grafana

Motivation #

JupyterHub and user groups #

Exporting user group memberships to Prometheus #

Visualizing user group resource usage with Grafana #

Future work #

Acknowledgements #

Track and manage cloud costs using Grafana

Accessing the cloud cost dashboard #

Understanding the cloud cost dashboard #

Sharing cost reports #

Next steps #

Acknowledgements #

Using `jupyterhub-home-nfs` for near-instant disk usage metrics #