Trusted CI has published a new success story on its collaboration with Tapis. In 2023, the Texas Advanced Computing Center engaged Trusted CI, the NSF Cybersecurity Center of Excellence, to assess the security of its Tapis software. Applying First Principles Vulnerability Assessment methodology, the Trusted CI team found four serious security vulnerabilities and one bug in the Tapis code and made several recommendations to improve Tapis’ security.
Trusted CI’s first Framework Cohort has successfully completed its initial six-month period of workshops designed to improve NSF Major Facilities’ alignment to the Trusted CI Framework. Each cohort member adopted the Trusted CI Framework as the foundation for their cybersecurity program. Additionally, each cohort member worked closely with Trusted CI to produce 1) a validated self-assessment of their cybersecurity program’s alignment with the Trusted CI Framework; and 2) a draft Cybersecurity Program Strategic Plan identifying priorities and directions for further refining their cybersecurity programs.The inaugural Cohort included the following NSF Major Facilities:
The success of the Framework Cohort is particularly notable as each of these facilities voluntarily adopted and rallied around the Trusted CI Framework as the foundation for their cybersecurity programs.
The foundation of the Cohort program is the Trusted CI Framework, which was created as a minimum standard for cybersecurity programs. In contrast to cybersecurity guidance focused narrowly on cybersecurity controls, the Trusted CI Framework provides a more holistic and mission-focused standard for managing cybersecurity.
For GAGE, LIGO, NRAO, NSO, and OOI, the Cohort was their first formal training in the Trusted CI Framework’s “Pillars” and “Musts” and how to apply these fundamental principles to assess and strengthen their cybersecurity programs. NOIRLab contributed their experience as an early adopter of the Framework, having previously completed a one-on-one Framework engagement with Trusted CI.
Feedback from members of the first cohort on their experience has been strongly positive:
Eric Cross, Head of Information Technology, National Solar Observatory, said the following about his experience:
"The TrustedCI Framework Cohort was a valuable experience. The process required us to research and reflect on our internal cybersecurity policies and procedures. The Cohort provided a platform to meet with other facilities and work through challenges with feedback from peers. The experience resulted in formal documentation that provided our organization's leadership clear direction to improve our cybersecurity program with specific short-term and long-term goals. I highly recommend this exercise for all NSF facilities."
Craig Risien, CI Systems Project Manager, Ocean Observatories Initiative, said the following about his experience:
“I found participating in Trusted CI’s first Framework Cohort to be exceptionally instructive and really enjoyed the opportunities to discuss cybersecurity challenges and lessons learned with Trusted CI and colleagues at other NSF Major Facilities. Working with Trusted CI on creating a validated self-assessment based on the Trusted CI Framework over the past six months has helped the Ocean Observatories Initiative (OOI) better understand the current state of its cybersecurity program. Being part of this cohort has also assisted the OOI with the development of a plan to fully implement the Trusted CI Framework and create a well-established and mature cybersecurity program. I look forward to the follow-on cohort sessions in the coming months.”
Trusted CI is continuing to support the first cohort through the end of 2022 by facilitating monthly workshops. Each facility will have the opportunity to lead a workshop in which they are encouraged to share their specific challenges and seek advice among the other cohort members.
Concurrently, Trusted CI is conducting its second cohort engagement leveraging the lessons learned from the first cohort. The second cohort includes the following organizations:
Trusted CI is excited to be working with these new facilities to advance their understanding and implementation of cybersecurity programs and best practices!
For more information, please contact us at info@trustedci.org.
The Framework Cohort is a six month, group engagement aimed at facilitating adoption and implementation of the Trusted CI Framework among NSF Major Facilities. During the engagement, members of the cohort will work closely with Trusted CI to adopt the Trusted CI Framework at their facility, emerging with a validated assessment of their cybersecurity program and a strategic plan detailing their path to fully implement each Framework Must.Cohort members will participate in six monthly workshops (each three hours) and spend no more than eight hours each month outside of the workshops on cohort assignments. The second cohort will meet from July to December 2022.
If you have any questions, please contact us at info@trustedci.org.
The Ocean Observatories Initiative (OOI, https://oceanobservatories.org/), funded by the NSF OCE Division of Ocean Sciences #1743430, is a science-driven ocean observing network that delivers real-time data from more than 800 instruments to address critical science questions regarding the world’s oceans. OOI data are freely available online to anyone with an Internet connection.
The OOI provides an exponential increase in the scope and timescale of observations of the world’s oceans. Present and future educators, scientists, and researchers are able to draw conclusions about climatological and environmental processes based on these measurements, requiring the data to be accurate, with a flawless pedigree. As a result, the OOI has a requirement to protect its data from being altered by any external agent.
To this end, OOI-CI (OOI Cyberinfrastructure) solicited a consultation from Trusted CI to evaluate their current security program, along with guidance on reviewing and evaluating potential alternatives for an enhanced security posture. We refined and prioritized OOI’s needs to the following goals: (i) perform a security review of OOI’s cyberinfrastructure using the Trusted CI Security Program Evaluation worksheet, (ii) take steps toward adopting the Trusted CI Framework by developing a “master information security policies and procedures” document (MISPP), (iii) investigate and document missing policies and procedures, including questions and concerns raised by OOI, and unknowns discovered in above exercises, and (iv) provide guidance on creating an asset inventory, applying a control set, and creating and maintaining a risk registry.
The OOI team completed the Trusted CI Security Program Evaluation spreadsheet. This exercise started the OOI team thinking about and discussing cybersecurity concerns that were raised in the evaluation, both in previously known topics, but also unknown or undefined areas. The Trusted CI team created a list of prioritized recommendations aligned with Framework Musts -- core concepts that every cybersecurity program should have -- that the OOI team can use in addressing or documenting gaps.
We introduced OOI to the Framework and Implementation Guide, and had discussions concerning the Musts, what they entail, and how they apply to and define a mature security program. The OOI team attended the 2021 NSF Cybersecurity Summit and specifically The Framework Workshop, where they were able to benefit from a deeper dive into the Framework and implementation guidance.
OOI displayed a solid grasp of the suggested security program solution, the Trusted CI Framework, and of what needs to be done to adopt it. Completely adopting the Framework was beyond the scope of this engagement, however OOI focused on (i) developing the top-level Master Information Security Policy & Procedures (MISPP) document, (ii) develop a Cybersecurity Strategic Plan, and (iii) develop supplemental security program policies, e.g., Incident Response Plan, Disaster Recovery, and Acceptable Use Policies.
In addition to creating top level policy documents, Trusted CI stressed the importance of having an up to date asset inventory as well as selecting and applying a base-line control set. The OOI team began identifying their critical assets as well as selecting CIS v8 as a control set and then aimed to apply controls from Implementation Groups 1 and 2. Trusted CI staff also provided a list of ‘high priority’ controls to focus on that would provide the best ROI for time and resources spent implementing.
We are pleased to announce that OOI is a participant in Trusted CI’s Framework Cohort taking place the first half of 2022 (1H2022). This will allow them to continue their work on creating and refining a mature security program while working with other NSF Major Facilities under the guidance and expertise of Trusted CI’s Framework team.
The engagement ran from August 16, 2021 to December 31, 2021, and was recorded in the document “OOI / Trusted CI Engagement Final Report” (https://hdl.handle.net/2022/27253).
In the first half of 2021, Trusted CI conducted an assessment of NOIRLab’s cybersecurity program using the Trusted CI Framework. The assessment culminated in the delivery of an Assessment Report [1] describing NOIRLab’s cybersecurity program and recommendations to improve. The report also included an “implementation rating” for each of the 16 Trusted CI Framework Musts.
In the second half of 2021, NOIRLab and Trusted CI continued the engagement with a series of monthly workshops designed to aid NOIRLab in implementing the highest priority recommendations from the Assessment Report. These workshops allowed Trusted CI to continue to provide input and guidance while NOIRLab tackled the most pressing changes needed to its cybersecurity program.
Engagement Outcomes
John Maclean, the Director of Center Operations Services for NOIRLab, said the following of the engagement:
“Trusted CI has given us a Framework, appropriate to our environment, with which to build our cybersecurity program. It allows us to do this in a manner that balances scientific productivity against organizational risk in a cost effective manner.”
Chris Morrison, the engagement lead for NOIRLab, said the following of the engagement:
“As we continue to merge technologies and processes throughout our constituent programs, the Framework assessment helped us focus our cybersecurity effort and think strategically. The programmatic focus on the initiatives is helping us make cybersecurity visible and understandable across the organization. The follow-on activities will unquestionably support this systematic deployment and facilitate communication and decision-making with NOIRLab’s senior leadership. We are incredibly pleased with the process and outcome of the engagement with Trusted CI, and we now have a clear and prioritized path forward.”
[1] This assessment was based on the PACT cybersecurity assessment methodology. PACT was developed by the Center for Applied Cybersecurity Research in collaboration with the US Navy. For more information about PACT, see https://cacr.iu.edu/pact/index.html.
The cohort pilot officially begins in January 2022 and will include the following NSF Major Facilities:
The Trusted CI Framework is a resource to help organizations establish and refine their cybersecurity programs. It is the product of Trusted CI’s many years of accumulated experience conducting cybersecurity research, training, assessments, consultations, and collaborating closely with the research community. In March 2021 Trusted CI published the Trusted CI Framework Implementation Guide (FIG) for Research Cyberinfrastructure Operators as the standard for cybersecurity programs among NSF funded organizations. Publishing the FIG represented a major step forward in advancing Trusted CI’s mission to enable trustworthy science through cybersecurity guidance, templates, and tools, empowering those projects to focus on their science endeavors.
Now that the FIG has been published, Trusted CI’s aim is to help facilitate Framework adoption and implementation across the broader NSF community. To fully realize the cybersecurity benefits provided by Framework implementation, community adoption must be facilitated at a much faster pace than is possible through the traditional one-on-one engagements undertaken by Trusted CI. To address this challenge, Trusted CI launched the “cohort” approach, where representatives from multiple NSF Major Facilities will participate in a group engagement with Trusted CI focused on adoption and implementation of the Framework.
Trusted CI anticipates the cohort project will span from CY2021 to CY2024 to reach the 25-30 NSF Major Facilities and other NSF research programs targeted for this effort. Trusted CI leadership will discuss the timing and plans for future cohorts in early spring based on the progress and success of this pilot. As Trusted CI gains experience from this initial Framework Cohort, we will keep the community informed of upcoming plans and opportunities for additional facilitated Framework adoption.
[1] “Adoption” refers to an organizational commitment to use the Framework as the foundation for its cybersecurity program, and to make the Musts a strategic priority. Adoption is designed to be a low bar, and does not require any implementation. “Implementation” refers to bringing all Musts to (at least) a minimum level of competence. This is a longer term goal.
Regardless, the engagees agreed that there was value in continuing the original engagement tasks, which include the following.
Three documents were produced from these engagement tasks.
All documentation produced during this engagement has been published to a GitHub repository.
Concurrent with this Trusted CI engagement, the Jupyter Security Coordination Team began working with the Jupyter Steering Council to address security issues across the Jupyter project as a whole. This effort led to the following milestones.
This engagement represents the start of a bigger conversation focused on Jupyter security concerns. It is our hope that the documentation produced by this engagement will be incorporated by Jupyter developers into their project documentation to assist administrators and users in securing their deployments.
The OOI provides an exponential increase in the scope and timescale of observations of the world’s oceans. Present and future educators, scientists, and researchers will draw conclusions about climatological and environmental processes based on these measurements, which sets a requirement for the data to be accurate, with a flawless pedigree. As a result, the OOI has a requirement to protect its data from being altered by any external agent.
To this end, OOI-CI (OOI Cyberinfrastructure) is seeking consultation from Trusted CI on evaluation of their current security program, along with guidance on reviewing and evaluating potential alternatives for an enhanced security posture. Through a kick-off meeting, Trusted CI and OOI discussed their concerns, questions, and goals, including: penetration testing; system and software vulnerability scanning and remediation; gaps in current policies and procedures; developing periodic security tasks; and identifying ‘unknowns’. These topics were refined and prioritized based on their needs using a subset of tasks outlining the goals of the engagement, specifically:
Additionally, broader impacts from this engagement can be realized as the OOI-CI is connected to several locations around the country. Lessons learned and recommendations from the engagement will be implemented at the other sites, which consist of Woods Hole Oceanographic Institute (WHOI) administration, and the three MIO’s (Marine Implementing Organizations) that provide data from Oregon State University, University of Washington, and WHOI.
The engagement will run from September 2021 to December 2021.
Open OnDemand, funded by NSF OAC, is an open-source HPC portal based on the Ohio Supercomputer Center’s original OnDemand portal. The goal of Open OnDemand is to provide an easy way for system administrators to provide web access to their HPC resources.
Open OnDemand is facing increased community adoption. As a result, it is becoming a critical production service for many HPC centers and clients. Open OnDemand engaged with Trusted CI to improve the overall security of the project, ensuring that it continues to be a trusted and reliable platform for the hundreds of centers and tens of thousands of clients that regularly utilize it.
Our engagement centered on providing the Open OnDemand team with the skills, tools and resources needed to ensure their software security. This included using the FPVA methodology to conduct in-depth vulnerability assessments independently. In addition, we evaluated the static analysis and dependency checking tools used by Open OnDemand. The analysis of this evaluation led to interesting findings regarding the way tools behave and a set of recommendations regarding which tools to use and how to most effectively configure them.
Trusted-CI has performed in-depth assessments for NSF projects in the past. In this engagement with Open OnDemand, we took a step forward as Trusted CI taught a group how to perform the assessment themselves. In general, the NSF community benefits from being able to carry out that kind of activity in an autonomous way. In addition, the lessons in this engagement related to automated tools will benefit any NSF software project.
Open OnDemand Software Engineer, Jeff Ohrstrom, shared positive feedback regarding the value of the engagement, stating “The biggest takeaway for me was just getting muscle memory around security to start to think about attack vectors in every change, every commit, every time.”
Our findings and recommendations are summarized in our engagement report, which can be found here.
The University of Arkansas has engaged with Trusted CI and the Engagement and Performance Operations Center (EPOC) to review their plans for a Science DMZ that will serve institutions for higher education across Arkansas. Trusted CI and EPOC will also help create training and policy materials that can be reused by other institutions both in the state of Arkansas and beyond.
Science DMZs are a secure architecture for providing high throughput transfer of science data between two points. By placing data transfer nodes outside each institution's cononical network and into a specially controlled zone, the Science DMZ is able to increase speed by reducing the friction created by firewalls, other traffic, and switches and routers that are tuned for more diverse traffic.
The University of Arkansas via its Data Analytics that are Robust and Trusted (DART) project, is funded by NSF GRANT #194639 for EPSCoR RII.
The Jupyter Security Best Practices documentation produced by this engagement will be shared with Project Jupyter for inclusion in their documentation, and also presented at the workshop.
To read Jupyter's blog post about the engagement, click here.
Ransomware is a form of cybercrime that has risen to the same level of concern as terrorism by the U.S. Department of Justice. The United States suffered more than 65,000 ransomware attacks last year and victims paid $350 million in ransom, with an unknown amount of collateral costs due to lost productivity. Historically, research organizations have been largely ignored by cybercriminals since they do not typically have data that is easily sold or otherwise monetized. Unfortunately, since ransomware works by extorting payments from victims to get their own data back, research organizations are no longer immune to being targeted by criminals.
An event of this nature occurred in the Physics and Astronomy department at Michigan State University (MSU), which experienced a ransomware attack in May 2020. While many organizations attempt to keep the public from finding out about cyberattacks for fear of loss of reputation or follow-up attacks, MSU has decided to make elements and factors of its attack public in the interests of transparency, to encourage disclosure of similar types of attacks, and perhaps more importantly, to educate the open-science community about the threat of ransomware and other destructive types of cyberattacks. The overarching goal is to raise awareness about rising cybersecurity threats to higher education in hopes of driving safe cyberinfrastructure practices across university communities.
To achieve this, the CIO’s office at MSU engaged with Trusted CI, the NSF Cybersecurity Center of Excellence, in a collaborative review and analysis of the ransomware attack suffered by MSU last year. The culmination of the engagement—based on interviews of those involved in the incident—is the report “Research at Risk: Ransomware attack on Physics and Astronomy Case Study,” which focuses on lessons learned during the analysis. The report contains mitigation strategies that other researchers and their colleagues can apply to protect themselves. In the experience of Trusted CI, there was nothing extraordinary about the issues that led to this incident, and hence, we share these lessons with the goal of motivating other organizations to prevent future negative impacts to their research mission.
The engagement ran from January 2021 to July 2021.
The five-month engagement began in February and completed in June. In that time the teams worked together to review FABRIC’s project documentation, which included a deep analysis of the security architecture. We moved on to completing an asset inventory and risk assessment, covering over 70 project assets, identifying attack surfaces and potential threats, and documenting current and planned security controls. Lastly, we documented engagement findings in an internal report shared with FABRIC project leadership.
FABRIC also assisted with the Trusted CI 2021 Annual Challenge (Software Assurance) by participating in an interview with members of the software assurance team. The results of that interview will provide input to Trusted CI's forthcoming guide on software assurance for NSF projects.
Cybersecurity exploits are on the rise across university communities, costing valuable resources, and loss of productivity, research data, and personally identifiable information. In a DXC report, it was estimated that an average ransomware attack can take critical systems down for 16 days, and the overall worldwide cost of ransomware in 2020 was predicted to cost $170 billion. Additional reputational impacts of cybersecurity attacks, although hard to measure, regularly weigh in the minds of scientists and researchers.
An event of this nature occurred at Michigan State University (MSU), which experienced a ransomware attack in May 2020. While many organizations attempt to keep the public from finding out about cyberattacks for fear of loss of reputation or follow-up attacks, MSU has decided to make elements of its attack public in the interests of transparency, to encourage disclosure of similar types of attacks, and perhaps more importantly, to educate the open-science community about the threat of ransomware and other destructive types of cyberattacks. The overarching goal is to raise awareness about rising cybersecurity threats to higher education in hopes of driving safe cyberinfrastructure practices across university communities.
To achieve this, the CIO’s office at MSU has engaged with Trusted CI, the NSF Cybersecurity Center of Excellence, in a collaborative review and analysis of the ransomware attack suffered by MSU last year. The culmination of the engagement will be a report focusing on lessons learned during the analysis; these ‘Lessons Learned’ would then be disseminated to the research community. We expect the published report to be a clear guide for researchers and their colleagues who are security professionals to help identify, manage, and mitigate the risk of ransomware and other types of attacks.
The Geodetic Facility for the Advancement of Geoscience (GAGE), is operated by UNAVCO
and funded by the NSF and NASA. The GAGE project’s mission is to
provide support to the larger NSF investigator community for geodesy,
earth sciences research, education, and workforce development. During
the second half of 2020, GAGE and the Trusted CI/CI CoE Identity
Management working group collaborated on an engagement to design a
working proof of concept for integrating federated identity into GAGE’s
researcher data portal.
The Cyberinfrastructure Center of Excellence Pilot
(CI CoE) is a Trusted CI partner, specializing in providing expertise
and active support to CI practitioners at the NSF major facilities in
order to accelerate the data lifecycle and ensure the integrity and
effectiveness of the CI upon which research and discovery depends. The
Identity Management working group is a joint effort between the CI CoE
and Trusted CI to provide subject matter expertise and advice to major
facilities on trust and identity issues, best practices and
implementation. The working group's target audience is NSF funded major
facilities, but participation in the working group is open to anyone in
higher education and IAM.
The engagement began in July 2020 with a
month long series of interviews between working group members and GAGE
department leadership. GAGE came into the engagement with a series of
needs that had arisen from practice and with a request from NSF to
collect information on how their research data was being used. The
working group used the interviews to identify key systems and areas of
impact in order to present GAGE with a design for integrating federated
identity into their data portal using elements of InCommon’s Trusted Access Platform.
Over
the next three months, the engagement team met with members of GAGE’s
software development team, CILogon, and COmanage to finalize and
implement the proof of concept design. This design used CILogon to
consume federated identities from other InCommon member institutions and
then used COmanage registry to store GAGE specific attributes for those
identities to grant permission for accessing various data groups,
membership in research projects, and home institutions. Identities and
attributes stored in COmanage could then be passed to the GAGE data
portal using OIDC claim tokens; granting permissions appropriately at
the time of access and allowing GAGE to track which identities were
requesting what permissions for their data.
The engagement culminated with a 15-page report
delivered to GAGE in February 2021 containing detailed observations
from interviews, alternate design configurations and tools for the proof
of concept, lessons learned through the implementation process, and
identification of future opportunities for investment and collaboration
in IAM. Additionally, findings from this engagement will be included in
an IAM cookbook that the working group plans to release in 2022. The
Identity Management working group meets monthly on the second Monday at
2pm Eastern time. For more information about the Identity Management
working group, please see the Trusted CI IAM page, the CI CoE working group directory, or join our mailing list to receive updates on working group meetings and products.
GAGE is funded by an NSF award managed by the Division of Earth Sciences (Award #1724794) and is operated by UNAVCO.
The CI CoE Pilot is supported by a grant managed by the NSF Office of
Advanced Cyberinfrastructure (Award #1842042) and is a collaboration
between the University of Southern California, University of North
Carolina at Chapel Hill, University of Notre Dame, University of Utah,
and Indiana University. The working group would like to thank the
following institutions and organizations for the collaboration and
contributions to the engagement: Internet2 and InCommon, the CILogon
team, the COmanage team, and the Globus team.
The Partnership to Advance Throughput and Computing (PATh) is a project funded by NSF’s OAC Campus Cyberinfrastructure (CC*) program and brings together the Center for High Throughput Computing (CHTC) and the Open Science Grid (OSG) in order to advance the nation’s campuses and science communities through the use of distributed High Throughput Computing. The PATh project offers technologies and services that enable researchers to harness through a single interface, and from the comfort of their “home directory”, computing capacity offered by a global and diverse collection of resources.
PATh is collaborating with Trusted CI on adapting and rewriting PATh’s security program. Through a pre-kickoff meeting and their proposed security program plan submitted to the NSF, we have prioritized their needs using a subset of tasks to outline the goals of the engagement, specifically:
The engagement began in January 2021 and will run until the end of June 2021.
Trusted CI Engagement Application Period is
Open
Applications Due April 2, 2021
Apply for a one-in-one engagement with Trusted CI for early 2021
Trusted CI is
accepting applications for one-on-one engagements to be executed in July-Dec 2021.
Applications are due April 2, 2020 (Slots are limited and in demand, so this is
a hard deadline!)
To learn more about the process and criteria, and to complete the
application form, visit our site:
http://trustedci.org/application
During Trusted CI’s first 5 years, we’ve conducted more than 24 one-on-one engagements with NSF-funded projects,
Large Facilities, and major science service providers representing the full
range of NSF science missions. We support a variety of engagement types including:
assistance in developing, improving, or evaluating an information security
program; software assurance-focused efforts; identity management; technology or
architectural evaluation; training for staff; and more.
As the NSF Cybersecurity Center of Excellence,
Trusted CI’s mission is to provide the NSF community a coherent understanding
of cybersecurity’s role in producing trustworthy science and the information
and know-how required to achieve and maintain effective cybersecurity programs.
Open OnDemand is funded by NSF OAC and is an open-source HPC portal based on the Ohio Supercomputer Center original OnDemand portal. The goal of Open OnDemand is to provide an easy way for system administrators to provide web access to their HPC resources.
Open OnDemand is now facing increased community adoption. As a result, it is becoming a critical production service for many HPC centers and clients. By improving the overall security of the project, we will ensure that it continues to be a trusted and reliable platform for the hundreds of centers and tens of thousands of clients that regularly utilize it.
Open OnDemand has engaged with Trusted CI to support their efforts to further develop the project’s ability to produce secure software. Trusted CI previously conducted an in-depth vulnerability assessment applying the FPVA methodology to Open OnDemand software. The results of this prior assessment will help to inform the activities of this engagement. During the course of the prior FPVA assessment, Trusted CI staff worked directly to test Open OnDemand’s software to identify vulnerabilities with support from the Open OnDemand team. Trusted CI will now work with Open OnDemand to improve the project’s ability to maintain the security of their software as changes are made and to identify and mitigate future vulnerabilities.
Upon completion of the engagement, Trusted CI will produce a published report describing the work performed, potential impact to the open-science community, and areas Open OnDemand may find appropriate for future engagements.
