EXTENSION OF THE MICROSOFT COPILOT TRIAL

FOI 24/25-1345 - DISCLOSURE LOG

OFFICIAL: SENSITIVE

DOCUMENT 3

EXECUTIVE BRIEF

Required Action: For Decision

PDR: EC24-001866
Div: CIO

Critical Date: N/A

1. Purpose

1.1. To seek Chief Operating Officer (COO) approval to extend the Digital Transformation Agency (DTA) Microsoft Copilot (Copilot) trial through to July 2025 and continue to use Microsoft Copilot and safely engage on the usage of Generative Artificial Intelligence (AI) within the National Disability Insurance Agency (NDIA or Agency).

1.2. To update the COO on the DTA Microsoft Copilot trial to date, including learnings, use case and risks.

2. Background

2.1. In November 2023, the NDIA joined a managed Copilot trial for Microsoft 365 (M365), coordinated by the DTA, as one of 30 agencies across government. This Copilot trial is a government-wide first, aimed at evaluating policy and employee use of AI tools within a secure framework.

2.2. The trial, costing redacted: s47G - business inform GST was inclusive for 300 licences to members of the NDIA Employee Disability Network (EDN), took place until June 2024. 225 licences went to NDIA staff, providing insights into Copilot’s functionality, security compliance, and user experience.

2.3. Copilot, integrated into M365, functions as an AI admin assistant, is limited to files users are authorised to access and not reaching beyond the Microsoft suite.

2.4. Findings (Attachment A) from the completion of the Copilot trial indicate:

2.4.1. Support for SES in M365 applications: Users have seen significant timesaving benefits using Copilot to summarise information from the Microsoft suite of applications. This means users can search for key information and Copilot can provide a summary combining information from all applications.

2.4.2. Extensive use in M365Teams: Copilot is being utilised for minute taking, recapping meetings, and providing actions from meetings in Microsoft Teams.

2.4.3. Enhanced Accessibility across M365 applications: Copilot has significantly improved accessibility within M365 applications for employees with disabilities, enabling them to interact with technology more efficiently.

2.5. A survey of trial participants in the Agency conducted in May 2024 found that Copilot exceeded expectations in Word, Outlook, and PowerPoint. Copilot exceeded user expectations for 67% of trial members and 84% of users were now highly confident with the use of AI due to their participation in the trial.

3. Extension of the Copilot Trial

3.1. The DTA Copilot trial found value in the AI tool for Agency users, noting there continues to be broad discussion regarding aspects such as value for money, integration into the operating environment and risk management.

FOI 24/25-1345 - DISCLOSURE LOG

OFFICIAL: SENSITIVE

EXECUTIVE BRIEF

3.2. While the Agency seeks approval to continue the Copilot trial from July 2024 to June 2025, further work will be undertaken to understand the risk exposure that AI poses to the Agency and will evaluate key mitigation strategies that are in place on completion of the trial.

3.3. The majority of DTA trial participants across government have elected, where possible to continue using Copilot. Departments such as Home Affairs have a Protected network and are unable to continue the trial in this environment. The Agency doesn’t have this restriction and so is not limited by its operating environment and policy settings.

3.4. Continuation of the trial will allow further opportunities for the EDN to explore the accessibility benefits of using Copilot in their business-as-usual activities. This will provide further data on the use of Copilot as an accessibility assistant and provide opportunities for the Agency to work with Microsoft as an active partner in improving accessibility features.

3.5. An extension of the trial will be separate from the current DTA trial, which finished in June 2024. Microsoft has quoted redacted: s47G - business per Copilot licence to allow a continuation of the trial, with our current pool of 300 licences, equating to a cost of redacted: s47G - business inform (GST Inc.) for 2024/25.

3.6. The extended trial at the NDIA will utilise all 300 licences by including most NDIA SES, and their Executive Assistants. This will expand the trial participants into more business areas, further testing the capability, value, and risks of Copilot.

4. Risk Mitigation

Risks

4.1. There is a recognised potential for Copilot to generate inaccurate, misleading, or inappropriate content that may affect the quality and integrity of NDIA’s documents and communications.

4.2. The possible legal and ethical implications of using Copilot-generated content without proper verification, attribution, or consent, especially when dealing with personal or sensitive information of participants, staff, or stakeholders is problematic where use of the tool is unregulated, and users are not trained and supported in their use.

4.3. The continued development of a robust governance framework and accountability mechanism for the continued use of Copilot, including clear roles and responsibilities, escalation and approval processes, and feedback and evaluation mechanisms is essential.

4.4. Such a governance framework is further enhanced by ongoing communication and engagement with Copilot users to raise awareness of functions and risks inherent in the tool, whilst encouraging best practice and soliciting feedback on the benefits and challenges of using Copilot.

Copilot’s access and separation

4.5. Prior to the implementation the trial the Agency conducted a Cyber Security risk assessment (Authority to Operate) on the Microsoft suite of applications that incorporate Copilot. The DTA has also undertaken an assessment of Copilot, which was reviewed to identify risks that may impact the Agency. The Cyber Security risks of using Copilot are considered LOW due to the following:

FOI 24/25-1345 - DISCLOSURE LOG

OFFICIAL: SENSITIVE

EXECUTIVE BRIEF

4.5.1. Copilot operates as a private tool, which is separated from other Microsoft customers and the public. Copilot only accesses information and data in the Microsoft suite of applications which the user has authorisation to access.

4.5.2. Sensitive NDIA systems and data (such as PACE, SAP CRM and Protected Enclave) cannot be accessed by Copilot.

4.5.3. The NDIA’s use of Copilot is designed to work within the boundaries of the NDIA ICT environment and does not access the wider internet.

4.5.4. Microsoft has made a commitment to the Australian Government that the use of Copilot will never train its large language model to gain insights and intelligence from Government processes and decision making.

Artificial Intelligence and Machine Learning Policy

4.6. Risk will continue to exist where a virtual office assistant is being used based on a Large Language Model of Artificial Intelligence (LLM). LLM is the deep learning algorithm in AI tools which summarise, translate and generate replicated human responses, ideas and concepts.

4.7. The Use of Artificial Intelligence, Generative AI, and Machine Learning – Interim Policy was implemented in April 2024 to address the inherent risks of using AI and machine learning within the Agency (Interim Policy)

4.8. The Interim Policy, which was developed in consultation with the DTA to support users of the technology and is accompanied by mandatory learning modules for users that address the ethical, legal and security risks (Attachment B).

4.9. The Interim Policy has been compared with the National Framework for the Assurance of Artificial Intelligence in Government published by the DTA on June 21, 2024, to determine if adjustments are necessary.

4.10. The review concluded that no modifications to the Interim Policy are needed at this time regarding Copilot usage. Another review is expected by December 2024 to include insights from the DTA’s Copilot survey conducted across all participants in the trial and the Privacy Assurance Advice provided to the Agency on August 8, 2024 (Attachment C).

Privacy and data protection measures

4.11. Some to the updates anticipated are:

4.11.1. Establishing clear and transparent criteria for the selection of Copilot users and ensuring that they are adequately trained and informed of their privacy obligations.

4.11.2. Implementing a Privacy Impact Assessment (PIA) process for any new use cases of Copilot that involve personal information or sensitive data.

4.12. Developing and enforcing clear guidelines and protocols for the use of Copilot, including the types of documents and data that can or cannot be accessed or generated by Copilot, the retention and deletion of Copilot outputs, and the reporting and resolution of any privacy incidents or breaches.

4.13. Conducting regular audits and reviews of Copilot’s performance, accuracy, and compliance with privacy laws and NDIA policies.

Risk management and governance

FOI 24/25-1345 - DISCLOSURE LOG

OFFICIAL: SENSITIVE

EXECUTIVE BRIEF

4.14. The Agency’s AI Working Group collaborated with the Agency Risk, Audit and Resilience Division to undertake a risk assessment (Attachment D), consolidating existing risk mitigation strategies and to identify any emerging risks (Attachment E).

4.15. The project team will engage with the Agency Ethics Committee anticipated to be established in October 2024 through the Integrity Agencies and Ethics team in the Legal Services Branch. The Agency Ethics Committee will maintain a standing item on the use of AI and will establish a review mechanism for all new AI and machine learning technology proposals. The project team will provide updates in status reporting to that Committee.

4.16. The Agency will continue to consult with the DTA on the Copilot trial to support the adoption of government wide practice and policies guard railing the use of AI.

5. Recommendation

5.1. The COO:

5.1.1. notes the preliminary findings of the Copilot trial and the evolving AI governance and policy.

5.1.2. approves the extension of the DTA Copilot trial through to July 2025.

5.1.3. notes that the project team will continue to engage on the ethical and responsible use of Copilot and inform our evolving governance and policy position on AI under consultation with the DTA.

6. Authorisation

Approval in PDMS History

…………………………………………………..

Date: 08 July 2024

Natasha Murphy
Branch Manager
Enterprise Architecture and Governance
Mobile: redacted: s47F - personal privacy

Prepared by: Carol redacted: s47F - personal privacy
Phone / Mobile: redacted: s47F - personal privacy

FOI 24/25-1345 - DISCLOSURE LOG

OFFICIAL: SENSITIVE

EXECUTIVE BRIEF

redacted: s47F - personal privacy

21 August 2024
Signature: …………………………………………………..
Date: …………………………………………………..
Samuel Porter
Deputy CEO Enabling Services / Chief Operating Officer

COO Comment

Cleared on the following basis:

Ethics committee consideration occurs asap and use thereafter is subject to the committee’s views
CIO and I taking the agency’s current policy on AI (at attachment B) through SLT and the ethics committee (in that order)
confirmation (in a separate brief) about the guardrails we’ve put around copilot use for EDN to ensure that it is separated from service delivery activities (in particular planning).

7. Attachments:

Attachment A: End of Trial Report (DTA Evaluation)

Attachment B: Use of AI Interim Policy

Attachment C: Privacy Assurance Advice — Copilot for Microsoft 365 Project

Attachment D: AI Working Group Terms of Reference

Attachment E: Final Copilot Project Risk Register