FOI 24/25-1356 - DISCLOSURE LOG DOCUMENT 4 OFFICIAL: SENSITIVE
EXECUTIVE BRIEF
Required Action: For Decision PDR: EC24-001866 Critical Date: N/A Div: ClO
EXTENSION OF THE MICROSOFT COPILOT TRIAL
- Purpose
1.1. To seek Chief Operating Officer (COO) approval to extend the Digital Transformation Agency (DTA) Microsoft Copilot (Copilot) trial through to July 2025 and continue to use Microsoft Copilot and safely engage on the usage of Generative Artificial Intelligence (Al) within the National Disability Insurance Agency (NDIA or Agency).
1.2. To update the COO on the DTA Microsoft Copilot trial to date, including learnings, use case and risks.
- Background
2.1. In November 2023, the NDIA joined a managed Copilot trial for Microsoft 365 (M365), coordinated by the DTA, as one of 30 agencies across government. This Copilot trial is a government-wide first, aimed at evaluating policy and employee use of Al tools within a secure framework.
2.2. The trial, costing redacted: s47G - business inform GST was inclusive for 300 licences to members of the
NDIA Employee Disability Network (EDN), took place until June 2024. 225 licences
went to NDIA staff, providing insights into Copilot’s functionality, security
compliance, and user experience.
2.3. Copilot, integrated into M365, functions as an Al admin assistant, is limited to files users are authorised to access and not reaching beyond the Microsoft suite.
2.4. Findings (Attachment A) from the completion of the Copilot trial indicate:
2.4.1. Support for SES in M365 applications: Users have seen significant timesaving benefits using Copilot to summarise information from the Microsoft suite of applications. This means users can search for key information and Copilot can provide a summary combining information from all applications.
2.4.2. Extensive use in M365Teams: Copilot is being utilised for minute taking, recapping meetings, and providing actions from meetings in Microsoft Teams.
2.4.3. Enhanced Accessibility across M365 applications: Copilot has significantly improved accessibility within M365 applications for employees with disabilities, enabling them to interact with technology more efficiently.
2.5. A survey of trial participants in the Agency conducted in May 2024 found that Copilot exceeded expectations in Word, Outlook, and PowerPoint. Copilot exceeded user expectations for 67% of trial members and 84% of users were now highly confident with the use of Al due to their participation in the trial.
- Extension of the Copilot Trial
3.1. The DTA Copilot trial found value in the Al tool for Agency users, noting there continues to be broad discussion regarding aspects such as value for money, integration into the operating environment and risk management.
OFFICIAL: SENSITIVE Extension of the Microsoft Copilot Trial 1 Page 68 of 189
FOI 24/25-1356 - DISCLOSURE LOG OFFICIAL: SENSITIVE
EXECUTIVE BRIEF
3.2. While the Agency seeks approval to continue the Copilot trial from July 2024 to June 2025, further work will be undertaken to understand the risk exposure that AI poses to the Agency and will evaluate key mitigation strategies that are in place on completion of the trial.
3.3. The majority of DTA trial participants across government have elected, where possible to continue using Copilot. Departments such as Home Affairs have a Protected network and are unable to continue the trial in this environment. The Agency doesn’t have this restriction and so is not limited by its operating environment and policy settings.
3.4. Continuation of the trial will allow further opportunities for the EDN to explore the accessibility benefits of using Copilot in their business-as-usual activities. This will provide further data on the use of Copilot as an accessibility assistant and provide opportunities for the Agency to work with Microsoft as an active partner in improving accessibility features.
3.5. An extension of the trial will be separate from the current DTA trial, which finished in
June 2024. Microsoft has quoted redacted: s47G - business inform per Copilot licence to allow a continuation of
the trial, with our current pool of 300 licences, equating to a cost of redacted: s47G - business inform
(GST Inc.) for 2024/25.
3.6. The extended trial at the NDIA will utilise all 300 licences by including most NDIA SES, and their Executive Assistants. This will expand the trial participants into more business areas, further testing the capability, value, and risks of Copilot.
- Risk Mitigation
Risks
4.1. There is a recognised potential for Copilot to generate inaccurate, misleading, or inappropriate content that may affect the quality and integrity of NDIA’s documents and communications.
4.2. The possible legal and ethical implications of using Copilot-generated content without proper verification, attribution, or consent, especially when dealing with personal or sensitive information of participants, staff, or stakeholders is problematic where use of the tool is unregulated, and users are not trained and supported in their use.
4.3. The continued development of a robust governance framework and accountability mechanism for the continued use of Copilot, including clear roles and responsibilities, escalation and approval processes, and feedback and evaluation mechanisms is essential.
4.4. Such a governance framework is further enhanced by ongoing communication and engagement with Copilot users to raise awareness of functions and risks inherent in the tool, whilst encouraging best practice and soliciting feedback on the benefits and challenges of using Copilot.
Copilot’s access and separation
4.5. Prior to the implementation the trial the Agency conducted a Cyber Security risk assessment (Authority to Operate) on the Microsoft suite of applications that incorporate Copilot. The DTA has also undertaken an assessment of Copilot, which was reviewed to identify risks that may impact the Agency. The Cyber Security risks of using Copilot are considered LOW due to the following:
OFFICIAL: SENSITIVE Extension of the Microsoft Copilot Trial 2 Page 69 of 189
FOI 24/25-1356 - DISCLOSURE LOG OFFICIAL: SENSITIVE
EXECUTIVE BRIEF
4.5.1. Copilot operates as a private tool, which is separated from other Microsoft customers and the public. Copilot only accesses information and data in the Microsoft suite of applications which the user has authorisation to access.
4.5.2. Sensitive NDIA systems and data (such as PACE, SAP CRM and Protected Enclave) cannot be accessed by Copilot.
4.5.3. The NDIA’s use of Copilot is designed to work within the boundaries of the NDIA ICT environment and does not access the wider internet.
4.5.4. Microsoft has made a commitment to the Australian Government that the use of Copilot will never train its large language model to gain insights and intelligence from Government processes and decision making.
Artificial Intelligence and Machine Learning Policy
4.6. Risk will continue to exist where a virtual office assistant is being used based on a Large Language Model of Artificial Intelligence (LLM). LLM is the deep learning algorithm in AI tools which summarise, translate and generate replicated human responses, ideas and concepts.
4.7. The Use of Artificial Intelligence, Generative AI, and Machine Learning – Interim Policy was implemented in April 2024 to address the inherent risks of using AI and machine learning within the Agency (Interim Policy)
4.8. The Interim Policy, which was developed in consultation with the DTA to support users of the technology and is accompanied by mandatory learning modules for users that address the ethical, legal and security risks (Attachment B).
4.9. The Interim Policy has been compared with the National Framework for the Assurance of Artificial Intelligence in Government published by the DTA on June 21, 2024, to determine if adjustments are necessary.
4.10. The review concluded that no modifications to the Interim Policy are needed at this time regarding Copilot usage. Another review is expected by December 2024 to include insights from the DTA’s Copilot survey conducted across all participants in the trial and the Privacy Assurance Advice provided to the Agency on August 8, 2024 (Attachment C).
Privacy and data protection measures
4.11. Some to the updates anticipated are:
4.11.1. Establishing clear and transparent criteria for the selection of Copilot users and ensuring that they are adequately trained and informed of their privacy obligations.
4.11.2. Implementing a Privacy Impact Assessment (PIA) process for any new use cases of Copilot that involve personal information or sensitive data.
4.12. Developing and enforcing clear guidelines and protocols for the use of Copilot, including the types of documents and data that can or cannot be accessed or generated by Copilot, the retention and deletion of Copilot outputs, and the reporting and resolution of any privacy incidents or breaches.
4.13. Conducting regular audits and reviews of Copilot’s performance, accuracy, and compliance with privacy laws and NDIA policies.
Risk management and governance
OFFICIAL: SENSITIVE Extension of the Microsoft Copilot Trial 3 Page 70 of 189
FOI 24/25-1356 - DISCLOSURE LOG OFFICIAL: SENSITIVE
EXECUTIVE BRIEF
4.14. The Agency’s AI Working Group collaborated with the Agency Risk, Audit and Resilience Division to undertake a risk assessment (Attachment D), consolidating existing risk mitigation strategies and to identify any emerging risks (Attachment E).
4.15. The project team will engage with the Agency Ethics Committee anticipated to be established in October 2024 through the Integrity Agencies and Ethics team in the Legal Services Branch. The Agency Ethics Committee will maintain a standing item on the use of AI and will establish a review mechanism for all new AI and machine learning technology proposals. The project team will provide updates in status reporting to that Committee.
4.16. The Agency will continue to consult with the DTA on the Copilot trial to support the adoption of government wide practice and policies guard railing the use of AI.
- Recommendation
5.1. The COO:
5.1.1. notes the preliminary findings of the Copilot trial and the evolving AI governance and policy.
5.1.2. approves the extension of the DTA Copilot trial through to July 2025.
5.1.3. notes that the project team will continue to engage on the ethical and responsible use of Copilot and inform our evolving governance and policy position on AI under consultation with the DTA.
- Authorisation
Approval in PDMS History
…………………………………….. Date: 08 July 2024
Natasha Murphy Prepared by: Carol redacted: s47F - personal privacy.
Branch Manager
Enterprise Architecture and Governance
Mobile: redacted: s47F - personal privacy Phone / Mobile redacted: s47F - personal privacy
OFFICIAL: SENSITIVE Extension of the Microsoft Copilot Trial 4 Page 71 of 189
FOI 24/25-1356 - DISCLOSURE LOG OFFICIAL: SENSITIVE
EXECUTIVE BRIEF
redacted: s47F - personal privacy
21 August 2024 Signature: ……………………. Date: ………………….. Samuel Porter Deputy CEO Enabling Services / Chief Operating Officer
COO Comment Cleared on the following basis:
-
Ethics committee consideration occurs asap and use thereafter is subject to the committee’s views
-
CIO and I taking the agency’s current policy on AI (at attachment B) through SLT and the ethics committee (in that order)
-
confirmation (in a separate brief) about the guardrails we’ve put around copilot use for EDN to ensure that it is separated from service delivery activities (in particular planning).
- Attachments:
Attachment A: End of Trial Report (DTA Evaluation)
Attachment B: Use of AI Interim Policy
Attachment C: Privacy Assurance Advice – Copilot for Microsoft 365 Project
Attachment D: AI Working Group Terms of Reference
Attachment E: Final Copilot Project Risk Register
OFFICIAL: SENSITIVE Extension of the Microsoft Copilot Trial 5 Page 72 of 189
End of Trial Report
Copilot for Microsoft 365
January — June 2024
Version: 1.0
Date: 17 June 2024
Author: Josiph redacted: s47F - personal privacy
Division: Office of the Chief Information Officer
ndis.gov.au
FOI 24/25-1356 - DISCLOSURE LOG OFFICIAL
End of Trial Report 1
Copilot for Microsoft 365 1
January – June 2024 1
- Executive summary 4
1.1 Background 4
1.2 Objectives of the trial 4
1.3 Purpose 5
-
Copilot at the NDIA 6
-
Implementation of Copilot for Microsoft 365 8
3.1 Approach 8
3.2 Planning and Configuration 8
3.3 Training and Support 9
3.4 Implementation and Monitoring 9
- Results 10
4.1 Using Copilot 10
4.2 Accessibility 11
Neurodivergent User 12
JAWS User 12
Dyslexic User 12
Hearing Impaired User 12
4.3 Ongoing Training, User Monitoring, and Guidance 13
4.4 Network Integration 13
-
Summary of Benefits 14
-
Key Findings 16
6.1 User Productivity Error! Bookmark not defined.
6.2 User Satisfaction Error! Bookmark not defined.
6.3 Accessibility Usage Error! Bookmark not defined.
6.4 Stakeholder Engagement Error! Bookmark not defined.
- Perceived Risks and Mitigation Strategies 17
ndis.gov.au 17 June 2024 | Copilot – End of Trial Report 2
Page 74 of 189 OFFICIAL
FOI 24/25-1356 - DISCLOSURE LOG OFFICIAL
- Controls for Mitigating Risks During the Trial 19
8.1 Data Privacy Controls 19
8.2 User Training and Support 19
8.3 Ethical Guidelines 19
-
Summary of Recommendations 20
-
Conclusion 20
ndis.gov.au 17 June 2024 | Copilot – End of Trial Report 3
Page 75 of 189 OFFICIAL
FOI 24/25-1356 - DISCLOSURE LOG OFFICIAL
- Executive summary
1.1 Background
In the wake of rapid advancements in Artificial Intelligence (AI) technologies, the Agency recognises the potential of AI to transform public service delivery, enhance operational efficiency, and foster innovation.
The Australian Public Service (APS) commenced a six-month trial of Copilot for Microsoft 365 in January 2024 to test new ways to enhance productivity and develop skills, capabilities and preparedness for the use of generative AI tools by APS staff. The trial was coordinated by the Digital Transformation Agency (DTA) and involved over 50 participating Commonwealth agencies.
The Agency is participating in the trial alongside numerous Australian Government Agencies and Departments and obtained 300 licenses. The trial started as a Proof of Concept (POC) for four months, and then continued as a Trial for the remaining trial period. A smaller group of users were chosen for the er group of users were chosen for the POC, more users were included for the Trial. Staff who participated were urged to use Copilot to help with their daily administrative tasks.
Over the six-month trial period, data was collected and analysed to assess security posture, user productivity, satisfaction, and accessibility improvements.
1.2 Objectives of the trial
- Evaluate effectiveness: Assess staff using Copilot impacts productivity and user satisfaction.
- Determine value for money: determine whether future use by all NDIA staff represents value for money.
- Ongoing AI risk mitigation: Assess if having an internal AI product like Copilot reduces the likelihood of staff accessing external/unapproved AI services, and the corresponding threat to Agency data that arises from unapproved use.
- Policy development: Use the trial to develop a policy position and unified approach to the use of Artificial Intelligence within ICT services.
- Accessibility impact: Measure the accessibility improvements for users requiring support.
- Gather data for informed decisions: Collect quantitative and qualitative data to inform future AI tool implementations.
ndis.gov.au 17 June 2024 | Copilot – End of Trial Report 4
Page 76 of 189 OFFICIAL
FOI 24/25-1356 - DISCLOSURE LOG OFFICIAL
1.3 Purpose
This purpose of this report is to:
- Describe the implementation of Copilot for Microsoft 365 at the Agency
- Illustrate use cases of Copilot at the Agency
- Evaluate the effectiveness of Copilot for Microsoft 365 on productivity and accessibility within the Agency.
ndis.gov.au 17 June 2024 | Copilot – End of Trial Report 5
Page 77 of 189 OFFICIAL
FOI 24/25-1356 - DISCLOSURE LOG OFFICIAL
- Copilot at the NDIA
A description of how Copilot can assist NDIA staff with their daily tasks is tabled below.
| Use | Description |
|---|---|
| Ensuring policy compliance | Assistance with preparing documents and emails and aligning with NDIA policies and guidelines by: |
- interpreting NDIA policies and generating a summary of the the purpose
- identifying amendments to a policy document and the responsible person | | Meeting scheduling | Quickly schedule and reschedule meetings:
- without manually comparing participants’ calendars for availability, and
- creating meeting invites, adding participants and setting the meeting date and time based on availability | | Checking document accessibility | Improve the accessibility of documents by providing suggestions for:
- readability and simplicity
- formatting
- complying with the NDIA’s accessibility standards
- colour contrast for colour vision deficiencies
- converting into structured PDF or plain text for accessibility
- accessibility best practices | | Email summary and creation | Assist users with:
- summarising key points and decisions from large email chains
- drafting email responses
- managing email inboxes (eg flagging important emails) | | Assisting with data analysis | Assist users with:
- creating complex formulas in Excel spreadsheets
- generating charts and graphs to visualise data
- interpretating numbers of trends |
ndis.gov.au 17 June 2024 | Copilot – End of Trial Report 6
Page 78 of 189 OFFICIAL
FOI 24/25-1356 - DISCLOSURE LOG OFFICIAL
| Use | Description |
|---|---|
| - sorting and filtering data |
- reminding users to keep data up to date | | Providing in-context training | Assist user with learning Microsoft 365:
- navigating tools and processes by providing step-by-step instructions
- helping learn more about advanced features | | Document summarisation and reading aloud | Assist visually impaired staff with reviewing extensive documents and reports by:
- providing summaries of Word documents with the key points, decisions and action items; and
- text-to-speech functionality. | | Presentation creation assistance | Assist staff with creating engaging and informative presentations by:
- suggesting a slide structure from an outline of the presentation
- using the appropriate NDIA templates
- retrieving and including relevant statistics, images and case studies from NDIA data
- assisting with presentation rehearsals
- assisting with managing live Q&A sessions by quickly providing information which answers the audience’s questions | | Accessible data analysis and reporting | Assist visually impaired users interpret complex data sets, charts and graphs by:
- interpreting data and providing simple and understandable insights in a format which is understandable without visual aids
- analysing trends, summarising data points and predicting future patterns based on historical data
- creating accessible reports through the translation of the data analysis into descriptive narratives in a format which is inclusive for all
- allowing reports to be easily shared through Microsoft 365 |
ndis.gov.au 17 June 2024 | Copilot – End of Trial Report 7
Page 79 of 189 OFFICIAL
FOI 24/25-1356 - DISCLOSURE LOG OFFICIAL
| Use | Description |
|---|---|
| - ensure documents created are compliant with accessibility standards and easy to navigate | |
| Accessible project management | Assist hearing impaired users with managing projects by: |
- enabling real-time transcription during Teams meetings so the user receives live captions and allow for confirmation of any discussion points which may have been missed
- generating notes, list tasks and assist the user in other ways during the meeting
- organising project tasks and timelines, the user can provide Copilot verbal instructions to update task statuses, add new assignments, and set reminders for deadlines. |
- Implementation of Copilot for Microsoft 365
3.1 Approach
The trial faced difficulties due to:
- scepticism about the usefulness of generative AI tools such as Copilot
- concerns for the security and protection of Agency data Copilot would access
- the perception that the trial was being conducted to evaluate it’s effect on productivity as part of a plan to reduce staff numbers
- findings of the recent “Robodebt” Royal Commission
To mitigate, the Copilot trial was reduced to a Proof of Concept (POC) for the initial four months of the trial. This provided the additional time required to carefully examine and mitigate risks before conducting the trial. Add some points in here about what was scaled down for the POC. A communication ban was enforced that impacted the ability to communicate broadly.
The POC was extended to a full trial in May 2024. Add some points in here about what was scaled up for the Trial. What happened to the communication ban?
3.2 Planning and Configuration
ndis.gov.au 17 June 2024 | Copilot – End of Trial Report 8
Page 80 of 189 OFFICIAL
FOI 24/25-1356 - DISCLOSURE LOG OFFICIAL
- Planning phase: Detailed planning sessions were held to define the scope, objectives, and success criteria for the trial.
- Infrastructure configuration: Necessary IT infrastructure and security measures were put in place to support the deployment of Copilot.
- Agency AI Working Group formation: Approximately 10 executives across different NDIA Business Areas, including staff requiring accessibility support.
- POC Group selection: Select up to 300 NDIA staff across different Groups, Divisions and Sections, including users requiring accessibility support.
- Trial Group selection: Select up to 300 NDIA staff across different Groups, Divisions and Sections, including users requiring accessibility support.
3.3 Training and Support
- Information sessions: Comprehensive training sessions were conducted to ensure users were familiar with Copilot features and functionalities.
- Learning modules: A mandatory learning module ensured that all users were taken through the ethical and professional obilgations of using AI.
- Support channels: Dedicated support channels were established, including a helpdesk, FAQs, and user guides.
3.4 Implementation and Monitoring
- Phased rollout: The implementation was carried out in phases to manage any potential issues effectively.
- Monitoring and feedback: Continuous monitoring of usage and performance, along with regular feedback collection from users, was conducted to identify areas for improvement.
ndis.gov.au 17 June 2024 | Copilot – End of Trial Report 9
Page 81 of 189 OFFICIAL
FOI 24/25-1356 - DISCLOSURE LOG OFFICIAL
- Results
4.1 Using Copilot
The following benefits were demonstrated by users using Copilot, and could also be realised by users across all Australian Government Departments:
| Benefit | Copilot Use |
|---|---|
| Enhanced productivity | Copilot assisted users achieve their tasks faster. For example, capturing summaries of meetings, assisting with summarising and prioritising email trails, assisting with scheduling meetings. This outcome indicates significant improvements to staff productivity. |
| Improved accessibility | Copilot enhanced accessibility support by providing guidance to enhance the accessibility of document content; interpreting complex text, graphs and analytical data; providing assistance during and after meetings through transcripts and summaries; and other capabilities that increased a staff member’s independence to complete tasks. This improved engagement and productivity of staff relying on accessibility tools. |
| Data-driven decision making | Copilot has the ability to search through NDIA data for relevant statistics, images, and case studies. This feature ensures that the content is accurate and visually appealing, aiding in informed decision-making. |
| Presentation preparation and delivery | Using Copilot’s assistance to prepare and rehearse presentations was a key finding. Copilot can generate slides with suggested titles and content, provide a script based on the slide content, and offer feedback on delivery, which analyses pacing, clarity, and tone. |
| Streamlining administrative activities | All use cases indicate that Copilot will streamline routine administrative activities, such as writing briefs and initial drafts of correspondence. This has the potential to provide |
ndis.gov.au 17 June 2024 | Copilot – End of Trial Report 10
Page 82 of 189 OFFICIAL
FOI 24/25-1356 - DISCLOSURE LOG OFFICIAL
| Benefit | Copilot Use |
|---|---|
| significant time savings to administrative staff within the Agency. |
4.2 Accessibility
The participation of Employee Disability Network (EDN) members in the NDIA trial provided valuable insights into Copilot’s impact on their work. The tool received overwhelmingly positive feedback, with notable benefits.
| User Group | Copilot Use |
|---|---|
| Neurodivergent Users | Copilot significantly improved task management, communication, and focus, offering tailored suggestions and automating routine tasks. |
| JAWS Users | Copilot’s integration with JAWS screen readers enhanced accessibility, making information more accessible and interactions more efficient.The ability to use Copilot to recap meetings allowed visually impaired users the freedom to participate fully in meetings rather than concentrating to ensure that transcript matched the tone of the discussion. |
| Hearing Impaired Users | Copilot’s application within Microsoft Teams was the biggest gamechanger for this group of users. The recap function, the live transcript in meetings, and the ability to ask Copilot for clarification when the conversation moved too quickly were revolutionary for hearing impaired users. |
| Users with Dyslexia | The ability to draft content from simple prompts or rewrite written content allowed users with Dyslexia the comfort of knowing that their work was correct, raising their confidence and shortening editing and review time across the board. |
ndis.gov.au 17 June 2024 | Copilot – End of Trial Report 11
Page 83 of 189 OFFICIAL
FOI 24/25-1356 - DISCLOSURE LOG OFFICIAL
4.2.1 Feedback
Neurodivergent User
Copilot has been a game-changer for me. It helps me stay organised and focused by providing real-time suggestions and automating repetitive tasks, reducing cognitive load and allowing me to concentrate on more complex work.
JAWS User
Using Copilot with JAWS has revolutionised how I interact with digital content. The AI assists in navigating complex documents and emails, making my workflow much smoother and more efficient.
Dyslexic User
Copilot has dramatically improved my experience with written communication. It helps me draft emails and documents by suggesting correct spelling, grammar, and even phrasing, reducing the time and stress associated with writing and ensures my communications are clear and professional. In Microsoft Teams, Copilot summarises long messages and threads, making it easier for me to understand and respond accurately.
Hearing Impaired User
Copilot has greatly enhanced my ability to participate in and manage digital communications. In Microsoft Teams, Copilot provides real-time transcription of audio and video meetings, allowing me to follow conversations more accurately and engage more effectively. This feature eliminates the need for separate transcription services and reduces delays in understanding meeting content. Additionally, Copilot’s ability to summarise key points and generate meeting notes ensures I never miss crucial information. In Outlook, Copilot allows me to access information in a way that suits my needs. Overall, Copilot has made my digital interactions more inclusive and efficient, significantly improving my work experience here in the Agency.
ndis.gov.au 17 June 2024 | Copilot — End of Trial Report 12
Page 84 of 189 OFFICIAL
FOI 24/25-1356 - DISCLOSURE LOG OFFICIAL
4.3 Ongoing Training, User Monitoring, and Guidance
Feedback from users indicated that Copilot made it very easy to slip into usage patterns outside of the trial and interim policy parameters. Ongoing education around the ethical use of AI would be required to ensure that users complied with data protection expectations.
| Training and Support | All use cases indicated that the implementation of any AI tool required ongoing and comprehensive training sessions. A new tool like Copilot also required dedicated support channels, including a helpdesk, FAQs, and user guides. |
|---|---|
| Ethical Use and Data Privacy | The establishment of clear guidelines, transparency, and accountability measures to monitor the ethical use of AI tools, along with strict data privacy controls, are critical use cases that demonstrate responsible AI implementation. |
4.4 Network Integration
| Microsoft Managed and Device (MMD) program | The Agency’s (MMD) program, coupled with its greenfield relationship with Microsoft, facilitated a smooth and efficient activation of Copilot.
This advanced setup allowed for quick deployment and reduced downtime, ensuring minimal disruption to ongoing operations. | | — | — | | Identity Management | Identity management functions were configured effectively, allowing users to access data and information strictly within Identity and Access Management (IDAM) rules.
This setup not only enhanced security but also ensured compliance with internal data governance policies, providing a robust framework for secure and efficient use of Copilot across the Agency. |
ndis.gov.au 17 June 2024 | Copilot – End of Trial Report 13
Page 85 of 189 OFFICIAL
FOI 24/25-1356 - DISCLOSURE LOG OFFICIAL
- Summary of Benefits
The following benefits were realised during the POC and trial of Copilot:
| Benefit | Description |
|---|---|
| Enhanced Collaboration | Copilot facilitates better collaboration by summarising conversations, providing real-time suggestions, and enabling seamless sharing of information, thus enhancing team productivity. |
| Process Optimisation | By automating routine tasks and providing intelligent suggestions, Copilot helps streamline workflows, reducing manual effort and minimising errors. |
| Time efficiencies | Copilot saves time by automating repetitive tasks, drafting emails, and summarising lengthy documents and conversation threads, allowing users to focus on more complex tasks. |
| Skill Development | Copilot assists users in learning and improving their skills by providing context-aware tips and recommendations, thus enhancing their overall proficiency in using Microsoft 365 tools. |
| Accesibility Improvements | Copilot’s accessibility features significantly improved the experience for all users with disabilities within the trial. |
| Cyber Security Mitigations | By providing an in-house AI tool like Copilot, the Agency mitigated the risk of users seeking potentially insecure, web-based AI tools, ensuring Agency data remains within the secure government ecosystem. |
| Improved Decision Making | Copilot supports better decision-making by providing data-driven insights, analysing trends, |
ndis.gov.au 17 June 2024 | Copilot – End of Trial Report 14
Page 86 of 189 OFFICIAL
FOI 24/25-1356 - DISCLOSURE LOG OFFICIAL
| Benefit | Description |
|---|---|
| and generating reports, enabling users to make informed and timely decisions. | |
| Enhanced User Satisfaction | The intuitive interface and helpful suggestions from Copilot lead to higher user satisfaction and engagement, contributing to a more positive work environment. |
| Reduced Workload | By automating repetitive and time-consuming tasks, Copilot reduces the overall workload on employees, helping them manage their time more effectively and reduce burnout. |
| Consistency in Work Output | Copilot ensures consistency in documents and communications by providing standard templates and suggesting improvements, leading to a more uniform and professional output. |
ndis.gov.au 17 June 2024 | Copilot – End of Trial Report 15
Page 87 of 189 OFFICIAL
FOI 24/25-1356 - DISCLOSURE LOG OFFICIAL
- Key Findings
ndis.gov.au 17 June 2024 | Copilot – End of Trial Report 16
Page 88 of 189 OFFICIAL
FOI 24/25-1356 - DISCLOSURE LOG OFFICIAL
- Perceived Risks and Mitigation Strategies
| Risk Category | Risk Description | Likelihood | Impact | Mitigation Strategy |
|---|---|---|---|---|
| Data Privacy External | Potential risks of data breaches or misuse of sensitive information by members | Medium | High | Strict adherence to data privacy regulations, regular security audits, and user awareness training. |
| Data Privacy Internal | Risks of accidental data exposure or unauthorised access by internal users | Medium | High | Implementation of robust access controls, continuous monitoring of data access and usage, regular internal audits, and comprehensive training for employees on data handling practices. |
| Integration Issues | Technical challenges in integrating Copilot with existing systems | Low | Low | Thorough testing, phased rollout, and collaboration with Microsoft for troubleshooting. |
| User Resistance | Resistance to adopting new AI tools | High | Medium | Comprehensive training, ongoing support, and communication plan that |
ndis.gov.au 17 June 2024 | Copilot – End of Trial Report 17
Page 89 of 189 OFFICIAL
FOI 24/25-1356 - DISCLOSURE LOG OFFICIAL
| Risk Category | Risk Description | Likelihood | Impact | Mitigation Strategy |
|---|---|---|---|---|
| highlight benefits and lowers expectations | ||||
| Dependence on AI | Over-reliance on AI tools might reduce critical thinking skills | Low | Low | Encouraging balanced use, where Copilot is seen as a support to repetitive tasks and not a replacement of human judgment. |
| Ethical Concerns | Ethical implications of using AI in decision-making processes | High | High | Preparation of the Interim Use of AI Policy and establishing clear supporting guidelines and ensuring transparency in how the Agency assesses AI. |
ndis.gov.au 17 June 2024 | Copilot – End of Trial Report 18
Page 90 of 189 OFFICIAL
FOI 24/25-1356 - DISCLOSURE LOG OFFICIAL
- Controls for Mitigating Risks During the Trial
8.1 Data Privacy Controls
Encryption: All sensitive data was encrypted both in transit and at rest.
Access Controls: Strict access controls were implemented to ensure only authorised personnel could access sensitive information.
Regular Audits: Periodic security audits were conducted to identify and mitigate potential vulnerabilities.
8.2 User Training and Support
Comprehensive Training: Detailed training programs were conducted to ensure users were well-versed in using Copilot.
Continuous Support: Ongoing support was provided through helpdesks, user guides, and FAQs to address any user concerns promptly.
8.3 Ethical Guidelines
Transparency: Clear guidelines were established to ensure transparency in how AI tools were used within the organisation.
Accountability: Accountability measures were put in place to monitor the ethical use of AI tools.
ndis.gov.au 17 June 2024 | Copilot – End of Trial Report 19
Page 91 of 189 OFFICIAL
FOI 24/25-1356 - DISCLOSURE LOG OFFICIAL
- Summary of Recommendations
Extend the Trial: Extend the trial for an additional 12 months to gather more data and further evaluate the long-term impact.
Expand user base: Include more users, particularly those requiring accessibility support, to get a broader understanding of the tool’s impact.
Enhance training: Continue to provide comprehensive training and support to maximise user engagement and effectiveness.
Strengthen partnerships: Foster stronger relationships with Microsoft and other AI tool providers to ensure continuous improvements and support.
Establish clear guidelines: Develop and implement clear guidelines for the ethical use of AI tools within the organisation.
- Conclusion
The Copilot for Microsoft 365 trial has shown significant potential in enhancing productivity, user satisfaction, and accessibility within the NDIA.
The six-month trial of Copilot for Microsoft 365 demonstrated significant benefits for the Agency, particularly for employees with accessibility needs. The tool’s positive impact on neurodivergent users and those relying on JAWS underscores its potential as an inclusive productivity aid.
Additionally, robust data security measures ensured the protection of Agency and participant data throughout the trial.
Extending the trial period and expanding the user base will provide more comprehensive data, allowing the agency to make informed decisions about the long-term adoption of AI tools.
The controls and mitigation strategies implemented during the trial have effectively managed risks, and the lessons learned will guide future initiatives to further leverage AI tools for organisational benefits.
Based on these findings, it is recommended that the Agency consider wider adoption of Copilot to further enhance productivity and accessibility for all employees.
ndis.gov.au 17 June 2024 | Copilot – End of Trial Report 20
Page 92 of 189 OFFICIAL