OFFICIAL DOCUMENT 4
Delivered by the National Disability Insurance Agency
Copilot for Microsoft 365 trial Factsheet
The Australian Government has announced a 6-month trial of Copilot for Microsoft 365 across the Australian Public Service (APS) commencing early 2024 to explore the safe, responsible and innovative use of generative Artificial Intelligence (AI).
About the trial
The trial will be administered by the Digital Transformation Agency (DTA) and will run from January to June 2024. The DTA is working with Microsoft to arrange training, onboarding, and implementation assistance with participating agencies. The DTA, in partnership with the AI in Government Taskforce, will establish whole- of government guardrails and security requirements, oversee the trial, and coordinate evaluation. This includes assisting agencies to work through the various considerations of trialling Copilot for Microsoft 365 for their circumstances, risk tolerances and potential use cases.
Participating agencies will be required to partake in a program of activities and contribute to trial deliverables including induction and training, reporting against evaluation measures and development of case studies.
The DTA in partnership with the AI in Government Taskforce will evaluate the experience of users to assess opportunities, feedback, and report to government.
Security of the trial
Important security considerations
The DTA is consulting with the Australian Cyber Security Centre to understand the security requirements. An Infosec Registered Assessors Program (IRAP) assessment of Copilot for Microsoft 365 is currently underway and is planned for completion prior to the commencement of the trial.
The NDIA is currently undertaking our own security risk assessment in line with our own tolerance and circumstances.
Copilot for Microsoft 365 is a part of the Microsoft 365 family of products, which means it’s built into Microsoft 365 and inherits all the same security, compliance, and privacy controls that underpin Microsoft 365.
For more information, see Data, Privacy, and Security for Copilot for Microsoft 365.
The trial was open to all agencies licensed for Microsoft 365 E3 or E5 on the Whole of Government Microsoft Volume Sourcing Arrangement (VSA).
The DTA has placed a centralised order on behalf of the participating agencies for a six month period. Following the placement of the order, licences were transferred to
PQ@SEFIC FAIS
the primary production tenancy of participating agencies which has been available for deployment since January 2024.
What were the Minimum requirements for trial participation?
The terms and minimum commitment for trial participation are an Agency commitment of:
- 300 Copilot for Microsoft 365 licences from agencies with 2,000 Microsoft 365 Qualified Users or more; or
- The greater of 15% of users or 10 Copilot for Microsoft 365 licences for agencies with fewer than 2,000 Microsoft 365 Qualified Users
- Commitment for the term of trial.
The overall benefit from Copilot for Microsoft 365 will be impacted by how heavily invested agencies are in their usage of Microsoft 365.
Agencies may opt to extend the term of their Copilot for Microsoft 365 licences after the trial for a further 12 months. The DTA will be contacting agencies with more information about extending their commitments as part of the annual licence reconciliation process.
Key steps and timeline
PageOFFICIAL13 of 76
Training and Support
- The DTA and Microsoft will be working with participating agencies to support them, through onboarding and readiness workshops, and additional training.
- Readiness workshops and training will be at no extra cost to agencies.
- An information pack containing FAQs addressing whole-of-government guardrails, possible use cases, evaluation approach and other information about the trial will be provided to trial participants early in the trial process.
Useful information
The following resources have been developed to assist implementing AI in the APS
Interim guidance on government use of public generative AI tools | aga (digital.gov.au)
Australia’s AI Ethics Principles | Australia’s Artificial Intelligence Ethics Framework - Department of Industry, Science and Resources
How might artificial intelligence affect the trustworthiness of public service delivery - PM&C (pmc.gov.au)
Product information
Copilot for Microsoft 365
Microsoft Copilot Copyright Commitment Security Information
Data, Privacy, and Security for Copilot for Microsoft 365
Microsoft Purview data security and compliance protections for Microsoft Copilot
Support and Feedback on the trial
The NDIA project team responsible for conducting the trial can be contacted via:
redacted: s47E(d) - certain operations of agencies
PageOFFICIAL14 of 76
DOCUMENTS FOI Dia Sggc
ie
End of Trial Report
Copilot for Microsoft 365
January — June 2024
Version: 1.0
Date: 17 June 2024
Author: Josiph redacted: s47F - personal privacy
Division: Office of the Chief Information Officer
ndis.gov.au
| End of Trial Report | 1 |
|---|---|
| Copilot for Microsoft 365 | 1 |
| January – June 2024 | 1 |
- Executive summary | 4 1.1 Background | 4 1.2 Objectives of the trial | 4 1.3 Purpose | 5
- Copilot at the NDIA | 6
- Implementation of Copilot for Microsoft 365 | 8 3.1 Approach | 8 3.2 Planning and Configuration | 8 3.3 Training and Support | 9 3.4 Implementation and Monitoring | 9
- Results | 10 4.1 Using Copilot | 10 4.2 Accessibility | 11 Neurodivergent User | 12 JAWS User | 12 Dyslexic User | 12 Hearing Impaired User | 12 4.3 Ongoing Training, User Monitoring, and Guidance | 13 4.4 Network Integration | 13
- Summary of Benefits | 14
- Key Findings | 16 6.1 User Productivity | Error! Bookmark not defined. 6.2 User Satisfaction | Error! Bookmark not defined. 6.3 Accessibility Usage | Error! Bookmark not defined. 6.4 Stakeholder Engagement | Error! Bookmark not defined.
- Perceived Risks and Mitigation Strategies | 17
ndis.gov.au 17 June 2024 | Copilot – End of Trial Report 2
Page 16 of 76
- Controls for Mitigating Risks During the Trial | 19 —|— 8.1 Data Privacy Controls | 19 8.2 User Training and Support | 19 8.3 Ethical Guidelines | 19
- Summary of Recommendations | 20
- Conclusion | 20
ndis.gov.au 17 June 2024 | Copilot – End of Trial Report 3
Page 17 of 76
1. Executive summary
1.1 Background
In the wake of rapid advancements in Artificial Intelligence (AI) technologies, the Agency recognises the potential of AI to transform public service delivery, enhance operational efficiency, and foster innovation.
The Australian Public Service (APS) commenced a six-month trial of Copilot for Microsoft 365 in January 2024 to test new ways to enhance productivity and develop skills, capabilities and preparedness for the use of generative AI tools by APS staff. The trial was coordinated by the Digital Transformation Agency (DTA) and involved over 50 participating Commonwealth agencies.
The Agency is participating in the trial alongside numerous Australian Government Agencies and Departments and obtained 300 licenses. The trial started as a Proof of Concept (POC) for four months, and then continued as a Trial for the remaining trial period. A smaller group of users were chosen for the er group of users were chosen for the POC, more users were included for the Trial. Staff who participated were urged to use Copilot to help with their daily administrative tasks.
Over the six-month trial period, data was collected and analysed to assess security posture, user productivity, satisfaction, and accessibility improvements.
1.2 Objectives of the trial
- Evaluate effectiveness: Assess staff using Copilot impacts productivity and user satisfaction.
- Determine value for money: determine whether future use by all NDIA staff represents value for money.
- Ongoing AI risk mitigation: Assess if having an internal AI product like Copilot reduces the likelihood of staff accessing external/unapproved AI services, and the corresponding threat to Agency data that arises from unapproved use.
- Policy development: Use the trial to develop a policy position and unified approach to the use of Artificial Intelligence within ICT services.
- Accessibility impact: Measure the accessibility improvements for users requiring support.
- Gather data for informed decisions: Collect quantitative and qualitative data to inform future AI tool implementations.
ndis.gov.au 17 June 2024 | Copilot – End of Trial Report 4
Page 18 of 76
1.3 Purpose
This purpose of this report is to:
- Describe the implementation of Copilot for Microsoft 365 at the Agency
- Illustrate use cases of Copilot at the Agency
- Evaluate the effectiveness of Copilot for Microsoft 365 on productivity and accessibility within the Agency.
ndis.gov.au 17 June 2024 | Copilot – End of Trial Report 5
Page 19 of 76
2. Copilot at the NDIA
A description of how Copilot can assist NDIA staff with their daily tasks is tabled below.
| Use | Description |
|---|---|
| Ensuring policy compliance | Assistance with preparing documents and emails and aligning with NDIA policies and guidelines by: • interpreting NDIA policies and generating a summary of the the purpose • identifying amendments to a policy document and the responsible person |
| Meeting scheduling | Quickly schedule and reschedule meetings: • without manually comparing participants’ calendars for availability, and • creating meeting invites, adding participants and setting the meeting date and time based on availability |
| Checking document accessibility | Improve the accessibility of documents by providing suggestions for: • readability and simplicity • formatting • complying with the NDIA’s accessibility standards • colour contrast for colour vision deficiencies • converting into structured PDF or plain text for accessibility • accessibility best practices |
| Email summary and creation | Assist users with: • summarising key points and decisions from large email chains • drafting email responses • managing email inboxes (eg flagging important emails) |
| Assisting with data analysis | Assist users with: • creating complex formulas in Excel spreadsheets • generating charts and graphs to visualise data • interpretating numbers of trends |
ndis.gov.au 17 June 2024 | Copilot – End of Trial Report 6
Page 20 of 76
| Use | Description |
|---|---|
| • sorting and filtering data • reminding users to keep data up to date |
|
| Providing in-context training | Assist user with learning Microsoft 365: • navigating tools and processes by providing step-by-step instructions • helping learn more about advanced features |
| Document summarisation and reading aloud | Assist visually impaired staff with reviewing extensive documents and reports by: • providing summaries of Word documents with the key points, decisions and action items; and • text-to-speech functionality. |
| Presentation creation assistance | Assist staff with creating engaging and informative presentations by: • suggesting a slide structure from an outline of the presentation • using the appropriate NDIA templates • retrieving and including relevant statistics, images and case studies from NDIA data • assisting with presentation rehearsals • assisting with managing live Q&A sessions by quickly providing information which answers the audience’s questions |
| Accessible data analysis and reporting | Assist visually impaired users interpret complex data sets, charts and graphs by: • interpreting data and providing simple and understandable insights in a format which is understandable without visual aids • analysing trends, summarising data points and predicting future patterns based on historical data • creating accessible reports through the translation of the data analysis into descriptive narratives in a format which is inclusive for all • allowing reports to be easily shared through Microsoft 365 |
ndis.gov.au 17 June 2024 | Copilot – End of Trial Report 7
Page 21 of 76
| Use | Description |
|---|---|
| • ensure documents created are compliant with accessibility standards and easy to navigate | |
| Accessible project management | Assist hearing impaired users with managing projects by: • enabling real-time transcription during Teams meetings so the user receives live captions and allow for confirmation of any discussion points which may have been missed • generating notes, list tasks and assist the user in other ways during the meeting • organising project tasks and timelines, the user can provide Copilot verbal instructions to update task statuses, add new assignments, and set reminders for deadlines. |
3. Implementation of Copilot for Microsoft 365
3.1 Approach
The trial faced difficulties due to:
- scepticism about the usefulness of generative AI tools such as Copilot
- concerns for the security and protection of Agency data Copilot would access
- the perception that the trial was being conducted to evaluate it’s effect on productivity as part of a plan to reduce staff numbers
- findings of the recent “Robodebt” Royal Commission
To mitigate, the Copilot trial was reduced to a Proof of Concept (POC) for the initial four months of the trial. This provided the additional time required to carefully examine and mitigate risks before conducting the trial. Add some points in here about what was scaled down for the POC. A communication ban was enforced that impacted the ability to communicate broadly.
The POC was extended to a full trial in May 2024. Add some points in here about what was scaled up for the Trial. What happened to the communication ban?
3.2 Planning and Configuration
ndis.gov.au 17 June 2024 | Copilot – End of Trial Report 8
Page 22 of 76
- Planning phase: Detailed planning sessions were held to define the scope, objectives, and success criteria for the trial.
- Infrastructure configuration: Necessary IT infrastructure and security measures were put in place to support the deployment of Copilot.
- Agency AI Working Group formation: Approximately 10 executives across different NDIA Business Areas, including staff requiring accessibility support.
- POC Group selection: Select up to 300 NDIA staff across different Groups, Divisions and Sections, including users requiring accessibility support.
- Trial Group selection: Select up to 300 NDIA staff across different Groups, Divisions and Sections, including users requiring accessibility support.
3.3 Training and Support
- Information sessions: Comprehensive training sessions were conducted to ensure users were familiar with Copilot features and functionalities.
- Learning modules: A mandatory learning module ensured that all users were taken through the ethical and professional obilgations of using AI.
- Support channels: Dedicated support channels were established, including a helpdesk, FAQs, and user guides.
3.4 Implementation and Monitoring
- Phased rollout: The implementation was carried out in phases to manage any potential issues effectively.
- Monitoring and feedback: Continuous monitoring of usage and performance, along with regular feedback collection from users, was conducted to identify areas for improvement.
ndis.gov.au 17 June 2024 | Copilot – End of Trial Report 9
Page 23 of 76
4. Results
4.1 Using Copilot
The following benefits were demonstrated by users using Copilot, and could also be realised by users across all Australian Government Departments:
| Benefit | Copilot Use |
|---|---|
| Enhanced productivity | Copilot assisted users achieve their tasks faster. For example, capturing summaries of meetings, assisting with summarising and prioritising email trails, assisting with scheduling meetings. This outcome indicates significant improvements to staff productivity. |
| Improved accessibility | Copilot enhanced accessibility support by providing guidance to enhance the accessibility of document content; interpreting complex text, graphs and analytical data; providing assistance during and after meetings through transcripts and summaries; and other capabilities that increased a staff member’s independence to complete tasks. This improved engagement and productivity of staff relying on accessibility tools. |
| Data-driven decision making | Copilot has the ability to search through NDIA data for relevant statistics, images, and case studies. This feature ensures that the content is accurate and visually appealing, aiding in informed decision-making. |
| Presentation preparation and delivery | Using Copilot’s assistance to prepare and rehearse presentations was a key finding. Copilot can generate slides with suggested titles and content, provide a script based on the slide content, and offer feedback on delivery, which analyses pacing, clarity, and tone. |
| Streamlining administrative activities | All use cases indicate that Copilot will streamline routine administrative activities, such as writing briefs and initial drafts of correspondence. This has the potential to provide |
ndis.gov.au 17 June 2024 | Copilot – End of Trial Report 10
Page 24 of 76
| Benefit | Copilot Use |
|---|---|
| significant time savings to administrative staff within the Agency. |
4.2 Accessibility
The participation of Employee Disability Network (EDN) members in the NDIA trial provided valuable insights into Copilot’s impact on their work. The tool received overwhelmingly positive feedback, with notable benefits.
| User Group | Copilot Use |
|---|---|
| Neurodivergent Users | Copilot significantly improved task management, communication, and focus, offering tailored suggestions and automating routine tasks. |
| JAWS Users | Copilot’s integration with JAWS screen readers enhanced accessibility, making information more accessible and interactions more efficient.The ability to use Copilot to recap meetings allowed visually impaired users the freedom to participate fully in meetings rather than concentrating to ensure that transcript matched the tone of the discussion. |
| Hearing Impaired Users | Copilot’s application within Microsoft Teams was the biggest gamechanger for this group of users. The recap function, the live transcript in meetings, and the ability to ask Copilot for clarification when the conversation moved too quickly were revolutionary for hearing impaired users. |
| Users with Dyslexia | The ability to draft content from simple prompts or rewrite written content allowed users with Dyslexia the comfort of knowing that their work was correct, raising their confidence and shortening editing and review time across the board. |
ndis.gov.au 17 June 2024 | Copilot – End of Trial Report 11
Page 25 of 76
4.2.1 Feedback
Neurodivergent User
Copilot has been a game-changer for me. It helps me stay organised and focused by providing real-time suggestions and automating repetitive tasks, reducing cognitive load and allowing me to concentrate on more complex work.
JAWS User
Using Copilot with JAWS has revolutionised how I interact with digital content. The AI assists in navigating complex documents and emails, making my workflow much smoother and more efficient.
Dyslexic User
Copilot has dramatically improved my experience with written communication. It helps me draft emails and documents by suggesting correct spelling, grammar, and even phrasing, reducing the time and stress associated with writing and ensures my communications are clear and professional. In Microsoft Teams, Copilot summarises long messages and threads, making it easier for me to understand and respond accurately.
Hearing Impaired User
Copilot has greatly enhanced my ability to participate in and manage digital communications. In Microsoft Teams, Copilot provides real-time transcription of audio and video meetings, allowing me to follow conversations more accurately and engage more effectively. This feature eliminates the need for separate transcription services and reduces delays in understanding meeting content. Additionally, Copilot’s ability to summarise key points and generate meeting notes ensures I never miss crucial information. In Outlook, Copilot allows me to access information in a way that suits my needs. Overall, Copilot has made my digital interactions more inclusive and efficient, significantly improving my work experience here in the Agency.
ndis.gov.au 17 June 2024 | Copilot — End of Trial Report 12
Page 26 of 76
4.3 Ongoing Training, User Monitoring, and Guidance
Feedback from users indicated that Copilot made it very easy to slip into usage patterns outside of the trial and interim policy parameters. Ongoing education around the ethical use of AI would be required to ensure that users complied with data protection expectations.
| Training and Support | All use cases indicated that the implementation of any AI tool required ongoing and comprehensive training sessions. A new tool like Copilot also required dedicated support channels, including a helpdesk, FAQs, and user guides. |
| Ethical Use and Data Privacy | The establishment of clear guidelines, transparency, and accountability measures to monitor the ethical use of AI tools, along with strict data privacy controls, are critical use cases that demonstrate responsible AI implementation. |
4.4 Network Integration
| Microsoft Managed and Device (MMD) program | The Agency’s (MMD) program, coupled with its greenfield relationship with Microsoft, facilitated a smooth and efficient activation of Copilot. This advanced setup allowed for quick deployment and reduced downtime, ensuring minimal disruption to ongoing operations. |
| Identity Management | Identity management functions were configured effectively, allowing users to access data and information strictly within Identity and Access Management (IDAM) rules. This setup not only enhanced security but also ensured compliance with internal data governance policies, providing a robust framework for secure and efficient use of Copilot across the Agency. |
ndis.gov.au 17 June 2024 | Copilot – End of Trial Report 13
Page 27 of 76
5. Summary of Benefits
The following benefits were realised during the POC and trial of Copilot:
| Benefit | Description |
|---|---|
| Enhanced Collaboration | Copilot facilitates better collaboration by summarising conversations, providing real-time suggestions, and enabling seamless sharing of information, thus enhancing team productivity. |
| Process Optimisation | By automating routine tasks and providing intelligent suggestions, Copilot helps streamline workflows, reducing manual effort and minimising errors. |
| Time efficiencies | Copilot saves time by automating repetitive tasks, drafting emails, and summarising lengthy documents and conversation threads, allowing users to focus on more complex tasks. |
| Skill Development | Copilot assists users in learning and improving their skills by providing context-aware tips and recommendations, thus enhancing their overall proficiency in using Microsoft 365 tools. |
| Accesibility Improvements | Copilot’s accessibility features significantly improved the experience for all users with disabilities within the trial. |
| Cyber Security Mitigations | By providing an in-house AI tool like Copilot, the Agency mitigated the risk of users seeking potentially insecure, web-based AI tools, ensuring Agency data remains within the secure government ecosystem. |
| Improved Decision Making | Copilot supports better decision-making by providing data-driven insights, analysing trends, |
ndis.gov.au 17 June 2024 | Copilot – End of Trial Report 14
Page 28 of 76
| Benefit | Description |
|---|---|
| and generating reports, enabling users to make informed and timely decisions. | |
| Enhanced User Satisfaction | The intuitive interface and helpful suggestions from Copilot lead to higher user satisfaction and engagement, contributing to a more positive work environment. |
| Reduced Workload | By automating repetitive and time-consuming tasks, Copilot reduces the overall workload on employees, helping them manage their time more effectively and reduce burnout. |
| Consistency in Work Output | Copilot ensures consistency in documents and communications by providing standard templates and suggesting improvements, leading to a more uniform and professional output. |
ndis.gov.au 17 June 2024 | Copilot – End of Trial Report 15
Page 29 of 76
6. Key Findings
ndis.gov.au 17 June 2024 | Copilot – End of Trial Report 16
Page 30 of 76
7. Perceived Risks and Mitigation Strategies
| Risk Category | Risk Description | Likelihood | Impact | Mitigation Strategy |
|---|---|---|---|---|
| Data Privacy External | Potential risks of data breaches or misuse of sensitive information by members | Medium | High | Strict adherence to data privacy regulations, regular security audits, and user awareness training. |
| Data Privacy Internal | Risks of accidental data exposure or unauthorised access by internal users | Medium | High | Implementation of robust access controls, continuous monitoring of data access and usage, regular internal audits, and comprehensive training for employees on data handling practices. |
| Integration Issues | Technical challenges in integrating Copilot with existing systems | Low | Low | Thorough testing, phased rollout, and collaboration with Microsoft for troubleshooting. |
| User Resistance | Resistance to adopting new AI tools | High | Medium | Comprehensive training, ongoing support, and communication plan that |
ndis.gov.au 17 June 2024 | Copilot – End of Trial Report 17
Page 31 of 76
| Risk Category | Risk Description | Likelihood | Impact | Mitigation Strategy |
|---|---|---|---|---|
| highlight benefits and lowers expectations | ||||
| Dependence on AI | Over-reliance on AI tools might reduce critical thinking skills | Low | Low | Encouraging balanced use, where Copilot is seen as a support to repetitive tasks and not a replacement of human judgment. |
| Ethical Concerns | Ethical implications of using AI in decision-making processes | High | High | Preparation of the Interim Use of AI Policy and establishing clear supporting guidelines and ensuring transparency in how the Agency assesses AI. |
ndis.gov.au 17 June 2024 | Copilot – End of Trial Report 18
Page 32 of 76
8. Controls for Mitigating Risks During the Trial
8.1 Data Privacy Controls
Encryption: All sensitive data was encrypted both in transit and at rest.
Access Controls: Strict access controls were implemented to ensure only authorised personnel could access sensitive information.
Regular Audits: Periodic security audits were conducted to identify and mitigate potential vulnerabilities.
8.2 User Training and Support
Comprehensive Training: Detailed training programs were conducted to ensure users were well-versed in using Copilot.
Continuous Support: Ongoing support was provided through helpdesks, user guides, and FAQs to address any user concerns promptly.
8.3 Ethical Guidelines
Transparency: Clear guidelines were established to ensure transparency in how AI tools were used within the organisation.
Accountability: Accountability measures were put in place to monitor the ethical use of AI tools.
ndis.gov.au 17 June 2024 | Copilot – End of Trial Report 19
Page 33 of 76
9. Summary of Recommendations
Extend the Trial: Extend the trial for an additional 12 months to gather more data and further evaluate the long-term impact.
Expand user base: Include more users, particularly those requiring accessibility support, to get a broader understanding of the tool’s impact.
Enhance training: Continue to provide comprehensive training and support to maximise user engagement and effectiveness.
Strengthen partnerships: Foster stronger relationships with Microsoft and other AI tool providers to ensure continuous improvements and support.
Establish clear guidelines: Develop and implement clear guidelines for the ethical use of AI tools within the organisation.
10. Conclusion
The Copilot for Microsoft 365 trial has shown significant potential in enhancing productivity, user satisfaction, and accessibility within the NDIA.
The six-month trial of Copilot for Microsoft 365 demonstrated significant benefits for the Agency, particularly for employees with accessibility needs. The tool’s positive impact on neurodivergent users and those relying on JAWS underscores its potential as an inclusive productivity aid.
Additionally, robust data security measures ensured the protection of Agency and participant data throughout the trial.
Extending the trial period and expanding the user base will provide more comprehensive data, allowing the agency to make informed decisions about the long-term adoption of AI tools.
The controls and mitigation strategies implemented during the trial have effectively managed risks, and the lessons learned will guide future initiatives to further leverage AI tools for organisational benefits.
Based on these findings, it is recommended that the Agency consider wider adoption of Copilot to further enhance productivity and accessibility for all employees.
ndis.gov.au 17 June 2024 | Copilot – End of Trial Report 20
Page 34 of 76