Guide to undertaking Privacy Impact Assessments on AI systems and projects

A PIA is a structured methodology that can help you to identify and minimise privacy risks, and realise privacy improvements, when you are starting a new project or making changes to existing initiatives. A PIA is one way to implement ‘privacy by design’ in your organisation’s practices. It can help you to understand the collection and flows of personal information while building and demonstrating compliance with privacy laws.

Refer to the Guide to Privacy Impact Assessments in NSW for more information about PIAs.

AI systems and projects could involve data that, at first glance, may not appear to be personal information, such as randomly assigned identifiers that distinguish individuals from each other but do not include attributes such as a name, address or driver licence number. Agencies should be aware that data of this kind could still be considered personal information if a person’s identity can be reasonably ascertained by referring to other data sources – even if the agency doesn’t have any specific intention to do such an identification.

Because personal information can include an opinion about an individual, AI-generated inferences about individuals are also considered personal information, even if they are incorrect.

See:  Fact Sheet - Reasonably Ascertainable Identity and Fact Sheet - De-identification of personal information

The IPPs and the HPPs are applicable to the activities of agencies involving personal information, whether the activity involves AI or not. AI is a broad label and there can be differing views on what constitutes AI. So the question might arise: “Why does AI require special treatment when it comes to privacy?”

For the purposes of privacy, what is important is that substantive privacy impacts on the community are assessed irrespective of the label applied to the technology. However, in practice AI can enable the processing and production of vast quantities of data on a scale which is uncommon among other technology solutions. AI can also be readily used to make or guide decision-making processes which have a profound impact on individuals. For these reasons, the impact and likelihood of traditional privacy risks can be greatly amplified by the use of AI technology. AI also intersects with other emerging technologies to create novel risks and challenges. 

These AI privacy risks are part of a range of ethical concerns arising from the technology. Ethical concerns include bias. fairness, transparency and accountability. They will often require careful consideration and for agencies to go beyond merely complying with privacy laws.

• Is the personal information that is being collected for the AI system or project directly related to the agency’s function or activities and necessary for that purpose?
• Does the AI system or project collect personal information beyond what is reasonably necessary for the agency’s function or activities? 
  • Be sure to review each input.
  • Consider also that the outputs of the AI system or project could be considered a “collection through creation” of personal information. Consider whether those outputs are all reasonably necessary for the agency’s functions or activities.
• Does the AI system or project clearly comply with the relevant legislation? 

AI technology enables both large-scale and novel kinds of data collection. Agencies should ensure the collection of personal information is reasonably necessary and directly related to the agency’s functions or activities.

AI systems can also create personal information through new insights about individuals. The creation of these new insights can be viewed as a collection of personal information. 

An example is a mobile fitness device and app that regularly ‘creates’ personal information about individuals through the monitoring of heart rate, pulse, walking or sleeping patterns. Over a period of time, new insights are created about an individual’s likely health outcomes, including the detection and prediction of disease.

Accordingly, agencies should ensure any new personal information created is also reasonably necessary and directly related to the function or activity of the agency.

If an AI solution can collect personal information that is beyond the scope of the agency’s activities, that capability should not be deployed. Configurations need to be carefully considered. Features that are merely interesting or fun should not be used if they involve collecting personal information that is not directly related to the agency’s activities.

AI systems which are not clearly explainable, or which do not transparently and comprehensively document the basis and evidence for their output, should not be relied upon for automated decision-making processes that affect individuals. Agencies that are seeking to use AI systems to make decisions that will impact individuals should carefully consider the legal basis underpinning the activity before proceeding and should ensure that there are appropriate opportunities for human review throughout the decision-making process.

• Does the AI system or project collect personal information directly from the individual or seek their authorisation to collect it from other sources?
• Consider personal information created through the use of AI systems and projects.

AI technology can involve collecting data from multiple sources – whether public or held by other organisations. Agencies should not collect personal information about individuals from other sources without consent from the individual. 

Agencies must consider whether the AI system creates personal information, and whether this “collection through creation” is without the user’s involvement or with the consent of the individual. Some common ways that AI can indirectly collect personal information is collection of facial images or biometric data from cameras or sensors, or inferences about a person’s emotional state derived from their movements, their voice or other physical characteristics.

Collecting data from other sources including from public sources like social media posts, or semi-public sources like the ASIC company register which charges a fee, is also an indirect collection of personal information which can be unexpected, unfair or intrusive, and could also result in the collection of unsolicited information.

When agencies enter into agreements to collect data from third party organisations, agencies should satisfy themselves that the data was collected from individuals lawfully and that the sharing to the agency is aligned with the original purpose of collection or individuals have consented.

In addition, commonly used software applications are integrating AI features to collect user data ‘behind the scenes’ in a way which is often not obvious to users. Agencies should satisfy themselves when deploying software applications as to whether this is a feature of the software.

• Are individuals notified that their personal information is being collected, the purposes for the collection and the intended recipients of their personal information?
• Consider third party providers managing the AI system.
• Consider whether the individual is notified of any personal information created through the use of the AI system.
• Consider whether personal information is being used to train the AI system, and whether that training is restricted to use on the agency’s system or also benefiting the third party provider, and whether that purpose should be notified to the individual at the time of collection.
• Is it clear whether the collection is required by law or is voluntary, and any consequences if the individual chooses not to provide the information?

AI technology may involve data collection that is covert or not obvious such as recording behaviour of users on a website or capturing data from a person’s voice or movements to ascertain their emotional state. Agencies should ensure individuals are informed in the manner that is most effective in the context of the collection.

Third party providers managing AI systems could be recipients of an individual’s personal information and should be mentioned in privacy notices in such circumstances.

If an agency is using an AI system that will use personal information solely for enhancing the agency’s service, a privacy statement or notice should include this. If the personal information is used beyond this, for example to allow the AI vendor to enhance their product, seek consent from affected individuals. 

Generally, using personal information to train AI systems outside of an agency use case is high risk and should be avoided wherever possible.

Agencies need to consider how to effectively communicate to individuals about novel forms of personal information collection. For example, where an AI-powered chatbot is engaging in a human-like conversation to elicit information, it is preferable to have the required notification communicated by the chatbot at the time the conversation is occurring, not posted on a web page which the user may not see.

• Is all the personal information collected for the AI system relevant to the purpose of collection?
  • Review each data point/input for relevance.
• How are you ensuring the personal information being collected is accurate, complete, up-to-date and not excessive?
  • Consider the source of the personal information being collected. Can it be relied on? 
  • Are there measures in place to ensure all required personal information is provided? What are the impacts if the individual provides only some of the required personal information?
  • Will the individual be prompted to provide all the required information?
  • Will the individual be prompted to provide the personal information in an appropriate format?
  • Is old information that may be out of date being used or relied upon?

Using poor quality information such as agency records which are out-of-date could result in unfair or wrong decisions. This is particularly concerning where it could limit an individual’s access to a service, opportunity or benefit, or result in a penalty.

Some AI tools, such as chatbots, may need to prompt an individual to provide personal information. It is important that this prompting occurs in a way that elicits relevant, accurate, complete and up to date information.

When using AI technology, it can be easy to collect more information than is required and automated collection may not be subject to the same quality assurance that applies to personal information collected through other means.

Agencies should guard against collecting excessive amount of data just because it is there or easy to do with the use of technology. If an AI tool needs to be fed data held by the agency, it should be limited to what is strictly required for a specific outcome. 

Agencies should consider whether the use of free text fields is necessary, as there can be a heightened risk of collecting unsolicited personal information. There is less risk in using structured data fields.

• Is personal information being stored securely within the AI system?
• Who has access to the personal information handled within the AI system?
• Is there a risk that an individual may see personal information of another individual when using the AI system?
• Have retention periods and appropriate data disposal methods been defined and implemented?
• If a third party is handling personal information, what due diligence has been undertaken to ensure the personal information is protected from unauthorised use or disclosure?

Data in an AI tool will need to be securely managed as it would in any other platform or system, on-premise or cloud-hosted. Data retention policies will need to incorporate mandatory minimum retention periods as required by the State Records Act or any other legislative requirements.

Contracts should be in place with third-party providers containing binding clauses on data security and compliance with the PPIP Act and the HRIP Act. Contracts should consider and account for obligations under the Mandatory Notification of Data Breach Scheme.

Use caution in using AI tools provided by third parties on the basis of standard terms. These terms are unlikely to require third parties to handle personal information in accordance with the PPIP Act and HRIP Act.

• How is the agency explaining to the person what personal information about them is being stored, why it is being used and any rights they have to access it?
• Can you describe what, how and why an individual’s personal information is being used in relation to the AI system?

Agencies should ensure clear information is made obviously and prominently available in a way that is appropriate to the situation, whether it is a website, in a live chat, on a sign in a public place or in a personal email addressed to the individual. Privacy Management Plans should be updated to include information about the use of AI. 

Individuals should not have to undertake onerous investigations or chase down information to know whether personal information about them is being stored and their rights in relation to that personal information.

• Are you prepared to meet requests for access to personal information, including insights or inferences, derived from the use of AI systems.
• Are you allowing people to access their personal information without excessive delay or expense?
  • Consider whether the AI system will enable or delay an individual’s access to their personal information.

• How can an individual update, correct or amend their personal information where necessary in relation to the AI system or project?
• Can you ensure the AI system or project will make decisions based on the updated personal information?

• What measures are in place to ensure the personal information is relevant, accurate, up to date and complete before being used by the agency?
  • Consider whether additional checks are required if relying on AI generated outputs.
  • Should AI generated outputs be limited to specific, low risk use by the agency?

AI technology such as automated decision-making can be used to make decisions or to recommend decisions to agency staff which will affect an individual’s access to a service, opportunity or benefit, or result in a penalty.

AI may produce outputs that are sufficiently reliable for some purposes (such as recommending a service), but not for other purposes (such as approving an application). Consider the accuracy of the information when deciding how it will be used.

Poor system design, or the use of poor-quality information such as historical agency records could result in biased, unfair or wrong decisions. Agencies must ensure training data and systems are reviewed with these risks in mind, especially where the AI system informs or makes decisions.

Agencies should ensure that decisions made by an AI tool are explainable. If there is uncertainty about the reasons why the technology is making certain decisions or recommendations, it should not be used.

Agencies should ensure that there is human validation of any AI process that uses personal information.

• Are you only using the personal information for the purpose it was collected unless the person has given their consent, or the purpose of use is directly related to the purpose for which it was collected, or to prevent or lessen a serious or imminent threat to any person’s health or safety?
• If using a third-party provider, have you ensured an individual’s personal information is not being used by that third party for their own purposes?
• When it comes to training AI models have you considered ways this can be done without using personal information?

AI systems can make it easy for agencies to use data for multiple purposes. Agencies should ensure each use case is assessed on its own merits against the IPPs and HPPs. For example, a database of facial images for security passes should not be used for research without a separate assessment.

Agencies should refrain from using personal information collected for a specific purpose for a different purpose unless individuals give their consent or an exemption applies. 

It is also common that AI system providers seek to use their customer’s data for their own purposes, such as training their AI models. This is distinct from an agency using personal information for training an AI system exclusively for internal purposes. 

Agencies should ensure external use of personal information does not occur unless affected individuals give consent or an exemption applies. Generally, using personal information to train AI systems outside of an agency use case is high risk and should be avoided wherever possible.

(In addition to the IPPs and HPPs, see Statutory Guidelines on exemptions)

IPP 11 – Restricted 

HPP 11 - Limited

• Does the AI system or project disclose personal information to another person or organisation?
• If so,
  • Is the disclosure directly related to the purpose the information was collected, and the individual is unlikely to object to the disclosure? or
  • Is the individual reasonably likely to have been aware of this disclosure? or
  • Is the disclosure necessary to prevent or lessen a serious and imminent threat to the life or health of an individual?

Where AI technology vendors, platform providers, or any other external organisation, will have access to personal information being processed by the technology, this could be considered a disclosure, and the agency should ensure it is able to satisfy one or more of the criteria in IPP 11 to permit the disclosure. 

Where agencies provide their contracted service providers, including AI system vendors, with access to personal information for the sole purpose of carrying out a contracted service on behalf of the agency, this is likely to be a 'use' (see IPP/HPP 10).

Agencies should also consider whether staff or users of a system may be able to view information of other individuals and implement measures to prevent this or ensure it occurs only with authorisation from the individual.

Agencies should consider what information needs to be provided to individuals who are users of a system and what consents might be needed if they are partnering with a third-party vendor.

In the PIA, agencies should document the disclosures which are intended and the rationale for how they comply with IPP 11. For example, AI vendor tech support is provided by a team of five staff who will have access to all data including personal information. This is a directly related purpose to which it may be assumed there is no objection if appropriate contractual obligations, access controls and security measures are in place.

• Have you ensured personal information relating to an individual’s ethnic or racial origin, political opinions, religious or philosophical belief, trade union membership or sexual activities are not being disclosed unless there is a serious and imminent threat to the life or health of an individual?
• Is the personal information being disclosed to a Commonwealth agency or a person or body outside of New South Wales?
  • Consider the location of any third-party providers engaged to support the AI system or project.

Agencies should carefully consider whether data they disclose could fall under the categories mentioned in IPP 12.

Agencies should refrain from disclosing datasets automatically and should ensure there is human review of data before it is disclosed.

Agencies should design systems and processes to remove sensitive information if it is included in chatbot free text fields. Agencies may also consider clear user guidance or notifications to deter users from entering sensitive information that is not required.

Facial images may be considered ‘sensitive information’ because it is potentially personal information about racial or ethnic origin or can indicate a person’s religion. Sensitive information points can be inferred from other personal information.

• In the event the AI system or project involves health information, can you achieve your objectives without using unique identifiers?

• In the event the AI system or project involves health information, and you are transferring data outside NSW, have you considered if the recipient is bound to standards which are equivalent to the NSW HPPs or if you can obtain individuals’ consent?

Agencies should ensure they are transferring health information only in accordance with the criteria in HPP 14. Key considerations are:

• whether the recipient is bound to standards which are equivalent to the NSW HPPs, 
• whether the individual has given consent, or
• whether there is a contract which would benefit the individual, it is impracticable to obtain consent, and the individual would be likely to consent.