Information Access Guideline 7 - Open Data

Read the guideline below or download it here Information Access Guideline 7 - Open Data May 2019.

1 What is open data?

Open data means simply that information an organisation holds that is stored digitally should be made freely available to the community. This is usually done by making a digitised dataset accessible from the organisation’s website. Anyone who accesses and downloads the data is then free to use it, rearrange it and publish it as they wish. They are not restricted in doing so by principles of copyright or original authorship or ownership.

Opening data means releasing data in a form that ensures that it is accessible and usable.  Government holds vast amounts of data that is, in its existing form or could become, open data. 

Government is also able to release data using the GIPA Act access pathways. Increasingly government is holding information in digital form. The GIPA Act encompasses all information held by a government agency – this includes data. This data might not ever fulfil the definition of open data in that it might be limited by conditions that could be imposed under the informal release pathway.^[1]

An open data strategy can be adopted by any organisation – government or non-government. More commonly the term is linked to government information management practices.

Partly this reflects that government agencies have extensive data holdings about all facets of society – economic management, law enforcement, health, corporate regulation, transport, agriculture, land planning, the environment, sporting, community assistance and many other areas. Data analysis underpins government policy-making and administration. Government data sets can have similar value to other bodies and individuals in their business and lifestyle planning. Importantly, Open Data strategies reflect an ongoing need to continually review and release data sets to expand the information asset and promote its integrity.

Another reason that open data and government are frequently linked is because democracy is underpinned by the principles of open government. This relationship is enshrined under the GIPA Act.^[2] Transparency, democratic accountability and public service go together. Information collected or created by government to perform its functions is an important public asset.

Like all government resources, information resources should be used to benefit the public, and ideally shared with them. This is captured in other terms that are often used interchangeably with open data – such as public sector information, and open government data.

2 Why is open data important?

We live in an information age – an age of big data.^[3] Performance in all walks of life is commonly tied to how effectively we acquire and use information about an activity or function. Information analysis and sharing is a platform for mapping and developing human progress. More than ever before, technology makes it easier to share information widely and inexpensively.

The NSW Government Open Data Policy published in 2016, notes that open data – shared information – can benefit the government and the community in four ways:

Improving government

Sharing information about a government activity means that more minds can contribute to planning, delivering, testing or assessing the activity. Open discussion – from different angles – about objectives, options and expectations can lead to better outcomes. Trends and cross-portfolio impacts can be better understood.

Information sharing also enables different activities in and outside government to be linked, to aid compatibility and efficiency. This has been vital to programs that affect all of us, such as the Smart Cities program that is based on linking infrastructure and service data about transport, utilities, health, workplaces and communities.

Empowering citizens

Knowledge is power. A better-informed community can engage and collaborate with government to improve government service delivery and its understanding of local community needs and challenges. Researchers and stakeholders can build on the work of government and offer fresh insights and analysis. Open data enables an open dialogue to take place.

Public trust in how government manages and shares information, and deals with privacy and security challenges, can also be strengthened by an open public data strategy.

Creating opportunity

Innovation and progress build on what is already known. As Isaac Newtown, observed, ‘If I have seen further, it is by standing on the shoulders of giants’.

Data holders may not realise that information they have collected for one purpose can have utility in a different context. The release of government data enables the business, research and community sectors to use or add value to the information for other activities.

Solving problems

The challenges facing government are complex. The information held by government on a topic may be incomplete or not current or clear. Most policies make assumptions or predictions about people’s behaviour that may be imperfect on the limited information that is first available.

Fresh insights and a stronger evidence base for problem solving can be developed when there is robust public participation supported by an open data culture. Government programs can be better targeted to ensure they are effective and tailored to client needs and characteristics.

These impacts or benefits have been echoed in many reports in Australia and internationally. A landmark Australian report in 2017 was the Productivity Commission report, Data Availability and Use. A key message in that report – under the heading ‘Access denied – Australia’s lost opportunities’^[4] – was that there is broad scope within Australian governments to increase the availability and productive use of data.

The tangible benefits of open data to government administration, business innovation, public participation, research and democratic processes were documented in the report and supported by hundreds of submissions the Commission received. As the Commission observed, government and the community must treat data as an asset and not a threat.

This challenge has also been taken up in the Australian Government’s second Open Government National Action Plan 2018-2020 (the NSW Information Commissioner is an appointed government member of the Open Government Forum that monitors and drives implementation of the National Action Plan). The 75 participating countries in the international Open Government Partnership are each required to prepare a national action plan to outline the country’s open government agenda. One of the eight commitments in Australia’s current national action plan is to ‘Improve the sharing, use and reuse of public sector data’.

3 Open data and the GIPA Act

The premise of the GIPA Act is that disclosure of government information is in the public interest. Only if there is an overriding public interest against disclosure should government information be withheld from the public (s 5).

To implement that presumption in favour of open government the GIPA Act sets out four information pathways. Each pathway can be activated to open data. Likewise open data can be released absent the authority provided under the GIPA Act when it is truly open data. To assist agencies in identifying open data the Information and Privacy Commission published a guide to open data.^[5]

Open data is a central element of two of the four information pathways – ‘mandatory proactive release’ (s 6) ^[6] and ‘authorised proactive release of government information’ (s 7).^[7]

Mandatory Proactive Release

An agency must facilitate access to ‘open access information’.  Open access information is information that all government agencies must provide free of charge under the GIPA Act.^[8] Additionally, different sectors such as the local council sector and the ministerial sector have specific requirements to provide additional information as open access.^[9]

An agency must facilitate access to open access information contained in a record by deleting information (if it is practicable to do so) where inclusion would otherwise result in an overriding public interest against disclosure.  The importance of accountability by agencies in meeting their open access responsibilities is also shown by the requirement to maintain a record of the open access information that they do not make available.^[10]

Authorised Proactive Release

Authorised proactive release means that agencies should look for ways to make information they hold available to the public. The most effective means for doing so will be publishing that information on an agency website so that it is publicly and freely accessible. Proactive release has other benefits too. The agency can take precautions to ensure that personal or confidential information is not improperly published by deleting information.^[11] This means that such information need not be an impediment to access. The information can be provided in a way that will make it more accessible or valuable to users. Links can also be drawn to other data sets or published information. 

Agencies are required by the GIPA Act to conduct at least an annual review of their program for proactively releasing information (s 7(3)). Agencies are also required to publish an agency information guide that identifies the various kinds of government information the agency holds and how it will be made publicly available (s 20(1)(d)-(f)). Agencies are encouraged to review datasets that can be prepared for proactive release.

The periodic review by agencies of their information guide and proactive release program should involve community and stakeholder input. Public input should be sought on the data categories that agencies both collect and publish, and how to make the data accessible and usable. Targeted consultation with organisations that regularly interact with the agency should form part of the public consultation.

An open data public consultation program aligns with another GIPA Act obligation on agencies to identify in their agency information guide the arrangements that exist to enable members of the public to participate in formulating agency policy and carrying out agency functions (s 20(1)(c)). The Information Commissioner has published a guide to assist agencies in seeking public input into the development and delivery of policies and services – Charter for Public Participation – a guide to assist agencies and promote citizen engagement (June 2018).

4 Considerations in favour of disclosure of information to open data

Other pathways under the GIPA Act can facilitate the release of information, including data upon request by an applicant. These are: the informal release of government information to a citizen who makes a request and access applications made in writing by citizens.^[12] In considering the release of data that is not in a format that is legally and technically open and freely accessible, it is important to consider other considerations that may permit its release. These considerations include deleting information that would act against disclosure or creating a new record.^[13]  The creation of a new record in relation to data may be achieved through a range of technical solutions including processes to de-identify information or the use of synthetic data sets.

The GIPA Act provides examples of public interest considerations in favour of disclosure of information.^[14]

1. Disclosure of the information could reasonably be expected to promote open discussion of public affairs, enhance Government accountability or contribute to positive and informed debate on issues of public importance.
2. Disclosure of the information could reasonably be expected to inform the public about the operations of agencies and, in particular, their policies and practices for dealing with members of the public.
3. Disclosure of the information could reasonably be expected to ensure effective oversight of the expenditure of public funds.
4. The information is personal information of the person to whom it is to be disclosed.
5. Disclosure of the information could reasonably be expected to reveal or substantiate that an agency (or a member of an agency) has engaged in misconduct or negligent, improper or unlawful conduct.

Importantly the considerations to be considered in releasing information are not limited.^[15]

Once a decision to provide access to information is made, an agency cannot impose conditions on the use or disclosure of that information/data.^[16]  Additionally agencies must maintain a disclosure logs to enable public access to information that the agency considers may be of interest to other members of the public.^[17] In this way the GIPA Act provides a mechanism to continuously supply information to the public that is not subject to conditions. When considered in the context of data the GIPA Act promotes on going public release of and access to data. 

Access decisions should be informed by the principles under the GIPA Act, other relevant guidance including policies, guidelines, contemporary social, economic and political issues and issues that might be unique to the information sought.^[18]

A number of relevant considerations that should be considered in releasing information are detailed below.

NSW Government Information Management Framework and Open Data Policy

The NSW Government adopted an Information Management (IM) Framework in 2013 as part of the NSW Government ICT Strategy. A revised framework was issued in late 2018. The purpose of the IM Framework is to set standards, policies, guidelines and procedures that enable government data to be managed in a secure, structured, consistent, accessible and usable manner.

A strong theme in the IM Framework is that data is an important government asset that, when widely shared and used, will enable strong governance, digital transformation and improved service delivery. Data can be an asset not only to the agency that holds it but to other agencies, business and the community. One of the six Principles of the IM Framework promotes information access and sharing:

Available and open to community and government

Information is publicly accessible and available in accordance with proactive release and open data principles, or shared within and between organisations to improve services, planning and innovation.

The IM Framework is supported by several policies and guidelines, some of which are referred to in this guideline. One such policy is the NSW Government Open Data Policy, first adopted in 2013 and revised in 2016.

The Open Data Policy provides a structure and principles for NSW Government agencies to adopt a Vision of ‘Better, faster, more open data’:

Better: Release better data in accessible, consumable formats with metadata and quality statements

Faster: Release data faster using automated processes, standard data categories, and trusted user models

More:   Release more data and make it discoverable through central portals

Three features of the Open Data Policy are notable. One is a set of six Open Data Principles (discussed in the next section of this guideline). The intention is that agencies will apply the Principles in publishing data sets on their own websites and on data.nsw.gov.au, which is the open data catalogue hosted by the Department of Customer Service. The number of published data sets in the catalogue has doubled annually, reaching 3,350 data sets by March 2019.

A second feature is that the Open Data Policy is linked to the work of the NSW Data Analytics Centre (DAC) that was established in 2015 and is supported by the Data Sharing (Government Sector) Act 2015. The DAC’s work involves providing guidance for data sharing between agencies as well as open data publication.

A third feature is that the Open Data Policy designates the NSW Information Commissioner as the NSW Open Data Advocate. This connects the Policy to the object and requirements of the GIPA Act. The Information Commissioner has a number of functions under the GIPA Act and the Government Information (Information Commissioner) Act 2009 (NSW) (GIIC Act) directed to ensuring compliance with the GIPA Act and its object.

Open Data Principles

The Open Data Principles form part of the NSW Open Data Policy, to guide agencies in publishing public sector data. The Principles reinforce the responsibility of agencies to have an active program of publishing data in a form that is accessible, usable and consistent across government. The Principles are summarised below, with additional commentary to link the Principles to the GIPA Act.

The Principles are complemented by the Rights and Responsibilities for Open Data Users, which is published as an Appendix to this guideline. The chart exemplifies that the responsibility of agencies to implement the six Open Data Principles should also be viewed as a correlative right belonging to open data users. For example, the responsibility of agencies to ensure that data is open by default and protected where required should also be viewed as a public right to access NSW Government data unless there is an overriding public interest against disclosure. Those rights are accompanied by responsibilities on open data users – for example, to recognise that there are legitimate reasons why some data cannot be released.

Following are the six Open Data Principles that outline the open data responsibilities of NSW Government agencies, linked to the GIPA Act and other IM Framework guidelines.

Open by default and protected where required

The presumption in the GIPA Act in favour of the disclosure of government information has a proactive element. Agencies should actively work to release data sets through data.nsw.gov.au and on linked agency websites. This pro-release focus is especially important when agencies are managing dynamic data sets that grow or change as new information is collected by agencies.

Data should be created and collected by agencies in ways that support downstream processing and release. Open data objectives should be built into program design and management – a policy of ‘open by design, public by default’.

An agency’s program design and information collection practices should identify information that must be protected and cannot be released – for example, to protect personal privacy, business confidentiality or state security.

Information labels and security classifications can be applied when data is collected, in line with the NSW Government Information Classification, Labelling and Handling Guidelines.

Potential obstacles to the release of data can also be addressed at the outset by agencies through the design of a data base or collection methods. For example, protected information can be redacted or de-identified, necessary consents or ethics approval for release can be obtained, and creative commons licensing can be applied.

Prioritised, discoverable and usable

The Open Data Policy urges that agencies prioritise high-value datasets for release. Stakeholder consultation and public and industry demand can help in identifying these data sets, as can an agency’s assessment of how open data may contribute to better service delivery.

Making data discoverable and usable requires expert technical evaluation, aided by resources such as the World Wide Web Consortium (W3C) open data standards, the NSW Standard for Data Quality Reporting and appropriate metadata practices.

Primary and timely

The Open Data Policy supports the release of primary data by agencies in the form it was collected, without modification or aggregation, except as required to safeguard confidential or personal data.

Timely release of the data may enhance its utility. The currency of the data should be recorded with timestamps or other information.

Well managed, trusted and authoritative

Data may be dynamic and conditional. Potential users should be alerted to possible limitations on the quality and reliability of data.

Data should be managed by agencies in accordance with relevant legal and other requirements. These include the NSW Data and Information Custodianship Policy, archival practices, and legislation protecting confidential, personal and privileged information.

Free where appropriate

Open data, as the term implies, is freely available, usually by download from a website. There may nevertheless be sound reason for an agency to impose a charge for specialised data services, particularly data services that are principally used for commercial purposes. Any such charges should be calculated by the agency on a cost recovery basis and be published.

This is consistent with one of the objects of the GIPA Act, which is ‘to facilitate and encourage, promptly and at the lowest reasonable cost, access to government information’ (s 3(2)(b)). The Act also lays down relevant charging principles for two of the four information pathways. Open access information, that agencies are required by the Act to make available, is to be made available free of charge either on an agency website or in some other form (s 6(3)). Information that an agency has decided to release proactively is to be made available either free of cost or at the lowest reasonable cost to the agency (s 7(1)).

Subject to public input

Agency decisions on open data – what and how to publish and in what form – should be made in consultation with the community, the research sector and industry. Community and industry feedback on the data should be published.

Agencies should also participate with government agencies in other jurisdictions to share and integrate data at the national level.

The open data governance framework

The adoption of an open data culture within government and by agencies requires a cultural shift – from controlling and shielding information to releasing it and allowing others to re-use it. This must be supported by a governance framework that is committed to an open data culture.

Three features of the governance framework are particularly important – public input and support, agency leadership, and dealing with privacy and other legal concerns about disclosure.

Public input and support

The importance of public input and support for open data practices has been discussed above. A key point was that business, researchers, community bodies and other stakeholders and individuals can provide helpful advice to an agency on the public sector data that is useful to them, how they like to access it, impediments to access, and uses and linkages for the data.

Public engagement is vital too in building public trust and confidence in open data practices. It is important to ease potential public concerns about government releasing data that derives from information collected from individuals and businesses.

An active strategy is required to encourage public participation. For example, this may be done through consultation and feedback channels, surveys, social media engagement, hackathons, analysis of popular data usage patterns, and expert research and advice. 

The modern view of open government is that it requires not only the release of government information but also an open dialogue model between government and the community. This added dimension is squarely recognised, for example, in the national action plans developed by many countries (including Australia) through their membership of the international Open Government Partnership.  This dialogue promotes openness through accountability and responsiveness and helps build community trust.

As noted above, public consultation on information release and related matters is required by the GIPA Act. The Information Commissioner has published a guide for agencies Charter for Public Participation – a guide to assist agencies and promote citizen engagement (June 2018).

Agency leadership

Agency leaders must promote and lead the open data commitment of their agency and build an open data governance culture.

This can be done in many ways – declaring the agency’s commitment to proactive release, appointing a senior officer as the ‘open data champion’, planning strategically to release data, regularly reviewing the agency’s open data practices, considering open data objectives when information is being collected and managed, allocating resources and skilled staff to open data tasks, encouraging staff to raise open data ideas, and ensuring consultation with stakeholders.

There is an expectation in the GIPA Act that agency leaders will play this role. The Act requires agencies – and, by implication, agency leaders (i.e. principal officers as defined in the GIPA Act) – to administer the Act and the four information pathways to promote the objects of the Act (s 15(a)). One of the objects is to authorise and encourage the proactive release of information (s 3(1)(a)).

Agencies and agency leaders are also required to have regard to guidelines issued by the Information Commissioner, such as this open data guideline.

A key role that agency leaders can play is to tackle head-on the cultural impediments that commonly exist within government to a more active open data approach. One impediment can be a heightened risk aversion based on privacy and security concerns, which is discussed in the next section.

Following are four other attitudinal impediments that have been identified in many studies, and the strategies for dealing with them:

• Agencies may be reluctant to release unrefined, incomplete or unstructured data that may not be of high or reliable quality. The understandable reason may be to guard against the risk of the data being misunderstood, misconstrued or misrepresented. A related concern is that an agency may be criticised for weaknesses in the data, misapprehensions to which it gives rise or inappropriate user reliance on the data.

This concern is taken up in two of the Open Data Principles, which direct agencies to release primary data in a timely manner, and to alert potential users to possible limitations on the quality and reliability of the data. Simply stated, a quest for faultless data could result in it never being released.

It is important to remember that the GIPA Act requires that a document to which access has been requested must be released in the form in which it was prepared, unless to do so would satisfy one of the categories set out in section 72(2) of the GIPA Act.

As to access generally, it would be inconsistent with open government to apply an information quality filter to block information from public release because it was, for example, unrefined or simplistic. Indeed, the GIPA Act states that the fact that disclosure of information might cause embarrassment or a loss of confidence in the Government is irrelevant. In addition, whether disclosure of information might be misinterpreted or misunderstood by any person is irrelevant and must not be taken into account in determining whether there is an overriding public interest against disclosure of government information (s 15). A core assumption of open government is that disclosure can facilitate an open dialogue that will lead to more polished or perfected ideas being developed.

• Agencies can be averse to allowing the private sector to capitalise freely on valuable government data. Some may object to the idea that a business that has not contributed to the preparation of government data can exploit it for commercial profit without any direct return to government.

Some allowance is made for this view in the Open Data Principles which accept that a cost recovery charge can be imposed for access to specialised data services. The GIPA Act also allows processing costs to be charged for access requests.

More generally, however, it is counter to the philosophy of open government and open data to suppress the opportunity for the private sector to realise a commercial potential from information that is a public asset. It has been assembled with public resources and should be available for community access and re-use.

Nor can it be assumed that there is no correlative public benefit when public sector data is made freely available. Information sharing can have direct economic and social benefits.

Two common examples are the public efficiencies that stem from the release of government transport and meteorological data. In short, the task of government is easier if the economy is growing and the community is lively and engaged. 

• The release of some data sets could have unintended consequences. Common examples are that the release of crime statistics may expose law enforcement weaknesses that can be exploited; corruption or anti-social behaviour may flourish if regulatory patterns are known; and endangered or protected species may be more at risk if their habitat or situation is revealed.

Those examples illustrate that some data sets cannot easily be released and may need careful preparation and cleansing before release. Open data poses risks that must be managed. Special preparation of the data set to enable its release may have the beneficial consequence that the community takes a special interest in working with government to pursue the law and order, business regulation or environmental protection objectives that were a cause for concern.

• There can be special interest objections to the release of specific data sets. The objection may come from political or business leaders or practitioners arguing that the data presents an inaccurate picture of program performance or business or professional conduct. Researchers may similarly object to the release to a broader research community of a data set they played a special role in developing.

Concerns of this nature illustrate that open government can be contentious. Special interest objections to release must be considered in applying the public interest test.  That is easier to do when there is a strong underlying commitment to an open data culture.

Privacy and other legal concerns about disclosure

Numerous laws and legal doctrines control how information is to be managed, used and disclosed. Sound information management is both a legal and a cultural requirement.

A central open data concern is the need to safeguard personal privacy. A great deal of information held by government contains personal information – such as names, addresses, occupations, and health and family details. Indeed, the population picture that aggregated data can reveal makes it particularly valuable as open data.

There is a clear legal obligation on government to safeguard the confidentiality of personal information. This obligation partly stems from statutory secrecy provisions and specialist statutes such as the Privacy and Personal Information Protection Act 1998 (PPIP Act) and the Health Records and Information Privacy Act 2002 (HRIP Act).

There is an equally strong community expectation that government will safeguard personal information against inappropriate and unauthorised release. People are understandably concerned that there may be adverse personal consequences if personal information is released without their knowledge or consent. A related and deeply held concern is that data matching capability enables apparently innocuous items of information to be linked and traced to individuals.

A frequent observation in independent studies is that privacy and related legal concerns have instilled a risk aversion culture in government that is a major impediment to an open data commitment being more actively embraced. Opinions will understandably differ on that assessment. There are nevertheless three matters that agencies should bear in mind.

First, the legal focus is primarily aimed at regulating how information can be released, rather than preventing release altogether. For example, the GIPA, PPIP and HRIP Acts regulate the disclosure of ‘personal information’ and ‘health information’, which is defined as information or an opinion about an individual whose identity is apparent or can reasonably be obtained from the information or opinion.^[19] Personal information that has been de-identified or anonymised is no longer personal information, and falls beyond the scope of those and similar laws. This interpretation has also been confirmed in guidance by the United Kingdom Office of the Information Commissioner.^[20]

Secondly, the central thread of the GIPA Act is that a balance must be struck between confidentiality and public release. A declared object of the Act is that ‘access to government information is restricted only when there is an overriding public interest against disclosure’ (s 3(1)(c)). The Act spells out this public interest test and states that there is an overriding public interest against disclosure ‘if (and only if) there are public interest considerations against disclosure and, on balance, those considerations outweigh the public interest considerations in favour of disclosure’ (s 13). The Act identifies an exhaustive number of matters that qualify as public interest considerations against disclosure.

The public interest balancing test applies to each of the four information pathways in the Act. The IPC has provided statutory guidance to assist in decision making that involves personal information.^[21]

The public interest balancing test is reinforced in other provisions of the GIPA Act that qualify the weight that can be placed on some considerations that support non-disclosure. For example, law enforcement and security considerations generally apply only if disclosure would ‘prejudice’ a law enforcement activity; the same concept of ‘prejudice’ applies to the protection of business and financial interests, the effectiveness of research and the environment and heritage; and a qualification of ‘damage’ or ‘prejudice’ applies to disclosure related to economic management (s 14).

A balancing test is present also in provisions of the PPIP Act that qualify the operation of the Information Protection Principles in that Act. Importantly, the Act provides that a public sector agency to which the Act applies is not required to comply with the information protection principles when using or disclosing personal information that ‘is reasonably necessary for the purpose of research, or the compilation or analysis of statistics in the public interest, and … reasonable steps are taken to de-identify the information’ (s 27B). Other exemptions from the Information protection principles apply to a use of information to which an individual has consented (s 17) and to a disclosure of information that an individual is reasonably likely to be aware of (s 18(1)(b)).

Thirdly, expert technical guidance is available from many sources in Australia and internationally on matters such as de-identification and risk management. The Information and Privacy Commission published research in 2017 to examine the Conditions enabling Open Data and Promoting a Data Sharing Culture.^[22]  This resource provides contemporary case studies and guidance regarding enablers including operational and technical enablers that support Open Data.

The Information and Privacy Commission urges agencies to undertake a privacy impact assessment of new projects. This can apply to the publication of new data sets. The Commission has published a Guide to Privacy Impact Assessments.

Agencies are also required by the PPIP Act to prepare a privacy management plan dealing with the protection of personal information by the agency (s 33). The agency’s open data program can be an element of the plan.

Conclusion

Within NSW there is a well-established and operative legislative mandate to open government information, including data. That mandate, coupled with a leadership culture that promotes openness and responsible stewardship of information, is crucial to harnessing the benefits of opening data. Decision makers play a vital role in upholding the rights of New South Wales citizens. In an increasingly digital environment the application of this guidance by agencies will support those rights and importantly promote open government to citizens.