Phase 1: Standardisation

All datasets are run through the same standardisation process to ensure that variables are treated the same during the linkage process. This can involve:

  • Removing invalid or incorrect names (“Unknown”, “Baby”, “Missing”)
  • Converting all Sex/Gender variables to a coded value (“Male”, “Boy” -> “1”)
  • Converting all birthdates to the format of YYYY-MM-DD
  • Padding out values such as UR Number to a full 10 digits (“584” -> “0000000584”)
  • Utilising a Geocoder service to standardise all addresses against an accepted list of all Australian addresses

Phase 2: Name Phonetic Hashing

In this phase, hash values are generated for First and Last Name values using the Soundex algorithm. These hash values aim to overcome spelling differences and enable matching of names when the spelling slightly differs (Fuzzy matching). These values were used as blocking values during the Probabilistic linkage phase described below.

Phase 3: Linkage Criteria

Following the Data Standardisation step, all datasets are linked to one another in the Victorian Linkage Map using a combination of deterministic and probabilistic criteria. Deterministic refers to links formed between records only where values match exactly. Probabilistic refers to links formed where scores between records are generated based on matching and non-matching information, and where a score passes a certain threshold, it is considered a match.

Phase 4: Specified Dataset Lookup tables

The linkage process outputs a table that lists a Person Key for all the records in the input datasets. This table allows all records in the input datasets to be linked together.

This table only contains IDs (i.e. no personal information) to protect people’s privacy and these IDs can then be used by the CVDL Content Data Team to extract the relevant de-identified content records that match the input datasets.

Phase 5: Linkage Analysis

Following linkage – a number of data quality analyses are undertaken:

  1. The overlap between datasets is calculated, and where possible – is compared to previous results to ensure consistency. Validity checks are also performed on this overlap (e.g., does the Birth Mothers dataset have a high overlap with the Perinatal Mothers dataset? Does the Births Baby dataset have higher overlap with Child Protection/Youth Justice dataset than datasets with only adults?)
  2. Manual sampling of the data is undertaken to check for false positive rates. All records associated with a person key are examined and checked for consistency against an approximate 2% expected error rate.
  3. The unmatched population is investigated to check there is no bias or errors causing records to not match. For example – gender and birthdates should be equally distributed, if many records are located interstate there is a general expectation for them to not match etc. This helps identity if a specific error is causing records to not match.
  4. Changes in familial structure is also examined to ensure no significant variations occur which may indicate a problem in how children-father-mother relationships are being formed.