Finding & Merging Duplicates

Unfortunately, one of the issues with a large patient database can be the creation of duplicate records. In fast-paced environments with data flowing from multiple sources, these duplicates can accumulate to significant levels.

This is where the Duplicate Patient Check becomes invaluable. Using AI-generated weighting algorithms, it scans all patient records and assigns each one a likelihood score indicating how likely it is to be a duplicate.

Once you have selected the records you believe are duplicates, the system can merge all associated data into a single primary patient record (typically the most recently created one). The real advantage of this process is that it intelligently handles missing data fields in the primary record, consolidates all appointments and financial information, and merges clinical notes and patient documents. It also redirects any live e-Docs so they correctly link to the final master record.

You will find the Duplicate Patient Check in ClinicOffice under Home > Patients > Data Cleansing. (you must have a sufficient access level)

This will open the Duplicate Patients window.

Before searching for duplicates, you can adjust the percentage probability selector. The default value is 70%, but you can choose any value from 60% to 90%.

You also have the option to show additional patient details. Enabling this will display the details of the two records that the system has identified as potential duplicates within the probability range you selected.

Once you are happy with these settings, simply click Search for Duplicates to begin the process.

After the search has completed, you will be presented with a list of records that may be duplicate patients. If you are working with a large number of records, this process may take some time, as the system performs tens of thousands of calculations and checks.

In the example below, we have two possible duplicates (four patient records), but you will likely have many more the first time this is run.

If we look at the first duplicate found it is highly likely this a duplicate record as the name, address and date of birth all match.

If we tick the select box against this record and click Merge Selected Patients the most recent record for Lisa will be retained as the master record and relevant details merged from the older record across.

Please note!

We strongly recommend performing record merges outside of normal working hours to ensure that patients’ records are not being edited while these updates are being carried out.
Although you can select all records we recommend that possible duplicates are reviewed individually before selecting them.
You will be asked to type YES before merging — since this process is NOT REVERSIBLE!

If we now look at the notes against Lisa’s record you will note the following information has been added.

Refining Our Matches

The Duplicate Patients window has 8 columns that can be filtered.

There are many scenarios where these filters can help you narrow down the records you want to merge. One example is with family members: they are likely to share the same last name and address, but their dates of birth will almost always be different. Filtering by date of birth can therefore help ensure that only true duplicate records are identified.

Text Field

The first Description column has a text based filter which enable you to filter as either a tick record box against records found using the search or the text filter which offers various criteria.

Value Fields

The Probability, Name Score and Addr Score columns are are all values fields and can be filtered using value sliders on min / max input fields.

Match Fields

The DOB, Email, Mobile and Clinic Match are state fields. Y for a match and Blank for no match. You can filter on three states of All Records, Matches Only or No Match.

Multiple duplicates

It is possible for a patient’s record to be duplicated more than once. The system will identify all duplicates as long as they meet the selected criteria.

For example, let’s say Jim Jones has registered three times, creating record IDs 101, 102, and 103.

The system will correctly detect all three duplicate combinations for Jim:

101 ↔ 102
101 ↔ 103
102 ↔ 103

If all three records are selected, they will be merged. However, you may see a message stating that one or more problems were encountered and you will be given the option to view the log file.

The log will likely contain a message such as:
“Merge Failed! Likely one of the records has just been deleted (or merged).”

This is expected behavior, and the logic is as follows:

Record 101 is merged into 102.
Record 101 into 103 is not possible as 101 was merged into 102 and no longer exists—which triggers the error message.
Record 102 is merged into 103 (the latest record of all three selected).

Note:
The number of duplicate combinations increases rapidly as more duplicate records exist. For example, if there are four duplicate records, the system will display six combinations:

Record 1 ↔ 2, Record 1 ↔ 3, Record 1 ↔ 4, Record 2 ↔ 3, Record 2 ↔ 4, Record 3 ↔ 4 : 6 rows for 4 records

This is expected behavior, as the system evaluates every possible pairing when identifying duplicates.

Duplicate Patient Records