Medical imaging data for AI: DICOM de-identification, annotation formats and labeling tools

The short answer

To send medical images to a labeling team, de-identify the DICOM with a named PS3.15 Annex E profile and options, check the pixels for burned-in text and faces, and agree on the label format before work starts. Use DICOM SEG if labels must stay tied to the source study, RTSTRUCT only when a radiation-therapy system needs it, and NIfTI for research pipelines. 3D Slicer, MONAI Label and CVAT suit different jobs: 3D Slicer for expert 3D segmentation, MONAI Label for AI-assisted labeling of CT and MRI at scale, and CVAT for 2D images and video. CVAT does not read DICOM natively.

Removing tags is not the same as de-identifying a study. The DICOM standard says the profiles “should be part of” a de-identification process, not a replacement for it.

De-identification: DICOM PS3.15 Annex E

DICOM PS3.15 Annex E (Attribute Confidentiality Profiles) defines action codes for each attribute: D (replace with a dummy value), Z (zero length or dummy), X (remove), K (keep), C (clean) and U (replace the UID consistently across related instances). It also requires the de-identifier to set Patient Identity Removed (0012,0062) to YES and to record what was done in De-identification Method (0012,0063) and the Method Code Sequence (0012,0064), using codes from CID 7050. Ask for that record. It tells the labeling team, and any later auditor, exactly which profile and options were applied.

The same standard notes that de-identifying the attributes “does not imply de-identification of the Information Object”. Pixels, private tags and free text can still carry identity.

How this lines up with HIPAA

Under 45 CFR 164.514, data is de-identified either by Expert Determination (a qualified expert finds the risk “very small” and documents the methods and results) or by Safe Harbor (removing 18 kinds of identifiers and having no actual knowledge that the rest could identify someone). The Safe Harbor list includes all elements of dates except the year, device identifiers and serial numbers, biometric identifiers, “full face photographic images and any comparable images”, and any other unique identifying number, characteristic or code. The HHS OCR de-identification guidance says an identifier must be removed wherever it appears, including free text, and that neither method brings risk to zero.

Our reading, not a legal opinion: some DICOM options sit uneasily with Safe Harbor. Retain Full Dates keeps dates that Safe Harbor removes, and Retain Device Identity keeps serial numbers that Safe Harbor lists. If a project needs them, plan for Expert Determination or another route your privacy officer approves. As one public example, TCIA submission and de-identification overview describes TCIA's process: the Basic Profile plus Clean Pixel Data, Clean Descriptors, Retain Longitudinal Temporal Information with Modified Dates, Retain Patient Characteristics and Retain Safe Private. TCIA says this process meets Safe Harbor. It shifts dates by a random offset that keeps intervals and has curators check every image for burned-in PHI.

Sources: PS3.15 Annex E, PS3.15 E.3 (options), E.3.9 Retain UIDs, E.3.10 Retain Safe Private, 45 CFR 164.514, HHS OCR de-identification guidance, TCIA submission and de-identification overview.

Burned-in text and faces

Tag rules don’t touch the pixels. PS3.15 notes that CT images “do not normally contain such burned in annotation, whereas Ultrasound images routinely do”. Screen captures, scanned documents and secondary-capture images deserve the same check. Under Clean Pixel Data the stored pixel values must be changed. Covering them with an overlay or shutter is not enough, because a receiving system may ignore it.

Faces are the other pixel-level risk. E.3.2 Clean Recognizable Visual Features says a 3D rendering of high-resolution head and neck imaging may be enough to identify someone. In a published test with 84 volunteers, face-recognition software was given each person’s photographs and faces reconstructed from research head MRI; for 70 of 84 (83%), the correct participant’s MRI was the top match (Schwarz et al., N Engl J Med 2019;381(17):1684–1686). The authors note that defacing software is rarely used because it can degrade some brain measurements. Decide early whether head scans will be defaced, and whether your labels need the facial region at all.

Sources: PS3.15 Annex E, E.3.2 Clean Recognizable Visual Features, Schwarz et al., N Engl J Med 2019;381(17):1684–1686.

Annotation formats: DICOM SEG, RTSTRUCT and NIfTI

Two practical points. First, keep geometry: DICOM SEG and RTSTRUCT carry the Frame of Reference, and a NIfTI mask is only usable if its affine matches the image it was drawn on. Second, NIfTI has no patient fields, but its header has an 80-character free-text description field (“any text you like”), an auxiliary file name field and optional extensions. In our view these are worth checking for identifiers after conversion. For writing DICOM SEG from research formats, the open-source libraries dcmqi (Herz et al., Cancer Res 2017;77(21):e87–e90 (dcmqi)) and highdicom (Bridge et al., J Digit Imaging 2022;35(6):1719–1737 (highdicom)) are described in the peer-reviewed literature.

Sources: PS3.3 A.51 Segmentation IOD, C.8.20.2 Segmentation Image Module, C.8.20.4 Segment Description, PS3.3 A.19 RT Structure Set IOD, C.8.8.5 Structure Set Module, C.8.8.6 ROI Contour Module, NIfTI-1, NIfTI-1 header fields, NIfTI-2, 3D Slicer Segmentations module, 3D Slicer data loading and saving.

Labeling tools: 3D Slicer, MONAI Label and CVAT

MONAI Label’s README says 3D Slicer is its most tested client for radiology studies. For CVAT, the repository’s CVAT dicom_converter utility turns DICOM into ordinary images. Our reading of its script is that it rescales pixel values into PNG, so treat any CVAT mask on converted slices as 2D work that needs to be mapped back to the original geometry.

Sources: 3D Slicer Segment Editor, 3D Slicer Segmentations module, 3D Slicer DICOM module, MONAI Label documentation, MONAI Label on GitHub, Diaz-Pinto et al., Med Image Anal 2024;95:103207 (MONAI Label), CVAT overview, CVAT tasks page, CVAT dicom_converter utility.

What to send an imaging labeling team

Our practice summary, based on the sources above:

• De-identified DICOM, with its record. Name the profile and options used, and keep (0012,0062), (0012,0063) and (0012,0064) populated.
• A pixel check result. Say which series were screened for burned-in text, and whether head scans were defaced.
• Consistent pseudonyms. Keep replacement UIDs and patient IDs consistent across related series, and keep the mapping table at your site. Don’t leave Patient Name, Patient ID or Series Instance UID empty: 3D Slicer warns that empty or missing values in these fields are a common cause of loading failure.
• A label specification. List the structures, with codes where you have them (DICOM SEG uses coded category and type), say whether segments can overlap, and give the output format (SEG, RTSTRUCT or NIfTI) and naming rules.
• An annotation guideline and a review plan. Include edge cases, who adjudicates disagreements, and how agreement will be measured.
• The tool decision. Name the viewer and version, any AI pre-labeling model, and whether its output is recorded as AUTOMATIC or SEMIAUTOMATIC in the segment metadata.
• The legal and access basis. State the de-identification route (Safe Harbor or Expert Determination), the agreements in place, and where data may be stored.

Common pitfalls

• Over-deleting. Deleting every private tag is common, but it removes scientifically useful data along with PHI (Moore et al., RadioGraphics 2015;35(3):727–735). TCIA asks submitters not to use PACS de-identification for this reason.
• Under-checking pixels. Ultrasound, secondary captures and screenshots can carry names and dates in the image itself.
• Losing geometry. Converting to PNG or re-slicing without keeping the affine or Frame of Reference makes masks hard to map back.
• Label maps that can’t overlap. In a LABELMAP-type DICOM SEG, each pixel holds one segment number, so overlapping structures need BINARY or FRACTIONAL segments.
• Dates. Shifted dates keep intervals, but the original dates are gone. Decide up front whether the analysis needs real calendar time.

Key references

• DICOM PS3.15 Annex E (Attribute Confidentiality Profiles); PS3.15 E.3 (options); CID 7050
• PS3.3 A.51 Segmentation IOD; PS3.3 A.19 RT Structure Set IOD
• NIfTI-1; NIfTI-1 header fields; NIfTI-2
• 3D Slicer Segmentations module; 3D Slicer DICOM module; 3D Slicer data loading and saving; Fedorov et al., Magn Reson Imaging 2012;30(9):1323–1341 (3D Slicer)
• MONAI Label documentation; MONAI Label on GitHub; Diaz-Pinto et al., Med Image Anal 2024;95:103207 (MONAI Label)
• CVAT overview; CVAT dicom_converter utility
• 45 CFR 164.514; HHS OCR de-identification guidance
• TCIA submission and de-identification overview; TCIA De-identification Knowledge Base (wiki)
• Schwarz et al., N Engl J Med 2019;381(17):1684–1686; Moore et al., RadioGraphics 2015;35(3):727–735; Herz et al., Cancer Res 2017;77(21):e87–e90 (dcmqi); Bridge et al., J Digit Imaging 2022;35(6):1719–1737 (highdicom)

Related resources

• Medical data labeling
• Expert clinical labeling
• HIPAA compliance consulting
• Medical data annotation guideline template
• Inter-rater agreement for medical data labeling
• Choosing a medical data labeling approach
• ECG annotation and EC57 evaluation
• Medical AI test set sample size
• Where to get health data for medical AI