SEMI-SUPERVISED VARIATIONAL AUTOENCODER FOR CELL

FEATURE EXTRACTION IN MULTIPLEXED

IMMUNOFLUORESCENCE IMAGES
Piumi Sandarenu1, Julia Chen2,3,4, Iveta Slapetova3, Lois Browne3, Peter H. Graham3,4,
Alexander Swarbrick2,5, Ewan K.A. Millar4,6,7,8, Yang Song1, Erik Meijering1 897
1 School of Computer Science and Engineering, University of New South Wales., 2 Cancer Ecosystems Program, Garvan Institute of Medical Research., 3 Cancer Care Centre, St George Hospital.,
4 St. George & Sutherland Clinical School, UNSW Sydney., 5 School of Clinical Medicine, Faculty of Medicine, UNSW Sydney., 6 Dept. of Anatomical Pathology, NSW Health Pathology, St. George Hospital.,
7 Faculty of Medicine & Health Sciences, Sydney Western University., 8 University of Technology Sydney.

Abstract Background
Current multiplexed immunofluorescence (mIF) image Tissue microarrays (TMAs) with each having 9 channels (corresponding to 8 biomarker stains PD1, CD140b,
analysis pipelines depend on cell feature representations CD146, Thy1, PanCK, CD8, α-SMA, CD31 and DAPI as a nuclear counterstain), and an additional
characterised by simple morphological and stain intensity- autofluorescence channel that are all registered. Each TMA contains cores of approximately 1.25 mm in
based metrics derived from nuclei and cell segmentation diameter scanned at 0.5 µm/ pixel resolution. Dataset consists of 18 TMA slides having 1,093 TMA cores
masks generated using simple statistical and machine collected from a cohort of
learning-based tools. Due to the noisy background and 450 breast cancer patients
extreme stain variability in mIF images, such methods are not belonging to multiple
reliable descriptors of cells. We propose a deep learning- molecular subtypes (Luminal
based cell feature extraction model using a variational A, B, HER2, and TNBC).
autoencoder (VAE) with supervision using a latent subspace to
extract cell features in mIF images. We perform cell Example mIF image patches.
phenotype classification using a cohort of more than 44,000 (a) Expanded views of cells
mIF cell image patches extracted across 1,093 tissue in a TMA core.
microarray cores of breast cancer patients. (b) Each channel of
the mIF image captures
the presence of a
Motivation and contribution respective biomarker.

• Cell phenotype labels can be used as a supervision signal to

enhance generalisability and facilitate learning of cell Proposed architecture
phenotype-related features.
• However, obtaining cell phenotype labels annotated by
expert pathologists is a costly and time-consuming task
with potentially high inter- and intra-observer variability. A subspace z is
• In addition, cell phenotype labels can be spuriously extracted from the
correlated with the presence (or absence) of respective latent space z
biomarkers which can limit the learning of meaningful where the latent space
feature representations due to the inherent inductive is now represented as
biases in neural networks. z = (z − z′, z′). The
• We use the labels automatically generated using built-in classifier Cγ uses the
tools of QuPath [1] and propose to use the full latent subspace μφ(x)′
representation for image reconstruction and a latent for classification,
subspace for the joint supervision task.
while the full latent space is used for reconstruction. The subspace μφ(x)′ generated by Eφ can be used to
predict the class label probabilities using cross-entropy loss as p(y|x) = lCE (y, Cγ (μφ(x)′)) and the combined
objective function L(θ, φ, γ) is defined as,
Ethics declaration
Ethical approval provided by the South Eastern Sydney Local Health District
Human Research Ethics Committee at Prince of Wales Hospital (2018/ETH00138
and HREC 96/16). All methods were performed in accordance with the relevant The size of z' is fixed and dependent on the complexity of data and level of correlation between the label and
institutional guidelines and regulations. input. In our study, we use a proportion of 1/8 for z'/z.

Experimental results

• The dataset consists of n = 44,400 randomly selected cell detection representing six cell phenotypes (tumour, iCAFs, myCAFs, T-cells, dPVLs, and exhausted T-cells) in equal
proportion across all TMA slides.
• We use the predicted cell centres as approximations of the actual cell centres and extract cell patches of size 48 × 48 × 9 pixels (corresponds to a tissue area of 24 × 24 μm2) [2].

[3]
[1]
[4]

Reconstruction and classification References

results. (a, b) Reconstruction
results of our model accurately [1] P. Bankhead et al., “QuPath: Open source software
captures cellular features while for digital pathology image analysis,” Scientific Reports, vol.
7, no. 1, pp. 1–7, 2017.
minimising noise. (c) Confusion [2] B. Shashni et al., “Size-based differentiation of cancer
matrix shows our model is and normal cells by a particle size analyzer assisted by a
capable of classifying all cell cell-recognition pc software,” Biological and Pharmaceutical
phenotypes with high accuracy. Bulletin, vol. 41, no. 4, pp. 487–503, 2018.
[3] D. P. Kingma and M. Welling, “Auto-encoding variational
Exhausted T-cells are more prone Bayes,” Proceedings of the International Conference on
to be categorised as T-cells which Learning Representations (ICLR), pp. 1–14, 2014.
may be due to the variation in [4] L. Le, A. Patterson, and M. White, “Supervised
staining strength of CD8+ and autoencoders: Improving generalization performance with
unsupervised regularizers,” Advances in Neural Information
PD1+ biomarkers. Processing Systems, vol. 31, pp. 107–117, 2018.