GAN with Skip Patch Discriminator for Biological Electron Microscopy Image Generation 1 Introduction 1.1 Dataset (Drosophila) 2 Methodology 2.1 Skip patch discriminator 2.2 Mitochondria and cell membrane mask (Fig.  1 -A.i) 2.3 EM Image of cellular structure (Fig.  1 -A.ii) 3 Results and conclusions GAN with Skip Patch Discriminator for Biological Electron Microscopy Image Generation Nishith Ranjon Roy 1 Nailah Rawnaq 1 Tulin Kaman 1 1 Department of Mathematical Sciences University of Arkansas Fayetteville AR 72701 USA Abstract Generating realistic electron microscopy (EM) images has been a challenging problem due to their complex global and local structures. Isola et al. proposed pix2pix, a conditional Generative Adversarial Network (GAN), for the general purpose of image-to-image translation; which fails to generate realistic EM images. We propose a new architecture for the discriminator in the GAN providing access to multiple patch sizes using skip patches and generating realistic EM images. Keywords— generalized adversarial network, biological image generation,  multi-scale patch 1 Introduction In classical GAN, the adversarial modeling framework consists of a generator G 𝐺 G italic_G and a discrimination D 𝐷 D italic_D model. A generator model mimics the data distribution and a discrimination model determines if an observation is from generated or real data  [ 1 ] . The pix2pix model  [ 2 ] is suitable for image-to-image translation of real-life objects like cars, roads, and buildings but not for EM images. The reason is that EM images are high-resolution microscopy images containing complex structures consisting of intricate, asymmetric and sparse patterns which are different from any regular objects. In addition, the limited data availability for EM images is a concern to train a neural network. We address the challenges and propose a new approach for training GAN models in settings with limited data availability and the presence of both global and local structures for generating realistic biological images. 1.1 Dataset (Drosophila) We run our studies on the Drosophila dataset  [ 4 ] which consists of 20 20 20 20 ( 1024 × 1024 1024 1024 1024\times 1024 1024 × 1024 resolution) images of the mitochondria and cell membrane masks (see Fig.  1 -A.i), and EM images (see Fig.  1 -A.ii). Figure 1: (A) The process of generating masks ( X 𝑋 X italic_X ) and EM images ( Y 𝑌 Y italic_Y ). The generator and discriminator are denoted by G 𝐺 G italic_G and D 𝐷 D italic_D respectively. The first stage of this process involves generating masks (iii) from random noise ( z 𝑧 z italic_z ), while the second stage involves using these generated masks to produce EM images (iv). (B) The principal component analysis (PCA) plot shows the similarity among the distribution of the generated mask with the real mask. The red points comparatively far on the right side indicate that the generated mask with 16 × 16 16 16 16\times 16 16 × 16 patch does not show any alignment with the true mask. On the contrary, the mixture in the center of three different points suggests that the generated mask with 70 × 70 70 70 70\times 70 70 × 70 patch, skip patch (ours), and real mask have approximately the same type of distribution. (C) The four images show the real and generated masks via different methods. A fixed 6000 training iterations (or epochs) has been performed for all three methods. 2 Methodology The process of generating EM images consists of two steps. The first step is to create the mitochondria and cell membrane masks, and the second step is to create EM images of tissue organization. In this paper, we present a new architecture for the discriminator in training GAN models. In both steps of the GAN training, the skip patch discriminator (proposed here) and the patch discriminator (initially proposed by [ 2 ] ) are used and compared. In Fig.  1 -A, we show the overall process of generating masks and images. 2.1 Skip patch discriminator The patch-based discriminator lacks the capability of simultaneously accessing both the global and local structures of the generated image. The discriminator outputs a matrix where each element of that matrix is calculated by applying convolution on the input image/tensor. The backtracking of the calculation for one pixel in the final layer of the patch-based discriminator is responsible for 70 × 70 70 70 70\times 70 70 × 70 pixels. Fig.  2 -A shows the 70 × 70 70 70 70\times 70 70 × 70 patches in the input tensor/image which allows the discriminator to evaluate the generated image on a 70 × 70 70 70 70\times 70 70 × 70 window. As shown in Fig.  1 -C, the patch size 70 × 70 70 70 70\times 70 70 × 70 captures the global structure but fails to capture the local structure and results in holes inside some of the mitochondria. Cutting down the depth of the number of layers from four to two reduces the patch size from 70 × 70 70 70 70\times 70 70 × 70 to 16 × 16 16 16 16\times 16 16 × 16 which captures the local structure but fails to capture the global structure and results in unrealistic symmetrical repetitive shapes. Therefore, we propose a skip patch discriminator (see Fig.  2 -B), where, due to the skip connections, each pixel of the final layer’s output tensor/matrix has access to the patch sizes of 16 × 16 16 16 16\times 16 16 × 16 , 20 × 20 20 20 20\times 20 20 × 20 , 32 × 32 32 32 32\times 32 32 × 32 , and 70 × 70 70 70 70\times 70 70 × 70 . Consequently, this discriminator is now capable of detecting trends, symmetry, and asymmetry across a range of different window sizes, from a minimum of 16 × 16 16 16 16\times 16 16 × 16 to a maximum of 70 × 70 70 70 70\times 70 70 × 70 parches. See Fig.  1 -C “skip patch” and Fig.  3 -B “cGAN S.Patch”. Figure 2: (A) The discriminator architecture proposed by [ 1 ] . (B) The new discriminator architecture proposed in this paper. (C) U-Net architecture for the generator in the conditional GAN (cGAN) EM images generation part. 2.2 Mitochondria and cell membrane mask (Fig.  1 -A.i) Generator : The generator is defined as G : N ⁢ ( 0 , 1 ) 8 × 8 × 8 → ( 0 , 1 ) 3 ∗ × 512 × 512 : 𝐺 → 𝑁 superscript 0 1 8 8 8 superscript 0 1 superscript 3 512 512 G:N(0,1)^{8\times 8\times 8}\to(0,1)^{3^{*}\times 512\times 512} italic_G : italic_N ( 0 , 1 ) start_POSTSUPERSCRIPT 8 × 8 × 8 end_POSTSUPERSCRIPT → ( 0 , 1 ) start_POSTSUPERSCRIPT 3 start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT × 512 × 512 end_POSTSUPERSCRIPT where 3 ∗ superscript 3 3^{*} 3 start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT represents the number of channels corresponding to the mitochondria (red) and cell membrane (green) mask, and the background (blue), and N ⁢ ( 0 , 1 ) 𝑁 0 1 N(0,1) italic_N ( 0 , 1 ) is a Gaussian distribution with a mean 0 0 and a standard deviation 1 1 1 1 . The architecture of the model is as follows: C ⁢ 256 ⁢ N ⁢ A → C ⁢ 256 ⁢ N ⁢ A → C ⁢ 128 ⁢ N ⁢ A → C ⁢ 64 ⁢ N ⁢ A → C ⁢ 32 ⁢ N ⁢ A → C ⁢ 3 ⁢ A → 𝐶 256 𝑁 𝐴 𝐶 256 𝑁 𝐴 → 𝐶 128 𝑁 𝐴 → 𝐶 64 𝑁 𝐴 → 𝐶 32 𝑁 𝐴 → 𝐶 3 𝐴 C256NA\to C256NA\to C128NA\to C64NA\to C32NA\to C3A italic_C 256 italic_N italic_A → italic_C 256 italic_N italic_A → italic_C 128 italic_N italic_A → italic_C 64 italic_N italic_A → italic_C 32 italic_N italic_A → italic_C 3 italic_A . Here, C i 𝑖 {i} italic_i denotes the transposed convolution with the number of output channels i 𝑖 i italic_i (kernel 4, stride 2), followed by the instance normalization (N) and ReLU activation (A). In the final layer, softmax activation is used without a normalization layer. Discriminator : A skip patch discriminator resulted in a realistic-looking mask compared to a patch discriminator (see Fig.  1 -C). All of these models have been trained on a fixed number (6000) of epochs and the setup with skip patch discriminator starts to converge within 50% of the total epoch. Fig.  2 -B shows the new architecture of the network where the discriminator is defined as D : ( 0 , 1 ) 3 × 512 × 512 → { 0 , 1 } 64 × 64 : 𝐷 → superscript 0 1 3 512 512 superscript 0 1 64 64 D:(0,1)^{3\times 512\times 512}\to\{0,1\}^{64\times 64} italic_D : ( 0 , 1 ) start_POSTSUPERSCRIPT 3 × 512 × 512 end_POSTSUPERSCRIPT → { 0 , 1 } start_POSTSUPERSCRIPT 64 × 64 end_POSTSUPERSCRIPT . Training : The tensor containing the information of the mask was resized down to 512 × 512 512 512 512\times 512 512 × 512 resolution. For augmentation, random vertical and horizontal flipping were applied. Additionally, cropping with 90% of the area (keeping the aspect ratio fixed) was applied to the mask tensor to address overfitting. The objective function of GAN for mask generation is min ( G ) ⁡ max ( D ) ⁡ V ⁢ ( G , D ) = ℒ ⁢ ( Prob= ⁢ D y ⁢ ( G x ⁢ ( z ) ) , Target= ⁢ { 0 } 64 × 64 ) + ℒ ⁢ ( Prob= ⁢ D y ⁢ ( X ) , Target= ⁢ { 1 } 64 × 64 ) subscript 𝐺 subscript 𝐷 𝑉 𝐺 𝐷 ℒ Prob= subscript 𝐷 𝑦 subscript 𝐺 𝑥 𝑧 Target= superscript 0 64 64 ℒ Prob= subscript 𝐷 𝑦 𝑋 Target= superscript 1 64 64 \min_{(G)}~{}\max_{(D)}~{}V(G,D)={{\cal L}(\text{Prob=}D_{y}(G_{x}(z)),\text{% Target=}\{0\}^{64\times 64})+{\cal L}(\text{Prob=}D_{y}(X),\text{Target=}\{1\}% ^{64\times 64})} roman_min start_POSTSUBSCRIPT ( italic_G ) end_POSTSUBSCRIPT roman_max start_POSTSUBSCRIPT ( italic_D ) end_POSTSUBSCRIPT italic_V ( italic_G , italic_D ) = caligraphic_L ( Prob= italic_D start_POSTSUBSCRIPT italic_y end_POSTSUBSCRIPT ( italic_G start_POSTSUBSCRIPT italic_x end_POSTSUBSCRIPT ( italic_z ) ) , Target= { 0 } start_POSTSUPERSCRIPT 64 × 64 end_POSTSUPERSCRIPT ) + caligraphic_L ( Prob= italic_D start_POSTSUBSCRIPT italic_y end_POSTSUBSCRIPT ( italic_X ) , Target= { 1 } start_POSTSUPERSCRIPT 64 × 64 end_POSTSUPERSCRIPT ) where z 𝑧 z italic_z is the random noise, X 𝑋 X italic_X is the mask, Y 𝑌 Y italic_Y is the image and ℒ ℒ {\cal L} caligraphic_L is the binary cross entropy. 2.3 EM Image of cellular structure (Fig.  1 -A.ii) Generator : The generator is defined as G : { 0 , 1 } 3 × 512 × 512 → ( 0 , 1 ) 1 × 512 × 512 : 𝐺 → superscript 0 1 3 512 512 superscript 0 1 1 512 512 G:\{0,1\}^{3\times 512\times 512}\to(0,1)^{1\times 512\times 512} italic_G : { 0 , 1 } start_POSTSUPERSCRIPT 3 × 512 × 512 end_POSTSUPERSCRIPT → ( 0 , 1 ) start_POSTSUPERSCRIPT 1 × 512 × 512 end_POSTSUPERSCRIPT is a U-Net based architecture used by Isola et al.   [ 2 ] . Due to the tendency to show artifacts in the generated images [ 3 ] changed the batch normalization layer with instance normalization (see Fig.  2 -C). Discriminator : In total three different conditional GAN models have been fit. Fig.  3 -PCA plot shows that all three models have the distribution of generated images as the real images. However, after observing the generated images the conclusion can be made that the cGAN model with skip patch (ours) discriminator provides a better result. A closer examination of the images reveals the artifact inside the generated mitochondria images by the patch discriminator. Besides, the model with skip patch discriminator converges 50% faster than the patch discriminator. All models have been trained for a 1500 fixed epoch size. Since the 3 channel of the mask is being concatenated with the image, the discriminator is defined as D : ( 0 , 1 ) 4 × 512 × 512 → { 0 , 1 } 64 × 64 : 𝐷 → superscript 0 1 4 512 512 superscript 0 1 64 64 D:(0,1)^{4\times 512\times 512}\to\{0,1\}^{64\times 64} italic_D : ( 0 , 1 ) start_POSTSUPERSCRIPT 4 × 512 × 512 end_POSTSUPERSCRIPT → { 0 , 1 } start_POSTSUPERSCRIPT 64 × 64 end_POSTSUPERSCRIPT Training : As the dataset is reasonably small with only 20 examples it is quite hard to retain the model from overfitting. As a remedy, a comparatively lower parameterized generator architecture has been used with an initial number of channels equal to 16 (see Fig.  2 -C). Additionally, for the augmentation random vertical flipping, horizontal flipping, and cropping with 98% of the area (keeping the aspect ratio fixed) were used. The objective of cGAN is arg ⁡ min ( G ) ⁡ max ( D ) ⁡ E x , y ⁢ [ log ⁡ D ⁢ ( x , y ) ] + E x , z ⁢ [ log ⁡ ( 1 − D ⁢ ( x , G ⁢ ( x , z ) ) ) ] + λ ⁢ L L ⁢ 1 subscript 𝐺 subscript 𝐷 subscript 𝐸 𝑥 𝑦 delimited-[] 𝐷 𝑥 𝑦 subscript 𝐸 𝑥 𝑧 delimited-[] 1 𝐷 𝑥 𝐺 𝑥 𝑧 𝜆 subscript 𝐿 𝐿 1 \arg\min_{(G)}\max_{(D)}E_{x,y}[\log~{}D(x,y)]+E_{x,z}[\log(1-D(x,G(x,z)))]+% \lambda L_{L1} roman_arg roman_min start_POSTSUBSCRIPT ( italic_G ) end_POSTSUBSCRIPT roman_max start_POSTSUBSCRIPT ( italic_D ) end_POSTSUBSCRIPT italic_E start_POSTSUBSCRIPT italic_x , italic_y end_POSTSUBSCRIPT [ roman_log italic_D ( italic_x , italic_y ) ] + italic_E start_POSTSUBSCRIPT italic_x , italic_z end_POSTSUBSCRIPT [ roman_log ( 1 - italic_D ( italic_x , italic_G ( italic_x , italic_z ) ) ) ] + italic_λ italic_L start_POSTSUBSCRIPT italic_L 1 end_POSTSUBSCRIPT where L L ⁢ 1 subscript 𝐿 𝐿 1 L_{L1} italic_L start_POSTSUBSCRIPT italic_L 1 end_POSTSUBSCRIPT is the L ⁢ 1 𝐿 1 L1 italic_L 1 loss and λ 𝜆 \lambda italic_λ is the weight for the L ⁢ 1 𝐿 1 L1 italic_L 1 loss. 3 Results and conclusions For the generation of EM images, the PCA method suggests that any of the options (patch 16 × 16 16 16 16\times 16 16 × 16 , patch 70 × 70 70 70 70\times 70 70 × 70 , or skip patch discriminator) will have a similar image distribution as the real EM images. However, a closer inspection of the generated images will reveal that the discriminator with a 16 × 16 16 16 16\times 16 16 × 16 patch produces blurry images, and a discriminator with a 70 × 70 70 70 70\times 70 70 × 70 patch produces images with a subtle square artifact inside all of the mitochondria. Using a skip patch discriminator provides a better result than using a patch discriminator even though the empirical result of PCA suggests that all models are equally good (see Fig  3 -B). Also, the cGAN model with skip patch discriminator converges approximately with a total epoch of 50% compared to the patch discriminator cGAN. Though for unbiased comparison all models have been trained for the same number of epochs indifferent of convergence. All of the networks produce similar types of images on the training data and the new unseen data which suggests that the models have not overfitted the training data. So, for a small dataset, a cGAN with a smaller number of parameters in the generator architecture is quite helpful. For generating the mitochondria and cell membrane mask (the main focus of our study), a patch discriminator does not perform well on both the global and local structures. Replacing the patch discriminator with the skip patch discriminator forces the generator to learn about both the global and local structures. Additionally, approximately 50% less epoch is required to obtain the desired performance compared to the patch discriminator. Without using the skip patch discriminator it is quite impossible to generate intricate structures such as the mitochondria and cell membrane mask (see Fig.  1 -C). Finally, when a generated image requires some global and local patterns to exhibit, a GAN with a skip-patch discriminator is always recommended. Figure 3: (A) Outcome of the conditional GAN (cGAN) models with the training data where S.Patch, P(16x16) and P(70x70) correspond to the skip-patch, 16 × 16 16 16 16\times 16 16 × 16 patch and 70 × 70 70 70 70\times 70 70 × 70 patch discriminators respectively. (B) The outcome of the cGAN models with the newly generated mask (zoomed). All of the points mix evenly in the PCA plot of the generated images vs the real image. Hence according to PCA, the distribution of the generated images with all of the three methods has a distribution close to the real EM images. References [1] J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial networks,” (2014), arXiv:1406.2661 [stat.ML]. [2] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” (2018), arXiv:1611.07004 [cs.CV]. [3] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycleconsistent adversarial networks,” (2020), arXiv:1703.10593 [cs.CV]. [4] S. Gerhard, J. Funke, J.  Martel, A. Cardona, and R. Fetter. “Segmented anisotropic ssTEM dataset of neural tissue,” (2013), figshare. Dataset. https://doi.org/10.6084/m9.figshare.856713 Generated on Thu May 2 19:33:42 2024 by L a T e XML