MetaFormer Baselines for Vision.

MetaFormer, the abstracted architecture of Transformer, has been found to play a significant role in achieving competitive performance. In this paper, we further explore the capacity of MetaFormer, again, by migrating our focus away from the token mixer design: we introduce several baseline models under MetaFormer using the most basic or common mixers, and demonstrate their gratifying performance. We summarize our observations as follows: (1) MetaFormer ensures solid lower bound of performance. By merely adopting identity mapping as the token mixer, the MetaFormer model, termed IdentityFormer, achieves [Formula: see text]80% accuracy on ImageNet-1 K. (2) MetaFormer works well with arbitrary token mixers. When specifying the token mixer as even a random matrix to mix tokens, the resulting model RandFormer yields an accuracy of [Formula: see text]81%, outperforming IdentityFormer. Rest assured of MetaFormer's results when new token mixers are adopted. (3) MetaFormer effortlessly offers state-of-the-art results. With just conventional token mixers dated back five years ago, the models instantiated from MetaFormer already beat state of the art. (a) ConvFormer outperforms ConvNeXt. Taking the common depthwise separable convolutions as the token mixer, the model termed ConvFormer, which can be regarded as pure CNNs, outperforms the strong CNN model ConvNeXt. (b) CAFormer sets new record on ImageNet-1 K. By simply applying depthwise separable convolutions as token mixer in the bottom stages and vanilla self-attention in the top stages, the resulting model CAFormer sets a new record on ImageNet-1 K: it achieves an accuracy of 85.5% at 224 ×224 resolution, under normal supervised training without external data or distillation. In our expedition to probe MetaFormer, we also find that a new activation, StarReLU, reduces 71% FLOPs of activation compared with commonly-used GELU yet achieves better performance. Specifically, StarReLU is a variant of Squared ReLU dedicated to alleviating distribution shift. We expect StarReLU to find great potential in MetaFormer- like models alongside other neural networks. Code and models are available at https://github.com/sail-sg/metaformer.

Contrast-Reconstruction Representation Learning for Self-Supervised Skeleton-Based Action Recognition.

Skeleton-based action recognition is widely used in varied areas, e.g., surveillance and human-machine interaction. Existing models are mainly learned in a supervised manner, thus heavily depending on large-scale labeled data, which could be infeasible when labels are prohibitively expensive. In this paper, we propose a novel Contrast-Reconstruction Representation Learning network (CRRL) that simultaneously captures postures and motion dynamics for unsupervised skeleton-based action recognition. It consists of three parts: Sequence Reconstructor (SER), Contrastive Motion Learner (CML), and Information Fuser (INF). SER learns representation from skeleton coordinate sequence via reconstruction. However the learned representation tends to focus on trivial postural coordinates and be hesitant in motion learning. To enhance the learning of motions, CML performs contrastive learning between the representation learned from coordinate sequences and additional velocity sequences, respectively. Finally, in the INF module, we explore varied strategies to combine SER and CML, and propose to couple postures and motions via a knowledge-distillation based fusion strategy which transfers the motion learning from CML to SER. Experimental results on several benchmarks, i.e., NTU RGB+D 60/120, PKU-MMD, CMU, and NW-UCLA, demonstrate the promise of the our method by outperforming state-of-the-art approaches.

BRN2 as a key gene drives the early primate telencephalon development.

Evolutionary mutations in primate-specific genes drove primate cortex expansion. However, whether conserved genes with previously unidentified functions also play a key role in primate brain expansion remains unknown. Here, we focus on BRN2 (POU3F2), a gene encoding a neural transcription factor commonly expressed in both primates and mice. Compared to the limited effects on mouse brain development, BRN2 biallelic knockout in cynomolgus monkeys (Macaca fascicularis) is lethal before midgestation. Histology analysis and single-cell transcriptome show that BRN2 deficiency decreases RGC expansion, induces precocious differentiation, and alters the trajectory of neurogenesis in the telencephalon. BRN2, serving as an upstream factor, controls specification and differentiation of ganglionic eminences. In addition, we identified the conserved function of BRN2 in cynomolgus monkeys to human RGCs. BRN2 may function by directly regulating SOX2 and STAT3 and maintaining HOPX. Our findings reveal a previously unknown mechanism that BRN2, a conserved gene, drives early primate telencephalon development by gaining novel mechanistic functions.

The Long Terminal Repeats of ERV6 Are Activated in Pre-Implantation Embryos of Cynomolgus Monkey.

Precise gene regulation is critical during embryo development. Long terminal repeat elements (LTRs) of endogenous retroviruses (ERVs) are dynamically expressed in blastocysts of mammalian embryos. However, the expression pattern of LTRs in monkey blastocyst is still unknown. By single-cell RNA-sequencing (seq) data of cynomolgus monkeys, we found that LTRs of several ERV families, including MacERV6, MacERV3, MacERV2, MacERVK1, and MacERVK2, were highly expressed in pre-implantation embryo cells including epiblast (EPI), trophectoderm (TrB), and primitive endoderm (PrE), but were depleted in post-implantation. We knocked down MacERV6-LTR1a in cynomolgus monkeys with a short hairpin RNA (shRNA) strategy to examine the potential function of MacERV6-LTR1a in the early development of monkey embryos. The silence of MacERV6-LTR1a mainly postpones the differentiation of TrB, EPI, and PrE cells in embryos at day 7 compared to control. Moreover, we confirmed MacERV6-LTR1a could recruit Estrogen Related Receptor Beta (ESRRB), which plays an important role in the maintenance of self-renewal and pluripotency of embryonic and trophoblast stem cells through different signaling pathways including FGF and Wnt signaling pathways. In summary, these results suggest that MacERV6-LTR1a is involved in gene regulation of the pre-implantation embryo of the cynomolgus monkeys.

Analysis of developmental imprinting dynamics in primates using SNP-free methods to identify imprinting defects in cloned placenta.

Our knowledge of genomic imprinting in primates is lagging behind that of mice largely because of the difficulties of allelic analyses in outbred animals. To understand imprinting dynamics in primates, we profiled transcriptomes, DNA methylomes, and H3K27me3 in uniparental monkey embryos. We further developed single-nucleotide-polymorphism (SNP)-free methods, TARSII and CARSII, to identify germline differentially methylated regions (DMRs) in somatic tissues. Our comprehensive analyses showed that allelic DNA methylation, but not H3K27me3, is a major mark that correlates with paternal-biasedly expressed genes (PEGs) in uniparental monkey embryos. Interestingly, primate germline DMRs are different from PEG-associated DMRs in early embryos and are enriched in placenta. Strikingly, most placenta-specific germline DMRs are lost in placenta of cloned monkeys. Collectively, our study establishes SNP-free germline DMR identification methods, defines developmental imprinting dynamics in primates, and demonstrates imprinting defects in cloned monkey placenta, which provides important clues for improving primate cloning.

Amnion signals are essential for mesoderm formation in primates.

Embryonic development is largely conserved among mammals. However, certain genes show divergent functions. By generating a transcriptional atlas containing >30,000 cells from post-implantation non-human primate embryos, we uncover that ISL1, a gene with a well-established role in cardiogenesis, controls a gene regulatory network in primate amnion. CRISPR/Cas9-targeting of ISL1 results in non-human primate embryos which do not yield viable offspring, demonstrating that ISL1 is critically required in primate embryogenesis. On a cellular level, mutant ISL1 embryos display a failure in mesoderm formation due to reduced BMP4 signaling from the amnion. Via loss of function and rescue studies in human embryonic stem cells we confirm a similar role of ISL1 in human in vitro derived amnion. This study highlights the importance of the amnion as a signaling center during primate mesoderm formation and demonstrates the potential of in vitro primate model systems to dissect the genetics of early human embryonic development.

Chimeric contribution of human extended pluripotent stem cells to monkey embryos ex vivo.

Interspecies chimera formation with human pluripotent stem cells (hPSCs) represents a necessary alternative to evaluate hPSC pluripotency in vivo and might constitute a promising strategy for various regenerative medicine applications, including the generation of organs and tissues for transplantation. Studies using mouse and pig embryos suggest that hPSCs do not robustly contribute to chimera formation in species evolutionarily distant to humans. We studied the chimeric competency of human extended pluripotent stem cells (hEPSCs) in cynomolgus monkey (Macaca fascicularis) embryos cultured ex vivo. We demonstrate that hEPSCs survived, proliferated, and generated several peri- and early post-implantation cell lineages inside monkey embryos. We also uncovered signaling events underlying interspecific crosstalk that may help shape the unique developmental trajectories of human and monkey cells within chimeric embryos. These results may help to better understand early human development and primate evolution and develop strategies to improve human chimerism in evolutionarily distant species.

Generation of a Hutchinson-Gilford progeria syndrome monkey model by base editing.

Many human genetic diseases, including Hutchinson-Gilford progeria syndrome (HGPS), are caused by single point mutations. HGPS is a rare disorder that causes premature aging and is usually caused by a de novo point mutation in the LMNA gene. Base editors (BEs) composed of a cytidine deaminase fused to CRISPR/Cas9 nickase are highly efficient at inducing C to T base conversions in a programmable manner and can be used to generate animal disease models with single amino-acid substitutions. Here, we generated the first HGPS monkey model by delivering a BE mRNA and guide RNA (gRNA) targeting the LMNA gene via microinjection into monkey zygotes. Five out of six newborn monkeys carried the mutation specifically at the target site. HGPS monkeys expressed the toxic form of lamin A, progerin, and recapitulated the typical HGPS phenotypes including growth retardation, bone alterations, and vascular abnormalities. Thus, this monkey model genetically and clinically mimics HGPS in humans, demonstrating that the BE system can efficiently and accurately generate patient-specific disease models in non-human primates.

Modulation of Wnt and Activin/Nodal supports efficient derivation, cloning and suspension expansion of human pluripotent stem cells.

Various culture systems have been used to derive and maintain human pluripotent stem cells (hPSCs), but they are inefficient in sustaining cloning and suspension expansion of hPSCs. Through systematically modulating Wnt and Activin/Nodal signaling, we developed a defined medium (termed AIC), which enables efficient cloning and long-term expansion of hPSCs (AIC-hPSCs) through single-cell passage on feeders, matrix or in suspension (25-fold expansion in 4 days) and maintains genomic stability of hPSCs over extensive expansion. Moreover, the AIC medium supports efficient derivation of hPSCs from blastocysts or somatic cells under feeder-free conditions. Compared to conventional hPSCs, AIC-hPSCs have similar gene expression profiles but down-regulated differentiation genes and display higher metabolic activity. Additionally, the AIC medium shows a good compatibility for different hPSC lines under various culture conditions. Our study provides a robust culture system for derivation, cloning and suspension expansion of high-quality hPSCs that benefits GMP production and processing of therapeutic hPSC products.

Dissecting primate early post-implantation development using long-term in vitro embryo culture.

The transition from peri-implantation to gastrulation in mammals entails the specification and organization of the lineage progenitors into a body plan. Technical and ethical challenges have limited understanding of the cellular and molecular mechanisms that underlie this transition. We established a culture system that enabled the development of cynomolgus monkey embryos in vitro for up to 20 days. Cultured embryos underwent key primate developmental stages, including lineage segregation, bilaminar disc formation, amniotic and yolk sac cavitation, and primordial germ cell-like cell (PGCLC) differentiation. Single-cell RNA-sequencing analysis revealed development trajectories of primitive endoderm, trophectoderm, epiblast lineages, and PGCLCs. Analysis of single-cell chromatin accessibility identified transcription factors specifying each cell type. Our results reveal critical developmental events and complex molecular mechanisms underlying nonhuman primate embryogenesis in the early postimplantation period, with possible relevance to human development.

Gene Delivery to Nonhuman Primate Preimplantation Embryos Using Recombinant Adeno-Associated Virus.

Delivery of genome editing tools to mammalian zygotes has revolutionized animal modeling. However, the mechanical delivery method to introduce genes and proteins to zygotes remains a challenge for some animal species that are important in biomedical research. Here, an approach to achieve gene delivery and genome editing in nonhuman primate embryos is presented by infecting zygotes with recombinant adeno-associated viruses (rAAVs). Together with previous reports from the authors of this paper and others, this approach is potentially applicable to a broad range of mammals. In addition to genome editing and animal modeling, this rAAV-based method can facilitate gene function studies in early-stage embryos.

Homologous recombination-mediated targeted integration in monkey embryos using TALE nucleases.

BACKGROUND: Non-human primate (NHP) models can closely mimic human physiological functions and are therefore highly valuable in biomedical research. Genome editing is now developing rapidly due to the precision and efficiency offered by engineered site-specific endonuclease-based systems, such as transcription activator-like effector nucleases (TALENs) and the clustered regularly interspaced short palindromic repeats (CRISPR)/CRISPR-associated protein-9 nuclease (Cas9) system. It has been demonstrated that these programmable nucleases can introduce genetic changes in embryos from many species including NHPs. In 2014, we reported the first genetic editing of macaques using TALENs and CRISPR/Cas9. Subsequently, we characterized the phenotype of a methyl CpG binding protein 2 (MECP2)-mutant cynomolgus monkey model of Rett syndrome generated using the TALEN approach. These efforts not only accelerated the advance of modeling genetic diseases in NHPs, but also encouraged us to develop specific gene knock-in monkeys. In this study, we assess the possibility of homologous recombination (HR)-mediated gene replacement using TALENs in monkeys, and generate preimplantation embryos carrying an EmGFP fluorescent reporter constructed in the OCT4 gene. RESULT: We assembled a pair of TALENs specific to the first exon of the OCT4 gene and constructed a donor vector consisting of the homology arms cloned from the monkey genome DNA, flanking an EmGFP cassette. Next, we co-injected the TALENs-coding plasmid and donor plasmid into the cytoplasm of 122 zygotes 6-8 h after fertilization. Sequencing and immunofluorescence revealed that the OCT4-EmGFP knock-in allele had been successfully generated by TALENs-mediated HR at an efficiency of 11.3% (7 out of 62) or 11.1% (1 out of 9), respectively, in monkey embryos. CONCLUSION: We have successfully, for the first time, obtained OCT4-EmGFP knock-in monkey embryos via HR mediated by TALENs. Our results suggest that gene targeting through TALEN-assisted HR is a useful approach to introduce precise genetic modification in NHPs.

Improving Cell Survival in Injected Embryos Allows Primed Pluripotent Stem Cells to Generate Chimeric Cynomolgus Monkeys.

Monkeys are an optimal model species for developing stem cell therapies. We previously reported generating chimeric cynomolgus monkey fetuses using dome-shaped embryonic stem cells (dESCs). However, conventional primed pluripotent stem cells (pPSCs) lack chimera competency. Here, by altering the media in which injected morulae are cultured, we observed increased survival of cynomolgus monkey primed ESCs, induced PSCs, and somatic cell nuclear transfer-derived ESCs, thereby enabling chimeric contributions with 0.1%-4.5% chimerism into the embryonic and placental tissues, including germ cell progenitors in chimeric monkeys. Mechanically, dESCs and pPSCs belong to different cell types and similarly express epiblast ontogenic genes. The host embryonic microenvironment could reprogram injected PSCs to embryonic-like cells. However, the reprogramming level and chimerism were associated with the cell state of injected PSCs. Our findings provide a method to understand pluripotency and broaden the use of embryonic chimeras for basic developmental biology research and regenerative medicine.

Transabdominal ultrasound-guided multifetal pregnancy reduction in 10 cases of monkeys.

Intracytoplasmic sperm injection (ICSI) and embryo transfer (ET) in nonhuman primates, e.g. rhesus and cynomolgus monkeys, has been widely used in researches of reproductive and developmental biology, and the success rate has been improved significantly. However, unwanted multiple pregnancy occurs frequently during the ICSI-ET in monkeys, most of which leads to miscarriages. To improve the birth rate of pregnancies and to safeguard health of host and baby monkeys, multifetal pregnancy reduction (MPR) is necessary. In this study, a total of 10 monkeys with multiple pregnancies received MPR through transabdominal ultrasound-guided potassium chloride injection into beating hearts of selective fetuses. To assess MPR efficiency, 31 monkeys with normal singleton pregnancies and 25 monkeys with twin pregnancies without MPR were used as controls. The aim of the reduction is to keep only one fetus, no matter twin or triplet pregnancy originally. Our results show that six cases of MPR were successful and all of them retained single fetus. Moreover, about 1 month (30.2 ± 1.2 days) of gestation is a better timing for MPR than later stage (50.7 ± 1.9 days). We also found that the remaining fetuses developed normally with full-term gestation and normal birth weight. In conclusion, transabdominal ultrasound-guided potassium chloride injection is a safe and effective MPR method for monkeys with multiple pregnancies.

Modeling Rett Syndrome Using TALEN-Edited MECP2 Mutant Cynomolgus Monkeys.

Gene-editing technologies have made it feasible to create nonhuman primate models for human genetic disorders. Here, we report detailed genotypes and phenotypes of TALEN-edited MECP2 mutant cynomolgus monkeys serving as a model for a neurodevelopmental disorder, Rett syndrome (RTT), which is caused by loss-of-function mutations in the human MECP2 gene. Male mutant monkeys were embryonic lethal, reiterating that RTT is a disease of females. Through a battery of behavioral analyses, including primate-unique eye-tracking tests, in combination with brain imaging via MRI, we found a series of physiological, behavioral, and structural abnormalities resembling clinical manifestations of RTT. Moreover, blood transcriptome profiling revealed that mutant monkeys resembled RTT patients in immune gene dysregulation. Taken together, the stark similarity in phenotype and/or endophenotype between monkeys and patients suggested that gene-edited RTT founder monkeys would be of value for disease mechanistic studies as well as development of potential therapeutic interventions for RTT.

De novo DNA methylation during monkey pre-implantation embryogenesis.

Critical epigenetic regulation of primate embryogenesis entails DNA methylome changes. Here we report genome-wide composition, patterning, and stage-specific dynamics of DNA methylation in pre-implantation rhesus monkey embryos as well as male and female gametes studied using an optimized tagmentation-based whole-genome bisulfite sequencing method. We show that upon fertilization, both paternal and maternal genomes undergo active DNA demethylation, and genome-wide de novo DNA methylation is also initiated in the same period. By the 8-cell stage, remethylation becomes more pronounced than demethylation, resulting in an increase in global DNA methylation. Promoters of genes associated with oxidative phosphorylation are preferentially remethylated at the 8-cell stage, suggesting that this mode of energy metabolism may not be favored. Unlike in rodents, X chromosome inactivation is not observed during monkey pre-implantation development. Our study provides the first comprehensive illustration of the 'wax and wane' phases of DNA methylation dynamics. Most importantly, our DNA methyltransferase loss-of-function analysis indicates that DNA methylation influences early monkey embryogenesis.

Rhesus monkey model of liver disease reflecting clinical disease progression and hepatic gene expression analysis.

Alcoholic liver disease (ALD) is a significant public health issue with heavy medical and economic burdens. The aetiology of ALD is not yet completely understood. The development of drugs and therapies for ALD is hampered by a lack of suitable animal models that replicate both the histological and metabolic features of human ALD. Here, we characterize a rhesus monkey model of alcohol-induced liver steatosis and hepatic fibrosis that is compatible with the clinical progression of the biochemistry and pathology in humans with ALD. Microarray analysis of hepatic gene expression was conducted to identify potential molecular signatures of ALD progression. The up-regulation of expression of hepatic genes related to liver steatosis (CPT1A, FASN, LEPR, RXRA, IGFBP1, PPARGC1A and SLC2A4) was detected in our rhesus model, as was the down-regulation of such genes (CYP7A1, HMGCR, GCK and PNPLA3) and the up-regulation of expression of hepatic genes related to liver cancer (E2F1, OPCML, FZD7, IGFBP1 and LEF1). Our results demonstrate that this ALD model reflects the clinical disease progression and hepatic gene expression observed in humans. These findings will be useful for increasing the understanding of ALD pathogenesis and will benefit the development of new therapeutic procedures and pharmacological reagents for treating ALD.

Functional disruption of the dystrophin gene in rhesus monkey using CRISPR/Cas9.

CRISPR/Cas9 has been used to genetically modify genomes in a variety of species, including non-human primates. Unfortunately, this new technology does cause mosaic mutations, and we do not yet know whether such mutations can functionally disrupt the targeted gene or cause the pathology seen in human disease. Addressing these issues is necessary if we are to generate large animal models of human diseases using CRISPR/Cas9. Here we used CRISPR/Cas9 to target the monkey dystrophin gene to create mutations that lead to Duchenne muscular dystrophy (DMD), a recessive X-linked form of muscular dystrophy. Examination of the relative targeting rate revealed that Crispr/Cas9 targeting could lead to mosaic mutations in up to 87% of the dystrophin alleles in monkey muscle. Moreover, CRISPR/Cas9 induced mutations in both male and female monkeys, with the markedly depleted dystrophin and muscle degeneration seen in early DMD. Our findings indicate that CRISPR/Cas9 can efficiently generate monkey models of human diseases, regardless of inheritance patterns. The presence of degenerated muscle cells in newborn Cas9-targeted monkeys suggests that therapeutic interventions at the early disease stage may be effective at alleviating the myopathy.

TALEN-mediated gene mutagenesis in rhesus and cynomolgus monkeys.

Recent advances in gene editing technology have introduced the potential for application of mutagenesis approaches in nonhuman primates to model human development and disease. Here we report successful TALEN-mediated mutagenesis of an X-linked, Rett syndrome (RTT) gene, methyl-CpG binding protein 2 (MECP2), in both rhesus and cynomolgus monkeys. Microinjection of MECP2-targeting TALEN plasmids into rhesus and cynomolgus zygotes leads to effective gene editing of MECP2 with no detected off-target mutagenesis. Male rhesus (2) and cynomolgous (1) fetuses carrying MECP2 mutations in various tissues including testes were miscarried during midgestation, consistent with RTT-linked male embryonic lethality in humans. One live delivery of a female cynomolgus monkey occurred after 162 days of gestation, with abundant MECP2 mutations in peripheral tissues. We conclude that TALEN-mediated mutagenesis can be an effective tool for genetic modeling of human disease in nonhuman primates.