PCA with ATLAS

1. Produce a glf file for each sample BAM file with ATLAS tool glf
2. Estimate the pairwise distances with ATLAS tool estimateDist. Use "euclidian" as the distance type.
3. Use the distance matrix produced in step 2 to perform an MDS. An MDS based on the euclidian genetic distances is identical to PCA.

PCAngsd

1. Produce a glf file for each sample BAM file with ATLAS tool glf
2. Use produced glf files as input to ATLAS tool majorMinor
3. Because PCAngsd only accepts files in beagle format, convert the produced vcf to beagle with VCFToBeagle. Make use of the filtering options to reduce the size of the beagle file and to speed up PCAngsd.

Remarks

With these approaches to perform PCA you can mitigate biases caused by mixing modern and ancient samples and different depths: you take PMD into account when creating glf's for the ancient samples, and there is no need for genotype calling, thus no problems due to differences in genotyping uncertainty