Bos taurus redo: Difference between revisions
Jump to navigation
Jump to search
| (891 intermediate revisions by the same user not shown) | |||
| Line 7: | Line 7: | ||
* [http://www.ncbi.nlm.nih.gov/sites/entrez?db=genomeprj&cmd=search&term=bos%20taurus Genome Projects] | * [http://www.ncbi.nlm.nih.gov/sites/entrez?db=genomeprj&cmd=search&term=bos%20taurus Genome Projects] | ||
* [http://www.ncbi.nlm.nih.gov/Traces/trace.cgi?cmd=retrieve&size=0&val=SPECIES_CODE+%3D+%22BOS+TAURUS%22 TA search] | * [http://www.ncbi.nlm.nih.gov/Traces/trace.cgi?cmd=retrieve&size=0&val=SPECIES_CODE+%3D+%22BOS+TAURUS%22 TA search] | ||
* ftp | * [ftp://ftp.ncbi.nih.gov/pub/TraceDB/bos_taurus/ TA ftp] | ||
* 91 volumes: 87 with qual & 4 with no quality (85 volumes contain BCM reads) | |||
* 14 centers | |||
* 21 center/trace_type_codes | |||
* Avg LEN=984 | * Avg LEN=984 | ||
* Avg CLIP (CLB intersect CLV)=760 | * Avg CLIP (CLB intersect CLV)=760 | ||
* Avg CLV=997 ( | * Avg CLV=997 > Avg LEN ??? | ||
* | * Avg QUAL=38.96 (27.51 for the 2.59M reads not in the UMD assembly) | ||
* Avg UMDoverlapper CLIP=778 | |||
* | Problems: | ||
* 0 QUAL reads 650,133 ( 551,114 are BCM.WGS ) | |||
* the quality lines in several qual. files start with space; need to remove it otherwise tarchive2ca errors out saying that the len(quality)=len(seq)+1 | |||
* several xml contained the "&" character => XML parser error | |||
* xml.bos_taurus.087 contained 2 trace_volumes => XML parser error | |||
* BCCAGSC.CLONEEND : all reads have LIBRARY_ID=CH240, SEQ_LIB_ID=. ; the INSERT_SIZE & INSERT_STDEV vary within the library: set to 150,000 & 30,000 | |||
* UIUC.CLONEEND: INSERT_SIZE & INSERT_STDEV missing: set to 150,000 & 30,000 | |||
== CENTER_NAME counts == | == CENTER_NAME counts == | ||
COUNT CENTER_NAME | |||
35629020 BCM Baylor College of Medicine | 1 35629020 BCM Baylor College of Medicine | ||
737900 NISC NIH Intramural Sequencing Center | 2 737900 NISC NIH Intramural Sequencing Center | ||
652614 BCCAGSC British Columbia Cancer Agency Genome Sciences Centre # TA query_tracedb CENTER_NAME = "BCCAGSC" => 652,510 | 3 652614 BCCAGSC British Columbia Cancer Agency Genome Sciences Centre # TA query_tracedb CENTER_NAME = "BCCAGSC" => 652,510 | ||
378871 MARC USDA, ARS, US Meat Animal Research Center | 4 378871 MARC USDA, ARS, US Meat Animal Research Center | ||
114753 UIUC University of Illinois at Urbana-Champaign # TA query_tracedb CENTER_NAME = "UIUC" => 106,368 | 5 114753 UIUC University of Illinois at Urbana-Champaign # TA query_tracedb CENTER_NAME = "UIUC" => 106,368 | ||
107367 BARC USDA, ARS, Beltsville Agricultural Research Center | 6 107367 BARC USDA, ARS, Beltsville Agricultural Research Center | ||
65171 TIGR The Institute for Genome Research | 7 65171 TIGR The Institute for Genome Research | ||
53556 GSC Genoscope | 8 53556 GSC Genoscope | ||
43033 CENARGEN Embrapa Genetic Resources and Biotechnology | 9 43033 CENARGEN Embrapa Genetic Resources and Biotechnology | ||
18623 SC The Sanger Center | 10 18623 SC The Sanger Center | ||
15301 UOKNOR University of Oklahoma Norman Campus, Advanced Center for Genome Technology | 11 15301 UOKNOR University of Oklahoma Norman Campus, Advanced Center for Genome Technology | ||
10651 TIGR_JCVIJTC The Institute for Genomic Research, Traces generated at JCVIJTC # TA query_tracedb CENTER_NAME="JCVI" | 12 10651 TIGR_JCVIJTC The Institute for Genomic Research, Traces generated at JCVIJTC # TA query_tracedb CENTER_NAME="JCVI" | ||
2485 UIACBCB University of Iowa Center for Bioinformatics and Computation Biology (UIACBCB) | 13 2485 UIACBCB University of Iowa Center for Bioinformatics and Computation Biology (UIACBCB) | ||
49 WUGSC Washington University, Genome Sequencing Center # TA query_tracedb CENTER_NAME = "WUGSC" => 9 | 14 49 WUGSC Washington University, Genome Sequencing Center # TA query_tracedb CENTER_NAME = "WUGSC" => 9 | ||
'''37829394''' total total # TA query_tracedb SPECIES_CODE = "BOS TAURUS" => 37,788,710 | |||
== TRACE_TYPE_CODE counts == | == TRACE_TYPE_CODE counts == | ||
COUNT CENTER_NAME TRACE_TYPE_CODE | |||
24863599 BCM | 1 24863599 BCM* WGS SEQ_LIB_ID:89 | ||
10748529 BCM | 2 10748529 BCM* SHOTGUN SEQ_LIB_ID:15543 | ||
737900 NISC SHOTGUN | 3 737900 NISC SHOTGUN SEQ_LIB_ID:247 | ||
125597 BCCAGSC CLONEEND | 4 125597 BCCAGSC CLONEEND LIBRARY_ID:1 large insert size; some qualityless; !!! almost all have CLIP3=0 | ||
114753 UIUC CLONEEND | 5 114753 UIUC CLONEEND LIBRARY_ID:2 insert size missing , no frequent kmers | ||
65171 TIGR CLONEEND | 6 65171 TIGR CLONEEND SEQ_LIB_ID:1 2K & use TRACE_DIRECTION instead of TRACE_END | ||
53556 GSC CLONEEND | 7 53556 GSC CLONEEND SEQ_LIB_ID:1 large insert size; !!! all have qual=0 and were excluded | ||
26246 CENARGEN WGS | 8 26246 CENARGEN WGS . no LIBRARY_ID; no SEQ_LIB_ID; no INSERT_SIZE; no INSERT_STDEV; reads have no direction; ~21954 could be paired (same TEMPLATE_ID) | ||
25454 BARC CLONEEND | 9 25454 BARC CLONEEND SEQ_LIB_ID:14304 !!! all have CLIP3=0 | ||
16892 BCM | 10 16892 BCM* CLONEEND LIBRARY_ID:1 VBBAA mea=167000 std=25000 | ||
16787 CENARGEN CLONEEND | 11 16787 CENARGEN CLONEEND LIBRARY_ID:1 | ||
15150 UOKNOR SHOTGUN | 12 15150 UOKNOR SHOTGUN LIBRARY_ID:1 some qualityless | ||
10651 TIGR_JCVIJTC CLONEEND | 13 10651 TIGR_JCVIJTC CLONEEND SEQ_LIB_ID:2 | ||
151 UOKNOR FINISHING | 14 151 UOKNOR FINISHING LIBRARY_ID:1 some qualityless, no direction(TRACE_END=N); no INSERT_SIZE; no INSERT_STDEV | ||
49 WUGSC CLONEEND | 15 49 WUGSC CLONEEND SEQ_LIB_ID:1 | ||
'''36820485''' total | |||
527017 BCCAGSC EST | 16 527017 BCCAGSC EST | ||
207204 MARC EST | 17 207204 MARC EST | ||
171667 MARC PCR | 18 171667 MARC PCR | ||
81913 BARC EST | 19 81913 BARC EST | ||
20 18623 SC EST | |||
2485 UIACBCB EST | 21 2485 UIACBCB EST | ||
'''1008909''' total | |||
== STRATEGY & TRACE_TYPE_CODE counts == | == STRATEGY & TRACE_TYPE_CODE counts == | ||
| Line 93: | Line 102: | ||
18623 SC EST EST | 18623 SC EST EST | ||
2485 UIACBCB . EST | 2485 UIACBCB . EST | ||
== BCM.SHOTGUN libraries == | |||
* The long inserts are probably wrong !!! | |||
SIZE STDEV COUNT | |||
3500 1500 4502569 | |||
2000 1000 3244493 | |||
3000 1000 1021577 | |||
180000 1000 840528 | |||
6500 1500 429026 | |||
180000 13000 320208 | |||
6000 2000 208192 | |||
167000 13000 96337 | |||
3500 15000 85599 | |||
SIZE COUNT | |||
3500 4588168 | |||
2000 3244493 | |||
180000 1160736 | |||
3000 1021577 | |||
6500 429026 | |||
6000 208192 | |||
167000 96337 | |||
== 3' VECTOR TRIMMED counts == | == 3' VECTOR TRIMMED counts == | ||
CENTER_NAME TRACE_TYPE_CODE TOTAL 3'CLV<LEN QUAL==0 UMD.FRG | |||
BCM WGS 24863599 10968979 551114 | 1 BCM WGS 24863599 10968979 551114 24050767 | ||
BCM SHOTGUN 10748529 5052692 23419 | 2 BCM SHOTGUN 10748529 5052692 23419 10068499 | ||
NISC SHOTGUN 737900 28972 0 | 3 NISC SHOTGUN 737900 28972 0 735488 | ||
BCCAGSC CLONEEND 125597 125484 8926 | 4 BCCAGSC CLONEEND 125597 125484 8926 113790 | ||
UIUC CLONEEND 114753 90243 0 | 5 UIUC CLONEEND 114753 90243 0 106247 | ||
TIGR CLONEEND 65171 46389 0 | 6 TIGR CLONEEND 65171 46389 0 64903 | ||
GSC CLONEEND 53556 53556 53556 (all) | 7 GSC CLONEEND 53556 53556 53556 (all) 0 !!! all have 0 quals and were excluded | ||
CENARGEN WGS 26246 26246 0 | 8 CENARGEN WGS 26246 26246 0 25976 | ||
BARC CLONEEND 25454 25454 0 | 9 BARC CLONEEND 25454 25454 0 25387 | ||
BCM CLONEEND 16892 6751 0 | 10 BCM CLONEEND 16892 6751 0 16863 | ||
CENARGEN CLONEEND 16787 16787 0 | 11 CENARGEN CLONEEND 16787 16787 0 16628 | ||
UOKNOR SHOTGUN 15150 2885 12195 | 12 UOKNOR SHOTGUN 15150 2885 12195 0 | ||
TIGR_JCVIJTC CLONEEND 10651 339 0 | 13 TIGR_JCVIJTC CLONEEND 10651 339 0 10644 | ||
UOKNOR FINISHING 151 0 151 | 14 UOKNOR FINISHING 151 0 151 151 | ||
WUGSC CLONEEND 49 0 0 | 15 WUGSC CLONEEND 49 0 0 0 | ||
16 BCCAGSC EST 527017 524173 772 0 | |||
17 MARC EST 207204 207204 0 0 | |||
18 MARC PCR 171667 171667 0 0 | |||
19 BARC EST 81913 78597 0 0 | |||
20 SC EST 18623 7350 0 0 | |||
21 UIACBCB EST 2485 2485 0 0 | |||
== ZERO QUALITY COUNTS == | |||
* Counts | |||
CENTER_NAME TRACE_TYPE_CODE COUNT | |||
BCM WGS 551114 | |||
GSC CLONEEND 53556 | |||
BCM SHOTGUN 23419 | |||
UOKNOR SHOTGUN 12195 | |||
BCCAGSC CLONEEND 8926 | |||
BCCAGSC EST 772 | |||
UOKNOR FINISHING 151 | |||
TOTAL 650134 | |||
* For 0 quality reads, assign quality 20 to bases 1..700, 0 to bases 701.. | |||
* Volumes 026..039 have been fixed | |||
= Local Data = | = Local Data = | ||
| Line 131: | Line 177: | ||
Figaro | Figaro | ||
* trims vector only at 5' end | * trims vector only at 5' end | ||
* call lucy trimming for | * call lucy trimming for qualities | ||
Lucy | Lucy | ||
| Line 141: | Line 187: | ||
= Contaminant search = | = Contaminant search = | ||
nucmer reads CLIPPING range to UniVec & EcoliK12 | |||
== UniVec == | == UniVec == | ||
Ref | |||
#seqs min max mean median n50 sum | #seqs min max mean median n50 sum | ||
UniVec 2861 12 48551 231 99 781 660,151 | UniVec 2861 12 48551 231 99 781 660,151 | ||
UniVec_Core 1348 12 48551 243 98 967 327,641 | UniVec_Core 1348 12 48551 243 98 967 327,641 | ||
Hits: alignment length | |||
bp #reads min max mean median n50 sum | |||
19 4548466 19 1045 28.37 23 27 129025025 | |||
20 3684852 20 1045 30.56 25 28 112616359 | |||
30 1097357 30 1045 48.04 38 43 52714583 | |||
40 484661 40 1045 66.36 47 53 32163896 | |||
100 54334 100 1045 198 116 223 10772815 # many are ESTs | |||
== Ecoli == | == Ecoli == | ||
K12 4,639,675 bp | Ref: | ||
K12 4,639,675 bp | |||
Hits: alignment length | |||
bp #reads min max mean median n50 sum | |||
19 275109 19 1223 30.66 19 20 8435470 | |||
20 102550 20 1223 50.29 21 161 5156849 | |||
30 19032 30 1223 178 37 706 3381214 | |||
40 9234 40 1223 329 171 738 3034293 | |||
100 6781 100 1223 424 223 749 2876432 | |||
200 4378 200 1223 575 696 771 2516916 | |||
== BCM vectors == | == BCM vectors == | ||
#seqs min max mean median n50 sum | #seqs min max mean median n50 sum | ||
BCM 14 2580 33180 9379 5821 32705 131312 | BCM 14 2580 33180 9379 5821 32705 131312 | ||
= Vector/Splice site search = | |||
== Strategy == | |||
* 1. Select all the reads in the same volume that belong to one particular library; same CENTER_NAME, STRATEGY & TRACE_TYPE_CODE | |||
* 2. Get the quality clipping trim: CLIP_QUALITY_LEFT & CLIP_QUALITY_RIGHT | |||
* 3. Separate reads in 2 sets according to direction TRACE_END: FORWARD & REVERSE | |||
* 4. Get the most frequent kmers in each set (24 & 8 bp) | |||
* 5. Check if the most frequent kmers are overrepresented | |||
* 6. Check if the most frequent 8mers are present in the most frequent 24mers | |||
* 7. Try to extend the 24mers by a few bp => linkers | |||
* 8. Align linkers to the opposite stand sequences using nucmer | |||
* 9. Extract the subsequences adjacent(following) to linker (50..150bp) | |||
* 10. Align the subsequences; if they align we've probably identified the vector | |||
* 11. Identify the vector name/id by alignment to UniVec => several alignments | |||
* 12. Check if the forward/reverse vector(s) are the same : we should find a common vector sequence; the UniVec alignments should be adjacent | |||
* 13. create the Lucy vector & splice files; the splice contains the linker+vector | |||
* 14. run lucy & trim input reads according to Lucy clr | |||
* 15. align lucy trimmed reads to linker,vector,splice & UniVec.dust | |||
* 16. align input reads to linker,vector,splice & UniVec.dust | |||
* 17. compare the 15. & 16. counts | |||
== Example == | |||
* 1. volume 011 : 500,000 reads CENTER_NAME=BCM, TRACE_TYPE_CODE=WGS | |||
* 2. | |||
* 3. 249,611 TRACE_END=F & 250,389 TRACE_END=R | |||
* 4. kmers: 8 8bp most frequent kmers are shared by the FORWARD & REVERSE strands ; no 24bp kmers are shared | |||
==> 24.fwd/kmers.tab <== | |||
AGTTCGACTGCAAGTAGTTCATCA TGATGAACTACTTGCAGTCGAACT 2463 # contains AGTAGTTC | |||
GAGTTCGACTGCAAGTAGTTCATC GATGAACTACTTGCAGTCGAACTC 2189 | |||
CGAGTTCGACTGCAAGTAGTTCAT ATGAACTACTTGCAGTCGAACTCG 1996 | |||
TCGAGTTCGACTGCAAGTAGTTCA TGAACTACTTGCAGTCGAACTCGA 1593 | |||
GTTCGACTGCAAGTAGTTCATCAA TTGATGAACTACTTGCAGTCGAAC 1023 | |||
GAGTTCGACTGCAGTAGTTCATCA TGATGAACTACTGCAGTCGAACTC 812 | |||
CGAGTTCGACTGCAGTAGTTCATC GATGAACTACTGCAGTCGAACTCG 777 | |||
GTTCGACTGCAAGTAGTTCATCAT ATGATGAACTACTTGCAGTCGAAC 769 | |||
TCGAGTTCGACTGCAGTAGTTCAT ATGAACTACTGCAGTCGAACTCGA 637 | |||
ATCGAGTTCGACTGCAAGTAGTTC GAACTACTTGCAGTCGAACTCGAT 594 | |||
==> 08.fwd/kmers.tab <== | |||
AGTAGTTC GAACTACT 86477 | |||
CAGTAGTT AACTACTG 67681 | |||
AGTTCTCA TGAGAACT 61556 | |||
TAGTTCTC GAGAACTA 60964 | |||
GTAGTTCT AGAACTAC 57866 | |||
AGTTCATC GATGAACT 49676 | |||
TAGTTCAT ATGAACTA 45298 | |||
GTTCATCA TGATGAAC 42117 | |||
GCAGTAGT ACTACTGC 41391 | |||
GTAGTTCA TGAACTAC 40694 | |||
==> 24.rev/kmers.tab <== | |||
TATCGATGGTACAGTAGTTCATCA TGATGAACTACTGTACCATCGATA 999 # contains AGTAGTTC | |||
CTATCGATGGTACAGTAGTTCATC GATGAACTACTGTACCATCGATAG 774 | |||
GCTATCGATGGTACAGTAGTTCAT ATGAACTACTGTACCATCGATAGC 600 | |||
CGCTATCGATGGTACAGTAGTTCA TGAACTACTGTACCATCGATAGCG 432 | |||
ATCGATGGTACAGTAGTTCATCAT ATGATGAACTACTGTACCATCGAT 417 | |||
ATCGATGGTACAGTAGTTCATCAA TTGATGAACTACTGTACCATCGAT 380 | |||
ATCAGATGGTACAGTAGTTCATCA TGATGAACTACTGTACCATCTGAT 373 | |||
ATCGATGGTACAGTAGTTCATCAC GTGATGAACTACTGTACCATCGAT 265 | |||
CTATCGATGGTAAGTAGTTCATCA TGATGAACTACTTACCATCGATAG 235 | |||
TCAGATGGTACAGTAGTTCATCAA TTGATGAACTACTGTACCATCTGA 224 | |||
==> 08.rev/kmers.tab <== | |||
AGTTCATC GATGAACT 85127 | |||
TAGTTCAT ATGAACTA 77902 | |||
GTTCATCA TGATGAAC 75585 | |||
TAGTTCTC GAGAACTA 68057 | |||
AGTTCTCA TGAGAACT 67277 | |||
GTAGTTCT AGAACTAC 64894 | |||
GTAGTTCA TGAACTAC 62607 | |||
CGTAGTTC GAACTACG 52031 | |||
AGTAGTTC GAACTACT 51013 | |||
ACGTAGTT AACTACGT 31552 | |||
* 7. Get linker sequences | |||
>linker.fwd 27bp | |||
TCGAGTTCGACTGCAAGTAGTTCATCA | |||
>linker.rev 27bp | |||
CTAATCAGATGGTACAGTAGTTCATCA | |||
#>linker.rev 40 bp Art's (13 more bp at 5') | |||
#TATGACCATGCGCCTAATCAGATGGTACAGTAGTTCATCA | |||
#GCTATCGATGGTACAGTAGTTCATCAT is the most frequent rev seq 27 kmers but not the linker (few snp differences) | |||
* 8 & 9 Align reads to linkers using nucmer | |||
Fwd: | |||
nucmer -l 12 -c 24 -r linker.fwd.seq ../bos_taurus.$v.r.fasta | |||
# nucmer -l 12 -c 24 -r kmers.seq ../bos_taurus.$v.r.fasta | |||
show-coords out.delta | awk '{print $19,$5,$13}' > ! out.clr | |||
extractfromfastanames.pl -clr -f out.clr < ../bos_taurus.$v.r.fasta >! out.seq | |||
Rev: | |||
nucmer -l 12 -c 24 -r linker.rev.seq ../bos_taurus.$v.f.fasta | |||
# nucmer -l 12 -c 24 -r kmers.seq ../bos_taurus.$v.f.fasta | |||
show-coords out.delta | awk '{print $19,$5,$13}' > ! out.clr | |||
extractfromfastanames.pl -clr -f out.clr < ../bos_taurus.$v.f.fasta >! out.seq | |||
Both: | |||
clrFasta out.seq >! out.cseq | |||
fasta2tab.pl out.cseq | sort -k2 > ! out.tab | |||
nucmer -c 40 out.cseq ~/db/UniVec -p vector | |||
delta-filter -q vector.delta >! vector.filter-q.delta | |||
show-coords vector.filter-q.delta | sort -n | head | |||
cat vector.filter-q.delta | grep "^>" | count.pl -c 1 -m 2 | |||
* 10. Extract "vector reads" | |||
>399553028 # 24.fwd | |||
TGATGAACTACTGTACCATCTGATTAGGCGCATGGTCATAGCTGTTTCCTGTGTGAAATT | |||
GCTATCCGCTCACAATTCCACACAACATACGAGCCGGAAGCATAAAGTGTAAAGCCTGGG | |||
GTGTCAAATGAGAGACCTAACTCACATTCAACTTTTTTTTTTTTTCTGCCCTCTATTCTA | |||
... | |||
>400269118 #24.rev | |||
TGATGAACTACTTGCAGTCGAAATCGAATCATCACTGGCCGTCCTTTTACAACGTCGTGA | |||
CTGGGAAAACCCTGGCGTTACCCAACTTAATCCGCCTTGCAGCACATCCCCCTTTCCCCC | |||
AGCTGGCGTAAAAACGTAAAAAGCCCCGCACCGATCGCCCTTTCCCAACAGGTTGCCCAG | |||
* 11. Align "vector reads" to UniVec; identify vector | |||
show-coords 24.fwd/400269118-UniVec.delta 24.rev/399553028-UniVec.delta | grep J01636.1 | |||
31 148 | 1175 1292 | 118 118 | 95.76 | 1276 7477 | 9.25 1.58 | 399553028.rev gnl|uv|J01636.1:1-7477 | |||
32 199 | 1302 1463 | 168 162 | 90.48 | 653 7477 | 25.73 2.17 | 400269118 gnl|uv|J01636.1:1-7477 | |||
* 12. 10bp distance between the 2 alignments | |||
* 13. Lucy files | |||
$ more vector.seq | |||
>J01636 E.coli lactose operon with lacI, lacZ, lacY and lacA genes | |||
GACACCATCGAATGGCGCAAAACCTTTCGCGGTATGGCATGATAGCGCCCGGAAGAGAGTCAATTCAGGG | |||
TGGTGAATGTGAAACCAGTAACGTTATACGATGTCGCAGAGTATGCCGGTGTCTCTTATCAGACCGTTTC | |||
CCGCGTGGTGAACCAGGCCAGCCACGTTTCTGCGAAAACGCGGGAAAAAGTGGAAGCGGCGATGGCGGAG | |||
CTGAATTACATTCCCAACCGCGTGGCACAACAACTGGCGGGCAAACAGTCGTTGCTGATTGGCGTTGCCA | |||
... | |||
$ more splice.seq | |||
>J01636.for.begin vector+linker.rev | |||
TGAATGTGAGTTAGGTCTCTCATTTGACACCCCAGGCTTTACACTTTATGCTTCCGGCTC | |||
GTATGTTGTGTGGAATTGTGAGCGGATAGCAATTTCACACAGGAAACAGCTATGACCATG | |||
CGCCTAATCAGATGGTACAGTAGTTCATCA | |||
>J01636.for.end rev(linker.fwd)+vector | |||
TGATGAACTACTTGCAGTCGAAATCGAATCATCACTGGCCGTCCTTTTACAACGTCGTGA | |||
CTGGGAAAACCCTGGCGTTACCCAACTTAATCCGCCTTGCAGCACATCCCCCTTTCCCCC | |||
AGCTGGCGTAAAAACGTAAAAAGCCCCGCA | |||
>J01636.rev.begin (revcomp of J01636.for.end) | |||
TGCGGGGCTTTTTACGTTTTTACGCCAGCTGGGGGAAAGGGGGATGTGCTGCAAGGCGGA | |||
TTAAGTTGGGTAACGCCAGGGTTTTCCCAGTCACGACGTTGTAAAAGGACGGCCAGTGAT | |||
GATTCGATTTCGACTGCAAGTAGTTCATCA | |||
>J01636.rev.end (revcomp of J01636.for.begin) | |||
TGATGAACTACTGTACCATCTGATTAGGCGCATGGTCATAGCTGTTTCCTGTGTGAAATT | |||
GCTATCCGCTCACAATTCCACACAACATACGAGCCGGAAGCATAAAGTGTAAAGCCTGGG | |||
GTGTCAAATGAGAGACCTAACTCACATTCA | |||
# splice=linker+vector | |||
3 120 | 1175 1292 | 118 118 | 95.76 | 150 7477 | 78.67 1.58 | J01636.for.begin J01636 | |||
32 131 | 1302 1399 | 100 98 | 96.00 | 150 7477 | 66.67 1.31 | J01636.for.end J01636 | |||
* 13.1 Align vector & splice to Ecoli | |||
1 7474 | 366812 359335 | 7474 7478 | 99.91 | 7477 4639675 | 99.96 0.16 | J01636 NC_000913.2 [CONTAINED] | |||
20 119 | 65 162 | 100 98 | 96.00 | 150 395 | 66.67 24.81 | J01636.rev.begin NC_000913.2 | |||
31 148 | 172 289 | 118 118 | 95.76 | 150 395 | 78.67 29.87 | J01636.rev.end NC_000913.2 | |||
1069 1463 | 395 1 | 395 395 | 100.00 | 7477 395 | 5.28 100.00 | J01636 NC_000913.2.365350-365744 | |||
* 14. Run lucy & trim reads | |||
$ /nfshomes/dpuiu/szdevel/SourceForge/lucy-1.19p/lucy \ | |||
-v vector.seq splice.seq | |||
-o bos_taurus.lucy.seq bos_taurus.lucy.qual \ | |||
-debug bos_taurus.lucy.info \ | |||
bos_taurus.seq bos_taurus.qual | |||
# Trim clr | |||
$ clrFasta bos_taurus.seq > bos_taurus.cseq | |||
* 15. Align lucy output to linker, vector, splice & UniVec.dust | |||
$ nucmer -l 12 -c 24 ~/db/vector.seq bos_taurus.lucy.cseq -p vector-bos_taurus.lucy | |||
$ nucmer -l 16 -c 30 ~/db/vector.seq bos_taurus.lucy.cseq -p vector-bos_taurus.lucy | |||
$ nucmer -l 16 -c 30 ~/db/splice.seq bos_taurus.lucy.cseq -p splice-bos_taurus.lucy | |||
$ nucmer -l 16 -c 30 ~/db/UniVec.dust bos_taurus.lucy.cseq -p UniVec.dust-bos_taurus.lucy | |||
* 16. Align input to linker, vector, splice & UniVec.dust | |||
$ nucmer -l 12 -c 24 ~/db/linker.seq bos_taurus.seq -p linker-bos_taurus | |||
$ nucmer -l 16 -c 30 ~/db/vector.seq bos_taurus.seq -p vector-bos_taurus | |||
$ nucmer -l 16 -c 30 ~/db/splice.seq bos_taurus.seq -p splice-bos_taurus | |||
$ nucmer -l 16 -c 30 ~/db/UniVec.dust bos_taurus.seq -p UniVec.dust-bos_taurus | |||
Count how many reads got trimmed | |||
infoseq *seq | getSummary.pl -c 1 -t original.LEN | |||
cat bos_taurus.lucy.info | awk '{print $4-$3}' | getSummary.pl -t lucy.CLR >! bos_taurus.lucy.summary | |||
cat bos_taurus.lucy.info | getSummary.pl -c 14 -t lucy.CLV5 -nh >> bos_taurus.lucy.summary | |||
cat bos_taurus.lucy.info | getSummary.pl -c 15 -t lucy.CLV3 -nh >> bos_taurus.lucy.summary | |||
== Libraries == | |||
=== 011.BCM.WGS FORWARD === | |||
* vector: J01636 | |||
* UniVec: gnl|uv|J01636.1:1-7477 E.coli lactose operon with lacI, lacZ, lacY and lacA genes | |||
ll ~dpuiu/db/J01636* | |||
-rw-rw-r-- 1 dpuiu dpuiu 7651 Jan 9 15:56 /nfshomes/dpuiu/db/J01636 | |||
-rw-rw-r-- 1 dpuiu dpuiu 105 Jan 14 07:17 /nfshomes/dpuiu/db/J01636linker | |||
-rw-rw-r-- 1 dpuiu dpuiu 840 Jan 13 13:43 /nfshomes/dpuiu/db/J01636splice | |||
cat ~dpuiu/db/J01636* | infoseq | |||
J01636 7477 53.43 | |||
J01636.linker.fwd 27 44.44 | |||
J01636.linker.rev 27 37.04 | |||
J01636.for.begin 150 44.67 | |||
J01636.for.end 150 51.33 | |||
J01636.rev.begin 150 51.33 | |||
J01636.rev.end 150 44.67 | |||
* 249,611 reads: | |||
* 91% got vector trimmed at the 5' | |||
* 0.4% (1149) got vector trimmed at the 3' | |||
#elem #0s min max mean median n50 sum | |||
original.LEN 249611 0 437 2349 1082 991 1009 270035781 | |||
lucy.CLV5 249611 21215 0 741 25.03 25 27 6247415 | |||
lucy.CLV3 249611 248462 0 1047 3.49 0 859 870344 | |||
* Original reads hit counts: | |||
10975 linker.fwd | |||
133 linker.rev | |||
166 splice | |||
152 vector | |||
228 UniVec.dust | |||
* Lucy trimmed read counts | |||
2 linker.fwd | |||
0 linker.rev | |||
1 splice | |||
1 vector | |||
6 UniVec.dust (only 3 are >40bp) | |||
=== 011.BCM.WGS REVERSE === | |||
#elem #0s min max mean median n50 sum | |||
original.LEN 250389 0 502 2148 1085 993 1012 271691094 | |||
lucy.CLR 250389 7345 0 1281 795 876 892 198982171 | |||
lucy.CLV5 250389 20271 0 668 26.52 27 29 6641362 | |||
lucy.CLV3 250389 249269 0 997 3.35 0 861 839029 | |||
* Original reads hit counts: | |||
linker.fwd 113 | |||
linker.rev 3812 | |||
splice 143 | |||
UniVec.dust 237 | |||
vector 4318 | |||
* Lucy trimmed reads hit counts: | |||
linker.fwd 1 | |||
linker.rev 0 | |||
splice 1 | |||
UniVec.dust 10 | |||
vector 1 | |||
=== 030.BCM.SHOTGUN === | |||
* same linker/vector/splice as BCM.WGS | |||
* 2.5% (4K out of 160K) reads contain linker & vector at 3' | |||
#elem #0s min max mean median n50 sum | |||
original.LEN 8411 0 325 1685 1181 1240 1314 9933150 | |||
lucy.CLR 8411 8 0 1054 841 863 874 7070994 | |||
lucy.CLV5 8411 568 0 232 27.01 28 29 227206 | |||
lucy.CLV3 8411 2325 0 1040 597 794 851 5023445 | |||
* Original reads hit counts: | |||
linker.fwd 4314 | |||
linker.rev 4125 | |||
splice 7816 | |||
UniVec.dust 4212 | |||
vector 6750 | |||
vector 27235 | |||
* Lucy trimmed reads hit counts: | |||
linker.fwd 3 | |||
linker.rev 1 | |||
splice 1 | |||
UniVec.dust 13 | |||
vector 0 | |||
=== 001.NISC.SHOTGUN === | |||
* Vector: pOTW13 | |||
* UniVec: 3 partial seqs | |||
gnl|uv|NGB00080.1:1-198 pOTW13 with linkers | |||
gnl|uv|NGB00080.1:718-888 pOTW13 with linkers | |||
gnl|uv|NGB00080.1:1490-1654-49 pOTW13 with linkers | |||
ll /nfshomes/dpuiu/db/NGB00080* | |||
-rw-rw-r-- 1 dpuiu dpuiu 1083 Jan 14 20:43 /nfshomes/dpuiu/db/NGB00080 | |||
-rw-r--r-- 1 dpuiu dpuiu 94 Jan 14 21:01 /nfshomes/dpuiu/db/NGB00080linker | |||
-rw-r--r-- 1 dpuiu dpuiu 2183 Jan 14 20:44 /nfshomes/dpuiu/db/NGB00080splice | |||
cat /nfshomes/dpuiu/db/NGB00080* | infoseq | |||
NGB00080 1054 50.00 | |||
NGB00080.linker.fwd 24 45.83 | |||
NGB00080.linker.rev 26 53.85 | |||
NGB00080.for.beg 518 46.14 | |||
NGB00080.for.end 518 50.48 | |||
NGB00080.rev.begin 518 50.48 | |||
NGB00080.rev.beg 518 46.14 | |||
* 944 read sample | |||
#elem #0s min max mean median n50 sum | |||
original.LEN 944 0 652 1017 735 721 722 693668 | |||
lucy.CLR 944 39 0 886 415 422 522 391333 | |||
lucy.CLV5 944 121 0 275 34.05 33 35 32143 | |||
lucy.CLV3 944 18 0 885 410 409 511 387007 | |||
* Original reads hit counts: | |||
linker.fwd 479 | |||
linker.rev 492 | |||
splice 910 | |||
UniVec.dust 0 | |||
vector 939 | |||
* Lucy trimmed reads hit counts: | |||
linker.fwd 1 | |||
linker.rev 0 | |||
splice 0 | |||
UniVec.dust 9 | |||
vector 1 | |||
=== 060.BCCAGSC.CLONEEND === | |||
* Linkers: | |||
linker.fwd CCCTGCTTTGTCTGGAAGGGGTTCCCGACCT | |||
linker.rev CAGGAGGGGAGAAAGGGCTCAGAGG | |||
* No common vector !!! | |||
wc -l *clb | |||
60746 bos_taurus.060.f.clb #18 reads original align to UniVec (nucmer default params) | |||
60836 bos_taurus.060.r.clb | |||
Fwd: | |||
329 428 | 440 535 | 100 96 | 91.00 | 503 1585 | 19.88 6.06 | 723951410 gnl|uv|U30497.1:3230-4814 Cloning vector pAS2-1 | |||
330 370 | 89 49 | 41 41 | 100.00 | 503 143 | 8.15 28.67 | 723951410 gnl|uv|U67875.1:6541-6683 pESP-I yeast expression vector | |||
330 370 | 94 54 | 41 41 | 100.00 | 503 143 | 8.15 28.67 | 723951410 gnl|uv|U67875.1:6541-6683 pESP-I yeast expression vector | |||
Rev: | |||
1 96 | 71 165 | 96 95 | 93.81 | 203 165 | 47.29 57.58 | 724018013 gnl|uv|AF133437.1:16659-16823 Cloning vector pCYPAC6 | |||
50 143 | 1 94 | 94 94 | 92.71 | 203 94 | 46.31 100.00 | 724018013 gnl|uv|U80929.2:2858-2951 Cloning vector pBACe3.6 | |||
=== 017.UIUC.CLONEEND === | |||
* No overrepresented kmers | |||
wc -l *clb | |||
17978 bos_taurus.017.f.clb | |||
17911 bos_taurus.017.r.clb | |||
==> 24.fwd/kmers.tab <== | |||
CCCTGCTTTGTCTGGAAGGGGTTC GAACCCCTTCCAGACAAAGCAGGG 9 | |||
CTGCTTTGTCTGGAAGGGGTTCCC GGGAACCCCTTCCAGACAAAGCAG 9 | |||
==> 24.rev/kmers.tab <== | |||
GAATGTTGAGCTTTAGCCAACTTT AAAGTTGGCTAAAGCTCAACATTC 4 | |||
TCTGAATGTTGAGCTTTAGCCAAC GTTGGCTAAAGCTCAACATTCAGA 4 | |||
==> 8.fwd/kmers.tab <== | |||
TTTTTTTT AAAAAAAA 55 | |||
AAGGGGTT AACCCCTT 35 | |||
==> 8.rev/kmers.tab <== | |||
GTCTGGAA TTCCAGAC 41 | |||
TCTGGAAG CTTCCAGA 39 | |||
* No UniVec hits | |||
=== 010.TIGR.CLONEEND === | |||
* No overrepresented kmers | |||
wc -l *clb | |||
5479 bos_taurus.032.f.clb | |||
5174 bos_taurus.032.r.clb | |||
==> 24.fwd/kmers.tab <== | |||
CTTGTGTTGGCCCAGGCAAGTCCA TGGACTTGCCTGGGCCAACACAAG 30 | |||
TTGTGTTGGCCCAGGCAAGTCCAA TTGGACTTGCCTGGGCCAACACAA 30 | |||
==> 24.rev/kmers.tab <== | |||
CTGCCTCTTGTGTTGGCCCAGGCA TGCCTGGGCCAACACAAGAGGCAG 16 | |||
GCTGCCTCTTGTGTTGGCCCAGGC GCCTGGGCCAACACAAGAGGCAGC 15 | |||
==> 8.fwd/kmers.tab <== | |||
GAGTGGGT ACCCACTC 176 | |||
GGAGTGGG CCCACTCC 171 | |||
==> 8.rev/kmers.tab <== | |||
TGGAGTGG CCACTCCA 182 | |||
GGAGTGGG CCCACTCC 181 | |||
* No UniVec hits | |||
=== ... === | |||
=== 070.BCM.CLONEEND === | |||
* No frequent kmers | |||
wc -l *clb | |||
6027 bos_taurus.070.f.clb | |||
6236 bos_taurus.070.r.clb | |||
==> 24.fwd/kmers.tab <== | |||
GGACTCTCAGAGTCTTCTCCAACA TGTTGGAGAAGACTCTGAGAGTCC 18 | |||
ACTGGTTGGATCTCCTTGCAGTCC GGACTGCAAGGAGATCCAACCAGT 18 | |||
==> 24.rev/kmers.tab <== | |||
ATAAAATCTGAGCCACCAGGGAAG CTTCCCTGGTGGCTCAGATTTTAT 1 | |||
CTATTGGTTCATATGGTCAACGTC GACGTTGACCATATGAACCAATAG 1 | |||
==> 8.fwd/kmers.tab <== | |||
TTTTTTTT AAAAAAAA 86 | |||
CTTCTCCA TGGAGAAG 75 | |||
==> 8.rev/kmers.tab <== | |||
TATAGTGT ACACTATA 9 | |||
ATATAGGG CCCTATAT 8 | |||
* No alignments to BCM WGS vector | |||
= Running Lucy = | |||
* Default parameters with vector trimming | |||
* BCM vector/splice | |||
/nfshomes/dpuiu/db/vector.BCM.seq | |||
/nfshomes/dpuiu/db/splice.BCM.seq | |||
* NISC vector/splice | |||
/nfshomes/dpuiu/db/vector.NISC.seq | |||
/nfshomes/dpuiu/db/splice.NISC.seq | |||
== BCM.WGS (all reads)== | |||
* orig.CLR < lucy.CLR ( 765 < 792 ) | |||
* orig.CLV > lucy.CLV ( 1015 > 973 ) | |||
* 739,529 out of 24,863,599 reads (3%) deleted by Lucy (CLR=-1,-1) | |||
* 21,728,592 out of 24,863,599 reads (87%) vector trimmed at the 5' end | |||
* 92,646 out of 24,863,599 reads (0.3%) vector trimmed at the 3' end | |||
'''elem <0 0 >0 min max mean median n50 sum''' | |||
orig.LEN 24863599 0 0 24863599 5 3097 1002 997 1015 24915462033 | |||
orig.CLR 24863599 463669 7 24399923 -1143 1833 '''765''' 836 864 19036744256 | |||
orig.CLR5 24863599 0 359245 24504354 0 2103 42 22 58 1047922451 | |||
orig.CLR3 24863599 463404 0 24400195 -1 2169 807 872 895 20084666707 | |||
lucy.CLR 24863599 0 739529 24124070 0 1219 '''792''' 878 904 19695000417 | |||
lucy.CLR5 24863599 '''739529''' 36108 24087962 -1 1753 43 29 42 1086413880 | |||
lucy.CLR3 24863599 '''739529''' 0 24124070 -1 1894 835 915 939 20781414297 | |||
orig.CLR5-lucy.CLR5 24863599 16299521 215345 8348733 -1186 2104 -1 -10 -1186 -38491429 | |||
orig.CLR3-lucy.CLR3 24863599 14858542 1494794 8510263 -1273 2170 -28 -20 -1273 -696747590 | |||
orig.CLV 24863599 1053 1920 24860626 -2 5345 '''1015''' 1002 1017 25260581538 | |||
orig.CLV5 8841849 0 0 8841849 1 1219 33 46 49 295011460 | |||
orig.CLV3 24861698 1053 0 24860645 -1 5346 1027 1005 1019 25555592998 | |||
lucy.CLV 24863599 10694 707 24852198 -469 3096 '''973''' 968 987 24195085877 | |||
lucy.CLV5 24863599 0 3135007 21728592 0 1359 25 27 29 623457486 | |||
lucy.CLV3 24863599 0 0 24863599 4 3096 998 995 1014 24818543363 | |||
lucy.CLVABS5 24863599 0 3135007 '''21728592''' 0 1359 25 27 29 623457486 | |||
lucy.CLVABS3 24863599 0 24770953 '''92646''' 0 1343 2 0 880 72055071 | |||
orig.CLV5-lucy.CLV5 24863599 17216820 1512453 6134326 -1312 1219 -13 -25 -1312 -328446026 | |||
orig.CLV3-lucy.CLV3 24863599 1519132 18579609 4764858 -1832 4672 29 0 479 737049635 | |||
== BCM.WGS (0 quality reads)== | |||
* orig.CLR > lucy.CLR (mean) | |||
* orig.CLV > lucy.CLV (mean) | |||
* 7,153 out of 551,114 reads (1.3%) deleted by Lucy (CLR=-1,-1) | |||
* 508,166 out of 551,114 reads (92%) vector trimmed at the 5' end | |||
* 1,946 out of 551,114 reads (0.35%) vector trimmed at the 3' end | |||
'''elem <0 0 >0 min max mean median n50 sum''' | |||
orig.LEN 551114 0 0 551114 5 1464 872 946 959 480705828 | |||
orig.CLR 551114 7754 0 543360 -770 1175 708 786 807 390325117 | |||
orig.CLR5 551114 0 6773 544341 0 1519 44 20 111 24582849 | |||
orig.CLR3 551114 7744 0 543370 -1 1638 752 818 833 414907966 | |||
lucy.CLR 551114 0 7153 543961 0 699 636 671 671 350759771 | |||
lucy.CLR5 551114 7153 35872 508089 -1 201 26 27 28 14442310 | |||
lucy.CLR3 551114 7153 0 543961 -1 699 662 699 699 365202081 | |||
orig.CLR5-lucy.CLR5 551114 364282 8801 178031 -198 1500 18 -8 215 10140539 | |||
orig.CLR3-lucy.CLR3 551114 85058 2962 463094 -700 1472 90 123 178 49705885 | |||
orig.CLV 551114 971 0 550143 -2 2037 974 978 981 537127121 | |||
orig.CLV5 5100 0 0 5100 1 845 35 29 31 180490 | |||
orig.CLV3 551114 971 0 550143 -1 2037 974 978 981 537307611 | |||
lucy.CLV 551114 58 6 551050 -84 1456 841 917 930 463903233 | |||
lucy.CLV5 551114 0 42948 508166 0 202 27 28 29 14964546 | |||
lucy.CLV3 551114 0 0 551114 4 1463 868 945 958 478867779 | |||
lucy.CLVABS5 551114 0 42948 508166 0 202 27 28 29 14964546 | |||
lucy.CLVABS3 551114 0 549168 1946 0 700 2 0 686 1286935 | |||
orig.CLV5-lucy.CLV5 551114 506108 42215 2791 -202 845 -26 -28 -202 -14784056 | |||
orig.CLV3-lucy.CLV3 551114 134959 23422 392733 -967 1614 106 7 459 58439832 | |||
== BCM.SHOTGUN == | |||
* orig.CLR < lucy.CLR (mean) | |||
* orig.CLV > lucy.CLV (mean) | |||
* 98,070 out of 10,748,529 reads (0.9%) deleted by Lucy (CLR=-1,-1) | |||
* 9,737,008 out of 10,748,529 reads (90%) vector trimmed at the 5' end | |||
* 294,942 out of 10,748,529 reads (2.7%) vector trimmed at the 3' end | |||
'''elem <0 0 >0 min max mean median n50 sum''' | |||
orig.LEN 10748529 0 0 10748529 5 2043 975 950 964 10486690472 | |||
orig.CLR 10748529 17308 2 10731219 -1293 1467 809 833 847 8701344571 | |||
orig.CLR5 10748529 0 68 10748461 0 1315 26 16 38 288662580 | |||
orig.CLR3 10748529 16780 0 10731749 -1 1647 836 851 863 8990007151 | |||
lucy.CLR 10748529 0 98070 10650459 0 1337 833 854 868 8955866769 | |||
lucy.CLR5 10748529 98070 1973 10648486 -1 1307 35 28 32 376276188 | |||
lucy.CLR3 10748529 98070 0 10650459 -1 1553 868 882 896 9332142957 | |||
orig.CLR5-lucy.CLR5 10748529 9498290 65171 1185068 -1099 1293 -8 -11 -1099 -87613608 | |||
orig.CLR3-lucy.CLR3 10748529 6879532 671097 3197900 -1149 1437 -31 -26 -1149 -342135806 | |||
orig.CLV 10748529 16779 412 10731338 -2 3919 974 948 964 10472347908 | |||
orig.CLV5 8594910 0 0 8594910 1 1239 3 1 49 28350257 | |||
orig.CLV3 10748349 16779 0 10731570 -1 3919 976 950 965 10500698165 | |||
lucy.CLV 10748529 7026 614 10740889 -268 2042 930 924 940 9997862132 | |||
lucy.CLV5 10748529 0 1011521 9737008 0 855 24 24 27 257993796 | |||
lucy.CLV3 10748529 0 0 10748529 4 2042 954 945 962 10255855928 | |||
lucy.CLVABS5 10748529 0 1011521 9737008 0 855 24 24 27 257993796 | |||
lucy.CLVABS3 10748529 0 10453587 294942 0 1214 20 0 847 220086015 | |||
orig.CLV5-lucy.CLV5 10748529 9538738 138680 1071111 -854 1239 -21 -23 -854 -229643539 | |||
orig.CLV3-lucy.CLV3 10748529 357934 9324166 1066429 -1328 2846 22 0 704 244842237 | |||
== NISC.SHOTGUN == | |||
* orig.CLR < lucy.CLR (mean) | |||
* orig.CLV > lucy.CLV (mean) | |||
* 8,248 out of 737,900 reads (1.1%) deleted by Lucy (CLR=-1,-1) | |||
* 633,409 out of 737,900 reads (85%) vector trimmed at the 5' end | |||
* 7,201 out of 737,900 reads (0.97%) vector trimmed at the 3' end | |||
'''elem <0 0 >0 min max mean median n50 sum''' | |||
orig.LEN 737900 0 0 737900 104 2104 784 729 734 579172842 | |||
orig.CLR 737900 5988 2 731910 -636 1033 651 668 676 480400909 | |||
orig.CLR5 737900 0 0 737900 1 1407 47 40 51 34857531 | |||
orig.CLR3 737900 0 5879 732021 0 1470 698 710 715 515258440 | |||
lucy.CLR 737900 0 8248 729652 0 1035 658 670 676 485757685 | |||
lucy.CLR5 737900 8248 56 729596 -1 1091 45 35 46 33811606 | |||
lucy.CLR3 737900 8248 0 729652 -1 1391 704 710 714 519569291 | |||
orig.CLR5-lucy.CLR5 737900 253727 89345 394828 -566 1408 1 1 485 1045925 | |||
orig.CLR3-lucy.CLR3 737900 177007 31 560862 -867 1471 -5 1 -867 -4310851 | |||
orig.CLV 737900 3224 2655 732021 -636 2103 771 725 730 569178445 | |||
orig.CLV5 734026 0 0 734026 1 987 5 1 35 4375315 | |||
orig.CLV3 732021 0 0 732021 35 2104 783 729 734 573553760 | |||
lucy.CLV 737900 1335 55 736510 -200 2104 747 696 702 551392388 | |||
lucy.CLV5 737900 104491 0 633409 -1 1199 30 31 34 22784742 | |||
lucy.CLV3 737900 0 0 737900 15 2103 778 728 733 574177130 | |||
lucy.CLVABS5 737900 0 104491 633409 0 1200 31 32 35 23522642 | |||
lucy.CLVABS3 737900 0 730699 7201 0 1076 5 0 686 4257812 | |||
orig.CLV5-lucy.CLV5 737900 561851 66390 109659 -1198 983 -24 -29 -1198 -18409427 | |||
orig.CLV3-lucy.CLV3 737900 8386 1 729513 -950 1077 0 1 -950 -623370 | |||
= Fragment files = | |||
* Locations: | |||
/fs/szasmg3/bos_taurus/data/frg | |||
/fs/szasmg3/bos_taurus/data/frg.new | |||
* All DST messages are unique | |||
* bos_taurus.clv : contains the vector clipping points | |||
** BCM.WGS, BCM.SHOTGUN & NISC.SHOTGUN: lucy.clv | |||
** others: the TA clv | |||
** 374,454 reads don't have valid clv's | |||
** 36,446,031 reads have valid clv's with avg=955 | |||
== Message counts (original) == | |||
DST FRG LKG | |||
bos_taurus.BCM.WGS.frg 79 24124070 11311841 | |||
#bos_taurus.BCM.SHOTGUN.frg 7339 10650459 1799069 # some libs & mates are missing due to a tarchive2ca crash (used by UMD2.1) | |||
#bos_taurus.BCM.SHOTGUN.new.frg 18208 10650459 4715172 # split the libraries by VOL & SEQ_LIB_ID (used by UMD2.2) | |||
#bos_taurus.BCM.SHOTGUN.new.frg 13826 10650459 5046435 # double check the FRG count !!! (used by UMD2.3) | |||
bos_taurus.BCM.SHOTGUN.new.frg 7 10650459 5046435 # UMD2.4 | |||
bos_taurus.NISC.SHOTGUN.frg 246 729652 344932 | |||
bos_taurus.BCCAGSC.CLONEEND.frg 1 125241 59505 | |||
bos_taurus.UIUC.CLONEEND.frg 2 114750 46319 | |||
bos_taurus.TIGR.CLONEEND.frg 1 65171 27067 | |||
bos_taurus.GSC.CLONEEND.frg 1 53521 25889 | |||
bos_taurus.CENARGEN.WGS.frg 0 26246 0 | |||
#bos_taurus.BARC.CLONEEND.frg 11150 25454 11150 # (used by UMD2.3) | |||
bos_taurus.BARC.CLONEEND.frg 1 25454 11150 # (used by UMD2.4) | |||
bos_taurus.BCM.CLONEEND.frg 1 16875 7103 | |||
bos_taurus.CENARGEN.CLONEEND.frg 1 16787 6269 | |||
bos_taurus.UOKNOR.SHOTGUN.frg 1 14651 4910 | |||
bos_taurus.TIGR_JCVIJTC.CLONEEND.frg 2 10651 4803 | |||
bos_taurus.UOKNOR.FINISHING.frg 0 151 0 | |||
bos_taurus.WUGSC.COLONEEND.frg 1 49 21 | |||
#total 25312 35973728 16896244 # (UMD2.3) <font color=red> | |||
total 344 35973728 16896244 # (UMD2.4) </font> | |||
== Message counts (quality) == | |||
DST FRG LKG | |||
bos_taurus.BCM.WGS.qual.count 79 23580109 11035582 | |||
#bos_taurus.BCM.SHOTGUN.qual.count 7339 10644092 1799069 | |||
bos_taurus.BCM.SHOTGUN.qual.new.count 18208 10644092 4712446 | |||
bos_taurus.NISC.SHOTGUN.count 246 729652 344932 | |||
bos_taurus.BCCAGSC.CLONEEND.qual.count 1 116484 53585 | |||
bos_taurus.UIUC.CLONEEND.count 2 114750 46319 | |||
bos_taurus.TIGR.CLONEEND.count 1 65171 27067 | |||
bos_taurus.CENARGEN.WGS.count 0 26246 0 | |||
bos_taurus.BARC.CLONEEND.count 11150 25454 11150 | |||
bos_taurus.BCM.CLONEEND.count 1 16875 7103 | |||
bos_taurus.CENARGEN.CLONEEND.count 1 16787 6269 | |||
bos_taurus.TIGR_JCVIJTC.CLONEEND.count 2 10651 4803 | |||
bos_taurus.UOKNOR.SHOTGUN.qual.count 1 2456 813 | |||
bos_taurus.WUGSC.COLONEEND.count 1 49 21 | |||
== Message counts (0quality) == | |||
DST FRG LKG | |||
bos_taurus.BCM.WGS.0qual.count 79 543961 234397 | |||
bos_taurus.GSC.CLONEEND.0qual.count 1 53521 25889 | |||
bos_taurus.UOKNOR.SHOTGUN.0qual.count 1 12195 4097 | |||
bos_taurus.BCCAGSC.CLONEEND.0qual.count 1 8757 2114 | |||
bos_taurus.BCM.SHOTGUN.0qual.count 7339 6367 0 | |||
bos_taurus.UOKNOR.FINISHING.0qual.count 0 151 0 | |||
= Assemblies = | |||
[[Media:Bt.qc.combine|Bt.qc.combine]] UMD2.0 ... UMD2.5 combine stats | |||
= UMD2.1(2009_0122_CA; Quality reads) = | |||
== Issues == | |||
# Uses only quality reads | |||
# BCM.SHOTGUN library : ~ 4715172-1799069=2.9M mates were missed due to a tarchive2ca crash ; some libraries got merged (were assigned the same lib_id) | |||
# All reads except for BCM.WGS were set as nonrandom | |||
# Update the runCA script to run overlapper concurently; new "ovlConcurrency" parameter added to the .spec file !!! | |||
# consensus after cgw crashed in MultiAlignContig() ... use "consensus -D forceunitigabut" !!! | |||
# cgw crashed after updating gkpStore with new lib/mate info => edit Input_CGW.c, remove the assert in line 117 | |||
== Info == | |||
host: walnut | |||
assembly version: wgs-5.2 stable | |||
dir: /scratch1/bos_taurus/Assembly/2009_0122_CA | |||
command: /fs/szdevel/dpuiu/SourceForge/wgs/Linux-amd64/bin/runCA-test -d . -p bt -s bt01.specFile *.frg | |||
spec file: | |||
cgwDistanceSampleSize = 1000 # ??? too big; more than 50% of the BCM.SHOTGUN reads are in libraries with less than 1000 inserts | |||
cnsConcurrency = 15 | |||
cnsMinFrags = 200000 | |||
doOverlapTrimming = 1 | |||
frgCorrBatchSize = 100000 | |||
frgCorrConcurrency = 15 | |||
merylMemory = 24000 | |||
merylThreads = 15 | |||
obtMerThreshold = 200 | |||
obtOverlapper = ovl | |||
ovlConcurrency = 8 | |||
ovlCorrBatchSize = 100000 | |||
ovlCorrConcurrency = 15 | |||
ovlHashBlockSize = 1200000 | |||
ovlMemory = 8GB --hashload 0.8 --hashstrings 400000 | |||
ovlMerThreshold = 500 | |||
ovlOverlapper = ovl | |||
ovlRefBlockSize = 7200000 | |||
ovlThreads = 2 | |||
unitigger = utg | |||
utgErrorRate = 0.015 | |||
vectorIntersect = bos_taurus.clv | |||
doExtendClearRanges = 2 | |||
== Steps == | |||
1. Run up till after initialStoreBuilding | |||
runCA stopAfter=initialStoreBuilding ... | |||
2. Update gkpStore with nonrandom frg flag | |||
cat bos_taurus.nonrandom.clv | perl -ane 'print "frg uid $F[0] isnonrandom 1\n";' > bos_taurus.nonrandom.edit | |||
gatekeeper -edit bos_taurus.nonrandom.edit bt.gkpStore | |||
== Input == | |||
gatekeeper -dumpinfo -lastfragiid bt.gkpStore | |||
... | |||
Last frag in store is iid = 35348776 | |||
== OBT == | |||
elem <0 0 >0 min max mean median n50 sum | |||
CLV5 35085508 0 3387027 31698481 0 970 25 27 29 891007232 | |||
CLV3 35164784 0 0 35164784 15 2974 984 980 1000 34612019144 | |||
CLR_ORIG5 35348776 0 43354 35305422 0 1753 42 29 38 1502168205 | |||
CLR_ORIG3 35348776 0 0 35348776 70 1894 864 905 927 30547294868 | |||
CLR_OBT5 35348776 0 26513 35322263 0 1690 49 30 73 1756346429 | |||
CLR_OBT3 35348776 0 23477 35325299 0 1813 843 895 914 29824543869 | |||
<font color=red> | |||
* 421,379 reads deleted by OBT: why so many??? | |||
</font> | |||
* Chimera: | |||
20297 reads too short => deleted | |||
* more 0-overlaptrim/bt.mergeLog.stats | |||
... | |||
211037: short or inconsistent | |||
253536: deleted fragment due to zero clear | |||
* Example: | |||
gatekeeper -dumpfragments 516316990 bt.gkpStore | |||
fragmentIdent = 516316990,14 | |||
fragmentMate = 0,0 | |||
fragmentLibrary = 27473,1563 | |||
fragmentIsDeleted = 1 | |||
fragmentIsNonRandom = 1 | |||
fragmentStatus = G | |||
fragmentOrientation = I | |||
fragmentHasVectorClear = 0 | |||
fragmentHasQualityClear = 0 | |||
fragmentPlate = 0 | |||
fragmentPlateLocation = 0 | |||
fragmentSeqLen = 862 | |||
fragmentHPSLen = 0 | |||
fragmentSrcLen = 17 | |||
fragmentClearORIG = 38,553 | |||
fragmentClearQLT = 1,0 | |||
fragmentClearVEC = 1,0 | |||
fragmentClearOBTINI = 35,578 | |||
fragmentClearOBT = 35,578 | |||
fragmentClearUTG = 35,578 | |||
fragmentClearECR1 = 35,578 | |||
fragmentClearECR2 = 35,578 | |||
fragmentSeqOffset = 5376 | |||
fragmentQltOffset = 11038 | |||
fragmentHpsOffset = 53 | |||
fragmentSrcOffset = 287 | |||
cat 0-overlaptrim/bt.mergeLog | grep 516316990 | |||
516316990,14 412 412 0 0 (deleted, too short) | |||
zcat *r000*gz | convertOverlap -a -obt | |||
... | |||
14 12128740 f 377 478 292 393 2.97 | |||
14 15226267 f 397 446 31 80 2.04 | |||
14 19071241 f 4 513 199 708 1.18 | |||
14 20073917 f 7 478 36 508 4.88 | |||
14 20042424 f 4 419 299 714 1.93 | |||
14 20212935 f 7 478 234 706 4.88 | |||
14 20073828 r 7 478 507 35 4.67 | |||
14 20212846 r 7 478 557 85 4.67 | |||
14 27089060 r 491 534 836 793 2.33 | |||
14 29061748 f 489 540 86 137 1.96 | |||
14 32105697 f 455 543 381 469 2.27 | |||
14 32187461 f 430 534 105 209 1.92 | |||
14 32027289 f 4 419 493 907 4.59 | |||
... | |||
#read aligns to contigs | |||
show-coords 516316990-ctg.filter-r.strict.delta | |||
35 531 | 97 594 | 497 498 | 99.20 | 862 2759 | 57.66 18.05 | 516316990 ctg7180001872751 | |||
45 678 | 931 1564 | 634 634 | 97.00 | 862 1567 | 73.55 40.46 | 516316990 ctg7180001837311 | |||
* OBT deleted reads: | |||
BCM WGS 253816 | |||
BCM SHOTGUN 151770 | |||
BCCAGSC CLONEEND 7510 | |||
NISC SHOTGUN 4757 | |||
TIGR CLONEEND 1577 | |||
CENARGEN WGS 599 | |||
CENARGEN CLONEEND 431 | |||
TIGR_JCVIJTC CLONEEND 377 | |||
UIUC CLONEEND 182 | |||
BCM CLONEEND 150 | |||
BARC CLONEEND 125 | |||
UOKNOR SHOTGUN 85 | |||
total . 421379 | |||
OBT deleted reads: | |||
elem >0 min max mean med n50 sum | |||
len 421379 421379 98 2974 862 927 968 363280405 | |||
avgQual 421379 421379 1 57 28 24 36 11852865 | |||
== Overlapper == | |||
** 98.33% of the reads (34,761,786 out of 35,348,776 reads) had overlaps | |||
** 1.66% of the reads had no overlaps | |||
** 6.68% of the BCCAGSC.CLONEEND reads had no overlaps | |||
** 4.95% of the TIGR_JCVIJTC.CLONEEND reads had no overlaps | |||
** 3.48% of the TIGR.CLONEEND reads had no overlaps | |||
** the median number of overlaps is 20 | |||
Overlaps | |||
reads min max mean median n50 sum | |||
qual 35348776 0 5592 106 20 769 3777789082 | |||
** the median number of overlaps for the BCM.WGS reads is 16 | |||
** the median number of overlaps for the BCM.SHOTGUN reads is 16 !!! | |||
** the median number of overlaps for the NISC.SHOTGUN reads is 40 !!! | |||
** the median number of overlaps for the BCM.CLONEEND reads is 16 !!! | |||
[[Media:Bt.ovlStore.big.png]] , [[Media:Bt.ovlStore.small.png]] | |||
== Unitigger == | |||
more 4-unitigger/bt.cga.0 | |||
UNITIG OVERLAP GRAPH INFORMATION | |||
5208738 : Total number of unitigs | |||
2527051 : Total number of singleton, contained unitigs | |||
1814842 : Total number of singleton, non-contained unitigs | |||
180910 : Total number of non-singleton, spanned unitigs | |||
685935 : Total number of non-singleton, non-spanned unitigs | |||
34927397 : Total number of fragments | |||
34927397 : Total number of fragments in all unitigs | |||
21521581 : Total number of essential fragments in all unitigs | |||
13405816 : Total number of contained fragments in all unitigs | |||
0.0076239952 : Randomly sampled fragment arrival rate per bp | |||
2510896132 : The sum of overhangs in all the unitigs | |||
6400342737 : Total number of bases in all unitigs | |||
0 : Estimated number of base pairs in the genome. | |||
0 : Total number of contained fragments not connected | |||
by containment edges to essential fragments. | |||
Total rho = 2510896132 | |||
Total nfrags = 19143061 | |||
Estimated genome length = 0 | |||
Estimated global_fragment_arrival_rate=0.007624 | |||
Computed global_fragment_arrival_rate =0.007624 | |||
Total number of randomly sampled fragments in genome = 23326293 | |||
Computed genome length = 3059589120.000000 | |||
Used global_fragment_arrival_rate=0.007624 | |||
Used global_fragment_arrival_distance=131.164826 | |||
Histogram of the number of base pairs in a chunk | |||
100292 - 159434: 22 | |||
90010 - 99906: 25 | |||
80043 - 89676: 73 | |||
70013 - 79966: 162 | |||
60010 - 69988: 389 | |||
50008 - 59983: 977 | |||
40000 - 49998: 2434 | |||
30000 - 39997: 6458 | |||
20000 - 29999: 18957 | |||
10000 - 19999: 57442 | |||
Unitigs >=10kb | |||
NewAsm UMd2Asm | |||
Number 86,939 57,204 | |||
Mean 19,464 15,140 | |||
Sum 1,692.1Mb 866.0Mb | |||
max 159,434bp 78,570bp | |||
Contigs >=10Kb: | |||
NewAsm UMd2Asm | |||
n 42,343 45,958 | |||
mean 59,856 55,473 | |||
sum 2,534.5Mb 2,549.4Mb | |||
Contigs >=100Kb: | |||
NewAsm UMd2Asm | |||
n 7,051 6,683 | |||
mean 163,170 162,357 | |||
sum 1,150.5Mb 1,085.0Mb | |||
max 627,705 742,802 | |||
Scaffolds >=10Mb: | |||
NewAsm UMd2Asm | |||
n 30 3 | |||
mean 14.10Mb 11.36Mb | |||
sum 422.95Mb 340.70Mb | |||
max 26.54Mb 13.36Mb | |||
== CGW & ECR == | |||
* Checkpoints: | |||
cat 7-0-CGW/bt.timing | grep ^Checkpoint | |||
Checkpoint 3 written during MergeScaffoldsAggressive at iteration 49 | |||
Checkpoint 4 written during MergeScaffoldsAggressive at iteration 85 | |||
Checkpoint 5 written after 1st Scaffold Merge | |||
Checkpoint 6 written after 2nd Aggressive Scaffold Merge | |||
Checkpoint 7 written after Final Rocks | |||
cat 7-2-CGW/bt.timing | grep ^Checkpoint | |||
Checkpoint 19 written during MergeScaffoldsAggressive at iteration 12 | |||
Checkpoint 20 written during MergeScaffoldsAggressive at iteration 31 | |||
Checkpoint 21 written after 1st Scaffold Merge | |||
Checkpoint 22 written after 2nd Aggressive Scaffold Merge | |||
Checkpoint 23 written after Final Rocks | |||
cat 7-4-CGW/bt.timing | grep ^Checkpoint | |||
Checkpoint 34 written during MergeScaffoldsAggressive at iteration 12 | |||
Checkpoint 35 written during MergeScaffoldsAggressive at iteration 49 | |||
Checkpoint 36 written after 1st Scaffold Merge | |||
Checkpoint 37 written during Stones CleanupScaffolds after scaffold 32436 | |||
Checkpoint 38 written during Stones CleanupScaffolds after scaffold 34939 | |||
Checkpoint 39 written after Stone Throwing and CleanupScaffolds | |||
Checkpoint 40 written after 2nd Aggressive Scaffold Merge | |||
Checkpoint 41 written after Final Rocks | |||
Checkpoint 42 written after Partial Stones | |||
Checkpoint 43 written after Final Contained Stones | |||
Checkpoint 44 written after resolveSurrogates | |||
* Get early CTG/SCF stats | |||
cat 7-CGW/bt.cgw_scaffolds | countMessages.pl | |||
ICL 451555 # ??? | |||
ICP 116455 # CTG | |||
ISF 66141 # SCF | |||
ISL 711 # SLK | |||
* Clear read extension: | |||
elem <0 0 >0 min max mean median n50 sum | |||
ClearORIG 35348776 4 0 35348772 -1147 1572 821 870 893 29045126663 | |||
ClearQLT 35348776 35348776 0 0 -1 -1 -1 -1 -1 -35348776 | |||
ClearVEC 35348776 299034 20323 35029419 -1 2043 952 953 975 33658445088 | |||
ClearOBTINI 35348776 0 31254 35317522 0 1364 831 879 902 29394688367 | |||
ClearOBT 35348776 0 31254 35317522 0 1318 794 854 877 28068197440 | |||
ClearECR1 35348776 0 31254 35317522 0 1329 794 854 877 28072014464 | |||
ClearECR2 35348776 0 31254 35317522 0 1329 794 854 877 28072365712 | |||
sum(ClearECR1)-sum(ClearUTG) = 3,817,024 | |||
sum(ClearECR2)-sum(ClearECR1)= 351,248 | |||
* Scaffold length stats: | |||
cat 7-0-CGW/stat/final0.Scaffolds.nodelength.cgm | grep -v ^Sca | getSummary.pl -t 0 # 0,2,4 | |||
... | |||
step scaff min max mean med n50 sum | |||
0 7048 2249 19719008 385020 21967 3114907 2713622175 | |||
2 4960 2249 21907006 540915 21181 4490171 2682939682 | |||
4 4006 2391 26541374 668427 29193 4590744 2677722052 | |||
* Last cgw | |||
cat 7-4-CGW/stat/final0.*Scaffolds.nodelength.cgm | grep -v ^Scaff | getSummary.pl -t scf | |||
cat 7-4-CGW/stat/final0.PlacedContig.n | grep -v ^Scaff | getSummary.pl -t scf | |||
elem min max mean med n50 sum | |||
scf 66141 432 26541374 42648 1347 4349378 2820819506 | |||
ctg 120461 65 627705 22421 2018 84989 2700959854 | |||
== QC stats == | |||
* [[Media:Bos_taurus.qc|Bos_taurus.qc]] this assembly stats | |||
* [[Media:Bos_taurus.qc.combine|Bos_taurus.qc.combine]] UMD2 vs this assembly stats | |||
TotalScaffolds=66,141 | |||
MaxBasesInScaffolds=26,048,998 | |||
MeanBasesInScaffolds=40,861 | |||
TotalContigsInScaffolds=120,461 | |||
MaxContigLength=627,911 | |||
MeanContigLength=22,436 | |||
TotalDegenContigs=269,031 | |||
MaxDegenContig=33,824 | |||
SingletonReads=3,721,123 | |||
* Posmap info | |||
cat bt.posmap.mates | awk '{print $3}' |count.pl -p 100 | |||
good 10338164 | |||
bothChaff 1160137 | |||
oneChaff 695982 | |||
oneSurrogate 233151 | |||
bothDegen 218198 | |||
diffScaffold 150423 | |||
badShort 138464 | |||
oneDegen 118232 | |||
badLong 23196 | |||
badSame 22451 | |||
badOuttie 8751 | |||
bothSurrogate 589 | |||
total 13107738 | |||
cat bt.posmap.frags | awk '{print $4,$5}' |count.pl -p 100 | |||
placed good 20676328 | |||
placed notMated 8007072 | |||
chaff bothChaff 2320274 | |||
chaff notMated 704849 | |||
placed oneChaff 695982 | |||
chaff oneChaff 695982 | |||
placed oneSurrogate 466302 | |||
placed bothDegen 436396 | |||
placed diffScaffold 300846 | |||
placed badShort 276928 | |||
placed oneDegen 236464 | |||
placed badLong 46392 | |||
placed badSame 44902 | |||
placed badOuttie 17502 | |||
placed bothSurrogate 1178 | |||
total 34927397 | |||
== Log files == | |||
* [[Media:Bt.runCA.log|Bt.runCA.log]] | |||
* [[Media:Bt.runCA.hourly.runtimes|Bt.runCA.hourly.runtimes]] approximate running times (in hours) | |||
== Analysis == | |||
=== Insert libraries === | |||
1. BCM.WGS : ok | |||
* FRG.mea: 1750-7000 | |||
* ASM.mea: 1594-6727 | |||
* Most libs have > 1000 reads & get reestimated | |||
* All libs have ASM.std< ASM.mea/3 | |||
2. BCM.SHOTGUN | |||
* only ~ 50% of the inserts are in libs with >1000 inserts and get reestimated by the assembly | |||
* if the thold is dropped from 1000 to 100, we'd get ~ 95% of the inserts reestimated | |||
elem <0 0 >0 min max mean median n50 sum | |||
0 7339 0 0 7339 1 11237 245 135 1137 1799069 | |||
100 4361 0 0 4361 100 11237 395 157 1252 1725604 | |||
1000 440 0 0 440 1008 11237 2075 1791 2323 913086 | |||
3. NISC.SHOTGUN: ok | |||
* Most libs have > 1000 reads & get reestimated | |||
* All libs have ASM.std< ASM.mea/3 | |||
4. BCCAGSC.CLONEEND: ok | |||
LIB.id FRG.mea FRG.std FRG.count CENTER.TYPE ASM.mea ASM.std | |||
125606 150000 30000 59505 BCCAGSC.CLONEEND 161998 20133 | |||
5. UIUC.CLONEEND: ok | |||
LIB.id FRG.mea FRG.std FRG.count CENTER.TYPE ASM.mea ASM.std | |||
114892 150000 30000 31063 UIUC.CLONEEND 175594 41208 | |||
115020 150000 30000 15256 UIUC.CLONEEND 162488 26358 | |||
6. TIGR.CLONEEND: originally wrong; gets reestimated | |||
LIB.id FRG.mea FRG.std FRG.count CENTER.TYPE ASM.mea ASM.std | |||
65177 2000 600 27067 TIGR.CLONEEND 161761 34938 | |||
7. GSC.CLONEEND: not used (all 53556 are 0 qual) | |||
8. CENARGEN.WGS: "not used" (all 26246 are unmated) | |||
9. BARC.CLONEEND: each library contains 1 template id => inserts did not get reestimated (25454 reads/11151 inserts) | |||
10. BCM cloneend: ok | |||
LIB.id FRG.mea FRG.std FRG.count CENTER.TYPE ASM.mea ASM.std | |||
19070 167000 25000 7103 BCM.CLONEEND 171244 18555 | |||
11. CENARGEN.CLONEEND: large stdev | |||
LIB.id FRG.mea FRG.std FRG.count CENTER.TYPE ASM.mea ASM.std | |||
17249 202000 20200 6269 CENARGEN.CLONEEND 158938 55165 | |||
12. UOKNOR.SHOTGUN: ok ? | |||
LIB.id FRG.mea FRG.std FRG.count CENTER.TYPE ASM.mea ASM.std | |||
15158 3000 1000 4910 UOKNOR.SHOTGUN 3000 1000 | |||
13. TIGR_JCVI.CLONEEND: originally wrong; gets reestimated | |||
LIB.id FRG.mea FRG.std FRG.count CENTER.TYPE ASM.mea ASM.std | |||
10691 2500 750 2763 TIGR_JCVI.CLONEEND 160363 29580 | |||
10738 2500 750 2040 TIGR_JCVI.CLONEEND 161915 29343 | |||
14. UOKNOR.FINISHING: only 151 reads | |||
15. WUGSC.CLONEEND: only 49 reads | |||
=== Contigs Vs UMD2 contaminants & Ecoli === | |||
4865 contigs in list.exclude_contigs.fa | |||
34404 exclude-ctg.qry_hits | |||
3763 exclude-ctg.ref_hits | |||
1204 exclude-ctg.CBE.qry_hits CONTAIN|IDENTITY|BEGIN|END | |||
748 exclude-ctg.CBE.ref_hits CONTAIN|IDENTITY|BEGIN|END | |||
559 Ecoli.365350-365744-ctg.qry_hits : max ctg aligned is 179K bp; 10 are > 10K bp | |||
=== Contigs Vs UMD2 chromosomes === | |||
* Split 120,461 contigs into 100 files; degeneartes not split | |||
* Align them to the 31 chromosomes 1..30,U (ref) => 101*31 jobs | |||
#Alignment stats | |||
cat chr*ctg*delta | grep "^>" | awk '{print $2}' | count.pl -f ../9-terminator/bt.ctg.infoseq | getSummary.pl -i 1 -z 1 | |||
ctg 0 1 >1 min max mean med n50 sum | |||
120461 652 37540 82269 0 176 11 3 33 1422808 | |||
#Unaligned ctg lengths | |||
ctg min max mean med n50 sum | |||
652 65 5849 1146 1134 1194 747295 | |||
* 50% of the contigs aligned uniquely | |||
cat chr*-ctg*.delta | ~/bin/mergeDelta.pl > chr-ctg.delta | |||
# degens? | |||
delta-filter -q chr-ctg.delta >> chr-ctg.filter-q.delta | |||
cat chr1-*.delta | ~/bin/delta2cvg.pl -M 0 | getSummary.pl -i 4 | |||
elem 0 >0 min max mean med n50 sum | |||
6681 1 6680 0 12892 366 142 1095 2450106 | |||
* There are disagreements: | |||
/fs/sz-user-supported/Linux-x86_64/bin/show-coords -l -r -H chr1-ctg.filter-q.delta | p 'print $F[-1],"\n";' | count.pl | head | |||
ctg7180001761585 24 | |||
... | |||
ctg7180001634116 7 | |||
... | |||
show-coords -d chr1-ctg.filter-q.delta | grep ctg7180001761585 | p 'print " $_";' | |||
142115744 142188863 | 383463 310345 | 73120 73119 | 99.98 | 157714772 383463 | 0.05 19.07 | 1 -1 chr1 ctg7180001761585 | |||
142188878 142286012 | 310361 213223 | 97135 97139 | 99.94 | 157714772 383463 | 0.06 25.33 | 1 -1 chr1 ctg7180001761585 | |||
142287100 142287675 | 212133 211556 | 576 578 | 98.27 | 157714772 383463 | 0.00 0.15 | 1 -1 chr1 ctg7180001761585 | |||
142288052 142288602 | 211182 210633 | 551 550 | 99.09 | 157714772 383463 | 0.00 0.14 | 1 -1 chr1 ctg7180001761585 | |||
142288652 142295709 | 210586 203531 | 7058 7056 | 99.87 | 157714772 383463 | 0.00 1.84 | 1 -1 chr1 ctg7180001761585 | |||
142295709 142342174 | 203512 157047 | 46466 46466 | 100.00 | 157714772 383463 | 0.03 12.12 | 1 -1 chr1 ctg7180001761585 | |||
142346440 142367791 | 156958 135606 | 21352 21353 | 99.99 | 157714772 383463 | 0.01 5.57 | 1 -1 chr1 ctg7180001761585 | |||
142367822 142370681 | 135597 132737 | 2860 2861 | 99.93 | 157714772 383463 | 0.00 0.75 | 1 -1 chr1 ctg7180001761585 | |||
142370660 142382289 | 132746 121117 | 11630 11630 | 99.88 | 157714772 383463 | 0.01 3.03 | 1 -1 chr1 ctg7180001761585 | |||
142382282 142411927 | 120984 91339 | 29646 29646 | 99.96 | 157714772 383463 | 0.02 7.73 | 1 -1 chr1 ctg7180001761585 | |||
142411941 142419553 | 91339 83728 | 7613 7612 | 99.66 | 157714772 383463 | 0.00 1.99 | 1 -1 chr1 ctg7180001761585 | |||
142419553 142434546 | 83721 68728 | 14994 14994 | 99.79 | 157714772 383463 | 0.01 3.91 | 1 -1 chr1 ctg7180001761585 | |||
142434506 142437288 | 68778 65996 | 2783 2783 | 99.86 | 157714772 383463 | 0.00 0.73 | 1 -1 chr1 ctg7180001761585 | |||
142437389 142439015 | 66757 65131 | 1627 1627 | 99.94 | 157714772 383463 | 0.00 0.42 | 1 -1 chr1 ctg7180001761585 | |||
142439271 142440703 | 65629 64197 | 1433 1433 | 100.00 | 157714772 383463 | 0.00 0.37 | 1 -1 chr1 ctg7180001761585 | |||
142441869 142442975 | 63548 62442 | 1107 1107 | 100.00 | 157714772 383463 | 0.00 0.29 | 1 -1 chr1 ctg7180001761585 | |||
142446690 142449325 | 30312 32945 | 2636 2634 | 99.58 | 157714772 383463 | 0.00 0.69 | 1 1 chr1 ctg7180001761585 | |||
142451384 142452476 | 63510 64603 | 1093 1094 | 99.91 | 157714772 383463 | 0.00 0.29 | 1 1 chr1 ctg7180001761585 | |||
142452577 142454379 | 61000 62806 | 1803 1807 | 99.78 | 157714772 383463 | 0.00 0.47 | 1 1 chr1 ctg7180001761585 | |||
142454487 142456821 | 59122 61456 | 2335 2335 | 100.00 | 157714772 383463 | 0.00 0.61 | 1 1 chr1 ctg7180001761585 | |||
142458383 142459582 | 57978 59177 | 1200 1200 | 100.00 | 157714772 383463 | 0.00 0.31 | 1 1 chr1 ctg7180001761585 | |||
142459738 142472295 | 32272 44828 | 12558 12557 | 99.92 | 157714772 383463 | 0.01 3.27 | 1 1 chr1 ctg7180001761585 | |||
142472300 142485640 | 44828 58163 | 13341 13336 | 99.89 | 157714772 383463 | 0.01 3.48 | 1 1 chr1 ctg7180001761585 | |||
142501686 142530021 | 28336 1 | 28336 28336 | 99.99 | 157714772 383463 | 0.02 7.39 | 1 -1 chr1 ctg7180001761585 | |||
show-coords -d chr1-ctg.filter-q.delta | grep ctg7180001634116 | |||
116312 162914 | 1 46603 | 46603 46603 | 99.99 | 157714772 122722 | 0.03 37.97 | 1 1 chr1 ctg7180001634116 | |||
164916 201988 | 58062 95135 | 37073 37074 | 99.99 | 157714772 122722 | 0.02 30.21 | 1 1 chr1 ctg7180001634116 | |||
203244 213377 | 48198 58331 | 10134 10134 | 100.00 | 157714772 122722 | 0.01 8.26 | 1 1 chr1 ctg7180001634116 | |||
261393 264506 | 45949 49062 | 3114 3114 | 100.00 | 157714772 122722 | 0.00 2.54 | 1 1 chr1 ctg7180001634116 | |||
264607 268579 | 94345 98317 | 3973 3973 | 100.00 | 157714772 122722 | 0.00 3.24 | 1 1 chr1 ctg7180001634116 | |||
268586 274734 | 98323 104471 | 6149 6149 | 100.00 | 157714772 122722 | 0.00 5.01 | 1 1 chr1 ctg7180001634116 | |||
274835 293945 | 103611 122722 | 19111 19112 | 99.99 | 157714772 122722 | 0.01 15.57 | 1 1 chr1 ctg7180001634116 | |||
~/bin/delta2breaks.pl -m 200 < chr1-ctg.filter-q.delta | awk '{print $8}' | count.pl | |||
AGREEMENT 9827 | |||
INVERSION 283 | |||
TRANSLOCATION+ 230 | |||
TRANSLOCATION- 154 | |||
~/bin/delta2breaks.pl -m 1000 < chr1-ctg.filter-q.delta | awk '{print $8}' | count.pl | |||
AGREEMENT 7564 | |||
INVERSION 216 | |||
TRANSLOCATION+ 192 | |||
TRANSLOCATION- 127 | |||
~/bin/delta2breaks.pl -m 10000 < chr1-ctg.filter-q.delta | awk '{print $8}' | count.pl | |||
AGREEMENT 3394 | |||
INVERSION 62 | |||
TRANSLOCATION+ 50 | |||
TRANSLOCATION- 29 | |||
= Assembly UMD2.2 (Quality reads) = | |||
* Try to add the missing BCM.SHOTGUN reads at the assembly | |||
* Assign new BCM.SHOTGUN library ID's base on volume & SEQ_LIB_ID : same library might have different insert size in different volume => might loose some correct mates from different volumes | |||
cat bos_taurus.summary | grep BCM | grep SHOTG | cut -f6,7,8,10 | sort | more | |||
FAAEP 180000 13000 252 | |||
FAAEP 2000 1000 84 | |||
... | |||
FAAHP 180000 13000 77 | |||
FAAHP 2000 1000 230 | |||
... | |||
* => 20,538 libraries out of which 18,208 contain mated reads | |||
* create DST messages & add them to gkpStore | |||
gatekeeper -a -o bt.gkpStore -T -F bos_taurus.BCM.SHOTGUN.new.DST | |||
* generate gatekeeper edit file that maps each TI to the new library id | |||
head bos_taurus.BCM.SHOTGUN.new.ti2libinfo.edit | |||
frg uid 499507131 libuid 601081 | |||
frg uid 499507132 libuid 601081 | |||
... | |||
* generate gatekeeper edit file that deletes all mate information | |||
head bos_taurus.BCM.SHOTGUN.new.mate.delete | |||
frg uid 500086180 mateuid 0 | |||
frg uid 500084310 mateuid 0 | |||
... | |||
* pair forward/reverse read that have the same new library id, same TEMPLATE_ID | |||
head bos_taurus.BCM.SHOTGUN.new.mate.edit | |||
frg uid 583866821 mateuid 583872364 | |||
frg uid 583866822 mateuid 583872408 | |||
... | |||
* run gatekeeper --edit for each edit/delete file | |||
gatekeeper --edit ... bt.gkpStore | |||
* restart assembly at cgw (doExtendClearRanges=1) | |||
* consensus after cgw failed on job 25 on CTG 5597062 : cannot create consensus from multialignment ... | |||
Fix: delete failed message | |||
cp bt.cgw_contigs.25 bt.cgw_contigs.25.FAILED | |||
delete "{ICM acc:5597062 pla:P len:20889 ..." from bt.cgw_contigs.25 | |||
* terminator fail; message: | |||
ICL: reference before definition error for contig ID 5597062 | |||
= Assembly UMD2.3 (2009_0210_CA; all reads) = | |||
<font color=red> | |||
* 35,973,728 reads : 35,348,776 quality & 624,952 quality-less | |||
* 16,896,244 mates | |||
* 25,312 libraries | |||
</font> | |||
Issues (not solved): | |||
* 10420 contain at least 1 "NN" in their clr (50.. min(len,600)) | |||
* 5973 contain at least 1 "NNN" in their clr (50.. min(len,600)) | |||
== Quality-less clrs == | |||
* 624,952 quality-less reads | |||
* Quality-less read stats: : alignment CLR or 50..min(len,600) trimming | |||
elem min max mean median n50 sum | |||
len 624952 5 1495 887 947 961 554429198 | |||
5 624952 6 1584 51 51 51 32150411 | |||
3 624952 5 1495 695 699 699 434960697 | |||
53 624952 -1579 1444 644 648 648 402810286 | |||
<font color=red> | |||
* Align 624,952 to the 120,461 Assembly1 contigs (no degenerates) : 1 day on 13 cpus | |||
* 572,140(91.5%) reads aligned and 52,812(8.5%) did not align to the contigs | |||
</font> | |||
1. Launch jobs in parallel: 12766 jobs on 13 processors | |||
nucmer -l 50 -c 200 -b 10 -g 5 -d 0.05 bt.ctg.001.fasta bos_taurus.0qual.01.seq -p ctg.001-seq.01 | |||
... | |||
nucmer -l 50 -c 200 -b 10 -g 5 -d 0.05 bt.ctg.982.fasta bos_taurus.0qual.13.seq -p ctg.001-seq.01 | |||
* CPU usage: 100% /job | |||
* Max mem usage: 0.1% /job | |||
2. Get maximum extended clrs | |||
cat *delta | ~/bin/delta2qryClr.pl -best | sort > bos_taurus.0qual.best.clr | |||
Length stats | |||
elem min max mean median n50 sum | |||
all 624952 5 1495 887 947 961 554429198 | |||
aligned 572140 221 1416 912 953 964 522281354 | |||
unaligned 52812 5 1495 608 580 754 32147844 | |||
Best/Max/Max+extended alignment coord stats: | |||
elem min max mean median n50 sum | |||
53.best 572140 94 1208 766 841 877 438793102 | |||
53.max 572140 170 1208 794 863 888 454816817 | |||
53.extend 572140 170 1208 797 865 889 456014184 | |||
Unaligned read counts: | |||
unaligned total quality quality-less | |||
BCM.WGS 42595 | |||
UOKNOR.SHOTGUN 5787 14651 2456 12195 | |||
GSC.CLONEEND 2294 53521 0 53521 | |||
BCCAGSC.CLONEEND 1869 125241 116484 8757 | |||
BCM.SHOTGUN 186 | |||
UOKNOR.FINISHING 81 | |||
* 52,812 quality-less unaligned reads to the contigs using less strict nucmer parameters: -l 30 -c 50 -b 50 -g 50 -d 0.12 | |||
* 9,269 reads aligned at an average 92% identity (min 81% identity) : not too good | |||
3. Get reads without clrs: set their clr to maximum 50..600 | |||
cp bos_taurus.0qual.extended.clr bos_taurus.0qual.clr | |||
difference.pl bos_taurus.0qual.infoseq bos_taurus.0qual.extended.clr | perl -ane '$three=600; $three=$F[1] if ($F[1]<600); print "$F[0] 50 $three\n";' >> bos_taurus.0qual.clr | |||
== Quality clrs == | |||
* Use Assembly1 OBT clrs | |||
* Delete reads deleted in the OBT process | |||
== Gatekeeper == | |||
Load order: | |||
# Add quality FRG : "gatekeeper -T -F ..." | |||
# Add quality-less FRG "gatekeeper -F -a ..." # -T should be removed | |||
# Delete quality FRG (deleted by UMD2.1 OBT) | |||
# Add DST | |||
# Add LKG | |||
Edit | |||
# Loads clrs | |||
# Loads clvs | |||
# Loads nonrandom info | |||
== Meryl == | |||
Use Assembly1 kmer counts | |||
== Overlapper == | |||
* Use 80/90 Assembly1 overlap results | |||
* Rerun 10 overlap jobs | |||
* 96.64% of the quality-less reads have overlaps (vs 98.33% of the quality reads) | |||
reads 0ovl 1+ovl min max mean median n50 sum | |||
0qual(all) 624831 20941 603890 0 4350 96 19 740 60494730 # 96.64% | |||
0qual(unaligned) 52691 15384 37307 0 3229 50 5 349 2655545 # 70.80% | |||
== Unitigger == | |||
* More unitigs, more bases in unitigs | |||
* Few of the longest unitigs got broken: Example 138,294(UMD2.3) vs 159,434(UMD2.1) | |||
UNITIG OVERLAP GRAPH INFORMATION | |||
5333434 : Total number of unitigs | |||
2595174 : Total number of singleton, contained unitigs | |||
1865473 : Total number of singleton, non-contained unitigs | |||
183693 : Total number of non-singleton, spanned unitigs | |||
689094 : Total number of non-singleton, non-spanned unitigs | |||
35551316 : Total number of fragments | |||
35551316 : Total number of fragments in all unitigs | |||
21830994 : Total number of essential fragments in all unitigs | |||
13720322 : Total number of contained fragments in all unitigs | |||
0.0077856472 : Randomly sampled fragment arrival rate per bp | |||
2514833413 : The sum of overhangs in all the unitigs | |||
6483064813 : Total number of bases in all unitigs | |||
0 : Estimated number of base pairs in the genome. | |||
0 : Total number of contained fragments not connected | |||
by containment edges to essential fragments. | |||
Total rho = 2514833413 | |||
Total nfrags = 19579606 | |||
Estimated genome length = 0 | |||
Estimated global_fragment_arrival_rate=0.007786 | |||
Computed global_fragment_arrival_rate =0.007786 | |||
Total number of randomly sampled fragments in genome = 23870254 | |||
Computed genome length = 3065930496.000000 | |||
Used global_fragment_arrival_rate=0.007786 | |||
Used global_fragment_arrival_distance=128.441474 | |||
Histogram of the number of base pairs in a chunk | |||
100406 - 138294: 21 | |||
90330 - 99887: 23 | |||
80042 - 89675: 79 | |||
70014 - 79943: 169 | |||
60002 - 69792: 374 | |||
50000 - 59982: 1008 | |||
40002 - 49995: 2440 | |||
30001 - 39994: 6509 | |||
20000 - 29999: 18989 | |||
10000 - 19999: 57404 | |||
== Consensus after unitigger == | |||
Problems: | |||
* job 120 executed partially (see bt_120.cgi_tmp); Solution: split into 3 parts, run separately, merge results | |||
* failed on 19 unitigs (587..7447 bp) | |||
rm 5-consensus/*failed | |||
touch 5-consensus/consensus.success | |||
== Cgw == | |||
* Failure 1 : because job 120 was run partially => missing mates | |||
* Failure 2 : because of /5-consensus/FAILED/bt_???.cgi.failed => missing mates => delete 356 mates | |||
Error: | |||
ProcessFrags()-- WARNING! fragiid=35973388,index=33600942 mateiid=35973363,index=0 -- MATE DOESN'T EXIST! | |||
cgw: Input_CGW.c:117: ProcessFrags: Assertion `err == 0' failed. | |||
Fix: | |||
cat cgw.out | grep MATE | p '/mateiid=(\d+)/; print $1,"\n";' >! cgw.out.mateiid | |||
gatekeeper -dumpfragments -tabular -iid cgw.out.mateiid bt.gkpStore/ | cut -f1,3 | ~/bin/mate2lkg.pl -a D >! cgw.out.delete.LKG | |||
gatekeeper -a -o bt.gkpStore -T -F -L cgw.out.delete.LKG | |||
* Failure 3: because of cgwOutputIntermediate=1 | |||
Try to restart from ckp : die with assertion failure | |||
cgw -y -R 8 -N 12 -j 1 -k 5 -r 5 -s 2 -S 0 -z -m 100 -g ./bt.gkpStore -o ./7-0-CGW.8_12/bt ./5-consensus/bt_001.cgi | |||
cgw -y -R 8 -j 1 -k 5 -r 5 -s 2 -S 0 -z -m 100 -g ./bt.gkpStore -o ./7-0-CGW.8_12/bt ./5-consensus/bt_001.cgi | |||
Fix: Restart cgw from the beginning | |||
* cgw does update bt.SeqStore - OpenSequenceDB() | |||
== ECR (eventually skipped) == | |||
* Failed after running for 1 day | |||
/fs/szdevel/dpuiu/SourceForge/wgs-5.2/Linux-amd64/bin/extendClearRanges -g ./bt.gkpStore -n 15 -c bt -b 146216 -e 167100 -i 1 > 7-1-ECR/extendClearRanges-scaffold.146216.err | |||
sh: line 1: 17016 Aborted | |||
* Last ckp : bt.ckp.15 | |||
* Try to fix: | |||
touch 7-1-ECR/cgw.success | |||
runCA "doExtendClearRanges = 1" | |||
* Runs too slow !!! | |||
* Can specify a scaffold range to process: -b ? -e ? => ckp files; could we merge them? | |||
* Failed after running for 1 day | |||
== Consensus after CGW == | |||
* Failed on job 56 | |||
tail 8-consensus/bt.cns_contigs.56.err | |||
... | |||
Could (really) not find overlap between 153923 (U) and 2508303 (R) estimated ahang: 0 (ejecting frag 2508303 from contig) | |||
consensus: math_AS.h:51: ceil_log2: Assertion `x > 0' failed. | |||
cat 7-CGW/bt.cgw_contigs.56 | countMessages.pl | |||
ICM 440 | |||
IMP 281412 | |||
IUP 12715 | |||
cat 8-consensus/bt.cns_contigs.56_tmp | countMessages.pl | |||
ICM 115 | |||
IMP 103322 | |||
IMV 8122 | |||
IUP 4849 | |||
* Fix: split ICM messages 1..115,116,116+ and run consensus on each set | |||
== QC == | |||
elem min max mean med n50 sum | |||
scf 56891 407 33129045 50871 1378 4716077 2894145150 | |||
ctg 122851 64 651167 21957 3647 71561 2697514858 | |||
deg 268237 65 30246 1019 985 997 273575106 | |||
* Compared with UMD2.1 : better scaffols, worse contigs & unitigs | |||
== Analysis == | |||
Issues: | |||
* Identify bacterial & mito contigs: [http://www.bch.umontreal.ca/ogmp/projects/other/genbank/AY676873.txt mito seq] | |||
* Align ctg°en to UMD2 chromosomes | |||
** the chromosomes should have no 0cvg regions | |||
** possible inversions, translocations (UMD2 used markers) | |||
** if align breaks/indels, which assembly is correct? | |||
= Assembly UMD2.4 (2004_0217_CA; All reads) = | |||
<font color=red> | |||
* 35,973,728 reads : 35,348,776 quality & 624,952 quality-less | |||
* 16,896,244 mates | |||
* 344 libraries | |||
</font> | |||
== Fix quality-less read clrs (N's) (temporary solution) == | |||
* 10420 contain at least 1 "NN" in their clr (50.. min(len,600)) | |||
* 5973 contain at least 1 "NNN" in their clr (50.. min(len,600)) | |||
Fix: | |||
frg2seq.pl < bos_taurus.0qual.frg > bos_taurus.0qual.seq | |||
fasta2qual.pl bos_taurus.0qual.seq > ! bos_taurus.0qual.qual | |||
lucy \ | |||
-o bos_taurus.0qual.lucy.seq bos_taurus.0qual.lucy.qual \ | |||
-debug bos_taurus.0qual.lucy.info \ | |||
bos_taurus.0qual.seq bos_taurus.0qual.qual | |||
cat bos_taurus.0qual.lucy.info | cut -f1,3,4 -d ' ' | sort >! bos_taurus.0qual.lucy.clr | |||
* 624,952 quality-less reads | |||
* Quality-less read stats: 50..min(len,600) & lucy trimming | |||
elem 0 >0 min max mean med n50 sum | |||
5 624952 2857 622095 0 501 52 52 52 33012433 | |||
3 624952 2857 622095 0 600 579 600 600 361980208 | |||
53 624952 2857 622095 0 548 526 548 548 328967775 | |||
== Fix quality-less read clrs (low complexity) == | |||
* Run dust filter on seq (before qual & lucy) | |||
elem 0 >0 min max mean med n50 sum | |||
5 624952 3564 621388 0 501 75 52 52 47385578 | |||
3 624952 3564 621388 0 600 554 600 600 346470473 | |||
53 624952 3564 621388 0 548 478 548 548 299084895 | |||
* Merge dust.lucy clrs with the alignment clrs | |||
elem 0 >0 min max mean med n50 sum | |||
5 624952 4488 620464 0 599 93 52 126 58378496 | |||
3 624952 4488 620464 0 600 547 600 600 342258160 | |||
53 624952 4488 620464 0 548 454 512 548 283879664 | |||
* Test seq | |||
gatekeeper -dumpfastaseq -b 35348777 -e 35973728 bt.gkpStore | grep NNN | |||
gatekeeper -dumpfastaseq bt.gkpStore | perl -ane 'if(/^>(\d+)/) { $id=$1} elsif(/NNN/) { print $id,"\n";} ' | uniq -c | awk '{print $2,$1}' > bt.NNN.seqs # 2411 seqs (all have the N's "in the middle") | |||
gatekeeper -dumpfastaseq -uid bt.NNN.seqs bt.gkpStore > bt.NNN.cseqs | |||
== Consolidate libraries == | |||
<font color=red> Drop from 25,312 to 344 libs </font> | |||
=== BCM.SHOTGUN === | |||
UMD2.4 reestimated 10,117 out of 13,826 libs (have > 100mates) | |||
==== Base on initial estimates ==== | |||
* Reduce the total number from 13826 to 2 libs: 3000 & 6000 | |||
* UMD2.3 mean estimates (Initial vs Final): | |||
meanI #libs minF maxF meanF medF n50F sumF uid | |||
180000 436 1636 5199 2475 2410 2458 1079407 #3000 | |||
167000 86 1585 2948 2264 2258 2285 194775 #3000 | |||
6500 31 5212 6636 5837 5867 5924 180951 #6000 | |||
6000 11 4556 6272 5389 5421 5421 59286 #6000 | |||
3500 949 1670 4769 2668 2608 2645 2532027 #3000 | |||
3000 2511 1483 5250 2715 2662 2723 6818678 #3000 | |||
2000 6093 1157 6443 2526 2487 2554 15391160 #3000 | |||
==== Base on final estimates ==== | |||
* Reduce the total number from 13826 to 7 libs: 6500,5500,...1500, un-estimates (2501) | |||
meanF #libs min max mean med n50 sum uid(new) mean(new) std(new) | |||
6K<=mea<7K 15 6010 6636 6176 6159 6159 92650 6500 6500 | |||
5K<=mea<6K 29 5121 5985 5540 5536 5577 160673 5500 5500 | |||
4K<=mea<5K 67 4017 4939 4284 4266 4274 287072 4500 4500 | |||
3K<=mea<4K 1401 3000 3998 3276 3209 3226 4590323 3500 3500 | |||
2K<=mea<3K 7998 2000 2999 2502 2498 2532 20017767 2500 2500 | |||
1K<=mea<2K 607 1157 1999 1825 1882 1890 1107798 1500 1200 | |||
un-estimated 3709 2501 2501 | |||
=== BARC.CLONEEND === | |||
Collapse all 11150 into 1: | |||
uid:25456 | |||
mea:165000 | |||
std:43000 | |||
== Overlapper == | |||
* Quality-less reads overlaps: fewer than in the UMD2.3 assembly | |||
elem 0 >0 min max mean med n50 sum | |||
0qual(all) 624830 35692 589138 0 3237 60 14 439 37578899 # 94.39% | |||
== Unitigger == | |||
UNITIG OVERLAP GRAPH INFORMATION | |||
5356408 : Total number of unitigs | |||
2613795 : Total number of singleton, contained unitigs | |||
1870448 : Total number of singleton, non-contained unitigs | |||
182878 : Total number of non-singleton, spanned unitigs | |||
689287 : Total number of non-singleton, non-spanned unitigs | |||
35547861 : Total number of fragments | |||
35547861 : Total number of fragments in all unitigs | |||
21685943 : Total number of essential fragments in all unitigs | |||
13861918 : Total number of contained fragments in all unitigs | |||
0.0077797328 : Randomly sampled fragment arrival rate per bp | |||
2513424271 : The sum of overhangs in all the unitigs | |||
6468428782 : Total number of bases in all unitigs | |||
0 : Estimated number of base pairs in the genome. | |||
0 : Total number of contained fragments not connected | |||
by containment edges to essential fragments. | |||
Total rho = 2513424271 | |||
Total nfrags = 19553770 | |||
Estimated genome length = 0 | |||
Estimated global_fragment_arrival_rate=0.007780 | |||
Computed global_fragment_arrival_rate =0.007780 | |||
Total number of randomly sampled fragments in genome = 23868770 | |||
Computed genome length = 3068070656.000000 | |||
Used global_fragment_arrival_rate=0.007780 | |||
Used global_fragment_arrival_distance=128.539119 | |||
Histogram of the number of base pairs in a chunk | |||
100292 - 138301: 19 | |||
90052 - 99906: 23 | |||
80043 - 89676: 79 | |||
70013 - 79966: 164 | |||
60010 - 69988: 390 | |||
50008 - 59983: 949 | |||
40000 - 49998: 2433 | |||
30000 - 39997: 6437 | |||
20000 - 29999: 18808 | |||
10000 - 19999: 57634 | |||
== Bog == | |||
!!! Much bigger unitigs than default unitigger | |||
Global Arrival Rate: 0.013829 | |||
212260 - 224992: 4 | |||
100099 - 186873: 372 | |||
90015 - 99973: 353 | |||
80045 - 89988: 582 | |||
70011 - 79999: 1084 | |||
60000 - 69994: 1856 | |||
50001 - 59996: 3162 | |||
40002 - 49994: 5407 | |||
30000 - 39999: 9767 | |||
20000 - 29996: 18981 | |||
10000 - 19999: 39641 | |||
== Consensus after Unitigger == | |||
* Failed on jobs 120 & 121 ( _tmp file) | |||
cat 4-unitigger/*120* | countMessages.pl | |||
IMP 280264 | |||
IUM 124707 | |||
cat 4-unitigger/*121* | countMessages.pl | |||
IMP 282146 | |||
IUM 245650 | |||
cat 5-consensus/bt_120.cgi_tmp | countMessages.pl | |||
IMP 34348 | |||
IUM 19222 | |||
cat 5-consensus/bt_121.cgi_tmp | countMessages.pl | |||
IMP 51833 | |||
IUM 16805 | |||
* Fix 120: split IUM messages | |||
extractfromfrgMSG.pl -b 0 -e 19222 bt_120.cgb.orig IUM >! bt_120.cgb & | |||
extractfromfrgMSG.pl -b 19222 bt_120.cgb.orig IUM >! bt_120.cgb & | |||
* Fix 121: remove assertion in AS_CNS/MultiAlignment_CNS.c | |||
if(to <= from || to > ma_length-1){ | |||
fprintf(stderr, "AbacusRefine range (to) invalid"); | |||
//assert(0); | |||
} | |||
== CGW == | |||
* Failed after Ckp3(7-0-CGW/bt.ckp.3; MergeScaffoldsAggressive 2nd itteration) | |||
CI extends beyond end of scaffold! | |||
offsetAEnd = 254204 offsetBEnd = 252250 scaffoldLength = 253268 | |||
cgw: CIScaffoldT_Merge_CGW.c:307: InsertScaffoldContentsIntoScaffold: Assertion `0' failed. | |||
* Last cgw | |||
Scaffold lengths: | |||
cat 7-4-CGW/stat/final0.*Scaffolds.nodelength.cgm | grep -v ^Scaff | getSummary.pl -t scf | |||
cat 7-4-CGW/stat/final0.PlacedContig.n | grep -v ^Scaff | getSummary.pl -t scf | |||
elem min max mean med n50 sum | |||
scf 45826 385 34263871 59591 1349 7059820 2730853790 | |||
ctg 96562 65 738899 27789 3657 93988 2683452359 | |||
Library insert estimates: | |||
cat 7-4-CGW/stat/scaffold_final.distupdate.dst | grep ^# | awk '{print $3,int($8),int($10)}' > 7-4-CGW/bt.dst | |||
join2.pl bt.dst 7-4-CGW/bt.dst | p 'print join "\t",@F[0,1,2,5,6,3,4]; print "\n";' > bt.dst.combine | |||
CLONEEND inserts: | |||
UID MEANI STDI MEANF STDF COUNT LIB | |||
114892 150000 30000 175701 40732 31063 UIUC.CLONEEND | |||
19070 167000 25000 171349 18253 7103 BCM.CLONEEND | |||
118 167000 16700 167000 16700 21 WUGSC.CLONEEND | |||
25456 165000 43000 163044 25849 11150 BARC.CLONEEND | |||
115020 150000 30000 162719 25343 15256 UIUC.CLONEEND | |||
65177 2000 600 162540 34155 27067 TIGR.CLONEEND | |||
125606 150000 30000 162396 19319 59505 BCCAGSC.CLONEEND | |||
10738 2500 750 162386 27567 2040 TIGR_JCVIJTC.CLONEEND | |||
10691 2500 750 161540 28239 2763 TIGR_JCVIJTC.CLONEEND | |||
17249 202000 20200 157496 55375 6269 CENARGEN.CLONEEND | |||
54017 120000 12000 115671 27594 25889 GSC.CLONEEND | |||
total 188126 CLONEENDs | |||
== Consensus == | |||
* Failed on job 34 with segmentation fault | |||
* 9kbp contig, made out of 3007 reads (24 of which are quality-less) | |||
cat 7-CGW/bt.cgw_contigs.34.1 | grep "^{" | uniq -c | awk '{print $2,$1}' | |||
{ICM 1 | |||
{IMP 3007 | |||
{IUP 329 | |||
* Fix : edit AS_CNS/MultiAlignment_CNS.c; add | |||
if(!ungappedSequence->Elements) { ungappedSequence->numElements=0; } | |||
if(!ungappedQuality->Elements) { ungappedQuality->numElements=0; } | |||
== Analysis == | |||
=== Contigs Vs possible contaminants === | |||
* nucmer alignment parameters: -l 40 -c 100 -b 10 -g 5 -d 0.05 | |||
* have to redo alignments using -maxmatch !!! | |||
* file location: | |||
reference seqs: | |||
/nfshomes/dpuiu/db/Ecoli.365350-365744 # Ecoli K12 region with most alignments (BCM WGS splice site) | |||
/nfshomes/dpuiu/db/Ecoli # Ecoli K12 substrain MG1655 (NC_000913 ; 1st completed) | |||
/nfshomes/dpuiu/db/Ecoli.all # 22 Ecoli completed genomes ( + plasmids) | |||
/nfshomes/dpuiu/db/UniVec_Core # UniVec Core seqs | |||
/nfshomes/dpuiu/db/OtherVec # 100 other vector sequences identified by aligning UMD2.0 contaminants to GenBank; align also to 110 UniVec core using nucmer (params above) | |||
/nfshomes/dpuiu/db/bos_taurus.UMD2.contaminant.fasta # 4813 whole contigs and 30329 contig regions identified by NCBI as UMD2 contamination | |||
/nfshomes/dpuiu/db/bos_taurus.UMD2.contaminant.organism_count # organism counts: vector is the most abundant | |||
/nfshomes/dpuiu/db/bos_taurus.UMD2.contaminant.infoseq # grep -v 'coli|vector|7180003101029' => 905 other contamiants | |||
query seqs: | |||
/scratch1/bos_taurus/Assembly/2009_0217_CA/9-terminator/ctg.split100/*fasta # latest assembly contigs (no degenerates) | |||
delta files: | |||
/scratch1/bos_taurus/Assembly/2009_0217_CA/nucmer_ctg/no_maxmatch/*delta | |||
Ecoli K12 substrains: | |||
NC_010473.1 4686137 50.78 Escherichia coli str. K-12 substr. DH10B, complete genome | |||
NC_000913.2 4639675 50.79 Escherichia coli str. K-12 substr. MG1655, complete genome | |||
AC_000091.1 4646332 50.80 Escherichia coli str. K-12 substr. W3110, complete genome | |||
==== no maxmatch ==== | |||
* fewer alignments in UMD2.4 than in UMD2 | |||
UMD2 (all): just a few degens | |||
15102 Ecoli.365350-365744-ctg.qry_hits | |||
15943 Ecoli-ctg.qry_hits | |||
17308 Ecoli.all-ctg.qry_hits | |||
79065 UMD2.contaminant-ctg.qry_hits # 55877 new hits | |||
20105 UMD2.contaminant-ctg.CBE.qry_hits # CONTAIN|BEGIN|END|IDENTITY | |||
19839 UniVec_Core-ctg.qry_hits | |||
UMD2.4 | |||
559 Ecoli.365350-365744-ctg.qry_hits | |||
1215 Ecoli-ctg.qry_hits | |||
2767 Ecoli.all-ctg.qry_hits # most 2 frequenct starins are UMN026 & ATCC 8739; K12 DH10B is rank 5th; K12 MG1655 is ranked 19th (out 31 seqs) | |||
44112 UMD2.contaminant-ctg.qry_hits | |||
5286 UniVec_Core-ctg.qry_hits | |||
Length of the reference seqs used for screening: | |||
#seqs min max mean med n50 sum | |||
Ecoli.365350-365744 1 395 395 395 395 395 395 # Ecoli K12 regions with most alignments (BCM WGS splice site) | |||
Ecoli 1 4639675 4639675 4639675 4639675 4639675 4639675 # Ecoli K12 substrain MG1655 | |||
Ecoli.all 49 3306 5572075 2293320 130440 5065741 112372708 # 22 Ecoli's | |||
UniVec_Core 1348 12 48551 243 98 967 327641 | |||
OtherVec 100 1702 739874 15419 5027 166744 1541984 | |||
UMD2.contaminant 35142 48 16661 512 362 674 18022349 | |||
Length of UMD2.4 contigs that contain contaminant (0+ bp from end): | |||
#ctgs <2000bp >=2000bp min max mean med n50 sum | |||
Ecoli.365350-365744-ctg 559 534 25 1001 179527 2467 1341 1894 1379440 | |||
Ecoli-ctg 1215 1086 129 1001 360312 4326 1347 71372 5256540 | |||
Ecoli.all-ctg 2767 2455 312* 1001 453627* 8031 1366 134516 22224468 | |||
UniVec_Core-ctg 5286 4718 568* 882 651163* 9820 1337 136090 51909339 | |||
UMD2.contaminant-ctg.CBE 4976 4410 566* 738 651163* 8497 1339 122281 42281715 #annotated alignments: CONTAIN|BEGIN|END|IDENTITY | |||
UMD2.contaminant-ctg 44112 12813 31299 268 739442 50591 27461 111598 2231701788 | |||
Length of UMD2.4 contigs that contain contaminant in the middle (500+ bp from end): | |||
#ctgs <2000bp >=2000bp min max mean med n50 sum | |||
Ecoli.365350-365744-ctg 144 136 8 1286 2053 1779 1811 1814 256259 | |||
Ecoli-ctg 171 152 19 1286 4703 1835 1807 1821 313820 | |||
Ecoli.all-ctg 197 160 37*(81) 1228 351373* 6516 1815 125069 1283728 #81 2K+ ctgs using -maxmatch | |||
UniVec_Core-ctg 1278 1110 168*(276) 1085 651163* 12266 1496 160336 15676765 #276 2K+ ctgs using -maxmatch | |||
UMD2.contaminant-ctg.CBE 52 25 27* 1249 351373* 22195 2054 125069 1154142 #annotated alignments: CONTAIN|BEGIN|END|IDENTITY | |||
UMD2.contaminant-ctg 31019 1437 29582 1113 739442 70665 50798 113684 2191986214 | |||
Length of the UMD2.4 contaminant seqeunece (0+ bp from end): | |||
#align <200bp >=200bp min max mean med n50 sum | |||
Ecoli.365350-365744-ctg 1066 537 529 104 225 192 162 224 205379 | |||
Ecoli-ctg 1793 587 1206 50 4440 496 224 994 889798 | |||
Ecoli.all-ctg 4074 1132 2942* 40 17075* 380 254 441 1551783 | |||
UniVec_Core-ctg 14425 9819 4606* 40 1801* 236 162 325 3409187 | |||
UMD2.contaminant-ctg 144843 96008 48835 40 16661 199 169 209 28912002 | |||
Length of the UMD2.4 contaminant seqeunece (500+ bp from end) | |||
alignm <200bp >=200bp min max mean med n50 sum | |||
Ecoli.365350-365744-ctg 243 136 107 162 224 189 162 224 46000 | |||
Ecoli-ctg 273 149 124 106 1341 219 162 224 59923 | |||
Ecoli.all-ctg 294 153 141* 106 2150* 251 162 224 73992 | |||
UniVec_Core-ctg 2144 2035 109* 50 1340* 122 121 121 261821 | |||
UMD2.contaminant-ctg 121331 86985 34346 40 2738 171 162 184 20753580 | |||
* Problem: 8 long ctgs contain Ecoli in the middle (1000+ bp from end) | |||
show-coords Ecoli.all-ctg.filter-q.delta | ~/bin/filterQryCoords.pl -i 1000 | sort -nk13 -r | |||
[S1] [E1] | [S2] [E2] | [LEN 1] [LEN 2] | [% IDY] | [LEN R] [LEN Q] | [COV R] [COV Q] | [TAGS] | |||
=============================================================================================================================== | |||
4640589 4641890 | 161712 160411 | 1302 1302 | 99.46 | 4686137 351373 | 0.03 0.37 | gi|170079663|ref|NC_010473.1| ctg7180001872124 | |||
5068908 5069620 | 87386 86679 | 713 708 | 98.88 | 5209548 91972 | 0.01 0.77 | gi|218687878|ref|NC_011745.1| ctg7180002055226 | |||
3087480 3088683 | 50423 51620 | 1204 1198 | 99.00 | 5202090 88182 | 0.02 1.36 | gi|218703261|ref|NC_011751.1| ctg7180002054092 | |||
4640580 4641890 | 19953 18646 | 1311 1308 | 99.08 | 4686137 31157 | 0.03 4.20 | gi|170079663|ref|NC_010473.1| ctg7180001875158 | |||
131462 133564 | 1247 3349 | 2103 2103 | 98.19 | 241387 5751 | 0.87 36.57 | gi|157412014|ref|NC_009838.1| ctg7180002043242 | |||
82801 83166 | 2986 2621 | 366 366 | 98.09 | 241387 4709 | 0.15 7.77 | gi|157412014|ref|NC_009838.1| ctg7180001714551 | |||
82264 82793 | 3523 2994 | 530 530 | 98.49 | 241387 4709 | 0.22 11.26 | gi|157412014|ref|NC_009838.1| ctg7180001714551 | |||
1652253 1652545 | 1487 1195 | 293 293 | 98.63 | 4700560 2492 | 0.01 11.76 | gi|218552585|ref|NC_011741.1| ctg7180001754941 | |||
* Regions present in DH10B but not MG1655 | |||
delta2cvg -M 0 < DH10B-MG1655.delta | |||
gi|170079663|ref|NC_010473.1| 1349629 1378243 28614 0 | |||
gi|170079663|ref|NC_010473.1| 1391006 1396986 5980 0 | |||
gi|170079663|ref|NC_010473.1| 3199469 3200798 1329 0 | |||
gi|170079663|ref|NC_010473.1| 3211928 3213257 1329 0 | |||
gi|170079663|ref|NC_010473.1| 4640588 4641918 1330 0 !!! | |||
* Problem: 10 long ctgs contain Vector in the middle (1000+ bp from end) | |||
show-coords UniVec_Core-ctg.filter-q.delta | ~/bin/filterQryCoords.pl -i 1000 | sort -nk13 -r | |||
[S1] [E1] | [S2] [E2] | [LEN 1] [LEN 2] | [% IDY] | [LEN R] [LEN Q] | [COV R] [COV Q] | [TAGS] | |||
=============================================================================================================================== | |||
1 121 | 215495 215615 | 121 121 | 99.17 | 170 271477 | 71.18 0.04 | gnl|uv|U09128.1:15891-16011-49 ctg7180002047604 # pSacBII P1 cloning vector | |||
2252 2435 | 1334 1151 | 184 184 | 100.00 | 2485 160336 | 7.40 0.11 | gnl|uv|U75992.1:16925-19409 ctg7180001808271 | |||
180 312 | 1153 1020 | 133 134 | 99.25 | 312 160336 | 42.63 0.08 | gnl|uv|NGB00145.1:2378-2689 ctg7180001808271 | |||
1 121 | 1367 1487 | 121 121 | 100.00 | 170 160336 | 71.18 0.08 | gnl|uv|U09128.1:15891-16011-49 ctg7180001808271 | |||
1 103 | 1286 1388 | 103 103 | 100.00 | 103 160336 | 100.00 0.06 | gnl|uv|U80929.2:11415-11517 ctg7180001808271 [CONTAINED] | |||
4 121 | 68269 68386 | 118 118 | 100.00 | 170 111913 | 69.41 0.11 | gnl|uv|U09128.1:15891-16011-49 ctg7180002052060 | |||
40 152 | 30255 30142 | 113 114 | 99.12 | 1663 42854 | 6.79 0.27 | gnl|uv|U09128.1:1-1663 ctg7180002053344 | |||
1 121 | 34358 34238 | 121 121 | 100.00 | 170 35471 | 71.18 0.34 | gnl|uv|U09128.1:15891-16011-49 ctg7180002046164 | |||
1 103 | 34439 34337 | 103 103 | 100.00 | 103 35471 | 100.00 0.29 | gnl|uv|U80929.2:11415-11517 ctg7180002046164 [CONTAINED] | |||
46 1385 | 8928 10267 | 1340 1340 | 100.00 | 1413 17587 | 94.83 7.62 | gnl|uv|X65279.1:5941-7353 ctg7180002043597 [CONTAINED] | |||
* ctg7180001872124 : 351373 bp; region 160411..161712 contaminated by Ecoli | |||
cat 9-terminator/bt.posmap.utgctg | grep 7180001872124 | wc -l # 329 | |||
cat 9-terminator/bt.posmap.utgctg | grep 7180001872124 | perl -ane '@F[2,3]=@F[3,2] if($F[2]>$F[3]); print $_ if($F[2]<160411 and 160411<$F[3] or $F[2]<161712 and 161712<$F[3]);' | |||
7180000441625 7180001872124 159483 161201 r | |||
7180000441788 7180001872124 160330 161329 f #Ecoli | |||
7180000442730 7180001872124 160368 161010 r #Ecoli | |||
7180000441635 7180001872124 160740 162700 f #Ecoli | |||
cat 9-terminator/bt.utg.info | |||
utg7180000441625 length=1715 num_frags=12 Astat=7.00 | |||
utg7180000441788 length=999 num_frags=1 Astat=0.00 | |||
utg7180000442730 length=640 num_frags=1 Astat=0.00 | |||
utg7180000441635 length=1957 num_frags=9 Astat=7.00 | |||
cat 9-terminator/bt.posmap.frgctg | perl -ane '@F[2,3]=@F[3,2] if($F[2]>$F[3]); print $_ if($F[2]<160411 and 160411<$F[3] or $F[2]<161712 and 161712<$F[3]);' | |||
1237446426 7180001872124 160117 161201 f | |||
1238816835 7180001872124 160133 160993 f | |||
1238817728 7180001872124 160322 161123 r | |||
1244436200 7180001872124 159976 160984 f | |||
1238817676 7180001872124 160105 160890 r | |||
1237443253 7180001872124 160106 160900 f | |||
1237471027 7180001872124 159930 160928 f | |||
1238822613 7180001872124 159774 160782 f | |||
1238816875 7180001872124 159878 160728 f | |||
1244436248 7180001872124 159483 160553 f | |||
1238818306 7180001872124 159718 160489 f | |||
1238818332 7180001872124 159722 160483 f | |||
1237476824 7180001872124 160330 161329 f | |||
1238817689 7180001872124 160368 161010 r | |||
1237447135 7180001872124 160740 161768 r | |||
1237483546 7180001872124 160814 161790 r | |||
1237483530 7180001872124 160818 161856 r | |||
1237471108 7180001872124 161003 162009 f | |||
1238817744 7180001872124 161151 161978 f | |||
1237446441 7180001872124 161050 162107 f | |||
1244436201 7180001872124 161117 162164 f | |||
1237446407 7180001872124 161586 162699 r | |||
1237471055 7180001872124 161652 162700 r | |||
# 23 BCM SHOTGUN RP42 VVHNP reads (1369 read lib; 1341 of the reads in this ctg) | |||
* ctg7180002047604 : Vctor in the middle | |||
[S1] [E1] | [S2] [E2] | [LEN 1] [LEN 2] | [% IDY] | [LEN R] [LEN Q] | [COV R] [COV Q] | [TAGS] | |||
=============================================================================================================================== | |||
1 121 | 215495 215615 | 121 121 | 99.17 | 170 271477 | 71.18 0.04 | gnl|uv|U09128.1:15891-16011-49 ctg7180002047604 # pSacBII P1 cloning vector | |||
cat 9-terminator/bt.posmap.utgctg | grep 7180002047604 perl -ane '@F[2,3]=@F[3,2] if($F[2]>$F[3]); print $_ if($F[2]<215495 and 215495<$F[3] or $F[2]<215615 and 215615<$F[3]);' | |||
7180000441711 7180001872124 214458 219678 r | |||
cat 9-terminator/bt.posmap.frgctg | grep 7180002047604 | perl -ane '@F[2,3]=@F[3,2] if($F[2]>$F[3]); print $_ if($F[2]<215495 and 215495<$F[3] or $F[2]<215615 and 215615<$F[3]);' | |||
498776751 7180001872124 215425 216426 r | |||
1236502885 7180001872124 215514 216377 r | |||
379408823 7180001872124 215572 216340 f | |||
1244436224 7180001872124 215388 216405 f | |||
1237471071 7180001872124 215229 216234 r | |||
1233297450 7180001872124 215234 216046 f | |||
1233363357 7180001872124 215267 215687 f | |||
937200686 7180001872124 215300 216129 r | |||
937254901 7180001872124 215321 216160 f | |||
1233294025 7180001872124 215383 216204 r | |||
1237446444 7180001872124 215146 216187 f | |||
1232033776 7180001872124 215193 215996 r | |||
671976381 7180001872124 215035 216021 r | |||
514932286 7180001872124 215043 216008 f | |||
500723879 7180001872124 215043 215802 f | |||
671927656 7180001872124 215116 215733 r | |||
381173692 7180001872124 214947 215877 r | |||
1233303570 7180001872124 214963 215803 f | |||
1232037705 7180001872124 214990 215803 f | |||
490852264 7180001872124 214923 215843 f | |||
1237447184 7180001872124 214684 215646 f | |||
668822243 7180001872124 214586 215572 f | |||
#22 reads ; ~half come from BCM SHOTGUN RP42 VVFOP | |||
==== maxmatch ctg ==== | |||
Parameters: | |||
nucmer -maxmatch -l 40 -c 100 -b 10 -g 5 -d 0.05 ... | |||
AllVec: UniVec_Core + 100 more vector seqs | |||
Length of UMD2.4 contigs that contain contaminant (0+ bp from end): | |||
elem <2000 >2000 min max mean med n50 sum | |||
Ecoli.all 2951* 2602 349 1001 453627 8252 1367 132226 24352779 | |||
UniVec_Core 5387* 4802 585 882 651163 9979 1334 136556 53760575 | |||
OtherVec 5657 5062 595 882 651163 9726 1320 136556 55021803 | |||
UMD2.cont.other 3976 3430 546 804 651163 11217 1346 130385 44601117 #18 aligned to Acinetobacter; longest is 56467bp | |||
Length of UMD2.4 contigs that contain contaminant (500+ bp from end): | |||
elem <2000 >2000 min max mean med n50 sum | |||
Ecoli.all 182 156 26* 1286 351373 6525 1811 125069 1187706 # 7* are >5K; 321* come from multi-ctg scaffolds | |||
UniVec_Core 2532 2220 312* 1065 651163 10593 1481 128344 26821960 # 267* are >5K ; 655* come from multi-ctg scaffolds | |||
OtherVec 376 323 53 1184 361749 13278 1508 139997 4992774 | |||
UMD2.cont.other ... | |||
Length of UMD2.4 contigs that contain contaminant (1000+ bp from end): | |||
elem <2000 >2000 min max mean med n50 sum | |||
Ecoli.all 8 0 8* 4709 351373 73065 31157 351373 584520 | |||
UniVec_Core 11 0 11* 2600 334933 93674 37847 271477 1030414 | |||
OtherVec 5 0 5* 3717 271477 131604 111913 228060 658021 | |||
UMD2.cont.other 54 0 54* 2398 522682 110947 88182 189352 5991164 | |||
total 67* # 18 of them are CONTAINED by UMD2.0 chromosomes | |||
Length of the UMD2.4 contaminant sequence (0+ bp from end): | |||
elem <200 >200 min max mean med n50 sum | |||
Ecoli.all 4775 1610 3165 39 17072 381 236 502 1823278 | |||
UniVec_Core 16985 12380 4605 39 1800 207 132 300 3519080 | |||
OtherVec 7563 1372 6191 39 1800 509 548 643 3849567 | |||
UMD2.cont.other 6626 343 6283 39 8228 543 573 615 3602329 | |||
==== maxmatch deg ==== | |||
All degenerates aligned are <2K | |||
Length of UMD2.4 deg that contain contaminant (0+ bp from end): | |||
elem <2000 >2000 min max mean med n50 sum | |||
Ecoli.all 1266 1266 0 104 1611 783 833 869 991447 | |||
UniVec_Core 1908 1908 0 147 1510 872 896 910 1664746 | |||
OtherVec 1963 1963 0 147 1510 872 898 911 1712703 | |||
UMD2.cont.other 1609 1609 0 132 1611 852 892 914 1372106 | |||
==== maxmatch utg ==== | |||
Unitig stats: | |||
elem <2000 >2000 min max mean med n50 sum | |||
1707816 1434164 273652 21 138676 2228 937 8002 3805166508 | |||
Parameters: | |||
nucmer -maxmatch -l 40 -c 100 -b 10 -g 5 -d 0.05 ... | |||
Files: | |||
/scratch1/bos_taurus/Assembly/2009_0217_CA/nucmer_utg/ | |||
<font color=red> | |||
Length of UMD2.4 unitigs that align to contaminants | |||
elem <2000 >2000 min max mean med n50 sum | |||
Ecoli.all 4275 4110 165 104 71709 1442 1212 1398 6166566 | |||
UniVec_Core 7563 7409 154 139 71709 1397 1182 1331 10570512 | |||
OtherVec 8208 8054 154 139 71709 1370 1159 1308 11248775 | |||
UMD2.cont.other 6094 5849 245 132 53113 1546 1163 1401 9422951 #80 aligned to Acinetobacter; longest is 9114bp | |||
Contaminants(all above) 10264 9895 369 104 71709 1471 1148 1359 15107544 | |||
Acinetobacter 2306** 0 2306 154 71709 1451 1316 1412 3347230 #2182 already in the Cont set | |||
Length of UMD2.4 unitigs that have contaminants 500+bp from ends | |||
elem <2000 >2000 min max mean med n50 sum | |||
Ecoli.all 172 156 16 1286 4852 1820 1805 1815 313185 | |||
UniVec_Core 2491 2422 69 1065 71709 1722 1457 1523 4291584 | |||
OtherVec 364 358 6 1167 71709 1795 1478 1538 653595 | |||
UMD2.cont.other 156 108 48 1213 50248 5344 1838 17518 833673 | |||
Length of the UMD2.4 alignments of unitigs to contaminants (unique unitig regions) | |||
elem <200 >200 min max mean med n50 sum reads(all unitig reads for unitgs with alignments>1K) | |||
Ecoli.all 5975 1686 4289 40 8184 397 268 542 2374366 12112(12142) | |||
UniVec_Core 8754 1674 7080 40 1801 474 490 645 4153030 26590(26849) | |||
OtherVec 8919 1250 7669 40 1801 511 536 629 4562326 30268(30268) | |||
UMD2.cont.other 6752 896 5856 40 6012 529 555 651 3573528 25006(25328) | |||
Contaminants(all above) 10992 1396 9596 40 8184 571 573 684 6280759 40351(40699) | |||
Acinetobacter (8286)</font> | |||
40699 reads aligned back to contaminants: nucmer -maxmatch | |||
* 35919 align | |||
* 34400 align 100+bp | |||
* 27742 align 200+bp | |||
* 14211 align 500+bp | |||
==== utg 5'& 3' ==== | |||
Unitig stats: | |||
elem <200 >200 min max mean med n50 sum | |||
utg 1,707,816 81200 1626616 21 138676 2228 937 8002 3805166508 | |||
utg5'&3' 3,334,432 0 3334432 21 199 100 100 100 335263271 | |||
Align utg5'&3' to Ecoli.all using: | |||
* nucmer -l 40 -c 100 -b 10 -g 5 -d 0.05 : 4,275 hits | |||
* nucmer -l 20 -c 40 : 6,617 hits | |||
* nucmer -l 20 -c 20 : 23,350 | |||
* blastall : 2,895,506 out of 3,334,432 (86%) aligned | |||
==== Acinetobacter contamination ==== | |||
Database: | |||
~dpuiu/db/Acinetobacter.all : 7 complete genomes, 19 seqs | |||
Seq len summary: | |||
elem min max mean med n50 sum | |||
19 2726 4050513 1418094 28279 3904116 26943793 | |||
Align all unitigs to Acinetobacter.all; Longest alignments is 8517bp | |||
show-coords Acinetobacter.all-utg.filter-q.delta | sort -nk8 -r | head | |||
[S1] [E1] | [S2] [E2] | [LEN 1] [LEN 2] | [% IDY] | [LEN R] [LEN Q] | [COV R] [COV Q] | [GenBank] [UMD2.4 utg] | |||
=============================================================================================================================== | |||
20644 29164 | 62 8578 | 8521 8517 | 98.80 | 94413 8578 | 9.03 99.29 | gi|169786889|ref|NC_010404.1| utg7180000281954* [CONTAINS] | |||
3395586 3401299 | 5712 1 | 5714 5712 | 99.79 | 3976747 8015 | 0.14 71.27 | gi|126640115|ref|NC_009085.1| utg7180000212251 | |||
3400344 3404485 | 1 4142 | 4142 4142 | 99.66 | 3976747 9114 | 0.10 45.45 | gi|126640115|ref|NC_009085.1| utg7180000277331 | |||
... | |||
utg7180000281954* -> ctg7180002053982 (28140bp; 78 unitigs) | |||
grep 7180002053982 ../9-terminator/bt.posmap.utgctg | nl | |||
1 7180000185222 7180002053982 0 3019 f | |||
2 7180000314302 7180002053982 2151 5706 r | |||
3 7180001463328 7180002053982 2256 2869 f | |||
... | |||
75 7180000281954* 7180002053982 17862 26442 r | |||
76 7180001471348 7180002053982 17886 18723 r | |||
77 7180001468075 7180002053982 17919 18732 f | |||
78 7180000280508 7180002053982 25672 28140 r | |||
show-coords UMD2.contaminant.other-ctg.filter-q.delta | grep 7180002053982 | |||
[S1] [E1] | [S2] [E2] | [LEN 1] [LEN 2] | [% IDY] | [LEN R] [LEN Q] | [COV R] [COV Q] | [UMD2.0 contam] [UMD2.4 ctg] | |||
=============================================================================================================================== | |||
1394 2508 | 28140 27024 | 1115 1117 | 98.75 | 7098 28140 | 15.71 3.97 | 7180003313366 ctg7180002053982 | |||
2561 2871 | 26971 26661 | 311 311 | 97.11 | 7098 28140 | 4.38 1.11 | 7180003313366 ctg7180002053982 | |||
2934 5670 | 26599 23862 | 2737 2738 | 97.99 | 7098 28140 | 38.56 9.73 | 7180003313366 ctg7180002053982 | |||
...gap... | |||
5930 7098 | 17270 16101 | 1169 1170 | 98.46 | 7098 28140 | 16.47 4.16 | 7180003313366 ctg7180002053982 | |||
...gap... | |||
281 1981 | 10335 8635 | 1701 1701 | 99.41 | 13090 28140 | 12.99 6.04 | 7180003320028 ctg7180002053982 | |||
1992 2672 | 8635 7954 | 681 682 | 99.71 | 13090 28140 | 5.20 2.42 | 7180003320028 ctg7180002053982 | |||
3302 5376 | 6719 4643 | 2075 2077 | 99.33 | 13090 28140 | 15.85 7.38 | 7180003320028 ctg7180002053982 | |||
8469 9021 | 4642 4090 | 553 553 | 99.10 | 13090 28140 | 4.22 1.97 | 7180003320028 ctg7180002053982 | |||
9038 9313 | 4073 3798 | 276 276 | 98.55 | 13090 28140 | 2.11 0.98 | 7180003320028 ctg7180002053982 | |||
9780 13090 | 3331 19 | 3311 3313 | 99.19 | 13090 28140 | 25.29 11.77 | 7180003320028 ctg7180002053982 | |||
grep 7180002053982 ../9-terminator/bt.posmap.utgctg | awk '{print $1,$4-$3+1}' | sed 's/^/utg/' >! ctg7180002053982.utgs | |||
intersect.pl UMD2.contaminant.other-utg.qry_hits ctg7180002053982.utgs | wc -l | |||
37 # only 37 out of 78 unitigs were detected | |||
[http://blast.ncbi.nlm.nih.gov/Blast.cgi?CMD=Get&VIEW_RESULTS=FromRes&RID=VNC3Z5NU01R&UNIQ_OBJ_NAME=A_SearchResults_1LiCpb_3X0f_1Z58HQ4MV_23torn_22Zi4F&QUERY_INDEX=0 ctg7180002053982 is Acinetobacter] | |||
= UMD2.5 (2004_0312_CA; delete 40699 contam reads & 22607 mates ) = | |||
40699 reads: | |||
* 25803 mated + 14896 unmated | |||
* 6392 mated reads had the mate also contaminated | |||
Location: | |||
/scratch1/bos_taurus/Assembly/2009_0312_CA | |||
== UNITIGGER == | |||
UNITIG OVERLAP GRAPH INFORMATION | |||
5322910 : Total number of unitigs | |||
2595715 : Total number of singleton, contained unitigs | |||
1869655 : Total number of singleton, non-contained unitigs | |||
182193 : Total number of non-singleton, spanned unitigs | |||
675347 : Total number of non-singleton, non-spanned unitigs | |||
35507162 : Total number of fragments | |||
35507162 : Total number of fragments in all unitigs | |||
21641007 : Total number of essential fragments in all unitigs | |||
13866155 : Total number of contained fragments in all unitigs | |||
0.0077909501 : Randomly sampled fragment arrival rate per bp | |||
2511009753 : The sum of overhangs in all the unitigs | |||
6442095933 : Total number of bases in all unitigs | |||
0 : Estimated number of base pairs in the genome. | |||
0 : Total number of contained fragments not connected | |||
by containment edges to essential fragments. | |||
Total rho = 2511009753 | |||
Total nfrags = 19563152 | |||
Estimated genome length = 0 | |||
Estimated global_fragment_arrival_rate=0.007791 | |||
Computed global_fragment_arrival_rate =0.007791 | |||
Total number of randomly sampled fragments in genome = 23866135 | |||
Computed genome length = 3063315200.000000 | |||
Used global_fragment_arrival_rate=0.007791 | |||
Used global_fragment_arrival_distance=128.354050 | |||
Histogram of the number of base pairs in a chunk | |||
100292 - 138301: 22 # 19 in UMD2.4 | |||
90020 - 99906: 28 # 23 | |||
80043 - 89676: 90 # 79 | |||
70013 - 79966: 190 # 164 | |||
60010 - 69988: 423 | |||
50008 - 59983: 1016 | |||
40000 - 49998: 2558 | |||
30000 - 39997: 6660 | |||
20000 - 29999: 18927 | |||
10000 - 19999: 57057 | |||
== CONSENSUS after CGW == | |||
<font color=red> | |||
* failed on job 80 : ctg 5706539, len=180,024, 159 unitigs, 1,851 reads | |||
</font> | |||
head 80/bt.cns_contigs.80.failed | |||
{ICM | |||
acc:5706539 | |||
pla:P | |||
len:180024 | |||
cns: | |||
. | |||
qlt: | |||
. | |||
for:0 | |||
npc:1851 | |||
more ../9-terminator/bt.asm | |||
... | |||
{CCO | |||
acc:(7180002022380*,5706539) | |||
pla:P | |||
len:180024 | |||
cns: | |||
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN | |||
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN | |||
... | |||
cat 80/bt.cns_contigs.80.failed | countMessages.pl | |||
ICM 1 | |||
IUP 159 #unitigs | |||
IMP 1851 #reads : 1264 are BCM.WGS, 389 are BCM.SHOTGUN ... | |||
# 254 contig scaffold | |||
cat ../9-terminator/bt.posmap.ctgscf | grep 7180002041301 | nl | |||
1 7180002022042 7180002041301 0 14121 f | |||
... | |||
183 7180002022380* 7180002041301 9164814 9344838 f | |||
.. | |||
254 7180002022240 7180002041301 12874874 12893460 f | |||
# the 5 UMD2.4 contigs below have the same number of reads with the ones that matched => CONTAINED | |||
cat UMD2.4-7180002022380.posma p.frgctg | grep -v ^$ | awk '{print $2}' | uniq -c | |||
#reads ctgid | |||
6 7180001712307 | |||
209 7180002028662 | |||
6 7180002028663 | |||
1552 7180002028664 | |||
21 7180002032323 | |||
1794 total => 1851-1794=57 additional reads | |||
cat ../../2009_0217_CA/9-terminator/bt.posmap.ctgscf | nl | egrep '7180001712307|7180002028662|7180002028663|7180002028664|7180002032323' | |||
#nl ctgid scfid start end dir | |||
34791 7180002028662 7180002069912 1425623 1446022 f | |||
34793 7180002032323 7180002069912 1448133 1450475 f | |||
34794 7180002028663 7180002069912 1450495 1451918 f | |||
34795 7180002028664 7180002069912 1452234 1602973 f | |||
64110 7180001712307 7180002071598 0 1441 f | |||
* Solution 1: | |||
* consensus -Dforceunitigabut => new assembly, new UID's | |||
ctg7180002022636 179505 38.53 # => ctg7180002022380 | |||
scf7180002041557 12892941 40.58 # => scf7180002041301 | |||
* Solution 2: | |||
* Reassemble 1851 reads ; clr=ECR2; doOBT=no | |||
* Asm dir: | |||
/scratch1/bos_taurus/Assembly/2009_0312_CA/8-consensus/80.ECR2.asm | |||
* It contains one 179,530 bp scaffold that has two contigs. | |||
* One contig is 156,349 bp and the other one is 23,181 bp. | |||
* The estimated gap between them is 231 bp. | |||
show-coords ctg7180002022636-80.ECR2.filter-r.delta | |||
1 156326 | 1 156331 | 156326 156331 | 99.99 | 179505 156349 | 87.09 99.99 | ctg7180002022636 ctg7180000000103 [CONTAINS] | |||
156345 179505 | 21 23181 | 23161 23161 | 99.99 | 179505 23181 | 12.90 99.91 | ctg7180002022636 ctg7180000000104 [CONTAINS] | |||
>ctg7180002022636_156327_156344 | |||
TTGTAAAAACCATCCCCT | |||
# ~ 20 bp unaligned on ctg7180002022636 & Chr1 | |||
show-coords ctg7180002022636-Chr1.filter-r.delta | more | |||
... | |||
151839 156326 | 61124826 61120339* | 4488 4488 | 99.91 | 179505 157590899 | 2.50 0.00 | ctg7180002022636 Chr1 | |||
156347 157501 | 61111219* 61110064 | 1155 1156 | 99.91 | 179505 157590899 | 0.64 0.00 | ctg7180002022636 Chr1 | |||
... | |||
# 2 UMD2.0 ctg & 2 UMD2.0 deg in this region | |||
more Chr1.agp | |||
... | |||
Chr1 61110064 61111482 3579 W deg0003139347 1 1419 + | |||
Chr1 61111483 61114130 3580 N 2648 fragment yes | |||
Chr1 61114131 61115490 3581 W deg0002967451 1 1360 + | |||
Chr1 61115491 61118114 3582 N 2624 fragment yes | |||
Chr1 61118115 61120117 3583 W 7180002846553 1 2003 + | |||
Chr1 61120118 61120217 3584 U 100 fragment yes | |||
Chr1 61120218 61145567 3585 W 7180003318962 1 25350 + | |||
... | |||
== QC == | |||
<font color=red> | |||
Lengths: | |||
elem <2000 >=2000 min max mean med n50 sum | |||
scf 39978* 31311 8667 316 34167202 68129 1360 8217662 2723691675 | |||
ctg 90135* 36140 53995 65 1160130 29693 5124 95988 2676390147 | |||
deg 251413 249285 2128 65 39964 1003 984 994 252279234 | |||
utg 1689033 1419729 269304 21 138676 2242 936 8213 3788090224 | |||
elem <0 0 >0 min max mean med n50 sum | |||
gaps(ca2scf) 50157 10759 3296 36102 -20 177144 929 20 34357 46620040 | |||
gaps(posmap) 50157 0 0 50157 20 177144 943 20 34065 47301528</font> | |||
Fragment happiness: | |||
placed good 27263138 | |||
chaff bothChaff 2467462 | |||
placed notMated 2050084 | |||
placed oneChaff 732517 | |||
chaff oneChaff 732517 | |||
placed oneSurrogate 555510 | |||
placed bothDegen 465114 | |||
chaff notMated 434052 | |||
placed diffScaffold 369,294 * | |||
placed oneDegen 213,768 * | |||
placed badSame 96862 | |||
placed badLong 76640 | |||
placed badOuttie 41142 | |||
placed badShort 5784 | |||
placed bothSurrogate 3278 | |||
Mate happiness: | |||
good 13631569 | |||
bothChaff 1233731 | |||
oneChaff 732517 | |||
oneSurrogate 277755 | |||
bothDegen 232557 | |||
diffScaffold 184647 | |||
oneDegen 106884 | |||
badSame 48431 | |||
badLong 38320 | |||
badOuttie 20571 | |||
badShort 2892 | |||
bothSurrogate 1639 | |||
Scaffold zero read/mate cvg regions: | |||
elem <2000 >2000 min max mean med n50 sum | |||
read 57011 55048 1963 1 177144 913 57 29302 52084484 | |||
mate 10507 8945 1562 1 30014 996 493 2367 10466518 | |||
Scaffold 10K+ zero read/mate cvg regions (2K+ inside) (some might be a result of surrogates?): | |||
elem <2000 >2000 min max mean med n50 sum | |||
read 51747 49878 1869 1 177144 958 38 32625 49613432 | |||
mate 1290 201 1089 15 30014 3560 3047 4017 4593586 | |||
mate(ignore seq len) 2599 1051 1548 1 72706 3321 2541 5513 8633691 | |||
== Contaminant search == | |||
=== ctg === | |||
elem <0 >0 min max mean med n50 sum | |||
ctg 90135 0 90135 65 1160130 29693 5124 95988 2676390147 | |||
nucmer -maxmatch -l 40 -c 100 -b 10 -g 5 -d 0.05 ... | |||
#OLD table | |||
elem <2000 >2000 min max mean med n50 sum | |||
Ecoli.all 71 66 5 1006 129770 4180 1127 45234 296830 | |||
UniVec_Core 120 111 9 1000 426540 7366 1130 100944 884023 | |||
OtherVec 121 112 9 1000 426540 7314 1127 100944 885022 | |||
UMD2.cont.other 98(-3) 83 15(-3) 1000 426540 13523 1190 199700 1325332 # 3 are 1000bp+ from ctg ends; these are actually "fake" contaminants | |||
# 7 are Acinetobacter baumannii min=1059 , max=9765 | |||
total 152*(-3) 133 19*(-3) 1000 426540 10649 1150 199700 1618735 | |||
Acinetobacter 65 53 12 1013 44359 2586 1212 3847 168130 # 46 out of 65 are in the 152* set; 19 are new; 13 have lots of alignments to other contigs (probably fake contaminants) | |||
total(new) 171*(-3) 144 27*(-3) 1000 426540 10013 1189 129770 1712376 # 65 are Acinetobacter and should be removed | |||
cat UMD2.contaminant.other-ctg.filter-q.coords | grep Acinetobacter | |||
UMD2.0 UMD2.5 | |||
1 285 | 285 1 | 285 285 | 99.65 | 287 8096 | 99.30 3.52 | 7180003292866_1_288 ctg7180002015457 [CONTAINED] Acinetobacter baumannii | |||
1422 2500 | 1078 1 | 1079 1078 | 99.63 | 7098 1078 | 15.20 100.00 | 7180003313366 ctg7180001706852 [CONTAINS] Acinetobacter baumannii | |||
2934 3940 | 1008 1 | 1007 1008 | 98.61 | 7098 1059 | 14.19 95.18 | 7180003313366 ctg7180001709709 Acinetobacter baumannii | |||
6281 7098 | 1 818 | 818 818 | 99.76 | 7098 1553 | 11.52 52.67 | 7180003313366 ctg7180001716052 [END] Acinetobacter baumannii | |||
1 790 | 790 1 | 790 790 | 100.00 | 1822 9765 | 43.36 8.09 | 7180003319195_8956_10778 ctg7180002015485 [BEGIN] Acinetobacter calcoaceticus | |||
285 1981 | 1 1697 | 1697 1697 | 99.59 | 13090 1856 | 12.96 91.43 | 7180003320028 ctg7180001706656* Acinetobacter baumannii | |||
1992 2148 | 1697 1856 | 157 160 | 98.12 | 13090 1856 | 1.20 8.62 | 7180003320028 ctg7180001706656* Acinetobacter baumannii | |||
12210 13090 | 63 943 | 881 881 | 99.89 | 13090 2556 | 6.73 34.47 | 7180003320028 ctg7180002007423 Acinetobacter baumannii | |||
# 7 Acinetobacter baumannii ctgs | |||
# no Serratia "best hits" | |||
# 3 mitochondrion ctgs, all < 2Kbp | |||
<font color=red> | |||
Delete summary: 65 Acinetobacter ctgs + 91 contaminant ctgs <2000bp => 156 ctgs , 152 scf => 4105 reads | |||
ctgs <2000 >2000 min max mean med n50 sum reads | |||
156 144 12 1000 44359 1782 1150 1483 278009 4105 # all ctg | |||
Trim summary: 12 contigs >=2000bp & 44 reads that overlap at least 10bp | |||
ctgs <2000 >2000 min max mean med n50 sum reads | |||
12 0 12 7072 426540 78470 45234 129770 941646 ? # all ctg | |||
12 12 0 172 935 532 618 750 6393 44 # ctg regions | |||
Files | |||
/scratch1/bos_taurus/Assembly/2009_0312_CA/nucmer_ctg/TO_DELETE/ctg.delete.uid | |||
/scratch1/bos_taurus/Assembly/2009_0312_CA/nucmer_ctg/TO_DELETE/scf.delete.uid | |||
/scratch1/bos_taurus/Assembly/2009_0312_CA/nucmer_ctg/TO_TRIM/ctg.trim.uid | |||
</font> | |||
=== ctg 5'&3' === | |||
elem <0 >0 min max mean med n50 sum | |||
ctg53 180044 0 180044 65 598 300 300 300 54033229 | |||
nucmer -maxmatch -l 17 -c 35 ... | |||
#ctgEnds #ctgs min max mean med n50 sum | |||
Ecoli.all 180 149 300 300 300 300 300 54000 | |||
UniVec_Core 312 277 300 300 300 300 300 93600 | |||
OtherVec 1211 1167 300 553 300 300 300 363989 | |||
UMD2.cont.other 15689 14693 257 598 300 300 300 4712162 | |||
=== deg === | |||
nucmer -maxmatch -l 40 -c 100 -b 10 -g 5 -d 0.05 ... | |||
elem <2000 >2000 min max mean med n50 sum | |||
Ecoli.all 387 387 0 131 1099 756 806 835 292892 | |||
UniVec_Core 569 569 0 101 1115 763 822 843 434400 | |||
OtherVec 579 579 0 101 1115 752 819 840 435549 | |||
UMD2.cont.other 539 539 0 131 1483 792 838 873 427408 | |||
total 810* 810 0* 101 1483 784 838 869 63547 | |||
== Scaffolds vs UMD2.0 chromosome alignments == | |||
Directory: | |||
/scratch1/bos_taurus/Assembly/2009_0312_CA/nucmer_scf | |||
Depening on the ref/qry seq and nucmer parameters, the number of unaligned gaps in UMD2.0 can vary between: | |||
101M: REF=Chr, QRY=scf, nucmer -l 100 -c 500 | |||
6M: REF=ChrPlaced, QRY=scf-deg, nucmer -maxmatch -l 50 -c 250 | |||
=== nucmer -l 100 -c 500 === | |||
Chr-scf.summary | |||
elem <2000 >2000 min max mean med n50 sum | |||
Chr-scf.qry_hits 32901 24546 8355 723 34167202 82494 1405 8217662 2714164100 | |||
Chr-scf.qry_nohits 7077 6765 312 316 12006 1346 1239 1291 9527056 | |||
Chr-scf.10K.qry_hits2+ 574 0 574 10308 34167202 4006753 1887309 9586144 2299876795 | |||
Chr-scf.0cvg 144712 125933 18779 1 102265 900 178 2968 130248709 | |||
Chr-scf.0cvg.clean 148556 143283 5273 1 39625 683 280 1363 101526883(101M) | |||
Chr-scf-deg.summary | |||
elem <2000 >2000 min max mean med n50 sum | |||
Chr-scf-deg.qry_hits 210225 199781 10444 501 34167202 13785 1007 7328685 2898141592 | |||
Chr-scf-deg.qry_nohits 81166 80815 351 65 12006 958 972 989 77828798 | |||
Chr-scf-deg.10K.qry_hits2+ 574 0 574 10308 34167202 4006753 1887309 9586144 2299876795 | |||
Chr-scf-deg.0cvg 175952 168553 7399 1 22067 445 120 1329 78433265 | |||
Chr-scf-deg.0cvg.clean 133809 132381 1428 1 20512 371 124 1101 49711440(49M) | |||
ChrPlaced-scf.summary | |||
elem <2000 >2000 min max mean med n50 sum | |||
ChrPlaced-scf.qry_hits 19488 13112 6376 723 34167202 137773 1527 8428844 2684927057 | |||
ChrPlaced-scf.qry_nohits 20490 18199 2291 316 192648 1891 1276 1569 38764099 | |||
ChrPlaced-scf.10K.qry_hits2+ 139 0 139 10486 31959312 6786671 4979278 12956086 943347316 | |||
ChrPlaced-scf.0cvg 76271 71816 4455 1 102265 568 179 1710 43339413 | |||
ChrPlaced-scf.0cvg.clean 72865 70541 2324 1 39625 356 94 1413 25951987(25M) | |||
ChrPlaced-scf-deg.summary | |||
elem <2000 >2000 min max mean med n50 sum | |||
ChrPlaced-scf-deg.qry_hits 130125 122000 8125 501 34167202 21530 1009 7515049 2801670853 | |||
ChrPlaced-scf-deg.qry_nohits 161266 158596 2670 65 192648 1080 987 1012 174299537 | |||
ChrPlaced-scf-deg.10K.qry_hits2+ 139 0 139 10486 31959312 6786671 4979278 12956086 943347316 | |||
ChrPlaced-scf-deg.0cvg 79041 76374 2667 1 22067 395 157 948 31251753 | |||
ChrPlaced-scf-deg.0cvg.clean 69012 68271 741 1 20512 200 81 592 13864328(13M) | |||
=== nucmer -maxmatch -l 100 -c 500 === | |||
Dir: | |||
/scratch1/bos_taurus/Assembly/2009_0312_CA/nucmer_scf.2 | |||
ChrPlaced-scf-deg.summary | |||
elem <2000 >2000 min max mean med n50 sum | |||
ChrPlaced-scf-deg.qry_hits 130510 122377 8133 501 34167202 21470 1009 7515049 2802100771 | |||
ChrPlaced-scf-deg.qry_nohits 160881 158219 2662 65 192648 1080 986 1012 173869619 | |||
ChrPlaced-scf-deg.10K.qry_hits2+ 120 0 120 20022 31959312 7587296 5639522 13010806 910475551 | |||
ChrPlaced-scf-deg.0cvg 82159 80425 1734 1 7002 321 145 647 26444796 | |||
ChrPlaced-scf-deg.0cvg.clean 111645 111546 99 1 6248 81 13 272 9057424(9M) | |||
=== nucmer -maxmatch -l 50 -c 250 === | |||
Dir: | |||
/scratch1/bos_taurus/Assembly/2009_0312_CA/nucmer_scf.3 | |||
ChrPlaced-scf-deg.summary | |||
elem <2000 >2000 min max mean med n50 sum | |||
ChrPlaced-scf-deg.qry_hits 204673 195653 9020 251 34167202 14088 1005 7329288 2883625712 | |||
ChrPlaced-scf-deg.qry_nohits 86718 84943 1775 65 192648 1064 970 1007 92344678 | |||
ChrPlaced-scf-deg.10K.qry_hits2+ 148 0 148 10486 31959312 6814292 5135095 12792673 1008515300 | |||
ChrPlaced-scf-deg.0cvg 86085 84614 1471 1 4565 279 123 557 24101912 | |||
ChrPlaced-scf-deg.0cvg.clean 113796 113791 5 1 2822 59 7 176 6714556(6M) | |||
=== nucmer -maxmatch -l 50 -c 250 ; delta-fileter -q === | |||
Dir: | |||
/scratch1/bos_taurus/Assembly/2009_0312_CA/nucmer_scf.3 | |||
ChrPlaced-scf-deg.filter-q.summary | |||
elem <2000 >2000 min max mean med n50 sum | |||
ChrPlaced-scf-deg.qry_hits 204673 195653 9020 251 34167202 14088 1005 7329288 2883625712 | |||
ChrPlaced-scf-deg.qry_nohits 86718 84943 1775 65 192648 1064 970 1007 92344678 | |||
ChrPlaced-scf-deg.10K.qry_hits2+ 118 0 118 20022 31959312 7633834 5639522 13010806 900792422 | |||
ChrPlaced-scf-deg.0cvg 77864 73686 4178 1 28711 529 181 1523 41240419 | |||
ChrPlaced-scf-deg.0cvg.clean 74172 72150 2022 1 28331* 321 89 1415 23852952(23M) | |||
Max gap is 28331; Duplicate region in UMD2.0? | |||
ChrPlaced-scf-deg.coords | |||
70739616 70767946 | 2993389 2965048 | 28331 28342 | 99.17 | 85187327 19514159 | 0.03 0.15 | Chr15 scf7180002041107 | |||
70768054 70808299 | 3005305 2965048 | 40246 40258 | 99.56 | 85187327 19514159 | 0.05 0.21 | Chr15 scf7180002041107 | |||
=> | |||
ChrPlaced-scf-deg.filter-q.coords | |||
70768054 70808299 | 3005305 2965048 | 40246 40258 | 99.56 | 85187327 19514159 | 0.05 0.21 | Chr15 scf7180002041107 | |||
== Markers == | |||
ALL: | |||
head /fs/szasmg3/bos_taurus/UMD_Freeze2.5/markers/markers_contigs_Art.txt | |||
'''Marker Chr_BTA Pos(K) Pos_from Pos_to UMD_Ctg_Pos Match_Len %IDY %Match UMD_Ctg_name''' | |||
BZ945871 1 47501 1 95001 7622 515 100.00 99.61 ctg7180002007845 | |||
BZ953651 1 80001 47501 112501 10786 700 99.57 100.00 ctg7180002026484 | |||
CC504788 1 118751 80001 157501 54583 862 100.00 100.00 ctg7180002026483 | |||
CC484491 1 123751 90001 157501 50169 77 98.72 100.00 ctg7180002026482 | |||
CZ415082 1 125001 92501 157501 75850 507 99.21 99.80 ctg7180002026483 | |||
CC475154 1 130001 97501 162501 40013 666 99.25 100.00 ctg7180002026482 | |||
CC561114 1 182501 145001 220001 1130 709 99.02 100.00 ctg7180002026482 | |||
CC578374 1 190001 155001 225001 170145 647 100.00 100.00 ctg7180002026481 | |||
BZ911787 1 278751 232501 325001 na na na na na | |||
... | |||
* 126,014 markers & 90,135 ctgs total | |||
* 107,271 markers align to 31,407 ctg & 2640 scf: | |||
** 85% of the markers align to 85% of the ctg sequence | |||
** avg distance between markers is 25Kbp | |||
* 188 questionable ctgs & 219 questionable scf (2 out of 4 disagreeing markers) | |||
UNIQ: | |||
head /fs/szasmg3/bos_taurus/UMD_Freeze2.5/markers/markers_contigs_Art.unique_only.txt | p 'print " ",$_;' | |||
'''Marker Chr_BTA Pos(Kbp) CI_Pos_from CI_Pos_to UMD_Scaff_Pos Match_Len %IDY %Matched UMD_Scaff_name''' | |||
BZ945871 1 52251 1 104501 na na na na na | |||
BZ953651 1 88001 52251 123751 na na na na na | |||
CC504788 1 130626 88001 173251 54583 862 100.00 100.00 ctg7180002026483 | |||
CC484491 1 136126 99001 173251 50169 77 98.72 100.00 ctg7180002026482 | |||
CZ415082 1 137501 101751 173251 na na na na na | |||
CC475154 1 143001 107251 178751 40013 666 99.25 100.00 ctg7180002026482 | |||
CC561114 1 200751 159501 242001 1130 709 99.02 100.00 ctg7180002026482 | |||
CC578374 1 209001 170501 247501 170145 647 100.00 100.00 ctg7180002026481 | |||
BZ911787 1 306626 255751 357501 na na na na na | |||
... | |||
* 93,508 markers align to 28,752 ctgs & 1,476 scf: | |||
* 109 questionable ctgs & 153 questionable scf (2 out of 4 disagreeing markers) | |||
---- | |||
Scripts: | |||
~/bin/marker2pos.pl markers_contigs_Art.unique_only.txt | sed 's/ctg//' | sort -nk1 -nk2 > markers_ctg.pos | |||
~/bin/translatePosMap.pl markers_ctg.pos bt.posmap.ctgscf | ~/bin/tab2tab.pl > markers_scf.pos | |||
Ctg vs markers summary: | |||
#ctg <10000 >10000 min max mean med n50 sum file | |||
ctg (all) 90135* 51024 39111 65 1160130 29693 5124 95988 2676390147 | |||
no markers 58728 48324 10404 65 322949* 6573 1597 21989 386064754 | |||
markers from 1+ Chr 31407 2700 28707 442 1160130 72924 52693 111252 2290325393 markers_ctg.Chr.count | |||
markers from 2+ Chr 2987 25 2962 1002 1160130 132480 104807 179692 395718221 markers_ctg.Chr.count2+ | |||
2+ markers from 2+ Chr 26 0 26* 15228 604155 221354 192182 298848 5755227 markers_ctg.Chr.count2.2+ | |||
2+ adjacent markers from 2+ Chr 15 0 15** 15228 368879 202728 194749 294623 3040932 markers_ctg.Chr.count2+a | |||
Scf vs markers summary: | |||
#scf <10000 >10000 min max mean med n50 sum | |||
scf(all) 39978* 37135 2843 316 34167202 68129 1360 8217662 2723691675 | |||
no markers 37338 36038 1300 316 754615* 2601 1336 3957 97140879 | |||
markers from 1+ Chr 2640 1097 1543 1000 34167202 994905 16220 8661690 2626550796 markers_scf.Chr.count | |||
markers from 2+ Chr 552 10 542 1002 34167202 4526814 2714036 9167014 2498801557 markers_scf.Chr.count2+ | |||
2+ markers from 2+ Chr 212 0 212* 15228 34167202 8579232 7358307 10521496 1818797327 markers_scf.Chr.count2.2+ | |||
2+ adjacent markers from 2+ Chr 38 0 38** 15228 25078118 8419544 7176534 13458592 319942681 markers_scf.Chr.count2+a | |||
=== 212* scaffolds === | |||
scf_id scf_len #Chr/2+markers #/Chr/2+adjmarkers reads | |||
7180002041381 31959312 15 0 469503 | |||
7180002041358 25078118 13 2 291163 | |||
7180002041386 21280754 12 0 | |||
... | |||
==== scf7180002041381.1 ==== | |||
* no low cvg regions in the middle | |||
* 1281 markers: 1231 on Chr4, 12 on Ch11 | |||
#1- mate cvg regions: at the ends !!! | |||
#scfid begin end scf_len cvg_len cvg | |||
7180002041381 1 1173 31959312 1173 0 | |||
7180002041381 1174 1454 31959312 281 1 | |||
7180002041381 31959139 31959312 31959312 174 1 | |||
==== scf7180002041358.2 ==== | |||
* one low cvg region & real break | |||
* 1111 markers: 869 on Chr14, 193 on Chr26, ... | |||
#1- mate cvg regions: middle | |||
#scfid begin end scf_len cvg_len cvg | |||
7180002041358 20970531 20970827 25078118 297 1 | |||
7180002041358 20970828 20970949 25078118 122 0 | |||
7180002041358 20970950 20971112 25078118 163 1 | |||
# markers in the regions | |||
#scfid begin end makerid Chr | |||
7180002041358 20964100 20964829 BZ839784 14 | |||
7180002041358 21002368 21003219 CC527932 26 | |||
==== scf7180002041386.3 ==== | |||
* one low cvg region but no markers in that region | |||
* 939 markers: 902 on Chr24, 3 on Chr6 ... | |||
#1- mate cvg regions: at the ends | |||
#scfid begin end scf_len cvg_len cvg | |||
7180002041386 26382 27557 21280754 1176 1 | |||
7180002041386 27558 27607 21280754 50 0 | |||
7180002041386 27608 28031 21280754 424 1 | |||
... | |||
==== scf7180002041368.128 ==== | |||
* one low cvg region | |||
#scfid begin end scf_len cvg_len cvg | |||
7180002041368 1356953 1357221 1404851 269 1 | |||
* markers:Chr7 & Chr5 at 3' | |||
#scfid begin end markerid Chr | |||
7180002041368 10436 11033 BZ840669 7 | |||
... | |||
7180002041368 1278950 1279444 CZ404867 7 | |||
7180002041368 1319489 1320264 CC593534 7 | |||
7180002041368 1344001 1344347 BZ885865 7 * | |||
7180002041368 1371526 1372308 BZ867572 5 * | |||
7180002041368 1386933 1387750 CC534893 5 | |||
* [[Media:7180002041368.0cvg.png|7180002041368.0cvg.png]] | |||
* subassembly: extract all reads & extra mates(265) and reassemble | |||
** Msg Counts | |||
DST 40 | |||
FRG 13762 | |||
LKG 6482 | |||
** qc | |||
[Top5Scaffolds=contigs,size,span,avgContig,avgGap,EUID] | |||
0=37,1211999,1358964,32757,4082,7180000000571 | |||
1=2,53844,53824,26922,-20,7180000000566 | |||
2=4,50178,50238,12544,20,7180000000570 | |||
3=1,28553,28553,28553,0,7180000000568 | |||
4=1,17195,17195,17195,0,7180000000569 | |||
total=45,1361769,1508774,30262,3675 | |||
show-coords scf7180002041368-7180002041368.update.scf.filter-q.delta | |||
... | |||
1269662 1338725 | 69065 1 | 69064 69065 | 99.97 | 1404851 1359628 | 4.92 5.08 | scf7180002041368 scf7180000000571 | |||
1340619 1341912 | 1 1295 | 1294 1295 | 98.77 | 1404851 1295 | 0.09 100.00 | scf7180002041368 scf7180000000574 [CONTAINS] | |||
1345570 1347105 | 1537 1 | 1536 1537 | 98.18 | 1404851 1537 | 0.11 100.00 | scf7180002041368 scf7180000000572 [CONTAINS] | |||
1347683 1356952 | 12617 3352 | 9270 9266 | 99.85 | 1404851 12617 | 0.66 73.44 | scf7180002041368 scf7180000000567 | |||
1356901 1361861 | 1767 6728 | 4961 4962 | 99.96 | 1404851 50318 | 0.35 9.86 | scf7180002041368 scf7180000000570 | |||
... | |||
==== scf7180002041061.187 ==== | |||
* infoseq | |||
scf7180002041061 509987 39.73 | |||
* 22 markers | |||
#scfid begin end makerid Chr | |||
7180002041061 2061 2904 BZ847430 15 | |||
7180002041061 8979 9756 CC481592 15 | |||
7180002041061 10856 11485 BZ839377 15 | |||
7180002041061 47578 48186 BZ836581 15 | |||
7180002041061 80485 81237 CC553472 15 | |||
7180002041061 117811 118488 CC918533 15 | |||
7180002041061 151253 152009 CC477055 15 | |||
7180002041061 213959 214640 CC572066 15 * | |||
7180002041061 236304 236479 CC550436 29 * | |||
7180002041061 242590 243462 BZ848041 29 | |||
7180002041061 267880 268614 BZ877402 29 | |||
7180002041061 268891 269493 CC580499 29 | |||
7180002041061 282248 282861 BZ885584 29 | |||
7180002041061 295221 295987 CC572941 29 | |||
7180002041061 337785 338517 CC923898 29 | |||
7180002041061 338464 339092 BZ921430 29 | |||
7180002041061 341219 341971 CZ415932 29 | |||
7180002041061 382509 383281 CC581748 29 | |||
7180002041061 387548 388176 CC572064 29 | |||
7180002041061 415263 415788 BZ878185 29 | |||
7180002041061 470138 470773 CC558303 29 | |||
7180002041061 493006 493794 CC565740 29 | |||
* mate happiness: | |||
cat 7180002041061.posmap.mates | cut -f3 | count.pl | |||
good 2501 | |||
diffScaffold 86 | |||
oneSurrogate 69 | |||
oneChaff 40 | |||
oneDegen 11 | |||
badSame 9 | |||
badOuttie 4 | |||
badLong 4 | |||
* low cvg region: | |||
#scfid begin end scf_len cvg_len cvg | |||
7180002041061 232849 232871 509987 23 1 | |||
#it is in the middle of a unitig | |||
intersectPos.pl -i 1 7180002041061.posmap.utgscf 7180002041061.posmap.frgscf.10K.2K.mate_cvg.1- | |||
7180000849321 7180002041061 229782 238910 r | |||
[[Media:7180002041061.0cvg.png|7180002041061.0cvg.png]] | |||
* subassembly: extract all reads & mates(139) and reassemble | |||
** Msg Counts | |||
DST 38 | |||
FRG 5769 | |||
LKG 2724 | |||
Dirs: | |||
/scratch1/bos_taurus/Assembly/2009_0312_CA/markers/scf7180002041061.187.mates/asm.ECR2 | |||
/scratch1/bos_taurus/Assembly/2009_0312_CA/markers/scf7180002041061.187.mates/asm.ECR2.mates | |||
Qc stats: | |||
asm.ECR2 asm.ECR2.bog | |||
... | |||
[Top5Scaffolds=contigs,size,span,avgContig,avgGap,EUID] | |||
0 3,277300,277925,92433,312,7180000000278 5,279938,399837,55988,29975,7180000000155 | |||
1 6,234445,280546,39074,9220,7180000000277 6,234463,280485,39077,9204,7180000000156 | |||
2 1,1630,1630,1630,0,7180000000276 1,1799,1799,1799,0,7180000000154 | |||
3 NA 1,1674,1674,1674,0,7180000000153 | |||
4 NA 1,1183,1183,1183,0,7180000000158 | |||
total 10,513375,560101,51338,6675 14,519057,684978,37076,18436 | |||
* Some of the 139 mates assemble into scaffolds | |||
* There is slightly more sequence in the bog assembly | |||
* Mean/Max bog utg size is twice larger than default utg (scf, ctg sizes are about the same) | |||
* align asm.ECR2 scaffolds to scf7180002041061 | |||
** 2919 bp at the 3' end of the new scf7180000000277 don't align | |||
** 1990 bp at the 3' end of the new scf7180000000278 don't align | |||
** new scf7180000000277 & new scf7180000000278 align for ~ 1058bp | |||
** most of the mated read added assembled at new scf7180000000277 & new scf7180000000278 3' | |||
nucmer -l 100 -c 500 ../scf7180002041061.fasta 9-terminator/7180002041061.update.scf.fasta -p scf7180002041061-7180002041061.update.scf | |||
delta-filter -q scf7180002041061-7180002041061.update.scf.delta > scf7180002041061-7180002041061.update.scf.filter-q.delta | |||
show-coords scf7180002041061-7180002041061.update.scf.filter-q.delta | |||
1 46099 | 1 46099 | 46099 46099 | 99.99 | 509987 280626 | 9.04 16.43 | scf7180002041061 scf7180000000277 | |||
46273 48914 | 46120 48761 | 2642 2642 | 100.00 | 509987 280626 | 0.52 0.94 | scf7180002041061 scf7180000000277 | |||
50810 56904 | 95425 101521 | 6095 6097 | 99.43 | 509987 280626 | 1.20 2.17 | scf7180002041061 scf7180000000277 | |||
57301 233303 | 101699 277707 | 176003 176009 | 99.97 | 509987 280626 | 34.51 62.72 | scf7180002041061 scf7180000000277 * | |||
232245 444418 | 275975 63802 | 212174 212174 | 99.99 | 509987 277965 | 41.60 76.33 | scf7180002041061 scf7180000000278 * | |||
444964 469975 | 63156 38151 | 25012 25006 | 99.97 | 509987 277965 | 4.90 9.00 | scf7180002041061 scf7180000000278 | |||
449967 479221 | 58152 28900 | 29255 29253 | 99.96 | 509987 277965 | 5.74 10.52 | scf7180002041061 scf7180000000278 | |||
480298 509987 | 29688 1 | 29690 29688 | 99.99 | 509987 277965 | 5.82 10.68 | scf7180002041061 scf7180000000278 | |||
==== scf7180002041163.214 ==== | |||
* infoseq | |||
scf7180002041234 10336067 41.93 | |||
* Markers | |||
#scfid begin end makerid Chr | |||
... | |||
7180002041163 770783 771508 CC562874 30 * | |||
7180002041163 791050 791484 BZ887794 11 * | |||
... | |||
* [[Media:7180002041163.0cvg.png|7180002041163.0cvg.png]] | |||
* Markers on different contigs !!! | |||
cat 7180002041163.posmap.ctgscf | |||
.. | |||
7180001926720 7180002041163 750392 787668 f | |||
7180001926721 7180002041163 787911 815738 f | |||
... | |||
* subassembly: extract all reads & extra mates(293) and reassemble | |||
** Msg Counts | |||
DST 39 | |||
FRG 17287 | |||
LKG 8113 | |||
** qc | |||
[Top5Scaffolds=contigs,size,span,avgContig,avgGap,EUID] | |||
0=54,1742235,1760846,32264,351,7180000000507 | |||
1=1,6224,6224,6224,0,7180000000506 | |||
2=1,1472,1472,1472,0,7180000000508 | |||
total=56,1749931,1768542,31249,351 | |||
* alignments : | |||
show-coords scf7180002041163-7180002041163.update.scf.filter-q.delta | |||
1748 7972 | 6224 1 | 6225 6224 | 99.84 | 1757217 6224 | 0.35 100.00 | scf7180002041163 scf7180000000506 [CONTAINS] | |||
4012 5490 | 1 1470 | 1479 1470 | 98.85 | 1757217 1472 | 0.08 99.86 | scf7180002041163 scf7180000000508 [CONTAINS] | |||
4567 18273 | 1 13697 | 13707 13697 | 99.92 | 1757217 1761853 | 0.78 0.78 | scf7180002041163 scf7180000000507 | |||
... | |||
750393 787668 | 757092 794367 | 37276 37276 | 100.00 | 1757217 1761853 | 2.12 2.12 | scf7180002041163 scf7180000000507 | |||
787912 815738 | 794644 822455 | 27827 27812 | 99.81 | 1757217 1761853 | 1.58 1.58 | scf7180002041163 scf7180000000507 | |||
... | |||
1755928 1757217 | 1760564 1761853 | 1290 1290 | 100.00 | 1757217 1761853 | 0.07 0.07 | scf7180002041163 scf7180000000507 | |||
=== Manually curated === | |||
==== Markers within 50K of a low mate cvg region ==== | |||
* 13 scaffolds (22 before) | |||
* 14 breaks : 9 on the same contig , 2 on adjacent contigs , 3 on non adjacent contigs | |||
* File | |||
/scratch1/bos_taurus/Assembly/2009_0312_CA/markers/scfproblems.mates//scfproblems | |||
* scfproblems.markers.txt | |||
#scfid begin end markerid Chr(break) comment | |||
7180002040911 2210713 2211037 CZ405316 29 half Chr30, half Chr29 ; SAME ctg | |||
7180002041061 213959 214640 CC572066 15 half Chr29, half Chr15; SAME ctg | |||
7180002041103 5296272 5297045 CC587675 15 Chr18 & Chr15 at 3'; diff ctg (2 ctgs in between) | |||
7180002041107 7302203 7302874 CC906829 15 half Chr9, half Chr15; SAME ctg | |||
7180002041200 4920833 4921695 BZ877250 20 half Chr20, half Chr2; SAME ctg | |||
7180002041216 307121 307653 CL609365 5 Chr5 at 5' & Chr15; SAME ctg | |||
7180002041259 4315523 4316178 BZ922220 3 Chr5, Chr3, Chr24 !!! ; diff ctgs (4 ctgs in between) | |||
7180002041281 2884855 2885652 CC538348 19 half Chr19, half Chr28 ; SAME ctg | |||
7180002041315 2196754 2197298 BZ838379 1 Chr13 & Chr1 at 3'; diff ctg (2 ctgs in between) | |||
7180002041325 2597590 2598283 CC472212 10 half Chr10, half Chr5; SAME ctg | |||
7180002041348 3028112 3028921 CC531996 8 half Chr8, half Chr7; SAME ctg | |||
7180002041356 6828214 6828925 CC531427 16 half Chr16, half Chr25; SAME ctg | |||
7180002041358 21002368 21003219 CC527932 26 Chr14 & Chr26 at 3' (long chunck); diff ctg | |||
Details: | |||
#scfid begin end markerid Chr #ctgid begin end markerid Chr | |||
7180002040911 2189844 2190490 BZ923031 30 7180001730191 42851 43497 BZ923031 30 * | |||
7180002040911 2210713 2211037 CZ405316 29 7180001730191 63720 64044 CZ405316 29 * | |||
-- | |||
7180002041061 213959 214640 CC572066 15 7180001852904 29927 30608 CC572066 15 * | |||
7180002041061 236304 236479 CC550436 29 7180001852904 52272 52447 CC550436 29 * | |||
-- | |||
7180002041103 5220856 5221407 CG984741 18 7180001854649 21126 21677 CG984741 18 * | |||
7180002041103 5296272 5297045 CC587675 15 7180001854651 837 1610 CC587675 15 * | |||
-- | |||
7180002041107 7302203 7302874 CC906829 15 7180001855003 77842 78513 CC906829 15 ** | |||
7180002041107 7311399 7312254 CC479102 9 7180001855003 87038 87893 CC479102 9 ** | |||
-- | |||
7180002041200 4940131 4940949 CC500137 20 7180002002029 31935 32753 CC500137 20 ** | |||
7180002041200 4956412 4957105 CZ428497 2 7180002002029 48216 48909 CZ428497 2 ** | |||
-- | |||
7180002041216 307121 307653 CL609365 5 7180002003578 6871 7403 CL609365 5 * | |||
7180002041216 310832 311546 CL865591 15 7180002003578 10582 11296 CL865591 15 * | |||
-- | |||
7180002041259 4224828 4225527 CC920177 5 7180002012718 112862 113561 CC920177 5 * | |||
7180002041259 4315523 4316178 BZ922220 3 7180002012722 19376 20031 BZ922220 3 * | |||
-- | |||
7180002041259 6220638 6221268 BZ869532 3 7180002012752 199799 200429 BZ869532 3 * | |||
7180002041259 6239728 6240375 CZ413142 24 7180002012753 2600 3247 CZ413142 24 * | |||
-- | |||
7180002041281 2927406 2928016 CC573399 19 7180002018361 74539 75149 CC573399 19 ** | |||
7180002041281 2938896 2939696 CC513914 28 7180002018361 86029 86829 CC513914 28 ** | |||
-- | |||
7180002041315 2152291 2153097 BZ836343 13 7180001862977 3397 4203 BZ836343 13 * | |||
7180002041315 2196754 2197298 BZ838379 1 7180002024308 7255 7799 BZ838379 1 * | |||
-- | |||
7180002041325 2608389 2609213 CC506736 10 7180002025880 237956 238780 CC506736 10 ** | |||
7180002041325 2638591 2639207 CC770009 5 7180002025880 268158 268774 CC770009 5 ** | |||
-- | |||
7180002041348 3044263 3044902 BZ872906 8 7180002030033 152672 153311 BZ872906 8 ** | |||
7180002041348 3092547 3093121 BZ924509 7 7180002030033 200956 201530 BZ924509 7 ** | |||
-- | |||
7180002041356 6828214 6828925 CC531427 16 7180001722964 12571 13282 CC531427 16 * | |||
7180002041356 6832720 6833356 CC494876 25 7180001722964 17077 17713 CC494876 25 * | |||
-- | |||
7180002041358 20964100 20964829 BZ839784 14 7180001723456 140919 141648 BZ839784 14 * | |||
7180002041358 21002368 21003219 CC527932 26 7180001723457 15983 16834 CC527932 26 * | |||
* scfproblems.low_mate_cvg.txt | |||
#scfid begin end ctglen len mate_cvg | |||
7180002040911 2205494 2205582 3008363 89 1 | |||
7180002041061 232849 232871 509987 23 1 | |||
7180002041103 5285314 5285479 5341215 166 1 | |||
7180002041107 7306552 7306653 19514159 102 1 | |||
7180002041200 4954073 4954180 16995932 108 1 | |||
7180002041216 307614 307741 1444165 128 0 | |||
7180002041259 4313297 4313311 15600612 15 0 | |||
7180002041281 2933371 2933618 12907599 248 1 | |||
7180002041315 2193142 2193167 2232736 26 1 | |||
7180002041325 2623956 2624190 11153784 235 1 | |||
7180002041348 3064633 3064750 19180127 118 1 | |||
7180002041356 6831454 6831811 14697197 358 1 | |||
7180002041358 20970950 20971112 25078118 163 1 | |||
==== No low cvg regions ==== | |||
* 9 scaffolds | |||
* 14 breaks : 5 on the same contig , 6 on adjacent contigs , 3 on non adjacent contigs | |||
7180002040844 : 3 consecutive Chr30 markers (in the middle); mate_cvg > 18 ; markers CC576837,CC516738,CC543771; SAME ctg | |||
7180002041163 : 5' Chr30, 3' Chr11 ; mate_cvg~=10; marker BZ887794 ; diff ctg (1 ctg in between) | |||
7180002041234 : 5' Chr10, 3' Chr5; mate_cvg=10; marker BZ889975; scf=1757217bp; diff ctg | |||
7180002041235 : 2 consecutive Chr14 markers; mate_cvg=15..20; markers CC561837 & CC585677; NOT uniq; SAME ctg | |||
7180002041279 : 5 consecutive Chr23 markers; markers CC472696,CC522963; cvg=9..18; diff ctg | |||
7180002041306 : 5' Chr14, 3' Chr6; marker CC549871; cvg=2; diff ctg (22 ctgs in between) | |||
7180002041308 : 2 consecutive Chr20 markers; markers CC571631 & BZ832318; cvg=20; SAME/diff ctg | |||
7180002041321 : 5' Chr2, 3' Chr15 ; marker CC513377 cvg=10; diff ctg (1 ctgs in between) | |||
7180002041350 : 2 consecutive Chr3 markers ; markers BZ837387 & CC571149; cvg 17..26; NOT uniq; SAME/diff ctg | |||
Details: | |||
#scfid begin end markerid Chr #ctgid begin end markerid Chr | |||
7180002040844 8369147 8369989 CC521620 17 7180001727899 132573 133415 CC521620 17 ** | |||
7180002040844 8437707 8438324 CC576837 30 7180001727899 201133 201750 CC576837 30 ** | |||
... | |||
7180002040844 8467493 8468096 CC543771 30 7180001727899 230919 231522 CC543771 30 * | |||
7180002040844 8522732 8523598 CC513544 17 7180001727901 34970 35836 CC513544 17 * | |||
--- | |||
7180002041163 770783 771508 CC562874 30 7180001926720 20391 21116 CC562874 30 * | |||
7180002041163 791050 791484 BZ887794 11 7180001926721 3139 3573 BZ887794 11 * | |||
-- | |||
7180002041234 3856848 3857673 BZ889975 10 7180002010181 56608 57433 BZ889975 10 * | |||
7180002041234 3949893 3950652 CC509477 5 7180002010182 28610 29369 CC509477 5 * | |||
-- | |||
7180002041235 1117908 1118686 CC579933 21 7180002010285 4086 4864 CC579933 21 ** | |||
7180002041235 1118939 1119675 CC561837 14 7180002010285 5117 5853 CC561837 14 ** | |||
7180002041235 1158300 1159022 CC585677 14 7180002010285 44478 45200 CC585677 14 ** | |||
7180002041235 1164296 1164891 BZ924510 21 7180002010285 50474 51069 BZ924510 21 ** | |||
-- | |||
7180002041279 6755839 6756519 BZ849919 1 7180002018195 417990 418670 BZ849919 1 * | |||
7180002041279 6786086 6786957 CC472696 23 7180002018196 12703 13574 CC472696 23 * | |||
... | |||
7180002041279 6866378 6866890 CC522963 23 7180002018196 92995 93507 CC522963 23 * | |||
7180002041279 6911185 6911897 CC574255 1 7180002018197 15411 16123 CC574255 1 * | |||
-- | |||
7180002041306 248713 249457 CC503129 14 7180002022875 59534 60278 CC503129 14 * | |||
7180002041306 910997 911554 BZ839769 6 7180002022892 21939 22496 BZ839769 6 * | |||
-- | |||
7180002041308 1919399 1920196 BZ883381 18 7180002023013 57093 57890 BZ883381 18 * | |||
7180002041308 1958658 1959168 CC571631 20 7180002023014 29833 30343 CC571631 20 * | |||
7180002041308 1963333 1964083 BZ832318 20 7180002023014 34508 35258 BZ832318 20 ** | |||
7180002041308 2009574 2010367 CC499423 18 7180002023014 80749 81542 CC499423 30 ** | |||
-- | |||
7180002041321 4820048 4820542 BZ846646 2 7180002025240 77429 77923 BZ846646 2 * | |||
7180002041321 4908631 4909160 CC513377 15 7180002025242 13302 13831 CC513377 15 * | |||
-- | |||
7180002041350 2423855 2424205 CG983886 13 7180002030523 183644 183994 CG983886 13 * | |||
7180002041350 2446763 2447125 BZ837387 3 7180002030524 9191 9553 BZ837387 3 * | |||
7180002041350 2457417 2457704 CC571149 3 7180002030524 19845 20132 CC571149 3 ** | |||
7180002041350 2486474 2487050 CC490214 13 7180002030524 48902 49478 CC490214 13 ** | |||
==== Scaffold splitting ==== | |||
Before: | |||
* 22 scaffolds | |||
* 28 breaks : 14 on the same contig , 8 on adjacent contigs , 6 on non adjacent contigs | |||
Now: | |||
* 14 scaffolds | |||
* 15 breaks : 8 on the same contig , 3 on adjacent contigs , 4 on non adjacent contigs | |||
===== Scaffold to break ===== | |||
nl scfid breaks | |||
#1 7180002040844 2 | |||
2 7180002040911 1 | |||
3 7180002041061 1 | |||
#4 7180002041103 1 | |||
5 7180002041107 1 | |||
#6 7180002041163 1 | |||
7 7180002041200 1 | |||
#8 7180002041216 1 | |||
9 7180002041234 1 | |||
#10 7180002041235 2 | |||
11 7180002041259 2 | |||
#12 7180002041279 2 | |||
13 7180002041281 1 | |||
14 7180002041306 1 | |||
#15 7180002041308 2 | |||
16 7180002041315 1 | |||
17 7180002041321 1 | |||
18 7180002041325 1 | |||
19 7180002041348 1 | |||
#20 7180002041350 2 | |||
21 7180002041356 1 | |||
22 7180002041358 1 | |||
===== Contigs to break ===== | |||
nl ctgid | |||
1 7180001722964 | |||
2 7180001723456 | |||
3 7180001730191 | |||
4 7180001852904 | |||
5 7180001855003 | |||
6 7180002002029 | |||
7 7180002010182 | |||
8 7180002012722 | |||
9 7180002012752 | |||
10 7180002018361 | |||
11 7180002024308 | |||
12 7180002025240 | |||
13 7180002025880 | |||
14 7180002030033 | |||
===== Marker pairs ===== | |||
nl scfid begin1 end2 markerid1 Chr1 markerid2 Chr2 ctg1 ctg2 dist(end2-begin1) dist(ctg2-ctg1) | |||
#1 7180002040844 8369147 8438324 CC521620 17 CC576837 30 7180001727899 7180001727899 69177 0 | |||
#2 7180002040844 8467493 8523598 CC543771 30 CC513544 17 7180001727899 7180001727901 56105 2 | |||
3 7180002040911 2189844 2211037 BZ923031 30 CZ405316 29 7180001730191 7180001730191 21193 0 | |||
4 7180002041061 213959 236479 CC572066 15 CC550436 29 7180001852904 7180001852904 22520 0 | |||
#5 7180002041103 5220856 5297045 CG984741 18 CC587675 15 7180001854649 7180001854651 76189 2 | |||
6 7180002041107 7302203 7312254 CC906829 15 CC479102 9 7180001855003 7180001855003 10051 0 | |||
#7 7180002041163 770783 791484 CC562874 30 BZ887794 11 7180001926720 7180001926721 20701 1 | |||
8 7180002041200 4940131 4957105 CC500137 20 CZ428497 2 7180002002029 7180002002029 16974 0 | |||
#9 7180002041216 307121 311546 CL609365 5 CL865591 15 7180002003578 7180002003578 4425 0 | |||
10 7180002041234 3856848 3950652 BZ889975 10 CC509477 5 7180002010181 7180002010182 93804 1 | |||
#11 7180002041235 1117908 1119675 CC579933 21 CC561837 14 7180002010285 7180002010285 1767 0 | |||
#12 7180002041235 1158300 1164891 CC585677 14 BZ924510 21 7180002010285 7180002010285 6591 0 | |||
13 7180002041259 4224828 4316178 CC920177 5 BZ922220 3 7180002012718 7180002012722 91350 4 | |||
14 7180002041259 6220638 6240375 BZ869532 3 CZ413142 24 7180002012752 7180002012753 19737 1 | |||
#15 7180002041279 6755839 6786957 BZ849919 1 CC472696 23 7180002018195 7180002018196 31118 1 | |||
#16 7180002041279 6866378 6911897 CC522963 23 CC574255 1 7180002018196 7180002018197 45519 1 | |||
17 7180002041281 2927406 2939696 CC573399 19 CC513914 28 7180002018361 7180002018361 12290 0 | |||
18 7180002041306 248713 911554 CC503129 14 BZ839769 6 7180002022875 7180002022892 662841 22 | |||
#19 7180002041308 1919399 1959168 BZ883381 18 CC571631 20 7180002023013 7180002023014 39769 1 | |||
#20 7180002041308 1963333 2010367 BZ832318 20 CC499423 18 7180002023014 7180002023014 47034 0 | |||
21 7180002041315 2152291 2197298 BZ836343 13 BZ838379 1 7180001862977 7180002024308 45007 2* | |||
22 7180002041321 4820048 4909160 BZ846646 2 CC513377 15 7180002025240 7180002025242 89112 2 | |||
23 7180002041325 2608389 2639207 CC506736 10 CC770009 5 7180002025880 7180002025880 30818 0 | |||
24 7180002041348 3044263 3093121 BZ872906 8 BZ924509 7 7180002030033 7180002030033 48858 0 | |||
#25 7180002041350 2423855 2447125 CG983886 13 BZ837387 3 7180002030523 7180002030524 23270 1 | |||
#26 7180002041350 2457417 2487050 CC571149 3 CC490214 13 7180002030524 7180002030524 29633 0 | |||
27 7180002041356 6828214 6833356 CC531427 16 CC494876 25 7180001722964 7180001722964 5142 0 | |||
28 7180002041358 20964100 21003219 BZ839784 14 CC527932 26 7180001723456 7180001723457 39119 1 | |||
===== Break intervals ===== | |||
#scfid begin end scflen len allread goodmate badmate Chr1 Chr2 mark1 mark2 ctgid | |||
#1 7180002040844 8412735 8412739 9565303 5 1 12 15 17 30 357 3 | |||
#2 7180002040844 8485384 8485869 9565303 486 0 12 4 30 17 3 53 | |||
3 7180002040911 2205044 2205247 3009964 204 3 0 8 30 29 44 34 7180001730191 | |||
4 7180002041061 232819 232843 510915 25 5 0 23 15 29 8 14 7180001852904 | |||
#5 7180002041103 5295117 5295435 5342873 319 0 11 2 18 15 221 4 | |||
6 7180002041107 7306552 7306620 19515586 69 2 0 11 15 9 320 549 7180001855003 | |||
#7 7180002041163 787669 787911 1752998 243 0 10 0 30 11 16 8 | |||
8 7180002041200 4954103 4954180 16991478 78 1 0 27 20 2 219 532 7180002002029 | |||
#9 7180002041216 309331 309337 1435859 7 1 4 3 5 15 7 50 | |||
10 7180002041234 3939493 3939548 10292558 56 1 0 20 10 5 160 227 7180002010182 | |||
#11 7180002041235 1117752 1117987 1742396 236 5 20 2 21 14 43 2 | |||
#12 7180002041235 1161294 1161363 1742396 70 3 16 0 14 21 2 32 | |||
13 7180002041259 4313381 4313547 15594966 167 1 0 12 5 3 193 75 7180002012722 | |||
14 7180002041259 6231717 6231762 15594966 46 8 4 18 3 24 75 406 7180002012752 | |||
#15 7180002041279 6773364 6773383 9784709 20 0 4 14 1 23 292 5 | |||
#16 7180002041279 6894603 6894618 9784709 16 0 4 16 23 1 5 141 | |||
17 7180002041281 2931877 2931885 12908817 9 3 5 10 19 28 83 441 7180002018361 | |||
18 7180002041306 267445 267446 4484402 2 0 2 14 14 6 49 141 7180002022875,7180002005935 | |||
#19 7180002041308 1928806 1928825 23070077 20 0 10 0 18 20 63 2 | |||
#20 7180002041308 1977759 1977828 23070077 70 1 21 1 20 18 2 836 | |||
21 7180002041315 2192754 2192893 2233575 140 7 0 22 13 1 59 2 7180002024308 | |||
22 7180002041321 4840934 4841070 6492983 137 1 0 16 2 15 227 72 7180002025240 | |||
23 7180002041325 2624153 2624190 11155368 38 3 0 22 10 5 86 357 7180002025880 | |||
24 7180002041348 3064691 3064730 19179560 40 1 0 16 8 7 116 634 7180002030033 | |||
#25 7180002041350 2437553 2437572 23353087 20 0 19 1 13 3 107 2 | |||
#26 7180002041350 2479000 2479029 23353087 30 2 23 1 3 13 2 880 | |||
27 7180002041356 6831454 6831783 14699461 330 1 0 9 16 25 299 319 7180001722964 | |||
28 7180002041358 20970057 20970139 25079321 83 7 0 25 14 26 868 189 7180001723456 | |||
Where: | |||
Chr1: The most frequent chromosome markers with alignment at coordinates <= begin | |||
Chr2: The most frequent chromosome markers with alignment at coordinates >= end | |||
mark1: Number of Chr1 markers with alignment at coordinates <= begin | |||
mark2: Number of Chr2 markers with alignment at coordinates >=end | |||
Lines starting with # should be ignored. | |||
ctgid: ctg to break | |||
===== New scaffolds ===== | |||
#scfid(new) scfid begin end scflen(new) | |||
===== Files ===== | |||
# ctg:scf new name mapping | |||
/scratch1/bos_taurus/Assembly/2009_0312_CA/markers/scfproblems.both.filtered/scfproblems.posmap.ctbscb | |||
# scf:scf new:original name mapping | |||
/scratch1/bos_taurus/Assembly/2009_0312_CA/markers/scfproblems.both.filtered/scfproblems.posmap.scbscf | |||
# ctg:ctg new:original name mapping | |||
/scratch1/bos_taurus/Assembly/2009_0312_CA/markers/scfproblems.both.filtered/ctgproblems.posmap.ctbctg | |||
# ctg FASTA sequences | |||
/scratch1/bos_taurus/Assembly/2009_0312_CA/markers/scfproblems.both.filtered/ctbproblems.fasta | |||
# scf FASTA sequences | |||
/scratch1/bos_taurus/Assembly/2009_0312_CA/markers/scfproblems.both.filtered/scbproblems.fasta | |||
= UMD2.6 (UMD2.5 without contam ctg/scaff; split ctg/scaff) = | |||
== Contaminants & MarkerBreaks == | |||
Delete summary: | |||
ctg scf scf->ctg | |||
contaminants(delete) 156 152 666 | |||
contaminants(trim) 12 12 1328 | |||
markerBreaks 14+1 14+1 2875+1 # 1 more break in UMD2.6.1 | |||
total 182 178 4869 | |||
Add summary: | |||
ctg scf | |||
contaminants(delete) 0 2 | |||
contaminants(trim) 12 12 | |||
markerBreaks 28+2 29+2 # 1 more break in UMD2.6.1 | |||
total 40 43 | |||
Summary: | |||
ctg scf markers | |||
original 90135 39978 | |||
del/add -142 -135 | |||
final 89993 39843 -17 | |||
Files: | |||
/scratch1/bos_taurus/Assembly/2009_0312_CA/new/bt.ctg.fasta : contig FASTA sequence | |||
/scratch1/bos_taurus/Assembly/2009_0312_CA/new/bt.scf.fasta : scaffold FASTA sequence | |||
/scratch1/bos_taurus/Assembly/2009_0312_CA/new/bt.posmap.ctglen : contig lengths | |||
/scratch1/bos_taurus/Assembly/2009_0312_CA/new/bt.posmap.scflen : scaffold lengths | |||
/scratch1/bos_taurus/Assembly/2009_0312_CA/new/bt.posmap.ctgscf : mapping of contigs to scaffolds (posmap format) | |||
/scratch1/bos_taurus/Assembly/2009_0312_CA/new/bt.posmap.scaff : mapping of contigs to scaffolds (scaff format) | |||
/scratch1/bos_taurus/Assembly/2009_0312_CA/new/ctg.delete.uid : UID of the contigs which were deleted | |||
/scratch1/bos_taurus/Assembly/2009_0312_CA/new/scf.delete.uid : UID of the scaffolds which were deleted | |||
/scratch1/bos_taurus/Assembly/2009_0312_CA/new/ctg.add.uid : UID of the contigs which were added : UID =~/brk\d+[abc]/ OR UID =~/cnt\d+/ | |||
/scratch1/bos_taurus/Assembly/2009_0312_CA/new/scf.add.uid : UID of the contigs which were added : UID =~/brk\d+[abc]/ OR UID =~/cnt\d+/ | |||
/scratch1/bos_taurus/Assembly/2009_0312_CA/new/markers.delete.uid : markers which got deleted | |||
/scratch1/bos_taurus/Assembly/2009_0312_CA/new/bt.ctg.break : 15 contig break regions | |||
/scratch1/bos_taurus/Assembly/2009_0312_CA/new/bt.scf.break : 16 scaffold break regions | |||
Scripts: | |||
~/bin/breakPosmapKeep.amos : pipeline for breaking scf/ctg | |||
* Final: | |||
elem min max mean med n50 sum | |||
ctg 89993 65 1160130(1.1M) 29736 5180 95952 2676109306 | |||
scf 39843 316 34167202(34M) 68353 1361 7451988 2723419943 | |||
scf<50K 38915 316 49898 2765 1349 5107 107632139 | |||
scf<5K 35229 316 4999 1518 1306 1435 53495879 | |||
== Ctg Markers == | |||
Filtered: | |||
%IDY>90 | |||
%Matched>85 | |||
* ~30% of alignments agree to this condition | |||
Markers: | |||
total: 126111 # from 31372 ctgs | |||
Contigs | |||
elem min max mean med n50 sum | |||
placed 31372* 442 1160130 72908 52734 111147 2287291732* | |||
unplaced 58621 65 425922 6632 1600 22204 388817574 | |||
total 89993 65 1160130 29736 5180 95952 2676109306 | |||
== Scf Markers == | |||
Markers: | |||
total: 126111 # from 2641 scaffolds; 1595 scaffolds have 2+ markers | |||
na: 18744 # not placed | |||
del: 17 # the scaffolds they belong to were deleted | |||
other: 3722 # not on the "main" chromosome; "main" chromosome determined by a majority rule; if it's a tie check markers for uniqueness | |||
outliers: 411 # interquartile range method (IQR) q1-1.5*(q3-q1) .. q1+1.5*(q3-q1) | |||
filtered: 103217 # "non conflicting" markers | |||
Scaffolds: | |||
elem min max mean med n50 sum #ctgs | |||
placed 2641 1000 34167202 994523 16673 8170786 2626536153* 50528 | |||
unplaced 37202 316 754615 2604 1337 3964 96883790 39465 | |||
total 39843 316 34167202 68353 1361 7451988 2723419943 89993 | |||
scf <2 >=2 >=10 min max mean med n50 sum | |||
markers/scf 2641 1595 1046 562 1 1418 39 1 354 103479 | |||
ctg/scf 2641 1546 1095 559 1 545 19 1 130 50528 | |||
cat markers_scf.mainChr.*summary | count.pl -c 7 | sort -n | |||
1 1564 # 1 marker/scf | |||
2 276 # 2 markers/scf | |||
3 116 # ... | |||
4 52 | |||
... | |||
1470 1 # scaffold 7180002041371 has 1470 markers | |||
* Scaffold position: | |||
filter outliers (interquartile range method); use median value | |||
problem: only 2 markers far apart: choose randomly or check for uniqueness | |||
Summary (approximate) | |||
Chr #Ctg #Scaff ScaffSpanSum MaxMarkerPos | |||
1 2989 112 157088082 167097751 | |||
2 2488 73 138112445 141135901 | |||
3 2257 109 120984003 128677351 | |||
4 2079 111 124695956 123662451 | |||
5 2271 98 120470218 130242001 | |||
6 2244 127 117350442 127208151 | |||
7 2169 93 109780318 114917551 | |||
8 2069 95 111646872 114607251 | |||
9 1937 72 103790361 106365151 | |||
10 1829 96 102878815 108508301 | |||
11 2030 62 106593132 107458151 | |||
12 1789 114 89109155 97406401 | |||
13 1498 73 83821399 88539451 | |||
14 1482 142 84084175 89211101 | |||
15 1734 105 84680500 91332551 | |||
16 1710 111 80727432 86838601 | |||
17 1384 51 72913556 78195801 | |||
18 1446 86 65689468 70299751 | |||
19 1338 56 63372609 69847351 | |||
20 1454 54 71941707 75982901 | |||
21 1405 63 70035525 72193201 | |||
22 1077 36 60892135 > 60178851 | |||
23 1021 44 51791473 54886001 | |||
24 1059 26 61662407 > 61466101 | |||
25 783 41 42670836 > 45254751 | |||
26 991 40 50640267 52316851 | |||
27 920 70 45768018 48911451 | |||
28 810 52 45884054 50753001 | |||
29 1143 89 51657687 55219751 | |||
30 3122 340 135803106 152429101 | |||
U 39465 37202 94983049 | |||
* Scaffold orientation | |||
filter outliers (interquartile range method) | |||
use LeastSequareFit method to estimate the orientation : if slope is positive => forward; if slope is negative => reverse; | |||
problem: slope ~=0 => which direction ? | |||
cat Chr.summary | getSummary.pl -i 5 | |||
cat Chr.agp | grep W | awk '{print $9}' | count.pl | |||
elem <0 0 >0 min max mean med | |||
scf 2641 516 1610 515 -31 61 0 0 | |||
ctg 50528 24885 2236 23407 | |||
Use slope thold to determine direction? | |||
cat markers_scf.mainChr.noOutliers.summary | p 'print $_ if(abs($F[5])>0.5);' | wc -l # 634 | |||
Ambiguity examples: | |||
BZ908653 6 114061501 114016501 114106501 172149 597 98.66 100.00 7180002040834 | |||
BZ891600 6 114085251 114051501 114119001 4834 504 99.80 99.02 7180002040834 | |||
CZ411135 6 114094001 114059001 114129001 242710 609 98.20 100.00 7180002040834 | |||
BZ854276 6 114132751 114101501 114164001 100980 580 99.31 99.83 7180002040834 | |||
CC524983 30 115669901 115622401 115717401 96634 715 98.74 100.00 7180002041003 | |||
BZ869249 30 115791151 115717401 115864901 86931 448 99.78 100.00 7180002041003 | |||
BZ867530 30 115798651 115732401 115864901 69671 572 99.13 100.00 7180002041003 | |||
CC585731 30 115808651 115752401 115864901 54950 737 99.86 100.00 7180002041003 | |||
CC469285 30 115818651 115772401 115864901 125555 550 94.57 100.00 7180002041003 | |||
* Scaffold overlaps: some small scaffolds might be contained by bigger ones | |||
cat markers_scf.mainChr.noOutliers.posmap.scfchrabs | ~/bin/posmap2ovl.pl | sort -nk6 -r | ~/bin/tab2tab.pl -f -15 | head | |||
Chr ref qry begin end end-begin | |||
30 7180002041328 7180002041078 72536051 76993951 4457900 | |||
4 7180002041381 7180002041269 41976451 46175201 4198750 | |||
30 7180002040852 7180002041077 142001501 145251501 3250000 | |||
30 7180002038569 7180002041121 33012001 35518351 2506350 | |||
30 7180002040971 7180002034501 59411001 61836101 2425100 | |||
... | |||
== Marker Issues == | |||
=== Placement === | |||
* 37202 scaffolds are unplaced (3.5% of total scaffold span); max=0.7M | |||
* 87 unplaced scaffolds (1.5Mbp total) could be placed using SLK messages | |||
perl ~/bin/difference21.pl bt.slk markers_scf.scflen | head | |||
7180002030632 7180002040171 I -147744.734 21214.779 2 UP | |||
7180002030792 7180002031221 I -15042.301 558.223 2 UP | |||
7180002030849 7180002041244 N -1609089.875 18145.568 4 UP | |||
... | |||
elem min max mean med n50 sum | |||
Unplaced 87 740 102909 17259 8544 34550 1501601 | |||
Placed 78 16177 27139572 5757234 4037663 10989230 449064304 | |||
* ambiguous assignment to chromosmes: | |||
cat markers_scf.count | p 'print $_ if($F[2]*2==$F[3]);' | nl | |||
#k1 k2 count12 count1 | |||
1 7180002030741 15 2 4 | |||
2 7180002031341 11 1 2 | |||
... | |||
36 7180002068140 25 1 2 | |||
# scaffold assigned differently | |||
~/bin/difference12.pl markers.all/markers_scf.mainChr.count markers.all_plus_uniq/markers_scf.mainChr.count | nl | |||
1 7180002032536 19 1 2 23480 | |||
2 7180002037223 8 1 2 31516 | |||
3 7180002040013 25 2 4 15228 | |||
4 7180002040262 30 1 2 3087 | |||
5 7180002040378 10 1 2 50907 | |||
6 7180002040523 29 1 2 9105 | |||
7 7180002040769 4 1 3 827526 | |||
8 7180002041203 14 2 4 1141560 | |||
9 7180002044555 18 1 2 1566 | |||
* Chr30 has many scaff aligned to it | |||
* how reliable are the markers: 1151 out of 2641 placed scaffolds have no unique markers ? | |||
* what measure is best for placing the scaffolds? | |||
* can some scaffolds go in the gaps ? | |||
* AGP: | |||
** gaps<=0 or 20 set to 100 | |||
** unoriented ctgs set to + | |||
* Marker positions are not uniformly distributed; they tend to "custer" | |||
elem 0 >0 min max mean med n50 sum | |||
markers_chr 107337 8405 98932 0 446250 25800 8750 75000 2769379200 | |||
markers_scf 94171 123 94048 0 2565850 24143 15856 42236 2273610521 | |||
markers_chr.mainChr.noOutliers 101084 7609 93475 0 1253750 27383 10000 78750 2768040450 # filtered IRQx,xy method | |||
markers_scf.mainChr.noOutliers 88450 52 88398 0 2669361 25327 16587 44374 2240202221 | |||
=== Orientation === | |||
* at least 1564 out of 2641 placed scaffolds cannot be oriented; max=1.39M | |||
=== Possible misassemblies === | |||
nl #scfid Ch medianPos rangePos scfLen slope #ChMark #Mark #ctg | |||
1 7180002041225 1 120473301 9866350 9279975(9M) 0.0103 381 390 184 : break interval: 1674263..1674319 (1X frg_cvg region) | |||
2 7180002041078 30 77506551 6310600 3094999(3M) 0.0148 65 69 46 : no clear position | |||
==== scf7180002041225 : 9.27 Mbp on Chr1 ==== | |||
scflen=9279975 | |||
[[Media:Markers_7180002041225.txt|7180002041225.markers]] | |||
[[Media:7180002041225.png|7180002041225.markers]] | |||
[[Media:7180002041225.cvg.png|7180002041225.cvg]] | |||
ChrPos ScfPos | |||
115272051 56300 | |||
... | |||
116848301 1618430 | |||
116953301 9190560 | |||
... | |||
125138401 1691145 | |||
break interval: 1674263..1674319 = 57bp frg_cvg=1 mate_cvg=0 bad_mate_cvg(nearby)=7 | |||
==== scf7180002041078 : 3.09 Mbp (not broken) ==== | |||
[[Media:Markers_7180002041078.txt|7180002041078.markers]] | |||
[[Media:7180002041078.png|7180002041078.markers]] | |||
[[Media:7180002041078.1.cvg.png|7180002041078.1.cvg]] xrange [315268:466827] | |||
[[Media:7180002041078.2.cvg.png|7180002041078.2.cvg]] xrange [1191000:1229713] | |||
[[Media:7180002041078.3.cvg.png|7180002041078.3.cvg]] xrange [2733146:2750017] | |||
ChrPos ScfPos | |||
72536051 2893791 | |||
... | |||
72682301 2750017 | |||
72728651 1191000 | |||
... | |||
73273651 466827 | |||
77112801 2733146 | |||
... | |||
78464051 1229713 | |||
.. | |||
78595401 315268 | |||
==== scf7180002040971 : 1.93 Mbp (not broken) ==== | |||
==== Files ==== | |||
/scratch1/bos_taurus/Assembly/2009_0312_CA/new/markers.all.redo/ | |||
* Nucmer alignments to UMD2.0 chromosomes: all seem to agree pretty well | |||
/scratch1/bos_taurus/Assembly/2009_0312_CA/new/markers.all.redo2/nucmer_UMD2.0/UMD2.0.Chr26-Chr26.png | |||
/scratch1/bos_taurus/Assembly/2009_0312_CA/new/markers.all.redo2/nucmer_UMD2.0/UMD2.0.Chr27-Chr27.png | |||
/scratch1/bos_taurus/Assembly/2009_0312_CA/new/markers.all.redo2/nucmer_UMD2.0/UMD2.0.Chr29-Chr29.png | |||
/scratch1/bos_taurus/Assembly/2009_0312_CA/new/markers.all.redo2/nucmer_UMD2.0/UMD2.0.ChrX-Chr30.png | |||
== Homo sapiens alignments == | |||
Citations: | |||
* mice closer to human than cow | |||
* human and cow have approximately 201 homologous blocks of DNA | |||
* Independently generated mapping data provide another measure of the quality of the assembly. Snelling et al. [4] created a B. taurus map from three radiation hybrid panels, two genetic maps, and bacterial artificial chromosome (BAC) end sequences. We aligned all of the 17,254 markers (of which 17,193 are unique) in their composite map (Cmap) to both assemblies. A marker was considered as matching a chromosome if 90% of the marker sequence aligned with at least 95% identity. Of the Cmap markers, 14,620 align to the UMD2 assembly's chromosomes, versus 13,699 markers (6.3% fewer) for the BCM4 assembly. A small number of Cmap markers (119 and 82 for UMD2 and BCM4, respectively) mapped to a different chromosome from the one indicated in the Cmap data. | |||
* homologous synteny block (HSB) : | |||
** human-cow alignment extended for at least 250 Kbp | |||
** it was not interrupted by an inversion or by an HSB on another chromosome. | |||
** If two HSBs were interrupted by a gap of <3 Mbp and nothing else fell in that gap, the two blocks were merged. (Note that if a large region of synteny is interrupted by a distinct HSB, the interruption creates three HSBs.) | |||
** [http://genomebiology.com/2009/10/4/R42/table/T3 Number of homologous synteny blocks on each chromosome of the cow and human genomes] 201 -> 268 blocks | |||
* Orienting contigs using cow-human alignments: | |||
** Scaffolds (sets of linked contigs) that were mapped onto chromosomes using only a single marker could not be oriented from the marker information alone. We oriented many of these scaffolds by taking advantage of the overall conserved synteny between cow and human. First, all cow scaffolds were aligned to the human genome using nucmer [14] with its maximal unique match (mum) option in order to avoid alignments of repetitive sequence. For each alignment of a previously unoriented scaffold to human, all alignments within 100 Kbp on each side were pulled out for analysis. A score S was computed for each unoriented scaffold, taking into account whether the scaffolds surrounding S on both sides (in cow) were mapped to a consistent set of locations in human. If the scaffolds surrounding S were oriented, and if a large majority of these scaffolds on both the left and right agreed on the orientation, then S was assigned that orientation. Using this procedure, 1,840 scaffolds containing 4,011 contigs were oriented. | |||
** We developed a similar procedure to assign unplaced contigs to chromosomes, again relying on conserved synteny between cow and human. First, all unplaced contigs were aligned as above. Mummer's 'delta-filter' program was then used to compute a one-to-one mapping of the unplaced contigs to human so that only the best aligning contig was considered at each region in human. For each unplaced contig's best alignment to human, the matching region in cow was identified via our human-cow syntenic map, and all contigs from this region were extracted for examination. We only considered placing a contig on a B. taurus chromosome if the order and direction of the surrounding contigs in cow matched the corresponding region in human. As above, we examined the alignments of nearby cow contigs that aligned within 100 kb of the unplaced contig's alignment in human. If the region of cow-human synteny contained no rearrangements, then the unplaced contig was placed at the location indicated by these alignments. Using this procedure, 1,046 contigs were placed on chromosomes. One consequence of this procedure was that a number of incompletely mapped genes (based on mRNA alignments) were completed. | |||
Issues: | |||
* which alignment program to use? | |||
** nucmer | |||
** blastz: difficult to parse | |||
** blat | |||
* nucmer: what parameters? | |||
** default | |||
** loose : -mum -l 12 -c 30 -g 1000 | |||
** ref: 24 homo sapiens chromosomes files | |||
** query: 26 bos taurus scaffold files | |||
Total scaffolds: | |||
elem min max mean med n50 sum | |||
'''scf(len) 39844 316 34167202 68352 1361 7451988 2723419938''' | |||
Aligned to HS: | |||
elem min max mean med n50 sum | |||
'''scf(len) 8860 385 34167202 301202 2257 7740810 2668658140''' # each scf aligns in avg to 4 Chr | |||
scf(aligLen) 789272 250 19097 514 399 550 405946972 | |||
scf(alig%) 789272 58.04 100.00 79 78.54 78.72 . | |||
scf(len,maxX) 727 1002 7069350 194658 6530 1895564 141516604 # 220 in common with the 339 ones that have mark | |||
scf(len,noMark) 7025 385 7605708 7783 1654 27630 54680173 | |||
'''scf(len,mark) 1835 1001 34167202 1424511 40503 7970944 2613977967''' | |||
Not aligned to HS: | |||
elem min max mean med n50 sum | |||
scf(len,notAligned) 30984 316 163953 1767 1309 1555 54762139 | |||
Aligned to markers: | |||
elem min max mean med n50 sum | |||
'''scf(len) 2641 1000 34167202 994523 16673 8170786 2626536153''' | |||
1 marker 1595 1000 1396951 17482 4595 48276 27884078 | |||
2+ markers 1046 1055 34167202 2484371 596240 8217662 2598652075 | |||
abs(slope)<0.25 1946 1000 3094999 29136 7585 114274 56700529 | |||
abs(slope)>=0.25 696 4674 34167202 3692292 1845828 8401441 2569835619 | |||
Scf summary: | |||
elem min max mean med n50 sum | |||
all 39844 316 34167202 68352 1361 7451988 2723419938 | |||
1+mark 2641 1000 34167202 994523 16673 8170786 2626536153 # best marker alignment | |||
1+align 8860 385 34167202 301202 2257 7740810 2668658140 # alignments > 250bp | |||
1+align(new) 9880 385 34167202 270458 1996 7740810 2672129298 # alignments > 200bp (3.74M more than 250bp align) | |||
1+mark or 1+align 9669 385 34167202 276443 2051 7740810 2672936792 | |||
0 mark and 1+align 7027 385 754,615 6603 1654 20219 46,400,303 !!! | |||
1+mark and 1+align 1791 1001 34167202 1305861 37914 7473583 2338798064 | |||
0 mark and 0 align 30179 316 88,326 1705 1305 1517 51,483,587 !!! | |||
1+mark or 1+align 9669 385 34167202 276443 2051 7740810 2,672,936,792 | |||
Degenerate: | |||
elem min max mean med n50 sum | |||
all 251413 65 39964 1003 984 994 252,279,234 | |||
all(2000+bp) 2128 2000 39964 3753 2827 3946 7,986,872 | |||
1+mark 562 200 30168 2731 1274 5029 1,535,011 | |||
1+mark(2000+bp) 180 2007 30168 6079 4404 7510 1,094,252 | |||
1+align 6429 251 39964 1487 1013 1287 9,566,273 | |||
1+align(2000+bp) 756 2004 39964 4820 3624 5646 3,644,556 | |||
Issues: | |||
* 24 scaffolds that have 200+ alignments to at least 2 HS chromosomes | |||
* 37 scaffolds that have 100+ alignments to at least 2 HS chromosomes | |||
* 83 scaffolds that have 50+ alignments to at least 2 HS chromosomes | |||
File location: | |||
/scratch1/bos_taurus/Assembly/2009_0312_CA/nucmer_human | |||
=== Synteny method === | |||
File location: | |||
/scratch1/bos_taurus/Assembly/2009_0312_CA/synteny | |||
* Position & orient scaffolds that aligned to HS | |||
* find placed neighboring that don't "disagree" | |||
elem min max mean med n50 sum | |||
total 7027 385 754615 6603 1654 20219 46,400,303 !!! | |||
* Scaffolds possibly assigned to the wrong chromosome | |||
cat summary.nucmer-markers.txt | perl ~/bin/synteny/getSynteny2.nucmer-marker.pl -minMarkers 2 | grep update:chr | getSummary.pl -i 6 -t "<2 markers" | |||
cat summary.nucmer-markers.txt | perl ~/bin/synteny/getSynteny2.nucmer-marker.pl -minMarkers 4 | grep update:chr | getSummary.pl -i 6 -t "<4 markers" | |||
elem min max mean med n50 sum | |||
<2 markers 307 1001 937279 27898 9212 98674 8,564,850 | |||
<4 markers 383 1001 1141560 33822 13426 160445 12,953,875 | |||
==== UMD2.0 ==== | |||
elem min max mean med N50 sum | |||
ctg 74337 88 840370 35148 14148 79144 2612810882* | |||
==== UMD2.6.1 noVariants,noCont ==== | |||
elem min max mean med N50 sum | |||
scf 2646 385 34167202 994701 20787 7139718 2631980624 | |||
ctg 50755 65 1160130 50966 29450 88583 2586785910* | |||
==== UMD2.6.1 noVariants ==== | |||
elem min max mean med n50 sum | |||
scf 39844 316 34167202 68352 1361 7451988 2,723,419,938 | |||
scf.placed 4707 385 34167202 564225 10413 7800796 2655811430 | |||
scf.variants 29436 723 51828 1714 1298 1514 50,461,989 | |||
scf.unplaced 30575 316 451968 1845 1309 1601 56440663 | |||
elem min max mean med n50 sum | |||
ctg 89994 65 1160130 29736 5180 95952 2676109378 | |||
ctg.placed 53646 65 1160130 48650 26840 98428 2609925446 | |||
ctg.unplaced 31510 101 207476 1761 1314 1556 55511410 | |||
elem min max mean med n50 sum | |||
deg(all) 251413 65 39964 1003 984 994 252,279,234 | |||
deg(>2Kbp) 2128 2000 39964 3753 2827 3946 7,986,872 | |||
deg.placed 883 200 30246 2711 1303 4845 2,393,905 | |||
deg.variants 4654 331 8039 1117 989 1024 5,200,553 | |||
deg.unplaced(?) 245343 65 39964 994 984 993 244047258 | |||
scf.unplaced 978 316 451968 7932 3736 15828 7757882 | |||
scf.uplaced.0cvg 863 316 451968 7521 2286 15744 6,491,342 | |||
deg.unplaced 747 2002 39964 4365 3509 4819 3261181 | |||
deg.unplaced.0cvg 734 2002 39964 4358 3507 4811 3,199,221 | |||
ct_deg.placed 53646 65 1160130 48650 26840 98428 2,609,925,446* | |||
UMD2.0-UMD2.6.1 gaps>1K summary | |||
id count min max median sum | |||
1 775 999 64953 1683 2884541 | |||
... | |||
30 2174 999 167808 1528 7739170 | |||
total 14936 999 167808 1638 53,526,388 | |||
* align UMD2.6 Chr1..30 to UMD2.0 Chr1..30 0cvg regions | |||
elem min max mean med n50 sum | |||
all 14936 1000 167809 3584 1639 6465 53541324 | |||
aligned 11091 1000 167809 4010 1692 7977 44475715 | |||
not_aligned 3845 1000 50546 2357 1529 2735 9065609 | |||
Files: UMD2.0 regions not covered by UMD2.6.1 (chr aligned to itself) | |||
/scratch1/bos_taurus/Assembly/2009_0312_CA/synteny/agp_markers.noVariants_nucmer.noVariants/nucmer_UMD2.0/Chr.0cvg.fa | |||
/scratch1/bos_taurus/Assembly/2009_0312_CA/synteny/agp_markers.noVariants_nucmer.noVariants/nucmer_UMD2.0/Chr.0cvg.posmap | |||
/scratch1/bos_taurus/Assembly/2009_0312_CA/synteny/agp_markers.noVariants_nucmer.noVariants/nucmer_UMD2.0/Chr.0cvg.summary | |||
* UMD2.6.1 vs UMD2.0 Chr1..30 | |||
* nucmer -mum -l 50 -c 250 | |||
* max gap: Chr8:60134434..60228381=93948 | |||
* gaps>10K: 191 , 4.2M total, 4.19M aligned without "-mum" | |||
elem min max mean med n50 sum | |||
gaps>10K 191 10077 93948 22106 15599 26551 4222342 | |||
* realign all scaff without using nucmer "-mum" option: all gaps >10K align to large scaffolds !!! | |||
==== UMD2.6.1 noVariants, add deg & UMD2.0 alignments ==== | |||
--[[User:Dpuiu|Dpuiu]] 09:44, 22 June 2009 (EDT) | |||
elem min max mean med n50 sum | |||
scf 39844 316 34167202 68352 1361 7451988 2723419938 | |||
scf.variants 29436 723 51828 1714 1298 1514 50461989 | |||
scf.noVariants 10580 316 34167202 252690 1957 7740810 2673467579 | |||
scf.noVariants.ignore 4079 316 4985 1852 1541 1842 7558196 # less than 5K and placed through alignments inside a larger scaffold | |||
scf.noVariants.noIgnore 6506 385 34167202 409763 6673 7740810 2665923763 | |||
ctg.variants 29967 723 51828 1641 1298 1475 49178546 | |||
deg.variants 4654 331 8039 1117 989 1024 5200553 | |||
scf_deg.placed 4933 385 34167202 540139 11548 7740810 2664507818 | |||
scf_deg.markers 2003 1020 34167202 1302992 30456 8170786 2609894380 | |||
scf_deg.alignHS 1818 385 3278163 27070 12322 47406 49214215 | |||
scf_deg.alignUMD2.0 1112 1002 73626 4855 2948 6898 5399223 | |||
scf.placed 4044 385 34167202 657786 15617 7740810 2660090134 | |||
scf.markers 1825 1020 34167202 1429490 37672 8170786 2608820353 | |||
scf.alignHS 1587 385 3278163 30006 13998 50604 47620297 | |||
scf.alignUMD2.0 632 1002 73626 5774 5137 9046 3649484 | |||
deg.placed 889 2002 39964 4969 3771 6165 4417684 | |||
deg.markers 178 2007 30168 6033 4336 7350 1074027 | |||
deg.alignHS 231 2049 39964 6900 6289 7337 1593918 | |||
deg.alignUMD2.0 480 2002 15992 3645 2767 3842 1749739 | |||
ctg_deg.placed 54129 65 1160130 48371 26493 98075 2,618,296,162* | |||
ctg.placed 53240 65 1160130 49096 27309 98285 2,613,878,478 | |||
deg.placed 889 2002 39964 4969 3771 6165 4,417,684 | |||
==== HS-BT Synteny map ==== | |||
* Trust scaffolds with 4+ markers | |||
* Scaffolds with 3- markers must have a close neighbor that agrees with them | |||
same HS & BT Chr | |||
maxCount=2 # at most 2 scaffolds away | |||
minMarkers=4 | |||
minRatio=0.66 & maxRatio=1.5 # distance ratio; maxDistance not used !!! | |||
=> 226 synteny regions !!! | |||
join2.pl nucmer_*lsf markers_scf.*lsf | \ | |||
~/bin/filterMarkers.pl -minMarkers 4 | \ | |||
~/bin/getSyntenyBlock.pl | \ | |||
grep -v ^# | grep -v ^$ | ~/bin/flipSummary.pl | sort -nk2 -nk5 | ~/bin/tab2tab.pl | \ | |||
perl -ane 'print $P[13]," ",$F[13],"\n" if($F[13]-$P[13]==1); print $F[13]," ",$P[13],"\n" if($P[13]-$F[13]==1); @P=@F;' | sort -u -n | \ | |||
~/bin/mergeMap.pl >! hs-bt.map.tmp | |||
join2.pl nucmer_*lsf markers_scf.*lsf | \ | |||
~/bin/filterMarkers.pl -minMarkers 4 | \ | |||
~/bin/getSyntenyBlock.pl -map hs-bt.map.tmp | \ | |||
~/bin/tab2tab.pl | grep # | sed 's/#//' > hs-bt.map | |||
~/bin/map-draw.pl -refLen hs.infoseq -qryLen bt.infoseq hs-bt.map > ! hs-bt.png | |||
Problems: | |||
7180002041220: BT.Chr2 ok | |||
7180002041025: BT.Chr4 | |||
7180002041222: BT.Chr7 | |||
7180002041228: BT.Chr8 | |||
7180002040195: BT.Chr8; HS.Ch23 del !!! | |||
7180002041001: BT.Chr8; HS.Ch23 del !!! | |||
7180002040851: BT.Chr30; HS.Ch3 -> HS.Ch23 | |||
7180002041008: BT.Chr30; HS.Ch7 -> HS.Ch23 | |||
* [[Media:hs-bt.png|hs-bt.png]] Map picture | |||
* Map: | |||
#HS-ref begin end len HS-clust BT-ref begin end BT-clust #scf | |||
01 870247 12637274 11767027 16 46893147 58621226 11728079 -1 5 | |||
01 16674545 30031540 13356995 2 126668764 139027145 12358381 -2 4 | |||
01 32016003 68894564 36878561 3 83066354 120874874 37808520 -3 10 | |||
01 68907947 122485896 53577949 3 23819464 83221225 59401761 -4 6 | |||
01 143725713 143737025 11312 3 25097798 25109110 11312 5 1 | |||
... | |||
23 148835146 151656716 2821570 30 32889252 35700335 2811083 -276 3 | |||
23 152157769 154641974 2484205 30 39322867 42104054 2781187 -277 2 | |||
23 154387850 154403684 15834 30 39353299 39369133 15834 278 1 | |||
Files: | |||
/scratch1/bos_taurus/Assembly/2009_0312_CA/synteny/map.4/hs-bt.map | |||
/scratch1/bos_taurus/Assembly/2009_0312_CA/synteny/map.4/hs-bt.png | |||
==== Overlaps ==== | |||
Cases: | |||
1. CONTAINED scaffolds (clear variants) | |||
2. single BEGIN/END between 2 scaffolds: 2 scaffolds could be merged | |||
3. scaffold closing a sequence: 3 scaffolds could be merged | |||
4. multiple BEGIN/END/CONTAIN* between 2 scaffold contigs | |||
1: CONTAINED ~ 4731 cases | |||
cat nucmer_scf.ovl/all-all.contained.ids marker_scf.ovl/all-all.contained.ids | sort -u > all-all.contained.ids | |||
* Summary | |||
elem <2Kbp 2..10Kbp >10Kbp min max mean med n50 sum | |||
variants(all) 4731 3503 1085 143 723 75436 2513 1429 3469 11891882 | |||
variants(1+markers) 541 311 170 60 1001 75421 4466 1690 9947 2416312 | |||
* Example: longest contained scaffold 75Kbp | |||
#scaffold alignments | |||
117893 126367 | 8520 1 | 8475 8520 | 94.06 | 562884 75576 | 1.51 11.27 | 7180002040891 7180002040646 | |||
286811 289412 | 72974 75576 | 2602 2603 | 93.29 | 562884 75576 | 0.46 3.44 | 7180002040891 7180002040646 | |||
421477 421977 | 9727 10223 | 501 497 | 95.41 | 562884 75576 | 0.09 0.66 | 7180002040891 7180002040646 | |||
171970 180765 | 73605 64798 | 8796 8808 | 96.74 | 274750 75576 | 3.20 11.65 | 7180002040912 7180002040646 | |||
180765 195989 | 64652 49398 | 15225 15255 | 96.05 | 274750 75576 | 5.54 20.18 | 7180002040912 7180002040646 | |||
196223 203368 | 49244 42105 | 7146 7140 | 96.81 | 274750 75576 | 2.60 9.45 | 7180002040912 7180002040646 | |||
205935 208456 | 38652 36130 | 2522 2523 | 96.12 | 274750 75576 | 0.92 3.34 | 7180002040912 7180002040646 | |||
207096 210496 | 42082 38673 | 3401 3410 | 96.07 | 274750 75576 | 1.24 4.51 | 7180002040912 7180002040646 | |||
209529 217353 | 36109 28272 | 7825 7838 | 95.50 | 274750 75576 | 2.85 10.37 | 7180002040912 7180002040646 | |||
213859 219155 | 25520 20220 | 5297 5301 | 95.80 | 274750 75576 | 1.93 7.01 | 7180002040912 7180002040646 | |||
221183 227630 | 20236 13870 | 6448 6367 | 93.63 | 274750 75576 | 2.35 8.42 | 7180002040912 7180002040646 | |||
227923 228686 | 13884 13126 | 764 759 | 95.16 | 274750 75576 | 0.28 1.00 | 7180002040912 7180002040646 | |||
228084 231413 | 13104 9760 | 3330 3345 | 94.10 | 274750 75576 | 1.21 4.43 | 7180002040912 7180002040646 | |||
#contig alignments | |||
1929 10403 | 8520 1 | 8475 8520 | 94.06 | 26592 8520 | 31.87 100.00 | 7180002040891.4.10 7180002040646.1.8 [CONTAINS] | |||
9084 11685 | 30871 33473 | 2602 2603 | 93.29 | 276174 33473 | 0.94 7.78 | 7180002040891.8.10 7180002040646.8.8 | |||
143750 144250 | 1187 1683 | 501 497 | 95.41 | 276174 4564 | 0.18 10.89 | 7180002040891.8.10 7180002040646.2.8 | |||
163797 172592 | 31502 22695 | 8796 8808 | 96.74 | 233551 33473 | 3.77 26.31 | 7180002040912.3.5 7180002040646.8.8 | |||
172592 187816 | 22549 7295 | 15225 15255 | 96.05 | 233551 33473 | 6.52 45.57 | 7180002040912.3.5 7180002040646.8.8 | |||
188050 195195 | 7141 2 | 7146 7140 | 96.81 | 233551 33473 | 3.06 21.33 | 7180002040912.3.5 7180002040646.8.8 | |||
197762 200283 | 2523 1 | 2522 2523 | 96.12 | 233551 2523 | 1.08 100.00 | 7180002040912.3.5 7180002040646.6.8 [CONTAINS] | |||
198923 202323 | 3410 1 | 3401 3410 | 96.07 | 233551 3411 | 1.46 99.97 | 7180002040912.3.5 7180002040646.7.8 [CONTAINS] | |||
201356 209180 | 7838 1 | 7825 7838 | 95.50 | 233551 7838 | 3.35 100.00 | 7180002040912.3.5 7180002040646.5.8 [CONTAINS] | |||
205686 210982 | 12396 7096 | 5297 5301 | 95.80 | 233551 12396 | 2.27 42.76 | 7180002040912.3.5 7180002040646.3.8 | |||
213010 219457 | 7112 746 | 6448 6367 | 93.63 | 233551 12396 | 2.76 51.36 | 7180002040912.3.5 7180002040646.3.8 | |||
219750 220513 | 760 2 | 764 759 | 95.16 | 233551 12396 | 0.33 6.12 | 7180002040912.3.5 7180002040646.3.8 | |||
219911 223240 | 4564 1220 | 3330 3345 | 94.10 | 233551 4564 | 1.43 73.29 | 7180002040912.3.5 7180002040646.2.8 [CONTAINS] | |||
#marker & alignment summary | |||
#id BT-ref #markers slope begin end len HS-ref #align slope begin end len | |||
7180002040891 4 1 0 13956053 14518937 562884 22 14 -1.5687 20838730 21401614 562884 | |||
7180002040163 8 1 -1 38079677 38178351 98674 22 3 1.0264 21133894 21232568 98674 update:dir:7180002040646 | |||
7180002040646 8 . 1 38088797 38164373 75576 22 4 -0.9833 21143014 21218590 75576 assign:Chr:7180002040163,7180002040163 | |||
2. ~ 349 cases (-3 cases 3.) | |||
cat all-all.begin.ids all-all.end.ids | sort -u | wc -l | |||
3. ~ 3 cases | |||
intersect.pl all-all.begin.ids all-all.end.ids | perl -ane 'print $_ if($F[1]=~/1.1$/);' | |||
7180002032818.1.1 23259 | |||
7180002036943.1.1 28355 | |||
7180002040409.1.1 25219 # merges 2 scaff | |||
1 2811 | 2807 1 | 2811 2807 | 99.11 | 23259 5134 | 12.09 54.67 | 7180002032818.1.1 7180002032811.1.2[BEGIN] [BEGIN] | |||
22101 23259 | 7238 6079 | 1159 1160 | 99.66 | 23259 7238 | 4.98 16.03 | 7180002032818.1.1 7180002032811.2.2[END] [END] | |||
1 2811 | 2807 1 | 2811 2807 | 99.11 | 23259 5134 | 12.09 54.67 | 7180002032818.1.1 7180002032811.1.2[BEGIN] [BEGIN] | |||
22101 23259 | 7238 6079 | 1159 1160 | 99.66 | 23259 7238 | 4.98 16.03 | 7180002032818.1.1 7180002032811.2.2[END] [END] | |||
1 2025 | 2066 4090 | 2025 2025 | 99.80 | 25219 4090 | 8.03 49.51 | 7180002040409.1.1 7180002033541.1.1[BEGIN] [END] | |||
23394 25219 | 5761 3930 | 1826 1832 | 98.42 | 25219 5761 | 7.24 31.80 | 7180002040409.1.1 7180002033538.1.1[END] [END] | |||
cd nucmer_scf.ovl | |||
intersect.pl all-all.begin.ids all-all.end.ids | |||
7180002032818 23259 | |||
7180002036943 28355 | |||
7180002040409 25219 | |||
4. ~10 cases | |||
cd nucmer_ctg.ovl/translated/ | |||
cat all-all.annotated.coords | egrep 'BEGIN|END|CONTAIN' | p 'next if($F[6]<5000); next if($F[7]<5000); print $_;' | p '$F[17]=~/^([^.]+)(.+)/; $F[17]=$1 ; $F[18]=~/^([^.]+)/; $F[18]=$1; print $F[17],"\t",$F[18],"\n";' | count.pl -m 2 | |||
7180002041235 brk002041306a 3 | |||
7180002038888 7180002034914 2 | |||
7180002041059 7180002040095 2 | |||
7180002040879 7180002040934 2 | |||
cnt0002041350 7180002040907 2 | |||
7180002041341 7180002040894 2 | |||
7180002041015 cnt0002040938 2 | |||
7180002039401 7180002040397 2 | |||
7180002037358 7180002031425 2 | |||
7180002040915 7180002039470 2 | |||
cat all-all.annotated.coords | egrep 'BEGIN|END|CONTAIN' | grep ... | |||
12363 21143 | 1 8837 | 8781 8837 | 93.43 | 21143 50443 | 41.53 17.52 | 7180002041235.23.33 brk002041306a.5.7[END] [BEGIN] | |||
1 9272 | 5591 14944 | 9272 9354 | 94.47 | 9272 50443 | 100.00 18.54 | 7180002041235.25.33 brk002041306a.5.7[CONTAINED] | |||
71049 97689 | 31941 5220 | 26641 26722 | 96.71 | 97689 31941 | 27.27 83.66 | 7180002041235.29.33 brk002041306a.6.7[END] [END] | |||
57 3896 | 14743 10903 | 3840 3841 | 95.94 | 3896 22667 | 98.56 16.95 | 7180002041235.32.33 brk002041306a.3.7[CONTAINED] | |||
1 8942 | 1 8947 | 8942 8947 | 99.49 | 20792 8947 | 43.01 100.00 | 7180002038888.1.1 7180002034914.1.2[CONTAINS] | |||
9670 20792 | 1 11091 | 11123 11091 | 99.27 | 20792 11091 | 53.50 100.00 | 7180002038888.1.1 7180002034914.2.2[CONTAINS] | |||
1 8705 | 3932 12610 | 8705 8679 | 96.17 | 8705 30654 | 100.00 28.31 | 7180002041059.1.26 7180002040095.1.1[CONTAINED] | |||
1 4777 | 12989 17697 | 4777 4709 | 95.32 | 4777 30654 | 100.00 15.36 | 7180002041059.2.26 7180002040095.1.1[CONTAINED] | |||
1 6588 | 17999 24607 | 6588 6609 | 96.15 | 6591 30654 | 99.95 21.56 | 7180002041059.3.26 7180002040095.1.1[CONTAINED] | |||
630 16709 | 16024 1 | 16080 16024 | 97.09 | 16901 16024 | 95.14 100.00 | 7180002040879.2.14 7180002040934.2.87[CONTAINS] | |||
1 5876 | 5882 1 | 5876 5882 | 98.51 | 39657 13643 | 14.82 43.11 | 7180002040879.3.14 7180002040934.1.87[BEGIN] [BEGIN] | |||
== Scaffold links == | |||
Try to identify scaffold that fit the following criteria: | |||
* have no markers and no alignments to HS | |||
* linked by 2+ links to a single scaffold that has markers/alignments to HS | |||
* is not a variant (2Kbp+ of unique sequnece) | |||
elem min max mean med n50 sum | |||
linked 6169 316 88326 2408 1480 2709 14,860,479 | |||
linked(2+mates) 2109 316 44120 2595 1558 3092 5,474,510 | |||
linked(2+mates to a single scf) 2057 316 44120 2516 1547 2850 5,177,182 | |||
linked(2+mates to a single scf, no variant) 112 316 43782 7278 1987 23338 815,223 # 46>2Kbp; 25>10Kbp | |||
=> 112 scaffold & 0.81Mbp could be added to the chromosomes !!! | |||
== BT alignments == | |||
* UMD2.6.1 vs UMD2.6.1 | |||
# Marker scaffolds against themselves (2642 total scaffolds) : nucmer -maxmatch -l 40 -c 2500 -g 250 | |||
# Mapped scaffolds without markers against marker scaffolds: nucmer -maxmatch -l 40 -c 250 | |||
# Mapped scaffolds without markers against themselves: nucmer -maxmatch -l 40 -c 250 | |||
File location: | |||
/scratch1/bos_taurus/Assembly/2009_0312_CA/nucmer_Chr/ | |||
== Other issues == | |||
=== Chr10 gene duplication === | |||
* LOC100298457 | |||
* See /nfshomes/dpuiu/Readmes/bos_taurus.runCA.13.txt | |||
=== Scf breaks (Mike Robers) === | |||
:::::::::::::: | |||
scf.break.ids | |||
:::::::::::::: | |||
#ctg scf beg end dir | |||
7180002015438 7180002038435 0 32822 f # ok: 4 markers from Chr18 | |||
7180002021916 7180002040442 0 24559 f # ok: 1 marker from Chr10 | |||
7180001725791 7180002040808 466123 562461 f # ok: 25 marker from Chr1, 1 marker from Chr2 | |||
7180001727899 7180002040844 8236574 8479429 f # also found by us; 3 Chr30 markers in the middle | |||
7180001854650 7180002041103 5230302 5294734 f # also found by us; 4 Chr15 markers at 3' | |||
7180002003578 7180002041216 300250 392069 f # also found by us; 7 Chr5 markers at 5' | |||
7180002020010 7180002041293 4482477 4600045 f # ok | |||
7180001722390 7180002041353 0 51674 f # ok | |||
:::::::::::::: | |||
scf.excised.ids | |||
:::::::::::::: | |||
#ctg scf beg end dir | |||
7180002008629 7180002034664 82740 105723 f # no markers | |||
7180001786240 7180002040913 2076746 2112375 f # ok | |||
7180001787022 7180002040927 1075352 1109350 f # ok | |||
7180001787352 7180002040931 4956099 4967823 f # ok | |||
7180001789387 7180002040981 6641924 6671967 f # ok | |||
7180001789575 7180002040984 3095587 3154302 f # ok | |||
7180002003269 7180002041209 6350890 6470971 f # ok | |||
7180002025281 7180002041322 989366 1227636 f # ok | |||
7180002026741 7180002041328 343726 414329 f # ok | |||
7180002029433 7180002041343 834493 888144 f # ok | |||
7180001726451 7180002041383 7893629 7901645 f # ok | |||
=== LOC100298457 (duplicate gene on Chr10) === | |||
Problem: | |||
Is LOC100298457 gene (cow Chr10) a variation of MFSD3 gene (cow Chr14)? | |||
UMD2.0 | |||
Chr10 15186759 15189292 833 W 7180003260677 1 2534 + # gene LOC100298457 | |||
Chr14 1839355 1958202 533 W 7180003326040 1 118848 + # gene MFSD3 | |||
* LOC100298457 aligns to 2 UMD2.6 scaffolds: | |||
scf7180002041112 (1.56Mbp; 34 contigs) | |||
scf7180002033841 (2.44Kbp; 1 contig) | |||
* scf7180002041112 contains 39 bos taurus Chr14 markers and has 206 alignments to human chromosome 8 | |||
* scf7180002033841 contains no bos taurus markers and has 1 alignments to human chromosome 8. | |||
* scf7180002033841 5' aligns to scf7180002041112 (1.56Mbp; 34 contigs, cow Chr14) and to a human Chr8 region that maps to cow Chr14 | |||
* scf7180002033841 3' aligns to scf7180002041157 (6.15Mbp; 99 contigs, cow Chr10) | |||
* scf7180002033841 & scf7180002041157 are linked by 2 mate pairs (inserts from a 3kbp BCM shotgun library) | |||
* LOC100298457 vs UMD2.6 scf: | |||
[S1] [E1] | [S2] [E2] | [LEN 1] [LEN 2] | [% IDY] | [LEN R] [LEN Q] | [COV R] [COV Q] | [TAGS] | |||
=============================================================================================================================== | |||
1 88 | 1138313 1138226 | 88 88 | 100.00 | 1344 1561688 | 6.55 0.01 | LOC100298457 scf7180002041112 | |||
80 192 | 1138150 1138038 | 113 113 | 100.00 | 1344 1561688 | 8.41 0.01 | LOC100298457 scf7180002041112 | |||
190 316 | 1137974 1137848 | 127 127 | 100.00 | 1344 1561688 | 9.45 0.01 | LOC100298457 scf7180002041112 | |||
315 464 | 1137770 1137621 | 150 150 | 100.00 | 1344 1561688 | 11.16 0.01 | LOC100298457 scf7180002041112 | |||
455 1344 | 1137322 1136432 | 890 891 | 99.78 | 1344 1561688 | 66.22 0.06 | LOC100298457 scf7180002041112 | |||
1 1344 | 1349 6 | 1344 1344 | 100.00 | 1344 2445 | 100.00 54.97 | LOC100298457 scf7180002033841 | |||
* UMD2.6 scf* vs scf7180002033841: | |||
[S1] [E1] | [S2] [E2] | [LEN 1] [LEN 2] | [% IDY] | [LEN R] [LEN Q] | [COV R] [COV Q] | [TAGS] | |||
=============================================================================================================================== | |||
1136427 1137322 | 1 895 | 896 895 | 99.78 | 1561688 2445 | 0.06 36.61 | scf7180002041112 scf7180002033841 | |||
4060495 4061426 | 2445 1521 | 932 925 | 97.76 | 6157473 2445 | 0.02 37.83 | scf7180002041157 scf7180002033841 | |||
Bos taurus marker summary: | |||
#id BT-ref #markers slope begin end len | |||
scf7180002041112 Chr14 39 -0.8518 3377707 4939395 1561688 | |||
scf7180002041157 Chr10 255 1.0035 12831593 18989066 6157473 | |||
scf7180002033841 ? 0 | |||
Homo sapiens alignment summary: | |||
#id HS-ref #align slope begin end len | |||
scf7180002041112 Chr8 206 0.6701 144381103 145942791 1561688 | |||
scf7180002041157 Chr15 1667 0.987 62245836 68403309 6157473 | |||
scf7180002033841 Chr8 1 1 145705320 145707765 2445 | |||
[S1] [E1] | [S2] [E2] | [LEN 1] [LEN 2] | [% IDY] | [LEN R] [LEN Q] | [COV R] [COV Q] | [TAGS] | |||
=============================================================================================================================== | |||
143996007 143996305 | 74278 74572 | 299 295 | 82.45 | 146274826 1561688 | 0.00 0.02 | 1 1 NC_000008 7180002041112 | |||
... | |||
146249212 146249657 | 1268887 1268446 | 446 442 | 80.89 | 146274826 1561688 | 0.00 0.03 | 1 -1 NC_000008 7180002041112 | |||
63013167 63013429 | 887697 887966 | 263 270 | 78.15 | 100338915 6157473 | 0.00 0.00 | 1 1 NC_000015 7180002041157 | |||
... | |||
68369047 68369417 | 6154757 6155112 | 371 356 | 83.06 | 100338915 6157473 | 0.00 0.01 | 1 1 NC_000015 7180002041157 | |||
145705522 145706206 | 202 886 | 685 685 | 81.28 | 146274826 2445 | 0.00 28.02 | 1 1 NC_000008 7180002033841 | |||
scf7180002033841: 6 reads | |||
read scf begin end dir | |||
1120017508 7180002033841 0 1018 f | |||
1120020725 7180002033841 227 1109 f | |||
1120020722 7180002033841 724 1720 r | |||
1120020726 7180002033841 1006 2073 f diffScaffold | |||
1120017511 7180002033841 1303 2175 r | |||
1120020728 7180002033841 1370 2445 f diffScaffold | |||
scf7180002033841: 4 mates | |||
read1 read2 scf1 scf2 | |||
1120017508 1120020722 7180002033841 7180002033841 | |||
1120017511 1120020725 7180002033841 7180002033841 | |||
1120020726 1120017512 7180002033841 7180002041157 diffScaffold | |||
1120020728 1120017514 7180002033841 7180002041157 diffScaffold | |||
scf7180002041157: 69095 reads; frg 1120017514 & 1120017512 positions close to the region aligned to scf7180002033841 (4060495-4061426) | |||
read scf begin end dir | |||
607312594 7180002041157 0 866 f | |||
... | |||
1120017512 7180002041157 4059332 4060396 f diffScaffold | |||
1120017514 7180002041157 4058851 4059931 f diffScaffold | |||
.... | |||
583956204 7180002041157 6157144 6157228 r | |||
scf7180002041157: 99 contigs | |||
count ctg scf begin end dir | |||
1 ctg7180001926175 scf7180002041157 0 81467 f | |||
.. | |||
64 ctg7180001926231 scf7180002041157 4039922 4046680 f # 6758bp ctg; 800bp gap following the contig | |||
65 ctg7180001926232 scf7180002041157 4047480 4193693 f # 146213bp ctg ; 20bp gap following the contig | |||
66 ctg7180001926233 scf7180002041157 4193713 4208047 f # 14334bp ctg | |||
.. | |||
99 ctg7180001926260 scf7180002041157 6152691 6157473 f | |||
frg 1120017514 & 1120017512 positions on ctg7180001926232 | |||
1120017514 ctg7180001926232 11371 12451 f # reads are 11kbp(>3Kb lib mean) inside the contig | |||
1120017512 ctg7180001926232 11852 12916 f # reads are 11kbp(>3Kb lib mean) inside the contig | |||
=== Missing genes === | |||
* 5 genes not found by Liliana using ESTaligner but found using gmap (%id<95) : they were on the haplotype variants | |||
=== Chr27 -> Chr21 === | |||
* The centromeric end of BTA27 is actually placed on BTA21 in UMD2.6 (which version?) | |||
== Chromosomes == | |||
=== synteny.redo2 === | |||
* summary.txt : 8122 scf + 6322 deg => 14444 seqs (placed using markers or synteny to HS) | |||
* from summary.txt removed: | |||
58 scf.questionable.ids (44 placed, 14 linked) | |||
100 ctg.questionable.ids | |||
931 deg.questionable.ids | |||
=== UMD_2.6.d_g === | |||
. elem <2000 >2000 min max mean med n50 sum | |||
haplotypes_contigs 3828 2701 1127 471 123243 2503 1580 2590 9,584,985 | |||
=== UMD_2.6.a_g_070109 === | |||
Gaps: 65900 | |||
* all 65900 gaps are "fragment yes" | |||
=== UMD_2.6.a_g_070509 === | |||
* Combine UMD_2.6.a_g_070109 (Guillaume's assembly) and UMD2.6.1 (Daniela's) | |||
=== UMD_2.6.a_g_070709 === | |||
* Remove from UMD_2.6.a_g_070509 | |||
** ~ 21 ChrY ctg & ~ 39 ChrY deg | |||
** ~ 4206 haplotype variants (6.63Mbp) within 1K from one another | |||
. elem <2000 >2000 min max mean med n50 sum | |||
ctg_deg.variants.placed 6654 5504 1150 263 42158 1761 1189 1772 11719653 | |||
ctg_deg.variants.placed.sameChr 5374 4455 919 263 42158 1748 1152 1775 9393815 | |||
ctg_deg.variants.placed.within_100K 4790 4100 690 263 42158 1651 1128 1605 7911160 | |||
ctg_deg.variants.placed.within_1K* 4206 3679 527 263 42158 1577 1107 1508 6633118 | |||
Summary: | |||
ctg+deg <2000 >=2000 min max mean med n50 sum | |||
Chr1..29,X 72197 20763 51434 65 1160130 36536 13055 97328 2,637,809,286 | |||
ChrU 3752 2587 1165 362 179692 3284 1447 6427 12,324,356 | |||
ChrY-contigs 315 266 49 224 26490 2249 974 6679 708,535 | |||
contigs.haplotype-variants 40198 36720 3478 263 51828 1460 1203 1361 58,698,457 | |||
deg.unplaced.less_2K 224945 224945 0 65 1996 972 983 990 218,847,978 | |||
=== Issues === | |||
* 490 scf don't have all ctgs placed (865 ctgs) | |||
* 699 reliable contigs (3.25Mbp) unplaced | |||
difference.pl ctg.reliable.ids UMD_2.6.a_g_070709/Chr.posmap | getSummary.pl -i 2 -z 2000 | |||
elem <2000 >2000 min max mean med n50 sum | |||
699 405 294 362 123243 4660 1759 14809 3257635 | |||
* Rearrangements UMD2.0 vs UMD_2.6.a_g_070709 : ~ 25 ctgs>50K ; ~ 12 scf | |||
* AGP file format: "fragment yes" should be preserved even if gap type=U | |||
grep -A 2 7180001925241 UMD_2.6.a_g_070109/Chr.agp | |||
Chr30 131686408 131762880 14157 W 7180001925241 1 76473 + | |||
Chr30 131762881 131762980 14158 U 100 fragment yes | |||
Chr30 131762981 131796167 14159 W 7180001925242 1 33187 + | |||
grep -A 2 7180001925241 ../UMD_2.6.a_g_070709/Chr.agp | |||
ChrX 133855417 133931889 15581 W 7180001925241 1 76473 + | |||
ChrX 133931890 133931989 15582 U 100 contig no | |||
ChrX 133931990 133965176 15583 W 7180001925242 1 33187 + | |||
* Liliana found 18 ctg + deg that have genes | |||
Fixed case B (7180001932648,7180001925237) | |||
* Bob (Missouri) | |||
# scf7180002041216 (43 ctgs, 1.44Bmp) should be split : Chr15 (36 ctgs) , Chr5 (first 7 ctgs) ctg7180002003576,ctg7180002003577 moved from Chr9=>Chr5 | |||
=> Chr5 | |||
1 7180002003574 7180002041216 0 19886 f # already on Chr5 | |||
2 7180001694221 7180002041216 19906 20911 r | |||
3 7180001978231 7180002041216 20931 52725 r | |||
4 7180001946738 7180002041216 53559 55402 r | |||
5 7180002003575 7180002041216 55422 92145 f | |||
6 7180002003576 7180002041216 92165 200007 f | |||
7 7180002003577 7180002041216 200456 300048 f | |||
# scf7180002041153 (48 ctgs, 2.36Mbp): assignment to Chr11 seems correct, Chr6(0) | |||
=== UMD_2.6.a_g_071709 -> UMD_Freeze2.99 === | |||
Changes vs UMD_2.6.a_g_070709: | |||
* 17 contigs/degenerates recruited by Liliana based on mRNA alignments got placed on chromosomes | |||
* Chr9,15 => Chr5 correction : a scaffold got broken between 2 chromosomes & and several contigs got moved to Chr5 | |||
* 697 unplaced contigs from placed scaffolds got placed as well | |||
Contig placement summary: | |||
#ctg+deg <2Kbp >=2Kbp min max mean med n50 sum | |||
================================================================================================== | |||
Chr1..29,X 72911 21180 51731 65 1160130 36223 12706 97232 2641097363 | |||
ChrU 3365 2448 917 224 179692 2898 1348 5399 9754701 | |||
contigs.haplotype-variants 40198 36720 3478 263 51828 1460 1203 1361 58698457 | |||
deg.unplaced.less_2K 224933 224933 0 65 1996 972 983 990 218837572 | |||
Chr1..29,X(new) 714 417 297 177 123243 4605 1741 14809 3288596 | |||
ChrY-contigs 315 266 49 224 26490 2249 974 6679 708535 | |||
ChrY-contigs.SHOTGUN_ONLY 144 140 4 804 4224 993 882 888 143047 | |||
=================================================================================================== | |||
Comments: | |||
* "Chr1..29,X", ChrU, contigs.haplotype-variants, deg.unplaced.less_2K are mutually exclusive sets | |||
* "Chr1..29,X(new)" are contigs which were not placed in UMD_2.6.a_g_071709. | |||
** 17 contig them were added by Liliana (1 failed) | |||
** the rest are reliable contigs left unplaced by Aleksey/Guillaume program | |||
* ChrY-contigs.SHOTGUN_ONLY are a subset of ChrY-contigs which don't contain only SHOTGUN reads | |||
* ChrY-contigs are part of ChrU | |||
Files (walnut): | |||
/scratch1/bos_taurus/Assembly/2009_0312_CA/scf_placements/UMD_2.6.a_g_071709/ # FASTA & AGP format | |||
/scratch1/bos_taurus/Assembly/2009_0312_CA/scf_placements/UMD_2.6.a_g_071709/nucmer_UMD2.0/ # nucmer alignments to UMD2.0 | |||
Files (freeze): | |||
/fs/szasmg3/bos_taurus/UMD_Freeze2.99/ # FASTA & AGP format | |||
/fs/szasmg3/bos_taurus/UMD_Freeze2.99/ncbi_files # SEQUIN format | |||
Ftp: | |||
ftp://ftp.cbcb.umd.edu/pub/salzberg/Bos_taurus_2.99/ -> /fs/ftp-cbcb/pub/salzberg/Bos_taurus_2.99/ | |||
=== UMD_2.6.a_g_072109 -> UMD_Freeze3.0 === | |||
Changes vs UMD_2.6.a_g_070709: | |||
* Delete 97 contaminated sequences found by NCBI (all except the primates) : http://www.ncbi.nlm.nih.gov/projects/WGS/screens/DAAA02_071709/ | |||
* Delete 441 haplotype variants found by Guillaume | |||
* Trim 54 partial contaminants (contaminants were on the ends) | |||
* Trim 7 terminal N's | |||
Gaps: 75739 | |||
* all 27103 N gaps are "fragment yes" | |||
* all 48636 U gaps are "contig no" | |||
Files (walnut): | |||
/scratch1/bos_taurus/Assembly/2009_0312_CA/scf_placements/UMD_2.6.a_g_072109/ # FASTA & AGP format | |||
Files (freeze): | |||
/fs/szasmg3/bos_taurus/UMD_Freeze3.0/ # FASTA & AGP format | |||
/fs/szasmg3/bos_taurus/UMD_Freeze3.0/ncbi_files # SEQUIN format | |||
Ftp: | |||
ftp://ftp.cbcb.umd.edu/pub/data/assembly/Bos_taurus/Bos_taurus_UMD_3.0/ -> pub/data/assembly/Bos_taurus/Bos_taurus_UMD_3.0/ | |||
Issues: | |||
* 7180001836672 941bp deg on Chr4 : aligns on all its length to the cow mitochondrion; placed based on human synteny | |||
* mitochondrion screening was done only on contigs, not on degenerates | |||
* Align all Chr*.fasta files to cow mitochondrion; show-coords -I 90 -L 600 | |||
[S1] [E1] | [S2] [E2] | [LEN 1] [LEN 2] | [% IDY] | [LEN R] [LEN Q] | [COV R] [COV Q] | [TAGS] | |||
=============================================================================================================================== | |||
3603 4930 | 6 1333 | 1328 1328 | 90.89 | 16339 1333 | 8.13 99.62 | AY676873 ctg7180001759596 [CONTAINS] | |||
13284 14521 | 1214 1 | 1238 1214 | 97.90 | 16339 1216 | 7.58 99.84 | AY676873 deg7180001872458 [CONTAINS] | |||
15731 16339 | 1 608 | 609 608 | 99.18 | 16339 941 | 3.73 64.61 | AY676873 deg7180001836672 [END] | |||
* Delete 2 degenerates | |||
* Summary 3.a --[[User:Dpuiu|Dpuiu]] 11:04, 5 August 2009 (EDT) | |||
. ctg+deg <2Kbp >=2Kbp min max mean med n50 sum | |||
====================================================================================================== | |||
Chr1..29,X 72479 20862 51617 65 1160130 36424 12941 103785 2639984487 | |||
ChrU 3285 2404 881 224 179692 2890 1338 5425 9496583 | |||
Chr 75764 23266 52498 65 1160130 34970 11207 96955 2649481070 | |||
contigs.haplotype-variants 40611 36984 3627 263 97877 1476 1205 1372 59958728 | |||
deg.unplaced.less_2K 224933 224933 0 65 1996 972 983 990 218837572 | |||
ChrY-contigs 314 266 48 224 26490 2210 973 6539 694140 | |||
ChrY-contigs.SHOTGUN_ONLY 144 140 4 804 4224 993 882 888 143047 | |||
====================================================================================================== | |||
=== UMD_Freeze3.1 === | |||
--[[User:Dpuiu|Dpuiu]] 11:42, 19 November 2009 (EST) | |||
* Only changed some of the gap specifications | |||
* Original(CA): | |||
ctg: 90135 | |||
deg: 251413 | |||
scf: 39978 | |||
* UMD3.1 AGP: | |||
ctg: 60499 | |||
deg: 15229 | |||
scf(CA): 11458 | |||
ctg(unoriented) 2118 | |||
scf(placed chr) 3193 | |||
scf(unplaced chr) 3285 | |||
Files: | |||
/fs/szasmg3/bos_taurus/UMD_Freeze3.1 | |||
== ToDo == | |||
* Align all UMD2.0 ChrU ctg/deg 10Kbp+ to our assembly; make sure everything aligns (Steven's suggestion) | |||
/scratch1/bos_taurus/Assembly/2009_0312_CA/scf_placements/UMD_2.6.a_g_071709/nucmer_UMD2.0 | |||
** all ctgs: 113K, 244Mbp | |||
** 10Kbp+ ctgs: 2561, 54Mbp | |||
** 53,951,168 out of 54,971,011 bp covered (98% of the sequence) | |||
** Most ctgs were added to ChrU | |||
join2.pl -i 4 UMD2.0.ChrU.10K.maxCvg.pair ../other/Chr.posmap | awk '{print $10,$2}' | ~/bin/sum2.pl | sort -nk3 -r | |||
Chr30 252 7,511,975 | |||
Chr1 171 3263575 | |||
Chr6 138 3043388 | |||
Chr12 152 2676615 | |||
ChrU 100 2600562 | |||
... | |||
Chr28 29 473523 | |||
all 2651 54,960,233 | |||
1. Remove remain_haps | |||
. elem <2000 >2000 min max mean med n50 sum | |||
remain_haps 436 282 154 471 37860 2600 1693 2819 1,133,874 | |||
remain_haps(Chr1..30,U) 408 264 144 471 37860 2636 1692 2883 1,075,616 # 77 from ChrU | |||
remain_haps(Chr1..30) 335 226 109 471 37860 2480 1661 2728 831,068 | |||
2. Remove/trim contaminants | |||
3. Remove/trim N's | |||
Latest revision as of 18:11, 8 April 2011
BCM
NCBI Data
- Genome Projects
- TA search
- TA ftp
- 91 volumes: 87 with qual & 4 with no quality (85 volumes contain BCM reads)
- 14 centers
- 21 center/trace_type_codes
- Avg LEN=984
- Avg CLIP (CLB intersect CLV)=760
- Avg CLV=997 > Avg LEN ???
- Avg QUAL=38.96 (27.51 for the 2.59M reads not in the UMD assembly)
- Avg UMDoverlapper CLIP=778
Problems:
- 0 QUAL reads 650,133 ( 551,114 are BCM.WGS )
- the quality lines in several qual. files start with space; need to remove it otherwise tarchive2ca errors out saying that the len(quality)=len(seq)+1
- several xml contained the "&" character => XML parser error
- xml.bos_taurus.087 contained 2 trace_volumes => XML parser error
- BCCAGSC.CLONEEND : all reads have LIBRARY_ID=CH240, SEQ_LIB_ID=. ; the INSERT_SIZE & INSERT_STDEV vary within the library: set to 150,000 & 30,000
- UIUC.CLONEEND: INSERT_SIZE & INSERT_STDEV missing: set to 150,000 & 30,000
CENTER_NAME counts
COUNT CENTER_NAME
1 35629020 BCM Baylor College of Medicine
2 737900 NISC NIH Intramural Sequencing Center
3 652614 BCCAGSC British Columbia Cancer Agency Genome Sciences Centre # TA query_tracedb CENTER_NAME = "BCCAGSC" => 652,510
4 378871 MARC USDA, ARS, US Meat Animal Research Center
5 114753 UIUC University of Illinois at Urbana-Champaign # TA query_tracedb CENTER_NAME = "UIUC" => 106,368
6 107367 BARC USDA, ARS, Beltsville Agricultural Research Center
7 65171 TIGR The Institute for Genome Research
8 53556 GSC Genoscope
9 43033 CENARGEN Embrapa Genetic Resources and Biotechnology
10 18623 SC The Sanger Center
11 15301 UOKNOR University of Oklahoma Norman Campus, Advanced Center for Genome Technology
12 10651 TIGR_JCVIJTC The Institute for Genomic Research, Traces generated at JCVIJTC # TA query_tracedb CENTER_NAME="JCVI"
13 2485 UIACBCB University of Iowa Center for Bioinformatics and Computation Biology (UIACBCB)
14 49 WUGSC Washington University, Genome Sequencing Center # TA query_tracedb CENTER_NAME = "WUGSC" => 9
37829394 total total # TA query_tracedb SPECIES_CODE = "BOS TAURUS" => 37,788,710
TRACE_TYPE_CODE counts
COUNT CENTER_NAME TRACE_TYPE_CODE
1 24863599 BCM* WGS SEQ_LIB_ID:89
2 10748529 BCM* SHOTGUN SEQ_LIB_ID:15543
3 737900 NISC SHOTGUN SEQ_LIB_ID:247
4 125597 BCCAGSC CLONEEND LIBRARY_ID:1 large insert size; some qualityless; !!! almost all have CLIP3=0
5 114753 UIUC CLONEEND LIBRARY_ID:2 insert size missing , no frequent kmers
6 65171 TIGR CLONEEND SEQ_LIB_ID:1 2K & use TRACE_DIRECTION instead of TRACE_END
7 53556 GSC CLONEEND SEQ_LIB_ID:1 large insert size; !!! all have qual=0 and were excluded
8 26246 CENARGEN WGS . no LIBRARY_ID; no SEQ_LIB_ID; no INSERT_SIZE; no INSERT_STDEV; reads have no direction; ~21954 could be paired (same TEMPLATE_ID)
9 25454 BARC CLONEEND SEQ_LIB_ID:14304 !!! all have CLIP3=0
10 16892 BCM* CLONEEND LIBRARY_ID:1 VBBAA mea=167000 std=25000
11 16787 CENARGEN CLONEEND LIBRARY_ID:1
12 15150 UOKNOR SHOTGUN LIBRARY_ID:1 some qualityless
13 10651 TIGR_JCVIJTC CLONEEND SEQ_LIB_ID:2
14 151 UOKNOR FINISHING LIBRARY_ID:1 some qualityless, no direction(TRACE_END=N); no INSERT_SIZE; no INSERT_STDEV
15 49 WUGSC CLONEEND SEQ_LIB_ID:1
36820485 total
16 527017 BCCAGSC EST
17 207204 MARC EST
18 171667 MARC PCR
19 81913 BARC EST
20 18623 SC EST
21 2485 UIACBCB EST
1008909 total
STRATEGY & TRACE_TYPE_CODE counts
COUNT CENTER_NAME STRATEGY TRACE_TYPE_CODE 12545304 BCM . WGS 11425910 BCM WGA WGS 5223683 BCM CLONE SHOTGUN 4479883 BCM POOLCLONE SHOTGUN 1044963 BCM . SHOTGUN 892385 BCM SNP WGS 737900 NISC CLONE SHOTGUN 125597 BCCAGSC CLONEEND CLONEEND 114753 UIUC CLONEEND CLONEEND 65171 TIGR CLONEEND CLONEEND 53556 GSC CLONEEND CLONEEND 26246 CENARGEN . WGS 25454 BARC . CLONEEND 16892 BCM CLONEEND CLONEEND 16787 CENARGEN CLONEEND CLONEEND 12195 UOKNOR . SHOTGUN 10651 TIGR_JCVIJTC CLONEEND CLONEEND 2955 UOKNOR CLONE SHOTGUN 151 UOKNOR . FINISHING 49 WUGSC CLONEEND CLONEEND
527017 BCCAGSC EST EST 145820 MARC EST EST 117958 MARC COMPARATIVE PCR 81913 BARC EST EST 61384 MARC CLONE EST 53709 MARC Re-Sequencing PCR 18623 SC EST EST 2485 UIACBCB . EST
BCM.SHOTGUN libraries
- The long inserts are probably wrong !!!
SIZE STDEV COUNT 3500 1500 4502569 2000 1000 3244493 3000 1000 1021577 180000 1000 840528 6500 1500 429026 180000 13000 320208 6000 2000 208192 167000 13000 96337 3500 15000 85599
SIZE COUNT 3500 4588168 2000 3244493 180000 1160736 3000 1021577 6500 429026 6000 208192 167000 96337
3' VECTOR TRIMMED counts
CENTER_NAME TRACE_TYPE_CODE TOTAL 3'CLV<LEN QUAL==0 UMD.FRG 1 BCM WGS 24863599 10968979 551114 24050767 2 BCM SHOTGUN 10748529 5052692 23419 10068499 3 NISC SHOTGUN 737900 28972 0 735488 4 BCCAGSC CLONEEND 125597 125484 8926 113790 5 UIUC CLONEEND 114753 90243 0 106247 6 TIGR CLONEEND 65171 46389 0 64903 7 GSC CLONEEND 53556 53556 53556 (all) 0 !!! all have 0 quals and were excluded 8 CENARGEN WGS 26246 26246 0 25976 9 BARC CLONEEND 25454 25454 0 25387 10 BCM CLONEEND 16892 6751 0 16863 11 CENARGEN CLONEEND 16787 16787 0 16628 12 UOKNOR SHOTGUN 15150 2885 12195 0 13 TIGR_JCVIJTC CLONEEND 10651 339 0 10644 14 UOKNOR FINISHING 151 0 151 151 15 WUGSC CLONEEND 49 0 0 0 16 BCCAGSC EST 527017 524173 772 0 17 MARC EST 207204 207204 0 0 18 MARC PCR 171667 171667 0 0 19 BARC EST 81913 78597 0 0 20 SC EST 18623 7350 0 0 21 UIACBCB EST 2485 2485 0 0
ZERO QUALITY COUNTS
- Counts
CENTER_NAME TRACE_TYPE_CODE COUNT BCM WGS 551114 GSC CLONEEND 53556 BCM SHOTGUN 23419 UOKNOR SHOTGUN 12195 BCCAGSC CLONEEND 8926 BCCAGSC EST 772 UOKNOR FINISHING 151 TOTAL 650134
- For 0 quality reads, assign quality 20 to bases 1..700, 0 to bases 701..
- Volumes 026..039 have been fixed
Local Data
Files & Dirs
/fs/szasmg3/bos_taurus/data/ /fs/szasmg2/Drosophila/D_pseudoobscura/Vectors /nfshomes/dpuiu/db/UniVec
Software
Figaro
- trims vector only at 5' end
- call lucy trimming for qualities
Lucy
- both vector sequence and splice sites are required
Atlas
- web site
- atlas-screen-trim-file : "calls cross_match and atlas-screen-window to create trimmed reads file (scan in from each end of read looking for 50-base windows of high quality and no vector); "
Contaminant search
nucmer reads CLIPPING range to UniVec & EcoliK12
UniVec
Ref
#seqs min max mean median n50 sum UniVec 2861 12 48551 231 99 781 660,151 UniVec_Core 1348 12 48551 243 98 967 327,641
Hits: alignment length
bp #reads min max mean median n50 sum 19 4548466 19 1045 28.37 23 27 129025025 20 3684852 20 1045 30.56 25 28 112616359 30 1097357 30 1045 48.04 38 43 52714583 40 484661 40 1045 66.36 47 53 32163896 100 54334 100 1045 198 116 223 10772815 # many are ESTs
Ecoli
Ref:
K12 4,639,675 bp
Hits: alignment length
bp #reads min max mean median n50 sum 19 275109 19 1223 30.66 19 20 8435470 20 102550 20 1223 50.29 21 161 5156849 30 19032 30 1223 178 37 706 3381214 40 9234 40 1223 329 171 738 3034293 100 6781 100 1223 424 223 749 2876432 200 4378 200 1223 575 696 771 2516916
BCM vectors
#seqs min max mean median n50 sum BCM 14 2580 33180 9379 5821 32705 131312
Vector/Splice site search
Strategy
- 1. Select all the reads in the same volume that belong to one particular library; same CENTER_NAME, STRATEGY & TRACE_TYPE_CODE
- 2. Get the quality clipping trim: CLIP_QUALITY_LEFT & CLIP_QUALITY_RIGHT
- 3. Separate reads in 2 sets according to direction TRACE_END: FORWARD & REVERSE
- 4. Get the most frequent kmers in each set (24 & 8 bp)
- 5. Check if the most frequent kmers are overrepresented
- 6. Check if the most frequent 8mers are present in the most frequent 24mers
- 7. Try to extend the 24mers by a few bp => linkers
- 8. Align linkers to the opposite stand sequences using nucmer
- 9. Extract the subsequences adjacent(following) to linker (50..150bp)
- 10. Align the subsequences; if they align we've probably identified the vector
- 11. Identify the vector name/id by alignment to UniVec => several alignments
- 12. Check if the forward/reverse vector(s) are the same : we should find a common vector sequence; the UniVec alignments should be adjacent
- 13. create the Lucy vector & splice files; the splice contains the linker+vector
- 14. run lucy & trim input reads according to Lucy clr
- 15. align lucy trimmed reads to linker,vector,splice & UniVec.dust
- 16. align input reads to linker,vector,splice & UniVec.dust
- 17. compare the 15. & 16. counts
Example
- 1. volume 011 : 500,000 reads CENTER_NAME=BCM, TRACE_TYPE_CODE=WGS
- 2.
- 3. 249,611 TRACE_END=F & 250,389 TRACE_END=R
- 4. kmers: 8 8bp most frequent kmers are shared by the FORWARD & REVERSE strands ; no 24bp kmers are shared
==> 24.fwd/kmers.tab <== AGTTCGACTGCAAGTAGTTCATCA TGATGAACTACTTGCAGTCGAACT 2463 # contains AGTAGTTC GAGTTCGACTGCAAGTAGTTCATC GATGAACTACTTGCAGTCGAACTC 2189 CGAGTTCGACTGCAAGTAGTTCAT ATGAACTACTTGCAGTCGAACTCG 1996 TCGAGTTCGACTGCAAGTAGTTCA TGAACTACTTGCAGTCGAACTCGA 1593 GTTCGACTGCAAGTAGTTCATCAA TTGATGAACTACTTGCAGTCGAAC 1023 GAGTTCGACTGCAGTAGTTCATCA TGATGAACTACTGCAGTCGAACTC 812 CGAGTTCGACTGCAGTAGTTCATC GATGAACTACTGCAGTCGAACTCG 777 GTTCGACTGCAAGTAGTTCATCAT ATGATGAACTACTTGCAGTCGAAC 769 TCGAGTTCGACTGCAGTAGTTCAT ATGAACTACTGCAGTCGAACTCGA 637 ATCGAGTTCGACTGCAAGTAGTTC GAACTACTTGCAGTCGAACTCGAT 594 ==> 08.fwd/kmers.tab <== AGTAGTTC GAACTACT 86477 CAGTAGTT AACTACTG 67681 AGTTCTCA TGAGAACT 61556 TAGTTCTC GAGAACTA 60964 GTAGTTCT AGAACTAC 57866 AGTTCATC GATGAACT 49676 TAGTTCAT ATGAACTA 45298 GTTCATCA TGATGAAC 42117 GCAGTAGT ACTACTGC 41391 GTAGTTCA TGAACTAC 40694 ==> 24.rev/kmers.tab <== TATCGATGGTACAGTAGTTCATCA TGATGAACTACTGTACCATCGATA 999 # contains AGTAGTTC CTATCGATGGTACAGTAGTTCATC GATGAACTACTGTACCATCGATAG 774 GCTATCGATGGTACAGTAGTTCAT ATGAACTACTGTACCATCGATAGC 600 CGCTATCGATGGTACAGTAGTTCA TGAACTACTGTACCATCGATAGCG 432 ATCGATGGTACAGTAGTTCATCAT ATGATGAACTACTGTACCATCGAT 417 ATCGATGGTACAGTAGTTCATCAA TTGATGAACTACTGTACCATCGAT 380 ATCAGATGGTACAGTAGTTCATCA TGATGAACTACTGTACCATCTGAT 373 ATCGATGGTACAGTAGTTCATCAC GTGATGAACTACTGTACCATCGAT 265 CTATCGATGGTAAGTAGTTCATCA TGATGAACTACTTACCATCGATAG 235 TCAGATGGTACAGTAGTTCATCAA TTGATGAACTACTGTACCATCTGA 224 ==> 08.rev/kmers.tab <== AGTTCATC GATGAACT 85127 TAGTTCAT ATGAACTA 77902 GTTCATCA TGATGAAC 75585 TAGTTCTC GAGAACTA 68057 AGTTCTCA TGAGAACT 67277 GTAGTTCT AGAACTAC 64894 GTAGTTCA TGAACTAC 62607 CGTAGTTC GAACTACG 52031 AGTAGTTC GAACTACT 51013 ACGTAGTT AACTACGT 31552
- 7. Get linker sequences
>linker.fwd 27bp TCGAGTTCGACTGCAAGTAGTTCATCA >linker.rev 27bp CTAATCAGATGGTACAGTAGTTCATCA #>linker.rev 40 bp Art's (13 more bp at 5') #TATGACCATGCGCCTAATCAGATGGTACAGTAGTTCATCA
#GCTATCGATGGTACAGTAGTTCATCAT is the most frequent rev seq 27 kmers but not the linker (few snp differences)
- 8 & 9 Align reads to linkers using nucmer
Fwd:
nucmer -l 12 -c 24 -r linker.fwd.seq ../bos_taurus.$v.r.fasta
# nucmer -l 12 -c 24 -r kmers.seq ../bos_taurus.$v.r.fasta
show-coords out.delta | awk '{print $19,$5,$13}' > ! out.clr
extractfromfastanames.pl -clr -f out.clr < ../bos_taurus.$v.r.fasta >! out.seq
Rev:
nucmer -l 12 -c 24 -r linker.rev.seq ../bos_taurus.$v.f.fasta
# nucmer -l 12 -c 24 -r kmers.seq ../bos_taurus.$v.f.fasta
show-coords out.delta | awk '{print $19,$5,$13}' > ! out.clr
extractfromfastanames.pl -clr -f out.clr < ../bos_taurus.$v.f.fasta >! out.seq
Both:
clrFasta out.seq >! out.cseq fasta2tab.pl out.cseq | sort -k2 > ! out.tab nucmer -c 40 out.cseq ~/db/UniVec -p vector delta-filter -q vector.delta >! vector.filter-q.delta show-coords vector.filter-q.delta | sort -n | head
cat vector.filter-q.delta | grep "^>" | count.pl -c 1 -m 2
- 10. Extract "vector reads"
>399553028 # 24.fwd TGATGAACTACTGTACCATCTGATTAGGCGCATGGTCATAGCTGTTTCCTGTGTGAAATT GCTATCCGCTCACAATTCCACACAACATACGAGCCGGAAGCATAAAGTGTAAAGCCTGGG GTGTCAAATGAGAGACCTAACTCACATTCAACTTTTTTTTTTTTTCTGCCCTCTATTCTA ... >400269118 #24.rev TGATGAACTACTTGCAGTCGAAATCGAATCATCACTGGCCGTCCTTTTACAACGTCGTGA CTGGGAAAACCCTGGCGTTACCCAACTTAATCCGCCTTGCAGCACATCCCCCTTTCCCCC AGCTGGCGTAAAAACGTAAAAAGCCCCGCACCGATCGCCCTTTCCCAACAGGTTGCCCAG
- 11. Align "vector reads" to UniVec; identify vector
show-coords 24.fwd/400269118-UniVec.delta 24.rev/399553028-UniVec.delta | grep J01636.1
31 148 | 1175 1292 | 118 118 | 95.76 | 1276 7477 | 9.25 1.58 | 399553028.rev gnl|uv|J01636.1:1-7477
32 199 | 1302 1463 | 168 162 | 90.48 | 653 7477 | 25.73 2.17 | 400269118 gnl|uv|J01636.1:1-7477
- 12. 10bp distance between the 2 alignments
- 13. Lucy files
$ more vector.seq >J01636 E.coli lactose operon with lacI, lacZ, lacY and lacA genes GACACCATCGAATGGCGCAAAACCTTTCGCGGTATGGCATGATAGCGCCCGGAAGAGAGTCAATTCAGGG TGGTGAATGTGAAACCAGTAACGTTATACGATGTCGCAGAGTATGCCGGTGTCTCTTATCAGACCGTTTC CCGCGTGGTGAACCAGGCCAGCCACGTTTCTGCGAAAACGCGGGAAAAAGTGGAAGCGGCGATGGCGGAG CTGAATTACATTCCCAACCGCGTGGCACAACAACTGGCGGGCAAACAGTCGTTGCTGATTGGCGTTGCCA ... $ more splice.seq >J01636.for.begin vector+linker.rev TGAATGTGAGTTAGGTCTCTCATTTGACACCCCAGGCTTTACACTTTATGCTTCCGGCTC GTATGTTGTGTGGAATTGTGAGCGGATAGCAATTTCACACAGGAAACAGCTATGACCATG CGCCTAATCAGATGGTACAGTAGTTCATCA >J01636.for.end rev(linker.fwd)+vector TGATGAACTACTTGCAGTCGAAATCGAATCATCACTGGCCGTCCTTTTACAACGTCGTGA CTGGGAAAACCCTGGCGTTACCCAACTTAATCCGCCTTGCAGCACATCCCCCTTTCCCCC AGCTGGCGTAAAAACGTAAAAAGCCCCGCA >J01636.rev.begin (revcomp of J01636.for.end) TGCGGGGCTTTTTACGTTTTTACGCCAGCTGGGGGAAAGGGGGATGTGCTGCAAGGCGGA TTAAGTTGGGTAACGCCAGGGTTTTCCCAGTCACGACGTTGTAAAAGGACGGCCAGTGAT GATTCGATTTCGACTGCAAGTAGTTCATCA >J01636.rev.end (revcomp of J01636.for.begin) TGATGAACTACTGTACCATCTGATTAGGCGCATGGTCATAGCTGTTTCCTGTGTGAAATT GCTATCCGCTCACAATTCCACACAACATACGAGCCGGAAGCATAAAGTGTAAAGCCTGGG GTGTCAAATGAGAGACCTAACTCACATTCA
# splice=linker+vector
3 120 | 1175 1292 | 118 118 | 95.76 | 150 7477 | 78.67 1.58 | J01636.for.begin J01636
32 131 | 1302 1399 | 100 98 | 96.00 | 150 7477 | 66.67 1.31 | J01636.for.end J01636
- 13.1 Align vector & splice to Ecoli
1 7474 | 366812 359335 | 7474 7478 | 99.91 | 7477 4639675 | 99.96 0.16 | J01636 NC_000913.2 [CONTAINED]
20 119 | 65 162 | 100 98 | 96.00 | 150 395 | 66.67 24.81 | J01636.rev.begin NC_000913.2
31 148 | 172 289 | 118 118 | 95.76 | 150 395 | 78.67 29.87 | J01636.rev.end NC_000913.2
1069 1463 | 395 1 | 395 395 | 100.00 | 7477 395 | 5.28 100.00 | J01636 NC_000913.2.365350-365744
- 14. Run lucy & trim reads
$ /nfshomes/dpuiu/szdevel/SourceForge/lucy-1.19p/lucy \
-v vector.seq splice.seq
-o bos_taurus.lucy.seq bos_taurus.lucy.qual \
-debug bos_taurus.lucy.info \
bos_taurus.seq bos_taurus.qual
# Trim clr $ clrFasta bos_taurus.seq > bos_taurus.cseq
- 15. Align lucy output to linker, vector, splice & UniVec.dust
$ nucmer -l 12 -c 24 ~/db/vector.seq bos_taurus.lucy.cseq -p vector-bos_taurus.lucy $ nucmer -l 16 -c 30 ~/db/vector.seq bos_taurus.lucy.cseq -p vector-bos_taurus.lucy $ nucmer -l 16 -c 30 ~/db/splice.seq bos_taurus.lucy.cseq -p splice-bos_taurus.lucy $ nucmer -l 16 -c 30 ~/db/UniVec.dust bos_taurus.lucy.cseq -p UniVec.dust-bos_taurus.lucy
- 16. Align input to linker, vector, splice & UniVec.dust
$ nucmer -l 12 -c 24 ~/db/linker.seq bos_taurus.seq -p linker-bos_taurus $ nucmer -l 16 -c 30 ~/db/vector.seq bos_taurus.seq -p vector-bos_taurus $ nucmer -l 16 -c 30 ~/db/splice.seq bos_taurus.seq -p splice-bos_taurus $ nucmer -l 16 -c 30 ~/db/UniVec.dust bos_taurus.seq -p UniVec.dust-bos_taurus
Count how many reads got trimmed
infoseq *seq | getSummary.pl -c 1 -t original.LEN
cat bos_taurus.lucy.info | awk '{print $4-$3}' | getSummary.pl -t lucy.CLR >! bos_taurus.lucy.summary
cat bos_taurus.lucy.info | getSummary.pl -c 14 -t lucy.CLV5 -nh >> bos_taurus.lucy.summary
cat bos_taurus.lucy.info | getSummary.pl -c 15 -t lucy.CLV3 -nh >> bos_taurus.lucy.summary
Libraries
011.BCM.WGS FORWARD
- vector: J01636
- UniVec: gnl|uv|J01636.1:1-7477 E.coli lactose operon with lacI, lacZ, lacY and lacA genes
ll ~dpuiu/db/J01636* -rw-rw-r-- 1 dpuiu dpuiu 7651 Jan 9 15:56 /nfshomes/dpuiu/db/J01636 -rw-rw-r-- 1 dpuiu dpuiu 105 Jan 14 07:17 /nfshomes/dpuiu/db/J01636linker -rw-rw-r-- 1 dpuiu dpuiu 840 Jan 13 13:43 /nfshomes/dpuiu/db/J01636splice
cat ~dpuiu/db/J01636* | infoseq J01636 7477 53.43 J01636.linker.fwd 27 44.44 J01636.linker.rev 27 37.04 J01636.for.begin 150 44.67 J01636.for.end 150 51.33 J01636.rev.begin 150 51.33 J01636.rev.end 150 44.67
- 249,611 reads:
- 91% got vector trimmed at the 5'
- 0.4% (1149) got vector trimmed at the 3'
#elem #0s min max mean median n50 sum original.LEN 249611 0 437 2349 1082 991 1009 270035781 lucy.CLV5 249611 21215 0 741 25.03 25 27 6247415 lucy.CLV3 249611 248462 0 1047 3.49 0 859 870344
- Original reads hit counts:
10975 linker.fwd 133 linker.rev 166 splice 152 vector 228 UniVec.dust
- Lucy trimmed read counts
2 linker.fwd 0 linker.rev 1 splice 1 vector 6 UniVec.dust (only 3 are >40bp)
011.BCM.WGS REVERSE
#elem #0s min max mean median n50 sum original.LEN 250389 0 502 2148 1085 993 1012 271691094 lucy.CLR 250389 7345 0 1281 795 876 892 198982171 lucy.CLV5 250389 20271 0 668 26.52 27 29 6641362 lucy.CLV3 250389 249269 0 997 3.35 0 861 839029
- Original reads hit counts:
linker.fwd 113 linker.rev 3812 splice 143 UniVec.dust 237 vector 4318
- Lucy trimmed reads hit counts:
linker.fwd 1 linker.rev 0 splice 1 UniVec.dust 10 vector 1
030.BCM.SHOTGUN
- same linker/vector/splice as BCM.WGS
- 2.5% (4K out of 160K) reads contain linker & vector at 3'
#elem #0s min max mean median n50 sum original.LEN 8411 0 325 1685 1181 1240 1314 9933150 lucy.CLR 8411 8 0 1054 841 863 874 7070994 lucy.CLV5 8411 568 0 232 27.01 28 29 227206 lucy.CLV3 8411 2325 0 1040 597 794 851 5023445
- Original reads hit counts:
linker.fwd 4314 linker.rev 4125 splice 7816 UniVec.dust 4212 vector 6750 vector 27235
- Lucy trimmed reads hit counts:
linker.fwd 3 linker.rev 1 splice 1 UniVec.dust 13 vector 0
001.NISC.SHOTGUN
- Vector: pOTW13
- UniVec: 3 partial seqs
gnl|uv|NGB00080.1:1-198 pOTW13 with linkers gnl|uv|NGB00080.1:718-888 pOTW13 with linkers gnl|uv|NGB00080.1:1490-1654-49 pOTW13 with linkers
ll /nfshomes/dpuiu/db/NGB00080* -rw-rw-r-- 1 dpuiu dpuiu 1083 Jan 14 20:43 /nfshomes/dpuiu/db/NGB00080 -rw-r--r-- 1 dpuiu dpuiu 94 Jan 14 21:01 /nfshomes/dpuiu/db/NGB00080linker -rw-r--r-- 1 dpuiu dpuiu 2183 Jan 14 20:44 /nfshomes/dpuiu/db/NGB00080splice
cat /nfshomes/dpuiu/db/NGB00080* | infoseq NGB00080 1054 50.00 NGB00080.linker.fwd 24 45.83 NGB00080.linker.rev 26 53.85 NGB00080.for.beg 518 46.14 NGB00080.for.end 518 50.48 NGB00080.rev.begin 518 50.48 NGB00080.rev.beg 518 46.14
- 944 read sample
#elem #0s min max mean median n50 sum original.LEN 944 0 652 1017 735 721 722 693668 lucy.CLR 944 39 0 886 415 422 522 391333 lucy.CLV5 944 121 0 275 34.05 33 35 32143 lucy.CLV3 944 18 0 885 410 409 511 387007
- Original reads hit counts:
linker.fwd 479 linker.rev 492 splice 910 UniVec.dust 0 vector 939
- Lucy trimmed reads hit counts:
linker.fwd 1 linker.rev 0 splice 0 UniVec.dust 9 vector 1
060.BCCAGSC.CLONEEND
- Linkers:
linker.fwd CCCTGCTTTGTCTGGAAGGGGTTCCCGACCT linker.rev CAGGAGGGGAGAAAGGGCTCAGAGG
- No common vector !!!
wc -l *clb
60746 bos_taurus.060.f.clb #18 reads original align to UniVec (nucmer default params)
60836 bos_taurus.060.r.clb
Fwd:
329 428 | 440 535 | 100 96 | 91.00 | 503 1585 | 19.88 6.06 | 723951410 gnl|uv|U30497.1:3230-4814 Cloning vector pAS2-1
330 370 | 89 49 | 41 41 | 100.00 | 503 143 | 8.15 28.67 | 723951410 gnl|uv|U67875.1:6541-6683 pESP-I yeast expression vector
330 370 | 94 54 | 41 41 | 100.00 | 503 143 | 8.15 28.67 | 723951410 gnl|uv|U67875.1:6541-6683 pESP-I yeast expression vector
Rev:
1 96 | 71 165 | 96 95 | 93.81 | 203 165 | 47.29 57.58 | 724018013 gnl|uv|AF133437.1:16659-16823 Cloning vector pCYPAC6
50 143 | 1 94 | 94 94 | 92.71 | 203 94 | 46.31 100.00 | 724018013 gnl|uv|U80929.2:2858-2951 Cloning vector pBACe3.6
017.UIUC.CLONEEND
- No overrepresented kmers
wc -l *clb 17978 bos_taurus.017.f.clb 17911 bos_taurus.017.r.clb ==> 24.fwd/kmers.tab <== CCCTGCTTTGTCTGGAAGGGGTTC GAACCCCTTCCAGACAAAGCAGGG 9 CTGCTTTGTCTGGAAGGGGTTCCC GGGAACCCCTTCCAGACAAAGCAG 9 ==> 24.rev/kmers.tab <== GAATGTTGAGCTTTAGCCAACTTT AAAGTTGGCTAAAGCTCAACATTC 4 TCTGAATGTTGAGCTTTAGCCAAC GTTGGCTAAAGCTCAACATTCAGA 4 ==> 8.fwd/kmers.tab <== TTTTTTTT AAAAAAAA 55 AAGGGGTT AACCCCTT 35 ==> 8.rev/kmers.tab <== GTCTGGAA TTCCAGAC 41 TCTGGAAG CTTCCAGA 39
- No UniVec hits
010.TIGR.CLONEEND
- No overrepresented kmers
wc -l *clb 5479 bos_taurus.032.f.clb 5174 bos_taurus.032.r.clb ==> 24.fwd/kmers.tab <== CTTGTGTTGGCCCAGGCAAGTCCA TGGACTTGCCTGGGCCAACACAAG 30 TTGTGTTGGCCCAGGCAAGTCCAA TTGGACTTGCCTGGGCCAACACAA 30 ==> 24.rev/kmers.tab <== CTGCCTCTTGTGTTGGCCCAGGCA TGCCTGGGCCAACACAAGAGGCAG 16 GCTGCCTCTTGTGTTGGCCCAGGC GCCTGGGCCAACACAAGAGGCAGC 15 ==> 8.fwd/kmers.tab <== GAGTGGGT ACCCACTC 176 GGAGTGGG CCCACTCC 171 ==> 8.rev/kmers.tab <== TGGAGTGG CCACTCCA 182 GGAGTGGG CCCACTCC 181
- No UniVec hits
...
070.BCM.CLONEEND
- No frequent kmers
wc -l *clb 6027 bos_taurus.070.f.clb 6236 bos_taurus.070.r.clb ==> 24.fwd/kmers.tab <== GGACTCTCAGAGTCTTCTCCAACA TGTTGGAGAAGACTCTGAGAGTCC 18 ACTGGTTGGATCTCCTTGCAGTCC GGACTGCAAGGAGATCCAACCAGT 18 ==> 24.rev/kmers.tab <== ATAAAATCTGAGCCACCAGGGAAG CTTCCCTGGTGGCTCAGATTTTAT 1 CTATTGGTTCATATGGTCAACGTC GACGTTGACCATATGAACCAATAG 1 ==> 8.fwd/kmers.tab <== TTTTTTTT AAAAAAAA 86 CTTCTCCA TGGAGAAG 75 ==> 8.rev/kmers.tab <== TATAGTGT ACACTATA 9 ATATAGGG CCCTATAT 8
- No alignments to BCM WGS vector
Running Lucy
- Default parameters with vector trimming
- BCM vector/splice
/nfshomes/dpuiu/db/vector.BCM.seq /nfshomes/dpuiu/db/splice.BCM.seq
- NISC vector/splice
/nfshomes/dpuiu/db/vector.NISC.seq /nfshomes/dpuiu/db/splice.NISC.seq
BCM.WGS (all reads)
- orig.CLR < lucy.CLR ( 765 < 792 )
- orig.CLV > lucy.CLV ( 1015 > 973 )
- 739,529 out of 24,863,599 reads (3%) deleted by Lucy (CLR=-1,-1)
- 21,728,592 out of 24,863,599 reads (87%) vector trimmed at the 5' end
- 92,646 out of 24,863,599 reads (0.3%) vector trimmed at the 3' end
elem <0 0 >0 min max mean median n50 sum orig.LEN 24863599 0 0 24863599 5 3097 1002 997 1015 24915462033 orig.CLR 24863599 463669 7 24399923 -1143 1833 765 836 864 19036744256 orig.CLR5 24863599 0 359245 24504354 0 2103 42 22 58 1047922451 orig.CLR3 24863599 463404 0 24400195 -1 2169 807 872 895 20084666707 lucy.CLR 24863599 0 739529 24124070 0 1219 792 878 904 19695000417 lucy.CLR5 24863599 739529 36108 24087962 -1 1753 43 29 42 1086413880 lucy.CLR3 24863599 739529 0 24124070 -1 1894 835 915 939 20781414297 orig.CLR5-lucy.CLR5 24863599 16299521 215345 8348733 -1186 2104 -1 -10 -1186 -38491429 orig.CLR3-lucy.CLR3 24863599 14858542 1494794 8510263 -1273 2170 -28 -20 -1273 -696747590 orig.CLV 24863599 1053 1920 24860626 -2 5345 1015 1002 1017 25260581538 orig.CLV5 8841849 0 0 8841849 1 1219 33 46 49 295011460 orig.CLV3 24861698 1053 0 24860645 -1 5346 1027 1005 1019 25555592998 lucy.CLV 24863599 10694 707 24852198 -469 3096 973 968 987 24195085877 lucy.CLV5 24863599 0 3135007 21728592 0 1359 25 27 29 623457486 lucy.CLV3 24863599 0 0 24863599 4 3096 998 995 1014 24818543363 lucy.CLVABS5 24863599 0 3135007 21728592 0 1359 25 27 29 623457486 lucy.CLVABS3 24863599 0 24770953 92646 0 1343 2 0 880 72055071 orig.CLV5-lucy.CLV5 24863599 17216820 1512453 6134326 -1312 1219 -13 -25 -1312 -328446026 orig.CLV3-lucy.CLV3 24863599 1519132 18579609 4764858 -1832 4672 29 0 479 737049635
BCM.WGS (0 quality reads)
- orig.CLR > lucy.CLR (mean)
- orig.CLV > lucy.CLV (mean)
- 7,153 out of 551,114 reads (1.3%) deleted by Lucy (CLR=-1,-1)
- 508,166 out of 551,114 reads (92%) vector trimmed at the 5' end
- 1,946 out of 551,114 reads (0.35%) vector trimmed at the 3' end
elem <0 0 >0 min max mean median n50 sum orig.LEN 551114 0 0 551114 5 1464 872 946 959 480705828 orig.CLR 551114 7754 0 543360 -770 1175 708 786 807 390325117 orig.CLR5 551114 0 6773 544341 0 1519 44 20 111 24582849 orig.CLR3 551114 7744 0 543370 -1 1638 752 818 833 414907966 lucy.CLR 551114 0 7153 543961 0 699 636 671 671 350759771 lucy.CLR5 551114 7153 35872 508089 -1 201 26 27 28 14442310 lucy.CLR3 551114 7153 0 543961 -1 699 662 699 699 365202081 orig.CLR5-lucy.CLR5 551114 364282 8801 178031 -198 1500 18 -8 215 10140539 orig.CLR3-lucy.CLR3 551114 85058 2962 463094 -700 1472 90 123 178 49705885 orig.CLV 551114 971 0 550143 -2 2037 974 978 981 537127121 orig.CLV5 5100 0 0 5100 1 845 35 29 31 180490 orig.CLV3 551114 971 0 550143 -1 2037 974 978 981 537307611 lucy.CLV 551114 58 6 551050 -84 1456 841 917 930 463903233 lucy.CLV5 551114 0 42948 508166 0 202 27 28 29 14964546 lucy.CLV3 551114 0 0 551114 4 1463 868 945 958 478867779 lucy.CLVABS5 551114 0 42948 508166 0 202 27 28 29 14964546 lucy.CLVABS3 551114 0 549168 1946 0 700 2 0 686 1286935 orig.CLV5-lucy.CLV5 551114 506108 42215 2791 -202 845 -26 -28 -202 -14784056 orig.CLV3-lucy.CLV3 551114 134959 23422 392733 -967 1614 106 7 459 58439832
BCM.SHOTGUN
- orig.CLR < lucy.CLR (mean)
- orig.CLV > lucy.CLV (mean)
- 98,070 out of 10,748,529 reads (0.9%) deleted by Lucy (CLR=-1,-1)
- 9,737,008 out of 10,748,529 reads (90%) vector trimmed at the 5' end
- 294,942 out of 10,748,529 reads (2.7%) vector trimmed at the 3' end
elem <0 0 >0 min max mean median n50 sum orig.LEN 10748529 0 0 10748529 5 2043 975 950 964 10486690472 orig.CLR 10748529 17308 2 10731219 -1293 1467 809 833 847 8701344571 orig.CLR5 10748529 0 68 10748461 0 1315 26 16 38 288662580 orig.CLR3 10748529 16780 0 10731749 -1 1647 836 851 863 8990007151 lucy.CLR 10748529 0 98070 10650459 0 1337 833 854 868 8955866769 lucy.CLR5 10748529 98070 1973 10648486 -1 1307 35 28 32 376276188 lucy.CLR3 10748529 98070 0 10650459 -1 1553 868 882 896 9332142957 orig.CLR5-lucy.CLR5 10748529 9498290 65171 1185068 -1099 1293 -8 -11 -1099 -87613608 orig.CLR3-lucy.CLR3 10748529 6879532 671097 3197900 -1149 1437 -31 -26 -1149 -342135806 orig.CLV 10748529 16779 412 10731338 -2 3919 974 948 964 10472347908 orig.CLV5 8594910 0 0 8594910 1 1239 3 1 49 28350257 orig.CLV3 10748349 16779 0 10731570 -1 3919 976 950 965 10500698165 lucy.CLV 10748529 7026 614 10740889 -268 2042 930 924 940 9997862132 lucy.CLV5 10748529 0 1011521 9737008 0 855 24 24 27 257993796 lucy.CLV3 10748529 0 0 10748529 4 2042 954 945 962 10255855928 lucy.CLVABS5 10748529 0 1011521 9737008 0 855 24 24 27 257993796 lucy.CLVABS3 10748529 0 10453587 294942 0 1214 20 0 847 220086015 orig.CLV5-lucy.CLV5 10748529 9538738 138680 1071111 -854 1239 -21 -23 -854 -229643539 orig.CLV3-lucy.CLV3 10748529 357934 9324166 1066429 -1328 2846 22 0 704 244842237
NISC.SHOTGUN
- orig.CLR < lucy.CLR (mean)
- orig.CLV > lucy.CLV (mean)
- 8,248 out of 737,900 reads (1.1%) deleted by Lucy (CLR=-1,-1)
- 633,409 out of 737,900 reads (85%) vector trimmed at the 5' end
- 7,201 out of 737,900 reads (0.97%) vector trimmed at the 3' end
elem <0 0 >0 min max mean median n50 sum orig.LEN 737900 0 0 737900 104 2104 784 729 734 579172842 orig.CLR 737900 5988 2 731910 -636 1033 651 668 676 480400909 orig.CLR5 737900 0 0 737900 1 1407 47 40 51 34857531 orig.CLR3 737900 0 5879 732021 0 1470 698 710 715 515258440 lucy.CLR 737900 0 8248 729652 0 1035 658 670 676 485757685 lucy.CLR5 737900 8248 56 729596 -1 1091 45 35 46 33811606 lucy.CLR3 737900 8248 0 729652 -1 1391 704 710 714 519569291 orig.CLR5-lucy.CLR5 737900 253727 89345 394828 -566 1408 1 1 485 1045925 orig.CLR3-lucy.CLR3 737900 177007 31 560862 -867 1471 -5 1 -867 -4310851 orig.CLV 737900 3224 2655 732021 -636 2103 771 725 730 569178445 orig.CLV5 734026 0 0 734026 1 987 5 1 35 4375315 orig.CLV3 732021 0 0 732021 35 2104 783 729 734 573553760 lucy.CLV 737900 1335 55 736510 -200 2104 747 696 702 551392388 lucy.CLV5 737900 104491 0 633409 -1 1199 30 31 34 22784742 lucy.CLV3 737900 0 0 737900 15 2103 778 728 733 574177130 lucy.CLVABS5 737900 0 104491 633409 0 1200 31 32 35 23522642 lucy.CLVABS3 737900 0 730699 7201 0 1076 5 0 686 4257812 orig.CLV5-lucy.CLV5 737900 561851 66390 109659 -1198 983 -24 -29 -1198 -18409427 orig.CLV3-lucy.CLV3 737900 8386 1 729513 -950 1077 0 1 -950 -623370
Fragment files
- Locations:
/fs/szasmg3/bos_taurus/data/frg /fs/szasmg3/bos_taurus/data/frg.new
- All DST messages are unique
- bos_taurus.clv : contains the vector clipping points
- BCM.WGS, BCM.SHOTGUN & NISC.SHOTGUN: lucy.clv
- others: the TA clv
- 374,454 reads don't have valid clv's
- 36,446,031 reads have valid clv's with avg=955
Message counts (original)
DST FRG LKG bos_taurus.BCM.WGS.frg 79 24124070 11311841 #bos_taurus.BCM.SHOTGUN.frg 7339 10650459 1799069 # some libs & mates are missing due to a tarchive2ca crash (used by UMD2.1) #bos_taurus.BCM.SHOTGUN.new.frg 18208 10650459 4715172 # split the libraries by VOL & SEQ_LIB_ID (used by UMD2.2) #bos_taurus.BCM.SHOTGUN.new.frg 13826 10650459 5046435 # double check the FRG count !!! (used by UMD2.3) bos_taurus.BCM.SHOTGUN.new.frg 7 10650459 5046435 # UMD2.4 bos_taurus.NISC.SHOTGUN.frg 246 729652 344932 bos_taurus.BCCAGSC.CLONEEND.frg 1 125241 59505 bos_taurus.UIUC.CLONEEND.frg 2 114750 46319 bos_taurus.TIGR.CLONEEND.frg 1 65171 27067 bos_taurus.GSC.CLONEEND.frg 1 53521 25889 bos_taurus.CENARGEN.WGS.frg 0 26246 0 #bos_taurus.BARC.CLONEEND.frg 11150 25454 11150 # (used by UMD2.3) bos_taurus.BARC.CLONEEND.frg 1 25454 11150 # (used by UMD2.4) bos_taurus.BCM.CLONEEND.frg 1 16875 7103 bos_taurus.CENARGEN.CLONEEND.frg 1 16787 6269 bos_taurus.UOKNOR.SHOTGUN.frg 1 14651 4910 bos_taurus.TIGR_JCVIJTC.CLONEEND.frg 2 10651 4803 bos_taurus.UOKNOR.FINISHING.frg 0 151 0 bos_taurus.WUGSC.COLONEEND.frg 1 49 21 #total 25312 35973728 16896244 # (UMD2.3) total 344 35973728 16896244 # (UMD2.4)
Message counts (quality)
DST FRG LKG bos_taurus.BCM.WGS.qual.count 79 23580109 11035582 #bos_taurus.BCM.SHOTGUN.qual.count 7339 10644092 1799069 bos_taurus.BCM.SHOTGUN.qual.new.count 18208 10644092 4712446 bos_taurus.NISC.SHOTGUN.count 246 729652 344932 bos_taurus.BCCAGSC.CLONEEND.qual.count 1 116484 53585 bos_taurus.UIUC.CLONEEND.count 2 114750 46319 bos_taurus.TIGR.CLONEEND.count 1 65171 27067 bos_taurus.CENARGEN.WGS.count 0 26246 0 bos_taurus.BARC.CLONEEND.count 11150 25454 11150 bos_taurus.BCM.CLONEEND.count 1 16875 7103 bos_taurus.CENARGEN.CLONEEND.count 1 16787 6269 bos_taurus.TIGR_JCVIJTC.CLONEEND.count 2 10651 4803 bos_taurus.UOKNOR.SHOTGUN.qual.count 1 2456 813 bos_taurus.WUGSC.COLONEEND.count 1 49 21
Message counts (0quality)
DST FRG LKG bos_taurus.BCM.WGS.0qual.count 79 543961 234397 bos_taurus.GSC.CLONEEND.0qual.count 1 53521 25889 bos_taurus.UOKNOR.SHOTGUN.0qual.count 1 12195 4097 bos_taurus.BCCAGSC.CLONEEND.0qual.count 1 8757 2114 bos_taurus.BCM.SHOTGUN.0qual.count 7339 6367 0 bos_taurus.UOKNOR.FINISHING.0qual.count 0 151 0
Assemblies
Bt.qc.combine UMD2.0 ... UMD2.5 combine stats
UMD2.1(2009_0122_CA; Quality reads)
Issues
- Uses only quality reads
- BCM.SHOTGUN library : ~ 4715172-1799069=2.9M mates were missed due to a tarchive2ca crash ; some libraries got merged (were assigned the same lib_id)
- All reads except for BCM.WGS were set as nonrandom
- Update the runCA script to run overlapper concurently; new "ovlConcurrency" parameter added to the .spec file !!!
- consensus after cgw crashed in MultiAlignContig() ... use "consensus -D forceunitigabut" !!!
- cgw crashed after updating gkpStore with new lib/mate info => edit Input_CGW.c, remove the assert in line 117
Info
host: walnut assembly version: wgs-5.2 stable dir: /scratch1/bos_taurus/Assembly/2009_0122_CA command: /fs/szdevel/dpuiu/SourceForge/wgs/Linux-amd64/bin/runCA-test -d . -p bt -s bt01.specFile *.frg spec file: cgwDistanceSampleSize = 1000 # ??? too big; more than 50% of the BCM.SHOTGUN reads are in libraries with less than 1000 inserts cnsConcurrency = 15 cnsMinFrags = 200000 doOverlapTrimming = 1 frgCorrBatchSize = 100000 frgCorrConcurrency = 15 merylMemory = 24000 merylThreads = 15 obtMerThreshold = 200 obtOverlapper = ovl ovlConcurrency = 8 ovlCorrBatchSize = 100000 ovlCorrConcurrency = 15 ovlHashBlockSize = 1200000 ovlMemory = 8GB --hashload 0.8 --hashstrings 400000 ovlMerThreshold = 500 ovlOverlapper = ovl ovlRefBlockSize = 7200000 ovlThreads = 2 unitigger = utg utgErrorRate = 0.015 vectorIntersect = bos_taurus.clv doExtendClearRanges = 2
Steps
1. Run up till after initialStoreBuilding
runCA stopAfter=initialStoreBuilding ...
2. Update gkpStore with nonrandom frg flag
cat bos_taurus.nonrandom.clv | perl -ane 'print "frg uid $F[0] isnonrandom 1\n";' > bos_taurus.nonrandom.edit gatekeeper -edit bos_taurus.nonrandom.edit bt.gkpStore
Input
gatekeeper -dumpinfo -lastfragiid bt.gkpStore ... Last frag in store is iid = 35348776
OBT
elem <0 0 >0 min max mean median n50 sum CLV5 35085508 0 3387027 31698481 0 970 25 27 29 891007232 CLV3 35164784 0 0 35164784 15 2974 984 980 1000 34612019144 CLR_ORIG5 35348776 0 43354 35305422 0 1753 42 29 38 1502168205 CLR_ORIG3 35348776 0 0 35348776 70 1894 864 905 927 30547294868 CLR_OBT5 35348776 0 26513 35322263 0 1690 49 30 73 1756346429 CLR_OBT3 35348776 0 23477 35325299 0 1813 843 895 914 29824543869
- 421,379 reads deleted by OBT: why so many???
- Chimera:
20297 reads too short => deleted
- more 0-overlaptrim/bt.mergeLog.stats
... 211037: short or inconsistent 253536: deleted fragment due to zero clear
- Example:
gatekeeper -dumpfragments 516316990 bt.gkpStore fragmentIdent = 516316990,14 fragmentMate = 0,0 fragmentLibrary = 27473,1563 fragmentIsDeleted = 1 fragmentIsNonRandom = 1 fragmentStatus = G fragmentOrientation = I fragmentHasVectorClear = 0 fragmentHasQualityClear = 0 fragmentPlate = 0 fragmentPlateLocation = 0 fragmentSeqLen = 862 fragmentHPSLen = 0 fragmentSrcLen = 17 fragmentClearORIG = 38,553 fragmentClearQLT = 1,0 fragmentClearVEC = 1,0 fragmentClearOBTINI = 35,578 fragmentClearOBT = 35,578 fragmentClearUTG = 35,578 fragmentClearECR1 = 35,578 fragmentClearECR2 = 35,578 fragmentSeqOffset = 5376 fragmentQltOffset = 11038 fragmentHpsOffset = 53 fragmentSrcOffset = 287
cat 0-overlaptrim/bt.mergeLog | grep 516316990 516316990,14 412 412 0 0 (deleted, too short)
zcat *r000*gz | convertOverlap -a -obt
...
14 12128740 f 377 478 292 393 2.97
14 15226267 f 397 446 31 80 2.04
14 19071241 f 4 513 199 708 1.18
14 20073917 f 7 478 36 508 4.88
14 20042424 f 4 419 299 714 1.93
14 20212935 f 7 478 234 706 4.88
14 20073828 r 7 478 507 35 4.67
14 20212846 r 7 478 557 85 4.67
14 27089060 r 491 534 836 793 2.33
14 29061748 f 489 540 86 137 1.96
14 32105697 f 455 543 381 469 2.27
14 32187461 f 430 534 105 209 1.92
14 32027289 f 4 419 493 907 4.59
...
#read aligns to contigs
show-coords 516316990-ctg.filter-r.strict.delta
35 531 | 97 594 | 497 498 | 99.20 | 862 2759 | 57.66 18.05 | 516316990 ctg7180001872751
45 678 | 931 1564 | 634 634 | 97.00 | 862 1567 | 73.55 40.46 | 516316990 ctg7180001837311
- OBT deleted reads:
BCM WGS 253816 BCM SHOTGUN 151770 BCCAGSC CLONEEND 7510 NISC SHOTGUN 4757 TIGR CLONEEND 1577 CENARGEN WGS 599 CENARGEN CLONEEND 431 TIGR_JCVIJTC CLONEEND 377 UIUC CLONEEND 182 BCM CLONEEND 150 BARC CLONEEND 125 UOKNOR SHOTGUN 85 total . 421379
OBT deleted reads:
elem >0 min max mean med n50 sum len 421379 421379 98 2974 862 927 968 363280405 avgQual 421379 421379 1 57 28 24 36 11852865
Overlapper
- 98.33% of the reads (34,761,786 out of 35,348,776 reads) had overlaps
- 1.66% of the reads had no overlaps
- 6.68% of the BCCAGSC.CLONEEND reads had no overlaps
- 4.95% of the TIGR_JCVIJTC.CLONEEND reads had no overlaps
- 3.48% of the TIGR.CLONEEND reads had no overlaps
- the median number of overlaps is 20
Overlaps
reads min max mean median n50 sum
qual 35348776 0 5592 106 20 769 3777789082
- the median number of overlaps for the BCM.WGS reads is 16
- the median number of overlaps for the BCM.SHOTGUN reads is 16 !!!
- the median number of overlaps for the NISC.SHOTGUN reads is 40 !!!
- the median number of overlaps for the BCM.CLONEEND reads is 16 !!!
Media:Bt.ovlStore.big.png , Media:Bt.ovlStore.small.png
Unitigger
more 4-unitigger/bt.cga.0
UNITIG OVERLAP GRAPH INFORMATION
5208738 : Total number of unitigs
2527051 : Total number of singleton, contained unitigs
1814842 : Total number of singleton, non-contained unitigs
180910 : Total number of non-singleton, spanned unitigs
685935 : Total number of non-singleton, non-spanned unitigs
34927397 : Total number of fragments
34927397 : Total number of fragments in all unitigs
21521581 : Total number of essential fragments in all unitigs
13405816 : Total number of contained fragments in all unitigs
0.0076239952 : Randomly sampled fragment arrival rate per bp
2510896132 : The sum of overhangs in all the unitigs
6400342737 : Total number of bases in all unitigs
0 : Estimated number of base pairs in the genome.
0 : Total number of contained fragments not connected
by containment edges to essential fragments.
Total rho = 2510896132
Total nfrags = 19143061
Estimated genome length = 0
Estimated global_fragment_arrival_rate=0.007624
Computed global_fragment_arrival_rate =0.007624
Total number of randomly sampled fragments in genome = 23326293
Computed genome length = 3059589120.000000
Used global_fragment_arrival_rate=0.007624
Used global_fragment_arrival_distance=131.164826
Histogram of the number of base pairs in a chunk
100292 - 159434: 22
90010 - 99906: 25
80043 - 89676: 73
70013 - 79966: 162
60010 - 69988: 389
50008 - 59983: 977
40000 - 49998: 2434
30000 - 39997: 6458
20000 - 29999: 18957
10000 - 19999: 57442
Unitigs >=10kb
NewAsm UMd2Asm
Number 86,939 57,204
Mean 19,464 15,140
Sum 1,692.1Mb 866.0Mb
max 159,434bp 78,570bp
Contigs >=10Kb:
NewAsm UMd2Asm
n 42,343 45,958
mean 59,856 55,473
sum 2,534.5Mb 2,549.4Mb
Contigs >=100Kb:
NewAsm UMd2Asm
n 7,051 6,683
mean 163,170 162,357
sum 1,150.5Mb 1,085.0Mb
max 627,705 742,802
Scaffolds >=10Mb:
NewAsm UMd2Asm
n 30 3
mean 14.10Mb 11.36Mb
sum 422.95Mb 340.70Mb
max 26.54Mb 13.36Mb
CGW & ECR
- Checkpoints:
cat 7-0-CGW/bt.timing | grep ^Checkpoint Checkpoint 3 written during MergeScaffoldsAggressive at iteration 49 Checkpoint 4 written during MergeScaffoldsAggressive at iteration 85 Checkpoint 5 written after 1st Scaffold Merge Checkpoint 6 written after 2nd Aggressive Scaffold Merge Checkpoint 7 written after Final Rocks
cat 7-2-CGW/bt.timing | grep ^Checkpoint Checkpoint 19 written during MergeScaffoldsAggressive at iteration 12 Checkpoint 20 written during MergeScaffoldsAggressive at iteration 31 Checkpoint 21 written after 1st Scaffold Merge Checkpoint 22 written after 2nd Aggressive Scaffold Merge Checkpoint 23 written after Final Rocks cat 7-4-CGW/bt.timing | grep ^Checkpoint Checkpoint 34 written during MergeScaffoldsAggressive at iteration 12 Checkpoint 35 written during MergeScaffoldsAggressive at iteration 49 Checkpoint 36 written after 1st Scaffold Merge Checkpoint 37 written during Stones CleanupScaffolds after scaffold 32436 Checkpoint 38 written during Stones CleanupScaffolds after scaffold 34939 Checkpoint 39 written after Stone Throwing and CleanupScaffolds Checkpoint 40 written after 2nd Aggressive Scaffold Merge Checkpoint 41 written after Final Rocks
Checkpoint 42 written after Partial Stones Checkpoint 43 written after Final Contained Stones Checkpoint 44 written after resolveSurrogates
- Get early CTG/SCF stats
cat 7-CGW/bt.cgw_scaffolds | countMessages.pl ICL 451555 # ??? ICP 116455 # CTG ISF 66141 # SCF ISL 711 # SLK
- Clear read extension:
elem <0 0 >0 min max mean median n50 sum ClearORIG 35348776 4 0 35348772 -1147 1572 821 870 893 29045126663 ClearQLT 35348776 35348776 0 0 -1 -1 -1 -1 -1 -35348776 ClearVEC 35348776 299034 20323 35029419 -1 2043 952 953 975 33658445088 ClearOBTINI 35348776 0 31254 35317522 0 1364 831 879 902 29394688367 ClearOBT 35348776 0 31254 35317522 0 1318 794 854 877 28068197440
ClearECR1 35348776 0 31254 35317522 0 1329 794 854 877 28072014464 ClearECR2 35348776 0 31254 35317522 0 1329 794 854 877 28072365712
sum(ClearECR1)-sum(ClearUTG) = 3,817,024 sum(ClearECR2)-sum(ClearECR1)= 351,248
- Scaffold length stats:
cat 7-0-CGW/stat/final0.Scaffolds.nodelength.cgm | grep -v ^Sca | getSummary.pl -t 0 # 0,2,4 ...
step scaff min max mean med n50 sum 0 7048 2249 19719008 385020 21967 3114907 2713622175 2 4960 2249 21907006 540915 21181 4490171 2682939682 4 4006 2391 26541374 668427 29193 4590744 2677722052
- Last cgw
cat 7-4-CGW/stat/final0.*Scaffolds.nodelength.cgm | grep -v ^Scaff | getSummary.pl -t scf
cat 7-4-CGW/stat/final0.PlacedContig.n | grep -v ^Scaff | getSummary.pl -t scf
elem min max mean med n50 sum
scf 66141 432 26541374 42648 1347 4349378 2820819506
ctg 120461 65 627705 22421 2018 84989 2700959854
QC stats
- Bos_taurus.qc this assembly stats
- Bos_taurus.qc.combine UMD2 vs this assembly stats
TotalScaffolds=66,141 MaxBasesInScaffolds=26,048,998 MeanBasesInScaffolds=40,861 TotalContigsInScaffolds=120,461 MaxContigLength=627,911 MeanContigLength=22,436 TotalDegenContigs=269,031 MaxDegenContig=33,824 SingletonReads=3,721,123
- Posmap info
cat bt.posmap.mates | awk '{print $3}' |count.pl -p 100
good 10338164
bothChaff 1160137
oneChaff 695982
oneSurrogate 233151
bothDegen 218198
diffScaffold 150423
badShort 138464
oneDegen 118232
badLong 23196
badSame 22451
badOuttie 8751
bothSurrogate 589
total 13107738
cat bt.posmap.frags | awk '{print $4,$5}' |count.pl -p 100
placed good 20676328
placed notMated 8007072
chaff bothChaff 2320274
chaff notMated 704849
placed oneChaff 695982
chaff oneChaff 695982
placed oneSurrogate 466302
placed bothDegen 436396
placed diffScaffold 300846
placed badShort 276928
placed oneDegen 236464
placed badLong 46392
placed badSame 44902
placed badOuttie 17502
placed bothSurrogate 1178
total 34927397
Log files
- Bt.runCA.log
- Bt.runCA.hourly.runtimes approximate running times (in hours)
Analysis
Insert libraries
1. BCM.WGS : ok
- FRG.mea: 1750-7000
- ASM.mea: 1594-6727
- Most libs have > 1000 reads & get reestimated
- All libs have ASM.std< ASM.mea/3
2. BCM.SHOTGUN
- only ~ 50% of the inserts are in libs with >1000 inserts and get reestimated by the assembly
- if the thold is dropped from 1000 to 100, we'd get ~ 95% of the inserts reestimated
elem <0 0 >0 min max mean median n50 sum 0 7339 0 0 7339 1 11237 245 135 1137 1799069 100 4361 0 0 4361 100 11237 395 157 1252 1725604 1000 440 0 0 440 1008 11237 2075 1791 2323 913086
3. NISC.SHOTGUN: ok
- Most libs have > 1000 reads & get reestimated
- All libs have ASM.std< ASM.mea/3
4. BCCAGSC.CLONEEND: ok
LIB.id FRG.mea FRG.std FRG.count CENTER.TYPE ASM.mea ASM.std 125606 150000 30000 59505 BCCAGSC.CLONEEND 161998 20133
5. UIUC.CLONEEND: ok
LIB.id FRG.mea FRG.std FRG.count CENTER.TYPE ASM.mea ASM.std 114892 150000 30000 31063 UIUC.CLONEEND 175594 41208 115020 150000 30000 15256 UIUC.CLONEEND 162488 26358
6. TIGR.CLONEEND: originally wrong; gets reestimated
LIB.id FRG.mea FRG.std FRG.count CENTER.TYPE ASM.mea ASM.std 65177 2000 600 27067 TIGR.CLONEEND 161761 34938
7. GSC.CLONEEND: not used (all 53556 are 0 qual)
8. CENARGEN.WGS: "not used" (all 26246 are unmated)
9. BARC.CLONEEND: each library contains 1 template id => inserts did not get reestimated (25454 reads/11151 inserts)
10. BCM cloneend: ok
LIB.id FRG.mea FRG.std FRG.count CENTER.TYPE ASM.mea ASM.std 19070 167000 25000 7103 BCM.CLONEEND 171244 18555
11. CENARGEN.CLONEEND: large stdev
LIB.id FRG.mea FRG.std FRG.count CENTER.TYPE ASM.mea ASM.std 17249 202000 20200 6269 CENARGEN.CLONEEND 158938 55165
12. UOKNOR.SHOTGUN: ok ?
LIB.id FRG.mea FRG.std FRG.count CENTER.TYPE ASM.mea ASM.std 15158 3000 1000 4910 UOKNOR.SHOTGUN 3000 1000
13. TIGR_JCVI.CLONEEND: originally wrong; gets reestimated
LIB.id FRG.mea FRG.std FRG.count CENTER.TYPE ASM.mea ASM.std 10691 2500 750 2763 TIGR_JCVI.CLONEEND 160363 29580 10738 2500 750 2040 TIGR_JCVI.CLONEEND 161915 29343
14. UOKNOR.FINISHING: only 151 reads
15. WUGSC.CLONEEND: only 49 reads
Contigs Vs UMD2 contaminants & Ecoli
4865 contigs in list.exclude_contigs.fa 34404 exclude-ctg.qry_hits 3763 exclude-ctg.ref_hits 1204 exclude-ctg.CBE.qry_hits CONTAIN|IDENTITY|BEGIN|END 748 exclude-ctg.CBE.ref_hits CONTAIN|IDENTITY|BEGIN|END
559 Ecoli.365350-365744-ctg.qry_hits : max ctg aligned is 179K bp; 10 are > 10K bp
Contigs Vs UMD2 chromosomes
- Split 120,461 contigs into 100 files; degeneartes not split
- Align them to the 31 chromosomes 1..30,U (ref) => 101*31 jobs
#Alignment stats
cat chr*ctg*delta | grep "^>" | awk '{print $2}' | count.pl -f ../9-terminator/bt.ctg.infoseq | getSummary.pl -i 1 -z 1
ctg 0 1 >1 min max mean med n50 sum
120461 652 37540 82269 0 176 11 3 33 1422808
#Unaligned ctg lengths ctg min max mean med n50 sum 652 65 5849 1146 1134 1194 747295
- 50% of the contigs aligned uniquely
cat chr*-ctg*.delta | ~/bin/mergeDelta.pl > chr-ctg.delta
# degens?
delta-filter -q chr-ctg.delta >> chr-ctg.filter-q.delta
cat chr1-*.delta | ~/bin/delta2cvg.pl -M 0 | getSummary.pl -i 4 elem 0 >0 min max mean med n50 sum 6681 1 6680 0 12892 366 142 1095 2450106
- There are disagreements:
/fs/sz-user-supported/Linux-x86_64/bin/show-coords -l -r -H chr1-ctg.filter-q.delta | p 'print $F[-1],"\n";' | count.pl | head ctg7180001761585 24 ... ctg7180001634116 7 ...
show-coords -d chr1-ctg.filter-q.delta | grep ctg7180001761585 | p 'print " $_";' 142115744 142188863 | 383463 310345 | 73120 73119 | 99.98 | 157714772 383463 | 0.05 19.07 | 1 -1 chr1 ctg7180001761585 142188878 142286012 | 310361 213223 | 97135 97139 | 99.94 | 157714772 383463 | 0.06 25.33 | 1 -1 chr1 ctg7180001761585 142287100 142287675 | 212133 211556 | 576 578 | 98.27 | 157714772 383463 | 0.00 0.15 | 1 -1 chr1 ctg7180001761585 142288052 142288602 | 211182 210633 | 551 550 | 99.09 | 157714772 383463 | 0.00 0.14 | 1 -1 chr1 ctg7180001761585 142288652 142295709 | 210586 203531 | 7058 7056 | 99.87 | 157714772 383463 | 0.00 1.84 | 1 -1 chr1 ctg7180001761585 142295709 142342174 | 203512 157047 | 46466 46466 | 100.00 | 157714772 383463 | 0.03 12.12 | 1 -1 chr1 ctg7180001761585 142346440 142367791 | 156958 135606 | 21352 21353 | 99.99 | 157714772 383463 | 0.01 5.57 | 1 -1 chr1 ctg7180001761585 142367822 142370681 | 135597 132737 | 2860 2861 | 99.93 | 157714772 383463 | 0.00 0.75 | 1 -1 chr1 ctg7180001761585 142370660 142382289 | 132746 121117 | 11630 11630 | 99.88 | 157714772 383463 | 0.01 3.03 | 1 -1 chr1 ctg7180001761585 142382282 142411927 | 120984 91339 | 29646 29646 | 99.96 | 157714772 383463 | 0.02 7.73 | 1 -1 chr1 ctg7180001761585 142411941 142419553 | 91339 83728 | 7613 7612 | 99.66 | 157714772 383463 | 0.00 1.99 | 1 -1 chr1 ctg7180001761585 142419553 142434546 | 83721 68728 | 14994 14994 | 99.79 | 157714772 383463 | 0.01 3.91 | 1 -1 chr1 ctg7180001761585 142434506 142437288 | 68778 65996 | 2783 2783 | 99.86 | 157714772 383463 | 0.00 0.73 | 1 -1 chr1 ctg7180001761585 142437389 142439015 | 66757 65131 | 1627 1627 | 99.94 | 157714772 383463 | 0.00 0.42 | 1 -1 chr1 ctg7180001761585 142439271 142440703 | 65629 64197 | 1433 1433 | 100.00 | 157714772 383463 | 0.00 0.37 | 1 -1 chr1 ctg7180001761585 142441869 142442975 | 63548 62442 | 1107 1107 | 100.00 | 157714772 383463 | 0.00 0.29 | 1 -1 chr1 ctg7180001761585 142446690 142449325 | 30312 32945 | 2636 2634 | 99.58 | 157714772 383463 | 0.00 0.69 | 1 1 chr1 ctg7180001761585 142451384 142452476 | 63510 64603 | 1093 1094 | 99.91 | 157714772 383463 | 0.00 0.29 | 1 1 chr1 ctg7180001761585 142452577 142454379 | 61000 62806 | 1803 1807 | 99.78 | 157714772 383463 | 0.00 0.47 | 1 1 chr1 ctg7180001761585 142454487 142456821 | 59122 61456 | 2335 2335 | 100.00 | 157714772 383463 | 0.00 0.61 | 1 1 chr1 ctg7180001761585 142458383 142459582 | 57978 59177 | 1200 1200 | 100.00 | 157714772 383463 | 0.00 0.31 | 1 1 chr1 ctg7180001761585 142459738 142472295 | 32272 44828 | 12558 12557 | 99.92 | 157714772 383463 | 0.01 3.27 | 1 1 chr1 ctg7180001761585 142472300 142485640 | 44828 58163 | 13341 13336 | 99.89 | 157714772 383463 | 0.01 3.48 | 1 1 chr1 ctg7180001761585 142501686 142530021 | 28336 1 | 28336 28336 | 99.99 | 157714772 383463 | 0.02 7.39 | 1 -1 chr1 ctg7180001761585
show-coords -d chr1-ctg.filter-q.delta | grep ctg7180001634116 116312 162914 | 1 46603 | 46603 46603 | 99.99 | 157714772 122722 | 0.03 37.97 | 1 1 chr1 ctg7180001634116 164916 201988 | 58062 95135 | 37073 37074 | 99.99 | 157714772 122722 | 0.02 30.21 | 1 1 chr1 ctg7180001634116 203244 213377 | 48198 58331 | 10134 10134 | 100.00 | 157714772 122722 | 0.01 8.26 | 1 1 chr1 ctg7180001634116 261393 264506 | 45949 49062 | 3114 3114 | 100.00 | 157714772 122722 | 0.00 2.54 | 1 1 chr1 ctg7180001634116 264607 268579 | 94345 98317 | 3973 3973 | 100.00 | 157714772 122722 | 0.00 3.24 | 1 1 chr1 ctg7180001634116 268586 274734 | 98323 104471 | 6149 6149 | 100.00 | 157714772 122722 | 0.00 5.01 | 1 1 chr1 ctg7180001634116 274835 293945 | 103611 122722 | 19111 19112 | 99.99 | 157714772 122722 | 0.01 15.57 | 1 1 chr1 ctg7180001634116
~/bin/delta2breaks.pl -m 200 < chr1-ctg.filter-q.delta | awk '{print $8}' | count.pl
AGREEMENT 9827
INVERSION 283
TRANSLOCATION+ 230
TRANSLOCATION- 154
~/bin/delta2breaks.pl -m 1000 < chr1-ctg.filter-q.delta | awk '{print $8}' | count.pl
AGREEMENT 7564
INVERSION 216
TRANSLOCATION+ 192
TRANSLOCATION- 127
~/bin/delta2breaks.pl -m 10000 < chr1-ctg.filter-q.delta | awk '{print $8}' | count.pl
AGREEMENT 3394
INVERSION 62
TRANSLOCATION+ 50
TRANSLOCATION- 29
Assembly UMD2.2 (Quality reads)
- Try to add the missing BCM.SHOTGUN reads at the assembly
- Assign new BCM.SHOTGUN library ID's base on volume & SEQ_LIB_ID : same library might have different insert size in different volume => might loose some correct mates from different volumes
cat bos_taurus.summary | grep BCM | grep SHOTG | cut -f6,7,8,10 | sort | more FAAEP 180000 13000 252 FAAEP 2000 1000 84 ... FAAHP 180000 13000 77 FAAHP 2000 1000 230 ...
- => 20,538 libraries out of which 18,208 contain mated reads
- create DST messages & add them to gkpStore
gatekeeper -a -o bt.gkpStore -T -F bos_taurus.BCM.SHOTGUN.new.DST
- generate gatekeeper edit file that maps each TI to the new library id
head bos_taurus.BCM.SHOTGUN.new.ti2libinfo.edit frg uid 499507131 libuid 601081 frg uid 499507132 libuid 601081 ...
- generate gatekeeper edit file that deletes all mate information
head bos_taurus.BCM.SHOTGUN.new.mate.delete frg uid 500086180 mateuid 0 frg uid 500084310 mateuid 0 ...
- pair forward/reverse read that have the same new library id, same TEMPLATE_ID
head bos_taurus.BCM.SHOTGUN.new.mate.edit frg uid 583866821 mateuid 583872364 frg uid 583866822 mateuid 583872408 ...
- run gatekeeper --edit for each edit/delete file
gatekeeper --edit ... bt.gkpStore
- restart assembly at cgw (doExtendClearRanges=1)
- consensus after cgw failed on job 25 on CTG 5597062 : cannot create consensus from multialignment ...
Fix: delete failed message
cp bt.cgw_contigs.25 bt.cgw_contigs.25.FAILED
delete "{ICM acc:5597062 pla:P len:20889 ..." from bt.cgw_contigs.25
- terminator fail; message:
ICL: reference before definition error for contig ID 5597062
Assembly UMD2.3 (2009_0210_CA; all reads)
- 35,973,728 reads : 35,348,776 quality & 624,952 quality-less
- 16,896,244 mates
- 25,312 libraries
Issues (not solved):
- 10420 contain at least 1 "NN" in their clr (50.. min(len,600))
- 5973 contain at least 1 "NNN" in their clr (50.. min(len,600))
Quality-less clrs
- 624,952 quality-less reads
- Quality-less read stats: : alignment CLR or 50..min(len,600) trimming
elem min max mean median n50 sum len 624952 5 1495 887 947 961 554429198 5 624952 6 1584 51 51 51 32150411 3 624952 5 1495 695 699 699 434960697 53 624952 -1579 1444 644 648 648 402810286
- Align 624,952 to the 120,461 Assembly1 contigs (no degenerates) : 1 day on 13 cpus
- 572,140(91.5%) reads aligned and 52,812(8.5%) did not align to the contigs
1. Launch jobs in parallel: 12766 jobs on 13 processors
nucmer -l 50 -c 200 -b 10 -g 5 -d 0.05 bt.ctg.001.fasta bos_taurus.0qual.01.seq -p ctg.001-seq.01 ... nucmer -l 50 -c 200 -b 10 -g 5 -d 0.05 bt.ctg.982.fasta bos_taurus.0qual.13.seq -p ctg.001-seq.01
- CPU usage: 100% /job
- Max mem usage: 0.1% /job
2. Get maximum extended clrs
cat *delta | ~/bin/delta2qryClr.pl -best | sort > bos_taurus.0qual.best.clr
Length stats
elem min max mean median n50 sum
all 624952 5 1495 887 947 961 554429198
aligned 572140 221 1416 912 953 964 522281354
unaligned 52812 5 1495 608 580 754 32147844
Best/Max/Max+extended alignment coord stats:
elem min max mean median n50 sum
53.best 572140 94 1208 766 841 877 438793102
53.max 572140 170 1208 794 863 888 454816817
53.extend 572140 170 1208 797 865 889 456014184
Unaligned read counts:
unaligned total quality quality-less
BCM.WGS 42595
UOKNOR.SHOTGUN 5787 14651 2456 12195
GSC.CLONEEND 2294 53521 0 53521
BCCAGSC.CLONEEND 1869 125241 116484 8757
BCM.SHOTGUN 186
UOKNOR.FINISHING 81
- 52,812 quality-less unaligned reads to the contigs using less strict nucmer parameters: -l 30 -c 50 -b 50 -g 50 -d 0.12
- 9,269 reads aligned at an average 92% identity (min 81% identity) : not too good
3. Get reads without clrs: set their clr to maximum 50..600
cp bos_taurus.0qual.extended.clr bos_taurus.0qual.clr difference.pl bos_taurus.0qual.infoseq bos_taurus.0qual.extended.clr | perl -ane '$three=600; $three=$F[1] if ($F[1]<600); print "$F[0] 50 $three\n";' >> bos_taurus.0qual.clr
Quality clrs
- Use Assembly1 OBT clrs
- Delete reads deleted in the OBT process
Gatekeeper
Load order:
- Add quality FRG : "gatekeeper -T -F ..."
- Add quality-less FRG "gatekeeper -F -a ..." # -T should be removed
- Delete quality FRG (deleted by UMD2.1 OBT)
- Add DST
- Add LKG
Edit
- Loads clrs
- Loads clvs
- Loads nonrandom info
Meryl
Use Assembly1 kmer counts
Overlapper
- Use 80/90 Assembly1 overlap results
- Rerun 10 overlap jobs
- 96.64% of the quality-less reads have overlaps (vs 98.33% of the quality reads)
reads 0ovl 1+ovl min max mean median n50 sum 0qual(all) 624831 20941 603890 0 4350 96 19 740 60494730 # 96.64% 0qual(unaligned) 52691 15384 37307 0 3229 50 5 349 2655545 # 70.80%
Unitigger
- More unitigs, more bases in unitigs
- Few of the longest unitigs got broken: Example 138,294(UMD2.3) vs 159,434(UMD2.1)
UNITIG OVERLAP GRAPH INFORMATION
5333434 : Total number of unitigs
2595174 : Total number of singleton, contained unitigs
1865473 : Total number of singleton, non-contained unitigs
183693 : Total number of non-singleton, spanned unitigs
689094 : Total number of non-singleton, non-spanned unitigs
35551316 : Total number of fragments
35551316 : Total number of fragments in all unitigs
21830994 : Total number of essential fragments in all unitigs
13720322 : Total number of contained fragments in all unitigs
0.0077856472 : Randomly sampled fragment arrival rate per bp
2514833413 : The sum of overhangs in all the unitigs
6483064813 : Total number of bases in all unitigs
0 : Estimated number of base pairs in the genome.
0 : Total number of contained fragments not connected
by containment edges to essential fragments.
Total rho = 2514833413
Total nfrags = 19579606
Estimated genome length = 0
Estimated global_fragment_arrival_rate=0.007786
Computed global_fragment_arrival_rate =0.007786
Total number of randomly sampled fragments in genome = 23870254
Computed genome length = 3065930496.000000
Used global_fragment_arrival_rate=0.007786
Used global_fragment_arrival_distance=128.441474
Histogram of the number of base pairs in a chunk
100406 - 138294: 21
90330 - 99887: 23
80042 - 89675: 79
70014 - 79943: 169
60002 - 69792: 374
50000 - 59982: 1008
40002 - 49995: 2440
30001 - 39994: 6509
20000 - 29999: 18989
10000 - 19999: 57404
Consensus after unitigger
Problems:
- job 120 executed partially (see bt_120.cgi_tmp); Solution: split into 3 parts, run separately, merge results
- failed on 19 unitigs (587..7447 bp)
rm 5-consensus/*failed touch 5-consensus/consensus.success
Cgw
- Failure 1 : because job 120 was run partially => missing mates
- Failure 2 : because of /5-consensus/FAILED/bt_???.cgi.failed => missing mates => delete 356 mates
Error: ProcessFrags()-- WARNING! fragiid=35973388,index=33600942 mateiid=35973363,index=0 -- MATE DOESN'T EXIST! cgw: Input_CGW.c:117: ProcessFrags: Assertion `err == 0' failed. Fix: cat cgw.out | grep MATE | p '/mateiid=(\d+)/; print $1,"\n";' >! cgw.out.mateiid gatekeeper -dumpfragments -tabular -iid cgw.out.mateiid bt.gkpStore/ | cut -f1,3 | ~/bin/mate2lkg.pl -a D >! cgw.out.delete.LKG gatekeeper -a -o bt.gkpStore -T -F -L cgw.out.delete.LKG
- Failure 3: because of cgwOutputIntermediate=1
Try to restart from ckp : die with assertion failure cgw -y -R 8 -N 12 -j 1 -k 5 -r 5 -s 2 -S 0 -z -m 100 -g ./bt.gkpStore -o ./7-0-CGW.8_12/bt ./5-consensus/bt_001.cgi cgw -y -R 8 -j 1 -k 5 -r 5 -s 2 -S 0 -z -m 100 -g ./bt.gkpStore -o ./7-0-CGW.8_12/bt ./5-consensus/bt_001.cgi Fix: Restart cgw from the beginning
- cgw does update bt.SeqStore - OpenSequenceDB()
ECR (eventually skipped)
- Failed after running for 1 day
/fs/szdevel/dpuiu/SourceForge/wgs-5.2/Linux-amd64/bin/extendClearRanges -g ./bt.gkpStore -n 15 -c bt -b 146216 -e 167100 -i 1 > 7-1-ECR/extendClearRanges-scaffold.146216.err sh: line 1: 17016 Aborted
- Last ckp : bt.ckp.15
- Try to fix:
touch 7-1-ECR/cgw.success runCA "doExtendClearRanges = 1"
- Runs too slow !!!
- Can specify a scaffold range to process: -b ? -e ? => ckp files; could we merge them?
- Failed after running for 1 day
Consensus after CGW
- Failed on job 56
tail 8-consensus/bt.cns_contigs.56.err ... Could (really) not find overlap between 153923 (U) and 2508303 (R) estimated ahang: 0 (ejecting frag 2508303 from contig) consensus: math_AS.h:51: ceil_log2: Assertion `x > 0' failed.
cat 7-CGW/bt.cgw_contigs.56 | countMessages.pl ICM 440 IMP 281412 IUP 12715 cat 8-consensus/bt.cns_contigs.56_tmp | countMessages.pl ICM 115 IMP 103322 IMV 8122 IUP 4849
- Fix: split ICM messages 1..115,116,116+ and run consensus on each set
QC
elem min max mean med n50 sum scf 56891 407 33129045 50871 1378 4716077 2894145150 ctg 122851 64 651167 21957 3647 71561 2697514858 deg 268237 65 30246 1019 985 997 273575106
- Compared with UMD2.1 : better scaffols, worse contigs & unitigs
Analysis
Issues:
- Identify bacterial & mito contigs: mito seq
- Align ctg°en to UMD2 chromosomes
- the chromosomes should have no 0cvg regions
- possible inversions, translocations (UMD2 used markers)
- if align breaks/indels, which assembly is correct?
Assembly UMD2.4 (2004_0217_CA; All reads)
- 35,973,728 reads : 35,348,776 quality & 624,952 quality-less
- 16,896,244 mates
- 344 libraries
Fix quality-less read clrs (N's) (temporary solution)
- 10420 contain at least 1 "NN" in their clr (50.. min(len,600))
- 5973 contain at least 1 "NNN" in their clr (50.. min(len,600))
Fix:
frg2seq.pl < bos_taurus.0qual.frg > bos_taurus.0qual.seq fasta2qual.pl bos_taurus.0qual.seq > ! bos_taurus.0qual.qual
lucy \
-o bos_taurus.0qual.lucy.seq bos_taurus.0qual.lucy.qual \
-debug bos_taurus.0qual.lucy.info \
bos_taurus.0qual.seq bos_taurus.0qual.qual
cat bos_taurus.0qual.lucy.info | cut -f1,3,4 -d ' ' | sort >! bos_taurus.0qual.lucy.clr
- 624,952 quality-less reads
- Quality-less read stats: 50..min(len,600) & lucy trimming
elem 0 >0 min max mean med n50 sum 5 624952 2857 622095 0 501 52 52 52 33012433 3 624952 2857 622095 0 600 579 600 600 361980208 53 624952 2857 622095 0 548 526 548 548 328967775
Fix quality-less read clrs (low complexity)
- Run dust filter on seq (before qual & lucy)
elem 0 >0 min max mean med n50 sum 5 624952 3564 621388 0 501 75 52 52 47385578 3 624952 3564 621388 0 600 554 600 600 346470473 53 624952 3564 621388 0 548 478 548 548 299084895
- Merge dust.lucy clrs with the alignment clrs
elem 0 >0 min max mean med n50 sum 5 624952 4488 620464 0 599 93 52 126 58378496 3 624952 4488 620464 0 600 547 600 600 342258160 53 624952 4488 620464 0 548 454 512 548 283879664
- Test seq
gatekeeper -dumpfastaseq -b 35348777 -e 35973728 bt.gkpStore | grep NNN
gatekeeper -dumpfastaseq bt.gkpStore | perl -ane 'if(/^>(\d+)/) { $id=$1} elsif(/NNN/) { print $id,"\n";} ' | uniq -c | awk '{print $2,$1}' > bt.NNN.seqs # 2411 seqs (all have the N's "in the middle")
gatekeeper -dumpfastaseq -uid bt.NNN.seqs bt.gkpStore > bt.NNN.cseqs
Consolidate libraries
Drop from 25,312 to 344 libs
BCM.SHOTGUN
UMD2.4 reestimated 10,117 out of 13,826 libs (have > 100mates)
Base on initial estimates
- Reduce the total number from 13826 to 2 libs: 3000 & 6000
- UMD2.3 mean estimates (Initial vs Final):
meanI #libs minF maxF meanF medF n50F sumF uid 180000 436 1636 5199 2475 2410 2458 1079407 #3000 167000 86 1585 2948 2264 2258 2285 194775 #3000 6500 31 5212 6636 5837 5867 5924 180951 #6000 6000 11 4556 6272 5389 5421 5421 59286 #6000 3500 949 1670 4769 2668 2608 2645 2532027 #3000 3000 2511 1483 5250 2715 2662 2723 6818678 #3000 2000 6093 1157 6443 2526 2487 2554 15391160 #3000
Base on final estimates
- Reduce the total number from 13826 to 7 libs: 6500,5500,...1500, un-estimates (2501)
meanF #libs min max mean med n50 sum uid(new) mean(new) std(new) 6K<=mea<7K 15 6010 6636 6176 6159 6159 92650 6500 6500 5K<=mea<6K 29 5121 5985 5540 5536 5577 160673 5500 5500 4K<=mea<5K 67 4017 4939 4284 4266 4274 287072 4500 4500 3K<=mea<4K 1401 3000 3998 3276 3209 3226 4590323 3500 3500 2K<=mea<3K 7998 2000 2999 2502 2498 2532 20017767 2500 2500 1K<=mea<2K 607 1157 1999 1825 1882 1890 1107798 1500 1200 un-estimated 3709 2501 2501
BARC.CLONEEND
Collapse all 11150 into 1:
uid:25456 mea:165000 std:43000
Overlapper
- Quality-less reads overlaps: fewer than in the UMD2.3 assembly
elem 0 >0 min max mean med n50 sum 0qual(all) 624830 35692 589138 0 3237 60 14 439 37578899 # 94.39%
Unitigger
UNITIG OVERLAP GRAPH INFORMATION
5356408 : Total number of unitigs
2613795 : Total number of singleton, contained unitigs
1870448 : Total number of singleton, non-contained unitigs
182878 : Total number of non-singleton, spanned unitigs
689287 : Total number of non-singleton, non-spanned unitigs
35547861 : Total number of fragments
35547861 : Total number of fragments in all unitigs
21685943 : Total number of essential fragments in all unitigs
13861918 : Total number of contained fragments in all unitigs
0.0077797328 : Randomly sampled fragment arrival rate per bp
2513424271 : The sum of overhangs in all the unitigs
6468428782 : Total number of bases in all unitigs
0 : Estimated number of base pairs in the genome.
0 : Total number of contained fragments not connected
by containment edges to essential fragments.
Total rho = 2513424271
Total nfrags = 19553770
Estimated genome length = 0
Estimated global_fragment_arrival_rate=0.007780
Computed global_fragment_arrival_rate =0.007780
Total number of randomly sampled fragments in genome = 23868770
Computed genome length = 3068070656.000000
Used global_fragment_arrival_rate=0.007780
Used global_fragment_arrival_distance=128.539119
Histogram of the number of base pairs in a chunk
100292 - 138301: 19
90052 - 99906: 23
80043 - 89676: 79
70013 - 79966: 164
60010 - 69988: 390
50008 - 59983: 949
40000 - 49998: 2433
30000 - 39997: 6437
20000 - 29999: 18808
10000 - 19999: 57634
Bog
!!! Much bigger unitigs than default unitigger
Global Arrival Rate: 0.013829 212260 - 224992: 4 100099 - 186873: 372 90015 - 99973: 353 80045 - 89988: 582 70011 - 79999: 1084 60000 - 69994: 1856 50001 - 59996: 3162 40002 - 49994: 5407 30000 - 39999: 9767 20000 - 29996: 18981 10000 - 19999: 39641
Consensus after Unitigger
- Failed on jobs 120 & 121 ( _tmp file)
cat 4-unitigger/*120* | countMessages.pl IMP 280264 IUM 124707 cat 4-unitigger/*121* | countMessages.pl IMP 282146 IUM 245650
cat 5-consensus/bt_120.cgi_tmp | countMessages.pl IMP 34348 IUM 19222 cat 5-consensus/bt_121.cgi_tmp | countMessages.pl IMP 51833 IUM 16805
- Fix 120: split IUM messages
extractfromfrgMSG.pl -b 0 -e 19222 bt_120.cgb.orig IUM >! bt_120.cgb & extractfromfrgMSG.pl -b 19222 bt_120.cgb.orig IUM >! bt_120.cgb &
- Fix 121: remove assertion in AS_CNS/MultiAlignment_CNS.c
if(to <= from || to > ma_length-1){
fprintf(stderr, "AbacusRefine range (to) invalid");
//assert(0);
}
CGW
- Failed after Ckp3(7-0-CGW/bt.ckp.3; MergeScaffoldsAggressive 2nd itteration)
CI extends beyond end of scaffold! offsetAEnd = 254204 offsetBEnd = 252250 scaffoldLength = 253268 cgw: CIScaffoldT_Merge_CGW.c:307: InsertScaffoldContentsIntoScaffold: Assertion `0' failed.
- Last cgw
Scaffold lengths:
cat 7-4-CGW/stat/final0.*Scaffolds.nodelength.cgm | grep -v ^Scaff | getSummary.pl -t scf
cat 7-4-CGW/stat/final0.PlacedContig.n | grep -v ^Scaff | getSummary.pl -t scf
elem min max mean med n50 sum
scf 45826 385 34263871 59591 1349 7059820 2730853790
ctg 96562 65 738899 27789 3657 93988 2683452359
Library insert estimates:
cat 7-4-CGW/stat/scaffold_final.distupdate.dst | grep ^# | awk '{print $3,int($8),int($10)}' > 7-4-CGW/bt.dst
join2.pl bt.dst 7-4-CGW/bt.dst | p 'print join "\t",@F[0,1,2,5,6,3,4]; print "\n";' > bt.dst.combine
CLONEEND inserts: UID MEANI STDI MEANF STDF COUNT LIB 114892 150000 30000 175701 40732 31063 UIUC.CLONEEND 19070 167000 25000 171349 18253 7103 BCM.CLONEEND 118 167000 16700 167000 16700 21 WUGSC.CLONEEND 25456 165000 43000 163044 25849 11150 BARC.CLONEEND 115020 150000 30000 162719 25343 15256 UIUC.CLONEEND 65177 2000 600 162540 34155 27067 TIGR.CLONEEND 125606 150000 30000 162396 19319 59505 BCCAGSC.CLONEEND 10738 2500 750 162386 27567 2040 TIGR_JCVIJTC.CLONEEND 10691 2500 750 161540 28239 2763 TIGR_JCVIJTC.CLONEEND 17249 202000 20200 157496 55375 6269 CENARGEN.CLONEEND 54017 120000 12000 115671 27594 25889 GSC.CLONEEND total 188126 CLONEENDs
Consensus
- Failed on job 34 with segmentation fault
- 9kbp contig, made out of 3007 reads (24 of which are quality-less)
cat 7-CGW/bt.cgw_contigs.34.1 | grep "^{" | uniq -c | awk '{print $2,$1}'
{ICM 1
{IMP 3007
{IUP 329
- Fix : edit AS_CNS/MultiAlignment_CNS.c; add
if(!ungappedSequence->Elements) { ungappedSequence->numElements=0; }
if(!ungappedQuality->Elements) { ungappedQuality->numElements=0; }
Analysis
Contigs Vs possible contaminants
- nucmer alignment parameters: -l 40 -c 100 -b 10 -g 5 -d 0.05
- have to redo alignments using -maxmatch !!!
- file location:
reference seqs: /nfshomes/dpuiu/db/Ecoli.365350-365744 # Ecoli K12 region with most alignments (BCM WGS splice site) /nfshomes/dpuiu/db/Ecoli # Ecoli K12 substrain MG1655 (NC_000913 ; 1st completed) /nfshomes/dpuiu/db/Ecoli.all # 22 Ecoli completed genomes ( + plasmids) /nfshomes/dpuiu/db/UniVec_Core # UniVec Core seqs /nfshomes/dpuiu/db/OtherVec # 100 other vector sequences identified by aligning UMD2.0 contaminants to GenBank; align also to 110 UniVec core using nucmer (params above)
/nfshomes/dpuiu/db/bos_taurus.UMD2.contaminant.fasta # 4813 whole contigs and 30329 contig regions identified by NCBI as UMD2 contamination /nfshomes/dpuiu/db/bos_taurus.UMD2.contaminant.organism_count # organism counts: vector is the most abundant /nfshomes/dpuiu/db/bos_taurus.UMD2.contaminant.infoseq # grep -v 'coli|vector|7180003101029' => 905 other contamiants query seqs: /scratch1/bos_taurus/Assembly/2009_0217_CA/9-terminator/ctg.split100/*fasta # latest assembly contigs (no degenerates) delta files: /scratch1/bos_taurus/Assembly/2009_0217_CA/nucmer_ctg/no_maxmatch/*delta
Ecoli K12 substrains:
NC_010473.1 4686137 50.78 Escherichia coli str. K-12 substr. DH10B, complete genome NC_000913.2 4639675 50.79 Escherichia coli str. K-12 substr. MG1655, complete genome AC_000091.1 4646332 50.80 Escherichia coli str. K-12 substr. W3110, complete genome
no maxmatch
- fewer alignments in UMD2.4 than in UMD2
UMD2 (all): just a few degens
15102 Ecoli.365350-365744-ctg.qry_hits 15943 Ecoli-ctg.qry_hits 17308 Ecoli.all-ctg.qry_hits 79065 UMD2.contaminant-ctg.qry_hits # 55877 new hits 20105 UMD2.contaminant-ctg.CBE.qry_hits # CONTAIN|BEGIN|END|IDENTITY 19839 UniVec_Core-ctg.qry_hits
UMD2.4
559 Ecoli.365350-365744-ctg.qry_hits 1215 Ecoli-ctg.qry_hits 2767 Ecoli.all-ctg.qry_hits # most 2 frequenct starins are UMN026 & ATCC 8739; K12 DH10B is rank 5th; K12 MG1655 is ranked 19th (out 31 seqs) 44112 UMD2.contaminant-ctg.qry_hits 5286 UniVec_Core-ctg.qry_hits
Length of the reference seqs used for screening:
#seqs min max mean med n50 sum Ecoli.365350-365744 1 395 395 395 395 395 395 # Ecoli K12 regions with most alignments (BCM WGS splice site) Ecoli 1 4639675 4639675 4639675 4639675 4639675 4639675 # Ecoli K12 substrain MG1655 Ecoli.all 49 3306 5572075 2293320 130440 5065741 112372708 # 22 Ecoli's UniVec_Core 1348 12 48551 243 98 967 327641 OtherVec 100 1702 739874 15419 5027 166744 1541984
UMD2.contaminant 35142 48 16661 512 362 674 18022349
Length of UMD2.4 contigs that contain contaminant (0+ bp from end):
#ctgs <2000bp >=2000bp min max mean med n50 sum Ecoli.365350-365744-ctg 559 534 25 1001 179527 2467 1341 1894 1379440 Ecoli-ctg 1215 1086 129 1001 360312 4326 1347 71372 5256540 Ecoli.all-ctg 2767 2455 312* 1001 453627* 8031 1366 134516 22224468 UniVec_Core-ctg 5286 4718 568* 882 651163* 9820 1337 136090 51909339 UMD2.contaminant-ctg.CBE 4976 4410 566* 738 651163* 8497 1339 122281 42281715 #annotated alignments: CONTAIN|BEGIN|END|IDENTITY UMD2.contaminant-ctg 44112 12813 31299 268 739442 50591 27461 111598 2231701788
Length of UMD2.4 contigs that contain contaminant in the middle (500+ bp from end):
#ctgs <2000bp >=2000bp min max mean med n50 sum Ecoli.365350-365744-ctg 144 136 8 1286 2053 1779 1811 1814 256259 Ecoli-ctg 171 152 19 1286 4703 1835 1807 1821 313820 Ecoli.all-ctg 197 160 37*(81) 1228 351373* 6516 1815 125069 1283728 #81 2K+ ctgs using -maxmatch UniVec_Core-ctg 1278 1110 168*(276) 1085 651163* 12266 1496 160336 15676765 #276 2K+ ctgs using -maxmatch UMD2.contaminant-ctg.CBE 52 25 27* 1249 351373* 22195 2054 125069 1154142 #annotated alignments: CONTAIN|BEGIN|END|IDENTITY UMD2.contaminant-ctg 31019 1437 29582 1113 739442 70665 50798 113684 2191986214
Length of the UMD2.4 contaminant seqeunece (0+ bp from end):
#align <200bp >=200bp min max mean med n50 sum Ecoli.365350-365744-ctg 1066 537 529 104 225 192 162 224 205379 Ecoli-ctg 1793 587 1206 50 4440 496 224 994 889798 Ecoli.all-ctg 4074 1132 2942* 40 17075* 380 254 441 1551783 UniVec_Core-ctg 14425 9819 4606* 40 1801* 236 162 325 3409187 UMD2.contaminant-ctg 144843 96008 48835 40 16661 199 169 209 28912002
Length of the UMD2.4 contaminant seqeunece (500+ bp from end)
alignm <200bp >=200bp min max mean med n50 sum Ecoli.365350-365744-ctg 243 136 107 162 224 189 162 224 46000 Ecoli-ctg 273 149 124 106 1341 219 162 224 59923 Ecoli.all-ctg 294 153 141* 106 2150* 251 162 224 73992 UniVec_Core-ctg 2144 2035 109* 50 1340* 122 121 121 261821 UMD2.contaminant-ctg 121331 86985 34346 40 2738 171 162 184 20753580
- Problem: 8 long ctgs contain Ecoli in the middle (1000+ bp from end)
show-coords Ecoli.all-ctg.filter-q.delta | ~/bin/filterQryCoords.pl -i 1000 | sort -nk13 -r
[S1] [E1] | [S2] [E2] | [LEN 1] [LEN 2] | [% IDY] | [LEN R] [LEN Q] | [COV R] [COV Q] | [TAGS] =============================================================================================================================== 4640589 4641890 | 161712 160411 | 1302 1302 | 99.46 | 4686137 351373 | 0.03 0.37 | gi|170079663|ref|NC_010473.1| ctg7180001872124 5068908 5069620 | 87386 86679 | 713 708 | 98.88 | 5209548 91972 | 0.01 0.77 | gi|218687878|ref|NC_011745.1| ctg7180002055226 3087480 3088683 | 50423 51620 | 1204 1198 | 99.00 | 5202090 88182 | 0.02 1.36 | gi|218703261|ref|NC_011751.1| ctg7180002054092 4640580 4641890 | 19953 18646 | 1311 1308 | 99.08 | 4686137 31157 | 0.03 4.20 | gi|170079663|ref|NC_010473.1| ctg7180001875158 131462 133564 | 1247 3349 | 2103 2103 | 98.19 | 241387 5751 | 0.87 36.57 | gi|157412014|ref|NC_009838.1| ctg7180002043242 82801 83166 | 2986 2621 | 366 366 | 98.09 | 241387 4709 | 0.15 7.77 | gi|157412014|ref|NC_009838.1| ctg7180001714551 82264 82793 | 3523 2994 | 530 530 | 98.49 | 241387 4709 | 0.22 11.26 | gi|157412014|ref|NC_009838.1| ctg7180001714551 1652253 1652545 | 1487 1195 | 293 293 | 98.63 | 4700560 2492 | 0.01 11.76 | gi|218552585|ref|NC_011741.1| ctg7180001754941
- Regions present in DH10B but not MG1655
delta2cvg -M 0 < DH10B-MG1655.delta gi|170079663|ref|NC_010473.1| 1349629 1378243 28614 0 gi|170079663|ref|NC_010473.1| 1391006 1396986 5980 0 gi|170079663|ref|NC_010473.1| 3199469 3200798 1329 0 gi|170079663|ref|NC_010473.1| 3211928 3213257 1329 0 gi|170079663|ref|NC_010473.1| 4640588 4641918 1330 0 !!!
- Problem: 10 long ctgs contain Vector in the middle (1000+ bp from end)
show-coords UniVec_Core-ctg.filter-q.delta | ~/bin/filterQryCoords.pl -i 1000 | sort -nk13 -r
[S1] [E1] | [S2] [E2] | [LEN 1] [LEN 2] | [% IDY] | [LEN R] [LEN Q] | [COV R] [COV Q] | [TAGS]
===============================================================================================================================
1 121 | 215495 215615 | 121 121 | 99.17 | 170 271477 | 71.18 0.04 | gnl|uv|U09128.1:15891-16011-49 ctg7180002047604 # pSacBII P1 cloning vector
2252 2435 | 1334 1151 | 184 184 | 100.00 | 2485 160336 | 7.40 0.11 | gnl|uv|U75992.1:16925-19409 ctg7180001808271
180 312 | 1153 1020 | 133 134 | 99.25 | 312 160336 | 42.63 0.08 | gnl|uv|NGB00145.1:2378-2689 ctg7180001808271
1 121 | 1367 1487 | 121 121 | 100.00 | 170 160336 | 71.18 0.08 | gnl|uv|U09128.1:15891-16011-49 ctg7180001808271
1 103 | 1286 1388 | 103 103 | 100.00 | 103 160336 | 100.00 0.06 | gnl|uv|U80929.2:11415-11517 ctg7180001808271 [CONTAINED]
4 121 | 68269 68386 | 118 118 | 100.00 | 170 111913 | 69.41 0.11 | gnl|uv|U09128.1:15891-16011-49 ctg7180002052060
40 152 | 30255 30142 | 113 114 | 99.12 | 1663 42854 | 6.79 0.27 | gnl|uv|U09128.1:1-1663 ctg7180002053344
1 121 | 34358 34238 | 121 121 | 100.00 | 170 35471 | 71.18 0.34 | gnl|uv|U09128.1:15891-16011-49 ctg7180002046164
1 103 | 34439 34337 | 103 103 | 100.00 | 103 35471 | 100.00 0.29 | gnl|uv|U80929.2:11415-11517 ctg7180002046164 [CONTAINED]
46 1385 | 8928 10267 | 1340 1340 | 100.00 | 1413 17587 | 94.83 7.62 | gnl|uv|X65279.1:5941-7353 ctg7180002043597 [CONTAINED]
- ctg7180001872124 : 351373 bp; region 160411..161712 contaminated by Ecoli
cat 9-terminator/bt.posmap.utgctg | grep 7180001872124 | wc -l # 329
cat 9-terminator/bt.posmap.utgctg | grep 7180001872124 | perl -ane '@F[2,3]=@F[3,2] if($F[2]>$F[3]); print $_ if($F[2]<160411 and 160411<$F[3] or $F[2]<161712 and 161712<$F[3]);' 7180000441625 7180001872124 159483 161201 r 7180000441788 7180001872124 160330 161329 f #Ecoli 7180000442730 7180001872124 160368 161010 r #Ecoli 7180000441635 7180001872124 160740 162700 f #Ecoli
cat 9-terminator/bt.utg.info utg7180000441625 length=1715 num_frags=12 Astat=7.00 utg7180000441788 length=999 num_frags=1 Astat=0.00 utg7180000442730 length=640 num_frags=1 Astat=0.00 utg7180000441635 length=1957 num_frags=9 Astat=7.00
cat 9-terminator/bt.posmap.frgctg | perl -ane '@F[2,3]=@F[3,2] if($F[2]>$F[3]); print $_ if($F[2]<160411 and 160411<$F[3] or $F[2]<161712 and 161712<$F[3]);' 1237446426 7180001872124 160117 161201 f 1238816835 7180001872124 160133 160993 f 1238817728 7180001872124 160322 161123 r 1244436200 7180001872124 159976 160984 f 1238817676 7180001872124 160105 160890 r 1237443253 7180001872124 160106 160900 f 1237471027 7180001872124 159930 160928 f 1238822613 7180001872124 159774 160782 f 1238816875 7180001872124 159878 160728 f 1244436248 7180001872124 159483 160553 f 1238818306 7180001872124 159718 160489 f 1238818332 7180001872124 159722 160483 f 1237476824 7180001872124 160330 161329 f 1238817689 7180001872124 160368 161010 r 1237447135 7180001872124 160740 161768 r 1237483546 7180001872124 160814 161790 r 1237483530 7180001872124 160818 161856 r 1237471108 7180001872124 161003 162009 f 1238817744 7180001872124 161151 161978 f 1237446441 7180001872124 161050 162107 f 1244436201 7180001872124 161117 162164 f 1237446407 7180001872124 161586 162699 r 1237471055 7180001872124 161652 162700 r # 23 BCM SHOTGUN RP42 VVHNP reads (1369 read lib; 1341 of the reads in this ctg)
- ctg7180002047604 : Vctor in the middle
[S1] [E1] | [S2] [E2] | [LEN 1] [LEN 2] | [% IDY] | [LEN R] [LEN Q] | [COV R] [COV Q] | [TAGS]
===============================================================================================================================
1 121 | 215495 215615 | 121 121 | 99.17 | 170 271477 | 71.18 0.04 | gnl|uv|U09128.1:15891-16011-49 ctg7180002047604 # pSacBII P1 cloning vector
cat 9-terminator/bt.posmap.utgctg | grep 7180002047604 perl -ane '@F[2,3]=@F[3,2] if($F[2]>$F[3]); print $_ if($F[2]<215495 and 215495<$F[3] or $F[2]<215615 and 215615<$F[3]);' 7180000441711 7180001872124 214458 219678 r
cat 9-terminator/bt.posmap.frgctg | grep 7180002047604 | perl -ane '@F[2,3]=@F[3,2] if($F[2]>$F[3]); print $_ if($F[2]<215495 and 215495<$F[3] or $F[2]<215615 and 215615<$F[3]);' 498776751 7180001872124 215425 216426 r 1236502885 7180001872124 215514 216377 r 379408823 7180001872124 215572 216340 f 1244436224 7180001872124 215388 216405 f 1237471071 7180001872124 215229 216234 r 1233297450 7180001872124 215234 216046 f 1233363357 7180001872124 215267 215687 f 937200686 7180001872124 215300 216129 r 937254901 7180001872124 215321 216160 f 1233294025 7180001872124 215383 216204 r 1237446444 7180001872124 215146 216187 f 1232033776 7180001872124 215193 215996 r 671976381 7180001872124 215035 216021 r 514932286 7180001872124 215043 216008 f 500723879 7180001872124 215043 215802 f 671927656 7180001872124 215116 215733 r 381173692 7180001872124 214947 215877 r 1233303570 7180001872124 214963 215803 f 1232037705 7180001872124 214990 215803 f 490852264 7180001872124 214923 215843 f 1237447184 7180001872124 214684 215646 f 668822243 7180001872124 214586 215572 f #22 reads ; ~half come from BCM SHOTGUN RP42 VVFOP
maxmatch ctg
Parameters:
nucmer -maxmatch -l 40 -c 100 -b 10 -g 5 -d 0.05 ... AllVec: UniVec_Core + 100 more vector seqs
Length of UMD2.4 contigs that contain contaminant (0+ bp from end):
elem <2000 >2000 min max mean med n50 sum Ecoli.all 2951* 2602 349 1001 453627 8252 1367 132226 24352779 UniVec_Core 5387* 4802 585 882 651163 9979 1334 136556 53760575 OtherVec 5657 5062 595 882 651163 9726 1320 136556 55021803 UMD2.cont.other 3976 3430 546 804 651163 11217 1346 130385 44601117 #18 aligned to Acinetobacter; longest is 56467bp
Length of UMD2.4 contigs that contain contaminant (500+ bp from end):
elem <2000 >2000 min max mean med n50 sum Ecoli.all 182 156 26* 1286 351373 6525 1811 125069 1187706 # 7* are >5K; 321* come from multi-ctg scaffolds UniVec_Core 2532 2220 312* 1065 651163 10593 1481 128344 26821960 # 267* are >5K ; 655* come from multi-ctg scaffolds OtherVec 376 323 53 1184 361749 13278 1508 139997 4992774 UMD2.cont.other ...
Length of UMD2.4 contigs that contain contaminant (1000+ bp from end):
elem <2000 >2000 min max mean med n50 sum Ecoli.all 8 0 8* 4709 351373 73065 31157 351373 584520 UniVec_Core 11 0 11* 2600 334933 93674 37847 271477 1030414 OtherVec 5 0 5* 3717 271477 131604 111913 228060 658021 UMD2.cont.other 54 0 54* 2398 522682 110947 88182 189352 5991164 total 67* # 18 of them are CONTAINED by UMD2.0 chromosomes
Length of the UMD2.4 contaminant sequence (0+ bp from end):
elem <200 >200 min max mean med n50 sum Ecoli.all 4775 1610 3165 39 17072 381 236 502 1823278 UniVec_Core 16985 12380 4605 39 1800 207 132 300 3519080 OtherVec 7563 1372 6191 39 1800 509 548 643 3849567 UMD2.cont.other 6626 343 6283 39 8228 543 573 615 3602329
maxmatch deg
All degenerates aligned are <2K
Length of UMD2.4 deg that contain contaminant (0+ bp from end):
elem <2000 >2000 min max mean med n50 sum Ecoli.all 1266 1266 0 104 1611 783 833 869 991447 UniVec_Core 1908 1908 0 147 1510 872 896 910 1664746 OtherVec 1963 1963 0 147 1510 872 898 911 1712703 UMD2.cont.other 1609 1609 0 132 1611 852 892 914 1372106
maxmatch utg
Unitig stats:
elem <2000 >2000 min max mean med n50 sum 1707816 1434164 273652 21 138676 2228 937 8002 3805166508
Parameters:
nucmer -maxmatch -l 40 -c 100 -b 10 -g 5 -d 0.05 ...
Files:
/scratch1/bos_taurus/Assembly/2009_0217_CA/nucmer_utg/
Length of UMD2.4 unitigs that align to contaminants
elem <2000 >2000 min max mean med n50 sum Ecoli.all 4275 4110 165 104 71709 1442 1212 1398 6166566 UniVec_Core 7563 7409 154 139 71709 1397 1182 1331 10570512 OtherVec 8208 8054 154 139 71709 1370 1159 1308 11248775 UMD2.cont.other 6094 5849 245 132 53113 1546 1163 1401 9422951 #80 aligned to Acinetobacter; longest is 9114bp Contaminants(all above) 10264 9895 369 104 71709 1471 1148 1359 15107544 Acinetobacter 2306** 0 2306 154 71709 1451 1316 1412 3347230 #2182 already in the Cont set
Length of UMD2.4 unitigs that have contaminants 500+bp from ends
elem <2000 >2000 min max mean med n50 sum Ecoli.all 172 156 16 1286 4852 1820 1805 1815 313185 UniVec_Core 2491 2422 69 1065 71709 1722 1457 1523 4291584 OtherVec 364 358 6 1167 71709 1795 1478 1538 653595 UMD2.cont.other 156 108 48 1213 50248 5344 1838 17518 833673
Length of the UMD2.4 alignments of unitigs to contaminants (unique unitig regions)
elem <200 >200 min max mean med n50 sum reads(all unitig reads for unitgs with alignments>1K) Ecoli.all 5975 1686 4289 40 8184 397 268 542 2374366 12112(12142) UniVec_Core 8754 1674 7080 40 1801 474 490 645 4153030 26590(26849) OtherVec 8919 1250 7669 40 1801 511 536 629 4562326 30268(30268) UMD2.cont.other 6752 896 5856 40 6012 529 555 651 3573528 25006(25328) Contaminants(all above) 10992 1396 9596 40 8184 571 573 684 6280759 40351(40699) Acinetobacter (8286)
40699 reads aligned back to contaminants: nucmer -maxmatch
- 35919 align
- 34400 align 100+bp
- 27742 align 200+bp
- 14211 align 500+bp
utg 5'& 3'
Unitig stats:
elem <200 >200 min max mean med n50 sum utg 1,707,816 81200 1626616 21 138676 2228 937 8002 3805166508 utg5'&3' 3,334,432 0 3334432 21 199 100 100 100 335263271
Align utg5'&3' to Ecoli.all using:
- nucmer -l 40 -c 100 -b 10 -g 5 -d 0.05 : 4,275 hits
- nucmer -l 20 -c 40 : 6,617 hits
- nucmer -l 20 -c 20 : 23,350
- blastall : 2,895,506 out of 3,334,432 (86%) aligned
Acinetobacter contamination
Database:
~dpuiu/db/Acinetobacter.all : 7 complete genomes, 19 seqs
Seq len summary:
elem min max mean med n50 sum 19 2726 4050513 1418094 28279 3904116 26943793
Align all unitigs to Acinetobacter.all; Longest alignments is 8517bp
show-coords Acinetobacter.all-utg.filter-q.delta | sort -nk8 -r | head [S1] [E1] | [S2] [E2] | [LEN 1] [LEN 2] | [% IDY] | [LEN R] [LEN Q] | [COV R] [COV Q] | [GenBank] [UMD2.4 utg] =============================================================================================================================== 20644 29164 | 62 8578 | 8521 8517 | 98.80 | 94413 8578 | 9.03 99.29 | gi|169786889|ref|NC_010404.1| utg7180000281954* [CONTAINS] 3395586 3401299 | 5712 1 | 5714 5712 | 99.79 | 3976747 8015 | 0.14 71.27 | gi|126640115|ref|NC_009085.1| utg7180000212251 3400344 3404485 | 1 4142 | 4142 4142 | 99.66 | 3976747 9114 | 0.10 45.45 | gi|126640115|ref|NC_009085.1| utg7180000277331 ...
utg7180000281954* -> ctg7180002053982 (28140bp; 78 unitigs)
grep 7180002053982 ../9-terminator/bt.posmap.utgctg | nl
1 7180000185222 7180002053982 0 3019 f
2 7180000314302 7180002053982 2151 5706 r
3 7180001463328 7180002053982 2256 2869 f
...
75 7180000281954* 7180002053982 17862 26442 r
76 7180001471348 7180002053982 17886 18723 r
77 7180001468075 7180002053982 17919 18732 f
78 7180000280508 7180002053982 25672 28140 r
show-coords UMD2.contaminant.other-ctg.filter-q.delta | grep 7180002053982
[S1] [E1] | [S2] [E2] | [LEN 1] [LEN 2] | [% IDY] | [LEN R] [LEN Q] | [COV R] [COV Q] | [UMD2.0 contam] [UMD2.4 ctg]
===============================================================================================================================
1394 2508 | 28140 27024 | 1115 1117 | 98.75 | 7098 28140 | 15.71 3.97 | 7180003313366 ctg7180002053982
2561 2871 | 26971 26661 | 311 311 | 97.11 | 7098 28140 | 4.38 1.11 | 7180003313366 ctg7180002053982
2934 5670 | 26599 23862 | 2737 2738 | 97.99 | 7098 28140 | 38.56 9.73 | 7180003313366 ctg7180002053982
...gap...
5930 7098 | 17270 16101 | 1169 1170 | 98.46 | 7098 28140 | 16.47 4.16 | 7180003313366 ctg7180002053982
...gap...
281 1981 | 10335 8635 | 1701 1701 | 99.41 | 13090 28140 | 12.99 6.04 | 7180003320028 ctg7180002053982
1992 2672 | 8635 7954 | 681 682 | 99.71 | 13090 28140 | 5.20 2.42 | 7180003320028 ctg7180002053982
3302 5376 | 6719 4643 | 2075 2077 | 99.33 | 13090 28140 | 15.85 7.38 | 7180003320028 ctg7180002053982
8469 9021 | 4642 4090 | 553 553 | 99.10 | 13090 28140 | 4.22 1.97 | 7180003320028 ctg7180002053982
9038 9313 | 4073 3798 | 276 276 | 98.55 | 13090 28140 | 2.11 0.98 | 7180003320028 ctg7180002053982
9780 13090 | 3331 19 | 3311 3313 | 99.19 | 13090 28140 | 25.29 11.77 | 7180003320028 ctg7180002053982
grep 7180002053982 ../9-terminator/bt.posmap.utgctg | awk '{print $1,$4-$3+1}' | sed 's/^/utg/' >! ctg7180002053982.utgs
intersect.pl UMD2.contaminant.other-utg.qry_hits ctg7180002053982.utgs | wc -l
37 # only 37 out of 78 unitigs were detected
ctg7180002053982 is Acinetobacter
UMD2.5 (2004_0312_CA; delete 40699 contam reads & 22607 mates )
40699 reads:
- 25803 mated + 14896 unmated
- 6392 mated reads had the mate also contaminated
Location:
/scratch1/bos_taurus/Assembly/2009_0312_CA
UNITIGGER
UNITIG OVERLAP GRAPH INFORMATION
5322910 : Total number of unitigs
2595715 : Total number of singleton, contained unitigs
1869655 : Total number of singleton, non-contained unitigs
182193 : Total number of non-singleton, spanned unitigs
675347 : Total number of non-singleton, non-spanned unitigs
35507162 : Total number of fragments
35507162 : Total number of fragments in all unitigs
21641007 : Total number of essential fragments in all unitigs
13866155 : Total number of contained fragments in all unitigs
0.0077909501 : Randomly sampled fragment arrival rate per bp
2511009753 : The sum of overhangs in all the unitigs
6442095933 : Total number of bases in all unitigs
0 : Estimated number of base pairs in the genome.
0 : Total number of contained fragments not connected
by containment edges to essential fragments.
Total rho = 2511009753
Total nfrags = 19563152
Estimated genome length = 0
Estimated global_fragment_arrival_rate=0.007791
Computed global_fragment_arrival_rate =0.007791
Total number of randomly sampled fragments in genome = 23866135
Computed genome length = 3063315200.000000
Used global_fragment_arrival_rate=0.007791
Used global_fragment_arrival_distance=128.354050
Histogram of the number of base pairs in a chunk
100292 - 138301: 22 # 19 in UMD2.4
90020 - 99906: 28 # 23
80043 - 89676: 90 # 79
70013 - 79966: 190 # 164
60010 - 69988: 423
50008 - 59983: 1016
40000 - 49998: 2558
30000 - 39997: 6660
20000 - 29999: 18927
10000 - 19999: 57057
CONSENSUS after CGW
- failed on job 80 : ctg 5706539, len=180,024, 159 unitigs, 1,851 reads
head 80/bt.cns_contigs.80.failed
{ICM
acc:5706539
pla:P
len:180024
cns:
.
qlt:
.
for:0
npc:1851
more ../9-terminator/bt.asm
...
{CCO
acc:(7180002022380*,5706539)
pla:P
len:180024
cns:
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
...
cat 80/bt.cns_contigs.80.failed | countMessages.pl ICM 1 IUP 159 #unitigs IMP 1851 #reads : 1264 are BCM.WGS, 389 are BCM.SHOTGUN ...
# 254 contig scaffold cat ../9-terminator/bt.posmap.ctgscf | grep 7180002041301 | nl 1 7180002022042 7180002041301 0 14121 f ... 183 7180002022380* 7180002041301 9164814 9344838 f .. 254 7180002022240 7180002041301 12874874 12893460 f
# the 5 UMD2.4 contigs below have the same number of reads with the ones that matched => CONTAINED
cat UMD2.4-7180002022380.posma p.frgctg | grep -v ^$ | awk '{print $2}' | uniq -c
#reads ctgid
6 7180001712307
209 7180002028662
6 7180002028663
1552 7180002028664
21 7180002032323
1794 total => 1851-1794=57 additional reads
cat ../../2009_0217_CA/9-terminator/bt.posmap.ctgscf | nl | egrep '7180001712307|7180002028662|7180002028663|7180002028664|7180002032323' #nl ctgid scfid start end dir 34791 7180002028662 7180002069912 1425623 1446022 f 34793 7180002032323 7180002069912 1448133 1450475 f 34794 7180002028663 7180002069912 1450495 1451918 f 34795 7180002028664 7180002069912 1452234 1602973 f 64110 7180001712307 7180002071598 0 1441 f
- Solution 1:
* consensus -Dforceunitigabut => new assembly, new UID's ctg7180002022636 179505 38.53 # => ctg7180002022380 scf7180002041557 12892941 40.58 # => scf7180002041301
- Solution 2:
* Reassemble 1851 reads ; clr=ECR2; doOBT=no
* Asm dir:
/scratch1/bos_taurus/Assembly/2009_0312_CA/8-consensus/80.ECR2.asm
* It contains one 179,530 bp scaffold that has two contigs.
* One contig is 156,349 bp and the other one is 23,181 bp.
* The estimated gap between them is 231 bp.
show-coords ctg7180002022636-80.ECR2.filter-r.delta
1 156326 | 1 156331 | 156326 156331 | 99.99 | 179505 156349 | 87.09 99.99 | ctg7180002022636 ctg7180000000103 [CONTAINS]
156345 179505 | 21 23181 | 23161 23161 | 99.99 | 179505 23181 | 12.90 99.91 | ctg7180002022636 ctg7180000000104 [CONTAINS]
>ctg7180002022636_156327_156344 TTGTAAAAACCATCCCCT
# ~ 20 bp unaligned on ctg7180002022636 & Chr1 show-coords ctg7180002022636-Chr1.filter-r.delta | more ... 151839 156326 | 61124826 61120339* | 4488 4488 | 99.91 | 179505 157590899 | 2.50 0.00 | ctg7180002022636 Chr1 156347 157501 | 61111219* 61110064 | 1155 1156 | 99.91 | 179505 157590899 | 0.64 0.00 | ctg7180002022636 Chr1 ...
# 2 UMD2.0 ctg & 2 UMD2.0 deg in this region more Chr1.agp ... Chr1 61110064 61111482 3579 W deg0003139347 1 1419 + Chr1 61111483 61114130 3580 N 2648 fragment yes Chr1 61114131 61115490 3581 W deg0002967451 1 1360 + Chr1 61115491 61118114 3582 N 2624 fragment yes Chr1 61118115 61120117 3583 W 7180002846553 1 2003 + Chr1 61120118 61120217 3584 U 100 fragment yes Chr1 61120218 61145567 3585 W 7180003318962 1 25350 + ...
QC
Lengths:
elem <2000 >=2000 min max mean med n50 sum scf 39978* 31311 8667 316 34167202 68129 1360 8217662 2723691675 ctg 90135* 36140 53995 65 1160130 29693 5124 95988 2676390147 deg 251413 249285 2128 65 39964 1003 984 994 252279234 utg 1689033 1419729 269304 21 138676 2242 936 8213 3788090224
elem <0 0 >0 min max mean med n50 sum gaps(ca2scf) 50157 10759 3296 36102 -20 177144 929 20 34357 46620040 gaps(posmap) 50157 0 0 50157 20 177144 943 20 34065 47301528
Fragment happiness:
placed good 27263138 chaff bothChaff 2467462 placed notMated 2050084 placed oneChaff 732517 chaff oneChaff 732517 placed oneSurrogate 555510 placed bothDegen 465114 chaff notMated 434052 placed diffScaffold 369,294 * placed oneDegen 213,768 * placed badSame 96862 placed badLong 76640 placed badOuttie 41142 placed badShort 5784 placed bothSurrogate 3278
Mate happiness:
good 13631569 bothChaff 1233731 oneChaff 732517 oneSurrogate 277755 bothDegen 232557 diffScaffold 184647 oneDegen 106884 badSame 48431 badLong 38320 badOuttie 20571 badShort 2892 bothSurrogate 1639
Scaffold zero read/mate cvg regions:
elem <2000 >2000 min max mean med n50 sum read 57011 55048 1963 1 177144 913 57 29302 52084484 mate 10507 8945 1562 1 30014 996 493 2367 10466518
Scaffold 10K+ zero read/mate cvg regions (2K+ inside) (some might be a result of surrogates?):
elem <2000 >2000 min max mean med n50 sum read 51747 49878 1869 1 177144 958 38 32625 49613432 mate 1290 201 1089 15 30014 3560 3047 4017 4593586 mate(ignore seq len) 2599 1051 1548 1 72706 3321 2541 5513 8633691
Contaminant search
ctg
elem <0 >0 min max mean med n50 sum ctg 90135 0 90135 65 1160130 29693 5124 95988 2676390147
nucmer -maxmatch -l 40 -c 100 -b 10 -g 5 -d 0.05 ...
#OLD table
elem <2000 >2000 min max mean med n50 sum
Ecoli.all 71 66 5 1006 129770 4180 1127 45234 296830
UniVec_Core 120 111 9 1000 426540 7366 1130 100944 884023
OtherVec 121 112 9 1000 426540 7314 1127 100944 885022
UMD2.cont.other 98(-3) 83 15(-3) 1000 426540 13523 1190 199700 1325332 # 3 are 1000bp+ from ctg ends; these are actually "fake" contaminants
# 7 are Acinetobacter baumannii min=1059 , max=9765
total 152*(-3) 133 19*(-3) 1000 426540 10649 1150 199700 1618735
Acinetobacter 65 53 12 1013 44359 2586 1212 3847 168130 # 46 out of 65 are in the 152* set; 19 are new; 13 have lots of alignments to other contigs (probably fake contaminants)
total(new) 171*(-3) 144 27*(-3) 1000 426540 10013 1189 129770 1712376 # 65 are Acinetobacter and should be removed
cat UMD2.contaminant.other-ctg.filter-q.coords | grep Acinetobacter
UMD2.0 UMD2.5
1 285 | 285 1 | 285 285 | 99.65 | 287 8096 | 99.30 3.52 | 7180003292866_1_288 ctg7180002015457 [CONTAINED] Acinetobacter baumannii
1422 2500 | 1078 1 | 1079 1078 | 99.63 | 7098 1078 | 15.20 100.00 | 7180003313366 ctg7180001706852 [CONTAINS] Acinetobacter baumannii
2934 3940 | 1008 1 | 1007 1008 | 98.61 | 7098 1059 | 14.19 95.18 | 7180003313366 ctg7180001709709 Acinetobacter baumannii
6281 7098 | 1 818 | 818 818 | 99.76 | 7098 1553 | 11.52 52.67 | 7180003313366 ctg7180001716052 [END] Acinetobacter baumannii
1 790 | 790 1 | 790 790 | 100.00 | 1822 9765 | 43.36 8.09 | 7180003319195_8956_10778 ctg7180002015485 [BEGIN] Acinetobacter calcoaceticus
285 1981 | 1 1697 | 1697 1697 | 99.59 | 13090 1856 | 12.96 91.43 | 7180003320028 ctg7180001706656* Acinetobacter baumannii
1992 2148 | 1697 1856 | 157 160 | 98.12 | 13090 1856 | 1.20 8.62 | 7180003320028 ctg7180001706656* Acinetobacter baumannii
12210 13090 | 63 943 | 881 881 | 99.89 | 13090 2556 | 6.73 34.47 | 7180003320028 ctg7180002007423 Acinetobacter baumannii
# 7 Acinetobacter baumannii ctgs # no Serratia "best hits" # 3 mitochondrion ctgs, all < 2Kbp
Delete summary: 65 Acinetobacter ctgs + 91 contaminant ctgs <2000bp => 156 ctgs , 152 scf => 4105 reads
ctgs <2000 >2000 min max mean med n50 sum reads 156 144 12 1000 44359 1782 1150 1483 278009 4105 # all ctg
Trim summary: 12 contigs >=2000bp & 44 reads that overlap at least 10bp
ctgs <2000 >2000 min max mean med n50 sum reads 12 0 12 7072 426540 78470 45234 129770 941646 ? # all ctg 12 12 0 172 935 532 618 750 6393 44 # ctg regions
Files
/scratch1/bos_taurus/Assembly/2009_0312_CA/nucmer_ctg/TO_DELETE/ctg.delete.uid /scratch1/bos_taurus/Assembly/2009_0312_CA/nucmer_ctg/TO_DELETE/scf.delete.uid /scratch1/bos_taurus/Assembly/2009_0312_CA/nucmer_ctg/TO_TRIM/ctg.trim.uid
ctg 5'&3'
elem <0 >0 min max mean med n50 sum ctg53 180044 0 180044 65 598 300 300 300 54033229
nucmer -maxmatch -l 17 -c 35 ...
#ctgEnds #ctgs min max mean med n50 sum Ecoli.all 180 149 300 300 300 300 300 54000 UniVec_Core 312 277 300 300 300 300 300 93600 OtherVec 1211 1167 300 553 300 300 300 363989 UMD2.cont.other 15689 14693 257 598 300 300 300 4712162
deg
nucmer -maxmatch -l 40 -c 100 -b 10 -g 5 -d 0.05 ...
elem <2000 >2000 min max mean med n50 sum
Ecoli.all 387 387 0 131 1099 756 806 835 292892
UniVec_Core 569 569 0 101 1115 763 822 843 434400
OtherVec 579 579 0 101 1115 752 819 840 435549
UMD2.cont.other 539 539 0 131 1483 792 838 873 427408
total 810* 810 0* 101 1483 784 838 869 63547
Scaffolds vs UMD2.0 chromosome alignments
Directory:
/scratch1/bos_taurus/Assembly/2009_0312_CA/nucmer_scf
Depening on the ref/qry seq and nucmer parameters, the number of unaligned gaps in UMD2.0 can vary between:
101M: REF=Chr, QRY=scf, nucmer -l 100 -c 500 6M: REF=ChrPlaced, QRY=scf-deg, nucmer -maxmatch -l 50 -c 250
nucmer -l 100 -c 500
Chr-scf.summary
elem <2000 >2000 min max mean med n50 sum Chr-scf.qry_hits 32901 24546 8355 723 34167202 82494 1405 8217662 2714164100 Chr-scf.qry_nohits 7077 6765 312 316 12006 1346 1239 1291 9527056 Chr-scf.10K.qry_hits2+ 574 0 574 10308 34167202 4006753 1887309 9586144 2299876795 Chr-scf.0cvg 144712 125933 18779 1 102265 900 178 2968 130248709 Chr-scf.0cvg.clean 148556 143283 5273 1 39625 683 280 1363 101526883(101M)
Chr-scf-deg.summary
elem <2000 >2000 min max mean med n50 sum Chr-scf-deg.qry_hits 210225 199781 10444 501 34167202 13785 1007 7328685 2898141592 Chr-scf-deg.qry_nohits 81166 80815 351 65 12006 958 972 989 77828798 Chr-scf-deg.10K.qry_hits2+ 574 0 574 10308 34167202 4006753 1887309 9586144 2299876795 Chr-scf-deg.0cvg 175952 168553 7399 1 22067 445 120 1329 78433265 Chr-scf-deg.0cvg.clean 133809 132381 1428 1 20512 371 124 1101 49711440(49M)
ChrPlaced-scf.summary
elem <2000 >2000 min max mean med n50 sum ChrPlaced-scf.qry_hits 19488 13112 6376 723 34167202 137773 1527 8428844 2684927057 ChrPlaced-scf.qry_nohits 20490 18199 2291 316 192648 1891 1276 1569 38764099 ChrPlaced-scf.10K.qry_hits2+ 139 0 139 10486 31959312 6786671 4979278 12956086 943347316 ChrPlaced-scf.0cvg 76271 71816 4455 1 102265 568 179 1710 43339413 ChrPlaced-scf.0cvg.clean 72865 70541 2324 1 39625 356 94 1413 25951987(25M)
ChrPlaced-scf-deg.summary
elem <2000 >2000 min max mean med n50 sum ChrPlaced-scf-deg.qry_hits 130125 122000 8125 501 34167202 21530 1009 7515049 2801670853 ChrPlaced-scf-deg.qry_nohits 161266 158596 2670 65 192648 1080 987 1012 174299537 ChrPlaced-scf-deg.10K.qry_hits2+ 139 0 139 10486 31959312 6786671 4979278 12956086 943347316 ChrPlaced-scf-deg.0cvg 79041 76374 2667 1 22067 395 157 948 31251753 ChrPlaced-scf-deg.0cvg.clean 69012 68271 741 1 20512 200 81 592 13864328(13M)
nucmer -maxmatch -l 100 -c 500
Dir:
/scratch1/bos_taurus/Assembly/2009_0312_CA/nucmer_scf.2
ChrPlaced-scf-deg.summary
elem <2000 >2000 min max mean med n50 sum ChrPlaced-scf-deg.qry_hits 130510 122377 8133 501 34167202 21470 1009 7515049 2802100771 ChrPlaced-scf-deg.qry_nohits 160881 158219 2662 65 192648 1080 986 1012 173869619 ChrPlaced-scf-deg.10K.qry_hits2+ 120 0 120 20022 31959312 7587296 5639522 13010806 910475551 ChrPlaced-scf-deg.0cvg 82159 80425 1734 1 7002 321 145 647 26444796 ChrPlaced-scf-deg.0cvg.clean 111645 111546 99 1 6248 81 13 272 9057424(9M)
nucmer -maxmatch -l 50 -c 250
Dir:
/scratch1/bos_taurus/Assembly/2009_0312_CA/nucmer_scf.3
ChrPlaced-scf-deg.summary
elem <2000 >2000 min max mean med n50 sum ChrPlaced-scf-deg.qry_hits 204673 195653 9020 251 34167202 14088 1005 7329288 2883625712 ChrPlaced-scf-deg.qry_nohits 86718 84943 1775 65 192648 1064 970 1007 92344678 ChrPlaced-scf-deg.10K.qry_hits2+ 148 0 148 10486 31959312 6814292 5135095 12792673 1008515300 ChrPlaced-scf-deg.0cvg 86085 84614 1471 1 4565 279 123 557 24101912 ChrPlaced-scf-deg.0cvg.clean 113796 113791 5 1 2822 59 7 176 6714556(6M)
nucmer -maxmatch -l 50 -c 250 ; delta-fileter -q
Dir:
/scratch1/bos_taurus/Assembly/2009_0312_CA/nucmer_scf.3
ChrPlaced-scf-deg.filter-q.summary
elem <2000 >2000 min max mean med n50 sum
ChrPlaced-scf-deg.qry_hits 204673 195653 9020 251 34167202 14088 1005 7329288 2883625712
ChrPlaced-scf-deg.qry_nohits 86718 84943 1775 65 192648 1064 970 1007 92344678
ChrPlaced-scf-deg.10K.qry_hits2+ 118 0 118 20022 31959312 7633834 5639522 13010806 900792422
ChrPlaced-scf-deg.0cvg 77864 73686 4178 1 28711 529 181 1523 41240419
ChrPlaced-scf-deg.0cvg.clean 74172 72150 2022 1 28331* 321 89 1415 23852952(23M)
Max gap is 28331; Duplicate region in UMD2.0?
ChrPlaced-scf-deg.coords 70739616 70767946 | 2993389 2965048 | 28331 28342 | 99.17 | 85187327 19514159 | 0.03 0.15 | Chr15 scf7180002041107 70768054 70808299 | 3005305 2965048 | 40246 40258 | 99.56 | 85187327 19514159 | 0.05 0.21 | Chr15 scf7180002041107 => ChrPlaced-scf-deg.filter-q.coords 70768054 70808299 | 3005305 2965048 | 40246 40258 | 99.56 | 85187327 19514159 | 0.05 0.21 | Chr15 scf7180002041107
Markers
ALL:
head /fs/szasmg3/bos_taurus/UMD_Freeze2.5/markers/markers_contigs_Art.txt Marker Chr_BTA Pos(K) Pos_from Pos_to UMD_Ctg_Pos Match_Len %IDY %Match UMD_Ctg_name BZ945871 1 47501 1 95001 7622 515 100.00 99.61 ctg7180002007845 BZ953651 1 80001 47501 112501 10786 700 99.57 100.00 ctg7180002026484 CC504788 1 118751 80001 157501 54583 862 100.00 100.00 ctg7180002026483 CC484491 1 123751 90001 157501 50169 77 98.72 100.00 ctg7180002026482 CZ415082 1 125001 92501 157501 75850 507 99.21 99.80 ctg7180002026483 CC475154 1 130001 97501 162501 40013 666 99.25 100.00 ctg7180002026482 CC561114 1 182501 145001 220001 1130 709 99.02 100.00 ctg7180002026482 CC578374 1 190001 155001 225001 170145 647 100.00 100.00 ctg7180002026481 BZ911787 1 278751 232501 325001 na na na na na ...
- 126,014 markers & 90,135 ctgs total
- 107,271 markers align to 31,407 ctg & 2640 scf:
- 85% of the markers align to 85% of the ctg sequence
- avg distance between markers is 25Kbp
- 188 questionable ctgs & 219 questionable scf (2 out of 4 disagreeing markers)
UNIQ:
head /fs/szasmg3/bos_taurus/UMD_Freeze2.5/markers/markers_contigs_Art.unique_only.txt | p 'print " ",$_;' Marker Chr_BTA Pos(Kbp) CI_Pos_from CI_Pos_to UMD_Scaff_Pos Match_Len %IDY %Matched UMD_Scaff_name BZ945871 1 52251 1 104501 na na na na na BZ953651 1 88001 52251 123751 na na na na na CC504788 1 130626 88001 173251 54583 862 100.00 100.00 ctg7180002026483 CC484491 1 136126 99001 173251 50169 77 98.72 100.00 ctg7180002026482 CZ415082 1 137501 101751 173251 na na na na na CC475154 1 143001 107251 178751 40013 666 99.25 100.00 ctg7180002026482 CC561114 1 200751 159501 242001 1130 709 99.02 100.00 ctg7180002026482 CC578374 1 209001 170501 247501 170145 647 100.00 100.00 ctg7180002026481 BZ911787 1 306626 255751 357501 na na na na na ...
- 93,508 markers align to 28,752 ctgs & 1,476 scf:
- 109 questionable ctgs & 153 questionable scf (2 out of 4 disagreeing markers)
Scripts:
~/bin/marker2pos.pl markers_contigs_Art.unique_only.txt | sed 's/ctg//' | sort -nk1 -nk2 > markers_ctg.pos ~/bin/translatePosMap.pl markers_ctg.pos bt.posmap.ctgscf | ~/bin/tab2tab.pl > markers_scf.pos
Ctg vs markers summary:
#ctg <10000 >10000 min max mean med n50 sum file ctg (all) 90135* 51024 39111 65 1160130 29693 5124 95988 2676390147 no markers 58728 48324 10404 65 322949* 6573 1597 21989 386064754 markers from 1+ Chr 31407 2700 28707 442 1160130 72924 52693 111252 2290325393 markers_ctg.Chr.count markers from 2+ Chr 2987 25 2962 1002 1160130 132480 104807 179692 395718221 markers_ctg.Chr.count2+ 2+ markers from 2+ Chr 26 0 26* 15228 604155 221354 192182 298848 5755227 markers_ctg.Chr.count2.2+ 2+ adjacent markers from 2+ Chr 15 0 15** 15228 368879 202728 194749 294623 3040932 markers_ctg.Chr.count2+a
Scf vs markers summary:
#scf <10000 >10000 min max mean med n50 sum scf(all) 39978* 37135 2843 316 34167202 68129 1360 8217662 2723691675 no markers 37338 36038 1300 316 754615* 2601 1336 3957 97140879 markers from 1+ Chr 2640 1097 1543 1000 34167202 994905 16220 8661690 2626550796 markers_scf.Chr.count markers from 2+ Chr 552 10 542 1002 34167202 4526814 2714036 9167014 2498801557 markers_scf.Chr.count2+ 2+ markers from 2+ Chr 212 0 212* 15228 34167202 8579232 7358307 10521496 1818797327 markers_scf.Chr.count2.2+ 2+ adjacent markers from 2+ Chr 38 0 38** 15228 25078118 8419544 7176534 13458592 319942681 markers_scf.Chr.count2+a
212* scaffolds
scf_id scf_len #Chr/2+markers #/Chr/2+adjmarkers reads 7180002041381 31959312 15 0 469503 7180002041358 25078118 13 2 291163 7180002041386 21280754 12 0 ...
scf7180002041381.1
- no low cvg regions in the middle
- 1281 markers: 1231 on Chr4, 12 on Ch11
#1- mate cvg regions: at the ends !!! #scfid begin end scf_len cvg_len cvg 7180002041381 1 1173 31959312 1173 0 7180002041381 1174 1454 31959312 281 1 7180002041381 31959139 31959312 31959312 174 1
scf7180002041358.2
- one low cvg region & real break
- 1111 markers: 869 on Chr14, 193 on Chr26, ...
#1- mate cvg regions: middle #scfid begin end scf_len cvg_len cvg 7180002041358 20970531 20970827 25078118 297 1 7180002041358 20970828 20970949 25078118 122 0 7180002041358 20970950 20971112 25078118 163 1
# markers in the regions #scfid begin end makerid Chr 7180002041358 20964100 20964829 BZ839784 14 7180002041358 21002368 21003219 CC527932 26
scf7180002041386.3
- one low cvg region but no markers in that region
- 939 markers: 902 on Chr24, 3 on Chr6 ...
#1- mate cvg regions: at the ends #scfid begin end scf_len cvg_len cvg 7180002041386 26382 27557 21280754 1176 1 7180002041386 27558 27607 21280754 50 0 7180002041386 27608 28031 21280754 424 1
...
scf7180002041368.128
- one low cvg region
#scfid begin end scf_len cvg_len cvg 7180002041368 1356953 1357221 1404851 269 1
- markers:Chr7 & Chr5 at 3'
#scfid begin end markerid Chr 7180002041368 10436 11033 BZ840669 7 ... 7180002041368 1278950 1279444 CZ404867 7 7180002041368 1319489 1320264 CC593534 7 7180002041368 1344001 1344347 BZ885865 7 * 7180002041368 1371526 1372308 BZ867572 5 * 7180002041368 1386933 1387750 CC534893 5
- subassembly: extract all reads & extra mates(265) and reassemble
- Msg Counts
DST 40 FRG 13762 LKG 6482
- qc
[Top5Scaffolds=contigs,size,span,avgContig,avgGap,EUID] 0=37,1211999,1358964,32757,4082,7180000000571 1=2,53844,53824,26922,-20,7180000000566 2=4,50178,50238,12544,20,7180000000570 3=1,28553,28553,28553,0,7180000000568 4=1,17195,17195,17195,0,7180000000569 total=45,1361769,1508774,30262,3675
show-coords scf7180002041368-7180002041368.update.scf.filter-q.delta
... 1269662 1338725 | 69065 1 | 69064 69065 | 99.97 | 1404851 1359628 | 4.92 5.08 | scf7180002041368 scf7180000000571 1340619 1341912 | 1 1295 | 1294 1295 | 98.77 | 1404851 1295 | 0.09 100.00 | scf7180002041368 scf7180000000574 [CONTAINS] 1345570 1347105 | 1537 1 | 1536 1537 | 98.18 | 1404851 1537 | 0.11 100.00 | scf7180002041368 scf7180000000572 [CONTAINS] 1347683 1356952 | 12617 3352 | 9270 9266 | 99.85 | 1404851 12617 | 0.66 73.44 | scf7180002041368 scf7180000000567 1356901 1361861 | 1767 6728 | 4961 4962 | 99.96 | 1404851 50318 | 0.35 9.86 | scf7180002041368 scf7180000000570 ...
scf7180002041061.187
- infoseq
scf7180002041061 509987 39.73
- 22 markers
#scfid begin end makerid Chr 7180002041061 2061 2904 BZ847430 15 7180002041061 8979 9756 CC481592 15 7180002041061 10856 11485 BZ839377 15 7180002041061 47578 48186 BZ836581 15 7180002041061 80485 81237 CC553472 15 7180002041061 117811 118488 CC918533 15 7180002041061 151253 152009 CC477055 15 7180002041061 213959 214640 CC572066 15 * 7180002041061 236304 236479 CC550436 29 * 7180002041061 242590 243462 BZ848041 29 7180002041061 267880 268614 BZ877402 29 7180002041061 268891 269493 CC580499 29 7180002041061 282248 282861 BZ885584 29 7180002041061 295221 295987 CC572941 29 7180002041061 337785 338517 CC923898 29 7180002041061 338464 339092 BZ921430 29 7180002041061 341219 341971 CZ415932 29 7180002041061 382509 383281 CC581748 29 7180002041061 387548 388176 CC572064 29 7180002041061 415263 415788 BZ878185 29 7180002041061 470138 470773 CC558303 29 7180002041061 493006 493794 CC565740 29
- mate happiness:
cat 7180002041061.posmap.mates | cut -f3 | count.pl good 2501 diffScaffold 86 oneSurrogate 69 oneChaff 40 oneDegen 11 badSame 9 badOuttie 4 badLong 4
- low cvg region:
#scfid begin end scf_len cvg_len cvg 7180002041061 232849 232871 509987 23 1 #it is in the middle of a unitig intersectPos.pl -i 1 7180002041061.posmap.utgscf 7180002041061.posmap.frgscf.10K.2K.mate_cvg.1- 7180000849321 7180002041061 229782 238910 r
- subassembly: extract all reads & mates(139) and reassemble
- Msg Counts
DST 38 FRG 5769 LKG 2724
Dirs:
/scratch1/bos_taurus/Assembly/2009_0312_CA/markers/scf7180002041061.187.mates/asm.ECR2 /scratch1/bos_taurus/Assembly/2009_0312_CA/markers/scf7180002041061.187.mates/asm.ECR2.mates
Qc stats:
asm.ECR2 asm.ECR2.bog ... [Top5Scaffolds=contigs,size,span,avgContig,avgGap,EUID] 0 3,277300,277925,92433,312,7180000000278 5,279938,399837,55988,29975,7180000000155 1 6,234445,280546,39074,9220,7180000000277 6,234463,280485,39077,9204,7180000000156 2 1,1630,1630,1630,0,7180000000276 1,1799,1799,1799,0,7180000000154 3 NA 1,1674,1674,1674,0,7180000000153 4 NA 1,1183,1183,1183,0,7180000000158 total 10,513375,560101,51338,6675 14,519057,684978,37076,18436
- Some of the 139 mates assemble into scaffolds
- There is slightly more sequence in the bog assembly
- Mean/Max bog utg size is twice larger than default utg (scf, ctg sizes are about the same)
- align asm.ECR2 scaffolds to scf7180002041061
- 2919 bp at the 3' end of the new scf7180000000277 don't align
- 1990 bp at the 3' end of the new scf7180000000278 don't align
- new scf7180000000277 & new scf7180000000278 align for ~ 1058bp
- most of the mated read added assembled at new scf7180000000277 & new scf7180000000278 3'
nucmer -l 100 -c 500 ../scf7180002041061.fasta 9-terminator/7180002041061.update.scf.fasta -p scf7180002041061-7180002041061.update.scf
delta-filter -q scf7180002041061-7180002041061.update.scf.delta > scf7180002041061-7180002041061.update.scf.filter-q.delta
show-coords scf7180002041061-7180002041061.update.scf.filter-q.delta
1 46099 | 1 46099 | 46099 46099 | 99.99 | 509987 280626 | 9.04 16.43 | scf7180002041061 scf7180000000277
46273 48914 | 46120 48761 | 2642 2642 | 100.00 | 509987 280626 | 0.52 0.94 | scf7180002041061 scf7180000000277
50810 56904 | 95425 101521 | 6095 6097 | 99.43 | 509987 280626 | 1.20 2.17 | scf7180002041061 scf7180000000277
57301 233303 | 101699 277707 | 176003 176009 | 99.97 | 509987 280626 | 34.51 62.72 | scf7180002041061 scf7180000000277 *
232245 444418 | 275975 63802 | 212174 212174 | 99.99 | 509987 277965 | 41.60 76.33 | scf7180002041061 scf7180000000278 *
444964 469975 | 63156 38151 | 25012 25006 | 99.97 | 509987 277965 | 4.90 9.00 | scf7180002041061 scf7180000000278
449967 479221 | 58152 28900 | 29255 29253 | 99.96 | 509987 277965 | 5.74 10.52 | scf7180002041061 scf7180000000278
480298 509987 | 29688 1 | 29690 29688 | 99.99 | 509987 277965 | 5.82 10.68 | scf7180002041061 scf7180000000278
scf7180002041163.214
- infoseq
scf7180002041234 10336067 41.93
- Markers
#scfid begin end makerid Chr ... 7180002041163 770783 771508 CC562874 30 * 7180002041163 791050 791484 BZ887794 11 * ...
- Markers on different contigs !!!
cat 7180002041163.posmap.ctgscf .. 7180001926720 7180002041163 750392 787668 f 7180001926721 7180002041163 787911 815738 f ...
- subassembly: extract all reads & extra mates(293) and reassemble
- Msg Counts
DST 39 FRG 17287 LKG 8113
- qc
[Top5Scaffolds=contigs,size,span,avgContig,avgGap,EUID] 0=54,1742235,1760846,32264,351,7180000000507 1=1,6224,6224,6224,0,7180000000506 2=1,1472,1472,1472,0,7180000000508 total=56,1749931,1768542,31249,351
- alignments :
show-coords scf7180002041163-7180002041163.update.scf.filter-q.delta 1748 7972 | 6224 1 | 6225 6224 | 99.84 | 1757217 6224 | 0.35 100.00 | scf7180002041163 scf7180000000506 [CONTAINS] 4012 5490 | 1 1470 | 1479 1470 | 98.85 | 1757217 1472 | 0.08 99.86 | scf7180002041163 scf7180000000508 [CONTAINS] 4567 18273 | 1 13697 | 13707 13697 | 99.92 | 1757217 1761853 | 0.78 0.78 | scf7180002041163 scf7180000000507 ... 750393 787668 | 757092 794367 | 37276 37276 | 100.00 | 1757217 1761853 | 2.12 2.12 | scf7180002041163 scf7180000000507 787912 815738 | 794644 822455 | 27827 27812 | 99.81 | 1757217 1761853 | 1.58 1.58 | scf7180002041163 scf7180000000507 ... 1755928 1757217 | 1760564 1761853 | 1290 1290 | 100.00 | 1757217 1761853 | 0.07 0.07 | scf7180002041163 scf7180000000507
Manually curated
Markers within 50K of a low mate cvg region
- 13 scaffolds (22 before)
- 14 breaks : 9 on the same contig , 2 on adjacent contigs , 3 on non adjacent contigs
- File
/scratch1/bos_taurus/Assembly/2009_0312_CA/markers/scfproblems.mates//scfproblems
- scfproblems.markers.txt
#scfid begin end markerid Chr(break) comment 7180002040911 2210713 2211037 CZ405316 29 half Chr30, half Chr29 ; SAME ctg 7180002041061 213959 214640 CC572066 15 half Chr29, half Chr15; SAME ctg 7180002041103 5296272 5297045 CC587675 15 Chr18 & Chr15 at 3'; diff ctg (2 ctgs in between) 7180002041107 7302203 7302874 CC906829 15 half Chr9, half Chr15; SAME ctg 7180002041200 4920833 4921695 BZ877250 20 half Chr20, half Chr2; SAME ctg 7180002041216 307121 307653 CL609365 5 Chr5 at 5' & Chr15; SAME ctg 7180002041259 4315523 4316178 BZ922220 3 Chr5, Chr3, Chr24 !!! ; diff ctgs (4 ctgs in between) 7180002041281 2884855 2885652 CC538348 19 half Chr19, half Chr28 ; SAME ctg 7180002041315 2196754 2197298 BZ838379 1 Chr13 & Chr1 at 3'; diff ctg (2 ctgs in between) 7180002041325 2597590 2598283 CC472212 10 half Chr10, half Chr5; SAME ctg 7180002041348 3028112 3028921 CC531996 8 half Chr8, half Chr7; SAME ctg 7180002041356 6828214 6828925 CC531427 16 half Chr16, half Chr25; SAME ctg 7180002041358 21002368 21003219 CC527932 26 Chr14 & Chr26 at 3' (long chunck); diff ctg
Details:
#scfid begin end markerid Chr #ctgid begin end markerid Chr 7180002040911 2189844 2190490 BZ923031 30 7180001730191 42851 43497 BZ923031 30 * 7180002040911 2210713 2211037 CZ405316 29 7180001730191 63720 64044 CZ405316 29 * -- 7180002041061 213959 214640 CC572066 15 7180001852904 29927 30608 CC572066 15 * 7180002041061 236304 236479 CC550436 29 7180001852904 52272 52447 CC550436 29 * -- 7180002041103 5220856 5221407 CG984741 18 7180001854649 21126 21677 CG984741 18 * 7180002041103 5296272 5297045 CC587675 15 7180001854651 837 1610 CC587675 15 * -- 7180002041107 7302203 7302874 CC906829 15 7180001855003 77842 78513 CC906829 15 ** 7180002041107 7311399 7312254 CC479102 9 7180001855003 87038 87893 CC479102 9 ** -- 7180002041200 4940131 4940949 CC500137 20 7180002002029 31935 32753 CC500137 20 ** 7180002041200 4956412 4957105 CZ428497 2 7180002002029 48216 48909 CZ428497 2 ** -- 7180002041216 307121 307653 CL609365 5 7180002003578 6871 7403 CL609365 5 * 7180002041216 310832 311546 CL865591 15 7180002003578 10582 11296 CL865591 15 * -- 7180002041259 4224828 4225527 CC920177 5 7180002012718 112862 113561 CC920177 5 * 7180002041259 4315523 4316178 BZ922220 3 7180002012722 19376 20031 BZ922220 3 * -- 7180002041259 6220638 6221268 BZ869532 3 7180002012752 199799 200429 BZ869532 3 * 7180002041259 6239728 6240375 CZ413142 24 7180002012753 2600 3247 CZ413142 24 * -- 7180002041281 2927406 2928016 CC573399 19 7180002018361 74539 75149 CC573399 19 ** 7180002041281 2938896 2939696 CC513914 28 7180002018361 86029 86829 CC513914 28 ** -- 7180002041315 2152291 2153097 BZ836343 13 7180001862977 3397 4203 BZ836343 13 * 7180002041315 2196754 2197298 BZ838379 1 7180002024308 7255 7799 BZ838379 1 * -- 7180002041325 2608389 2609213 CC506736 10 7180002025880 237956 238780 CC506736 10 ** 7180002041325 2638591 2639207 CC770009 5 7180002025880 268158 268774 CC770009 5 ** -- 7180002041348 3044263 3044902 BZ872906 8 7180002030033 152672 153311 BZ872906 8 ** 7180002041348 3092547 3093121 BZ924509 7 7180002030033 200956 201530 BZ924509 7 ** -- 7180002041356 6828214 6828925 CC531427 16 7180001722964 12571 13282 CC531427 16 * 7180002041356 6832720 6833356 CC494876 25 7180001722964 17077 17713 CC494876 25 * -- 7180002041358 20964100 20964829 BZ839784 14 7180001723456 140919 141648 BZ839784 14 * 7180002041358 21002368 21003219 CC527932 26 7180001723457 15983 16834 CC527932 26 *
- scfproblems.low_mate_cvg.txt
#scfid begin end ctglen len mate_cvg 7180002040911 2205494 2205582 3008363 89 1 7180002041061 232849 232871 509987 23 1 7180002041103 5285314 5285479 5341215 166 1 7180002041107 7306552 7306653 19514159 102 1 7180002041200 4954073 4954180 16995932 108 1 7180002041216 307614 307741 1444165 128 0 7180002041259 4313297 4313311 15600612 15 0 7180002041281 2933371 2933618 12907599 248 1 7180002041315 2193142 2193167 2232736 26 1 7180002041325 2623956 2624190 11153784 235 1 7180002041348 3064633 3064750 19180127 118 1 7180002041356 6831454 6831811 14697197 358 1 7180002041358 20970950 20971112 25078118 163 1
No low cvg regions
- 9 scaffolds
- 14 breaks : 5 on the same contig , 6 on adjacent contigs , 3 on non adjacent contigs
7180002040844 : 3 consecutive Chr30 markers (in the middle); mate_cvg > 18 ; markers CC576837,CC516738,CC543771; SAME ctg 7180002041163 : 5' Chr30, 3' Chr11 ; mate_cvg~=10; marker BZ887794 ; diff ctg (1 ctg in between) 7180002041234 : 5' Chr10, 3' Chr5; mate_cvg=10; marker BZ889975; scf=1757217bp; diff ctg 7180002041235 : 2 consecutive Chr14 markers; mate_cvg=15..20; markers CC561837 & CC585677; NOT uniq; SAME ctg 7180002041279 : 5 consecutive Chr23 markers; markers CC472696,CC522963; cvg=9..18; diff ctg 7180002041306 : 5' Chr14, 3' Chr6; marker CC549871; cvg=2; diff ctg (22 ctgs in between) 7180002041308 : 2 consecutive Chr20 markers; markers CC571631 & BZ832318; cvg=20; SAME/diff ctg 7180002041321 : 5' Chr2, 3' Chr15 ; marker CC513377 cvg=10; diff ctg (1 ctgs in between) 7180002041350 : 2 consecutive Chr3 markers ; markers BZ837387 & CC571149; cvg 17..26; NOT uniq; SAME/diff ctg
Details:
#scfid begin end markerid Chr #ctgid begin end markerid Chr 7180002040844 8369147 8369989 CC521620 17 7180001727899 132573 133415 CC521620 17 ** 7180002040844 8437707 8438324 CC576837 30 7180001727899 201133 201750 CC576837 30 ** ... 7180002040844 8467493 8468096 CC543771 30 7180001727899 230919 231522 CC543771 30 * 7180002040844 8522732 8523598 CC513544 17 7180001727901 34970 35836 CC513544 17 * --- 7180002041163 770783 771508 CC562874 30 7180001926720 20391 21116 CC562874 30 * 7180002041163 791050 791484 BZ887794 11 7180001926721 3139 3573 BZ887794 11 * -- 7180002041234 3856848 3857673 BZ889975 10 7180002010181 56608 57433 BZ889975 10 * 7180002041234 3949893 3950652 CC509477 5 7180002010182 28610 29369 CC509477 5 * -- 7180002041235 1117908 1118686 CC579933 21 7180002010285 4086 4864 CC579933 21 ** 7180002041235 1118939 1119675 CC561837 14 7180002010285 5117 5853 CC561837 14 ** 7180002041235 1158300 1159022 CC585677 14 7180002010285 44478 45200 CC585677 14 ** 7180002041235 1164296 1164891 BZ924510 21 7180002010285 50474 51069 BZ924510 21 ** -- 7180002041279 6755839 6756519 BZ849919 1 7180002018195 417990 418670 BZ849919 1 * 7180002041279 6786086 6786957 CC472696 23 7180002018196 12703 13574 CC472696 23 * ... 7180002041279 6866378 6866890 CC522963 23 7180002018196 92995 93507 CC522963 23 * 7180002041279 6911185 6911897 CC574255 1 7180002018197 15411 16123 CC574255 1 * -- 7180002041306 248713 249457 CC503129 14 7180002022875 59534 60278 CC503129 14 * 7180002041306 910997 911554 BZ839769 6 7180002022892 21939 22496 BZ839769 6 * -- 7180002041308 1919399 1920196 BZ883381 18 7180002023013 57093 57890 BZ883381 18 * 7180002041308 1958658 1959168 CC571631 20 7180002023014 29833 30343 CC571631 20 * 7180002041308 1963333 1964083 BZ832318 20 7180002023014 34508 35258 BZ832318 20 ** 7180002041308 2009574 2010367 CC499423 18 7180002023014 80749 81542 CC499423 30 ** -- 7180002041321 4820048 4820542 BZ846646 2 7180002025240 77429 77923 BZ846646 2 * 7180002041321 4908631 4909160 CC513377 15 7180002025242 13302 13831 CC513377 15 * -- 7180002041350 2423855 2424205 CG983886 13 7180002030523 183644 183994 CG983886 13 * 7180002041350 2446763 2447125 BZ837387 3 7180002030524 9191 9553 BZ837387 3 * 7180002041350 2457417 2457704 CC571149 3 7180002030524 19845 20132 CC571149 3 ** 7180002041350 2486474 2487050 CC490214 13 7180002030524 48902 49478 CC490214 13 **
Scaffold splitting
Before:
- 22 scaffolds
- 28 breaks : 14 on the same contig , 8 on adjacent contigs , 6 on non adjacent contigs
Now:
- 14 scaffolds
- 15 breaks : 8 on the same contig , 3 on adjacent contigs , 4 on non adjacent contigs
Scaffold to break
nl scfid breaks
#1 7180002040844 2
2 7180002040911 1
3 7180002041061 1
#4 7180002041103 1
5 7180002041107 1
#6 7180002041163 1
7 7180002041200 1
#8 7180002041216 1
9 7180002041234 1
#10 7180002041235 2
11 7180002041259 2
#12 7180002041279 2
13 7180002041281 1
14 7180002041306 1
#15 7180002041308 2
16 7180002041315 1
17 7180002041321 1
18 7180002041325 1
19 7180002041348 1
#20 7180002041350 2
21 7180002041356 1
22 7180002041358 1
Contigs to break
nl ctgid
1 7180001722964
2 7180001723456
3 7180001730191
4 7180001852904
5 7180001855003
6 7180002002029
7 7180002010182
8 7180002012722
9 7180002012752
10 7180002018361
11 7180002024308
12 7180002025240
13 7180002025880
14 7180002030033
Marker pairs
nl scfid begin1 end2 markerid1 Chr1 markerid2 Chr2 ctg1 ctg2 dist(end2-begin1) dist(ctg2-ctg1)
#1 7180002040844 8369147 8438324 CC521620 17 CC576837 30 7180001727899 7180001727899 69177 0
#2 7180002040844 8467493 8523598 CC543771 30 CC513544 17 7180001727899 7180001727901 56105 2
3 7180002040911 2189844 2211037 BZ923031 30 CZ405316 29 7180001730191 7180001730191 21193 0
4 7180002041061 213959 236479 CC572066 15 CC550436 29 7180001852904 7180001852904 22520 0
#5 7180002041103 5220856 5297045 CG984741 18 CC587675 15 7180001854649 7180001854651 76189 2
6 7180002041107 7302203 7312254 CC906829 15 CC479102 9 7180001855003 7180001855003 10051 0
#7 7180002041163 770783 791484 CC562874 30 BZ887794 11 7180001926720 7180001926721 20701 1
8 7180002041200 4940131 4957105 CC500137 20 CZ428497 2 7180002002029 7180002002029 16974 0
#9 7180002041216 307121 311546 CL609365 5 CL865591 15 7180002003578 7180002003578 4425 0
10 7180002041234 3856848 3950652 BZ889975 10 CC509477 5 7180002010181 7180002010182 93804 1
#11 7180002041235 1117908 1119675 CC579933 21 CC561837 14 7180002010285 7180002010285 1767 0
#12 7180002041235 1158300 1164891 CC585677 14 BZ924510 21 7180002010285 7180002010285 6591 0
13 7180002041259 4224828 4316178 CC920177 5 BZ922220 3 7180002012718 7180002012722 91350 4
14 7180002041259 6220638 6240375 BZ869532 3 CZ413142 24 7180002012752 7180002012753 19737 1
#15 7180002041279 6755839 6786957 BZ849919 1 CC472696 23 7180002018195 7180002018196 31118 1
#16 7180002041279 6866378 6911897 CC522963 23 CC574255 1 7180002018196 7180002018197 45519 1
17 7180002041281 2927406 2939696 CC573399 19 CC513914 28 7180002018361 7180002018361 12290 0
18 7180002041306 248713 911554 CC503129 14 BZ839769 6 7180002022875 7180002022892 662841 22
#19 7180002041308 1919399 1959168 BZ883381 18 CC571631 20 7180002023013 7180002023014 39769 1
#20 7180002041308 1963333 2010367 BZ832318 20 CC499423 18 7180002023014 7180002023014 47034 0
21 7180002041315 2152291 2197298 BZ836343 13 BZ838379 1 7180001862977 7180002024308 45007 2*
22 7180002041321 4820048 4909160 BZ846646 2 CC513377 15 7180002025240 7180002025242 89112 2
23 7180002041325 2608389 2639207 CC506736 10 CC770009 5 7180002025880 7180002025880 30818 0
24 7180002041348 3044263 3093121 BZ872906 8 BZ924509 7 7180002030033 7180002030033 48858 0
#25 7180002041350 2423855 2447125 CG983886 13 BZ837387 3 7180002030523 7180002030524 23270 1
#26 7180002041350 2457417 2487050 CC571149 3 CC490214 13 7180002030524 7180002030524 29633 0
27 7180002041356 6828214 6833356 CC531427 16 CC494876 25 7180001722964 7180001722964 5142 0
28 7180002041358 20964100 21003219 BZ839784 14 CC527932 26 7180001723456 7180001723457 39119 1
Break intervals
#scfid begin end scflen len allread goodmate badmate Chr1 Chr2 mark1 mark2 ctgid
#1 7180002040844 8412735 8412739 9565303 5 1 12 15 17 30 357 3
#2 7180002040844 8485384 8485869 9565303 486 0 12 4 30 17 3 53
3 7180002040911 2205044 2205247 3009964 204 3 0 8 30 29 44 34 7180001730191
4 7180002041061 232819 232843 510915 25 5 0 23 15 29 8 14 7180001852904
#5 7180002041103 5295117 5295435 5342873 319 0 11 2 18 15 221 4
6 7180002041107 7306552 7306620 19515586 69 2 0 11 15 9 320 549 7180001855003
#7 7180002041163 787669 787911 1752998 243 0 10 0 30 11 16 8
8 7180002041200 4954103 4954180 16991478 78 1 0 27 20 2 219 532 7180002002029
#9 7180002041216 309331 309337 1435859 7 1 4 3 5 15 7 50
10 7180002041234 3939493 3939548 10292558 56 1 0 20 10 5 160 227 7180002010182
#11 7180002041235 1117752 1117987 1742396 236 5 20 2 21 14 43 2
#12 7180002041235 1161294 1161363 1742396 70 3 16 0 14 21 2 32
13 7180002041259 4313381 4313547 15594966 167 1 0 12 5 3 193 75 7180002012722
14 7180002041259 6231717 6231762 15594966 46 8 4 18 3 24 75 406 7180002012752
#15 7180002041279 6773364 6773383 9784709 20 0 4 14 1 23 292 5
#16 7180002041279 6894603 6894618 9784709 16 0 4 16 23 1 5 141
17 7180002041281 2931877 2931885 12908817 9 3 5 10 19 28 83 441 7180002018361
18 7180002041306 267445 267446 4484402 2 0 2 14 14 6 49 141 7180002022875,7180002005935
#19 7180002041308 1928806 1928825 23070077 20 0 10 0 18 20 63 2
#20 7180002041308 1977759 1977828 23070077 70 1 21 1 20 18 2 836
21 7180002041315 2192754 2192893 2233575 140 7 0 22 13 1 59 2 7180002024308
22 7180002041321 4840934 4841070 6492983 137 1 0 16 2 15 227 72 7180002025240
23 7180002041325 2624153 2624190 11155368 38 3 0 22 10 5 86 357 7180002025880
24 7180002041348 3064691 3064730 19179560 40 1 0 16 8 7 116 634 7180002030033
#25 7180002041350 2437553 2437572 23353087 20 0 19 1 13 3 107 2
#26 7180002041350 2479000 2479029 23353087 30 2 23 1 3 13 2 880
27 7180002041356 6831454 6831783 14699461 330 1 0 9 16 25 299 319 7180001722964
28 7180002041358 20970057 20970139 25079321 83 7 0 25 14 26 868 189 7180001723456
Where:
Chr1: The most frequent chromosome markers with alignment at coordinates <= begin Chr2: The most frequent chromosome markers with alignment at coordinates >= end mark1: Number of Chr1 markers with alignment at coordinates <= begin mark2: Number of Chr2 markers with alignment at coordinates >=end Lines starting with # should be ignored. ctgid: ctg to break
New scaffolds
#scfid(new) scfid begin end scflen(new)
Files
# ctg:scf new name mapping /scratch1/bos_taurus/Assembly/2009_0312_CA/markers/scfproblems.both.filtered/scfproblems.posmap.ctbscb # scf:scf new:original name mapping /scratch1/bos_taurus/Assembly/2009_0312_CA/markers/scfproblems.both.filtered/scfproblems.posmap.scbscf # ctg:ctg new:original name mapping /scratch1/bos_taurus/Assembly/2009_0312_CA/markers/scfproblems.both.filtered/ctgproblems.posmap.ctbctg # ctg FASTA sequences /scratch1/bos_taurus/Assembly/2009_0312_CA/markers/scfproblems.both.filtered/ctbproblems.fasta # scf FASTA sequences /scratch1/bos_taurus/Assembly/2009_0312_CA/markers/scfproblems.both.filtered/scbproblems.fasta
UMD2.6 (UMD2.5 without contam ctg/scaff; split ctg/scaff)
Contaminants & MarkerBreaks
Delete summary:
ctg scf scf->ctg contaminants(delete) 156 152 666 contaminants(trim) 12 12 1328 markerBreaks 14+1 14+1 2875+1 # 1 more break in UMD2.6.1 total 182 178 4869
Add summary:
ctg scf contaminants(delete) 0 2 contaminants(trim) 12 12 markerBreaks 28+2 29+2 # 1 more break in UMD2.6.1 total 40 43
Summary:
ctg scf markers original 90135 39978 del/add -142 -135 final 89993 39843 -17
Files:
/scratch1/bos_taurus/Assembly/2009_0312_CA/new/bt.ctg.fasta : contig FASTA sequence /scratch1/bos_taurus/Assembly/2009_0312_CA/new/bt.scf.fasta : scaffold FASTA sequence /scratch1/bos_taurus/Assembly/2009_0312_CA/new/bt.posmap.ctglen : contig lengths /scratch1/bos_taurus/Assembly/2009_0312_CA/new/bt.posmap.scflen : scaffold lengths /scratch1/bos_taurus/Assembly/2009_0312_CA/new/bt.posmap.ctgscf : mapping of contigs to scaffolds (posmap format) /scratch1/bos_taurus/Assembly/2009_0312_CA/new/bt.posmap.scaff : mapping of contigs to scaffolds (scaff format) /scratch1/bos_taurus/Assembly/2009_0312_CA/new/ctg.delete.uid : UID of the contigs which were deleted /scratch1/bos_taurus/Assembly/2009_0312_CA/new/scf.delete.uid : UID of the scaffolds which were deleted /scratch1/bos_taurus/Assembly/2009_0312_CA/new/ctg.add.uid : UID of the contigs which were added : UID =~/brk\d+[abc]/ OR UID =~/cnt\d+/ /scratch1/bos_taurus/Assembly/2009_0312_CA/new/scf.add.uid : UID of the contigs which were added : UID =~/brk\d+[abc]/ OR UID =~/cnt\d+/ /scratch1/bos_taurus/Assembly/2009_0312_CA/new/markers.delete.uid : markers which got deleted
/scratch1/bos_taurus/Assembly/2009_0312_CA/new/bt.ctg.break : 15 contig break regions /scratch1/bos_taurus/Assembly/2009_0312_CA/new/bt.scf.break : 16 scaffold break regions
Scripts:
~/bin/breakPosmapKeep.amos : pipeline for breaking scf/ctg
- Final:
elem min max mean med n50 sum ctg 89993 65 1160130(1.1M) 29736 5180 95952 2676109306
scf 39843 316 34167202(34M) 68353 1361 7451988 2723419943 scf<50K 38915 316 49898 2765 1349 5107 107632139 scf<5K 35229 316 4999 1518 1306 1435 53495879
Ctg Markers
Filtered:
%IDY>90 %Matched>85
- ~30% of alignments agree to this condition
Markers:
total: 126111 # from 31372 ctgs
Contigs
elem min max mean med n50 sum placed 31372* 442 1160130 72908 52734 111147 2287291732* unplaced 58621 65 425922 6632 1600 22204 388817574 total 89993 65 1160130 29736 5180 95952 2676109306
Scf Markers
Markers:
total: 126111 # from 2641 scaffolds; 1595 scaffolds have 2+ markers na: 18744 # not placed del: 17 # the scaffolds they belong to were deleted other: 3722 # not on the "main" chromosome; "main" chromosome determined by a majority rule; if it's a tie check markers for uniqueness outliers: 411 # interquartile range method (IQR) q1-1.5*(q3-q1) .. q1+1.5*(q3-q1) filtered: 103217 # "non conflicting" markers
Scaffolds:
elem min max mean med n50 sum #ctgs
placed 2641 1000 34167202 994523 16673 8170786 2626536153* 50528
unplaced 37202 316 754615 2604 1337 3964 96883790 39465
total 39843 316 34167202 68353 1361 7451988 2723419943 89993
scf <2 >=2 >=10 min max mean med n50 sum markers/scf 2641 1595 1046 562 1 1418 39 1 354 103479 ctg/scf 2641 1546 1095 559 1 545 19 1 130 50528
cat markers_scf.mainChr.*summary | count.pl -c 7 | sort -n 1 1564 # 1 marker/scf 2 276 # 2 markers/scf 3 116 # ... 4 52 ... 1470 1 # scaffold 7180002041371 has 1470 markers
- Scaffold position:
filter outliers (interquartile range method); use median value problem: only 2 markers far apart: choose randomly or check for uniqueness
Summary (approximate) Chr #Ctg #Scaff ScaffSpanSum MaxMarkerPos 1 2989 112 157088082 167097751 2 2488 73 138112445 141135901 3 2257 109 120984003 128677351 4 2079 111 124695956 123662451 5 2271 98 120470218 130242001 6 2244 127 117350442 127208151 7 2169 93 109780318 114917551 8 2069 95 111646872 114607251 9 1937 72 103790361 106365151 10 1829 96 102878815 108508301 11 2030 62 106593132 107458151 12 1789 114 89109155 97406401 13 1498 73 83821399 88539451 14 1482 142 84084175 89211101 15 1734 105 84680500 91332551 16 1710 111 80727432 86838601 17 1384 51 72913556 78195801 18 1446 86 65689468 70299751 19 1338 56 63372609 69847351 20 1454 54 71941707 75982901 21 1405 63 70035525 72193201 22 1077 36 60892135 > 60178851 23 1021 44 51791473 54886001 24 1059 26 61662407 > 61466101 25 783 41 42670836 > 45254751 26 991 40 50640267 52316851 27 920 70 45768018 48911451 28 810 52 45884054 50753001 29 1143 89 51657687 55219751 30 3122 340 135803106 152429101 U 39465 37202 94983049
- Scaffold orientation
filter outliers (interquartile range method) use LeastSequareFit method to estimate the orientation : if slope is positive => forward; if slope is negative => reverse; problem: slope ~=0 => which direction ?
cat Chr.summary | getSummary.pl -i 5
cat Chr.agp | grep W | awk '{print $9}' | count.pl
elem <0 0 >0 min max mean med scf 2641 516 1610 515 -31 61 0 0 ctg 50528 24885 2236 23407 Use slope thold to determine direction? cat markers_scf.mainChr.noOutliers.summary | p 'print $_ if(abs($F[5])>0.5);' | wc -l # 634
Ambiguity examples: BZ908653 6 114061501 114016501 114106501 172149 597 98.66 100.00 7180002040834 BZ891600 6 114085251 114051501 114119001 4834 504 99.80 99.02 7180002040834 CZ411135 6 114094001 114059001 114129001 242710 609 98.20 100.00 7180002040834 BZ854276 6 114132751 114101501 114164001 100980 580 99.31 99.83 7180002040834 CC524983 30 115669901 115622401 115717401 96634 715 98.74 100.00 7180002041003 BZ869249 30 115791151 115717401 115864901 86931 448 99.78 100.00 7180002041003 BZ867530 30 115798651 115732401 115864901 69671 572 99.13 100.00 7180002041003 CC585731 30 115808651 115752401 115864901 54950 737 99.86 100.00 7180002041003 CC469285 30 115818651 115772401 115864901 125555 550 94.57 100.00 7180002041003
- Scaffold overlaps: some small scaffolds might be contained by bigger ones
cat markers_scf.mainChr.noOutliers.posmap.scfchrabs | ~/bin/posmap2ovl.pl | sort -nk6 -r | ~/bin/tab2tab.pl -f -15 | head Chr ref qry begin end end-begin 30 7180002041328 7180002041078 72536051 76993951 4457900 4 7180002041381 7180002041269 41976451 46175201 4198750 30 7180002040852 7180002041077 142001501 145251501 3250000 30 7180002038569 7180002041121 33012001 35518351 2506350 30 7180002040971 7180002034501 59411001 61836101 2425100 ...
Marker Issues
Placement
- 37202 scaffolds are unplaced (3.5% of total scaffold span); max=0.7M
- 87 unplaced scaffolds (1.5Mbp total) could be placed using SLK messages
perl ~/bin/difference21.pl bt.slk markers_scf.scflen | head 7180002030632 7180002040171 I -147744.734 21214.779 2 UP 7180002030792 7180002031221 I -15042.301 558.223 2 UP 7180002030849 7180002041244 N -1609089.875 18145.568 4 UP ...
elem min max mean med n50 sum Unplaced 87 740 102909 17259 8544 34550 1501601 Placed 78 16177 27139572 5757234 4037663 10989230 449064304
- ambiguous assignment to chromosmes:
cat markers_scf.count | p 'print $_ if($F[2]*2==$F[3]);' | nl
#k1 k2 count12 count1
1 7180002030741 15 2 4
2 7180002031341 11 1 2
...
36 7180002068140 25 1 2
# scaffold assigned differently
~/bin/difference12.pl markers.all/markers_scf.mainChr.count markers.all_plus_uniq/markers_scf.mainChr.count | nl
1 7180002032536 19 1 2 23480
2 7180002037223 8 1 2 31516
3 7180002040013 25 2 4 15228
4 7180002040262 30 1 2 3087
5 7180002040378 10 1 2 50907
6 7180002040523 29 1 2 9105
7 7180002040769 4 1 3 827526
8 7180002041203 14 2 4 1141560
9 7180002044555 18 1 2 1566
- Chr30 has many scaff aligned to it
- how reliable are the markers: 1151 out of 2641 placed scaffolds have no unique markers ?
- what measure is best for placing the scaffolds?
- can some scaffolds go in the gaps ?
- AGP:
- gaps<=0 or 20 set to 100
- unoriented ctgs set to +
- Marker positions are not uniformly distributed; they tend to "custer"
elem 0 >0 min max mean med n50 sum markers_chr 107337 8405 98932 0 446250 25800 8750 75000 2769379200 markers_scf 94171 123 94048 0 2565850 24143 15856 42236 2273610521 markers_chr.mainChr.noOutliers 101084 7609 93475 0 1253750 27383 10000 78750 2768040450 # filtered IRQx,xy method markers_scf.mainChr.noOutliers 88450 52 88398 0 2669361 25327 16587 44374 2240202221
Orientation
- at least 1564 out of 2641 placed scaffolds cannot be oriented; max=1.39M
Possible misassemblies
nl #scfid Ch medianPos rangePos scfLen slope #ChMark #Mark #ctg 1 7180002041225 1 120473301 9866350 9279975(9M) 0.0103 381 390 184 : break interval: 1674263..1674319 (1X frg_cvg region) 2 7180002041078 30 77506551 6310600 3094999(3M) 0.0148 65 69 46 : no clear position
scf7180002041225 : 9.27 Mbp on Chr1
scflen=9279975
7180002041225.markers 7180002041225.markers 7180002041225.cvg
ChrPos ScfPos 115272051 56300 ... 116848301 1618430 116953301 9190560 ... 125138401 1691145
break interval: 1674263..1674319 = 57bp frg_cvg=1 mate_cvg=0 bad_mate_cvg(nearby)=7
scf7180002041078 : 3.09 Mbp (not broken)
7180002041078.markers 7180002041078.markers 7180002041078.1.cvg xrange [315268:466827] 7180002041078.2.cvg xrange [1191000:1229713] 7180002041078.3.cvg xrange [2733146:2750017]
ChrPos ScfPos 72536051 2893791 ... 72682301 2750017 72728651 1191000 ... 73273651 466827 77112801 2733146 ... 78464051 1229713 .. 78595401 315268
scf7180002040971 : 1.93 Mbp (not broken)
Files
/scratch1/bos_taurus/Assembly/2009_0312_CA/new/markers.all.redo/
- Nucmer alignments to UMD2.0 chromosomes: all seem to agree pretty well
/scratch1/bos_taurus/Assembly/2009_0312_CA/new/markers.all.redo2/nucmer_UMD2.0/UMD2.0.Chr26-Chr26.png /scratch1/bos_taurus/Assembly/2009_0312_CA/new/markers.all.redo2/nucmer_UMD2.0/UMD2.0.Chr27-Chr27.png /scratch1/bos_taurus/Assembly/2009_0312_CA/new/markers.all.redo2/nucmer_UMD2.0/UMD2.0.Chr29-Chr29.png /scratch1/bos_taurus/Assembly/2009_0312_CA/new/markers.all.redo2/nucmer_UMD2.0/UMD2.0.ChrX-Chr30.png
Homo sapiens alignments
Citations:
- mice closer to human than cow
- human and cow have approximately 201 homologous blocks of DNA
- Independently generated mapping data provide another measure of the quality of the assembly. Snelling et al. [4] created a B. taurus map from three radiation hybrid panels, two genetic maps, and bacterial artificial chromosome (BAC) end sequences. We aligned all of the 17,254 markers (of which 17,193 are unique) in their composite map (Cmap) to both assemblies. A marker was considered as matching a chromosome if 90% of the marker sequence aligned with at least 95% identity. Of the Cmap markers, 14,620 align to the UMD2 assembly's chromosomes, versus 13,699 markers (6.3% fewer) for the BCM4 assembly. A small number of Cmap markers (119 and 82 for UMD2 and BCM4, respectively) mapped to a different chromosome from the one indicated in the Cmap data.
- homologous synteny block (HSB) :
- human-cow alignment extended for at least 250 Kbp
- it was not interrupted by an inversion or by an HSB on another chromosome.
- If two HSBs were interrupted by a gap of <3 Mbp and nothing else fell in that gap, the two blocks were merged. (Note that if a large region of synteny is interrupted by a distinct HSB, the interruption creates three HSBs.)
- Number of homologous synteny blocks on each chromosome of the cow and human genomes 201 -> 268 blocks
- Orienting contigs using cow-human alignments:
- Scaffolds (sets of linked contigs) that were mapped onto chromosomes using only a single marker could not be oriented from the marker information alone. We oriented many of these scaffolds by taking advantage of the overall conserved synteny between cow and human. First, all cow scaffolds were aligned to the human genome using nucmer [14] with its maximal unique match (mum) option in order to avoid alignments of repetitive sequence. For each alignment of a previously unoriented scaffold to human, all alignments within 100 Kbp on each side were pulled out for analysis. A score S was computed for each unoriented scaffold, taking into account whether the scaffolds surrounding S on both sides (in cow) were mapped to a consistent set of locations in human. If the scaffolds surrounding S were oriented, and if a large majority of these scaffolds on both the left and right agreed on the orientation, then S was assigned that orientation. Using this procedure, 1,840 scaffolds containing 4,011 contigs were oriented.
- We developed a similar procedure to assign unplaced contigs to chromosomes, again relying on conserved synteny between cow and human. First, all unplaced contigs were aligned as above. Mummer's 'delta-filter' program was then used to compute a one-to-one mapping of the unplaced contigs to human so that only the best aligning contig was considered at each region in human. For each unplaced contig's best alignment to human, the matching region in cow was identified via our human-cow syntenic map, and all contigs from this region were extracted for examination. We only considered placing a contig on a B. taurus chromosome if the order and direction of the surrounding contigs in cow matched the corresponding region in human. As above, we examined the alignments of nearby cow contigs that aligned within 100 kb of the unplaced contig's alignment in human. If the region of cow-human synteny contained no rearrangements, then the unplaced contig was placed at the location indicated by these alignments. Using this procedure, 1,046 contigs were placed on chromosomes. One consequence of this procedure was that a number of incompletely mapped genes (based on mRNA alignments) were completed.
Issues:
- which alignment program to use?
- nucmer
- blastz: difficult to parse
- blat
- nucmer: what parameters?
- default
- loose : -mum -l 12 -c 30 -g 1000
- ref: 24 homo sapiens chromosomes files
- query: 26 bos taurus scaffold files
Total scaffolds:
elem min max mean med n50 sum scf(len) 39844 316 34167202 68352 1361 7451988 2723419938
Aligned to HS:
elem min max mean med n50 sum scf(len) 8860 385 34167202 301202 2257 7740810 2668658140 # each scf aligns in avg to 4 Chr scf(aligLen) 789272 250 19097 514 399 550 405946972 scf(alig%) 789272 58.04 100.00 79 78.54 78.72 . scf(len,maxX) 727 1002 7069350 194658 6530 1895564 141516604 # 220 in common with the 339 ones that have mark scf(len,noMark) 7025 385 7605708 7783 1654 27630 54680173 scf(len,mark) 1835 1001 34167202 1424511 40503 7970944 2613977967
Not aligned to HS:
elem min max mean med n50 sum scf(len,notAligned) 30984 316 163953 1767 1309 1555 54762139
Aligned to markers:
elem min max mean med n50 sum scf(len) 2641 1000 34167202 994523 16673 8170786 2626536153 1 marker 1595 1000 1396951 17482 4595 48276 27884078 2+ markers 1046 1055 34167202 2484371 596240 8217662 2598652075 abs(slope)<0.25 1946 1000 3094999 29136 7585 114274 56700529 abs(slope)>=0.25 696 4674 34167202 3692292 1845828 8401441 2569835619
Scf summary:
elem min max mean med n50 sum all 39844 316 34167202 68352 1361 7451988 2723419938 1+mark 2641 1000 34167202 994523 16673 8170786 2626536153 # best marker alignment 1+align 8860 385 34167202 301202 2257 7740810 2668658140 # alignments > 250bp 1+align(new) 9880 385 34167202 270458 1996 7740810 2672129298 # alignments > 200bp (3.74M more than 250bp align) 1+mark or 1+align 9669 385 34167202 276443 2051 7740810 2672936792 0 mark and 1+align 7027 385 754,615 6603 1654 20219 46,400,303 !!! 1+mark and 1+align 1791 1001 34167202 1305861 37914 7473583 2338798064 0 mark and 0 align 30179 316 88,326 1705 1305 1517 51,483,587 !!! 1+mark or 1+align 9669 385 34167202 276443 2051 7740810 2,672,936,792
Degenerate:
elem min max mean med n50 sum all 251413 65 39964 1003 984 994 252,279,234 all(2000+bp) 2128 2000 39964 3753 2827 3946 7,986,872 1+mark 562 200 30168 2731 1274 5029 1,535,011 1+mark(2000+bp) 180 2007 30168 6079 4404 7510 1,094,252
1+align 6429 251 39964 1487 1013 1287 9,566,273 1+align(2000+bp) 756 2004 39964 4820 3624 5646 3,644,556
Issues:
- 24 scaffolds that have 200+ alignments to at least 2 HS chromosomes
- 37 scaffolds that have 100+ alignments to at least 2 HS chromosomes
- 83 scaffolds that have 50+ alignments to at least 2 HS chromosomes
File location:
/scratch1/bos_taurus/Assembly/2009_0312_CA/nucmer_human
Synteny method
File location:
/scratch1/bos_taurus/Assembly/2009_0312_CA/synteny
- Position & orient scaffolds that aligned to HS
- find placed neighboring that don't "disagree"
elem min max mean med n50 sum total 7027 385 754615 6603 1654 20219 46,400,303 !!!
- Scaffolds possibly assigned to the wrong chromosome
cat summary.nucmer-markers.txt | perl ~/bin/synteny/getSynteny2.nucmer-marker.pl -minMarkers 2 | grep update:chr | getSummary.pl -i 6 -t "<2 markers"
cat summary.nucmer-markers.txt | perl ~/bin/synteny/getSynteny2.nucmer-marker.pl -minMarkers 4 | grep update:chr | getSummary.pl -i 6 -t "<4 markers"
elem min max mean med n50 sum
<2 markers 307 1001 937279 27898 9212 98674 8,564,850
<4 markers 383 1001 1141560 33822 13426 160445 12,953,875
UMD2.0
elem min max mean med N50 sum ctg 74337 88 840370 35148 14148 79144 2612810882*
UMD2.6.1 noVariants,noCont
elem min max mean med N50 sum scf 2646 385 34167202 994701 20787 7139718 2631980624 ctg 50755 65 1160130 50966 29450 88583 2586785910*
UMD2.6.1 noVariants
elem min max mean med n50 sum
scf 39844 316 34167202 68352 1361 7451988 2,723,419,938
scf.placed 4707 385 34167202 564225 10413 7800796 2655811430
scf.variants 29436 723 51828 1714 1298 1514 50,461,989
scf.unplaced 30575 316 451968 1845 1309 1601 56440663
elem min max mean med n50 sum
ctg 89994 65 1160130 29736 5180 95952 2676109378
ctg.placed 53646 65 1160130 48650 26840 98428 2609925446
ctg.unplaced 31510 101 207476 1761 1314 1556 55511410
elem min max mean med n50 sum
deg(all) 251413 65 39964 1003 984 994 252,279,234
deg(>2Kbp) 2128 2000 39964 3753 2827 3946 7,986,872
deg.placed 883 200 30246 2711 1303 4845 2,393,905 deg.variants 4654 331 8039 1117 989 1024 5,200,553 deg.unplaced(?) 245343 65 39964 994 984 993 244047258
scf.unplaced 978 316 451968 7932 3736 15828 7757882 scf.uplaced.0cvg 863 316 451968 7521 2286 15744 6,491,342
deg.unplaced 747 2002 39964 4365 3509 4819 3261181 deg.unplaced.0cvg 734 2002 39964 4358 3507 4811 3,199,221 ct_deg.placed 53646 65 1160130 48650 26840 98428 2,609,925,446*
UMD2.0-UMD2.6.1 gaps>1K summary id count min max median sum 1 775 999 64953 1683 2884541 ... 30 2174 999 167808 1528 7739170 total 14936 999 167808 1638 53,526,388
- align UMD2.6 Chr1..30 to UMD2.0 Chr1..30 0cvg regions
elem min max mean med n50 sum all 14936 1000 167809 3584 1639 6465 53541324 aligned 11091 1000 167809 4010 1692 7977 44475715 not_aligned 3845 1000 50546 2357 1529 2735 9065609
Files: UMD2.0 regions not covered by UMD2.6.1 (chr aligned to itself)
/scratch1/bos_taurus/Assembly/2009_0312_CA/synteny/agp_markers.noVariants_nucmer.noVariants/nucmer_UMD2.0/Chr.0cvg.fa /scratch1/bos_taurus/Assembly/2009_0312_CA/synteny/agp_markers.noVariants_nucmer.noVariants/nucmer_UMD2.0/Chr.0cvg.posmap /scratch1/bos_taurus/Assembly/2009_0312_CA/synteny/agp_markers.noVariants_nucmer.noVariants/nucmer_UMD2.0/Chr.0cvg.summary
- UMD2.6.1 vs UMD2.0 Chr1..30
- nucmer -mum -l 50 -c 250
- max gap: Chr8:60134434..60228381=93948
- gaps>10K: 191 , 4.2M total, 4.19M aligned without "-mum"
elem min max mean med n50 sum gaps>10K 191 10077 93948 22106 15599 26551 4222342
- realign all scaff without using nucmer "-mum" option: all gaps >10K align to large scaffolds !!!
UMD2.6.1 noVariants, add deg & UMD2.0 alignments
--Dpuiu 09:44, 22 June 2009 (EDT)
elem min max mean med n50 sum scf 39844 316 34167202 68352 1361 7451988 2723419938 scf.variants 29436 723 51828 1714 1298 1514 50461989 scf.noVariants 10580 316 34167202 252690 1957 7740810 2673467579 scf.noVariants.ignore 4079 316 4985 1852 1541 1842 7558196 # less than 5K and placed through alignments inside a larger scaffold scf.noVariants.noIgnore 6506 385 34167202 409763 6673 7740810 2665923763 ctg.variants 29967 723 51828 1641 1298 1475 49178546 deg.variants 4654 331 8039 1117 989 1024 5200553
scf_deg.placed 4933 385 34167202 540139 11548 7740810 2664507818 scf_deg.markers 2003 1020 34167202 1302992 30456 8170786 2609894380 scf_deg.alignHS 1818 385 3278163 27070 12322 47406 49214215 scf_deg.alignUMD2.0 1112 1002 73626 4855 2948 6898 5399223 scf.placed 4044 385 34167202 657786 15617 7740810 2660090134 scf.markers 1825 1020 34167202 1429490 37672 8170786 2608820353 scf.alignHS 1587 385 3278163 30006 13998 50604 47620297 scf.alignUMD2.0 632 1002 73626 5774 5137 9046 3649484 deg.placed 889 2002 39964 4969 3771 6165 4417684 deg.markers 178 2007 30168 6033 4336 7350 1074027 deg.alignHS 231 2049 39964 6900 6289 7337 1593918 deg.alignUMD2.0 480 2002 15992 3645 2767 3842 1749739 ctg_deg.placed 54129 65 1160130 48371 26493 98075 2,618,296,162* ctg.placed 53240 65 1160130 49096 27309 98285 2,613,878,478 deg.placed 889 2002 39964 4969 3771 6165 4,417,684
HS-BT Synteny map
- Trust scaffolds with 4+ markers
- Scaffolds with 3- markers must have a close neighbor that agrees with them
same HS & BT Chr maxCount=2 # at most 2 scaffolds away minMarkers=4 minRatio=0.66 & maxRatio=1.5 # distance ratio; maxDistance not used !!!
=> 226 synteny regions !!!
join2.pl nucmer_*lsf markers_scf.*lsf | \
~/bin/filterMarkers.pl -minMarkers 4 | \
~/bin/getSyntenyBlock.pl | \
grep -v ^# | grep -v ^$ | ~/bin/flipSummary.pl | sort -nk2 -nk5 | ~/bin/tab2tab.pl | \
perl -ane 'print $P[13]," ",$F[13],"\n" if($F[13]-$P[13]==1); print $F[13]," ",$P[13],"\n" if($P[13]-$F[13]==1); @P=@F;' | sort -u -n | \
~/bin/mergeMap.pl >! hs-bt.map.tmp
join2.pl nucmer_*lsf markers_scf.*lsf | \
~/bin/filterMarkers.pl -minMarkers 4 | \
~/bin/getSyntenyBlock.pl -map hs-bt.map.tmp | \
~/bin/tab2tab.pl | grep # | sed 's/#//' > hs-bt.map
~/bin/map-draw.pl -refLen hs.infoseq -qryLen bt.infoseq hs-bt.map > ! hs-bt.png
Problems:
7180002041220: BT.Chr2 ok 7180002041025: BT.Chr4 7180002041222: BT.Chr7 7180002041228: BT.Chr8 7180002040195: BT.Chr8; HS.Ch23 del !!! 7180002041001: BT.Chr8; HS.Ch23 del !!! 7180002040851: BT.Chr30; HS.Ch3 -> HS.Ch23 7180002041008: BT.Chr30; HS.Ch7 -> HS.Ch23
- hs-bt.png Map picture
- Map:
#HS-ref begin end len HS-clust BT-ref begin end BT-clust #scf 01 870247 12637274 11767027 16 46893147 58621226 11728079 -1 5 01 16674545 30031540 13356995 2 126668764 139027145 12358381 -2 4 01 32016003 68894564 36878561 3 83066354 120874874 37808520 -3 10 01 68907947 122485896 53577949 3 23819464 83221225 59401761 -4 6 01 143725713 143737025 11312 3 25097798 25109110 11312 5 1 ... 23 148835146 151656716 2821570 30 32889252 35700335 2811083 -276 3 23 152157769 154641974 2484205 30 39322867 42104054 2781187 -277 2 23 154387850 154403684 15834 30 39353299 39369133 15834 278 1
Files:
/scratch1/bos_taurus/Assembly/2009_0312_CA/synteny/map.4/hs-bt.map /scratch1/bos_taurus/Assembly/2009_0312_CA/synteny/map.4/hs-bt.png
Overlaps
Cases:
1. CONTAINED scaffolds (clear variants) 2. single BEGIN/END between 2 scaffolds: 2 scaffolds could be merged 3. scaffold closing a sequence: 3 scaffolds could be merged 4. multiple BEGIN/END/CONTAIN* between 2 scaffold contigs
1: CONTAINED ~ 4731 cases
cat nucmer_scf.ovl/all-all.contained.ids marker_scf.ovl/all-all.contained.ids | sort -u > all-all.contained.ids
- Summary
elem <2Kbp 2..10Kbp >10Kbp min max mean med n50 sum variants(all) 4731 3503 1085 143 723 75436 2513 1429 3469 11891882 variants(1+markers) 541 311 170 60 1001 75421 4466 1690 9947 2416312
- Example: longest contained scaffold 75Kbp
#scaffold alignments 117893 126367 | 8520 1 | 8475 8520 | 94.06 | 562884 75576 | 1.51 11.27 | 7180002040891 7180002040646 286811 289412 | 72974 75576 | 2602 2603 | 93.29 | 562884 75576 | 0.46 3.44 | 7180002040891 7180002040646 421477 421977 | 9727 10223 | 501 497 | 95.41 | 562884 75576 | 0.09 0.66 | 7180002040891 7180002040646 171970 180765 | 73605 64798 | 8796 8808 | 96.74 | 274750 75576 | 3.20 11.65 | 7180002040912 7180002040646 180765 195989 | 64652 49398 | 15225 15255 | 96.05 | 274750 75576 | 5.54 20.18 | 7180002040912 7180002040646 196223 203368 | 49244 42105 | 7146 7140 | 96.81 | 274750 75576 | 2.60 9.45 | 7180002040912 7180002040646 205935 208456 | 38652 36130 | 2522 2523 | 96.12 | 274750 75576 | 0.92 3.34 | 7180002040912 7180002040646 207096 210496 | 42082 38673 | 3401 3410 | 96.07 | 274750 75576 | 1.24 4.51 | 7180002040912 7180002040646 209529 217353 | 36109 28272 | 7825 7838 | 95.50 | 274750 75576 | 2.85 10.37 | 7180002040912 7180002040646 213859 219155 | 25520 20220 | 5297 5301 | 95.80 | 274750 75576 | 1.93 7.01 | 7180002040912 7180002040646 221183 227630 | 20236 13870 | 6448 6367 | 93.63 | 274750 75576 | 2.35 8.42 | 7180002040912 7180002040646 227923 228686 | 13884 13126 | 764 759 | 95.16 | 274750 75576 | 0.28 1.00 | 7180002040912 7180002040646 228084 231413 | 13104 9760 | 3330 3345 | 94.10 | 274750 75576 | 1.21 4.43 | 7180002040912 7180002040646 #contig alignments 1929 10403 | 8520 1 | 8475 8520 | 94.06 | 26592 8520 | 31.87 100.00 | 7180002040891.4.10 7180002040646.1.8 [CONTAINS] 9084 11685 | 30871 33473 | 2602 2603 | 93.29 | 276174 33473 | 0.94 7.78 | 7180002040891.8.10 7180002040646.8.8 143750 144250 | 1187 1683 | 501 497 | 95.41 | 276174 4564 | 0.18 10.89 | 7180002040891.8.10 7180002040646.2.8 163797 172592 | 31502 22695 | 8796 8808 | 96.74 | 233551 33473 | 3.77 26.31 | 7180002040912.3.5 7180002040646.8.8 172592 187816 | 22549 7295 | 15225 15255 | 96.05 | 233551 33473 | 6.52 45.57 | 7180002040912.3.5 7180002040646.8.8 188050 195195 | 7141 2 | 7146 7140 | 96.81 | 233551 33473 | 3.06 21.33 | 7180002040912.3.5 7180002040646.8.8 197762 200283 | 2523 1 | 2522 2523 | 96.12 | 233551 2523 | 1.08 100.00 | 7180002040912.3.5 7180002040646.6.8 [CONTAINS] 198923 202323 | 3410 1 | 3401 3410 | 96.07 | 233551 3411 | 1.46 99.97 | 7180002040912.3.5 7180002040646.7.8 [CONTAINS] 201356 209180 | 7838 1 | 7825 7838 | 95.50 | 233551 7838 | 3.35 100.00 | 7180002040912.3.5 7180002040646.5.8 [CONTAINS] 205686 210982 | 12396 7096 | 5297 5301 | 95.80 | 233551 12396 | 2.27 42.76 | 7180002040912.3.5 7180002040646.3.8 213010 219457 | 7112 746 | 6448 6367 | 93.63 | 233551 12396 | 2.76 51.36 | 7180002040912.3.5 7180002040646.3.8 219750 220513 | 760 2 | 764 759 | 95.16 | 233551 12396 | 0.33 6.12 | 7180002040912.3.5 7180002040646.3.8 219911 223240 | 4564 1220 | 3330 3345 | 94.10 | 233551 4564 | 1.43 73.29 | 7180002040912.3.5 7180002040646.2.8 [CONTAINS] #marker & alignment summary #id BT-ref #markers slope begin end len HS-ref #align slope begin end len 7180002040891 4 1 0 13956053 14518937 562884 22 14 -1.5687 20838730 21401614 562884 7180002040163 8 1 -1 38079677 38178351 98674 22 3 1.0264 21133894 21232568 98674 update:dir:7180002040646 7180002040646 8 . 1 38088797 38164373 75576 22 4 -0.9833 21143014 21218590 75576 assign:Chr:7180002040163,7180002040163
2. ~ 349 cases (-3 cases 3.)
cat all-all.begin.ids all-all.end.ids | sort -u | wc -l
3. ~ 3 cases
intersect.pl all-all.begin.ids all-all.end.ids | perl -ane 'print $_ if($F[1]=~/1.1$/);' 7180002032818.1.1 23259 7180002036943.1.1 28355 7180002040409.1.1 25219 # merges 2 scaff 1 2811 | 2807 1 | 2811 2807 | 99.11 | 23259 5134 | 12.09 54.67 | 7180002032818.1.1 7180002032811.1.2[BEGIN] [BEGIN] 22101 23259 | 7238 6079 | 1159 1160 | 99.66 | 23259 7238 | 4.98 16.03 | 7180002032818.1.1 7180002032811.2.2[END] [END] 1 2811 | 2807 1 | 2811 2807 | 99.11 | 23259 5134 | 12.09 54.67 | 7180002032818.1.1 7180002032811.1.2[BEGIN] [BEGIN] 22101 23259 | 7238 6079 | 1159 1160 | 99.66 | 23259 7238 | 4.98 16.03 | 7180002032818.1.1 7180002032811.2.2[END] [END] 1 2025 | 2066 4090 | 2025 2025 | 99.80 | 25219 4090 | 8.03 49.51 | 7180002040409.1.1 7180002033541.1.1[BEGIN] [END] 23394 25219 | 5761 3930 | 1826 1832 | 98.42 | 25219 5761 | 7.24 31.80 | 7180002040409.1.1 7180002033538.1.1[END] [END]
cd nucmer_scf.ovl intersect.pl all-all.begin.ids all-all.end.ids 7180002032818 23259 7180002036943 28355 7180002040409 25219
4. ~10 cases
cd nucmer_ctg.ovl/translated/ cat all-all.annotated.coords | egrep 'BEGIN|END|CONTAIN' | p 'next if($F[6]<5000); next if($F[7]<5000); print $_;' | p '$F[17]=~/^([^.]+)(.+)/; $F[17]=$1 ; $F[18]=~/^([^.]+)/; $F[18]=$1; print $F[17],"\t",$F[18],"\n";' | count.pl -m 2 7180002041235 brk002041306a 3 7180002038888 7180002034914 2 7180002041059 7180002040095 2 7180002040879 7180002040934 2 cnt0002041350 7180002040907 2 7180002041341 7180002040894 2 7180002041015 cnt0002040938 2 7180002039401 7180002040397 2 7180002037358 7180002031425 2 7180002040915 7180002039470 2 cat all-all.annotated.coords | egrep 'BEGIN|END|CONTAIN' | grep ... 12363 21143 | 1 8837 | 8781 8837 | 93.43 | 21143 50443 | 41.53 17.52 | 7180002041235.23.33 brk002041306a.5.7[END] [BEGIN] 1 9272 | 5591 14944 | 9272 9354 | 94.47 | 9272 50443 | 100.00 18.54 | 7180002041235.25.33 brk002041306a.5.7[CONTAINED] 71049 97689 | 31941 5220 | 26641 26722 | 96.71 | 97689 31941 | 27.27 83.66 | 7180002041235.29.33 brk002041306a.6.7[END] [END] 57 3896 | 14743 10903 | 3840 3841 | 95.94 | 3896 22667 | 98.56 16.95 | 7180002041235.32.33 brk002041306a.3.7[CONTAINED] 1 8942 | 1 8947 | 8942 8947 | 99.49 | 20792 8947 | 43.01 100.00 | 7180002038888.1.1 7180002034914.1.2[CONTAINS] 9670 20792 | 1 11091 | 11123 11091 | 99.27 | 20792 11091 | 53.50 100.00 | 7180002038888.1.1 7180002034914.2.2[CONTAINS] 1 8705 | 3932 12610 | 8705 8679 | 96.17 | 8705 30654 | 100.00 28.31 | 7180002041059.1.26 7180002040095.1.1[CONTAINED] 1 4777 | 12989 17697 | 4777 4709 | 95.32 | 4777 30654 | 100.00 15.36 | 7180002041059.2.26 7180002040095.1.1[CONTAINED] 1 6588 | 17999 24607 | 6588 6609 | 96.15 | 6591 30654 | 99.95 21.56 | 7180002041059.3.26 7180002040095.1.1[CONTAINED] 630 16709 | 16024 1 | 16080 16024 | 97.09 | 16901 16024 | 95.14 100.00 | 7180002040879.2.14 7180002040934.2.87[CONTAINS] 1 5876 | 5882 1 | 5876 5882 | 98.51 | 39657 13643 | 14.82 43.11 | 7180002040879.3.14 7180002040934.1.87[BEGIN] [BEGIN]
Scaffold links
Try to identify scaffold that fit the following criteria:
- have no markers and no alignments to HS
- linked by 2+ links to a single scaffold that has markers/alignments to HS
- is not a variant (2Kbp+ of unique sequnece)
elem min max mean med n50 sum linked 6169 316 88326 2408 1480 2709 14,860,479 linked(2+mates) 2109 316 44120 2595 1558 3092 5,474,510 linked(2+mates to a single scf) 2057 316 44120 2516 1547 2850 5,177,182 linked(2+mates to a single scf, no variant) 112 316 43782 7278 1987 23338 815,223 # 46>2Kbp; 25>10Kbp
=> 112 scaffold & 0.81Mbp could be added to the chromosomes !!!
BT alignments
- UMD2.6.1 vs UMD2.6.1
- Marker scaffolds against themselves (2642 total scaffolds) : nucmer -maxmatch -l 40 -c 2500 -g 250
- Mapped scaffolds without markers against marker scaffolds: nucmer -maxmatch -l 40 -c 250
- Mapped scaffolds without markers against themselves: nucmer -maxmatch -l 40 -c 250
File location:
/scratch1/bos_taurus/Assembly/2009_0312_CA/nucmer_Chr/
Other issues
Chr10 gene duplication
- LOC100298457
- See /nfshomes/dpuiu/Readmes/bos_taurus.runCA.13.txt
Scf breaks (Mike Robers)
:::::::::::::: scf.break.ids :::::::::::::: #ctg scf beg end dir 7180002015438 7180002038435 0 32822 f # ok: 4 markers from Chr18 7180002021916 7180002040442 0 24559 f # ok: 1 marker from Chr10 7180001725791 7180002040808 466123 562461 f # ok: 25 marker from Chr1, 1 marker from Chr2 7180001727899 7180002040844 8236574 8479429 f # also found by us; 3 Chr30 markers in the middle 7180001854650 7180002041103 5230302 5294734 f # also found by us; 4 Chr15 markers at 3' 7180002003578 7180002041216 300250 392069 f # also found by us; 7 Chr5 markers at 5' 7180002020010 7180002041293 4482477 4600045 f # ok 7180001722390 7180002041353 0 51674 f # ok :::::::::::::: scf.excised.ids :::::::::::::: #ctg scf beg end dir 7180002008629 7180002034664 82740 105723 f # no markers 7180001786240 7180002040913 2076746 2112375 f # ok 7180001787022 7180002040927 1075352 1109350 f # ok 7180001787352 7180002040931 4956099 4967823 f # ok 7180001789387 7180002040981 6641924 6671967 f # ok 7180001789575 7180002040984 3095587 3154302 f # ok 7180002003269 7180002041209 6350890 6470971 f # ok 7180002025281 7180002041322 989366 1227636 f # ok 7180002026741 7180002041328 343726 414329 f # ok 7180002029433 7180002041343 834493 888144 f # ok 7180001726451 7180002041383 7893629 7901645 f # ok
LOC100298457 (duplicate gene on Chr10)
Problem:
Is LOC100298457 gene (cow Chr10) a variation of MFSD3 gene (cow Chr14)?
UMD2.0
Chr10 15186759 15189292 833 W 7180003260677 1 2534 + # gene LOC100298457 Chr14 1839355 1958202 533 W 7180003326040 1 118848 + # gene MFSD3
- LOC100298457 aligns to 2 UMD2.6 scaffolds:
scf7180002041112 (1.56Mbp; 34 contigs) scf7180002033841 (2.44Kbp; 1 contig)
- scf7180002041112 contains 39 bos taurus Chr14 markers and has 206 alignments to human chromosome 8
- scf7180002033841 contains no bos taurus markers and has 1 alignments to human chromosome 8.
- scf7180002033841 5' aligns to scf7180002041112 (1.56Mbp; 34 contigs, cow Chr14) and to a human Chr8 region that maps to cow Chr14
- scf7180002033841 3' aligns to scf7180002041157 (6.15Mbp; 99 contigs, cow Chr10)
- scf7180002033841 & scf7180002041157 are linked by 2 mate pairs (inserts from a 3kbp BCM shotgun library)
- LOC100298457 vs UMD2.6 scf:
[S1] [E1] | [S2] [E2] | [LEN 1] [LEN 2] | [% IDY] | [LEN R] [LEN Q] | [COV R] [COV Q] | [TAGS]
===============================================================================================================================
1 88 | 1138313 1138226 | 88 88 | 100.00 | 1344 1561688 | 6.55 0.01 | LOC100298457 scf7180002041112
80 192 | 1138150 1138038 | 113 113 | 100.00 | 1344 1561688 | 8.41 0.01 | LOC100298457 scf7180002041112
190 316 | 1137974 1137848 | 127 127 | 100.00 | 1344 1561688 | 9.45 0.01 | LOC100298457 scf7180002041112
315 464 | 1137770 1137621 | 150 150 | 100.00 | 1344 1561688 | 11.16 0.01 | LOC100298457 scf7180002041112
455 1344 | 1137322 1136432 | 890 891 | 99.78 | 1344 1561688 | 66.22 0.06 | LOC100298457 scf7180002041112
1 1344 | 1349 6 | 1344 1344 | 100.00 | 1344 2445 | 100.00 54.97 | LOC100298457 scf7180002033841
- UMD2.6 scf* vs scf7180002033841:
[S1] [E1] | [S2] [E2] | [LEN 1] [LEN 2] | [% IDY] | [LEN R] [LEN Q] | [COV R] [COV Q] | [TAGS] =============================================================================================================================== 1136427 1137322 | 1 895 | 896 895 | 99.78 | 1561688 2445 | 0.06 36.61 | scf7180002041112 scf7180002033841 4060495 4061426 | 2445 1521 | 932 925 | 97.76 | 6157473 2445 | 0.02 37.83 | scf7180002041157 scf7180002033841
Bos taurus marker summary:
#id BT-ref #markers slope begin end len scf7180002041112 Chr14 39 -0.8518 3377707 4939395 1561688 scf7180002041157 Chr10 255 1.0035 12831593 18989066 6157473 scf7180002033841 ? 0
Homo sapiens alignment summary:
#id HS-ref #align slope begin end len scf7180002041112 Chr8 206 0.6701 144381103 145942791 1561688 scf7180002041157 Chr15 1667 0.987 62245836 68403309 6157473 scf7180002033841 Chr8 1 1 145705320 145707765 2445
[S1] [E1] | [S2] [E2] | [LEN 1] [LEN 2] | [% IDY] | [LEN R] [LEN Q] | [COV R] [COV Q] | [TAGS] =============================================================================================================================== 143996007 143996305 | 74278 74572 | 299 295 | 82.45 | 146274826 1561688 | 0.00 0.02 | 1 1 NC_000008 7180002041112 ... 146249212 146249657 | 1268887 1268446 | 446 442 | 80.89 | 146274826 1561688 | 0.00 0.03 | 1 -1 NC_000008 7180002041112 63013167 63013429 | 887697 887966 | 263 270 | 78.15 | 100338915 6157473 | 0.00 0.00 | 1 1 NC_000015 7180002041157 ... 68369047 68369417 | 6154757 6155112 | 371 356 | 83.06 | 100338915 6157473 | 0.00 0.01 | 1 1 NC_000015 7180002041157 145705522 145706206 | 202 886 | 685 685 | 81.28 | 146274826 2445 | 0.00 28.02 | 1 1 NC_000008 7180002033841
scf7180002033841: 6 reads
read scf begin end dir 1120017508 7180002033841 0 1018 f 1120020725 7180002033841 227 1109 f 1120020722 7180002033841 724 1720 r 1120020726 7180002033841 1006 2073 f diffScaffold 1120017511 7180002033841 1303 2175 r 1120020728 7180002033841 1370 2445 f diffScaffold
scf7180002033841: 4 mates
read1 read2 scf1 scf2 1120017508 1120020722 7180002033841 7180002033841 1120017511 1120020725 7180002033841 7180002033841 1120020726 1120017512 7180002033841 7180002041157 diffScaffold 1120020728 1120017514 7180002033841 7180002041157 diffScaffold
scf7180002041157: 69095 reads; frg 1120017514 & 1120017512 positions close to the region aligned to scf7180002033841 (4060495-4061426)
read scf begin end dir 607312594 7180002041157 0 866 f ... 1120017512 7180002041157 4059332 4060396 f diffScaffold 1120017514 7180002041157 4058851 4059931 f diffScaffold .... 583956204 7180002041157 6157144 6157228 r
scf7180002041157: 99 contigs
count ctg scf begin end dir 1 ctg7180001926175 scf7180002041157 0 81467 f .. 64 ctg7180001926231 scf7180002041157 4039922 4046680 f # 6758bp ctg; 800bp gap following the contig 65 ctg7180001926232 scf7180002041157 4047480 4193693 f # 146213bp ctg ; 20bp gap following the contig 66 ctg7180001926233 scf7180002041157 4193713 4208047 f # 14334bp ctg .. 99 ctg7180001926260 scf7180002041157 6152691 6157473 f
frg 1120017514 & 1120017512 positions on ctg7180001926232
1120017514 ctg7180001926232 11371 12451 f # reads are 11kbp(>3Kb lib mean) inside the contig 1120017512 ctg7180001926232 11852 12916 f # reads are 11kbp(>3Kb lib mean) inside the contig
Missing genes
- 5 genes not found by Liliana using ESTaligner but found using gmap (%id<95) : they were on the haplotype variants
Chr27 -> Chr21
- The centromeric end of BTA27 is actually placed on BTA21 in UMD2.6 (which version?)
Chromosomes
synteny.redo2
- summary.txt : 8122 scf + 6322 deg => 14444 seqs (placed using markers or synteny to HS)
- from summary.txt removed:
58 scf.questionable.ids (44 placed, 14 linked) 100 ctg.questionable.ids 931 deg.questionable.ids
UMD_2.6.d_g
. elem <2000 >2000 min max mean med n50 sum haplotypes_contigs 3828 2701 1127 471 123243 2503 1580 2590 9,584,985
UMD_2.6.a_g_070109
Gaps: 65900
- all 65900 gaps are "fragment yes"
UMD_2.6.a_g_070509
- Combine UMD_2.6.a_g_070109 (Guillaume's assembly) and UMD2.6.1 (Daniela's)
UMD_2.6.a_g_070709
- Remove from UMD_2.6.a_g_070509
- ~ 21 ChrY ctg & ~ 39 ChrY deg
- ~ 4206 haplotype variants (6.63Mbp) within 1K from one another
. elem <2000 >2000 min max mean med n50 sum ctg_deg.variants.placed 6654 5504 1150 263 42158 1761 1189 1772 11719653 ctg_deg.variants.placed.sameChr 5374 4455 919 263 42158 1748 1152 1775 9393815 ctg_deg.variants.placed.within_100K 4790 4100 690 263 42158 1651 1128 1605 7911160 ctg_deg.variants.placed.within_1K* 4206 3679 527 263 42158 1577 1107 1508 6633118
Summary:
ctg+deg <2000 >=2000 min max mean med n50 sum Chr1..29,X 72197 20763 51434 65 1160130 36536 13055 97328 2,637,809,286 ChrU 3752 2587 1165 362 179692 3284 1447 6427 12,324,356 ChrY-contigs 315 266 49 224 26490 2249 974 6679 708,535 contigs.haplotype-variants 40198 36720 3478 263 51828 1460 1203 1361 58,698,457 deg.unplaced.less_2K 224945 224945 0 65 1996 972 983 990 218,847,978
Issues
- 490 scf don't have all ctgs placed (865 ctgs)
- 699 reliable contigs (3.25Mbp) unplaced
difference.pl ctg.reliable.ids UMD_2.6.a_g_070709/Chr.posmap | getSummary.pl -i 2 -z 2000 elem <2000 >2000 min max mean med n50 sum 699 405 294 362 123243 4660 1759 14809 3257635
- Rearrangements UMD2.0 vs UMD_2.6.a_g_070709 : ~ 25 ctgs>50K ; ~ 12 scf
- AGP file format: "fragment yes" should be preserved even if gap type=U
grep -A 2 7180001925241 UMD_2.6.a_g_070109/Chr.agp Chr30 131686408 131762880 14157 W 7180001925241 1 76473 + Chr30 131762881 131762980 14158 U 100 fragment yes Chr30 131762981 131796167 14159 W 7180001925242 1 33187 +
grep -A 2 7180001925241 ../UMD_2.6.a_g_070709/Chr.agp ChrX 133855417 133931889 15581 W 7180001925241 1 76473 + ChrX 133931890 133931989 15582 U 100 contig no ChrX 133931990 133965176 15583 W 7180001925242 1 33187 +
- Liliana found 18 ctg + deg that have genes
Fixed case B (7180001932648,7180001925237)
- Bob (Missouri)
- scf7180002041216 (43 ctgs, 1.44Bmp) should be split : Chr15 (36 ctgs) , Chr5 (first 7 ctgs) ctg7180002003576,ctg7180002003577 moved from Chr9=>Chr5
=> Chr5
1 7180002003574 7180002041216 0 19886 f # already on Chr5
2 7180001694221 7180002041216 19906 20911 r
3 7180001978231 7180002041216 20931 52725 r
4 7180001946738 7180002041216 53559 55402 r
5 7180002003575 7180002041216 55422 92145 f
6 7180002003576 7180002041216 92165 200007 f
7 7180002003577 7180002041216 200456 300048 f
- scf7180002041153 (48 ctgs, 2.36Mbp): assignment to Chr11 seems correct, Chr6(0)
UMD_2.6.a_g_071709 -> UMD_Freeze2.99
Changes vs UMD_2.6.a_g_070709:
- 17 contigs/degenerates recruited by Liliana based on mRNA alignments got placed on chromosomes
- Chr9,15 => Chr5 correction : a scaffold got broken between 2 chromosomes & and several contigs got moved to Chr5
- 697 unplaced contigs from placed scaffolds got placed as well
Contig placement summary:
#ctg+deg <2Kbp >=2Kbp min max mean med n50 sum ================================================================================================== Chr1..29,X 72911 21180 51731 65 1160130 36223 12706 97232 2641097363 ChrU 3365 2448 917 224 179692 2898 1348 5399 9754701 contigs.haplotype-variants 40198 36720 3478 263 51828 1460 1203 1361 58698457 deg.unplaced.less_2K 224933 224933 0 65 1996 972 983 990 218837572
Chr1..29,X(new) 714 417 297 177 123243 4605 1741 14809 3288596
ChrY-contigs 315 266 49 224 26490 2249 974 6679 708535 ChrY-contigs.SHOTGUN_ONLY 144 140 4 804 4224 993 882 888 143047 ===================================================================================================
Comments:
- "Chr1..29,X", ChrU, contigs.haplotype-variants, deg.unplaced.less_2K are mutually exclusive sets
- "Chr1..29,X(new)" are contigs which were not placed in UMD_2.6.a_g_071709.
- 17 contig them were added by Liliana (1 failed)
- the rest are reliable contigs left unplaced by Aleksey/Guillaume program
- ChrY-contigs.SHOTGUN_ONLY are a subset of ChrY-contigs which don't contain only SHOTGUN reads
- ChrY-contigs are part of ChrU
Files (walnut):
/scratch1/bos_taurus/Assembly/2009_0312_CA/scf_placements/UMD_2.6.a_g_071709/ # FASTA & AGP format /scratch1/bos_taurus/Assembly/2009_0312_CA/scf_placements/UMD_2.6.a_g_071709/nucmer_UMD2.0/ # nucmer alignments to UMD2.0
Files (freeze):
/fs/szasmg3/bos_taurus/UMD_Freeze2.99/ # FASTA & AGP format /fs/szasmg3/bos_taurus/UMD_Freeze2.99/ncbi_files # SEQUIN format
Ftp:
ftp://ftp.cbcb.umd.edu/pub/salzberg/Bos_taurus_2.99/ -> /fs/ftp-cbcb/pub/salzberg/Bos_taurus_2.99/
UMD_2.6.a_g_072109 -> UMD_Freeze3.0
Changes vs UMD_2.6.a_g_070709:
- Delete 97 contaminated sequences found by NCBI (all except the primates) : http://www.ncbi.nlm.nih.gov/projects/WGS/screens/DAAA02_071709/
- Delete 441 haplotype variants found by Guillaume
- Trim 54 partial contaminants (contaminants were on the ends)
- Trim 7 terminal N's
Gaps: 75739
- all 27103 N gaps are "fragment yes"
- all 48636 U gaps are "contig no"
Files (walnut):
/scratch1/bos_taurus/Assembly/2009_0312_CA/scf_placements/UMD_2.6.a_g_072109/ # FASTA & AGP format
Files (freeze):
/fs/szasmg3/bos_taurus/UMD_Freeze3.0/ # FASTA & AGP format /fs/szasmg3/bos_taurus/UMD_Freeze3.0/ncbi_files # SEQUIN format
Ftp:
ftp://ftp.cbcb.umd.edu/pub/data/assembly/Bos_taurus/Bos_taurus_UMD_3.0/ -> pub/data/assembly/Bos_taurus/Bos_taurus_UMD_3.0/
Issues:
- 7180001836672 941bp deg on Chr4 : aligns on all its length to the cow mitochondrion; placed based on human synteny
- mitochondrion screening was done only on contigs, not on degenerates
- Align all Chr*.fasta files to cow mitochondrion; show-coords -I 90 -L 600
[S1] [E1] | [S2] [E2] | [LEN 1] [LEN 2] | [% IDY] | [LEN R] [LEN Q] | [COV R] [COV Q] | [TAGS] =============================================================================================================================== 3603 4930 | 6 1333 | 1328 1328 | 90.89 | 16339 1333 | 8.13 99.62 | AY676873 ctg7180001759596 [CONTAINS] 13284 14521 | 1214 1 | 1238 1214 | 97.90 | 16339 1216 | 7.58 99.84 | AY676873 deg7180001872458 [CONTAINS] 15731 16339 | 1 608 | 609 608 | 99.18 | 16339 941 | 3.73 64.61 | AY676873 deg7180001836672 [END]
- Delete 2 degenerates
- Summary 3.a --Dpuiu 11:04, 5 August 2009 (EDT)
. ctg+deg <2Kbp >=2Kbp min max mean med n50 sum ====================================================================================================== Chr1..29,X 72479 20862 51617 65 1160130 36424 12941 103785 2639984487 ChrU 3285 2404 881 224 179692 2890 1338 5425 9496583 Chr 75764 23266 52498 65 1160130 34970 11207 96955 2649481070 contigs.haplotype-variants 40611 36984 3627 263 97877 1476 1205 1372 59958728 deg.unplaced.less_2K 224933 224933 0 65 1996 972 983 990 218837572 ChrY-contigs 314 266 48 224 26490 2210 973 6539 694140 ChrY-contigs.SHOTGUN_ONLY 144 140 4 804 4224 993 882 888 143047 ======================================================================================================
UMD_Freeze3.1
--Dpuiu 11:42, 19 November 2009 (EST)
- Only changed some of the gap specifications
- Original(CA):
ctg: 90135 deg: 251413 scf: 39978
- UMD3.1 AGP:
ctg: 60499 deg: 15229 scf(CA): 11458 ctg(unoriented) 2118 scf(placed chr) 3193 scf(unplaced chr) 3285
Files:
/fs/szasmg3/bos_taurus/UMD_Freeze3.1
ToDo
- Align all UMD2.0 ChrU ctg/deg 10Kbp+ to our assembly; make sure everything aligns (Steven's suggestion)
/scratch1/bos_taurus/Assembly/2009_0312_CA/scf_placements/UMD_2.6.a_g_071709/nucmer_UMD2.0
- all ctgs: 113K, 244Mbp
- 10Kbp+ ctgs: 2561, 54Mbp
- 53,951,168 out of 54,971,011 bp covered (98% of the sequence)
- Most ctgs were added to ChrU
join2.pl -i 4 UMD2.0.ChrU.10K.maxCvg.pair ../other/Chr.posmap | awk '{print $10,$2}' | ~/bin/sum2.pl | sort -nk3 -r
Chr30 252 7,511,975
Chr1 171 3263575
Chr6 138 3043388
Chr12 152 2676615
ChrU 100 2600562
...
Chr28 29 473523
all 2651 54,960,233
1. Remove remain_haps
. elem <2000 >2000 min max mean med n50 sum remain_haps 436 282 154 471 37860 2600 1693 2819 1,133,874 remain_haps(Chr1..30,U) 408 264 144 471 37860 2636 1692 2883 1,075,616 # 77 from ChrU remain_haps(Chr1..30) 335 226 109 471 37860 2480 1661 2728 831,068
2. Remove/trim contaminants
3. Remove/trim N's