Problem in generating references #5
Description
Hello,
I'm interested in trying "CircMiMi" because my research focuses on the interactions of circRNAs.
I attempted to generate the reference files, but I always encountered an error when adding them to the database.
Could you please advise me on how to resolve this? Thanks.
(base) yu@yu-virtual-machine:~/Downloads/CircMiMi-master/circmimi$ circmimi_tools genref --species hsa --source gencode --version 34 refs/
2023-01-20 14:09:28,704 - NumExpr defaulting to 4 threads.
--2023-01-20 14:09:36-- ftp://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_human/release_34/gencode.v34.annotation.gtf.gz
=> ‘./gencode.v34.annotation.gtf.gz’
Resolving ftp.ebi.ac.uk (ftp.ebi.ac.uk)... 193.62.193.138
Connecting to ftp.ebi.ac.uk (ftp.ebi.ac.uk)|193.62.193.138|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done. ==> PWD ... done.
==> TYPE I ... done. ==> CWD (1) /pub/databases/gencode/Gencode_human/release_34 ... done.
==> SIZE gencode.v34.annotation.gtf.gz ... 43164654
==> PASV ... done. ==> RETR gencode.v34.annotation.gtf.gz ... done.
Length: 43164654 (41M) (unauthoritative)
gencode.v34.annotat 100%[===================>] 41.16M 3.23MB/s in 13s
2023-01-20 14:09:51 (3.16 MB/s) - ‘./gencode.v34.annotation.gtf.gz’ saved [43164654]
--2023-01-20 14:09:51-- ftp://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_human/release_34/GRCh38.primary_assembly.genome.fa.gz
=> ‘./GRCh38.primary_assembly.genome.fa.gz’
Resolving ftp.ebi.ac.uk (ftp.ebi.ac.uk)... 193.62.193.138
Connecting to ftp.ebi.ac.uk (ftp.ebi.ac.uk)|193.62.193.138|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done. ==> PWD ... done.
==> TYPE I ... done. ==> CWD (1) /pub/databases/gencode/Gencode_human/release_34 ... done.
==> SIZE GRCh38.primary_assembly.genome.fa.gz ... 844691642
==> PASV ... done. ==> RETR GRCh38.primary_assembly.genome.fa.gz ... done.
Length: 844691642 (806M) (unauthoritative)
GRCh38.primary_asse 100%[===================>] 805.56M 2.46MB/s in 3m 51s
2023-01-20 14:13:45 (3.49 MB/s) - ‘./GRCh38.primary_assembly.genome.fa.gz’ saved [844691642]
--2023-01-20 14:13:45-- https://www.mirbase.org/ftp/22/mature.fa.gz
Resolving www.mirbase.org (www.mirbase.org)... 130.88.97.249
Connecting to www.mirbase.org (www.mirbase.org)|130.88.97.249|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 803746 (785K) [application/x-gzip]
Saving to: ‘./mature.fa.gz’
mature.fa.gz 100%[===================>] 784.91K 293KB/s in 2.7s
2023-01-20 14:13:48 (293 KB/s) - ‘./mature.fa.gz’ saved [803746/803746]
--2023-01-20 14:13:48-- https://mirtarbase.cuhk.edu.cn/~miRTarBase/miRTarBase_2019/cache/download/7.0/miRTarBase_MTI.xlsx
Resolving mirtarbase.cuhk.edu.cn (mirtarbase.cuhk.edu.cn)... 116.31.95.60
Connecting to mirtarbase.cuhk.edu.cn (mirtarbase.cuhk.edu.cn)|116.31.95.60|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 26285698 (25M) [application/vnd.openxmlformats-officedocument.spreadsheetml.sheet]
Saving to: ‘./miRTarBase_MTI.xlsx’
miRTarBase_MTI.xlsx 100%[===================>] 25.07M 1.18MB/s in 22s
2023-01-20 14:14:11 (1.15 MB/s) - ‘./miRTarBase_MTI.xlsx’ saved [26285698/26285698]
--2023-01-20 14:14:11-- https://treeslab1.genomics.sinica.edu.tw/CircMiMi/refs/miRDB_data/v6.0/miRDB_v6.0_hsa.simple.tsv.gz
Resolving treeslab1.genomics.sinica.edu.tw (treeslab1.genomics.sinica.edu.tw)... 140.109.55.12
Connecting to treeslab1.genomics.sinica.edu.tw (treeslab1.genomics.sinica.edu.tw)|140.109.55.12|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 6423753 (6.1M) [application/x-gzip]
Saving to: ‘./miRDB_v6.0_hsa.simple.tsv.gz’
miRDB_v6.0_hsa.simp 100%[===================>] 6.13M 1.59MB/s in 3.8s
2023-01-20 14:14:15 (1.59 MB/s) - ‘./miRDB_v6.0_hsa.simple.tsv.gz’ saved [6423753/6423753]
--2023-01-20 14:14:15-- https://treeslab1.genomics.sinica.edu.tw/CircMiMi/refs/ENCORI_miRNA/mir_target_ref.ENCORI.miRBase_v22.simple.tsv.gz
Resolving treeslab1.genomics.sinica.edu.tw (treeslab1.genomics.sinica.edu.tw)... 140.109.55.12
Connecting to treeslab1.genomics.sinica.edu.tw (treeslab1.genomics.sinica.edu.tw)|140.109.55.12|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4886391 (4.7M) [application/x-gzip]
Saving to: ‘./mir_target_ref.ENCORI.miRBase_v22.simple.tsv.gz’
mir_target_ref.ENCO 100%[===================>] 4.66M 1.16MB/s in 4.0s
2023-01-20 14:14:20 (1.16 MB/s) - ‘./mir_target_ref.ENCORI.miRBase_v22.simple.tsv.gz’ saved [4886391/4886391]
--2023-01-20 14:14:20-- ftp://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_human/release_34/gencode.v34.pc_transcripts.fa.gz
=> ‘./gencode.v34.pc_transcripts.fa.gz’
Resolving ftp.ebi.ac.uk (ftp.ebi.ac.uk)... 193.62.193.138
Connecting to ftp.ebi.ac.uk (ftp.ebi.ac.uk)|193.62.193.138|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done. ==> PWD ... done.
==> TYPE I ... done. ==> CWD (1) /pub/databases/gencode/Gencode_human/release_34 ... done.
==> SIZE gencode.v34.pc_transcripts.fa.gz ... 43164068
==> PASV ... done. ==> RETR gencode.v34.pc_transcripts.fa.gz ... done.
Length: 43164068 (41M) (unauthoritative)
gencode.v34.pc_tran 100%[===================>] 41.16M 3.69MB/s in 13s
2023-01-20 14:14:35 (3.10 MB/s) - ‘./gencode.v34.pc_transcripts.fa.gz’ saved [43164068]
--2023-01-20 14:14:35-- ftp://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_human/release_34/gencode.v34.lncRNA_transcripts.fa.gz
=> ‘./gencode.v34.lncRNA_transcripts.fa.gz’
Resolving ftp.ebi.ac.uk (ftp.ebi.ac.uk)... 193.62.193.138
Connecting to ftp.ebi.ac.uk (ftp.ebi.ac.uk)|193.62.193.138|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done. ==> PWD ... done.
==> TYPE I ... done. ==> CWD (1) /pub/databases/gencode/Gencode_human/release_34 ... done.
==> SIZE gencode.v34.lncRNA_transcripts.fa.gz ... 14764808
==> PASV ... done. ==> RETR gencode.v34.lncRNA_transcripts.fa.gz ... done.
Length: 14764808 (14M) (unauthoritative)
gencode.v34.lncRNA_ 100%[===================>] 14.08M 3.68MB/s in 4.7s
2023-01-20 14:14:42 (3.01 MB/s) - ‘./gencode.v34.lncRNA_transcripts.fa.gz’ saved [14764808]
--2023-01-20 14:14:42-- https://treeslab1.genomics.sinica.edu.tw/CircMiMi/refs/ENCORI_RBP/ENCORI_RBP_binding_sites.hg38.AGO.bed.gz
Resolving treeslab1.genomics.sinica.edu.tw (treeslab1.genomics.sinica.edu.tw)... 140.109.55.12
Connecting to treeslab1.genomics.sinica.edu.tw (treeslab1.genomics.sinica.edu.tw)|140.109.55.12|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 18687603 (18M) [application/x-gzip]
Saving to: ‘./ENCORI_RBP_binding_sites.hg38.AGO.bed.gz’
ENCORI_RBP_binding_ 100%[===================>] 17.82M 3.67MB/s in 7.5s
2023-01-20 14:14:50 (2.37 MB/s) - ‘./ENCORI_RBP_binding_sites.hg38.AGO.bed.gz’ saved [18687603/18687603]
--2023-01-20 14:19:22-- https://www.mirbase.org/ftp/21/miRNA.dat.gz
Resolving www.mirbase.org (www.mirbase.org)... 130.88.97.249
Connecting to www.mirbase.org (www.mirbase.org)|130.88.97.249|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3245620 (3.1M) [application/x-gzip]
Saving to: ‘./miRNA.dat.gz’
miRNA.dat.gz 100%[===================>] 3.09M 864KB/s in 4.3s
2023-01-20 14:19:27 (731 KB/s) - ‘./miRNA.dat.gz’ saved [3245620/3245620]
--2023-01-20 14:19:27-- https://www.mirbase.org/ftp/22/miRNA.dat.gz
Resolving www.mirbase.org (www.mirbase.org)... 130.88.97.249
Connecting to www.mirbase.org (www.mirbase.org)|130.88.97.249|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4115022 (3.9M) [application/x-gzip]
Saving to: ‘./miRNA.dat.gz’
miRNA.dat.gz 100%[===================>] 3.92M 1.10MB/s in 4.4s
2023-01-20 14:19:32 (906 KB/s) - ‘./miRNA.dat.gz’ saved [4115022/4115022]
--2023-01-20 14:19:32-- https://www.mirbase.org/ftp/22/miRNA.diff.gz
Resolving www.mirbase.org (www.mirbase.org)... 130.88.97.249
Connecting to www.mirbase.org (www.mirbase.org)|130.88.97.249|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 133886 (131K) [application/x-gzip]
Saving to: ‘./miRNA.diff.gz’
miRNA.diff.gz 100%[===================>] 130.75K 147KB/s in 0.9s
2023-01-20 14:19:34 (147 KB/s) - ‘./miRNA.diff.gz’ saved [133886/133886]
Killed
(base) yu@yu-virtual-machine:~/Downloads/CircMiMi-master/circmimi$ circmimi_tools genref --species hsa --source gencode --version 34 refs/
2023-01-20 14:20:40,995 - NumExpr defaulting to 4 threads.
Traceback (most recent call last):
File "/home/yu/anaconda3/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context
self.dialect.do_execute(
File "/home/yu/anaconda3/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
cursor.execute(statement, parameters)
sqlite3.IntegrityError: UNIQUE constraint failed: chromosome.name
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/yu/anaconda3/bin/circmimi_tools", line 8, in <module>
sys.exit(cli())
File "/home/yu/anaconda3/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "/home/yu/anaconda3/lib/python3.9/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/home/yu/anaconda3/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/yu/anaconda3/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/yu/anaconda3/lib/python3.9/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/home/yu/anaconda3/lib/python3.9/site-packages/circmimi/scripts/circmimi_tools.py", line 164, in generate_references
info, ref_files = genref.generate(species, source, version, ref_dir)
File "/home/yu/anaconda3/lib/python3.9/site-packages/circmimi/reference/genref.py", line 311, in generate
anno_ref.generate()
File "/home/yu/anaconda3/lib/python3.9/site-packages/circmimi/reference/genref.py", line 26, in generate
gendb.generate(self.src_name, self.filename)
File "/home/yu/anaconda3/lib/python3.9/site-packages/circmimi/reference/gendb.py", line 196, in generate
_write_data_to_db(session, tables_raw_data.chromosomes, Chromosome)
File "/home/yu/anaconda3/lib/python3.9/site-packages/circmimi/reference/gendb.py", line 179, in _write_data_to_db
session.commit()
File "/home/yu/anaconda3/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1451, in commit
self._transaction.commit(_to_root=self.future)
File "/home/yu/anaconda3/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 829, in commit
self._prepare_impl()
File "/home/yu/anaconda3/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl
self.session.flush()
File "/home/yu/anaconda3/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 3383, in flush
self._flush(objects)
File "/home/yu/anaconda3/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 3523, in _flush
transaction.rollback(_capture_exception=True)
File "/home/yu/anaconda3/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
compat.raise_(
File "/home/yu/anaconda3/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
raise exception
File "/home/yu/anaconda3/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 3483, in _flush
flush_context.execute()
File "/home/yu/anaconda3/lib/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute
rec.execute(self)
File "/home/yu/anaconda3/lib/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute
util.preloaded.orm_persistence.save_obj(
File "/home/yu/anaconda3/lib/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj
_emit_insert_statements(
File "/home/yu/anaconda3/lib/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements
result = connection._execute_20(
File "/home/yu/anaconda3/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20
return meth(self, args_10style, kwargs_10style, execution_options)
File "/home/yu/anaconda3/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 332, in _execute_on_connection
return connection._execute_clauseelement(
File "/home/yu/anaconda3/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement
ret = self._execute_context(
File "/home/yu/anaconda3/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context
self._handle_dbapi_exception(
File "/home/yu/anaconda3/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2043, in _handle_dbapi_exception
util.raise_(
File "/home/yu/anaconda3/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
raise exception
File "/home/yu/anaconda3/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context
self.dialect.do_execute(
File "/home/yu/anaconda3/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (sqlite3.IntegrityError) UNIQUE constraint failed: chromosome.name
[SQL: INSERT INTO chromosome (name) VALUES (?)]
[parameters: ('chr1',)]
(Background on this error at: https://sqlalche.me/e/14/gkpj)
(base) yu@yu-virtual-machine:~/Downloads/CircMiMi-master/circmimi$ circmimi_tools checking -r refs/ -i circRNAs.gencode_format.tsv -o circRNAs.gencode_format. --dist 5000
2023-01-20 14:24:13,174 - NumExpr defaulting to 4 threads.
Traceback (most recent call last):
File "/home/yu/anaconda3/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context
self.dialect.do_execute(
File "/home/yu/anaconda3/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
cursor.execute(statement, parameters)
sqlite3.OperationalError: no such table: chromosome
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/yu/anaconda3/bin/circmimi_tools", line 8, in <module>
sys.exit(cli())
File "/home/yu/anaconda3/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "/home/yu/anaconda3/lib/python3.9/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/home/yu/anaconda3/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/yu/anaconda3/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/yu/anaconda3/lib/python3.9/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/home/yu/anaconda3/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/yu/anaconda3/lib/python3.9/site-packages/circmimi/scripts/circmimi_tools.py", line 324, in default_checking
ctx.invoke(
File "/home/yu/anaconda3/lib/python3.9/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/home/yu/anaconda3/lib/python3.9/site-packages/circmimi/scripts/circmimi_tools.py", line 208, in check_annotation
circ_events.check_annotation(anno_db)
File "/home/yu/anaconda3/lib/python3.9/site-packages/circmimi/circ.py", line 146, in check_annotation
self._annotator = Annotator(anno_db_file)
File "/home/yu/anaconda3/lib/python3.9/site-packages/circmimi/annotation.py", line 116, in __init__
self._db = Annotation(anno_db_file)
File "/home/yu/anaconda3/lib/python3.9/site-packages/circmimi/annotation.py", line 17, in __init__
self.session.query(Chromosome.name, Chromosome.id).all()
File "/home/yu/anaconda3/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 2768, in all
return self._iter().all()
File "/home/yu/anaconda3/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 2903, in _iter
result = self.session.execute(
File "/home/yu/anaconda3/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1712, in execute
result = conn._execute_20(statement, params or {}, execution_options)
File "/home/yu/anaconda3/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20
return meth(self, args_10style, kwargs_10style, execution_options)
File "/home/yu/anaconda3/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 332, in _execute_on_connection
return connection._execute_clauseelement(
File "/home/yu/anaconda3/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement
ret = self._execute_context(
File "/home/yu/anaconda3/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context
self._handle_dbapi_exception(
File "/home/yu/anaconda3/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2043, in _handle_dbapi_exception
util.raise_(
File "/home/yu/anaconda3/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
raise exception
File "/home/yu/anaconda3/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context
self.dialect.do_execute(
File "/home/yu/anaconda3/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table: chromosome
[SQL: SELECT chromosome.name AS chromosome_name, chromosome.id AS chromosome_id
FROM chromosome]
(Background on this error at: https://sqlalche.me/e/14/e3q8)