LocalDataTransfer - Revision history

Mbaney at 16:23, 25 October 2024

2024-10-25T16:23:25Z

Mbaney: Mbaney moved page Data Transfer to LocalDataTransfer

2024-10-25T16:21:08Z

Mbaney moved page Data Transfer to LocalDataTransfer

← Older revision	Revision as of 16:21, 25 October 2024
(No difference)

Mbaney at 15:45, 8 April 2024

2024-04-08T15:45:17Z

← Older revision		Revision as of 15:45, 8 April 2024
Line 1:		Line 1:
	Depending on the amount of data you are trying to transfer there are different commands that you should use. The classic <code>cp</code> command has a number of edge cases in which it will not copy everything that is expected, so the '''only time''' that one should use the <code>cp</code> command is if the copy can be verified, such as if you are moving a single file. The better choice for transferring data is by using <code>tar</code> or <code>gtar</code>. After transferring files or directories, you can use <code>rsync</code> to check that everything moved correctly and ~~it will update~~ files ~~that have been changed~~.		Depending on the amount of data you are trying to transfer there are different commands that you should use. The classic <code>cp</code> command has a number of edge cases in which it will not copy everything that is expected, so the '''only time''' that one should use the <code>cp</code> command is if the copy can be verified, such as if you are moving a single file. The better choice for transferring data is by using <code>tar</code> or <code>gtar</code>. After transferring files or directories, you can use <code>rsync</code> to check that everything moved correctly and to perform a final sync on changed files.

	==Transfer a single file==		==Transfer a single file==
Line 22:		Line 22:

	==Transfer a directory or large amounts of data to another location==		==Transfer a directory or large amounts of data to another location==

	If you want to transfer a whole directory or a large amount of data to another location you can use the <code>gtar</code> command.		If you want to transfer a whole directory or a large amount of data to another location you can use the <code>gtar</code> command.
	* To combine two files in your current directory in an archive: <pre>gtar -cpvf archive.tar file1 file2</pre>		* To combine two files in your current directory in an archive: <pre>gtar -cpvf archive.tar file1 file2</pre>
Line 55:		Line 54:

	<code>rsync</code> can also be used for the initial transfer of data if you expect the transfer to be interrupted. Otherwise, this method should not be used as it takes more time and memory.		<code>rsync</code> can also be used for the initial transfer of data if you expect the transfer to be interrupted. Otherwise, this method should not be used as it takes more time and memory.

			To run rsync to copy files, the format for the command is the same as written below under "Verifying transfer."

	==Verifying transfer==		==Verifying transfer==

	To verify that your transfer copied everything you can use the <code>rsync</code> command, which will compare the two directories' contents and will update the files in which it sees differences.		To verify that your transfer copied everything you can use the <code>rsync</code> command, which will compare the two directories' contents and will update the files in which it sees differences.

Mbaney at 19:48, 20 November 2023

2023-11-20T19:48:22Z

← Older revision		Revision as of 19:48, 20 November 2023
Line 5:		Line 5:
	If you want to transfer a single file into another directory you can use the <code>cp</code> command. The format for the <code>cp</code> command is: <code>cp Source Destination</code>. If destination is omitted, it will make a copy of the file(s) in the current directory.		If you want to transfer a single file into another directory you can use the <code>cp</code> command. The format for the <code>cp</code> command is: <code>cp Source Destination</code>. If destination is omitted, it will make a copy of the file(s) in the current directory.

	* To copy the file foo.txt ~~from~~ your home directory ~~into~~ Documents		* To copy the file foo.txt into your home directory's Documents directory

	<pre>cp foo.txt ~/Documents</pre>		<pre>cp foo.txt ~/Documents</pre>

	* To transfer the file foo.txt from your current directory to another host use '''<tt>scp</tt>''' (secure copy) with $USERNAME@<FQDN>:<PATH>		* To transfer the file foo.txt from your current directory to another host use '''<tt>scp</tt>''' (secure copy) with <USERNAME>@<FQDN>:<PATH>

	<pre>scp foo.txt USERNAME@example1.umiacs.umd.edu:~/Documents</pre>		<pre>scp foo.txt USERNAME@example1.umiacs.umd.edu:~/Documents</pre>

Mbaney at 18:01, 6 March 2023

2023-03-06T18:01:31Z

← Older revision		Revision as of 18:01, 6 March 2023
Line 23:		Line 23:
	==Transfer a directory or large amounts of data to another location==		==Transfer a directory or large amounts of data to another location==

	If you want to transfer a whole directory or a large amount of data to another location you can ~~use the <code>code</code> command. Even though we~~ use the <code>gtar</code> command ~~to copy and transfer a directory, the more common use of <code>gtar</code> is to make tar balls (to create archives of specified directories or files)~~.		If you want to transfer a whole directory or a large amount of data to another location you can use the <code>gtar</code> command.
			* To combine two files in your current directory in an archive: <pre>gtar -cpvf archive.tar file1 file2</pre>
	* To combine two files in your current directory in an archive:		* To re-archive the data back in a different directory, for example Documents in your home directory: <pre>gtar -C ~/Documents/ -xpvf archive.tar</pre>

	<pre>gtar -cpvf archive.tar file1 file2</pre>

	To re-archive the data back in a different directory, for example Documents in your home directory:

	<pre>gtar -C ~/Documents/ -xpvf archive.tar</pre>

	This command will archive all files (including those in subdirectories) within the current directory and re-create the files in the directory that you specify.		This command will archive all files (including those in subdirectories) within the current directory and re-create the files in the directory that you specify.
Line 40:		Line 34:
	In this command, the target is a dash <tt>'-'</tt> which stands for standard output, and the source is a period <tt>'.'</tt> which is interpreted as all files in your current directory. The standard output is piped to the second command, which has as a source a dash <tt>'-'</tt> (which the shell interprets as standard input). The output of the first command is piped and becomes the input of the second command. The target directory in your second command is <tt>/dir</tt>.		In this command, the target is a dash <tt>'-'</tt> which stands for standard output, and the source is a period <tt>'.'</tt> which is interpreted as all files in your current directory. The standard output is piped to the second command, which has as a source a dash <tt>'-'</tt> (which the shell interprets as standard input). The output of the first command is piped and becomes the input of the second command. The target directory in your second command is <tt>/dir</tt>.

	* To transfer all files from your documents to a folder in your home directory called foo:		* To transfer all files from your documents to a folder in your home directory called foo: <pre>cd ~/Documents gtar -cpf - . \| gtar -C ~/foo -xpvf -</pre>

	<pre>cd ~/Documents

	gtar -cpf - . \| gtar -C ~/foo -xpvf -</pre>

	When you use this command it will display a list of the files it has transferred.		When you use this command it will display a list of the files it has transferred.

Mbaney at 15:10, 14 March 2022

2022-03-14T15:10:22Z

← Older revision		Revision as of 15:10, 14 March 2022
Line 62:		Line 62:
	<pre>gtar -cpf - /foo/ \| ssh USERNAME@example1.umiacs.umd.edu gtar -C /tmp -xpvf -</pre>		<pre>gtar -cpf - /foo/ \| ssh USERNAME@example1.umiacs.umd.edu gtar -C /tmp -xpvf -</pre>

	If your data transfer is interrupted, you can use the ~~'''~~<tt>rsync</tt>~~'''~~ command listed below to copy the rest of the files without creating doubles of files that have already been transferred.		If your data transfer is interrupted, you can use the <code>rsync</code> command listed below to copy the rest of the files without creating doubles of files that have already been transferred.

	~~'''~~<tt>rsync</tt>~~'''~~ can also be used for the initial transfer of data if you expect the transfer to be interrupted. Otherwise, this method should not be used as it takes more time and memory.		<code>rsync</code> can also be used for the initial transfer of data if you expect the transfer to be interrupted. Otherwise, this method should not be used as it takes more time and memory.

	==Verifying transfer==		==Verifying transfer==

	To verify that your transfer copied everything you can use the ~~'''~~<tt>rsync</tt>~~'''~~ command, which will compare the two directories contents and will update the files in which it sees differences.		To verify that your transfer copied everything you can use the <code>rsync</code> command, which will compare the two directories' contents and will update the files in which it sees differences.

	The format for the ~~'''~~<tt>rsync</tt>~~'''~~ command is:		The format for the <code>rsync</code> command is:

	<pre>rsync -aH /source/ /target</pre>		<pre>rsync -aH /source/ /target</pre>

Mbaney at 15:06, 14 March 2022

2022-03-14T15:06:32Z

← Older revision		Revision as of 15:06, 14 March 2022
Line 1:		Line 1:
	Depending on the amount of data you are trying to transfer there are different commands that you should use. The classic cp command has a number of edge cases in which it will not copy everything that is expected, so the '''only time''' that one should use the cp command is if the copy can be verified, such as if you are moving a single file. The better choice for ~~transfering~~ data is by using tar/gtar. After ~~transfering~~ files or directories you can use rsync to check that everything moved correctly and it will update files that have been changed.		Depending on the amount of data you are trying to transfer there are different commands that you should use. The classic <code>cp</code> command has a number of edge cases in which it will not copy everything that is expected, so the '''only time''' that one should use the <code>cp</code> command is if the copy can be verified, such as if you are moving a single file. The better choice for transferring data is by using <code>tar</code> or <code>gtar</code>. After transferring files or directories, you can use <code>rsync</code> to check that everything moved correctly and it will update files that have been changed.

	==Transfer a single file==		==Transfer a single file==

	If you want to transfer a single file into another directory you can use the ~~'''~~<tt>cp</tt>~~'''~~ command. The format for the ~~'''~~<tt>cp</tt>~~'''~~ command is: <tt>cp Source Destination</tt>. If destination is omitted, it will make a copy of the file(s) in the current directory.		If you want to transfer a single file into another directory you can use the <code>cp</code> command. The format for the <code>cp</code> command is: <code>cp Source Destination</code>. If destination is omitted, it will make a copy of the file(s) in the current directory.

	* To copy the file foo.txt from your home directory into Documents		* To copy the file foo.txt from your home directory into Documents
Line 19:		Line 19:
	<pre>scp USERNAME@example1.umiacs.umd.edu:~/Documents/foo.txt ~/Videos</pre>		<pre>scp USERNAME@example1.umiacs.umd.edu:~/Documents/foo.txt ~/Videos</pre>

	'''Important:''' The ~~'''~~<tt>cp</tt>~~'''~~ command should only be used for small file transfers. If you try to transfer a large amount it is possible that ~~'''~~<tt>cp</tt>~~'''~~ will not copy all the files over properly. To learn more about the ~~'''~~<tt>cp</tt>~~'''~~ command type <tt>man cp</tt> on the in the terminal.		'''Important:''' The <code>cp</code> command should only be used for small file transfers. If you try to transfer a large amount it is possible that <code>cp</code> will not copy all the files over properly. To learn more about the <code>cp</code> command type <code>man cp</code> on the in the terminal.

	==Transfer a directory or large amounts of data to another location==		==Transfer a directory or large amounts of data to another location==

	If you want to transfer a whole directory or a large amount of data to another location you can use the ~~'''~~<tt>~~gtar~~</tt>~~'''~~ command. Even though we use the ''<tt>gtar</tt>~~'''~~ command to copy and transfer a directory, the more common use of ~~'''~~<tt>gtar</tt>~~'''~~ is to make tar balls (to create archives of specified directories or files).		If you want to transfer a whole directory or a large amount of data to another location you can use the <code>code</code> command. Even though we use the <code>gtar</code> command to copy and transfer a directory, the more common use of <code>gtar</code> is to make tar balls (to create archives of specified directories or files).

	* To combine two files in your current directory in an archive:		* To combine two files in your current directory in an archive:
Line 35:		Line 35:
	This command will archive all files (including those in subdirectories) within the current directory and re-create the files in the directory that you specify.		This command will archive all files (including those in subdirectories) within the current directory and re-create the files in the directory that you specify.

	The format for the gtar command for data transfer is:		The format for the <code>gtar</code> command for data transfer is:
	<pre>gtar -cpf - . \| gtar -C /dir -xpvf -</pre>		<pre>gtar -cpf - . \| gtar -C /dir -xpvf -</pre>

Jayid07 at 19:42, 2 February 2021

2021-02-02T19:42:41Z

← Older revision		Revision as of 19:42, 2 February 2021
Line 62:		Line 62:
	<pre>gtar -cpf - /foo/ \| ssh USERNAME@example1.umiacs.umd.edu gtar -C /tmp -xpvf -</pre>		<pre>gtar -cpf - /foo/ \| ssh USERNAME@example1.umiacs.umd.edu gtar -C /tmp -xpvf -</pre>

	If your data transfer is interrupted, you can use the '''<tt>rsync<tt>''' command listed below to copy the rest of the files without creating doubles of files that have already been transferred.		If your data transfer is interrupted, you can use the '''<tt>rsync</tt>''' command listed below to copy the rest of the files without creating doubles of files that have already been transferred.

	'''<tt>rsync<tt>''' can also be used for the initial transfer of data if you expect the transfer to be interrupted. Otherwise, this method should not be used as it takes more time and memory.		'''<tt>rsync</tt>''' can also be used for the initial transfer of data if you expect the transfer to be interrupted. Otherwise, this method should not be used as it takes more time and memory.

	==Verifying transfer==		==Verifying transfer==

Jayid07: /* Transfer a directory or large amounts of data to another location */

2021-02-02T19:40:07Z

Transfer a directory or large amounts of data to another location

← Older revision		Revision as of 19:40, 2 February 2021
Line 40:		Line 40:
	In this command, the target is a dash <tt>'-'</tt> which stands for standard output, and the source is a period <tt>'.'</tt> which is interpreted as all files in your current directory. The standard output is piped to the second command, which has as a source a dash <tt>'-'</tt> (which the shell interprets as standard input). The output of the first command is piped and becomes the input of the second command. The target directory in your second command is <tt>/dir</tt>.		In this command, the target is a dash <tt>'-'</tt> which stands for standard output, and the source is a period <tt>'.'</tt> which is interpreted as all files in your current directory. The standard output is piped to the second command, which has as a source a dash <tt>'-'</tt> (which the shell interprets as standard input). The output of the first command is piped and becomes the input of the second command. The target directory in your second command is <tt>/dir</tt>.

	~~'''Example 2:'''~~ To transfer all files from your documents to a folder in your home directory called foo:		* To transfer all files from your documents to a folder in your home directory called foo:

	<tt>cd ~/Documents~~</tt>~~		<pre>cd ~/Documents

	~~<tt>~~gtar -cpf - . \| gtar -C ~/foo -xpvf -</tt>		gtar -cpf - . \| gtar -C ~/foo -xpvf -</pre>

	When you use this command it will display a list of the files it has transferred.		When you use this command it will display a list of the files it has transferred.

Jayid07: /* Transfer between two different hosts */

2021-02-02T19:29:13Z

Transfer between two different hosts

← Older revision		Revision as of 19:29, 2 February 2021
Line 54:		Line 54:
	===Transfer between two different hosts===		===Transfer between two different hosts===

	~~'''~~*~~'''~~ To transfer files from the <tt>tmp</tt> directory on example1.umiacs.umd.edu to the <tt>foo</tt> directory on your current host:		* To transfer files from the <tt>tmp</tt> directory on example1.umiacs.umd.edu to the <tt>foo</tt> directory on your current host:

	<pre>ssh USERNAME@example1.umiacs.umd.edu gtar -cpf - /tmp \| gtar -C /foo -xpvf -</pre>		<pre>ssh USERNAME@example1.umiacs.umd.edu gtar -cpf - /tmp \| gtar -C /foo -xpvf -</pre>

	~~'''~~*~~'''~~ To transfer files from <tt>foo</tt> directory on your current host to the <tt>tmp</tt> directory on example1.umiacs.umd.edu:		* To transfer files from <tt>foo</tt> directory on your current host to the <tt>tmp</tt> directory on example1.umiacs.umd.edu:

	<pre>gtar -cpf - /foo/ \| ssh USERNAME@example1.umiacs.umd.edu gtar -C /tmp -xpvf -</pre>		<pre>gtar -cpf - /foo/ \| ssh USERNAME@example1.umiacs.umd.edu gtar -C /tmp -xpvf -</pre>

← Older revision		Revision as of 16:23, 25 October 2024
Line 1:		Line 1:
	Depending on the amount of data you are trying to transfer there are different commands that you should use. The classic <code>cp</code> command has a number of edge cases in which it will not copy everything that is expected, so the '''only time''' that one should use the <code>cp</code> command is if the copy can be verified, such as if you are moving a single file. The better choice for transferring data is by using <code>tar</code> or <code>gtar</code>. After transferring files or directories, you can use <code>rsync</code> to check that everything moved correctly and to perform a final sync on changed files.		Depending on the amount of data you are trying to transfer, there are different commands that you should use. The classic <code>cp</code> command has a number of edge cases in which it will not copy everything that is expected, so the '''only time''' that one should use the <code>cp</code> command is if the copy can be verified, such as if you are moving a single file. The better choice for transferring data is by using <code>tar</code> or <code>gtar</code>. After transferring files or directories, you can use <code>rsync</code> to check that everything moved correctly and to perform a final sync on changed files.

	==Transfer a single file==		==Transfer a single file==

	If you want to transfer a single file into another directory you can use the <code>cp</code> command. The format for the <code>cp</code> command is: <code>cp Source Destination</code>. If destination is omitted, it will make a copy of the file(s) in the current directory.		If you want to transfer a single file into another directory you can use the <code>cp</code> command. The format for the <code>cp</code> command is: <code>cp Source Destination</code>. If destination is omitted, it will make a copy of the file(s) in the current directory.

	* To copy the file foo.txt into your home directory's Documents directory		* To copy the file foo.txt into your home directory's Documents directory

	<pre>cp foo.txt ~/Documents</pre>		<pre>cp foo.txt ~/Documents</pre>

	* To transfer the file foo.txt from your current directory to another host use '''<tt>scp</tt>''' (secure copy) with <USERNAME>@<FQDN>:<PATH>		* To transfer the file foo.txt from your current directory to another host use '''<tt>scp</tt>''' (secure copy) with <USERNAME>@<FQDN>:<PATH>

	<pre>scp foo.txt USERNAME@example1.umiacs.umd.edu:~/Documents</pre>		<pre>scp foo.txt USERNAME@example1.umiacs.umd.edu:~/Documents</pre>

Line 16:		Line 13:

	* To transfer back the file foo.txt from the host example1 to another directory in your current host		* To transfer back the file foo.txt from the host example1 to another directory in your current host

	<pre>scp USERNAME@example1.umiacs.umd.edu:~/Documents/foo.txt ~/Videos</pre>		<pre>scp USERNAME@example1.umiacs.umd.edu:~/Documents/foo.txt ~/Videos</pre>

Line 33:		Line 29:
	In this command, the target is a dash <tt>'-'</tt> which stands for standard output, and the source is a period <tt>'.'</tt> which is interpreted as all files in your current directory. The standard output is piped to the second command, which has as a source a dash <tt>'-'</tt> (which the shell interprets as standard input). The output of the first command is piped and becomes the input of the second command. The target directory in your second command is <tt>/dir</tt>.		In this command, the target is a dash <tt>'-'</tt> which stands for standard output, and the source is a period <tt>'.'</tt> which is interpreted as all files in your current directory. The standard output is piped to the second command, which has as a source a dash <tt>'-'</tt> (which the shell interprets as standard input). The output of the first command is piped and becomes the input of the second command. The target directory in your second command is <tt>/dir</tt>.

	* To transfer all files from your documents to a folder in your home directory called foo: <pre>cd ~/Documents gtar -cpf - . \| gtar -C ~/foo -xpvf -</pre>		* To transfer all files from your documents to a folder in your home directory called foo:
			<pre>cd ~/Documents gtar -cpf - . \| gtar -C ~/foo -xpvf -</pre>

	When you use this command it will display a list of the files it has transferred.		When you use this command it will display a list of the files it has transferred.
Line 42:		Line 39:

	===Transfer between two different hosts===		===Transfer between two different hosts===

	* To transfer files from the <tt>tmp</tt> directory on example1.umiacs.umd.edu to the <tt>foo</tt> directory on your current host:		* To transfer files from the <tt>tmp</tt> directory on example1.umiacs.umd.edu to the <tt>foo</tt> directory on your current host:

	<pre>ssh USERNAME@example1.umiacs.umd.edu gtar -cpf - /tmp \| gtar -C /foo -xpvf -</pre>		<pre>ssh USERNAME@example1.umiacs.umd.edu gtar -cpf - /tmp \| gtar -C /foo -xpvf -</pre>

	* To transfer files from <tt>foo</tt> directory on your current host to the <tt>tmp</tt> directory on example1.umiacs.umd.edu:		* To transfer files from <tt>foo</tt> directory on your current host to the <tt>tmp</tt> directory on example1.umiacs.umd.edu:

	<pre>gtar -cpf - /foo/ \| ssh USERNAME@example1.umiacs.umd.edu gtar -C /tmp -xpvf -</pre>		<pre>gtar -cpf - /foo/ \| ssh USERNAME@example1.umiacs.umd.edu gtar -C /tmp -xpvf -</pre>

	If your data transfer is interrupted, you can use the <code>rsync</code> command listed below to copy the rest of the files without creating doubles of files that have already been transferred.		If your data transfer is interrupted, you can use the <code>rsync</code> command listed below to copy the rest of the files without creating doubles of files that have already been transferred.

	<code>rsync</code> can also be used for the initial transfer of data if you expect the transfer to be interrupted. Otherwise, this method should not be used as it takes more time and memory.		<code>rsync</code> can also be used for the initial transfer of data if you expect the transfer to be interrupted. Otherwise, this method should not be used as the overhead takes more time and memory.

	To run rsync to copy files, the format for the command is the same as written below under "Verifying transfer."		To run rsync to copy files, the format for the command is the same as written below under "Verifying transfer."