Migrating your data safely - Revision history

Johann at 20:10, 13 January 2021

2021-01-13T20:10:00Z

2018-05-12T00:35:54Z

2018-05-12T00:16:44Z

2018-05-09T07:06:42Z

2018-05-09T07:05:41Z

2018-05-08T09:33:57Z

2018-05-08T09:06:53Z

‎Migrating large data sets:

2018-05-08T08:53:00Z

‎Migrating large data sets:

2018-05-08T08:52:10Z

‎Migrating large data sets:

2018-05-08T08:17:24Z

‎Migrating large data sets:

@@ Line 43: / Line 43: @@
     # cd /home/<username>
-    # rm -rf /home/<username>/some_directory   ##### PROCEED WITH EXTREME CAUTION! #####
+    # rm -rf ./some_directory   '''##### PROCEED WITH EXTREME CAUTION! #####'''
-Double check your command line when using 'rm -rf’ before you hit enter. You absolutely need to use full paths.
+'''Double check your command line when using 'rm -rf’ before you hit enter.'''
 Its effects are immediate, and irreversible, so mistakes will be costly.
 If all went well, you’ll end up with an exact copy of your source files, with all attributes, permissions and time stamps preserved, at the new destination.

@@ Line 26: / Line 26: @@
     # tmux
     # cp -r -v --preserve=all /home/<username>/some_directory /nlustre/users/<username>
-    # rsync -raH —progress /home/<username>/some_directory/* /nlustre/users/<username>/some_directory     ##### Note path specs are different
+    # rsync -raH --progress /home/<username>/some_directory/* /nlustre/users/<username>/some_directory     ##### Note path specs are different
 then again (just hit up-arrow) :-
-    # rsync -raH —progress /home/<username>/some_directory/* /nlustre/users/<username>/some_directory
+    # rsync -raH --progress /home/<username>/some_directory/* /nlustre/users/<username>/some_directory
 '''Note: Always use full explicit paths when doing this kind of operation! Also - be careful when pasting code from browsers.'''

@@ Line 25: / Line 25: @@
     # tmux
-    # cp -r -v --preserve=all /home/<username>/some_directory /nlustre/users/<username>/some_directory
+    # cp -r -v --preserve=all /home/<username>/some_directory /nlustre/users/<username>
     # rsync -raH —progress /home/<username>/some_directory/* /nlustre/users/<username>/some_directory     ##### Note path specs are different
 then again (just hit up-arrow) :-

@@ Line 27: / Line 27: @@
     # cp -r -v --preserve=all /home/<username>/some_directory /nlustre/users/<username>/some_directory
     # rsync -raH —progress /home/<username>/some_directory/* /nlustre/users/<username>/some_directory     ##### Note path specs are different
-then again (just hit up-arrow):
+then again (just hit up-arrow) :-
     # rsync -raH —progress /home/<username>/some_directory/* /nlustre/users/<username>/some_directory
@@ Line 40: / Line 40: @@
 You do not need to learn all of tmux to use it - just the 3 or so basic commands will do.
-Then, finally, you can remove your source directories:
+Then, finally, you can remove your source directories:-
     # cd /home/<username>

@@ Line 3: / Line 3: @@
 Moving large data sets can be problematic. It cannot be done with drag-and-drop operations, or even with the ‘mv’ command due to drops in network connectivity, for example.
-'''''A much better and tried-and-tested workflow is as follows:'''''
+'''A much better and tried-and-tested workflow is as follows:'''
 * First, do a ‘cp’ with proper construction of the command line.
@@ Line 19: / Line 19: @@
 '''Example:'''
-First, log onto one of our general-use servers such as Zoidberg (do not do this from our Wonko headnode please)
+First, log onto one of our general-use servers such as Zoidberg (do not do this from our Wonko headnode please) :-
     # ssh <username>@zoidberg.bi.up.ac.za

← Older revision		Revision as of 09:06, 8 May 2018
Line 1:		Line 1:

−	~~== Migrating large data sets:==~~

	Moving large data sets can be problematic. It cannot be done with drag-and-drop operations, or even with the ‘mv’ command due to drops in network connectivity, for example.		Moving large data sets can be problematic. It cannot be done with drag-and-drop operations, or even with the ‘mv’ command due to drops in network connectivity, for example.

@@ Line 32: / Line 32: @@
 '''Note: Always use full explicit paths when doing this kind of operation! Also - be careful when pasting code from browsers.'''
-You are done when rsync reports no further files copied. It won't say that explicitly, but you'll notice the total file sizes are the same, and no further file copy stats will come up.
 Any or all of these operations can take many hours, even days - hence the use of ‘tmux’ to preserve the session.

← Older revision		Revision as of 08:52, 8 May 2018
Line 32:		Line 32:

	'''Note: Always use full explicit paths when doing this kind of operation! Also - be careful when pasting code from browsers.'''		'''Note: Always use full explicit paths when doing this kind of operation! Also - be careful when pasting code from browsers.'''
		+	You are done when rsync reports no further files copied. It won't say that explicitly, but you'll notice the total file sizes are the same, and no further file copy stats will come up.

	Any or all of these operations can take many hours, even days - hence the use of ‘tmux’ to preserve the session.		Any or all of these operations can take many hours, even days - hence the use of ‘tmux’ to preserve the session.

← Older revision		Revision as of 08:17, 8 May 2018
Line 47:		Line 47:
	Its effects are immediate, and irreversible, so mistakes will be costly.		Its effects are immediate, and irreversible, so mistakes will be costly.

−	If all went well, you’ll end up with an exact copy of your source files, with all attributes, permissions and time stamps preserved.	+	If all went well, you’ll end up with an exact copy of your source files, with all attributes, permissions and time stamps preserved, at the new destination.