2021-01-13T20:03:35Z

Johann:

'''Please familiarise yourself thoroughly with the following:'''

* All users should use the PBS Torque queue manager for submitting jobs.

* Please do '''not''' run jobs directly on the main headnode wonko.bi.up.ac.za.

* Computing resources are managed by the queuing system.

* Storage resources however, have user quotas, and are billed for, according to our billing policy (See [[Storage Quotas and Charges]]).

* Both users, and their supervisors should familiarise themselves with the quota system, and the costs incurred.

* Home directories - every user gets allocated 500 GB on the file server at /home/<username>. The purpose of this is to serve as a landing spot when you log in, and for storing documents and similar files. It is expressly '''not intended''' for research data.

* Data directories - every user also gets allocated by default 1 TB on our fast Lustre storage at /nlustre/users/<username>.

* This is where your large files must go, and where you should process from, and store towards during processing.

* This storage may '''not be used''' for backup of any other systems, PC's, laptops or data. You may also not store personal data such as movies, music files and photos here. Its intended for bona fide research activities on Bioinformatics servers only.

* The home directories get backed up daily with a retention period of 2 weeks for multiple versions.

* Due to its size, the Lustre directories get replicated to our replication storage once per week.

* You must request installation of utilities and software applications from the systems admin, who will evaluate the best solution for you, and perform the installation system-wide. '''You may not install or attempt to install software of any kind on our servers,''' even under your own account (trying to use utilities like brew or conda will simply not work).

* DISCLAIMER: Take note that whilst we do backups to the best of our ability of all servers and data, '''the onus of securing and ownership of your data remain with you, the user, and your supervisor. We do not accept responsibility for the loss of users' data or intellectual property.''' We do not have control over the stability and availability of reliable power supply to the facility. Hence unplanned outages may occur at any time, and this may cause data loss, even on our backup systems.

Guidelines and Terms of use

2021-01-13T20:01:52Z

Johann:

'''Please familiarise yourself thoroughly with the following:'''

* All users should use the PBS Torque queue manager for submitting jobs.

* Please do '''not''' run jobs directly on the main headnode wonko.bi.up.ac.za.

* Computing resources are managed by the queuing system.

* Storage resources however, have user quotas, and are billed for, according to our billing policy (See [[Storage Quotas and Charges]]).

* Both users, and their supervisors should familiarise themselves with the quota system, and the costs incurred.

* Home directories - every user gets allocated 500 GB on the file server at /home/<username>. The purpose of this is to serve as a landing spot when you log in, and for storing documents and similar files. It is expressly '''not intended''' for research data.

* Data directories - every user also gets allocated by default 1 TB on our fast Lustre storage at /nlustre/users/<username>.

* This is where your large files must go, and where you should process from, and store towards during processing.

* This storage may '''not be used''' for backup of any other systems, PC's, laptops or data. You may also not store personal data such as movies, music files and photos here. Its intended for bona fide research activities on Bioinformatics servers only.

* The home directories get backed up daily with a retention period of 2 weeks for multiple versions.

* Due to its size, the Lustre directories get replicated to our replication storage once per week.

* You must request installation of utilities and software applications from the systems admin, who will evaluate the best solution for you, and perform the installation system-wide. '''You may not install or attempt to install software of any kind on our servers,''' even under your own account (trying to use utilities like brew or conda will simply not work).

* DISCLAIMER: Take note that whilst we do backups to the best of our ability of all servers and data, '''the onus of securing and ownership of your data remain with you. We do not accept responsibility for the loss of users' data or intellectual property.''' We do not have control over the stability and availability of reliable power supply to the facility. Hence unplanned outages may occur at any time, and this may cause data loss, even on our backup systems.

2018-12-15T10:05:27Z

Johann:

90% of problems with scripts and file handling are due to incorrect file permission and ownership settings. Every system user should familiarise themselves with this important aspect of Linux.

Before a new script can be run, its execute permission must be set by the user. This is by default unset for safety, and causes problems every day.

When a new file tst.sh is created, it will typically have the following default permissions:

[user@wonko:~]$ ls -al tst.sh

-rw-r--r-- 1 user users 3 Dec 15 11:58 tst.sh

This file cannot be executed or submitted to the queue.

By changing user permissions and setting the execute flag:

[user@wonko:~]$ chmod u+x tst.sh

[user@wonko:~]$ ls -al tst.sh

-rwxr--r-- 1 user users 3 Dec 15 11:58 tst.sh

this script can now be run or submiited.

'''Permission Groups -:'''

Each file and directory has three user based permission groups:

'''owner -''' The Owner permissions apply only the owner of the file or directory, they will not impact the actions of other users.

'''group -''' The Group permissions apply only to the group that has been assigned to the file or directory, they will not effect the actions of other users.

'''all users (world, others) -''' The All Users permissions apply to all other users on the system, this is the permission group that you want to watch the most. There's usually no reason to have this set, and for safety we recommend leaving this off.

'''Permission Types -: '''

Each file or directory has three basic permission types:

'''read -''' The Read permission refers to a user's capability to read the contents of the file.

'''write -''' The Write permissions refer to a user's capability to write or modify a file or directory.

'''execute -''' The Execute permission affects a user's capability to execute a file or view the contents of a directory.

'''Viewing the Permissions -:'''

You can view the permissions by checking the file or directory permissions by reviewing the output of the "ls -al" command while in the terminal and while working in the directory which contains the file or folder.

The permission in the command line is displayed as: _ rwxrwxrwx 1 <owner>:<group>

'''User rights/Permissions -:'''

The first character that I marked with an underscore is the special permission flag that can vary.
The following set of three characters (rwx) is for the owner permissions.
The second set of three characters (rwx) is for the Group permissions.
The third set of three characters (rwx) is for the All Users permissions.
Following that grouping since the integer/number displays the number of hardlinks to the file.
The last piece is the Owner and Group assignment formatted as Owner:Group.

'''Modifying the Permissions-:'''

When in the command line, the permissions are edited by using the command chmod. You can assign the permissions explicitly or by using a binary reference as described below.

'''Explicitly Defining Permissions -:'''

To explicity define permissions you will need to reference the Permission Group and Permission Types.

The Permission Groups used are:
u - Owner
g - Group
o - Others
a - All users
The potential Assignment Operators are + (plus) and - (minus); these are used to tell the system whether to add or remove the specific permissions.

The Permission Types that are used are:
r - Read
w - Write
x - Execute

So for an example, lets say I have a file named file1 that currently has the permissions set to rw_rw_rw_, which means that the owner, group and all users have read and write permission. Now we want to remove the read and write permissions from the all users group.

To make this modification you would invoke the command: chmod a-rw file1
To add the permissions above you would invoke the command: chmod a+rw file1

As you can see, if you want to grant those permissions you would change the minus character to a plus to add those permissions.

Using Binary References to Set permissions
Now that you understand the permissions groups and types this one should feel natural. To set the permission using binary references you must first understand that the input is done by entering three octal integers/numbers.

A sample permission string would be chmod 640 file1, which means that the owner has read and write permissions, the group has read permissions, and all other user have no rights to the file.

The first number represents the Owner permission; the second represents the Group permissions; and the last number represents the permissions for all other users. The numbers are a binary representation of the rwx string.

r = 4
w = 2
x = 1
You add the numbers to get the integer/number representing the permissions you wish to set. You will need to include the binary permissions for each of the three permission groups.

So to set a file to permissions on file1 to read rwxr_ _ _ _ _, you would enter chmod 740 file1.

'''Owners and Groups -:'''

File and directory ownership is usually set by the system defaults, or by the systems administrator, and will be of the form <username>:<primary group>, where the group will typically be "users".

I have made several references to Owners and Groups above, but have not yet told you how to assign or change the Owner and Group assigned to a file or directory. There's usually no need to change these, just take note that they are properly set.

You use the chown command to change owner and group assignments, the syntax is simple chown owner:group filename, so to change the owner of file1 to user1 and the group to family you would enter chown user1:family file1.

'''One more thing - you should be very careful when copy/pasting code from web pages. They usually do not translate all characters accurately or correctly from web to terminal and can lead to unexpected behaviour and bugs.'''

The above just touches the tip of the iceberg on this topic, and you really should consult one of the many websites dealing with the subject in detail.

File and directory permissions and ownership

2018-12-15T10:04:59Z

Johann:

90% of problems with scripts and file handling are due to incorrect file permission and ownership settings. Every system user should familiarise themselves with this important aspect of Linux.

Before a new script can be run, its execute permission must be set by the user. This is by default unset for safety, and causes problems every day.

When a new file tst.sh is created, it will typically have the following default permissions:

[user@wonko:~]$ ls -al tst.sh

-rw-r--r-- 1 user users 3 Dec 15 11:58 tst.sh

This file cannot be executed or submitted to the queue.

By changing user permissions and setting the execute flag:

[user@wonko:~]$ chmod u+x tst.sh
[user@wonko:~]$ ls -al tst.sh

-rwxr--r-- 1 user users 3 Dec 15 11:58 tst.sh

this script can now be run or submiited.

'''Permission Groups -:'''

Each file and directory has three user based permission groups:

'''owner -''' The Owner permissions apply only the owner of the file or directory, they will not impact the actions of other users.

'''group -''' The Group permissions apply only to the group that has been assigned to the file or directory, they will not effect the actions of other users.

'''all users (world, others) -''' The All Users permissions apply to all other users on the system, this is the permission group that you want to watch the most. There's usually no reason to have this set, and for safety we recommend leaving this off.

'''Permission Types -: '''

Each file or directory has three basic permission types:

'''read -''' The Read permission refers to a user's capability to read the contents of the file.

'''write -''' The Write permissions refer to a user's capability to write or modify a file or directory.

'''execute -''' The Execute permission affects a user's capability to execute a file or view the contents of a directory.

'''Viewing the Permissions -:'''

You can view the permissions by checking the file or directory permissions by reviewing the output of the "ls -al" command while in the terminal and while working in the directory which contains the file or folder.

The permission in the command line is displayed as: _ rwxrwxrwx 1 <owner>:<group>

'''User rights/Permissions -:'''

The first character that I marked with an underscore is the special permission flag that can vary.
The following set of three characters (rwx) is for the owner permissions.
The second set of three characters (rwx) is for the Group permissions.
The third set of three characters (rwx) is for the All Users permissions.
Following that grouping since the integer/number displays the number of hardlinks to the file.
The last piece is the Owner and Group assignment formatted as Owner:Group.

'''Modifying the Permissions-:'''

When in the command line, the permissions are edited by using the command chmod. You can assign the permissions explicitly or by using a binary reference as described below.

'''Explicitly Defining Permissions -:'''

To explicity define permissions you will need to reference the Permission Group and Permission Types.

The Permission Groups used are:
u - Owner
g - Group
o - Others
a - All users
The potential Assignment Operators are + (plus) and - (minus); these are used to tell the system whether to add or remove the specific permissions.

The Permission Types that are used are:
r - Read
w - Write
x - Execute

So for an example, lets say I have a file named file1 that currently has the permissions set to rw_rw_rw_, which means that the owner, group and all users have read and write permission. Now we want to remove the read and write permissions from the all users group.

To make this modification you would invoke the command: chmod a-rw file1
To add the permissions above you would invoke the command: chmod a+rw file1

As you can see, if you want to grant those permissions you would change the minus character to a plus to add those permissions.

Using Binary References to Set permissions
Now that you understand the permissions groups and types this one should feel natural. To set the permission using binary references you must first understand that the input is done by entering three octal integers/numbers.

A sample permission string would be chmod 640 file1, which means that the owner has read and write permissions, the group has read permissions, and all other user have no rights to the file.

The first number represents the Owner permission; the second represents the Group permissions; and the last number represents the permissions for all other users. The numbers are a binary representation of the rwx string.

r = 4
w = 2
x = 1
You add the numbers to get the integer/number representing the permissions you wish to set. You will need to include the binary permissions for each of the three permission groups.

So to set a file to permissions on file1 to read rwxr_ _ _ _ _, you would enter chmod 740 file1.

'''Owners and Groups -:'''

File and directory ownership is usually set by the system defaults, or by the systems administrator, and will be of the form <username>:<primary group>, where the group will typically be "users".

I have made several references to Owners and Groups above, but have not yet told you how to assign or change the Owner and Group assigned to a file or directory. There's usually no need to change these, just take note that they are properly set.

You use the chown command to change owner and group assignments, the syntax is simple chown owner:group filename, so to change the owner of file1 to user1 and the group to family you would enter chown user1:family file1.

'''One more thing - you should be very careful when copy/pasting code from web pages. They usually do not translate all characters accurately or correctly from web to terminal and can lead to unexpected behaviour and bugs.'''

The above just touches the tip of the iceberg on this topic, and you really should consult one of the many websites dealing with the subject in detail.

File and directory permissions and ownership

2018-12-15T10:04:24Z

Johann:

90% of problems with scripts and file handling are due to incorrect file permission and ownership settings. Every system user should familiarise themselves with this important aspect of Linux.

Before a new script can be run, its execute permission must be set by the user. This is by default unset for safety, and causes problems every day.

When a new file tst.sh is created, it will typically have the following default permissions:

[user@wonko:~]$ ls -al tst.sh
-rw-r--r-- 1 user users 3 Dec 15 11:58 tst.sh

This file cannot be executed or submitted to the queue.

By changing user permissions and setting the execute flag:

[user@wonko:~]$ chmod u+x tst.sh
[user@wonko:~]$ ls -al tst.sh
-rwxr--r-- 1 user users 3 Dec 15 11:58 tst.sh

this script can now be run or submiited.

'''Permission Groups -:'''

Each file and directory has three user based permission groups:

'''owner -''' The Owner permissions apply only the owner of the file or directory, they will not impact the actions of other users.

'''group -''' The Group permissions apply only to the group that has been assigned to the file or directory, they will not effect the actions of other users.

'''all users (world, others) -''' The All Users permissions apply to all other users on the system, this is the permission group that you want to watch the most. There's usually no reason to have this set, and for safety we recommend leaving this off.

'''Permission Types -: '''

Each file or directory has three basic permission types:

'''read -''' The Read permission refers to a user's capability to read the contents of the file.

'''write -''' The Write permissions refer to a user's capability to write or modify a file or directory.

'''execute -''' The Execute permission affects a user's capability to execute a file or view the contents of a directory.

'''Viewing the Permissions -:'''

You can view the permissions by checking the file or directory permissions by reviewing the output of the "ls -al" command while in the terminal and while working in the directory which contains the file or folder.

The permission in the command line is displayed as: _ rwxrwxrwx 1 <owner>:<group>

'''User rights/Permissions -:'''

The first character that I marked with an underscore is the special permission flag that can vary.
The following set of three characters (rwx) is for the owner permissions.
The second set of three characters (rwx) is for the Group permissions.
The third set of three characters (rwx) is for the All Users permissions.
Following that grouping since the integer/number displays the number of hardlinks to the file.
The last piece is the Owner and Group assignment formatted as Owner:Group.

'''Modifying the Permissions-:'''

When in the command line, the permissions are edited by using the command chmod. You can assign the permissions explicitly or by using a binary reference as described below.

'''Explicitly Defining Permissions -:'''

To explicity define permissions you will need to reference the Permission Group and Permission Types.

The Permission Groups used are:
u - Owner
g - Group
o - Others
a - All users
The potential Assignment Operators are + (plus) and - (minus); these are used to tell the system whether to add or remove the specific permissions.

The Permission Types that are used are:
r - Read
w - Write
x - Execute

So for an example, lets say I have a file named file1 that currently has the permissions set to rw_rw_rw_, which means that the owner, group and all users have read and write permission. Now we want to remove the read and write permissions from the all users group.

To make this modification you would invoke the command: chmod a-rw file1
To add the permissions above you would invoke the command: chmod a+rw file1

As you can see, if you want to grant those permissions you would change the minus character to a plus to add those permissions.

Using Binary References to Set permissions
Now that you understand the permissions groups and types this one should feel natural. To set the permission using binary references you must first understand that the input is done by entering three octal integers/numbers.

A sample permission string would be chmod 640 file1, which means that the owner has read and write permissions, the group has read permissions, and all other user have no rights to the file.

The first number represents the Owner permission; the second represents the Group permissions; and the last number represents the permissions for all other users. The numbers are a binary representation of the rwx string.

r = 4
w = 2
x = 1
You add the numbers to get the integer/number representing the permissions you wish to set. You will need to include the binary permissions for each of the three permission groups.

So to set a file to permissions on file1 to read rwxr_ _ _ _ _, you would enter chmod 740 file1.

'''Owners and Groups -:'''

File and directory ownership is usually set by the system defaults, or by the systems administrator, and will be of the form <username>:<primary group>, where the group will typically be "users".

I have made several references to Owners and Groups above, but have not yet told you how to assign or change the Owner and Group assigned to a file or directory. There's usually no need to change these, just take note that they are properly set.

You use the chown command to change owner and group assignments, the syntax is simple chown owner:group filename, so to change the owner of file1 to user1 and the group to family you would enter chown user1:family file1.

'''One more thing - you should be very careful when copy/pasting code from web pages. They usually do not translate all characters accurately or correctly from web to terminal and can lead to unexpected behaviour and bugs.'''

The above just touches the tip of the iceberg on this topic, and you really should consult one of the many websites dealing with the subject in detail.