Hoffman2:Compiling MATLAB - Revision history

Ccn admin: 12 revisions

2014-01-16T03:14:46Z

12 revisions

← Older revision	Revision as of 03:14, 16 January 2014
(No difference)

Alhead: added a line for loading modules prior to calling "module load matlab"

2013-03-06T01:40:01Z

added a line for loading modules prior to calling "module load matlab"

← Older revision		Revision as of 01:40, 6 March 2013
Line 180:		Line 180:
	You could also make a shell script that contains		You could also make a shell script that contains
	#!/bin/bash		#!/bin/bash
			source /u/local/Modules/default/init/modules.sh
	module load matlab		module load matlab
	/path/to/compiled/matlab/script		/path/to/compiled/matlab/script

Elau at 18:01, 23 April 2012

2012-04-23T18:01:06Z

← Older revision		Revision as of 18:01, 23 April 2012
Line 4:		Line 4:

	Furthermore, if you need to run the same MATLAB script on multiple subjects and would like to do so in parallel, you will find that you can't simply open up multiple instances of MATLAB on Hoffman2. This is because each user is only allotted one MATLAB license. To parallelize your processing, it would be better to compile your MATLAB script and submit a job for each subject using the compiled code. '''Compiled code does not require a MATLAB license in order to run.'''		Furthermore, if you need to run the same MATLAB script on multiple subjects and would like to do so in parallel, you will find that you can't simply open up multiple instances of MATLAB on Hoffman2. This is because each user is only allotted one MATLAB license. To parallelize your processing, it would be better to compile your MATLAB script and submit a job for each subject using the compiled code. '''Compiled code does not require a MATLAB license in order to run.'''



			==Version Notice - 2012.04.23==
			'''As of 2012.04.23 ATS updated the default MATLAB version from 7.11 (R2010b) to 7.14 (R2012a).'''

			This means that any MATLAB code previously compiled will need to load the old MATLAB module in order to run. e.g.
			$ module load matlab/7.11
			$ /path/to/my/old/compiled/code

			And the <code>matexe.q</code> tool will be using the newer MATLAB module. Which means you can't use it to submit previously compiled code. Please recompile your code to work with the new MATLAB version before submitting jobs with this tool.

Elau at 20:42, 18 April 2012

2012-04-18T20:42:29Z

← Older revision		Revision as of 20:42, 18 April 2012
Line 61:		Line 61:
	$ cd ~/mccExample		$ cd ~/mccExample
	$ mcc -m /u/home/FMRI/apps/examples/mcc/exampleMatlabScript.m -R -singleCompThread -I /u/home/FMRI/apps/examples/mcc/subfunctions		$ mcc -m /u/home/FMRI/apps/examples/mcc/exampleMatlabScript.m -R -singleCompThread -I /u/home/FMRI/apps/examples/mcc/subfunctions

			; <code>-m /u/home/FMRI/apps/examples/mcc/exampleMatlabScript.m</code>
			: This flag is used to indicate which file is to be compiled.
			; <code>-R -singleCompThread</code>
			: This flag indicates what resources are needed. Default is to ask for all available cores, but by only requesting a single core you allow yourself more flexibility. If your code needed all the cores on a node, any jobs you submit to the queue will probably wait much longer to execute. If your code only needs one core, you job can be more flexible in scheduling and will probably execute sooner.
			; <code>-I /u/home/FMRI/apps/examples/mcc/subfunctions</code>
			: This flag is used to indicate the subdirectories where any support files are located so that they can be included in the compiling. If you had support .m files in multiple directories, you would need to include each separately e.g.
			:: <pre>mcc -m /path/to/main/file -R -singleCompThread -I /path/to/file/one -I /path/to/file/two -I /path/to/file/three</pre>

	The output will probably start with		The output will probably start with
	Ifconfig uses the ioctl access method to get the full address information, which limits hardware addresses to 8 bytes.		Ifconfig uses the ioctl access method to get the full address information, which limits hardware addresses to 8 bytes.

Elau at 20:31, 18 April 2012

2012-04-18T20:31:28Z

← Older revision		Revision as of 20:31, 18 April 2012
Line 60:		Line 60:
	$ mkdir ~/mccExample		$ mkdir ~/mccExample
	$ cd ~/mccExample		$ cd ~/mccExample
	$ mcc -m /u/home/FMRI/apps/examples/mcc/exampleMatlabScript.m -R -singleCompThread -I /u/home/FMRI/apps/examples/mcc/		$ mcc -m /u/home/FMRI/apps/examples/mcc/exampleMatlabScript.m -R -singleCompThread -I /u/home/FMRI/apps/examples/mcc/subfunctions
	The output will probably start with		The output will probably start with
	Ifconfig uses the ioctl access method to get the full address information, which limits hardware addresses to 8 bytes.		Ifconfig uses the ioctl access method to get the full address information, which limits hardware addresses to 8 bytes.

Elau at 01:38, 13 April 2012

2012-04-13T01:38:49Z

← Older revision		Revision as of 01:38, 13 April 2012
Line 125:		Line 125:
	# When two files named ''exampleMatlabScript.output.[JOBID]'' and ''exampleMatlabScript.joblog.[JOBID]'' (where JOBID is your job's unique identifier) appear, your job has run.		# When two files named ''exampleMatlabScript.output.[JOBID]'' and ''exampleMatlabScript.joblog.[JOBID]'' (where JOBID is your job's unique identifier) appear, your job has run.
	#: ''exampleMatlabScript.output.[JOBID]''		#: ''exampleMatlabScript.output.[JOBID]''
	#:: This file has all the standard output generated by your script. In this case it will be		#:: This file has all the standard output generated by your script. In this case it will be something like
	#::: <code>Passing numbers to the subfunction...</code>		#::: <code>Passing numbers to the subfunction...</code>
	#::: <code>This is a MATLAB function that takes input arguments.</code>		#::: <code>This is a MATLAB function that takes input arguments.</code>

Elau at 01:37, 13 April 2012

2012-04-13T01:37:18Z

Elau at 01:30, 13 April 2012

2012-04-13T01:30:07Z

← Older revision		Revision as of 01:30, 13 April 2012
Line 88:		Line 88:

	==Submitting==		==Submitting==
	You've compiled code and tested that it works. Time to submit it as a job~~. ATS has a tool for this (similar to [[Hoffman2:Submitting Jobs#job.q\|job.q]]) called <code>matexe.q</code>~~.		You've compiled code and tested that it works. Time to submit it as a job.

			===matexe.q===
			ATS has a tool for this (similar to [[Hoffman2:Submitting Jobs#job.q\|job.q]]) called <code>matexe.q</code>. It's a step-by-step tool that's pretty self explanatory, but we provided an example anyway.

			====Example====
			# Once on Hoffman2, you'll need to edit one file so pull out your favorite [[Text Editors\|text editor]] and edit the file
			#: <pre>~/.queuerc</pre>
			# Add the line (if it isn't already there)
			#: <pre>set qqodir = ~/job-output</pre>
			# You've just set the default directory where your job command files will be created. Save the configuration file and close your text editor.
			# Make that directory using the command
			#: <pre>$ mkdir ~/job-output</pre>
			# Now run
			#:<pre>$ matexe.q</pre>
			# Press enter to acknowledge any messages that may appear (READ IT FIRST THOUGH).
			# Type ''Build <ENTER>'' to begin creating an SGE command file.
			# The program now asks you which script you'd like to run, enter the following text to use our example script
			#: <pre>~/mccExample/exampleMatlabScript</pre>
			# The program now asks how much memory the job will need (in [http://en.wikipedia.org/wiki/Megabyte Megabytes]). This script is really simple, but let's go with the default 1024.
			# The program now asks how long will the job take (in hours). Go with the minimum 1 hour; it will complete in much less than this.
			# The program now asks if your job should be limited to only your resource group's cores. Answer ''n'' because you do not need to be limiting yourself here and the job is not going to be running for more than 24 hours.
			# It will ask for any arguments you'd like to supply. Feel free to put random values here or nothing at all to see how the example script treats your arguments.
			# Soon, the program will tell you that ''exampleMatlabScript.cmd'' has been built and saved.
			# When it asks you if you would like to submit your job, say no. Then type ''Quit <ENTER>'' to leave the program.
			# Now you should be able to run
			#: <pre>$ ls ~/job-output</pre>
			#: and see ''exampleMatlabScript.cmd''. This file will stay there until you delete it and can be run over and over again. Making a command file like this is especially useful if there is a task you'll be running repeatedly on Hoffman2. But if this is something you only need to run once, you should delete the file so you don't needlessly approach your [[Hoffman2:Quotas\|quota]].
			# The time has come to actually run the program (thought we'd never get to that, didn't you?). Type
			#: <pre>$ qsub job-output/exampleMatlabScript.cmd</pre>
			#: and after hitting enter, a message similar to this will pop up:
			#: <pre>Your job 1882940 ("exampleMatlabScript.cmd") has been submitted</pre>
			#: where the number is your JobID, a unique numerical identifier for the computer job you have submitted to the queue.
			# Now you can check if the job has finished running by doing
			#: <pre>$ ls ~/job-output</pre>
			#: when two files named ''exampleMatlabScript.output.[JOBID]'' and ''exampleMatlabScript.joblog.[JOBID]'' (where JOBID is your job's unique identifier) appear, your job has run.
			#:: ''exampleMatlabScript.output.[JOBID]''
			#::: This file has all the standard output generated by your script. In this case it will be
			#:::: <code>Passing numbers to the subfunction...</code>
			#:::: <code>This is a MATLAB function that takes input arguments.</code>
			#:::: <code>Argument #1 -- -- is a double</code>
			#:::: <code>Argument #2 -- -- is a double</code>
			#:::: <code>Argument #3 -- -- is a double</code>
			#:::: <code>Passing characters to the subfunction...</code>
			#:::: <code>This is a MATLAB function that takes input arguments.</code>
			#:::: <code>Argument #1 -- a -- is a char</code>
			#:::: <code>Argument #2 -- b -- is a char</code>
			#:::: <code>Argument #3 -- c -- is a char</code>
			#:::: <code>Passing whatever arguments you gave me on to the subfunction...</code>
			#:::: <code>This is a MATLAB function that takes input arguments.</code>
			#:::: <code>Argument #1 -- 1 -- is a char</code>
			#:::: <code>This is a MATLAB function that takes input arguments.</code>
			#:::: <code>Argument #1 -- a -- is a char</code>
			#:::: <code>This is a MATLAB function that takes input arguments.</code>
			#:::: <code>Argument #1 -- 2 -- is a char</code>
			#:::: <code>This is a MATLAB function that takes input arguments.</code>
			#:::: <code>Argument #1 -- b -- is a char</code>
			#:::: <code>This is a MATLAB function that takes input arguments.</code>
			#:::: <code>Argument #1 -- 3 -- is a char</code>
			#:::: <code>This is a MATLAB function that takes input arguments.</code>
			#:::: <code>Argument #1 -- c -- is a char</code>
			#:::: <code>Warning: No display specified. You will not be able to display graphics on the screen.</code>
			#:: ''exampleMatlabScript.joblog.[JOBID]''
			#::: This file has all the details about when, where, and how your job was processed. Useful information if you are going to be running this job over and over and need to fine tune the resources it uses.
			# Finally, go check the inbox of the email you used to sign up for your Hoffman2 account. There will be two emails from "root@mail.hoffman2.idre.ucla.edu" that indicate when the job was started and when the job was completed. This is one of the neat features of the queue so that you can be alerted about the progress of your job without having to stay logged into Hoffman2 and checking on it constantly.


			===By hand===
			You could also make a shell script that contains
			#!/bin/bash
			module load matlab
			/path/to/compiled/matlab/script
			and submit this shell script using [[Hoffman2:Submitting Jobs#qsub\|qsub]] or [[Hoffman2:Submitting Jobs#q.sh\|q.sh]] to achieve similar results.



			==About those arguments==
			If you played around with submitting different arguments to the sample script we compiled above, you will have noticed that '''compiled MATLAB treats all arguments as if they are strings.''' So if you plan to leverage arguments in your code to make your functions more useful, and we recommend that you do, you may need to make use of the command
			str2num()
			to convert certain arguments into usable numbers.

Elau at 00:08, 11 April 2012

2012-04-11T00:08:58Z

← Older revision		Revision as of 00:08, 11 April 2012
Line 55:		Line 55:

	end		end


	To compile this example script, first [[Hoffman2:Interactive Sessions\|check out an interactive node]] '''because this can be a computational intensive process'''. Then run the following commands		To compile this example script, first [[Hoffman2:Interactive Sessions\|check out an interactive node]] '''because this can be a computational intensive process'''. Then run the following commands
Line 60:		Line 61:
	$ cd ~/mccExample		$ cd ~/mccExample
	$ mcc -m /u/home/FMRI/apps/examples/mcc/exampleMatlabScript.m -R -singleCompThread -I /u/home/FMRI/apps/examples/mcc/		$ mcc -m /u/home/FMRI/apps/examples/mcc/exampleMatlabScript.m -R -singleCompThread -I /u/home/FMRI/apps/examples/mcc/
	The output will start with		The output will probably start with
	Ifconfig uses the ioctl access method to get the full address information, which limits hardware addresses to 8 bytes.		Ifconfig uses the ioctl access method to get the full address information, which limits hardware addresses to 8 bytes.
	Because Infiniband address has 20 bytes, only the first 8 bytes are displayed correctly.		Because Infiniband address has 20 bytes, only the first 8 bytes are displayed correctly.

Elau at 04:41, 5 April 2012

2012-04-05T04:41:26Z

← Older revision		Revision as of 04:41, 5 April 2012
Line 2:		Line 2:

	Running the MATLAB GUI is not always desirable. If you already know that your code works and no visuals or user interaction are needed, then using the MATLAB GUI is unnecessary and adds extra work for the computer. Instead, it may be better to compile your MATLAB code into an executable that can be submitted as a job on Hoffman2.		Running the MATLAB GUI is not always desirable. If you already know that your code works and no visuals or user interaction are needed, then using the MATLAB GUI is unnecessary and adds extra work for the computer. Instead, it may be better to compile your MATLAB code into an executable that can be submitted as a job on Hoffman2.

			Furthermore, if you need to run the same MATLAB script on multiple subjects and would like to do so in parallel, you will find that you can't simply open up multiple instances of MATLAB on Hoffman2. This is because each user is only allotted one MATLAB license. To parallelize your processing, it would be better to compile your MATLAB script and submit a job for each subject using the compiled code. '''Compiled code does not require a MATLAB license in order to run.'''


	==Compiling==		==Compiling==
	There is a handy tool <code>mcc</code> which can take a MATLAB .m file and compile it into an executable that can be run outside of MATLAB. It is expensive to buy, but luckily Hoffman2 has already done the purchasing for you.		There is a handy tool, <code>mcc</code>, which can take a MATLAB .m file and compile it into an executable that can be run outside of MATLAB. It is expensive to buy, but luckily Hoffman2 has already done the purchasing for you.

			'''Remember that to use <code>mcc</code>, you need an open MATLAB license. If you are running MATLAB already and open up a new Hoffman2 session intending to compile something with <code>mcc</code>, know that you will see errors and compilation will fail because you are already using your one allotted MATLAB license.'''

	We have two example MATLAB scripts:		We have two example MATLAB scripts:

	'''/u/home/FMRI/apps/examples/mcc/~~matlabScript~~.m'''		'''/u/home/FMRI/apps/examples/mcc/subfunctions/exampleSubfunction.m'''

	This function can take any number of arguments and the text it prints out is based on the type of variables it receives.		This function can take any number of arguments and the text it prints out is based on the type of variables it receives. It is kept in a separate directory from the exampleMatlabScript.m file to illustrate how to work with dependent files in different parts of the filesystem.

	function ~~subfunction~~(varargin)		function exampleSubfunction(varargin)

	fprintf('This is a MATLAB function that takes input arguments.\n');		fprintf('This is a MATLAB function that takes input arguments.\n');
Line 27:		Line 31:
	fprintf('Argument #%d\t -- is an unknown\n', i);		fprintf('Argument #%d\t -- is an unknown\n', i);
	end		end
			else
			fprintf('Argument #%d\t -- is empty\n', i);
	end		end
	end		end
Line 32:		Line 38:


	'''/u/home/FMRI/apps/examples/mcc/~~subfunction~~.m'''		'''/u/home/FMRI/apps/examples/mcc/exampleMatlabScript.m'''

	Makes use of the ~~subfunction~~() function.		Makes use of the exampleSubfunction() function.
	function matlabScript(varargin)		function matlabScript(varargin)

Line 50:		Line 56:
	end		end

	To compile this ~~use~~ the ~~command~~		To compile this example script, first [[Hoffman2:Interactive Sessions\|check out an interactive node]] '''because this can be a computational intensive process'''. Then run the following commands
	mcc -m /u/home/FMRI/apps/examples/mcc/~~matlabScript~~.m -R -singleCompThread -I /u/home/FMRI/apps/examples/mcc/~~subfunction~~.m		$ mkdir ~/mccExample
			$ cd ~/mccExample
			$ mcc -m /u/home/FMRI/apps/examples/mcc/exampleMatlabScript.m -R -singleCompThread -I /u/home/FMRI/apps/examples/mcc/
			The output will start with
			Ifconfig uses the ioctl access method to get the full address information, which limits hardware addresses to 8 bytes.
			Because Infiniband address has 20 bytes, only the first 8 bytes are displayed correctly.
			Ifconfig is obsolete! For replacement check ip.
			which can be ignored and will be followed by any other messages related to the compiling, or error messages if something went wrong.




			==Testing==
			Now that you've compiled your code, you should test it to make sure it works before submitting 100,000 jobs using it.

			# [[Hoffman2:Interactive Sessions\|Check out an interactive node]]
			# Execute
			#: <pre>$ module load matlab</pre>
			#: so that the computing node knows how to speak MATLAB
			# Then change to the directory where the compiled code is
			#: <pre>$ cd ~/mccExample</pre>
			# And run your compiled code
			#: <pre>$ ./exampleMatlabScript </pre>
			#: Feel free to add any arguments to your function call, e.g.
			#: <pre>$ ./exampleMatlabScript ARG1 2 3 4</pre>
			#: to see how the output changes.



	==Submitting==		==Submitting==
			You've compiled code and tested that it works. Time to submit it as a job. ATS has a tool for this (similar to [[Hoffman2:Submitting Jobs#job.q\|job.q]]) called <code>matexe.q</code>.




	==Known Issues==		==Known Issues==
	Compiling EEGLAB has thus far failed on Hoffman2. Compiling SPM5 or SPM8 may fail similarly. If you figure out how to do this, let us know.		* The official mcc documentation cites a <code>-a</code> flag for including a directory and all of its subdirectories recursively. We haven't found this flag to work very effectively and strongly suggest you use <code>-I</code> to explicitly name all the directories that have .m files your code depends on.
			* Compiling EEGLAB has thus far failed on Hoffman2. Compiling SPM5 or SPM8 may fail similarly. If you figure out how to do this, let us know.
			* You cannot compile something while you have another MATLAB session open. Each Hoffman2 user is allotted one MATLAB license at a time, and <code>mcc</code> requires a license to run.

← Older revision		Revision as of 01:37, 13 April 2012
Line 123:		Line 123:
	# Now you can check if the job has finished running by doing		# Now you can check if the job has finished running by doing
	#: <pre>$ ls ~/job-output</pre>		#: <pre>$ ls ~/job-output</pre>
	#~~: when~~ two files named ''exampleMatlabScript.output.[JOBID]'' and ''exampleMatlabScript.joblog.[JOBID]'' (where JOBID is your job's unique identifier) appear, your job has run.		# When two files named ''exampleMatlabScript.output.[JOBID]'' and ''exampleMatlabScript.joblog.[JOBID]'' (where JOBID is your job's unique identifier) appear, your job has run.
	#:: ''exampleMatlabScript.output.[JOBID]''		#: ''exampleMatlabScript.output.[JOBID]''
	#::: This file has all the standard output generated by your script. In this case it will be		#:: This file has all the standard output generated by your script. In this case it will be
	#:::: <code>Passing numbers to the subfunction...</code>		#::: <code>Passing numbers to the subfunction...</code>
	#:::: <code>This is a MATLAB function that takes input arguments.</code>		#::: <code>This is a MATLAB function that takes input arguments.</code>
	#:::: <code>Argument #1 -- -- is a double</code>		#::: <code>Argument #1 -- -- is a double</code>
	#:::: <code>Argument #2 -- -- is a double</code>		#::: <code>Argument #2 -- -- is a double</code>
	#:::: <code>Argument #3 -- -- is a double</code>		#::: <code>Argument #3 -- -- is a double</code>
	#:::: <code>Passing characters to the subfunction...</code>		#::: <code>Passing characters to the subfunction...</code>
	#:::: <code>This is a MATLAB function that takes input arguments.</code>		#::: <code>This is a MATLAB function that takes input arguments.</code>
	#:::: <code>Argument #1 -- a -- is a char</code>		#::: <code>Argument #1 -- a -- is a char</code>
	#:::: <code>Argument #2 -- b -- is a char</code>		#::: <code>Argument #2 -- b -- is a char</code>
	#:::: <code>Argument #3 -- c -- is a char</code>		#::: <code>Argument #3 -- c -- is a char</code>
	#:::: <code>Passing whatever arguments you gave me on to the subfunction...</code>		#::: <code>Passing whatever arguments you gave me on to the subfunction...</code>
	#:::: <code>This is a MATLAB function that takes input arguments.</code>		#::: <code>This is a MATLAB function that takes input arguments.</code>
	#:::: <code>Argument #1 -- 1 -- is a char</code>		#::: <code>Argument #1 -- 1 -- is a char</code>
	#:::: <code>This is a MATLAB function that takes input arguments.</code>		#::: <code>This is a MATLAB function that takes input arguments.</code>
	#:::: <code>Argument #1 -- a -- is a char</code>		#::: <code>Argument #1 -- a -- is a char</code>
	#:::: <code>This is a MATLAB function that takes input arguments.</code>		#::: <code>This is a MATLAB function that takes input arguments.</code>
	#:::: <code>Argument #1 -- 2 -- is a char</code>		#::: <code>Argument #1 -- 2 -- is a char</code>
	#:::: <code>This is a MATLAB function that takes input arguments.</code>		#::: <code>This is a MATLAB function that takes input arguments.</code>
	#:::: <code>Argument #1 -- b -- is a char</code>		#::: <code>Argument #1 -- b -- is a char</code>
	#:::: <code>This is a MATLAB function that takes input arguments.</code>		#::: <code>This is a MATLAB function that takes input arguments.</code>
	#:::: <code>Argument #1 -- 3 -- is a char</code>		#::: <code>Argument #1 -- 3 -- is a char</code>
	#:::: <code>This is a MATLAB function that takes input arguments.</code>		#::: <code>This is a MATLAB function that takes input arguments.</code>
	#:::: <code>Argument #1 -- c -- is a char</code>		#::: <code>Argument #1 -- c -- is a char</code>
	#:::: <code>Warning: No display specified. You will not be able to display graphics on the screen.</code>		#::: <code>Warning: No display specified. You will not be able to display graphics on the screen.</code>
	#:: ''exampleMatlabScript.joblog.[JOBID]''		#: ''exampleMatlabScript.joblog.[JOBID]''
	#::: This file has all the details about when, where, and how your job was processed. Useful information if you are going to be running this job over and over and need to fine tune the resources it uses.		#:: This file has all the details about when, where, and how your job was processed. Useful information if you are going to be running this job over and over and need to fine tune the resources it uses.
			# Better ways of checking on your job can be found [[Hoffman2:Monitoring Jobs\|here]].
	# Finally, go check the inbox of the email you used to sign up for your Hoffman2 account. There will be two emails from "root@mail.hoffman2.idre.ucla.edu" that indicate when the job was started and when the job was completed. This is one of the neat features of the queue so that you can be alerted about the progress of your job without having to stay logged into Hoffman2 and checking on it constantly.		# Finally, go check the inbox of the email you used to sign up for your Hoffman2 account. There will be two emails from "root@mail.hoffman2.idre.ucla.edu" that indicate when the job was started and when the job was completed. This is one of the neat features of the queue so that you can be alerted about the progress of your job without having to stay logged into Hoffman2 and checking on it constantly.