mdm-0.1.3/.hg_archival.txt0000644000000000000000000000013611255624051015521 0ustar00usergroup00000000000000repo: 6ecda764486ad000d011b6a60934053e1a556162 node: 6fb04ba5125f59f154166448cc159e0ab93ff617 mdm-0.1.3/.hgignore0000644000000000000000000000006611255624051014240 0ustar00usergroup00000000000000syntax: glob *.o mdm-master mdm-run mdm-slave mdm-top mdm-0.1.3/.hgtags0000644000000000000000000000033411255624051013711 0ustar00usergroup00000000000000d862df56868f4a9fc9b3157cd27a085b9665f4e8 release-0.1.0 06f194232f2c64383a5aa73628790893fe26ccca release-0.1.1 d7494b4db676ad9642f46c0c124138f3161b7f9b release-0.1.2 097ac17e7888f4194b1cf38a596e585b23d9ef7e release-0.1.3 mdm-0.1.3/CHANGES0000644000000000000000000000162411255624051013431 0ustar00usergroup00000000000000Version 0.1.3 Mon Sep 21 00:12:38 2009 -0700 http://hg.berlios.de/repos/mdm/rev/097ac17e7888 * Support CC, CFLAGS, LDFLAGS override in Makefile * Support DESTDIR in Makefile install targets * Install manual pages into share/man/man1 directory * Prioritize MDM_LIB over $PATH in mdm.screen Version 0.1.2 Fri Mar 20 00:57:59 2009 -0700 http://hg.berlios.de/repos/mdm/rev/d7494b4db676 * Require user acknowledge parallel commands with non-zero exit status * Add ncpus(1) script which detects the number of processors/cores * Pass working directory as a file descriptor instead of as a string * Make screen(1) start slaves (instead of shells) in new windows Version 0.1.1 Fri Mar 06 23:54:18 2009 -0800 http://hg.berlios.de/repos/mdm/rev/06f194232f2c * Add manual pages for user commands Version 0.1.0 Fri Mar 06 15:14:38 2009 -0800 http://hg.berlios.de/repos/mdm/rev/d862df56868f * Initial release mdm-0.1.3/LICENSE0000644000000000000000000002613611255624051013450 0ustar00usergroup00000000000000 Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright [yyyy] [name of copyright owner] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. mdm-0.1.3/Makefile0000644000000000000000000000324611255624051014100 0ustar00usergroup00000000000000# Time-stamp: <2009-09-20 23:58:48 cklin> override CC := $(shell which mdm-run > /dev/null && echo mdm-run) $(CC) override CFLAGS += -Wall -D_GNU_SOURCE -Iinclude SED := /bin/sed INSTALL := /usr/bin/install LN := /bin/ln GZIP := /bin/gzip LIB := library/buffer.o library/comms.o library/socket.o PROG := $(patsubst programs/%.c,%,$(wildcard programs/*.c)) PREFIX ?= /usr/local BIN_DIR := $(PREFIX)/bin LIB_DIR := $(PREFIX)/lib/mdm MAN_DIR := $(PREFIX)/share/man/man1 BIN_D_DIR := $(DESTDIR)$(BIN_DIR) LIB_D_DIR := $(DESTDIR)$(LIB_DIR) MAN_D_DIR := $(DESTDIR)$(MAN_DIR) all : $(PROG) mdm-master : library/hazard.o mdm-top : library/procfs.o mdm-top : override LDFLAGS += -lcurses mdm-% : programs/mdm-%.c $(LIB) $(CC) $(CFLAGS) $(LDFLAGS) -o $@ $+ LIB += library/hazard.o library/procfs.o $(LIB) : include/middleman.h $(PROG) : include/middleman.h MAN := $(wildcard documents/*.1) HTML := $(patsubst %,%.html,$(MAN)) %.1.html : %.1 rman -f html -r '%s.%s.html' $+ > $@ man-html : $(HTML) install : install-bin install-docs install-bin : all $(INSTALL) -d $(BIN_D_DIR) $(LIB_D_DIR) $(INSTALL) scripts/mdm.screen scripts/ncpus $(BIN_D_DIR) $(INSTALL) -s mdm-run $(BIN_D_DIR) $(LN) -f -s mdm-run $(BIN_D_DIR)/mdm-sync $(INSTALL) -s mdm-master mdm-slave mdm-top $(LIB_D_DIR) $(SED) -i -e "s:MDM_LIB:$(LIB_DIR):" $(BIN_D_DIR)/mdm.screen install-docs : $(INSTALL) -d $(MAN_D_DIR) $(INSTALL) -m 644 $(MAN) $(MAN_D_DIR) $(GZIP) -f -9 $(patsubst documents/%,$(MAN_D_DIR)/%,$(MAN)) $(LN) -f -s mdm-run.1.gz $(MAN_D_DIR)/mdm-sync.1.gz clean : $(RM) library/*.o dist-clean : clean $(RM) mdm-* documents/*.html .PHONY : all man-html install install-bin install-docs clean dist-clean mdm-0.1.3/README0000644000000000000000000001160011255624051013311 0ustar00usergroup00000000000000THE MIDDLEMAN PROJECT ===================== Homepage http://mdm.berlios.de/ Project Summary: http://developer.berlios.de/projects/mdm/ Code Repository: http://hg.berlios.de/repos/mdm Forums: http://developer.berlios.de/forum/?group_id=10680 Bug Tracker: http://developer.berlios.de/bugs/?group_id=10680 ABOUT THE PROJECT ================= The Middleman Project (mdm) aims to create utility programs that unleash the power of multi-processor and multi-core computer systems. It does so by helping you parallelize your shell scripts, Makefiles, or any other program that invoke external programs. SOFTWARE REQUIREMENTS ===================== To run mdm, you need a modern (2.6+) Linux system, GNU screen and the ncurses library. It should be easy to port to other Unix systems by writing new /proc parsers and fixing some library incompatibilities. BUILDING AND INSTALLING MDM =========================== To build mdm, simply run "make" at the toplevel. This project is simple enough so that there is no need for autoconf and automake. To install, use "make install" as follows: $ make install PREFIX=/install/directory/prefix Without the PREFIX override, "make install" installs mdm to /usr/local. HOW DOES IT WORK? ================= The philosophy behind mdm is that users should benefit from their multi-core systems without making drastic changes to their shell scripts. With mdm, you annotate your scripts to specify which commands might benefit from parallelization, and then you run it under the supervision of the mdm system. At runtime, the mdm system dynamically discovers parallelization opportunities and run the annotated commands in parallel as appropriate. MDM IN 3 EASY STEPS =================== Suppose you use the following shell script (encode.sh) for encoding your music library. It works, but it leaves your quad-core computer mostly idle because it processes only one file at a time. #!/bin/bash for i in */*.wav do echo $i ffmpeg -i "$i" "${i%%.wav}.mp3" done You can parallelize this shell script in three easy steps. 1. Find commands that you think are suitable for parallel execution, and annotate them with mdm-run. Here is the modified encode.sh: #!/bin/bash for i in *.wav do echo $i mdm-run ffmpeg -i "$i" "${i%%.wav}.mp3" done 2. Specify the I/O behavior of your parallel commands in an iospec file. You know ffmpeg reads from its -i option argument and writes to its command argument (w/o option), so this is what you write in your iospec file: ffmpeg R-i W You can skip this step if you are certain the parallel command cannot interfere with any other command in the script. 3. Run the script under mdm.screen as follows: $ mdm.screen -c iospec encode.sh You should see a monitoring program (mdm-top) displaying the execution status of your parallel commands, and the encoding process should (hopefully) complete a lot sooner because you are giving all processing cores a good workout! WHEN NOT TO ANNOTATE WITH MDM-RUN ================================= There are a few cases where you should not annotate a command with mdm-run. They are: 1. The command is a shell built-in, 2. You need to know the exit status of the command, or 3. You perform I/O redirection on the command THE I/O SPECIFICATION FILE ========================== The I/O specification file (iospec) specifies the I/O behavior of programs. The mdm system use these specifications to decide whether it is okay to run two annotated commands at the same time. Each line of the file describes a program. Here are a few examples: ffmpeg R-i W rm W cc W-o 0-c Rbusy R date Wbusy In plain English: * ffmpeg reads from the option argument of -i and writes to all its non-option arguments, * rm writes to all its non-option arguments, * cc writes to its -o argument, -c takes no arguments, reads from the (abstract) file "busy" and from its non-option arguments, and * date writes to the (abstract) file "busy". Adding the abstract file "busy" to the iospec ensures that mdm will never schedule the date command to run when any "cc" command is still running (and vice versa). Beware that the iospec format is subject to change in the future. WHEN TO USE MDM-SYNC ==================== The mdm-sync command is just like mdm-run, except that it does not submit the command for parallel execution. Use mdm-sync to annotate a command when you don't want it to run in parallel, but you think it might interfere with a command annotated by mdm-run. QUESTIONS? COMMENTS? ===================== Please feel free to leave questions or comments on the mdm support forum on BerliOS . If you prefer, you can also write to me at . -- Chuan-kai Lin Thu Mar 12 16:30:13 PDT 2009 mdm-0.1.3/documents/mdm-run.10000644000000000000000000000341211255624051016075 0ustar00usergroup00000000000000.TH MDM-RUN 1 "2009-03-06" Linux "User Commands" .SH NAME mdm-run \- run a command in parallel with mdm-master .SH SYNOPSIS .B mdm-run .I command .br .B mdm-sync .I command .SH DESCRIPTION .B mdm-run annotates .I command as a candidate for parallel execution. If an .B mdm-master process is present (i.e., the environment variable .B MDM_CMD_SOCK is defined), .B mdm-run submits .I command to the .BR mdm-master , and it exits when the master issues the .I command to an .B mdm-slave process (which may be connected to another terminal) for execution. .B mdm-sync runs .I command in place, but it coordinates with .B mdm-master to ensure that executing .I command does not interfere with commands that are currently executing in parallel. If there is no .B mdm-master process present (i.e., the environment variable .B MDM_CMD_SOCK is not defined), both .B mdm-run and .B mdm-sync simply run .I command as if you entered .I command all by itself. These programs are a part of the Middleman System (mdm). .SH OPTIONS Neither .B mdm-run nor .B mdm-sync accepts any options. .SH EXIT STATUS If there is no .B mdm-master process present, the exit status of .B mdm-run is the exit status of .IR command . If an .B mdm-master process is present, the exit status is 0 if and only if .I command is successfully issued to an .B mdm-slave process for execution. The exit status of .B mdm-sync is always the exit status of .IR command . .SH ENVIRONMENT The mdm system uses the following environment variable for internal communication, and thus you should not try to set them yourself. .IP MDM_CMD_SOCK .B mdm-run uses this environment variable to decide whether a .B mdm-master process is present. .SH EXAMPLE See .BR mdm.screen (1) for an example use of .BR mdm-run . .SH SEE ALSO .BR mdm.screen (1) mdm-0.1.3/documents/mdm.screen.10000644000000000000000000000570511255624051016560 0ustar00usergroup00000000000000.TH MDM.SCREEN 1 "2009-03-06" Linux "User Commands" .SH NAME mdm.screen \- run a command under mdm-master .SH SYNOPSIS .B mdm.screen .RI [ "-c iospec" ] .RI [ "-n slaves" ] .I command .SH DESCRIPTION .B mdm.screen runs .I command under .B mdm-master so that .B mdm-run commands in .I command can run in parallel. This program is a part of the Middleman System (mdm). .SH OPTIONS The .B mdm.screen command has two optional options. .IP "-c iospec" The iospec file (see below). .IP "-n slaves" The number of .B mdm-slave process to start (default 6). .SH THE IOSPEC FILE .B mdm-master uses the .I iospec file to decide whether two commands interfere with each other, and it only runs two commands at the same time if they do not interfere. Each line of the .I iospec file specifies the I/O behavior of a program, unless the line starts with a '#' character, in which case the line is treated as comments and ignored. The format is as follows: .RS .I program spec spec ... .RE .P .I program names the program whose behavior you are specifying. For each .I spec element, the first character indicates .IR usage , and the remaining characters indicate .IR resource . A .I resource that starts with a '-' character represents the argument of the corresponding program option. A .I resource that is an empty string represents a program argument. A .I resource that is not empty and does not start with a '-' character represents the .I resource string itself. A .I usage is typically an upper-case character. If two commands access the same resource with different .IR usage , .B mdm-master will not run these two commands at the same time. Furthermore, if a command access a resource with \'W\' .IR usage , .B mdm-master will not run it with any command that access the same resource in any way (including 'W'). There is a special case: if .I usage is '0' and .I resource starts with a '-' character, it means that the .I resource program option does not take any arguments. All programs has an implicit .IR spec " 'Rglobal'," so if you mark a program as 'Wglobal', it will not run while any other command is running. Here is an example to help clarify matters. Suppose we have the following iospec file: .RS cc W-o 0-c R .br rm W .RE .P Then these two commands .B can run at the same time: .RS cc -o pa pa.o lib.o .br cc -o pb pb.o lib.o .RE .P These two commands .B cannot run at the same time: .RS cc -o pa.o -c pa.c .br cc -o pa pa.o lib.o .RE .P And these two commands also .B cannot run at the same time: .RS cc -o pa pa.o lib.o .br rm pa .RE .SH EXIT STATUS The exit status of .B mdm.screen is 0. .SH EXAMPLE Here is what you do to decompress all files with .gz extension in the current directory tree. .RS mdm.screen find . -iname '*.gz' -exec mdm-run gunzip {} \\; .RE .P Since we run .B find under .B mdm.screen and invoking .B gunzip under .BR mdm-run , we specify that we wish to run the .B gunzip commands in parallel. .SH SEE ALSO .BR mdm-run "(1), " mdm-sync "(1), " ncpus (1) mdm-0.1.3/documents/ncpus.10000644000000000000000000000116611255624051015652 0ustar00usergroup00000000000000.TH NCPUS 1 "2009-03-10" Linux "User Commands" .SH NAME ncpus \- print the number of virtual CPUs .SH SYNOPSIS .B ncpus [ .I +n | .I n | .I -n ] .SH DESCRIPTION .B ncpus prints the number of virtual CPUs to standard output. With the .I +n argument, it prints the number of processors +n; with the .I -n argument, it prints the number of processors -n (if the result is less than 1, .B ncpus prints 1). With the .I n argument, .B ncpus prints .IR n . .B ncpus ignores malformed argument. This program is a part of the Middleman System (mdm). .SH EXIT STATUS The exit status of .B ncpus is 0. .SH SEE ALSO .BR mdm.screen (1) mdm-0.1.3/include/middleman.h0000644000000000000000000000450411255624051016164 0ustar00usergroup00000000000000// Time-stamp: <2009-03-11 22:40:13 cklin> /* middleman.h - Middleman System Header File Copyright 2009 Chuan-kai Lin Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ #ifndef __COMMS_H__ #define __COMMS_H__ #include #include #include #define MAX_SLAVES 16 #define MAX_HISTORY 60 #define CMD_SOCK_VAR "MDM_CMD_SOCK" #define FETCH_SOCK "fetch" #define ISSUE_SOCK "issue" #define MON_SOCK "monitor" #define LOG_FILE "messages" #define TOP_OP_EXIT 0 #define TOP_OP_FETCH 1 #define TOP_OP_ISSUE 2 #define TOP_OP_DONE 3 #define TOP_OP_ATTN 4 #define TOP_OP_ONLINE 10 #define TOP_OP_OFFLINE 11 typedef struct { char *buffer; char **svec; } sv; typedef struct { sv cmd, env; int cwd; } job; int serv_listen(const char *name); int serv_accept(int listenfd); int cli_conn(const char *name); int send_fd(int sockfd, int fd); int recv_fd(int sockfd); void check_sockdir(const char *path); ssize_t readn(int fd, void *vptr, size_t n); ssize_t writen(int fd, const void *vptr, size_t n); ssize_t write_int(int fd, int v); ssize_t write_pid(int fd, pid_t v); ssize_t write_time(int fd, time_t v); void read_job(int fd, job *job); void write_job(int fd, const job *job); int read_sv(int fd, sv *sv); int write_sv(int fd, char *const svec[]); void *xmalloc(size_t size); char *path_join(const char *path, const char *name); char *xstrdup(const char *s); void release_sv(sv *sv); void release_job(job *job); void flatten_sv(sv *sv); void init_iospec(const char *config_name); bool validate_job(sv *cmd); void register_job(sv *cmd); void unregister_job(sv *cmd); typedef struct { char state; pid_t ppid; unsigned long utime; time_t start_time; } proc; bool proc_stat(pid_t pid, proc *pptr); char *time_string(unsigned long seconds); #endif mdm-0.1.3/library/buffer.c0000644000000000000000000000423411255624051015517 0ustar00usergroup00000000000000// Time-stamp: <2009-03-11 22:48:46 cklin> /* buffer.c - Middleman System String and Buffer Procedures Copyright 2009 Chuan-kai Lin Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ #include #include #include #include #include "middleman.h" void *xmalloc(size_t size) { void *ptr = malloc(size); if (size > 0 && ptr == NULL) errx(100, "Memory allocation failure (%u bytes)", size); return ptr; } char *path_join(const char *path, const char *name) { int path_len, name_len; char *pathname; if (path == NULL || name == NULL) errx(101, "NULL argument in path_join"); if (path[0] != '/') errx(102, "Relative path argument 1 in path_join"); if (strchr(name, '/')) errx(103, "Relative path argument 2 in path_join"); path_len = strlen(path); name_len = strlen(name); pathname = xmalloc(path_len+name_len+2); strcpy(pathname, path); if (pathname[path_len-1] != '/') pathname[path_len++] = '/'; strcpy(pathname+path_len, name); return pathname; } char *xstrdup(const char *s) { char *dup = strdup(s); if (dup == NULL) err(104, "strdup"); return dup; } void release_sv(sv *sv) { free(sv->buffer); free(sv->svec); sv->buffer = NULL; sv->svec = NULL; } void release_job(job *job) { release_sv(&(job->cmd)); release_sv(&(job->env)); } void flatten_sv(sv *sv) { char *vp, *bp; int index; assert(sv->buffer != NULL && sv->svec != NULL); assert(sv->svec[0] == sv->buffer); for (index=0, bp=sv->buffer; sv->svec[index]; index++) { vp = sv->svec[index]; while (*vp) *(bp++) = *(vp++); *(bp++) = ' '; } *(--bp) = '\0'; free(sv->svec); sv->svec = NULL; } mdm-0.1.3/library/comms.c0000644000000000000000000000502111255624051015357 0ustar00usergroup00000000000000// Time-stamp: <2009-03-11 22:49:55 cklin> /* comms.c - Middleman System Communications Procedures Copyright 2009 Chuan-kai Lin Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ #include #include #include #include "middleman.h" // Read variable recorded-size block from file descriptor static int read_block(int fd, char **buffer) { int size = 0; readn(fd, &size, sizeof (int)); if (size == 0) { *buffer = NULL; return 0; } *buffer = xmalloc(size); readn(fd, *buffer, size); if ((*buffer)[size-1]) { warnx("read_block: input is not null-terminated"); return -2; } return size; } // Unpack string vector written by write_sv static int unpack_svec(sv *sv, int size) { int ch, seg; ch = seg = 0; sv->svec[seg++] = sv->buffer; while (ch < size) { if (sv->buffer[ch++]) continue; sv->svec[seg++] = sv->buffer+ch; } sv->svec[--seg] = NULL; return seg; } // Read string vector from file descriptor int read_sv(int fd, sv *sv) { int size, count; readn(fd, &count, sizeof (int)); sv->svec = xmalloc(++count * sizeof (char *)); size = read_block(fd, &(sv->buffer)); if (size > 0) unpack_svec(sv, size); return size; } // Write zero-terminated string with size to file descriptor int write_string(int fd, const char buffer[]) { int size; size = strlen(buffer)+1; write_int(fd, size); writen(fd, buffer, size); return size; } // Write NULL-terminated string vector to file descriptor int write_sv(int fd, char *const svec[]) { int index, size; for (index=0, size=0; svec[index]; index++, size++) size += strlen(svec[index]); write_int(fd, index); write_int(fd, size); for (index=0; svec[index]; index++) writen(fd, svec[index], strlen(svec[index])+1); return 0; } void read_job(int fd, job *job) { readn(fd, &(job->cwd), sizeof (int)); read_sv(fd, &(job->cmd)); read_sv(fd, &(job->env)); } void write_job(int fd, const job *job) { write_int(fd, job->cwd); write_sv(fd, job->cmd.svec); write_sv(fd, job->env.svec); } mdm-0.1.3/library/hazard.c0000644000000000000000000001371211255624051015520 0ustar00usergroup00000000000000// Time-stamp: <2009-03-04 13:52:53 cklin> /* hazard.c - Job Interference Decision Procedures Copyright 2009 Chuan-kai Lin Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ #include #include #include #include #include #include #include #include #include "middleman.h" #define IOS_MAX 4096 #define UTIL_MAX 4096 #define BUF_MAX 1024 typedef struct { char *name; char *resources; int res_count; } iospec; static void read_iospec(const char *spec, iospec *ios) { int count; bool new_token; char *cp; ios->resources = NULL; ios->name = xstrdup(spec); new_token = false; for (cp=ios->name, count=0; *cp; cp++) { if (isspace(*cp)) { new_token = true; *cp = '\0'; continue; } if (new_token) { if (ios->resources == NULL) ios->resources = cp; new_token = false; count++; } } ios->res_count = count; } static iospec iost[IOS_MAX]; static int ios_count; void init_iospec(const char *config_name) { char buffer[BUF_MAX]; FILE *config; config = fopen(config_name, "r"); if (config == NULL) err(120, "Cannot open config file %s", config_name); ios_count = 0; while (fgets(buffer, BUF_MAX, config)) { if (strlen(buffer) == BUF_MAX-1) err(121, "Line too long in config file"); if (buffer[0] == '#') continue; if (isspace(buffer[0])) errx(122, "Line begins with space"); if (ios_count == IOS_MAX) errx(123, "Too many lines in config file"); read_iospec(buffer, &(iost[ios_count++])); } fclose(config); } static iospec *find_iospec(const char *name) { int index; for (index=0; indexresources; return false; } if (ios == NULL) return false; for ( ; count < ios->res_count; cp++) { if (*cp == '\0') { new_token = true; continue; } if (new_token) { new_token = false; count++; *res = cp; return true; } } return false; } static char calc_usage(const char *opt, iospec *ios) { char *cp; iterate_spec(ios, NULL); while (iterate_spec(NULL, &cp)) if (strcmp(opt, cp+1) == 0) { if (*cp == '0' && *opt == '-') return calc_usage("", ios); return *cp; } return '-'; } typedef struct { char usage; char *name; int count; } util; static util utit[UTIL_MAX]; static int uti_count; static bool conflict(char c1, char c2) { return (c1 == 'W' || c1 != c2); } static bool check_conflict(char usage, const char *name) { int index; for (index=0; indexsvec[0]); base = basename(name); ios = find_iospec(base); opt = nul; ptr = cmd->svec; fixed = true; initial = true; if (ios) iterate_spec(ios, NULL); return false; } assert(ar && au); if (fixed) { *ar = "global"; *au = 'R'; fixed = false; return true; } if (ios == NULL) return false; if (initial) { while (iterate_spec(NULL, &spec)) if (spec[1] != '-' && spec[1] != '\0') { *ar = spec+1; *au = *spec; return true; } initial = false; } while (*(++ptr)) if (**ptr != '-') { char usage = calc_usage(opt, ios); opt = nul; if (usage != '-') { *ar = *ptr; *au = usage; return true; } } else opt = *ptr; ptr--; return false; } bool validate_job(sv *cmd) { char *res, usage; assert(cmd->buffer); iterate_usage(cmd, NULL, NULL); while (iterate_usage(NULL, &res, &usage)) if (check_conflict(usage, res)) return false; return true; } void register_job(sv *cmd) { char *res, usage; assert(cmd->buffer); iterate_usage(cmd, NULL, NULL); while (iterate_usage(NULL, &res, &usage)) add_utilization(usage, res); } void unregister_job(sv *cmd) { char *res, usage; assert(cmd->buffer); iterate_usage(cmd, NULL, NULL); while (iterate_usage(NULL, &res, &usage)) del_utilization(usage, res); } mdm-0.1.3/library/procfs.c0000644000000000000000000000540311255624051015541 0ustar00usergroup00000000000000// Time-stamp: <2009-02-28 11:32:50 cklin> /* procfs.c - Middleman System Linux /proc Procedures Copyright 2009 Chuan-kai Lin Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ #include #include #include #include #include #include #include #include "middleman.h" static time_t boot_time(void) { static unsigned long boot = 0; if (boot == 0) { char buffer[1024]; FILE *stat_fp = fopen("/proc/stat", "r"); while (fgets(buffer, sizeof (buffer), stat_fp)) if (strncmp(buffer, "btime ", 6) == 0) { sscanf(buffer+6, "%lu", &boot); break; } fclose(stat_fp); assert(boot != 0); } return boot; } static time_t jiffy_to_sec(unsigned long long jiffy) { static unsigned long long clk_tck; if (clk_tck == 0) clk_tck = sysconf(_SC_CLK_TCK); return jiffy/clk_tck; } #pragma GCC diagnostic ignored "-Wformat" bool proc_stat(pid_t pid, proc *pptr) { unsigned long long start_jiffies; char path[40], buffer[1024], *start; int stat_fd, num; snprintf(path, sizeof (path), "/proc/%d/stat", pid); stat_fd = open(path, O_RDONLY); if (stat_fd < 0) return false; num = read(stat_fd, buffer, sizeof (buffer)-1); close(stat_fd); if (num < 0) return false; buffer[num] = '\0'; start = strrchr(buffer, ')')+2; if (start == NULL) return false; num = sscanf(start, "%c " // state "%d %*d %*d %*d %*d " // ppid "%*u %*lu %*lu %*lu %*lu " // flags "%lu %*lu %*ld %*ld " // utime "%*ld %*ld %*ld %*ld " // priority "%llu", // starttime &(pptr->state), &(pptr->ppid), &(pptr->utime), &start_jiffies); if (num != 4) return false; pptr->utime = jiffy_to_sec(pptr->utime); pptr->start_time = boot_time() + jiffy_to_sec(start_jiffies); return true; } char *time_string(unsigned long seconds) { static char buffer[10]; unsigned long minutes; minutes = seconds / 60; seconds %= 60; if (minutes > 99) return "99:59+"; sprintf(buffer, "%2lu:%02lu ", minutes, seconds); return buffer; } mdm-0.1.3/library/socket.c0000644000000000000000000001315111255624051015534 0ustar00usergroup00000000000000// Time-stamp: <2009-03-11 22:14:47 cklin> /* socket.c - Middleman System Socket Procedures Copyright 2009 Chuan-kai Lin Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ #include #include #include #include #include #include #include "middleman.h" // Advanced Programming in the Unix Environment, Program 15.22 int serv_listen(const char *name) { int sockfd, addrlen; struct sockaddr_un unix_addr; struct sockaddr *sock_addr; sockfd = socket(AF_UNIX, SOCK_STREAM, 0); if (sockfd < 0) { warn("socket"); return -1; } addrlen = strlen(name); if (addrlen >= sizeof (unix_addr.sun_path)) { warnx("serv_listen: socket name too long"); return -2; } addrlen += sizeof (unix_addr.sun_family); memset(&unix_addr, 0, sizeof (unix_addr)); unix_addr.sun_family = AF_UNIX; strcpy(unix_addr.sun_path, name); unlink(name); sock_addr = (struct sockaddr *) &unix_addr; if (bind(sockfd, sock_addr, addrlen) < 0) { warn("bind"); return -3; } if (listen(sockfd, 5) < 0) { warn("listen"); return -4; } return sockfd; } // Advanced Programming in the Unix Environment, Program 15.24 int serv_accept(int listenfd) { int connfd; connfd = accept(listenfd, NULL, NULL); if (connfd < 0) warn("accept"); return connfd; } // Advanced Programming in the Unix Environment, Program 15.23 int cli_conn(const char *name) { int sockfd, addrlen; struct sockaddr_un unix_addr; struct sockaddr *sock_addr; sockfd = socket(AF_UNIX, SOCK_STREAM, 0); if (sockfd < 0) { warn("socket"); return -1; } addrlen = strlen(name); if (addrlen >= sizeof (unix_addr.sun_path)) { warnx("serv_listen: socket name too long"); return -2; } addrlen += sizeof (unix_addr.sun_family); memset(&unix_addr, 0, sizeof (unix_addr)); unix_addr.sun_family = AF_UNIX; strcpy(unix_addr.sun_path, name); sock_addr = (struct sockaddr *) &unix_addr; if (connect(sockfd, sock_addr, addrlen) < 0) { warn("connect"); return -3; } return sockfd; } // Advanced Programming in the Unix Environment, Program 12.13 ssize_t readn(int fd, void *vptr, size_t n) { size_t remaining; ssize_t received; char *ptr; remaining = n; ptr = vptr; while (remaining > 0) { received = read(fd, ptr, remaining); if (received <= 0) { if (received < 0) warn("read"); return received; } remaining -= received; ptr += received; } return n; } // Advanced Programming in the Unix Environment, Program 12.12 ssize_t writen(int fd, const void *vptr, size_t n) { size_t remaining; ssize_t written; const char *ptr; remaining = n; ptr = vptr; while (remaining > 0) { written = write(fd, ptr, remaining); if (written <= 0) { if (written < 0) warn("write"); return written; } remaining -= written; ptr += written; } return n; } // Advanced Programming in the Unix Environment, Program 15.9 #define CONTROLLEN (sizeof (struct cmsghdr)+sizeof (int)) static struct cmsghdr *cmptr = NULL; int send_fd(int sockfd, int fd) { struct iovec iov; struct msghdr msg; char ch; ch = 0; iov.iov_base = &ch; iov.iov_len = 1; msg.msg_iov = &iov; msg.msg_iovlen = 1; msg.msg_name = NULL; msg.msg_namelen = 0; if (!cmptr) cmptr = xmalloc(CONTROLLEN); cmptr->cmsg_level = SOL_SOCKET; cmptr->cmsg_type = SCM_RIGHTS; cmptr->cmsg_len = CONTROLLEN; msg.msg_control = (caddr_t) cmptr; msg.msg_controllen = CONTROLLEN; *(int *) CMSG_DATA(cmptr) = fd; if (sendmsg(sockfd, &msg, 0) != 1) { warn("send_fd sendmsg"); return -1; } return 0; } // Advanced Programming in the Unix Environment, Program 15.10 int recv_fd(int sockfd) { struct cmsghdr *cmptr = NULL; struct iovec iov; struct msghdr msg; char ch; int nread; iov.iov_base = &ch; iov.iov_len = 1; msg.msg_iov = &iov; msg.msg_iovlen = 1; msg.msg_name = NULL; msg.msg_namelen = 0; if (!cmptr) cmptr = xmalloc(CONTROLLEN); msg.msg_control = (caddr_t) cmptr; msg.msg_controllen = CONTROLLEN; nread = recvmsg(sockfd, &msg, 0); if (nread < 0) { warn("recv_fd recvmsg"); return -1; } if (nread == 0) { warnx("recv_fd: connection closed by server"); return -2; } if (ch == 0) if (msg.msg_controllen == CONTROLLEN) return *(int *) CMSG_DATA(cmptr); warnx("recv_fd: protocol error"); return -3; } // Write a scalar value to a file descriptor ssize_t write_int(int fd, int v) { return writen(fd, &v, sizeof (int)); } ssize_t write_pid(int fd, pid_t v) { return writen(fd, &v, sizeof (pid_t)); } ssize_t write_time(int fd, time_t v) { return writen(fd, &v, sizeof (time_t)); } // Basic security checks for the IPC directory void check_sockdir(const char *path) { struct stat st; if (lstat(path, &st) < 0) err(110, "stat(\"%s\")", path); if (! S_ISDIR(st.st_mode)) errx(111, "%s is not a directory", path); if (st.st_mode & S_IWOTH) errx(112, "%s is world-writable", path); } mdm-0.1.3/programs/mdm-master.c0000644000000000000000000001462711255624051016511 0ustar00usergroup00000000000000// Time-stamp: <2009-03-11 22:46:32 cklin> /* mdm-master.c - Middleman System Main Controller Copyright 2009 Chuan-kai Lin Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ #include #include #include #include #include #include #include #include #include #include "middleman.h" typedef struct { job job; int issue_fd; pid_t pid, run_pid; bool idle; } slave; #define GOOD_SLAVE(x) (0 < (x) && (x) < sc) static slave slaves[1+MAX_SLAVES]; static fd_set openfds; static int maxfd, sc; static void bump_maxfd(int fd) { if (fd > maxfd) maxfd = fd; } static void slave_init(int fd) { assert(sc < sizeof (slaves)); slaves[sc].issue_fd = fd; readn(fd, &(slaves[sc].pid), sizeof (pid_t)); slaves[sc].idle = true; FD_SET(fd, &openfds); bump_maxfd(fd); sc++; } static void slave_exit(int slave, bool halt) { int slave_fd; assert(GOOD_SLAVE(slave)); slave_fd = slaves[slave].issue_fd; release_job(&(slaves[slave].job)); slaves[slave] = slaves[--sc]; FD_CLR(slave_fd, &openfds); if (halt) write_int(slave_fd, 0); close(slave_fd); } static char *sockdir; static int init_issue(void) { char *issue_addr = path_join(sockdir, ISSUE_SOCK); int issue_fd = serv_listen(issue_addr); free(issue_addr); bump_maxfd(issue_fd); return issue_fd; } static int mon_fd; static void init_monitor(void) { char *mon_addr = path_join(sockdir, MON_SOCK); int mon_sock_fd = serv_listen(mon_addr); mon_fd = serv_accept(mon_sock_fd); close(mon_sock_fd); free(mon_addr); } static void init_mesg_log(void) { char *mesg_file = path_join(sockdir, LOG_FILE); int mesg_fd = open(mesg_file, O_WRONLY | O_CREAT, S_IRUSR); if (mesg_fd == -1) err(2, "Log file %s", mesg_file); dup2(mesg_fd, STDERR_FILENO); close(mesg_fd); free(mesg_file); } static int init_fetch(void) { char *fetch_addr = path_join(sockdir, FETCH_SOCK); int fetch_fd = serv_listen(fetch_addr); setenv(CMD_SOCK_VAR, fetch_addr, 1); free(fetch_addr); bump_maxfd(fetch_fd); return fetch_fd; } static int slave_wait(slave *slv, int *stp) { ssize_t n = readn(slv->issue_fd, stp, sizeof (int)); if (n && *stp == 0) { if (slv != slaves) unregister_job(&slv->job.cmd); slv->idle = true; } return n; } static bool wind_down, pending, sync_mode; static job job_pending; static int run_fd; static void fetch(int fetch_fd) { int opcode; assert(!pending); run_fd = serv_accept(fetch_fd); readn(run_fd, &opcode, sizeof (int)); pending = true; switch (opcode) { case 1: sync_mode = false; read_job(run_fd, &job_pending); write_int(mon_fd, TOP_OP_FETCH); write_sv(mon_fd, job_pending.cmd.svec); break; case 2: sync_mode = true; read_job(run_fd, &job_pending); break; default: warnx("Unknown mdm-run opcode %d", opcode); pending = false; break; } } static void issue(slave *slv) { assert(slv->idle); release_job(&(slv->job)); slv->job = job_pending; write_int(slv->issue_fd, 1); write_job(slv->issue_fd, &(slv->job)); readn(slv->issue_fd, &(slv->run_pid), sizeof (pid_t)); slv->idle = false; } static void issue_ack(int slave_index) { slave *slv = slaves+slave_index; assert(pending); issue(slv); pending = false; write_int(mon_fd, TOP_OP_ISSUE); write_pid(mon_fd, slv->pid); write_pid(mon_fd, slv->run_pid); write_int(run_fd, 0); close(run_fd); } static void process_tick(void) { int index; if (pending && sync_mode) if (validate_job(&job_pending.cmd)) { pending = false; write_int(run_fd, 0); close(run_fd); } if (pending) for (index=1; index0; index--) if (slaves[index].idle) { write_int(mon_fd, TOP_OP_OFFLINE); slave_exit(index, true); } } static void run_main(int issue_fd, char *argv[]) { assert(sc == 0); slave_init(serv_accept(issue_fd)); job_pending.cwd = open(".", O_RDONLY); job_pending.cmd.svec = argv; job_pending.env.svec = environ; issue(slaves); } int main(int argc, char *argv[]) { int issue_fd, fetch_fd, status; int slave_index; if (argc < 4) errx(1, "Need comms directory, iospec file, and command"); sockdir = *(++argv); check_sockdir(sockdir); init_iospec(*(++argv)); issue_fd = init_issue(); daemon(1, 0); init_mesg_log(); init_monitor(); fetch_fd = init_fetch(); run_main(issue_fd, argv+1); wind_down = false; while (sc > 1 || !wind_down) { fd_set readfds = openfds; if (!pending && !wind_down) FD_SET(fetch_fd, &readfds); if (sc < sizeof (slaves) && !wind_down) FD_SET(issue_fd, &readfds); if (select(maxfd+1, &readfds, NULL, NULL, NULL) < 0) err(3, "select"); if (FD_ISSET(fetch_fd, &readfds)) fetch(fetch_fd); if (FD_ISSET(slaves->issue_fd, &readfds)) { slave_wait(slaves, &status); wind_down = true; } for (slave_index=sc-1; slave_index>0; slave_index--) { slave *slv = &slaves[slave_index]; if (FD_ISSET(slv->issue_fd, &readfds)) { if (slave_wait(slv, &status) > 0) { if (status) { write_int(mon_fd, TOP_OP_ATTN); write_pid(mon_fd, slv->pid); write_int(mon_fd, status); } else { write_int(mon_fd, TOP_OP_DONE); write_pid(mon_fd, slv->pid); } } else { write_int(mon_fd, TOP_OP_OFFLINE); slave_exit(slave_index, false); } } } if (FD_ISSET(issue_fd, &readfds)) { slave_init(serv_accept(issue_fd)); write_int(mon_fd, TOP_OP_ONLINE); } process_tick(); } write_int(slaves->issue_fd, 0); write_int(mon_fd, TOP_OP_EXIT); return 0; } mdm-0.1.3/programs/mdm-run.c0000644000000000000000000000374011255624051016014 0ustar00usergroup00000000000000// Time-stamp: <2009-03-11 22:42:20 cklin> /* mdm-run.c - Middleman System Job Proxy Copyright 2009 Chuan-kai Lin Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ #include #include #include #include #include #include #include "middleman.h" extern char **environ; int main(int argc, char *argv[]) { struct stat sock_stat; char *master_addr; bool sync_mode; int master_fd, status; job job; if (argc < 2) errx(1, "Please supply command as arguments"); master_addr = getenv(CMD_SOCK_VAR); if (!master_addr) if (execvp(*argv, ++argv) < 0) errx(2, "execve: %s", *argv); if (lstat(master_addr, &sock_stat) < 0) err(3, "%s: Cannot stat master socket", master_addr); if (!S_ISSOCK(sock_stat.st_mode)) errx(4, "%s: Not a socket", master_addr); if (sock_stat.st_uid != geteuid()) errx(5, "%s: Belongs to someone else", master_addr); master_fd = cli_conn(master_addr); if (master_fd < 0) errx(6, "%s: cli_conn error", master_addr); sync_mode = !strcmp(basename(*argv), "mdm-sync"); write_int(master_fd, sync_mode ? 2 : 1); job.cwd = open(".", O_RDONLY); job.cmd.svec = ++argv; job.env.svec = environ; write_job(master_fd, &job); readn(master_fd, &status, sizeof (int)); if (sync_mode) if (execvp(*argv, argv) < 0) errx(2, "execve: %s", *argv); readn(master_fd, &status, sizeof (int)); close(master_fd); return status; } mdm-0.1.3/programs/mdm-slave.c0000644000000000000000000000475311255624051016327 0ustar00usergroup00000000000000// Time-stamp: <2009-03-11 22:46:42 cklin> /* mdm-slave.c - Middleman System Job Runner Copyright 2009 Chuan-kai Lin Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ #include #include #include #include #include #include #include #include #include "middleman.h" extern char **environ; static int hookup(const char *sockdir) { char *master_addr; int master_fd; check_sockdir(sockdir); master_addr = path_join(sockdir, ISSUE_SOCK); master_fd = cli_conn(master_addr); if (master_fd < 0) errx(2, "Cannot connect to mdm-master"); free(master_addr); return master_fd; } void screen_title(const char *format, ...) { va_list argp; if (isatty(STDIN_FILENO)) { va_start(argp, format); printf("\ek"); vprintf(format, argp); printf("\e\\\n"); va_end(argp); } } static void wait_user_ack(pid_t pid, sv *sv, int status) { char *buffer = NULL; size_t n; screen_title("Process %5u ATTN", pid); flatten_sv(sv); printf("\n%s", sv->buffer); printf("\nExit status %u: Press ENTER... ", status); if (getline(&buffer, &n, stdin) < 0) warn("getline"); free(buffer); } int main(int argc, char *argv[]) { job job; int master_fd, op, status; pid_t pid; if (argc != 2) errx(1, "Need socket directory argument"); master_fd = hookup(argv[1]); write_pid(master_fd, getpid()); for ( ; ; ) { screen_title("Idle"); readn(master_fd, &op, sizeof (int)); if (op == 0) break; read_job(master_fd, &job); pid = fork(); if (pid == 0) { screen_title("Process %5u", getpid()); close(master_fd); fchdir(job.cwd); environ = job.env.svec; execvp(job.cmd.svec[0], job.cmd.svec); } write_pid(master_fd, pid); wait(&status); write_int(master_fd, status); if (status) { if (isatty(STDIN_FILENO)) wait_user_ack(pid, &(job.cmd), status); write_int(master_fd, 0); } } return 0; } mdm-0.1.3/programs/mdm-top.c0000644000000000000000000001334111255624051016010 0ustar00usergroup00000000000000// Time-stamp: <2009-03-09 00:31:30 cklin> /* mdm-top.c - Middleman System Monitoring Utility Copyright 2009 Chuan-kai Lin Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ #include #include #include #include #include #include #include #include #include #include #include #include "middleman.h" static int hookup(const char *sockdir) { char *master_addr; int master_fd; check_sockdir(sockdir); master_addr = path_join(sockdir, MON_SOCK); master_fd = cli_conn(master_addr); if (master_fd < 0) errx(2, "Cannot connect to mdm-master"); free(master_addr); return master_fd; } #define STAGE_FETCH 300 #define STAGE_ISSUE 301 #define STAGE_DONE 302 #define STAGE_ATTN 303 typedef struct { sv cmd; proc pc; pid_t pid, run_pid; int stage; int status; } run; static run runs[MAX_HISTORY]; static int rc, sc, ac; static void release_run(void) { int index; assert(rc == MAX_HISTORY); for (index=0; runs[index].stage != STAGE_DONE; index++); release_sv(&(runs[index].cmd)); while (++index < rc) runs[index-1] = runs[index]; rc--; } static void init_run(sv *cmd) { if (rc == MAX_HISTORY) release_run(); runs[rc].pid = 0; runs[rc].run_pid = 0; runs[rc].stage = STAGE_FETCH; runs[rc].cmd = *cmd; flatten_sv(&(runs[rc].cmd)); rc++; } static int find_run(pid_t pid) { int index; for (index=rc-1; index>=0; index--) if (runs[index].pid == pid) return index; errx(3, "Cannot find run with pid %d", pid); return -1; } static void start_run(pid_t pid, pid_t run_pid) { int index = rc-1; assert(rc > 0); assert(runs[index].stage == STAGE_FETCH); runs[index].pid = pid; runs[index].run_pid = run_pid; runs[index].stage = STAGE_ISSUE; ac++; } static void end_run(pid_t pid, int status) { int index = find_run(pid); run *rptr = runs+index; if (status > 255) status = 255; switch (rptr->stage) { case STAGE_ISSUE: ac--; rptr->status = status; rptr->pc.state = ' '; if (status) { rptr-> stage = STAGE_ATTN; break; } case STAGE_ATTN: assert(status == 0); rptr->stage = STAGE_DONE; break; default: errx(6, "Job stage error (%d)", rptr->stage); } } void update_display(void) { char start[9], *utime; struct tm *ltime; time_t now; run *rptr; proc *pptr; int index, row, col, y, x; now = time(NULL); mvprintw(0, 2, "Active Jobs: %2d/%2d", ac, sc); mvprintw(0, 28, "%s", ctime(&now)); getmaxyx(stdscr, row, col); mvaddstr(2, 3, "START PID ST CPU COMMAND"); for (index=0, y=3; indexstage == STAGE_DONE) continue; rptr = runs+index; pptr = &(rptr->pc); if (rptr->stage == STAGE_ISSUE) { pptr->state = '!'; proc_stat(rptr->run_pid, pptr); attron(A_REVERSE); } if (rptr->stage == STAGE_ATTN) attron(A_BLINK); move(y++, 0); if (rptr->stage != STAGE_FETCH) { utime = time_string(pptr->utime); ltime = localtime(&(pptr->start_time)); strftime(start, sizeof (start), "%T", ltime); printw("%s %5d ", start, rptr->run_pid); if (rptr->stage == STAGE_ISSUE) printw(" %c %s ", pptr->state, utime); else printw("%3d %s ", rptr->status, utime); } else { addstr(" - - "); addstr(" - "); } addnstr(rptr->cmd.buffer, col-27); for (x=27+strlen(rptr->cmd.buffer); x&2 "usage: $name [-c iospec] [-n slaves] cmd ..." exit 1 } while getopts c:n: OPTION do case $OPTION in c) iospec="$OPTARG" ;; n) slaves="$OPTARG" if [ $slaves -lt 1 ] then echo >&2 "error: must have at least 1 slave" exit 2 fi ;; [?]) usage esac done shift $[OPTIND-1] [ $# -eq 0 ] && usage function print_screenrc { echo shell $TMPFILE/shell echo screen mdm-top $TMPFILE while [ $slaves -ge 0 ] do echo screen slaves=$[slaves-1] done echo select 0 } function write_slave_proxy { cat < $TMPFILE/shell #!/bin/sh exec mdm-slave $TMPFILE EOF chmod 700 $TMPFILE/shell } TMPFILE=`mktemp -d -t mdm-XXXXXXXXXX` || exit 1 print_screenrc > $TMPFILE/screenrc write_slave_proxy mdm-master $TMPFILE "$iospec" "$@" screen -c $TMPFILE/screenrc cat $TMPFILE/messages rm -Rf $TMPFILE mdm-0.1.3/scripts/ncpus0000755000000000000000000000214011255624051015175 0ustar00usergroup00000000000000#!/bin/bash # ncpus - Print the number of (virtual) processors # # Copyright 2009 Chuan-kai Lin # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. CPUS=`grep '^processor' /proc/cpuinfo | wc -l` function isnumber { [ "$1" -eq "$1" ] 2> /dev/null } function limit { if [ "$1" -gt 0 ] then echo "$1" else echo 1 fi } function offset { if isnumber "$1" then limit "$[CPUS$1]" else echo "$CPUS" fi } if [ -z "$1" ] then echo "$CPUS" elif [ "$1" != "${1##+}" ] then offset "$1" elif [ "$1" != "${1##-}" ] then offset "$1" elif isnumber "$1" then limit "$1" else echo "$CPUS" fi