pax_global_header00006660000000000000000000000064145714124300014513gustar00rootroot0000000000000052 comment=e090c478140941e125c716b5c8c43f8e9fcb4f3d damo-2.2.4/000077500000000000000000000000001457141243000124405ustar00rootroot00000000000000damo-2.2.4/.codespell_ignore000066400000000000000000000000061457141243000157520ustar00rootroot00000000000000damon damo-2.2.4/.gitignore000066400000000000000000000002101457141243000144210ustar00rootroot00000000000000__pycache__/* *.pyc damon.data damon.data.old damon.data.perf.data.old damon.snap.data damon.adjusted.data tests/schemes/alloc_1gb_spin damo-2.2.4/.pre-commit-config.yaml000066400000000000000000000007461457141243000167300ustar00rootroot00000000000000repos: - repo: https://github.com/pre-commit/pre-commit-hooks rev: v4.4.0 hooks: - id: check-yaml - id: end-of-file-fixer exclude: tests/report/expects - id: trailing-whitespace exclude: tests/report/expects - repo: https://github.com/PyCQA/isort rev: 5.12.0 hooks: - id: isort - repo: https://github.com/codespell-project/codespell rev: v2.2.5 hooks: - id: codespell args: ['-I', '.codespell_ignore'] damo-2.2.4/CONTRIBUTING000066400000000000000000000031461457141243000142760ustar00rootroot00000000000000General Process =============== For contributions, please refer to the linux kernel development process[1] and send patches to sj@kernel.org or pull-requests via Github. The contributions should have proper 'Signed-off-by:' tags[2]. The tag will be considered the same as that of the linux kernel development process. There are two branches, 'master' and 'next'. Changes first merged in 'next', get tested, and finally merged in 'master', if the tests show no problem. So, please base your work on 'next'. [1] https://docs.kernel.org/process/index.html [2] https://docs.kernel.org/process/submitting-patches.html#sign-your-work-the-developer-s-certificate-of-origin What To Do, First? ================== Testing First ------------- If you want to participate in the development but are unsure what to do first, you could consider running DAMO tests on your system. It might reveal some issues in DAMO. You could also contribute more tests. You can run the tests via below command from the root of DAMO. $ sudo ./tests/run.sh If you want to run wider range of tests, or have interests in not only DAMO but also DAMON, you could also run DAMON tests suite (https://github.com/awslabs/damon-tests) on your system. It might reveal some issues in DAMO or DAMON. TODO list --------- There is a list of todo items for DAMO in 'TODO' file of this repo. If you don't have specific task to do at the moment, you could consider finding one from the file. The list is not well managed at the moment, and the items may not well explained. Please feel free to reach out to the original author of the item for asking details. damo-2.2.4/COPYING000066400000000000000000000432541457141243000135030ustar00rootroot00000000000000 GNU GENERAL PUBLIC LICENSE Version 2, June 1991 Copyright (C) 1989, 1991 Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. Preamble The licenses for most software are designed to take away your freedom to share and change it. By contrast, the GNU General Public License is intended to guarantee your freedom to share and change free software--to make sure the software is free for all its users. This General Public License applies to most of the Free Software Foundation's software and to any other program whose authors commit to using it. (Some other Free Software Foundation software is covered by the GNU Lesser General Public License instead.) You can apply it to your programs, too. When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for this service if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs; and that you know you can do these things. To protect your rights, we need to make restrictions that forbid anyone to deny you these rights or to ask you to surrender the rights. These restrictions translate to certain responsibilities for you if you distribute copies of the software, or if you modify it. For example, if you distribute copies of such a program, whether gratis or for a fee, you must give the recipients all the rights that you have. You must make sure that they, too, receive or can get the source code. And you must show them these terms so they know their rights. We protect your rights with two steps: (1) copyright the software, and (2) offer you this license which gives you legal permission to copy, distribute and/or modify the software. Also, for each author's protection and ours, we want to make certain that everyone understands that there is no warranty for this free software. If the software is modified by someone else and passed on, we want its recipients to know that what they have is not the original, so that any problems introduced by others will not reflect on the original authors' reputations. Finally, any free program is threatened constantly by software patents. We wish to avoid the danger that redistributors of a free program will individually obtain patent licenses, in effect making the program proprietary. To prevent this, we have made it clear that any patent must be licensed for everyone's free use or not licensed at all. The precise terms and conditions for copying, distribution and modification follow. GNU GENERAL PUBLIC LICENSE TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION 0. This License applies to any program or other work which contains a notice placed by the copyright holder saying it may be distributed under the terms of this General Public License. The "Program", below, refers to any such program or work, and a "work based on the Program" means either the Program or any derivative work under copyright law: that is to say, a work containing the Program or a portion of it, either verbatim or with modifications and/or translated into another language. (Hereinafter, translation is included without limitation in the term "modification".) Each licensee is addressed as "you". Activities other than copying, distribution and modification are not covered by this License; they are outside its scope. The act of running the Program is not restricted, and the output from the Program is covered only if its contents constitute a work based on the Program (independent of having been made by running the Program). Whether that is true depends on what the Program does. 1. You may copy and distribute verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice and disclaimer of warranty; keep intact all the notices that refer to this License and to the absence of any warranty; and give any other recipients of the Program a copy of this License along with the Program. You may charge a fee for the physical act of transferring a copy, and you may at your option offer warranty protection in exchange for a fee. 2. You may modify your copy or copies of the Program or any portion of it, thus forming a work based on the Program, and copy and distribute such modifications or work under the terms of Section 1 above, provided that you also meet all of these conditions: a) You must cause the modified files to carry prominent notices stating that you changed the files and the date of any change. b) You must cause any work that you distribute or publish, that in whole or in part contains or is derived from the Program or any part thereof, to be licensed as a whole at no charge to all third parties under the terms of this License. c) If the modified program normally reads commands interactively when run, you must cause it, when started running for such interactive use in the most ordinary way, to print or display an announcement including an appropriate copyright notice and a notice that there is no warranty (or else, saying that you provide a warranty) and that users may redistribute the program under these conditions, and telling the user how to view a copy of this License. (Exception: if the Program itself is interactive but does not normally print such an announcement, your work based on the Program is not required to print an announcement.) These requirements apply to the modified work as a whole. If identifiable sections of that work are not derived from the Program, and can be reasonably considered independent and separate works in themselves, then this License, and its terms, do not apply to those sections when you distribute them as separate works. But when you distribute the same sections as part of a whole which is a work based on the Program, the distribution of the whole must be on the terms of this License, whose permissions for other licensees extend to the entire whole, and thus to each and every part regardless of who wrote it. Thus, it is not the intent of this section to claim rights or contest your rights to work written entirely by you; rather, the intent is to exercise the right to control the distribution of derivative or collective works based on the Program. In addition, mere aggregation of another work not based on the Program with the Program (or with a work based on the Program) on a volume of a storage or distribution medium does not bring the other work under the scope of this License. 3. You may copy and distribute the Program (or a work based on it, under Section 2) in object code or executable form under the terms of Sections 1 and 2 above provided that you also do one of the following: a) Accompany it with the complete corresponding machine-readable source code, which must be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange; or, b) Accompany it with a written offer, valid for at least three years, to give any third party, for a charge no more than your cost of physically performing source distribution, a complete machine-readable copy of the corresponding source code, to be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange; or, c) Accompany it with the information you received as to the offer to distribute corresponding source code. (This alternative is allowed only for noncommercial distribution and only if you received the program in object code or executable form with such an offer, in accord with Subsection b above.) The source code for a work means the preferred form of the work for making modifications to it. For an executable work, complete source code means all the source code for all modules it contains, plus any associated interface definition files, plus the scripts used to control compilation and installation of the executable. However, as a special exception, the source code distributed need not include anything that is normally distributed (in either source or binary form) with the major components (compiler, kernel, and so on) of the operating system on which the executable runs, unless that component itself accompanies the executable. If distribution of executable or object code is made by offering access to copy from a designated place, then offering equivalent access to copy the source code from the same place counts as distribution of the source code, even though third parties are not compelled to copy the source along with the object code. 4. You may not copy, modify, sublicense, or distribute the Program except as expressly provided under this License. Any attempt otherwise to copy, modify, sublicense or distribute the Program is void, and will automatically terminate your rights under this License. However, parties who have received copies, or rights, from you under this License will not have their licenses terminated so long as such parties remain in full compliance. 5. You are not required to accept this License, since you have not signed it. However, nothing else grants you permission to modify or distribute the Program or its derivative works. These actions are prohibited by law if you do not accept this License. Therefore, by modifying or distributing the Program (or any work based on the Program), you indicate your acceptance of this License to do so, and all its terms and conditions for copying, distributing or modifying the Program or works based on it. 6. Each time you redistribute the Program (or any work based on the Program), the recipient automatically receives a license from the original licensor to copy, distribute or modify the Program subject to these terms and conditions. You may not impose any further restrictions on the recipients' exercise of the rights granted herein. You are not responsible for enforcing compliance by third parties to this License. 7. If, as a consequence of a court judgment or allegation of patent infringement or for any other reason (not limited to patent issues), conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot distribute so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not distribute the Program at all. For example, if a patent license would not permit royalty-free redistribution of the Program by all those who receive copies directly or indirectly through you, then the only way you could satisfy both it and this License would be to refrain entirely from distribution of the Program. If any portion of this section is held invalid or unenforceable under any particular circumstance, the balance of the section is intended to apply and the section as a whole is intended to apply in other circumstances. It is not the purpose of this section to induce you to infringe any patents or other property right claims or to contest validity of any such claims; this section has the sole purpose of protecting the integrity of the free software distribution system, which is implemented by public license practices. Many people have made generous contributions to the wide range of software distributed through that system in reliance on consistent application of that system; it is up to the author/donor to decide if he or she is willing to distribute software through any other system and a licensee cannot impose that choice. This section is intended to make thoroughly clear what is believed to be a consequence of the rest of this License. 8. If the distribution and/or use of the Program is restricted in certain countries either by patents or by copyrighted interfaces, the original copyright holder who places the Program under this License may add an explicit geographical distribution limitation excluding those countries, so that distribution is permitted only in or among countries not thus excluded. In such case, this License incorporates the limitation as if written in the body of this License. 9. The Free Software Foundation may publish revised and/or new versions of the General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. Each version is given a distinguishing version number. If the Program specifies a version number of this License which applies to it and "any later version", you have the option of following the terms and conditions either of that version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of this License, you may choose any version ever published by the Free Software Foundation. 10. If you wish to incorporate parts of the Program into other free programs whose distribution conditions are different, write to the author to ask for permission. For software which is copyrighted by the Free Software Foundation, write to the Free Software Foundation; we sometimes make exceptions for this. Our decision will be guided by the two goals of preserving the free status of all derivatives of our free software and of promoting the sharing and reuse of software generally. NO WARRANTY 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. END OF TERMS AND CONDITIONS How to Apply These Terms to Your New Programs If you develop a new program, and you want it to be of the greatest possible use to the public, the best way to achieve this is to make it free software which everyone can redistribute and change under these terms. To do so, attach the following notices to the program. It is safest to attach them to the start of each source file to most effectively convey the exclusion of warranty; and each file should have at least the "copyright" line and a pointer to where the full notice is found. Copyright (C) This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. Also add information on how to contact you by electronic and paper mail. If the program is interactive, make it output a short notice like this when it starts in an interactive mode: Gnomovision version 69, Copyright (C) year name of author Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. This is free software, and you are welcome to redistribute it under certain conditions; type `show c' for details. The hypothetical commands `show w' and `show c' should show the appropriate parts of the General Public License. Of course, the commands you use may be called something other than `show w' and `show c'; they could even be mouse-clicks or menu items--whatever suits your program. You should also get your employer (if you work as a programmer) or your school, if any, to sign a "copyright disclaimer" for the program, if necessary. Here is a sample; alter the names: Yoyodyne, Inc., hereby disclaims all copyright interest in the program `Gnomovision' (which makes passes at compilers) written by James Hacker. , 1 April 1989 Ty Coon, President of Vice This General Public License does not permit incorporating your program into proprietary programs. If your program is a subroutine library, you may consider it more useful to permit linking proprietary applications with the library. If this is what you want to do, use the GNU Lesser General Public License instead of this License. damo-2.2.4/LICENSE000066400000000000000000000432541457141243000134550ustar00rootroot00000000000000 GNU GENERAL PUBLIC LICENSE Version 2, June 1991 Copyright (C) 1989, 1991 Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. Preamble The licenses for most software are designed to take away your freedom to share and change it. By contrast, the GNU General Public License is intended to guarantee your freedom to share and change free software--to make sure the software is free for all its users. This General Public License applies to most of the Free Software Foundation's software and to any other program whose authors commit to using it. (Some other Free Software Foundation software is covered by the GNU Lesser General Public License instead.) You can apply it to your programs, too. When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for this service if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs; and that you know you can do these things. To protect your rights, we need to make restrictions that forbid anyone to deny you these rights or to ask you to surrender the rights. These restrictions translate to certain responsibilities for you if you distribute copies of the software, or if you modify it. For example, if you distribute copies of such a program, whether gratis or for a fee, you must give the recipients all the rights that you have. You must make sure that they, too, receive or can get the source code. And you must show them these terms so they know their rights. We protect your rights with two steps: (1) copyright the software, and (2) offer you this license which gives you legal permission to copy, distribute and/or modify the software. Also, for each author's protection and ours, we want to make certain that everyone understands that there is no warranty for this free software. If the software is modified by someone else and passed on, we want its recipients to know that what they have is not the original, so that any problems introduced by others will not reflect on the original authors' reputations. Finally, any free program is threatened constantly by software patents. We wish to avoid the danger that redistributors of a free program will individually obtain patent licenses, in effect making the program proprietary. To prevent this, we have made it clear that any patent must be licensed for everyone's free use or not licensed at all. The precise terms and conditions for copying, distribution and modification follow. GNU GENERAL PUBLIC LICENSE TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION 0. This License applies to any program or other work which contains a notice placed by the copyright holder saying it may be distributed under the terms of this General Public License. The "Program", below, refers to any such program or work, and a "work based on the Program" means either the Program or any derivative work under copyright law: that is to say, a work containing the Program or a portion of it, either verbatim or with modifications and/or translated into another language. (Hereinafter, translation is included without limitation in the term "modification".) Each licensee is addressed as "you". Activities other than copying, distribution and modification are not covered by this License; they are outside its scope. The act of running the Program is not restricted, and the output from the Program is covered only if its contents constitute a work based on the Program (independent of having been made by running the Program). Whether that is true depends on what the Program does. 1. You may copy and distribute verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice and disclaimer of warranty; keep intact all the notices that refer to this License and to the absence of any warranty; and give any other recipients of the Program a copy of this License along with the Program. You may charge a fee for the physical act of transferring a copy, and you may at your option offer warranty protection in exchange for a fee. 2. You may modify your copy or copies of the Program or any portion of it, thus forming a work based on the Program, and copy and distribute such modifications or work under the terms of Section 1 above, provided that you also meet all of these conditions: a) You must cause the modified files to carry prominent notices stating that you changed the files and the date of any change. b) You must cause any work that you distribute or publish, that in whole or in part contains or is derived from the Program or any part thereof, to be licensed as a whole at no charge to all third parties under the terms of this License. c) If the modified program normally reads commands interactively when run, you must cause it, when started running for such interactive use in the most ordinary way, to print or display an announcement including an appropriate copyright notice and a notice that there is no warranty (or else, saying that you provide a warranty) and that users may redistribute the program under these conditions, and telling the user how to view a copy of this License. (Exception: if the Program itself is interactive but does not normally print such an announcement, your work based on the Program is not required to print an announcement.) These requirements apply to the modified work as a whole. If identifiable sections of that work are not derived from the Program, and can be reasonably considered independent and separate works in themselves, then this License, and its terms, do not apply to those sections when you distribute them as separate works. But when you distribute the same sections as part of a whole which is a work based on the Program, the distribution of the whole must be on the terms of this License, whose permissions for other licensees extend to the entire whole, and thus to each and every part regardless of who wrote it. Thus, it is not the intent of this section to claim rights or contest your rights to work written entirely by you; rather, the intent is to exercise the right to control the distribution of derivative or collective works based on the Program. In addition, mere aggregation of another work not based on the Program with the Program (or with a work based on the Program) on a volume of a storage or distribution medium does not bring the other work under the scope of this License. 3. You may copy and distribute the Program (or a work based on it, under Section 2) in object code or executable form under the terms of Sections 1 and 2 above provided that you also do one of the following: a) Accompany it with the complete corresponding machine-readable source code, which must be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange; or, b) Accompany it with a written offer, valid for at least three years, to give any third party, for a charge no more than your cost of physically performing source distribution, a complete machine-readable copy of the corresponding source code, to be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange; or, c) Accompany it with the information you received as to the offer to distribute corresponding source code. (This alternative is allowed only for noncommercial distribution and only if you received the program in object code or executable form with such an offer, in accord with Subsection b above.) The source code for a work means the preferred form of the work for making modifications to it. For an executable work, complete source code means all the source code for all modules it contains, plus any associated interface definition files, plus the scripts used to control compilation and installation of the executable. However, as a special exception, the source code distributed need not include anything that is normally distributed (in either source or binary form) with the major components (compiler, kernel, and so on) of the operating system on which the executable runs, unless that component itself accompanies the executable. If distribution of executable or object code is made by offering access to copy from a designated place, then offering equivalent access to copy the source code from the same place counts as distribution of the source code, even though third parties are not compelled to copy the source along with the object code. 4. You may not copy, modify, sublicense, or distribute the Program except as expressly provided under this License. Any attempt otherwise to copy, modify, sublicense or distribute the Program is void, and will automatically terminate your rights under this License. However, parties who have received copies, or rights, from you under this License will not have their licenses terminated so long as such parties remain in full compliance. 5. You are not required to accept this License, since you have not signed it. However, nothing else grants you permission to modify or distribute the Program or its derivative works. These actions are prohibited by law if you do not accept this License. Therefore, by modifying or distributing the Program (or any work based on the Program), you indicate your acceptance of this License to do so, and all its terms and conditions for copying, distributing or modifying the Program or works based on it. 6. Each time you redistribute the Program (or any work based on the Program), the recipient automatically receives a license from the original licensor to copy, distribute or modify the Program subject to these terms and conditions. You may not impose any further restrictions on the recipients' exercise of the rights granted herein. You are not responsible for enforcing compliance by third parties to this License. 7. If, as a consequence of a court judgment or allegation of patent infringement or for any other reason (not limited to patent issues), conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot distribute so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not distribute the Program at all. For example, if a patent license would not permit royalty-free redistribution of the Program by all those who receive copies directly or indirectly through you, then the only way you could satisfy both it and this License would be to refrain entirely from distribution of the Program. If any portion of this section is held invalid or unenforceable under any particular circumstance, the balance of the section is intended to apply and the section as a whole is intended to apply in other circumstances. It is not the purpose of this section to induce you to infringe any patents or other property right claims or to contest validity of any such claims; this section has the sole purpose of protecting the integrity of the free software distribution system, which is implemented by public license practices. Many people have made generous contributions to the wide range of software distributed through that system in reliance on consistent application of that system; it is up to the author/donor to decide if he or she is willing to distribute software through any other system and a licensee cannot impose that choice. This section is intended to make thoroughly clear what is believed to be a consequence of the rest of this License. 8. If the distribution and/or use of the Program is restricted in certain countries either by patents or by copyrighted interfaces, the original copyright holder who places the Program under this License may add an explicit geographical distribution limitation excluding those countries, so that distribution is permitted only in or among countries not thus excluded. In such case, this License incorporates the limitation as if written in the body of this License. 9. The Free Software Foundation may publish revised and/or new versions of the General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. Each version is given a distinguishing version number. If the Program specifies a version number of this License which applies to it and "any later version", you have the option of following the terms and conditions either of that version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of this License, you may choose any version ever published by the Free Software Foundation. 10. If you wish to incorporate parts of the Program into other free programs whose distribution conditions are different, write to the author to ask for permission. For software which is copyrighted by the Free Software Foundation, write to the Free Software Foundation; we sometimes make exceptions for this. Our decision will be guided by the two goals of preserving the free status of all derivatives of our free software and of promoting the sharing and reuse of software generally. NO WARRANTY 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. END OF TERMS AND CONDITIONS How to Apply These Terms to Your New Programs If you develop a new program, and you want it to be of the greatest possible use to the public, the best way to achieve this is to make it free software which everyone can redistribute and change under these terms. To do so, attach the following notices to the program. It is safest to attach them to the start of each source file to most effectively convey the exclusion of warranty; and each file should have at least the "copyright" line and a pointer to where the full notice is found. Copyright (C) This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. Also add information on how to contact you by electronic and paper mail. If the program is interactive, make it output a short notice like this when it starts in an interactive mode: Gnomovision version 69, Copyright (C) year name of author Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. This is free software, and you are welcome to redistribute it under certain conditions; type `show c' for details. The hypothetical commands `show w' and `show c' should show the appropriate parts of the General Public License. Of course, the commands you use may be called something other than `show w' and `show c'; they could even be mouse-clicks or menu items--whatever suits your program. You should also get your employer (if you work as a programmer) or your school, if any, to sign a "copyright disclaimer" for the program, if necessary. Here is a sample; alter the names: Yoyodyne, Inc., hereby disclaims all copyright interest in the program `Gnomovision' (which makes passes at compilers) written by James Hacker. , 1 April 1989 Ty Coon, President of Vice This General Public License does not permit incorporating your program into proprietary programs. If your program is a subroutine library, you may consider it more useful to permit linking proprietary applications with the library. If this is what you want to do, use the GNU Lesser General Public License instead of this License. damo-2.2.4/README.md000066400000000000000000000206551457141243000137270ustar00rootroot00000000000000DAMO: Data Access Monitoring Operator ===================================== `damo` is a user space tool for [DAMON](https://damonitor.github.io). Using this, you can monitor the data access patterns of your system or workloads and make data access-aware memory management optimizations. ![damo monitor demo for water_nsquared](images/damo_monitor_water_nsquared.gif) Demo Video ========== Please click the below thumbnail to show the short demo video. [![DAMON: a demo for the Kernel Summit 2020]( http://img.youtube.com/vi/l63eqbVBZRY/0.jpg)]( http://www.youtube.com/watch?v=l63eqbVBZRY "DAMON: a demo for the Kernel Summit 2020") Getting Started =============== [![Packaging status](https://repology.org/badge/vertical-allrepos/damo.svg)](https://repology.org/project/damo/versions) Follow below instructions and commands to monitor and visualize the access pattern of your workload. $ # ensure DAMON is enabled on your kernel $ # install damo from PyPI, or use your distribution's package manager $ sudo pip3 install damo $ sudo damo record $(pidof ) $ sudo damo report heats --heatmap stdout --stdout_heatmap_color emotion The last command will show the access pattern of your workload, like below: ![masim zigzag heatmap in ascii](images/masim_zigzag_heatmap_ascii.png) ![masim stairs heatmap in ascii](images/masim_stairs_heatmap_ascii.png) FAQs ==== How can I ensure DAMON is enabled on my kernel? ----------------------------------------------- Please refer to 'Install' [section](https://sjp38.github.io/post/damon/#install) of the project webpage. Where can I get more detailed usage? ------------------------------------ The below sections provide quick introductions for `damo`'s major features. For more detailed usage, please refer to [USAGE.md](USAGE.md) file. What does the version numbers mean? ----------------------------------- Nothing at all but indicate which version is more fresh. A higher version number means it is more recently released. Will `pip3 install damo` install the latest version of `damo`? -------------------------------------------------------------- It will install the latest _stable_ version of `damo`. If you want, you can also install less stable but more fresh `damo` from source code. For that, fetch the `next` branch of the source tree and use `damo` executable file in the tree. $ git clone https://github.com/awslabs/damo -b next How can I participate in the development of `damo`? --------------------------------------------------- Please refer to [CONTRIBUTING](https://github.com/awslabs/damo/blob/next/CONTRIBUTING) file. Why some subcommands are not documented on [USAGE.md](USAGE.md) file? --------------------------------------------------------------------- Only sufficiently stabilized features are documented there. In other words, any feature that not documented on [USAGE.md](USAGE.md) are in experimental stage. Such experimental features could be deprecated and removed without any notice and grace periods. The documented features could also be deprecated, but those will provide some notification and grace periods. Snapshot Data Access Pattern ============================ Below commands repeatedly get a snapshot of the access pattern of a program for every second. $ git clone https://github.com/sjp38/masim $ cd masim; make; ./masim ./configs/zigzag.cfg --silent & $ sudo damo start --target_pid $(pidof masim) $ while :; do sudo damo show; sleep 1; done The first two lines of the commands get an artificial memory access generator program and run it in the background. It will repeatedly access two 100 MiB-sized memory regions one by one. You can substitute this with your real workload. The third line asks ``damo`` to start monitoring the access pattern of the process. Finally, the last line retries a snapshot of the monitoring results every second and show on terminal. Recording Data Access Patterns ============================== Below commands record memory access patterns of a program and save the monitoring results in `damon.data` file. $ git clone https://github.com/sjp38/masim $ cd masim; make; ./masim ./configs/zigzag.cfg --silent & $ sudo damo record -o damon.data $(pidof masim) The first two lines of the commands get an artificial memory access generator program and run it in the background. It will repeatedly access two 100 MiB-sized memory regions one by one. You can substitute this with your real workload. The last line asks ``damo`` to record the access pattern in ``damon.data`` file. Visualizing Recorded Patterns ============================= Below three commands visualize the recorded access patterns into three image files. $ damo report heats --heatmap stdout $ damo report wss --range 0 101 1 $ damo report wss --range 0 101 1 --sortby time --plot - ``access_pattern_heatmap.png`` will show the data access pattern in a heatmap, which shows when (x-axis) what memory region (y-axis) is how frequently accessed (color). - ``wss_dist.png`` will show the distribution of the working set size. - ``wss_chron_change.png`` will show how the working set size has chronologically changed. You can show the images on a web page [1]. Those made with other realistic workloads are also available [2,3,4]. [1] https://damonitor.github.io/doc/html/latest/admin-guide/mm/damon/start.html#visualizing-recorded-patterns
[2] https://damonitor.github.io/test/result/visual/latest/rec.heatmap.1.png.html
[3] https://damonitor.github.io/test/result/visual/latest/rec.wss_sz.png.html
[4] https://damonitor.github.io/test/result/visual/latest/rec.wss_time.png.html Data Access Pattern Aware Memory Management =========================================== Below command makes every memory region of size >=4K that hasn't accessed for >=60 seconds in your workload to be swapped out. By doing this, you can make your workload more memory efficient with only modest performance overhead. $ sudo damo start --damos_access_rate 0 0 --damos_sz_region 4K max \ --damos_age 60s max --damos_action pageout \ Deprecated, or Will be Deprecated Features ========================================== Below are features that recently deprecated, or will be deprecated. If you depend on any of those, please report your usecase to the community via github issue, sj@kernel.org, damon@lists.linux.dev, and/or linux-mm@kvack.org. `damo translate_damos` ---------------------- Deprecated. Use the command of v2.0.2 or lower version of DAMO instead. DAMON record binary format -------------------------- Deprecated. Use `json_compressed` format instead. At the beginning, DAMO used its special binary format, namely `record`. It is designed for lightweight saving of the monitoring results. It is difficult to read, and not that efficient compared to fancy compression techniques. `json` based monitoring results can be easier to read, and more efficient when compression technique is used. Hence, the format is deprecated. You may use `damo convert_record_format` of v2.0.2 or lower version of DAMO to convert your old record binary format monitoring results files to the new format. Python2 support --------------- Deprecated. Use Python3. For some old distros, DAMO initially supported Python2. Because Python2 is really old now, the support has deprecated. Please use Python3 or newer. DAMOS single line format ------------------------ Deprecated. Use `--damos_*` command line options or json format input. A simple DAMOS scheme specification format called one-line scheme specification was initially supported. Because it is not flexible for extension of features, it has deprecated now. You may use `--damos_*` command line options or json format instead. You could use `damo translate_damos` ov v2.0.2 or lower version of DAMO to convert your old single line DAMOS schemes specification to the new json format. --rbuf option of `damo record` ------------------------------ Deprecated. Early versions of DAMON supported in-kernel direct monitoring results record file generation. To control the overhead of it, DAMO allowed user to specify the size of buffer for the work. The feature has not merged into the mainline, and discarded. Hence the option was available for only few kernels that ported the feature. For most of kernels, tracepoint based record file generation is being used, and the overhead of the approach is subtle. Hence, the option has deprecated. damo-2.2.4/SECURITY.md000066400000000000000000000005361457141243000142350ustar00rootroot00000000000000Reporting a Vulnerability ------------------------- If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/) or directly via email to aws-security@amazon.com. Please do **not** create a public GitHub issue. damo-2.2.4/TODO000066400000000000000000000025771457141243000131430ustar00rootroot00000000000000- Test --numa_node 0 case - Support profiling - Support rss recording and reporting wss/rss - Support flamegraph-wss-heatmap visualization - Implement histogram stat regions output - Deprecate old features - Deprecate translate_damos (done.) - Deprecate single line per-scheme (done. code moved to _damon_deprecated) - Deprecate v1-v3 schemes (done. necessary code moved to _damon_deprecated) - Deprecate python2 support (done. code moved to _damon_deprecated) - Deprecate scheme subcommand - Can be replaced by start or tune - Deprecate old scheme usages in tests - Deprecate DAMON results record binary format (done. code moved to _damon_deprecated) - damo show features implementation - support target damos filter - time based filtering - collapse by time - collapse by priority value (more histogram control) - Implement min/max snapshot format keyword - Test quota goals (might make more sense to test from in-tree DAMON tests) - Handle quota goals tuning error / invalid inputs better - damo reclaim: add a command for basic investigation - helps ensure DAMON_RECLAIM is working as expected - helps reporting issue - Handle _damon_sysfs file i/o error for unsupported kernel features and conflicts with concurrent ones - Let DAMOS quota specified in more easy way - Optimize page management in 'replay' - Maybe we could manage it in 2 MiB granularity. damo-2.2.4/USAGE.md000066400000000000000000001032571457141243000136360ustar00rootroot00000000000000This document describes the detailed usage of `damo`. This doesn't cover all details of `damo` but only major features. This document may not complete and up to date sometimes. Please don't hesitate at asking questions and improvement of this document via GitHub [issues](https://github.com/awslabs/damo/issues) or [mails](https://lore.kernel.org/damon). Table of Contents ================= - [Prerequisites](#prerequisites) * [Kernel](#kernel) * [Sysfs or Debugfs](#sysfs-or-debugfs) * [Perf](#perf) * [Basic Concepts of DAMON](#basic-concepts-of-damon) - [Install](#install) - [Overview](#overview) - [DAMON Control (Access Monitoring and Monitoring-based System Optimization)](#damon-control--access-monitoring-and-monitoring-based-system-optimization-) * [`damo start`](#-damo-start-) + [Simple Target Argument](#simple-target-argument) + [Partial DAMON Parameters Update](#partial-damon-parameters-update) + [Partial DAMOS Parameters Update](#partial-damos-parameters-update) + [Full DAMON Parameters Update](#full-damon-parameters-update) + [Full DAMOS Parameters Update](#full-damos-parameters-update) * [`damo tune`](#-damo-tune-) * [`damo stop`](#-damo-stop-) - [Snapshot and Visualization of DAMON Monitoring Results and Running Status](#snapshot-and-visualization-of-damon-monitoring-results-and-running-status) * [`damo show`](#-damo-show-) + [DAMON Monitoring Results Structure](#damon-monitoring-results-structure) + [`damo`'s way of showing DAMON Monitoring Results](#-damo--s-way-of-showing-damon-monitoring-results) + [Customization of The Output](#customization-of-the-output) - [Region Visualization via Boxes](#region-visualization-via-boxes) - [Sorting and Filtering Regions Based on Access Pattern](#sorting-and-filtering-regions-based-on-access-pattern) * [`damo status`](#-damo-status-) - [For recording the access monitoring results and visualizing those](#for-recording-the-access-monitoring-results-and-visualizing-those) * [`damo record` (Recording Data Access Pattern)](#-damo-record---recording-data-access-pattern-) + [Recording Profile Information](#recording-profile-information) * [`damo report` (Visualizing Recorded Data Access Pattern)](#-damo-report---visualizing-recorded-data-access-pattern-) + [raw](#raw) + [heats](#heats) + [wss](#wss) + [profile](#profile) + [times](#times) * [`damo replay` (Replay Recorded Data Access Pattern)](#-damo-replay---replay-recorded-data-access-pattern-) - [Miscelleneous Helper Commands](#miscelleneous-helper-commands) * [`damo version`](#-damo-version-) * [`damo fmt_json`](#-damo-fmt-json-) Prerequisites ============= Kernel ------ You should first ensure your system is running on a kernel built with at least ``CONFIG_DAMON``, ``CONFIG_DAMON_VADDR``, ``CONFIG_DAMON_PADDR``, ``CONFIG_DAMON_SYSFS``, and ``CONFIG_DAMON_DBGFS``. Sysfs or Debugfs ---------------- Because `damo` is using the sysfs or debugfs interface of DAMON, you should ensure at least one of those is mounted. Note that DAMON debugfs interface is deprecated. Please use sysfs. If you depend on DAMON debugfs interface and cannot use sysfs interface, report your usecase to sj@kernel.org, damon@lists.linux.dev and linux-mm@kvack.org. Perf ---- `damo` uses `perf`[1] for recording DAMON's access monitoring results. Please ensure your system is having it if you will need to do record DAMON's monitoring resluts. If you will not do the recording, you don't need to install `perf` on your system, though. [1] https://perf.wiki.kernel.org/index.php/Main_Page Basic Concepts of DAMON ----------------------- `damo` is a user space tool for `DAMON`. Hence, for advanced and optimized use of `damo` rather than simple "Getting Started" tutorial, you should first understand the concepts of DAMON. There are a number of links to resources including DAMON introduction talks and publications at the project [site](https://damonitor.github.io). The official design [document](https://docs.kernel.org/mm/damon/design.html) is recommended among those, since we will try to keep it up to date always, and appropriate for DAMON users. Install ======= You can install `damo` via the official Python packages system, PyPi: $ sudo pip3 install damo Or, you can use your distribution's package manager if available. Refer to below [repology](https://repology.org/project/damo) data to show the packaging status of `damo` for each distribution. [![Packaging status](https://repology.org/badge/vertical-allrepos/damo.svg)](https://repology.org/project/damo/versions) If none of above options work for you, you can simply download the source code and use `damo` file at the root of the source tree. Optionally, you could add the path to the source code directory to your `$PATH`. Overview ======== `damo` provides a subcommands-based interface. You can show the list of the available commands and brief descripton of those via `damo --help`. The major commands can be categorized as below: - For controlling DAMON (monitoring and monitoring-based system optimization) - `start`, `tune`, and `stop` are included - For snapshot and visualization of DAMON's monitoring results and running status - `show`, and `status` are included - For recording the access monitoring results and utilizing the records - `record`, `report` and `replay` are included - For more convenient use of `damo` - `version` and `fmt_json` are included Every subcommand also provides `--help` option, which shows the basic usage of it. Below sections introduce more details about the major subcommands. Note that some of the subcommands that not described in this document would be in experimental stage, or not assumed to be used in major use cases. Those could be deprecated and removed without any notice and grace periods. DAMON Control (Access Monitoring and Monitoring-based System Optimization) ========================================================================== The main purposes of `damo` is operating DAMON, as the name says (DAMO: Data Access Monitor Operator). In other words, `damo` is for helping control of DAMON and retrieval/interpretation of the results. `damo start` ------------ `damo start` starts DAMON as users request. Specifically, users can specify how and to what address spaces DAMON should do monitor accesses, and what access monitoring-based system optimization to do. The request can be made via several command line options of the command. You can get the full list of the options via `damo start --help`. The command exits immediately after starting DAMON as requested. It exits with exit value `0` if it successfully started DAMON. Otherwise, the exit value will be non-zero. ### Simple Target Argument The command recieves one positional argument called deducible target. It could be used for specifying monitoring target, or full DAMON parameters in a json format. The command will try to deduce the type of the argument value and use it. With the argument, users can specify the monitoring target with 1) the command for execution of the monitoring target process, 2) pid of running target process, or 3) the special keyword, `paddr`, if you want to monitor the system's physical memory address space. Below example shows a command target usage: # damo start "sleep 5" The command will execute ``sleep 5`` by itself and start monitoring the data access patterns of the process. Note that the command requires root permission, and hence executes the monitoring target command as a root. This means that the user could execute arbitrary commands with root permission. Hence, sysadmins should allow only trusted users to use ``damo``. Below example shows a pid target usage: # sleep 5 & # damo start $(pidof sleep) Finally, below example shows the use of the special keyword, `paddr`: # damo start paddr In this case, the monitoring target regions defaults to the largest 'System RAM' region specified in `/proc/iomem` file. Note that the initial monitoring target region is maintained rather than dynamically updated like the virtual memory address spaces monitoring case. Users can specify full DAMON parameters at once in json format, by passing the json string or a path to a file containing the json string. Refer to "Full DAMON Parameters Update" section below for the detail of the concept, and "`damo fmt_json`" section below for the format of the json input. ### Partial DAMON Parameters Update The command line options basically support specification of partial DAMON parameters such as monitoring intervals and DAMOS action. With a good understanding of DAMON's core concepts, understanding what each of such options mean with their brief description on the help message wouldn't be that difficult. Note that these command line options support only single kdamond, single DAMON context, and single monitoring target case at the moment. Users can make requests without such limitation using json format input. Refer to 'Full DAMON Parameters Update' section below for the detail. ### Partial DAMOS Parameters Update Command line options having prefix of `--damos_` are for DAMON-based operation schemes. Those options are allowed to be specified multiple times for requesting multiple schemes. For example, below shows how you can start DAMON with two DAMOS schemes, one for proactive LRU-prioritization of hot pages and the other one for proactive LRU-deprioritization of cold pages. # damo start \ --damos_action lru_prio --damos_access_rate 50% max --damos_age 5s max \ --damos_action lru_deprio --damos_access_rate 0% 0% --damos_age 5s max This command will ask DAMON to find memory regions that showing >=50% access rate for >=5 seconds and prioritize the pages of the regions on the Linux kernel's LRU lists, while finding memory regions that not accessed for >=5 seconds and deprioritizes the pages of the regions from the LRU lists. ### Full DAMON Parameters Update As mentioned above, the partial DAMON parameters update command line options support only single kdamond and single DAMON context. That should be enough for many use cases, but for system-wide dynamic DAMON usages, that could be restrictive. Also, specifying each parameter that different from their default values could be not convenient. Users may want to specify full parameters at once in such cases. For such users, the command supports `--kdamonds` option. It receives a json-format specification of kdamonds that would contains all DAMON parameters. Then, `damo` starts DAMON with the specification. For the format of the json input, please refer to `damo fmt_json` documentation below, or simply try the command. The `--kdamonds` option keyword can also simply omitted because the json input can used as is for the `deducible target` (refer to "Simple Target Argument" section above). Note that multiple DAMON contexts per kdamond is not supported as of 2023-09-12, though. ### Full DAMOS Parameters Update The Partial DAMOS parameters update options support multiple schemes as abovely mentioned. However, it could be still too manual in some cases and users may want to provide all inputs at once. For such cases, `--schemes` option receives a json-format specification of DAMOS schemes. The format is same to schemes part of the `--kdamonds` input. You could get some example json format input for `--schemes` option from any `.json` files in `damon-tests` [repo](https://github.com/awslabs/damon-tests/tree/next/perf/schemes). `damo tune` ----------- `damo tune` updates the DAMON parameters while DAMON is running. It provides the set of command line options that same to that of `damo start`. Note that users should provide the full request specification to this command. If only a partial parameters are specified via the command line options of this command, unspecified parameters of running DAMON will be updated to their default values. The command exits immediately after updating DAMON parameters as requested. It exits with exit value `0` if the update successed. Otherwise, the exit value will be non-zero. `damo stop` ----------- `damo stop` stops the running DAMON. The command exits immediately after stopping DAMON. It exits with exit value `0` if it successfully terminated DAMON. Otherwise, the exit value will be non-zero. Snapshot and Visualization of DAMON Monitoring Results and Running Status ========================================================================= `damo show` ----------- `damo show` takes a snapshot of running DAMON's monitoring results and show it. For example: # damo start # damo show 0 addr [4.000 GiB , 16.245 GiB ) (12.245 GiB ) access 0 % age 7 m 32.100 s 1 addr [16.245 GiB , 28.529 GiB ) (12.284 GiB ) access 0 % age 12 m 40.500 s 2 addr [28.529 GiB , 40.800 GiB ) (12.271 GiB ) access 0 % age 15 m 10.100 s 3 addr [40.800 GiB , 52.866 GiB ) (12.066 GiB ) access 0 % age 15 m 58.600 s 4 addr [52.866 GiB , 65.121 GiB ) (12.255 GiB ) access 0 % age 16 m 15.900 s 5 addr [65.121 GiB , 77.312 GiB ) (12.191 GiB ) access 0 % age 16 m 22.400 s 6 addr [77.312 GiB , 89.537 GiB ) (12.225 GiB ) access 0 % age 16 m 24.200 s 7 addr [89.537 GiB , 101.824 GiB) (12.287 GiB ) access 0 % age 16 m 25 s 8 addr [101.824 GiB , 126.938 GiB) (25.114 GiB ) access 0 % age 16 m 25.300 s total size: 122.938 GiB ### DAMON Monitoring Results Structure The biggest unit of the monitoring result is called 'record'. Each record contains monitoring results snapshot that retrieved for each kdamond/context/target combination. Hence, the number of records that `damo show` will show depends on how many kdamond/context/target combination exists. Each record contains multiple snapshots of the monitoring results that retrieved for each `aggregation interval`. For `damo show`, therefore, each record will contain only one single snapshot. Each snapshot contains regions information. Each region information contains the monitoring results for the region including the start and end addresses of the memory region, `nr_accesses`, and `age`. The number of regions per snapshot would depend on the `min_nr_regions` and `max_nr_regions` DAMON parameters, and actual data access pattern of the monitoring target address space. ### `damo`'s way of showing DAMON Monitoring Results `damo show` shows the information in an enclosed hierarchical way like below: [...] [...] [...] That is, information of record and snapshot can be shown twice, once at the beginning (before showing it's internal data), and once at the end. Meanwhile, the information of regions can be shown only once since it is the lowest level that not encloses anything. By default, record and snapshot head/tail are skipped if there is only one record and one snapshot. That's why above `damo show` example output shows only regions information. ### Customization of The Output Users can customize what information to be shown in which way for the each position using `--format_{record,snapshot,region}[_{head,tail}]` option. Each of the option receives a string for the template. The template can have any words and special format keywords for each position. For example, ``, ``, ``, or `` keywords are available for `--foramt_region` option's value. The template can also have arbitrary strings. The newline character (`\n`) is also supported. Each of the keywords for each position and their brief description can be shown via `--ls_{record,snapshot,region}_format_keywords` option. Actually, `damo show` also internally uses the customization feature with its default templates. For example: # damo start # damo show --format_region "region that starts from and ends at was having access rate for ." region that starts from 4.000 GiB and ends at 16.251 GiB was having 0 % access rate for 40.700 s. region that starts from 16.251 GiB and ends at 126.938 GiB was having 0 % access rate for 47.300 s. total size: 122.938 GiB #### Region Visualization via Boxes For region information customization, a special keyword called `` is provided. It represents each region's access pattern with its shape and color. By default it represents each region's relative age, access rate (`nr_accesses`), and size with its length, color, and height, respectively. That is, `damo show --format_region ""` shows visualization of the access pattern, by showing location of each region in Y-axis, the hotness with color of each box, and how long the hotness has continued in X-axis. Showing only the first column of the output would be somewhat similar to an access heatmap of the target address space. For convenient use of it with a default format, `damo show` provides `--region_box` option. Output of the command with the option would help users better to understand. Users can further customize the box using `damo show` options that having `--region_box_` prefix. For example, users can set what access information to be represented by the length, color, and height, and whether the values should be represented in logscale or linearscale. #### Sorting and Filtering Regions Based on Access Pattern By default, `damo show` shows all regions that sorted by their start address. Different users would have different interest to regions having specific access pattern. Someone would be interested in hot and small regions, while some others are interested in cold and big regions. For such cases, users can make it to sort regions with specific access pattern values as keys including `access_rate`, `age`, and `size` via `--sort_regions_by` option. `--sort_regions_dsc` option can be used to do desscending order sorting. Further, users can make `damo show` to show only regions of specific access pattern and address ranges using options including `--sz_region`, `--access_rate`, `--age`, and `--address`. Note that the filtering could reduce DAMON's overhead, and therefore recommended to be used if you don't need full results and your system is sensitive to any resource waste. `damo status` ------------- `damo status` shows the status of DAMON. It shows every kdamond with the parameters that applied to it, running status (`on` or `off`), and DAMOS schemes status including their statistics and detailed applied regions information. Note that users can use `--json` to represent the status in a json format. And the json format output can again be used for `--kdamonds` or the positional option of some DAMON control commands including `damo start` and `damo tune`. The command exits immediately after showing the current status. It exits with exit value `0` if it successfully retrieved and shown the status of DAMON. Otherwise, the exit value will be non-zero. For recording the access monitoring results and visualizing those ================================================================= `damo show` shows only a snapshot. Since it contains the `age` of each region, it can be useful enough for online profiling or debugging. For detailed offline profiling or debugging, though, recording every changing monitoring results and analyzing the record could be more helpful. In this case, the `record` would same to that for `damo show`, but simply contains multiple `snapshot`s. `damo record` (Recording Data Access Pattern) --------------------------------------------- `damo record` records the data access pattern of target workloads in a file (`./damon.data` by default). The path to the file can be set with `--out` option. The command requires root permission. The output file will be owned by `root` and have `600` permission by default, so only root can read it. Users can change the permission via `--output_permission` option. Other than the two options, `damo record` receives command line options that same to those for `damo start` and `damo tune`. If DAMON is already running, users can simply record the monitoring results of the running DAMON by providing no DAMON parameter options. For example, below will start DAMON for physical address space monitoring, record the monitoring results, and save the records in `damon.data` file. # damo start # damo record Or, users can ask `damo record` to start DAMON by themselves, together with the monitoring target command, like below: # damo record "sleep 5" or, for already running process, like below: # damo record $(pidof my_workload) ### Recording Profile Information Note: This feature is an experimental one. Some changes could be made, or the support can be dropped in future. Users can record profiling information of the system together with the access pattern by adding `--profile` command line option to the `damo record` command. Internally, it runs `perf record` while `damo record` is running, and store the `perf` output as a file of name same to the access pattern record file (specified by `--out` option of `damo record`) except having `.profile` suffix. Hence, `damon.data.profile` is the default name of the profile information. Because the profile information record file is simply `perf record` output, users can further analyze the profile information using `perf` or any `perf record` output compatible tools. `damo report` (Visualizing Recorded Data Access Pattern) -------------------------------------------------------- `damo report` reads a data access pattern record file (if not explicitly specified using ``-i`` option, reads ``./damon.data`` file by default) and generates human-readable reports. Users can specify what type of report they want using a sub-subcommand to `damo report`. `raw`, `heats`, and `wss` report types are supported. ### raw `raw` sub-subcommand directly transforms the binary record into a human-readable text. For example: $ damo report raw base_time_absolute: 8 m 59.809 s monitoring_start: 0 ns monitoring_end: 104.599 ms monitoring_duration: 104.599 ms target_id: 18446623438842320000 nr_regions: 3 563ebaa00000-563ebc99e000( 31.617 MiB): 1 7f938d7e1000-7f938ddfc000( 6.105 MiB): 0 7fff66b0a000-7fff66bb2000( 672.000 KiB): 0 monitoring_start: 104.599 ms monitoring_end: 208.590 ms monitoring_duration: 103.991 ms target_id: 18446623438842320000 nr_regions: 4 563ebaa00000-563ebc99e000( 31.617 MiB): 1 7f938d7e1000-7f938d9b5000( 1.828 MiB): 0 7f938d9b5000-7f938ddfc000( 4.277 MiB): 0 7fff66b0a000-7fff66bb2000( 672.000 KiB): 5 The first line shows the recording started timestamp. Records of data access patterns follow. Each record is separated by a blank line. Each record first specifies when the record started (`monitoring_start`) and ended (`monitoring_end`) relative to the start time, the duration for the recording (`monitoring_duration`). Recorded data access patterns of each target follow. Each data access pattern for each task shows the target's id (``target_id``) and a number of monitored address regions in this access pattern (``nr_regions``) first. After that, each line shows the start/end address, size, and the number of observed accesses of each region. ### heats The `raw` output is very detailed but hard to manually read. `heats` sub-subcommand plots the data in 3-dimensional form, which represents the time in x-axis, address of regions in y-axis, and the access frequency in z-axis. Users can optionally set the resolution of the map (`--resol`) and start/end point of each axis (`--time_range` and `--address_range`). For example: # damo report heats --resol 3 3 0 0 0.0 0 7609002 0.0 0 15218004 0.0 66112620851 0 0.0 66112620851 7609002 0.0 66112620851 15218004 0.0 132225241702 0 0.0 132225241702 7609002 0.0 132225241702 15218004 0.0 This command shows a recorded access pattern in a heatmap of 3x3 resolution. Therefore it shows 9 data points in total. Each line shows each of the data points. The three numbers in each line represent time in nanoseconds, address in bytes and the observed access frequency. Users can convert this text output into a heatmap image (represents z-axis values with colors) or other 3D representations using various tools such as `gnuplot`. For more convenience, `heats` sub-subcommand provides the `gnuplot` based heatmap image creation. For this, `--heatmap` option can be used. Also, note that because it uses `gnuplot` internally, it will fail if `gnuplot` is not installed on your system. For example: $ ./damo report heats --heatmap heatmap.png Creates the heatmap image in ``heatmap.png`` file. It supports ``pdf``, ``png``, ``jpeg``, and ``svg``. If the target address space is a virtual memory address space and the user plots the entire address space, the huge unmapped regions will make the picture looks only black. Therefore the user should do proper zoom in / zoom out using the resolution and axis boundary-setting arguments. To make this effort minimal, `--guide` option can be used as below: $ ./damo report heats --guide target_id:18446623438842320000 time: 539914032967-596606618651 (56.693 s) region 0: 00000094827419009024-00000094827452162048 (31.617 MiB) region 1: 00000140271510761472-00000140271717171200 (196.848 MiB) region 2: 00000140734916239360-00000140734916927488 (672.000 KiB) The output shows unions of monitored regions (start and end addresses in byte) and the union of monitored time duration (start and end time in nanoseconds) of each target task. Therefore, it would be wise to plot the data points in each union. If no axis boundary option is given, it will automatically find the biggest union in ``--guide`` output and set the boundary in it. ### wss The `wss` type extracts the distribution and chronological working set size changes from the record. By default, the working set is defined as memory regions shown any access within each snapshot. Hence, for example, if a record is having N snapshots, the record is having N working set size values, and `wss` report type shows the distribution of the N values in size order, or chronological order. For example: $ ./damo report wss # # target_id 18446623438842320000 # avr: 107.767 MiB 0 0 B | | 25 95.387 MiB |**************************** | 50 95.391 MiB |**************************** | 75 95.414 MiB |**************************** | 100 196.871 MiB |***********************************************************| Without any option, it shows the distribution of the working set sizes as above. It shows 0th, 25th, 50th, 75th, and 100th percentile and the average of the measured working set sizes in the access pattern records. In this case, the working set size was 95.387 MiB for 25th to 75th percentile but 196.871 MiB in max and 107.767 MiB on average. By setting the sort key of the percentile using `--sortby`, you can show how the working set size has chronologically changed. For example: $ ./damo report wss --sortby time # # target_id 18446623438842320000 # avr: 107.767 MiB 0 0 B | | 25 95.418 MiB |***************************** | 50 190.766 MiB |***********************************************************| 75 95.391 MiB |***************************** | 100 95.395 MiB |***************************** | The average is still 107.767 MiB, of course. And, because the access was spiked in very short duration and this command plots only 4 data points, we cannot show when the access spikes made. Users can specify the resolution of the distribution (``--range``). By giving more fine resolution, the short duration spikes could be more easily found. Similar to that of ``heats --heatmap``, it also supports `gnuplot` based simple visualization of the distribution via ``--plot`` option. ### profile Note: This feature is an experimental one. Some changes could be made, or the support can be dropped in future. The `profile` type shows profiling report for specific access pattern. It requires two files, namely an access pattern record file and a profiling information record file that recorded together with the access pattern record file. Those can be generated by `damo record` with `--profile` option (Refer to 'Recording Profile Information' [section](#recording-profile-information) for details). Users can further describe access pattern of their interest that they want to know what happens when the access pattern occurs. Then, `damo report profile` command read the access pattern, find times when the specific access pattern happened, collect profiling information for the time ranges, and generate the report with the filtered information. For example, below shows what was consuming CPU while 50% or more rate of access was made towards 50 MiB size address range starting from `139,798,348,038,144`, and the total size of the memory regions that got the access in the address range was 40 or more MiB. $ sudo ./damo report profile --access_rate 50% 100% \ --address 139798348038144 $((139798348038144 + 50 * 1024 * 1024)) \ --sz_snapshot 40MiB max Samples: 69K of event 'cpu-clock:pppH', Event count (approx.): 17449500000 Overhead Command Shared Object Symbol 70.32% swapper [kernel.vmlinux] [k] pv_native_safe_halt 28.83% masim masim [.] do_seq_wo 0.03% masim [kernel.vmlinux] [k] _raw_spin_unlock_irqrestore 0.03% ps [kernel.vmlinux] [k] do_syscall_64 0.03% swapper [kernel.vmlinux] [k] __do_softirq ### times Note: This feature is an experimental one. Some changes could be made, or the support can be dropped in future. The `times` type shows time intervals in an access pattern record that showing specific access pattern. This can be useful when user runs `damo` together with other tools such as profilers. For example, below shows when there was no access to 50 MiB size address range starting from `139,798,348,038,144`. $ sudo ./damo report times --access_rate 0% 0% \ --address 139798348038144 $((139798348038144 + 50 * 1024 * 1024)) 93904.291408-93904.393156 93905.919058-93910.903176 93915.994039-93920.876248 93926.049032-93930.918094 93935.988105-93940.956402 93946.027539-93950.997432 93956.067597-93961.036500 93966.101779-93966.910657 `damo replay` (Replay Recorded Data Access Pattern) --------------------------------------------------- Note: This feature is an experimental one. Some changes could be made, or the support can be dropped in future. `damo replay` receives a `damo record`-generated data access pattern record file that specified via command line argument (`./damon.data` by default). Then, the command reproduces the recorded accesses in the file by making articial memory accesses. This could be useful for some types of system analysis or experiments with real-world memory access pattern. Note that current implementation of `damo replay` runs on Python with single thread. Hence it might not performant enough to saturate full memory bandwidth of the system. If the record is made by workloads and/or systems that utilize memory bandwidth more than 'damo replay' and/or replaying systems could, and as the difference of the performance is big, the replayed accesses would be less similar to the original one. To show the real memory access performance of `damo replay` on specific system, users could use `--test_perf` option. Miscelleneous Helper Commands ============================= Abovely explained commands are all core functions of `damo`. For more convenient use of `damo` and debugging of DAMON or `damo` itself, `damo` supports more commands. This section explains some of those that could be useful for some cases. `damo version` -------------- `damo version` shows the version of the installed `damo`. The version number is constructed with three numbers. `damo` is doing chronological release (about once per week), so the version number means nothing but the relative time of the release. Later one would have more features and bug fixes. `damo fmt_json` --------------- As mentioned for `damo start` above, DAMON control commands including `start`, `tune`, and additionally `record` allows passing DAMON parameters or DAMOS specification all at once via a json format. That's for making specifying and managing complex requests easier, but writing the whole json manually could be annoying, while the partial DAMON/DAMOS parameters setup command line options are easy for simple use case. To help formatting the json input easier, `damo fmt_json` receives the partial DMAON/DAMOS parameters setup options and print out resulting json format Kdamond parameters. For example, # damo fmt_json --damos_action stat prints json format DAMON parameters specification that will be result in a DAMON configuration that same to one that can be made with `damo start --damos_action stat`. In other words, `damo start $(damo fmt_json --damos_action stat)` will be same to `damo start --damos_action stat`. Note that starting DAMON with the partial DAMON parameter command line option and then getting the DAMON parameters in the json format using `damo status` could also be one way for easily starting write of the json format specification. damo-2.2.4/_damo_ascii_color.py000066400000000000000000000026421457141243000164430ustar00rootroot00000000000000# SPDX-License-Identifier: GPL-2.0 # {name: [[background colors], [foreground colors]]} colorsets = { 'gray':[ [232] * 10, [237, 239, 241, 243, 245, 247, 249, 251, 253, 255]], 'flame':[ [232, 1, 1, 2, 3, 3, 20, 21,26, 27, 27], [239, 235, 237, 239, 243, 245, 247, 249, 251, 255]], 'emotion':[ [232, 234, 20, 21, 26, 2, 3, 1, 1, 1], [239, 235, 237, 239, 243, 245, 247, 249, 251, 255]], } def max_color_level(): return len(colorsets['gray'][0]) - 1 def color_mode_start_txt(colorset_name, level): if not colorset_name in colorsets: raise Exception('wrong colorset (%s)' % colorset) colorset = colorsets[colorset_name] bg = colorset[0][level] fg = colorset[1][level] return u'\u001b[48;5;%dm\u001b[38;5;%dm' % (bg, fg) def color_mode_end_txt(): return u'\u001b[0m' def colored(txt, colorset_name, level): return ''.join([color_mode_start_txt(colorset_name, level), txt, color_mode_end_txt()]) def color_samples(colorset_name): samples = [] for level in range(max_color_level() + 1): samples.append('%s%s' % (color_mode_start_txt(colorset_name, level), '%d' % level)) samples.append(color_mode_end_txt()) return ''.join(samples) def main(): for colorset_name in colorsets: print('colorset_name') print(color_samples(colorset_name)) if __name__ == '__main__': main() damo-2.2.4/_damo_deprecated.py000066400000000000000000000162751457141243000162640ustar00rootroot00000000000000# SPDX-License-Identifier: GPL-2.0 ''' Keep code for deprecated features, which still need to help old users migrate, e.g., 'translate_damos' and 'convert_record_format'. ''' import json import os import subprocess import sys import _damo_deprecation_notice import _damo_fmt_str import _damon ''' Python2 support ''' if sys.version.startswith('2.'): _damo_deprecation_notice.deprecated(feature='Python2 support of damo', deadline='2023-Q2') # For supporting python 2.6 try: subprocess.DEVNULL = subprocess.DEVNULL except AttributeError: subprocess.DEVNULL = open(os.devnull, 'wb') try: subprocess.check_output = subprocess.check_output except AttributeError: def check_output(*popenargs, **kwargs): process = subprocess.Popen(stdout=subprocess.PIPE, *popenargs, **kwargs) output, err = process.communicate() rc = process.poll() if rc: raise subprocess.CalledProcessError(rc, popenargs[0]) return output subprocess.check_output = check_output ''' DAMOS single-line scheme specification input. Change human readable data access monitoring-based operation schemes input for 'damo' to a '_damon.Damos' object. This format has inspired by DAMON debugfs 'schemes' file input/output. It was enough to be used for the initial version, but later extending it made it to receive more than 15 fields, and became hard to understand and maintain. Hence, replaced by more intuitive command line options and json format. Below are examples of the single-line scheme input. # format is: # # # B/K/M/G/T for Bytes/KiB/MiB/GiB/TiB # us/ms/s/m/h/d for micro-seconds/milli-seconds/seconds/minutes/hours/days # 'min/max' for possible min/max value. # if a region keeps a high access frequency for >=100ms, put the region on # the head of the LRU list (call madvise() with MADV_WILLNEED). min max 80 max 100ms max willneed # if a region keeps a low access frequency for >=200ms and <=one hour, put # the region on the tail of the LRU list (call madvise() with MADV_COLD). min max 10 20 200ms 1h cold # if a region keeps a very low access frequency for >=60 seconds, swap out # the region immediately (call madvise() with MADV_PAGEOUT). min max 0 10 60s max pageout # if a region of a size >=2MiB keeps a very high access frequency for # >=100ms, let the region to use huge pages (call madvise() with # MADV_HUGEPAGE). 2M max 90 100 100ms max hugepage # If a regions of a size >=2MiB keeps small access frequency for >=100ms, # avoid the region using huge pages (call madvise() with MADV_NOHUGEPAGE). 2M max 0 25 100ms max nohugepage ''' def fields_to_v0_scheme(fields): scheme = _damon.Damos() scheme.access_pattern = _damon.DamosAccessPattern( sz_bytes = [_damo_fmt_str.text_to_bytes(fields[0]), _damo_fmt_str.text_to_bytes(fields[1])], nr_accesses = [_damo_fmt_str.text_to_percent(fields[2]), _damo_fmt_str.text_to_percent(fields[3])], nr_accesses_unit = _damon.unit_percent, age = [_damo_fmt_str.text_to_us(fields[4]), _damo_fmt_str.text_to_us(fields[5])], age_unit = _damon.unit_usec) scheme.action = fields[6].lower() return scheme def fields_to_v1_scheme(fields): scheme = fields_to_v0_scheme(fields) scheme.quotas.sz_bytes = _damo_fmt_str.text_to_bytes(fields[7]) scheme.quotas.reset_interval_ms = _damo_fmt_str.text_to_ms( fields[8]) return scheme def fields_to_v2_scheme(fields): scheme = fields_to_v1_scheme(fields) scheme.quotas.weight_sz_permil = int(fields[9]) scheme.quotas.weight_nr_accesses_permil = int(fields[10]) scheme.quotas.weight_age_permil = int(fields[11]) return scheme def fields_to_v3_scheme(fields): scheme = fields_to_v2_scheme(fields) scheme.watermarks.metric = fields[12].lower() scheme.watermarks.interval_us = _damo_fmt_str.text_to_us( fields[13]) scheme.watermarks.high_permil = int(fields[14]) scheme.watermarks.mid_permil = int(fields[15]) scheme.watermarks.low_permil = int(fields[16]) return scheme def fields_to_v4_scheme(fields): scheme = fields_to_v0_scheme(fields) scheme.quotas.time_ms = _damo_fmt_str.text_to_ms(fields[7]) scheme.quotas.sz_bytes = _damo_fmt_str.text_to_bytes(fields[8]) scheme.quotas.reset_interval_ms = _damo_fmt_str.text_to_ms( fields[9]) scheme.quotas.weight_sz_permil = int(fields[10]) scheme.quotas.weight_nr_accesses_permil = int(fields[11]) scheme.quotas.weight_age_permil = int(fields[12]) scheme.watermarks.metric = fields[13].lower() scheme.watermarks.interval_us = _damo_fmt_str.text_to_us( fields[14]) scheme.watermarks.high_permil = int(fields[15]) scheme.watermarks.mid_permil = int(fields[16]) scheme.watermarks.low_permil = int(fields[17]) return scheme avoid_crashing_single_line_scheme_for_testing = False avoid_crashing_v1_v3_schemes_for_testing = False def damo_single_line_scheme_to_damos(line): '''Returns Damos object and err''' _damo_deprecation_notice.deprecated( feature='single line scheme input', deadline='2023-Q2', do_exit=not avoid_crashing_single_line_scheme_for_testing, exit_code=1, additional_notice='Please use json format or --damo_* options') fields = line.split() # Remove below if someone depends on the v1-v3 DAMOS input is found. if len(fields) in [9, 12, 17]: _damo_deprecation_notice.deprecated( feature='9, 12, or 17 fields single line scheme input', do_exit=not avoid_crashing_v1_v3_schemes_for_testing, exit_code=1, deadline='2023-Q2') try: if len(fields) == 7: return fields_to_v0_scheme(fields), None elif len(fields) == 9: return fields_to_v1_scheme(fields), None elif len(fields) == 12: return fields_to_v2_scheme(fields), None elif len(fields) == 17: return fields_to_v3_scheme(fields), None elif len(fields) == 18: return fields_to_v4_scheme(fields), None else: return None, 'expected %s fields, but \'%s\'' % ( [7, 9, 12, 17, 18], line) except: return None, 'wrong input field' return None, 'unsupported version of single line scheme' def damo_single_line_schemes_to_damos(schemes): if os.path.isfile(schemes): with open(schemes, 'r') as f: schemes = f.read() # remove comments, empty lines, and unnecessary white spaces damo_schemes_lines = [l.strip() for l in schemes.strip().split('\n') if not l.strip().startswith('#') and l.strip() != ''] damos_list = [] for line in damo_schemes_lines: damos, err = damo_single_line_scheme_to_damos(line) if err != None: return None, 'invalid input: %s' % err damos.name = '%d' % len(damos_list) damos_list.append(damos) return damos_list, None damo-2.2.4/_damo_deprecation_notice.py000066400000000000000000000017031457141243000200100ustar00rootroot00000000000000# SPDX-License-Identifier: GPL-2.0 import sys def will_be_deprecated(feature, deadline, additional_notice=''): sys.stderr.write('\n'.join([ '', 'WARNING: %s will be deprecated by %s.' % (feature, deadline), ' %s' % additional_notice, ' Please report your usecase to Github issues[1], sj@kernel.org,', ' damon@lists.linux.dev and/or linux-mm@kvack.org if you depend on those.', '', ' [1] https://github.com/awslabs/damo/issues', '', ''])) def deprecated(feature, deadline, do_exit=False, exit_code=1, additional_notice=''): sys.stderr.write('\n'.join([ '', 'WARNING: %s is deprecated.' % feature, ' The support will be removed by %s.' % deadline, ' %s' % additional_notice, ' Please report your usecase to Github issues[1], sj@kernel.org,', ' damon@lists.linux.dev and/or linux-mm@kvack.org if you depend on those.', '', ' [1] https://github.com/awslabs/damo/issues', '', ''])) if do_exit: exit(exit_code) damo-2.2.4/_damo_dist.py000066400000000000000000000011541457141243000151150ustar00rootroot00000000000000# SPDX-License-Identifier: GPL-2.0 import os import subprocess 'return error' def plot_dist(data_file, output_file, xlabel, ylabel): terminal = output_file.split('.')[-1] if not terminal in ['pdf', 'jpeg', 'png', 'svg']: os.remove(data_file) return 'Unsupported plot output type.' gnuplot_cmd = """ set term %s; set output '%s'; set key off; set xlabel '%s'; set ylabel '%s'; plot '%s' with linespoints;""" % (terminal, output_file, xlabel, ylabel, data_file) subprocess.call(['gnuplot', '-e', gnuplot_cmd]) os.remove(data_file) return None damo-2.2.4/_damo_fmt_str.py000066400000000000000000000236111457141243000156320ustar00rootroot00000000000000# SPDX-License-Identifier: GPL-2.0 import platform def format_nr(nr, machine_friendly): raw_string = '%d' % nr if machine_friendly: return raw_string fields = [] for i in range(0, len(raw_string), 3): start_idx = max(0, len(raw_string) - i - 3) end_idx = len(raw_string) - i fields = [raw_string[start_idx:end_idx]] + fields return ','.join(fields) def format_sz(sz_bytes, machine_friendly): if machine_friendly: return '%d' % sz_bytes sz_bytes = float(sz_bytes) if sz_bytes == float(ulong_max): return 'max' if sz_bytes > 1<<60: return '%.3f EiB' % (sz_bytes / (1<<60)) if sz_bytes > 1<<50: return '%.3f PiB' % (sz_bytes / (1<<50)) if sz_bytes > 1<<40: return '%.3f TiB' % (sz_bytes / (1<<40)) if sz_bytes > 1<<30: return '%.3f GiB' % (sz_bytes / (1<<30)) if sz_bytes > 1<<20: return '%.3f MiB' % (sz_bytes / (1<<20)) if sz_bytes > 1<<10: return '%.3f KiB' % (sz_bytes / (1<<10)) return '%d B' % sz_bytes def format_addr_range(start, end, machine_friendly): return '[%s, %s) (%s)' % ( format_nr(start, machine_friendly), format_nr(end, machine_friendly), format_sz(end - start, machine_friendly)) ns_ns = 1 us_ns = 1000 ms_ns = 1000 * us_ns sec_ns = 1000 * ms_ns minute_ns = 60 * sec_ns hour_ns = 60 * minute_ns day_ns = 24 * hour_ns nsecs_to_unit = {1: 'ns', us_ns: 'us', ms_ns: 'ms', sec_ns: 's', minute_ns: 'm', hour_ns: 'h', day_ns: 'd'} def format_time_ns_min_unit(time_ns, min_unit, machine_friendly): if machine_friendly: return '%d' % time_ns if time_ns >= ulong_max: return 'max' for unit_nsecs in sorted(nsecs_to_unit.keys(), reverse=True): if time_ns < unit_nsecs: continue if unit_nsecs == min_unit: if time_ns % unit_nsecs: return '%.3f %s' % (time_ns / unit_nsecs, nsecs_to_unit[unit_nsecs]) else: return '%d %s' % (time_ns / unit_nsecs, nsecs_to_unit[unit_nsecs]) unit_nr = int(time_ns / unit_nsecs) unit_str = '%s %s' % ( format_nr(unit_nr, False), nsecs_to_unit[unit_nsecs]) less_unit_ns = time_ns - unit_nr * unit_nsecs if less_unit_ns == 0: return unit_str else: return '%s %s' % (unit_str, format_time_ns_min_unit(less_unit_ns, min_unit, False)) return '0 ns' def format_time_ns_exact(time_ns, machine_friendly): return format_time_ns_min_unit(time_ns, ns_ns, machine_friendly) def format_time_us_exact(time_us, machine_friendly): return format_time_ns_exact(time_us * us_ns, machine_friendly) def format_time_ms_exact(time_ms, machine_friendly): return format_time_ns_exact(time_ms * ms_ns, machine_friendly) def format_time_ns(time_ns, machine_friendly): if machine_friendly: return '%d' % time_ns time_ns = float(time_ns) if time_ns >= ulong_max: return 'max' if time_ns >= hour_ns: hour = int(time_ns / hour_ns) hour_str = '%d h' % hour less_hour_ns = time_ns - (hour * hour_ns) if less_hour_ns == 0: return hour_str return '%s %s' % (hour_str, format_time_ns(less_hour_ns, False)) if time_ns >= minute_ns: if time_ns % minute_ns == 0: return '%d m' % (time_ns / minute_ns) if time_ns % sec_ns == 0: return '%d m %d s' % (time_ns / minute_ns, (time_ns % minute_ns) / sec_ns) return '%d m %.3f s' % (time_ns / minute_ns, (time_ns % minute_ns) / sec_ns) if time_ns >= sec_ns: if time_ns % sec_ns == 0: return '%d s' % (time_ns / sec_ns) return '%.3f s' % (time_ns / sec_ns) if time_ns >= ms_ns: if time_ns % ms_ns == 0: return '%d ms' % (time_ns / ms_ns) return '%.3f ms' % (time_ns / ms_ns) if time_ns >= us_ns: if time_ns % us_ns == 0: return '%d us' % (time_ns / us_ns) return '%.3f us' % (time_ns / us_ns) return '%d ns' % time_ns def format_time_us(time_us, machine_friendly): if machine_friendly: return '%d' % time_us return format_time_ns(time_us * 1000, machine_friendly) def format_time_ms(time_ms, machine_friendly): if machine_friendly: return '%d' % time_ms return format_time_ns(time_ms * 1000000, machine_friendly) def format_time_sec(time_sec, machine_friendly): if machine_friendly: return '%d' % time_sec return format_time_ns(time_sec * 1000000000, machine_friendly) def format_ratio(ratio, machine_friendly): if machine_friendly: return '%f' % ratio over_percent = int(ratio * 100) over_percent_str = format_nr(over_percent, machine_friendly) under_percent = ratio * 100 % 1 under_percent_str = ('%.7f' % under_percent).rstrip('0') # cut '0.' prefix under_percent_str = under_percent_str[2:] if under_percent_str == '': return '%s %%' % over_percent_str return '%s.%s %%' % (over_percent_str, under_percent_str) def format_percent(percent, machine_friendly): if machine_friendly: return '%f' % percent return format_ratio(float(percent) / 100, machine_friendly) def format_permil(permil, machine_friendly): if machine_friendly: return '%f' % permil return format_ratio(float(permil) / 1000, machine_friendly) def format_bp(bp, machine_friendly): if machine_friendly: return '%f' % bp return format_ratio(float(bp) / 10000, machine_friendly) def indent_lines(string, indent_width): return '\n'.join([' ' * indent_width + l for l in string.split('\n')]) number_types = [int, float] try: # for python2 number_types.append(long) except: pass uint_max = 2**32 - 1 ulong_max = 2**64 - 1 if platform.architecture()[0] != '64bit': ulong_max = 2**32 - 1 unit_to_bytes = {'B': 1, 'K': 1 << 10, 'KB': 1 << 10, 'KiB': 1 << 10, 'M': 1 << 20, 'MB': 1 << 20, 'MiB': 1 << 20, 'G': 1 << 30, 'GB': 1 << 30, 'GiB': 1 << 30, 'T': 1 << 40, 'TB': 1 << 40, 'TiB': 1 << 40, 'P': 1 << 50, 'PB': 1 << 50, 'PiB': 1 << 50, 'E': 1 << 60, 'EB': 1 << 60, 'EiB': 1 << 60} def text_to_nr(txt): if type(txt) in number_types: return txt new_txt = ''.join([c for c in txt if c != ',']) try: return int(new_txt) except: pass return float(new_txt) def try_common_input(txt, min_val=0, max_val=ulong_max): 'return success and number' if txt == 'min': return True, min_val if txt == 'max': return True, max_val try: return True, text_to_nr(txt) except: pass return False, None def text_to_bytes(txt): success, number = try_common_input(txt) if success: return number unit = None if len(txt) > 3: unit = txt[len(txt) - 3:] if unit in unit_to_bytes: number = text_to_nr(txt[:-3]) else: unit = None if unit == None: if len(txt) > 2: unit = txt[len(txt) - 2:] if unit in unit_to_bytes: number = text_to_nr(txt[:-2]) else: unit = None if unit == None: if txt[-1] in unit_to_bytes: unit = txt[-1] number = text_to_nr(txt[:-1]) else: unit = 'B' number = text_to_nr(txt) return min(ulong_max, int(number * unit_to_bytes[unit])) unit_to_nsecs = {'ns': ns_ns, 'us': us_ns, 'ms': ms_ns, 's': sec_ns, 'm': minute_ns, 'h': hour_ns, 'd': day_ns} def text_to_ns(txt): success, number = try_common_input(txt) if success: return number fields = txt.split() if len(fields) > 1: result_us = 0 for i in range(0, len(fields), 2): result_us += text_to_ns(''.join(fields[i: i + 2])) return result_us if not txt[-2:] in unit_to_nsecs and not txt[-1] in unit_to_nsecs: return float(txt) unit = txt[-2:] if unit in ['ns', 'us', 'ms']: number = text_to_nr(txt[:-2]) else: unit = txt[-1] number = text_to_nr(txt[:-1]) return number * unit_to_nsecs[unit] def text_to_us(txt): success, number = try_common_input(txt) if success: return number return text_to_ns(txt) / us_ns def text_to_ms(txt): success, number = try_common_input(txt) if success: return number return text_to_us(txt) / 1000 def text_to_sec(txt): success, number = try_common_input(txt) if success: return number return text_to_ms(txt) / 1000 def text_to_ratio(txt): success, number = try_common_input(txt, 0.0) if success: return number is_percent = False if txt[-1] == '%': is_percent = True txt = txt[:-1] ratio = text_to_nr(txt) if is_percent: ratio /= 100.0 return ratio def text_to_bp(txt): success, number = try_common_input(txt, 0) if success: return number return text_to_ratio(txt) * 10000 def text_to_permil(txt): success, number = try_common_input(txt, 0) if success: return number return text_to_ratio(txt) * 1000 def text_to_percent(txt): success, number = try_common_input(txt, 0) if success: return number return text_to_ratio(txt) * 100 def text_to_nr_unit(txt): fields = txt.split() if len(fields) != 2: raise Exception('text_to_nr_unit requires two fields') return text_to_nr(fields[0]), fields[1] def text_to_bool(txt): if type(txt) == bool: return txt true_txts = ['y', 'yes', 'true'] false_txts = ['n', 'no', 'false'] txt = txt.lower() if txt in true_txts: return True elif txt in false_txts: return False else: raise Exception('txt should be one of %s but %s' % (' '.join(true_txts + false_txts), txt)) damo-2.2.4/_damo_fs.py000066400000000000000000000063351457141243000145700ustar00rootroot00000000000000# SPDX-License-Identifier: GPL-2.0 import os import _damon '''Returns content and error''' def read_file(filepath): try: with open(filepath, 'r') as f: content = f.read() except Exception as e: return None, 'reading %s failed (%s)' % (filepath, e) if _damon.pr_debug_log: print('read \'%s\': \'%s\'' % (filepath, content.strip())) return content, None def read_files(root): contents = {} for filename in os.listdir(root): filepath = os.path.join(root, filename) if os.path.isdir(filepath): contents[filename] = read_files(filepath) else: contents[filename], err = read_file(filepath) if err != None: contents[filename] = 'read failed (%s)' % err return contents ''' Returns None if success error string otherwise ''' def write_file(filepath, content): if _damon.pr_debug_log: print('write \'%s\' to \'%s\'' % (content.strip(), filepath)) try: with open(filepath, 'w') as f: f.write(content) except Exception as e: return 'writing %s to %s failed (%s)' % (content.strip(), filepath, e) return None ''' operations can be either {path: content}, or [operations]. In the former case, this function writes content to path, for all path/content pairs in the dictionary. In the latter case, operations in the list is executed sequentially. If the path is for a file, content should be a string. If the path is for a directory, the content should be yet another operations. In the latter case, upper-level path is prefixed to paths of the lower-level operations paths. For example: { 'foo': 'bar', 'dirA': { 'fileA': '123', 'fileB': '456', }, [ {'fileA': '42'}, {'fileB': '4242'}, ] } writes 'bar' to 'foo', writes '123' to 'dirA/fileA', writes '456' to 'dirA/fileB', and writes '42' to 'fileA' then writes '4242' to 'fileB', in any order Return an error string if fails any write, or None otherwise. ''' def write_files(operations, root=''): if not type(operations) in [list, dict]: return ('write_files() received none-list, none-dict content: %s' % operations) if isinstance(operations, list): for o in operations: err = write_files(o, root) if err != None: return err return None for filename in operations: filepath = os.path.join(root, filename) content = operations[filename] if os.path.isfile(filepath): err = write_file(filepath, content) if err != None: return err elif os.path.isdir(filepath): err = write_files(content, filepath) if err != None: return err else: return 'filepath (%s) is neither dir nor file' % (filepath) return None def dev_mount_point(dev): '''Returns mount point of specific device. None if not mounted''' with open('/proc/mounts', 'r') as f: for line in f: dev_name, mount_point = line.split()[:2] if dev_name == dev: return mount_point return None damo-2.2.4/_damo_paddr_layout.py000066400000000000000000000126371457141243000166510ustar00rootroot00000000000000# SPDX-License-Identifier: GPL-2.0 import argparse import os import _damon class PaddrRange: start = None end = None nid = None state = None name = None def __init__(self, start, end, nid, state, name): self.start = start self.end = end self.nid = nid self.state = state self.name = name def __str__(self): return '%x-%x, nid %s, state %s, name %s' % (self.start, self.end, self.nid, self.state, self.name) class MemBlock: nid = None index = None state = None def __init__(self, nid, index, state): self.nid = nid self.index = index self.state = state def __str__(self): return '%d (%s)' % (self.index, self.state) def __repr__(self): return self.__str__() def readfile(file_path): with open(file_path, 'r') as f: return f.read() def collapse_ranges(ranges): ranges = sorted(ranges, key=lambda x: x.start) merged = [] for r in ranges: if not merged: merged.append(r) continue last = merged[-1] if last.end != r.start or last.nid != r.nid or last.state != r.state: merged.append(r) else: last.end = r.end return merged def memblocks_to_ranges(blocks, block_size): ranges = [] for b in blocks: ranges.append(PaddrRange(b.index * block_size, (b.index + 1) * block_size, b.nid, b.state, None)) return collapse_ranges(ranges) def memblock_ranges(): SYSFS='/sys/devices/system/node' sz_block = int(readfile('/sys/devices/system/memory/block_size_bytes'), 16) sys_nodes = [x for x in os.listdir(SYSFS) if x.startswith('node')] blocks = [] for sys_node in sys_nodes: nid = int(sys_node[4:]) sys_node_files = os.listdir(os.path.join(SYSFS, sys_node)) for f in sys_node_files: if not f.startswith('memory'): continue if f[6:] == '_failure': continue index = int(f[6:]) sys_state = os.path.join(SYSFS, sys_node, f, 'state') state = readfile(sys_state).strip() blocks.append(MemBlock(nid, index, state)) return memblocks_to_ranges(blocks, sz_block) def iomem_ranges(): ranges = [] with open('/proc/iomem', 'r') as f: # example of the line: '100000000-42b201fff : System RAM' for line in f: fields = line.split(':') if len(fields) < 2: continue name = ':'.join(fields[1:]).strip() addrs = fields[0].split('-') if len(addrs) != 2: continue start = int(addrs[0], 16) end = int(addrs[1], 16) + 1 ranges.append(PaddrRange(start, end, None, None, name)) return ranges def integrate(memblock_parsed, iomem_parsed): merged = [] for r in iomem_parsed: for r2 in memblock_parsed: if r2.start <= r.start and r.end <= r2.end: r.nid = r2.nid r.state = r2.state merged.append(r) elif r2.start <= r.start and r.start < r2.end and r2.end < r.end: sub = PaddrRange(r2.end, r.end, None, None, r.name) iomem_parsed.append(sub) r.end = r2.end r.nid = r2.nid r.state = r2.state merged.append(r) merged = sorted(merged, key=lambda x: x.start) return merged def paddr_ranges(): return integrate(memblock_ranges(), iomem_ranges()) def pr_ranges(ranges): print('#%12s %13s\tnode\tstate\tresource\tsize' % ('start', 'end')) for r in ranges: print('%13d %13d\t%s\t%s\t%s\t%d' % (r.start, r.end, r.nid, r.state, r.name, r.end - r.start)) def default_paddr_region(): "Largest System RAM region becomes the default" ret = [] with open('/proc/iomem', 'r') as f: # example of the line: '100000000-42b201fff : System RAM' for line in f: fields = line.split(':') if len(fields) != 2: continue name = fields[1].strip() if not name.startswith('System RAM'): continue addrs = fields[0].split('-') if len(addrs) != 2: continue start = int(addrs[0], 16) end = int(addrs[1], 16) sz_region = end - start if not ret or sz_region > (ret[1] - ret[0]): ret = [start, end] return ret def paddr_region_of(numa_node): if not os.path.isdir('/sys/devices/system/memory'): return None, '/sys/devices/system/memory not found. You may need CONFIG_MEMORY_HOTPLUG enabled.' regions = [] paddr_ranges_ = paddr_ranges() for r in paddr_ranges_: if r.nid == numa_node and r.name.startswith('System RAM'): regions.append([r.start, r.end]) return regions, None def main(): parser = argparse.ArgumentParser() parser.add_argument('--numa_node', type=int, metavar='', help='print ranges of this numa node only') args = parser.parse_args() _damon.ensure_root_permission() ranges = [] for r in paddr_ranges(): if args.numa_node and r.nid != args.numa_node: continue ranges.append(r) pr_ranges(ranges) print('largest system RAM region: %s' % default_paddr_region()) if __name__ == '__main__': main() damo-2.2.4/_damo_print.py000066400000000000000000000010121457141243000152770ustar00rootroot00000000000000# SPDX-License-Identifier: GPL-2.0 import os import subprocess import tempfile def pr_with_pager_if_needed(text): try: nr_terminal_lines = os.get_terminal_size().lines except: nr_terminal_lines = 50 if text.count('\n') <= nr_terminal_lines: print(text) return fd, tmp_path = tempfile.mkstemp(prefix='damo_show-') with open(tmp_path, 'w') as f: f.write(text) subprocess.call(['less', '--RAW-CONTROL-CHARS', '--no-init', tmp_path]) os.remove(tmp_path) damo-2.2.4/_damo_subcmds.py000066400000000000000000000014141457141243000156110ustar00rootroot00000000000000# SPDX-License-Identifier: GPL-2.0 import argparse class DamoSubCmdModule: set_argparser = lambda self, args: args main = lambda self, args: args def __init__(self, set_argparser, main): if set_argparser != None: self.set_argparser = set_argparser if main != None: self.main = main class DamoSubCmd: name = None msg = None module = None def __init__(self, name, module, msg): self.name = name self.module = module self.msg = msg def add_parser(self, subparsers): subparser = subparsers.add_parser(self.name, help=self.msg) subparser.description = self.msg self.module.set_argparser(subparser) def execute(self, args): self.module.main(args) damo-2.2.4/_damon.py000066400000000000000000001330521457141243000142530ustar00rootroot00000000000000# SPDX-License-Identifier: GPL-2.0 """ Contains core functions for DAMON control. """ import collections import copy import json import os import random import time import _damo_fmt_str # Core data structures class DamonIntervals: sample = None aggr = None ops_update = None def __init__(self, sample='5ms', aggr='100ms', ops_update='1s'): self.sample = _damo_fmt_str.text_to_us(sample) self.aggr = _damo_fmt_str.text_to_us(aggr) self.ops_update = _damo_fmt_str.text_to_us(ops_update) def to_str(self, raw): return 'sample %s, aggr %s, update %s' % ( _damo_fmt_str.format_time_us(self.sample, raw), _damo_fmt_str.format_time_us(self.aggr, raw), _damo_fmt_str.format_time_us(self.ops_update, raw)) def __str__(self): return self.to_str(False) def __eq__(self, other): return type(self) == type(other) and '%s' % self == '%s' % other @classmethod def from_kvpairs(cls, kvpairs): return DamonIntervals( kvpairs['sample_us'], kvpairs['aggr_us'], kvpairs['ops_update_us']) def to_kvpairs(self, raw=False): return collections.OrderedDict([ ('sample_us', _damo_fmt_str.format_time_us(self.sample, raw)), ('aggr_us', _damo_fmt_str.format_time_us(self.aggr, raw)), ('ops_update_us', _damo_fmt_str.format_time_us(self.ops_update, raw)), ]) class DamonNrRegionsRange: minimum = None maximum = None def __init__(self, min_=10, max_=1000): self.minimum = _damo_fmt_str.text_to_nr(min_) self.maximum = _damo_fmt_str.text_to_nr(max_) def to_str(self, raw): return '[%s, %s]' % ( _damo_fmt_str.format_nr(self.minimum, raw), _damo_fmt_str.format_nr(self.maximum, raw)) def __str__(self): return self.to_str(False) def __eq__(self, other): return type(self) == type(other) and '%s' % self == '%s' % other @classmethod def from_kvpairs(cls, kvpairs): return DamonNrRegionsRange(kvpairs['min'], kvpairs['max']) def to_kvpairs(self, raw=False): return collections.OrderedDict([ ('min', _damo_fmt_str.format_nr(self.minimum, raw)), ('max', _damo_fmt_str.format_nr(self.maximum, raw)), ]) unit_percent = 'percent' unit_samples = 'samples' unit_usec = 'usec' unit_aggr_intervals = 'aggr_intervals' class DamonNrAccesses: samples = None percent = None def __init__(self, val, unit): if val == None or unit == None: return if unit == unit_samples: self.samples = _damo_fmt_str.text_to_nr(val) elif unit == unit_percent: self.percent = _damo_fmt_str.text_to_percent(val) else: raise Exception('invalid DamonNrAccesses unit \'%s\'' % unit) def __eq__(self, other): return (type(self) == type(other) and ((self.samples != None and self.samples == other.samples) or (self.percent != None and self.percent == other.percent))) def add_unset_unit(self, intervals): if self.samples != None and self.percent != None: return max_val = intervals.aggr / intervals.sample if self.samples == None: self.samples = int(self.percent * max_val / 100) elif self.percent == None: self.percent = int(self.samples * 100.0 / max_val) def to_str(self, unit, raw): if unit == unit_percent: return '%s %%' % (_damo_fmt_str.format_nr(self.percent, raw)) elif unit == unit_samples: return '%s %s' % (_damo_fmt_str.format_nr(self.samples, raw), unit_samples) raise Exception('unsupported unit for NrAccesses (%s)' % unit) @classmethod def from_kvpairs(cls, kv): ret = DamonNrAccesses(None, None) if 'samples' in kv and kv['samples'] != None: ret.samples = _damo_fmt_str.text_to_nr(kv['samples']) if 'percent' in kv and kv['percent'] != None: ret.percent = _damo_fmt_str.text_to_percent(kv['percent']) return ret def to_kvpairs(self, raw=False): return collections.OrderedDict( [('samples', self.samples), ('percent', self.percent)]) class DamonAge: usec = None aggr_intervals = None def __init__(self, val, unit): if val == None and unit != None: self.unit = unit return if val == None and unit == None: return if unit == unit_usec: self.usec = _damo_fmt_str.text_to_us(val) elif unit == unit_aggr_intervals: self.aggr_intervals = _damo_fmt_str.text_to_nr(val) else: raise Exception('DamonAge unsupported unit (%s)' % unit) def __eq__(self, other): return (type(self) == type(other) and ((self.usec != None and self.usec == other.usec) or (self.aggr_intervals != None and self.aggr_intervals == other.aggr_intervals))) def add_unset_unit(self, intervals): if self.usec != None and self.aggr_intervals != None: return if self.usec == None: self.usec = self.aggr_intervals * intervals.aggr elif self.aggr_intervals == None: self.aggr_intervals = int(self.usec / intervals.aggr) def to_str(self, unit, raw): if unit == unit_usec: return _damo_fmt_str.format_time_us_exact(self.usec, raw) return '%s %s' % (_damo_fmt_str.format_nr(self.aggr_intervals, raw), unit_aggr_intervals) @classmethod def from_kvpairs(cls, kv): ret = DamonAge(None, None) if kv['usec'] != None: ret.usec = _damo_fmt_str.text_to_us(kv['usec']) if kv['aggr_intervals'] != None: ret.aggr_intervals = _damo_fmt_str.text_to_nr(kv['aggr_intervals']) return ret def to_kvpairs(self, raw=False): return collections.OrderedDict( [('usec', _damo_fmt_str.format_time_us_exact(self.usec, raw) if self.usec != None else None), ('aggr_intervals', _damo_fmt_str.format_nr(self.aggr_intervals, raw) if self.aggr_intervals != None else None)]) class DamonRegion: # [start, end) start = None end = None # nr_accesses and age could be None nr_accesses = None age = None scheme = None # non-None if tried region def __init__(self, start, end, nr_accesses=None, nr_accesses_unit=None, age=None, age_unit=None): self.start = _damo_fmt_str.text_to_bytes(start) self.end = _damo_fmt_str.text_to_bytes(end) if nr_accesses == None: return self.nr_accesses = DamonNrAccesses(nr_accesses, nr_accesses_unit) self.age = DamonAge(age, age_unit) def to_str(self, raw, intervals=None): if self.nr_accesses == None: return _damo_fmt_str.format_addr_range(self.start, self.end, raw) if intervals != None: self.nr_accesses.add_unset_unit(intervals) self.age.add_unset_unit(intervals) if raw == False and intervals != None: nr_accesses_unit = unit_percent age_unit = unit_usec else: nr_accesses_unit = unit_samples age_unit = unit_aggr_intervals return '%s: nr_accesses: %s, age: %s' % ( _damo_fmt_str.format_addr_range(self.start, self.end, raw), self.nr_accesses.to_str(nr_accesses_unit, raw), self.age.to_str(age_unit, raw)) def __str__(self): return self.to_str(False) def __eq__(self, other): if self.nr_accesses == None: return type(self) == type(other) and '%s' % self == '%s' % other # For aggregate_snapshots() support def __hash__(self): identification = '%s-%s' % (self.start, self.end) return hash(identification) @classmethod def from_kvpairs(cls, kvpairs): if not 'nr_accesses' in kvpairs: return DamonRegion(kvpairs['start'], kvpairs['end']) region = DamonRegion(kvpairs['start'], kvpairs['end']) region.nr_accesses = DamonNrAccesses.from_kvpairs( kvpairs['nr_accesses']) region.age = DamonAge.from_kvpairs(kvpairs['age']) return region def to_kvpairs(self, raw=False): if self.nr_accesses == None: return collections.OrderedDict([ ('start', _damo_fmt_str.format_nr(self.start, raw)), ('end', _damo_fmt_str.format_nr(self.end, raw))]) return collections.OrderedDict([ ('start', _damo_fmt_str.format_nr(self.start, raw)), ('end', _damo_fmt_str.format_nr(self.end, raw)), ('nr_accesses', self.nr_accesses.to_kvpairs(raw)), ('age', self.age.to_kvpairs(raw))]) def size(self): return self.end - self.start class DamonTarget: pid = None regions = None context = None def __init__(self, pid, regions): self.pid = pid self.regions = regions def to_str(self, raw): lines = ['pid: %s' % self.pid] for region in self.regions: lines.append('region %s' % region.to_str(raw)) return '\n'.join(lines) def __str__(self): return self.to_str(False) def __eq__(self, other): return type(self) == type(other) and '%s' % self == '%s' % other @classmethod def from_kvpairs(cls, kvpairs): regions = [DamonRegion.from_kvpairs(kvp) for kvp in kvpairs['regions']] return DamonTarget(kvpairs['pid'], regions) def to_kvpairs(self, raw=False): kvp = collections.OrderedDict() kvp['pid'] = self.pid kvp['regions'] = [r.to_kvpairs(raw) for r in self.regions] return kvp class DamosAccessPattern: sz_bytes = None nr_acc_min_max = None # [min/max DamonNrAccesses] nr_accesses_unit = None age_min_max = None # [min/max DamonAge] age_unit = None # every region by default, so that it can be used for monitoring def __init__(self, sz_bytes=['min', 'max'], nr_accesses=['min', 'max'], nr_accesses_unit=unit_percent, age=['min', 'max'], age_unit=unit_usec): self.sz_bytes = [_damo_fmt_str.text_to_bytes(sz_bytes[0]), _damo_fmt_str.text_to_bytes(sz_bytes[1])] self.nr_acc_min_max = [ DamonNrAccesses(nr_accesses[0], nr_accesses_unit), DamonNrAccesses(nr_accesses[1], nr_accesses_unit)] self.nr_accesses_unit = nr_accesses_unit self.age_min_max = [ DamonAge(age[0], age_unit), DamonAge(age[1], age_unit)] self.age_unit = age_unit def to_str(self, raw): lines = [ 'sz: [%s, %s]' % (_damo_fmt_str.format_sz(self.sz_bytes[0], raw), _damo_fmt_str.format_sz(self.sz_bytes[1], raw)), ] lines.append('nr_accesses: [%s, %s]' % ( self.nr_acc_min_max[0].to_str(self.nr_accesses_unit, raw), self.nr_acc_min_max[1].to_str(self.nr_accesses_unit, raw))) lines.append('age: [%s, %s]' % ( self.age_min_max[0].to_str(self.age_unit, raw), self.age_min_max[1].to_str(self.age_unit, raw))) return '\n'.join(lines) def __str__(self): return self.to_str(False) def __eq__(self, other): return (type(self) == type(other) and self.sz_bytes == other.sz_bytes and self.nr_acc_min_max == other.nr_acc_min_max and self.age_min_max == other.age_min_max) @classmethod def from_kvpairs(cls, kv): sz_bytes = [_damo_fmt_str.text_to_bytes(kv['sz_bytes']['min']), _damo_fmt_str.text_to_bytes(kv['sz_bytes']['max'])] kv_ = kv['nr_accesses'] try: nr_accesses = [_damo_fmt_str.text_to_percent(kv_['min']), _damo_fmt_str.text_to_percent(kv_['max'])] nr_accesses_unit = unit_percent except: min_, nr_accesses_unit = _damo_fmt_str.text_to_nr_unit(kv_['min']) max_, nr_accesses_unit2 = _damo_fmt_str.text_to_nr_unit(kv_['max']) if nr_accesses_unit != nr_accesses_unit2: raise Exception('nr_accesses units should be same') nr_accesses = [min_, max_] kv_ = kv['age'] try: age = [_damo_fmt_str.text_to_us(kv_['min']), _damo_fmt_str.text_to_us(kv_['max'])] age_unit = unit_usec except: min_age, age_unit = _damo_fmt_str.text_to_nr_unit(kv_['min']) max_age, age_unit2 = _damo_fmt_str.text_to_nr_unit(kv_['max']) if age_unit != age_unit2: raise Exception('age units should be same') age = [min_age, max_age] return DamosAccessPattern(sz_bytes, nr_accesses, nr_accesses_unit, age, age_unit) def to_kvpairs(self, raw=False): min_nr_accesses = self.nr_acc_min_max[0].to_str( self.nr_accesses_unit, raw) max_nr_accesses = self.nr_acc_min_max[1].to_str( self.nr_accesses_unit, raw) min_age = self.age_min_max[0].to_str(self.age_unit, raw) max_age = self.age_min_max[1].to_str(self.age_unit, raw) return collections.OrderedDict([ ('sz_bytes', (collections.OrderedDict([ ('min', _damo_fmt_str.format_sz(self.sz_bytes[0], raw)), ('max', _damo_fmt_str.format_sz(self.sz_bytes[1], raw))]))), ('nr_accesses', (collections.OrderedDict([ ('min', min_nr_accesses), ('max', max_nr_accesses)]))), ('age', (collections.OrderedDict([ ('min', min_age), ('max', max_age)]))), ]) def convert_for_units(self, nr_accesses_unit, age_unit, intervals): self.nr_acc_min_max[0].add_unset_unit(intervals) self.nr_acc_min_max[1].add_unset_unit(intervals) self.age_min_max[0].add_unset_unit(intervals) self.age_min_max[1].add_unset_unit(intervals) self.nr_accesses_unit = nr_accesses_unit self.age_unit = age_unit def converted_for_units(self, nr_accesses_unit, age_unit, intervals): copied = copy.deepcopy(self) copied.convert_for_units(nr_accesses_unit, age_unit, intervals) return copied def effectively_equal(self, other, intervals): return ( self.converted_for_units( unit_samples, unit_aggr_intervals, intervals) == other.converted_for_units( unit_samples, unit_aggr_intervals, intervals)) class DamosQuotaGoal: metric = None target_value = None current_value = None quotas = None def __init__(self, metric='user_input', target_value='0', current_value='0'): self.metric = metric # todo: support time string for some_mem_psi_us self.target_value = _damo_fmt_str.text_to_nr(target_value) self.current_value = _damo_fmt_str.text_to_nr(current_value) def to_str(self, raw): return 'metric %s target %s current %s' % ( self.metric, _damo_fmt_str.format_nr(self.target_value, raw), _damo_fmt_str.format_nr(self.current_value, raw),) def __str__(self): return self.to_str(False) def __eq__(self, other): return (type(self) == type(other) and self.metric == other.metric and self.target_value == other.target_value and self.current_value == other.current_value) @classmethod def from_kvpairs(cls, kv): if 'target_value_bp' in kv: # For supporing old version of bad naming. Should deprecate later. return DamosQuotaGoal(target_value=kv['target_value_bp'], current_value=kv['current_value_bp']) return DamosQuotaGoal( metric=kv['metric'], target_value=kv['target_value'], current_value=kv['current_value']) def to_kvpairs(self, raw=False): return collections.OrderedDict([ ('metric', self.metric), ('target_value', _damo_fmt_str.format_nr(self.target_value, raw)), ('current_value', _damo_fmt_str.format_nr(self.current_value, raw))]) class DamosQuotas: time_ms = None sz_bytes = None reset_interval_ms = None weight_sz_permil = None weight_nr_accesses_permil = None weight_age_permil = None goals = None effective_sz_bytes = None scheme = None def __init__(self, time_ms=0, sz_bytes=0, reset_interval_ms='max', weights=['0 %', '0 %', '0 %'], goals=[], effective_sz_bytes=0): self.time_ms = _damo_fmt_str.text_to_ms(time_ms) self.sz_bytes = _damo_fmt_str.text_to_bytes(sz_bytes) self.reset_interval_ms = _damo_fmt_str.text_to_ms(reset_interval_ms) self.weight_sz_permil = _damo_fmt_str.text_to_permil(weights[0]) self.weight_nr_accesses_permil = _damo_fmt_str.text_to_permil( weights[1]) self.weight_age_permil = _damo_fmt_str.text_to_permil(weights[2]) self.goals = goals self.effective_sz_bytes = _damo_fmt_str.text_to_bytes( effective_sz_bytes) for goal in self.goals: goal.quotas = self def __str__(self): return self.to_str(False) def __eq__(self, other): return (type(self) == type(other) and self.time_ms == other.time_ms and self.sz_bytes == other.sz_bytes and self.reset_interval_ms == other.reset_interval_ms and self.weight_sz_permil == other.weight_sz_permil and self.weight_nr_accesses_permil == other.weight_nr_accesses_permil and self.weight_age_permil == other.weight_age_permil and self.goals == other.goals) @classmethod def from_kvpairs(cls, kv): if 'goals' in kv: goals = [DamosQuotaGoal.from_kvpairs(goal) for goal in kv['goals']] else: goals = [] return DamosQuotas(kv['time_ms'], kv['sz_bytes'], kv['reset_interval_ms'], [kv['weights']['sz_permil'], kv['weights']['nr_accesses_permil'], kv['weights']['age_permil'],], goals, kv['effective_sz_bytes'] if 'effective_sz_bytes' in kv else 0) def to_str(self, raw): lines = [ '%s / %s / %s per %s' % ( _damo_fmt_str.format_time_ns(self.time_ms * 1000000, raw), _damo_fmt_str.format_sz(self.sz_bytes, raw), _damo_fmt_str.format_sz(self.effective_sz_bytes, raw), _damo_fmt_str.format_time_ms(self.reset_interval_ms, raw))] for idx, goal in enumerate(self.goals): lines.append('goal %d: %s' % (idx, goal.to_str(raw))) lines.append( 'priority: sz %s, nr_accesses %s, age %s' % ( _damo_fmt_str.format_permil(self.weight_sz_permil, raw), _damo_fmt_str.format_permil( self.weight_nr_accesses_permil, raw), _damo_fmt_str.format_permil(self.weight_age_permil, raw))) return '\n'.join(lines) def to_kvpairs(self, raw=False): return collections.OrderedDict([ ('time_ms', _damo_fmt_str.format_time_ms_exact(self.time_ms, raw)), ('sz_bytes', _damo_fmt_str.format_sz(self.sz_bytes, raw)), ('reset_interval_ms', _damo_fmt_str.format_time_ms_exact( self.reset_interval_ms, raw)), ('goals', [goal.to_kvpairs(raw) for goal in self.goals]), ('effective_sz_bytes', _damo_fmt_str.format_sz(self.effective_sz_bytes, raw)), ('weights', (collections.OrderedDict([ ('sz_permil', _damo_fmt_str.format_permil(self.weight_sz_permil, raw)), ('nr_accesses_permil', _damo_fmt_str.format_permil( self.weight_nr_accesses_permil, raw)), ('age_permil', _damo_fmt_str.format_permil(self.weight_age_permil, raw))]) ))]) damos_wmarks_metric_none = 'none' damos_wmarks_metric_free_mem_rate = 'free_mem_rate' class DamosWatermarks: metric = None interval_us = None high_permil = None mid_permil = None low_permil = None # no limit by default def __init__(self, metric=damos_wmarks_metric_none, interval_us=0, high='0 %', mid='0 %', low='0 %'): # 'none' or 'free_mem_rate' if not metric in [damos_wmarks_metric_none, damos_wmarks_metric_free_mem_rate]: raise Exception('wrong watermark metric (%s)' % metric) self.metric = metric self.interval_us = _damo_fmt_str.text_to_us(interval_us) self.high_permil = _damo_fmt_str.text_to_permil(high) self.mid_permil = _damo_fmt_str.text_to_permil(mid) self.low_permil = _damo_fmt_str.text_to_permil(low) def to_str(self, raw): return '\n'.join([ 'metric %s, interval %s' % (self.metric, _damo_fmt_str.format_time_us(self.interval_us, raw)), '%s, %s, %s' % ( _damo_fmt_str.format_permil(self.high_permil, raw), _damo_fmt_str.format_permil(self.mid_permil, raw), _damo_fmt_str.format_permil(self.low_permil, raw)), ]) def __str__(self): return self.to_str(False) def __eq__(self, other): return (type(self) == type(other) and self.metric == other.metric and self.interval_us == other.interval_us and self.high_permil == other.high_permil and self.mid_permil == other.mid_permil and self.low_permil == other.low_permil) @classmethod def from_kvpairs(cls, kv): return DamosWatermarks(*[kv[x] for x in ['metric', 'interval_us', 'high_permil', 'mid_permil', 'low_permil']]) def to_kvpairs(self, raw=False): return collections.OrderedDict([ ('metric', self.metric), ('interval_us', _damo_fmt_str.format_time_us_exact( self.interval_us, raw)), ('high_permil', _damo_fmt_str.format_permil(self.high_permil, raw)), ('mid_permil', _damo_fmt_str.format_permil(self.mid_permil, raw)), ('low_permil', _damo_fmt_str.format_permil(self.low_permil, raw)), ]) class DamosFilter: filter_type = None # anon, memcg, addr, or target matching = None memcg_path = None address_range = None # DamonRegion damon_target_idx = None scheme = None def __init__(self, filter_type, matching, memcg_path=None, address_range=None, damon_target_idx=None): self.filter_type = filter_type self.matching = _damo_fmt_str.text_to_bool(matching) self.memcg_path = memcg_path self.address_range = address_range if damon_target_idx != None: self.damon_target_idx = _damo_fmt_str.text_to_nr(damon_target_idx) def to_str(self, raw): txt = '%s %s' % (self.filter_type, 'matching' if self.matching else 'nomatching') if self.filter_type == 'anon': return txt if self.filter_type == 'memcg': return '%s %s' % (txt, self.memcg_path) if self.filter_type == 'addr': return '%s %s' % (txt, self.address_range.to_str(raw)) if self.filter_type == 'target': return '%s %s' % (txt, _damo_fmt_str.format_nr( self.damon_target_idx, raw)) def __str__(self): return self.to_str(False) def __eq__(self, other): return type(self) == type(other) and '%s' % self == '%s' % other @classmethod def from_kvpairs(cls, kv): return DamosFilter(kv['filter_type'], kv['matching'], kv['memcg_path'] if kv['filter_type'] == 'memcg' else '', DamonRegion.from_kvpairs(kv['address_range']) if kv['filter_type'] == 'addr' else None, kv['damon_target_idx'] if kv['filter_type'] == 'target' else None) def to_kvpairs(self, raw=False): return collections.OrderedDict([ ('filter_type', self.filter_type), ('matching', self.matching), ('memcg_path', self.memcg_path), ('address_range', self.address_range.to_kvpairs(raw) if self.address_range != None else None), ('damon_target_idx', _damo_fmt_str.format_nr(self.damon_target_idx, raw) if self.damon_target_idx != None else None)]) class DamosStats: nr_tried = None sz_tried = None nr_applied = None sz_applied = None qt_exceeds = None def __init__(self, nr_tried=0, sz_tried=0, nr_applied=0, sz_applied=0, qt_exceeds=0): self.nr_tried = nr_tried self.sz_tried = sz_tried self.nr_applied = nr_applied self.sz_applied = sz_applied self.qt_exceeds = qt_exceeds def to_str(self, raw): return '\n'.join([ 'tried %d times (%s)' % (self.nr_tried, _damo_fmt_str.format_sz(self.sz_tried, raw)), 'applied %d times (%s)' % (self.nr_applied, _damo_fmt_str.format_sz(self.sz_applied, raw)), 'quota exceeded %d times' % self.qt_exceeds, ]) def __str__(self): return self.to_str(False) def to_kvpairs(self, raw=False): kv = collections.OrderedDict() kv['nr_tried'] = _damo_fmt_str.format_nr(self.nr_tried, raw) kv['sz_tried'] = _damo_fmt_str.format_sz(self.sz_tried, raw) kv['nr_applied'] = _damo_fmt_str.format_nr(self.nr_applied, raw) kv['sz_applied'] = _damo_fmt_str.format_sz(self.sz_applied, raw) kv['qt_exceeds'] = _damo_fmt_str.format_nr(self.qt_exceeds, raw) return kv # TODO: check support of pageout and lru_(de)prio damos_actions = [ 'willneed', 'cold', 'pageout', 'hugepage', 'nohugepage', 'lru_prio', 'lru_deprio', 'stat', ] damos_action_willneed = damos_actions[0] damos_action_cold = damos_actions[1] damos_action_pageout = damos_actions[2] damos_action_hugepage = damos_actions[3] damos_action_nohugepage = damos_actions[4] damos_action_lru_prio = damos_actions[5] damos_action_lru_deprio = damos_actions[6] damos_action_stat = damos_actions[7] class Damos: access_pattern = None action = None apply_interval_us = None quotas = None watermarks = None filters = None stats = None tried_regions = None tried_bytes = None context = None # for monitoring only by default def __init__(self, access_pattern=None, action=damos_action_stat, apply_interval_us=None, quotas=None, watermarks=None, filters=None, stats=None, tried_regions=None, tried_bytes=None): self.access_pattern = (access_pattern if access_pattern != None else DamosAccessPattern()) if not action in damos_actions: raise Exception('wrong damos action: %s' % action) self.action = action if apply_interval_us != None: self.apply_interval_us = _damo_fmt_str.text_to_us( apply_interval_us) else: self.apply_interval_us = 0 self.quotas = quotas if quotas != None else DamosQuotas() self.quotas.scheme = self self.watermarks = (watermarks if watermarks != None else DamosWatermarks()) self.filters = filters if filters != None else [] for filter_ in self.filters: filter_.scheme = self self.stats = stats if stats != None else DamosStats() self.tried_regions = tried_regions if tried_regions != None else [] for tried_region in self.tried_regions: tried_region.scheme = self self.tried_bytes = 0 if tried_bytes: self.tried_bytes = _damo_fmt_str.text_to_bytes( tried_bytes) else: for region in self.tried_regions: self.tried_bytes += region.size() def to_str(self, raw): lines = ['action: %s per %s' % (self.action, _damo_fmt_str.format_time_us(self.apply_interval_us, raw) if self.apply_interval_us != 0 else 'aggr interval')] lines.append('target access pattern') lines.append(_damo_fmt_str.indent_lines( self.access_pattern.to_str(raw), 4)) lines.append('quotas') lines.append(_damo_fmt_str.indent_lines(self.quotas.to_str(raw), 4)) lines.append('watermarks') lines.append(_damo_fmt_str.indent_lines( self.watermarks.to_str(raw), 4)) for idx, damos_filter in enumerate(self.filters): lines.append('filter %d' % idx) lines.append(_damo_fmt_str.indent_lines( damos_filter.to_str(raw), 4)) if self.stats != None: lines.append('statistics') lines.append(_damo_fmt_str.indent_lines(self.stats.to_str(raw), 4)) if self.tried_regions != None: lines.append('tried regions (%s)' % _damo_fmt_str.format_sz( self.tried_bytes, raw)) for region in self.tried_regions: lines.append(_damo_fmt_str.indent_lines(region.to_str(raw), 4)) return '\n'.join(lines) def __str__(self): return self.to_str(False) def __repr__(self): return self.__str__() def __eq__(self, other): return (type(self) == type(other) and self.access_pattern == other.access_pattern and self.action == other.action and self.apply_interval_us == other.apply_interval_us and self.quotas == other.quotas and self.watermarks == other.watermarks and self.filters == other.filters) @classmethod def from_kvpairs(cls, kv): filters = [] if 'filters' in kv: for damos_filter_kv in kv['filters']: filters.append(DamosFilter.from_kvpairs(damos_filter_kv)) return Damos(DamosAccessPattern.from_kvpairs(kv['access_pattern']) if 'access_pattern' in kv else DamosAccessPattern(), kv['action'] if 'action' in kv else damos_action_stat, kv['apply_interval_us'] if 'apply_interval_us' in kv else None, DamosQuotas.from_kvpairs(kv['quotas']) if 'quotas' in kv else DamosQuotas(), DamosWatermarks.from_kvpairs(kv['watermarks']) if 'watermarks' in kv else DamosWatermarks(), filters, None, None) def to_kvpairs(self, raw=False): kv = collections.OrderedDict() kv['action'] = self.action kv['access_pattern'] = self.access_pattern.to_kvpairs(raw) kv['apply_interval_us'] = self.apply_interval_us kv['quotas'] = self.quotas.to_kvpairs(raw) kv['watermarks'] = self.watermarks.to_kvpairs(raw) filters = [] for damos_filter in self.filters: filters.append(damos_filter.to_kvpairs(raw)) kv['filters'] = filters if self.stats != None: kv['stats'] = self.stats.to_kvpairs(raw) return kv def effectively_equal(self, other, intervals): return (type(self) == type(other) and self.access_pattern.effectively_equal( other.access_pattern, intervals) and self.action == other.action and self.apply_interval_us == other.apply_interval_us and self.quotas == other.quotas and self.watermarks == other.watermarks and self.filters == other.filters) class DamonCtx: ops = None targets = None intervals = None nr_regions = None schemes = None kdamond = None def __init__(self, ops, targets, intervals, nr_regions, schemes): self.ops = ops self.targets = targets for target in self.targets: target.context = self self.intervals = intervals self.nr_regions = nr_regions self.schemes = schemes for scheme in self.schemes: scheme.context = self def to_str(self, raw): lines = ['ops: %s' % self.ops] for idx, target in enumerate(self.targets): lines.append('target %d' % idx) lines.append(_damo_fmt_str.indent_lines(target.to_str(raw), 4)) lines.append('intervals: %s' % self.intervals.to_str(raw)) lines.append('nr_regions: %s' % self.nr_regions.to_str(raw)) for idx, scheme in enumerate(self.schemes): lines.append('scheme %d' % idx) lines.append(_damo_fmt_str.indent_lines(scheme.to_str(raw), 4)) return '\n'.join(lines) def __str__(self): return self.to_str(False) def __eq__(self, other): return type(self) == type(other) and '%s' % self == '%s' % other def __hash__(self): return hash(self.__str__()) @classmethod def from_kvpairs(cls, kv): ctx = DamonCtx( kv['ops'], [DamonTarget.from_kvpairs(t) for t in kv['targets']], DamonIntervals.from_kvpairs(kv['intervals']) if 'intervals' in kv else DamonIntervals(), DamonNrRegionsRange.from_kvpairs(kv['nr_regions']) if 'nr_regions' in kv else DAmonNrRegionsRange(), [Damos.from_kvpairs(s) for s in kv['schemes']] if 'schemes' in kv else []) return ctx def to_kvpairs(self, raw=False): kv = collections.OrderedDict({}) kv['ops'] = self.ops kv['targets'] = [t.to_kvpairs(raw) for t in self.targets] kv['intervals'] = self.intervals.to_kvpairs(raw) kv['nr_regions'] = self.nr_regions.to_kvpairs(raw) kv['schemes'] = [s.to_kvpairs(raw) for s in self.schemes] return kv def target_has_pid(ops): return ops in ['vaddr', 'fvaddr'] class Kdamond: state = None pid = None contexts = None def __init__(self, state, pid, contexts): self.state = state self.pid = pid self.contexts = contexts for ctx in self.contexts: ctx.kdamond = self def summary_str(self): return 'state: %s, pid: %s' % (self.state, self.pid) def to_str(self, raw): lines = [self.summary_str()] for idx, ctx in enumerate(self.contexts): lines.append('context %d' % idx) lines.append(_damo_fmt_str.indent_lines(ctx.to_str(raw), 4)) return '\n'.join(lines) def __str__(self): return self.to_str(False) def __eq__(self, other): return type(self) == type(other) and '%s' % self == '%s' % other def __hash__(self): return hash(self.__str__()) @classmethod def from_kvpairs(cls, kv): return Kdamond( kv['state'] if 'state' in kv else 'off', kv['pid'] if 'pid' in kv else None, [DamonCtx.from_kvpairs(c) for c in kv['contexts']]) def to_kvpairs(self, raw=False): kv = collections.OrderedDict() kv['state'] = self.state kv['pid'] = self.pid kv['contexts'] = [c.to_kvpairs(raw) for c in self.contexts] return kv import _damo_fs import _damon_dbgfs import _damon_sysfs import damo_version # System check # damo supports all DAMON-enabled kernels. For that, damo maintains list of # the DAMON features, and a dict saying whether the feature is supported on the # running kernel. Since the supports depend on underlying DAMON interface, the # dict is populated by _damon_fs, and saved as _damon_fs.feature_supports. # # The feature_supports population cannot be fully done while DAMON is running, # particularly in case of debugfs. Sysfs is ok for now, but similar issue # could happen in future. For the reason, the feature_supports are saved at # feature_supports_file_path file. damo commands can ask # _damon.ensure_initialized() to load/save the file. features = ['record', # was in DAMON patchset, but not merged in mainline 'vaddr', # merged in v5.15, thebeginning 'schemes', # merged in v5.16 'init_regions', # merged in v5.16 (90bebce9fcd6) 'paddr', # merged in v5.16 (a28397beb55b) 'schemes_speed_limit', # merged in v5.16 (2b8a248d5873) 'schemes_quotas', # merged in v5.16 (1cd243030059) 'schemes_prioritization', # merged in v5.16 (38683e003153) 'schemes_wmarks', # merged in v5.16 (ee801b7dd782) 'schemes_stat_succ', # merged in v5.17 (0e92c2ee9f45) 'schemes_stat_qt_exceed', # merged in v5.17 (0e92c2ee9f45) 'init_regions_target_idx', # merged in v5.18 (144760f8e0c3) 'fvaddr', # merged in v5.19 (b82434471cd2) 'schemes_tried_regions', # merged in v6.2-rc1 'schemes_filters', # merged in v6.3-rc1 'schemes_filters_anon', # merged in v6.3-rc1 'schemes_filters_memcg', # merged in v6.3-rc1 'schemes_tried_regions_sz', # merged in v6.6-rc1 'schemes_filters_addr', # merged in v6.6-rc1 'schemes_filters_target', # merged in v6.6-rc1 'schemes_apply_interval', # merged in v6.7-rc1 'schemes_quota_goals', # merged in v6.8-rc1 'schemes_quota_effective_bytes', # in development 'schemes_quota_goal_metric', # in development 'schemes_quota_goal_some_psi', # in development ] _damon_fs = None pr_debug_log = False def ensure_root_permission(): if os.geteuid() != 0: print('Run as root') exit(1) feature_supports_file_path = os.path.join(os.environ['HOME'], '.damo.damon_feature_supports') feature_support_file_format_ver = 1 def read_feature_supports_file(): '''Return error string''' if not os.path.isfile(feature_supports_file_path): return '%s not exist' % feature_supports_file_path try: with open(feature_supports_file_path, 'r') as f: feature_supports = json.load(f) except Exception as e: return 'reading feature supports failed (%s)' % e if not 'file_format_ver' in feature_supports: # The initial format set_feature_supports(feature_supports) return None file_format_ver = feature_supports['file_format_ver'] # support only the init version and current version for now. if file_format_ver != feature_support_file_format_ver: return 'unsupported format version %s' % file_format_ver if not damon_interface() in feature_supports: return 'no feature_supports for %s interface saved' % damon_interface() set_feature_supports(feature_supports[damon_interface()]) def write_feature_supports_file(): '''Return error string''' feature_supports, err = get_feature_supports() if err != None: return 'get_feature_supports() failed (%s)' % err to_save = {} if os.path.isfile(feature_supports_file_path): with open(feature_supports_file_path, 'r') as f: try: to_save = json.load(f) except: # Maybe previous writing was something wrong. Just overwirte. to_save = {} if not 'file_format_ver' in to_save: to_save = {} elif to_save['file_format_ver'] != feature_support_file_format_ver: to_save = {} if to_save == {}: to_save['file_format_ver'] = feature_support_file_format_ver to_save[damon_interface()] = feature_supports with open(feature_supports_file_path, 'w') as f: json.dump(to_save, f, indent=4, sort_keys=True) def feature_supported(feature): return _damon_fs.feature_supported(feature) def get_feature_supports(): err = _damon_fs.update_supported_features() if err != None: return None, err return _damon_fs.feature_supports, None def set_feature_supports(feature_supports): _damon_fs.feature_supports = feature_supports def initialize(damon_interface, debug_damon, save_feature_supports, load_feature_supports): global _damon_fs if damon_interface == 'sysfs': _damon_fs = _damon_sysfs elif damon_interface == 'debugfs': _damon_fs = _damon_dbgfs elif damon_interface == 'auto': if _damon_sysfs.supported(): _damon_fs = _damon_sysfs else: _damon_fs = _damon_dbgfs if not _damon_fs.supported(): return 'DAMON interface (%s) not supported' % damon_interface global pr_debug_log if debug_damon: pr_debug_log = True if load_feature_supports: err = read_feature_supports_file() if err: return err if save_feature_supports: write_feature_supports_file() return None initialized = False def ensure_initialized(args, save_feature_supports, load_feature_supports): global initialized if initialized: return err = initialize(args.damon_interface_DEPRECATED, args.debug_damon, save_feature_supports, load_feature_supports) if err != None: print(err) exit(1) initialized = True def ensure_root_and_initialized(args, save_feature_supports=False, load_feature_supports=False): ensure_root_permission() ensure_initialized(args, save_feature_supports, load_feature_supports) def damon_interface(): if _damon_fs == _damon_sysfs: return 'sysfs' elif _damon_fs == _damon_dbgfs: return 'debugfs' raise Exception('_damo_fs is neither _damon_sysfs nor _damon_dbgfs') # DAMON control def stage_kdamonds(kdamonds): return _damon_fs.stage_kdamonds(kdamonds) def commit_staged(kdamond_idxs): if _damon_fs == _damon_dbgfs: return 'debugfs interface does not support commit_staged()' return _damon_fs.commit_staged(kdamond_idxs) def commit_quota_goals(kdamond_idxs): if _damon_fs == _damon_dbgfs: return 'debugfs interface does not support commit_quota_goals()' return _damon_fs.commit_quota_goals(kdamond_idxs) def commit(kdamonds, commit_quota_goals_only=False): if not commit_quota_goals_only: err = stage_kdamonds(kdamonds) if err: return 'staging updates failed (%s)' % err kdamond_idxs = ['%s' % idx for idx, k in enumerate(kdamonds)] if commit_quota_goals_only: err = commit_quota_goals(kdamond_idxs) if err: return 'commit quotas failed (%s)' % err return None err = commit_staged(kdamond_idxs) if err: return 'commit staged updates filed (%s)' % err return None def update_schemes_stats(kdamond_idxs=None): if kdamond_idxs == None: kdamond_idxs = running_kdamond_idxs() return _damon_fs.update_schemes_stats(kdamond_idxs) def update_schemes_tried_bytes(kdamond_idxs=None): if kdamond_idxs == None: kdamond_idxs = running_kdamond_idxs() return _damon_fs.update_schemes_tried_bytes(kdamond_idxs) def update_schemes_tried_regions(kdamond_idxs=None): if kdamond_idxs == None: kdamond_idxs = running_kdamond_idxs() return _damon_fs.update_schemes_tried_regions(kdamond_idxs) def update_schemes_quota_effective_bytes(kdamond_idxs=None): if kdamond_idxs == None: kdamond_idxs = running_kdamond_idxs() return _damon_fs.update_schemes_quota_effective_bytes(kdamond_idxs) def update_schemes_status(stats=True, tried_regions=True, quota_effective_bytes=False): '''Returns error string or None''' idxs = running_kdamond_idxs() if len(idxs) == 0: return None if stats: err = update_schemes_stats(idxs) if err != None: return err if tried_regions and feature_supported('schemes_tried_regions'): err = update_schemes_tried_regions(idxs) if err != None: return err if quota_effective_bytes and feature_supported('schemes_quota_effective_bytes'): return update_schemes_quota_effective_bytes(idxs) return None def turn_damon_on(kdamonds_idxs): err = _damon_fs.turn_damon_on(kdamonds_idxs) if err: return err wait_kdamonds_turned_on() def turn_damon_off(kdamonds_idxs): err = _damon_fs.turn_damon_off(kdamonds_idxs) if err: return err wait_kdamonds_turned_off() # DAMON status reading def is_kdamond_running(kdamond_idx): return _damon_fs.is_kdamond_running(kdamond_idx) def current_kdamonds(): return _damon_fs.current_kdamonds() def update_read_kdamonds(nr_retries=0, update_stats=True, update_tried_regions=True, update_quota_effective_bytes=False): err = 'assumed error' nr_tries = 0 while True: err = update_schemes_status(update_stats, update_tried_regions, update_quota_effective_bytes) nr_tries += 1 if err == None or nr_tries > nr_retries: break time.sleep(random.randrange(2**(nr_tries - 1), 2**nr_tries) / 100) if err: return None, err return current_kdamonds(), None def nr_kdamonds(): return _damon_fs.nr_kdamonds() def running_kdamond_idxs(): return [idx for idx in range(nr_kdamonds()) if is_kdamond_running(idx)] def any_kdamond_running(): for idx in range(nr_kdamonds()): if is_kdamond_running(idx): return True return False def wait_kdamonds_turned_on(): for idx in range(nr_kdamonds()): while not is_kdamond_running(idx): time.sleep(1) def wait_kdamonds_turned_off(): for idx in range(nr_kdamonds()): while is_kdamond_running(idx): time.sleep(1) damo-2.2.4/_damon_args.py000066400000000000000000000510441457141243000152670ustar00rootroot00000000000000# SPDX-License-Identifier: GPL-2.0 """ Command line arguments handling """ import argparse import json import os import subprocess import _damo_paddr_layout import _damon # Kdamonds construction from command line arguments def init_regions_for(args): init_regions = [] if args.regions: for region in args.regions.split(): addrs = region.split('-') try: if len(addrs) != 2: raise Exception ('two addresses not given') region = _damon.DamonRegion(addrs[0], addrs[1]) if region.start >= region.end: raise Exception('start >= end') if init_regions and init_regions[-1].end > region.start: raise Exception('regions overlap') except Exception as e: return None, 'Wrong \'--regions\' argument (%s)' % e init_regions.append(region) if args.ops == 'paddr' and not init_regions: if args.numa_node != None: init_regions, err = _damo_paddr_layout.paddr_region_of( args.numa_node) if err != None: return None, err else: init_regions = [_damo_paddr_layout.default_paddr_region()] try: init_regions = [_damon.DamonRegion(r[0], r[1]) for r in init_regions] except Exception as e: return None, 'Wrong \'--regions\' argument (%s)' % e return init_regions, None def damon_intervals_for(args): default_intervals = _damon.DamonIntervals() intervals1 = _damon.DamonIntervals(args.sample, args.aggr, args.updr) intervals2 = _damon.DamonIntervals(*args.monitoring_intervals) if not intervals1 == default_intervals: return intervals1 if not intervals2 == default_intervals: return intervals2 return default_intervals def damon_nr_regions_range_for(args): default_range = _damon.DamonNrRegionsRange() range1 = _damon.DamonNrRegionsRange(args.minr, args.maxr) range2 = _damon.DamonNrRegionsRange(*args.monitoring_nr_regions_range) if not range1 == default_range: return range1 if not range2 == default_range: return range2 return default_range def schemes_option_to_damos(schemes): if os.path.isfile(schemes): with open(schemes, 'r') as f: schemes = f.read() try: kvpairs = json.loads(schemes) return [_damon.Damos.from_kvpairs(kv) for kv in kvpairs], None except Exception as json_err: return None, '%s' % json_err def damos_options_to_filters(filters_args): filters = [] if filters_args == None: return filters, None for fields in filters_args: if len(fields) < 2: return None, '<2 filter field legth (%s)' % fields ftype = fields[0] fmatching = fields[1] fargs = fields[2:] if not fmatching in ['matching', 'nomatching']: return None, 'unsupported matching keyword (%s)' % fmatching fmatching = fmatching == 'matching' if ftype == 'anon': if len(fargs): return (None, 'anon filter receives no arguments but (%s)' % fargs) filters.append(_damon.DamosFilter(ftype, fmatching)) elif ftype == 'memcg': if len(fargs) != 1: return None, 'wrong number of memcg arguments (%s)' % fargs memcg_path = fargs[0] filters.append(_damon.DamosFilter(ftype, fmatching, memcg_path)) elif ftype == 'addr': if len(fargs) != 2: return None, 'wrong number of addr arguments (%s)' % fargs try: addr_range = _damon.DamonRegion(fargs[0], fargs[1]) except Exception as e: return None, 'wrong addr range (%s, %s)' % (fargs, e) filters.append( _damon.DamosFilter(ftype, fmatching, None, addr_range)) elif ftype == 'target': if len(fargs) != 1: return None, 'wrong number of target argument (%s)' % fargs try: filters.append( _damon.DamosFilter(ftype, fmatching, None, None, fargs[0])) except Exception as e: return None, 'target filter creation failed (%s, %s)' % ( fargs[0], e) else: return None, 'unsupported filter type' return filters, None def damos_quotas_cons_arg(cmd_args): time_ms = 0 sz_bytes = 0 reset_interval_ms = 'max' weights = ['0 %', '0 %', '0 %'] nr_cmd_args = len(cmd_args) if nr_cmd_args >= 1: time_ms = cmd_args[0] if nr_cmd_args >= 2: sz_bytes = cmd_args[1] if nr_cmd_args >= 3: reset_interval_ms = cmd_args[2] if nr_cmd_args >= 4: weights[0] = cmd_args[3] if nr_cmd_args >= 5: weights[1] = cmd_args[4] if nr_cmd_args >= 6: weights[2] = cmd_args[5] return [time_ms, sz_bytes, reset_interval_ms, weights] def damos_options_to_scheme(sz_region, access_rate, age, action, apply_interval, quotas, goals, wmarks, filters): if quotas != None: gargs = goals # garg should be [current value] # [current value] is given for only 'user_input' for garg in gargs: if not len(garg) in [2, 3]: return None, 'Wrong --damos_quota_goal (%s)' % garg if garg[0] == 'user_input' and len(garg) != 3: return None, 'Wrong --damos_quota_goal (%s)' % garg if garg[0] != 'user_input' and len(garg) != 2: return None, 'Wrong --damos_quota_goal (%s)' % garg try: goals = [_damon.DamosQuotaGoal(*garg) for garg in gargs] except Exception as e: return None, 'Wrong --damos_quota_goal (%s, %s)' % (gargs, e) qargs = quotas if len(qargs) > 6: return None, 'Wrong --damos_quotas (%s, >6 parameters)' % qargs try: quotas = _damon.DamosQuotas(*damos_quotas_cons_arg(qargs), goals=goals) except Exception as e: return None, 'Wrong --damos_quotas (%s, %s)' % (qargs, e) if wmarks != None: wargs = wmarks try: wmarks = _damon.DamosWatermarks(wargs[0], wargs[1], wargs[2], wargs[3], wargs[4]) except Exception as e: return None, 'Wrong --damos_wmarks (%s, %s)' % (wargs, e) filters, err = damos_options_to_filters(filters) if err != None: return None, err try: return _damon.Damos( access_pattern=_damon.DamosAccessPattern(sz_region, access_rate, _damon.unit_percent, age, _damon.unit_usec), action=action, apply_interval_us=apply_interval, quotas=quotas, watermarks=wmarks, filters=filters), None except Exception as e: return None, 'Wrong \'--damos_*\' argument (%s)' % e def damos_options_to_schemes(args): nr_schemes = len(args.damos_action) if len(args.damos_sz_region) > nr_schemes: return [], 'too much --damos_sz_region' if len(args.damos_access_rate) > nr_schemes: return [], 'too much --damos_access_rate' if len(args.damos_age) > nr_schemes: return [], 'too much --damos_age' if len(args.damos_apply_interval) > nr_schemes: return [], 'too much --damos_apply_interval' if len(args.damos_quotas) > nr_schemes: return [], 'too much --damos_quotas' if len(args.damos_quota_goal) > 0 and nr_schemes > 1: if len(args.damos_nr_quota_goals) == 0: return [], '--damos_nr_quota_goals required' if nr_schemes == 1 and args.damos_nr_quota_goals == []: args.damos_nr_quota_goals = [len(args.damos_quota_goal)] if sum(args.damos_nr_quota_goals) != len(args.damos_quota_goal): return [], 'wrong --damos_nr_quota_goals' if len(args.damos_wmarks) > nr_schemes: return [], 'too much --damos_wmarks' # for multiple schemes, number of filters per scheme is required if len(args.damos_filter) > 0 and nr_schemes > 1: if len(args.damos_nr_filters) == 0: return [], '--damos_nr_filters required' if nr_schemes == 1 and args.damos_nr_filters == []: args.damos_nr_filters = [len(args.damos_filter)] if sum(args.damos_nr_filters) != len(args.damos_filter): return [], 'wrong --damos_nr_filters' args.damos_sz_region += [['min', 'max']] * ( nr_schemes - len(args.damos_sz_region)) args.damos_access_rate += [['min', 'max']] * ( nr_schemes - len(args.damos_access_rate)) args.damos_age += [['min', 'max']] * (nr_schemes - len(args.damos_age)) args.damos_apply_interval += [0] * ( nr_schemes - len(args.damos_apply_interval)) args.damos_quotas += [None] * (nr_schemes - len(args.damos_quotas)) args.damos_wmarks += [None] * (nr_schemes - len(args.damos_wmarks)) schemes = [] for i in range(nr_schemes): qgoals = [] if args.damos_quota_goal: goal_start_index = sum(args.damos_nr_quota_goals[:i]) goal_end_index = goal_start_index + args.damos_nr_quota_goals[i] qgoals = args.damos_quota_goal[goal_start_index:goal_end_index] filters = [] if args.damos_filter: filter_start_index = sum(args.damos_nr_filters[:i]) filter_end_index = filter_start_index + args.damos_nr_filters[i] filters = args.damos_filter[filter_start_index:filter_end_index] scheme, err = damos_options_to_scheme(args.damos_sz_region[i], args.damos_access_rate[i], args.damos_age[i], args.damos_action[i], args.damos_apply_interval[i], args.damos_quotas[i], qgoals, args.damos_wmarks[i], filters) if err != None: return [], err schemes.append(scheme) return schemes, None def damos_for(args): if args.damos_action: schemes, err = damos_options_to_schemes(args) if err != None: return None, err return schemes, None if not 'schemes' in args or args.schemes == None: return [], None schemes, err = schemes_option_to_damos(args.schemes) if err: return None, 'failed damo schemes arguments parsing (%s)' % err return schemes, None def damon_ctx_for(args): try: intervals = damon_intervals_for(args) except Exception as e: return None, 'invalid intervals arguments (%s)' % e try: nr_regions = damon_nr_regions_range_for(args) except Exception as e: return None, 'invalid nr_regions arguments (%s)' % e ops = args.ops init_regions, err = init_regions_for(args) if err: return None, err try: target = _damon.DamonTarget(args.target_pid if _damon.target_has_pid(ops) else None, init_regions) except Exception as e: return 'Wrong \'--target_pid\' argument (%s)' % e schemes, err = damos_for(args) if err: return None, err try: ctx = _damon.DamonCtx(ops, [target], intervals, nr_regions, schemes) return ctx, None except Exception as e: return None, 'Creating context from arguments failed (%s)' % e def kdamonds_from_json_arg(arg): try: if os.path.isfile(arg): with open(arg, 'r') as f: kdamonds_str = f.read() else: kdamonds_str = arg kdamonds_kvpairs = json.loads(kdamonds_str)['kdamonds'] return [_damon.Kdamond.from_kvpairs(kvp) for kvp in kdamonds_kvpairs], None except Exception as e: return None, e target_type_explicit = 'explicit' target_type_cmd = 'cmd' target_type_pid = 'pid' target_type_unknown = None def deduced_target_type(target): if target in ['vaddr', 'paddr', 'fvaddr']: return target_type_explicit try: subprocess.check_output(['which', target.split()[0]]) return target_type_cmd except: pass try: pid = int(target) return target_type_pid except: pass return target_type_unknown def warn_option_override(option_name): print('warning: %s is overridden by ' % option_name) def deduce_target_update_args(args): args.self_started_target = False target_type = deduced_target_type(args.deducible_target) if target_type == target_type_unknown: return 'target \'%s\' is not supported' % args.deducible_target if target_type == target_type_explicit and args.deducible_target == 'paddr': if not args.ops in ['paddr', None]: warn_option_override('--ops') args.ops = 'paddr' if args.target_pid != None: warn_option_override('--target_pid') args.target_pid = None return None if target_type == target_type_cmd: p = subprocess.Popen(args.deducible_target, shell=True, executable='/bin/bash') pid = p.pid args.self_started_target = True elif target_type == target_type_pid: pid = int(args.deducible_target) if args.target_pid != None: print('warning: --target_pid will be ignored') args.target_pid = pid if not args.regions: if not args.ops in ['vaddr', None]: warn_option_override('--ops') args.ops = 'vaddr' if args.regions: if not args.ops in ['fvaddr', None]: print('warning: override --ops by and --regions') args.ops = 'fvaddr' def kdamonds_for(args): if args.kdamonds: return kdamonds_from_json_arg(args.kdamonds) if args.deducible_target: kdamonds, e = kdamonds_from_json_arg(args.deducible_target) if e == None: return kdamonds, e err = deduce_target_update_args(args) if err: return None, err if args.ops == None: if args.target_pid == None: args.ops = 'paddr' else: args.ops = 'vaddr' ctx, err = damon_ctx_for(args) if err: return None, err return [_damon.Kdamond(state=None, pid=None, contexts=[ctx])], None def self_started_target(args): return 'self_started_target' in args and args.self_started_target # Command line processing helpers def is_ongoing_target(args): return args.deducible_target == 'ongoing' def stage_kdamonds(args): kdamonds, err = kdamonds_for(args) if err: return None, 'cannot create kdamonds from args (%s)' % err err = _damon.stage_kdamonds(kdamonds) if err: return None, 'cannot apply kdamonds from args (%s)' % err return kdamonds, None def commit_kdamonds(args, commit_quota_goals_only): kdamonds, err = kdamonds_for(args) if err: return None, 'cannot create kdamonds to commit from args (%s)' % err err = _damon.commit(kdamonds, commit_quota_goals_only) if err: return None, 'cannot commit kdamonds (%s)' % err return kdamonds, None def turn_damon_on(args): kdamonds, err = stage_kdamonds(args) if err: return err, None return _damon.turn_damon_on( ['%s' % kidx for kidx, k in enumerate(kdamonds)]), kdamonds # Commandline options setup helpers def set_common_argparser(parser): parser.add_argument('--damon_interface_DEPRECATED', choices=['sysfs', 'debugfs', 'auto'], default='auto', help='underlying DAMON interface to use (!! DEPRECATED)') parser.add_argument('--debug_damon', action='store_true', help='Print debugging log') def set_monitoring_attrs_pinpoint_argparser(parser): # for easier pinpoint setup parser.add_argument('-s', '--sample', metavar='', default=5000, help='sampling interval (us)') parser.add_argument('-a', '--aggr', metavar='', default=100000, help='aggregate interval (us)') parser.add_argument('-u', '--updr', metavar='', default=1000000, help='regions update interval (us)') parser.add_argument('-n', '--minr', metavar='<# regions>', default=10, help='minimal number of regions') parser.add_argument('-m', '--maxr', metavar='<# regions>', default=1000, help='maximum number of regions') def set_monitoring_attrs_argparser(parser): # for easier total setup parser.add_argument('--monitoring_intervals', nargs=3, default=['5ms', '100ms', '1s'], metavar=('', '', ''), help='monitoring intervals (us)') parser.add_argument('--monitoring_nr_regions_range', nargs=2, metavar=('', ''), default=[10, 1000], help='min/max number of monitoring regions') def set_monitoring_argparser(parser): parser.add_argument('--ops', choices=['vaddr', 'paddr', 'fvaddr'], help='monitoring operations set') parser.add_argument('--target_pid', type=int, metavar='', help='pid of monitoring target process') parser.add_argument('-r', '--regions', metavar='"- ..."', type=str, default='', help='monitoring target address regions') parser.add_argument('--numa_node', metavar='', type=int, help='if target is \'paddr\', limit it to the numa node') set_monitoring_attrs_argparser(parser) def set_damos_argparser(parser): parser.add_argument('--damos_action', metavar='', choices=_damon.damos_actions, action='append', default=[], help='damos action to apply to the target regions') parser.add_argument('--damos_sz_region', metavar=('', ''), nargs=2, default=[], action='append', help='min/max size of damos target regions (bytes)') parser.add_argument('--damos_access_rate', metavar=('', ''), nargs=2, default=[], action='append', help='min/max access rate of damos target regions (percent)') parser.add_argument('--damos_age', metavar=('', ''), nargs=2, default=[], action='append', help='min/max age of damos target regions (microseconds)') parser.add_argument('--damos_apply_interval', metavar='', action='append', default=[], help='the apply interval for the scheme') parser.add_argument('--damos_quotas', default=[], metavar='', nargs='+', action='append', help=' '.join([ 'damos quotas (